面向局部时序异常的窗口引导音视频深度伪造检测方法

doi:10.3969/j.issn.1671-1122.2026.09.008

摘要/Abstract

摘要：

随着音视频生成技术的发展，深度伪造内容逐渐呈现出局部短时伪造和多模态协同生成的特点，少量伪造线索容易在长视频全局建模过程中被正常帧稀释。针对上述问题，文章提出一种面向局部时序异常的窗口引导音视频深度伪造检测方法。该方法基于时间对齐的音视频特征构造局部动态信息和跨模态差异信息，将连续序列组织为“帧-窗口-全局”的分层结构。模型在局部窗口内保留短时异常响应，通过跨窗口上下文建模建立窗口间联系，并利用全局语义反馈增强帧级表示，实现局部异常感知与视频级真伪判别。实验结果表明，该方法在FakeAVCeleb、DFDC和LAV-DF数据集上的库内检测AUC分别达到99.54%、99.90%和100%，跨库泛化和跨伪造类型实验检验了模型在不同数据分布和伪造方式下的检测性能。窗口结构消融实验验证了局部窗口、异常峰值提取和全局语义反馈的作用。视频边缘裁剪和后端效率实验表明，模型在边缘扰动条件下保持稳定，窗口级上下文建模能够降低长序列后端处理的计算量和显存占用；窗口引导分层聚合能够在保留局部短时伪造线索的同时，兼顾视频级判别与长序列后端计算开销。

关键词: 深度伪造检测, 音视频取证, 局部时序异常, 窗口引导聚合, 多粒度预测

Abstract:

With the development of audio-visual generation techniques, deepfake videos increasingly exhibit localized, short-duration, and multimodal manipulation patterns. Short-term forgery cues may be weakened by numerous normal frames during global temporal modeling. To address this issue, this paper proposed a window-guided audio-visual deepfake detection method for local temporal anomalies. Based on temporally aligned audio and visual features, our model constructed local dynamic information and cross-modal difference information, and organized the sequence into a frame-window-global hierarchy. The model preserved short-term anomalous responses within local windows, and enhances frame-level representations through cross-window contextual modeling and global semantic feedback, thereby balancing local anomaly perception and video-level authenticity prediction. Experimental results show that the proposed method achieves intra-dataset AUC scores of 99.54%, 99.90%, and 100% on FakeAVCeleb, DFDC, and LAV-DF, respectively. Cross-dataset and cross-manipulation experiments further evaluate the model under changes in data distribution and manipulation type. Ablation studies further verify the contributions of local-window modeling, anomaly-peak extraction, and global semantic feedback. Additional boundary-cropping and backend-efficiency experiments show that the model maintains stable performance under boundary perturbations, while window-level contextual modeling effectively reduces the computational and memory costs of backend long-sequence modeling. These results demonstrate that window-guided hierarchical aggregation preserves local short-term forgery cues while supporting video-level discrimination and efficient backend modeling of long sequences.

Key words: deepfake detection, audio-visual forensics, local temporal anomaly, window-guided aggregation, multi-granularity prediction

中图分类号:

TP309

张璐, 芦天亮. 面向局部时序异常的窗口引导音视频深度伪造检测方法[J]. 信息网络安全, 2026, 26(9): 1422-1434.

Zhang Lu, Lu Tianliang. Window-guided audio-visual deepfake detection for local temporal anomalies[J]. Netinfo Security, 2026, 26(9): 1422-1434.

图/表 10

图1

图2

表1

表2

表3

表4

表5

表6

表7

表8

参考文献 52

[1]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144. doi: 10.1145/3422622 URL
[2]	Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
[3]	Prajwal K R, Mukhopadhyay R, Namboodiri V P, et al. A lip sync expert is all you need for speech to lip generation in the wild[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 484-492.
[4]	Afchar D, Nozick V, Yamagishi J, et al. MesoNet: a compact facial video forgery detection network[C]// 2018 IEEE International Workshop on Information Forensics and Security (WIFS). New York: IEEE, 2018: 1-7.
[5]	Rossler A, Cozzolino D, Verdoliva L, et al. FaceForensics++: learning to detect manipulated facial images[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2019: 1-11.
[6]	Li Lingzhi, Bao Jianmin, Zhang Ting, et al. Face X-ray for more general face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2020: 5001-5010.
[7]	Qian Yuyang, Yin Guojun, Sheng Lu, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]// The European Conference on Computer Vision. Heidelberg: Springer, 2020: 86-103.
[8]	Chung J S, Zisserman A. Out of time: automated lip sync in the wild[C]// The Asian Conference on Computer Vision. Heidelberg: Springer, 2016: 251-263.
[9]	Chugh K, Gupta P, Dhall A, et al. Not made for each other-audio-visual dissonance-based deepfake detection and localization[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 439-447.
[10]	Zhou Yipin, Lim S N. Joint audio-visual deepfake detection[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2021: 14800-14809.
[11]	Haliassos A, Mira R, Petridis S, et al. Leveraging real talking faces via self-supervision for robust forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 14950-14962.
[12]	Yin Qilin, Lu Wei, Cao Xiaochun, et al. Fine-grained multimodal deepfake classification via heterogeneous graphs[J]. International Journal of Computer Vision, 2024, 132(11): 5255-5269. doi: 10.1007/s11263-024-02128-1
[13]	Bertasius G, Wang Heng, Torresani L. Is space-time attention all you need for video understanding?[C]// The 38th International Conference on Machine Learning. Vienna: PMLR, 2021, 139: 813-824.
[14]	Liu Ze, Ning Jia, Cao Yue, et al. Video swin transformer[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 3202-3211.
[15]	Liu Honggu, Li Xiaodan, Zhou Wenbo, et al. Spatial-Phase shallow learning: rethinking face forgery detection in frequency domain[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 772-781.
[16]	Cao Junyi, Ma Chao, Yao Taiping, et al. End-to-end reconstruction-classification learning for face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 4113-4122.
[17]	Shiohara K, Yamasaki T. Detecting deepfakes with self-blended images[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 18720-18729.
[18]	Miao Changtao, Tan Zichang, Chu Qi, et al. Hierarchical frequency-assisted interactive networks for face manipulation detection[J]. IEEE Transactions on Information Forensics and Security, 2022, 17: 3008-3021. doi: 10.1109/TIFS.2022.3198275 URL
[19]	Jung J, Heo H S, Kim J, et al. RawNet: advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification[C]// The 20th Annual Conference of the International Speech Communication Association (INTERSPEECH 2019). Baixas: ISCA, 2019: 1268-1272.
[20]	Jung J, Heo H S, Tak H, et al. Aasist: audio anti-spoofing using integrated spectro-temporal graph attention networks[C]// 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New York: IEEE, 2022: 6367-6371.
[21]	Chen Sanyuan, Wang Chengyi, Chen Zhengyang, et al. WavLM: large-scale self-supervised pre-training for full stack speech processing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6): 1505-1518. doi: 10.1109/JSTSP.2022.3188113 URL
[22]	Yang Wenyuan, Zhou Xiaoyu, Chen Zhikai, et al. AVoiD-DF: audio-visual joint learning for detecting deepfake[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 2015-2029. doi: 10.1109/TIFS.2023.3262148 URL
[23]	Yu Yang, Liu Xiaolong, Ni Rongrong, et al. PVASS-MDD: predictive visual-audio alignment self-supervision for multimodal deepfake detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 34(8): 6926-6936. doi: 10.1109/TCSVT.2023.3309899 URL
[24]	Oorloff T, Koppisetti S, Bonettini N, et al. AVFF: audio-visual feature fusion for video deepfake detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2024: 27102-27112.
[25]	Nie Fan, Ni Jiangqun, Zhang Jian, et al. FRADE: forgery-aware audio-distilled multimodal learning for deepfake detection[C]// The 32nd ACM International Conference on Multimedia. New York: ACM, 2024: 6297-6306.
[26]	Miao Hui, Guo Yuanfang, Liu Zeming, et al. Multi-modal deepfake detection via multi-task audio-visual prompt learning[C]// The AAAI Conference on Artificial Intelligence. Washington: AAAI, 2025: 612-621.
[27]	Nguyen D, Mejri N, Singh I P, et al. Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2024: 17395-17405.
[28]	Mittal T, Bhattacharya U, Chandra R, et al. Emotions don't lie: an audio-visual deepfake detection method using affective cues[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2823-2832.
[29]	Güera D, Delp E J. Deepfake video detection using recurrent neural networks[C]// The 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). New York: IEEE, 2018: 1-6.
[30]	Carreira J, Zisserman A. Quo vadis action recognition? A new model and the kinetics dataset[C]// The IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 6299-6308.
[31]	Zheng Yinglin, Bao Jianmin, Chen Dong, et al. Exploring temporal coherence for more general video face forgery detection[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2021: 15044-15054.
[32]	Gu Zhihao, Chen Yang, Yao Taiping, et al. Spatiotemporal inconsistency learning for deepfake video detection[C]// The 29th ACM International Conference on Multimedia. New York: ACM, 2021: 3473-3481.
[33]	Gu Zhihao, Chen Yang, Yao Taiping, et al. Delving into the local: dynamic inconsistency learning for deepfake video detection[C]// The 36th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2022: 744-752.
[34]	Wang Zhendong, Bao Jianmin, Zhou Wengang, et al. AltFreezing for more general video face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2023: 4129-4138.
[35]	Wang Junke, Wu Zuxuan, Ouyang Wenhao, et al. M2TR: multi-modal multi-scale transformers for deepfake detection[C]// The 2022 International Conference on Multimedia Retrieval. New York: ACM, 2022: 615-623.
[36]	Zhao Cairong, Wang Chutian, Hu Guosheng, et al. ISTVT: interpretable spatial-temporal video transformer for deepfake detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1335-1348. doi: 10.1109/TIFS.2023.3239223 URL
[37]	Lin T-Y, Goyal P, Girshick R, et al. Focal loss for dense object detection[C]// The IEEE International Conference on Computer Vision. New York: IEEE, 2017: 2980-2988.
[38]	Khalid H, Tariq S, Kim M, et al. FakeAVCeleb: a novel audio-video multimodal deepfake dataset[EB/OL]. (2022-03-01)[2026-05-14]. https://arxiv.org/abs/2108.05080.
[39]	Dolhansky B, Bitton J, Pflaum B, et al. The deepfake detection challenge (DFDC) dataset[EB/OL]. (2020-10-28)[2026-05-14]. https://arxiv.org/abs/2006.07397.
[40]	Cai Zhixi, Ghosh S, Dhall A, et al. Glitch in the matrix: a large scale benchmark for content driven audio-visual forgery detection and localization[J]. Computer Vision and Image Understanding, 2023, 236: 103818. doi: 10.1016/j.cviu.2023.103818 URL
[41]	Wang Jian, Wu Baoyuan, Liu Li, et al. FauForensics: boosting audio-visual deepfake detection with facial action units[J]. IEEE Transactions on Information Forensics and Security, 2026, 21: 3353-3365. doi: 10.1109/TIFS.2026.3674401 URL
[42]	Smeu S, Boldisor D A, Oneata D, et al. Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2025: 18815-18825.
[43]	Peng Jielun, Wang Yabin, Li Yaqi, et al. Leave no stone unturned: uncovering holistic audio-visual intrinsic coherence for deepfake detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings. New York: IEEE, 2026: 6655-6666.
[44]	Cai Zhixi, Stefanov K, Dhall A, et al. Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization[C]// 2022 International Conference on Digital Image Computing:Techniques and Applications (DICTA). New York: IEEE, 2022: 1-10.
[45]	Koutlis C, Papadopoulos S. DiMoDif: discourse modality-information differentiation for audio-visual deepfake detection and localization[EB/OL]. (2025-04-11)[2026-05-14]. https://arxiv.org/abs/2411.10193.
[46]	Anshul A, Gopal S, Rajan D, et al. Intra-modal and cross-modal synchronization for audio-visual deepfake detection and temporal localization[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2025: 13826-13836.
[47]	Datta S K, Jia Shan, Lyu Siwei. LoCC: detection and localization of lip-syncing deepfakes via counterfactual frame consistency[EB/OL]. (2026-05-22)[2026-05-14]. https://arxiv.org/abs/2606.22772.
[48]	Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer[EB/OL]. (2021-03-11)[2026-05-14]. https://arxiv.org/abs/2102.11126.
[49]	Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer[EB/OL]. arXiv: 2102.11126, 2021.
[50]	Haliassos A, Vougioukas K, Petridis S, et al. Lips Don't Lie: a generalisable and robust approach to face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 5039-5049.
[51]	Wang Rui, Ye Dengpan, Tang Long, et al. AVT²-DWF: improving deepfake detection with audio-visual fusion and dynamic weighting strategies[J]. IEEE Signal Processing Letters, 2024, 31: 1960-1964. doi: 10.1109/LSP.2024.3433596 URL
[52]	Anshul A, Gopal S, Rajan D, et al. Next-frame feature prediction for multimodal deepfake detection and temporal localization[EB/OL]. (2025-11-13)[2026-05-14]. https://arxiv.org/abs/2511.10212.

方法	发表出处	年份	FakeAVCeleb	DFDC	LAV-DF
文献[9]方法	ACM MM	2020	86.50%	91.50%	97.83%
文献[22]方法	TIFS	2023	89.20%	94.80%	—
文献[23]方法	TCSVT	2023	96.50%	98.90%	—
文献[26]方法	AAAI	2025	99.98%	—	—
文献[41]方法	TIFS	2026	99.91%	—	99.97%
文献[42]方法	CVPR	2025	99.20%	—	—
文献[43]方法	CVPR	2026	99.65%	—	—
本文方法	—	—	99.54%	99.90%	100%

方法	年份	AP@0.50	AP@0.75	AP@0.95
文献[9]方法	2020	12.78%	1.62%	0
文献[44]方法	2022	76.90%	38.50%	0.25%
文献[45]方法	2024	95.50%	87.90%	20.60%
文献[46]方法	2025	87.40%	66.80%	5.72%
文献[47]方法	2026	91.00%	74.07%	2.46%
本文方法	—	89.55%	77.06%	23.47%

训练集→测试集	方法	年份	AUC
FakeAVCeleb→ LAV-DF	文献[12]方法	2024	63.80%
	文献[48]方法	2024	95.80%
	本文方法	—	95.69%
LAV-DF→ FakeAVCeleb	文献[12]方法	2024	65.52%
	文献[41]方法	2026	83.77%
	本文方法	—	92.64%
FakeAVCeleb→ DFDC	文献[5]方法	2019	50.00%
	文献[9]方法	2020	63.50%
	文献[49]方法	2021	51.10%
	文献[50]方法	2021	49.00%
	文献[51]方法	2024	74.60%
	本文方法	—	76.50%
DFDC→ FakeAVCeleb	文献[5]方法	2019	61.43%
	文献[9]方法	2020	47.62%
	文献[49]方法	2021	57.57%
	文献[50]方法	2021	56.14%
	文献[51]方法	2024	77.20%
	本文方法	—	68.68%

方法	年份	模态	RVFA	FVRA-WL	FVFA-WL	FVFA-GAN	FVFA-FS	AVG-FV
文献[5]方法	2019	V	—	88.20%	91.00%	69.60%	92.30%	84.80%
文献[11]方法	2022	V	—	88.80%	93.40%	99.80%	99.30%	95.30%
文献[43]方法	2026	AV	96.70%	100%	100%	100%	100%	100%
文献[50]方法	2021	V	—	97.80%	98.60%	64.50%	99.90%	89.40%
文献[52]方法	2025	AV	95.90%	88.60%	97.60%	97.80%	98.40%	95.60%
本文方法	—	AV	96.06%	87.52%	99.94%	100%	100%	96.87%

方法	年份	模态	RVFA	FVRA-WL	FVFA-WL	FVFA-GAN	FVFA-FS	AVG-FV
文献[5]方法	2019	V	—	88.30%	91.00%	68.50%	93.50%	85.30%
文献[11]方法	2022	V	—	93.00%	96.70%	99.80%	99.10%	97.10%
文献[43]方法	2026	AV	98.60%	100%	100%	100%	100%	100%
文献[50]方法	2021	V	—	97.37%	98.70%	68.10%	99.90%	91.10%
文献[52]方法	2025	AV	96.40%	98.50%	99.70%	99.70%	99.70%	99.40%
本文方法	—	AV	99.20%	99.70%	100%	100%	100%	99.93%