Window-guided audio-visual deepfake detection for local temporal anomalies

doi:10.3969/j.issn.1671-1122.2026.09.008

Abstract

Abstract:

With the development of audio-visual generation techniques, deepfake videos increasingly exhibit localized, short-duration, and multimodal manipulation patterns. Short-term forgery cues may be weakened by numerous normal frames during global temporal modeling. To address this issue, this paper proposed a window-guided audio-visual deepfake detection method for local temporal anomalies. Based on temporally aligned audio and visual features, our model constructed local dynamic information and cross-modal difference information, and organized the sequence into a frame-window-global hierarchy. The model preserved short-term anomalous responses within local windows, and enhances frame-level representations through cross-window contextual modeling and global semantic feedback, thereby balancing local anomaly perception and video-level authenticity prediction. Experimental results show that the proposed method achieves intra-dataset AUC scores of 99.54%, 99.90%, and 100% on FakeAVCeleb, DFDC, and LAV-DF, respectively. Cross-dataset and cross-manipulation experiments further evaluate the model under changes in data distribution and manipulation type. Ablation studies further verify the contributions of local-window modeling, anomaly-peak extraction, and global semantic feedback. Additional boundary-cropping and backend-efficiency experiments show that the model maintains stable performance under boundary perturbations, while window-level contextual modeling effectively reduces the computational and memory costs of backend long-sequence modeling. These results demonstrate that window-guided hierarchical aggregation preserves local short-term forgery cues while supporting video-level discrimination and efficient backend modeling of long sequences.

Key words: deepfake detection, audio-visual forensics, local temporal anomaly, window-guided aggregation, multi-granularity prediction

CLC Number:

TP309

Zhang Lu, Lu Tianliang. Window-guided audio-visual deepfake detection for local temporal anomalies[J]. Netinfo Security, 2026, 26(9): 1422-1434.

Figures/Tables 10

References 52

[1]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144. doi: 10.1145/3422622 URL
[2]	Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
[3]	Prajwal K R, Mukhopadhyay R, Namboodiri V P, et al. A lip sync expert is all you need for speech to lip generation in the wild[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 484-492.
[4]	Afchar D, Nozick V, Yamagishi J, et al. MesoNet: a compact facial video forgery detection network[C]// 2018 IEEE International Workshop on Information Forensics and Security (WIFS). New York: IEEE, 2018: 1-7.
[5]	Rossler A, Cozzolino D, Verdoliva L, et al. FaceForensics++: learning to detect manipulated facial images[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2019: 1-11.
[6]	Li Lingzhi, Bao Jianmin, Zhang Ting, et al. Face X-ray for more general face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2020: 5001-5010.
[7]	Qian Yuyang, Yin Guojun, Sheng Lu, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]// The European Conference on Computer Vision. Heidelberg: Springer, 2020: 86-103.
[8]	Chung J S, Zisserman A. Out of time: automated lip sync in the wild[C]// The Asian Conference on Computer Vision. Heidelberg: Springer, 2016: 251-263.
[9]	Chugh K, Gupta P, Dhall A, et al. Not made for each other-audio-visual dissonance-based deepfake detection and localization[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 439-447.
[10]	Zhou Yipin, Lim S N. Joint audio-visual deepfake detection[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2021: 14800-14809.
[11]	Haliassos A, Mira R, Petridis S, et al. Leveraging real talking faces via self-supervision for robust forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 14950-14962.
[12]	Yin Qilin, Lu Wei, Cao Xiaochun, et al. Fine-grained multimodal deepfake classification via heterogeneous graphs[J]. International Journal of Computer Vision, 2024, 132(11): 5255-5269. doi: 10.1007/s11263-024-02128-1
[13]	Bertasius G, Wang Heng, Torresani L. Is space-time attention all you need for video understanding?[C]// The 38th International Conference on Machine Learning. Vienna: PMLR, 2021, 139: 813-824.
[14]	Liu Ze, Ning Jia, Cao Yue, et al. Video swin transformer[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 3202-3211.
[15]	Liu Honggu, Li Xiaodan, Zhou Wenbo, et al. Spatial-Phase shallow learning: rethinking face forgery detection in frequency domain[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 772-781.
[16]	Cao Junyi, Ma Chao, Yao Taiping, et al. End-to-end reconstruction-classification learning for face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 4113-4122.
[17]	Shiohara K, Yamasaki T. Detecting deepfakes with self-blended images[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 18720-18729.
[18]	Miao Changtao, Tan Zichang, Chu Qi, et al. Hierarchical frequency-assisted interactive networks for face manipulation detection[J]. IEEE Transactions on Information Forensics and Security, 2022, 17: 3008-3021. doi: 10.1109/TIFS.2022.3198275 URL
[19]	Jung J, Heo H S, Kim J, et al. RawNet: advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification[C]// The 20th Annual Conference of the International Speech Communication Association (INTERSPEECH 2019). Baixas: ISCA, 2019: 1268-1272.
[20]	Jung J, Heo H S, Tak H, et al. Aasist: audio anti-spoofing using integrated spectro-temporal graph attention networks[C]// 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New York: IEEE, 2022: 6367-6371.
[21]	Chen Sanyuan, Wang Chengyi, Chen Zhengyang, et al. WavLM: large-scale self-supervised pre-training for full stack speech processing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6): 1505-1518. doi: 10.1109/JSTSP.2022.3188113 URL
[22]	Yang Wenyuan, Zhou Xiaoyu, Chen Zhikai, et al. AVoiD-DF: audio-visual joint learning for detecting deepfake[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 2015-2029. doi: 10.1109/TIFS.2023.3262148 URL
[23]	Yu Yang, Liu Xiaolong, Ni Rongrong, et al. PVASS-MDD: predictive visual-audio alignment self-supervision for multimodal deepfake detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 34(8): 6926-6936. doi: 10.1109/TCSVT.2023.3309899 URL
[24]	Oorloff T, Koppisetti S, Bonettini N, et al. AVFF: audio-visual feature fusion for video deepfake detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2024: 27102-27112.
[25]	Nie Fan, Ni Jiangqun, Zhang Jian, et al. FRADE: forgery-aware audio-distilled multimodal learning for deepfake detection[C]// The 32nd ACM International Conference on Multimedia. New York: ACM, 2024: 6297-6306.
[26]	Miao Hui, Guo Yuanfang, Liu Zeming, et al. Multi-modal deepfake detection via multi-task audio-visual prompt learning[C]// The AAAI Conference on Artificial Intelligence. Washington: AAAI, 2025: 612-621.
[27]	Nguyen D, Mejri N, Singh I P, et al. Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2024: 17395-17405.
[28]	Mittal T, Bhattacharya U, Chandra R, et al. Emotions don't lie: an audio-visual deepfake detection method using affective cues[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2823-2832.
[29]	Güera D, Delp E J. Deepfake video detection using recurrent neural networks[C]// The 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). New York: IEEE, 2018: 1-6.
[30]	Carreira J, Zisserman A. Quo vadis action recognition? A new model and the kinetics dataset[C]// The IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 6299-6308.
[31]	Zheng Yinglin, Bao Jianmin, Chen Dong, et al. Exploring temporal coherence for more general video face forgery detection[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2021: 15044-15054.
[32]	Gu Zhihao, Chen Yang, Yao Taiping, et al. Spatiotemporal inconsistency learning for deepfake video detection[C]// The 29th ACM International Conference on Multimedia. New York: ACM, 2021: 3473-3481.
[33]	Gu Zhihao, Chen Yang, Yao Taiping, et al. Delving into the local: dynamic inconsistency learning for deepfake video detection[C]// The 36th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2022: 744-752.
[34]	Wang Zhendong, Bao Jianmin, Zhou Wengang, et al. AltFreezing for more general video face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2023: 4129-4138.
[35]	Wang Junke, Wu Zuxuan, Ouyang Wenhao, et al. M2TR: multi-modal multi-scale transformers for deepfake detection[C]// The 2022 International Conference on Multimedia Retrieval. New York: ACM, 2022: 615-623.
[36]	Zhao Cairong, Wang Chutian, Hu Guosheng, et al. ISTVT: interpretable spatial-temporal video transformer for deepfake detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1335-1348. doi: 10.1109/TIFS.2023.3239223 URL
[37]	Lin T-Y, Goyal P, Girshick R, et al. Focal loss for dense object detection[C]// The IEEE International Conference on Computer Vision. New York: IEEE, 2017: 2980-2988.
[38]	Khalid H, Tariq S, Kim M, et al. FakeAVCeleb: a novel audio-video multimodal deepfake dataset[EB/OL]. (2022-03-01)[2026-05-14]. https://arxiv.org/abs/2108.05080.
[39]	Dolhansky B, Bitton J, Pflaum B, et al. The deepfake detection challenge (DFDC) dataset[EB/OL]. (2020-10-28)[2026-05-14]. https://arxiv.org/abs/2006.07397.
[40]	Cai Zhixi, Ghosh S, Dhall A, et al. Glitch in the matrix: a large scale benchmark for content driven audio-visual forgery detection and localization[J]. Computer Vision and Image Understanding, 2023, 236: 103818. doi: 10.1016/j.cviu.2023.103818 URL
[41]	Wang Jian, Wu Baoyuan, Liu Li, et al. FauForensics: boosting audio-visual deepfake detection with facial action units[J]. IEEE Transactions on Information Forensics and Security, 2026, 21: 3353-3365. doi: 10.1109/TIFS.2026.3674401 URL
[42]	Smeu S, Boldisor D A, Oneata D, et al. Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2025: 18815-18825.
[43]	Peng Jielun, Wang Yabin, Li Yaqi, et al. Leave no stone unturned: uncovering holistic audio-visual intrinsic coherence for deepfake detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings. New York: IEEE, 2026: 6655-6666.
[44]	Cai Zhixi, Stefanov K, Dhall A, et al. Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization[C]// 2022 International Conference on Digital Image Computing:Techniques and Applications (DICTA). New York: IEEE, 2022: 1-10.
[45]	Koutlis C, Papadopoulos S. DiMoDif: discourse modality-information differentiation for audio-visual deepfake detection and localization[EB/OL]. (2025-04-11)[2026-05-14]. https://arxiv.org/abs/2411.10193.
[46]	Anshul A, Gopal S, Rajan D, et al. Intra-modal and cross-modal synchronization for audio-visual deepfake detection and temporal localization[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2025: 13826-13836.
[47]	Datta S K, Jia Shan, Lyu Siwei. LoCC: detection and localization of lip-syncing deepfakes via counterfactual frame consistency[EB/OL]. (2026-05-22)[2026-05-14]. https://arxiv.org/abs/2606.22772.
[48]	Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer[EB/OL]. (2021-03-11)[2026-05-14]. https://arxiv.org/abs/2102.11126.
[49]	Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer[EB/OL]. arXiv: 2102.11126, 2021.
[50]	Haliassos A, Vougioukas K, Petridis S, et al. Lips Don't Lie: a generalisable and robust approach to face forgery detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 5039-5049.
[51]	Wang Rui, Ye Dengpan, Tang Long, et al. AVT²-DWF: improving deepfake detection with audio-visual fusion and dynamic weighting strategies[J]. IEEE Signal Processing Letters, 2024, 31: 1960-1964. doi: 10.1109/LSP.2024.3433596 URL
[52]	Anshul A, Gopal S, Rajan D, et al. Next-frame feature prediction for multimodal deepfake detection and temporal localization[EB/OL]. (2025-11-13)[2026-05-14]. https://arxiv.org/abs/2511.10212.

方法	发表出处	年份	FakeAVCeleb	DFDC	LAV-DF
文献[9]方法	ACM MM	2020	86.50%	91.50%	97.83%
文献[22]方法	TIFS	2023	89.20%	94.80%	—
文献[23]方法	TCSVT	2023	96.50%	98.90%	—
文献[26]方法	AAAI	2025	99.98%	—	—
文献[41]方法	TIFS	2026	99.91%	—	99.97%
文献[42]方法	CVPR	2025	99.20%	—	—
文献[43]方法	CVPR	2026	99.65%	—	—
本文方法	—	—	99.54%	99.90%	100%

方法	年份	AP@0.50	AP@0.75	AP@0.95
文献[9]方法	2020	12.78%	1.62%	0
文献[44]方法	2022	76.90%	38.50%	0.25%
文献[45]方法	2024	95.50%	87.90%	20.60%
文献[46]方法	2025	87.40%	66.80%	5.72%
文献[47]方法	2026	91.00%	74.07%	2.46%
本文方法	—	89.55%	77.06%	23.47%

训练集→测试集	方法	年份	AUC
FakeAVCeleb→ LAV-DF	文献[12]方法	2024	63.80%
	文献[48]方法	2024	95.80%
	本文方法	—	95.69%
LAV-DF→ FakeAVCeleb	文献[12]方法	2024	65.52%
	文献[41]方法	2026	83.77%
	本文方法	—	92.64%
FakeAVCeleb→ DFDC	文献[5]方法	2019	50.00%
	文献[9]方法	2020	63.50%
	文献[49]方法	2021	51.10%
	文献[50]方法	2021	49.00%
	文献[51]方法	2024	74.60%
	本文方法	—	76.50%
DFDC→ FakeAVCeleb	文献[5]方法	2019	61.43%
	文献[9]方法	2020	47.62%
	文献[49]方法	2021	57.57%
	文献[50]方法	2021	56.14%
	文献[51]方法	2024	77.20%
	本文方法	—	68.68%

方法	年份	模态	RVFA	FVRA-WL	FVFA-WL	FVFA-GAN	FVFA-FS	AVG-FV
文献[5]方法	2019	V	—	88.20%	91.00%	69.60%	92.30%	84.80%
文献[11]方法	2022	V	—	88.80%	93.40%	99.80%	99.30%	95.30%
文献[43]方法	2026	AV	96.70%	100%	100%	100%	100%	100%
文献[50]方法	2021	V	—	97.80%	98.60%	64.50%	99.90%	89.40%
文献[52]方法	2025	AV	95.90%	88.60%	97.60%	97.80%	98.40%	95.60%
本文方法	—	AV	96.06%	87.52%	99.94%	100%	100%	96.87%

方法	年份	模态	RVFA	FVRA-WL	FVFA-WL	FVFA-GAN	FVFA-FS	AVG-FV
文献[5]方法	2019	V	—	88.30%	91.00%	68.50%	93.50%	85.30%
文献[11]方法	2022	V	—	93.00%	96.70%	99.80%	99.10%	97.10%
文献[43]方法	2026	AV	98.60%	100%	100%	100%	100%	100%
文献[50]方法	2021	V	—	97.37%	98.70%	68.10%	99.90%	91.10%
文献[52]方法	2025	AV	96.40%	98.50%	99.70%	99.70%	99.70%	99.40%
本文方法	—	AV	99.20%	99.70%	100%	100%	100%	99.93%