A Multi-Scale Feature Fusion Deepfake Detection Algorithm Based on Reconstruction Learning

doi:10.3969/j.issn.1671-1122.2024.08.004

Abstract

Abstract:

With the rapid development of deepfake technology, the detection of deepfake faces has become a research hotspot in the field of computer vision. Although existing detection methods based on noise, local texture, or frequency features can exhibit good detection performance to a certain extent or in specific scenarios, these methods lack in-depth exploration of fine-grained facial representation features, limiting their generalization ability. To address the above issues, this paper proposed a novel classification network model based on multi-scale feature fusion reconstruction MSFFR. This network explored fine-grained facial content and gradient representation features from the perspective of reconstruction learning and achieved deepfake face detection through multi-scale feature fusion. The model included three innovative modules, a dual-branch feature extraction module designed to reveal distribution differences between real and fake faces; a fine-grained content and gradient feature fusion module to explore the correlation between fine-grained content features and gradient features of faces; a bidirectional attention module based on reconstruction disparity, effectively guiding the model to classify the fused features. Extensive experiments conducted on large-scale benchmark datasets demonstrate that, compared with existing state-of-the-art techniques, the proposed method significantly improves detection performance, especially in terms of generalization ability.

Key words: deepfake detection, multi-scale feature fusion, reconstruction learning, deep generative model

CLC Number:

TP309

XU Kaiwen, ZHOU Yichao, GU Wenquan, CHEN Chen, HU Xiyuan. A Multi-Scale Feature Fusion Deepfake Detection Algorithm Based on Reconstruction Learning[J]. Netinfo Security, 2024, 24(8): 1173-1183.

Figures/Tables 13

References 28

[1]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[2]	KINGMA D P, WELLING M. An Introduction to Variational Autoencoders[J]. Foundations and Trends® in Machine Learning, 2019, 12(4): 307-392.
[3]	PENG Shufan, CAI Manchun, LIU Xiaowen, et al. Deepfake Detection Algorithm Based on Image Fine-Grained Features[J]. Netinfo Security, 2022, 22(11): 77-84.
	彭舒凡, 蔡满春, 刘晓文, 等. 基于图像细粒度特征的深度伪造检测算法[J]. 信息网络安全, 2022, 22(11): 77-84.
[4]	ZI Bojia, CHANG Minghao, CHEN Jingjing, et al. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection[C]// ACM. Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2382-2390.
[5]	GU Qiqi, CHEN Shen, YAO Taiping, et al. Exploiting Fine-Grained Face Forgery Clues via Progressive Enhancement Learning[C]// AAAI. Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2022, 36(1): 735-743.
[6]	GU Zhihao, CHEN Yang, YAO Taiping, et al. Delving into the Local: Dynamic Inconsistency Learning for DeepFake Video Detection[C]// AAAI. Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2022, 36(1): 744-752.
[7]	LI Jiaming, XIE Hongtao, LI Jiahong, et al. Frequency-Aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection[C]// IEEE. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2021: 6458-6467.
[8]	CAO Junyi, MA Chao, YAO Taiping, et al. End-to-End Reconstruction-Classification Learning for Face Forgery Detection[C]// IEEE. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2022: 4113-4122.
[9]	CHOI Y, CHOI M, KIM M, et al. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation[C]// IEEE. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 8789-8797.
[10]	ABDAL R, QIN Yipeng, WONKA P. Image2StyleGAN: How to Embed Images into the StyleGAN Latent Space?[C]// IEEE. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2019: 4432-4441.
[11]	ZHAO Hanqing, WEI Tianyi, ZHOU Wenbo, et al. Multi-Attentional Deepfake Detection[C]// IEEE. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2021: 2185-2194.
[12]	LI Jiaming, XIE Hongtao, LI Jiahong, et al. Frequency-Aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection[C]// IEEE. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2021: 6458-6467.
[13]	ZHAO Tianchen, XU Xiang, XU Mingze, et al. Learning Self-Consistency for Deepfake Detection[C]// IEEE. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2021: 15023-15033.
[14]	DONG Shichao, WANG Jin, LIANG Jianjun, et al. Explaining Deepfake Detection by Analysing Image Matching[C]// Springer. European Conference on Computer Vision. Heidelberg: Springer, 2022: 18-35.
[15]	KHALID H, WOO S S. OC-FakeDect: Classifying Deepfakes Using One-Class Variational Autoencoder[C]// IEEE. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). New York: IEEE, 2020: 656-657.
[16]	TAN Mingxing, LE Q V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks[C]// PMLR. International Conference on Machine Learning. New York: PMLR, 2019: 6105-6114.
[17]	ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: Learning to Detect Manipulated Facial Images[C]// IEEE. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2019: 1-11.
[18]	DOLHANSKY B, BITTON J, PFLAUM B, et al. The DeepFake Detection Challenge (DFDC) Dataset[EB/OL]. (2020-06-12)[2024-04-30]. https://doi.org/10.48550/arXiv.2006.07397.
[19]	LI Yuezun, YANG Xin, SUN Pu, et al. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics[C]// IEEE. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2020: 3207-3216.
[20]	DUFOUR N, GULLY A. Contributing Data to Deepfake Detection Research[EB/OL]. [2024-04-30]. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html.
[21]	QIAN Yuyang, YIN Guojun, SHENG Lu, et al. Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues[C]// Springer. European Conference on Computer Vision. Heidelberg: Springer, 2020: 86-103.
[22]	SUN Ke, LIU Hong, YAO Taiping, et al. An Information Theoretic Approach For Attention-Driven Face Forgery Detection[C]// Springer. European Conference on Computer Vision. Heidelberg: Springer, 2022: 111-127.
[23]	WANG Yuan, YU Kun, CHEN Chen, et al. Dynamic Graph Learning with Content-Guided Spatial-Frequency Relation Reasoning for Deepfake Detection[C]// IEEE. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2023: 7278-7287.
[24]	YANG Ziming, LIANG Jian, XU Yuting, et al. Masked Relation Learning for DeepFake Detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1696-1708.
[25]	WANG Jian, DU Xiaoyu, CHENG Yu, et al. SI-Net: Spatial Interaction Network for Deepfake Detection[J]. Multimedia Systems, 2023, 29(5): 3139-3150.
[26]	PANG Guilin, ZHANG Baopeng, TENG Zhu, et al. MRE-Net: Multi-Rate Excitation Network for Deepfake Video Detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3663-3676.
[27]	SHI Zenan, CHEN Haipeng, CHEN Long, et al. Discrepancy-Guided Reconstruction Learning for Image Forgery Detection[EB/OL]. (2023-04-26)[2024-04-30]. https://arxiv.org/abs/2304.13349v2.
[28]	HUANG Baojin, WANG Zhongyuan, YANG Jifan, et al. Implicit Identity Driven Deepfake Face Swapping Detection[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2023: 4490-4499.

模型	FF++（LQ）		FF++（HQ）		WildDeepfake		Celeb-DF
模型	Acc	AUC	Acc	AUC	Acc	AUC	Acc	AUC
ADD-Net^[4]	87.50%	91.01%	96.78%	97.74%	76.25%	86.17%	96.93%	99.55%
SCL^[7]	89.00%	92.40%	96.69%	99.30%	—	—	—	—
MADD^[11]	88.69%	90.40%	97.60%	99.29%	82.62%	90.71%	97.92%	99.94%
F3Net^[21]	90.43%	93.30%	97.52%	98.10%	80.66%	87.53%	95.95%	98.93%
PEL^[5]	90.52%	94.28%	97.63%	99.32%	84.14%	91.62%	—	—
RECCE^[8]	91.03%	95.02%	97.06%	99.32%	83.25%	92.02%	98.59%	99.94%
ITA-SIA^[22]	90.23%	93.45%	97.64%	99.35%	83.95%	91.34%	98.48%	99.96%
SFDG^[23]	92.28%	95.98%	98.19%	99.53%	84.41%	92.57%	99.22%	99.96%
MRL^[24]	91.81%	96.18%	93.82%	98.27%	—	—	—	99.96%
SI-Net^[25]	87.90%	90.53%	96.85%	99.08%	84.08%	91.33%	—	—
MRE-Net^[26]	91.60%	96.55%	97.76%	99.57%	85.61%	91.23%	—	—
DisGRL^[27]	91.27%	95.19%	97.69%	99.48%	84.53%	93.27%	98.71%	99.91%
MSFFR	93.73%	97.12%	98.43%	99.99%	86.76%	95.27%	99.99%	99.99%

训练集	模型	测试集
		Celeb-DF		DFD		WildDeepfake		DFDC
		AUC	EER↓	AUC	EER↓	AUC	EER↓	AUC	EER↓
FF++	ADD-Net	57.83%	0.444	57.16%	0.453	54.21%	0.462	51.60%	0.548
	MADD	68.64%	0.371	74.19%	0.327	65.65%	0.397	63.02%	0.410
	F3Net	67.95%	0.368	69.50%	0.354	60.49%	0.434	57.87%	0.442
	PEL	69.18%	0.357	75.86%	0.308	67.39%	0.383	63.31%	0.404
	RECCE	68.71%	0.357	—	—	64.31%	0.405	69.06%	0.360
	SFDG	75.83%	0.303	88.00%	0.197	69.27%	0.377	73.64%	0.337
	IID^[28]	83.80%	0.248	93.92%	0.140	—	—	81.23%	0.268
	DisGRL	70.03%	0.342	—	—	66.73%	0.392	70.89%	0.342
	MSFFR	91.20%	0.127	94.25%	0.098	87.09%	0.195	86.53%	0.162

方法	PEL		SFDG		MSFFR
方法	FF	WLD	FF	WLD	FF	WLD
+高斯噪声	-0.10%	-0.86%	-0.10%	-0.75%	-0.23%	-0.47%
+椒盐噪声	-9.39%	-4.25%	-10.10%	-3.74%	-1.57%	-1.77%
+高斯模糊	-7.41%	-10.88%	-3.76%	-5.12%	-3.13%	-2.55%

序号	Rec	DFE	Rec disparity	Acc	AUC
a）	—	—	—	75.26%	84.10%
b）	√	—	—	77.37%	86.83%
c）	√	√	—	80.16%	89.44%
d）	√		√	79.83%	88.86%
MSFFR	√	√	√	86.76%	95.27%

特征融合方法	FF++（LQ）		WildDeepfake		Celeb-DF
特征融合方法	Acc	AUC	Acc	AUC	Acc	AUC
简单拼接	91.08%	95.23%	85.09%	94.16%	99.53%	99.86%
本文方法	93.73%	97.12%	86.76%	95.27%	99.99%	99.99%