基于多尺度特征融合重建学习的深度伪造人脸检测算法

doi:10.3969/j.issn.1671-1122.2024.08.004

摘要/Abstract

摘要：

随着深度伪造技术的快速发展，针对深度伪造人脸的检测已经成为计算机视觉领域的研究热点。虽然现有的基于噪声、局部纹理或频率特征的检测方法能够在特定场景中表现出良好的检测效果，但这些方法缺乏对人脸细粒度表征特征的深入挖掘，限制了其泛化能力。为了解决上述问题，文章提出了一种新型的基于多尺度特征融合重建的分类网络模型MSFFR，该网络模型从重建学习的角度学习挖掘人脸细粒度内容和梯度表征特征信息，并采用多尺度特征融合的方式实现伪造人脸的检测，通过融合这两种信息来识别伪造面孔。文章提出的模型包含3个创新模块,设计了双分支特征提取模块，用于揭示真实人脸与伪造人脸之间的分布差异；提出了细粒度内容和梯度特征融合模块，用于探索挖掘人脸细粒度内容特征与梯度特征之间的相关性；引入了基于重建视差的双向注意力模块，有效地指导模型对融合后的特征进行分类。在大规模基准数据集上进行的广泛实验表明，与现有技术相比，文章提出的方法在检测性能方面具有显著提高，尤其是在泛化能力方面表现出色。

关键词: 深伪检测, 多尺度特征融合, 重建学习, 深度生成模型

Abstract:

With the rapid development of deepfake technology, the detection of deepfake faces has become a research hotspot in the field of computer vision. Although existing detection methods based on noise, local texture, or frequency features can exhibit good detection performance to a certain extent or in specific scenarios, these methods lack in-depth exploration of fine-grained facial representation features, limiting their generalization ability. To address the above issues, this paper proposed a novel classification network model based on multi-scale feature fusion reconstruction MSFFR. This network explored fine-grained facial content and gradient representation features from the perspective of reconstruction learning and achieved deepfake face detection through multi-scale feature fusion. The model included three innovative modules, a dual-branch feature extraction module designed to reveal distribution differences between real and fake faces; a fine-grained content and gradient feature fusion module to explore the correlation between fine-grained content features and gradient features of faces; a bidirectional attention module based on reconstruction disparity, effectively guiding the model to classify the fused features. Extensive experiments conducted on large-scale benchmark datasets demonstrate that, compared with existing state-of-the-art techniques, the proposed method significantly improves detection performance, especially in terms of generalization ability.

Key words: deepfake detection, multi-scale feature fusion, reconstruction learning, deep generative model

中图分类号:

TP309

许楷文, 周翊超, 谷文权, 陈晨, 胡晰远. 基于多尺度特征融合重建学习的深度伪造人脸检测算法[J]. 信息网络安全, 2024, 24(8): 1173-1183.

XU Kaiwen, ZHOU Yichao, GU Wenquan, CHEN Chen, HU Xiyuan. A Multi-Scale Feature Fusion Deepfake Detection Algorithm Based on Reconstruction Learning[J]. Netinfo Security, 2024, 24(8): 1173-1183.

图/表 13

图1

图2

图3

图4

表1

表2

图5

表3

表4

表5

图6

图7

图8

参考文献 28

[1]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[2]	KINGMA D P, WELLING M. An Introduction to Variational Autoencoders[J]. Foundations and Trends® in Machine Learning, 2019, 12(4): 307-392.
[3]	PENG Shufan, CAI Manchun, LIU Xiaowen, et al. Deepfake Detection Algorithm Based on Image Fine-Grained Features[J]. Netinfo Security, 2022, 22(11): 77-84.
	彭舒凡, 蔡满春, 刘晓文, 等. 基于图像细粒度特征的深度伪造检测算法[J]. 信息网络安全, 2022, 22(11): 77-84.
[4]	ZI Bojia, CHANG Minghao, CHEN Jingjing, et al. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection[C]// ACM. Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2382-2390.
[5]	GU Qiqi, CHEN Shen, YAO Taiping, et al. Exploiting Fine-Grained Face Forgery Clues via Progressive Enhancement Learning[C]// AAAI. Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2022, 36(1): 735-743.
[6]	GU Zhihao, CHEN Yang, YAO Taiping, et al. Delving into the Local: Dynamic Inconsistency Learning for DeepFake Video Detection[C]// AAAI. Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2022, 36(1): 744-752.
[7]	LI Jiaming, XIE Hongtao, LI Jiahong, et al. Frequency-Aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection[C]// IEEE. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2021: 6458-6467.
[8]	CAO Junyi, MA Chao, YAO Taiping, et al. End-to-End Reconstruction-Classification Learning for Face Forgery Detection[C]// IEEE. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2022: 4113-4122.
[9]	CHOI Y, CHOI M, KIM M, et al. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation[C]// IEEE. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 8789-8797.
[10]	ABDAL R, QIN Yipeng, WONKA P. Image2StyleGAN: How to Embed Images into the StyleGAN Latent Space?[C]// IEEE. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2019: 4432-4441.
[11]	ZHAO Hanqing, WEI Tianyi, ZHOU Wenbo, et al. Multi-Attentional Deepfake Detection[C]// IEEE. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2021: 2185-2194.
[12]	LI Jiaming, XIE Hongtao, LI Jiahong, et al. Frequency-Aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection[C]// IEEE. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2021: 6458-6467.
[13]	ZHAO Tianchen, XU Xiang, XU Mingze, et al. Learning Self-Consistency for Deepfake Detection[C]// IEEE. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2021: 15023-15033.
[14]	DONG Shichao, WANG Jin, LIANG Jianjun, et al. Explaining Deepfake Detection by Analysing Image Matching[C]// Springer. European Conference on Computer Vision. Heidelberg: Springer, 2022: 18-35.
[15]	KHALID H, WOO S S. OC-FakeDect: Classifying Deepfakes Using One-Class Variational Autoencoder[C]// IEEE. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). New York: IEEE, 2020: 656-657.
[16]	TAN Mingxing, LE Q V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks[C]// PMLR. International Conference on Machine Learning. New York: PMLR, 2019: 6105-6114.
[17]	ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: Learning to Detect Manipulated Facial Images[C]// IEEE. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2019: 1-11.
[18]	DOLHANSKY B, BITTON J, PFLAUM B, et al. The DeepFake Detection Challenge (DFDC) Dataset[EB/OL]. (2020-06-12)[2024-04-30]. https://doi.org/10.48550/arXiv.2006.07397.
[19]	LI Yuezun, YANG Xin, SUN Pu, et al. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics[C]// IEEE. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2020: 3207-3216.
[20]	DUFOUR N, GULLY A. Contributing Data to Deepfake Detection Research[EB/OL]. [2024-04-30]. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html.
[21]	QIAN Yuyang, YIN Guojun, SHENG Lu, et al. Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues[C]// Springer. European Conference on Computer Vision. Heidelberg: Springer, 2020: 86-103.
[22]	SUN Ke, LIU Hong, YAO Taiping, et al. An Information Theoretic Approach For Attention-Driven Face Forgery Detection[C]// Springer. European Conference on Computer Vision. Heidelberg: Springer, 2022: 111-127.
[23]	WANG Yuan, YU Kun, CHEN Chen, et al. Dynamic Graph Learning with Content-Guided Spatial-Frequency Relation Reasoning for Deepfake Detection[C]// IEEE. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2023: 7278-7287.
[24]	YANG Ziming, LIANG Jian, XU Yuting, et al. Masked Relation Learning for DeepFake Detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1696-1708.
[25]	WANG Jian, DU Xiaoyu, CHENG Yu, et al. SI-Net: Spatial Interaction Network for Deepfake Detection[J]. Multimedia Systems, 2023, 29(5): 3139-3150.
[26]	PANG Guilin, ZHANG Baopeng, TENG Zhu, et al. MRE-Net: Multi-Rate Excitation Network for Deepfake Video Detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3663-3676.
[27]	SHI Zenan, CHEN Haipeng, CHEN Long, et al. Discrepancy-Guided Reconstruction Learning for Image Forgery Detection[EB/OL]. (2023-04-26)[2024-04-30]. https://arxiv.org/abs/2304.13349v2.
[28]	HUANG Baojin, WANG Zhongyuan, YANG Jifan, et al. Implicit Identity Driven Deepfake Face Swapping Detection[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2023: 4490-4499.

模型	FF++（LQ）		FF++（HQ）		WildDeepfake		Celeb-DF
模型	Acc	AUC	Acc	AUC	Acc	AUC	Acc	AUC
ADD-Net^[4]	87.50%	91.01%	96.78%	97.74%	76.25%	86.17%	96.93%	99.55%
SCL^[7]	89.00%	92.40%	96.69%	99.30%	—	—	—	—
MADD^[11]	88.69%	90.40%	97.60%	99.29%	82.62%	90.71%	97.92%	99.94%
F3Net^[21]	90.43%	93.30%	97.52%	98.10%	80.66%	87.53%	95.95%	98.93%
PEL^[5]	90.52%	94.28%	97.63%	99.32%	84.14%	91.62%	—	—
RECCE^[8]	91.03%	95.02%	97.06%	99.32%	83.25%	92.02%	98.59%	99.94%
ITA-SIA^[22]	90.23%	93.45%	97.64%	99.35%	83.95%	91.34%	98.48%	99.96%
SFDG^[23]	92.28%	95.98%	98.19%	99.53%	84.41%	92.57%	99.22%	99.96%
MRL^[24]	91.81%	96.18%	93.82%	98.27%	—	—	—	99.96%
SI-Net^[25]	87.90%	90.53%	96.85%	99.08%	84.08%	91.33%	—	—
MRE-Net^[26]	91.60%	96.55%	97.76%	99.57%	85.61%	91.23%	—	—
DisGRL^[27]	91.27%	95.19%	97.69%	99.48%	84.53%	93.27%	98.71%	99.91%
MSFFR	93.73%	97.12%	98.43%	99.99%	86.76%	95.27%	99.99%	99.99%

训练集	模型	测试集
		Celeb-DF		DFD		WildDeepfake		DFDC
		AUC	EER↓	AUC	EER↓	AUC	EER↓	AUC	EER↓
FF++	ADD-Net	57.83%	0.444	57.16%	0.453	54.21%	0.462	51.60%	0.548
	MADD	68.64%	0.371	74.19%	0.327	65.65%	0.397	63.02%	0.410
	F3Net	67.95%	0.368	69.50%	0.354	60.49%	0.434	57.87%	0.442
	PEL	69.18%	0.357	75.86%	0.308	67.39%	0.383	63.31%	0.404
	RECCE	68.71%	0.357	—	—	64.31%	0.405	69.06%	0.360
	SFDG	75.83%	0.303	88.00%	0.197	69.27%	0.377	73.64%	0.337
	IID^[28]	83.80%	0.248	93.92%	0.140	—	—	81.23%	0.268
	DisGRL	70.03%	0.342	—	—	66.73%	0.392	70.89%	0.342
	MSFFR	91.20%	0.127	94.25%	0.098	87.09%	0.195	86.53%	0.162

方法	PEL		SFDG		MSFFR
方法	FF	WLD	FF	WLD	FF	WLD
+高斯噪声	-0.10%	-0.86%	-0.10%	-0.75%	-0.23%	-0.47%
+椒盐噪声	-9.39%	-4.25%	-10.10%	-3.74%	-1.57%	-1.77%
+高斯模糊	-7.41%	-10.88%	-3.76%	-5.12%	-3.13%	-2.55%

序号	Rec	DFE	Rec disparity	Acc	AUC
a）	—	—	—	75.26%	84.10%
b）	√	—	—	77.37%	86.83%
c）	√	√	—	80.16%	89.44%
d）	√		√	79.83%	88.86%
MSFFR	√	√	√	86.76%	95.27%

特征融合方法	FF++（LQ）		WildDeepfake		Celeb-DF
特征融合方法	Acc	AUC	Acc	AUC	Acc	AUC
简单拼接	91.08%	95.23%	85.09%	94.16%	99.53%	99.86%
本文方法	93.73%	97.12%	86.76%	95.27%	99.99%	99.99%