A Data-Free Personalized Federated Learning Algorithm Based on Knowledge Distillation

doi:10.3969/j.issn.1671-1122.2024.10.010

Abstract

Abstract:

Federated learning algorithms usually face the problem of huge differences between clients, and these heterogeneities degrade the global model performance, which are mitigated by knowledge distillation approaches. In order to further liberate public data and improve the model performance, DFP-KD trained a robust federated learning global model using datad-free knowledge distillation methods; used ReACGAN as the generator part; and adopted a step-by-step EMA fast updating strategy, which speeded up the update rate of the global model while avoiding catastrophic forgetting. Comparison experiments, ablation experiments, and parameter value influence experiments show that DFP-KD is more advantageous than the classical data-free knowledge distillation algorithms in terms of accuracy, stability, and update rate.

Key words: federated learning, heterogeneity, knowledge distillation, image generation

CLC Number:

TP309

CHEN Jing, ZHANG Jian. A Data-Free Personalized Federated Learning Algorithm Based on Knowledge Distillation[J]. Netinfo Security, 2024, 24(10): 1562-1569.

Figures/Tables 7

References 20

[1]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-Efficient Learning of Deep Networks from Decentralized Data[C]// AISTATS. 20th International Conference on Artificial Intelligence and Statistics. New York: PMLR, 2017: 1273-1282.
[2]	ZHANG L. Federated Learning-Cracking the Smart Healthcare Data Security Privacy Puzzle[EB/OL]. (2020-08-04)[2024-04-20]. https://zhuanlan.zhihu.com/p/166482616.
[3]	LI Qinbin, DIAO Yiqun, CHEN Quan, et al. Federated Learning on Non-IID Data Silos: An Experimental Study[C]// IEEE. 38th International Conference on Data Engineering (ICDE). New York: IEEE, 2022: 965-978.
[4]	GONG Yanxia. Research on Personalized Method of Federated Learning in Heterogeneous Scenario[D]. GuiLin: Guangxi Normal University, 2023.
	龚艳霞. 异质场景下联邦学习的个性化方法研究[D]. 桂林: 广西师范大学, 2023.
[5]	HUANG Hua. Research on Key Technologies of Federated Learning with Heterogeneous Data[D]. Xi’an: Xidian University, 2022.
	黄华. 异质性数据的联邦学习关键技术研究[D]. 西安: 西安电子科技大学, 2022.
[6]	SHEN Tao, KUANG Kun, WU Chao, et al. The Challenge of Heterogeneity in Privacy Computing: Exploring the Co-Optimization Problem of Federated Learning in Distributed Heterogeneous Environments[J]. Artificial Intelligence, 2023(6): 1-13.
	沈弢, 况琨, 吴超, 等. 隐私计算中的异质性挑战:探索分布式异质环境下联邦学习的协同优化问题[J]. 人工智能, 2023(6): 1-13.
[7]	ZHAN Fan. Research on Weighted Federated Distillation Algorithm for Non-IID Data[D]. Wuhan: Huazhong University of Science and Technology, 2022.
	詹帆. 面向非独立同分布数据的加权联邦蒸馏算法研究[D]. 武汉: 华中科技大学, 2022.
[8]	CHEN Xuebin, REN Zhiqiang. PFKD: A Personalized Federated Learning Framework that Integrates Data Heterogeneity and Model Heterogeneity[J]. Journal of Nanjing University of Information Science & Technology, 2024, 16(4): 513-519.
	陈学斌, 任志强. PFKD: 综合考虑数据异构和模型异构的个性化联邦学习框架[J]. 南京信息工程大学学报, 2024, 16(4): 513-519.
[9]	LIN Tao, KONG Lingjing, STICH S U, et al. Ensemble Distillation for Robust Model Fusion in Federated Learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 2351-2363.
[10]	YANG Qiang. A Study of Image Classification Algorithm Based on Heterogeneous Federated Learning[D]. Chengdu: University of Electronic Science and Technology of China, 2023.
	杨强. 基于异构联邦学习的图像分类算法研究[D]. 成都: 电子科技大学, 2023.
[11]	ZHUANG Yulin, LYU Lingjuan, SHI Chuan, et al. Data-Free Adversarial Knowledge Distillation for Graph Neural Networks[C]// IJCAI. 31st International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann, 2022: 2441-2447.
[12]	ZHU Zhuangdi, HONG Junyuan, ZHOU Jiayu. Data-Free Knowledge Distillation for Heterogeneous Federated Learning[C]// ICML. 38th International Conference on Machine Learning. New York: PMLR, 2021: 12878-12889.
[13]	ZHANG Lin, SHEN Li, DING Liang, et al. Fine-Tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning[C]// IEEE. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 10164-10173.
[14]	ZHANG Jie, CHEN Chen, LI Bo, et al. Dense: Data-Free One-Shot Federated Learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 21414-21428.
[15]	LUO Kangyang, WANG Shuai, FU Yexuan, et al. DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning[C]// NeurIPS. 37th International Conference on Neural Information Processing Systems. New York: Curran Associates, 2023: 17854-17866.
[16]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[17]	JIANG Jinyang, WU Haotian, WANG Fengjuan, et al. Cross-Dimensional Reconstruction of Microstructures and Transport Properties Characterization of Porous Materials via Deep Convolutional Generative Adversarial Network[J]. Engineering Mechanics, 2024, 41: 1-14.
[18]	ODENA A, OLAH C, SHLENS J. Conditional Image Synthesis with Auxiliary Classifier Gans[C]// ICML. 34th International Conference on Machine Learning. New York: PMLR, 2017: 2642-2651.
[19]	KANG M, SHIM W, CHO M, et al. Rebooting Acgan: Auxiliary Classifier Gans with Stable Training[J]. Advances in Neural Information Processing Systems, 2021, 34: 23505-23518.
[20]	DO K, LE T H, NGUYEN D, et al. Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation[J]. Advances in Neural Information Processing Systems, 2022, 35: 10055-10067.

算法	数据集	L.acc	G.acc
DFP-KD	SVHN	22.34%	17.53%
DFP-KD	FMNIST	26.58%	34.96%
DENSE	SVHN	20.30%	13.25%
DENSE	FMNIST	26.41%	29.79%
FedFTG	SVHN	13.12%	9.16%
FedFTG	FMNIST	20.78%	19.69%

实验算法	准确率
实验算法	L.acc	G.acc
DFP-KD	26.58%	34.96%
-分步EMA	26.58%	31.72%
-${{L}_{fid}}$	26.58%	21.78%
-分步EMA-${{L}_{fid}}$	26.58%	20.30%

实验算法	准确率
实验算法	L.acc	G.acc
DFP-KD	22.34%	17.53%
-分步EMA	22.34%	19.33%
-${{L}_{fid}}$	22.34%	15.88%
-分步EMA-${{L}_{fid}}$	22.34%	14.68%

$\lambda$	准确率
$\lambda$	L.acc	G.acc
0.1	22.34%	17.63%
0.3	22.34%	18.54%
0.5	22.34%	17.53%
0.7	22.34%	18.50%
0.9	22.34%	15.48%