基于知识蒸馏的医疗诊断差分隐私方法研究

doi:10.3969/j.issn.1671-1122.2025.04.002

摘要/Abstract

摘要：

随着智能医疗系统的快速发展，标注数据的匮乏已成为制约研究进展的关键因素之一，知识蒸馏作为一种有效的数据利用策略能够缓解这一问题。然而，在智能医疗领域，模型通常用于替代人工进行影像、数据的诊断，这不仅对医疗信息隐私保护提出了更高要求，还强调了模型精度对诊断结果准确性的决定性影响。因此，文章提出一种结合差分隐私的知识蒸馏方案，并将其应用于图神经网络模型，在知识蒸馏过程中保护用户敏感信息的同时，确保较高的医疗诊断准确率。为验证所提方法的有效性，文章构建了图注意力网络（GAT）模型和卷积神经网络（CNN）模型作为对照组，并采用3种实际医疗图像数据集进行实验。结果表明，文章所提方法在GAT模型的准确率较在CNN模型的准确率有所提升，对应在3个数据集上分别由61%提升至68%、83%提升至93%、67%提升至80%。鉴于GAT模型的高资源开销，文章进一步设计了一种轻量化GAT模型架构。该轻量化模型在显著降低资源消耗的同时，仍保持优于CNN模型的分类性能，从而在差分隐私保护的前提下，有效提升医疗诊断效果。

关键词: 知识蒸馏, 差分隐私, 图神经网络, 智能医疗, 轻量化

Abstract:

With the rapid development of intelligent medical systems, the lack of labeled data has become a key factor restricting research progress. Knowledge distillation, as an effective data utilization strategy, can alleviate this problem. However, in intelligent medical field, models are usually used to replace manual diagnosis of images and data. This not only puts forward higher requirements for the protection of medical information privacy, but also emphasizes the decisive impact of model accuracy on the accuracy of diagnostic results.Therefore, this paper proposed a knowledge distillation scheme combined with differential privacy, and applied it to graph neural network models, aiming to protect users’ sensitive information in the knowledge distillation process while ensuring high medical diagnostic accuracy. To verify the effectiveness of the proposed method, this paper constructed a graph attention network (GAT) model and a convolutional neural network (CNN) model as control groups, and conducted experimental verification using three practical medical image datasets. The experimental results show that the accuracy of the GAT model proposed in this paper is higher than that of the CNN model, which is improved from 61% to 68%, 83% to 93%, and 67% to 80% on the three datasets respectively. Given the high resource overhead of the GAT model, this paper further designed a lightweight GAT model architecture. The lightweight model significantly reduces resource consumption while maintaining classification performance superior to the CNN model, thereby effectively improving medical diagnostic outcomes under the premise of differential privacy protection.

Key words: knowledge distillation, differential privacy, graph neural networks, intelligent healthcare, lightweight

中图分类号:

TP309

李骁, 宋晓, 李勇. 基于知识蒸馏的医疗诊断差分隐私方法研究[J]. 信息网络安全, 2025, 25(4): 524-535.

LI Xiao, SONG Xiao, LI Yong. Research on Differential Privacy Methods for Medical Diagnosis Based on Knowledge Distillation[J]. Netinfo Security, 2025, 25(4): 524-535.

图/表 17

图1

图2

图3

表1

表2

图4

图5

图6

图7

表3

图8

图9

表4

表5

表6

表7

表8

参考文献 31

[1]	WAC M, SANTOS-RODRIGUEZ R, MCWILLIAMS C, et al. Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study[EB/OL]. (2023-09-28)[2024-11-15]. https://arxiv.org/abs/2309.16500.
[2]	DAI Enyan, ZHAO Tianxiang, ZHU Huaisheng, et al. A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability[J]. Machine Intelligence Research, 2024, 21(6): 1011-1061.
[3]	WU Zonghan, PAN Shirui, CHEN Fengwen, et al. A Comprehensive Survey on Graph Neural Networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(1): 4-24.
[4]	DILLON T W, LENDING D. Will They Adopt? Effects of Privacy and Accuracy[J]. Journal of Computer Information Systems, 2010, 50(4): 20-29.
[5]	ZHOU Jie, CUI Ganqu, HU Shengding, et al. Graph Neural Networks: A Review of Methods and Applications[J]. AI Open, 2020, 1: 57-81.
[6]	LIU Zhiyuan, ZHOU Jie. Graph Attention Networks[M]. Beijing: Posts & Telecom Press, 2021.
[7]	ZHANG Si, TONG Hanghang, XU Jiejun, et al. Graph Convolutional Networks: A Comprehensive Review[EB/OL]. (2019-09-10)[2024-11-15]. https://doi.org/10.1186/s40649-019-0069-y.
[8]	DWORK C. Differential Privacy[M]. Heidelberg: Springer, 2006.
[9]	DWORK C, ROTH A. The Algorithmic Foundations of Differential Privacy[J]. Foundations and Trends® in Theoretical Computer Science, 2014, 9(3-4): 211-407.
[10]	DWORK C, MCSHERRY F, NISSIM K, et al. Calibrating Noise to Sensitivity in Private Data Analysis[C]// Springer. Proceedings of the Third Conference on Theory of Cryptography. Heidelberg: Springer, 2006: 265-284.
[11]	WEN Jie, ZHANG Zhixia, LAN Yang, et al. A Survey on Federated Learning: Challenges and Applications[J]. International Journal of Machine Learning and Cybernetics, 2022, 14(2): 513-535.
[12]	YU Da, ZHANG Huishuai, CHEN Wei, et al. Large Scale Private Learning via Low-Rank Reparametrization[C]// PMLR. International Conference on Machine Learning. New York: PMLR, 2021: 12208-12218.
[13]	UNIYAL A, NAIDU R, KOTTI S, et al. DP-SGD vs PATE: Which Has Less Disparate Impact on Model Accuracy?[EB/OL]. (2024-03-19)[2024-11-15]. https://ar5iv.labs.arxiv.org/html/2106.12576.
[14]	CHEN Jing, ZHANG Jian. A Data-Free Personalized Federated Learning Algorithm Based on Knowledge Distillation[J]. Netinfo Security, 2024, 24(10): 1562-1569.
	陈婧, 张健. 基于知识蒸馏的无数据个性化联邦学习算法[J]. 信息网络安全, 2024, 24(10):1562-1569.
[15]	LIU Zilong, WANG Xuequn. How to Regulate Individuals’ Privacy Boundaries on Social Network Sites: A Cross-Cultural Comparison[J]. Information & Management, 2018, 55(8): 1005-1023.
[16]	INAN A, GURSOY M E, SAYGIN Y. Sensitivity Analysis for Non-Interactive Differential Privacy: Bounds and Efficient Algorithms[J]. IEEE Transactions on Dependable and Secure Computing, 2017(1): 194-207.
[17]	ABADI M, CHU A, GOODFELLOW I, et al. Deep Learning with Differential Privacy[C]// ACM. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2016: 308-318.
[18]	PAPERNOT N, ABADI M, ERLINGSSON Ú, et al. Semi-Supervised Knowledge Transfer for Deep Learning from Private Training Data[EB/OL]. (2017-02-06)[2024-11-15]. https://arxiv.org/abs/1610.05755v4.
[19]	KRICHEN M. Convolutional Neural Networks: A Survey[EB/OL]. (2023-06-17)[2024-11-15]. https://doi.org/10.3390/computers12080151.
[20]	ZHAO Xia, WANG Limin, ZHANG Yufei, et al. A Review of Convolutional Neural Networks in Computer Vision[EB/OL]. (2024-03-23)[2024-11-25]. https://doi.org/10.1007/s10462-024-10721-6.
[21]	VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph Attention Networks[EB/OL]. (2017-10-30)[2024-11-15]. https://arxiv.org/abs/1710.10903.
[22]	NGONG I. Maintaining Privacy in Medical Data with Differential Privacy[EB/OL]. (2020-03-12)[2024-11-15]. https://openmined.org/blog/maintaining-privacy-in-medical-data-with-differential-privacy.
[23]	LIU Weikang, ZHANG Yanchun, YANG Hong, et al. A Survey on Differential Privacy for Medical Data Analysis[J]. Annals of Data Science, 2024, 11(2): 733-747.
[24]	WEI Li, DUAN Qin, LIU Zhiwei. A Summary of Medical Information Security Management in Internet Era[J]. Netinfo Security, 2019, 19(12): 88-92.
	韦力, 段沁, 刘志伟. 互联网时代医院网络安全管理综述[J]. 信息网络安全, 2019, 19(12):88-92.
[25]	YAN Haicao, YIN Menghan, YAN Chaokun, et al. A Survey of Privacy Preserving Methods Based on Differential Privacy for Medical Data[C]// IEEE. 2024 7th World Conference on Computing and Communication Technologies (WCCCT). New York: IEEE, 2024: 104-108.
[26]	YANG Xingyi, HE Xuehai, ZHAO Jinyu, et al. COVID-CT-Dataset:A CT Scan Dataset about COVID-19[EB/OL]. (2020-03-30)[2024-11-15]. https://arxiv.org/abs/2003.13865v3.
[27]	YANG Feng, POOSTCHI M, YU Hang, et al. Deep Learning for Smartphone-Based Malaria Parasite Detection in Thick Blood Smears[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(5): 1427-1438. doi: 10.1109/JBHI.2019.2939121 pmid: 31545747
[28]	KASSIM Y M, YANG Feng, YU Hang, et al. Diagnosing Malaria Patients with Plasmodium Falciparum and Vivax Using Deep Learning for Thick Smear Images[EB/OL]. (2021-10-27)[2024-11-15]. https://doi.org/10.3390/diagnostics11111994.
[29]	KASSIM Y M, PALANIAPPAN K, YANG Feng, et al. Clustering-Based Dual Deep Learning Architecture for Detecting Red Blood Cells in Malaria Diagnostic Smears[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25(5): 1735-1746.
[30]	YANG Feng, QUIZON N, YU Hang, et al. Cascading YOLO: Automated Malaria Parasite Detection for Plasmodium Vivax in Thin Blood Smears[C]// SPIE. Medical Imaging 2020:Computer-Aided Diagnosis. Bellingham: SPIE, 2020: 404-410.
[31]	RAJARAMAN S, ANTANI S K, POOSTCHI M, et al. Pre-Trained Convolutional Neural Networks as Feature Extractors Toward Improved Malaria Parasite Detection in Thin Blood Smear Images[EB/OL]. (2018-04-16)[2024-11-15]. https://doi.org/10.7717/peerj.4568.

层类型	名称	输入尺寸	输出尺寸	参数描述
输入层	—	(3,32,32)	(3,32,32)	输入图像，3通道，32×32像素
卷积层	Conv2d	(3,32,32)	(32,32,32)	卷积核，32个3×3卷积核，步长1，填充1
激活层	ReLU	(32,32,32)	(32,32,32)	ReLU激活函数
池化层	MaxPool2d	(32,32,32)	(32,16,16)	最大池化，2×2池化窗口，步长2
展平层	—	(32,16,16)	(401408)	展平操作，转换为1D向量
全连接层	Linear	(401418)	(2)	全连接层，输出两个类别

层类型	名称	输入尺寸	输出尺寸	参数描述
输入层	—	(N,3)	(N,3)	输入节点特征矩阵，N为节点数量
图卷积层	GATConv(conv1)	(N,3)	(N,96)	96个卷积核，每个卷积核有27个权重和一个偏置，总计2688个参数
激活层	ELU	(N,96)	(N,96)	应用ELU激活函数
丢弃层	Dropout	(N,96)	(N,96)	随机丢弃比例为0.5
图卷积层	GATConv(conv2)	(N,96)	(N,32)	32个卷积核，每个卷积核有288个权重和一个偏置，总计9248个参数
激活层	ELU	(N,32)	(N,32)	应用ELU激活函数
丢弃层	Dropout	(N,32)	(N,32)	随机丢弃比例为0.5
图卷积层	GATConv(conv3)	(N,32)	(N,32)	32个卷积核，每个卷积核有96个权重和一个偏置，总计3104个参数
激活层	ELU	(N,32)	(N,32)	应用ELU激活函数
池化层	Global Mean Pooling	(N,32)	(1,32)	全局平均池化，整合整个图的特征
丢弃层	Dropout	(1,32)	(1,32)	随机丢弃比例为0.5
全连接层	Linear	(1,32)	(1,2)	全连接层，将32个隐藏单元映射到2个输出类别，共66个参数
激活函数	Log Softmax	(1,2)	(1,2)	使用log_softmax激活函数作为输出

数据集	数据量/张	教师训练集/张	教师测试集/张	学生训练集/张	学生测试集/张
COVID-CT	746	100	24	100	24
骨折影像	3996	320	79	320	79
疟疾细胞图像	27558	2204	551	2204	551

数据集	教师模型/个
数据集	1	3	5	7	9
COVID-CT	61%	67%	68%	—	—
骨折影像	87%	92%	93%	88%	90%
疟疾细胞图像	68%	74%	80%	81%	79%

模型	时间开销/s	内存占用/MB
未轻量化GAT模型训练	637.0	404
未轻量化GAT模型测试	14.8	412
轻量化GAT模型训练	159.0	378
轻量化GAT模型测试	8.3	382
CNN模型训练	133.0	360
CNN模型测试	6.9	359