Research on Membership Inference Attack Method Based on Double Threshold Function

doi:10.3969/j.issn.1671-1122.2023.02.008

Abstract

Abstract:

The emergence of massive data and powerful computing power has brought deep learning to an unprecedented height, and its wide application in areas such as intelligent transportation and medical diagnosis has brought many conveniences to people’s daily lives. However, privacy leakage in machine learning cannot be ignored. Among them, the membership inference attack infers that whether the data sample can used in the training set of the machine learning model, thus interfering with the user’s training data. Firstly, this paper introduced the single-threshold-based membership inference attack and its characteristics, visualized the data distribution of members and non-members for different attack methods, then analyzed the internal mechanism of the successful membership inference attack, and proposed an attack model based on a double-threshold function, and systematically analyzed and compared single-threshold and double-threshold membership inference attacks through experiments, and analyzed the attack performance of threshold-based membership inference attacks on different models and different datasets. The comparative experiments on multiple groups of control variables show that the membership inference attack based on the double-threshold function has better performance on some data sets and models, and the overall performance is more stable.

Key words: deep learning, membership inference attack, privacy leak, double-threshold function

CLC Number:

TP309

CHEN Depeng, LIU Xiao, CUI Jie, ZHONG Hong. Research on Membership Inference Attack Method Based on Double Threshold Function[J]. Netinfo Security, 2023, 23(2): 64-75.

Figures/Tables 16

References 16

[1]	SHOKRI R, STRONATI M, SONG Congzheng, et al. Membership Inference Attacks Against Machine Learning Models[C]// IEEE. Symposium on Security and Privacy. New York: IEEE, 2017: 3-18.
[2]	SALEM A, ZHANG Yang, HUMBERT M, et al. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models[C]// The Internet Society. Network and Distributed System Security Symposium. New York: The Internet Society, 2019: 1-15.
[3]	LI Zheng, ZHANG Yang. Membership Leakage in Label-Only Exposures[C]// ACM. Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2021: 880-895.
[4]	YEOM S, GIACOMELLI I, FREDRIKSON M, et al. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting[C]// IEEE. 31st Computer Security Foundations Symposium. New York: IEEE, 2018: 268-282.
[5]	CHOO C C A, TRAMER F, CARLINI N, et al. Label-Only Membership Inference Attacks[C]// PMLR. Proceedings of the 38th International Conference on Machine Learning. New York: PMLR, 2021: 1964-1974.
[6]	HUI Bo, YANG Yuchen, YUAN Haolin, et al. Practical Blind Membership Inference Attack via Differential Comparisons[C]// The Internet Society. Network and Distributed System Security Symposium, New York: The Internet Society, 2021: 1-17.
[7]	TRUEX S, LIU Ling, GURSOY M E, et al. Demystifying Membership Inference Attacks in Machine Learning as a Service[J]. IEEE Transactions on Services Computing, 2019, 14(6): 2073-2089. doi: 10.1109/TSC.2019.2897554 URL
[8]	CHEN Jiale, ZHANG Jiale, ZHAO Yanchao, et al. Beyond Model-Level Membership Privacy Leakage: An Adversarial Approach in Federated Learning[C]// IEEE. 29th International Conference on Computer Communications and Networks. New York: IEEE, 2020: 1-9.
[9]	PICHLER G, ROMANELLI M, VEGA L R, et al. Perfectly Accurate Membership Inference by a Dishonest Central Server in Federated Learning[EB/OL]. (2022-03-30)[2022-10-18]. https://arxiv.org/ftp/arxiv/papers/2203/2203.16463.pdf.
[10]	HILPRECHT B, HRTERICH M, BERNAU D. Reconstruction and Membership Inference Attacks Against Generative Models[EB/OL]. (2019-06-07)[2022-10-18]. https://arxiv.org/pdf/1906.03006.pdf.
[11]	CHEN Dingfan, YU Ning, ZHANG Yang, et al. GAN-Leaks: A Taxonomy of Membership Inference Attacks Against Generative Models[C]// ACM. Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2020: 343-362.
[12]	HU Hailong, PANG Jun. Membership Inference Attacks Against GANs by Leveraging Over-representation Regions[C]// ACM. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2021: 2387-2389.
[13]	LIU Hongbin, JIA Jinyuan, QU Wenjie, et al. EncoderMI: Membership Inference Against Pre-Trained Encoders in Contrastive Learning[C]// ACM. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2021: 2081-2095.
[14]	LI Jiacheng, LI Ninghui, RIBEIRO B. Membership Inference Attacks and Defenses in Classification Models[C]// ACM. The Eleventh ACM Conference on Data and Application Security and Privacy. New York: ACM, 2021: 5-16.
[15]	LONG Yunhui, BINDSCHAEDLER V, WANG Lei, et al. Understanding Membership Inferences on Well-Generalized Learning Models[EB/OL]. (2018-02-13)[2022-10-18]. https://arxiv.org/pdf/1802.04889.pdf.
[16]	REZAEI S, LIU Xin. On the Difficulty of Membership Inference Attacks[C]// IEEE. Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 7892-7900.

环境及实验配置	对应值
CPU	Intel(R) Core(TM) i7-11700 @2.50 GHz(16 CPUs)
内存	16 GB
显卡	NVIDIA Tesla T4
数据集	CIFAR10、CIFAR100、STL10
目标模型结构	CNN、Vgg19、Resnet18

混淆矩阵		预测值
混淆矩阵		1	0
真实值	1	TP	FN
真实值	0	FP	TN

模型表现				基于阈值的成员推理攻击准确率
数据集	模型结构	训练准确度	测试准确度	${{A}_{\text{correct}}}$[4]	${{A}_{\text{max}}}$[2]	${{A}_{\text{entropy}}}$[2]	${{A}_{\text{double}}}$
CIFAR10	CNN	100%	77.55%	77.84%	74.84%	80.49%	75.88%
	VGG19	100%	92.69%	73.52%	73.64%	74.13%	74.30%
	Resnet18	100%	88.72%	74.65%	79.38%	76.53%	79.83%
CIFAR100	CNN	99.97%	43.74%	87.49%	79.23%	90.05%	81.74%
	VGG19	99.98%	71.46%	79.57%	79.74%	80.80%	79.73%
	Resnet18	99.98%	67.95%	80.57%	91.25%	89.41%	93.84%
STL10	CNN	100%	58.38%	71.24%	63.78%	69.22%	68.68%
	VGG19	100%	86.23%	56.92%	66.08%	56.94%	66.12%
	Resnet18	100%	77.50%	60.84%	70.74%	63.38%	70.90%
平均值	—	—	—	73.63%	75.41%	75.66%	76.78%

模型表现		基于阈值的成员推理攻击召回率
数据集	模型结构	${{A}_{\text{correct}}}$[4]	${{A}_{\text{max}}}$[2]	${{A}_{\text{entropy}}}$[2]	${{A}_{\text{double}}}$
CIFAR10	CNN	100%	95.78%	99.94%	97.57%
	VGG19	100%	98.26%	99.96%	98.38%
	Resnet18	100%	99.45%	100%	98.84%
CIFAR100	CNN	99.98%	95.25%	99.52%	95.93%
	VGG19	99.98%	98.21%	99.92%	98.44%
	Resnet18	99.98%	99.76%	99.93%	99.28%
STL10	CNN	100%	82.48%	100%	86.68%
	VGG19	100%	89.96%	100%	94.48%
	Resnet18	100%	93.28%	99.96%	92.92%

模型表现		基于阈值的成员推理攻击精确率和 F1分数
数据集	模型结构	${{A}_{\text{correct}}}$[4]		${{A}_{\max }}$[2]		${{A}_{\text{entropy}}}$[2]		${{A}_{\text{double}}}$
数据集	模型结构	精确率	F1 分数	精确率	F1 分数	精确率	F1 分数	精确率	F1 分数
CIFAR10	CNN	76.32%	0.866	74.85%	0.847	78.57%	0.879	76.93%	0.853
	VGG19	72.95%	0.844	74.17%	0.845	73.70%	0.847	74.11%	0.846
	Resnet18	73.81%	0.849	77.84%	0.873	75.27%	0.859	78.39%	0.874
CIFAR100	CNN	85.12%	0.920	79.66%	0.868	88.09%	0.935	81.05%	0.879
	VGG19	77.77%	0.874	78.70%	0.874	78.85%	0.881	78.59%	0.874
	Resnet18	78.63%	0.880	89.13%	0.942	89.41%	0.944	92.61%	0.958
STL10	CNN	63.48%	0.777	60.03%	0.695	61.90%	0.765	63.74%	0.738
	VGG19	53.72%	0.699	60.52%	0.726	53.73%	0.699	60.29%	0.737
	Resnet18	56.08%	0.719	64.30%	0.761	57.73%	0.732	64.58%	0.762