Research on Attacks and Defenses towards Machine Learning Systems

doi:10.3969/j.issn.1671-1122.2018.09.002

Abstract

Abstract:

Recent research shows that almost all stages of machine learning system pipeline may encounter data contamination attack, attack on learning algorithm and dependency library, escape attack, model theft and model reasoning attack. These attacks not only affect the learning process of the machine learning system, but also may affect the performance of the model or make the model appear the errors which attackers want the model to appear under specific input, thus affecting the accuracy of the model. So, understanding the security of machine learning algorithms and systems, and exploring their security improvements, has increasingly become a research direction in cross field of computer security and machine learning. This paper firstly defines the machine learning system pipeline, and then studies the possible attacks and potential solutions on the pipeline. Finally, this paper summarizes the full text and prospects the future research directions.

Key words: machine learning, data contamination, escape attack, model reasoning

CLC Number:

TP309

Yingchao YU, Lin DING, Zuoning CHEN. Research on Attacks and Defenses towards Machine Learning Systems[J]. Netinfo Security, 2018, 18(9): 10-18.

Figures/Tables 2

References 42

[1]	BIGGIO B, NELSON B, LASKOV P.Poisoning Attacks against Support Vector Machines[C]//ACM. The 29th International Conference on International Conference on Machine Learning, June 26-July 1, 2012, Edinburgh, Scotland.New York:ACM,2012:1467-1474.
[2]	KEARNS M, LI Ming.Learning in the Presence of Malicious Errors[C]//ACM. the Twentieth Annual ACM Symposium on Theory of Computing, May 2 - 4, 1988, Chicago, Illinois, USA.New York:ACM,1988:267-280.
[3]	The Guardian. Microsoft 'deeply sorry' for Racist and Sexist Tweets by AI Chatbot[EB/OL] . ,2018-6-30.
[4]	CHEN Xinyun, LIU Chang, LI Bo, et al.Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning[EB/OL]. 2018-6-30.
[5]	HUANG Ling, JOSEPH A D, NELSON B, et al.Adversarial Machine Learning[C]//ACM. The 4th ACM Workshop on Security and Artificial Intelligence, October 21, 2011,Chicago, USA.New York: ACM, 2011: 43-58.
[6]	RUBINSTEIN B I P, NELSON B, HUANG Ling, et al. ANTIDOTE: Understanding and Defending against Poisoning of Anomaly Detectors[C]//ACM. The 9th ACM Sigcomm Internet Measurement Conference, November 4-6, 2009,Chicago,USA. New York: ACM, 2009: 1-14.
[7]	BARRENO M, NELSON B, SEARS R, et al.Can Machine Learning Be Secure[C]//ACM. The 2006 ACM Symposium on Information, Computer and Communications Security, March 21 - 24, 2006, Taipei,China. New York: ACM, 2006: 16-25.
[8]	BIGGIO B, FUMERA G, ROLI F.Multiple Classifier Systems for Robust Classifier Design in Adversarial Environments[J]. Inter- national Journal of Machine Learning and Cybernetics, 2010, 1(1-4): 27-41.
[9]	BIGGIO B, CORONA I, FUMERA G, et al.Bagging Classifiers for Fighting Poisoning Attacks in Adversarial Classification Tasks[C]//LNCS. The 10th International Conference on Multiple Classifier Systems, June 15-17, 2011, Naples, Italy. Heidelberg: Springer-Verlag Berlin, 2011: 350-359.
[10]	Google Cloud Platform.CLOUD AI[EB/OL]. ,2018-6-30.
[11]	aws.Amazon machine learning[EB/OL].,2018-6-30.
[12]	BigML[EB/OL]., 2018-6-30.
[13]	SONG Congzheng, RISTENPART T,SHMATIKOV V.Machine Learning Models that RememberToo Much[C]//ACM. The 2017 ACM Sigsac Conference on Computer and Communications Security, October 30 - November 3, 2017, Dallas, Texas, USA. New York:ACM,2017:587-601.
[14]	HUNT T, SONG Congzheng, SHOKRI R, et al.Chiron: Privacy-preserving Machine Learning as a Service[J]. Proceedings on Privacy Enhancing Technologies, 2018(3):123-142.
[15]	OHRIMENKO O, SCHUSTER F, FOURNET C, et al.Oblivious Multi-party Machine Learning on Trusted Processors[EB/OL].,2016-7-13.
[16]	SZEGEDY C,ZAREMBA W, SUTSKEVER I, J.et al.Intriguing Properties of Neural Networks[EB/OL]..
[17]	SHARIF M, BHAGAVATULA S, BAUER L, et al.Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition[C]//ACM. The 2016 ACM Sigsac Conference on Computer and Communications Security, October 24 - 28, 2016, Vienna, Austria. New York: ACM, 2016:1528-1540.
[18]	GROSSE K, PAPERNOT N, MANOHARAN P, et al.Adversarial Perturbations Against Deep Neural Networks for Malware Classification[EB/OL]. , 2018-6-30.
[19]	PAPERNOT N, ABADI M, Erlingsson Ú, et al. Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data[EB/OL].,2018-6-30.
[20]	PRANAV R, ZHANG Jian, KONSTANTIN L, et al.Squad: 100,000+ Questions for Machine Comprehension of Text[EB/OL]. , 2016-6-16.
[21]	EYKHOLT K, EVTIMOV I, FERNANDES E,et al. Robust Physical-world Attacks on Deep Learning Models[EB/OL]. .
[22]	SZEGEDY C, ZAREMBA W, SUTSKEVER I,et al. Intriguing Properties of Neural Networks[EB/OL]. .
[23]	GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and Harnessing Adversarial Examples[EB/OL]. .
[24]	LU Jiajun, ISSARANON T, Forsyth D.Safetynet: Detecting and Rejecting
	Adversarial Examples Robustly[C]//IEEE. 2017 IEEE International Conference on Computer Vision, October 22-29, 2017, Venice, Italy .NJ:IEEE,2017:446-454.
[25]	METZEN J H, GENEWEIN T, FISCHER V, et al.On Detecting Adversarial Perturbations[EB/OL]. ,2017-2-21.
[26]	PAPERNOT N, MCDANIEL P, WU Xi, et al.Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks[C]//IEEE. 2016 IEEE Symposium on Security and Privacy (SP), May 22-26, 2016, San Jose, CA, USA.NJ:IEEE,2016: 582-597.
[27]	HUANG Ruitong, XU Bing, SCHUURMANS Dale, et al. Learning with a Strong Adversary[EB/OL]. 2016-1-16.
[28]	TRAMÈR F, KURAKIN A, PAPERNOT N, et al. Ensemble Adversarial Training: Attacks and Defenses[EB/OL].,2017-5-19.
[29]	TRAMER F,ZHANG Fan, JUELS A, et al.Stealing Machine Learing Models via Prediction APIs[EB/OL]. .
[30]	DANG H,HUANG Y,Chang E C.Evading Classifiers by Morphing in the Dark[EB/OL]. .
[31]	PAPERNOT N, MCDANIEL P, GOODFELLOW I, et al.Practical Black-box Attacks against Machine Learning[C]// ACM. The 2017 ACM on Asia Conference on Computer and Communications Security, April 2 - 6, 2017, Abu Dhabi, United Arab Emirates. New York:ACM,2017:506-519.
[32]	SHOKRI R, STRONATI M, SONG Congzheng, et al.Membership Inference Attacks against Machine Learning Models[C]//IEEE. 2017 IEEE Symposium on Security and Privacy (SP), May 22-26, 2017, San Jose, CA, USA.NJ: IEEE, 2017: 3-18.
[33]	ABADI M, CHU A, GOODFELLOW I, et al.Deep Learning with Differential Privac[C]//ACM. the2016 ACM Sigsac Conference on Computer and Communications Security,October 24-28, 2016 Vienna,Austria.New York: ACM, 2016:308-318.
[34]	PAPERNOT N, MCDANIEL P, JHA S, et al. The Limitations of Deep Learning in Adversarial Settings [C]//IEEE.2016 IEEE European Symposium on Security and Privacy(EuroS&P), March 21-24,2016, Saarbrucken,Germany.March 21-24, 2016, Saarbrucken, Germany.NJ:IEEE, 2016:372-387.
[35]	FREDRIKSON M,JHA S,RISTENPART T.Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures[C]//ACM. The 22nd ACM Sigsac Conference on Computer and Communications Security, October 12 - 16, 2015, Denver, Colorado, USA. New York:ACM,2015:1322-1333.
[36]	ERKIN Z, VEUGEN T, TOFT T, et al.Generating Private Recommendations Efficiently Using Homomorphic Encryption and Data Packing[J].IEEE. IEEE Transactions on Information Forensics and Security, 2012,7(3): 1053-1066.
[37]	DOWLIN N, GILAD-BACHRACH R, LAINE K, et al.CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy[EB/OL].,2018-6-30.
[38]	XIE Pengtao, BILENKO M, FINLEY T, et al. Crypto-nets: Neural Networks over Encrypted Data[EB/OL]. , 2014-11-24.
[39]	ERLINGSSON Ú, PIHUR V, KOROLOVA A.RAPPOR: Randomized Aggregatable Privacy- preserving Ordinal Response[C]//ACM. The 2014 ACM Sigsac Conference on Computer and Communications Security, November 3-7, 2014, Scottsdale, Arizona, USA. New York: ACM, 2014: 1054-1067.
[40]	anquanke.2017 AI White Paper on Security Risks[EB/OL].
	安全客. 2017年AI安全风险白皮书[EB/OL].
[41]	GU Tianyu, DOLAN-GAVITT B, GARG S. Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain[EB/OL]. .
[42]	CARLINI N, WAGNER D.Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods[C]//ACM. The 10th ACM Workshop on Artificial Intelligence and Security, November 3 , 2017, Dallas, Texas, USA. New York:ACM,2017:3-14.