人工智能时代面向运维数据的异常检测技术研究与分析

doi:10.3969/j.issn.1671-1122.2019.11.004

摘要/Abstract

摘要： 随着信息化时代的到来以及人工智能技术在各个领域的落地及实践,IT运维也迎来一个智能化运维的新时代。为了确保大型软硬件系统安全、可靠地运行,需要有专业的运维人员进行系统的部署、运行和维护。运维数据是一系列与大型软硬件系统运行状态相关的参数。运维数据异常检测技术旨在检测大型系统的健康状态,并把运维人员从纷繁复杂的告警和噪声中解放出来。但是,标记数据的稀缺以及企业对准确率的高要求等问题给运维数据异常检测技术的实际应用带来了严峻挑战。文章对运维数据异常进行了描述,并详细介绍了运维数据异常检测的研究现状;在此基础上提出了一种初步的解决方案并给出了实验结果。文章阐述了运维数据异常检测的潜在问题和可能的发展方向,力求为运维数据异常检测技术的发展提供可行的研究思路。

关键词: 人工智能, 运维数据, 异常检测, 深度学习

Abstract: With the arrival of the information age and the landing and practice of artificial intelligent technology in various fields, IT operation and maintenance ushers in a new era of IT intelligent operations and maintenance. In order to ensure the safe and reliable operation of large-scale hardware and software system, it is necessary to have professional operation and maintenance personnel to deploy, operate and maintain the system. Operation and maintenance data is a series of parameters related to the running state of large-scale hardware and software system. The anomaly detection technology of operation and maintenance data is designed to detect the health status of large-scale system and free operation and maintenance personnel from complicated alarms and noises. However, the scarcity of labeled data and the high requirements of enterprises for accuracy bring severe challenges to the practical application of operation and maintenance data anomaly detection technology. This paper describes the abnormal operation and maintenance data, and introduces the research status of operation and maintenance data anomaly detection in detail. On this basis, this paper presents a preliminary solution and gives experimental results. This paper expounds the potential problems and possible development directions of operation and maintenance data anomaly detection, and tries to provide feasible research ideas for the development of operation and maintenance data anomaly detection technology.

Key words: artificial intelligence, operations data, anomaly detection, deep learning

中图分类号:

TP309

朱海麒, 姜峰. 人工智能时代面向运维数据的异常检测技术研究与分析[J]. 信息网络安全, 2019, 19(11): 24-35.

ZHU Haiqi, JIANG Feng. Research and Analysis of Anomaly Detection Technology for Operation and Maintenance Data in the Era of Artificial Intelligence[J]. Netinfo Security, 2019, 19(11): 24-35.

参考文献

[1] PEI Dan, ZHANG Shenglin, PEI Changhua.Intelligent Operation and Maintenance Based on Machine Learning[J]. Communications of CCF, 2017, 13(12): 67-73.
裴丹,张圣林,裴昶华.基于机器学习的智能运维[J].中国计算机学会通讯,2017,13(12):67-73.
[2] LIU Dapeng, ZHAO Youjian, XU Haowen, et al.Opprentice: Towards Practical and Automatic Anomaly Detection through Machine Learning[C]//ACM. 15th Internet Measurement Conference, October 28-30, 2015, Tokyo, Japan. New York: ACM, 2015: 211-224.
[3] LAPTEV N, AMIZADEH S, FLINT I.Generic and Scalable Framework for Automated Time-series Anomaly Detection[C]//ACM. 21th ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, August 10-13, 2015, Sydney, Australia. New York: ACM, 2015: 1939-1947.
[4] HAWKINS S, HE Hongxing, WILLIAMS G J, et al.Outlier Detection Using Replicator Neural Networks[M]//Springer. Data Warehousing and Knowledge Discovery. Heidelberg: Springer-Verlag, 2002: 170-180.
[5] CHANDOLA V, BANERJEE A, KUMAR V.Anomaly Detection: A Survey[J]. ACM Computing Surveys, 2009, 41(3): 1-58.
[6] KUMAR V.Parallel and Distributed Computing for Cybersecurity[J]. IEEE Distributed Systems Online, 2005, 6(10): 1.
[7] SPENCE C, PARRA L, SAJDA P.Detection, Synthesis and Compression in Mammographic Image Analysis with a Hierarchical Image Probability Model[C]//IEEE. Mathematical Methods in Biomedical Image Analysis, December 9-10, 2001, Kauai, HI, USA. NJ: IEEE, 2001: 3.
[8] FUJIMAKI R, YAIRI T, MACHIDA K.An Approach to Spacecraft Anomaly Detection Problem Using Kernel Feature Space[C]//ACM. 11th ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, August 21-24, 2005, Chicago, lllinoism, USA. New York: ACM, 2005: 401-410.
[9] ADEREMI O, ANDRONICUS A A.A Survey of Machine-learning and Nature-inspired Based Credit Card Fraud Detection Techniques[J]. International Journal of System Assurance Engineering and Management, 2017, 8(2): 937-953.
[10] KWON D, KIM H, KIM J, et al.A Survey of Deep Learning-based Network Anomaly Detection[J]. Cluster Computing, 2019, 22(1): 949-961.
[11] LITJENS G, KOOI T, BEJNORDI B E, et al.A Survey on Deep Learning in Medical Image Analysis[J]. Medical Image Analysis, 2017, 42: 60-88.
[12] MOHAMMADI M, ALA A F, SAMEH S, et al.Deep Learning for IoT Big Data and Streaming Analytics: A Survey[J]. IEEE Communications Surveys & Tutorials, 2018, 1: 1-40.
[13] BALL J E, DEREK T A, CHAN C S.Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools, and Challenges for the Community[J]. Journal of Applied Remote Sensing, 2017, 11(4): 1-64.
[14] KIRAN B, DILIP T, RANJITH P.An Overview of Deep Learning Based Methods for Unsupervised and Semi-supervised Anomaly Detection in Videos[J]. Journal of Imaging, 2018, 4(2): 36-42.
[15] NICK D, HAFFNER P, KRISHNAMURTHY E, et al.Rule-based Anomaly Detection on IP Flows[C]//IEEE. 28th IEEE International Conference on Computer Communications, April 19-25, 2009, Rio de Janeiro, Brazil. NJ: IEEE, 2009: 424-432.
[16] LI Lei, YANG Dezhang, SHEN Fangcheng.A Novel Rule-based Intrusion Detection System Using Data Mining[C]//IEEE. Computer Science and Information Technology(ICCSIT), 2010 3rd IEEE International Conference on, July 9-11, 2010, Chengdu, China. NJ: IEEE. 2010: 169-172.
[17] KHAN M S A. Rule-based Network Intrusion Detection Using Genetic Algorithm[J]. International Journal of Computer Applications, 2011, 18(8): 26-29.
[18] ANSCOMBE F J.Rejection of Outliers[J]. Technometrics, 1960, 2(2): 123-146.
[19] SOULE A, SALAMATIAN K, TAFT N.Combining Filtering and Statistical Methods for Anomaly Detection[C]//ACM. 5th ACM SIGCOMM Conference on Internet Measurement, August 22-26, 2005, Philadelphia, PA. New York: ACM, 2005: 1-14.
[20] PINCOMBE B.Anomaly Detection in Time Series of Graphs Using ARMA Processes[J]. Asor Bulletin, 2005, 24(1): 67-75.
[21] XU Chengcheng, LI Zhibin, WANG Wei.Short-term Traffic Flow Prediction Using a Methodology Based on Autoregressive Integrated Moving Average and Genetic Programming[J]. Transport, 2016, 31(3): 343-358.
[22] SHI Yantao.Holter-winter Forecast Model of Inflow Runoff of Dahuofang Reservoir: Also a Discussion with Fu Wenyi[J]. Journal of Liaodong University(Natural Sciences), 2017, 24(2): 147-153.
石岩涛. 大伙房水库入库径流的Holter-winter预报模型——兼与付文艺老师商榷[J]. 辽东学院学报(自然科学版),2017,24(2):147-153.
[23] CASTRO-NETO M, JEONG Y S, JEONG M K, et al.Online-svr for Short-term Traffic Flow Prediction under Typical and Atypical Traffic Conditions[J]. Expert Systems with Applications. 2009, 36: 164-173.
[24] GOLDSTEIN M, DENGEL A.Histogram-based Outlier Score(HBOS): A Fast Unsupervised Anomaly Detection Algorithm[C]//IEEE. Poster and Demo Track 35th German Conference on Artificial Intelligence(KI2012), September 24-25, 2012, Saarbrücken, Germany. NJ: IEEE, 2012: 59-63.
[25] LAPTEV N, AMIZADEH S, FLINT I.Generic and Scalable Framework for Automated Time-series Anomaly Detection[C]//ACM. 21th ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, August 10-13, 2015, Sydney, Australia. New York: ACM, 2015: 1939-1947.
[26] KHREICH W, KHOSRAVIFAR B, HAMOU-LHADJ A, et al.An Anomaly Detection System Based on Variable N-gram Features and One-class SVM[J]. Information and Software Technology, 2017, 91: 186-197.
[27] WANG Guiping, YANG Jianxi, LI Ren.Imbalanced SVM-based Anomaly Detection Algorithm for Imbalanced Training Datasets[J]. Etri Journal, 2017, 39(5): 621-631.
[28] YANG Xingwei, LATECKI L J, POKRAJAC D.Outlier Detection with Globally Optimal Exemplar-based GMM[C]//SDM. International Conference on Data Mining, April 30 - May 2, 2009, Sparks, Nevada, USA. New York: SDM, 2009: 145-154.
[29] RAMASWAMY S, RASTOGI R, SHIM K.Efficient Algorithms for Mining Outliers from Large Data Sets[C]//ACM. SIGMOD International Conference on Management of Data. May 16-18, 2000, Dallas, Texas, USA. New York: ACM, 2000: 427-438.
[30] TANG Jian, CHEN Zhixiang, FU A W, et al.Enhancing Effectiveness of Outlier Detections for Low Density Patterns[M]// Springer. Advance in Knowledge Discovery and Data Mining. Heidelberg: Springer-Verlag, 2002: 535-548.
[31] JIN Wen, TUNG A K H, HAN Jiawei, et al. Ranking Outliers Using Symmetric Neighborhood Relationship[M]// Springer. Advances in Knowledge Discovery and Data Mining. Heidelberg: Springer-Verlag, 2006: 577-593.
[32] LIU F T, TING Kaiming, ZHOU Zhihua.Isolation-based Anomaly Detection[J]. ACM Transactions on Knowledge Discovery from Data, 2012, 6(1): 1-39.
[33] ARYAL S, TING Kaiming, WELLS J R, et al.Improving iForest with Relative Mass[M]//Springer. Advances in Knowledge Discovery and Data Mining. Cham: Springer, 2014: 510-521.
[34] JAVAID A Y, NIYAZ Q, SUN Weiqing, et al.A Deep Learning Approach for Network Intrusion Detection System[C]//ACM. 9th EAI International Conference on Bio-inspired Information and Communications Technologies, December 3- 5, 2015, New York City, United States. New York: ACM, 2015: 21-26.
[35] PENG Huankai, MARCULESCU R.Multi-scale Compositionality: Identifying the Compositional Structures of Social Dynamics Using Deep Learning[J]. PloS one, 2015, 10(4): 45-57.
[36] CHALAPATHY R, BORZESHI E Z, PICCARDI M. An Investigation of Recurrent Neural Architectures for Drug Name Recognition[EB/OL]. https://www.aclweb.org/anthology/W16-6101/, 2019-4-28.
[37] CORREA B A. Building AI Applications using Deep Learning[EB/OL]. https://blog.easysol.net/wp-content/uploads/2017/06/image1.png, 2019-4-28.
[38] WULSIN D, BLANCO J, MANI R, et al.Semi-supervised Anomaly Detection for EEG Waveforms Using Deep Belief Nets[C]//IEEE. 2010 9th International Conference on Machine Learning and Applications, December 18-21, 2011, Honolulu, USA. NJ: IEEE, 2011: 436-441.
[39] NADEEM M, MARSHALL O, SINGH S, et al.Semi-supervised Deep Neural Network for Network Intrusion Detection[C]//IEEE. ICACCI 2017 Sixth International Conference on Advances in Computing, Communications and Informatics, September 13-16, 2017, Manipal, India. NJ: IEEE, 2016: 454-467.
[40] SONG Hongchao, JIANG Zhuqing, MEN Aidong.A Hybrid Semi-supervised Anomaly Detection Model for High-dimensional Data[J]. Computational Intelligence and Neuroscience, 2017, 17: 1-9.
[41] PATTERSON J, GIBSON A.Deep Learning: A Practitioner’s Approach[M]. Sebastopol: O’Reilly Media, Inc., 2017.
[42] TUOR A, KAPLAN S, HUTCHINSON B, et al. Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams[EB/OL]. https://arxiv.org/abs/1710.00811, 2019-4-28.
[43] WOLD S, ESBENSEN K, GELADI P.Principal Component Analysis[J]. Chemometrics and Intelligent Laboratory Systems, 1987, 2(1-3): 37-52.
[44] CORTES C, VAPNIK V.Support-vector Networks[J]. Machine Learning, 1995, 20(3): 273-297.
[45] LIU F T, TING Kaiming, ZHOU Zhihua.Isolation Forest[C]//IEEE. The 2008 Eighth IEEE International Conference on Data Mining, December 15- 19, 2008, Pisa, Italy. NJ: IEEE, 2008: 413-422.
[46] SUTSKEVER I, HINTON G E, TAYLOR G W.The Recurrent Temporal Restricted Boltzmann Machine[J]. Advances in Neural Information Processing Systems, 2008, 20: 1601-1608.
[47] SALAKHUTDINOV R, LAROCHELLE H.Efficient Learning of Deep Boltzmann Machines[C]//IEEE. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, May 13-15, 2010, Sardinia, Italy. NJ: IEEE, 2010: 693-700.
[48] VINCENT P, LAROCHELLE H, BENGIO Y, et al.Extracting and Composing Robust Features with Denoising Autoencoders[C]//ACM. The 25th International Conference on Machine Learning, July 5- 9, 2008, Helsinki, Finland. New York: ACM, 2008: 1096-1103.
[49] RODRIGUEZ P, WILES J, ELMAN J L.A Recurrent Neural Network that Learns to Count[J]. Connection Science, 1999, 11(1): 5-40.
[50] LAMPLE G, BALLESTEROS M, SUBRAMANIAN S, et al. Neural Architectures for Named Entity Recognition[EB/OL]. https://arxiv.org/abs/1603.01360, 2019-4-28.
[51] FARHADI A, HEJRATI M, SADEGHI M A, et al.Every Picture Tells a Atory: Generating Sentences from Images[J]. Lecture Notes in Computer Science, 2010, 21(10): 15-29.
[52] RAKTHANMANON T, CAMPANA B, MUEEN A, et al.Keogh Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping[C]//ACM. 18th ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, August 12-16, 2012, Beijing, China. New York: ACM, 2012: 262-270.
[53] PEI Jian, HAN Jiawei, LAKSHMANAN L V S. Mining Frequent Item Sets with Convertible Constraints[C]//IEEE. 17th International Conference on Data Engineering, April 2-6, 2001, Heidelberg, Germany. NJ: IEEE, 2001: 433-442.
[54] PEI Jian, HAN Jiawei, MORTAZAVIASL B, et al.PrefixSpan: Mining Sequential Patterns Efficiently by Prefix-projected Pattern Growth[C]//IEEE. International Conference on Data Engineering, April 2-6, 2001, Heidelberg, Germany. NJ: IEEE, 2001: 215.
[55] LIU F T, TING Kaiming, ZHOU Zhihua.Isolation Forest[C]//IEEE. 8th IEEE International Conference on Data Mining, December 6-9, 2009, Miami, Florida, USA. NJ: IEEE, 2009: 413-422.
[56] Breunig M M, KRIEGEL H P, NG R T, et al.LOF: Identifying Density-based Local Outliers[J]. Acm Sigmod Record, 2000, 29(2): 93-104.
[57] PANG Guansong, CAO Longbing, CHEN Ling.Outlier Detection in Complex Categorical Data by Modeling the Feature Value Couplings[C]//IJCAI. International Joint Conference on Artificial Intelligence, July 9-15, 2016, New York, USA. New York: ACM, 2016: 1902-1908.
[58] MA Minghua, ZHANG Shenglin, PEI Dan, et al.Robust and Rapid Adaption for Concept Drift in Software System Anomaly Detection[C]//IEEE. 29th IEEE International Symposium on Software Reliability Engineering ISSRE 2018, October 15-18, 2018, Memphis, TN, USA. NJ: IEEE, 2018: 13-24.
[59] DING Rui, WANG Qiang, DANG Yingnong.YADING: Fast Clustering of Large-scale Time Series Data[C]//IEEE. International Conference on Very Large Data Bases, August 31, 2015, Kohala Coast, Hawaii, USA. NJ: IEEE, 2015: 154-165.
[60] SHERVASHIDZE N, BACH F.Learning the Structure for Structured Sparsity[J]. IEEE Transactions on Signal Processing, 2015, 63(18): 4894-4902.
[61] ZHANG Quanshi, ZHU Songchun.Visual Interpretability for Deep Learning: A survey[J]. Frontiers of Information Technology ＆ Electronic Engineering, 2018, 19(1): 27-39.
[62] RIBEIRO M T, SINGH S, GUESTRIN C.Explaining the Predictions of Any Classifier[C]//ACM. 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, August 24-27, 2016, San Francisco, USA. New York: ACM, 2016: 1135-1144.