一种融合变量的日志异常检测方法

doi:10.3969/j.issn.1671-1122.2023.10.003

信息网络安全 ›› 2023, Vol. 23 ›› Issue (10): 16-20.doi: 10.3969/j.issn.1671-1122.2023.10.003

一种融合变量的日志异常检测方法

张玉臣, 李亮辉, 马辰阳, 周洪伟()

中国人民解放军信息工程大学密码工程学院，郑州 450001

收稿日期:2023-06-04 出版日期:2023-10-10 发布日期:2023-10-11
通讯作者: 周洪伟 E-mail:hong_wei_zhou@126.com
作者简介:张玉臣（1977—），男，河南，教授，博士，主要研究方向为保密管理|李亮辉（2000—），男，河北，本科，主要研究方向为保密管理|马辰阳（2000—），男，河南，本科，主要研究方向为网络信息防御|周洪伟（1979—），男，重庆，副教授，博士，主要研究方向为网络信息安全
基金资助:
国家自然科学基金(61902427)

A Log Anomaly Detection Method with Variables

ZHANG Yuchen, LI Lianghui, MA Chenyang, ZHOU Hongwei()

Department of Cryptographic Engineering, Information Engineering University of PLA, Zhengzhou 450001, China

Received:2023-06-04 Online:2023-10-10 Published:2023-10-11

摘要/Abstract

摘要：

为了充分挖掘日志中变量的潜能，优化日志异常检测效果，文章提出一种融合变量的日志异常检测方法SiEv。首先，该方法可以识别主体变量，并根据主体变量将日志划分为不同片段；然后，SiEv以这些日志片段为输入，基于长短期记忆网络（Long Short-Term Memory，LSTM）训练或检测异常，从而避免不同主体的日志序列特征相互干扰；最后，根据日志片段将SiEv划分为多个类别，从不同角度检测日志。为了验证文章所提方法的有效性，SiEv对Loghub所提供的日志数据集进行测试。实验结果表明，SiEv能够发现多种类型日志中存在的异常，识别同一主体的活动行为模式和变化趋势。

关键词: 日志, 异常检测, LSTM, 变量

Abstract:

In order to fully tap the potential of variables in logs and optimize the effectiveness of log anomaly detection, this paper proposed a novel log anomaly detection method SiEv with the variables. Firstly, this method identified the subject variable in the log, and divided the log into different fragments based on the subject variable. Then, SiEv took these fragments as input for LSTM to avoid mutual interference between log sequence features of different subjects. Finally, according to different log fragments, SiEv was able to be divided into multiple categories to detect logs with the view of different perspectives. To verify the effectiveness of the method, SiEv was tested with the log dataset provided by the Loghub. The experimental results indicate that SiEv is able to detect anomalies in various types of logs, identify the activity behavior patterns and trends of the same subject.

Key words: log, anomaly detection, LSTM, variables

中图分类号:

TP309

张玉臣, 李亮辉, 马辰阳, 周洪伟. 一种融合变量的日志异常检测方法[J]. 信息网络安全, 2023, 23(10): 16-20.

ZHANG Yuchen, LI Lianghui, MA Chenyang, ZHOU Hongwei. A Log Anomaly Detection Method with Variables[J]. Netinfo Security, 2023, 23(10): 16-20.

图/表 4

参考文献 21

[1]	HE Shilin, HE Pinjia, CHEN Zhuangbin, et al. A Survey on Automated Log Analysis for Reliability Engineering[J]. ACM Computing Surveys (CSUR), 2021, 54(6): 1-37.
[2]	OLINER A, GANAPATHI A, XU Wei. Advances and Challenges in Log Analysis[J]. Communications of the ACM, 2012, 55(2): 55-61.
[3]	LIAO H J, LIN C H R, LIN Y C, et al. Intrusion Detection System: A Comprehensive Review[J]. Journal of Network and Computer Applications, 2013, 36(1): 16-24. doi: 10.1016/j.jnca.2012.09.004 URL
[4]	LANDAUER M, SKOPIK F, WURZENBERGER M, et al. System Log Clustering Approaches for Cyber Security Applications: A Survey[EB/OL]. (2020-05-01) [2023-05-30]. https://www.sciencedirect.com/science/article/pii/S0167404820300250.
[5]	HE Shilin, ZHU Jieming, HE Pinjia, et al. Experience Report: System Log Analysis for Anomaly Detection[C]// IEEE. 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). New York: IEEE, 2016: 207-218.
[6]	KRUEGEL C, VIGNA G. Anomaly Detection of Web-Based Attacks[C]// ACM. Proceedings of the 10th ACM Conference on Computer and Communications Security. New York: ACM, 2003: 251-261.
[7]	LANDAUER M, WURZENBERGER M, SKOPIK F, et al. Dynamic Log File Analysis: An Unsupervised Cluster Evolution Approach for Anomaly Detection[J]. Computers & Security, 2018(79): 94-116.
[8]	DU Min, LI Feifei, ZHENG Guineng, et al. Deeplog: Anomaly Detection and Diagnosis from System Logs Through Deep Learning[C]// ACM. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2017: 1285-1298.
[9]	MENG Weibin, LIU Ying, ZHU Yichen, et al. LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs[C]// ACM. Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI). New York: ACM, 2019: 4739-4745.
[10]	YANG Ruipeng, QU Dan, GAO Ying, et al. nLSALog: An Anomaly Detection Framework for Log Sequence in Security Management[J]. IEEE Access, 2019(7): 181152-181164.
[11]	ZHU Bei, LI Jing, GU Rongbin, et al. An Approach to Cloud Platform Log Anomaly Detection Based on Natural Language Processing and LSTM[C]// ACM. 2020 3rd International Conference on Algorithms, Computing and Artificial Intelligence. New York: ACM, 2020: 1-7.
[12]	LANDAUER M, ONDER S, SKOPIK F, et al. Deep Learning for Anomaly Detection in Log Data: A Survey[EB/OL]. (2023-05-15) [2023-05-30]. https://arxiv.org/abs/2207.03820.
[13]	LE V H,ZHANG Hongyu. Log-Based Anomaly Detection without Log Parsing[C]// IEEE. 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). New York: IEEE, 2021: 492-504.
[14]	HE Shilin, ZHU Jieming, HE Pinjia, et al. Loghub: A Large Collection of System Log Datasets Towards Automated Log Analytics[EB/OL]. (2020-08-14) [2023-05-30]. https://arxiv.org/abs/2008.06448.
[15]	XU Wei, HUANG Ling, FOX A, et al. Detecting Large-Scale System Problems by Mining Console Logs[C]// ACM. Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles. New York: ACM, 2009: 117-132.
[16]	HAN Shangbin, WU Qianhong, ZHANG Han, et al. Log-Based Anomaly Detection with Robust Feature Extraction and Online Learning[J]. IEEE Transactions on Information Forensics and Security, 2021(16): 2300-2311.
[17]	CAO Qimin, QIAO Yinrong, LYU Zhong. Machine Learning to Detect Anomalies in Web Log Analysis[C]// IEEE. 2017 3rd IEEE International Conference on Computer and Communications (ICCC). New York: IEEE, 2017: 519-523.
[18]	YING Shi, WANG Bingming, WANG Lu, et al. An Improved KNN-Based Efficient Log Anomaly Detection Method with Automatically Labeled Samples[J]. ACM Transactions on Knowledge Discovery from Data (TKDD), 2021, 15(3): 1-22.
[19]	WURZENBERGER M, SKOPIK F, LANDAUER M, et al. Incremental Clustering for Semi-Supervised Anomaly Detection Applied on Log Data[C]// ACM. Proceedings of the 12th International Conference on Availability, Reliability and Security. New York: ACM, 2017: 1-6.
[20]	ZHANG Xu, XU Yong, LIN Qingwei, et al. Robust Log-Based Anomaly Detection on Unstable Log Data[C]// ACM. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. New York: ACM, 2019: 807-817.
[21]	BARIL X, COUSTIE O, MOTHE J, et al. Application Performance Anomaly Detection with LSTM on Temporal Irregularities in Logs[C]// ACM. Proceedings of the 29th ACM International Conference on Information & Knowledge Management. New York: ACM, 2020: 1961-1964.

日志类型	主体变量位置规律
Hadoop	<yyyy-MM-dd HH:mm:ss, SSS>\|<Log Level>\|<主体变量>\|<log中的message>
HDFS	<yyMMdd HHmmss>\|<Pid>\|<Log Level>\|<主体变量>\|<log中的message>
BGL	<Lable>\|<nnnnnnnnnn>\|<yyyy.MM.dd>\|<Node><Time>\|<NodeRepeat>\|<Type>\| <主体变量>\|<Log level>\|<log中的message>
OpenStack	<Logrecord>\|< yyyy-MM-dd HH:mm:ss, SSS>\|<Pid>\|<Log Level>\|<主体变量>\| <ADDR>\|< log中的message>

一种融合变量的日志异常检测方法

A Log Anomaly Detection Method with Variables

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 21

相关文章 15

编辑推荐

Metrics

本文评价

[1]	秦中元, 马楠, 余亚聪, 陈立全. 基于双重图神经网络和自编码器的网络异常检测[J]. 信息网络安全, 2023, 23(9): 1-11.
[2]	蒋英肇, 陈雷, 闫巧. 基于双通道特征融合的分布式拒绝服务攻击检测算法[J]. 信息网络安全, 2023, 23(7): 86-97.
[3]	施园, 李杨, 詹孟奇. 一种面向微服务的多维度根因定位算法[J]. 信息网络安全, 2023, 23(3): 73-83.
[4]	谭柳燕, 阮树骅, 杨敏, 陈兴蜀. 基于深度学习的教育数据分类方法[J]. 信息网络安全, 2023, 23(3): 96-102.
[5]	陈立全, 薛雨欣, 江英华, 朱雅晴. 基于国密SM2算法的证书透明日志系统设计[J]. 信息网络安全, 2023, 23(11): 9-16.
[6]	廖丽云, 张伯雷, 吴礼发. 基于代价敏感学习的物联网异常检测模型[J]. 信息网络安全, 2023, 23(11): 94-103.
[7]	张光华, 刘永升, 王鹤, 于乃文. 基于BiLSTM和注意力机制的智能合约漏洞检测方案[J]. 信息网络安全, 2022, 22(9): 46-54.
[8]	顾兆军, 刘婷婷, 高冰, 隋翯. 基于GAN-Cross的工控系统类不平衡数据异常检测[J]. 信息网络安全, 2022, 22(8): 81-89.
[9]	周婧怡, 李红娇. 针对PMU测量的虚假数据注入攻击检测方法[J]. 信息网络安全, 2022, 22(5): 75-83.
[10]	刘吉强, 何嘉豪, 张建成, 黄学臻. 基于解析器树的日志压缩优化方法[J]. 信息网络安全, 2022, 22(4): 30-39.
[11]	陈彬杰, 魏福山, 顾纯祥. 基于KNN的具有隐私保护功能的区块链异常交易检测[J]. 信息网络安全, 2022, 22(3): 78-84.
[12]	徐茹枝, 吕畅冉, 龙燕, 刘远彬. 工业控制系统高隐蔽性数据攻击防御方法研究[J]. 信息网络安全, 2022, 22(12): 34-46.
[13]	银鹰, 周志洪, 姚立红. 基于LSTM的CAN入侵检测模型研究[J]. 信息网络安全, 2022, 22(12): 57-66.
[14]	郭森森, 王同力, 慕德俊. 基于生成对抗网络与自编码器的网络流量异常检测模型[J]. 信息网络安全, 2022, 22(12): 7-15.
[15]	牛艺诺, 张逸飞, 高能, 马存庆. 融合时序和逻辑关系的日志异常检测系统设计[J]. 信息网络安全, 2022, 22(11): 1-6.