基于参数语义的日志解析方法

doi:10.3969/j.issn.1671-1122.2025.04.009

信息网络安全 ›› 2025, Vol. 25 ›› Issue (4): 610-618.doi: 10.3969/j.issn.1671-1122.2025.04.009

• 专题论文：智能系统安全 • 上一篇下一篇

基于参数语义的日志解析方法

邢瀚韬¹^,²^,³, 阮树骅¹^,²^,³(), 陈良国¹^,²^,³, 曾雪梅²^,³

1.四川大学网络空间安全学院，成都 610065
2.数据安全防护与智能治理教育部重点实验室，成都 610065
3.四川大学网络空间安全研究院，成都 610065

收稿日期:2024-12-30 出版日期:2025-04-10 发布日期:2025-04-25
通讯作者: 阮树骅 ruanshuhua@scu.edu.cn
作者简介:邢瀚韬（2000—），男，山西，硕士研究生，主要研究方向为数据安全|阮树骅（1966—），女，四川，副教授，硕士，主要研究方向为云计算与大数据安全、区块链安全|陈良国（1993—），男，贵州，博士研究生，主要研究方向为大数据和网络安全|曾雪梅（1976—），女，四川，工程师，博士，主要研究方向为网络威胁检测、网络行为分析
基金资助:
中央高校基本科研业务费专项资金(SCU2024D012);四川大学理工学科内涵发展项目(2020SCUNG129)

Log Parsing Method Based on Semantic of Parameters

XING Hantao¹^,²^,³, RUAN Shuhua¹^,²^,³(), CHEN Liangguo¹^,²^,³, ZENG Xuemei²^,³

1. School of Cyber Science and Engineering, Sichuan University, Chengdu 610065, China
2. Key Laboratory of Data Protection and Intelligent Management, Chengdu 610065, China
3. Cyber Science Research Institute, Sichuan University, Chengdu 610065, China

Received:2024-12-30 Online:2025-04-10 Published:2025-04-25

摘要/Abstract

摘要：

现代信息系统规模日益扩大，通过分析结构各异的多源日志可以快速了解系统行为。日志参数的语义表征了系统中的实体信息，对实现多源日志的联合分析至关重要。但现有解析方法对日志参数的语义特征捕捉不足，存在语义缺失、语义覆盖范围不广、语义识别准确率不足等问题。因此，文章提出一种基于参数语义的日志解析方法（PS-Parser），该方法通过构建BERT模型捕捉日志上下文语义特征，提取日志参数的语义，并通过常规参数语义特征库，补全日志参数不同层次的语义，最终根据参数语义表征系统实体，实现多源日志联合分析。文章在6个多源真实数据集上进行实验，日志参数解析的平均准确率为94.7%，平均语义覆盖率为81.7%，语义解析的平均F1分数为0.991，相较于现有方法有显著提升，验证了所提方法的有效性。最后，针对大数据系统下的日志分析场景，验证了基于参数语义的日志解析方法对多源日志联合分析工作的支持作用。

关键词: 日志解析, 参数语义提取, 多源日志分析

Abstract:

Modern information systems are increasingly large, and their behavior is reflected in diverse multi-source logs. The semantics of log parameters represent entity information within the system, which is crucial for the joint analysis of multi-source logs. However, existing parsing methods inadequately capture the semantic features of log parameters, leading to issues such as semantic gaps, limited coverage, and insufficient accuracy in semantic recognition. To address this, this paper proposed a parameter semantics-based log parsing method, (PS-Parser), which captured the semantic features of log context using a BERT model, extracted the semantics of log parameters, and complemented the semantics at different levels through a conventional parameter semantic feature library. Ultimately, it represented system entities based on parameter semantics to achieve joint analysis of multi-source logs. Experiments on six multi-source real datasets show an average accuracy of 94.7% for log parameter parsing, an average semantic coverage of 81.7%, and an average F1 score of 0.991 for semantic parsing, significantly improving upon existing methods and validating the effectiveness of the proposed approach. Finally, the support of the parameter semantics-based log parsing method for joint analysis of multi-source logs in big data system scenarios is verified.

Key words: log parsing, semantic of parameters extraction, multi-source log analysis

中图分类号:

TP309

邢瀚韬, 阮树骅, 陈良国, 曾雪梅. 基于参数语义的日志解析方法[J]. 信息网络安全, 2025, 25(4): 610-618.

XING Hantao, RUAN Shuhua, CHEN Liangguo, ZENG Xuemei. Log Parsing Method Based on Semantic of Parameters[J]. Netinfo Security, 2025, 25(4): 610-618.

图/表 12

图1

图2

图3

图4

图5

表1

图6

表2

表3

表4

表5

图7

参考文献 18

[1]	HE Shilin, ZHU Jieming, HE Pinjia, et al. Experience Report: System Log Analysis for Anomaly Detection[C]// IEEE. 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). New York: IEEE, 2016: 207-218.
[2]	DU Min, LI Feifei, ZHENG Guineng, et al. Deeplog: Anomaly Detection and Diagnosis from System Logs through Deep Learning[C]// ACM. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2017: 1285-1298.
[3]	MENG Weibin, LIU Ying, ZHU Yichen, et al. Loganomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs[C]// IJCAI. The 28th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann, 2019: 4739-4745.
[4]	ZOU Deqing, QIN Hao, JIN Hai. Uilog: Improving Log-Based Fault Diagnosis by Log Analysis[J]. Journal of Computer Science and Technology, 2016, 31(5): 1038-1052.
[5]	SUN Yang, LI Huajing, COUNCILL I G, et al. Personalized Ranking for Digital Libraries Based on Log Analysis[C]// ACM. Proceedings of the 10th ACM Workshop on Web Information and Data Management. New York: ACM, 2008: 133-140.
[6]	YU Xiuming, LI Meijing, PAIK I, et al. Prediction of Web User Behavior by Discovering Temporal Relational Rules from Web Log Data[C]// Springer. Database and Expert Systems Applications:The 23rd International Conference. Heidelberg: Springer, 2012: 31-38.
[7]	HUO Yintong, SU Yuxin, LEE C, et al. Semparser: A Semantic Parser for Log Analytics[C]// IEEE. 2023 IEEE/ACM 45th International Conference on Software Engineering. New York: IEEE, 2023: 881-893.
[8]	VAARANDI R. A Data Clustering Algorithm for Mining Patterns from Event Logs[C]// IEEE. Proceedings of the 3rd IEEE Workshop on IP Operations Management. New York: IEEE, 2003: 119-126.
[9]	DAI Hetong, LI Heng, CHEN Cheshao, et al. Logram: Efficient Log Parsing Using n-Gram Dictionaries[J]. IEEE Transactions on Software Engineering, 2020, 48(3): 879-892.
[10]	MIZUTANI M. Incremental Mining of System Log Format[C]// IEEE. 2013 IEEE International Conference on Services Computing. New York: IEEE, 2013: 595-602.
[11]	HE Pinjia, ZHU Jieming, ZHENG Zibin, et al. Drain: An Online Log Parsing Approach with Fixed Depth Tree[C]// IEEE. 2017 IEEE International Conference on Web Services. New York: IEEE, 2017: 33-40.
[12]	JIANG Jinzhao, FU Yuanyuan, XU Jian. Heuristic Online Log Parsing Method Based on Part-of-Speech Tagging[J]. Application Research of Computers, 2024, 41 (1): 217-221.
	蒋金钊, 傅媛媛, 徐建. 基于词性标注的启发式在线日志解析方法[J]. 计算机应用研究, 2024, 41(1): 217-221.
[13]	LU Jiawei, LU Shida, LIU Sisi, et al. Online Log Parsing Method Based on Bert and Adaptive Clustering[J]. Computer Science, 2024, 51(11): 65-72. doi: 10.11896/jsjkx.230900161
	卢家伟, 卢士达, 刘思思, 等. 基于Bert和自适应聚类的在线日志解析方法[J]. 计算机科学, 2024, 51(11): 65-72. doi: 10.11896/jsjkx.230900161
[14]	DEVLIN J, CHANG M, LEE K, et al. BERT: Pre-Training of Deep Bidirectional Transformersfor Language Understanding[C]// ACL. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg: ACL, 2019: 4171-4186.
[15]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All You Need[C]// NIPS. 31st Conference on Neural Information Processing Systems. New York: Curran Associates Incorporated, 2017: 5998-6008.
[16]	ZHU Jieming, HE Shilin, HE Pinjia, et al. Loghub: A Large Collection of System Log Datasets for AI-Driven Log Analytics[C]// IEEE. 2023 IEEE 34th International Symposium on Software Reliability Engineerin. New York: IEEE, 2023: 355-366.
[17]	LE V H, ZHANG Hongyu. Log-Based Anomaly Detection with Deep Learning: How Far are We?[C]// ACM. Proceedings of the 44th International Conference on Software Engineering. New York: ACM, 2022: 1356-1367.
[18]	SUI Yicheng, ZHANG Yuzhe, SUN Jianjun. LogKG: Log Failure Diagnosis through Knowledge Graph[J]. IEEE Transactions on Services Computing, 2023, 16(5): 3493-3507.

参数语义	参数特征	参数实例
IPv4地址	(0~255).(0~255).(0~255).(0~255)	10.251.42.84
文件路径	///(代表任意字符串)	/user/root
域名	..(代表任意字符串)	msra-sa.com
时间	YYYY/MM/DD	2024/09/30

数据集	IPLoM	SHISO	Drain	PosParser	PS-Parser
Andriod	0.682	0.585	0.911	0.926	0.937
Hadoop	0.954	0.867	0.948	0.989	0.981
HDFS	1.000	0.998	0.998	1.000	1.000
Linux	0.672	0.701	0.690	0.754	0.793
OpenStack	0.758	0.722	0.733	1.000	0.981
Zookeeper	0.921	0.660	0.967	0.995	0.989
平均值	0.831	0.756	0.874	0.944	0.947

数据集	Semparser		PS-Parser		提升幅度
数据集	SC	Pairs	SC	Pairs	提升幅度
Andriod	70.7%	6370	77.9%	7023	7.24%
Hadoop	63.4%	2552	89.9%	3620	26.53%
HDFS	57.4%	3015	100.0%	5253	42.60%
Linux	31.8%	2844	47.0%	4201	15.18%
OpenStack	54.9%	4337	84.4%	6669	29.52%
Zookeeper	66.6%	1176	90.7%	1602	24.13%
平均语义覆盖率	57.5%	81.7%	24.2%

数据集	Semparser			PS-Parser
数据集	P	R	F	P	R	F
Andriod	0.951	0.935	0.943	0.974	0.964	0.969
Hadoop	0.993	0.978	0.985	0.994	0.989	0.991
HDFS	1	1	1	1	1	1
Linux	0.998	0.977	0.987	0.998	0.979	0.988
OpenStack	0.999	0.998	0.999	0.999	0.998	0.999
Zookeeper	1	0.989	0.995	1	0.997	0.998
平均F1分数	0.985			0.991

参数类型	Semparser	PS-Parser
Block	1176	1662
PacketResponder	316	425
blockMap	297	362
NameSystem	230	271
Size	94	94
Src	451	542
Dest	451	542
IP	0	1084
Path	0	271
合计	3015	5253

基于参数语义的日志解析方法

Log Parsing Method Based on Semantic of Parameters

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 18

相关文章 1

编辑推荐

Metrics

本文评价