Log Parsing Method Based on Semantic of Parameters

doi:10.3969/j.issn.1671-1122.2025.04.009

Abstract

Abstract:

Modern information systems are increasingly large, and their behavior is reflected in diverse multi-source logs. The semantics of log parameters represent entity information within the system, which is crucial for the joint analysis of multi-source logs. However, existing parsing methods inadequately capture the semantic features of log parameters, leading to issues such as semantic gaps, limited coverage, and insufficient accuracy in semantic recognition. To address this, this paper proposed a parameter semantics-based log parsing method, (PS-Parser), which captured the semantic features of log context using a BERT model, extracted the semantics of log parameters, and complemented the semantics at different levels through a conventional parameter semantic feature library. Ultimately, it represented system entities based on parameter semantics to achieve joint analysis of multi-source logs. Experiments on six multi-source real datasets show an average accuracy of 94.7% for log parameter parsing, an average semantic coverage of 81.7%, and an average F1 score of 0.991 for semantic parsing, significantly improving upon existing methods and validating the effectiveness of the proposed approach. Finally, the support of the parameter semantics-based log parsing method for joint analysis of multi-source logs in big data system scenarios is verified.

Key words: log parsing, semantic of parameters extraction, multi-source log analysis

CLC Number:

TP309

XING Hantao, RUAN Shuhua, CHEN Liangguo, ZENG Xuemei. Log Parsing Method Based on Semantic of Parameters[J]. Netinfo Security, 2025, 25(4): 610-618.

Figures/Tables 12

References 18

[1]	HE Shilin, ZHU Jieming, HE Pinjia, et al. Experience Report: System Log Analysis for Anomaly Detection[C]// IEEE. 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). New York: IEEE, 2016: 207-218.
[2]	DU Min, LI Feifei, ZHENG Guineng, et al. Deeplog: Anomaly Detection and Diagnosis from System Logs through Deep Learning[C]// ACM. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2017: 1285-1298.
[3]	MENG Weibin, LIU Ying, ZHU Yichen, et al. Loganomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs[C]// IJCAI. The 28th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann, 2019: 4739-4745.
[4]	ZOU Deqing, QIN Hao, JIN Hai. Uilog: Improving Log-Based Fault Diagnosis by Log Analysis[J]. Journal of Computer Science and Technology, 2016, 31(5): 1038-1052.
[5]	SUN Yang, LI Huajing, COUNCILL I G, et al. Personalized Ranking for Digital Libraries Based on Log Analysis[C]// ACM. Proceedings of the 10th ACM Workshop on Web Information and Data Management. New York: ACM, 2008: 133-140.
[6]	YU Xiuming, LI Meijing, PAIK I, et al. Prediction of Web User Behavior by Discovering Temporal Relational Rules from Web Log Data[C]// Springer. Database and Expert Systems Applications:The 23rd International Conference. Heidelberg: Springer, 2012: 31-38.
[7]	HUO Yintong, SU Yuxin, LEE C, et al. Semparser: A Semantic Parser for Log Analytics[C]// IEEE. 2023 IEEE/ACM 45th International Conference on Software Engineering. New York: IEEE, 2023: 881-893.
[8]	VAARANDI R. A Data Clustering Algorithm for Mining Patterns from Event Logs[C]// IEEE. Proceedings of the 3rd IEEE Workshop on IP Operations Management. New York: IEEE, 2003: 119-126.
[9]	DAI Hetong, LI Heng, CHEN Cheshao, et al. Logram: Efficient Log Parsing Using n-Gram Dictionaries[J]. IEEE Transactions on Software Engineering, 2020, 48(3): 879-892.
[10]	MIZUTANI M. Incremental Mining of System Log Format[C]// IEEE. 2013 IEEE International Conference on Services Computing. New York: IEEE, 2013: 595-602.
[11]	HE Pinjia, ZHU Jieming, ZHENG Zibin, et al. Drain: An Online Log Parsing Approach with Fixed Depth Tree[C]// IEEE. 2017 IEEE International Conference on Web Services. New York: IEEE, 2017: 33-40.
[12]	JIANG Jinzhao, FU Yuanyuan, XU Jian. Heuristic Online Log Parsing Method Based on Part-of-Speech Tagging[J]. Application Research of Computers, 2024, 41 (1): 217-221.
	蒋金钊, 傅媛媛, 徐建. 基于词性标注的启发式在线日志解析方法[J]. 计算机应用研究, 2024, 41(1): 217-221.
[13]	LU Jiawei, LU Shida, LIU Sisi, et al. Online Log Parsing Method Based on Bert and Adaptive Clustering[J]. Computer Science, 2024, 51(11): 65-72. doi: 10.11896/jsjkx.230900161
	卢家伟, 卢士达, 刘思思, 等. 基于Bert和自适应聚类的在线日志解析方法[J]. 计算机科学, 2024, 51(11): 65-72. doi: 10.11896/jsjkx.230900161
[14]	DEVLIN J, CHANG M, LEE K, et al. BERT: Pre-Training of Deep Bidirectional Transformersfor Language Understanding[C]// ACL. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg: ACL, 2019: 4171-4186.
[15]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All You Need[C]// NIPS. 31st Conference on Neural Information Processing Systems. New York: Curran Associates Incorporated, 2017: 5998-6008.
[16]	ZHU Jieming, HE Shilin, HE Pinjia, et al. Loghub: A Large Collection of System Log Datasets for AI-Driven Log Analytics[C]// IEEE. 2023 IEEE 34th International Symposium on Software Reliability Engineerin. New York: IEEE, 2023: 355-366.
[17]	LE V H, ZHANG Hongyu. Log-Based Anomaly Detection with Deep Learning: How Far are We?[C]// ACM. Proceedings of the 44th International Conference on Software Engineering. New York: ACM, 2022: 1356-1367.
[18]	SUI Yicheng, ZHANG Yuzhe, SUN Jianjun. LogKG: Log Failure Diagnosis through Knowledge Graph[J]. IEEE Transactions on Services Computing, 2023, 16(5): 3493-3507.

参数语义	参数特征	参数实例
IPv4地址	(0~255).(0~255).(0~255).(0~255)	10.251.42.84
文件路径	///(代表任意字符串)	/user/root
域名	..(代表任意字符串)	msra-sa.com
时间	YYYY/MM/DD	2024/09/30

数据集	IPLoM	SHISO	Drain	PosParser	PS-Parser
Andriod	0.682	0.585	0.911	0.926	0.937
Hadoop	0.954	0.867	0.948	0.989	0.981
HDFS	1.000	0.998	0.998	1.000	1.000
Linux	0.672	0.701	0.690	0.754	0.793
OpenStack	0.758	0.722	0.733	1.000	0.981
Zookeeper	0.921	0.660	0.967	0.995	0.989
平均值	0.831	0.756	0.874	0.944	0.947

数据集	Semparser		PS-Parser		提升幅度
数据集	SC	Pairs	SC	Pairs	提升幅度
Andriod	70.7%	6370	77.9%	7023	7.24%
Hadoop	63.4%	2552	89.9%	3620	26.53%
HDFS	57.4%	3015	100.0%	5253	42.60%
Linux	31.8%	2844	47.0%	4201	15.18%
OpenStack	54.9%	4337	84.4%	6669	29.52%
Zookeeper	66.6%	1176	90.7%	1602	24.13%
平均语义覆盖率	57.5%	81.7%	24.2%

数据集	Semparser			PS-Parser
数据集	P	R	F	P	R	F
Andriod	0.951	0.935	0.943	0.974	0.964	0.969
Hadoop	0.993	0.978	0.985	0.994	0.989	0.991
HDFS	1	1	1	1	1	1
Linux	0.998	0.977	0.987	0.998	0.979	0.988
OpenStack	0.999	0.998	0.999	0.999	0.998	0.999
Zookeeper	1	0.989	0.995	1	0.997	0.998
平均F1分数	0.985			0.991

参数类型	Semparser	PS-Parser
Block	1176	1662
PacketResponder	316	425
blockMap	297	362
NameSystem	230	271
Size	94	94
Src	451	542
Dest	451	542
IP	0	1084
Path	0	271
合计	3015	5253