面向智能系统的内部威胁多源日志分析与检测方法

doi:10.3969/j.issn.1671-1122.2025.04.001

摘要/Abstract

摘要：

在智能系统安全领域，异常检测尤其是内部威胁的识别是一项极具挑战性的任务。现有方法通常依赖预定义规则或时序建模学习，但在面对未知威胁模式时存在局限，且难以充分挖掘日志数据的深层特征。针对这一问题，文章提出一种基于 Transformer 编码器（Trans-Encoder）与长短期记忆网络（LSTM）融合的内部威胁检测方法，旨在仅使用正常类数据训练实现日志中隐蔽异常的高效识别。首先，文章提出的方法通过改进 Transformer 编码器结构，增加屏蔽机制，从而增强了从多源日志数据中提取特征的能力。然后，应用 LSTM 进行时间序列建模，以捕捉提取特征之间的时间相关性，从而提高模型分析顺序依赖关系的能力。最后，计算预测值与对应特征值的差异度，并与阈值进行对比，以判断是否为异常操作。实验结果表明，该方法在内部威胁检测任务上的性能优于现有方法，其准确率提高1.5%，召回率提高4.8%，F1分数提高1.3%，在仅有10%训练数据的情况下，仍能保持稳定性能。此外，在训练阶段和测试阶段的计算效率都高于MTSAD，验证了其在智能系统安全中的应用潜力，为提升系统防护能力提供了一种高效可靠的解决方案。

关键词: 内部威胁检测, 用户行为分析, 语义分析, 时序分析

Abstract:

In the field of intelligent system security, the anomaly detection domain, especially the identification of insider threats, is a challenging task. Existing methods usually rely on predefined rules or temporal modelling learning, but are prone to limitations when facing unknown threat patterns, and it is difficult to fully explore the deep features of log data. To address this problem, this paper proposed an insider threat detection method based on the fusion of Transformer Encoder (Trans-Encoder) and Long Short-Term Memory (LSTM) networks, aiming to achieve efficient identification of hidden anomalies in logs by using only normal class data for training. Firstly, the method proposed in this paper enhanced the ability to extract features from multi-source log data by improving the Transformer encoder structure and adding a masking mechanism. Then LSTM was applied for time series modelling to capture the temporal correlation between the extracted features, which improved the model’s ability to analyze sequential dependencies. Finally, the degree of difference between the predicted value and the corresponding feature value was calculated and compared with the threshold value to determine whether the operation was anomalous or not. The experimental results show that the method outperforms the existing state-of-the-art methods on the insider threat detection task, with a 1.5% improvement in Precision, a 4.8% improvement in Recall, a 1.3% improvement in F1-score, and a stable performance with only 10% training data. In addition, the computational efficiency is higher than that of MTSAD in both the training and testing phases, which verifies its potential application in intelligent system security and provides an efficient and reliable solution for improving system protection.

Key words: insider threat detection, user behavior analysis, semantic analysis, temporal analysis

中图分类号:

TP309

李涛, 毕悦, 胡爱群. 面向智能系统的内部威胁多源日志分析与检测方法[J]. 信息网络安全, 2025, 25(4): 509-523.

LI Tao, BI Yue, HU Aiqun. Insider Threat Multi-Source Log Analysis and Detection Method for Intelligent Systems[J]. Netinfo Security, 2025, 25(4): 509-523.

图/表 13

表1

图1

图2

图3

图4

表2

表3

图5

表4

图6

图7

图8

表5

参考文献 28

[1]	LIANG Xiao, GUI Xiaolin, DAI Haojun, et al. Research on Cache Side Channel Attack Technology of Virtual Machines in Cloud Environment[J]. Chinese Journal of Computers, 2017, 40(2): 317-336.
[2]	Verizon. 2023 Data Breach Investigations Report[EB/OL]. [2025-01-20]. https://www.verizon.com/business/resources/T2b1/reports/2023-data-breach-investigations-report-dbir.pdf.
[3]	AKASH T, JAYAN A, GAUR N, et al. Identifying Insider Cyber Threats Using Behaviour Modelling and Analysis[C]// IEEE. 2023 OITS International Conference on Information Technology (OCIT). New York: IEEE, 2023: 35-40.
[4]	LE D C, ZINCIR-HEYWOOD A N. Evaluating Insider Threat Detection Workflow Using Supervised and Unsupervised Learning[C]// IEEE. 2018 IEEE Security and Privacy Workshops (SPW). New York: IEEE, 2018: 270-275.
[5]	MENG Fanzhi, LU Peng, LI Junhao, et al. GRU and Multi-Autoencoder Based Insider Threat Detection for Cyber Security[C]// IEEE. 2021 IEEE Sixth International Conference on Data Science in Cyberspace (DSC). New York: IEEE, 2021: 203-210.
[6]	TIAN Zhihong, SHI Wei, TAN Zhiyuan, et al. Deep Learning and Dempster-Shafer Theory Based Insider Threat Detection[J]. Mobile Networks and Applications, 2020: 1-10.
[7]	YUAN Shuhan, WU Xintao. Deep Learning for Insider Threat Detection: Review, Challenges and Opportunities[EB/OL]. (2021-05-01)[2025-01-20]. https://doi.org/10.1016/j.cose.2021.102221.
[8]	TUOR A, KAPLAN S, HUTCHINSON B, et al. Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams[EB/OL]. (2017-10-02)[2025-01-20]. https://arxiv.org/abs/1710.00811.
[9]	ELDARDIRY H, BART E, LIU Juan, et al. Multi-Domain Information Fusion for Insider Threat Detection[C]// IEEE. 2013 IEEE Security and Privacy Workshops. New York: IEEE, 2013: 45-51.
[10]	ZHANG Lili, XIE Yuxiang, LUAN Xidao, et al. Multi-Source Heterogeneous Data Fusion[C]// IEEE. 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD). New York: IEEE, 2018: 47-51.
[11]	LEE S, PARK S, HONG K. RDFNet: RGB-D Multi-Level Residual Feature Fusion for Indoor Semantic Segmentation[C]// IEEE. 2017 IEEE International Conference on Computer Vision (ICCV). New York: IEEE, 2017: 4990-4999.
[12]	BILINSKI P, PRISACARIU V. Dense Decoder Shortcut Connections for Single-Pass Semantic Segmentation[C]// IEEE. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 6596-6605.
[13]	LE V H, ZHANG Hongyu. Log-Based Anomaly Detection without Log Parsing[C]// IEEE. 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). New York: IEEE, 2021: 492-504.
[14]	LIU Shuo, XU Liwen, WANG Jinru, et al. LCR-GAN: Learning Crucial Representation for Anomaly Detection[C]// ACM. 2021 5th International Conference on Computer Science and Artificial Intelligence. New York: ACM, 2021: 162-167.
[15]	WU Shuguang, WANG Hongyan, WANG Yu, et al. Technology Analysis of Network Anomalous Behavior Detection Based on Machine Learning[C]// IEEE. Proceedings of the 2022 3rd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE). New York: IEEE, 2022: 730-737.
[16]	MAO Sheng, GUO Jiansheng, GU Taoyong, et al. Dis-AE-LSTM: Generative Adversarial Networks for Anomaly Detection of Time Series Data[C]// IEEE. 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE). New York: IEEE, 2020: 330-336.
[17]	SAID E M, LE-KHAC N A, DEV S, et al. Network Anomaly Detection Using LSTM Based Autoencoder[C]// ACM. Proceedings of the 16th ACM Symposium on QoS and Security for Wireless and Mobile Networks. New York: ACM, 2020: 37-45.
[18]	XUE Yapeng. Research on Time Series Anomaly Detection Based on Graph Neural Network[C]// IEEE. 2023 IEEE International Conference on Electrical, Automation and Computer Engineering (ICEACE). New York: IEEE, 2023: 1670-1674.
[19]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All you Need[J]. Neural Information Processing Systems, 2017, 30: 5998-6008.
[20]	YIN Chuanlong, ZHU Yuefei, FEI Jinlong, et al. A Deep Learning Approach for Intrusion Detection Using Recurrent Neural Networks[J]. IEEE Access, 2017, 5: 21954-21961.
[21]	XU Wei, HUANG Ling, FOX A, et al. Detecting Large-Scale System Problems by Mining Console Logs[C]// ACM. Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles. New York: ACM, 2009: 117-132.
[22]	XU Wei, HUANG Ling, FOX A, et al. Online System Problem Detection by Mining Patterns of Console Logs[C]// IEEE. 2009 Ninth IEEE International Conference on Data Mining. New York: IEEE, 2009: 588-597.
[23]	LOU Jianguang, FU Qiang, YANG Shengqi, et al. Mining Invariants from Console Logs for System Problem Detection[J]. IEEE Transactions on Computers, 2010, 59(6): 759-774.
[24]	OLINER A, STEARLEY J. What Supercomputers Say: A Study of Five System Logs[C]// IEEE. The 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07). New York: IEEE, 2007: 575-584.
[25]	DU Min, LI Feifei, ZHENG Guineng, et al. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning[C]// ACM. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2017: 1285-1298.
[26]	AUDIBERT J, MICHIARDI P, GUYARD F, et al. USAD: UnSupervised Anomaly Detection on Multivariate Time Series[C]// ACM. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM, 2020: 3395-3404.
[27]	LI Sainan, YIN Qilei, LI Guoliang, et al. Unsupervised Contextual Anomaly Detection for Database Systems[C]// ACM. Proceedings of the 2022 International Conference on Management of Data. New York: ACM, 2022: 788-802.
[28]	SOWMYA K, RAMESH K. Enhancing Anomaly Detection in Multivariate Time Series with Stacked Transformer Encoders and Adaptive Positional Embeddings[EB/OL]. (2024-12-06)[2025-01-20]. https://doi.org/10.1007/s13369-024-09821-w.

传统方法	文本方法
对用户属性要求严格^[9]，数据融合算法导致数据规模过大，影响分析效率和准确性^[10]	通过提取语义特征避免了严格的用户组属性一致性假设，通过滑动窗口对高频并发操作的日志数据处理
无法有效处理日志格式和结构不统一的情况，且对异构日志没有优化^[13,14]	对日志数据进行结构化处理，优化了异构日志的特征提取
主要侧重时间依赖性^[16-18]，未充分考虑日志数据的复杂语义关系和多源信息融合	对日志数据进行多维度分析，即分析语义特征和时间特征

数据集	数据量		词汇表大小/个
数据集	训练数据	测试数据	词汇表大小/个
HDFS	4855个正常数据	553366个正常数据和 16838个异常数据	29
BGL	831个正常数据	5990个正常数据和 453个异常数据	656

参数	最优值
Hidden layers	2
hidden_size	48
Batch size	1024
Optimizer	Adam
function	MSE
Activation function	tanh
Learning rate	0.0001
Number of epochs	100

数据集	方法	Deeplog	USAD	UCAD	MTSAD	TE-LSTM
HDFS	Precision	0.8110	0.7945	0.7751	0.8191	0.8340
	Recall	0.9199	0.8692	0.9181	0.9206	0.9689
	F1-score	0.8621	0.8503	0.8406	0.8833	0.8964
BGL	Precision	0.8974	0.7697	0.9045	0.8843	0.9104
	Recall	0.8278	0.9831	0.9583	0.9154	0.9778
	F1-score	0.8612	0.8186	0.9306	0.9097	0.9314

方法	训练时间/s	测试时间/s
MTSAD	38.83	434.29
TE-LSTM	15.95	290.04