基于溯因学习的无监督网络流量异常检测

doi:10.3969/j.issn.1671-1122.2024.11.007

摘要/Abstract

摘要：

当前计算机网络流量异常检测面临缺乏标注信息的挑战，同时用户需要自行选择合适的技术并调整参数，但没有标签可用于交叉验证。为此，文章提出一种基于溯因学习的无监督网络流量异常检测（ABL-ATD）模型。该模型通过自动生成伪标签，并利用演绎与一致性验证生成高质量标签，避免人工干预。ABL-ATD从多种无监督异常检测模型中提取有效信号，并通过验证与修正，可靠区分异常流量和正常流量。实验结果表明，该模型在多个数据集上展现出与使用真实标签训练的监督学习模型相当的准确性。

关键词: 异常流量检测, 无监督学习, 溯因学习

Abstract:

The current challenge in computer network traffic anomaly detection is the lack of labeled information, while users must select appropriate technologies and adjust parameters without any labels for cross-validation. To address this issue, this paper proposed an abductive learning-based anomaly traffic detection (ABL-ATD) model, which operated in an unsupervised manner. This model automatically generated pseudo-labels and utilized deductive reasoning and consistency verification to produce high-quality labels, thereby avoiding manual intervention. The innovation of ABL-ATD lied in its ability to extract effective signals from multiple unsupervised anomaly detection models and reliably distinguish between anomalous and normal traffic through validation and correction. Experimental results demonstrate that this model exhibits accuracy comparable to that of supervised learning models trained with real labels across multiple datasets.

Key words: traffic anomaly detection, unsupervised learning, abductive learning

中图分类号:

TP309

胡文涛, 徐靖凯, 丁伟杰. 基于溯因学习的无监督网络流量异常检测[J]. 信息网络安全, 2024, 24(11): 1675-1684.

HU Wentao, XU Jingkai, DING Weijie. Unsupervised Network Traffic Anomaly Detection Based on Abductive Learning[J]. Netinfo Security, 2024, 24(11): 1675-1684.

图/表 6

图1

图2

表1

表2

图3

表3

参考文献 25

[1]	HE Ke, KIM D D, ASGHAR M R. Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey[J]. IEEE Communications Surveys & Tutorials, 2023, 25(1): 538-566.
[2]	QIN Zhongyuan, MA Nan, YU Yacong, et al. Network Anomaly Detection Based on Dual Graph Convolutional Network and Autoencoders[J]. Netinfo Security, 2023, 23(9): 1-11.
	秦中元, 马楠, 余亚聪, 等. 基于双重图神经网络和自编码器的网络异常检测[J]. 信息网络安全, 2023, 23(9):1-11.
[3]	CHANG Zhihao, YU Linzhu, XU Yanchao, et al. Neural Embeddings for KNN Search in Biological Sequence[C]//AAAI. Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2024: 38-45.
[4]	DAI Wangzhou, XU Qiuling, YU Yang, et al. Bridging Machine Learning and Logical Reasoning by Abductive Learning[C]//ACM. Proceedings of the 33rd International Conference on Neural Information Processing Systems. New York: ACM, 2019: 32-44.
[5]	LIU F T, TINGKaiming, ZHOU Zhihua. Isolation Forest[C]//IEEE. 2008 Eighth IEEE International Conference on Data Mining. New York: IEEE, 2008: 413-422.
[6]	BREUNIG M M, KRIEGEL H P, NG R T, et al. LOF: Identifying Density-Based Local Outliers[C]//ACM. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2000: 93-104.
[7]	ZHANG Minling, ZHOU Zhihua. ML-KNN: A Lazy Learning Approach to Multi-Label Learning[J]. Pattern Recognition, 2007, 40(7): 2038-2048.
[8]	HU Wentao, JIANG Dawei, WU Sai, et al. Distributional Constraint Discovery for Intelligent Auditing[J]. Knowledge and Information Systems, 2023, 65(12): 5195-5229.
[9]	HU Wentao, JIANG Dawei, WU Sai, et al. IceBerg: Deep Generative Modeling for Constraint Discovery and Anomaly Detection[C]//IEEE. 2022 IEEE International Conference on Parallel & Distributed Processing with Applications. New York: IEEE, 2022: 74-81.
[10]	LIU Yuxiao, CHEN Wei, ZHANG Tianyue, et al. Explainable Anomaly Traffic Detection Based on Sparse Autoencoders[J]. Netinfo Security, 2023, 23(7): 74-85.
	刘宇啸, 陈伟, 张天月, 等. 基于稀疏自动编码器的可解释性异常流量检测[J]. 信息网络安全, 2023, 23(7):74-85.
[11]	HAN Qi, QIAN Xin, XU Hongxiang, et al. DM-CNN: Dynamic Multi-Scale Convolutional Neural Network with Uncertainty Quantification for Medical Image Classification[J]. Computers in Biology and Medicine, 2024, 168: 107758.
[12]	ACHARYA T, ANNAMALAI A, CHOUIKHA M F. Optimizing the Performance of Network Anomaly Detection Using Bidirectional Long Short-Term Memory (Bi-LSTM) and over-Sampling for Imbalance Network Traffic Data[J]. Advances in Science Technology and Engineering Systems Journal, 2023, 8(6): 144-154.
[13]	YUAN Lining, LI Xin, WANG Xiaodong, et al. Graph Embedding Models: A Survey[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(1): 59-87.
	袁立宁, 李欣, 王晓冬, 等. 图嵌入模型综述[J]. 计算机科学与探索, 2022, 16(1):59-87. doi: 10.3778/j.issn.1673-9418.2104020
[14]	FAN Wenfei, HAN Ziyan, WANG Yaoshu, et al. Discovering Top-K Rules Using Subjective and Objective Criteria[J]. Proceedings of the ACM on Management of Data, 2023, 1(1): 1-29.
[15]	HUANG Qiang, LUO Pingyi, TUNG A K H. A New Sparse Data Clustering Method Based on Frequent Items[J]. Proceedings of the ACM on Management of Data, 2023, 1(1): 29-58.
[16]	ZONG Bo, SONG Qi, MIN M R, et al. Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection[C]//ICLR. InInternational Conference on Learning Representations (ICLR). San Diego: ICLR, 2020: 1-19.
[17]	DONG Shuqin, ZHANG Bin. Network Traffic Anomaly Detection Method Based on Deep Features Learning[J]. Journal of Electronics & Information Technology, 2020, 42(3): 1-9.
	董书琴, 张斌. 基于深度特征学习的网络流量异常检测方法[J]. 电子与信息学报, 2020, 42(3):1-9.
[18]	YE Huanrong, LI Muyuan, JIANG Bo. Research on DGA Malicious Domain Name Detection Method Based on Transfer Learning and Threat Intelligence[J]. Netinfo Security, 2023, 23(10): 8-15.
	叶桓荣, 李牧远, 姜波. 基于迁移学习和威胁情报的DGA恶意域名检测方法研究[J]. 信息网络安全, 2023, 23(10):8-15.
[19]	FU Yu, WANG Kun, DUAN Xueyuan, et al. Survey of Research on Abnormal Traffic Detection for Software Defined Networks[J]. Journal on Communications, 2024, 45(3): 208-226. doi: 10.11959/j.issn.1000-436x.2024016
	付钰, 王坤, 段雪源, 等. 面向软件定义网络的异常流量检测研究综述[J]. 通信学报, 2024, 45(3):208-226. doi: 10.11959/j.issn.1000-436x.2024016
[20]	ZHANG Yuchen, LI Lianghui, MA Chenyang, et al. A Log Anomaly Detection Method with Variables[J]. Netinfo Security, 2023, 23(10): 16-20.
	张玉臣, 李亮辉, 马辰阳, 等. 一种融合变量的日志异常检测方法[J]. 信息网络安全, 2023, 23(10):16-20.
[21]	HU Wentao, JIANG Dawei, WU Sai, et al. Complex Integrity Constraint Discovery: Measuring Trust in Modern Intelligent Railroad Systems[J]. Journal of Zhejiang University-Science A, 2022, 23(10): 832-837.
[22]	BREVE B, CARUCCIO L, CIRILLO S, et al. IndiBits: Incremental Discovery of Relaxed Functional Dependencies Using Bitwise Similarity[C]//IEEE. 2023 IEEE 39th International Conference on Data Engineering (ICDE). New York: IEEE, 2024: 1393-1405.
[23]	CAO Lei, YAN Yizhou, WANG Yu, et al. AutoOD: Automatic Outlier Detection[J]. Proceedings of the ACM on Management of Data, 2023, 1(1): 1-27.
[24]	IMAN S, ARASH H, ALI A. Ghorba-Ni, Intrusion Detection Evaluation Dataset(CICIDS2017)[EB/OL]. (2022-11-29)[2024-05-30]. https://www.unb.ca/cic/datasets/ids-2017.html.
[25]	ZHONG Ying, WANG Zhiliang, SHI Xingang, et al. RFG-HELAD: A Robust Fine-Grained Network Traffic Anomaly Detection Model Based on Heterogeneous Ensemble Learning[J]. IEEE Transactions on Information Forensics and Security, 2024, 19(1): 5895-5910.

数据集	特征数/个	网络流量/GB	攻击类型/种	实验数据划分比例
CIC-IDS-2017	80	7.8	12	8:1:1
UNSW-NB15	42	0.5	49	8:1:1

方法	数据集	F1-score	AUC	Precision	Recall
隔离森林	CIC-IDS-2017	58%	72%	61%	56%
隔离森林	UNSW-NB15	59%	74%	62%	58%
最佳无监督模型	CIC-IDS-2017	65%	79%	68%	64%
最佳无监督模型	UNSW-NB15	69%	81%	71%	67%
DAGMM	CIC-IDS-2017	73%	83%	75%	72%
DAGMM	UNSW-NB15	78%	85%	77%	76%
RFG-HELAD	CIC-IDS-2017	82%	88%	80%	78%
RFG-HELAD	UNSW-NB15	83%	89%	81%	79%
ABL-ATD	CIC-IDS-2017	86%	91%	88%	85%
ABL-ATD	UNSW-NB15	87%	93%	89%	86%
有监督模型（GT）	CIC-IDS-2017	92%	95%	93%	91%
有监督模型（GT）	UNSW-NB15	94%	96%	95%	93%

模型设置	数据集	F1-$score$	AUC	Precision	Recall
完整ABL-ATD	CIC-IDS-2017	86%	91%	88%	85%
完整ABL-ATD	UNSW-NB15	87%	93%	89%	86%
移除一致性优化模块	CIC-IDS-2017	81%	85%	82%	80%
移除一致性优化模块	UNSW-NB15	83%	88%	84%	82%
移除逻辑演绎与标签修正模块	CIC-IDS-2017	78%	82%	80%	77%
移除逻辑演绎与标签修正模块	UNSW-NB15	79%	84%	81%	78%