基于数据增强和模型更新的异常流量检测技术

doi:10.3969/j.issn.1671-1122.2020.02.009

摘要/Abstract

摘要：

网络攻击手段层出不穷,使得数据样本不断变化,导致异常检测精度低。传统网络异常流量检测方法通过规则匹配进行检测,该方法检测手段较简单,很难适应复杂灵活的大规模网络环境。为此,文章提出一种基于数据增强和模型更新的异常流量检测技术。为解决数据不平衡问题,文章引入SMOTE算法进行少数类样本的过采样,并结合ENN算法剔除噪音数据。通过随机森林算法提取样本特征的重要性,并在改进的KNN算法中以特征重要性作为距离度量实现模型更新。最后,采用带有分类特性的CatBoost分类算法对网络流量数据进行分类。该模型在模型迭代更新过程中,对异常流量的检测效果较好,与HCPTC-IDS等方法比较,检测精度和误报率都有所提升。利用KDD 99数据集进行实验的结果表明,该模型的多分类检测精度高达96.52%,并且误报率仅为0.92%。

关键词: 网络异常流量检测, 数据不平衡, 特征重要性, 模型更新, KDD 99

Abstract:

Due to the endless network attack means, the data samples are constantly changing, resulting in low accuracy of anomaly detection. The traditional network abnormal traffic detection method is detected by rule matching. The detection method is relatively simple, and it is difficult to adapt to a complex and flexible large-scale network environment. To this end, this paper proposes an abnormal traffic detection technology based on data augmentation and model update. In order to solve the problem of data imbalance, this paper introduces the SMOTE algorithm to oversample the minority samples, and removes the noise data with the ENN algorithm. The important features are extracted by the random forest algorithm, and the model update is implemented with the feature importance as the distance metric in the improved KNN algorithm. Finally, the CatBoost classification algorithm is used to classify network traffic data. In the model iterative update process, the detection of abnormal traffic is better. Compared with HCPTC-IDS, the detection accuracy and false positive rate are improved. The experimental results on the KDD 99 dataset show that the multi-classification detection accuracy of this model is as high as 96.52%, and the false positive rate is only 0.92%.

Key words: network abnormal traffic detection, data imbalance, character importance, model updating, KDD 99

中图分类号:

TP309

张浩, 陈龙, 魏志强. 基于数据增强和模型更新的异常流量检测技术[J]. 信息网络安全, 2020, 20(2): 66-74.

ZHANG Hao, CHEN Long, WEI Zhiqiang. Abnormal Traffic Detection Technology Based on Data Augmentation and Model Update[J]. Netinfo Security, 2020, 20(2): 66-74.

图/表 16

图1

图2

图3

图4

图5

图6

表1

表2

表3

图7

表4

表5

图8

表6

表7

图9

参考文献 15

[1]	Network Security Law of the People’s Republic of China[EB/OL]. , 2016-11-7.
	中华人民共和国网络安全法[EB/OL]. , 2016- 11-7.
[2]	BELOUCH M, EL S, IDHAMMAD M.A Two-Stage Classifier Approach Using RepTree Algorithm for Network Intrusion Detection[J]. International Journal of Advanced Computer Science and Applications, 2017, 8(6): 137-142.
[3]	JIA Fan, YAN Yan, ZHANG Jiaqi.k-means Based Feature Reduction for Network Anomaly Detection[J]. Journal of Tsinghua University(Science and Technology), 2018, 58(2): 137-142.
	贾凡,严妍,张家琪.基于k-means聚类特征消减的网络异常检测[J]. 清华大学学报(自然科学版),2018,58(2):137-142.
[4]	ASHFAQ R A R, WANG Xizhao, HUANG Zhexue, et al. Fuzziness Based Semi-Supervised Learning Approach for Intrusion Detection System[J]. Information Sciences, 2016, 378(C): 484-497.
[5]	AHMIM A, DERDOUR M, FERRAG M A.An Intrusion Detection System Based on Combining Probability Predictions of a Tree of Classifiers[J]. International Journal of Communication Systems, 2018, 31(9): e3547.
[6]	YANG Xudong, GAO Ling, WANG Hai, et al.A Cooperative Deep Belief Network for Intrusion Detection[C]//IEEE. 2018 Sixth International Conference on Advanced Cloud and Big Data, August 12-15, Lanzhou, China. New York: IEEE, 2018: 230-236.
[7]	GREGGIO N.Anomaly Detection in IDSs by Means of Unsupervised Greedy Learning of Finite Mixture Models[J]. Soft Computing, 2018, 22(10): 3357-3372.
[8]	CHAWLA N V, BOWYER K W, HALL L O, et al.SMOTE: Synthetic Minority Over-Sampling Technique[J]. Journal of Artificial Intelligence Research, 2002, 16(1): 321-357.
[9]	WILSON D L. Asymptotic Properties of Nearest Neighbor Rules Using Edited Data[J]. IEEE Transactions on Systems, Man and Cybernetics, 1972, SMC-2(3): 408-421.
[10]	BREIMAN L.Random Forests[J]. Machine Learning, 2001, 45(1): 5-32.
[11]	SYLVESTER E V A, BENTZEN P, BRADBURY I R, et al. Applications of Random Forest Feature Selection for Fine-Scale Genetic Population Assignment[J]. Evolutionary Applications, 2018, 11(2): 153-165.
[12]	PROKHORENKOVA L, GUSEV G, VOROBEV A, et al.CatBoost: Unbiased Boosting with Categorical Features[J]. Advances in Neural Information Processing Systems, 2018, 31(3): 6638-6648.
[13]	HART P E.The Condensed Nearest Neighbor Rule[J]. IEEE Transactions on Information Theory, 1968, 14(3): 515-516.
[14]	COHEN W W.Fast Effective Rule Induction[C]//ACM. Proceedings of the Twelfth International Conference on Machine Learning Machine Learning, July 9-12, 1995, Tahoe City, California. New York: ELSEVISER, 1995: 115-123.
[15]	JORDAN M I, BISHOP C.Neural Networks[J]. ACM Computing Surveys, 1996, 28(1): 73-75.

类别	KDD 99中10%训练集数据		KDD 99中10%测试集数据
类别	所有	去冗余	所有	去冗余
Normal	97278	87832	60593	47913
DoS	391458	54572	229853	23568
Probe	4107	2130	4166	2678
R2L	1126	999	16189	2913
U2R	52	52	228	215
Total	494021	145585	311029	77287

类型	Normal	DoS	Probe	R2L	U2R	Total
训练数据集	12000	24819	2130	999	52	40000
测试数据集	47913	23568	2678	2913	215	77287

特征	重要性	特征	重要性
dst_host_srv_count	0.1026	diff_srv_rate	0.0349
dst_bytes	0.0905	dst_host_same_srv_rate	0.0321
service	0.0828	dst_host_count	0.0303
logged_in	0.0750	flag	0.0284
dst_host_diff_srv_rate	0.0555	protocol_type	0.0271
count	0.0526	num_compromised	0.0269
srv_count	0.0483	num_file_creations	0.0265
serror_rate	0.0475	dst_host_srv_diff_host_rate	0.0238
dst_host_same_src_port_rate	0.0399	hot	0.0220
same_srv_rate	0.0361	duration	0.0181

类别	Normal	DoS	Probe	R2L	U2R	Total	类型
Num	12000	24819	2130	999	52	40000	Training
Num	24784	24801	24784	24799	24819	123987	SMOTE-ENN

		预测类型
		负类	正类
实际类型	负类	TN	FP
实际类型	正类	FN	TP