A Covert Tunnel and Encrypted Malicious Traffic Detection Method Based on Multi-Model Fusion

doi:10.3969/j.issn.1671-1122.2024.05.004

Abstract

Abstract:

To evade detection, advanced persistent threat(APT) attackers often employ strategies such as encrypted malicious traffic and covert tunnels to conceal malicious activities, thereby increasing the difficulty of detection. Currently, most methods for detecting DNS covert tunnels are based on characteristics such as statistics, frequency, and packets. These methods are not well-suited for real-time detection, which can lead to data leaks. Therefore, it is necessary to detect based on individual DNS requests rather than performing statistical analysis on traffic, to achieve real-time and reliable detection. When the system determines that a single DNS request is tunnel traffic, it can respond accordingly to prevent data leaks. However, existing methods for detecting encrypted malicious traffic have issues such as the inability to fully extract traffic feature information, limited means of feature extraction, and underutilization of features. Thus, this paper proposed a method for detecting covert tunnel malicious encrypted traffic based on multi-model fusion. For DNS covert tunnels, the paper proposed a detection method that fused MLP, 1D-CNN, and RNN models and calculates the fusion results based on a proposed mathematical model. This method can monitor covert tunnels in real-time, further improving the overall detection accuracy. For encrypted malicious traffic, the paper proposed a parallel fusion detection method combining 1D-CNN and LSTM models. The parallel fusion model can more comprehensively extract feature information and reflect the full scope of the traffic data, thereby enhancing the detection accuracy of the model.

Key words: encrypt malicious traffic detection, DNS hidden tunnel detection, multi model fusion

CLC Number:

TP309

GU Guomin, CHEN Wenhao, HUANG Weida. A Covert Tunnel and Encrypted Malicious Traffic Detection Method Based on Multi-Model Fusion[J]. Netinfo Security, 2024, 24(5): 694-708.

Figures/Tables 28

References 37

[1]	REZAEI S, LIU XIN. Deep Learning for Encrypted Traffic Classification: An Overview[J]. IEEE Communications Magazine, 2019, 57(5): 76-81. doi: 10.1109/MCOM.2019.1800819
[2]	TANG Zhengzhi, ZENG Xuewen, GUO Zhichuan, et al. Malware Traffic Classification Based on Recurrence Quantification Analysis[J]. International Journal of Networking and Security, 2020, 22(3): 449-459.
[3]	WU Kemeng, ZHANG Yongzheng, TAO Yin. TDAE: Autoencoder-Based Automatic Feature Learning Method for the Detection of DNS Tunnel[C]// IEEE. 2020 IEEE International Conference on Communications (ICC)2020. New York: IEEE, 2020: 1-7.
[4]	YAN Chuyu, GAO Songfeng, WANG Baohui. Research on Encrypted Malicious Traffic Detection[J]. New Industrialization, 2021, 11(10): 59-61.
	闫楚玉, 高嵩峰, 王宝会. 加密恶意流量检测研究[J]. 新型工业化, 2021, 11(10):59-61.
[5]	ANDERSON B, MCGREW D. Identifying Encrypted Malware Traffic with Contextual Flow Data[C]// ACM. Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security. New York: ACM, 2016: 35-46.
[6]	LI Yanmiao, GUO Hao, HOU Jiangang, et al. A Survey of Encrypted Malicious Traffic Detection[C]// IEEE. 2021 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI). New York: IEEE, 2021: 1-7.
[7]	WANG Wei, ZHU Ming, ZENG Xuewen, et al. Malware Traffic Classification Using Convolutional Neural Network for Representation Learning[C]// IEEE. 2017 International Conference on Information Networking (ICOIN). New York: IEEE, 2017: 712-717.
[8]	RADIVILOVA T, KIRICHENKO L, AGEYEV D, et al. Decrypting SSL/TLS Traffic for Hidden Threats Detection[C]// IEEE. 2018 IEEE 9th International Conference on Dependable Systems, Services and Technologies (DESSERT). New York: IEEE, 2018: 143-146.
[9]	KILIC F, ECKERT C. iDeFEND: Intrusion Detection Framework for Encrypted Network Data[C]// Springer. Cryptology and Network Security. Heidelberg: Springer, 2015: 111-118.
[10]	ZHANG Han, PAPADOPOULOS C, MASSEY D. Detecting Encrypted Botnet Traffic[C]// IEEE. 2013 Proceedings IEEE INFOCOM. New York: IEEE, 2013: 3453-3458.
[11]	HU Bin. Research on Detection of Malicious SSL/TLS Encrypted Traffic[D]. Shanghai: Shanghai Jiao Tong University, 2020.
	胡斌. 恶意 SSL/TLS 加密流量检测研究[D]. 上海: 上海交通大学, 2020.
[12]	SHEKHAWAT A S. Analysis of Encrypted Malicious Traffic[EB/OL]. (2018-05-18)[2023-09-05]. https://scholarworks.sjsu.edu/etd_projects/622/.
[13]	LEE I, ROH H, LEE W. Encrypted Malware Traffic Detection Using Incremental Learning[C]// IEEE. IEEE INFOCOM 2020-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). New York: IEEE, 2020: 1348-1349.
[14]	LI Li, REN Yifei, LOU Jiapeng. A Method for Malicious Encrypted Traffic Detection Based on Ensemble Learning[J]. Journal of Beijing Electronic Science and Technology Institute, 2021, 29(2): 8-16.
	李莉, 任逸飞, 娄嘉鹏. 一种基于集成学习的恶意加密流量检测方法[J]. 北京电子科技学院学报, 2021, 29(2):8-16.
[15]	LUO Ziming, XU Shubin, LIU Xiaodong. A TLS Malicious Encrypted Traffic Detection Scheme Based on Machine Learning[J]. Journal of Network and Information Security, 2020, 6(1): 77-83.
	骆子铭, 许书彬, 刘晓东. 基于机器学习的TLS恶意加密流量检测方案[J]. 网络与信息安全学报, 2020, 6(1):77-83.
[16]	JIANG Peng. Analysis and Detection of C&C Encrypted Channel Network Behavior Under APT Attacks[D]. Shanghai: Shanghai Normal University, 2018.
	姜鹏. APT 攻击下的 C&C 加密信道网络行为分析与检测[D]. 上海: 上海师范大学, 2018.
[17]	PRASSE P, MACHLICA L, PEVNÝ T, et al. Malware Detection by Analysing Network Traffic with Neural Networks[C]// IEEE. 2017 IEEE Security and Privacy Workshops (SPW). New York: IEEE, 2017: 205-210.
[18]	CHEN Lin, JIANG Yixi, KUANG Xiaoyun, et al. Deep Learning Detection Method of Encrypted Malicious Traffic for Power Grid[C]// IEEE. 2020 IEEE International Conference on Energy Internet (ICEI). New York: IEEE, 2020: 86-91.
[19]	ZENG Yi, GU Huaxi, WEI Wenting, et al. $ Deep-Full-Range $: A Deep Learning Based Network Encrypted Traffic Classification and Intrusion Detection Framework[J]. IEEE Access, 2019, 7: 45182-45190.
[20]	WANG Yue, ZHOU Anmin, LIAO Shan, et al. A Comprehensive Survey on DNS Tunnel Detection[EB/OL]. (2021-10-09)[2023-09-07]. https://www.sciencedirect.com/science/article/pii/S1389128621003248.
[21]	CROTTI M, DUSI M, GRINGOLI F, et al. Detecting Http Tunnels with Statistical Mechanisms[C]// IEEE. 2007 IEEE International Conference on Communications. New York: IEEE, 2007: 6162-6168.
[22]	DUSI M, CROTTI M, GRINGOLI F, et al. Tunnel Hunter: Detecting Application-Layer Tunnels with Statistical Fingerprinting[J]. Computer Networks, 2009, 53(1): 81-97.
[23]	CASAS P, MAZEL J, OWEZARSKI P. MINETRAC: Mining Flows for Unsupervised Analysis & Semi-Supervised Classification[C]// IEEE. 2011 23rd International Teletraffic Congress (ITC). New York: IEEE, 2011: 87-94.
[24]	WANG Hao. Research on Anomaly DNS Traffic Detection Based on Machine Learning[D]. Nanjing: Nanjing University of Posts and Telecommunications, 2019.
	王浩. 基于机器学习的异常 DNS 流量检测研究[D]. 南京: 南京邮电大学, 2019.
[25]	SAKARKAR G, KOLEKAR M K H, PAITHANKAR K, et al. Advance Approach for Detection of DNS Tunneling Attack from Network Packets Using Deep Learning Algorithms[J]. Advances in Distributed Computing and Artificial Intelligence Journal, 2021, 10(3): 241-266.
[26]	SHERIDAN S, KEANE A. Detection of DNS Based Covert Channels[EB/OL]. (2015-01-01)[2023-09-05]. https://www.researchgate.net/publication/282931276_Detection_of_DNS_based_covert_channels.
[27]	LIU Xiaolei, ZHANG Qiongyin, REN Lei, et al. Exploration of DNS Tunneling Trojan Detection Technology Based on Communication Behavior Analysis[J]. Science and Technology Information, 2018, 16(34): 17-18.
	刘晓蕾, 张琼尹, 任磊, 等. 基于通信行为分析的DNS隧道木马检测技术探究[J]. 科技资讯, 2018, 16(34):17-18.
[28]	NADLER A, AMINOV A, SHABTAI A. Detection of Malicious and Low Throughput Data Exfiltration over the DNS Protocol[J]. Computers & Security, 2019, 80: 36-53.
[29]	QI Cheng, CHEN Xiaojun, XU Cui, et al. A Bigram Based Real Time DNS Tunnel Detection Approach[J]. Procedia Computer Science, 2013, 17: 852-860.
[30]	YU Bin, SMITH L, THREEFOOT M, et al. Behavior Analysis Based DNS Tunneling Detection and Classification with Big Data Technologies[C]// IoTBD. Behavior Analysis Based DNS Tunneling Detection and Classification with Big Data Technologies. Rome: IoTBD, 2016: 284-290.
[31]	CHEN Shaojie, LANG Bo, LIU Hongyu, et al. DNS Covert Channel Detection Method Using the LSTM Model[EB/OL]. (2021-01-28)[2023-09-05]. https://www.sciencedirect.com/science/article/pii/S0167404820303680.
[32]	WANG Qi, XIE Kun, MA Yan, et al. DNS Tunnel Detection Based on Log Statistical Features[J]. Journal of Zhejiang University (Engineering Science), 2020, 54(9): 1753-1760.
	王琪, 谢坤, 马严, 等. 基于日志统计特征的 DNS 隧道检测[J]. 浙江大学学报(工学版), 2020, 54(9):1753-1760.
[33]	LIN Huaqing, LIU Gao, YAN Zheng. Detection of Application-Layer Tunnels with Rules and Machine Learning[C]// Springer. International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage. Heidelberg: Springer, 2019: 441-455.
[34]	ZHANG Meng, SUN Haoliang, YANG Peng. DNS Covert Channel Recognition Based on Improved Convolutional Neural Network[J]. Journal on Communications, 2020, 41(1): 169-179. doi: 10.11959/j.issn.1000-436x.2020017
	张猛, 孙昊良, 杨鹏. 基于改进卷积神经网络识别 DNS 隐蔽信道[J]. 通信学报, 2020, 41(1):169-179. doi: 10.11959/j.issn.1000-436x.2020017
[35]	FARNHAM G, ATLASIS A. Detecting DNS Tunneling[J]. SANS Institute InfoSec Reading Room, 2013, 9: 1-32.
[36]	CHEN Yang, LI Xiaoyong. A High Accuracy DNS Tunnel Detection Method without Feature Engineering[C]// IEEE. 2020 16th International Conference on Computational Intelligence and Security (CIS). New York: IEEE, 2020: 374-377.
[37]	GARCIA S, GRILL M, STIBOREK J, et al. An Empirical Comparison of Botnet Detection Methods[J]. Computers & Security, 2014, 45: 100-123.

模型	准确率	召回率	精确率	F1分数
MLP_1	0.966250	0.932510	0.999989	0.965072
MLP_2	0.966350	0.932720	0.999978	0.965179
MLP_3	0.970900	0.941830	0.999968	0.970029
MLP_4	0.978960	0.957960	0.999958	0.978509

模型	准确率	召回率	精确率	F1分数
1D-CNN_1	0.981635	0.963280	0.999990	0.981292
1D-CNN_2	0.981640	0.963370	0.999907	0.981298
1D-CNN_3	0.987630	0.975270	0.999990	0.987475
1D-CNN_4	0.981625	0.963250	1.000000	0.981281

模型	准确率	召回率	精确率	F1分数
RNN_1	0.967180	0.934420	0.999936	0.966068
RNN_2	0.966165	0.932390	0.999936	0.964982
RNN_3	0.981650	0.963400	0.999990	0.981309
RNN_4	0.980765	0.961600	0.999927	0.980389

模型	权重
MLP	33.08%
1D-CNN	33.66%
RNN	33.26%

模型	准确率	召回率	精确率	F1分数
MLP	0.978960	0.957960	0.999958	0.978509
1D-CNN	0.987630	0.975270	0.999990	0.987475
RNN	0.981650	0.963400	0.999896	0.981309
融合模型	0.990975	0.982560	0.999380	0.990898