A Dynamic Malware Detection Method Based on Ensemble Learning

doi:10.3969/j.issn.1671-1122.2025.01.014

Abstract

Abstract:

In the current network environment, constantly upgrading variants of malicious code pose significant challenges to network security. Although existing artificial intelligence models have shown significant effectiveness in detecting malicious code, there are still two undeniable shortcomings. Firstly, their generalization ability is poor. Although they perform well on training data, their performance is not ideal in actual testing due to the phenomenon of concept drift. Secondly, their robustness is poor and they are susceptible to attacks from adversarial samples. To solve the above problems, this paper proposed a dynamic detection method for malicious code based on ensemble learning. According to the different features of API sequences, statistical feature analysis module, semantic feature analysis module, and structural feature analysis module were respectively constructed. Each module performed targeted malicious code detection, and finally integrated the analysis results of each module to obtain the final detection conclusion. The experimental results on the Speakeasy dataset show that compared with existing research methods, this method has significant advantages in various performance indicators and good robustness, which can effectively resist two adversarial attack methods against API sequences.

Key words: malware detection, n-gram algorithm, Transformer encoder, graph neural network, adversarial attack

CLC Number:

TP309

LIU Qiang, WANG Jian, WANG Yanan, WANG Shan. A Dynamic Malware Detection Method Based on Ensemble Learning[J]. Netinfo Security, 2025, 25(1): 159-172.

Figures/Tables 28

References 32

[1]	Kaspersky Security Network. The Mobile Malware Threat Landscape in 2023[EB/OL]. (2024-02-06)[2024-09-10]. https://securelist.com/mobile-malware-report-2023/111964.
[2]	LI Sicong, WANG Jian, SONG Yafei, et al. Malicious Code Classification Method Based on BiTCN-DLP[J]. Netinfo Security, 2023, 23(11): 104-117.
	李思聪, 王坚, 宋亚飞, 等. 基于BiTCN-DLP的恶意代码分类方法[J]. 信息网络安全, 2023, 23(11):104-117.
[3]	SUN Hongzhe, WANG Jian, WANG Peng, et al. Network Intrusion Detection Method Based on Attention-BiTCN[J]. Netinfo Security, 2024, 24(2): 309-318.
	孙红哲, 王坚, 王鹏, 等. 基于Attention-BiTCN的网络入侵检测方法[J]. 信息网络安全, 2024, 24(2): 309-318.
[4]	ZHANG Dandan, SONG Yafei, LIU Shu. MalMKNet: A Multi-Scale Convolutional Neural Network Used for Malware Classification[J]. Acta Electronica Sinica, 2023, 51(5): 1359-1369. doi: 10.12263/DZXB.20221069
	张丹丹, 宋亚飞, 刘曙. MalMKNet:一种用于恶意代码分类的多尺度卷积神经网络[J]. 电子学报, 2023, 51(5): 1359-1369. doi: 10.12263/DZXB.20221069
[5]	GALLORO N, POLINO M, CARMINATI M, et al. A Systematical and Longitudinal Study of Evasive Behaviors in Windows Malware[EB/OL]. (2021-12-05)[2024-09-10]. https://doi.org/10.1016/j.cose.2021.102550.
[6]	CHAI Yuhan. Research on Key Technologies for Malware Classification in Open World[D]. Guangzhou: Guangzhou University, 2023.
	柴瑜晗. 开放场景下恶意软件分类关键技术研究[D]. 广州: 广州大学, 2023.
[7]	QIAO Yong, YANG Yueyang, JI Lin, et al. Analyzing Malware by Abstracting the Frequent Itemsets in API Call Sequences[C]// IEEE. 2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications. New York: IEEE, 2013: 265-270.
[8]	UPPAL D, SINHA R, MEHRA V, et al. Malware Detection and Classification Based on Extraction of API Sequences[C]// IEEE. 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI). New York: IEEE, 2014: 2337-2342.
[9]	ALAZAB M, ALAZAB M, SHALAGINOV A, et al. Intelligent Mobile Malware Detection Using Permission Requests and API Calls[J]. Future Generation Computer Systems, 2020, 107: 509-521.
[10]	KOLOSNJAJI B, ZARRAS A, WEBSTER G, et al. Deep Learning for Classification of Malware System Call Sequences[C]// Springer. Advances in Artificial Intelligence:29th Australasian Joint Conference (AI 2016). Heidelberg: Springer, 2016: 137-149.
[11]	AGRAWAL R, STOKES J W, MARINESCU M, et al. Neural Sequential Malware Detection with Parameters[C]// IEEE. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New York: IEEE, 2018: 2656-2660.
[12]	ZHANG Zhaoqi, QI Panpan, WANG Wei. Dynamic Malware Analysis with Feature Engineering and Feature Learning[C]// AAAI. Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2020, 34(1): 1210-1217.
[13]	LI Ce, LYU Qiujian, LI Ning, et al. A Novel Deep Framework for Dynamic Malware Detection Based on API Sequence Intrinsic Features[EB/OL]. (2022-03-17)[2024-09-10]. https://doi.org/10.1016/j.cose.2022.102686.
[14]	ZHANG Sanfeng, WU Jiahao, ZHANG Mengzhe, et al. Dynamic Malware Analysis Based on API Sequence Semantic Fusion[EB/OL]. (2023-05-26)[2024-09-10]. https://doi.org/10.3390/app13116526.
[15]	DEMIRKIRAN F, ÇAYIR A, UNAL U, et al. An Ensemble of Pre-Trained Transformer Models for Imbalanced Multiclass Malware Classification[EB/OL]. (2022-08-06)[2024-09-10]. https://doi.org/10.1016/j.cose.2022.102846.
[16]	LI Yaping, LI Yuancheng. IoT Malware Threat Hunting Method Based on Improved Transformer[J]. International Journal of Network Security, 2023, 25(2): 267-276.
[17]	TRIZNA D, DEMETRIO L, BIGGIO B, et al. Nebula: Self-Attention for Dynamic Malware Analysis[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 6155-6167.
[18]	JIANG Haodi, TURKI T, WANG J T L. DLGraph: Malware Detection Using Deep Learning and Graph Embedding[C]// IEEE. 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). New York: IEEE, 2018: 1029-1033.
[19]	AMER E, ZELINKA I. A Dynamic Windows Malware Detection and Prediction Method Based on Contextual Understanding of API Call Sequence[EB/OL]. (2020-02-20)[2024-09-10]. https://doi.org/10.1016/j.cose.2020.101760.
[20]	XIAO Fei, LIN Zhaowen, SUN Yi, et al. Malware Detection Based on Deep Learning of Behavior Graphs[EB/OL]. (2019-02-11)[2024-09-10]. https://doi.org/10.1155/2019/8195395.
[21]	LI Ce, CHENG Zijun, ZHU He, et al. DMalNet: Dynamic Malware Analysis Based on API Feature Engineering and Graph Learning[EB/OL]. (2022-08-21)[2024-09-10]. https://doi.org/10.1016/j.cose.2022.102872.
[22]	DONG Shishi, HUANG Zhexue. A Brief Theoretical Overview of Random Forests[J]. Journal of Integration Technology, 2013, 2(1): 1-7.
[23]	PARMAR A, KATARIYA R, PATEL V. A Review on Random Forest: An Ensemble Classifier[C]// Springer. International Conference on Intelligent Data Communication Technologies and Internet of Things (ICICI). Heidelberg: Springer, 2019: 758-763.
[24]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All You Need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.
[25]	MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient Estimation of Word Representations in Vector Space[EB/OL]. (2013-09-07)[2024-09-10]. https://arxiv.org/pdf/1301.3781.
[26]	XU Keyulu, HU Weihua, LESKOVEC J, et al. How Powerful are Graph Neural Networks?[EB/OL]. (2018-10-04)[2024-09-10]. https://arxiv.org/pdf/1810.00826.pdf.
[27]	TRIZNA D. Quo Vadis: Hybrid Machine Learning Meta-Model Based on Contextual and Behavioral malware Representations[C]// ACM.Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security. New York: ACM, 2022: 127-136.
[28]	MANDIANT. Speakeasy: Portable, Modular, Binary Emulator Designed to Emulate Windows Kernel and User Mode Malware[EB/OL]. (2021-10-11)[2024-09-10]. https://github.com/mandiant/speakeasy.
[29]	JINDAL C, SALLS C, AGHAKHANI H, et al. Neurlux: Dynamic Malware Analysis without Feature Engineering[C]// ACM. Proceedings of the 35th Annual Computer Security Applications Conference. New York: ACM, 2019: 444-455.
[30]	ROSENBERG I, SHABTAI A, ELOVICI Y, et al. Query-Efficient Black-Box Attack against Sequence-Based Malware Classifiers[C]// ACM.Proceedings of the 36th Annual Computer Security Applications Conference. New York: ACM, 2020: 611-626.
[31]	YU Lantao, ZHANG Weinan, WANG Jun, et al. Seqgan: Sequence Generative Adversarial Nets with Policy Gradient[EB/OL]. (2017-02-13)[2024-09-10]. https://doi.org/10.1609/aaai.v31i1.10804.
[32]	Cuckoo Sandbox. Cuckoo Sandbox Hooked APIs and Categories[EB/OL]. (2019-08-24)[2024-09-10]. https://github.com/cuckoosandbox/cuckoo/wiki/Hooked-APIs-and-Categories.

参数	设置
n_estimators	100
criterion	gini
min_samples_split	2
min_samples_leaf	1
max_features	sqrt
random_state	42

类型	训练集/个	测试集/个
clean	24434	7944
backdoor	11062	1940
coinminer	6891	1684
dropper	8243	252
keylogger	4378	1041
ransomware	9627	2139
rat	1697	1258
trojan	8733	1085

实验环境	具体配置
操作系统	Windows 11
CPU	Intel（R） Core（TM） i7-13620H CPU @2.40GHz
内存	16GB
硬盘	1TB
显卡	NVIDIA GeForce RTX 4050
开发框架	PyTorch
开发语言	Python 3.10

	训练集				测试集
	Accuracy	F1-值	TPR（FPR=10^-3）	AUC	Accuracy	F1-值	TPR（FPR=10^-3）	AUC
2-gram	0.9592	0.9701	0.7962	0.9918	0.8432	0.8446	0.3879	0.9360
3-gram	0.9515	0.9646	0.7616	0.9879	0.8334	0.8363	0.3806	0.9351
4-gram	0.9348	0.9529	0.7207	0.9793	0.8317	0.8367	0.3856	0.9284
5-gram	0.9159	0.9404	0.6827	0.9665	0.8261	0.8386	0.3694	0.9202

集成方式	评价指标
集成方式	Accuracy	F1值	TPR（FPR=10^-3）	AUC
仅模块A	0.8432	0.8446	0.3879	0.9360
仅模块B	0.8711	0.8718	0.4112	0.9518
仅模块C	0.8300	0.8390	0.3467	0.8937
模块A+B	0.9255	0.9281	0.5242	0.9745
模块A+C	0.8827	0.8862	0.4448	0.9573
模块B+C	0.9121	0.9163	0.4926	0.9648
模块A+B+C	0.9338	0.9352	0.6370	0.9821