A Static Detection Method of ROP Traffic Based on Bytes Fluctuation Characteristics

doi:10.3969/j.issn.1671-1122.2022.07.008

Abstract

Abstract:

Under the function of vulnerability mitigation mechanism of modern computer system, the traditional injection attack cannot realize function. Return-oriented programming (ROP) has become an indispensable part of vulnerability attack, which uses multiple gadgets to form the ROP chain to achieve the function of arbitrary operation execution. The detection of ROP chains in network traffic plays a vital role in preventing vulnerability attacks. This paper proposed a static detection method of ROP traffic that combined information entropy and variance to quantify the byte fluctuation characteristics of ROP chains through sequence extraction. Then, this paper leveraged CNN to capture such characteristics to precisely detect ROP chains in the traffic. The ROP chain was extracted from the real-world ROP code and randomly mixed with normal traffic to form a dataset for classification training. The model’s highest accuracy can reach 99.6%, the false negative rate can be kept below 2%, and the false positive rate can be kept below 1%. The method proposed in this paper realizes pure static ROP traffic detection with low system overhead and does not rely on information about memory addresses.

Key words: ROP, static detection, entropy, variance quantization

CLC Number:

TP309

ZHANG Mengjie, WANG Jian, HUANG Kaijie, YANG Gang. A Static Detection Method of ROP Traffic Based on Bytes Fluctuation Characteristics[J]. Netinfo Security, 2022, 22(7): 64-72.

Figures/Tables 12

References 18

[1]	ANDERSEN S. Changes to Functionality in Microsoft Windows XP Service Pack 2: Part 3: Memory Protection Technologies[EB/OL]. (2004-11-04)[2021-12-22]. https://oriolrius.cat/article_fitxers/1901/winsock2.pdf.
[2]	BLETSCH T, JIANG Xuxian, FREEH V W, et al. Jump-Oriented Programming: A New Class of Code-Reuse Attack[C]// ACM. 6th ACM Symposium on Information, Computer and Communications Security. New York: ACM, 2011: 30-40.
[3]	CHECKOWAY S, DAVI L, DMITRIENKO A, et al. Return-Oriented Programming without Returns[C]// ACM. 17th ACM Conference on Computer and Communications Security. New York: ACM, 2010: 559-572.
[4]	ROEMER R, BUCHANAN E, SHACHAM H, et al. Return-Oriented Programming: Systems, Languages, and Applications[J]. ACM Transactions on Information and System Security (TISSEC), 2012, 15(1): 1-34.
[5]	SNOW K Z, MONROSE F, DAVI L, et al. Just-in-Time Code Reuse: on the Effectiveness of Fine-Grained Address Space Layout Randomization[C]// IEEE. 2013 IEEE Symposium on Security and Privacy. New York: IEEE, 2013: 574-588.
[6]	CHEN Ping, XIAO Hai, SHEN Xiaobin, et al. DROP: Detecting Return-Oriented Programming Malicious Code[C]// Springer. International Conference on Information Systems Security. Heidelberg: Springer, 2009: 163-177.
[7]	PAPPAS V, POLYCHRONAKIS M, KEROMYTIS A D. Transparent ROP Exploit Mitigation Using Indirect Branch Tracing[C]// USENIX. 22nd USENIX Security Symposium. Berkeley: USENIX, 2013: 447-462.
[8]	CHENG Yueqiang, ZHOU Zongwei, MIAO Yu, et al. ROPecker: A Generic and Practical Approach for Defending against ROP Attack[EB/OL]. (2014-02-23)[2021-11-26]. https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=2972&context=sis_research.
[9]	ELSABAGH M, BARBARA D, FLECK D, et al. Detecting ROP with Statistical Learning of Program Characteristics[C]// ACM. 7th ACM on Conference on Data and Application Security and Privacy. New York: ACM, 2017: 219-226.
[10]	PFAFF D, HACK S, HAMMER C. Learning How to Prevent Return-Oriented Programming Efficiently[C]// Springer. International Symposium on Engineering Secure Software and Systems. Heidelberg: Springer, 2015: 68-85.
[11]	POLYCHRONAKIS M, KEROMYTIS A D. ROP Payload Detection Using Speculative Code Execution[C]// IEEE. 2011 6th International Conference on Malicious and Unwanted Software. New York: IEEE, 2011: 58-65.
[12]	USUI T, IKUSE T, OTSUKI Y, et al. ROPminer: Learning-Based Static Detection of ROP Chain Considering Linkability of ROP Gadgets[J]. IEICE Transactions on Information and Systems, 2020, 103(7): 1476-1492.
[13]	JÄMTHAGEN C, KARLSSON L, STANKOVSKI P, et al. EavesROP: Listening for ROP Payloads in Data Streams[C]// Springer. International Conference on Information Security. Heidelberg: Springer, 2014: 413-424.
[14]	TANAKA Y, GOTO A. N-ROPdetector: Proposal of a Method to Detect the ROP Attack Code on the Network[C]// ACM. 2014 Workshop on Cyber Security Analytics, Intelligence and Automation. New York: ACM, 2014: 33-36.
[15]	STANCILL B, SNOW K Z, OTTERNESS N, et al. Check My Profile: Leveraging Static Analysis for Fast and Accurate Detection of ROP Gadgets[C]// Springer. International Workshop on Recent Advances in Intrusion Detection. Heidelberg: Springer, 2013: 62-81.
[16]	LI Xusheng, HU Zhisheng, WANG Haizhou, et al. DeepReturn: A Deep Neural Network Can Learn How to Detect Previously-Unseen ROP Payloads without Using Any Heuristics[J]. Journal of Computer Security, 2020, 28(5): 499-523. doi: 10.3233/JCS-191368 URL
[17]	RONG Zelin, XIE Peidai, WANG Jingyuan, et al. Clean the Scratch Registers: A Way to Mitigate Return-Oriented Programming Attacks[C]// IEEE. 2018 IEEE 29th International Conference on Application-Specific Systems, Architectures and Processors (ASAP). New York: IEEE, 2018: 1-8.
[18]	WANG Wei, ZHU Ming, ZENG Xuewen, et al. Malware Traffic Classification Using Convolutional Neural Network for Representation Learning[C]// IEEE. 2017 International Conference on Information Networking (ICOIN). New York: IEEE, 2017: 712-717.

特点方法	低漏报	低误报	低系统开销	对运行程序的影响小
基于程序运行状态信息的检测方法	是	是	否	否
基于网络流量分析的检测方法	是	否	是	是

长度	准确率	误报率	漏报率	查准率	查全率	F1
5	0.9809	0.0042	0.0351	0.9953	0.9649	0.9798
6	0.9927	0.0028	0.0122	0.9969	0.9878	0.9923
7	0.9927	0.0028	0.0122	0.9969	0.9878	0.9923
8	0.9868	0.0071	0.0198	0.9923	0.9802	0.9862
9	0.9860	0.0085	0.0198	0.9907	0.9802	0.9854
10	0.9875	0.0085	0.0168	0.9908	0.9832	0.9870
11	0.9882	0.0085	0.0153	0.9908	0.9847	0.9877
12	0.9909	0.0056	0.0122	0.9913	0.9878	0.9895
13	0.9904	0.0042	0.0153	0.9954	0.9847	0.9900
14	0.9912	0.0028	0.0153	0.9969	0.9847	0.9908
15	0.9963	0.0014	0.0061	0.9985	0.9939	0.9962
16	0.9912	0.0014	0.0168	0.9984	0.9832	0.9908
17	0.9919	0.0000	0.0168	1.0000	0.9832	0.9915
18	0.9927	0.0000	0.0153	1.0000	0.9847	0.9923
19	0.9927	0.0000	0.0153	1.0000	0.9847	0.9923
20	0.9912	0.0000	0.0183	1.0000	0.9817	0.9908
21	0.9912	0.0000	0.0183	1.0000	0.9817	0.9908
22	0.9904	0.0000	0.0198	1.0000	0.9802	0.9900
23	0.9934	0.0014	0.0122	0.9985	0.9878	0.9931
24	0.9831	0.0000	0.0351	1.0000	0.9649	0.9821
25	0.9838	0.0000	0.0336	1.0000	0.9664	0.9829

检测数据	检测网络	准确率
原始流量	CNN	0.655
特征处理流量	CNN	0.996