Automatic Intrusion Response Decision-making Method Based on Q-Learning

doi:10.3969/j.issn.1671-1122.2021.06.004

Abstract

Abstract:

Aiming at the problem of poor adaptability of existing automatic intrusion response decision-making, this paper proposes an automatic intrusion response decision-making method based on Q-Learning (Q-AIRD). Q-AIRD formalizes the states and actions of network attack and defense based on the attack graph, and introduces the attack mode layer to identify attackers with different abilities, so as to make more targeted response actions. According to the characteristics of intrusion response, the Softmax algorithm is adopted and the security threshold θ, stable reward factor μ and penalty factor ν are introduced to select the response strategy. Based on the voting mechanism, the multi-response purpose evaluation of the strategy is realized to meet the needs of the multi-response purpose. On this basis, an automatic intrusion response decision algorithm based on Q-Learning is designed. The simulation results show that Q-AIRD has good adaptability and can realize timely and effective intrusion response decision-making.

Key words: reinforcement learning, automatic intrusion response, Softmax algorithm, multi-objective decision-making

CLC Number:

TP309

LIU Jing*, ZHANG Yuchen, ZHANG Hongqi. Automatic Intrusion Response Decision-making Method Based on Q-Learning[J]. Netinfo Security, 2021, 21(6): 26-35.

Figures/Tables 22

$R_{s\to s'}^{{{a}_{1}}}$	描述	赋值
高	能够确认攻击者身份并有足够证据	1
中	能够确认攻击者身份并有少量证据	0.7
较低	能够确认攻击者身份但没有证据	0.4
低	只有少量用以猜测攻击者身份的信息	0.1
无	没有任何攻击者身份信息和证据	0

$R_{s\to s'}^{{{a}_{2}}}$	描述	赋值
高	完全掌握攻击方式、路径和意图	1
中	一定程度掌握攻击方式、路径和意图	0.7
较低	掌握部分攻击方式和路径	0.4
低	只发现受到攻击,无其他任何信息	0.1

$R_{s\to s'}^{{{a}_{3}}}$	描述	赋值
高	完全终止攻击	1
中	使攻击在可控范围内	0.7
较低	部分终止攻击但服务遭到破坏	0.4
低	小部分终止攻击,服务遭到严重破坏	0.1
无	无任何效果	0

$W_{k}^{c}$	描述	赋值
高	信息泄露会造成极其严重危害	1
中	信息泄露危害在“高”和“较低”之间	0.7
较低	信息泄露造成一定危害	0.4
低	信息可在需要时泄露	0.1
无	信息不涉密	0

$W_{k}^{i}$	描述	赋值
高	数据被篡改会造成极其严重危害	1
中	数据被篡改危害在“高”和“较低”之间	0.7
较低	数据被篡改造成一定危害	0.4
低	数据在部分情况下允许被篡改	0.1
无	数据被篡改无任何影响	0

$R_{s\to s'}^{{{a}_{8}}}$	描述	赋值
高	响应在生效时间内占用较多系统资源	-1
中	响应在生效时间内占用较少系统资源	-0.7
低	操作代价非常小	-0.4
无	不采取响应,无代价	0

服务器	脆弱性（CVE编号）	描述
Web服务器	${{v}_{1}}$ （CVE-2016-3338）	获得root权限
	${{v}_{2}}$ （CVE-2016-3387）	获得user权限
	${{v}_{3}}$ （CVE-2017-0893）	获得user权限
应用服务器	${{v}_{4}}$ （CVE-2016-3343）	获得root权限
	${{v}_{5}}$ （CVE-2017-9758）	获得root权限
	${{v}_{6}}$ （CVE-2016-1261）	发起DoS攻击
数据库服务器	${{v}_{7}}$ （CVE-2016-2555）	获得root权限

$A({{s}_{2}})$	$A({{s}_{3}})$	$A({{s}_{4}})$	$A({{s}_{5}})$	$A({{s}_{6}})$
${{a}_{1}}$	${{a}_{1}}$	${{a}_{1}}$	${{a}_{1}}$	${{a}_{1}}$
${{a}_{2}}$	${{a}_{2}}$	${{a}_{2}}$	${{a}_{3}}$	${{a}_{3}}$
${{a}_{4}}$	${{a}_{3}}$	${{a}_{3}}$	${{a}_{4}}$	${{a}_{5}}$
${{a}_{7}}$	${{a}_{4}}$	${{a}_{9}}$	${{a}_{5}}$	${{a}_{9}}$
${{a}_{8}}$	${{a}_{5}}$		${{a}_{6}}$	${{a}_{10}}$
	${{a}_{6}}$		${{a}_{10}}$	${{a}_{11}}$

$\gamma$	$\alpha$	$\theta$	$\mu$	$\nu$	$\tau$
0.3	0.6	0.5	1.1	0.6	0.3

$r{{d}_{1}}$	$r{{d}_{2}}$	$r{{d}_{3}}$	$r{{d}_{4}}$	$r{{d}_{5}}$	$r{{d}_{6}}$	$r{{d}_{7}}$	$r{{d}_{8}}$
0	0	0	0.6	0	0.4	0	0

$r{{d}_{1}}$	$r{{d}_{2}}$	$r{{d}_{3}}$	$r{{d}_{4}}$	$r{{d}_{5}}$	$r{{d}_{6}}$	$r{{d}_{7}}$	$r{{d}_{8}}$
0	0	0	0.6	0	0.4	0	0
0.4	0	0	0	0	0.3	0	0.3

References 21

[1]	ZHANG Hengwei, HUANG Shirui. Markov Differential Game Model and Its Application in Network Security[J]. Acta Electronica Sinica, 2019,47(3):606-612.
	张恒巍, 黄世锐. Markov 微分博弈模型及其在网络安全中的应用[J]. 电子学报, 2019,47(3):606-612.
[2]	QIAN Yaguan, LU Hongbo, JI Shouling, et al. A Poisoning Attack on Intrusion Detection System Based on SVM[J]. Acta Electronica Sinica, 2019,47(1):59-65.
	钱亚冠, 卢红波, 纪守领, 等. 一种针对基于 SVM 入侵检测系统的毒性攻击方法[J]. 电子学报, 2019,47(1):59-65.
[3]	SRINIVASAN T, SESHADRI J, JONATHAN J, et al. A System for Power-aware Agent-based Intrusion Detection (SPAID) in Wireless Ad Hoc Networks[J]. Lecture Notes in Computer Science, 2005,3619(4):153-162.
[4]	INAYAT Z, GANI A, ANUAR N B, et al. Intrusion Response Systems: Foundations, Design, and Challenges[J]. Journal of Network and Computer Applications, 2016,62(2):53-74. doi: 10.1016/j.jnca.2015.12.006 URL
[5]	ANWAR S, ZAIN M J, ZOLKIPLI M F, et al. From Intrusion Detection to An Intrusion Response System: Fundamentals, Requirements, and Future Directions[J]. Algorithms, 2017,10(2):1-24. doi: 10.3390/a10010001 URL
[6]	KHOLIDY H A, ERRADI A, ABDELWAHED S, et al. A Risk Mitigation Approach for Autonomous Cloud Intrusion Response System[J]. Computing, 2016,98(11):1111-1135. doi: 10.1007/s00607-016-0495-8 URL
[7]	SHAMELI-SENDI A, LOUAFI H, HE Wenbo, et al. Dynamic Optimal Countermeasure Selection for Intrusion Response System[J]. IEEE Transactions on Dependable and Secure Computing, 2016,15(5):755-770. doi: 10.1109/TDSC.8858 URL
[8]	WU Y S, FOO B R, MAO Yuchun, et al. Automated Adaptive Intrusion Containment in Systems of Interacting Services[J]. Computer Networks, 2007,51(5):1334-1360. doi: 10.1016/j.comnet.2006.09.006 URL
[9]	UPPULURI P, SEKAR R. Experiences with Specification-based Intrusion Detection[C]// Springer. International Symposium on Recent Advances in Intrusion Detection, October 10-12, 2001, Davis, CA, USA. Heidelberg: Springer, 2001: 172-189.
[10]	SHI Jin, LU Yin, XIE Li. Dynamic Intrusion Response Based on Game Theory[J]. Journal of Computer Research and Development, 2008,45(5):747-757.
	石进, 陆音, 谢立. 基于博弈理论的动态入侵响应[J]. 计算机研究与发展, 2008,45(5):747-757.
[11]	SCHNACKENGERG D, HOLLIDAY H, SMITH R, et al. Cooperative Intrusion Traceback and Response Architecture (CITRA)[C]// IEEE. DARPA Information Survivability Conference & Exposition II, June 12-14, 2001, Anaheim, CA, USA. NJ: IEEE, 2001: 56-68.
[12]	NADEEM A, HOWARTH M P. An Intrusion Detection & Adaptive Response Mechanism for MANETs[J]. Ad Hoc Networks, 2014,13(2):368-380. doi: 10.1016/j.adhoc.2013.08.017 URL
[13]	MU Chengpo, LI Yingjiu. An Intrusion Response Decision-making Model Based on Hierarchical Task Network Planning[J]. Expert Systems with Applications, 2010,37(3):2465-2472. doi: 10.1016/j.eswa.2009.07.079 URL
[14]	STAKHANOVA N, BASU S, WONG J. A Cost-sensitive Model for Preemptive Intrusion Response Systems[C]// IEEE. 21st International Conference on Advanced Information Networking and Applications, May 21-23, 2007, Niagara Falls, ON, Canada. NJ: IEEE, 2007: 428-435.
[15]	HEI H, HOEGG C L, MCANDREW K, et al. It Rained What in Where? A Collaborative Approach to Improve Response and Remediation of Water Intrusions in Clinical Areas[J]. American Journal of Infection Control, 2020,48(8):27.
[16]	CARVER C A. Adaptive Agent-based Intrusion Response[D]. College Station: Texas A&M University, 2001.
[17]	HUTTER F, KOTTHOFF L, VANSCHOREN J. Automated Machine Learning: Methods, Systems, Challenges[M]. Heidelberg:Springer Nature, 2019.
[18]	YE Yun, XU Xishan, QI Zhichang, et al. Attack Graph Generation Algorithm for Large-scale Network System[J]. Journal of Computer Research and Development, 2013,50(10):2133-2139.
	叶云, 徐锡山, 齐治昌, 等. 大规模网络中攻击图自动构建算法研究[J]. 计算机研究与发展, 2013,50(10):2133-2139.
[19]	WANG Shuo, TANG Guangming, KOU Guang, et al. Attack Path Prediction Method Based on Causal Knowledge Net[J]. Journal on Communications, 2016,37(10):188-198.
	王硕, 汤光明, 寇广, 等. 基于因果知识网络的攻击路径预测方法[J]. 通信学报, 2016,37(10):188-198.
[20]	GHOSAL S, AAD V. Fundamentals of Nonparametric Bayesian Inference[M]. Cambridge: Cambridge University Press, 2017.
[21]	YANG Zhuoran, XIE Yuchen, WANG Zhaoran. A Theoretical Analysis of Deep Q-learning[EB/OL]. https://www.researchgate.net/publication/330102327_A_Theoretical_Analysis_of_Deep_Q-Learning , 2020-12-20.