基于Q-Learning的自动入侵响应决策方法

doi:10.3969/j.issn.1671-1122.2021.06.004

摘要/Abstract

摘要：

针对现有自动入侵响应决策自适应性差的问题,文章提出一种基于Q-Learning的自动入侵响应决策方法——Q-AIRD。Q-AIRD基于攻击图对网络攻防中的状态和动作进行形式化描述,通过引入攻击模式层识别不同能力的攻击者,从而做出有针对性的响应动作;针对入侵响应的特点,采用Softmax算法并通过引入安全阈值θ、稳定奖励因子μ和惩罚因子ν进行响应策略的选取;基于投票机制实现对策略的多响应目的评估,满足多响应目的的需求,在此基础上设计了基于Q-Learning的自动入侵响应决策算法。仿真实验表明,Q-AIRD具有很好的自适应性,能够实现及时、有效的入侵响应决策。

关键词: 强化学习, 自动入侵响应, Softmax算法, 多目标决策

Abstract:

Aiming at the problem of poor adaptability of existing automatic intrusion response decision-making, this paper proposes an automatic intrusion response decision-making method based on Q-Learning (Q-AIRD). Q-AIRD formalizes the states and actions of network attack and defense based on the attack graph, and introduces the attack mode layer to identify attackers with different abilities, so as to make more targeted response actions. According to the characteristics of intrusion response, the Softmax algorithm is adopted and the security threshold θ, stable reward factor μ and penalty factor ν are introduced to select the response strategy. Based on the voting mechanism, the multi-response purpose evaluation of the strategy is realized to meet the needs of the multi-response purpose. On this basis, an automatic intrusion response decision algorithm based on Q-Learning is designed. The simulation results show that Q-AIRD has good adaptability and can realize timely and effective intrusion response decision-making.

Key words: reinforcement learning, automatic intrusion response, Softmax algorithm, multi-objective decision-making

中图分类号:

TP309

刘璟, 张玉臣, 张红旗. 基于Q-Learning的自动入侵响应决策方法[J]. 信息网络安全, 2021, 21(6): 26-35.

LIU Jing*, ZHANG Yuchen, ZHANG Hongqi. Automatic Intrusion Response Decision-making Method Based on Q-Learning[J]. Netinfo Security, 2021, 21(6): 26-35.

图/表 22

图1

图2

图3

表1

表2

表3

表4

表5

表6

图4

表7

图5

表8

表9

表1

图6

图7

图8

图9

图10

参考文献 21

[1]	ZHANG Hengwei, HUANG Shirui. Markov Differential Game Model and Its Application in Network Security[J]. Acta Electronica Sinica, 2019,47(3):606-612.
	张恒巍, 黄世锐. Markov 微分博弈模型及其在网络安全中的应用[J]. 电子学报, 2019,47(3):606-612.
[2]	QIAN Yaguan, LU Hongbo, JI Shouling, et al. A Poisoning Attack on Intrusion Detection System Based on SVM[J]. Acta Electronica Sinica, 2019,47(1):59-65.
	钱亚冠, 卢红波, 纪守领, 等. 一种针对基于 SVM 入侵检测系统的毒性攻击方法[J]. 电子学报, 2019,47(1):59-65.
[3]	SRINIVASAN T, SESHADRI J, JONATHAN J, et al. A System for Power-aware Agent-based Intrusion Detection (SPAID) in Wireless Ad Hoc Networks[J]. Lecture Notes in Computer Science, 2005,3619(4):153-162.
[4]	INAYAT Z, GANI A, ANUAR N B, et al. Intrusion Response Systems: Foundations, Design, and Challenges[J]. Journal of Network and Computer Applications, 2016,62(2):53-74. doi: 10.1016/j.jnca.2015.12.006 URL
[5]	ANWAR S, ZAIN M J, ZOLKIPLI M F, et al. From Intrusion Detection to An Intrusion Response System: Fundamentals, Requirements, and Future Directions[J]. Algorithms, 2017,10(2):1-24. doi: 10.3390/a10010001 URL
[6]	KHOLIDY H A, ERRADI A, ABDELWAHED S, et al. A Risk Mitigation Approach for Autonomous Cloud Intrusion Response System[J]. Computing, 2016,98(11):1111-1135. doi: 10.1007/s00607-016-0495-8 URL
[7]	SHAMELI-SENDI A, LOUAFI H, HE Wenbo, et al. Dynamic Optimal Countermeasure Selection for Intrusion Response System[J]. IEEE Transactions on Dependable and Secure Computing, 2016,15(5):755-770. doi: 10.1109/TDSC.8858 URL
[8]	WU Y S, FOO B R, MAO Yuchun, et al. Automated Adaptive Intrusion Containment in Systems of Interacting Services[J]. Computer Networks, 2007,51(5):1334-1360. doi: 10.1016/j.comnet.2006.09.006 URL
[9]	UPPULURI P, SEKAR R. Experiences with Specification-based Intrusion Detection[C]// Springer. International Symposium on Recent Advances in Intrusion Detection, October 10-12, 2001, Davis, CA, USA. Heidelberg: Springer, 2001: 172-189.
[10]	SHI Jin, LU Yin, XIE Li. Dynamic Intrusion Response Based on Game Theory[J]. Journal of Computer Research and Development, 2008,45(5):747-757.
	石进, 陆音, 谢立. 基于博弈理论的动态入侵响应[J]. 计算机研究与发展, 2008,45(5):747-757.
[11]	SCHNACKENGERG D, HOLLIDAY H, SMITH R, et al. Cooperative Intrusion Traceback and Response Architecture (CITRA)[C]// IEEE. DARPA Information Survivability Conference & Exposition II, June 12-14, 2001, Anaheim, CA, USA. NJ: IEEE, 2001: 56-68.
[12]	NADEEM A, HOWARTH M P. An Intrusion Detection & Adaptive Response Mechanism for MANETs[J]. Ad Hoc Networks, 2014,13(2):368-380. doi: 10.1016/j.adhoc.2013.08.017 URL
[13]	MU Chengpo, LI Yingjiu. An Intrusion Response Decision-making Model Based on Hierarchical Task Network Planning[J]. Expert Systems with Applications, 2010,37(3):2465-2472. doi: 10.1016/j.eswa.2009.07.079 URL
[14]	STAKHANOVA N, BASU S, WONG J. A Cost-sensitive Model for Preemptive Intrusion Response Systems[C]// IEEE. 21st International Conference on Advanced Information Networking and Applications, May 21-23, 2007, Niagara Falls, ON, Canada. NJ: IEEE, 2007: 428-435.
[15]	HEI H, HOEGG C L, MCANDREW K, et al. It Rained What in Where? A Collaborative Approach to Improve Response and Remediation of Water Intrusions in Clinical Areas[J]. American Journal of Infection Control, 2020,48(8):27.
[16]	CARVER C A. Adaptive Agent-based Intrusion Response[D]. College Station: Texas A&M University, 2001.
[17]	HUTTER F, KOTTHOFF L, VANSCHOREN J. Automated Machine Learning: Methods, Systems, Challenges[M]. Heidelberg:Springer Nature, 2019.
[18]	YE Yun, XU Xishan, QI Zhichang, et al. Attack Graph Generation Algorithm for Large-scale Network System[J]. Journal of Computer Research and Development, 2013,50(10):2133-2139.
	叶云, 徐锡山, 齐治昌, 等. 大规模网络中攻击图自动构建算法研究[J]. 计算机研究与发展, 2013,50(10):2133-2139.
[19]	WANG Shuo, TANG Guangming, KOU Guang, et al. Attack Path Prediction Method Based on Causal Knowledge Net[J]. Journal on Communications, 2016,37(10):188-198.
	王硕, 汤光明, 寇广, 等. 基于因果知识网络的攻击路径预测方法[J]. 通信学报, 2016,37(10):188-198.
[20]	GHOSAL S, AAD V. Fundamentals of Nonparametric Bayesian Inference[M]. Cambridge: Cambridge University Press, 2017.
[21]	YANG Zhuoran, XIE Yuchen, WANG Zhaoran. A Theoretical Analysis of Deep Q-learning[EB/OL]. https://www.researchgate.net/publication/330102327_A_Theoretical_Analysis_of_Deep_Q-Learning , 2020-12-20.

$R_{s\to s'}^{{{a}_{1}}}$	描述	赋值
高	能够确认攻击者身份并有足够证据	1
中	能够确认攻击者身份并有少量证据	0.7
较低	能够确认攻击者身份但没有证据	0.4
低	只有少量用以猜测攻击者身份的信息	0.1
无	没有任何攻击者身份信息和证据	0

$R_{s\to s'}^{{{a}_{2}}}$	描述	赋值
高	完全掌握攻击方式、路径和意图	1
中	一定程度掌握攻击方式、路径和意图	0.7
较低	掌握部分攻击方式和路径	0.4
低	只发现受到攻击,无其他任何信息	0.1

$R_{s\to s'}^{{{a}_{3}}}$	描述	赋值
高	完全终止攻击	1
中	使攻击在可控范围内	0.7
较低	部分终止攻击但服务遭到破坏	0.4
低	小部分终止攻击,服务遭到严重破坏	0.1
无	无任何效果	0

$W_{k}^{c}$	描述	赋值
高	信息泄露会造成极其严重危害	1
中	信息泄露危害在“高”和“较低”之间	0.7
较低	信息泄露造成一定危害	0.4
低	信息可在需要时泄露	0.1
无	信息不涉密	0

$W_{k}^{i}$	描述	赋值
高	数据被篡改会造成极其严重危害	1
中	数据被篡改危害在“高”和“较低”之间	0.7
较低	数据被篡改造成一定危害	0.4
低	数据在部分情况下允许被篡改	0.1
无	数据被篡改无任何影响	0