Research on TTP Extraction Method Based on Pre-Trained Language Model and Chinese-English Threat Intelligence

doi:10.3969/j.issn.1671-1122.2024.07.009

Abstract

Abstract:

The tactics, techniques, and procedures (TTP) intelligence primarily resides in unstructured threat reports and serves as a valuable source of cyber threat intelligence. However, the existing open-source TTP classification label datasets are predominantly focused on the English domain, with limited coverage of source materials and TTP types, particularly lacking relevant data in the Chinese domain. To address this issue, this paper constructed a bilingual TTP intelligence dataset, bilingual threat intelligence classifying dataset (BTICD), which included 17700 samples and 236 corresponding TTPs. BTICD was the first to utilize publicly available Chinese threat report as corpora for TTP annotation and also annotated a portion of white-box samples that cannot be mapped to any TTP. This paper introduced and fine-tuned pre-trained models on the bilingual dataset to obtain a bilingual TTP identification model SecBiBERT. Experimental results show that SecBiBERT achieves a Micro F1 score of 86.49% on the 50 common TTP classification tasks and a Micro F1 score of 73.09% on the full set of 236 TTP classification tasks, which outperforms existing similar models.

Key words: TTP, threat intelligence, pre-trained language model

CLC Number:

TP309

REN Changyu, ZHANG Ling, JI Hangyuan, YANG Liqun. Research on TTP Extraction Method Based on Pre-Trained Language Model and Chinese-English Threat Intelligence[J]. Netinfo Security, 2024, 24(7): 1076-1087.

Figures/Tables 20

References 20

[1]	STROM B E, APPLEBAUM A, MILLER D P, et al. MITRE ATT&CK: Design and Philosophy[M]. Technical Report: The MITRE Corporation, 2018.
[2]	SHIN Y, KIM K, LEE J J, et al. ART: Automated Reclassification for Threat Actors Based on ATT&CK Matrix Similarity[C]// IEEE. 2021 World Automation Congress (WAC). New York: IEEE, 2021: 15-20.
[3]	JON B. Our TRAM Large Language Model Automates TTP Identification in CTI Reports[R]. Virginia: MITRE Engenuity, CT0075, 2023.
[4]	YOU Yizhe, JIANG Jun, JIANG Zhengwei, et al. TIM: Threat Context-Enhanced TTP Intelligence Mining on Unstructured Threat Data[J]. Cybersecurity, 2022, 5(1): 2523-3246.
[5]	ORBINATO V, BARBARACI M, NATELLA R, et al. Automatic Mapping of Unstructured Cyber Threat Intelligence: An Experimental Study: (Practical Experience Report)[C]// IEEE. 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE). New York: IEEE, 2022: 181-192.
[6]	LEGOY V, CASELLI M, SEIFERT C, et al. Automated Retrieval of Att&CK Tactics and Techniques for Cyber Threat Reports[EB/OL]. (2020-04-29)[2024-03-30]. https://arxiv.org/abs/2004.14322.
[7]	MARCHIORI F, CONTI M, VERDE N V. Stixnet: A Novel and Modular Solution for Extracting All Stix Objects in Cti Reports[C]// ACM. Proceedings of the 18th International Conference on Availability, Reliability and Security. New York: ACM, 2023: 1-11.
[8]	ABDEEN B, AL-SHAER E, SINGHAL A, et al. Smet: Semantic Mapping of Cve to ATT&CK and Its Application to Cybersecurity[C]// Springer. IFIP Annual Conference on Data and Applications Security and Privacy. Heidelberg: Springer, 2023: 243-260.
[9]	QIHOO360. Luwak TTP Extractor[EB/OL]. (2023-07-09)[2024-03-30]. https://github.com/Qihoo360/Luwak/tree/master.
[10]	RANI N, SAHA B, MAURYA V, et al. TTPHunter: Automated Extraction of Actionable Intelligence as TTPs from Narrative Threat Reports[C]// ACM. Proceedings of the 2023 Australasian Computer Science Week. New York: ACM, 2023: 126-134.
[11]	RANI N, SAHA B, MAURYA V, et al. TTPXHunter: Actionable Threat Intelligence Extraction as TTPs form Finished Cyber Threat Reports[EB/OL]. (2024-03-05)[2024-03-30]. https://arxiv.org/abs/2403.03267.
[12]	WU Shangyuan, SHEN Guowei, GUO Chun, et al. Threat Intelligence-Driven Dynamic Threat Hunting Method[J]. Netinfo Security, 2023, 23(6): 91-103.
	吴尚远, 申国伟, 郭春, 等. 威胁情报驱动的动态威胁狩猎方法[J]. 信息网络安全, 2023, 23(6):91-103.
[13]	AJMAL A B, ALAM M, KHALIQ A A, et al. Last Line of Defense: Reliability through Inducing Cyber Threat Hunting with Deception in Scada Networks[J]. IEEE Access, 2021, 9: 126789-126800.
[14]	BINDRA A. Securing the Power Grid: Protecting Smart Grids and Connected Power Systems from Cyberattacks[J]. IEEE Power Electronics Magazine, 2017, 4(3): 20-27.
[15]	ZHOU Yinghai, REN Yitong, YI Ming, et al. Cdtier: A Chinese Dataset of Threat Intelligence Entity Relationships[J]. IEEE Transactions on Sustainable Computing, 2023, 8(4): 627-638.
[16]	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding[C]// ACL. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: ACL, 2019: 4171-4186.
[17]	SUN Hongzhe, WANG Jian, WANG Peng, et al. Network Intrusion Detection Method Based on Attention-BiTCN[J]. Netinfo Security, 2024, 24(2): 309-318.
	孙红哲, 王坚, 王鹏, 等. 基于Attention-BiTCN的网络入侵检测方法[J]. 信息网络安全, 2024, 24(2):309-318.
[18]	HENDRYCKS D, GIMPEL K. Gaussian Error Linear Units (GELUs)[EB/OL]. (2016-06-27)[2024-03-30]. https://arxiv.org/abs/1606.08415.
[19]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal Loss for Dense Object Detection[C]// IEEE. 2017 IEEE International Conference on Computer Vision (ICCV). New York: IEEE, 2017: 2999-3007.
[20]	LIU Yinhan, OTT M, GOYAL N, et al. Roberta: A Robustly Optimized Bert Pretraining Approach[EB/OL]. (2019-07-26)[2024-03-30]. https://arxiv.org/abs/1907.11692.

参数名称	参数值
样本总数	17700条
中文样本数	8877条
英文样本数	8823条
样本平均长度	24.14词/条
样本最大长度	142词/条
样本最小长度	4词/条
单词总数	427263个

数据集	语种	样本数/条	TTP类别数/种
TRAM^[3]	英文	4070	50
文献[5]数据集	英文	12945	188
TTPHunter^[10]	英文	8387	50
BTICD（本文）	中文、英文	17700（含353条白样本）	236（含白标签）

样本内容	TTP标注
downpaper使用powershell执行	T1059.001
植入ngrok反向代理	T1090
netboy and revbshell, gather system information	T1082
hawkball 创建了一个 cmd.exe 反向 shell，执行命令并通过命令行上传输出	T1059.003
apt28 has used compromised email accounts to send credential phishing emails	T1586.002
blacklotus 尝试使用合法文件名隐藏其部署在 esp 上的文件，例如 grubx64.efi（如果在受感染计算机上启用了 uefi 安全启动）或 bootmgfw.efi（如果在受感染计算机上禁用了 uefi 安全启动）	T1036.005
apt32 has used scheduled task raw xml with a backdated timestamp of june 2, 2016. the group has also set the creation time of the files dropped by the second stage of the exploit to match the creation time of kernel32.dll. additionally, apt32 has used a random value to modify the timestamp of the file storing the clientid	T1070.006
当文件执行时，Windows 不会自动加载此数据，因为它位于 pe 结构之外	white
15c87b1820b67d4d2b082e81fd7946dd00a1072441b7551e38fccd5575bf18c2	white
In january 2023, there was a significant 41% decrease in ransomware victim posting rates across all groups compared to december 2022, signaling an overall decline in ransomware activities	white

划分	中文样本数/条	英文样本数/条	样本总数/条	TTP类别数/种
训练集	7028	7037	14065	236
测试集	1849	1786	3635	236

参数名称	参数值
文本特征向量维度	768
优化器	Adam
学习率	5e-5
损失函数	Focal loss
批次大小（条/批次）	12