Spam Filtering Model Based on ALBERT Dynamic Word Vector

doi:10.3969/j.issn.1671-1122.2020.09.022

Abstract

Abstract:

In order to solve the problem of insufficient word vector learning in spam classification, this paper introduces a model with ALBERT dynamic word vector, and proposes an ALBERT-RNN model which combines the ALBERT dynamic word vector with the recurrent neural network. In the open spam dataset (TEC06C), two traditional statistical models and four ALBERT-RNN models with different RNN structure are compared, and the cross entropy loss function of ALBERT-RNN is optimized by Focal Loss method. The experimental results show that the ALBERT-LSTM model with Focal Loss achieves the highest accuracy (99.13%) on the TEC06C dataset.

Key words: Chinese spam, recurrent neural network, ALBERT model, dynamic word vector

CLC Number:

TP309

ZHOU Zhining, WANG Binjun, ZHAI Yiming, TONG Xin. Spam Filtering Model Based on ALBERT Dynamic Word Vector[J]. Netinfo Security, 2020, 20(9): 107-111.

Figures/Tables 8

References 17

[1]	LIN Yanzhong, PEI Zhiyong, LIU Chuanqi, et al. Research Report on the Security of Chinese Enterprise Email in 2019[EB/OL]. https://shs3.b.qianxin.com/qax/345df09630c9033c11e8ad173c743fe4.pdf, 2020-1-13.
	林延忠, 裴志勇, 刘川琦, 等. 2019年中国企业邮箱安全性研究报告[EB/OL]. https://shs3.b.qianxin.com/qax/345df09630c9033c11e8ad173c743fe4.pdf, 2020-1-13.
[2]	GAO Feng. Will Spam Disappear in 2020[J]. Computer and Network, 2020,4:53.
	高枫. 垃圾邮件会在2020年消失吗[J]. 计算机与网络, 2020,4:53.
[3]	E Security. Ten Threats of Dark Network[J]. Information Security in China, 2018,9:110-111.
	E安全. 暗网的十大威胁[J]. 中国信息安全, 2018,9:110-111.
[4]	MOHAMED Bennasar, HICKS Yulia, SETCHI Rossitza. Feature Selection Using Joint Mutual Information Maximisation[EB/OL]. https://xueshu.baidu.com/usercenter/paper/show?paperid=c804e73158237c088c06a43ffb61aa54&site=xueshu_se, 2020-6-30.
[5]	LIU Li, CHEN Jie, FIEGUTH Paul, et al. From BoW to CNN: Two Decades of Texture Representation for Texture Classification[J]. International Journal of Computer Vision, 2019,127(1):74-109. doi: 10.1007/s11263-018-1125-z URL
[6]	MIKOLOV Tomas, CHEN Kai, CORRADO Greg S, et al. Efficient Estimation of Word Representations in Vector Space[EB/OL]. https://arxiv.org/abs/1301.3781, 2013-9-7.
[7]	PENNINGTON Jeffrey, SOCHER Richard, MANNING Christopher D. Glove: Global Vectors for Word Representation[C]// ACL. the 2014 Conference on Empirical Methods in Natural Language Processing, October 25-29, 2014, Doha, Qatar. Stroudsburg, PA: ACL, 2014: 1532-1543.
[8]	PETERS Matthew E, AMMAR Waleed, BHAGAVATULA Chandra, et al. Semi-supervised Sequence Tagging with Bidirectional Language Models[C]// ACL. the 55st Annual Meeting of the Association for Computational Linguistics, July 30-August 4, 2017, Vancouver, Canada. Stroudsburg, PA: ACL, 2017: 1756-1765.
[9]	PETERS Matthew E, NEUMANN Mark, IYYER Mohit, et al. Deep Contextualized Word Representations[EB/OL]. https://arxiv.org/abs/1802.05365, 2018-5-22.
[10]	DEVLIN Jacob, CHANG Mingwei, LEE Kenton, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[EB/OL]. https://arxiv.org/abs/1810.04805, 2019-5-24.
[11]	LAN Zhenzhong, CHEN Mingda, GOODMAN Sebastian, et al. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations[EB/OL]. https://arxiv.org/abs/1909.11942, 2020-2-9.
[12]	VASWANI Ashish, SHAZEER Noam, PARMAR Niki, et al. Attention is All You Need[EB/OL]. https://arxiv.org/abs/1706.03762, 2017-12-6.
[13]	ELMAN Jeffrey L. Finding Structure in Time[J]. Cognitive Science, 1990,14(2):179-211. doi: 10.1207/s15516709cog1402_1 URL
[14]	WANG Xin, LIU Yuanchao, SUN Chengjie, et al. Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory[C]// ACL. the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing,July 26-31, 2015, Beijing, China. Stroudsburg, PA: ACL, 2015: 1343-1353.
[15]	CHO Kyunghyun, MERRIENBOER Bart Van, GULCEHRE Caglar, et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[EB/OL]. https://arxiv.org/abs/1406.1078, 2014-9-3.
[16]	SRIVASTAVA Nitish, HINTON Geoffrey E, KRIZHEVSKY Alex, et al. Dropout: a Simple Way to Prevent Neural Networks from Overfitting[J]. Journal of Machine Learning Research, 2014,15(1):1929-1958.
[17]	LIN Tsungyi, GOYAL Priya, GIRSHICK Ross, et al. Focal Loss for Dense Object Detection[EB/OL]. https://arxiv.org/abs/1708.02002, 2018-2-7.

模型		隐藏层数	隐藏节点数	参数大小
BERT	base	12	768	108M
BERT	large	24	1024	334M
ALBERT	base	12	768	12M
ALBERT	large	24	1024	18M

模型名称	准确率	召回率	精确值	F1得分值
KNN	0.7890	0.7948	0.7379	0.7581
SVM	0.8180	0.7893	0.8380	0.8051
ALBERT-LSTM	0.9880	0.9828	0.9919	0.9867
ALBERT-BiLSTM	0.9857	0.9757	0.9937	0.9839
ALBERT-GRU	0.9823	0.9705	0.9910	0.9798
ALBERT-BiGRU	0.9870	0.9791	0.9935	0.9855

模型名称	准确率	召回率	精确值	F1值
ALBERT-LSTM	0.9913	0.9966	0.9865	0.9910
ALBERT-BiLSTM	0.9910	0.9949	0.9884	0.9908
ALBERT-GRU	0.9910	0.9955	0.9870	0.9907
ALBERT-BiGRU	0.9910	0.9932	0.9898	0.9910