Research on Chinese Question Answering Matching Based on Mutual Attention Mechanism and Bert

doi:10.3969/j.issn.1671-1122.2021.12.014

Abstract

Abstract:

Question and answer matching task is one of the key technologies of question and answer system. Focusing on the problems that the traditional question and answer matching model is not accurate enough in the representation of Chinese word vector and insufficient extraction of interactive features between texts, a bi-directional encoder representation algorithm based on attention is proposed. In Chinese vector representation, transfer learning is used to introduce the pretrained Chinese BERT model parameters, and further finetune the training set to obtain the optimal parameters. The Chinese character vector is represented by the BERT model, so as to solve the problem of insufficient representation ability of the traditional word vector model in Chinese vocabulary. At the text interaction layer, the interactive features of questions and answers are extracted by using the mutual attention mechanism, and the generated interactive features are combined with the input vector of the attention mechanism to form a feature combination. Then BiLSTM is used for reasoning combination, reducing the feature dimension and integrating the context semantic information. Finally, it is tested on the Chinese legal data set. The experimental results show that the model is better than many traditional models. Compared with ESIM, it improves the accuracy of Top-1 by 3.55%, MAP by 5.21% and MRR by 4.05%.

Key words: question and answer matching, bi-directional encoder representation from transformers, mutual attention mechanism

CLC Number:

TP309

DAI Xiang, SUN Haichun, NIU Shuo, ZHU Rongchen. Research on Chinese Question Answering Matching Based on Mutual Attention Mechanism and Bert[J]. Netinfo Security, 2021, 21(12): 102-108.

Figures/Tables 8

References 14

[1]	FENG Minwei, XIANG Bing, GLASS M R, et al. Applying Deep Learning to Answer Selection: A Study and An Open Task[C]// IEEE. 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), December 13-17, 2015, Scottsdale, AZ, USA. NJ: IEEE, 2015: 813-820.
[2]	TAN Ming, DOS S C, XIANG Bing, et al. Improved Representation Learning for Question Answer Matching[C]// ACL. 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), August 7-12, 2016, Berlin, Germany. Stroudsburg, PA: ACL, 2016: 464-473.
[3]	DOS S C, TAN Ming, XIANG Bing, et al. Attentive Pooling Networks[EB/OL]. https://arxiv.org/abs/1602.03609, 2021-06-23.
[4]	LIU Yang, AI Qingyao, GUO Jiafeng, et al. ANMM: Ranking Short Answer Texts with Attention-based Neural Matching Model[C]// ACM. 25th ACM International on Conference on Information and Knowledge Management, October 24-28, 2016, Indiana, Indianapolis, USA. New York: ACM, 2016: 287-296.
[5]	TAY Y, PHAN M C, TUAN L A, et al. Learning to Rank Question Answer Pairs with Holographic Dual Lstm Architecture[C]// ACM. 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, August 7-11, 2017, Tokyo, Shinjuku, Japan. New York: ACM, 2017: 695-704.
[6]	TAY Y, TUAN L A, HUI S C. Hyperbolic Representation Learning for Fast and Efficient Neural Question Answering[C]// ACM. Proceedings 11th ACM International Conference on Web Search and Data Mining, February 5-9, 2018, CA, Marina Del Rey, USA. New York: ACM, 2018: 583-591.
[7]	WANG Zhiguo, HAMZA W, FLORIAN R. Bilateral Multi-perspective Matching for Natural Language Sentences[C]// AAAI. 26th International Joint Conference on Artificial Intelligence, August 19-25, 2017, Melbourne, Australia. Menlo Park, Ka: AAAI, 2017: 4144-4150.
[8]	SHEN Gehui, YANG Yunlun, DENG Zhihong. Inter-weighted Alignment Network for Sentence Pair Modeling[C]// ACL. 2017 Conference on Empirical Methods in Natural Language Processing, September 9-11, 2017, Copenhagen, Denmark. Stroudsburg: ACL, 2017: 1179-1189.
[9]	TAY Y, TUAN L A, HUI S C. Multi-cast Attention Networks[C]// ACM. 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, August 19-23, 2018, London, United Kingdom. New York: ACM, 2018: 2299-2308.
[10]	CHEN Kejin, HOU Junan, GUO Zhi, et al. Answer Selection Algorithm Based on Multi-scale Similarity Feature[J]. Systems Engineering and Electronic Technology, 2018, 40(6):1398-1404.
	陈柯锦, 侯俊安, 郭智, 等. 基于多尺度相似度特征的答案选择算法[J]. 系统工程与电子技术, 2018, 40(6):1398-1404.
[11]	CHEN Zhihao, YU Xiang, LIU Zichen, et al. Chinese Medical Question Answering Matching Method Based on Attention and Word Embedding[J]. Computer Applications, 2019, 39(6):1639-1645.
	陈志豪, 余翔, 刘子辰, 等. 基于注意力和字嵌入的中文医疗问答匹配方法[J]. 计算机应用, 2019, 39(6):1639-1645.
[12]	XIE Zhengwen, BAI Junxian, XIONG Xi, et al. Research on the Algorithm of Answer Selection Based on The Expression of Enhanced Question Importance[J]. Journal of Sichuan University (Natural Science Edition), 2020, 57(1):66-72.
	谢正文, 柏钧献, 熊熙, 等. 基于增强问题重要性表示的答案选择算法研究[J]. 四川大学学报(自然科学版), 2020, 57(1):66-72.
[13]	ZHANG Yangsen, WANG Sheng, WEI Wenjie, et al. Multi Stage Attention Answer Selection Model Integrating Semantic Information and Key Information of Questions[J]. Journal of Computer Science, 2021, 44(3):491-507.
	张仰森, 王胜, 魏文杰, 等. 融合语义信息与问题关键信息的多阶段注意力答案选取模型[J]. 计算机学报, 2021, 44(3):491-507.
[14]	MIKOLOV T, SUTSKEVER I, CHEN Kai, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// NIPS. 27th Annual Conference on Neural Information Processing Systems 2013, December 5-8, 2013, Nevada, United States. CA: NIPS, 2013: 3111-3119.

Model	Top-1 Accuracy	MAP	MRR
Word2vec	40.95%	47.38%	54.49%
TextCNN	46.30%	52.89%	59.66%
QA-LSTM	47.15%	53.25%	60.00%
Word2vec-BA	51.20%	56.15%	63.50%
BERT	54.00%	61.16%	67.25%
BERT-TextCNN	55.25%	61.90%	67.90%
ESIM	58.85%	62.03%	69.29%
BERT-BA	62.40%	67.24%	73.34%

Margin值	Top-1 Accuracy	MAP	MRR
Margin=0.15	58.85%	64.74%	70.64%
Margin=0.2	62.40%	67.24%	73.34%
Margin=0.25	58.70%	65.47%	71.07%
Margin=0.3	58.65%	65.01%	70.88%

[1]	GUO Xiangmin, LIANG Guangjun, XIA Lingling. Domain-Flux Malicious Domain Name Detection and Analysis Based on HMM [J]. Netinfo Security, 2021, 21(12): 1-8.
[2]	MA Rui, CAI Manchun, PENG Shufan. A Deep Forgery Video Detection Model Based on Improved Xception Network [J]. Netinfo Security, 2021, 21(12): 109-117.
[3]	WANG Xirui, LU Tianliang, ZHANG Jianling, DING Meng. Tor Anonymous Traffic Identification Method Based on Weighted Stacking Ensemble Learning [J]. Netinfo Security, 2021, 21(12): 118-125.
[4]	WANG Wei, HUANG Shuhua. The Digital File Confidential Cabinet Based on Symmetric Encryption [J]. Netinfo Security, 2021, 21(12): 19-24.
[5]	LUO Wenhua, ZHANG Yaowen. Analysis on Characteristics of Victims of Telecom Network Fraud Based on Bayesian Network [J]. Netinfo Security, 2021, 21(12): 25-30.
[6]	KANG Wenjie, ZHAO Wei, LIU Xuchong, SU Xin. Adjoint Relation Mining Model of Key Personnel Based on Discrete Trajectory [J]. Netinfo Security, 2021, 21(12): 31-37.
[7]	LI Pengchao, LIU Yanfei. Research on Forensics Technology of Malicious Code Based on Deleted PE File Header [J]. Netinfo Security, 2021, 21(12): 38-43.
[8]	CHEN Guangxuan, WU Jiajian, CAO Danni, XIE Qingquan. Research on iPhone Forensic Method Based on Checkm8 Vulnerability [J]. Netinfo Security, 2021, 21(12): 44-50.
[9]	XIONG Jianying. Reputation Evaluation Model in Social Network Based on Information Behavior [J]. Netinfo Security, 2021, 21(12): 51-59.
[10]	NI Xueli, WANG Qun, LIANG Guangjun. Research on Authentication Method of WeChat Evidence [J]. Netinfo Security, 2021, 21(12): 60-69.
[11]	ZHANG Shuqing, CAI Zhiwen. Revocable Encryption Scheme Based on Accountability Attribute under Cloud Data [J]. Netinfo Security, 2021, 21(12): 70-77.
[12]	GAO Jian, WANG Kaiyue, HUANG Shuhua. Research on Security Evaluation Index System for Video Monitoring Network [J]. Netinfo Security, 2021, 21(12): 78-85.
[13]	HU Gangyi, PENG Jin. An Encryption Algorithm for Police Image [J]. Netinfo Security, 2021, 21(12): 86-90.
[14]	XU Guotian, LIU Mengmeng. Malware Detection Method Based on Improved Harris Hawks Optimization Synchronization Optimization Feature Selection [J]. Netinfo Security, 2021, 21(12): 9-18.
[15]	ZHU Rongchen, LI Xin, LIN Xiaonuan. The Security Risk Analysis Method for Video Private Network Based on Bayesian Network [J]. Netinfo Security, 2021, 21(12): 91-101.