融合互注意力机制与BERT的中文问答匹配技术研究

doi:10.3969/j.issn.1671-1122.2021.12.014

信息网络安全 ›› 2021, Vol. 21 ›› Issue (12): 102-108.doi: 10.3969/j.issn.1671-1122.2021.12.014

融合互注意力机制与BERT的中文问答匹配技术研究

代翔, 孙海春(), 牛硕, 朱容辰

中国人民公安大学信息网络安全学院,北京 100038

收稿日期:2021-09-29 出版日期:2021-12-10 发布日期:2022-01-11
通讯作者: 孙海春 E-mail:sunhaichun@ppsuc.edu.cn
作者简介:代翔（1996—）,男,河南,硕士研究生,主要研究方向为自然语言处理|孙海春（1985—）,女,讲师,博士,主要研究方向为信息服务、机器学习和面向服务的计算|牛硕（1999—）,男,本科,主要研究方向为网络安全、自然语言处理|朱容辰（1996—）,男, 硕士研究生,主要研究方向为网络安全、视频网络、机器学习
基金资助:
国家自然科学基金(41971367);国家重点研发计划(2017YFC0803700);公安部技术研究计划(2020JSYJC22ok)

Research on Chinese Question Answering Matching Based on Mutual Attention Mechanism and Bert

DAI Xiang, SUN Haichun(), NIU Shuo, ZHU Rongchen

School of Information and Network Security, People’s Public Security University of China, Beijing 100038, China

Received:2021-09-29 Online:2021-12-10 Published:2022-01-11
Contact: SUN Haichun E-mail:sunhaichun@ppsuc.edu.cn

摘要/Abstract

摘要：

问答匹配任务是问答系统关键技术之一,针对传统问答匹配模型对中文词向量表示不够精确、文本间交互特征提取不充分的问题,文章提出基于注意力的双向编码表征问答匹配模型。在中文向量表征上采用迁移学习引入预训练中文BERT模型参数,并在训练集上进一步微调获取最优参数,通过BERT模型对中文字向量进行表示,从而解决传统词向量模型在中文词汇上表征能力不足的问题。在文本交互层面,首先利用互注意力机制提取问题与答案的交互特征,并将生成的交互特征与注意力机制的输入向量形成特征组合;然后使用双向长短期记忆网络进行推理组合并降低特征维度,融入上下文语义信息;最后在中文法律数据集上进行测试。测试结果表明,该模型优于多项传统模型,与ESIM相比,在Top-1准确率上提高了3.55%,在MAP上提高了5.21%,在MRR上提高了4.05%。

关键词: 问答匹配, BERT, 互注意力机制

Abstract:

Question and answer matching task is one of the key technologies of question and answer system. Focusing on the problems that the traditional question and answer matching model is not accurate enough in the representation of Chinese word vector and insufficient extraction of interactive features between texts, a bi-directional encoder representation algorithm based on attention is proposed. In Chinese vector representation, transfer learning is used to introduce the pretrained Chinese BERT model parameters, and further finetune the training set to obtain the optimal parameters. The Chinese character vector is represented by the BERT model, so as to solve the problem of insufficient representation ability of the traditional word vector model in Chinese vocabulary. At the text interaction layer, the interactive features of questions and answers are extracted by using the mutual attention mechanism, and the generated interactive features are combined with the input vector of the attention mechanism to form a feature combination. Then BiLSTM is used for reasoning combination, reducing the feature dimension and integrating the context semantic information. Finally, it is tested on the Chinese legal data set. The experimental results show that the model is better than many traditional models. Compared with ESIM, it improves the accuracy of Top-1 by 3.55%, MAP by 5.21% and MRR by 4.05%.

Key words: question and answer matching, bi-directional encoder representation from transformers, mutual attention mechanism

中图分类号:

TP309

代翔, 孙海春, 牛硕, 朱容辰. 融合互注意力机制与BERT的中文问答匹配技术研究[J]. 信息网络安全, 2021, 21(12): 102-108.

DAI Xiang, SUN Haichun, NIU Shuo, ZHU Rongchen. Research on Chinese Question Answering Matching Based on Mutual Attention Mechanism and Bert[J]. Netinfo Security, 2021, 21(12): 102-108.

图/表 8

图1

图2

图3

表1

表2

图4

图5

图6

参考文献 14

[1]	FENG Minwei, XIANG Bing, GLASS M R, et al. Applying Deep Learning to Answer Selection: A Study and An Open Task[C]// IEEE. 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), December 13-17, 2015, Scottsdale, AZ, USA. NJ: IEEE, 2015: 813-820.
[2]	TAN Ming, DOS S C, XIANG Bing, et al. Improved Representation Learning for Question Answer Matching[C]// ACL. 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), August 7-12, 2016, Berlin, Germany. Stroudsburg, PA: ACL, 2016: 464-473.
[3]	DOS S C, TAN Ming, XIANG Bing, et al. Attentive Pooling Networks[EB/OL]. https://arxiv.org/abs/1602.03609, 2021-06-23.
[4]	LIU Yang, AI Qingyao, GUO Jiafeng, et al. ANMM: Ranking Short Answer Texts with Attention-based Neural Matching Model[C]// ACM. 25th ACM International on Conference on Information and Knowledge Management, October 24-28, 2016, Indiana, Indianapolis, USA. New York: ACM, 2016: 287-296.
[5]	TAY Y, PHAN M C, TUAN L A, et al. Learning to Rank Question Answer Pairs with Holographic Dual Lstm Architecture[C]// ACM. 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, August 7-11, 2017, Tokyo, Shinjuku, Japan. New York: ACM, 2017: 695-704.
[6]	TAY Y, TUAN L A, HUI S C. Hyperbolic Representation Learning for Fast and Efficient Neural Question Answering[C]// ACM. Proceedings 11th ACM International Conference on Web Search and Data Mining, February 5-9, 2018, CA, Marina Del Rey, USA. New York: ACM, 2018: 583-591.
[7]	WANG Zhiguo, HAMZA W, FLORIAN R. Bilateral Multi-perspective Matching for Natural Language Sentences[C]// AAAI. 26th International Joint Conference on Artificial Intelligence, August 19-25, 2017, Melbourne, Australia. Menlo Park, Ka: AAAI, 2017: 4144-4150.
[8]	SHEN Gehui, YANG Yunlun, DENG Zhihong. Inter-weighted Alignment Network for Sentence Pair Modeling[C]// ACL. 2017 Conference on Empirical Methods in Natural Language Processing, September 9-11, 2017, Copenhagen, Denmark. Stroudsburg: ACL, 2017: 1179-1189.
[9]	TAY Y, TUAN L A, HUI S C. Multi-cast Attention Networks[C]// ACM. 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, August 19-23, 2018, London, United Kingdom. New York: ACM, 2018: 2299-2308.
[10]	CHEN Kejin, HOU Junan, GUO Zhi, et al. Answer Selection Algorithm Based on Multi-scale Similarity Feature[J]. Systems Engineering and Electronic Technology, 2018, 40(6):1398-1404.
	陈柯锦, 侯俊安, 郭智, 等. 基于多尺度相似度特征的答案选择算法[J]. 系统工程与电子技术, 2018, 40(6):1398-1404.
[11]	CHEN Zhihao, YU Xiang, LIU Zichen, et al. Chinese Medical Question Answering Matching Method Based on Attention and Word Embedding[J]. Computer Applications, 2019, 39(6):1639-1645.
	陈志豪, 余翔, 刘子辰, 等. 基于注意力和字嵌入的中文医疗问答匹配方法[J]. 计算机应用, 2019, 39(6):1639-1645.
[12]	XIE Zhengwen, BAI Junxian, XIONG Xi, et al. Research on the Algorithm of Answer Selection Based on The Expression of Enhanced Question Importance[J]. Journal of Sichuan University (Natural Science Edition), 2020, 57(1):66-72.
	谢正文, 柏钧献, 熊熙, 等. 基于增强问题重要性表示的答案选择算法研究[J]. 四川大学学报(自然科学版), 2020, 57(1):66-72.
[13]	ZHANG Yangsen, WANG Sheng, WEI Wenjie, et al. Multi Stage Attention Answer Selection Model Integrating Semantic Information and Key Information of Questions[J]. Journal of Computer Science, 2021, 44(3):491-507.
	张仰森, 王胜, 魏文杰, 等. 融合语义信息与问题关键信息的多阶段注意力答案选取模型[J]. 计算机学报, 2021, 44(3):491-507.
[14]	MIKOLOV T, SUTSKEVER I, CHEN Kai, et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// NIPS. 27th Annual Conference on Neural Information Processing Systems 2013, December 5-8, 2013, Nevada, United States. CA: NIPS, 2013: 3111-3119.

Model	Top-1 Accuracy	MAP	MRR
Word2vec	40.95%	47.38%	54.49%
TextCNN	46.30%	52.89%	59.66%
QA-LSTM	47.15%	53.25%	60.00%
Word2vec-BA	51.20%	56.15%	63.50%
BERT	54.00%	61.16%	67.25%
BERT-TextCNN	55.25%	61.90%	67.90%
ESIM	58.85%	62.03%	69.29%
BERT-BA	62.40%	67.24%	73.34%

Margin值	Top-1 Accuracy	MAP	MRR
Margin=0.15	58.85%	64.74%	70.64%
Margin=0.2	62.40%	67.24%	73.34%
Margin=0.25	58.70%	65.47%	71.07%
Margin=0.3	58.65%	65.01%	70.88%

融合互注意力机制与BERT的中文问答匹配技术研究

Research on Chinese Question Answering Matching Based on Mutual Attention Mechanism and Bert

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 14

相关文章 3

编辑推荐

Metrics

本文评价

[1]	周枝凝, 王斌君, 翟一鸣, 仝鑫. 基于ALBERT动态词向量的垃圾邮件过滤模型[J]. 信息网络安全, 2020, 20(9): 107-111.
[2]	冯胥睿瑞, 刘嘉勇, 程芃森. 基于特征提取的恶意软件行为及能力分析方法研究[J]. 信息网络安全, 2019, 19(12): 72-78.
[3]	王武军. 基于Hilbert变换的数字图像置乱新算法[J]. , 2012, 12(3): 0-0.