Research on English-Chinese Machine Translation Based on Sentence Grouping

doi:10.3969/j.issn.1671-1122.2021.07.008

Abstract

Abstract:

Although neural machine translation models can obtain improvements when using larger data set for training, the information about categories and structures of sentences in the data set has not been properly utilized. This paper proposes a neural machine translation model based on sentence grouping, which adds a discriminator based on attention mechanism after encoders. In addition, this paper proposes a method to calculate the structural information vector of sentences as well. These vectors can be used to obtain the group labels by unsupervised method. Before training, sentences in the data set will be divided according to their content category and sentence structure to get group labels. Then the model is trained with these labels and parallel corpus at the same time, which will help the model identify the group that sentences belong to. In this way, the information in the data set can be more fully utilized. Sufficient comparative experiments show the rationality of the grouping idea. The translation results of Transformer model based on group architecture have been improved. Compared with the vanilla Transformer model, the BLEU score of our model has increased by at most 1.2.

Key words: machine translation, sentence grouping, structural information

CLC Number:

TP309

ZHAO Yuran, MENG Kui. Research on English-Chinese Machine Translation Based on Sentence Grouping[J]. Netinfo Security, 2021, 21(7): 63-71.

Figures/Tables 13

References 21

[1]	KALCHBRENNER N, BLUNSOM P. Recurrent Continuous Translation Models[C]// ACL. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, October 18-21, 2013, Washington, Stroudsburg: ACL, 2013: 1700-1709.
[2]	SUTSKEVER I, VINYALS O, LE Q V. Sequence to Sequence Learning with Neural Networks[C]// Neural Information Processing Systems Foundation. Proceedings of the 27th International Conference on Neural Information Processing Systems, December 8-13, 2014, Montreal, Quebec, Canada. New York: Curran Associates, 2014: 3104-3112.
[3]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All you Need[C]// Neural Information Processing Systems Foundation. Proceedings of the 31st International Conference on Neural Information Processing Systems, December 4-9, 2017, Long Beach, CA, USA. New York: Curran Associates, 2017: 5998-6008.
[4]	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// ACL. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, June 2-7, 2019, Minneapolis, MN, USA. Stroudsburg: ACL, 2019: 4171-4186.
[5]	SALTON G, FOX E A, WU H. Extended Boolean Information Retrieval[J]. Communications of the ACM, 1983, 26(11):1022-1036. doi: 10.1145/182.358466 URL
[6]	MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed Representations of Words and Phrases and their Compositionality[C]// Neural Information Processing Systems Foundation. Proceedings of the 26th International Conference on Neural Information Processing Systems, December 5-8, 2013, Lake Tahoe, Nevada, USA. New York: Curran Associates, 2013: 3111-3119.
[7]	PENNINGTON J, SOCHER R, MANNING C D. Glove: Global Vectors for Word Representation[C]// ACL. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, October 25-29, 2014, Doha, Qatar. Stroudsburg: ACL, 2014: 1532-1543.
[8]	DAI A M, LE Q V. Semi-supervised Sequence Learning[C]// Neural Information Processing Systems Foundation. Proceedings of the 28th International Conference on Neural Information Processing Systems, December 7-12, 2015, Montreal, Quebec, Canada. New York: Curran Associates, 2015: 3079-3087.
[9]	PETERS M, NEUMANN M, IYYER M, et al. Deep Contextualized Word Representations[C]// ACL. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, June 1-6, 2018, New Orleans, USA. Stroudsburg: ACL, 2018: 2227-2237.
[10]	CHEN T, GUESTRIN C. Xgboost: A Scalable Tree Boosting System[C]// ACM. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 13-17, 2016, San Francisco, CA, USA. New York: ACM, 2016: 785-794.
[11]	CORTES C, VAPNIK V. Support-vector Networks[J]. Machine Learning, 1995, 20(3):273-297.
[12]	LIU P, QIU X, HUANG X. Recurrent Neural Network for Text Classification with Multi-task Learning[C]// IJCAI. Proceedings of the 25th International Joint Conference on Artificial Intelligence, July 9-15, 2016, New York. Menlo Park: AAAI, 2016: 2873-2879.
[13]	KIM Y. Convolutional Neural Networks for Sentence Classification[C]// ACL. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, October 25-29, 2014, Doha, Qatar. Stroudsburg: ACL, 2014: 1746-1751.
[14]	LLOYD S. Least Squares Quantization in PCM[J]. IEEE Transactions on Information Theory, 1982, 28(2):129-136. doi: 10.1109/TIT.1982.1056489 URL
[15]	ARTHUR D, VASSILVITSKII S. K-means++: The Advantages of Careful Seeding[C]// ACM. Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms, January 7-9, 2007, New Orleans, Louisiana, USA. Philadelphia: SIAM, 2007: 1027-1035.
[16]	SCULLEY D. Web-scale K-means Clustering[C]// ACM. Proceedings of the 19th International Conference on World Wide Web, April 26-30, 2010, Raleigh, North Carolina, USA. New York: ACM, 2010: 1177-1178.
[17]	BRITZ D, LE Q, PRYZANT R. Effective Domain Mixing for Neural Machine Translation[C]// ACL. Proceedings of the Second Conference on Machine Translation, September 7-8, 2017, Copenhagen, Denmark. Stroudsburg: ACL, 2017: 118-126.
[18]	KIFER D, BEN-DAVID S, GEHRKE J. Detecting Change in Data Streams[C]// VLDB. Proceedings of the Thirtieth International Conference on Very Large Data Bases, August 31-September 3, 2004, Toronto, Canada. Trondheim, Norway: VLDB Endowment, 2004: 180-191.
[19]	BEND S, EIRON N, LONG P M. On the Difficulty of Approximately Maximizing Agreements[J]. Journal of Computer and System Sciences, 2003, 3(66):496-514.
[20]	BEND S, BLITZER J, CRAMMER K, et al. Analysis of Representations for Domain Adaptation[C]// Neural Information Processing Systems Foundation. Proceedings of the 20th Annual Conference on Neural Information Processing Systems, December 4-7, 2006, Vancouver, British Columbia, Canada. Cambridge, USA: MIT Press, 2007: 137-144.
[21]	HEWITT J, MANNING C D. A Structural Probe for Finding Syntax in Word Representations[C]// ACL. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, June 2-7, 2019, Minneapolis, MN, USA. Stroudsburg: ACL, 2019: 4129-4138.

数据集来源	句子对数目/个	所属领域
CWMT	3,050,000	新闻
UN	15,886,041	政治
AI Challenger 2018	3,262,499	口语
医疗数据集	369,984	医疗

数据集组别	训练集/条	符号数/个	验证集/条	测试集/条
新闻	354,500	7,795,353	1,250	1,250
政治	312,500	7,792,509	1,250	1,250
口语	637,500	7,796,759	1,250	1,250
医疗	365,000	7,798,532	1,250	1,250

数据集句子长度	训练集/条	验证集/条	测试集/条
[1,10]	322,382	1,368	1,443
[11,17]	366,472	1,462	1,590
[18,24]	368,691	944	851
[25, ∞]	402,455	1,226	1,126

数据集组别	训练集/条	验证集/条	测试集/条
组别1	722,172	3,036	2,534
组别2	109,725	372	468
组别3	330,716	818	1,136
组别4	297,387	774	862

组别组别	口语	新闻	政治	医疗
口语		1.01	1.81	1.98
新闻	1.01		1.48	1.88
政治	1.81	1.48		1.98
医疗	1.98	1.88	1.98