Abstract
Phrase information has been successfully integrated into current state-of-the-art neural machine translation (NMT) models. However, the natural property of the source and target phrase alignment has not been explored. In this paper, we propose a novel phrase-level agreement method to deal with this problem. First, we explore n-gram models over minimal translation units (MTUs) to explicitly capture aligned bilingual phrases from the parallel corpora. Then, we propose a phrase-level agreement loss that directly reduces the difference between the representations of the source-side and target-side phrase. Finally, we integrate the phrase-level agreement loss into the NMT models, to improve the translation performance. Empirical results on the NIST Chinese-to-English and the WMT English-to-German translation tasks demonstrate that the proposed phrase-level agreement method achieves significant improvements over state-of-the-art baselines, demonstrating the effectiveness and necessity of exploiting phrase-level agreement for NMT.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
The corpora include LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06.
- 2.
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of the 3rd ICLR (2015)
Chen, K., Wang, R., Utiyama, M., Sumita, E., Zhao, T.: Syntax-directed attention for neural machine translation. In: Proceedings of the 32nd AAAI (2018)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of EMNLP 2014 (2014)
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single \$&!#* vector: probing sentence embeddings for linguistic properties. In: Proceedings of the 56th ACL (2018)
Durrani, N., Schmid, H., Fraser, A.: A joint sequence translation model with integrated reordering. In: Proceedings of the 49th ACL (2011)
Hao, J., Wang, X., Shi, S., Zhang, J., Tu, Z.: Multi-granularity self-attention for neural machine translation. In: Proceedings of EMNLP-IJCNLP 2019 (2019)
Hao, J., Wang, X., Yang, B., Wang, L., Zhang, J., Tu, Z.: Modeling recurrence for transformer. In: NAACL 2019 (2019)
Hu, Y., Auli, M., Gao, Q., Gao, J.: Minimum translation modeling with recurrent neural networks. In: Proceedings of the 14th EACL (2014)
Huang, P., Wang, C., Huang, S., Zhou, D., Deng, L.: Towards neural phrase-based machine translation. In: Proceedings of the 6th ICLR (2018)
Kalchbrenner, N., Blunsom, P.: Recurrent continuous translation models. In: Proceedings of EMNLP 2013 (2013)
Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd ACL (2014)
Koehn, P.: Statistical Machine Translation, 1st edn. Cambridge University Press, Cambridge (2010)
Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Proceedings of NAACL 2003 (2003)
Kuang, S., Li, J., Branco, A., Luo, W., Xiong, D.: Attention focusing for neural machine translation by bridging source and target embeddings. In: Proceedings of the 56th ACL (2018)
Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. In: Proceedings of ICML 2014 (2014)
Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of EMNLP 2015 (2015)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. CoRR abs/1301.3781 (2013)
Mitchell, J., Lapata, M.: Composition in distributional models of semantics. Cogn. Sci. 38(4), 1388–1429 (2010)
Nguyen, P.X., Joty, S.: Phrase-based attentions. CoRR abs/1810.03444 (2018)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of 40th ACL (2002)
Park, C.Y., Tsvetkov, Y.: Learning to generate word- and phrase-embeddings for efficient phrase-based neural machine translation. In: Proceedings of the 3rd Workshop on Neural Generation and Translation (2019)
Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers (2018)
Quirk, C., Menezes, A.: Do we need phrases?: challenging the conventional wisdom in statistical machine translation. In: Proceedings of NAACL 2006 (2006)
Sennrich, R., Haddow, B., Birch, A.: Edinburgh neural machine translation systems for WMT 16. In: Proceedings of the First Conference on Machine Translation (2016)
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th ACL (2016)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, X., Lu, Z., Tu, Z., Li, H., Xiong, D., Zhang, M.: Neural machine translation advised by statistical machine translation. In: Proceedings of the 31th AAAI (2017)
Wang, X., Tu, Z., Wang, L., Shi, S.: Exploiting sentential context for neural machine translation. In: Proceedings of the 57th ACL (2019)
Wang, X., Tu, Z., Wang, L., Shi, S.: Self-attention with structural position representations. In: Proceedings of EMNLP-IJCNLP (2019)
Wang, X., Tu, Z., Xiong, D., Zhang, M.: Translating phrases in neural machine translation. In: Proceedings of EMNLP 2017 (2017)
Wuebker, J., Green, S., DeNero, J., Hasan, S., Luong, M.T.: Models and inference for prefix-constrained machine translation. In: Proceedings of the 54th ACL (2016)
Yang, B., Tu, Z., Wong, D.F., Meng, F., Chao, L.S., Zhang, T.: Modeling localness for self-attention networks. In: Proceedings of EMNLP 2018 (2018)
Yang, M., et al.: Sentence-level agreement for neural machine translation. In: Proceedings of the 57th ACL (2019)
Yang, M., Zhang, M., Chen, K., Wang, R., Zhao, T.: Neural machine translation with target-attention model. IEICE Trans. Inf. Syst. 103(3), 684–694 (2020)
Zhang, H., Toutanova, K., Quirk, C., Gao, J.: Beyond left-to-right: multiple decomposition structures for SMT. In: Proceedings of NAACL 2013 (2013)
Zhang, M., Wu, Y., Li, W., Li, W.: Learning universal sentence representations with mean-max attention autoencoder. In: Proceedings of EMNLP 2018 (2018)
Zhao, Y., Wang, Y., Zhang, J., Zong, C.: Phrase table as recommendation memory for neural machine translation. In: Proceedings of the 27th IJCAI (2018)
Zhou, L., Hu, W., Zhang, J., Zong, C.: Neural system combination for machine translation. In: Proceedings of the 55th ACL (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, M., Wang, X., Zhang, M., Zhao, T. (2020). Incorporating Phrase-Level Agreement into Neural Machine Translation. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-60450-9_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60449-3
Online ISBN: 978-3-030-60450-9
eBook Packages: Computer ScienceComputer Science (R0)