Advertisement

Neural Machine Translation with Attention Based on a New Syntactic Branch Distance

  • Ru Peng
  • Zhitao Chen
  • Tianyong HaoEmail author
  • Yi FangEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1104)

Abstract

Attention mechanism has been proved to be able to improve the quality of neural machine translation by selectively focusing on partial words of a source sentence during translation process. Attention mechanism usually focuses on local attention by using solely the linear index distance of words while ignores syntax structures of sentences. In this paper, we extend local attention through syntax distance constraint, and propose an attention mechanism based on a new syntactic branch distance, which simultaneously pays attention to words with similar linear index distances and syntax-related words. Based on the English-to-German translation task, experiment results showed that our model outperforms a recent baseline method with an improvement of 1.61 BLEU points, demonstrating the effectiveness of the proposed model.

Keywords

Neural machine translation Attention mechanism Syntactic branch distance Syntax structure 

Notes

Acknowledgements

This work was supported by National Natural Science Foundation of China (No.61772146).

References

  1. 1.
    Kalchbrenner, N., Blunsom, P.: Recurrent continuous translation models. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700–1709 (2013)Google Scholar
  2. 2.
    Cho, K., Merrienboer, B.V., Gülçehre, Ç., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  3. 3.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sutskever, I., et al.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar
  4. 4.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2015)
  5. 5.
    Luong, M.T., Sutskever, I., Le, Q.V., et al.: Addressing the rare word problem in neural machine translation. Bull. Univ. Agric. Sci. Vet. Med. Cluj-Napoca. Vet. Med. 27(2), 82–86 (2014)Google Scholar
  6. 6.
    Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 (2015)
  7. 7.
    Sennrich, R., Haddow, B.: Linguistic input features improve neural machine translation. In: Proceedings of the First Conference on Machine Translation, Berlin, Germany, pp. 83–91. ACL (2016)Google Scholar
  8. 8.
    Eriguchi, A., Tsuruoka, Y., Cho, K.: Learning to parse and translate improves neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, pp. 72–78. ACL (2017)Google Scholar
  9. 9.
    Li, J., Xiong, D., Tu, Z., Zhu, M., Zhou, G.: Modeling source syntax for neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, pp. 688–697. ACL (2017)Google Scholar
  10. 10.
    Wu, S., Zhang, D., Yang, N., Li, M., Zhou, M.: Sequence-to-dependency neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1), Vancouver, Canada, pp. 698–707 (2017)Google Scholar
  11. 11.
    Eriguchi, A., Hashimoto, K., Tsuruoka, Y.: Tree-to-sequence attentional neural machine translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, pp. 823–833 (2016)Google Scholar
  12. 12.
    Chen, H., Huang, S., Chiang, D., Chen, J.: Improved neural machine translation with a syntax-aware encoder and decoder. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, pp. 1936–1945 (2017)Google Scholar
  13. 13.
    Wu, S., Zhou, M., Zhang, D.: Improved neural machine translation with source syntax. In: Proceedings of the Twenty Sixth International Joint Conference on Artificial Intelligence, IJCAI-2017, pp. 4179–4185 (2017)Google Scholar
  14. 14.
    Chen, K., Wang, R., Utiyama, M., Liu, L., Zhao, T., et al.: Neural machine translation with source dependency representation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 23–32 (2017)Google Scholar
  15. 15.
    Zeiler M.D.: ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012)
  16. 16.
    Sennrich, R., Firat, O., Cho, K., et al.: Nematus: a toolkit for neural machine translation. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, pp. 65–68 (2017)Google Scholar
  17. 17.
    Chang, P.C., Tseng, H., Jurafsky, D., Manning, C.D.: Discriminative reordering with Chinese grammatical relations features. In: Proceedings of the Third Workshop on Syntax and Structure in Statistical Translation, Boulder, Colorado, pp. 51–59 (2009)Google Scholar
  18. 18.
    Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
  19. 19.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311–318 (2002)Google Scholar
  20. 20.
    Wang, X., Pham, H., Yin, P., Neubig, G.: A tree-based decoder for neural machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4772–4777(2018)Google Scholar
  21. 21.
    Ahmed, M., Samee, M. R., Mercer, R. E.: Improving tree-LSTM with tree attention. In: Proceedings of the 2019 IEEE 13th International Conference on Semantic Computing, pp. 247–254(2019)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.School of Information EngineeringGuangdong University of TechnologyGuangzhouChina
  2. 2.School of Computer ScienceSouth China Normal UniversityGuangzhouChina

Personalised recommendations