Skip to main content
SpringerLink
Log in

Neural Machine Translation with Soft Reordering Knowledge

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1332))

Included in the following conference series:

  • 2332 Accesses

Abstract

The Transformer architecture has been widely used in sequence to sequence tasks since it was proposed. However, it only adds the representations of absolute positions to its inputs to make use of the order information of the sequence. It lacks explicit structures to exploit the reordering knowledge of words. In this paper, we propose a simple but effective method to incorporate the reordering knowledge into the Transformer translation system. The reordering knowledge of each word is obtained by an additional reordering-aware attention sublayer based on its semantic and contextual information. The proposed approach can be easily integrated into the existing framework of the Transformer. Experimental results on two public translation tasks demonstrate that our proposed method can achieve significant translation improvements over the basic Transformer model and also outperforms the existing competitive systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://www.statmt.org/wmt19/translation-task.html.

  2. 2.

    https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl.

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)

    Google Scholar 

  2. Brown, P.F., Pietra, S.D., Pietra, V.J.D., Mercer, R.L.: The mathematics of statistical machine translation: parameter estimation. Comput. Linguistics 19(2), 263–311 (1993)

    Google Scholar 

  3. Cettolo, M., Girardi, C., Federico, M.: Wit3: Web inventory of transcribed and translated talks. In: Conference of European Association for Machine Translation, pp. 261–268 (2012)

    Google Scholar 

  4. Chen, K., Wang, R., Utiyama, M., Sumita, E.: Neural machine translation with reordering embeddings. In: ACL (1), pp. 1787–1799. Association for Computational Linguistics (2019)

    Google Scholar 

  5. Chiang, D.: A hierarchical phrase-based model for statistical machine translation. In: ACL, pp. 263–270. The Association for Computer Linguistics (2005)

    Google Scholar 

  6. Gehring, J., Auli, M., Grangier, D., Dauphin, Y.N.: A convolutional encoder model for neural machine translation. In: ACL (1), pp. 123–135. Association for Computational Linguistics (2017)

    Google Scholar 

  7. Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: ICML. Proceedings of Machine Learning Research, vol. 70, pp. 1243–1252. PMLR (2017)

    Google Scholar 

  8. Kawara, Y., Chu, C., Arase, Y.: Recursive neural network based preordering for English-to-Japanese machine translation. In: ACL (3), pp. 21–27. Association for Computational Linguistics (2018)

    Google Scholar 

  9. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (Poster) (2015)

    Google Scholar 

  10. Meng, F., Zhang, J.: DTMT: a novel deep transition architecture for neural machine translation. In: AAAI, pp. 224–231. AAAI Press (2019)

    Google Scholar 

  11. Nakagawa, T.: Efficient top-down BTG parsing for machine translation preordering. In: ACL (1), pp. 208–218. The Association for Computer Linguistics (2015)

    Google Scholar 

  12. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL (1). The Association for Computer Linguistics (2016)

    Google Scholar 

  13. Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. In: NAACL-HLT (2), pp. 464–468. Association for Computational Linguistics (2018)

    Google Scholar 

  14. So, D.R., Le, Q.V., Liang, C.: The evolved transformer. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 5877–5886. PMLR (2019)

    Google Scholar 

  15. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS, pp. 3104–3112 (2014)

    Google Scholar 

  16. Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)

    Google Scholar 

  17. Zhang, J., Wang, M., Liu, Q., Zhou, J.: Incorporating word reordering knowledge into attention-based neural machine translation. In: ACL (1), pp. 1524–1534. Association for Computational Linguistics (2017)

    Google Scholar 

  18. Zhu, Z.: Evaluating neural machine translation in English-Japanese task. In: WAT, pp. 61–68. Workshop on Asian Translation (2015)

    Google Scholar 

Download references

Acknowledgments

This research work has been funded by the National Natural Science Foundation of China (Grant No. 61772337), the National Key Research and Development Program of China NO. 2016QY03D0604.

Author information

Authors and Affiliations

  1. School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Leiying Zhou, Jie Zhou, Wenjie Lu, Kui Meng & Gongshen Liu

Authors
  1. Leiying Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenjie Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kui Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gongshen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Kui Meng or Gongshen Liu .

Editor information

Editors and Affiliations

  1. Department of AI, Ping An Life, Shenzhen, China

    Haiqin Yang

  2. Faculty of Information Technology, King Mongkut's Institute of Technology Ladkrabang, Bangkok, Thailand

    Kitsuchart Pasupa

  3. City University of Hong Kong, Kowloon, Hong Kong

    Andrew Chi-Sing Leung

  4. Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, Hong Kong

    James T. Kwok

  5. School of Information Technology, King Mongkut's University of Technology Thonburi, Bangkok, Thailand

    Jonathan H. Chan

  6. The Chinese University of Hong Kong, New Territories, Hong Kong

    Irwin King

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhou, L., Zhou, J., Lu, W., Meng, K., Liu, G. (2020). Neural Machine Translation with Soft Reordering Knowledge. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Communications in Computer and Information Science, vol 1332. Springer, Cham. https://doi.org/10.1007/978-3-030-63820-7_79

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-63820-7_79

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-63819-1

  • Online ISBN: 978-3-030-63820-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation