Skip to main content
SpringerLink
Log in

A user-study on online adaptation of neural machine translation to human post-edits

  • Published:
Machine Translation

Abstract

The advantages of neural machine translation (NMT) have been extensively validated for offline translation of several language pairs for different domains of spoken and written language. However, research on interactive learning of NMT by adaptation to human post-edits has so far been confined to simulation experiments. We present the first user study on online adaptation of NMT to user post-edits in the domain of patent translation. Our study involves 29 human subjects (translation students) whose post-editing effort and translation quality were measured on about 4500 interactions of a human post-editor and an NMT system integrating an online adaptive learning algorithm. Our experimental results show a significant reduction in human post-editing effort due to online adaptation in NMT according to several evaluation metrics, including hTER, hBLEU, and KSMR. Furthermore, we found significant improvements in BLEU/TER between NMT outputs and professional translations in granted patents, providing further evidence for the advantages of online adaptive NMT in an interactive setup.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. In addition, Green et al. (2014)—one of the first user studies on online adaptation to post-edits—performed system updates offline instead of online.

  2. http://data.statmt.org/wmt17/translation-task/training-parallel-nc-v12.tgz.

  3. http://www.cl.uni-heidelberg.de/statnlpgroup/pattr/.

  4. http://www.cl.uni-heidelberg.de/statnlpgroup/pattr/.

  5. e.g., https://www.linguee.com.

  6. http://www.ifs.tuwien.ac.at/imp/marec.shtml.

  7. http://www.dict.cc, http://dict.leo.org.

  8. https://de.wikipedia.org.

  9. A single document was used as an exam in the very last session and translated by all post-editors and without adaptation.

References

  • Baayen RH, Davidson DJ, Bates DM (2008) Mixed-effects modeling with crossed random effects for subjects and items. J Mem Lang 59(4):390–412

    Article  Google Scholar 

  • Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: Proceedings of the international conference on learning representations (ICLR), San Diego, pp 1–15

  • Barr DJ, Levy R, Scheepers C, Tilly HJ (2013) Random effects structure for confirmatory hypothesis testing: keep it maximal. J Mem Lang 68(3):255–278

    Article  Google Scholar 

  • Barrachina S, Bender O, Casacuberta F, Civera J, Cubel E, Khadivi S, Lagarda A, Ney H, Tomás J, Vidal E et al (2009) Statistical approaches to computer-assisted translation. Comput Linguist 35(1):3–28

    Article  MathSciNet  Google Scholar 

  • Bates D, Mächler M, Bolker B, Walker S (2015) Fitting linear mixed-effects models using lme4. J Stat Softw 67(1):1–48

    Article  Google Scholar 

  • Bentivogli L, Bertoldi N, Cettolo M, Federico M, Negri M, Turchi M (2016a) On the evaluation of adaptive machine translation for human post-editing. IEEE/ACM Trans Audio Speech Lang Process 24(2):388–399

    Article  Google Scholar 

  • Bentivogli L, Bisazza A, Cettolo M, Federico M (2016b) Neural versus phrase-based machine translation quality: a case study. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Austin, pp 257–267

  • Bentivogli L, Bisazza A, Cettolo M, Federico M (2018) Neural versus phrase-based MT quality: an in-depth analysis on English–German and English–French. Comput Speech Lang 49:52–70

    Article  Google Scholar 

  • Bertoldi N, Simianer P, Cettolo M, Wäschle K, Federico M, Riezler S (2014) Online adaptation to post-edits for phrase-based statistical machine translation. Mach Transl 28:309–339

    Article  Google Scholar 

  • Burchardt A, Macketanz V, Dehdari J, Heigold G, Peter JT, Williams P (2017) A linguistic evaluation of rule-based, phrase-based, and neural MT engines. Prague Bull Math Linguist 108(1):159–170

    Article  Google Scholar 

  • Castilho S, Moorkens J, Gaspari F, Calixto I, Tinsley J, Way A (2017a) Is neural machine translation the new state of the art? Prague Bull Math Linguist 108(1):109–120

    Article  Google Scholar 

  • Castilho S, Moorkens J, Gaspari F, Sennrich R, Sosoni V, Georgakopoulou Y, Lohar P, Way A, Miceli Barone A, Gialama M (2017b) A comparative quality evaluation of PBSMT and NMT using professional translators. In: Proceedings of MT Summit XVI, vol 1. Research Track, Nagoya, pp 116–131

  • Cesa-Bianchi N, Reverberi G, Szedmak S (2008) Online learning algorithms for computer-assisted translation. Technical report, SMART. http://www.smart-project.eu

  • Denkowski M, Dyer C, Lavie A (2014a) Learning from post-editing: online model adaptation for statistical machine translation. In: Proceedings of the conference of the European chapter of the association for computational linguistics (EACL), Gothenburg, pp 395–404

  • Denkowski M, Lavie A, Lacruz I, Dyer C (2014b) Real time adaptive machine translation for post-editing with cdec and transcenter. In: Proceedings of the EACL workshop on humans and computer-assisted translation, Gothenburg, pp 72–77

  • Farajian MA, Turchi M, Negri M, Bertoldi N, Federico M (2017) Neural vs. phrase-based machine translation in a multi-domain scenario. In: Proceedings of the conference of the european chapter of the association for computational linguistics (EACL), vol 2, Short Papers, Valencia, pp 280–284

  • Forcada ML (2017) Making sense of neural machine translation. Transl Spaces 6(2):291–309

    Article  Google Scholar 

  • Graham Y, Baldwin T, Moffat A, Zobel J (2016) Can machine translation systems be evaluated by the crowd alone? Nat Lang Eng 23(1):3–30

    Article  Google Scholar 

  • Green S, Heer J, Manning CD (2013) The efficacy of human post-editing for language translation. In: Proceedings of the SIGCHI conference on human factors in computing systems, Paris, pp 439–448

  • Green S, Wang S, Chuang J, Heer J, Schuster S, Manning CD (2014) Human effort and machine learnability in computer aided translation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Doha, pp 1225–1236

  • Hardt D, Elming J (2010) Incremental re-training for post-editing SMT. In: Proceedings of the conference of the association for machine translation in the Americas (AMTA), Denver, pp 1–10

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput. 9(8):1735–1780

    Article  Google Scholar 

  • Isabelle P, Cherry C, Foster G (2017) A challenge set approach to evaluating machine translation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Copenhagen, pp 2486–2496

  • Jean S, Firat O, Cho K, Memisevic R, Bengio Y (2015) Montreal neural machine translation systems for WMT’15. In: Proceedings of the workshop on statistical machine translation (WMT), Lisbon, pp 134–140

  • Junczys-Dowmunt M, Dwojak T, Hoang H (2016) Is neural machine translation ready for deployment? A case study on 30 translation directions. In: Proceedings of the international workshop on spoken language translation (IWSLT), Seattle, pp 1–8

  • Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the international conference on learning representations (ICLR), San Diego, pp 1–15

  • Klubička F, Toral A, Sánchez-Cartagena VM (2017) Fine-grained human evaluation of neural versus phrase-based machine translation. Prague Bull Math Linguist 108(1):121–132

    Article  Google Scholar 

  • Klubička F, Toral A, Sánchez-Cartagena VM (2018) Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian. Mach Transl 1–21

  • Knowles R, Koehn P (2016) Neural interactive translation prediction. In: Proceedings of the conference of the association for machine translation in the Americas (AMTA), Austin, pp 107–120

  • Koehn P (2005) Europarl: a parallel corpus for statistical machine translation. In: Conference proceedings: the tenth machine translation summit, Phuket, pp 79–86

  • Koehn P, Knowles R (2017) Six challenges for neural machine translation. In: Proceedings of the first workshop on neural machine translation, Vancouver, pp 28–39

  • Kreutzer J, Sokolov A, Riezler S (2017) Bandit structured prediction for neural sequence-to-sequence learning. In: Proceedings of the 55th annual meeting of the association for computational linguistics (ACL), vol 1, Long Papers, Vancouver, pp 1503–1513

  • Kreutzer J, Khadivi S, Matusov E, Riezler S (2018a) Can neural machine translation be improved with user feedback? In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 3, Industry Papers (NAACL-HLT), New Orleans, pp 92–105

  • Kreutzer J, Uyheng J, Riezler S (2018b) Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In: Proceedings of the 56th annual meeting of the association for computational linguistics (ACL), vol 1, Long Papers, Melbourne, pp 1777–1788

  • Lam TK, Kreutzer J, Riezler S (2018) A reinforcement learning approach to interactive-predictive neural machine translation. In: Proceedings of the 21st annual conference of the European association for machine translation (EAMT), Alicante, pp 169–178

  • López-Salcedo FJ, Sanchis-Trilles G, Casacuberta F (2012) Online learning of log-linear weights in interactive machine translation. In: Proceedings of IberSpeech: advances in speech and language technologies for Iberian languages, Madrid, pp 277–286

  • Luong M, Manning CD (2015) Stanford neural machine translation systems for spoken language domains. In: Proceedings of the international workshop on spoken language translation (IWSLT), Da Nang, pp 76–79

  • Luong M, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Lisbon, pp 1412–1421

  • Macketanz V, Avramidis E, Burchardt A, Helcl J, Srivastava A (2017) Machine translation: phrase-based, rule-based and neural approaches with linguistic evaluation. Cybern Inf Technol 17(2):28–43

    Google Scholar 

  • Martínez-Gómez P, Sanchis-Trilles G, Casacuberta F (2012) Online adaptation strategies for statistical machine translation in post-editing scenarios. Pattern Recogn 45(9):3193–3202

    Article  Google Scholar 

  • Nakov P, Guzman F, Vogel S (2012) Optimizing for sentence-level BLEU+1 yields short translations. In: Proceedings of the conference on computational linguistics (COLING), Mumbai, pp 1979–1994

  • Nepveu L, Lapalme G, Langlais P, Foster G (2004) Adaptive language and translation models for interactive machine translation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Barcelona, pp 190–197

  • Neubig G (2015) lamtram: a toolkit for language and translation modeling using neural networks. http://www.github.com/neubig/lamtram

  • Neubig G, Dyer C, Goldberg Y, Matthews A, Ammar W, Anastasopoulos A, Ballesteros M, Chiang D, Clothiaux D, Cohn T, Duh K, Faruqui M, Gan C, Garrette D, Ji Y, Kong L, Kuncoro A, Kumar G, Malaviya C, Michel P, Oda Y, Richardson M, Saphra N, Swayamdipta S, Yin P (2017) Dynet: the dynamic neural network toolkit. CoRR arxiv:1701.03980, pp 1–33

  • Nguyen K, Daumé H, Boyd-Graber J (2017) Reinforcement learning for bandit neural machine translation with simulated feedback. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), Copenhagen, pp 1464–1474

  • Ortiz-Martínez D, García-Varea I, Casacuberta F (2010) Online learning for interactive statistical machine translation. In: Proceedings of the human language technologies: the 2010 annual conference of the North American chapter of the association for computational linguistics (HLT-NAACL), Los Angeles, pp 546–554

  • Papineni K, Roukos S, Ward T, Zhu WJ (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics (ACL), Philadelphia, pp 311–318

  • Peris Á, Domingo M, Casacuberta F (2017) Interactive neural machine translation. Comput Speech Lang 45:201–220

    Article  Google Scholar 

  • Popović M (2017) Comparing language related issues for NMT and PBMT between German and English. Prague Bull Math Linguist 108(1):209–220

    Article  Google Scholar 

  • Sennrich R, Haddow B, Birch A (2016a) Edinburgh Neural Machine Translation Systems for WMT’16. In: Proceedings of the first conference on machine translation (WMT), Berlin, pp 371–376

  • Sennrich R, Haddow B, Birch A (2016b) Neural machine translation of rare words with subword units. In: Proceedings of the 54th annual meeting of the association for computational linguistics (ACL), vol 1, Long Papers, Berlin, pp 1715–1725

  • Shterionov D, Casanellas PNL, Superbo R, O’Dowd T (2017) Empirical evaluation of NMT and PBSMT quality for large-scale translation production. In: Proceedings of the annual conference of the european association for machine translation (EAMT): user studies and project/product descriptions, Prague, pp 74–79

  • Simianer P, Karimova S, Riezler S (2016) A post-editing interface for immediate adaptation in statistical machine translation. In: Proceedings of the conference on computational linguistics: system demonstrations (COLING Demos), Osaka, pp 16–20

  • Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of the conference of the association for machine translation in the Americas (AMTA), Cambridge, pp 223–231

  • Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  • Toral A, Sánchez-Cartagena VM (2017) A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. In: Proceedings of the conference of the European chapter of the association for computational linguistics (EACL), vol 1, Long Papers, Valencia, pp 1063–1073

  • Turchi M, Negri M, Farajian MA, Federico M (2017) Continuous learning from human post-edits for neural machine translation. Prague Bull Math Linguist 108(1):233–244

    Article  Google Scholar 

  • Wuebker J, Green S, DeNero J, Hasan S, Luong M (2016) Models and inference for prefix-constrained machine translation. In: Proceedings of the 54th annual meeting of the association for computational linguistics (ACL), vol 1, Long Papers, Berlin, pp 66–75

Download references

Acknowledgements

The research reported in this paper was supported in part by the German research foundation (DFG) under Grant RI-2221/4-1.

Author information

Authors and Affiliations

  1. Department of Computational Linguistics, Heidelberg University, 69120, Heidelberg, Germany

    Sariya Karimova, Patrick Simianer & Stefan Riezler

  2. Kazan Federal University, Kazan, Russia, 420008

    Sariya Karimova

Authors
  1. Sariya Karimova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick Simianer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Riezler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sariya Karimova.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karimova, S., Simianer, P. & Riezler, S. A user-study on online adaptation of neural machine translation to human post-edits. Machine Translation 32, 309–324 (2018). https://doi.org/10.1007/s10590-018-9224-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10590-018-9224-8

Keywords

Search

Navigation