Advertisement

Machine Translation

, Volume 29, Issue 3–4, pp 267–284 | Cite as

Correlations of perceived post-editing effort with measurements of actual effort

  • Joss MoorkensEmail author
  • Sharon O’Brien
  • Igor A. L. da Silva
  • Norma B. de Lima Fonseca
  • Fabio Alves
Article

Abstract

Human rating of predicted post-editing effort is a common activity and has been used to train confidence estimation models. However, the correlation between human ratings and actual post-editing effort is under-measured. Moreover, the impact of presenting effort indicators in a post-editing user interface on actual post-editing effort has hardly been researched. In this study, ratings of perceived post-editing effort are tested for correlations with actual temporal, technical and cognitive post-editing effort. In addition, the impact on post-editing effort of the presentation of post-editing effort indicators in the user interface is also tested. The language pair involved in this study is English-Brazilian Portuguese. Our findings, based on a small sample, suggest that there is little agreement between raters for predicted post-editing effort and that the correlations between actual post-editing effort and predicted effort are only moderate, and thus an inefficient basis for MT confidence estimation. Moreover, the presentation of post-editing effort indicators in the user interface appears not to impact on actual post-editing effort.

Keywords

Post-editing Post-editing effort Eye-tracking Confidence estimation Confidence indicators Machine translation user evaluation 

Notes

Acknowledgments

This research is supported by Science Foundation Ireland (Grant 12/CE/I2267) as part of the CNGL (www.cngl.ie) at Dublin City University, Research Brazil Ireland, and by the FALCON Project (falcon-project.eu), funded by the European Commission through the Seventh Framework Programme (FP7) Grant Agreement No. 610879. The authors would like to place on record their thanks to Research Brazil Ireland, and to staff and students at the Faculdade de Letras at UFMG.

References

  1. Alves F, Koglin A, Mesa-Lao B, García MM, Fonseca NBL, Sá AM, Gonçalves JL, Szpak KS, Sekino K, Aquino M (2015) Analysing the impact of interactive machine translation on post-editing effort. In: Carl M, Bangalore S, Schaeffer M (eds) New directions in empirical translation process research. Springer, New York, pp 77–95Google Scholar
  2. Biçici E, Way A (2014) Referential translation machines for predicting translation quality. WMT 2014: ACL 2014 Ninth Workshop On Statistical Machine Translation, BaltimoreGoogle Scholar
  3. Blain F, Senellart J, Schwenk H, Plitt M, Roturier J (2011) Qualitative analysis of post-editing for high quality machine translation. In: Machine Translation Summit XIII, Asia-Pacific Association for Machine Translation (AAMT), XiamenGoogle Scholar
  4. Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, Sanchis A, Ueffing N (2004) Confidence estimation for machine translation. Proceedings of the 20th international conference on computational linguistics, 23–27 Aug 2004, Geneva, pp 315–321Google Scholar
  5. Carl M, Dragsted B, Elming J, Hardt D, Jakobsen AL (2011) The process of post-editing: a pilot study. In: Proceedings of the 8th international natural language processing and cognitive science workshop, Frederiksberg, pp 131–142Google Scholar
  6. Cooper A (2004) The inmates are running the asylum: why hi-tech products drive us crazy and how to restore the sanity. Sams Publishing, IndianapolisGoogle Scholar
  7. Da Silva IAL, Schmaltz M, Alves F, Pagano A, Wong D, Chao L, Leal ALV, Quaresma P, Garcia C (2015) Translating and post-editing in the Chinese-Portuguese Language pair: insights from an exploratory study of key logging and eye tracking. Translation Spaces 4(1):145–169Google Scholar
  8. De Almeida G, O’Brien S (2010) Analysing post-editing performance: Correlations with years of translation experience. In: Proceedings of the 14th annual conference of the European Association for Machine Translation, St. RaphaëlGoogle Scholar
  9. DePalma DA, Hegde V, Pielmeier H, Stewart RG (2013) The language services market. Common Sense Advisory, BostonGoogle Scholar
  10. Doherty S (2012) Investigating the effects of controlled language on the reading and comprehension of machine translated texts: a mixed-methods approach. Dissertation, Dublin City UniversityGoogle Scholar
  11. Gaspari F, Toral A, Kumar NS, Groves D, Way A (2014) Perception vs. reality: measuring machine translation post-editing productivity. In: O’Brien S, Simard M, Specia L (eds) Proceedings of the 11th conference of the Association for Machine Translation in the Americas: workshop on post-editing technology and practice (WPTP3), Vancouver, pp 60–72Google Scholar
  12. Guerberof A (2009) Productivity and quality in the post-editing of outputs from translation memories and machine translation. Localisation Focus 7(1):11–21Google Scholar
  13. Hokamp C, Liu C (2015) HandyCAT. In: Durgar El-Kahlout İ, Özkan M, Sánchez-Martínez F, Ramírez-Sánchez G, Hollowood F, Way A (eds) Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, p 216Google Scholar
  14. Koponen M (2012) Comparing human perceptions of post-editing effort with post-editing operations. In: Proceedings of the 7th workshop on statistical machine translation, Montreal, pp 181–190Google Scholar
  15. Krings HP (2001) Repairing texts. Kent State University Press, KentGoogle Scholar
  16. Lacruz I, Shreve GM (2014) Pauses and cognitive effort in post-editing. In: O’Brien S, Balling LW, Carl M, Simard M, Specia L (eds) Post-editing of machine translation: processes and applications. Cambridge Scholars Publishing, Newcastle-Upon-Tyne, pp 246–272Google Scholar
  17. Läubli S, Fishel M, Massey G, Ehrensberger-Dow M, Volk M (2013) Assessing post-editing efficiency in a realistic translation environment. In: Simard M, Specia L (eds) O’Brien S. Proceedings of MT Summit XIV workshop on post-editing technology and practice, Nice, pp 83–91Google Scholar
  18. Moorkens J, O’Brien S (2013) User attitudes to the post-editing interface. In: Simard M, Specia L (eds) O’Brien S. Proceedings of MT Summit XIV workshop on post-editing technology and practice, Nice, pp 19–25Google Scholar
  19. Moorkens J, O’Brien S (2015) Post-editing evaluations: Trade-offs between novice and professional participants. In: Durgar El-Kahlout İ, Özkan M, Sánchez-Martínez F, Ramírez-Sánchez G, Hollowood F, Way A (eds) Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, pp 75–81Google Scholar
  20. O’Brien S (2005) Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation 19(1):37–58CrossRefGoogle Scholar
  21. O’Brien S (2010) Introduction to Post-Editing: Who, What, How and Where to Next? In: Proceedings of AMTA 2010, the 9th conference of the Association for Machine Translation in the Americas (online), DenverGoogle Scholar
  22. O’Brien S (2011) Towards predicting post-editing productivity. Machine Translation 25:197–215CrossRefGoogle Scholar
  23. Plitt M, Masselot F (2010) A productivity test of statistical machine translation post-editing in a typical localization context. Prague Bull Math Linguist 93:7–16CrossRefGoogle Scholar
  24. Quirk C (2004) Training a sentence-level machine translation confidence measure. In: Proceedings of the 4th conference on language resources and evaluation, Lisbon, pp 825–828Google Scholar
  25. Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124(3):372–422CrossRefGoogle Scholar
  26. Shah K, Specia L (2014) Quality estimation for translation selection. In: Tadic M, Koehn P, Roturier J, Way A (eds) Proceedings of the 17th annual conference of the European Association for Machine Translation (EAMT 2014), Dubrovnik, pp 109–116Google Scholar
  27. Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th conference of the Association for Machine Translation in the Americas (AMTA2006), “Visions for the Future of Machine Translation’, Cambridge, pp 223–231Google Scholar
  28. Soricut R, Narsale S (2012) Combining quality prediction and system selection for improved automatic translation output. In: Proceedings of the ACL 7th workshop on statistical machine translation (WMT-2012), Montreal, pp 163–170Google Scholar
  29. Specia L (2011) Exploiting objective annotations for measuring translation post-editing effort. In: Proceedings of the 15th conference of EAMT, Leuven, pp 73–80Google Scholar
  30. Specia L, Cancedda N, Dymetman M, Turchi M, Cristianini N (2009) Estimating the sentence-level quality of machine translation systems. In: Proceedings of the 13th annual conference of the EAMT, Barcelona, pp 28–35Google Scholar
  31. Sperber D, Wilson D (1986) Relevance: communication and cognition. Blackwell, OxfordGoogle Scholar
  32. Tatsumi M (2009) Correlation between automatic evaluation scores, post-editing speed and some other factors. In: Proceedings of MT Summit XII, Ottawa, pp 332–339Google Scholar
  33. Tatsumi M, Roturier J (2010) Source text characteristics and technical and temporal post-editing effort: what is their relationship? In: Zhechev V (ed) Proceedings of the 2nd joint EM+/CNGL workshop “Bringing MT to the User: Research on Integrating MT in the Translation Industry” (JEC ’10). Denver, pp 43–51Google Scholar
  34. Teixeira CSC (2014) Perceived vs. measured performance in the post-editing of suggestions from machine translation and translation memories. In: O’Brien S, Simard M, Specia L (eds) Proceedings of the 11th conference of the association for machine translation in the Americas: workshop on post-editing technology and practice (WPTP3), Vancouver, pp 45–59Google Scholar
  35. Temnikova I (2010) Cognitive evaluation approach for a controlled language post-editing experiment. In: Proceedings of the 7th international conference on language resources and evaluation, LREC 2010, Valletta, pp 3485–3490Google Scholar
  36. Turian J, Shen L, Melamed D (2003) Evaluation of machine translation and its evaluation. In: Proceedings of the MT Summit IX, New Orleans, pp 386–393Google Scholar
  37. Vieira L, Specia L (2011) A review of translation tools from a post-editing perspective. In: Proceedings of the 3rd joint EM+/CNGL workshop bringing MT to the users: research meets translators” (JEC ’11), Luxembourg, pp 33–42Google Scholar
  38. Vieira LN (2014) Indices of cognitive effort in machine translation post-editing. Machine Translation 28(3):187–216CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.ADAPT Centre/School of ComputingDublin City UniversityDublinIreland
  2. 2.ADAPT Centre/SALIS/CTTSDublin City UniversityDublinIreland
  3. 3.Instituto de Letras e LinguísticaFederal University of UberlândiaUberlândiaBrazil
  4. 4.Laboratory for Experimentation in Translation (LETRA)Federal University of Minas GeraisBelo HorizonteBrazil

Personalised recommendations