Machine Translation

, Volume 28, Issue 3–4, pp 165–186 | Cite as

Correlations between productivity and quality when post-editing in a professional context

  • Ana Guerberof ArenasEmail author


This article presents results on the correlation between machine-translated and fuzzy matches segments in terms of productivity and final quality in the context of a localization project. In order to explore these two aspects, we set up an experiment with a group of twenty four professional translators using an online post-editing tool and a customized Moses machine translation engine with a BLEU score of 0.60. The translators were asked to translate from English to Spanish, working on no-match, machine-translated and translation memory segments from the 85–94  % value, using a post-editing tool, without actually knowing if the segment came from machine translation or from translation memory. The texts were corrected by three professional reviewers to assess the final quality of the assignment. The findings suggest that translators have higher productivity and quality when using machine-translated output than when translating without it, and that this productivity and quality is not significantly different from the values obtained when processing fuzzy matches from translation memories in the range 85–94 %.


Post-editing Productivity Quality Machine translation Translation memory 



The research project this article stems from was funded by the European Association of Machine Translation (EAMT) and the Centre for Global Intelligent Content (CNGL). I would also like to thank Crosslang, MicroStrategy and Logoscript (now Pactera) for their key contribution to the project. I’m especially grateful to Dr. Anthony Pym and Dr. Sharon O’Brien for their continuous support on the completion of the doctoral thesis that this project is part of.


  1. Asia Online (2012) JABA-Translations machine translation workshop demonstrates MT equivalent to 85% fuzzy match productivity. Language Studio Newsletter. Accessed Jan 2014
  2. Autodesk (2011) Translation and post-editing productivity. Accessed Jan 2014
  3. Bier K, Herranz M (2011) MT experience at Sybase. Localization World Conference, Barcelona. Accessed Jan 2014
  4. Carl M, Dragsted B, Elming J, Hardt D, Jakobsen A (2011) The process of post-editing: a pilot study. In: Proceedings of the 8\({{\rm th}}\) International NLPSC Workshop. Copenhagen Studies in Language 41, Frederiksberg, pp 131–142 Accessed Jan 2014
  5. De Almeida G, O’Brien S (2010) Analysing post-editing performance: correlations with years of translation experience. In: Proceedings of the 14\({{\rm th}}\) EAMT Conference, St. Raphael. Accessed Jan 2014
  6. De Palma D, Sargent B, Bassetti T, Beninatto R (2008) The price of translation: a comprehensive analysis of pricing for globalization service buyers. Common Sense Advisory.
  7. De Sutter N, Depraetere I (2012) Post-edited translation quality, edit distance and fluency scores: report on a case study. In: Journée d’études-Traduction et qualité: Méthodologies en matière d’assurance qualité, Universtité Lille 3, sciences humaines et socials, Lille. Accessed Jan 2014
  8. Dove C, Way A, Johnson D (2011) Quality above price and speed. TAUS Conference, Santa Clara. Accessed Jan 2014
  9. Fiederer R, O’Brien S (2009) Quality and machine translation: a realistic objective? The Journal of Specialised Translation (11). Accessed Jan 2014
  10. Flournoy R, Duran C (2009) Machine translation and document localization at Adobe: from pilot to production. In: Proceedings of the 12\({{\rm th}}\) MT Summit, Ottawa. Accessed Jan 2014
  11. García I (2010) Is machine translation ready yet? Target, vol 22(1). Benjamins, Amsterdam and PhiladelphiaGoogle Scholar
  12. García I (2011) Translating by post-editing: Is it the way forward? Machine Translation, vol 25(3). Springer, Netherlands, pp 217–237Google Scholar
  13. Guerberof A (2008) Productivity and quality in machine translation and translation memory outputs. Dissertation, Universitat Rovira i Virgili. Accessed Jan 2014
  14. Guerberof A (2012) Productivity and quality in the post-editing of outputs from translation memories and machine translation. PhD Thesis, Universitat Rovira i Virgili. Accessed Jan 2014
  15. Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R, Dyer CJ, Bojar O, Constantin A, Herbst E (2007) Moses: open source toolkit for statistical machine translation. Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions. ACL, Stroudsburg, pp 177–180CrossRefGoogle Scholar
  16. O’Brien S (2002) Teaching post-editing: a proposal for course content. In: Proceedings for the 6\(^{{\rm th}}\) EAMT Conference, Manchester, pp 99–106. Accessed Jan 2014
  17. O’Brien S (2006a) Methodologies for measuring correlations between post-editing effort and machine translatability. Machine translation 19(1):37–58, Springer, NetherlandsGoogle Scholar
  18. O’Brien S (2006b) Eye tracking and translation memory matches. Perspect Stud Translatol 14(3):185–205Google Scholar
  19. O’Brien S (2011) Towards predicting post-editing productivity. Machine translation, vol 25(3). Springer, Netherlands, pp 197–215Google Scholar
  20. Offersgaard L, Povlsen C, Almsten L, Maegaard B (2008) Domain specific MT use. In: Proceedings of 12\({{\rm th}}\) EAMT Conference, Hamburg, pp 150–159. Accessed Jan 2014
  21. Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: A method for automatic evaluation of machine translation. In: Proceedings of the 40\({{\rm th}}\) Annual Meeting of the ACL, Stroudsburg, pp 311–318. Accessed Jan 2014
  22. Plitt M, Masselot F (2010) A productivity test of statistical machine translation post-editing in a typical localisation context. The Prague Bulletin of Mathematical Linguistics, Prague, pp 7–16. Accessed Jan 2014
  23. Roukos S, Rojas F, Ming Xu J, Pont Nesta S, Martínez Corriá A, Chapman H, Vohra S (2011) The value of post-editing: IBM case study. Localization World Conference, Barcelona. Accessed Ja 2014
  24. Tatsumi M (2010) Post-editing machine translated text in a commercial setting: observation and statistical analysis. PhD Thesis, Dublin City University. Accessed Mar 2012

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  1. 1.BarcelonaSpain

Personalised recommendations