Analysing the Impact of Interactive Machine Translation on Post-editing Effort

  • Fabio Alves
  • Arlene Koglin
  • Bartolomé Mesa-Lao
  • Mercedes García Martínez
  • Norma B. de Lima Fonseca
  • Arthur de Melo Sá
  • José Luiz Gonçalves
  • Karina Sarto Szpak
  • Kyoko Sekino
  • Marceli Aquino
Chapter
Part of the New Frontiers in Translation Studies book series (NFTS)

Abstract

The combination of temporal, technical and cognitive effort has been proposed as metrics to evaluate the feasibility of post-editing on machine-translation (MT) output (Krings, 2001). In this study, we investigate the impact of interactive machine translation on the post-editing effort required to post-edit two specialized texts under experimental conditions and correlate it with Translation Edit Rate (TER) scores. Using the CasMaCat workbench as a post-editing tool in conjunction with a Tobii T60 eye tracker, process data were collected from 16 participants with some training on postediting. They were asked to carry out post-editing tasks under two different conditions: i) traditional post-editing (MT) and ii) interactive post-editing (IMT). In the IMT condition, as the user types, the MT system suggests alternative target translations which the post-editor can interactively accept or overwrite, whereas in the traditional MT condition no aids are provided to the user while editing the raw MT output. Temporal effort is measured by the total time spent to complete the task whereas technical effort is measured by the number of keystrokes and mouse events performed by each participant. In turn, cognitive effort is measured by fixation duration and the number of eye fixations (fixation count) in each task. Results show that IMT post-editing had significantly lower fixation duration and fewer fixation counts in comparison to traditional post-editing.

Keywords

Post-editing effort Interactive post-editing Traditional post-editing TER scores CASMACAT workbench 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

The work described in this chapter was carried out within the framework of the EU project CASMACAT: Cognitive Analysis and Statistical Methods for Advanced Computer Aided Translation, funded by the European Union 7th Framework Programme Project 287576 (ICT-2011.4.2). Website: http://www.casmacat.eu. Brazilian researchers were funded by CNPq, the Brazilian Research Council (grant 307964/2011-6), and FAPEMIG, the Research Agency of the State of Minas Gerais (grant SHA/PPM-00170-14).

References

  1. Barrachina, S., Bender, O., Casacuberta, F., Civera, J., Cubel, E., Khadivi, S., Lagarda, A., Ney, H., Tomás, J., Vidal, E., & Vilar, J. M. (2009). Statistical approaches to computer-assisted translation. Computational Linguistics, 35(1), 3–28.MathSciNetCrossRefGoogle Scholar
  2. Carl, M., Dragsted, B., Elming, J., Hardt, D., & Jakobsen, A. L. (2011).The process of post-editing: A pilot study. In B. Sharp, M. Zock, M. Carl, & A. L. Jakobsen (orgs.), Proceedings of the 8th natural language processing and cognitive science workshop (Copenhagen studies in language series, Vol. 41, pp. 131–142).Google Scholar
  3. Casacuberta, F., Civera, J., Cubel, E., Lagarda, A. L., Lapalme, G., Macklovitch, E., & Vidal, E. (2009). Human interaction for high quality machine translation. Communications of the ACM, 52(10), 135–138.CrossRefGoogle Scholar
  4. Duchowski, A. (2007). Eye tracking methodology: theory and practice. Clemson: Springer.Google Scholar
  5. Federico, M., Cattelan, A., & Trombetti, M. (2012). Measuring user productivity in machine translation enhanced computer assisted translation. In Proceedings of the tenth conference of the association for machine translation in the americas (AMTA). AMTA 2012. Retrieved October 30, 2014.Google Scholar
  6. Flournoy, R., & Duran, C. (2009). Machine translation and document localization at adobe: From pilot to production. In MT Summit XII: Proceedings of the twelfth machine translation summit.Google Scholar
  7. Green, S., Heer, J., & Manning, C. D. (2013). The efficacy of human post-editing for language translation. In SIGCHI conference on human factors in computing systems (pp. 439–448). ACM.Google Scholar
  8. Isabelle, P., & Church, K. (1998). Special issue on: New tools for human translators. Machine Translation, 12(1/2).Google Scholar
  9. Jakobsen, A. L., & Jensen, K. T. H. (2008). Eye movement behaviour across four different types of reading task. Copenhagen Studies in Language, 36, 103–124.Google Scholar
  10. Kay, M., Gawron, J. M., & Norvig, P. (1994). Verbmobil: A translation system for face-to face dialog. Stanford: Center for the Study of Language and Information.Google Scholar
  11. Koehn, P. (2009). A process study of computer-aided translation. Machine Translation, 23(4), 241–263.CrossRefGoogle Scholar
  12. Krings, H. (2001). Repairing texts: Empirical investigations of machine translation port-editing processes (Trans. G. Koby, G. Shreve, K. Mischericow, & S.~Litzar). Ohio: Kent State University Press.Google Scholar
  13. Lacruz, I., Gregory, M. S., & Angelone, E. (2012). Average pause ratio as an indicator of cognitive effort in post-editing: A case study. In S. O’Brien, M. Simard, & L. Specia (Eds), Proceedings of the AMTA 2012 workshop on post-editing technology and practice (WPTP 2012). Retrieved from http://amta2012.amtaweb.org/AMTA2012Files/html/2/2_paper
  14. Langlais, P., & Lapalme, G. (2002). TransType: development-evaluation cycles to boost translator’s productivity. Machine Translation, 17(2), 77–98.CrossRefGoogle Scholar
  15. Mesa-Lao, B. (2013). Introduction to post-editing – The CasMaCat GUI. Retrieved from http://bridge.cbs.dk/projects/seecat/material/hand-out_post-editing_bmesa-lao.pdf
  16. O’Brien, S. (2004) Machine translatability and post-editing effort: How do they relate? In Translating and the computer. London: Aslib.Google Scholar
  17. O’Brien, S. (2005). Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation, 19, 37–58.Google Scholar
  18. O’Brien, S. (2007). An empirical investigation of temporal and technical post-editing effort. Translation and Interpreting Studies, 2(1), 83–136.CrossRefGoogle Scholar
  19. O’Brien, S. (2006). Pauses as indicators of cognitive effort in post-editing machine translation output. Across Language and Cultures, 7(1), 1–21.MathSciNetCrossRefGoogle Scholar
  20. Plitt, M., & Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a typical localisation context. In The Prague bulletin of mathematical linguistics no. 93 (pp. 7–16). ISBN 978-80-904175-4-0. doi: 10.2478/v10108-010-0010-x.
  21. Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52, 591–611.Google Scholar
  22. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of AMTA-2006 (pp. 223–231).Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Fabio Alves
    • 1
  • Arlene Koglin
    • 1
  • Bartolomé Mesa-Lao
    • 2
  • Mercedes García Martínez
    • 3
  • Norma B. de Lima Fonseca
    • 1
  • Arthur de Melo Sá
    • 1
  • José Luiz Gonçalves
    • 4
  • Karina Sarto Szpak
    • 1
  • Kyoko Sekino
    • 1
  • Marceli Aquino
    • 1
  1. 1.Laboratory for Experimentation in Translation (LETRA)Federal University of Minas Gerais (UFMG)Belo HorizonteBrazil
  2. 2.Center for Research and Innovation in Translation and Translation Technology, Department of International Business CommunicationCopenhagen Business SchoolFrederiksbergDenmark
  3. 3.Computer LaboratoryUniversity of MaineLe MansFrance
  4. 4.Laboratory for Experimentation in Translation (LETRA)Universidade Federal de Ouro Preto (UFOP)Belo HorizonteBrazil

Personalised recommendations