Advertisement

Machine Translation

, Volume 24, Issue 1, pp 1–13 | Cite as

Eye tracking as an MT evaluation technique

  • Stephen Doherty
  • Sharon O’Brien
  • Michael Carl
Article

Abstract

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.

Keywords

MT evaluation Eye tracking Gaze time Fixation count Fixation duration Pupil dilation HTER 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ball Linden J, Eger N, Stevens R, Dodd J (2006) Applying the post-experience eye-tracked protocol (PEEP) method in usability testing. Interfaces 67: 15–19Google Scholar
  2. Caffrey C (2008) Using pupillometric, fixation-based and subjective measures to measure the processing effort experienced when viewing subtitled TV anime with pop-up gloss. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 125–144Google Scholar
  3. Daneman M, Carpenter P (1980) Individual differences in working memory and reading. J Verbal Learn Verbal Behav 19(4): 450–466CrossRefGoogle Scholar
  4. Duchowski A (2003) Eye-tracking methodology—theory and practice. Springer-Verlag, LondonzbMATHGoogle Scholar
  5. Göpferich, S, Jakobsen, A, Mees, I (eds) (2009) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 125–144Google Scholar
  6. Hess E, Polt J (1964) Pupil size in relation to mental activity in simple problem solving. Science 143: 1190–1192CrossRefGoogle Scholar
  7. Hyönä J, Nurminen AM (2006) Do adult readers know how they read? Evidence from eye movement patterns and verbal reports. Br J Psychol 97: 31–50CrossRefGoogle Scholar
  8. Iqbal S, Adamzyck P, Zheng X, Bailey P (2005) Towards an index of opportunity: understanding changes in mental workload during task execution. In: Human factors in computing systems: proceedings of CHI’05. ACM Press, New York, pp 311–320Google Scholar
  9. Jakobsen AL, Jensen K (2009) Eye movement behaviour across four different types of reading task. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 103–124Google Scholar
  10. Jakobsen AL (1999) Logging target text production with Translog. In: Hansen G (eds) Probing the process in translation: methods and results. Copenhagen Studies in Language 24. Samfundslitteratur, Copenhagen, pp 9–20Google Scholar
  11. Kaakinen JK, Hyönä J (2005) Perspective effects on expository text comprehension: evidence from think-aloud protocols, eyetracking, and recalls. Discourse Process 40: 239–257CrossRefGoogle Scholar
  12. Kaakinen JK, Hyönä J, Keenan J (2003) How prior knowledge, WMC, and relevance of information affect eye fixations in expository text. J Exp Psychol 29(3): 447–457Google Scholar
  13. Kintsch W (1998) Comprehension: a paradigm for cognition. Cambridge University Press, Cambridge, EnglandGoogle Scholar
  14. Nakayama M, Koji T, Yasutaka S (2002) The act of task difficulty and eye-movement frequency for the oculo-motor indices. In: Proceedings of the symposium on eye tracking research and application, New Orleans, Louisiana, pp 37–42Google Scholar
  15. O’Brien S (forthcoming) Controlled language and readability. In: Shreve G, Angelone E (eds) Translation and cognition. American Translators Association Scholarly Monograph Series. John Benjamins, AmsterdamGoogle Scholar
  16. O’Brien S (2008) Processing fuzzy matches in translation memory tools—an eye-tracking analysis. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 79–102Google Scholar
  17. O’Brien S (2006) Eye-tracking and translation memory matches. Perspectives 14(3): 185–205Google Scholar
  18. Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124: 372–422CrossRefGoogle Scholar
  19. Roturier J (2006) An investigation into the impact of controlled English rules on the comprehensibility, usefulness, and acceptability of machine-translated technical documentation for French and German users. PhD Dissertation, Dublin City UniversityGoogle Scholar
  20. Roturier J (2009) Deploying novel MT technology to raise the bar for quality: a review of key advantages and challenges. MT Summit XII: proceedings of the twelfth Machine Translation Summit, August 26–30, Ottawa, Ontario, CanadaGoogle Scholar
  21. Schultheis H, Jameson A (2004) Assessing cognitive load in adaptive hypermedia systems: Physiological and behavioural methods. In: Neijdlw W, Bra P (eds) Adaptive hypermedia and adaptive web-based systems. Springer Verlag, Eindhoven, pp 18–24Google Scholar
  22. Stanford Poynter Project: http://www.poynterextra.org/et/i.htm [Last accessed: 29/04/2009]
  23. Van Gog T, Kester L, Nievelstein F, Giesbers B, Paas F (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Comput Hum Behav 25: 325–331CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Stephen Doherty
    • 1
  • Sharon O’Brien
    • 1
  • Michael Carl
    • 2
  1. 1.Centre for Next Generation Localisation, School of Applied Language and Intercultural StudiesDublin City UniversityDublinIreland
  2. 2.Centre for Research and Innovation in Translation and Translation TechnologyCopenhagen Business SchoolFrederiksbergDenmark

Personalised recommendations