Skip to main content
SpringerLink
Log in

Eye tracking as an MT evaluation technique

  • Published:
Machine Translation

Abstract

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ball Linden J, Eger N, Stevens R, Dodd J (2006) Applying the post-experience eye-tracked protocol (PEEP) method in usability testing. Interfaces 67: 15–19

    Google Scholar 

  • Caffrey C (2008) Using pupillometric, fixation-based and subjective measures to measure the processing effort experienced when viewing subtitled TV anime with pop-up gloss. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 125–144

    Google Scholar 

  • Daneman M, Carpenter P (1980) Individual differences in working memory and reading. J Verbal Learn Verbal Behav 19(4): 450–466

    Article  Google Scholar 

  • Duchowski A (2003) Eye-tracking methodology—theory and practice. Springer-Verlag, London

    MATH  Google Scholar 

  • Göpferich, S, Jakobsen, A, Mees, I (eds) (2009) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 125–144

    Google Scholar 

  • Hess E, Polt J (1964) Pupil size in relation to mental activity in simple problem solving. Science 143: 1190–1192

    Article  Google Scholar 

  • Hyönä J, Nurminen AM (2006) Do adult readers know how they read? Evidence from eye movement patterns and verbal reports. Br J Psychol 97: 31–50

    Article  Google Scholar 

  • Iqbal S, Adamzyck P, Zheng X, Bailey P (2005) Towards an index of opportunity: understanding changes in mental workload during task execution. In: Human factors in computing systems: proceedings of CHI’05. ACM Press, New York, pp 311–320

  • Jakobsen AL, Jensen K (2009) Eye movement behaviour across four different types of reading task. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 103–124

    Google Scholar 

  • Jakobsen AL (1999) Logging target text production with Translog. In: Hansen G (eds) Probing the process in translation: methods and results. Copenhagen Studies in Language 24. Samfundslitteratur, Copenhagen, pp 9–20

    Google Scholar 

  • Kaakinen JK, Hyönä J (2005) Perspective effects on expository text comprehension: evidence from think-aloud protocols, eyetracking, and recalls. Discourse Process 40: 239–257

    Article  Google Scholar 

  • Kaakinen JK, Hyönä J, Keenan J (2003) How prior knowledge, WMC, and relevance of information affect eye fixations in expository text. J Exp Psychol 29(3): 447–457

    Google Scholar 

  • Kintsch W (1998) Comprehension: a paradigm for cognition. Cambridge University Press, Cambridge, England

    Google Scholar 

  • Nakayama M, Koji T, Yasutaka S (2002) The act of task difficulty and eye-movement frequency for the oculo-motor indices. In: Proceedings of the symposium on eye tracking research and application, New Orleans, Louisiana, pp 37–42

  • O’Brien S (forthcoming) Controlled language and readability. In: Shreve G, Angelone E (eds) Translation and cognition. American Translators Association Scholarly Monograph Series. John Benjamins, Amsterdam

  • O’Brien S (2008) Processing fuzzy matches in translation memory tools—an eye-tracking analysis. In: Göpferich S, Jakobsen A, Mees I (eds) Looking at eyes—eye tracking studies of reading and translation processing. Copenhagen Studies in Language 36. Samfundslitteratur, Copenhagen, pp 79–102

    Google Scholar 

  • O’Brien S (2006) Eye-tracking and translation memory matches. Perspectives 14(3): 185–205

    Google Scholar 

  • Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124: 372–422

    Article  Google Scholar 

  • Roturier J (2006) An investigation into the impact of controlled English rules on the comprehensibility, usefulness, and acceptability of machine-translated technical documentation for French and German users. PhD Dissertation, Dublin City University

  • Roturier J (2009) Deploying novel MT technology to raise the bar for quality: a review of key advantages and challenges. MT Summit XII: proceedings of the twelfth Machine Translation Summit, August 26–30, Ottawa, Ontario, Canada

  • Schultheis H, Jameson A (2004) Assessing cognitive load in adaptive hypermedia systems: Physiological and behavioural methods. In: Neijdlw W, Bra P (eds) Adaptive hypermedia and adaptive web-based systems. Springer Verlag, Eindhoven, pp 18–24

    Google Scholar 

  • Stanford Poynter Project: http://www.poynterextra.org/et/i.htm [Last accessed: 29/04/2009]

  • Van Gog T, Kester L, Nievelstein F, Giesbers B, Paas F (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Comput Hum Behav 25: 325–331

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Next Generation Localisation, School of Applied Language and Intercultural Studies, Dublin City University, Dublin, Ireland

    Stephen Doherty & Sharon O’Brien

  2. Centre for Research and Innovation in Translation and Translation Technology, Copenhagen Business School, Frederiksberg, Denmark

    Michael Carl

Authors
  1. Stephen Doherty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sharon O’Brien
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Carl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sharon O’Brien.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Doherty, S., O’Brien, S. & Carl, M. Eye tracking as an MT evaluation technique. Machine Translation 24, 1–13 (2010). https://doi.org/10.1007/s10590-010-9070-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10590-010-9070-9

Keywords

Search

Navigation