Measuring Difficulty in Translation and Post-editing: A Review

  • Sanjun Sun
Part of the New Frontiers in Translation Studies book series (NFTS)


Difficulty (or called mental load, cognitive effort) has been an importance topic in translation and interpreting process research. This article first clarifies conceptual issues and reviews difficulty, mental workload, cognitive load and other related terms, their histories and theories. Under the umbrella of cognitive science, it then reviews two lines of research, i.e., difficulty in human translation and in postediting of machine translation. Studies concerning methods for measuring difficulty in human translation and post-editing are presented and critically examined. Two assumptions in translation difficulty research are described towards the end of this article.


Mental Workload Human Translation Machine Translation (MT) Mental Load Translational Research Process (TPR) 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported by the Young Faculty Research Fund of Beijing Foreign Studies University (Grant No. 2016JT004) and by the Fundamental Research Funds for the Central Universities (Grant No. 2015JJ003).


  1. Alamargot, D., Chesnet, D., Dansac, C., & Ros, C. (2006). Eye and pen: A new device for studying reading during writing. Behavior Research Methods, 38(2), 287–299.CrossRefGoogle Scholar
  2. Alamargot, D., Dansac, C., Chesnet, D., & Fayol, M. (2007). Parallel processing before and after pauses: A combined analysis of graphomotor and eye movements during procedural text production. In M. Torrance, L. Van Waes, & D. Galbraith (Eds.), Writing and cognition: Research and applications (pp. 13–29). Amsterdam: Elsevier.Google Scholar
  3. Allen, J. (2003). Post-editing. In H. Somers (Ed.), Computers and translation: A translator’s guide (pp. 297–318). Amsterdam: John Benjamins.CrossRefGoogle Scholar
  4. Alves, F. (2015). Translation process research at the interface. In A. Ferreira & J. W. Schwieter (Eds.), Psycholinguistic and cognitive inquiries into translation and interpreting (pp. 17–39). Amsterdam: John Benjamins.Google Scholar
  5. Alves, F., Pagano, A., & da Silva, I. (2014). Effortful text production in translation: A study of grammatical (de)metaphorization drawing on product and process data. Translation and Interpreting Studies, 9(1), 25–51.CrossRefGoogle Scholar
  6. Anagnostou, N. K., & Weir, G. R. S. (2007). From corpus-based collocation frequencies to readability measure. In G. R. S. Weir & T. Ozasa (Eds.), Texts, textbooks and readability (pp. 34–48). Glasgow: University of Stratchclyde Publishing.Google Scholar
  7. Arenas, A. G. (2014). The role of professional experience in post-editing from a quality and productivity perspective. In S. O’Brien, L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation: Processes and applications (pp. 51–76). Newcastle: Cambridge Scholars Publishing.Google Scholar
  8. Aziz, W., & Specia, L. (2012). PET: A standalone tool for assessing machine translation through post-editing. Paper presented at the Translating and The Computer 34, London.Google Scholar
  9. Aziz, W., Mitkov, R., & Specia, L. (2013). Ranking machine translation systems via post-editing. In I. Habernal & V. Matoušek (Eds.), Text, speech, and dialogue (pp. 410–418). London: Springer.CrossRefGoogle Scholar
  10. Aziz, W., Koponen, M., & Specia, L. (2014). Sub-sentence level analysis of machine translation post-editing effort. In S. O’Brien, L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation: Processes and applications (pp. 170–199). Newcastle: Cambridge Scholars Publishing.Google Scholar
  11. Baddeley, A. D., Eysenck, M. W., & Anderson, M. C. (2015). Memory (2nd ed.). London: Psychology Press.CrossRefGoogle Scholar
  12. Baker, M. (2011). In other words: A coursebook on translation (2nd ed.). New York: Routledge.CrossRefGoogle Scholar
  13. Balling, L. W., Hvelplund, K. T., & Sjørup, A. C. (2014). Evidence of parallel processing during translation. Meta, 59(2), 234–259.CrossRefGoogle Scholar
  14. Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Paper presented at the Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43rd Annual Meeting of the Association of Computational Linguistics (ACL-2005), Ann Arbor, Michigan.Google Scholar
  15. Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276–292.CrossRefGoogle Scholar
  16. Bermúdez, J. L. (2014). Cognitive science: An introduction to the science of the mind (2nd ed.). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  17. Block, R. A., Hancock, P. A., & Zakay, D. (2010). How cognitive load affects duration judgments: A meta-analytic review. Acta Psychologica, 134(3), 330–343.CrossRefGoogle Scholar
  18. Bradshaw, J. L. (1968). Load and pupillary changes in continuous processing tasks. British Journal of Psychology, 59(3), 265–271.CrossRefGoogle Scholar
  19. Broadbent, D. E. (1958). Perception and communication. London: Pergamon Press.CrossRefGoogle Scholar
  20. Brünken, R. E., Plass, J. L., & Moreno, R. E. (2010). Current issues and open questions in cognitive load research. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  21. Busse, L. M., & Buhmann, J. M. (2011). Model-based clustering of inhomogeneous paired comparison data. In M. Pelillo & E. R. Hancock (Eds.), Similarity-based pattern recognition (pp. 207–221). Berlin: Springer.CrossRefGoogle Scholar
  22. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., & Schroeder, J. (2008). Further meta-evaluation of machine translation. In Proceedings of the third workshop on statistical machine translation (pp. 70–106). Columbus: Association for Computational Linguistics.CrossRefGoogle Scholar
  23. Campbell, S. (1999). A cognitive approach to source text difficulty in translation. Target, 11(1), 33–63.CrossRefGoogle Scholar
  24. Campbell, S. (2000). Choice network analysis in translation research. In M. Olohan (Ed.), Intercultural faultlines: Research models in translation studies: Textual and cognitive aspects (pp. 29–42). Manchester: St. Jerome.Google Scholar
  25. Campbell, S., & Hale, S. (1999). What makes a text difficult to translate? Refereed Proceedings of the 23rd Annual ALAA Congress. Retrieved March 1, 2015, from
  26. Cara, F. (1999). Cognitive ergonomics. In R. A. Wilson & F. C. Keil (Eds.), The MIT encyclopedia of the cognitive sciences (pp. 130–132). Cambridge: MIT Press.Google Scholar
  27. Carl, M., Bangalore, S., & Schaeffer, M. (2015). New directions in empirical translation process research: Exploring the CRITT TPR-DB. New York: Springer.Google Scholar
  28. Cassenti, D. N., & Kelley, T. D. (2006). Towards the shape of mental workload. Paper presented at the Human Factors and Ergonomics Society Annual Meeting, Boston, MA.CrossRefGoogle Scholar
  29. Cassenti, D. N., Kelley, T. D., & Carlson, R. A. (2013). Differences in performance with changing mental workload as the basis for an IMPRINT plug-in proposal. Paper presented at the 22nd Annual Conference on Behavior Representation in Modeling and Simulation, Ottawa, Canada.Google Scholar
  30. Cassin, B. (2014). Dictionary of untranslatables: A philosophical lexicon. Princeton: Princeton University Press.CrossRefGoogle Scholar
  31. Chall, J. S., & Dale, E. (1995). Readability revisited: The new Dale-Chall readability formula. Cambridge: Brookline Books.Google Scholar
  32. Charlton, S. G. (2002). Measurement of cognitive states in test and evaluation. In S. G. Charlton & T. G. O’Brien (Eds.), Handbook of human factors testing and evaluation (2nd ed., pp. 97–126). Mahwah: Lawrence Erlbaum.Google Scholar
  33. Common Sense Advisory. (2014). Ten concepts and data points to remember in 2014. MultiLingual, 1, 37-38.Google Scholar
  34. Conway, A. R. A., Cowan, N., Bunting, M. F., Therriault, D. J., & Minkoff, S. R. B. (2002). A latent variable analysis of working memory capacity, short-term memory capacity, processing speed, and general fluid intelligence. Intelligence, 30(2), 163–183.CrossRefGoogle Scholar
  35. Conway, A. R. A., Kane, M. J., Bunting, M. F., Hambrick, D. Z., Wilhelm, O., & Engle, R. W. (2005). Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review, 12(5), 769–786.CrossRefGoogle Scholar
  36. Daneman, M., & Carpenter, P. A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19(4), 450–466.CrossRefGoogle Scholar
  37. De Almeida, G. (2013). Translating the post-editor: An investigation of post-editing changes and correlations with professional experience across two Romance languages. PhD thesis. Dublin City University, Dublin.Google Scholar
  38. De Waard, D., & Lewis-Evans, B. (2014). Self-report scales alone cannot capture mental workload. Cognition, Technology & Work, 16(3), 303–305.CrossRefGoogle Scholar
  39. DeKeyser, R. (2003). Implicit and explicit learning. In C. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp. 313–348). Oxford: Blackwell.Google Scholar
  40. Denkowski, M., & Lavie, A. (2011). Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the 6th workshop on statistical machine translation (pp. 85–91). Edinburgh: Association for Computational Linguistics.Google Scholar
  41. Denkowski, M., & Lavie, A. (2012). TransCenter: Web-based translation research suite. Retrieved April 1, 2015, from
  42. DePalma, D. A., & Hegde, V. (2010). The market for MT post-editing. Lowell: Common Sense Advisory.Google Scholar
  43. DePalma, D. A., & Kelly, N. (2009). The business case for machine translation. Lowell: Common Sense Advisory.Google Scholar
  44. Doherty, S., O’Brien, S., & Carl, M. (2010). Eye tracking as an MT evaluation technique. Machine Translation, 24(1), 1–13.CrossRefGoogle Scholar
  45. Dorr, B., Olive, J., McCary, J., & Christianson, C. (2011). Machine translation evaluation and optimization. In J. Olive, C. Christianson, & J. McCary (Eds.), Handbook of natural language processing and machine translation (pp. 745–843). New York: Springer.CrossRefGoogle Scholar
  46. Dragsted, B. (2004). Segmentation in translation and translation memory systems: An empirical investigation of cognitive segmentation and effects of integrating a TM system into the translation process. PhD thesis. Copenhagen Business School, Frederiksberg, Denmark.Google Scholar
  47. Dragsted, B. (2012). Indicators of difficulty in translation: Correlating product and process data. Across Languages and Cultures, 13(1), 81–98.CrossRefGoogle Scholar
  48. Embrey, D., Blackett, C., Marsden, P., & Peachey, J. (2006). Development of a human cognitive workload assessment tool: MCA final report. Dalton: Human Reliability Associates.Google Scholar
  49. Engle, R. W., & Kane, M. J. (2004). Executive attention, working memory capacity, and a two-factor theory of cognitive control. In B. Ross (Ed.), The psychology of learning and motivation (pp. 145–199). New York: Elsevier.Google Scholar
  50. Englund Dimitrova, B. (2005). Expertise and explicitation in the translation process. Amsterdam: John Benjamins.CrossRefGoogle Scholar
  51. Felice, M., & Specia, L. (2012). Linguistic features for quality estimation. In Proceedings of the 7th workshop on statistical machine translation (pp. 96–103). Montréal: Association for Computational Linguistics.Google Scholar
  52. Fiedler, S., Glöckner, A., & Nicklisch, A. (2012). The influence of social value orientation on information processing in repeated voluntary contribution mechanism games: An eye-tracking analysis. In A. Innocenti & A. Sirigu (Eds.), Neuroscience and the Economics of Decision Making (pp. 21–53). London: Routledge.Google Scholar
  53. Frankish, K., & Ramsey, W. (Eds.). (2012). The Cambridge handbook of cognitive science. Cambridge: Cambridge University Press.Google Scholar
  54. Freeman, G. L., & Giese, W. J. (1940). The relationship between task difficulty and palmar skin resistance. The Journal of General Psychology, 23(1), 217–220.CrossRefGoogle Scholar
  55. Freixa, J. (2006). Causes of denominative variation in terminology: A typology proposal. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication, 12(1), 51–77.Google Scholar
  56. Fry, E. B. (1988). Writeability: The principles of writing for increased comprehension. In B. L. Zakaluk & S. J. Samuels (Eds.), Readability: Its past, present, and future (pp. 77–95). Newark: International Reading Association.Google Scholar
  57. Gallupe, R. B., DeSanctis, G., & Dickson, G. W. (1988). Computer-based support for group problem-finding: An experimental investigation. MIS Quarterly, 12(2), 277–296.CrossRefGoogle Scholar
  58. Gile, D. (1999). Testing the Effort Models’ tightrope hypothesis in simultaneous interpreting – A contribution. Hermes, 23, 153–172.Google Scholar
  59. Gile, D. (2009). Basic concepts and models for interpreter and translator training (Rev. Ed.). Amsterdam: John Benjamin.Google Scholar
  60. Goldberg, J. H., & Wichansky, A. M. (2003). Eye tracking in usability evaluation: A practitioner’s guide. In R. Radach, J. Hyona, & H. Deubel (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 493–516). Amsterdam: Elsevier.CrossRefGoogle Scholar
  61. Göpferich, S., Jakobsen, A. L., & Mees, I. M. (Eds.). (2008). Looking at eyes: Eye-tracking studies of reading and translation processing. Copenhagen: Sammfundslitteratur.Google Scholar
  62. Gopher, D. (1994). Analysis and measurement of mental load. In G. d’Ydewalle, P. Eelen, & P. Bertelson (Eds.), International perspectives on psychological science, Vol. II: The state of the art (pp. 265–292). East Sussex: Lawrence Erlbaum.Google Scholar
  63. Gray, W. S., & Leary, B. E. (1935). What makes a book readable. Chicago: The University of Chicago Press.Google Scholar
  64. Hale, S., & Campbell, S. (2002). The interaction between text difficulty and translation accuracy. Babel, 48(1), 14–33.CrossRefGoogle Scholar
  65. Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.CrossRefGoogle Scholar
  66. Hummel, K. M. (2002). Second language acquisition and working memory. In F. Fabbro (Ed.), Advances in the neurolinguistics of bilingualism (pp. 95–117). Udine: Forum.Google Scholar
  67. Hvelplund, K. T. (2011). Allocation of cognitive resources in translation: An eye-tracking and key-logging study. PhD thesis. Copenhagen Business School, Frederiksberg, Denmark.Google Scholar
  68. Ilkowska, M., & Engle, R. W. (2010). Trait and state differences in working memory capacity. In A. Gruszka, G. Matthews, & B. Szymura (Eds.), Handbook of individual differences in cognition (pp. 295–320). New York: Springer.CrossRefGoogle Scholar
  69. International Ergonomics Association. (2015). Definition and domains of ergonomics. Retrieved March 1, 2015, from
  70. Ivir, V. (1981). Formal correspondence vs. translation equivalence revisited. Poetics Today, 2(4), 51–59.CrossRefGoogle Scholar
  71. Jakobsen, A. L. (2011). Tracking translators’ keystrokes and eye movements with Translog. In C. Alvstad, A. Hild, & E. Tiselius (Eds.), Methods and strategies of process research (pp. 37–55). Amsterdam: John Benjamins.CrossRefGoogle Scholar
  72. Jensen, K. T. (2009). Indicators of text complexity. In S. Göpferich, A. L. Jakobsen, & I. M. Mees (Eds.), Behind the mind: Methods, models and results in translation process research (pp. 61–80). Amsterdam: John Benjamins.Google Scholar
  73. Jex, H. R. (1988). Measuring mental workload: Problems, progress, and promises. In P. A. Hancock & N. Meshkati (Eds.), Hman mental workload (pp. 5–38). Amsterdam: North-Holland.CrossRefGoogle Scholar
  74. Just, M. A., & Carpenter, P. A. (1976). Eye fixations and cognitive processes. Cognitive Psychology, 8(4), 441–480.CrossRefGoogle Scholar
  75. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354.CrossRefGoogle Scholar
  76. Kalsbeek, J. W. H., & Sykes, R. N. (1967). Objective measurement of mental load. Acta Psychologica, 27, 253–261.CrossRefGoogle Scholar
  77. Kalyuga, S. (2009). Managing cognitive load in adaptive multimedia learning. Hershey: Information Science Reference.CrossRefGoogle Scholar
  78. Kanazawa, S. (1998). In defense of unrealistic assumptions. Sociological Theory, 16(2), 193–204.CrossRefGoogle Scholar
  79. Karwowski, W. (2012). The discipline of human factors and ergonomics. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 1–37). Hoboken: Wiley.Google Scholar
  80. Kit, C. Y., & Wong, B. T. M. (2015). Evaluation in machine translation and computer-aided translation. In S. W. Chan (Ed.), Routledge encyclopedia of translation technology (pp. 213–236). London: Routledge.Google Scholar
  81. Klare, G. R. (1984). Readability. In P. D. Pearson & R. Barr (Eds.), Handbook of reading research (pp. 681–744). New York: Longman.Google Scholar
  82. Koehn, P. (2010). Statistical machine translation. New York: Cambridge University Press.Google Scholar
  83. Koehn, P., & Germann, U. (2014). The impact of machine translation quality on human post-editing. Paper presented at the Workshop on Humans and Computer-Assisted Translation (HaCaT), Gothenburg, Sweden.Google Scholar
  84. Koponen, M. (2012). Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the 7th Workshop on Statistical Machine Translation (pp. 181–190). Montreal: Association for Computational Linguistics.Google Scholar
  85. Koponen, M., Aziz, W., Ramos, L., & Specia, L. (2012). Post-editing time as a measure of cognitive effort. Paper presented at the AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), San Diego.Google Scholar
  86. Krings, H. P. (2001). Repairing texts: Empirical investigations of machine translation post-editing processes. (G. Koby, G. Shreve, K. Mischerikow & S. Litzer, Trans.). Kent, Ohio: Kent State University Press.Google Scholar
  87. Kuiken, F., & Vedder, I. (2007). Task complexity needs to be distinguished from task difficulty. In M. D. P. GarcíaMayo (Ed.), Investigating tasks in formal language learning (pp. 117–135). Clevedon: Multilingual Matters.Google Scholar
  88. Lacruz, I., & Shreve, G. M. (2014). Pauses and cognitive effort in post-editing. In S. O’Brien, L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation: Processes and applications (pp. 246–272). Newcastle: Cambridge Scholars Publishing.Google Scholar
  89. Lacruz, I., Shreve, G. M., & Angelone, E. (2012). Average pause ratio as an indicator of cognitive effort in post-editing: A case study. Paper presented at the AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012), San Diego.Google Scholar
  90. Liu, M., & Chiu, Y.-H. (2009). Assessing source material difficulty for consecutive interpreting: Quantifiable measures and holistic judgment. Interpreting, 11(2), 244–266.CrossRefGoogle Scholar
  91. Liu, Q., & Zhang, X. (2015). Machine translation: General. In S. W. Chan (Ed.), Routledge encyclopedia of translation technology (pp. 105–119). London: Routledge.Google Scholar
  92. Lusk, M. M., & Atkinson, R. K. (2007). Animated pedagogical agents: Does their degree of embodiment impact learning from static or animated worked examples? Applied Cognitive Psychology, 21(6), 747–764.CrossRefGoogle Scholar
  93. Martins, D. B., & Caseli, H. (2015). Automatic machine translation error identification. Machine Translation, 29(1), 1–24.CrossRefGoogle Scholar
  94. Mesa-Lao, B. (2013). Introduction to post-editing–The CasMaCat GUI. Retrieved March 1, 2015 from
  95. Meshkati, N. (1988). Toward development of a cohesive model of workload. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 305–314). Amsterdam: North-Holland.CrossRefGoogle Scholar
  96. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97.CrossRefGoogle Scholar
  97. Mishra, A., Bhattacharyya, P., & Carl, M. (2013, August 4–9). Automatically predicting sentence translation difficulty. Paper presented at the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria.Google Scholar
  98. Moors, A., & De Houwer, J. (2006). Automaticity: A theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326.CrossRefGoogle Scholar
  99. Moran, J., & Lewis, D. (2011). Unobtrusive methods for low-cost manual evaluation of machine translation. Retrieved April 1, 2015 from
  100. Moray, N. (1977). Models and measures of mental workload. In N. Moray (Ed.), Mental workload: Its theory and measurement (pp. 13–21). New York: Springer.Google Scholar
  101. Muñoz Martín, R. (2010). Leave no stone unturned: On the development of cognitive translatology. Translation and Interpreting Studies, 5(2), 145–162.CrossRefGoogle Scholar
  102. Muñoz Martín, R. (2012). Just a matter of scope. Translation Spaces, 1(1), 169–188.CrossRefGoogle Scholar
  103. Muñoz Martín, R. (2014). A blurred snapshot of advances in translation process research. MonTI. Special Issue (Minding Translation), 1, 49–84.Google Scholar
  104. Newell, W. H. (2001). A theory of interdisciplinary studies. Issues in Integrative Studies, 19, 1–25.Google Scholar
  105. Nimon, K. F. (2012). Statistical assumptions of substantive analyses across the general linear model: A mini-review. Frontiers in Psychology, 3, 1–5.Google Scholar
  106. Nkwake, A. M. (2013). Working with assumptions in international development program evaluation. New York: Springer.CrossRefGoogle Scholar
  107. Nord, C. (2005). Text analysis in translation: Theory, methodology, and didactic application of a model for translation-oriented text analysis (2nd ed.). Amsterdam: Rodopi.Google Scholar
  108. O’Brien, S. (2004). Machine Translatability and Post-Editing Effort: How do they relate? Paper presented at the 26th Translating and the Computer Conference (ASLIB), London.Google Scholar
  109. O’Brien, S. (2005). Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation, 19(1), 37–58.CrossRefGoogle Scholar
  110. O’Brien, S. (2006). Pauses as indicators of cognitive effort in post-editing machine translation output. Across Languages and Cultures, 7(1), 1–21.CrossRefGoogle Scholar
  111. O’Brien, S. (2007a). An empirical investigation of temporal and technical post-editing effort. Translation and Interpreting Studies, 2(1), 83–136.CrossRefGoogle Scholar
  112. O’Brien, S. (2007b). Eye-tracking and translation memory matches. Perspectives, 14(3), 185–205.CrossRefGoogle Scholar
  113. O’Brien, S. (2010). Controlled language and readability. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 143–165). Amsterdam: John Benjamins.CrossRefGoogle Scholar
  114. O’Brien, S. (2011). Cognitive explorations of translation. London: Continuum.Google Scholar
  115. O’Brien, S., Balling, L. W., Carl, M., Simard, M., & Specia, L. (Eds.). (2014). Post-editing of machine translation: Processes and applications. Newcastle: Cambridge Scholars Publishing.Google Scholar
  116. O’Donnell, R. D., & Eggemeier, F. T. (1986). Workload assessment methodology. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, Vol. II: Cognitive processes and performance (pp. 42/41–42–49). New York: Wiley.Google Scholar
  117. Onwuegbuzie, A. J., & Daniel, L. G. (1999, November 17–19). Uses and misuses of the correlation coefficient. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, Point Clear, AL.Google Scholar
  118. Ortiz-Martínez, D., Sanchis-Trilles, G., Casacuberta, F., Alabau, V., Vidal, E., Benedı, J.-M … González, J. (2012). The CASMACAT project: The next generation translator’s workbench. Paper presented at the 7th Jornadas en Tecnologıa del Habla and the 3rd Iberian SLTech Workshop (IberSPEECH), Madrid.Google Scholar
  119. Orzechowski, J. (2010). Working memory capacity and individual differences in higher-level cognition. In G. Matthews & B. Szymura (Eds.), Handbook of individual differences in cognition (pp. 353–368). New York: Springer.CrossRefGoogle Scholar
  120. Paas, F. G. W. C. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429–434.CrossRefGoogle Scholar
  121. Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994a). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6(4), 351–371.CrossRefGoogle Scholar
  122. Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994b). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86(1), 122–133.CrossRefGoogle Scholar
  123. Paas, F. G. W. C., Ayres, P., & Pachman, M. (2008). Assessment of cognitive load in multimedia learning. In D. H. Robinson & G. Schraw (Eds.), Assessment of cognitive load in multimedia learning: Theory, methods and applications (pp. 11–35). Charlotte, NC: Information Age Publishing.Google Scholar
  124. Palumbo, G. (2009). Key terms in translation studies. London: Continuum.Google Scholar
  125. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (pp. 311–318). University of Pennsylvania, Philadelphia: Association for Computational Linguistics.Google Scholar
  126. Poole, A., & Ball, L. J. (2006). Eye tracking in HCI and usability research. In C. Ghaoui (Ed.), Encyclopedia of human computer interaction (pp. 211–219). London: Idea Group.CrossRefGoogle Scholar
  127. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422.CrossRefGoogle Scholar
  128. Rayner, K., & Pollatsek, A. (1989). Psychology of reading. Hillsdale: Lawrence Erlbaum.Google Scholar
  129. Redfield, C. L. (1922). Mental levels. Journal of Education, 95(8), 214–216.CrossRefGoogle Scholar
  130. Reichle, E. D., Reineberg, A. E., & Schooler, J. W. (2010). Eye movements during mindless reading. Psychological Science, 21(9), 1300–1310.CrossRefGoogle Scholar
  131. Robinson, P. (2001). Task complexity, cognitive resources, and syllabus design: A triadic framework for examining task influences on SLA. In P. Robinson (Ed.), Cognition and second language instruction (pp. 287–318). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  132. Rosa, R. (2014). Depfix, a tool for automatic rule-based post-editing of SMT. The Prague Bulletin of Mathematical Linguistics, 102(1), 47–56.CrossRefGoogle Scholar
  133. Rost, M. (2006). Areas of research that influence L2 listening instruction. In E. Usó Juan & A. Martínez Flor (Eds.), Current trends in the development and teaching of the four language skills (pp. 47–74). Berlin: Mouton de Gruyter.CrossRefGoogle Scholar
  134. Schaeffer, M., & Carl, M. (2013). Shared representations and the translation process: A recursive model. Translation and Interpreting Studies, 8(2), 169–190.CrossRefGoogle Scholar
  135. Schaeffer, M., & Carl, M. (2014). Measuring the cognitive effort of literal translation processes. Paper presented at the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden.Google Scholar
  136. Schilperoord, J. (1996). It’s about time: Temporal aspects of cognitive processes in text production. Amsterdam: Rodopi.Google Scholar
  137. Schleiermacher, F. (2012). On the different methods of translating (S. Bernofsky, Trans.). In L. Venuti (Ed.), The translation studies reader (3rd ed., pp. 43–63). London: Routledge.Google Scholar
  138. Sharmin, S., Špakov, O., Räihä, K.-J., & Jakobsen, A. L. (2008). Where on the screen do translation students look while translating, and for how long? In S. Göpferich, A. L. Jakobsen, & I. M. Mees (Eds.), Looking at eyes: Eye-tracking studies of reading and translation processing (pp. 31–51). Copenhagen: Samfundslitteratur.Google Scholar
  139. Shreve, G. M. (2002). Knowing translation: Cognitive and experiential aspects of translation expertise from the perspective of expertise studies. In A. Ruiccardi (Ed.), Translation studies: Perspectives on an emerging discipline (pp. 150–173). Cambridge: Cambridge University Press.Google Scholar
  140. Shreve, G. M., & Angelone, E. (Eds.). (2010). Translation and cognition. Amsterdam: John Benjamins.Google Scholar
  141. Silveira, F. d. S. D. d. (2011). Working memory capacity and lexical access in advanced students of L2 English. PhD thesis. Universidade Federal do Rio Grande do Sul, Brazil. Retrieved from
  142. Sirén, S., & Hakkarainen, K. (2002). Expertise in translation. Across Languages and Cultures, 3(1), 71–82.CrossRefGoogle Scholar
  143. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (pp. 223–231). Cambridge, MA.Google Scholar
  144. Sousa, S. C. M. d., Aziz, W. F., & Specia, L. (2011). Assessing the post-editing effort for automatic and semi-automatic translations of DVD subtitles. In Proceedings of the International Conference of Recent Advances in Natural Language Processing (pp. 97–103). Bulgaria.Google Scholar
  145. Specia, L. (2011). Exploiting objective annotations for measuring translation post-editing effort. Paper presented at the 15th Conference of the European Association for Machine Translation, Leuven.Google Scholar
  146. Specia, L., Raj, D., & Turchi, M. (2010). Machine translation evaluation versus quality estimation. Machine Translation, 24(1), 39–50.CrossRefGoogle Scholar
  147. Stymne, S. (2011). Blast: A tool for error analysis of machine translation output. Paper presented at the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon.Google Scholar
  148. Sun, S. (2011). Think-aloud-based translation process research: Some methodological considerations. Meta, 56(4), 928–951.CrossRefGoogle Scholar
  149. Sun, S. (2015). Measuring translation difficulty: Theoretical and methodological considerations. Across Languages and Cultures, 16(1), 29–54.CrossRefGoogle Scholar
  150. Sun, S., & Shreve, G. M. (2014). Measuring translation difficulty: An empirical study. Target, 26(1), 98–127.CrossRefGoogle Scholar
  151. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.CrossRefGoogle Scholar
  152. Tatler, B. W., Kirtley, C., Macdonald, R. G., Mitchell, K. M., & Savage, S. W. (2014). The active eye: Perspectives on eye movement research. In M. Horsley, M. Eliot, B. A. Knight, & R. Reilly (Eds.), Current trends in eye tracking research (pp. 3–16). London: Springer.CrossRefGoogle Scholar
  153. TAUS. (2014). Post-editing: Championing MT. Retrieved March 1, 2015 from
  154. Thorndike, E. L., Bregman, E. O., Cobb, M. V., & Woodyard, E. (1927). The measurement of intelligence. New York: Bureau of Publications, Columbia University.Google Scholar
  155. Tirkkonen-Condit, S. (2005). The monitor model revisited: Evidence from process research. Meta, 50(2), 405–414.CrossRefGoogle Scholar
  156. Tokowicz, N., Kroll, J. F., De Groot, A. M. B., & Van Hell, J. G. (2002). Number-of-translation norms for Dutch – English translation pairs: A new tool for examining language production. Behavior Research Methods, Instruments, & Computers, 34(3), 435–451.CrossRefGoogle Scholar
  157. Tomporowski, P. D. (2003). Performance and perceptions of workload among young and older adults: Effects of practice during cognitively demanding tasks. Educational Gerontology, 29(5), 447–466.CrossRefGoogle Scholar
  158. Turner, M. L., & Engle, R. W. (1989). Is working memory capacity task dependent? Journal of Memory and Language, 28(2), 127–154.CrossRefGoogle Scholar
  159. Vasconcellos, M. (1987). A comparison of MT post-editing and traditional revision. In K. Kummer (Ed.), Proceedings of the 28th annual conference of the American Translators Association (pp. 409-416). Medford: Learned Information.Google Scholar
  160. Vidulich, M. A., & Tsang, P. S. (2012). Mental workload and situation awareness. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (4th ed., pp. 243–273). Hoboken: Wiley.CrossRefGoogle Scholar
  161. Vieira, L. N. (2014). Indices of cognitive effort in machine translation post-editing. Machine Translation, 28(3-4), 187–216.CrossRefGoogle Scholar
  162. Vonk, W., & Cozijn, R. (2003). On the treatment of saccades and regressions in eye movement measures of reading time. In J. Hyona, R. Radach, & H. deubel (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 291–312). London: Elsevier.CrossRefGoogle Scholar
  163. Wierwille, W. W., & Williges, B. H. (1980). An annotated bibliography on operator mental workload assessment (Naval Air Test Center Report No. SY-27R-80). Patuxent River: Naval Air Test Center, System Engineering Test Directorate.Google Scholar
  164. Wilson, R. A., & Keil, F. C. (Eds.). (1999). The MIT encyclopedia of the cognitive sciences. Cambridge: MIT Press.Google Scholar
  165. Wilss, W. (1982). The science of translation: Problems and methods. Tübingen: Gunter Narr.Google Scholar
  166. Wisniewski, G., Kübler, N., & Yvon, F. (2014). A corpus of machine translation errors extracted from translation students exercises. Paper presented at the International Conference on Language Resources and Evaluation (LREC), Iceland.
  167. Woodrow, H. (1936). The measurement of difficulty. Psychological Review, 43(4), 341–365.CrossRefGoogle Scholar
  168. Zipf, G. K. (1935). The psycho-biology of language: An introduction to dynamic philology. Boston: Houghton Mifflin.Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Sanjun Sun
    • 1
  1. 1.School of English and International StudiesBeijing Foreign Studies UniversityBeijingChina

Personalised recommendations