Abstract
Human rating of predicted post-editing effort is a common activity and has been used to train confidence estimation models. However, the correlation between human ratings and actual post-editing effort is under-measured. Moreover, the impact of presenting effort indicators in a post-editing user interface on actual post-editing effort has hardly been researched. In this study, ratings of perceived post-editing effort are tested for correlations with actual temporal, technical and cognitive post-editing effort. In addition, the impact on post-editing effort of the presentation of post-editing effort indicators in the user interface is also tested. The language pair involved in this study is English-Brazilian Portuguese. Our findings, based on a small sample, suggest that there is little agreement between raters for predicted post-editing effort and that the correlations between actual post-editing effort and predicted effort are only moderate, and thus an inefficient basis for MT confidence estimation. Moreover, the presentation of post-editing effort indicators in the user interface appears not to impact on actual post-editing effort.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Cognitive effort was also recorded for this cohort using eye tracking, but the data will be analysed at a later stage.
This is a fork of HandyCAT (Hokamp and Liu 2015).
Participant anonymity is a standard requirement of the university research ethics approval process.
Using the esurv.org platform.
Available from www.cs.umd.edu/~snover/tercom.
“Its main economic activities include agriculture, forestry, fishing, mining, and manufacturing goods such as textiles, clothing, refined metals, and refined petroleum. Bolivia is very wealthy in minerals, especially tin.” This was the only segment that contained two sentences due to a segmentation error.
References
Alves F, Koglin A, Mesa-Lao B, García MM, Fonseca NBL, Sá AM, Gonçalves JL, Szpak KS, Sekino K, Aquino M (2015) Analysing the impact of interactive machine translation on post-editing effort. In: Carl M, Bangalore S, Schaeffer M (eds) New directions in empirical translation process research. Springer, New York, pp 77–95
Biçici E, Way A (2014) Referential translation machines for predicting translation quality. WMT 2014: ACL 2014 Ninth Workshop On Statistical Machine Translation, Baltimore
Blain F, Senellart J, Schwenk H, Plitt M, Roturier J (2011) Qualitative analysis of post-editing for high quality machine translation. In: Machine Translation Summit XIII, Asia-Pacific Association for Machine Translation (AAMT), Xiamen
Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, Sanchis A, Ueffing N (2004) Confidence estimation for machine translation. Proceedings of the 20th international conference on computational linguistics, 23–27 Aug 2004, Geneva, pp 315–321
Carl M, Dragsted B, Elming J, Hardt D, Jakobsen AL (2011) The process of post-editing: a pilot study. In: Proceedings of the 8th international natural language processing and cognitive science workshop, Frederiksberg, pp 131–142
Cooper A (2004) The inmates are running the asylum: why hi-tech products drive us crazy and how to restore the sanity. Sams Publishing, Indianapolis
Da Silva IAL, Schmaltz M, Alves F, Pagano A, Wong D, Chao L, Leal ALV, Quaresma P, Garcia C (2015) Translating and post-editing in the Chinese-Portuguese Language pair: insights from an exploratory study of key logging and eye tracking. Translation Spaces 4(1):145–169
De Almeida G, O’Brien S (2010) Analysing post-editing performance: Correlations with years of translation experience. In: Proceedings of the 14th annual conference of the European Association for Machine Translation, St. Raphaël
DePalma DA, Hegde V, Pielmeier H, Stewart RG (2013) The language services market. Common Sense Advisory, Boston
Doherty S (2012) Investigating the effects of controlled language on the reading and comprehension of machine translated texts: a mixed-methods approach. Dissertation, Dublin City University
Gaspari F, Toral A, Kumar NS, Groves D, Way A (2014) Perception vs. reality: measuring machine translation post-editing productivity. In: O’Brien S, Simard M, Specia L (eds) Proceedings of the 11th conference of the Association for Machine Translation in the Americas: workshop on post-editing technology and practice (WPTP3), Vancouver, pp 60–72
Guerberof A (2009) Productivity and quality in the post-editing of outputs from translation memories and machine translation. Localisation Focus 7(1):11–21
Hokamp C, Liu C (2015) HandyCAT. In: Durgar El-Kahlout İ, Özkan M, Sánchez-Martínez F, Ramírez-Sánchez G, Hollowood F, Way A (eds) Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, p 216
Koponen M (2012) Comparing human perceptions of post-editing effort with post-editing operations. In: Proceedings of the 7th workshop on statistical machine translation, Montreal, pp 181–190
Krings HP (2001) Repairing texts. Kent State University Press, Kent
Lacruz I, Shreve GM (2014) Pauses and cognitive effort in post-editing. In: O’Brien S, Balling LW, Carl M, Simard M, Specia L (eds) Post-editing of machine translation: processes and applications. Cambridge Scholars Publishing, Newcastle-Upon-Tyne, pp 246–272
Läubli S, Fishel M, Massey G, Ehrensberger-Dow M, Volk M (2013) Assessing post-editing efficiency in a realistic translation environment. In: Simard M, Specia L (eds) O’Brien S. Proceedings of MT Summit XIV workshop on post-editing technology and practice, Nice, pp 83–91
Moorkens J, O’Brien S (2013) User attitudes to the post-editing interface. In: Simard M, Specia L (eds) O’Brien S. Proceedings of MT Summit XIV workshop on post-editing technology and practice, Nice, pp 19–25
Moorkens J, O’Brien S (2015) Post-editing evaluations: Trade-offs between novice and professional participants. In: Durgar El-Kahlout İ, Özkan M, Sánchez-Martínez F, Ramírez-Sánchez G, Hollowood F, Way A (eds) Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, pp 75–81
O’Brien S (2005) Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation 19(1):37–58
O’Brien S (2010) Introduction to Post-Editing: Who, What, How and Where to Next? In: Proceedings of AMTA 2010, the 9th conference of the Association for Machine Translation in the Americas (online), Denver
O’Brien S (2011) Towards predicting post-editing productivity. Machine Translation 25:197–215
Plitt M, Masselot F (2010) A productivity test of statistical machine translation post-editing in a typical localization context. Prague Bull Math Linguist 93:7–16
Quirk C (2004) Training a sentence-level machine translation confidence measure. In: Proceedings of the 4th conference on language resources and evaluation, Lisbon, pp 825–828
Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124(3):372–422
Shah K, Specia L (2014) Quality estimation for translation selection. In: Tadic M, Koehn P, Roturier J, Way A (eds) Proceedings of the 17th annual conference of the European Association for Machine Translation (EAMT 2014), Dubrovnik, pp 109–116
Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th conference of the Association for Machine Translation in the Americas (AMTA2006), “Visions for the Future of Machine Translation’, Cambridge, pp 223–231
Soricut R, Narsale S (2012) Combining quality prediction and system selection for improved automatic translation output. In: Proceedings of the ACL 7th workshop on statistical machine translation (WMT-2012), Montreal, pp 163–170
Specia L (2011) Exploiting objective annotations for measuring translation post-editing effort. In: Proceedings of the 15th conference of EAMT, Leuven, pp 73–80
Specia L, Cancedda N, Dymetman M, Turchi M, Cristianini N (2009) Estimating the sentence-level quality of machine translation systems. In: Proceedings of the 13th annual conference of the EAMT, Barcelona, pp 28–35
Sperber D, Wilson D (1986) Relevance: communication and cognition. Blackwell, Oxford
Tatsumi M (2009) Correlation between automatic evaluation scores, post-editing speed and some other factors. In: Proceedings of MT Summit XII, Ottawa, pp 332–339
Tatsumi M, Roturier J (2010) Source text characteristics and technical and temporal post-editing effort: what is their relationship? In: Zhechev V (ed) Proceedings of the 2nd joint EM+/CNGL workshop “Bringing MT to the User: Research on Integrating MT in the Translation Industry” (JEC ’10). Denver, pp 43–51
Teixeira CSC (2014) Perceived vs. measured performance in the post-editing of suggestions from machine translation and translation memories. In: O’Brien S, Simard M, Specia L (eds) Proceedings of the 11th conference of the association for machine translation in the Americas: workshop on post-editing technology and practice (WPTP3), Vancouver, pp 45–59
Temnikova I (2010) Cognitive evaluation approach for a controlled language post-editing experiment. In: Proceedings of the 7th international conference on language resources and evaluation, LREC 2010, Valletta, pp 3485–3490
Turian J, Shen L, Melamed D (2003) Evaluation of machine translation and its evaluation. In: Proceedings of the MT Summit IX, New Orleans, pp 386–393
Vieira L, Specia L (2011) A review of translation tools from a post-editing perspective. In: Proceedings of the 3rd joint EM+/CNGL workshop bringing MT to the users: research meets translators” (JEC ’11), Luxembourg, pp 33–42
Vieira LN (2014) Indices of cognitive effort in machine translation post-editing. Machine Translation 28(3):187–216
Acknowledgments
This research is supported by Science Foundation Ireland (Grant 12/CE/I2267) as part of the CNGL (www.cngl.ie) at Dublin City University, Research Brazil Ireland, and by the FALCON Project (falcon-project.eu), funded by the European Commission through the Seventh Framework Programme (FP7) Grant Agreement No. 610879. The authors would like to place on record their thanks to Research Brazil Ireland, and to staff and students at the Faculdade de Letras at UFMG.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Moorkens, J., O’Brien, S., da Silva, I.A.L. et al. Correlations of perceived post-editing effort with measurements of actual effort. Machine Translation 29, 267–284 (2015). https://doi.org/10.1007/s10590-015-9175-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10590-015-9175-2