Abstract
This study investigates the relationship between machine translation (MT) and human translation (HT) through the lens of word translation entropy, also known as HTra (i.e., a metric that measures how many different translations a given source text word has). We aligned different translations from multiple MT systems (three different target languages: Japanese, Arabic, and Spanish) with the same English source texts (STs) to calculate HTra for each language, and we then compared these values to additional HT data sets of the same STs and languages. We found that MT HTra correlates strongly with HT HTra within and across the languages. We also annotated the ST in terms of word class, figurative expressions, voice, and anaphora in order to examine the relationships these ST features have with HTra. For this same purpose, we normalized all HTra values (nHTra) in order to compare HTra values across all six data sets. We found that these source text features are, in general, associated with HTra in the same manner regardless of target language or the distinction between MT and HT.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This online calculator (https://www.shannonentropy.netmark.pl/) can be a useful resource for understanding how entropy is calculated.
- 2.
We used the following MT systems. Amazon Translate, Bing, DayTranslations, Google, Online English Arabic Translator, Prompt Online, Reverso, Systran, Tradukka, Translator.eu, Translator, and Yandex for Arabic; Baidu, Bing, Excite, Google, Paralink ImTranslator, Infoseek, MiraiTranslate, Pragma, So-Net, Textra, Weblio, WorldLingo, and Yandex for Japanese; and Amazon Translate, Baidu, Bing, DeepL, Google, Lilt, Pragma, Yarakuzen, and Yandex for Spanish.
- 3.
This was evaluated manually by the researchers, based on criteria of simple usability.
- 4.
The data, including a description of the multiLing texts, is publicly available on the CRITT website under the following study IDs: ARMT19 for the Arabic data; JAMT19 for the Japanese data; and ESMT19 for the Spanish data.
- 5.
The version of the AR20 study that we used had less sessions than the current version and can be downloaded from https://sourceforge.net/projects/tprdb/ under the version number “r561.”
- 6.
All our annotations of the multiLing source texts can be downloaded from https://devrobgilb.com/Researcher/Repository/multiLing/.
- 7.
The tagset used for PoS is the Penn Treebank Project tagset and can be found at https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html.
- 8.
- 9.
Japanese target texts are morphologically divided into ‘words’ because there are no typed spaces in texts. This granular tokenization probably influences HTra, resulting in higher values compared to other languages.
References
Almazroei SA, Ogawa H, Gilbert D (2019) Investigating correlations between human translation and MT output. In: Proceedings of the second MEMENTO workshop on modelling parameters of cognitive effort in translation production, European association for machine translation, pp 11–13, MT Summit 2019, Second MEMENTO workshop on modelling parameters of cognitive effort in translation production; Dublin, 20 Aug 2019
Bernth A, Gdaniec C (2001) MTranslatability. Mach Transl 16(3):175–218
Bracken J, Degani T, Eddington C, Tokowicz N (2017) Translation semantic variability: how semantic relatedness affects learning of translation-ambiguous words. Biling Lang Congn 20(4):783–794
Carl M, Schaeffer M (2014) Word transition entropy as an indicator for expected machine translation quality. In: Proceedings of the workshop on automatic and manual metrics for operational translation evaluation, MTE 2014, European Language Resources Association, pp 45–50, LREC 2014 workshop on automatic and manual metrics for operational translation evaluation; Reykjavik, 26 May 2014
Carl M, Schaeffer MJ (2017) Why translation is difficult: a corpus-based study of non-literality in post-editing and from-scratch translation. HERMES J Lang Commun Bus (56):43–57
Carl M, Toledo Báez MC (2019) Machine translation errors and the translation process: a study across different languages. J Specialised Transl (31):107–132
Carl M, Aizawa A, Yamada M (2016a) English-to-Japanese translation vs. dictation vs. post-editing: comparing translation modes in a multilingual setting. In: Proceedings of the tenth international conference on language resources and evaluation (LREC 2016). European Language Resources Association, pp 4024–4031, LREC 2016, Tenth international conference on language resources and evaluation; Portorož, 23–28 May 2016
Carl M, Schaeffer M, Bangalore S (2016b) The CRITT translation process research database. In: Carl M, Schaeffer M, Bangalore S (eds) New directions in empirical translation process research. Springer, pp 13–54
Chiswick BR, Miller PW (2004) Linguistic distance: a quantitative measure of the distance between English and other languages. IZA Discussion Papers (1246)
Daems J, Macken L, Vandepitte S (2014) On the origin of errors: a fine-grained analysis of MT and PE errors and their relationship. In: Proceedings of the ninth international conference on language resources and evaluation (LREC 2014). European Language Resources Association, pp 62–66, Ninth International conference on language resources and evaluation; Reykjavik, 26–31 May 2014
Eddington CM, Tokowicz N (2013) Examining English-German translation ambiguity using primed translation recognition. Biling Lang Congn 16(2):442–457
Germann U (2008) Yawat: yet another word alignment tool. In: Proceedings of the 46th annual meeting of the association for computational linguistics on human language technologies: demo session. Association for Computational Linguistics, pp 20–23, 46th annual meeting of the association for computational linguistics on human language technologies: Demo Session; Columbus, 16 June 2008
Guerberof Arenas A (2009) Productivity and quality in MT post-editing. In: Proceedings of XII MT summit workshop: beyond translation memories. International Association for Machine Translation, XII MT Summit Workshop: beyond translation memories, Ottawa, 26–30 Aug 2009
Isphording IE, Otten S (2013) The costs of Babylon–linguistic distance in applied economics. Review of International Econ 21(2):354–369
Isphording IE, Otten S (2014) Linguistic barriers in the destination language acquisition of immigrants. J Econ Behav Organ 105:30–50
Jia Y, Carl M, Wang X (2019) How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study. J Specialised Transl (31):61–86
Koponen M, Salmi L, Nikulin M (2019) A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output. Mach Transl 33(1–2):61–90
Kudo T, Matsumoto Y (2002) Japanese dependency analysis using cascaded chunking. In: Proceedings of the 6th conference on natural language learning. Association for Computational Linguistics, pp 1–7, 6th conference on natural language learning; Taipei, Aug 31 to 1 Sept 2002
Laxén J, Lavaur JM (2010) The role of semantics in translation recognition: Effects of number of translations, dominance of translations and semantic relatedness of multiple translations. Biling Lang Congn 13(2):157–183
Läubli S, Castilho S, Neubig G, Sennrich R, Shen Q, Toral A (2020) A set of recommendations for assessing human-machine parity in language translation. J Artif Intell Res 67:653–672
Marzouk S, Hansen-Schirra S (2019) Evaluation of the impact of controlled language on neural machine translation compared to other MT architectures. Mach Transl 33(1–2):179–203
Meldrum YF (2009) Translationese-specific linguistic characteristics: a corpus-based study of contemporary Japanese translationese. Honyaku kenkyū e no shōtai 3:105–132
Mesa-Lao B (2014) Gaze behaviour on source texts: an exploratory study comparing translation and post-editing. In: O’Brien S, Winther Balling L, Carl M, Simard M, Specia L (eds) Post-editing of machine translation: processes and applications. Cambridge Scholars Publishing, pp 219–245
Mitkov R (2014) Anaphora resolution. Routledge
Moorkens J, O’Brien S (2015) Post-editing evaluations: trade-offs between novice and professional participants. In: Proceedings of the 18th annual conference of the European association for machine translation. European Association for Machine Translation, pp 75–81, 18th annual conference of the European association for machine translation; Antalya, 11–13 May 2015
O’Brien S (2006) Machine-translatability and post-editing effort: an empirical study using translog and choice network analysis. Doctoral Thesis, Dublin City University
Prior A, MacWhinney B, Kroll JF (2007) Translation norms for English and Spanish: the role of lexical variables, word class, and L2 proficiency in negotiating translation ambiguity. Behav Res Methods 39(4):1029–1038
Safir KJ (2004) The syntax of anaphora. Oxford University Press
Schaeffer MJ, Oster K, Nitzke J, Tardel A, Gros AK, Gutermuth S, Hansen-Schirra S, Carl M (2018) Cross-linguistic (dis)similarities in translation: process and product. In: Book of abstracts: using corpora in contrastive and translation studies conference (5th edition), centre for English corpus linguistics, pp 90–92, using corpora in contrastive and translation studies conference (5th edition); Louvain-la-Neuve, 12–14 Sept 2018
Schwartz L (2018) The history and promise of machine translation. In: Lacruz I, Jääskeläinen R (eds) Innovation and expansion in translation process research. John Benjamins Publishing Company, pp 161–190
Shannon CE (1951) Prediction and entropy of printed English. Bell Syst Tech J 30(1):50–64
Tokowicz N (2014) Translation ambiguity affects language processing, learning, and representation. In: Selected proceedings of the 2012 second language research forum. Cascadilla Press Somerville, pp 170–180, 31st Second Language Research Forum; Pittsburgh, 18–21 Oct 2012
Underwood N, Jongejan B (2001) Translatability checker: a tool to help decide whether to use MT. In: Proceedings of MT summit VIII: machine translation in the information age, European association for machine translation, pp 363–368, MT summit VIII: machine translation in the information age; Santiago de Compostela, 18–22 Sept 2001
Vanroy B, De Clercq O, Macken L (2019) Correlating process and product data to get an insight into translation difficulty. Perspectives 27(6):924–941
Voita E, Serdyukov P, Sennrich R, Titov I (2018) Context-aware neural machine translation learns anaphora resolution. In: Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers), Association for computational linguistics, pp 1264–1274, 56th Annual meeting of the association for computational linguistics; Melbourne, 15–20 July 2018
Acknowledgements
More thanks than we can give to Kristin Yeager, the Manager of Statistical Consulting at Kent State University Libraries. Also, a special thanks to Andrew Tucker for coming up with our project’s codename.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
H LMEM
H LMEM
For our exploratory analyses, we examined the data using linear mixed-effects models (LMEM) to see whether there is a significant difference among the groups in each ST linguistic feature in terms of the nHTra values. LMEM is a better choice than a t-test because the nHTra values were uniquely distributed across studies with a high frequency of zero values.
We first ran LMEM using the “lme4” package on RStudio with all the data points from six studies. We set nHTra as the dependent variable, Text (i.e., 1–6) and TextId (i.e., a combination of Text number and word ID as in “1_5” for the fifth word of Text 1) as a crossed random effect, and one of our syntactic/semantic categories as a fixed effect. We examined a three-way interaction effect among the category, the target language, and the distinction between HT and MT. We then ran a similar analysis with the data narrowed down by the target language. For WordClass, Verb was significantly different from the other four groups in all six studies. For Figurative, the three groups (Metaphoric, Fixed and Other) were significantly different from each other in all studies except the case of Fixed vs. Other in JA_MT. For Voice, we only observed a significant difference between Passive and Active in AR_MT and ES_HT. And for Anaphora, we found significant difference between Anaphoric and Other in all studies except JA_MT and ES_MT.
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Ogawa, H., Gilbert, D., Almazroei, S. (2021). RedBird: Rendering Entropy Data and ST-Based Information into a Rich Discourse on Translation. In: Carl, M. (eds) Explorations in Empirical Translation Process Research. Machine Translation: Technologies and Applications, vol 3. Springer, Cham. https://doi.org/10.1007/978-3-030-69777-8_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-69777-8_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69776-1
Online ISBN: 978-3-030-69777-8
eBook Packages: EducationEducation (R0)