Automatic Evaluation of MT Output and Post-edited MT Output for Genealogically Related Languages
The aim of the research is twofold: to evaluate the translation quality of the individual sentences of the MT output and also post-edited MT output on the basis of metrics of automatic MT evaluation from Slovak into the German language; and to compare the quality of MT output and post-edited MT output based on the same automatic metrics of MT evaluation. The icon graphs were used to visualize the results for individual sentences. A significant difference was found in sentence 36 in favor of the post-edited MT output and vice versa in sentence 5 in favor of MT output. Due to the error rate, a significant difference was in sentence 29 and 11 in favor of post-edited MT output and vice versa the sentence 26 in favor of MT output. Based on our results we can state that it is necessary to include into the evaluation of the quality of translation all automatic metrics for each sentence separately.
KeywordsLanguage processing Machine translation Automatic MT metrics Genealogically related languages
This work was supported by the SRD Agency under the contract No. APVV-18-0473 and Scientific Grant Agency of the ME SR and SAS under the contracts No. VEGA-1/0809/18.
This publication was supported by the Operational Program: Research and Innovation project “Fake news on the Internet - identification, content analysis, emotions”, co-funded by the European Regional Development Fund.
- 1.Munkova, D., Munk, M., Benko, L., Absolon, J.: From old fashioned “one size fits all” to tailor made online training. In: Auer, M.E., Tsiatsos, T. (eds.) ICL 2018. Advances in Intelligent Systems and Computing, vol. 916, pp. 365–376. Springer, Cham (2019)Google Scholar
- 2.Chéragui, M.A.: Theoretical overview of machine translation. In: Proceedings of ICWIT 2012, pp. 160–169. CEUR-WS, Sidi Bel Abbes, Algeria (2012)Google Scholar
- 3.Wallis, J.: Interactive translation vs. pre-translation in the context of translation memory systems: investigating the effects of translation method on productivity, quality and translator satisfaction. University of Ottawa, Ottawa (2006)Google Scholar
- 4.Munkova, D., Munk, M.: Evalvácia strojového prekladu. Univerzita Konštantína Filozofa v Nitre, Nitra (2016)Google Scholar
- 5.DePalma, D.A.: How to add post-edited MT to your service offerings. Common Sense Advisory, Cambridge, MA (2013)Google Scholar
- 6.Van Slype, G.: Critical study of methods for evaluating the quality of machine translation. Technical report, Bureau Marcel van Dijk/European Commission, Brussels (1979Google Scholar
- 7.Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, Pennsylvania, pp. 311–318 (2002)Google Scholar
- 8.Vilar, D., Xu, J., D’haro, L.F., Ney, H.: Error analysis of statistical machine translation output. Language Resources and Evaluation, Genoa, Italy, pp. 697–702 (2006)Google Scholar
- 9.Nießen, S., Och, F.J., Leusch, G., Ney, H.: An evaluation tool for machine translation: fast evaluation for MT research. In: Proceedings of the Second International Conference on Language Resources and Evaluation, Athens, Greece, pp. 39–45 (2000)Google Scholar
- 10.Tillmann, C., Vogel, S., Ney, H., Zubiaga, A., Sawaf, H.: Accelerated DP based search for statistical translation. In: Fifth European Conference on Speech Communication and Technology, Rhodes, Greece, pp. 2667–2670 (1997)Google Scholar
- 11.Leusch, G., Ueffing, N., Ney, H.: CDER: efficient MT evaluation using block movements. In: 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, pp. 241–248 (2006)Google Scholar
- 12.Munk, M., Munkova, D., Benko, L.: Identification of relevant and redundant automatic metrics for MT evaluation. In: Sombattheera, C., Stolzenburg, F., Lin, L., Nayak, A. (eds.) MIWAI 2016. Lecture Notes in Artificial Intelligence, vol. 10053, pp. 141–152. Springer, Cham (2016)Google Scholar
- 14.Seljan, S., Brkić, M., Kučiš, V.: Evaluation of free online machine translations for Croatian-English and English-Croatian language pairs. In: Proceedings of the 3rd International Conference on the Future of Information Sciences, Zagreb, pp. 331–345 (2011)Google Scholar
- 15.Brandt, D.: Developing an Icelandic to English shallow transfer machine translation system. Reykjavík University, Reykjavík (2011)Google Scholar
- 16.Babych, B., Hartley, A., Sharoff, S.: Translating from under-resourced languages: comparing direct transfer against pivot translation. In: Proceedings of the MT Summit XI. Citeseer, Copenhagen (2007)Google Scholar
- 17.Adly, N., Al Ansary, S.: Natural Language Processing and Information Systems. Springer, Heidelberg (2010)Google Scholar