Automatic Evaluation of MT Output and Post-edited MT Output for Genealogically Related Languages

  • Daša Munková
  • Michal MunkEmail author
  • Ján Skalka
  • Karol Kasaš
Conference paper
Part of the Learning and Analytics in Intelligent Systems book series (LAIS, volume 7)


The aim of the research is twofold: to evaluate the translation quality of the individual sentences of the MT output and also post-edited MT output on the basis of metrics of automatic MT evaluation from Slovak into the German language; and to compare the quality of MT output and post-edited MT output based on the same automatic metrics of MT evaluation. The icon graphs were used to visualize the results for individual sentences. A significant difference was found in sentence 36 in favor of the post-edited MT output and vice versa in sentence 5 in favor of MT output. Due to the error rate, a significant difference was in sentence 29 and 11 in favor of post-edited MT output and vice versa the sentence 26 in favor of MT output. Based on our results we can state that it is necessary to include into the evaluation of the quality of translation all automatic metrics for each sentence separately.


Language processing Machine translation Automatic MT metrics Genealogically related languages 



This work was supported by the SRD Agency under the contract No. APVV-18-0473 and Scientific Grant Agency of the ME SR and SAS under the contracts No. VEGA-1/0809/18.

This publication was supported by the Operational Program: Research and Innovation project “Fake news on the Internet - identification, content analysis, emotions”, co-funded by the European Regional Development Fund.


  1. 1.
    Munkova, D., Munk, M., Benko, L., Absolon, J.: From old fashioned “one size fits all” to tailor made online training. In: Auer, M.E., Tsiatsos, T. (eds.) ICL 2018. Advances in Intelligent Systems and Computing, vol. 916, pp. 365–376. Springer, Cham (2019)Google Scholar
  2. 2.
    Chéragui, M.A.: Theoretical overview of machine translation. In: Proceedings of ICWIT 2012, pp. 160–169. CEUR-WS, Sidi Bel Abbes, Algeria (2012)Google Scholar
  3. 3.
    Wallis, J.: Interactive translation vs. pre-translation in the context of translation memory systems: investigating the effects of translation method on productivity, quality and translator satisfaction. University of Ottawa, Ottawa (2006)Google Scholar
  4. 4.
    Munkova, D., Munk, M.: Evalvácia strojového prekladu. Univerzita Konštantína Filozofa v Nitre, Nitra (2016)Google Scholar
  5. 5.
    DePalma, D.A.: How to add post-edited MT to your service offerings. Common Sense Advisory, Cambridge, MA (2013)Google Scholar
  6. 6.
    Van Slype, G.: Critical study of methods for evaluating the quality of machine translation. Technical report, Bureau Marcel van Dijk/European Commission, Brussels (1979Google Scholar
  7. 7.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, Pennsylvania, pp. 311–318 (2002)Google Scholar
  8. 8.
    Vilar, D., Xu, J., D’haro, L.F., Ney, H.: Error analysis of statistical machine translation output. Language Resources and Evaluation, Genoa, Italy, pp. 697–702 (2006)Google Scholar
  9. 9.
    Nießen, S., Och, F.J., Leusch, G., Ney, H.: An evaluation tool for machine translation: fast evaluation for MT research. In: Proceedings of the Second International Conference on Language Resources and Evaluation, Athens, Greece, pp. 39–45 (2000)Google Scholar
  10. 10.
    Tillmann, C., Vogel, S., Ney, H., Zubiaga, A., Sawaf, H.: Accelerated DP based search for statistical translation. In: Fifth European Conference on Speech Communication and Technology, Rhodes, Greece, pp. 2667–2670 (1997)Google Scholar
  11. 11.
    Leusch, G., Ueffing, N., Ney, H.: CDER: efficient MT evaluation using block movements. In: 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, pp. 241–248 (2006)Google Scholar
  12. 12.
    Munk, M., Munkova, D., Benko, L.: Identification of relevant and redundant automatic metrics for MT evaluation. In: Sombattheera, C., Stolzenburg, F., Lin, L., Nayak, A. (eds.) MIWAI 2016. Lecture Notes in Artificial Intelligence, vol. 10053, pp. 141–152. Springer, Cham (2016)Google Scholar
  13. 13.
    Munk, M., Munkova, D.: Detecting errors in machine translation using residuals and metrics of automatic evaluation. J. Intell. Fuzzy Syst. 34(5), 3211–3223 (2018)CrossRefGoogle Scholar
  14. 14.
    Seljan, S., Brkić, M., Kučiš, V.: Evaluation of free online machine translations for Croatian-English and English-Croatian language pairs. In: Proceedings of the 3rd International Conference on the Future of Information Sciences, Zagreb, pp. 331–345 (2011)Google Scholar
  15. 15.
    Brandt, D.: Developing an Icelandic to English shallow transfer machine translation system. Reykjavík University, Reykjavík (2011)Google Scholar
  16. 16.
    Babych, B., Hartley, A., Sharoff, S.: Translating from under-resourced languages: comparing direct transfer against pivot translation. In: Proceedings of the MT Summit XI. Citeseer, Copenhagen (2007)Google Scholar
  17. 17.
    Adly, N., Al Ansary, S.: Natural Language Processing and Information Systems. Springer, Heidelberg (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Daša Munková
    • 1
  • Michal Munk
    • 1
    Email author
  • Ján Skalka
    • 1
  • Karol Kasaš
    • 2
  1. 1.Constantine the Philosopher University in NitraNitraSlovakia
  2. 2.University of PardubicePardubiceCzech Republic

Personalised recommendations