Automatic Translation Error Analysis

  • Mark Fishel
  • Ondřej Bojar
  • Daniel Zeman
  • Jan Berka
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6836)

Abstract

We propose a method of automatic identification of various error types in machine translation output. The approach is mostly based on monolingual word alignment of the hypothesis and the reference translation. In addition to common lexical errors misplaced words are also detected. A comparison to manually classified MT errors is presented. Our error classification is inspired by that of Vilar (2006; [17]), although distinguishing some of their categories is beyond the reach of the current version of our system.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Berka, J., Černý, M., Bojar, O.: Quiz-Based Evaluation of Machine Translation. Prague Bulletin of Mathematical Linguistics 95 (2011)Google Scholar
  2. 2.
    Birch, A., Osborne, M., Blunsom, P.: Metrics for mt evaluation: evaluating reordering. Machine Translation 24(1), 15–26 (2010)CrossRefGoogle Scholar
  3. 3.
    Bojar, O., Prokopová, M.: Czech-English Word Alignment. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), pp. 1236–1239. ELRA (May 2006)Google Scholar
  4. 4.
    Bojar, O.: Analyzing Error Types in English-Czech Machine Translation. Prague Bulletin of Mathematical Linguistics 95 (2011)Google Scholar
  5. 5.
    Callison-Burch, C., Koehn, P., Monz, C., Peterson, K., Przybocki, M., Zaidan, O.: Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In: Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR, pp. 17–53. Association for Computational Linguistics, Uppsala (July 2010), http://www.aclweb.org/anthology/W10-1703 (revised August 2010)Google Scholar
  6. 6.
    Callison-Burch, C., Koehn, P., Monz, C., Schroeder, J.: Findings of the 2009 Workshop on Statistical Machine Translation. In: Proceedings of the Fourth Workshop on Statistical Machine Translation, pp. 1–28. Association for Computational Linguistics, Athens (2009), http://www.aclweb.org/anthology/W/W09/W09-0401 CrossRefGoogle Scholar
  7. 7.
    DeNero, J., Klein, D.: Discriminative modeling of extraction sets for machine translation. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1453–1463. Association for Computational Linguistics, Uppsala (July 2010), http://www.aclweb.org/anthology/P10-1147 Google Scholar
  8. 8.
    Denkowski, M., Lavie, A.: Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 250–253 (2010)Google Scholar
  9. 9.
    Farrús, M., Costa-jussà, M.R., Mariño, J.B., Poch, M., Hernández, A., Henriquez, C., Fonollosa, J.A.R.: Overcoming statistical machine translation limitations: error analysis. In: Language Resources and Evaluation, pp. 1–28 (February 2011)Google Scholar
  10. 10.
    Giménez, J., Màrquez, L.: Towards heterogeneous automatic mt error analysis. In: ELRA, E.L.R.A. (ed.) Proceedings of the Sixth International Language Resources and Evaluation (LREC 2008), Marrakech, Morocco (May 2008)Google Scholar
  11. 11.
    He, X., Yang, M., Gao, J., Nguyen, P., Moore, R.: Indirect-hmm-based hypothesis alignment for combining outputs from machine translation systems. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, pp. 98–107. Association for Computational Linguistics, Stroudsburg (2008), http://portal.acm.org/citation.cfm?id=1613715.1613730 CrossRefGoogle Scholar
  12. 12.
    Matusov, E., Leusch, G., Banchs, R.E., Bertoldi, N., Dechelotte, D., Federico, M., Kolss, M., Lee, Y.S., Marino, J.B., Paulik, M., Roukos, S., Schwenk, H., Ney, H.: System Combination for Machine Translation of Spoken and Written Language. IEEE Transactions on Audio, Speech and Language Processing 16(7), 1222–1237 (2008)CrossRefGoogle Scholar
  13. 13.
    Popel, M., Mareček, D.: unpublished (2010), http://code.google.com/p/failfinder/
  14. 14.
    Popovic, M., de Gispert, A., Gupta, D., Lambert, P., Ney, H., Mariño, J.B., Federico, M., Banchs, R.: Morpho-syntactic information for automatic error analysis of statistical machine translation output. In: Proceedings on the Workshop on Statistical Machine Translation, pp. 1–6, New York, USA (2006)Google Scholar
  15. 15.
    Popović, M., Ney, H.: Word error rates: decomposition over pos classes and applications for error analysis. In: Proceedings of the Second Workshop on Statistical Machine Translation, StatMT 2007, pp. 48–55. Association for Computational Linguistics, Stroudsburg (2007), http://portal.acm.org/citation.cfm?id=1626355.1626362 CrossRefGoogle Scholar
  16. 16.
    Tiedemann, J.: Word to word alignment strategies. In: Proceedings of COLING 2004, pp. 212–218, Geneva, Switzerland (2004)Google Scholar
  17. 17.
    Vilar, D., Xu, J., D’Haro, L.F., Ney, H.: Error analysis of machine translation output. In: Proceedings of the 5th LREC, pp. 697–702, Genoa, Italy (2006)Google Scholar
  18. 18.
    Vogel, S., Ney, H., Tillmann, C.: HMM-based word alignment in statistical translation. In: International Conference on Computational Linguistics, pp. 836–841, Kopenhagen, Denmark (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Mark Fishel
    • 1
  • Ondřej Bojar
    • 2
  • Daniel Zeman
    • 2
  • Jan Berka
    • 2
  1. 1.Department of Computer ScienceUniversity of TartuEstonia
  2. 2.Institute of Formal and Applied Linguistics Faculty of Mathematics and PhysicsCharles UniversityPragueCzechia

Personalised recommendations