Machine Translation

, Volume 29, Issue 1, pp 1–24 | Cite as

Automatic machine translation error identification

  • Débora Beatriz de Jesus Martins
  • Helena de Medeiros Caseli


Although machine translation (MT) has been an object of study for decades now, the texts generated by the state-of-the-art MT systems still present several errors for many language pairs. Aiming at coping with this drawback, lots of efforts have been made to post-edit those errors either manually or automatically. Manual post-editing is more accurate but can be prohibitive when too many changes have to be made. Automatic post-editing demands less effort but can also be less effective and give rise to new errors. A way to avoid unnecessary automatic post-editing and new errors is by previously selecting only the machine-translated segments that really need to be post-edited. Thus, this paper describes the experiments carried out to automatically identify MT errors generated by a state-of-the-art phrase-based statistical MT system. Despite the fact that our experiments have been carried out using a statistical MT engine, we believe the approach can also be applied to other types of MT systems. The experiments investigated the well-known machine-learning algorithms Naive Bayes, Decision Trees and Support Vector Machines. Using the decision tree algorithm it was possible to identify wrong segments with around 77 % precision and recall when a small training corpus of only 2,147 error instances was used. Our experiments were performed on English-to-Brazilian Portuguese MT, and although some of the features are language-dependent, the proposed approach is language-independent and can be easily generalized to other language pairs.


Automatic error identification Automatic post-edition  Machine translation Machine learning 



This project was developed with support of the Grants #2011/03799-4, #2010/07517-0 and #2013/11811-0 from the São Paulo Research Foundation (FAPESP). We also thank Maria das Graças Volpe Nunes and Lucas Vinicius Avanço for their help in the corpus annotation process. This work is also part of the CAMELEON (CAPES-COFECUB #707-11) and AIM-WEST (FAPESP #2013/50757-0) projects.


  1. Allen J (2003) In: Somers H (ed) Computers and translation: a translator’s guide. John Benjamins Publishing Company, Amsterdam, pp 297–317Google Scholar
  2. Allen J, Hogan C (2000) Toward the development of a post-editing module for raw machine translation output: a controlled language perspective. Proceedings of the third international workshop on controlled language applications. Seattle, pp 62–71Google Scholar
  3. Armentano-Oller C, Carrasco RC, Corbí-Bellot AM, Forcada ML, Ginestí-Rosell M, Ortiz-Rojas S, Pérez-Ortiz J, Ramírez-Sánchez G, Sánchez-Martínez F, Scalco MA (2006) Open-source Portuguese-Spanish machine translation. Proceedings of the 7th international workshop on computational processing of written and spoken Portuguese. Itatiaia, pp 50–59Google Scholar
  4. Aziz W, Specia L (2011) Fully automatic compilation of Portuguese–English and Portuguese–Spanish parallel corpora. Proceedings of the 8th Brazilian symposium in information and human language technology (STIL 2011). Cuiabá, pp 234–238Google Scholar
  5. Aziz W, de Sousa SCM, Specia L (2012) PET: a Tool for post-editing and assessing machine translation. Eighth international conference on language resources and evaluation (LREC 2012). Istanbul, pp 3982–3987Google Scholar
  6. Béchara H, Ma Y, van Genabith J (2011) Statistical post-editing for a statistical MT system. Proceedings of the thirteenth machine translation summit (MT Summit XIII). Xiamen, pp 308–315Google Scholar
  7. Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, Sanchis A, Ueffing N (2004) Confidence estimation for machine translation. Twentieth international conference on computational linguistics. Proceedings, Geneva, pp 315–321Google Scholar
  8. Bottou L, Lin CJ (2007) Support vector machine solvers. In: L. Bottou, O. Chapelle, D. DeCoste and J. Weston (eds.) Large scale kernel machines, MIT Press, Cambridge, pp 301–320,
  9. Caseli HM (2007) Indução de léxicos bilíngües e regras para a tradução automática. PhD thesis, USP, São Carlos, São PauloGoogle Scholar
  10. Caseli HM, Nunes M, Forcada M (2006) Automatic induction of bilingual resources from aligned parallel corpora: application to shallow-transfer machine translation. Mach Transl 20(4):227–245. doi: 10.1007/s10590-007-9027-9 CrossRefGoogle Scholar
  11. Doddington G (2002) Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. HLT 2002: human language technology conference: proceedings of the second international conference on human language technology research. San Diego, pp 138–145Google Scholar
  12. Elming J (2006) Transformation-based correction of rule-based MT. Eleventh annual conference of the European association for machine translation. Proceedings, Oslo, pp 219–226Google Scholar
  13. Felice M, Specia L (2012) Linguistic features for quality estimation. Proceedings of the 7th workshop on statistical machine translation. Montreal, pp 96–103Google Scholar
  14. Fishel M, Sennrich R, Popovic M, Bojar O (2012) TerrorCat. Proceedings of the 7th workshop on statistical machine translation. Montreal, pp 64–70Google Scholar
  15. Font Llitjós A (2007) Automatic improvement of machine translation systems. PhD thesis, Carnegie Mellon University, PittsburghGoogle Scholar
  16. George C, Japkowicz N (2005) Automatic correction of French to English relative pronoun translations using natural language processing and machine learning techniques. In: Computational linguistics In the North East (CLiNE’05), OttawaGoogle Scholar
  17. Gomes FT, Pardo TAS (2008) Trapezio–Translation post editor. In: Anais do Congresso da Academia Trinacional de Ciências (C3N), Foz do Iguaçu, Paraná, pp 1–10 taspardo/C3N2008-TassarioPardo
  18. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software. SIGKDD Explor 11:10–18CrossRefGoogle Scholar
  19. Hastie T, Tibshirani R (1998) Classification by pairwise coupling. Ann Stat 26(2):451–471CrossRefzbMATHMathSciNetGoogle Scholar
  20. John GH, Langley P (1995) Estimating continuous distributions in Bayesian classifiers. Eleventh conference on uncertainty in artificial intelligence. San Mateo, pp 338–345Google Scholar
  21. Kawamorita C, Caseli HM (2012) Memórias de Tradução: auxiliando o humano a traduzir, trabalho apresentado no Encontro de Linguística de Corpus (ELC 2012), São Carlos, São Paulo, p 10.
  22. Keerthi S, Shevade S, Bhattacharyya C, Murthy K (2001) Improvements to Platt’s SMO algorithm for SVM classifier design. Neural Comput 13(3):637–649CrossRefzbMATHGoogle Scholar
  23. Knight K, Chander I (1994) Automated post-editing of documents. Proceedings of the 12th national conference on artificial intelligence. Seattle, pp 779–784Google Scholar
  24. Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R, Dyer C, Bojar O, Constantin A, Herbst E (2007) Moses: open source toolkit for statistical machine translation. Proceedings of the ACL 2007 demo and poster sessions. Czech Republic, Prague, pp 177–180Google Scholar
  25. Krings HP (2001) Repairing texts—empirical investigations of machine translation post-editing processes. The Kent State University Press, KentGoogle Scholar
  26. Lagarda AL, Alabau V, Casacuberta F, Silva R, DíAZ-DE-LIAO E (2009) Statistical post-editing of a rule-based machine translation system. In: human language technologies: the 2009 annual conference of the North American chapter of the association for computational linguistics, Proceedings of the conference, Boulder, pp 217–220Google Scholar
  27. Levenshtein VI (1966) Binary codes capable of correcting deletions. Insertions and reversals. Sov Phys Dokl 10(8):707–710MathSciNetGoogle Scholar
  28. Martins DBJ (2014) Pós-edição automática de textos traduzidos automaticamente de inglês para português do Brasil. Master’s thesis, Centro de Ciências Exatas e de Tecnologia - Programa de Pós-graduação em Ciência da Computação, Universidade Federal de São Carlos, São Paulo
  29. Martins DBJ, Caseli HM (2013) Anotação manual de erros de tradução automática em textos traduzidos de inglês para português do Brasil. Tech. Rep. NILC-TR-13-02, Série de Relatórios do NILC, Brazil
  30. Martins DBJ, Avanço LV, Nunes MGV, Caseli HM (2013) In: Hardie A, Love R (eds) Corpus linguistics. Lancaster, pp 189–192Google Scholar
  31. Nießen S, Och FJ, Leusch G, Ney H (2000) An evaluation tool for machine translation. Proceedings of the second international conference on language resources and evaluation (LREC). Athens, pp 39–45Google Scholar
  32. O’Brien S (2002) Teaching post-editing. Sixth EAMT workshop “Teaching machine translation”. Manchester, England, pp 99–106Google Scholar
  33. Och FJ, Ney H (2003) A systematic comparison of various statistical alignment models. Comput Linguist 29(1):19–51CrossRefzbMATHGoogle Scholar
  34. Och FJ, Ney H (2004) The alignment template approach to statistical machine translation. Comput Linguist 30(4):417–449CrossRefzbMATHGoogle Scholar
  35. Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. In: ACL-2002: 40th annual meeting of the association for computational linguistics, Philadelphia, pp 311–318Google Scholar
  36. Platt J (1998) Fast training of support vector machines using sequential minimal optimization. In: Schoelkopf B, Burges C, Smola A (eds) Advances in kernel methods—support vector learning. MIT Press, CambridgeGoogle Scholar
  37. Popovic M (2011) Hjerson. Prague Bull Math Linguist 96:59–67CrossRefGoogle Scholar
  38. Popovic M, Burchardt A (2011) From human to automatic error classification for machine translation output. EAMT 2011, proceedings of the 15th conference of the European association for machine translation. Belgium, pp 265–272Google Scholar
  39. Potet M, Esperança-Rodier E, Blanchon H, Besacier L (2011) Preliminary experiments on using users’ post-editions to enhance a SMT system. EAMT 2011, proceedings of the 15th conference of the European association for machine translation. Belgium, pp 161–168Google Scholar
  40. Quinlan JR (1993) C4.5. Morgan Kaufmann Publishers, San MateoGoogle Scholar
  41. Roturier J (2009) Deploying novel MT technology to raise the bar for quality: A review of key advantages and challenges. MT summit XII: proceedings of the twelfth machine translation summit. Ottawa, pp 1–8Google Scholar
  42. Sag IA, Baldwin T, Bond F, Copestake A, Flickinger D (2002) Multiword expressions: A pain in the neck for NLP. In: Proceedings of the third international conference on computational linguistics and intelligent text processing (CICLing-2002), Springer, London (lecture notes in computer science), vol 2276, pp 1–15Google Scholar
  43. Shah K, Cohn T, Specia L (2013) An investigation on the effectiveness of features for translation quality estimation. Proceedings of the XIV machine translation summit. Nice, pp 167–174Google Scholar
  44. Simard M, Goutte C, Isabelle P (2007) Statistical phrase-based post-editing. In: Human language Technologies 2007: the conference of the North American chapter of the association for computational linguistics, Proceedings of the main conference, Rochester, pp 508–515Google Scholar
  45. Specia L (2011) Exploiting objective annotations for measuring translation post-editing effort. In: EAMT 2011, proceedings of the 15th conference of the European association for machine translation, Leuven, pp 73–80Google Scholar
  46. Stymne S (2011) BLAST: A Tool for error analysis of machine translation output. Proceedings of the ACLHLT 2011 system demonstrations. Portland, pp 56–61Google Scholar
  47. Tillmann C, Vogel S, Ney H, Zubiaga A, Sawaf H (1997) Accelerated DP based search for statistical translation. European conference on speech communication and technology. Rhodes, pp 2667–2670Google Scholar
  48. Vieira TL, Caseli HM (2011) PorTAl: Recursos e Ferramentas de Tradução Automática para o Português do Brasil. In: Eighth Brazilian symposium in information and human language technology (STIL), Cuiabá, pp 179–183Google Scholar
  49. Vilar D, Xu J, D’Haro LF, Ney H (2006) Error analysis of statistical machine translation output. LREC-2006: fifth international conference on language resources and evaluation. Proceedings, Genoa, pp 22–28Google Scholar
  50. Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Springer, BerlinGoogle Scholar
  51. Zeman D, Fishel M, Berka J, Bojar O (2011) Addicter: what is wrong with my translations? Prague Bull Math Linguist 96:79–88CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Débora Beatriz de Jesus Martins
    • 1
  • Helena de Medeiros Caseli
    • 1
  1. 1.Federal University of São CarlosSão CarlosBrazil

Personalised recommendations