Machine Translation

, Volume 27, Issue 3–4, pp 193–212 | Cite as

Investigating the contribution of linguistic information to quality estimation

Article

Abstract

This paper describes a study on the contribution of linguistically-informed features to the task of quality estimation for machine translation at sentence level. A standard regression algorithm is used to build models using a combination of linguistic and non-linguistic features extracted from the input text and its machine translation. Experiments with three English–Spanish translation datasets show that linguistic features on their own are not able to outperform shallower features based on statistics from the input text, its translation and additional corpora. However, further analysis suggests that linguistic information can be useful to produce better results if carefully combined with other features. An in-depth analysis of the results highlights a number of issues related to the use of linguistic features.

Keywords

Machine translation Evaluation Quality estimation 

References

  1. Alpaydin E (2010) Introduction to machine learning (adaptive computation and machine learning), 2nd edn. MIT Press, CambridgeGoogle Scholar
  2. Avramidis E (2012) Quality estimation for machine translation output using linguistic analysis and decoding features. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 84–90Google Scholar
  3. Bach N, Huang F, Al-Onaizan Y (2011) Goodness: a method for measuring machine translation confidence. In: Proceedings of the 49th annual meeting of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Portland, Oregon, USA, pp 211–219Google Scholar
  4. Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, Sanchis A, Ueffing N (2004) Confidence estimation for machine translation. Final report of Johns Hopkins (2003) summer workshop on speech and language engineering. Johns Hopkins University, Baltimore, Maryland, USAGoogle Scholar
  5. Buck C (2012) Black box features for the wmt 2012 quality estimation shared task. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 91–95Google Scholar
  6. Callison-Burch C, Koehn P, Monz C, Post M, Soricut R, Specia L (2012) Findings of the 2012 workshop on statistical machine translation. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 10–51Google Scholar
  7. Carroll JB (1964) Language and thought. Prentice-Hall, Englewood CliffsGoogle Scholar
  8. Chang CC, Lin CJ (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):1–27CrossRefGoogle Scholar
  9. Dugast D (1980) La statistique lexicale. Slatkine, GenevaGoogle Scholar
  10. Felice M, Specia L (2012) Linguistic features for quality estimation. In: Proceedings of the seventh workshop on statistical machine translation. Association for Computational Linguistics, Montréal, Canada, pp 96–103Google Scholar
  11. Giménez J, Màrquez L (2010) Linguistic measures for automatic machine translation evaluation. Mach Transl 24(3):209–240CrossRefGoogle Scholar
  12. Guiraud P (1954) Les Caractères Statistiques du Vocabulaire. Presses Universitaires de France, ParisGoogle Scholar
  13. Halliday MAK, Hasan R (1976) Cohesion in english. Longman, LondonGoogle Scholar
  14. Hardmeier C (2011) Improving machine translation quality prediction with syntactic tree kernels. In: Proceedings of the 15th conference of the European Association for Machine Translation (EAMT 2011), Leuven, Belgium, pp 233–240Google Scholar
  15. Hardmeier C, Nivre J, Tiedemann J (2012) Tree kernels for machine translation quality estimation. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 109–113Google Scholar
  16. Herdan G (1960) Type-token mathematics: a textbook of mathematical linguistics. Mouton & Co., The HagueMATHGoogle Scholar
  17. Jarvis S (2002) Short texts, best-fitting curves and new measures of lexical diversity. Lang Test 19(1): 57–84Google Scholar
  18. Padró L, Collado M, Reese S, Lloberes M, Castellón I (2010) Freeling 2.1: five years of open-source language processing tools. In: Chair NCC, Choukri K, Maegaard B, Mariani J, Odijk J, Piperidis S, Rosner M, Tapias D (eds) Proceedings of the seventh conference on international language resources and evaluation (LREC’10). European Language Resources Association (ELRA), Valletta, MaltaGoogle Scholar
  19. Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics, pp 311–318Google Scholar
  20. Pighin D, Màrquez L (2011) Automatic projection of semantic structures: an application to pairwise translation ranking. In: Fifth workshop on syntax, semantics and structure in statistical translation (SSST-5), Portland, OregonGoogle Scholar
  21. Pighin D, González M, Màrquez L (2012) The UPC submission to the WMT 2012 shared task on quality estimation. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 127–132Google Scholar
  22. Popovic M (2012) Morpheme- and pos-based ibm1 and language model scores for translation quality estimation. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 133–137Google Scholar
  23. Quirk CB (2004) Training a sentence-level machine translation confidence metric. In: Proceedings of the international conference on language resources and evaluation, Lisbon, Portugal, LREC 2004, vol 4, pp 825–828Google Scholar
  24. Rubino R, Foster J, Wagner J, Roturier J, Samad Zadeh Kaljahi R, Hollowood F (2012) Dcu-symantec submission for the wmt 2012 quality estimation task. In: Proceedings of the seventh workshop on statistical machine translation, Montréal, Canada, pp 138–144Google Scholar
  25. Schmid H (1995) Improvements in part-of-speech tagging with an application to german. In: Proceedings of the ACL SIGDAT-Workshop, Dublin, Ireland, pp 47–50Google Scholar
  26. Soricut R, Bach N, Wang Z (2012) The SDL language weaver systems in the WMT12 quality estimation shared task. In: Proceedings of the seventh workshop on statistical machine translation. Association for Computational Linguistics, Montréal, Canada, pp 145–151Google Scholar
  27. Specia L (2011) Exploiting objective annotations for measuring translation post-editing effort. In: Proceedings of the 15th conference of the European Association for Machine Translation, Leuven, pp 73–80Google Scholar
  28. Specia L, Turchi M, Cancedda N, Dymetman M, Cristianini N (2009a) Estimating the sentence-level quality of machine translation systems. In: Proceedings of the 13th annual conference of the European Association for Machine Translation (EAMT), Barcelona, Spain, pp 28–35Google Scholar
  29. Specia L, Turchi M, Wang Z, Shawe-Taylor J, Saunders C (2009b) Improving the confidence of machine translation quality estimates. In: Proceedings of the twelfth machine translation summit (MT Summit XII), Ottawa, Canada, pp 136–143Google Scholar
  30. Specia L, Hajlaoui N, Hallett C, Aziz W (2011) Predicting machine translation adequacy. In: Machine translation summit XIII, Xiamen, China, pp 19–23Google Scholar
  31. Stolcke A (2002) Srilm-an extensible language modeling toolkit. In: Proceedings of the 7th international conference on spoken language processing (ICSLP 2002), Denver, USA, vol 2, pp 901–904Google Scholar
  32. Taulé M, Martí MA, Recasens M (2008) Ancora: multilevel annotated corpora for catalan and spanish. In: Chair NCC, Choukri K, Maegaard B, Mariani J, Odijk J, Piperidis S, Rosner M, Tapias D (eds) Proceedings of the sixth international conference on language resources and evaluation (LREC’08). European Language Resources Association (ELRA), Marrakech, MoroccoGoogle Scholar
  33. Vilar D, Xu J, D’Haro LF, Ney H (2006) Error analysis of machine translation output. In: International conference on language resources and evaluation, Genoa, Italy, pp 697–702Google Scholar
  34. Xiong D, Zhang M, Li H (2010) Error detection for statistical machine translation using linguistic features. In: Proceedings of the 48th annual meeting of the association for computational linguistics, Uppsala, Sweden, pp 604–611Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  1. 1.Computer LaboratoryUniversity of CambridgeCambridgeUK
  2. 2.Department of Computer ScienceUniversity of SheffieldSheffieldUK

Personalised recommendations