Abstract
We develop a top performing model for automatic, accurate, and language independent prediction of sentence-level statistical machine translation (SMT) quality with or without looking at the translation outputs. We derive various feature functions measuring the closeness of a given test sentence to the training data and the difficulty of translating the sentence. We describe mono feature functions that are based on statistics of only one side of the parallel training corpora and duo feature functions that incorporate statistics involving both source and target sides of the training data. Overall, we describe novel, language independent, and SMT system extrinsic features for predicting the SMT performance, which also rank high during feature ranking evaluations. We experiment with different learning settings, with or without looking at the translations, which help differentiate the contribution of different feature sets. We apply partial least squares and feature subset selection, both of which improve the results and we present ranking of the top features selected for each learning setting, providing an exhaustive analysis of the extrinsic features used. We show that by just looking at the test source sentences and not using the translation outputs at all, we can achieve better performance than a baseline system using SMT model dependent features that generated the translations. Furthermore, our prediction system is able to achieve the \(2\) nd best performance overall according to the official results of the quality estimation task (QET) challenge when also looking at the translation outputs. Our representation and features achieve the top performance in QET among the models using the SVR learning model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Version 1.0.0.2, available from http://www.seggu.net/ccl/.
\(P(m|l) = \frac{e^{-l r}(l r)^m}{m!}\) where \(r\) represents the ratio of the number of target words to the number of source words found in the training set.
Overview of statistical testing, especially for machine translation (Bicici 2011, App. B).
References
Albrecht JS, Hwa R (2008) Regression for machine translation evaluation at the sentence level. Mach Transl 22(1–2):1–27
Bicici E (2011) The regression model of machine translation. PhD thesis, Koç University.
Bicici E, Yuret D (2011) Instance selection for machine translation using feature decay algorithms. In: Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Edinburgh, pp 272–283
Birch A, Osborne M, Koehn P (2008) Predicting success in machine translation. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, pp 745–754
Blatz J, Fitzgerald E, Foster G, Gandrabur S, Goutte C, Kulesza A, Sanchis A, Ueffing N (2004) Confidence estimation for machine translation. In: Proceedings of COLING 2004, Geneva, pp 315–321
Brown PF, Pietra SAD, Pietra VJD, Mercer RL (1993) The mathematics of statistical machine translation: parameter estimation. Comput Linguist 19(2):263–311
Callison-Burch C, Koehn P, Monz C, Post M, Soricut R, Specia L (2012) Findings of the 2012 workshop on statistical machine translation. In: Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, Montréal, pp 10–51
Cover TM, Thomas JA (2006) Elements of information theory. Wiley-Interscience, New York
Drucker H, Burges CJC, Kaufman L, Smola AJ, Vapnik V (1996) Support vector regression machines. In: NIPS, pp 155–161
Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182
Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46(1–3):389–422
Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining inference and prediction, 2nd edn. Springer-Verlag, New York
He Y, Ma Y, van Genabith J, Way A (2010) Bridging SMT and TM with translation recommendation. In: Association for Computational Linguistics, Uppsala, pp 622–630
Knight K (1999) A statistical machine translation tutorial workbook. http://www.isi.edu/natural-language/mt/wkbk.rtf
Koehn P (2010) Statistical machine translation. Cambridge University Press, Cambridge
Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R, Dyer C, Bojar O, Constantin A, Herbst E (2007) Moses: open source toolkit for statistical machine translation. In: Annual Meeting of the Association for Computational Linguistics, Prague, pp 177–180
Moore RC (2002) Fast and accurate sentence alignment of bilingual corpora. In: AMTA ’02: Proceedings of the 5th Conference of the Association for Machine Translation in the Americas on Machine Translation: From Research to Real Users, Springer-Verlag, London, pp 135–144
Och FJ, Ney H (2003) A systematic comparison of various statistical alignment models. Comput Linguist 29(1):19–51
Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, pp 311–318
Rosipal R, Trejo LJ (2001) Kernel partial least squares regression in reproducing kernel hilbert space. J Mach Learn Res 2:97–123
Sarle WS (2002) Faq: should i normalize/standardize/rescale the data? http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-16.html
Seginer Y (2007) Learning syntactic structure. PhD thesis, Universiteit van Amsterdam.
Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14(3):199–222
Soricut R, Bach N, Wang Z (2012) The SDL language weaver systems in the wmt12 quality estimation shared task. In: Proceedings of the Seventh Workshop on Statistical Machine Translation. Association for Computational Linguistics, Montréal, pp 145–151
Specia L, Cancedda N, Dymetman M, Turchi M, Cristianini N (2009) Estimating the sentence-level quality of machine translation systems. In: Proceedings of the 13th Annual Conference of the European Association for Machine Translation (EAMT), Barcelona, pp 28–35
Specia L, Raj D, Turchi M (2010) Machine translation evaluation versus quality estimation. Mach Transl 24(1):39–50
Wasserman L (2004) All of statistics: a concise course in statistical inference. Springer, New York
Acknowledgments
This work was supported in part by SFI (07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at Dublin City University and in part by the European Commission through the QTLaunchPad FP7 project (No: 296347). The authors wish to acknowledge the SFI/HEA Irish Centre for High-End Computing (ICHEC) for the provision of computational facilities and support. We also thank the reviewers for their constructive comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Biçici, E., Groves, D. & van Genabith, J. Predicting sentence translation quality using extrinsic and language independent features. Machine Translation 27, 171–192 (2013). https://doi.org/10.1007/s10590-013-9138-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10590-013-9138-4