Advertisement

Domain Adaptation Techniques for Machine Translation and Their Evaluation in a Real-World Setting

  • Baskaran Sankaran
  • Majid Razmara
  • Atefeh Farzindar
  • Wael Khreich
  • Fred Popowich
  • Anoop Sarkar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7310)

Abstract

Statistical Machine Translation (SMT) is currently used in real-time and commercial settings to quickly produce initial translations for a document which can later be edited by a human. The SMT models specialized for one domain often perform poorly when applied to other domains. The typical assumption that both training and testing data are drawn from the same distribution no longer applies. This paper evaluates domain adaptation techniques for SMT systems in the context of end-user feedback in a real world application. We present our experiments using two adaptive techniques, one relying on log-linear models and the other using mixture models. We describe our experimental results on legal and government data, and present the human evaluation effort for post-editing in addition to traditional automated scoring techniques (BLEU scores). The human effort is based primarily on the amount of time and number of edits required by a professional post-editor to improve the quality of machine-generated translations to meet industry standards. The experimental results in this paper show that the domain adaptation techniques can yield a significant increase in BLEU score (up to four points) and a significant reduction in post-editing time of about one second per word.

Keywords

Statistical Machine Translation Machine Learning Domain Adaptation Model Adaptation Human Evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aziz, W., de Sousa, S.C.M., Specia, L.: PET: a tool for post-editing and assessing machine translation. In: The 8th International Conference on Language Resources and Evaluation (LREC 2012), Instanbul, Turkey (2012)Google Scholar
  2. 2.
    Bertoldi, N., Federico, M.: Domain adaptation for statistical machine translation with monolingual resources. In: Proceedings of the 4th Workshop on Statistical Machine Translation, StatMT 2009, pp. 182–189. Association for Computational Linguistics, USA (2009)CrossRefGoogle Scholar
  3. 3.
    Chiang, D., Marton, Y., Resnik, P.: Online large-margin training of syntactic and structural translation features. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. ACL (2008)Google Scholar
  4. 4.
    Civera, J., Juan, A.: Domain adaptation in statistical machine translation with mixture modelling. In: Proceedings of the Second Workshop on Statistical Machine Translation, StatMT 2007, pp. 177–180. Association for Computational Linguistics, Stroudsburg (2007)CrossRefGoogle Scholar
  5. 5.
    Daumé III, H.: Frustratingly easy domain adaptation. In: Conference of the Association for Computational Linguistics. ACL, Prague (2007)Google Scholar
  6. 6.
    DeNero, J., Kumar, S., Chelba, C., Och, F.: Model combination for machine translation. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT 2010, pp. 975–983. Association for Computational Linguistics, Stroudsburg (2010)Google Scholar
  7. 7.
    Farzindar, A., Lapalme, G.: Machine Translation of Legal Information and Its Evaluation. In: Gao, Y., Japkowicz, N. (eds.) AI 2009. LNCS, vol. 5549, pp. 64–73. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  8. 8.
    Foster, G., Kuhn, R.: Mixture-model adaptation for SMT. In: Proceedings of the Second Workshop on Statistical Machine Translation. ACL (2007)Google Scholar
  9. 9.
    Gotti, F., Farzindar, A., Lapalme, G., Macklovitch, E.: Automatic translation of court judgments. In: AMTA 2008 The Eighth Conference of the Association for Machine Translation in the Americas, pp. 1–10. Waikiki, Hawai’i (2008)Google Scholar
  10. 10.
    Hildebrand, A.S., Vogel, S.: CMU system combination for WMT 2009. In: Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT 2009, pp. 47–50. Association for Computational Linguistics, Stroudsburg (2009)CrossRefGoogle Scholar
  11. 11.
    Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Proceedings of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pp. 48–54. Association for Computational Linguistics (2003)Google Scholar
  12. 12.
    Koehn, P., Schroeder, J.: Experiments in domain adaptation for statistical machine translation. In: Proceedings of the Second Workshop on Statistical Machine Translation, StatMT 2007, pp. 224–227. Association for Computational Linguistics, Stroudsburg (2007)CrossRefGoogle Scholar
  13. 13.
    Och, F.J.: Minimum error rate training in statistical machine translation. In: Proceedings of the Association of Computational Linguistics. ACL (2003)Google Scholar
  14. 14.
    Och, F.J., Ney, H.: Discriminative training and maximum entropy models for statistical machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL 2002, pp. 295–302. ACL, Stroudsburg (2002)Google Scholar
  15. 15.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the Association for Computational Linguistics, pp. 311–318. ACL (2002)Google Scholar
  16. 16.
    Razmara, M., Foster, G., Sankaran, B., Sarkar, A.: Mixing multiple translation models in statistical machine translation. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Jeju (2012)Google Scholar
  17. 17.
    Sadat, F., Johnson, H., Agbago, A., Foster, G., Martin, J., Tikuisis, A.: Portage: A phrase-based machine translation system. In: Proceedings of the ACL Worskhop on Building and Using Parallel Texts. ACL (2005)Google Scholar
  18. 18.
    Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: Proceedings of Association for Machine Translation in the Americas, pp. 223–231 (2006)Google Scholar
  19. 19.
    Specia, L., Farzindar, A.: Estimating machine translation post-editing effort with HTER. In: AMTA 2010 Workshop, Bringing MT to the User: MT Research and the Translation Industry. The 9th Conference of the Association for Machine Translation in the Americas (2010)Google Scholar
  20. 20.
    Ueffing, N., Haffari, G., Sarkar, A.: Semi-supervised model adaptation for statistical machine translation. Machine Translation 21(2), 77–94 (2007)CrossRefGoogle Scholar
  21. 21.
    Wu, H., Wang, H., Zong, C.: Domain adaptation for statistical machine translation with domain dictionary and monolingual corpora. In: Proceedings of the 22nd International Conference on Computational Linguistics, COLING 2008, USA, pp. 993–1000 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Baskaran Sankaran
    • 1
  • Majid Razmara
    • 1
  • Atefeh Farzindar
    • 2
  • Wael Khreich
    • 2
  • Fred Popowich
    • 1
  • Anoop Sarkar
    • 1
  1. 1.School of Computing ScienceSimon Fraser UniversityBurnabyCanada
  2. 2.NLP Technologies Inc.MontrealCanada

Personalised recommendations