Iterative Rule Segmentation under Minimum Description Length for Unsupervised Transduction Grammar Induction

  • Markus Saers
  • Karteek Addanki
  • Dekai Wu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7978)


We argue that for purely incremental unsupervised learning of phrasal inversion transduction grammars, a minimum description length driven, iterative top-down rule segmentation approach that is the polar opposite of Saers, Addanki, and Wu’s previous 2012 bottom-up iterative rule chunking model yields significantly better translation accuracy and grammar parsimony. We still aim for unsupervised bilingual grammar induction such that training and testing are optimized upon the same exact underlying model—a basic principle of machine learning and statistical prediction that has become unduly ignored in statistical machine translation models of late, where most decoders are badly mismatched to the training assumptions. Our novel approach learns phrasal translations by recursively subsegmenting the training corpus, as opposed to our previous model—where we start with a token-based transduction grammar and iteratively build larger chunks. Moreover, the rule segmentation decisions in our approach are driven by a minimum description length objective, whereas the rule chunking decisions were driven by a maximum likelihood objective. We demonstrate empirically how this trades off maximum likelihood against model size, aiming for a more parsimonious grammar that escapes the perfect overfitting to the training data that we start out with, and gradually generalizes to previously unseen sentence translations so long as the model shrinks enough to warrant a looser fit to the training data. Experimental results show that our approach produces a significantly smaller and better model than the chunking-based approach.


Machine Translation Minimum Description Length Statistical Machine Translation Sentence Pair Defense Advance Research Project Agency 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wu, D.: Stochastic Inversion Transduction Grammars and Bilingual Parsing of Parallel Corpora. Computational Linguistics 23(3), 377–403 (1997)Google Scholar
  2. 2.
    Saers, M., Addanki, K., Wu, D.: From finite-state to inversion transductions: Toward unsupervised bilingual grammar induction. In: COLING 2012: Technical Papers, Mumbai, India, pp. 2325–2340 (December 2012)Google Scholar
  3. 3.
    Koehn, P., Och, F.J., Marcu, D.: Statistical Phrase-Based Translation. In: HLT/NAACL 2003, Edmonton, Canada, vol. 1 (May/June 2003)Google Scholar
  4. 4.
    Chiang, D.: A hierarchical phrase-based model for statistical machine translation. In: ACL 2005, Ann Arbor, Michigan, pp. 263–270 (June 2005)Google Scholar
  5. 5.
    Brown, P.F., Della Pietra, S.A., Della Pietra, V.J., Mercer, R.L.: The Mathematics of Machine Translation: Parameter estimation. CL 19(2) (1993)Google Scholar
  6. 6.
    Vogel, S., Ney, H., Tillmann, C.: HMM-based Word Alignment in Statistical Translation. In: COLING 1996, vol. 2, pp. 836–841 (1996)Google Scholar
  7. 7.
    Och, F.J., Ney, H.: A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics 29(1), 19–51 (2003)zbMATHCrossRefGoogle Scholar
  8. 8.
    Johnson, H., Martin, J., Foster, G., Kuhn, R.: Improving translation quality by discarding most of the phrasetable. In: EMNLP/CoNLL 2007, Prague, Czech Republic, pp. 967–975 (June 2007)Google Scholar
  9. 9.
    Cherry, C., Lin, D.: Inversion transduction grammar for joint phrasal translation modeling. In: SSST-1, Rochester, New York, pp. 17–24 (April 2007)Google Scholar
  10. 10.
    Zhang, H., Quirk, C., Moore, R.C., Gildea, D.: Bayesian learning of non-compositional phrases with synchronous parsing. In: ACL/HLT 2008, Columbus, Ohio, pp. 97–105 (June 2008)Google Scholar
  11. 11.
    Blunsom, P., Cohn, T., Dyer, C., Osborne, M.: A Gibbs sampler for phrasal synchronous grammar induction. In: ACL/IJCNLP 2009, Singapore (August 2009)Google Scholar
  12. 12.
    Haghighi, A., Blitzer, J., DeNero, J., Klein, D.: Better word alignments with supervised itg models. In: ACL/IJCNLP’09, Suntec, Singapore (August 2009)Google Scholar
  13. 13.
    Saers, M., Wu, D.: Improving phrase-based translation via word alignments from stochastic inversion transduction grammars. In: SSST-3, Boulder, CO (June 2009)Google Scholar
  14. 14.
    Saers, M., Wu, D.: Principled induction of phrasal bilexica. In: EAMT 2011, Leuven, Belgium, pp. 313–320 (May 2011)Google Scholar
  15. 15.
    Blunsom, P., Cohn, T.: Inducing synchronous grammars with slice sampling. In: HLT/NAACL 2010, Los Angeles, California, pp. 238–241 (June 2010)Google Scholar
  16. 16.
    Burkett, D., Blitzer, J., Klein, D.: Joint parsing and alignment with weakly synchronized grammars. In: HLT/NAACL 2010, Los Angeles, CA (June 2010)Google Scholar
  17. 17.
    Riesa, J., Marcu, D.: Hierarchical search for word alignment. In: ACL 2010, Uppsala, Sweden, pp. 157–166 (July 2010)Google Scholar
  18. 18.
    Saers, M., Nivre, J., Wu, D.: Word alignment with stochastic bracketing linear inversion transduction grammar. In: HLT/NAACL 2010, Los Angeles, California, pp. 341–344 (June 2010)Google Scholar
  19. 19.
    Neubig, G., Watanabe, T., Sumita, E., Mori, S., Kawahara, T.: An unsupervised model for joint phrase alignment and extraction. In: ACL/HLT 2011, Portland, Oregon (June 2011)Google Scholar
  20. 20.
    Neubig, G., Watanabe, T., Mori, S., Kawahara, T.: Machine translation without words through substring alignment. In: ACL 2012, Jeju, Korea (July 2012)Google Scholar
  21. 21.
    Galley, M., Graehl, J., Knight, K., Marcu, D., DeNeefe, S., Wang, W., Thayer, I.: Scalable inference and training of context-rich syntactic translation models. In: COLING/ACL 2006, Sydney, Australia (July 2006)Google Scholar
  22. 22.
    Stolcke, A., Omohundro, S.: Inducing probabilistic grammars by bayesian model merging. In: Carrasco, R.C., Oncina, J. (eds.) ICGI 1994. LNCS, vol. 862, pp. 106–118. Springer, Heidelberg (1994)CrossRefGoogle Scholar
  23. 23.
    Grünwald, P.: A minimum description length approach to grammar inference in symbolic. In: Wermter, S., Scheler, G., Riloff, E. (eds.) IJCAI-WS 1995. LNCS (LNAI), vol. 1040, pp. 203–216. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  24. 24.
    Si, Z., Pei, M., Yao, B., Zhu, S.-C.: Unsupervised learning of event and-or grammar and semantics from video. In: IEEE ICCV 2011 (November 2011)Google Scholar
  25. 25.
    Solomonoff, R.J.: A new method for discovering the grammars of phrase structure languages. In: IFIP Congress, pp. 285–289 (1959)Google Scholar
  26. 26.
    Rissanen, J.: A universal prior for integers and estimation by minimum description length. The Annals of Statistics 11(2), 416–431 (1983)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Shannon, C.E.: A mathematical theory of communication. The Bell System Technical Journal 27, 379–423, 623–656 (1948)Google Scholar
  28. 28.
    Saers, M., Nivre, J., Wu, D.: Learning stochastic bracketing inversion transduction grammars with a cubic time biparsing algorithm. In: IWPT 2009, Paris, France, pp. 29–32 (October 2009)Google Scholar
  29. 29.
    Fordyce, C.S.: Overview of the IWSLT 2007 evaluation campaign. In: IWSLT 2007 (2007)Google Scholar
  30. 30.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Cocke, J.: Programming languages and their compilers: Preliminary notes. Courant Institute of Mathematical Sciences, New York University (1969)Google Scholar
  32. 32.
    Kasami, T.: An efficient recognition and syntax analysis algorithm for context-free languages. Technical Report AFCRL-65-00143, Air Force Cambridge Research Laboratory (1965)Google Scholar
  33. 33.
    Younger, D.H.: Recognition and parsing of context-free languages in time n 3. Information and Control 10(2), 189–208 (1967)zbMATHCrossRefGoogle Scholar
  34. 34.
    Chiang, D.: Hierarchical phrase-based translation. CL 33(2) (2007)Google Scholar
  35. 35.
    Stolcke, A.: SRILM – an extensible language modeling toolkit. In: ICSLP 2002, Denver, Colorado, pp. 901–904 (September 2002)Google Scholar
  36. 36.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: ACL 2002, Philadelphia, PA (July 2002)Google Scholar
  37. 37.
    Doddington, G.: Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: HLT 2002, San Diego, California, pp. 138–145 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Markus Saers
    • 1
  • Karteek Addanki
    • 1
  • Dekai Wu
    • 1
  1. 1.Human Language Technology Center Dept. of Computer Science and EngineeringHong Kong University of Science and TechnologyHong Kong

Personalised recommendations