Dynamic Bayesian Networks for Language Modeling

  • Pascal Wiggers
  • Leon J. M. Rothkrantz
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4188)


Although n-gram models are still the de facto standard in language modeling for speech recognition, it has been shown that more sophisticated models achieve better accuracy by taking additional information, such as syntactic rules, semantic relations or domain knowledge into account. Unfortunately, most of the effort in developing such models goes into the implementation of handcrafted inference routines. What lacks is a generic mechanism to introduce background knowledge into a language model. We propose the use of dynamic Bayesian networks for this purpose. Dynamic Bayesian networks can be seen as a generalization of the n-gram models and hmms traditionally used in language modeling and speech recognition. Whereas those models use a single random variable to represent state, Bayesian networks can have any number of variables. As such they are particularly well-suited for the construction of models that take additional information into account. In this paper language modeling with belief networks is discussed. Examples of belief network implementations of well-known language models are given and a new model is presented that models dependencies between the content words in a sentence.


Bayesian Network Speech Recognition Language Model Content Word Dynamic Bayesian Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Chelba, C., Jelinek, F.: Exploiting Syntactic Structure for Language Modeling. In: Proc. of the 36th Annual Meeting of the ACL (August 1998)Google Scholar
  2. 2.
    Rosenfeld, R.: A Maximum Entropy Approach to Adaptive Statistical Language Modeling. Computer, Speech and Language 10, 187–228 (1996)CrossRefGoogle Scholar
  3. 3.
    Wiggers, P., Rothkrantz, L.J.M.: Using Confidence Measures and Domain Knowledge to Improve Speech Recognition. In: Proceedings of Eurospeech 2003, Geneva Switserland (September 2003)Google Scholar
  4. 4.
    Goodman, J.: A Bit of Progress in Language Modeling. Computer Speech and Language, pp. 403–434 (October 2001)Google Scholar
  5. 5.
    Iyer, R., Ostendorf, M.: Modeling Long Distance Dependence in Language: Topic Mixtures vs. Dynamic Cache Models. In: Proc. ICSLP 1996, Philadelphia, PA (1996)Google Scholar
  6. 6.
    Zweig, G.: Speech Recognition with Dynamic Bayesian Networks, Ph.D. Thesis, Computer Science Division, University of California at Berkeley (1998)Google Scholar
  7. 7.
    Bellegarda, J.R.: A Multispan Language Modeling Framework for Large Vocabulary Speech Recognition. IEEE Transactions on Speech and Audio Processing 6(5) (September 1998)Google Scholar
  8. 8.
    Gildea, D., Hoffman, T.: Topic-Based Language Models using EM. In: Eurospeech 1999, Budapest (1999)Google Scholar
  9. 9.
    Zhang, R., Rudnicky, A.I.: Improve Latent Semantic Analysis based Language Model by Integrating Multiple Level Knowledge. In: Proc. of ICSLP 2002, Denver, Colorado (2002)Google Scholar
  10. 10.
    Roark, B.: Probabilistic Top-Down Parsing and Language Modeling. Computational Linguistics 27(2)Google Scholar
  11. 11.
    Charniak, E.: Immediate-Head Parsing for Language Models. Meeting of the Association for Computational Linguistics, 116–123 (2001)Google Scholar
  12. 12.
    Murphy, K.: Dynamic Bayesian Networks: Representation, Inference and Learning, Ph.D. Thesis, University of California, Berkeley (2002)Google Scholar
  13. 13.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems - Networks of Plausible Inference. Morgan Kaufmann Publishers, San Francisco (1988)Google Scholar
  14. 14.
    Bilmes, J.: Natural Statistical Models for Automatic Speech Recognition, Ph.D. Thesis, Dept. of EECS, CS Devision, UC Berkeley (1999)Google Scholar
  15. 15.
    Nefian, A.V., Liang, L., Pi, X., Liu, X., Murphy, K.: Dynamic Bayesian Networks for Audio-Visual Speech Recognition. EURASIP Journal on Applied Signal Processing 11, 1–15 (2002)Google Scholar
  16. 16.
    Keizer, S.: Reasoning under Uncertainty in Natural Language Dialogue using Networks, Ph.D. Thesis Twenty University, 96 (2003)Google Scholar
  17. 17.
    Jelinek, F.: Self-organized language modeling for speech recognition. In: Waibel, A., Lee, K.-F. (eds.) Readings in Speech Recognition. Morgan Kaufman Publishers, San Mateo (1990)Google Scholar
  18. 18.
    Schuurman, I., Schouppe, M., Hoekstra, H., van der Wouden, T.: CGN, an Annotated Corpus of Spoken Dutch. In: Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora (LINC 2003), Budapest, Hungary, April 14 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Pascal Wiggers
    • 1
  • Leon J. M. Rothkrantz
    • 1
  1. 1.Man-Machine Interaction GroupDelft University of TechnologyDelftThe Netherlands

Personalised recommendations