Neural Network Language Model with Cache

  • Daniel Soutner
  • Zdeněk Loose
  • Luděk Müller
  • Aleš Pražák
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7499)


In this paper we investigate whether a combination of statistical, neural network and cache language models can outperform a basic statistical model. These models have been developed, tested and exploited for a Czech spontaneous speech data, which is very different from common written Czech and is specified by a small set of the data available and high inflection of the words. As a baseline model we used a trigram model and after its training several cache models interpolated with the baseline model have been tested and measured on a perplexity. Finally, an evaluation of the model with the lowest perplexity has been performed on speech recordings of phone calls.


neural networks language modelling automatic speech recognition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Stolcke, A.: SRILM – an extensible language modeling toolkit. In: INTERSPEECH (2002)Google Scholar
  2. 2.
    Mikolov, T., Kopecký, J., Burget, L., Glembek, O., Černocký, J.: Neural network based language models for highly inflective languages. In: ICASSP, pp. 4725–4728 (2009)Google Scholar
  3. 3.
    Schwenk, H., Gauvain, J.: Training Neural Network Language Models on Very Large Corpora. In: HLT/EMNLP (2005)Google Scholar
  4. 4.
    Brown, P.F., Pietra, V.J.D., Souza, P.V.D., Lai, J.C., Mercer, R.L.: Class-Based n-gram Models of Natural Language. Computational Linguistics, 467–479 (1992)Google Scholar
  5. 5.
    Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)zbMATHGoogle Scholar
  6. 6.
    Kuhn, R., De Mori, R.: A Cache-Based Natural Language Model for Speech Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 570–583 (June 1990)Google Scholar
  7. 7.
    Trmal, J., Zelinka, J., Müller, L.: Adaptation of a Feedforward Artificial Neural Network Using a Linear Transform. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) TSD 2010. LNCS, vol. 6231, pp. 423–430. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Pavel, I., Josef, P., Psutka Josef, V.: Using Morphological Information for Robust Language Modeling in Czech ASR System. IEEE Transactions on Audio Speech and Language Processing 17, 840–847 (2009)CrossRefGoogle Scholar
  9. 9.
    Skorkovská, L., Ircing, P., Pražák, A., Lehečka, J.: Automatic Topic Identification for Large Scale Language Modeling Data Filtering. In: Habernal, I., Matoušek, V. (eds.) TSD 2011. LNCS, vol. 6836, pp. 64–71. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  10. 10.
    Bacchiani, M., Roark, B.: Unsupervised language model adaptation. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 224–227 (2003)Google Scholar
  11. 11.
    Psutka, J., Švec, J., Psutka, J.V., Vaněk, J., Pražák, A., Šmídl, L., Ircing, P.: System for Fast Lexical and Phonetic Spoken Term Detection in a Czech Cultural Heritage Archive. EURASIP Journal on Audio, Speech, and Music Processing (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Daniel Soutner
    • 1
  • Zdeněk Loose
    • 1
  • Luděk Müller
    • 1
  • Aleš Pražák
    • 1
  1. 1.Faculty of Applied Science, Dept. of CyberneticsUniversity of West BohemiaPlzeňCzech Republic

Personalised recommendations