Biologically Plausible Speech Recognition with LSTM Neural Nets

  • Alex Graves
  • Douglas Eck
  • Nicole Beringer
  • Juergen Schmidhuber
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3141)


Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) are local in space and time and closely related to a biological model of memory in the prefrontal cortex. Not only are they more biologically plausible than previous artificial RNNs, they also outperformed them on many artificially generated sequential processing tasks. This encouraged us to apply LSTM to more realistic problems, such as the recognition of spoken digits. Without any modification of the underlying algorithm, we achieved results comparable to state-of-the-art Hidden Markov Model (HMM) based recognisers on both the TIDIGITS and TI46 speech corpora. We conclude that LSTM should be further investigated as a biologically plausible basis for a bottom-up, neural net-based approach to speech recognition.


Hide Markov Model Speech Recognition Recurrent Neural Network Memory Block Input Gate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Williams, R.J., Zipser, D.: Gradient-based learning algorithms for recurrent networks and their computational complexity. In: Chauvin, Y., Rumelhart, D.E. (eds.) Back-propagation: Theory, Architectures and Applications, pp. 433–486. Lawrence Erlbaum Publishers, Hillsdale (1995)Google Scholar
  2. 2.
    Werbos, P.J.: Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1 (1988)Google Scholar
  3. 3.
    Robinson, A.J., Fallside, F.: The utility driven dynamic error propagation network. Technical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department (1987)Google Scholar
  4. 4.
    Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: Kremer, S.C., Kolen, J.F. (eds.) A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press, Los Alamitos (2001)Google Scholar
  5. 5.
    Rabiner, L.R.: A tutorial on hidden markov models and selected applications in speech recognition. Proc. IEEE 77, 257–286 (1989)CrossRefGoogle Scholar
  6. 6.
    Bourlard, H., Morgan, N.: Connnectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, Dordrecht (1994)Google Scholar
  7. 7.
    Robinson, A.J.: An application of recurrent nets to phone probability estimation. IEEE Transactions on Neural Networks 5, 298–305 (1994)CrossRefGoogle Scholar
  8. 8.
    Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9, 1735–1780 (1997)CrossRefGoogle Scholar
  9. 9.
    Gers, F.: Long Short-Term Memory in Recurrent Neural Networks. PhD thesis (2001)Google Scholar
  10. 10.
    O’Reilly, R.: Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Technical Report ICS-03-03, ICS (2003)Google Scholar
  11. 11.
    Gers, F.A., Schmidhuber, J.: LSTM recurrent networks learn simple context free and context sensitive languages. IEEE Transactions on Neural Networks 12, 1333–1340 (2001)CrossRefGoogle Scholar
  12. 12.
    Eck, D., Schmidhuber, J.: Finding temporal structure in music: Blues improvisation with LSTM recurrent networks. In: Bourlard, H. (ed.) Proceedings of the 2002 IEEE Workshop in Neural Networks for Signal Processing XII, pp. 747–756. IEEE, New York (2002)CrossRefGoogle Scholar
  13. 13.
    Young, S.: The HTK Book. Cambridge University Press, Cambridge (1995/1996)Google Scholar
  14. 14.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Inc., Oxford (1995)Google Scholar
  15. 15.
    Plaut, D.C., Nowlan, S.J., Hinton, G.E.: Experiments on learning back propagation. Technical Report CMU–CS–86–126, Carnegie–Mellon University, Pittsburgh, PA (1986)Google Scholar
  16. 16.
    Zheng, F., Picone, J.: Robust low perplexity voice interfaces. Technical report, MITRE Corporation (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Alex Graves
    • 1
  • Douglas Eck
    • 1
  • Nicole Beringer
    • 1
  • Juergen Schmidhuber
    • 1
  1. 1.Istituto Dalle Molle di Studi sull’Intelligenza ArtificialeManno-LuganoSwitzerland

Personalised recommendations