Abstract
The purpose of this paper is to further investigate how and why long short-term memory networks (LSTM) perform so well on several pattern recognition problems. Our contribution is three-fold. First, we describe the main highlights of the LSTM architecture, especially when compared to standard recurrent neural networks (SRN). Second, we give an overview of previous studies to analyze the behavior of LSTMs on toy problems and some realistic data in the speech recognition domain. Third, the behavior of LSTMs is analyzed on novel problems which are relevant for pattern recognition research. Thereby, we analyze the ability of LSTMs to classify long sequences containing specific patterns at an arbitrary position on iteratively increasing the complexity of the problem under constant training conditions. We also compare the behavior of LSTMs to SRNs for text vs. non-text sequence classification on a real-world problem with significant non-local time-dependencies where the features are computed only locally. Finally, we discuss why LSTMs with standard training methods are not suited for the task of signature verification.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Eck, D., Schmidhuber, J.: Finding temporal structure in music: Blues improvisation with lstm recurrent networks. In: Proceedings of the 2002 IEEE Workshop on Neural Networks for Signal Processing XII, pp. 747–756. IEEE (2002)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with LSTM. Neural Computation 12, 2451–2471 (1999)
Gers, F.A., Schmidhuber, J.: LSTM recurrent networks learn simple context free and context sensitive languages. IEEE Transactions on Neural Networks 12, 1333–1340 (2001)
Gers, F.A., Schraudolph, N.N., Schmidhuber, J.: Learning precise timing with LSTM recurrent networks. Journal of Machine Learning Research 3, 115–143 (2002)
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(5), 855–868 (2009)
Graves, A.: A comparison of network architectures. In: Graves, A. (ed.) Supervised Sequence Labelling with Recurrent Neural Networks. SCI, vol. 385, pp. 49–58. Springer, Heidelberg (2012)
Graves, A., Bruegge, B., Schmidhuber, J., Kramer, S.: Supervised Sequence Labelling with Recurrent Neural Networks. Ph.D. thesis, Technische Universitaet Muenchen (2008)
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks: The Official Journal of the International Neural Network Society 18(5-6), 602–610 (2005)
Hochreiter, S., Heusel, M., Obermayer, K.: Fast model-based protein homology detection without alignment. Bioinformatics 23(14), 1728–1736 (2007)
Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Comput 9(8), 1735–1780 (1997)
Indermuehle, E., Liwicki, M., Bunke, H.: IAMonDo-database: an online handwritten document database with non-uniform contents. In: 9th Int. Workshop on Document Analysis Systems, pp. 97–104 (2010)
Liwicki, M., Malik, M.I., Heuvel, C., Chen, X., Berger, C., Stoel, R., Blumenstein, M., Found, B.: Signature Verification Competition for Online and Offline Skilled Forgeries (SigComp 2011). In: 2011 International Conference on Document Analysis and Recognition (ICDAR), pp. 1480–1484. IEEE (2011)
Marti, U.V., Bunke, H.: Using a statistical language model to improve the performance of an HMM-based cursive handwriting recognition system. Int. Journal of Pattern Recognition and Artificial Intelligence 15, 65–90 (2001)
Otte, S., Liwicki, M., Krechel, D., Dengel, A.: Local feature based online mode detection with recurrent neural networks. In: 2012 International Conference on Frontiers in Handwriting Recognition, ICFHR (2012)
Robinson, A.J., Fallside, F.: The utility driven dynamic error propagation network. Tech. Rep. CUED/F-INFENG/TR.1, Cambridge University Engineering Department, Cambridge (1987)
Schmidhuber, J., Gers, F., Eck, D.: Learning nonregular languages: A comparison of simple recurrent networks and lstm. Neural Computation 14 (2002)
Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by evolino. Neural Comput. 19(3), 757–779 (2007)
Schraudolph, N.N.: A fast, compact approximation of the exponential function. Neural Computation 11, 4–11 (1998)
Tiin, C., Omlin, C.: LSTM recurrent neural networks for signature veri cation. In: Proc. Southern African Telecommunication Networks & Applications Conference (2003)
Tiin, C.: LSTM Recurrent Neural Networks for Signature Veri cation: A Novel Approach. LAP LAMBERT Academic Publishing (2012)
Williams, R.J., Zipser, D.: Gradient-Based learning algorithms for recurrent networks and their computational complexity (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Otte, S., Liwicki, M., Krechel, D. (2014). Investigating Long Short-Term Memory Networks for Various Pattern Recognition Problems. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2014. Lecture Notes in Computer Science(), vol 8556. Springer, Cham. https://doi.org/10.1007/978-3-319-08979-9_37
Download citation
DOI: https://doi.org/10.1007/978-3-319-08979-9_37
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08978-2
Online ISBN: 978-3-319-08979-9
eBook Packages: Computer ScienceComputer Science (R0)