Self-organized Reservoirs and Their Hierarchies

  • Mantas Lukoševičius
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7552)


We investigate how unsupervised training of recurrent neural networks (RNNs) and their deep hierarchies can benefit a supervised task like temporal pattern detection. The RNNs are fully and fast trained by unsupervised algorithms and only supervised feed-forward readouts are used. The unsupervised RNNs are shown to perform better in a rigorous comparison against state-of-art random reservoir networks. Unsupervised greedy bottom-up trained hierarchies of such RNNs are shown being capable of big performance improvements over single layer setups.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Chappell, G.J., Taylor, J.G.: The temporal Kohonen map. Neural Networks 6(3), 441–445 (1993)CrossRefGoogle Scholar
  2. 2.
    Erhan, D., Bengio, Y., Courville, A.C., Manzagol, P.-A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11, 625–660 (2010)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Farkaš, I., Crocker, M.W.: Systematicity in sentence processing with a recursive self-organizing neural network. In: Proceedings of ESANN 2007, pp. 49–54 (2007)Google Scholar
  4. 4.
    Hammer, B., Micheli, A., Sperduti, A., Strickert, M.: Recursive self-organizing network models. Neural Networks 17(8-9), 1061–1085 (2004)zbMATHCrossRefGoogle Scholar
  5. 5.
    Jaeger, H.: Echo state network. Scholarpedia 2(9), 2330 (2007)CrossRefGoogle Scholar
  6. 6.
    Jaeger, H., Lukoševičius, M., Popovici, D., Siewert, U.: Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks 20(3), 335–352 (2007)zbMATHCrossRefGoogle Scholar
  7. 7.
    Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: Proceedings of ICCV 2009. IEEE (2009)Google Scholar
  8. 8.
    Kohonen, T., Honkela, T.: Kohonen network. Scholarpedia 2(1), 1568 (2007)CrossRefGoogle Scholar
  9. 9.
    Koutník, J.: Inductive modelling of temporal sequences by means of self-organization. In: Proceeding of IWIM 2007, pp. 269–277. CTU in Prague (2007)Google Scholar
  10. 10.
    Lazar, A., Pipa, G., Triesch, J.: SORN: a self-organizing recurrent neural network. Frontiers in Computational Neuroscience 4, 12 (2010)Google Scholar
  11. 11.
    LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Computation 1(4), 541–551 (1989)CrossRefGoogle Scholar
  12. 12.
    Lukoševičius, M.: Reservoir Computing and Self-Organized Neural Hierarchies. PhD thesis, Jacobs University Bremen, Bremen, Germany (2011)Google Scholar
  13. 13.
    Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Computer Science Review 3(3), 127–149 (2009)CrossRefGoogle Scholar
  14. 14.
    Mak, M.W.: A learning algorithm for recurrent radial basis function networks. Neural Processing Letters 2(1), 27–31 (1995)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Martinetz, T., Schulten, K.: A “neural-gas” network learns topologies. Artificial Neural Networks 1, 397–402 (1991)Google Scholar
  16. 16.
    Rahimi, A., Recht, B.: Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In: Proceedings of NIPS 2008, pp. 1313–1320. MIT Press, Cambridge (2009)Google Scholar
  17. 17.
    Saxe, A., Koh, P., Chen, Z., Bhand, M., Suresh, B., Ng, A.Y.: On random weights and unsupervised feature learning. In: Proceedings of ICML 2011 (2011)Google Scholar
  18. 18.
    Tiňo, P., Farkaš, I., van Mourik, J.: Dynamics and topographic organization of recursive self-organizing maps. Neural Computation 18(10), 2529–2567 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Varsta, M., Millán, J.D.R., Heikkonen, J.: A Recurrent Self-Organizing Map for Temporal Sequence Processing. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 421–426. Springer, Heidelberg (1997)Google Scholar
  20. 20.
    Voegtlin, T.: Recursive self-organizing maps. Neural Networks 15(8-9), 979–991 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Mantas Lukoševičius
    • 1
  1. 1.Jacobs University BremenBremenGermany

Personalised recommendations