Skip to main content
SpringerLink
Log in

Guided Self-Organization of Input-Driven Recurrent Neural Networks

  • Chapter
Guided Self-Organization: Inception

Part of the book series: Emergence, Complexity and Computation ((ECC,volume 9))

Abstract

To understand the world around us, our brains solve a variety of tasks. One of the crucial functions of a brain is to make predictions of what will happen next, or in the near future. This ability helps us to anticipate upcoming events and plan our reactions to them in advance. To make these predictions, past information needs to be stored, transformed or used otherwise. How exactly the brain achieves this information processing is far from clear and under heavy investigation. To guide this extraordinary research effort, neuroscientists increasingly look for theoretical frameworks that could help explain the data recorded from the brain, and to make the enormous task more manageable. This is evident, for instance, through the funding of the billion-dollar ”Human Brain Project”, of the European Union, amongst others. Mathematical techniques from graph and information theory, control theory, dynamical and complex systems (Sporns 2011), statistical mechanics (Rolls and Deco 2010), as well as machine learning and computer vision (Seung 2012; Hawkins and Blakeslee 2004), have provided new insights into brain structure and possible function, and continue to generate new hypotheses for future research.

Both authors contributed equally to this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Ashby, W.R.: An Introduction to Cybernetics. Chapman & Hall, London (1956)

    Google Scholar 

  • Baddeley, R., Abbott, L.F., Booth, M.C.A., Sengpiel, F., Freeman, T., Wakeman, E.A., Roll, E.T.: Responses of neurons in primary and inferior temporal visual cortices to natural scenes. Proc. R. Soc. Lond. B 264, 1775–1783 (1997)

    Article  Google Scholar 

  • Bell, A.J., Sejnowski, T.J.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7(6), 1129–1159 (1995)

    Article  Google Scholar 

  • Bengio, Y., Boulanger-Lewandowski, N., Pascanu, R.: Advances in Optimizing Recurrent Networks. arXiv preprint 1212.0901, arXiv.org (2012)

    Google Scholar 

  • Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transaction on Neural Networks 5(2), 157–166 (1994)

    Article  Google Scholar 

  • Bertschinger, N., Natschläger, T.: Real-time computation at the edge of chaos in recurrent neural networks. Neural Computation 16(7), 1413–1436 (2004)

    Article  MATH  Google Scholar 

  • Boedecker, J., Obst, O., Lizier, J.T., Mayer, N.M., Asada, M.: Information processing in echo state networks at the edge of chaos. Theory in biosciences Theorie in den Biowissenschaften 131(3), 1–9 (2011)

    Google Scholar 

  • Boedecker, J., Obst, O., Mayer, N.M., Asada, M.: Initialization and self-organized optimization of recurrent neural network connectivity. HFSP Journal 3(5), 340–349 (2009)

    Article  Google Scholar 

  • Crutchfield, J.P., Machta, J.: Introduction to focus issue on “Randomness, Structure, and Causality: Measures of complexity from theory to applications”. Chaos 21(3), 037101 (2011)

    Google Scholar 

  • Dambre, J., Verstraeten, D., Schrauwen, B., Massar, S.: Information processing capacity of dynamical systems. Scientific Reports 2, 514 (2012)

    Article  Google Scholar 

  • Douglas, R., Markram, H., Martin, K.: Neocortex. In: Shepherd, G. (ed.) Synaptic Organization In the Brain, pp. 499–558. Oxford University Press (2004)

    Google Scholar 

  • Doya, K.: Bifurcations in the learning of recurrent neural networks. In: IEEE International Symposium on Circuits and Systems, pp. 2777–2780. IEEE (1992)

    Google Scholar 

  • Ganguli, S., Huh, D., Sompolinsky, H.: Memory traces in dynamical systems. Proceedings of the National Academy of Sciences 105(48), 18970–18975 (2008)

    Article  Google Scholar 

  • Grassberger, P.: Toward a quantitative theory of self-generated complexity. International Journal of Theoretical Physics 25(9), 907–938 (1986)

    Article  MATH  MathSciNet  Google Scholar 

  • Grassberger, P.: Randomness, information, and complexity. Technical Report 1208.3459, arXiv.org (2012)

    Google Scholar 

  • Hawkins, J., Blakeslee, S.: On Intelligence. Times Books (2004)

    Google Scholar 

  • Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6(2), 107–116 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  • Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  • Jaeger, H.: Short term memory in echo state networks. Technical Report 152, GMD – German National Research Institute for Computer Science (2001)

    Google Scholar 

  • Jaeger, H., Haas, H.: Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication. Science 304(5667), 78–80 (2004)

    Article  Google Scholar 

  • Kohonen, T.: Self-Organizing Maps, 3rd, extended edn. Springer (2001)

    Google Scholar 

  • Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Problemy Peredachi Informatsii 1(1), 3–11 (1965)

    MATH  MathSciNet  Google Scholar 

  • Lazar, A., Pipa, G., Triesch, J.: SORN: a self-organizing recurrent neural network. Frontiers in Computational Neuroscience 3, 23 (2009)

    Google Scholar 

  • Legenstein, R., Maass, W.: What makes a dynamical system computationally powerful. In: Haykin, S., Principe, J.C., Sejnowski, T., McWhirter, J. (eds.) New Directions in Statistical Signal Processing: From Systems to Brains, pp. 127–154. MIT Press (2007)

    Google Scholar 

  • Linsker, R.: Towards an organizing principle for a layered perceptual network. In: Anderson, D.Z. (ed.) NIPS, pp. 485–494. American Institute of Physics (1987)

    Google Scholar 

  • Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Detecting non-trivial computation in complex dynamics. In: Almeida e Costa, F., Rocha, L.M., Costa, E., Harvey, I., Coutinho, A. (eds.) ECAL 2007. LNCS (LNAI), vol. 4648, pp. 895–904. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  • Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Local measures of information storage in complex distributed computation. Information Sciences 208, 39–54 (2012)

    Article  Google Scholar 

  • Lukoševičius, M.: A practical guide to applying echo state networks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade, 2nd edn. LNCS, vol. 7700, pp. 659–686. Springer, Heidelberg (2012a)

    Google Scholar 

  • Lukoševičius, M.: Self-organized reservoirs and their hierarchies. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012, Part I. LNCS, vol. 7552, pp. 587–595. Springer, Heidelberg (2012b)

    Chapter  Google Scholar 

  • Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Computer Science Review 3(3), 127–149 (2009)

    Article  Google Scholar 

  • Maass, W., Joshi, P., Sontag, E.D.: Computational aspects of feedback in neural circuits. PLOS Computational Biology 3(1), e165 (2007)

    Google Scholar 

  • Manjunath, G., Tino, P., Jaeger, H.: Theory of Input Driven Dynamical Systems. In: dice.ucl.ac.be, pp. 25–27 (April 2012)

    Google Scholar 

  • Martens, J., Sutskever, I.: Learning recurrent neural networks with hessian-free optimization. In: Proceedings of the 28th International Conference on Machine Learning, vol. 46, p. 68. Omnipress Madison, WI (2011)

    Google Scholar 

  • Martinetz, T., Schulten, K.: A “neural-gas” network learns topologies. Artificial Neural Networks 1, 397–402 (1991)

    Google Scholar 

  • Mitchell, M., Hraber, P.T., Crutchfield, J.P.: Revisiting the edge of chaos: Evolving cellular automata to perform computations. Complex Systems 7, 89–130 (1993)

    MATH  Google Scholar 

  • Obst, O., Boedecker, J., Asada, M.: Improving Recurrent Neural Network Performance Using Transfer Entropy. Neural Information Processing Models and Applications 6444, 193–200 (2010)

    Article  Google Scholar 

  • Obst, O., Boedecker, J., Schmidt, B., Asada, M.: On active information storage in input-driven systems. preprint 1303.5526v1, arXiv.org (2013)

    Google Scholar 

  • Ozturk, M.C., Xu, D., Príncipe, J.C.: Analysis and design of echo state networks. Neural Computation 19(1), 111–138 (2007)

    Article  MATH  Google Scholar 

  • Prokopenko, M., Lizier, J.T., Obst, O., Wang, X.R.: Relating Fisher information to order parameters. Physical Review E 84(4), 041116 (2011)

    Google Scholar 

  • Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the rprop algorithm. In: IEEE International Conference on Neural Networks, vol. 1, pp. 586–591 (1993)

    Google Scholar 

  • Rissanen, J.: Modeling by shortest data description. Automatica 14(5), 465–471 (1978)

    Article  MATH  Google Scholar 

  • Rolls, E.T., Deco, G.: The Noisy Brain - Stochastic Dynamics as a Principle of Brain Function. Oxford University Press (2010)

    Google Scholar 

  • Rumelhart, D., Hinton, G., Williams, R.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)

    Article  Google Scholar 

  • Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by evolino. Neural Computation 19(3), 757–779 (2007)

    Article  MATH  Google Scholar 

  • Schrauwen, B., Wardermann, M., Verstraeten, D., Steil, J.J., Stroobandt, D.: Improving reservoirs using intrinsic plasticity. Neurocomputing 71(7-9), 1159–1171 (2008)

    Article  Google Scholar 

  • Schreiber, T.: Measuring information transfer. Physical Review Letters 85(2), 461–464 (2000)

    Article  Google Scholar 

  • Seung, H.S.: Connectome: How the Brain’s Wiring Makes Us Who We Are. Houghton Mifflin Harcout, New York (2012)

    Google Scholar 

  • Sporns, O.: Networks Of the Brain. The MIT Press (2011)

    Google Scholar 

  • Sussillo, D., Barak, O.: Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Computation 25(3), 626–649 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  • Tino, P., Rodan, A.: Short term memory in input-driven linear dynamical systems. Neurocomputing (2013)

    Google Scholar 

  • Triesch, J.: A gradient rule for the plasticity of a neuron’s intrinsic excitability. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) ICANN 2005. LNCS, vol. 3696, pp. 65–70. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  • Voegtlin, T.: Recursive self-organizing maps. Neural Networks 15(8-9), 979–991 (2002)

    Article  Google Scholar 

  • Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10), 1550–1560 (1990)

    Article  Google Scholar 

  • Williams, P.L., Beer, R.D.: Information dynamics of evolved agents. From Animals to Animats 11, 38–49 (2010)

    Article  Google Scholar 

  • Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1(2), 270–280 (1989)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CSIRO Computational Informatics, PO Box 76, Epping, NSW, 1710, Australia

    Oliver Obst

  2. Machine Learning Lab, University of Freiburg, Freiburg, Germany

    Joschka Boedecker

Authors
  1. Oliver Obst
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joschka Boedecker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oliver Obst .

Editor information

Editors and Affiliations

  1. CSIRO ICT Centre, Epping, Australia

    Mikhail Prokopenko

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Obst, O., Boedecker, J. (2014). Guided Self-Organization of Input-Driven Recurrent Neural Networks. In: Prokopenko, M. (eds) Guided Self-Organization: Inception. Emergence, Complexity and Computation, vol 9. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53734-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-53734-9_11

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-53733-2

  • Online ISBN: 978-3-642-53734-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation