Skip to main content
Download PDF

Behavioral analysis of differential hebbian learning in closed-loop systems

Biological Cybernetics volume 103, pages 255–271 (2010)Cite this article

Abstract

Understanding closed loop behavioral systems is a non-trivial problem, especially when they change during learning. Descriptions of closed loop systems in terms of information theory date back to the 1950s, however, there have been only a few attempts which take into account learning, mostly measuring information of inputs. In this study we analyze a specific type of closed loop system by looking at the input as well as the output space. For this, we investigate simulated agents that perform differential Hebbian learning (STDP). In the first part we show that analytical solutions can be found for the temporal development of such systems for relatively simple cases. In the second part of this study we try to answer the following question: How can we predict which system from a given class would be the best for a particular scenario? This question is addressed using energy, input/output ratio and entropy measures and investigating their development during learning. This way we can show that within well-specified scenarios there are indeed agents which are optimal with respect to their structure and adaptive properties.

References

  • Ashby WR (1956) An introduction to cybernetics. Chapmann and Hall Ltd., London

    Google Scholar 

  • Ay N, Bertschinger N, Der R, Güttler F, Olbrich E (2008) Predictive information and explorative behavior of autonomous robots. Eur Phys J B 63: 329–339

    CAS  Article  Google Scholar 

  • Baldwin JM (1896) A new factor in evolution. Am Nat 30: 441–451

    Article  Google Scholar 

  • Bi GQ, Poo MM (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18: 10464–10472

    CAS  PubMed  Google Scholar 

  • Box G, Jenkins GM, Reinsel GC (1994) Time series analysis: forecasting and control. Prentice-Hall, Englewood Cliffs, NJ

    Google Scholar 

  • Braitenberg V (1986) Vehicles: experiments in synthetic psychology. The MIT Press, Cambridge, MA

    Google Scholar 

  • Der R, Güttler F, Ay N (2008) Predictive information and emergent cooperativity in a chain of mobile robots. In: Bullock S, Noble J, Watson R, Bedau MA (eds) Artificial life XI: proceedings of the eleventh international conference on the simulation and synthesis of living systems.. MIT Press, Cambridge, MA, pp 166–172

    Google Scholar 

  • Hebb DO (1949) The organization of behavior. Wiley, New York

    Google Scholar 

  • Hinton GE, Nowlan SJ (1987) How learning guides evolution. Complex Syst 1: 495–502

    Google Scholar 

  • Hofstötter C, Mintz M, Verschure PF (2002) The cerebellum in action: a simulation and robotics study. Eur J Neurosci 16: 1361–1376

    Article  PubMed  Google Scholar 

  • Iglesias R, Nehmzow U, Billings SA (2008) Model identification and model analysis in robot training. Robot Auton Syst 56: 1061–1067

    Article  Google Scholar 

  • Klopf AH (1988) A neuronal model of classical conditioning. Psychobiology 16(2): 85–123

    Google Scholar 

  • Klyubin AS, Polani D, Nehaniv CL (2004) Organization of the information flow in the perception-action loop of evolved agents. In: 2004 NASA/DoD conference on evolvable hardware. IEEE Computer Society, pp 177–180

  • Klyubin AS, Polani D, Nehaniv CL (2005) Empowerment: a universal agent-centric measure of control. In: IEEE congress on evolutionary computation (CEC 2005), pp 128–135

  • Klyubin AS, Polani D, Nehaniv CL (2007) Representations of space and time in the maximization of information flow in the perception-action loop. Neural Comput 19: 2387–2432

    Article  PubMed  Google Scholar 

  • Klyubin AS, Polani D, Nehaniv CL (2008) Keep your options open: an information-based driving principle for sensorimotor systems. PLoS ONE 3: e4018

    Article  PubMed  Google Scholar 

  • Kosco B (1986) Differential Hebbian learning. In: Denker JS (eds) Neural networks for computing: AIP conference proceedings, vol 151. American Institute of Physics, New York

    Google Scholar 

  • Kulvicius T, Porr B, Wörgötter F (2007) Chained learning architectures in a simple closed-loop behavioural context. Biol Cybern 97: 363–378

    Article  PubMed  Google Scholar 

  • Kyriacou T, Nehmzow U, Iglesias R, Billings SA (2008) Accurate robot simulation through system identification. Robot Auton Syst 56: 1082–1093

    Article  Google Scholar 

  • Lungarella M, Pegors T, Bulwinkle D, Sporns O (2005) Methods for quantifying the informational structure of sensory and motor data. Neuroinformatics 3: 243–262

    Article  PubMed  Google Scholar 

  • Lungarella M, Sporns O (2006) Mapping information flow in sensorimotor networks. PLoS Comput Biol 2: e144

    Article  PubMed  Google Scholar 

  • Markram H, Lübke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275: 213–215

    CAS  Article  PubMed  Google Scholar 

  • Porr B, Wörgötter F (2003a) Isotropic sequence order learning. Neural Comput 15: 831–864

    Article  PubMed  Google Scholar 

  • Porr B, Wörgötter F (2003b) Isotropic-sequence-order learning in a closed-loop behavioural system. Philos Transact A Math Phys Eng Sci 361: 2225–2244

    Article  PubMed  Google Scholar 

  • Porr B, Wörgötter F (2006) Strongly improved stability and faster convergence of temporal sequence learning by using input correlations only. Neural Comput 18: 1380–1412

    Article  PubMed  Google Scholar 

  • Porr B, Egerton A, Wörgötter F (2006) Towards closed loop information: Predictive information. Constr Found 1(2): 83–90

    Google Scholar 

  • Poupart P, Boutilier C (2002) Value-directed compression of POMDPs. In: Becker STS, Obermayer K (eds) Advances in neural information processing systems, vol 15. pp 1547–1554

  • Prokopenko M, Gerasimov V, Tanev I (2006) Evolving spatiotemporal coordination in a modular robotic system. In: SAB 2006. pp 558–569

  • Saudargiene A, Porr B, Wörgötter F (2004) How the shape of pre- and postsynaptic signals can influence STDP: a biophysical model. Neural Comput 16: 595–625

    Article  PubMed  Google Scholar 

  • Saudargiene A, Porr B, Wörgötter F (2005) Synaptic modifications depend on synapse location and activity: a biophysical model of STDP. BioSystems 79: 3–10

    CAS  Article  PubMed  Google Scholar 

  • Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27: 379–423

    Google Scholar 

  • Slonim N, Tishby N (2000) Document clustering using word clusters via the information bottleneck method. In: Proceedings of the 23rd annual international acm-sigir conference on research and development in information retrieval

  • Slonim N, Tishby N (2001) The power of word clustering for text classification. In: Proceedings of the 23rd European colloquium on information retrieval research

  • Slonim N, Somerville R, Tishby N, Lahav O (2001) Objective classification of galaxy spectra using the information bottleneck method. Mon Notes R Astron Soc 323: 270–284

    CAS  Article  Google Scholar 

  • Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88: 135–170

    CAS  Article  PubMed  Google Scholar 

  • Tishby N, Pereira FC, Bialek W (1999) The information bottleneck method. In: Proceedings of the 37-th annual allerton conference on communication, control and computing. pp 368–377

  • Touchette H, Lloyd S (2000) Information-theoretic approach to the study of control systems. Physica A 331: 140–172

    Article  Google Scholar 

  • Wolpert DM, Miall RC, Kawato M (1998) Internal models in the cerebellum. Trends Cogn Sci 2: 338–347

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the European funded PACO-PLUS project as well as by BMBF (Federal Ministry of Education and Research), BCCN (Bernstein Center for Computational Neuroscience)—Göttingen project W3 and BFNT project 3a

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Author information

Affiliations

  1. Bernstein Center for Computational Neuroscience, Department for Computational Neuroscience, III Physikalisches Institut - Biophysik, Georg-August-Universität Göttingen, Friedrich-Hund Platz 1, 37077, Göttingen, Germany

    Tomas Kulvicius, Christoph Kolodziejski & Florentin Wörgötter

  2. Department of Informatics, Vytautas Magnus University, Vileikos g. 8, Kaunas, Lithuania

    Minija Tamosiunaite

  3. Department of Electronics & Electrical Engineering, University of Glasgow, Glasgow, GT12 8LT, Scotland

    Bernd Porr

Authors
  1. Tomas Kulvicius
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Kolodziejski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Minija Tamosiunaite
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bernd Porr
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Florentin Wörgötter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tomas Kulvicius.

Additional information

Tomas Kulvicius and Christoph Kolodziejski have contributed equally to this work.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Kulvicius, T., Kolodziejski, C., Tamosiunaite, M. et al. Behavioral analysis of differential hebbian learning in closed-loop systems. Biol Cybern 103, 255–271 (2010). https://doi.org/10.1007/s00422-010-0396-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-010-0396-4

Keywords

  • Adaptive systems
  • Sensorimotor loop
  • Learning and plasticity
  • Entropy
  • Input/output ratio
  • Energy
  • Optimal agents