Abstract
Understanding closed loop behavioral systems is a non-trivial problem, especially when they change during learning. Descriptions of closed loop systems in terms of information theory date back to the 1950s, however, there have been only a few attempts which take into account learning, mostly measuring information of inputs. In this study we analyze a specific type of closed loop system by looking at the input as well as the output space. For this, we investigate simulated agents that perform differential Hebbian learning (STDP). In the first part we show that analytical solutions can be found for the temporal development of such systems for relatively simple cases. In the second part of this study we try to answer the following question: How can we predict which system from a given class would be the best for a particular scenario? This question is addressed using energy, input/output ratio and entropy measures and investigating their development during learning. This way we can show that within well-specified scenarios there are indeed agents which are optimal with respect to their structure and adaptive properties.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Ashby WR (1956) An introduction to cybernetics. Chapmann and Hall Ltd., London
Ay N, Bertschinger N, Der R, Güttler F, Olbrich E (2008) Predictive information and explorative behavior of autonomous robots. Eur Phys J B 63: 329–339
Baldwin JM (1896) A new factor in evolution. Am Nat 30: 441–451
Bi GQ, Poo MM (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18: 10464–10472
Box G, Jenkins GM, Reinsel GC (1994) Time series analysis: forecasting and control. Prentice-Hall, Englewood Cliffs, NJ
Braitenberg V (1986) Vehicles: experiments in synthetic psychology. The MIT Press, Cambridge, MA
Der R, Güttler F, Ay N (2008) Predictive information and emergent cooperativity in a chain of mobile robots. In: Bullock S, Noble J, Watson R, Bedau MA (eds) Artificial life XI: proceedings of the eleventh international conference on the simulation and synthesis of living systems.. MIT Press, Cambridge, MA, pp 166–172
Hebb DO (1949) The organization of behavior. Wiley, New York
Hinton GE, Nowlan SJ (1987) How learning guides evolution. Complex Syst 1: 495–502
Hofstötter C, Mintz M, Verschure PF (2002) The cerebellum in action: a simulation and robotics study. Eur J Neurosci 16: 1361–1376
Iglesias R, Nehmzow U, Billings SA (2008) Model identification and model analysis in robot training. Robot Auton Syst 56: 1061–1067
Klopf AH (1988) A neuronal model of classical conditioning. Psychobiology 16(2): 85–123
Klyubin AS, Polani D, Nehaniv CL (2004) Organization of the information flow in the perception-action loop of evolved agents. In: 2004 NASA/DoD conference on evolvable hardware. IEEE Computer Society, pp 177–180
Klyubin AS, Polani D, Nehaniv CL (2005) Empowerment: a universal agent-centric measure of control. In: IEEE congress on evolutionary computation (CEC 2005), pp 128–135
Klyubin AS, Polani D, Nehaniv CL (2007) Representations of space and time in the maximization of information flow in the perception-action loop. Neural Comput 19: 2387–2432
Klyubin AS, Polani D, Nehaniv CL (2008) Keep your options open: an information-based driving principle for sensorimotor systems. PLoS ONE 3: e4018
Kosco B (1986) Differential Hebbian learning. In: Denker JS (eds) Neural networks for computing: AIP conference proceedings, vol 151. American Institute of Physics, New York
Kulvicius T, Porr B, Wörgötter F (2007) Chained learning architectures in a simple closed-loop behavioural context. Biol Cybern 97: 363–378
Kyriacou T, Nehmzow U, Iglesias R, Billings SA (2008) Accurate robot simulation through system identification. Robot Auton Syst 56: 1082–1093
Lungarella M, Pegors T, Bulwinkle D, Sporns O (2005) Methods for quantifying the informational structure of sensory and motor data. Neuroinformatics 3: 243–262
Lungarella M, Sporns O (2006) Mapping information flow in sensorimotor networks. PLoS Comput Biol 2: e144
Markram H, Lübke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275: 213–215
Porr B, Wörgötter F (2003a) Isotropic sequence order learning. Neural Comput 15: 831–864
Porr B, Wörgötter F (2003b) Isotropic-sequence-order learning in a closed-loop behavioural system. Philos Transact A Math Phys Eng Sci 361: 2225–2244
Porr B, Wörgötter F (2006) Strongly improved stability and faster convergence of temporal sequence learning by using input correlations only. Neural Comput 18: 1380–1412
Porr B, Egerton A, Wörgötter F (2006) Towards closed loop information: Predictive information. Constr Found 1(2): 83–90
Poupart P, Boutilier C (2002) Value-directed compression of POMDPs. In: Becker STS, Obermayer K (eds) Advances in neural information processing systems, vol 15. pp 1547–1554
Prokopenko M, Gerasimov V, Tanev I (2006) Evolving spatiotemporal coordination in a modular robotic system. In: SAB 2006. pp 558–569
Saudargiene A, Porr B, Wörgötter F (2004) How the shape of pre- and postsynaptic signals can influence STDP: a biophysical model. Neural Comput 16: 595–625
Saudargiene A, Porr B, Wörgötter F (2005) Synaptic modifications depend on synapse location and activity: a biophysical model of STDP. BioSystems 79: 3–10
Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27: 379–423
Slonim N, Tishby N (2000) Document clustering using word clusters via the information bottleneck method. In: Proceedings of the 23rd annual international acm-sigir conference on research and development in information retrieval
Slonim N, Tishby N (2001) The power of word clustering for text classification. In: Proceedings of the 23rd European colloquium on information retrieval research
Slonim N, Somerville R, Tishby N, Lahav O (2001) Objective classification of galaxy spectra using the information bottleneck method. Mon Notes R Astron Soc 323: 270–284
Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88: 135–170
Tishby N, Pereira FC, Bialek W (1999) The information bottleneck method. In: Proceedings of the 37-th annual allerton conference on communication, control and computing. pp 368–377
Touchette H, Lloyd S (2000) Information-theoretic approach to the study of control systems. Physica A 331: 140–172
Wolpert DM, Miall RC, Kawato M (1998) Internal models in the cerebellum. Trends Cogn Sci 2: 338–347
Acknowledgements
This research was supported by the European funded PACO-PLUS project as well as by BMBF (Federal Ministry of Education and Research), BCCN (Bernstein Center for Computational Neuroscience)—Göttingen project W3 and BFNT project 3a
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Additional information
Tomas Kulvicius and Christoph Kolodziejski have contributed equally to this work.
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Kulvicius, T., Kolodziejski, C., Tamosiunaite, M. et al. Behavioral analysis of differential hebbian learning in closed-loop systems. Biol Cybern 103, 255–271 (2010). https://doi.org/10.1007/s00422-010-0396-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00422-010-0396-4