Abstract
When monkeys tackle novel complex behavioral tasks by trial-and-error they select actions from repertoires of sensorimotor primitives that allow them to search solutions in a space which is coarser than the space of fine movements. Neuroscientific findings suggested that upper-limb sensorimotor primitives might be encoded, in terms of the final goal-postures they pursue, in premotor cortex. A previous work by the authors reproduced these results in a model based on the idea that cortical pathways learn sensorimotor primitives while basal ganglia learn to assemble and trigger them to pursue complex reward-based goals. This paper extends that model in several directions: a) it uses a Kohonen network to create a neural map with population encoding of postural primitives; b) it proposes an actor-critic reinforcement learning algorithm capable of learning to select those primitives in a biologically plausible fashion (i.e., through a dynamic competition between postures); c) it proposes a procedure to pre-train the actor to select promising primitives when tackling novel reinforcement learning tasks. Some tests (obtained with a task used for studying monkeys engaged in learning reaching-action sequences) show that the model is computationally sound and capable of learning to select sensorimotor primitives from the postures’ continuous space on the basis of their population encoding.
This research has been supported by the project “MindRACES – From Reactive to Anticipatory Cognitive Embodied Systems”, European Commission’s grant FP6-511931.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Aflalo, T.N., Graziano, M.S.A.: Partial Tuning of Motor Cortex Neurons to Final Posture in a Free-Moving Paradigm. Proceedings of the National Academy of Science 103(8), 2909–2914 (2006)
Arbib, M.: Visuomotor Coordination: From Neural Nets to Schema Theory. Cognition and Brain Theory 4, 23–39 (1981)
Baldassarre, G.: A Modular Neural-Network Model of the Basal Ganglia’s Role in Learning and Selecting Motor Behaviours. Journal of Cognitive Systems Research 3, 5–13 (2002)
Barto, A.G., Mahadevan, S.: Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems 13, 341–379 (2003)
Girard, B., Filliat, D., Meyer, J.-A., Berthoz, A., Guillot, A.: Integration of Navigation and Action Selection Functionalities in a Computational Model of Cortico-Basal Ganglia-Thalamo-Cortical Loops. Adaptive Behavior 13(2), 115–130 (2005)
Giszter, S.F., Mussa-Ivaldi, F.A., Bizzi, E.: Convergent Force Fields Organised in the Frog’s Spinal Cord. Journal of Neuroscience 13(2), 467–491 (1993)
Graybiel, A.M.: The Basal Ganglia and Chunking of Action Repertoires. Neurobiology of Learning and Memory 70, 119–136 (1998)
Graziano, M.S., Taylor, C.S., Moore, T.: Complex Movements Evoked by Microstimulation of Precentral Cortex. Neuron 34, 841–851 (2002)
Gurney, K., Prescott, T.J., Redgrave, P.: A Computational Model of Action Selection in the Basal Ganglia I. A New Functional Anatomy. Biological Cybernetics 84, 401–410 (2001)
Houk, J.C., Davis, J.L., Beiser, D.G. (eds.): Models of Information Processing in the Basal Ganglia. MIT Press, Cambridge (1995)
Joel, D.E.E., Niv, Y., Ruppin, E.: Actor-critic Models of the Basal Ganglia: New Anatomical and Computational Perspectives. Neural Networks 15, 535–547 (2002)
Kandel, E.R., Schwartz, J.H., Jessell, T.M.: Principles of Neural Science. McGraw-Hill, New York (2000)
Kohonen, T.: Self-Organizing Maps, 3rd edn. Springer, Heidelberg (2001)
Kuperstein, M.: A Neural Model of Adaptive Hand-Eye Coordination for Single Postures. Science 239, 1308–1311 (1988)
Meltzoff, A.N., Moore, M.K.: Explaining Facial Imitation: A Theoretical Model. Early Development and Parenting 6, 179–192 (1997)
Ognibene, D., Mannella, F., Pezzulo, G., Baldassarre, G.: Integrating Reinforcement-Learning, Accumulator Models, and Motor-Primitives to Study Action Selection and Reaching in Monkeys. In: Fum, D., Del Missier, F., Stocco, A. (eds.) Proceedings of the 7th International Conference on Cognitive Modelling - ICCM 2006, pp. 214–219 (2006)
Pasupathy, A., Miller, E.K.: Different Time Courses of Learning-Related Activity In the Prefrontal Cortex and Striatum. Nature 433, 873–876 (2005)
Pouget, A., Lathaam, P.E.: Population Codes. In: Arbib, M.A. (ed.) The Handbook of Brain Theory and Neural Networks, pp. 893–897. MIT Press, Cambridge (2003)
Rand, M.K., Hikosaka, O., Miyachi, S., Lu, X., Miyashita, K.: Characteristics of a Long-Term Procedural Skill in the Monkey. Experimental Brain Research 118, 293–297 (1998)
Widrow, B., Hoff, M.E.: Adaptive Switching Circuits. IRE WESCON Convention Record 4, 96–104 (1960)
Schall, J.D.: Neural Basis of Deciding, Choosing and Acting. Nature Reviews Neuroscience 2, 33–42 (2001)
Shadmehr, R., Wise, S.: The Computational Neurobiology of Reaching and Pointing. MIT Press, Cambridge (2005)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Usher, M., McClelland, J.L.: On the Time Course of Perceptual Choice: The Leaky Competing Accumulator Model. Psychological Review 108, 550–592 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ognibene, D., Rega, A., Baldassarre, G. (2006). A Model of Reaching that Integrates Reinforcement Learning and Population Encoding of Postures. In: Nolfi, S., et al. From Animals to Animats 9. SAB 2006. Lecture Notes in Computer Science(), vol 4095. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11840541_32
Download citation
DOI: https://doi.org/10.1007/11840541_32
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-38608-7
Online ISBN: 978-3-540-38615-5
eBook Packages: Computer ScienceComputer Science (R0)