A Model of Reaching that Integrates Reinforcement Learning and Population Encoding of Postures
When monkeys tackle novel complex behavioral tasks by trial-and-error they select actions from repertoires of sensorimotor primitives that allow them to search solutions in a space which is coarser than the space of fine movements. Neuroscientific findings suggested that upper-limb sensorimotor primitives might be encoded, in terms of the final goal-postures they pursue, in premotor cortex. A previous work by the authors reproduced these results in a model based on the idea that cortical pathways learn sensorimotor primitives while basal ganglia learn to assemble and trigger them to pursue complex reward-based goals. This paper extends that model in several directions: a) it uses a Kohonen network to create a neural map with population encoding of postural primitives; b) it proposes an actor-critic reinforcement learning algorithm capable of learning to select those primitives in a biologically plausible fashion (i.e., through a dynamic competition between postures); c) it proposes a procedure to pre-train the actor to select promising primitives when tackling novel reinforcement learning tasks. Some tests (obtained with a task used for studying monkeys engaged in learning reaching-action sequences) show that the model is computationally sound and capable of learning to select sensorimotor primitives from the postures’ continuous space on the basis of their population encoding.
Unable to display preview. Download preview PDF.
- 2.Arbib, M.: Visuomotor Coordination: From Neural Nets to Schema Theory. Cognition and Brain Theory 4, 23–39 (1981)Google Scholar
- 6.Giszter, S.F., Mussa-Ivaldi, F.A., Bizzi, E.: Convergent Force Fields Organised in the Frog’s Spinal Cord. Journal of Neuroscience 13(2), 467–491 (1993)Google Scholar
- 10.Houk, J.C., Davis, J.L., Beiser, D.G. (eds.): Models of Information Processing in the Basal Ganglia. MIT Press, Cambridge (1995)Google Scholar
- 12.Kandel, E.R., Schwartz, J.H., Jessell, T.M.: Principles of Neural Science. McGraw-Hill, New York (2000)Google Scholar
- 16.Ognibene, D., Mannella, F., Pezzulo, G., Baldassarre, G.: Integrating Reinforcement-Learning, Accumulator Models, and Motor-Primitives to Study Action Selection and Reaching in Monkeys. In: Fum, D., Del Missier, F., Stocco, A. (eds.) Proceedings of the 7th International Conference on Cognitive Modelling - ICCM 2006, pp. 214–219 (2006)Google Scholar
- 18.Pouget, A., Lathaam, P.E.: Population Codes. In: Arbib, M.A. (ed.) The Handbook of Brain Theory and Neural Networks, pp. 893–897. MIT Press, Cambridge (2003)Google Scholar
- 20.Widrow, B., Hoff, M.E.: Adaptive Switching Circuits. IRE WESCON Convention Record 4, 96–104 (1960)Google Scholar
- 22.Shadmehr, R., Wise, S.: The Computational Neurobiology of Reaching and Pointing. MIT Press, Cambridge (2005)Google Scholar
- 23.Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar