The SURE_REACH Model for Motor Learning and Control of a Redundant Arm: From Modeling Human Behavior to Applications in Robotics

  • Oliver Herbort
  • Martin V. Butz
  • Gerulf Pedersen

Abstract

The recently introduced neural network SURE_REACH (sensorimotor unsupervised redundancy resolving control architecture) models motor cortical learning and control of human reaching movements. The model learns redundant, internal body models that are highly suitable to flexibly invoke effective motor commands. The encoded redundancy is used to adapt behavior flexible to situational constraints without the need for further learning. These adaptations to specific tasks or situations are realized by a neurally generated movement plan that adheres to various end-state or trajectory-related constraints. The movement plan can be implemented by proprioceptive or visual closed-loop control. This chapter briefly reviews the literature on computational models of motor learning and control and gives a description of SURE_REACH and its neural network implementation. Furthermore, we relate the model to human motor learning and performance and discuss its neural foundations. Finally, we apply the model to the control of a dynamic robot platform. In sum, SURE_REACH grounds highly flexible task-dependent behavior on a neural network framework for unsupervised learning. It accounts for the neural processes that underlie fundamental aspects of human behavior and is well applicable to the control of robots.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Adams, J.A.: A closed-loop theory of motor learning. J. Mot. Behav. 3(2), 111–149 (1971)Google Scholar
  2. 2.
    Aflalo, T.N., Graziano, M.S.A.: Partial tuning of motor cortex neurons to final posture in a free-moving paradigm. Proc. Natl. Acad. of Sci. 8, 2909–2914 (2006)CrossRefGoogle Scholar
  3. 3.
    Baraduc, P., Guigon, E., Burnod, Y.: Where does the population vector of motor cortical cells point during reaching movements? In: Kearns, M., Solla, S., Cohn, D. (eds.) Advances in Neural Information Processing Systems, vol. 11, pp. 83–89. MIT Press, Cambridge (1999)Google Scholar
  4. 4.
    Barto, A.G., Fagg, A.H., Sitkoff, N., Houk, J.C.: A cerebellar model of timing and prediction in the control of reaching. Neural Comp. 11, 565–594 (1999)CrossRefGoogle Scholar
  5. 5.
    Bastian, A., Schöner, G., Riehle, A.: Preshaping and continuous evolution of motor cortical representations during movement preparation. Eur. J. Neurosci. 18, 2047–2058 (2003)CrossRefGoogle Scholar
  6. 6.
    Bernstein, N.A.: The co-ordination and regulation of movements. Pergamon Press, Oxford (1967)Google Scholar
  7. 7.
    Berthier, N.E., Rosenstein, M.T., Barto, A.G.: Approximate optimal control as a model for motor learning. Psychol. Rev. 112(2), 329–346 (2005)CrossRefGoogle Scholar
  8. 8.
    Bowers, J.S.: On the biological plausibility of grandmother cells: Implications for neural network theories in psychology and neuroscience. Psychol. Rev. 116(1), 220–251 (2009)CrossRefGoogle Scholar
  9. 9.
    Bullock, D., Grossberg, S., Guenther, F.H.: A self-organizing neural model of motor equivalent reaching and tool use by a multijoint arm. J. Cogn. Neurosci. 5(4), 408–435 (1993)CrossRefGoogle Scholar
  10. 10.
    Butz, M.V., Herbort, O., Hoffmann, J.: Exploiting redundancy for flexible behavior: Unsupervised learning in a modular sensorimotor control architecture. Psychol. Rev. 114(4), 1015–1046 (2007)CrossRefGoogle Scholar
  11. 11.
    Butz, M.V., Herbort, O., Pezzulo, G.: Anticipatory, goal-directed behavior. In: Pezzulo, G., Butz, M.V., Castelfranchi, C., Falcone, R. (eds.) The Challenge of Anticipation. LNCS (LNAI), vol. 5225, pp. 85–113. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Cisek, P.: Integrated neural processes for defining potential actions and deciding between them: A computational model. J. Neurosci. 26(38), 9761–9770 (2006)CrossRefGoogle Scholar
  13. 13.
    Cisek, P., Kalaska, J.F.: Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action. Neuron 45(5), 801–814 (2005)CrossRefGoogle Scholar
  14. 14.
    Cruse, H., Steinkühler, U.: Solution of the direct and inverse kinematic problems by a common algorithm based on the mean of multiple computations. Biol. Cybern. 69, 341–351 (1993)CrossRefGoogle Scholar
  15. 15.
    Cruse, H., Steinkühler, U., Burkamp, C.: MMC - a recurrent neural network which can be used as manipulable body model. In: Pfeifer, R., Blumberg, B., Meyer, J.A., Wilson, S. (eds.) From Animals to Animats 5: The Fifth International Conference on the Simulation of Adaptive Behavior, pp. 381–389. MIT Press, Cambridge (1998)Google Scholar
  16. 16.
    de Freitas, S.M.S.F., Scholz, J.P., Stehman, A.J.: Effect of motor planning on use of motor abundance. Neurosci. Lett. 417(1), 66–71 (2007)CrossRefGoogle Scholar
  17. 17.
    Dean, J., Brüwer, M.: Control of human arm movements in two dimensions: Paths and joint control in avoiding simple linear obstacles. Exp. Brain Res. 97, 497–514 (1994)CrossRefGoogle Scholar
  18. 18.
    Desmurget, M., Grafton, S.: Forward modeling allows feedback control for fast reaching movements. Trends Cogn. Sci. 4(11), 423–431 (2000)CrossRefGoogle Scholar
  19. 19.
    Doya, K.: Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10(6), 732–739 (2000)CrossRefGoogle Scholar
  20. 20.
    Elliott, D., Helsen, W.F., Chua, R.: A century later: Woodworth’s two-component model of goal-directed aiming. Psychol. Bull. 127(3), 342–357 (1899)CrossRefGoogle Scholar
  21. 21.
    Elsner, B., Hommel, B.: Effect anticipations and action control. J. Exp. Psychol. 27(1), 229–240 (2001)Google Scholar
  22. 22.
    Fischer, M.H., Rosenbaum, D.A., Vaughan, J.: Speed and sequential effects in reaching. J. Exp. Psychol.: Hum. Percept. Perform. 23(2), 404–428 (1997)CrossRefGoogle Scholar
  23. 23.
    Flash, T., Hogan, N.: The coordination of arm movements: An experimentally confirmed mathematical model. J. Neurosci. 5(7), 1688–1703 (1985)Google Scholar
  24. 24.
    Georgopoulos, A.P.: Current issues in directional motor control. Trends Neurosci. 18(11), 506–510 (1995)CrossRefGoogle Scholar
  25. 25.
    Hallett, M.A., Pascual-Leone, A., Topka, H.: The acquisition of motor behavior in vertebrates. In: Bloedel, J.R., Ebner, T.J., Wise, S.P. (eds.) Adaptation and skill learning: Evidence for different neural substrates, pp. 289–301. MIT Press, Cambridge (1996)Google Scholar
  26. 26.
    Haruno, M., Wolpert, D.M., Kawato, M.: Hierarchical mosaic for movement generation. In: Ono, T., Matsumoto, G., Llinas, R., Berthoz, A., Norgren, R., Nishijo, H., Tamura, R. (eds.) Excepta Medica International Congress Series, vol. 1250 (2003)Google Scholar
  27. 27.
    Herbart, J.F.: Psychologie als Wissenschaft neu gegründet auf Erfahrung, Metaphysik und Mathematik. In: Zweiter analytischer Teil [Psychology as a Science newly founded on Experience, Metaphysics and Mathematics: Second, Analytical Part]. August Wilhelm Unzer., Königsberg (1825)Google Scholar
  28. 28.
    Herbort, O., Butz, M.V.: Encoding complete body models enables task dependent optimal control. Proc. Int. Jt. Conf. Neural Netw. 20, 1639–1644 (2007)CrossRefGoogle Scholar
  29. 29.
    Herbort, O., Butz, M.V.: Anticipatory planning of sequential hand and finger movements. J. Mot. Behav. (in press) Google Scholar
  30. 30.
    Herbort, O., Butz, M.V., Hoffmann, J.: Multimodal goal representations and feedback in hierarchical motor control. In: Proc. Int. Conf. Cogn. Syst. (2008)Google Scholar
  31. 31.
    Herbort, O., Ognibene, D., Butz, M.V., Baldassarre, G.: Learning to select targets within targets in reaching tasks. In: Proc. 6th Int. IEEE Conf. Dev. Learn., vol. 6, pp. 7–12 (2007)Google Scholar
  32. 32.
    Hikosaka, O., Nakamura, K., Sakai, K., Nakahara, H.: Central mechanisms of motor skill learning. Curr. Opin. Neurobiol. 12, 217–222 (2002)CrossRefGoogle Scholar
  33. 33.
    Hoffmann, J.: Vorhersage und Erkenntnis: Die Funktion von Antizipationen in der menschlichen Verhaltenssteuerung und Wahrnehmung [Anticipation and cognition: The function of anticipations in human behavioral control and perception]. Hogrefe, Göttingen (1993)Google Scholar
  34. 34.
    Hoffmann, J.: Anticipatory behavior control. In: Butz, M.V., Sigaud, O., Gérard, P. (eds.) Anticipatory Behavior in Adaptive Learning Systems. LNCS (LNAI), vol. 2684, pp. 44–65. Springer, Heidelberg (2003)Google Scholar
  35. 35.
    Hoffmann, J., Butz, M.V., Herbort, O., Kiesel, A., Lenhard, A.: Spekulationen zur Struktur ideo-motorischer Beziehungen [Speculations about the structure of ideomotor relations]. Z. Sportpsychol. 14(3), 95–103 (2007)CrossRefGoogle Scholar
  36. 36.
    Jackson, A., Mavoori, J., Fetz, E.E.: Long-term motor cortex plasticity induced by an electronic neural implant. Nat. 444, 56–60 (2006)CrossRefGoogle Scholar
  37. 37.
    James, W.: The principles of psychology, vol. 1. Holt, New York (1890)Google Scholar
  38. 38.
    Johnson-Frey, S.H., McCarty, M.E., Keen, R.: Reaching beyond spatial perception: Effects of intended future actions on visually guided prehension. Vis. Cogn. 11(2-3), 371–399 (2004)CrossRefGoogle Scholar
  39. 39.
    Jordan, M.I., Wolpert, D.M.: Computational motor control. In: Gazzaniga (ed.) The Cognitive Neuroscience, pp. 601–620. MIT Press, Cambridge (1999)Google Scholar
  40. 40.
    Karniel, A., Inbar, G.F.: A model for learning human reaching movements. Biol. Cybern. 77, 173–183 (1997)MATHCrossRefGoogle Scholar
  41. 41.
    Kawato, M.: Feedback-error-learning neural network for supervised learning. In: Eckmiller, R. (ed.) Advanced neural computers, pp. 365–372. North-Holland, Amsterdam (1990)Google Scholar
  42. 42.
    Kawato, M.: Internal models for motor control and trajectory planning. Curr. Opin. Neurobiol. 9, 718–727 (1999)CrossRefGoogle Scholar
  43. 43.
    Kawato, M., Furukawa, K., Suzuki, R.: A hierarchical neural-network model for control and learning of voluntary movement. Biol. Cybern. 57, 169–185 (1987)MATHCrossRefGoogle Scholar
  44. 44.
    Khan, M.A., Franks, I.M., Goodman, D.: The effect of practice on the control of rapid aiming movements: Evidence for an interdependency between programming and feedback processing. Q. J. Exp. Psychol. Section A 51(2), 425–443 (1998)Google Scholar
  45. 45.
    Klapp, S.T., Erwin, C.I.: Relation between programming time and duration of the response being programmed. J. Exp. Psychol.: Hum. Percept. Perform. 2(4), 591–598 (1976)CrossRefGoogle Scholar
  46. 46.
    Klein Breteler, M.D., Hondzinski, J.M., Flanders, M.: Drawing sequences of segments in 3d: Kinetic influences on arm configuration. J. Neurophysiol. 89, 3253–3263 (2003)CrossRefGoogle Scholar
  47. 47.
    Körding, K.P., Wolpert, D.M.: Bayesian integration in sensorimotor learning. Nat. 427, 244–247 (2004)CrossRefGoogle Scholar
  48. 48.
    Kunde, W., Koch, I., Hoffmann, J.: Anticipated action effects affect the selection, initiation, and execution of actions. Q. J. Exp. Psychol. Section A: Human Exp. Psychol. 57, 87–106 (2004)CrossRefGoogle Scholar
  49. 49.
    Kuperstein, M.: Neural model of adaptive hand-eye coordination for single postures. Sci. 239, 1308–1311 (1988)CrossRefGoogle Scholar
  50. 50.
    Latash, M.L., Scholz, J.P., Schöner, G.: Motor control strategies revealed in the structure of motor variability. Exerc. & Sport Sci. Rev. 30(1), 26–31 (2002)CrossRefGoogle Scholar
  51. 51.
    Lavrysen, A., Helsen, W.F., Tremblay, L., Elliott, D., Adam, J.J., Feys, P., Buekers, M.J.: The control of sequential aiming movements: The influence of practice and manual asymmetries on the one-target advantage. Cortex 39, 307–325 (2003)CrossRefGoogle Scholar
  52. 52.
    Lotze, H.R.: Medicinische Psychologie oder Physiologie der Seele [Medical Psychology or Physiology of the Soul]. Weidmannsche Buchhandlung, Leipzig (1852)Google Scholar
  53. 53.
    Ludwig, D.A.: Emg changes during the acquisition of a motor skill. Am. J. Phys. Medicine 61(5), 229–243 (1982)Google Scholar
  54. 54.
    Ma-Wyatt, A., McKee, S.P.: Visual information throughout a reach determines endpoint precision. Exp. Brain Res. 179(1), 55–64 (2007)CrossRefGoogle Scholar
  55. 55.
    Morasso, P., Sanguineti, V., Spada, G.: A computational theory of targeting movements based on force fields and topology representing networks. Neurocomputing 15(3-4), 411–434 (1997)CrossRefGoogle Scholar
  56. 56.
    Munro, H., Plumb, M.S., Wilson, A.D., Williams, J.H.G., Mon-Williams, M.: The effect of distance on reaction time in aiming movements. Exp. Brain Res. 183(2), 249–257 (2007)CrossRefGoogle Scholar
  57. 57.
    Mutsaarts, M., Steenbergen, B., Bekkering, H.: Anticipatory planning deficits and task context effects in hemiparetic cerebral palsy. Exp. Brain Res. 172(2), 151–162 (2006)CrossRefGoogle Scholar
  58. 58.
    Ognibene, D., Mannella, F., Pezzulo, G., Baldassarre, G.: Integrating reinforcement-learning, accumulator models, and motor-primitives to study action selection and reaching in monkeys. In: Fum, D., Del Missier, F., Stocco, A. (eds.) Proc. Seventh International Conference on Cognitive Modeling (ICCM 2006), pp. 214–219. Edizioni Goliardiche, Trieste (2006)Google Scholar
  59. 59.
    Pedersen, G., Butz, M.V., Herbort, O.: Integrating dynamics into a human behavior model for highly flexible autonomous manipulator control. IEEE Systems, Man & Cybernetics B (submitted)Google Scholar
  60. 60.
    Peters, J., Schaal, S.: Learning to control in operational space. Int. J. Robot. Res. 27, 197–212 (2008)CrossRefGoogle Scholar
  61. 61.
    Robertson, E.M., Miall, R.C.: Multi-joint limbs permit a flexible response to unpredictable events. Exp. Brain Res. 117, 148–152 (1997)CrossRefGoogle Scholar
  62. 62.
    Rosenbaum, D.A.: Human movement initiation: Specification of arm, direction and extent. J. Exp. Psychol.: Gen. 109, 444–474 (1980)CrossRefGoogle Scholar
  63. 63.
    Rosenbaum, D.A., Engelbrecht, S.E., Bushe, M.M., Loukopoulos, L.D.: A model for reaching control. Acta Psychol. 82(1-3), 237–250 (1993)CrossRefGoogle Scholar
  64. 64.
    Rosenbaum, D.A., Loukopoulos, L.D., Meulenbroek, R.G.J., Vaughan, J., Engelbrecht, S.E.: Planning reaches by evaluating stored postures. Psychol. Rev. 102(1), 28–67 (1995)CrossRefGoogle Scholar
  65. 65.
    Rosenbaum, D.A., Meulenbroek, R.G.J., Vaughan, J., Jansen, C.: Posture-based motion planning: Applications to grasping. Psychol. Rev. 108(4), 709–734 (2001)CrossRefGoogle Scholar
  66. 66.
    Salaun, C., Padois, V., Sigaud, O.: Learning forward models for the operational space control of redundant robots. In: Sigaud, O., Peters, J. (eds.) From Motor Learning to Interaction Learning in Robots. SCI, vol. 264, pp. 169–192. Springer, Heidelberg (2010)Google Scholar
  67. 67.
    Schmidt, R.A.: A schema theory of discrete motor skill-learning. Psychol. Rev. 82(4), 229–261 (1975)CrossRefGoogle Scholar
  68. 68.
    Schwartz, A.B., Moran, D.W., Reina, G.A.: Differential representation of perception and action in the frontal cortex. Sci. 303, 380–383 (2004)CrossRefGoogle Scholar
  69. 69.
    Shadmehr, R., Wise, S.P.: The Computational Neurobiology of Reaching and Pointing: A foundation for motor learning. MIT Press, Cambridge (2005)Google Scholar
  70. 70.
    Short, M.W., Cauraugh, J.H.: Precision hypothesis and the end-state comfort effect. Acta Psychol. 100(3), 243–252 (1999)CrossRefGoogle Scholar
  71. 71.
    Soechting, J.F., Buneo, C.A., Herrmann, U., Flanders, M.: Moving effortlessly in three dimensions: Does Donders’ law apply to arm movement? J. Neurosci. 15, 6271–6280 (1995)Google Scholar
  72. 72.
    Todorov, E., Jordan, M.I.: Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5(11), 1226–1235 (2002)CrossRefGoogle Scholar
  73. 73.
    Toussaint, M., Goerick, C.: A bayesian view on motor control and planning. In: Sigaud, O., Peters, J. (eds.) From Motor Learning to Interaction Learning in Robots. SCI, vol. 264, pp. 227–252. Springer, Heidelberg (2010)Google Scholar
  74. 74.
    Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement: Minimum torque-change model. Biol. Cybern. 61(2), 89–101 (1989)CrossRefGoogle Scholar
  75. 75.
    van Sonderen, J.F., Dernier van der Gon, J.J.: Reaction-time-dependent differences in the initial movement direction of fast goal-directed arm movements. Hum. Mov. Sci. 10(6), 713–726 (1991)CrossRefGoogle Scholar
  76. 76.
    von Hofsten, C.: An action perspective on motor development. Trends Cogn. Sci. 8(6), 266–272 (2004)CrossRefGoogle Scholar
  77. 77.
    Webots, C.L.: Commercial mobile robot simulation software, http://www.cyberbotics.com
  78. 78.
    Weigelt, M., Kunde, W., Prinz, W.: End-state comfort in bimanual object manipulation. Exp. Psychol. 53(2), 143–148 (2006)Google Scholar
  79. 79.
    Woodworth, R.S.: The Accuracy of Voluntary Movement. New Era, Lancaster (1899)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Oliver Herbort
    • 1
  • Martin V. Butz
    • 1
  • Gerulf Pedersen
    • 1
  1. 1.Department of PsychologyUniversität WürzburgWürzburgGermany

Personalised recommendations