Biological Cybernetics

, Volume 93, Issue 2, pp 119–130 | Cite as

Learning visuomotor transformations for gaze-control and grasping

  • Heiko Hoffmann
  • Wolfram Schenck
  • Ralf Möller


For reaching to and grasping of an object, visual information about the object must be transformed into motor or postural commands for the arm and hand. In this paper, we present a robot model for visually guided reaching and grasping. The model mimics two alternative processing pathways for grasping, which are also likely to coexist in the human brain. The first pathway directly uses the retinal activation to encode the target position. In the second pathway, a saccade controller makes the eyes (cameras) focus on the target, and the gaze direction is used instead as positional input. For both pathways, an arm controller transforms information on the target’s position and orientation into an arm posture suitable for grasping. For the training of the saccade controller, we suggest a novel staged learning method which does not require a teacher that provides the necessary motor commands. The arm controller uses unsupervised learning: it is based on a density model of the sensor and the motor data. Using this density, a mapping is achieved by completing a partially given sensorimotor pattern. The controller can cope with the ambiguity in having a set of redundant arm postures for a given target. The combined model of saccade and arm controller was able to fixate and grasp an elongated object with arbitrary orientation and at arbitrary position on a table in 94% of trials.


Target Position Combine Model Density Model Unsupervised Learning Motor Command 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Batista, AP, Buneo, CA, Snyder, LH, Andersen, RA 1999Reach plans in eye-centered coordinatesScience285257260CrossRefPubMedGoogle Scholar
  2. Blasdel, GG, Salama, G 1986Voltage-sensitive dyes reveal a modular organization in monkey striate cortexNature321579585CrossRefPubMedGoogle Scholar
  3. Bridgeman, B 1995A review of the role of efference copy in sensory and oculomotor control systemsAnn Biomed Eng23409422PubMedGoogle Scholar
  4. Buneo, CA, Jarvis, MR, Batista, AP, Andersen, RA 2002Direct visuomotor transformations for reachingNature416632636CrossRefPubMedGoogle Scholar
  5. Carey, DP, Coleman, RJ, Sala, SD 1997Magnetic misreachingCortex33639652PubMedGoogle Scholar
  6. Carey, DP, Sala, SD, Ietswaart, M 2002Neuropsychological perspectives on eye-hand coordination in visually-guided reaching. In: Progress in brain research vol 140ElsevierAmsterdam311327Google Scholar
  7. Cipolla, R, Hollinghurst, N 1997Visually guided grasping in unstructured environmentsRob Auton Syst19337346CrossRefGoogle Scholar
  8. Crawford, JD, Guitton, D 1997Visual-motor transformations required for accurate and kinematically correct saccadesJ Neurophysiol7814471467PubMedGoogle Scholar
  9. Crawford, JD, Medendorp, WP, Marotta, JJ 2004Spatial transformation for eye-hand coordinationJ Neurophysiol921019CrossRefPubMedGoogle Scholar
  10. Cruse, H, Steinkühler, U 1993Solution of the direct and inverse kinematic problems by a common algorithm based on the mean of multiple computationsBiol Cybern69345351CrossRefGoogle Scholar
  11. Diamantaras, KI, Kung, SY 1996Principal component neural networksWileyNew YorkGoogle Scholar
  12. Distante, C, Anglani, A, Taurisano, F 2000Target reaching by using visual information and Q-learning controllersAuton Robots94150CrossRefGoogle Scholar
  13. Fuentes, O, Nelson, RC 1998Learning dextrous manipulation skills for multifingered robot hands using the evolution strategyMach Learn31223237CrossRefGoogle Scholar
  14. Graziano, MS, Taylor, CS, Moore, T 2002Complex movements evoked by microstimulation of precentral cortexNeuron34841851CrossRefPubMedGoogle Scholar
  15. Hinton, GE, Dayan, P, Revow, M 1997Modeling the manifolds of images of handwritten digitsIEEE Trans Neural Netw86574CrossRefGoogle Scholar
  16. Hoffmann, H, Möller, R 2003Unsupervised learning of a kinematic arm modelKaynak, OAlpaydin, EOja, EXu, L eds. Artificial neural networks and neural information processing – ICANN/ICONIP, LNCS, vol 2714SpringerBerlin Heidelberg New York463470Google Scholar
  17. Hopfield, JJ 1982Neural networks and physical systems with emergent collective computational abilitiesProc Nat Acad Sci USA7925542558PubMedGoogle Scholar
  18. Hubel, DH, Wiesel, TN 1962Receptive fields, binocular interaction and functional architecture in the cat’s visual cortexJ Physiol160106154PubMedGoogle Scholar
  19. Jordan, MI, Rumelhart, DE 1992Forward models: supervised learning with a distal teacherCogn Sci16307354CrossRefGoogle Scholar
  20. Kawato, M 1990Feedback-error-learning neural network for supervised motor learningEckmiller, R eds. Advanced neural computersElsevierNorth-Holland365372Google Scholar
  21. Kawato, M, Furukawa, K, Suzuki, R 1987A hierarchical neural-network model for control and learning of voluntary movementBiol Cybern57169185CrossRefPubMedGoogle Scholar
  22. Klier, EM, Crawford, JD 1998Human oculomotor system accounts for 3-D eye orientation in the visual-motor transformation for saccadesJ Neurophysiol8022742294PubMedGoogle Scholar
  23. Kuperstein, M 1988Neural model of adaptive hand-eye coordination for single posturesScience23913081311PubMedGoogle Scholar
  24. Kuperstein, M 1990INFANT neural controller for adaptive sensory-motor coordinationNeural Netw4131145CrossRefGoogle Scholar
  25. Lacquaniti, F, Caminiti, R 1998Visuo-motor transformations for arm reachingEur J Neurosci10195203CrossRefPubMedGoogle Scholar
  26. Latham, PE, Deneve, S, Pouget, A 2003Optimal computation with attractor networksJ Physiol97683694Google Scholar
  27. Leigh, RJ, Zee, DS 1999The neurology of eye movementsOxford University PressNew YorkGoogle Scholar
  28. Martinetz, TM, Berkovich, SG, Schulten, KJ 1993‘‘Neural-Gas’’ network for vector quantization and its application to time-series predictionIEEE Trans Neural Netw4558569CrossRefGoogle Scholar
  29. Miller, JP, Jacobs, GA, Theunissen, FE 1991Representation of sensory information in the cricket cercal sensory system I. Response properties of the primary interneuronsJ Neurophysiol6616801689PubMedGoogle Scholar
  30. Molina-Vilaplana, J, Pedreño-Molina, JL, López-Coronado, J 2004Hyper RBF model for accurate reaching in redundant robotic systemsNeurocomputing61495501CrossRefGoogle Scholar
  31. Möller, R, Hoffmann, H 2004An extension of neural gas to local PCANeurocomputing62305326CrossRefGoogle Scholar
  32. Movellan, JR, McClelland, JL 1993Learning continuous probability distributions with symmetric diffusion networksCogn Sci17463496CrossRefGoogle Scholar
  33. Ouyang, S, Bao, Z, Liao, GS 2000Robust recursive least squares learning algorithm for principal component analysisIEEE Trans Neural Netw11215221CrossRefGoogle Scholar
  34. Oztop, E, Bradley, NS, Arbib, MA 2004Infant grasp learning: a computational modelExp Brain Res158480503CrossRefPubMedGoogle Scholar
  35. Riedmiller M, BraunH(1993)A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: Proceedings of the IEEE international conference on neural networks. San Francisco, pp 586–591Google Scholar
  36. Ritter, HJ 1993Parametrized self-organizing maps. In: Gielen S Kappen B (eds) Proceedings of the international conference on artificial neural networksSpringerBerlin Heidelberg New York568575Google Scholar
  37. Ritter, HJ, Martinetz, TM, Schulten, KJ 1989Topology-conserving maps for learning visuo-motor-coordinationNeural Netw2159168CrossRefGoogle Scholar
  38. Rizzolatti, G, Fadiga, L 1998Grasping objects and grasping action meanings: the dual role of monkey rostroventral premotor cortex (area F5)Novartis Found Symp21881103PubMedGoogle Scholar
  39. Ruskell, GL 1999Extraocular muscle proprioceptors and proprioceptionProg Retin Eye Res18269291CrossRefPubMedGoogle Scholar
  40. Salganicoff, M, Ungar, LH, Bajcsy, R 1996Active learning for vision-based robot graspingMach Learn23251278Google Scholar
  41. Schenck, W, Möller, R 2004Staged learning of saccadic eye movements with a robot camera headBowman, HLabiouse, C eds. Connectionist models of cognition and perception IIWorld ScientificLondon8291Google Scholar
  42. Schenck W, Hoffmann H, Möller R (2003) Learning internal models for eye-hand coordination in reaching and grasping. In: Proceedings of the European cognitive science conference, Erlbaum, Mahwah, pp 289-294Google Scholar
  43. Snyder, LH 2000Coordinate transformations for eye and arm movements in the brainCurr Opin Neurobiol10747754CrossRefPubMedGoogle Scholar
  44. Snyder, LH, Batista, AP, Andersen, RA 2000Saccade-related activity in the parietal reach regionJ Neurophysiol8310991102PubMedGoogle Scholar
  45. Sparks, DL, Mays, LE 1990Signal transformations required for the generation of saccadic eye movementsAnn Rev Neurosci13309336CrossRefPubMedGoogle Scholar
  46. Steinkühler, U, Cruse, H 1998A holistic model for an internal representation to control the movement of a manipulator with redundant degrees of freedomBiol Cybern79457466CrossRefGoogle Scholar
  47. Tipping, ME, Bishop, CM 1999Mixtures of probabilistic principal component analyzersNeural Comput11443482CrossRefPubMedGoogle Scholar
  48. Treue, S, Trujillo, JCM 1999Feature-based attention influences motion processing gain in macaque visual cortexNature399575579CrossRefPubMedGoogle Scholar
  49. Umbaugh SE (1998) Computer vision and image processing: a practical approach using CVIPtools. Prentice Hall.Google Scholar
  50. Uno, Y, Fukumura, N, Suzuki, R, Kawato, M 1995A computational model for recognizing objects and planning hand shapes in grasping movementsNeural Netw8839851CrossRefGoogle Scholar
  51. Walter JA, Nölker C, Ritter H (2000) The PSOM algorithm and applications. In: Proceedings of the symposium on neural computation, pp 758–764Google Scholar
  52. Wolpert, DM, Ghahramani, Z, Jordan, MI 1995Are arm trajectories planned in kinematic or dynamic coordinates? An adaptation studyExp Brain Res103460470CrossRefPubMedGoogle Scholar
  53. Zipser, D, Andersen, RA 1988A back-propagation programmed network that simulates response properties of a subset of posterior parietal neuronsNature331679684CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  1. 1.Department of Psychology, Max Planck Institute for Human Cognitive and Brain SciencesCognitive RoboticsMunichGermany
  2. 2.Computer Engineering Group, Faculty of TechnologyBielefeld UniversityBielefeldGermany

Personalised recommendations