Biological Cybernetics

, Volume 108, Issue 2, pp 223–248 | Cite as

Learning robotic eye–arm–hand coordination from human demonstration: a coupled dynamical systems approach

  • Luka Lukic
  • José Santos-Victor
  • Aude Billard
Original Paper


We investigate the role of obstacle avoidance in visually guided reaching and grasping movements. We report on a human study in which subjects performed prehensile motion with obstacle avoidance where the position of the obstacle was systematically varied across trials. These experiments suggest that reaching with obstacle avoidance is organized in a sequential manner, where the obstacle acts as an intermediary target. Furthermore, we demonstrate that the notion of workspace travelled by the hand is embedded explicitly in a forward planning scheme, which is actively involved in detecting obstacles on the way when performing reaching. We find that the gaze proactively coordinates the pattern of eye–arm motion during obstacle avoidance. This study provides also a quantitative assessment of the coupling between the eye–arm–hand motion. We show that the coupling follows regular phase dependencies and is unaltered during obstacle avoidance. These observations provide a basis for the design of a computational model. Our controller extends the coupled dynamical systems framework and provides fast and synchronous control of the eyes, the arm and the hand within a single and compact framework, mimicking similar control system found in humans. We validate our model for visuomotor control of a humanoid robot.


Visuomotor coordination in reach and grasp movements Coupled dynamical systems Forward planning Obstacle avoidance 



This work was supported in part by EU projects POETICON++ (FP7-ICT-288382) and FIRST-MM (FP7-ICT 248258) and Fundação para a Ciência e a Tecnologia (FCT) doctoral grant (SFRH/BD/51072/2010) under IST-EPFL Joint Doctoral Initiative.


  1. Abrams R, Meyer D, Kornblum S (1990) Eye-hand coordination: oculomotor control in rapid aimed limb movements. J Exp Psychol Hum Percept Perform 16(2):248PubMedCrossRefGoogle Scholar
  2. Aivar M, Brenner E, Smeets J (2008) Avoiding moving obstacles. Exp Brain Res 190(3):251–264PubMedCrossRefGoogle Scholar
  3. Alberts JL, Saling M, Stelmach GE (2002) Alterations in transport path differentially affect temporal and spatial movement parameters. Exper Brain Res 143(4):417–425CrossRefGoogle Scholar
  4. Aloimonos J, Weiss I, Bandyopadhyay A (1988) Active vision. Int J Comput Vis 1(4):333–356CrossRefGoogle Scholar
  5. Andersen RA, Cui H (2009) Intention, action planning, and decision making in parietal–frontal circuits. Neuron 63(5):568–583PubMedCrossRefGoogle Scholar
  6. Bajcsy R (1988) Active perception. Proc IEEE 76(8):966–1005CrossRefGoogle Scholar
  7. Bajcsy R, Campos M (1992) Active and exploratory perception. CVGIP Image Underst 56(1):31–40CrossRefGoogle Scholar
  8. Baldauf D, Deubel H (2010) Attentional landscapes in reaching and grasping. Vis Res 50(11):999–1013PubMedCrossRefGoogle Scholar
  9. Ballard D (1991) Animate vision. Artif Intell 48(1):57–86CrossRefGoogle Scholar
  10. Ballard DH, Hayhoe MM, Pelz JB (1995) Memory representations in natural tasks. J Cogn Neurosci 7(1):66–80PubMedCrossRefGoogle Scholar
  11. Bendahan P, Gorce P (2006) A neural network architecture to learn arm motion planning in grasping tasks with obstacle avoidance. Robotica 24(2):197–204CrossRefGoogle Scholar
  12. Berthier NE, Clifton RK, Gullapalli V, McCall DD, Robin DJ (1996) Visual information and object size in the control of reaching. J Mot Behav 28(3):187–197PubMedCrossRefGoogle Scholar
  13. Bishop C (2007) Pattern recognition and machine learning (information science and statistics). Pattern Recognit 4(2):1–748 Google Scholar
  14. Bowman M, Johannson R, Flanagan J (2009) Eye-hand coordination in a sequential target contact task. Exp Brain Res 195(2):273–283PubMedCrossRefGoogle Scholar
  15. Brouwer A, Franz V, Gegenfurtner K (2009) Differences in fixations between grasping and viewing objects. J Vis 9(1):1–8Google Scholar
  16. Castiello U, Bennett K, Mucignat C (1983) The reach to grasp movement of blind subjects. Exp Brain Res 96(1):152–162CrossRefGoogle Scholar
  17. Castiello U, Bennett K, Stelmach G (1993) Reach to grasp: the natural response to perturbation of object size. Exp Brain Res 94(1):163–178PubMedCrossRefGoogle Scholar
  18. Chaumette F, Hutchinson S (2008) Visual servoing and visual tracking. In: Siciliano B, Khatib O (eds) Springer Handbook of Robotics. Springer, Berlin, Heidelberg, pp 563–583Google Scholar
  19. Dalton K, Nacewicz B, Johnstone T, Schaefer H, Gernsbacher M, Goldsmith H, Alexander A, Davidson R (2005) Gaze fixation and the neural circuitry of face processing in autism. Nat Neurosci 8(4):519–526PubMedGoogle Scholar
  20. Dean J, Brüwer M (1994) Control of human arm movements in two dimensions: paths and joint control in avoiding simple linear obstacles. Exp Brain Res 97(3):497–514PubMedCrossRefGoogle Scholar
  21. Deubel H, O’Regan JK, Radach R (2000) Attention, information processing, and eye movement control. In: Kennedy A, ii Radach R, Heller D, Pynte J (eds) Reading as a perceptual process. Elsevier, Oxford, pp 355–374Google Scholar
  22. Engbert R, Kliegl R et al (2003) Microsaccades uncover the orientation of covert attention. Vis Res 43(9):1035–1045PubMedCrossRefGoogle Scholar
  23. Espiau B, Chaumette F, Rives P (1992) A new approach to visual servoing in robotics. IEEE Trans Robot Autom 8(3):313–326CrossRefGoogle Scholar
  24. Fisk J, Goodale M (1985) The organization of eye and limb movements during unrestricted reaching to targets in contralateral and ipsilateral visual space. Exp Brain Res 60(1):159–178PubMedCrossRefGoogle Scholar
  25. Gentilucci M, Toni I, Chieffi S, Pavesi G (1994) The role of proprioception in the control of prehension movements: a kinematic study in a peripherally deafferented patient and in normal subjects. Exp Brain Res 99(3):483–500PubMedCrossRefGoogle Scholar
  26. Gibson JJ (1950) The perception of the visual world. Houghton Mifflin, BostonGoogle Scholar
  27. González-Alvarez C, Subramanian A, Pardhan S (2007) Reaching and grasping with restricted peripheral vision. Ophthalmic Physiol Opt 27(3):265–274PubMedCrossRefGoogle Scholar
  28. Goodale MA (2011) Transforming vision into action. Vis Res 51(13):1567–1587PubMedCrossRefGoogle Scholar
  29. Goodale MA, Haffenden A (1998) Frames of reference for perception and action in the human visual system. Neurosci Biobehav Rev 22(2):161–172PubMedCrossRefGoogle Scholar
  30. Grasso R, Prévost P, Ivanenko Y, Berthoz A et al (1998) Eye–head coordination for the steering of locomotion in humans: an anticipatory synergy. Neurosci Lett 253(2):115–118PubMedCrossRefGoogle Scholar
  31. Haggard P, Wing A (1991) Remote responses to perturbation in human prehension. Neurosci Lett 122(1):103–108PubMedCrossRefGoogle Scholar
  32. Haggard P, Wing A (1995) Coordinated responses following mechanical perturbation of the arm during prehension. Exp Brain Res 102(3):483–494PubMedCrossRefGoogle Scholar
  33. Hayhoe M, Ballard D (2005) Eye movements in natural behavior. Trends Cogn Sci 9(4):188–194PubMedCrossRefGoogle Scholar
  34. Hayhoe M, Shrivastava A, Mruczek R, Pelz J (2003) Visual memory and motor planning in a natural task. J Vis 3(1):49–63Google Scholar
  35. Henderson JM, Hollingworth A (1999) The role of fixation position in detecting scene changes across saccades. Psychol Sci 10(5):438– 443CrossRefGoogle Scholar
  36. Hesse C, Deubel H (2010) Effects of altered transport paths and intermediate movement goals on human grasp kinematics. Exp Brain Res 201(1):93–109PubMedCrossRefGoogle Scholar
  37. Hesse C, Deubel H (2011) Efficient grasping requires attentional resources. Vis Res 51(11):1223–1231PubMedCrossRefGoogle Scholar
  38. Hicheur H, Berthoz A (2005) How do humans turn? head and body movements for the steering of locomotion. In: IEEE-RAS international conference on humanoid robots (Humanoids), IEEE, pp 265–270Google Scholar
  39. Hoffmann H, Schenck W, Möller R (2005) Learning visuomotor transformations for gaze-control and grasping. Biol Cybern 93(2):119–130PubMedCrossRefGoogle Scholar
  40. Hulse M, McBrid S, Lee M (2009) Robotic hand-eye coordination without global reference: a biologically inspired learning scheme. In: IEEE international conference on development and Learning (ICDL), IEEE, pp 1–6Google Scholar
  41. Inhoff AW, Radach R (1998) Definition and computation of oculomotor measures in the study of cognitive processes. In: Underwood G (ed) Eye guidance in reading and scene perception. Elsevier, Amsterdam, pp 29–53Google Scholar
  42. Jacob R, Karn K (2003) Eye tracking in human–computer interaction and usability research: ready to deliver the promises. Mind 2(3):4Google Scholar
  43. Jakobson L, Goodale M (1991) Factors affecting higher-order movement planning: a kinematic analysis of human prehension. Exp Brain Res 86(1):199–208PubMedCrossRefGoogle Scholar
  44. Jamone L, Natale L, Nori F, Metta G, Sandini G (2012) Autonomous online learning of reaching behavior in a humanoid robot. Int J Humanoid Robot 9(03):1–26Google Scholar
  45. Javier Traver V, Bernardino A (2010) A review of log-polar imaging for visual perception in robotics. Robot Auton Syst 58(4):378–398CrossRefGoogle Scholar
  46. Jeannerod M (1984) The timing of natural prehension movements. J Mot Behav 16(3):235–254Google Scholar
  47. Johansson R, Westling G, Bäckström A, Flanagan J (2001) Eye–hand coordination in object manipulation. J Neurosci 21(17):6917–6932PubMedGoogle Scholar
  48. Johansson RS, Flanagan JR, Johansson RS (2009) Sensory control of object manipulation. Sensorimotor control of grasping: physiology and pathophysiology. Cambridge University Press, CambridgeGoogle Scholar
  49. Kavraki LE, Svestka P, Latombe JC, Overmars MH (1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans Robotd Autom 12(4):566–580CrossRefGoogle Scholar
  50. Khansari-Zadeh S, Billard A (2011) Learning stable nonlinear dynamical systems with Gaussian mixture models. IEEE Trans Robot 27(5):943–957CrossRefGoogle Scholar
  51. Khansari-Zadeh SM, Billard A (2012) A dynamical system approach to realtime obstacle avoidance. Auton Robots 32(4):433–454CrossRefGoogle Scholar
  52. Khatib O (1986) Real-time obstacle avoidance for manipulators and mobile robots. Int J Robot Res 5(1):90–98CrossRefGoogle Scholar
  53. Kuffner Jr J, LaValle S (2000) Rrt-connect: an efficient approach to single-query path planning. In: IEEE international conference on robotics and automation (ICRA), IEEE, vol 2, pp 995–1001Google Scholar
  54. Land M (1999) Motion and vision: why animals move their eyes. J Comp Physiol A Neuroethol Sens Neural Behav Physiol 185(4):341–352CrossRefGoogle Scholar
  55. Land M, Mennie N, Rusted J et al (1999) The roles of vision and eye movements in the control of activities of daily living. Perception 28(11):1311–1328PubMedCrossRefGoogle Scholar
  56. Land MF, Furneaux S (1997) The knowledge base of the oculomotor system. Philos Trans R Soc Lond Ser B Biol Sci 352(1358):1231–1239CrossRefGoogle Scholar
  57. Liversedge S, Findlay J (2000) Saccadic eye movements and cognition. Trends Cogn Sci 4(1):6–14PubMedCrossRefGoogle Scholar
  58. Lukic L, Santos-Victor J, Billard A (2012) Learning coupled dynamical systems from human demonstration for robotic eye–arm–hand coordination. In: Proceedings of the IEEE-RAS international conference on humanoid robots (Humanoids), Osaka, JapanGoogle Scholar
  59. Lumelsky V, Skewis T (1990) Incorporating range sensing in the robot navigation function. IEEE Trans Syst Man Cybern 20(5):1058–1069CrossRefGoogle Scholar
  60. Mansard N, Lopes M, Santos-Victor J, Chaumette F (2006) Jacobian learning methods for tasks sequencing in visual servoing. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 4284–4290Google Scholar
  61. Metta G, Gasteratos A, Sandini G (2004) Learning to track colored objects with log-polar vision. Mechatronics 14(9):989–1006CrossRefGoogle Scholar
  62. Metta G, Natale L, Nori F, Sandini G, Vernon D, Fadiga L, Von Hofsten C, Rosander K, Lopes M, Santos-Victor J et al (2010) The icub humanoid robot: an open-systems platform for research in cognitive development. Neural Netw 23(8–9):1125–1134PubMedCrossRefGoogle Scholar
  63. Mishra A, Aloimonos Y, Fah CL (2009a) Active segmentation with fixation. In: 12th international conference on computer vision (ICCV), IEEE, pp 468–475Google Scholar
  64. Mishra A, Aloimonos Y, Fermuller C (2009b) Active segmentation for robotics. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 3133–3139Google Scholar
  65. Mon-Williams M, Tresilian J, Coppard V, Carson R (2001) The effect of obstacle position on reach-to-grasp movements. Exp Brain Res 137(3):497–501PubMedGoogle Scholar
  66. Natale L, Metta G, Sandini G (2005) A developmental approach to grasping. In: Developmental robotics AAAI spring symposium, vol 44Google Scholar
  67. Natale L, Nori F, Sandini G, Metta G (2007) Learning precise 3d reaching in a humanoid robot. In: IEEE international conference on development and learning (ICDL), IEEE, pp 324–329Google Scholar
  68. Neggers S, Bekkering H (2000) Ocular gaze is anchored to the target of an ongoing pointing movement. J Neurophysiol 83(2):639–651PubMedGoogle Scholar
  69. Noris B, Keller J, Billard A (2010) A wearable gaze tracking system for children in unconstrained environments. Comput Vis Image Underst 115(4):476–486CrossRefGoogle Scholar
  70. Paillard J (1982) The contribution of peripheral and central vision to visually guided reaching. In: Ingle D, Goodale M, Marsfield R (eds) Analysis of visual behavior. MIT Press, Cambridge, pp 367–385Google Scholar
  71. Pattacini U, Nori F, Natale L, Metta G, Sandini G (2010) An experimental evaluation of a novel minimum-jerk Cartesian controller for humanoid robots. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 1668–1674Google Scholar
  72. Paulignan Y, MacKenzie C, Marteniuk R, Jeannerod M (1991) Selective perturbation of visual input during prehension movements. Exp Brain Res 83(3):502–512PubMedCrossRefGoogle Scholar
  73. Pelisson D, Prablanc C, Goodale M, Jeannerod M (1986) Visual control of reaching movements without vision of the limb. Exp Brain Res 62(2):303–311PubMedCrossRefGoogle Scholar
  74. Prablanc C, Echallier J, Komilis E, Jeannerod M (1979) Optimal response of eye and hand motor systems in pointing at a visual target. Biol Cybern 35(2):113–124PubMedCrossRefGoogle Scholar
  75. Purdy KA, Lederman SJ, Klatzky RL (1999) Manipulation with no or partial vision. J Exp Psychol Hum Percept Perform 25(3):755PubMedCrossRefGoogle Scholar
  76. Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372PubMedCrossRefGoogle Scholar
  77. Rizzolatti G, Fogassi L, Gallese V (1997) Parietal cortex: from sight to action. Curr Opin Neurobiol 7(4):562–567PubMedCrossRefGoogle Scholar
  78. Rossetti Y, Stelmach G, Desmurget M, Prablanc C, Jeannerod M (1994) The effect of viewing the static hand prior to movement onset on pointing kinematics and variability. Exp Brain Res 101(2):323–330PubMedCrossRefGoogle Scholar
  79. Rothkopf C, Ballard D (2009) Image statistics at the point of gaze during human navigation. Vis Neurosci 26(01):81–92 Google Scholar
  80. Rothkopf C, Ballard D, Hayhoe M (2007) Task and context determine where you look. J Vis 7(14):1–16Google Scholar
  81. Sahbani A, El-Khoury S, Bidaud P (2012) An overview of 3d object grasp synthesis algorithms. Robot Auton Syst 60(3):326–336CrossRefGoogle Scholar
  82. Saling M, Alberts J, Stelmach G, Bloedel J (1998) Reach-to-grasp movements during obstacle avoidance. Exp Brain Res 118(2):251–258PubMedCrossRefGoogle Scholar
  83. Schenck W, Hoffmann H, Möller R (2011) Grasping of extrafoveal targets: a robotic model. New Ideas Psychol 29(3):235–259Google Scholar
  84. Seara JF, Strobl KH, Schmidt G (2003) Path-dependent gaze control for obstacle avoidance in vision guided humanoid walking. In: IEEE international conference on robotics and automation (ICRA), IEEE, vol 1, pp 887–892Google Scholar
  85. Shukla A, Billard A (2011) Coupled dynamical system based arm–hand grasping model for learning fast adaptation strategies. Robot Auton Syst 60(3):424–440CrossRefGoogle Scholar
  86. Simmons R (1996) The curvature–velocity method for local obstacle avoidance. In: IEEE international conference on robotics and automation (ICRA), IEEE, vol 4, pp 3375–3382Google Scholar
  87. Sivak B, MacKenzie CL (1990) Integration of visual information and motor output in reaching and grasping: the contributions of peripheral and central vision. Neuropsychologia 28(10):1095–1116PubMedCrossRefGoogle Scholar
  88. Spijkers WA, Lochner P (1994) Partial visual feedback and spatial end-point accuracy of discrete aiming movements. J Mot Behav 26(3):283–295PubMedCrossRefGoogle Scholar
  89. Srinivasa SS, Berenson D, Cakmak M, Collet A, Dogar MR, Dragan AD, Knepper RA, Niemueller T, Strabala K et al (2012) Herb 2.0: lessons learned from developing a mobile manipulator for the home. Proc IEEE 100(8):2410–2428CrossRefGoogle Scholar
  90. Sung HG (2004) Gaussian mixture regression and classification. PhD thesis, Rice UniversityGoogle Scholar
  91. Tatler BW, Hayhoe MM, Land MF, Ballard DH (2011) Eye guidance in natural vision: reinterpreting salience. J Vis 11(5):1–23Google Scholar
  92. Timmann D, Stelmach G, Bloedel J (1996) Grasping component alterations and limb transport. Exp Brain Res 108(3):486–492PubMedCrossRefGoogle Scholar
  93. Tresilian J (1998) Attention in action or obstruction of movement? A kinematic analysis of avoidance behavior in prehension. Exp Brain Res 120(3):352–368PubMedCrossRefGoogle Scholar
  94. Triesch J, Ballard DH, Hayhoe MM, Sullivan BT (2003) What you see is what you need. J Vis 3(1):86–94Google Scholar
  95. Vernon D, Hofsten C, Fadiga L (2010) A roadmap for cognitive development in humanoid robots, vol 11. Springer, BerlinGoogle Scholar
  96. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society conference on computer vision and pattern recognition, IEEE, vol 1, p I-511Google Scholar
  97. Wolpert D, Miall R, Kawato M (1998) Internal models in the cerebellum. Trends Cogn Sci 2(9):338–347PubMedCrossRefGoogle Scholar
  98. Wolpert D, Flanagan J et al (2001) Motor prediction. Curr Biol 11(18):729CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Learning Algorithms and Systems LaboratoryEcole Polytechnique Fédérale de Lausanne, EPFL-STI-I2S-LASALausanneSwitzerland
  2. 2.Computer Vision and Robotic LaboratoryInstituto Superior TécnicoLisbonPortugal

Personalised recommendations