Skip to main content

Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach


We propose an approach to efficiently teach robots how to perform dynamic manipulation tasks in cooperation with a human partner. The approach utilises human sensorimotor learning ability where the human tutor controls the robot through a multi-modal interface to make it perform the desired task. During the tutoring, the robot simultaneously learns the action policy of the tutor and through time gains full autonomy. We demonstrate our approach by an experiment where we taught a robot how to perform a wood sawing task with a human partner using a two-person cross-cut saw. The challenge of this experiment is that it requires precise coordination of the robot’s motion and compliance according to the partner’s actions. To transfer the sawing skill from the tutor to the robot we used Locally Weighted Regression for trajectory generalisation, and adaptive oscillators for adaptation of the robot to the partner’s motion.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10


  1. 1.

    Stiffness is the inverse of compliance.

  2. 2.

    Error is equal to the difference between the demonstrated (desired) and learnt performance.


  1. Adorno, B. V., Bo, A. P. L., Fraisse, P., & Poignet, P. (2011). Towards a cooperative framework for interactive manipulation involving a human and a humanoid. In 2011 IEEE international conference on robotics and automation (ICRA) (pp. 3777–3783).

  2. Ajoudani, A., Tsagarakis, N. G., & Bicchi, A. (2012). Tele-impedance: Teleoperation with impedance regulation using a body-machine interface. The International Journal of Robotics Research, 31(13), 1642–1656.

    Article  Google Scholar 

  3. Argall, B. D., & Billard, A. (2010). A survey of tactile human–robot interactions. Robotics and Autonomous Systems, 58(10), 1159–1176.

    Article  Google Scholar 

  4. Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009). A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5), 469–483.

    Article  Google Scholar 

  5. Atkeson, C. G., Hale, J. G., Pollick, F., Riley, M., Kotosaka, S., Schaal, S., et al. (2000). Using humanoid robots to study human behavior. IEEE Intelligent Systems, 15(4), 46–56.

    Article  Google Scholar 

  6. Babič, J., Hale, J. G., & Oztop, E. (2011). Human sensorimotor learning for humanoid robot skill synthesis. Adaptive Behavior—Animals, Animats, Software Agents, Robots, Adaptive Systems, 19, 250–263.

    Google Scholar 

  7. Ben Amor, H., Berger, E., Vogt, D., & Jung, B. (2009). Kinesthetic bootstrapping: Teaching motor skills to humanoid robots through physical interaction. In Proceedings of the 32nd annual German conference on advances in artificial intelligence, KI’09 (pp. 492–499). Berlin, Heidelberg: Springer.

  8. Ben Amor, H., Saxena, A., Kroemer, O., & Peters, J. (2012). Workshop: Beyond robot grasping—Modern approaches for learning dynamic manipulation. In 2012 IEEE/RSJ international conference on intelligent robots and systems.

  9. Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics (pp. 1371–1394). Berlin: Springer.

    Chapter  Google Scholar 

  10. Burdet, E., Osu, R., Franklin, D. W., Milner, T. E., & Kawato, M. (2001). The central nervous system stabilizes unstable dynamics by learning optimal impedance. Nature, 414(6862), 446–449.

    Article  Google Scholar 

  11. Chiaverini, S. (1997). Singularity-robust task-priority redundancy resolution for real-time kinematic control of robot manipulators. IEEE Transactions on Robotics and Automation, 13(3), 399–410.

    Google Scholar 

  12. De Santis, A., Siciliano, B., De Luca, A., & Bicchi, A. (2008). An atlas of physical human–robot interaction. Mechanism and Machine Theory, 43(3), 253–270.

    Article  MATH  Google Scholar 

  13. Edsinger, A., & Kemp, C. C. (2007). Human–robot interaction for cooperative manipulation: Handing objects to one another. In The 16th IEEE international symposium on robot and human interactive communication (2007) (pp. 1167–1172).

  14. Evrard, P., Gribovskaya, E., Calinon, S., Billard, A., & Kheddar, A. (2009). Teaching physical collaborative tasks: object-lifting case study with a humanoid. In IEEE-RAS international conference on humanoid robots (pp. 399–404).

  15. Franklin, D. W., Burdet, E., Osu, R., Kawato, M., & Milner, T. (2003a). Functional significance of stiffness in adaptation of multijoint arm movements to stable and unstable dynamics. Experimental Brain Research, 151(2), 145–157.

    Article  Google Scholar 

  16. Franklin, D. W., Osu, R., Burdet, E., Kawato, M., & Milner, T. E. (2003b). Adaptation to stable and unstable dynamics achieved by combined impedance control and inverse dynamics model. Journal of Neurophysiology, 90, 3270–3282.

    Google Scholar 

  17. Gams, A., Ijspeert, A. J., Schaal, S., & Lenarčič, J. (2009). On-line learning and modulation of periodic movements with nonlinear dynamical systems. Autonomous Robots, 27(1), 3–23.

    Article  Google Scholar 

  18. Gams, A., Petric, T., Ude, A., & Zlajpah, L. (2012). Performing periodic tasks: On-line learning, adaptation and synchronization with external signals. In R. Zaier (Ed.), The future of humanoid robots—research and applications (pp. 1–28). InTech.

  19. Haykin, S. (1999). Neural networks: A comprehensive foundation. New Jersey: Prentice Hall.

    MATH  Google Scholar 

  20. Hersch, M., Guenter, F., Calinon, S., & Billard, A. (2008). Dynamical system modulation for robot learning via kinesthetic demonstrations. IEEE Transactions on Robotics, 24(6), 1463–1467.

    Article  Google Scholar 

  21. Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2002). Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings 2002 IEEE international conference on robotics and automation (vol. 2, pp. 1398–1403).

  22. Ikemoto, S., Ben Amor, H., Minato, T., Jung, B., & Ishiguro, H (2012). Physical human–robot interaction: Mutual learning and adaptation. IEEE Robotics Automation Magazine, 19(4), 24–35.

    Google Scholar 

  23. Kober, J., Wilhelm, A., Oztop, E., & Peters, J. (2012). Reinforcement learning to adjust parametrized motor primitives to new situations. Autonomous Robots, 33(4), 361–379.

    Article  Google Scholar 

  24. Kormushev, P., Calinon, S., & Caldwell, D. G. (2011). Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input. Advanced Robotics, 25(5), 581–603.

    Article  Google Scholar 

  25. Kremer, P., Wimböck, T., Artigas, J., Schätzle, S., Jöhl, K., Schmidt, F., et al. (2009). Multimodal telepresent control of dlr’s rollin’ justin. In Proceedings of the 2009 IEEE international conference on robotics and automation, ICRA’09 (pp. 3096–3097). Piscataway, NJ: IEEE Press.

  26. Kroemer, O. B., Detry, R., Piater, J., & Peters, J. (2010). Combining active learning and reactive control for robot grasping. Robotics and Autonomous Systems, 58(9), 1105–1116.

    Article  Google Scholar 

  27. Kushida, D., Nakamura, M., Goto, S., & Kyura, N. (2001). Human direct teaching of industrial articulated robot arms based on force-free control. Artificial Life and Robotics, 5(1), 26–32.

    Article  Google Scholar 

  28. Lallee, S., Yoshida, E., Mallet, A., Nori, F., Natale, L., Metta, G., et al. (2010). Human–robot cooperation based on interaction learning. In O. Sigaud & J. Peters (Eds.), From motor learning to interaction learning in robots volume 264 of studies in computational intelligence (pp. 491–536). Berlin: Springer.

  29. Maciejewski, A. A., & Klein, C. A. (1985). Obstacle avoidance for kinematically redundant manipulators in dynamically varying environments. The International Journal of Robotics Research, 4(3), 109–117.

    Article  Google Scholar 

  30. Moore, B., & Oztop, E. (2012). Robotic grasping and manipulation through human visuomotor learning. Robotics and Autonomous Systems, 60(3), 441–451.

    Article  Google Scholar 

  31. Muelling, K., Kober, J., Kroemer, O., & Peters, J. (2012). Learning to select and generalize striking movements in robot table tennis. In AAAI fall symposium series.

  32. Niemeyer, G., Preusche, C., & Hirzinger, G. (2008). Telerobotics. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics (pp. 741–757). Berlin: Springer.

    Chapter  Google Scholar 

  33. Osu, R., & Gomi, H. (1999). Multijoint muscle regulation mechanism examined by measured human arm stiffness and emg signals. Journal of Neurophysiology, 81, 1458–1468.

    Google Scholar 

  34. Oztop, E., Lin, L.-H., Kawato, M., & Cheng, G. (2006). Dexterous skills transfer by extending human body schema to a robotic hand. In 2006 6th IEEE-RAS international conference on humanoid robots (pp. 82–87).

  35. Peternel, L., & Babič, J. (2013a). Humanoid robot posture-control learning in real-time based on human sensorimotor learning ability. In 2013 IEEE international conference on robotics and automation (pp. 5309–5314).

  36. Peternel, L., & Babič, J. (2013b). Learning of compliant human–robot interaction using full-body haptic interface. Advanced Robotics, 27(13), 1003–1012.

    Google Scholar 

  37. Petrič, T., Gams, A., Ijspeert, A. J., & Žlajpah, L. (2011). On-line frequency adaptation and movement imitation for rhythmic robotic tasks. The International Journal of Robotics Research, 30(14), 1775–1788.

    Article  Google Scholar 

  38. Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian processes for machine learning. International Journal of Neural Systems, 14(2), 69–106.

    Google Scholar 

  39. Sauser, E. L., Argall, B. D., Metta, G., & Billard, A. G. (2012). Iterative learning of grasp adaptation through human corrections. Robotics and Autonomous Systems, 60(1), 55–71.

    Article  Google Scholar 

  40. Schaal, S. (1999). Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3(6), 233–242.

    Article  Google Scholar 

  41. Schaal, S., & Atkeson, C. G. (1998). Constructive incremental learning from only local information. Neural Computation, 10(8), 2047–2084.

    Article  Google Scholar 

  42. Selen, L. P. J., Beek, P. J., & van Dieëbn, J. H. (2005). Can co-activation reduce kinematic variability? A simulation study. Biological Cybernetics, 93(5), 373–381.

    Article  MATH  Google Scholar 

  43. Tsumugiwa, T., Yokogawa, R., & Hara, K. (2002). Variable impedance control with virtual stiffness for human-robot cooperative peg-in-hole task. In 2002 IEEE/RSJ international conference on intelligent robots and systems (vol. 2, pp. 1075–1081).

Download references


We thank L. Žlajpah for discussions and for providing us with the robot control infrastructure. This work was supported by the Slovenian Research Agency, programme P2-0076 and grant J2-2272, and the Slovenian Ministry of Higher Education, Science and Technology.

Author information



Corresponding author

Correspondence to Luka Peternel.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mpg 25622 KB)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Peternel, L., Petrič, T., Oztop, E. et al. Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach. Auton Robot 36, 123–136 (2014).

Download citation


  • Human–robot interaction
  • Sensorimotor learning
  • Teleoperation
  • Impedance control
  • Adaptive oscillator