Robot Learning of Everyday Object Manipulation Using Kinect

  • Nan Chen
  • Ying Hu
  • Jun Zhang
  • Jianwei Zhang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 215)


In this paper, we provide a solution of teaching a robot to perform tasks through human demonstration. On the one hand, we consider many everyday complex task manipulations made up of manipulation primitives, consisting of a series of sequential rotations and translations. On the other hand, we design demonstration primitives which decompose a demonstrated task. We use Kinect sensor to locate some hot points’ position of a teacher’s hand during demonstration, in order to gain the axes of those rotations and translations which will support the manipulation primitives that will contribute to building task descriptors. Based on this, we quote a descriptor to represent this manipulation primitive. We also teach the robot where to start and where to end once a manipulation primitive is operating via recording the special two points’ coordinates in a real scene while demonstration, which could reinforce robots’ learning to those effective and sequential demonstrations and shorten the learning time. One manipulation primitive corresponds with one demonstration primitive. Two kinds of primitives will be connected to the axes of trajectory. After that, we perform experiments to discuss the universality of this framework.


Robot learning Human demonstration Kinect 



This research is supported by National Natural Science Foundation of China (No. 61210013, 61105130), Guangdong Innovative Research Team Program (No. 201001D0104648280). Key Fundamental Research Program of Shenzhen (No. JC201005270357A)


  1. 1.
    Klingbeil E, Saxena A, Ng AY (2008) Learning to open new doors. In: RSJ workshop on robot manipulationGoogle Scholar
  2. 2.
    Jain A, Kemp CC (2009) Pulling open novel doors and drawers with equilibrium point control. In: 9th IEEE-RAS international conference on Humanoid Robots, pp 498–505Google Scholar
  3. 3.
    Quigley M, Batra S, Gould S, Klingbeil E, Le Q, Wellman A, Ng AY (2009) High-accuracy 3D sensing for mobile manipulation: improving object detection and door opening. In: Proceedings of the IEEE international conference on Robotics and automation, ICRA ‘09, pp 2816–2822Google Scholar
  4. 4.
    Asfour T, Berns K, Dillmann R (2000) The humanoid robot armar: design and control. In: International conference on Humanoid RobotGoogle Scholar
  5. 5.
    Kaneko K, Kanehiro F, Kajita S, Hirukawa H, Kawasaki T, Hirata M, Akachi K, Isozumi T (2004) Humanoid robot HRP-2. In: Proceedings of the IEEE international conference on Robotics and automation, ICRA ‘04, vol 2, pp 1083−1090Google Scholar
  6. 6.
    Microsoft (2010) Kinect for X-BOX 360.
  7. 7.
  8. 8.
    Argall B, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57(5):469–483CrossRefGoogle Scholar
  9. 9.
    Pook PK, Ballard DH (1993) Recognizing teleoperated manipulations. In: Proceedings of the IEEE international conference on Robotics and automation, vol 2, pp 578–585Google Scholar
  10. 10.
    Billard A, Calinon S, Guenter F (2006) Discriminative and adaptive imitation in uni-manual and bi-manual tasks. Robot Auton Syst 54(5):370–384CrossRefGoogle Scholar
  11. 11.
    Aleotti J, Caselli S (2006) Robust trajectory learning and approximation for robot programming by demonstration. Robot Auton Syst 54(5):409–413CrossRefGoogle Scholar
  12. 12.
    Calinon S, D’halluin F, Sauser EL, Caldwell DG, Billard AG (2010) Learning and Reproduction of Gestures by Imitation. Robotics Automation Magazine, IEEE 17(2):44–54CrossRefGoogle Scholar
  13. 13.
    Openni homepage.
  14. 14.
    PrimeSense (2010) OpenNI User GuideGoogle Scholar
  15. 15.
    PrimeSense (2010) Prime sensor NITE 1.3 algorithms notes, 1st ednGoogle Scholar
  16. 16.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. The MIT Press, CambridgeGoogle Scholar
  17. 17.
    Dang H, Allen PK, (2010) Robot learning of everyday object manipulations via human demonstration. In: IEEE/RSJ international conference on intelligent Robots and systems (IROS), pp 1284–1289Google Scholar
  18. 18.
    Erol A, Bebis G, Nicolescu M, Boyle RD, Twombly X (2005) A review on vision-based full DOF hand motion estimation. In: IEEE computer society conference on computer vision and pattern recognition—workshops, CVPR workshops, p 75, pp 25–25Google Scholar
  19. 19.
    Comaniciu D, Ramesh V, Meer P (2001) The variable bandwidth mean shift and data-driven scale selection. In: Proceedings of the eighth IEEE international conference on computer vision, ICCV 2001, vol 1, pp 438–445Google Scholar
  20. 20.
    Fukunaga K, Hostetler LD (1975) The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans Inf Theory 21:32–40CrossRefMATHMathSciNetGoogle Scholar
  21. 21.
  22. 22.

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Nan Chen
    • 1
    • 2
  • Ying Hu
    • 1
    • 2
  • Jun Zhang
    • 1
    • 2
  • Jianwei Zhang
    • 3
  1. 1.Shenzhen Institutes of Advanced TechnologyChinese Academy of SciencesShenzhenChina
  2. 2.The Chinese University of Hong KongHong KongChina
  3. 3.TAMS, Department of InformaticsUniversity of HamburgHamburgGermany

Personalised recommendations