A motion imitation system for humanoid robots with inference-based optimization and an auditory user interface

  • Hideaki Itoh
  • Nozomi IharaEmail author
  • Hisao Fukumoto
  • Hiroshi Wakuya
Original Article


Humanoid robots are expected to perform various jobs replacing human beings, but it is difficult for non-expert users to control them. To control humanoid robots easily, various motion imitation systems, in which a humanoid robot does the same motion as a human teacher, have been studied. In the present study, we make two contributions to building easy-to-use imitation systems. The first contribution is a flexible learning system that enables both synchronous real-time imitation, in which the robot imitates human motion almost immediately, and batch imitation, in which the robot’s motion is optimized more accurately after longer measurement of human motion. We use an efficient inference-based optimization method to enable both learning modes. The other contribution is an auditory user interface that accepts voice commands from the user and at the same time identifies the target person to be imitated. This enables the user to easily use the imitation system even when other people are present. Experimental results show that the proposed system can be used in both imitation modes and that it can identify the target person.


Humanoid robot Sound source localization Imitation learning Approximate inference control (AICO) Kinect v2 sensor 



We would like to thank the anonymous reviewers for their valuable comments. This study was partially supported by the Ministry of Education, Culture, Sports, Science and Technology in Japan, Grant-in-Aid for Scientific Research (C) 15K00341 and 19K12157.


  1. 1.
    Ude A, Atkeson CG, Riley M (2004) Programming full-body movements for humanoid robots by observation. Robot Auton Syst 47(2–3):93–108CrossRefGoogle Scholar
  2. 2.
    Elbasiony R, Gomaa W (2018) Humanoids skill learning based on real-time human motion imitation using kinect. Intel Serv Robot 11(2):149–169CrossRefGoogle Scholar
  3. 3.
    Dariush B, Gienger M, Arumbakkam A, Zhu Y, Jian B, Fujimura K, Goerick C (2009) Online transfer of human motion to humanoids. Int J Hum Robot 6(02):265–289CrossRefGoogle Scholar
  4. 4.
    Montecillo-Puente FJ, Sreenivasa M, Laumond JP (2010) On real-time whole-body human to humanoid motion transfer. In: Proceedings of the 7th international conference on informatics in control, automation and robotics (ICINCO 2010), pp 22–31Google Scholar
  5. 5.
    Ou Y, Hu J, Wang Z, Fu Y, Wu X, Li X (2015) A real-time human imitation system using kinect. Int J Soc Robot 7(5):587–600CrossRefGoogle Scholar
  6. 6.
    Alibeigi M, Rabiee S, Ahmadabadi MN (2017) Inverse kinematics based human mimicking system using skeletal tracking technology. J Intell Robot Syst 85(1):27–45CrossRefGoogle Scholar
  7. 7.
    Nakaoka S, Nakazawa A, Kanehiro F, Kaneko K, Morisawa M, Hirukawa H, Ikeuchi K (2007) Learning from observation paradigm: leg task models for enabling a biped humanoid robot to imitate human dances. Int J Robot Res 26(8):829–844CrossRefGoogle Scholar
  8. 8.
    Munirathinam K, Chevallereau C, Sakka S (2014) Offline imitation of a human motion by a humanoid robot under balance constraint. Springer, Cham, pp 269–282Google Scholar
  9. 9.
    Toussaint M (2009) Robot trajectory optimization using approximate inference. In: Proceedings of the 26th international conference on machine learning, pp 1049–1056Google Scholar
  10. 10.
    Itoh H, Sakai Y, Kadoya T, Fukumoto H, Wakuya H, Furukawa T (2017) Using model uncertainty for robust optimization in approximate inference control. Artif Life Robot 22(3):327–335CrossRefGoogle Scholar
  11. 11.
    Zarubin D, Pokorny FT, Song D, Toussaint M, Kragic D (2013) Topological synergies for grasp transfer. In: Hand Synergies-how to tame the complexity of grapsing, Workshop, IEEE international conference on robotics and automationGoogle Scholar
  12. 12.
    Watter M, Springenberg J, Boedecker J, Riedmiller M (2015) Embed to control: a locally linear latent dynamics model for control from raw images. In: Advances in neural information processing systems, pp 2728–2736Google Scholar
  13. 13.
    Rawlik K, Zarubin D, Toussaint M, Vijayakumar S (2018) An approximate inference approach to temporal optimization for robotics. In: Robotics research. Springer, pp 145–162Google Scholar
  14. 14.
    Toussaint M (2009) Pros and cons of truncated Gaussian EP in the context of approximate inference control. NIPS workshop on probabilistic approaches for robotics and controlGoogle Scholar
  15. 15.
    Rüeckert E, Mindt M, Peters J, Neumann G (2014) Robust policy updates for stochastic optimal control. In: Humanoid robots (humanoids), 2014 14th IEEE-RAS international conference on, pp 388–393Google Scholar
  16. 16.
    Zarubin D, Ivan V, Toussaint M, Komura T, Vijayakumar S (2013) Hierarchical motion planning in topological representations. Robotics, pp 465–472Google Scholar
  17. 17.
    Schaal S (1999) Is imitation learning the route to humanoid robots? Trends Cogn Sci 3(6):233–242CrossRefGoogle Scholar
  18. 18.
    Breazeal C, Scassellati B (2002) Robots that imitate humans. Trends Cogn Sci 6(11):481–487CrossRefGoogle Scholar
  19. 19.
    Hussein A, Gaber MM, Elyan E, Jayne C (2017) Imitation learning: a survey of learning methods. ACM Comput Surv 50(2):21CrossRefGoogle Scholar
  20. 20.
    Grimes DB, Chalodhorn R, Rao RP (2006) Dynamic imitation in a humanoid robot through nonparametric probabilistic inference. In: Robotics: science and systems, pp 199–206Google Scholar
  21. 21.
    Cole JB, Grimes DB, Rao RP (2007) Learning full-body motions from monocular vision: dynamic imitation in a humanoid robot. In: 2007 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 240–246Google Scholar
  22. 22.
    Ding J, Shi JY (2017) Kinect microphone array-based speech and speaker recognition for the exhibition control of humanoid robots. Comput Electr Eng 62:719–729CrossRefGoogle Scholar
  23. 23.
    Nakashima M, Itoh H, Fukumoto H, Wakuya H, Furukawa T (2016) Developing a ROS node to control ICS servo motors. In: Proceedings of the 21st international symposium on artificial life and robotics, pp 107–110Google Scholar

Copyright information

© International Society of Artificial Life and Robotics (ISAROB) 2019

Authors and Affiliations

  • Hideaki Itoh
    • 1
  • Nozomi Ihara
    • 2
    Email author
  • Hisao Fukumoto
    • 1
  • Hiroshi Wakuya
    • 1
  1. 1.Department of Electrical and Electronic Engineering, Faculty of Science and EngineeringSaga UniversitySagaJapan
  2. 2.Department of Electrical and Electronic Engineering, Graduate School of Science and EngineeringSaga UniversitySagaJapan

Personalised recommendations