Skip to main content

Towards Deep Learning Based Robot Automatic Choreography System

Part of the Lecture Notes in Computer Science book series (LNAI,volume 11743)


It is a challenge task to enable a robot to dance according to different types of music. However, two problems have not been well resolved yet: (1) how to assign a dance to a certain type of music, and (2) how to ensure a dancing robot to keep in balance. To tackle these challenges, a robot automatic choreography system based on the deep learning technology is introduced in this paper. First, two deep learning neural network models are built to convert local and global features of music to corresponding features of dance, respectively. Then, an action graph is built based on the collected dance segments; the main function of the action graph is to generate a complete dance sequence based on the dance features generated by the two deep learning models. Finally, the generated dance sequence is performed by a humanoid robot. The experimental results shows that, according to the input music, the proposed model can successfully generate dance sequences that match the input music; also, the robot can maintain its balance while it is dancing. In addition, compared with the dance sequences in the training dataset, the dance sequences generated by the model has reached the level of artificial choreography in both diversity and innovation. Therefore, this method provides a promising solution for robotic choreography automation and design assistance.


  • Robot dance
  • Motion planning
  • Gesture relation
  • Action graph
  • Deep learning

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-27538-9_54
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-27538-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.


  1. 1.

    The complete dance performances can be seen on the website:


  1. Eaton, M.: An approach to the synthesis of humanoid robot dance using non-interactive evolutionary techniques. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics, pp. 3305–3309. IEEE (2013)

    Google Scholar 

  2. Peng, H., Zhou, C., Hu, H., Chao, F., Li, J.: Robotic dance in social robotics–a taxonomy. IEEE Trans. Hum.-Mach. Syst. 45(3), 281–293 (2015).

    CrossRef  Google Scholar 

  3. Peng, H., Hu, H., Chao, F., Zhou, C., Li, J.: Autonomous robotic choreography creation via semi-interactive evolutionary computation. Int. J. Soc. Robot. 8(5), 649–661 (2016)

    CrossRef  Google Scholar 

  4. Manfrè, A., Infantino, I., Vella, F., Gaglio, S.: An automatic system for humanoid dance creation. Biol. Inspired Cogn. Architect. 15, 1–9 (2016)

    CrossRef  Google Scholar 

  5. Meng, Q., Tholley, I., Chung, P.W.: Robots learn to dance through interaction with humans. Neural Comput. Appl. 24(1), 117–124 (2014)

    CrossRef  Google Scholar 

  6. Tholley, I.S.: Towards a framework to make robots learn to dance. Ph.D. thesis, Ibrahim S. Tholley (2012)

    Google Scholar 

  7. Xia, G., Tay, J., Dannenberg, R., Veloso, M.: Autonomous robot dancing driven by beats and emotions of music. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 205–212. International Foundation for Autonomous Agents and Multiagent Systems (2012)

    Google Scholar 

  8. Qin, R., Zhou, C., Zhu, H., Shi, M., Chao, F., Li, N.: A music-driven dance system of humanoid robots. Int. J. Hum. Robot. 15(05), 1850023 (2018)

    CrossRef  Google Scholar 

  9. Chu, H., Urtasun, R., Fidler, S.: Song from PI: a musically plausible network for pop music generation. CoRR abs/1611.03477 (2016).

  10. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  11. Crnkovic-Friis, L., Crnkovic-Friis, L.: Generative choreography using deep learning. In: Proceedings of the Seventh International Conference on Computational Creativity (2016)

    Google Scholar 

  12. Choi, K., Fazekas, G., Cho, K., Sandler, M.B.: A tutorial on deep learning for music information retrieval. CoRR abs/1709.04396 (2017).

  13. Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078 (2014).

  14. McFee, B., et al.: Librosa: audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference, pp. 18–25 (2015)

    Google Scholar 

  15. Ellis, D.P.: Beat tracking by dynamic programming. J. New Music Res. 36(1), 51–60 (2007)

    CrossRef  Google Scholar 

  16. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: Eleventh Annual Conference of the International Speech Communication Association (2010)

    Google Scholar 

  17. Zhao, L., Safonova, A.: Achieving good connectivity in motion graphs. Graph. Models 71(4), 139–152 (2009)

    CrossRef  Google Scholar 

  18. Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. In: ACM SIGGRAPH 2008 Classes, p. 51. ACM (2008)

    Google Scholar 

Download references


This work was supported by the National Natural Science Foundation of China (No. 61673322, 61673326, and 91746103), the Fundamental Research Funds for the Central Universities (No. 20720190142), Natural Science Foundation of Fujian Province of China (No. 2017J01128 and 2017J01129), and the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement (No. 663830).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Fei Chao .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Wu, R. et al. (2019). Towards Deep Learning Based Robot Automatic Choreography System. In: Yu, H., Liu, J., Liu, L., Ju, Z., Liu, Y., Zhou, D. (eds) Intelligent Robotics and Applications. ICIRA 2019. Lecture Notes in Computer Science(), vol 11743. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-27537-2

  • Online ISBN: 978-3-030-27538-9

  • eBook Packages: Computer ScienceComputer Science (R0)