Learning as Performance: Autoencoding and Generating Dance Movements in Real Time

  • Alexander BermanEmail author
  • Valencia James
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10783)


This paper describes the technology behind a performance where human dancers interact with an “artificial” performer projected on a screen. The system learns movement patterns from the human dancers in real time. It can also generate novel movement sequences that go beyond what it has been taught, thereby serving as a source of inspiration for the human dancers, challenging their habits and normal boundaries and enabling a mutual exchange of movement ideas. It is central to the performance concept that the system’s learning process is perceivable for the audience. To this end, an autoencoder neural network is trained in real time with motion data captured live on stage. As training proceeds, a “pose map” emerges that the system explores in a kind of improvisational state. The paper shows how this method is applied in the performance, and shares observations and lessons made in the process.


Human-computer co-creativity Movement generation Real-time learning Autoencoder 



The work presented in this paper was supported by Kulturbryggan/Swedish Arts Grants Committee, European Commission Culture Program, Life Long Burning, National Cultural Fund of Hungary, Trafó House of Contemporary Arts, CAFe Budapest Contemporary Arts Festival and 3:e Våningen.

Supplementary material

465633_1_En_17_MOESM1_ESM.mp4 (12.2 mb)
Supplementary material 1 (mp4 12506 KB)
465633_1_En_17_MOESM2_ESM.mp4 (9.9 mb)
Supplementary material 2 (mp4 10088 KB)


  1. 1.
    Schiphorst, T.: A case study of merce cunningham’s use of the lifeforms computer choreographic system in the making of trackers. M.A. thesis, Simon Fraser University (1993)Google Scholar
  2. 2.
    deLahunta, S.: The choreographic language agent. In: Stock, C. (ed.) Conference Proceedings of the 2008 World Dance Alliance Global Summit (2009)Google Scholar
  3. 3.
    Rothwell, N.: Programming languages, software thinking and creative process. In: Maragiannis, A. (ed.) Book of Abstracts DRHA 2014 (2014)Google Scholar
  4. 4.
    Leach, J., deLahunta, S.: Dance becoming knowledge: designing a digital “body”. Leonardo 50(5), 461–467 (2017)CrossRefGoogle Scholar
  5. 5.
    Carlson, K., Schiphorst, T., Pasquier, P.: Scuddle: generating movement catalysts for computer-aided choreography. In: Proceedings of the Second International Conference on Computational Creativity, pp. 123–128 (2011)Google Scholar
  6. 6.
    Augello, A., Cipolla, E., Infantino, I., Manfre, A., Pilato, G., Vella, F.: Creative robot dance with variational encoder. In: Goel, A., Jordanous, A., Pease, A. (eds.) Proceedings of the Eighth International Conference on Computational Creativity (2017)Google Scholar
  7. 7.
    Crnkovic-Friis, L., Crnkovic-Friis, L.: Generative choreography using deep learning. In: Proceedings of the Seventh International Conference on Computational Creativity (2016)Google Scholar
  8. 8.
    Bret, M., Tramus, M.H., Berthoz, A.: Interacting with an intelligent dancing figure: artistic experiments at the crossroads between art and cognitive science. Leonardo 38(1), 47–53 (2005). CrossRefGoogle Scholar
  9. 9.
    Arnal Romero, G.: Dancing with deep learning. B.Sc. thesis, Universitat Politècnica de Catalunya (2017)Google Scholar
  10. 10.
    McCormick, J., Vincs, K., Nahavandi, S., Creighton, D.: Learning to dance with a human. In: Proceedings of the 19th International Symposium on Electronic Art (2013)Google Scholar
  11. 11.
    Berman, A., James, V.: Kinetic imaginations: exploring the possibilities of combining AI and dance. In: Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI 2015, pp. 2431–2437. AAAI Press (2015).
  12. 12.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  13. 13.
    Jones, K., Bonafilia, D.: Gangogh: creating art with GANs. Accessed 15 Jan 2018
  14. 14.
    Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: controlling deep image synthesis with sketch and color. CoRR abs/1612.00835 (2016).
  15. 15.
    Elwes, J.: Latent space. Accessed 15 Jan 2018
  16. 16.
    Broad, T., Grierson, M.: Autoencoding blade runner: reconstructing films with artificial neural networks. Leonardo 50(4), 376–383 (2017)CrossRefGoogle Scholar
  17. 17.
    Elgammal, A., Liu, B., Elhoseiny, M., Mazzone, M.: CAN: creative adversarial networks: generating “art” by learning about styles and deviating from style norms. In: Proceedings of the Seventh International Conference on Computational Creativity (2016)Google Scholar
  18. 18.
    Baldi, P., Hornik, K.: Neural networks and principal component analysis: learning from examples without local minima. Neural Netw. 2(1), 53–58 (1989). CrossRefGoogle Scholar
  19. 19.
    Zeestraten, M.J.A., Havoutis, I., Silvério, J., Calinon, S., Caldwell, D.G.: An approach for imitation learning on riemannian manifolds. IEEE Robot. Autom. Lett. (RA-L) 2, 1240–1247 (2017)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.AI_amGothenburgSweden
  2. 2.AI_amRedwood CityUSA

Personalised recommendations