Investigating Embodied Music Expression Through the Leap Motion: Experimentations in Educational and Clinical Contexts

  • Adriano Baratè
  • Luca A. LudovicoEmail author
  • Eleonora Oriolo
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1022)


This paper describes a computer-based approach to foster embodied musical expression. The proposed system captures user’s hand gestures through a Leap Motion controller, and the signals thus generated are sent to a software tool that converts movements into music notes. When applied to an educational or a rehabilitation context, this solution may foster the development of communication and motor skills by using free-hand interaction. The conceptual framework was experimented in two contexts: technologically-augmented music lessons in a primary school, and music therapy sessions for the rehabilitation of physically and intellectually impaired people.


Leap Motion Music education Embodied music expression Music therapy Computer-supported education 



The authors wish to acknowledge the music therapist Antonio Elia, staff members and patients of the Cooperativa Sociale Sim-patia, Valmorea, Italy for experimenting the Leap Motion in a music therapy context. Besides, the authors would like to thank Monica Mengacci, Daniela Pina, third-grade students attending section B and the entire staff of the Scuola Elementare Del Centro, Erba, Italy for their support in classroom experimentation. Finally, we are grateful to Veronica Curioni, who coordinated and operatively performed the observation phase on primary school students.


  1. 1.
    Baratè, A., Elia, A., Ludovico, L.A., Oriolo, E.: The Leap Motion controller in clinical music therapy. A computer-based approach to intellectual and motor disabilities. In: McLaren, B.M., Reilly, R., Uhomoibhi, J., Zvacek, S. (eds.) Proceedings of the 10th International Conference on Computer Supported Education, CSEDU 2018, 15–17 March 2018, Funchal, Madeira, Portugal, pp. 461–469. SCITEPRESS - Science and Technology Publications, Lda., Setúbal (2018)Google Scholar
  2. 2.
    van Beurden, E., Zask, A., Barnett, L.M., Dietrich, U.: Fundamental movement skills: how do primary school children perform? The Move it Groove it program in rural Australia. J. Sci. Med. Sport 5(3), 244–252 (2002)CrossRefGoogle Scholar
  3. 3.
    Cattaneo, P.: La canzone come esperienza relazionale, educativa, terapeutica. Universal Music MGB Publications, Sesto Ulteriano, San Giuliano Milanese (2009)Google Scholar
  4. 4.
    Elia, A.: Con un fil di voce: dinamiche espressive tra parlato e canto. In: Cattaneo, P., Lopez, L. (eds.) Musicoterapia e Relazione. Franco Angeli, Milano (2017)Google Scholar
  5. 5.
    Gieser, S.N., Boisselle, A., Makedon, F.: Real-time static gesture recognition for upper extremity rehabilitation using the Leap Motion. In: Duffy, V.G. (ed.) DHM 2015. LNCS, vol. 9185, pp. 144–154. Springer, Cham (2015). Scholar
  6. 6.
    Godøy, R.I., Leman, M.: Musical Gestures: Sound, Movement, and Meaning. Routledge, Abingdon (2010)CrossRefGoogle Scholar
  7. 7.
    Gray, C.H., Figueroa-Sarriera, H.J., Mentor, S.: The Cyborg Handbook. Routledge, Abingdon (1995)Google Scholar
  8. 8.
    Halberstam, J.M., Livingston, I.: Posthuman Bodies. Indiana University Press, Bloomington (1995)Google Scholar
  9. 9.
    Hansson, S.O.: The ethics of enabling technology. Camb. Q. Healthc. Ethics 16(3), 257–267 (2007)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hantrakul, L., Kaczmarek, K.: Implementations of the Leap Motion device in sound synthesis and interactive live performance. In: Proceedings of the 2014 International Workshop on Movement and Computing, p. 142. ACM (2014)Google Scholar
  11. 11.
    Haraway, D.: Simians, Cyborgs, and Women: The Reinvention of Nature. Routledge, Abingdon (1991)Google Scholar
  12. 12.
    Hiebert, J., Morris, A.K., Berk, D., Jansen, A.: Preparing teachers to learn from teaching. J. Teach. Educ. 58(1), 47–61 (2007)CrossRefGoogle Scholar
  13. 13.
    Hmelo, C.E., Ferrari, M.: The problem-based learning tutorial: cultivating higher order thinking skills. J. Educ. Gifted 20(4), 401–422 (1997)CrossRefGoogle Scholar
  14. 14.
    Iosa, M., et al.: Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top. Stroke Rehabil. 22(4), 306–316 (2015)CrossRefGoogle Scholar
  15. 15.
    Karageorghis, C.I., Terry, P.C.: The psychophysical effects of music in sport and exercise: a review. J. Sport Behav. 20(1), 54 (1997)Google Scholar
  16. 16.
    Khademi, M., Mousavi Hondori, H., McKenzie, A., Dodakian, L., Lopes, C.V., Cramer, S.C.: Free-hand interaction with Leap Motion controller for stroke rehabilitation. In: Proceedings of the Extended Abstracts of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pp. 1663–1668. ACM (2014)Google Scholar
  17. 17.
    Lerner, N., Straus, J.: Sounding Off: Theorizing Disability in Music. Routledge, Abingdon (2006)Google Scholar
  18. 18.
    Liu, Z., Zhang, Y., Patrick Rau, P.-L., Choe, P., Gulrez, T.: Leap-Motion based online interactive system for hand rehabilitation. In: Rau, P.L.P. (ed.) CCD 2015. LNCS, vol. 9181, pp. 338–347. Springer, Cham (2015). Scholar
  19. 19.
    Lubet, A.: Music, Disability, and Society. Temple University Press, Philadelphia (2011)Google Scholar
  20. 20.
    Ludovico, L.A., Mangione, G.R.: Self-regulation competence in music education. In: McPherson, M., Nunes, M.B. (eds.) Proceedings of the International Conference e-Learning 2014, EL 2014, pp. 46–54. IADIS Press, Lisbon (2014)Google Scholar
  21. 21.
    Lupton, D., Seymour, W.: Technology, selfhood and physical disability. Soc. Sci. Med. 50(12), 1851–1862 (2000)CrossRefGoogle Scholar
  22. 22.
    Martin, A.J.: Motivation and engagement in music and sport: testing a multidimensional framework in diverse performance settings. J. Pers. 76(1), 135–170 (2008)CrossRefGoogle Scholar
  23. 23.
    McPherson, G.E., Renwick, J.M.: A longitudinal study of self-regulation in children’s musical practice. Music Educ. Res. 3(2), 169–186 (2001)CrossRefGoogle Scholar
  24. 24.
    Penley, C., Ross, A.: Technoculture, vol. 3. University of Minnesota Press, Minneapolis (1991)Google Scholar
  25. 25.
    Ritter, M., Aska, A.: Leap Motion as expressive gestural interface. In: Proceedings ICMC—SMC—2014, pp. 659–662 (2014)Google Scholar
  26. 26.
    Riva, G., Baños, R.M., Botella, C., Mantovani, F., Gaggioli, A.: Transforming experience: the potential of augmented reality and virtual reality for enhancing personal and clinical change. Front. Psychiatry 7, 164 (2016)CrossRefGoogle Scholar
  27. 27.
    Santagata, R., Zannoni, C., Stigler, J.W.: The role of lesson analysis in pre-service teacher education: an empirical investigation of teacher learning from a virtual video-based field experience. J. Math. Teach. Educ. 10(2), 123–140 (2007)CrossRefGoogle Scholar
  28. 28.
    Scheufele, P.M.: Effects of progressive relaxation and classical music on measurements of attention, relaxation, and stress responses. J. Behav. Med. 23(2), 207–228 (2000)CrossRefGoogle Scholar
  29. 29.
    Schlaug, G., Norton, A., Overy, K., Winner, E.: Effects of music training on the child’s brain and cognitive development. Ann. N.Y. Acad. Sci. 1060(1), 219–230 (2005)CrossRefGoogle Scholar
  30. 30.
    Straus, J.N.: Extraordinary Measures: Disability in Music. Oxford University Press, Oxford (2011)CrossRefGoogle Scholar
  31. 31.
    Taylor, J., Curran, K.: Using Leap Motion and gamification to facilitate and encourage rehabilitation for hand injuries: Leap Motion for rehabilitation. In: Handbook of Research on Holistic Perspectives in Gamification for Clinical Practice, pp. 183–192 (2015)Google Scholar
  32. 32.
    Weichert, F., Bachmann, D., Rudak, B., Fisseler, D.: Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13(5), 6380–6393 (2013)CrossRefGoogle Scholar
  33. 33.
    Zhu, G., Cai, S., Ma, Y., Liu, E.: A series of Leap Motion-based matching games for enhancing the fine motor skills of children with autism. In: 2015 IEEE 15th International Conference on Advanced Learning Technologies (ICALT), pp. 430–431. IEEE (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Adriano Baratè
    • 1
  • Luca A. Ludovico
    • 1
    Email author
  • Eleonora Oriolo
    • 1
  1. 1.Laboratorio di Informatica Musicale (LIM), Dipartimento di Informatica “Giovanni Degli Antoni”Università degli Studi di MilanoMilanoItaly

Personalised recommendations