Skip to main content
SpringerLink
Log in

MappEMG: Enhancing Music Pedagogy by Mapping Electromyography to Multimodal Feedback

  • Conference paper
  • First Online:
ArtsIT, Interactivity and Game Creation (ArtsIT 2023)

Included in the following conference series:

Abstract

Music learning and practice may be enhanced by the use of biofeedback based on both learners’ and teachers’ muscle activity, an essential component of music performance typically unavailable to listeners. By incorporating haptic vibrations, MappEMG enables the audience to experience the performers’ muscle effort. This paper updates the MappEMG system to make muscle effort explicit in music lessons. We integrated a low-cost EMG system (BITalino MuscleBIT) and modified processing, communication, and mobile application modules. We conducted a series of experimental teaching workshops where a piano professor guided beginner and intermediate piano students with the updated MappEMG. Four interaction scenarios with MappEMG were identified from these workshops, and we gathered feedback on the initial effectiveness of using MappEMG in music pedagogy.

Supported by Pôle lavallois d’enseignement supérieur en arts numériques et économie créative, Partnership Development program of Social Sciences and Humanities Research Council of Canada, Natural Sciences and Engineering Research Council of Canada Discovery grant, and CIRMMT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Biosiglive is a Python library that aims to provide a simple and efficient way to access and process biomechanical data in real-time. https://github.com/aceglia/biosiglive.

  2. 2.

    https://github.com/IDMIL/MappEMG.

References

  1. Ceglia, A., Verdugo, F., Begon, M.: Biosiglive: an open-source Python package for real-time biosignal processing. J. Open Source Softw. 8(83), 5091 (2023)

    Article  Google Scholar 

  2. Tom, A., Singh, A., Daigle, M., Marandola, F., Wanderley, M.M.: Haptic tutor-a haptics-based music education tool for beginners. In: Proceedings of International Workshop on Haptic and Audio Interaction Design (2020)

    Google Scholar 

  3. Verdugo, F., et al.: Feeling the effort of classical musicians-a pipeline from electromyography to smartphone vibration for live music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression. PubPub (2022). https://doi.org/10.21428%2F92fbeb44.3ce22588

  4. Verdugo, F.,: MappEMG: supporting musical expression with vibrotactile feedback by capturing gestural features through electromyography. In: Proceedings of International Workshop on Haptic and Audio Interaction Design (2020)

    Google Scholar 

  5. Verdugo, F., Pelletier, J., Michaud, B., Traube, C., Begon, M: Effects of trunk motion, touch, and articulation on upper-limb velocities and on joint contribution to endpoint velocities during the production of loud piano tones. Front. Psychol. 11, 1159 (2020)

    Google Scholar 

  6. Arifin, A., Mashuri, M.T., Lestari, N.C., Satria, E., Dewantara, R.: Application of interactive learning games in stimulating knowledge about object recognition in early childhood. Educenter: Jurnal Ilmiah Pendidikan 2(1) (2023)

    Google Scholar 

  7. Bremmer, M., Nijs, L.: The role of the body in instrumental and vocal music pedagogy: a dynamical systems theory perspective on the music Teacher’s bodily engagement in teaching and learning. Front. Educ. 5, 79. Frontiers Media SA (2020)

    Google Scholar 

  8. Chaffin, R., Imreh, G., Lemieux, A.F., Chen, C.: seeing the big picture: piano practice as expert problem solving. Music. Percept. 20(4), 465–490 (2003)

    Google Scholar 

  9. Dannenberg, R.B., Sanchez, M., Joseph, A., Joseph, R., Saul, R., Capell, P.: Results from the piano tutor project. In: Proceedings of the Fourth Biennial Arts and Technology Symposium, pp. 143–150 (1993)

    Google Scholar 

  10. Davids, K., Araújo, D., Hristovski, R., Passos, P., Chow, J.Y.: Ecological dynamics and motor learning design in sport. Skill acquisition in sport: Research, theory and practice, pp. 112–130 (2012)

    Google Scholar 

  11. Dittmar, C., Cano, E., Abeßer, J., Grollmisch, S.: Music information retrieval meets music education. In: Dagstuhl Follow-Ups. vol. 3. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2012)

    Google Scholar 

  12. Dupee, M., Forneris, T., Werthner, P.: Perceived outcomes of a biofeedback and neurofeedback training intervention for optimal performance: Learning to enhance self-awareness and self-regulation with olympic athletes. Sport Psychol. 30(4), 339–349 (2016)

    Google Scholar 

  13. Fitts, P.M., Posner, M.I.: Human performance (1967)

    Google Scholar 

  14. Fougner, A., Stavdahl, y., Kyberd, P.J., Losier, Y.G., Parker, P.A.: Control of upper limb prostheses: terminology and proportional myoelectric control-a review. IEEE Trans. Neural Syst. Rehab. Eng. 20(5), 663–677 (2012)

    Google Scholar 

  15. Furuya, S., Altenmüller, E.: Flexibility of movement organization in piano performance. Front. Hum. Neurosci. 7, 173 (2013)

    Article  Google Scholar 

  16. Gallahue, D.L., Donnelly, F.C.: Developmental physical education for all children. Human Kinetics (2007)

    Google Scholar 

  17. Gazzoni, M., Afsharipour, B., Merletti, R.: Surface EMG in ergonomics and occupational medicine. Surface electromyography: physiology, engineering, and applications, pp. 361–391 (2016), publisher: Wiley Online Library

    Google Scholar 

  18. Goebl, W., Bresin, R., Galembo, A.: Touch and temporal behavior of grand piano actions. J. Acoust. Society America 118(2), 1154–1165 (2005)

    Article  Google Scholar 

  19. Gonzalez-Sanchez, V., Dahl, S., Hatfield, J.L., Godøy, R.I.: Characterizing movement fluency in musical performance: toward a generic measure for technology enhanced learning. Front. Psychol. 10, 84 (2019)

    Article  Google Scholar 

  20. Goubault, E., Verdugo, F., Pelletier, J., Traube, C., Begon, M., Dal Maso, F.: Exhausting repetitive piano tasks lead to local forearm manifestation of muscle fatigue and negatively affect musical parameters. Sci. Rep. 11(1), 8117 (2021)

    Article  Google Scholar 

  21. Grindlay, G.: Haptic guidance benefits musical motor learning. In: Proceedings of symposium on haptic interfaces for virtual environment and teleoperator systems, pp. 397–404. IEEE (2008)

    Google Scholar 

  22. Grosshauser, T., Hermann, T.: Augmented haptics – an interactive feedback system for musicians. In: Altinsoy, M.E., Jekosch, U., Brewster, S. (eds.) HAID 2009. LNCS, vol. 5763, pp. 100–108. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04076-4_11

    Chapter  Google Scholar 

  23. Gruzelier, J.H., Egner, T.: Physiological self-regulation: biofeedback and neurofeedback. Musical excellence: strategies and techniques to enhance performance, pp. 197–219 (2004), publisher: Oxford University Press, Oxford, UK

    Google Scholar 

  24. Jensenius, A.R.: Action-Sound: Developing Methods and Tools to Study Music-related Body Movement. Ph.D. thesis, University of Oslo (2007)

    Google Scholar 

  25. Kinoshita, H., Furuya, S., Aoki, T., Altenmüller, E.: Loudness control in pianists as exemplified in keystroke force measurements on different touches. J. Acoust. Society America 121(5), 2959–2969 (2007)

    Article  Google Scholar 

  26. Kotov-Smolenskiy, A.M., Khizhnikova, A.E., Klochkov, A.S., Suponeva, N.A., Piradov, M.A.: Surface EMG: applicability in the motion analysis and opportunities for practical rehabilitation. Human Physiol. 47(2), 237–247 (2021), iSBN: 0362-1197 Publisher: Springer

    Google Scholar 

  27. Kugimoto, N., et al.: CG animation for piano performance. In: SIGGRAPH’09: Posters, pp. 1–1 (2009)

    Google Scholar 

  28. Labrou, K., Zaman, C.H., Turkyasar, A., Davis, R.: Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications. In: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–8. ACM, Hamburg Germany (Apr 2023). https://doi.org/10.1145/3544549.3585838, https://dl.acm.org/doi/10.1145/3544549.3585838

  29. LeVine, W.R., Irvine, J.K.: In vivo EMG biofeedback in violin and viola pedagogy. Biofeedback Self Regul. 9, 161–168 (1984)

    Article  Google Scholar 

  30. Levy, C.E., Lee, W.A., Brandfonbrener, A.G., Press, J., Levy, A.E.: Electromyographic analysis of muscular activity in the upper extremity generated by supporting a violin with and without a shoulder rest. Med. Probl. Perform. Artist. 7(4), 103–109 (1992)

    Google Scholar 

  31. Mani, S., Vinay, C.K., Deepika, P., Rao, M.: Surface EMG signal classification for unsupervised musical keyboard learning application. In: 2020 IEEE SENSORS, pp. 1–4. IEEE (2020)

    Google Scholar 

  32. Massie-Laberge, C., Cossette, I., Wanderley, M.M.: Kinematic analysis of pianists’ expressive performances of romantic excerpts: applications for enhanced pedagogical approaches. Front. Psychol. 9, 2725 (2019)

    Article  Google Scholar 

  33. Oku, T., Furuya, S.: A novel vibrotactile biofeedback device for optimizing neuromuscular control in piano playing. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 1554–1555. IEEE (2019)

    Google Scholar 

  34. Ramstein, C.: Analyse, représentation et traitement du geste instrumental: application aux instruments à clavier. Ph.D. thesis, Institut National Polytechnique de Grenoble-INPG (1991)

    Google Scholar 

  35. Reaz, M.B.I., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biol. Proc. online 8, 11–35 (2006)

    Article  Google Scholar 

  36. Reimer, P.C., Wanderley, M.M.: Embracing less common evaluation strategies for studying user experience in NIME. In: NIME 2021. PubPub (2021)

    Google Scholar 

  37. Remache-Vinueza, B., Trujillo-León, A., Zapata, M., Sarmiento-Ortiz, F., Vidal-Verdú, F.: Audio-tactile rendering: a review on technology and methods to convey musical information through the sense of touch. Sensors 21(19), 6575 (2021)

    Article  Google Scholar 

  38. Salehi, S.K., Tahmasebi, F., Talebrokni, F.S.: A different look at featured motor learning models: comparison exam of gallahue’s, fitts and posner’s and ann gentile’s motor learning models. Movement Sport Sci. 2, 53–63 (2021)

    Google Scholar 

  39. Turner, C., Goubault, E., Dal Maso, F., Begon, M., Verdugo, F.: The influence of proximal motor strategies on pianists’ upper-limb movement variability. Hum. Mov. Sci. 90, 103110 (2023)

    Article  Google Scholar 

  40. Turner, C., Visentin, P., Oye, D., Rathwell, S., Shan, G.: An examination of trunk and right-hand coordination in piano performance: a case comparison of three pianists. Front. Psychol. 13, 838554 (2022)

    Article  Google Scholar 

  41. Verdugo, F., Begon, M., Gibet, S., Wanderley, M.M.: Proximal-to-distal sequences of attack and release movements of expert pianists during pressed-staccato keystrokes. J. Mot. Behav. 54(3), 316–326 (2022)

    Article  Google Scholar 

  42. Visentin, P., Shan, G.: Applications of EMG pertaining to music performance-A review. Arts BioMechanics 1(1), 15 (2011)

    Google Scholar 

  43. Ziane, C., Goubault, E., Michaud, B., Begon, M., Dal Maso, F.: Muscle fatigue during assisted violin performance. Ergonomics (just-accepted), pp. 1–19 (2023)

    Google Scholar 

  44. Ziane, C., Michaud, B., Begon, M., Dal Maso, F.: How do violinists adapt to dynamic assistive support? a study focusing on kinematics, muscle activity, and musical performance. Human Factors (2021)

    Google Scholar 

Download references

Acknowledgment

The work is funded by Pôle lavallois d’enseignement supérieur en arts numériques et économie créative (call for projects 2021–2022), the Partnership Development program of Social Sciences and Humanities Research Council of Canada (SSHRC-890-2021-0072), a Natural Sciences and Engineering Research Council of Canada Discovery grant to the second author, and CIRMMT. We thank Amedeo Ceglia for the support in updating the pipeline to the new biosiglive version, former interns Karl Koerich and Noa Kemp for their work in the pipeline processing refactoring, Alex Burton for its work on the implementation of the mDNS protocol and on the new version of the hAPPtiks application, and Sylvie Gibet for discussions on the previous version of the MappEMG system. We also thank all IDMIL lab members’ suggestions and comments (especially Travis West, Bavo Van Kerrebroeck, Pierrick Uro, Paul Buser, and Erivan Duarte). Finally, we warmly thank the piano teachers and students of Quebec conservatories who participated in the workshops and provided feedback on our work.

Author information

Authors and Affiliations

  1. IDMIL, CIRMMT, McGill University, Montreal, Canada

    Ziyue Piao, Marcelo M. Wanderley & Felipe Verdugo

  2. Faculté de musique and S2M laboratory, Université de Montreal, Montreal, Canada

    Felipe Verdugo

Authors
  1. Ziyue Piao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcelo M. Wanderley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Felipe Verdugo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felipe Verdugo .

Editor information

Editors and Affiliations

  1. Aalborg University, Aalborg, Denmark

    Anthony L. Brooks

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Piao, Z., Wanderley, M.M., Verdugo, F. (2024). MappEMG: Enhancing Music Pedagogy by Mapping Electromyography to Multimodal Feedback. In: Brooks, A.L. (eds) ArtsIT, Interactivity and Game Creation. ArtsIT 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 565. Springer, Cham. https://doi.org/10.1007/978-3-031-55312-7_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-55312-7_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-55311-0

  • Online ISBN: 978-3-031-55312-7

Publish with us

Policies and ethics

Search

Navigation