Skip to main content

Movement Sonification: Intermodal Transformation of Movement Data to Enhance Motor Learning

  • Chapter
  • First Online:
Sports Technology

Abstract

Contrasted to ‘visualization’, the term ‘sonification’ refers to an acoustic transformation of data into sound. Human movement sonification based on kinematic and dynamic data offers multifaceted innovative applications in sports concerning enhancement in motor-perception, motor control and motor learning, mainly referring to closed skills. With the transformation of originally visual information into the auditory domain the specific power of the auditory and musical perceptual system will be integrated comprehensively into behavioral regulation. Particularly, real-time movement sonification can get effectively in novel ways, if configured convergently to visual or kinesthetic information: It has been shown that, seamlessly integrated with information from other perceptual modalities, it modulates and enhances perceptual-, control-, and even learning-processes—not only individually, but also interpersonally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Allerdissen, M., Güldenpenning, I., Schack, T., & Bläsing, B. (2017). Recognizing fencing attacks from auditory and visual information: A comparison between expert fencers and novices. Psychology of Sport and Exercise, 31, 123–130.

    Article  Google Scholar 

  • Boyer, E. O., Bevilacqua, F., Guigon, E., Hanneton, S., & Roby-Brami, A. (2020). Modulation of ellipses drawing by sonification. Experimental Brain Research, 238(4), 1011–1024.

    Article  PubMed  Google Scholar 

  • Brock, H., Schmitz, G., Baumann, J., & Effenberg, A. O. (2012). If motion sounds: Movement sonification based on inertial sensor data. Procedia Engineering, 34, 556–561.

    Article  Google Scholar 

  • Czeszumski, A., Gert, A. L., Keshava, A., Ghadirzadeh, A., Kalthoff, T., Ehinger, B. V., Tiessen, M., Björkman, M., Kragic, D., & König, P. (2021). Coordinating with a robot partner affects action monitoring related neural processing. Frontiers in Neurorobotics, 15, 686010. bioRxiv.

    Article  PubMed  PubMed Central  Google Scholar 

  • Demos, A. P., Chaffin, R., Begosh, K. T., Daniels, J. R., & Marsh, K. L. (2012). Rocking to the beat: Effects of music and partner's movements on spontaneous interpersonal coordination. Journal of Experimental Psychology: General, 141(1), 49.

    Article  PubMed  Google Scholar 

  • Dubus, G., & Bresin, R. (2013). A systematic review of mapping strategies for the sonification of physical quantities. PLoS One, 8(12), e82491.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dyer, J., Stapleton, P., & Rodger, M. (2017). Transposing musical skill: Sonification of movement as concurrent augmented feedback enhances learning in a bimanual task. Psychological Research, 81, 850–862.

    Article  PubMed  Google Scholar 

  • Effenberg, A. O. (2005). Movement sonification: Effects on perception and action. IEEE Multimedia, 12(2), 53–59.

    Article  Google Scholar 

  • Effenberg, A. O., & Schmitz, G. (2018). Acceleration and deceleration at constant speed: Systematic modulation of motion perception by kinematic sonification. Annals of the New York Academy of Sciences, 1425(1), 52–69.

    Article  PubMed  Google Scholar 

  • Effenberg, A. O., & Schmitz, G. (2022). Motorisches Lernen. In A. Güllich & M. Krüger (Eds.), Bewegung, training, Leistung und Gesundheit. Springer. https://doi.org/10.1007/978-3-662-53386-4_60-1

    Chapter  Google Scholar 

  • Effenberg, A. O., Fehse, U., Schmitz, G., Krueger, B., & Mechling, H. (2016). Movement sonification: Effects on motor learning beyond rhythmic adjustments. Frontiers in Neuroscience, 10, 219.

    Article  PubMed  PubMed Central  Google Scholar 

  • Effenberg, A. O., Hwang, T. H., Ghai, S., & Schmitz, G. (2018). Auditory modulation of multisensory representations. In International Symposium on Computer Music Multidisciplinary Research, CMMR 2017, LNCS 11265 (pp. 284–311). Springer.

    Google Scholar 

  • Eriksson, M., & Bresin, R. (2010). Improving running mechanics by use of interactive sonification. Proceedings of ISon, 95–98.

    Google Scholar 

  • Forsberg, J. (2014). A mobile application for improving running performance using interactive sonification. [Dissertation, KTH].

    Google Scholar 

  • Ghai, S., Schmitz, G., Hwang, T. H., & Effenberg, A. O. (2018). Auditory proprioceptive integration: Effects of real-time kinematic auditory feedback on knee proprioception. Frontiers in Neuroscience, 12, 142.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ghai, S., Schmitz, G., Hwang, T. H., & Effenberg, A. O. (2019). Training proprioception with sound: Effects of real-time auditory feedback on intermodal learning. Annals of the New York Academy of Sciences, 1438(1), 50–61.

    Article  PubMed  Google Scholar 

  • Ghez, C., Dubois, R. L., Rikakis, T., & Cook, P. R. (2000). An auditory display system for aiding interjoint coordination. In 6th International Conference on Auditory Display (ICAD2000), Atlanta, Georgia (pp. 1–6).

    Google Scholar 

  • Hasegawa, S., Ishijima, S., Kato, F., Mitake, H., & Sato, M. (2012). Realtime sonification of the center of gravity for skiing. In Proceedings of the 3rd Augmented Human International Conference (pp. 1–4).

    Google Scholar 

  • Hwang, T. H., & Effenberg, A. O. (2021). Head trajectory diagrams for gait symmetry analysis using a single head-worn IMU. Sensors, 21(19), 6621.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hwang, T. H., Schmitz, G., Klemmt, K., Brinkop, L., Ghai, S., Stoica, M., Maye, A., Blume, H., & Effenberg, A. O. (2018). Effect-and performance-based auditory feedback on interpersonal coordination. Frontiers in Psychology, 9, 404.

    Article  PubMed  PubMed Central  Google Scholar 

  • Keller, P. E. (2012). Mental imagery in music performance: Underlying mechanisms and potential benefits. Annals of the New York Academy of Sciences, 1252(1), 206–213.

    Article  PubMed  Google Scholar 

  • Masai, K., Kajiyama, T., Muramatsu, T., Sugimoto, M., & Kimura, T. (2022). Virtual reality sonification training system can improve a Novice’s Forehand Return of Serve in Tennis. In 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (pp. 845–849). IEEE.

    Chapter  Google Scholar 

  • Merchant, H., Zarco, W., & Prado, L. (2008). Do we have a common mechanism for measuring time in the hundreds of millisecond range? Evidence from multiple-interval timing tasks. Journal of Neurophysiology, 99(2), 939–949.

    Article  PubMed  Google Scholar 

  • Müller, M. A., Martínez-Guerrero, A., Corsi-Cabrera, M., Effenberg, A. O., Friedrich, A., Garcia-Madrid, I., Hornschuh, M., Schmitz, G., & Müller, M. F. (2022). How to orchestrate a soccer team: Generalized synchronization promoted by rhythmic acoustic stimuli. Frontiers in Human Neuroscience, 16, 909939.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nown, T. H., Upadhyay, P., Kerr, A., Andonovic, I., Tachtatzis, C., & Grealy, M. A. (2022). A mapping review of real-time movement sonification systems for movement rehabilitation. IEEE Reviews in Biomedical Engineering, 1–16.

    Google Scholar 

  • O’Brien, B., Juhas, B., Bieńkiewicz, M., Buloup, F., Bringoux, L., & Bourdin, C. (2020). Online sonification for golf putting gesture: Reduced variability of motor behaviour and perceptual judgement. Experimental Brain Research, 238, 883–895.

    Article  PubMed  Google Scholar 

  • Pasinski, A. C., McAuley, J. D., & Snyder, J. S. (2016). How modality specific is processing of auditory and visual rhythms? Psychophysiology, 53(2), 198–208.

    Article  PubMed  Google Scholar 

  • Pietschmann, J., Geu Flores, F., & Jöllenbeck, T. (2019). Gait training in orthopedic rehabilitation after joint replacement-back to normal gait with sonification? International Journal of Computer Science in Sport, 18(2).

    Google Scholar 

  • Reh, J., Hwang, T. H., Schmitz, G., & Effenberg, A. O. (2019). Dual mode gait sonification for rehabilitation after unilateral hip arthroplasty. Brain Sciences, 9(3), 66.

    Article  PubMed  PubMed Central  Google Scholar 

  • Reh, J., Schmitz, G., Hwang, T. H., & Effenberg, A. O. (2022). Loudness affects motion: Asymmetric volume of auditory feedback results in asymmetric gait in healthy young adults. BMC Musculoskeletal Disorders, 23(1), 586.

    Article  PubMed  PubMed Central  Google Scholar 

  • Repp, B. H., & Penel, A. (2002). Auditory dominance in temporal processing: New evidence from synchronization with simultaneous visual and auditory sequences. Journal of Experimental Psychology: Human Perception and Performance, 28(5), 1085.

    PubMed  Google Scholar 

  • Rodger, M. W., Young, W. R., & Craig, C. M. (2013). Synthesis of walking sounds for alleviating gait disturbances in Parkinson’s disease. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(3), 543–548.

    Article  PubMed  Google Scholar 

  • Schaffert, N., Mattes, K., & Effenberg, A. O. (2009). A sound design for the purposes of movement optimisation in elite sport (using the example of rowing). In Proceedings of the 15th International Conference on Auditory Display, Copenhagen, Denmark (pp. 1–4).

    Google Scholar 

  • Schaffert, N., Janzen, T. B., Mattes, K., & Thaut, M. H. (2019). A review on the relationship between sound and movement in sports and rehabilitation. Frontiers in Psychology, 10, 244.

    Article  PubMed  PubMed Central  Google Scholar 

  • Scheef, L., Boecker, H., Daamen, M., Fehse, U., Landsberg, M. W., Granath, D. O., Mechling, H., & Effenberg, A. O. (2009). Multimodal motion processing in area V5/MT: Evidence from an artificial class of audio-visual events. Brain Research, 1252, 94–104.

    Article  CAS  PubMed  Google Scholar 

  • Schmitz, G., Mohammadi, B., Hammer, A., Heldmann, M., Samii, A., Münte, T. F., & Effenberg, A. O. (2013). Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neuroscience, 14(1), 1–11.

    Article  Google Scholar 

  • Schmitz, G., Kroeger, D., & Effenberg, A. O. (2014). A mobile sonification system for stroke rehabilitation. In 20th International Conference on Auditory Display (ICAD2014), New York, NY (pp. 1–7).

    Google Scholar 

  • Schmitz, G., Bergmann, J., Effenberg, A. O., Krewer, C., Hwang, T. H., & Müller, F. (2018). Movement sonification in stroke rehabilitation. Frontiers in Neurology, 9, 389.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sigrist, R., Rauter, G., Riener, R., & Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic Bulletin & Review, 20(1), 21–53.

    Article  Google Scholar 

  • Tanaka, Y., Kon, H., & Koike, H. (2018). A real-time golf-swing training system using sonification and sound image localization. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (pp. 1–2).

    Google Scholar 

  • Trendafilov, D., Schmitz, G., Hwang, T. H., Effenberg, A. O., & Polani, D. (2020). Tilting together: An information-theoretic characterization of behavioral roles in rhythmic dyadic interaction. Frontiers in Human Neuroscience, 14, 185.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ungerechts, B., Cesarini, D., & Hermann, T. (2014). Real-time sonification in swimming–From pressure changes of displaced water to sound. In BMS2014—Proceedings.

    Google Scholar 

  • Vinken, P., Kroeger, D., Fehse, U., Schmitz, G., Brock, H., & Effenberg, A. O. (2013). Auditory coding of human movement kinematics. Multisensory Research, 26, 533–552.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alfred O. Effenberg .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer-Verlag GmbH, DE, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Effenberg, A.O., Hwang, TH. (2024). Movement Sonification: Intermodal Transformation of Movement Data to Enhance Motor Learning. In: Memmert, D. (eds) Sports Technology. Springer Spektrum, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-68703-1_20

Download citation

Publish with us

Policies and ethics