Skip to main content
SpringerLink
Log in

Auditory Modulation of Multisensory Representations

  • Conference paper
  • First Online:
Music Technology with Swing (CMMR 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11265))

Included in the following conference series:

Abstract

Motor control and motor learning as well as interpersonal coordination are based on motor perception and emergent perceptuomotor representations. At least in early stages motor learning and interpersonal coordination are emerging heavily on visual information in terms of observing others and transforming the information into internal representations to guide owns behavior. With progressing learning, also other perceptual modalities are added when a new motor pattern is established by repeated physical exercises. In contrast to the vast majority of publications on motor learning and interpersonal coordination referring to a certain perceptual modality here we look at the perceptual system as a unitary system coordinating and unifying the information of all involved perceptual modalities. The relation between perceptual streams of different modalities, the intermodal processing and multisensory integration of information as a basis for motor control and learning will be the main focus of this contribution.

Multi-/intermodal processing of perceptual streams results in multimodal representations and opens up new approaches to support motor learning and interpersonal coordination: Creating an additional perceptual stream adequately auditory movement information can be generated suitable to be integrated with information of other modalities and thereby modulating the resulting perceptuomotor representations without the need of attention and higher cognition. Here, the concept of a movement defined real-time acoustics is used to serve the auditory system in terms of an additional movement-auditory stream. Before the computational approach of kinematic real-time sonification is finally described, a special focus is directed to the level of adaptation modules of the internal models. Furthermore, this concept is compared with different approaches of additional acoustic movement information. Moreover, a perspective of this approach is given in a broad spectrum of new applications of supporting motor control and learning in sports and motor rehabilitation as well as a broad spectrum of joint action and interpersonal coordination between humans but also concerning human-robot-interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Asanuma, H., Keller, A.: Neuronal mechanisms of motor learning in mammals. NeuroReport 2(5), 217–224 (1991)

    Article  Google Scholar 

  • Atallah, L., Aziz, O., Lo, B., Yang, G.Z.: Detecting walking gait impairment with an ear-worn sensor. In: Sixth International Workshop on Wearable and Implantable Body Sensor Networks, BSN 2009, pp. 175–180. IEEE (2009)

    Google Scholar 

  • Bastian, A.J.: Understanding sensorimotor adaptation and learning for rehabilitation. Curr. Opin. Neurol. 21(6), 628–633 (2008)

    Article  Google Scholar 

  • Baumann, O., Greenlee, M.W.: Neural correlates of coherent audiovisual motion perception. Cereb. Cortex 17, 1433–1443 (2006)

    Article  Google Scholar 

  • Bernier, P., Gauthier, G., Blouin, J.: Evidence for distinct, differentially adaptable sensorimotor transformations for reaches to visual and proprioceptive targets. J. Neurophysiol. 98, 1815–1819 (2007)

    Article  Google Scholar 

  • Berthoz, A., Viaud-Delmon, I.: Multisensory integration in spatial orientation. Curr. Opin. Neurobiol. 9(6), 708–712 (1999)

    Article  Google Scholar 

  • Bialkowski, S.E.: Real-time digital filters: infinite impulse response filters. Anal. Chem. 60(6), 403A–413A (1988)

    Article  Google Scholar 

  • Bishop, L., Goebl, W.: Beating time: how ensemble musicians’ cueing gestures communicate beat position and tempo. Psychol. Music 46(1), 84–106 (2018). https://doi.org/10.1177/0305735617702971

    Article  Google Scholar 

  • Bock, O.: Basic principles of sensorimotor adaptation to different distortions with different effectors and movement types: a review and synthesis of behavioral findings. Front. Hum. Neurosci. 7, 81 (2013)

    Article  Google Scholar 

  • Bock, O., Schmitz, G.: Transfer of visuomotor adaptation to unpractised hands and sensory modalities. Psychology 4(12), 1004–1007 (2013). https://doi.org/10.4236/psych.2013.412145

    Article  Google Scholar 

  • Bock, O., Schmitz, G., Grigorova, V.: Transfer of adaptation between ocular saccades and arm movements. Hum. Mov. Sci. 27, 383–395 (2008)

    Article  Google Scholar 

  • Brodie, M., Walmsley, A., Page, W.: Fusion motion capture: a prototype system using inertial measurement units and GPS for the biomechanical analysis of ski racing. Sports Technol. 1(1), 17–28 (2008)

    Article  Google Scholar 

  • Buchecker, M., Wegenkittl, S., Stöggl, T., Müller, E.: Unstable footwear affects magnitude and structure of variability in postural control. Motor Control 22(1), 1–35 (2017)

    Article  Google Scholar 

  • Calvert, G.A., Spence, C., Stein, B.E. (eds.): The Handbook of Multisensory Processes. MIT Press, Cambridge (2004)

    Google Scholar 

  • Campos-Sousa, I.S., Campos-Sousa, R.N., Ataide Jr., L., Soares, M.M., Almeida, K.J.: Executive dysfunction and motor symptoms in Parkinson’s disease. Arq. Neuropsiquiatr. 68(2), 246–251 (2010)

    Article  Google Scholar 

  • Chen, L., Vroomen, Y.: Intersensory binding across space and time: a tutorial review. Atten. Percept. Psychophys. 75, 790–811 (2013)

    Article  Google Scholar 

  • Cohen, M.: Changes in auditory localization following prismatic exposure under continuous and terminal visual feedback. Percept. Mot. Skills 38, 1202 (1974)

    Article  Google Scholar 

  • Craske, B.: Intermodal transfer of adaptation to displacement. Nature 5037, 765 (1966)

    Article  Google Scholar 

  • Cromwell, R., Wellmon, R.: Sagittal plane head stabilization during level walking and ambulation on stairs. Physiotherapy Res. Int. 6(3), 179–192 (2001)

    Article  Google Scholar 

  • Cunnington, R., Iansek, R., Bradshaw, J.L., Phillips, J.G.: Movement-related potentials in Parkinson’s disease. Brain 118(4), 935–950 (1995)

    Article  Google Scholar 

  • D’Ausilio, A., Badino, L., Li, Y., Tokay, S., Craighero, L., Canto, R., Aloimonos, Y., Fadiga, L.: Leadership in orchestra emerges from the causal relationships of movement kinematics. PLoS ONE 7(5), e35757 (2012)

    Article  Google Scholar 

  • D’Ausilio, A., Novembre, G., Fadiga, L., Keller, P.E.: What can music tell us about social interaction? Trends Cogn. Sci. 19(3), 111–114 (2015)

    Article  Google Scholar 

  • Debaere, F., Wenderoth, N., Sunaert, S., Van Hecke, P., Swinnen, S.P.: Internal vs external generation of movements: differential neural pathways involved in bimanual coordination performed in the presence or absence of augmented visual feedback. Neuroimage 19(3), 764–776 (2003)

    Article  Google Scholar 

  • Delignières, D., Torre, K.: Fractal dynamics of human gait: a reassessment of the 1996 data of Hausdorff et al. J. Appl. Physiol. 106(4), 1272–1279 (2009)

    Article  Google Scholar 

  • Demos, A.P., Chaffin, R., Begosh, K.T., Daniels, J.R., Marsh, K.L.: Rocking to the beat: effects of music and partner’s movements on spontaneous interpersonal coordination. J. Exp. Psychol. Gen. 141(1), 49 (2012)

    Article  Google Scholar 

  • Dijkerman, H.C., McIntosh, R.D., Anema, H.A., de Haan, E.H., Kappelle, L.J., Milner, A.D.: Reaching errors in optic ataxia are linked to eye position rather than head or body position. Neuropsychologia 44(13), 2766–2773 (2006)

    Article  Google Scholar 

  • Dubus, G., Bresin, R.: A systematic review of mapping strategies for the sonification of physical quantities. PLoS ONE 8(12), e82491 (2013)

    Article  Google Scholar 

  • Effenberg, A.O.: Movement sonification: effects on perception and action. IEEE Multimedia 12(2), 53–59 (2005)

    Article  Google Scholar 

  • Effenberg, A.O., Fehse, U., Schmitz, G., Krueger, B., Mechling, H.: Movement sonification: effects on motor learning beyond rhythmic adjustments. Front. Neurosci. 10 (2016). https://doi.org/10.3389/fnins.2016.00219

  • Effenberg, A.O., Schmitz, G., Baumann, F., Rosenhahn, B., Kroeger, D.: Soundscript–supporting the acquisition of character writing by multisensory integration. Open Psychol. J. 8(3), 230–237 (2015). https://doi.org/10.2174/1874350101508010230

    Article  Google Scholar 

  • Effenberg, A.O., Schmitz, G.: Acceleration and deceleration at constant speed: systematic modulation of motion perception by kinematic sonification. Ann. N. Y. Acad. Sci. (2018). https://doi.org/10.1111/nyas.13693

    Article  Google Scholar 

  • El-Tamawy, M.S., Darwish, M.H., Khallaf, M.E.: Effects of augmented proprioceptive cues on the parameters of gait of individuals with Parkinson’s disease. Ann. Indian Acad. Neurol. 15(4), 267–272 (2012)

    Article  Google Scholar 

  • Felix, R.A., Fridberger, A., Leijon, S., Berrebi, A.S., Magnusson, A.K.: Sound rhythms are encoded by postinhibitory rebound spiking in the superior paraolivary nucleus. J. Neurosci. 31(35), 12566–12578 (2011)

    Article  Google Scholar 

  • Flannigan, J.C., Posthuma, R.J., Lombardo, J.N., Murray, C., Cressmann, E.K.: Adaptation to proprioceptive targets following visuomotor adaptation. Exp. Brain Res. 236, 419–432 (2018)

    Article  Google Scholar 

  • Ford, M.P., Malone, L.A., Nyikos, I., Yelisetty, R., Bickel, C.S.: Gait training with progressive external auditory cueing in persons with Parkinson’s disease. Arch. Phys. Med. Rehabil. 91(8), 1255–1261 (2010)

    Article  Google Scholar 

  • Fujioka, T., Trainor, L.J., Large, E.W., Ross, B.: Internalized timing of isochronous sounds is represented in neuromagnetic beta oscillations. J. Neurosci. 32(5), 1791–1802 (2012)

    Article  Google Scholar 

  • Galea, J., Miall, R.: Concurrent adaptation to opposing visual displacements during an alternating movement. Exp. Brain Res. 175, 676–688 (2006)

    Article  Google Scholar 

  • Ghai, S., Ghai, I., Effenberg, A.O.: Effect of rhythmic auditory cueing on aging gait: a systematic review and meta-analysis. Aging Dis. 131–200 (2017a)

    Google Scholar 

  • Ghai, S., Ghai, I., Effenberg, A.O.: Effects of dual-task training and dual-tasks on postural stability: a systematic review and meta-analysis. Clin. Interv. Aging 12, 557–577 (2017b)

    Google Scholar 

  • Ghai, S., Ghai, I., Effenberg, A.O.: Effect of rhythmic auditory cueing on gait in cerebral palsy: a systematic review and meta-analysis. Neuropsychiatric Dis. Treat. 14, 43–59 (2018a)

    Article  Google Scholar 

  • Ghai, S., Ghai, I., Schmitz, G., Effenberg, A.O.: Effect of rhythmic auditory cueing on Parkinsonian gait: a systematic review and meta-analysis. Sci. Rep. 8(1), 506 (2018b)

    Article  Google Scholar 

  • Ghai, S., Schmitz, G., Hwang, T.-H., Effenberg, A.O.: Auditory proprioceptive integration: effects of real-time kinematic auditory feedback on knee proprioception. Front. Neurosci. 12, 142 (2018c)

    Article  Google Scholar 

  • Gibson, J.J.: The Senses Considered as Perceptual Systems. Houghton-Mifflin, Boston (1966)

    Google Scholar 

  • Gibson, J.J.: The Ecological Approach to Visual Perception. Houghton-Mifflin, Boston (1979)

    Google Scholar 

  • Goodman, J.R., Isenhower, R.W., Marsh, K., Schmidt, R., Richardson, M.: The interpersonal phase entrainment of rocking chair movements. In: Heft, H., Marsh, K.L. (eds.) Studies in Perception and Action VIII: Thirteenth International Conference on Perception and Action (2005)

    Google Scholar 

  • Haar, S., Donchin, O., Dinstein, I.: Dissociating visual and motor directional selectivity using visuomotor adaptation. J. Neurosci. 35(17), 6813–6821 (2015)

    Article  Google Scholar 

  • Hameed, S., Ferris, T., Jayaraman, S., Sarter, N.: Using informative peripheral visual and tactile cues to support task and interruption management. Hum. Factors 51(2), 126–135 (2009)

    Article  Google Scholar 

  • Harris, C.S.: Adaptation to displaced vision: visual, motor, or proprioceptive change? Science 140, 812–813 (1963)

    Article  Google Scholar 

  • Harris, C.S.: Perceptual adaptation to inverted, reversed, and displaced vision. Psychol. Rev. 72(6), 419–444 (1965)

    Article  Google Scholar 

  • Hatada, Y., Miall, R.C., Rossetti, Y.: Two waves of a long-lasting aftereffect of prism adaptation measured over 7 days. Exp. Brain Res. 169(3), 417–426 (2006)

    Article  Google Scholar 

  • Hausdorff, J.M., Lowenthal, J., Herman, T., Gruendlinger, L., Peretz, C., Giladi, N.: Rhythmic auditory stimulation modulates gait variability in Parkinson’s disease. Eur. J. Neurosci. 26(8), 2369–2375 (2007)

    Article  Google Scholar 

  • Hausdorff, J.M., Purdon, P.L., Peng, C., Ladin, Z., Wei, J.Y., Goldberger, A.L.: Fractal dynamics of human gait: stability of long-range correlations in stride interval fluctuations. J. Appl. Physiol. 80(5), 1448–1457 (1996)

    Article  Google Scholar 

  • Hellmers, H., Norrdine, A., Blankenbach, J., Eichhorn, A.: An IMU/magnetometer-based indoor positioning system using Kalman filtering. In: 2013 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–9. IEEE (2013)

    Google Scholar 

  • Hesch, J.A., Kottas, D.G., Bowman, S.L., Roumeliotis, S.I.: Camera-IMU-based localization: observability analysis and consistency improvement. Int. J. Robot. Res. 33(1), 182–201 (2014)

    Article  Google Scholar 

  • Hopkins, K., Kass, S.J., Blalock, L.D., Brill, J.C.: Effectiveness of auditory and tactile crossmodal cues in a dual-task visual and auditory scenario. Ergonomics 60(5), 692–700 (2017)

    Article  Google Scholar 

  • Hove, M.J., Suzuki, K., Uchitomi, H., Orimo, S., Miyake, Y.: Interactive rhythmic auditory stimulation reinstates natural 1/f timing in gait of Parkinson’s patients. PLoS ONE 7(3), e32600 (2012)

    Article  Google Scholar 

  • Hwang, T.H., Reh, J., Effenberg, A.O., Blume, H.: Real-time gait analysis using a single head-worn inertial measurement unit. IEEE Trans. Consum. Electron. 64(2), 240–248 (2018a). https://doi.org/10.1109/tce.2018.2843289

    Article  Google Scholar 

  • Hwang, T.-H., et al.: Effect and performance-based auditory feedback on interpersonal coordination. Front. Psychol. 9, 404 (2018b). https://doi.org/10.3389/fpsyg.2018.00404

    Article  Google Scholar 

  • Imamizu, H., Shimojo, S.: The locus of visual-motor learning at the task or manipulator level: implications from intermanual transfer. J. Exp. Psychol. Hum. Percept. Perform. 21, 719–733 (1995)

    Article  Google Scholar 

  • Imamizu, H., et al.: Explicit contextual information selectively contributes to predictive switching of internal models. Exp. Brain Res. 181(3), 395–408 (2007)

    Article  Google Scholar 

  • Johansson, B.B.: Multisensory stimulation in stroke rehabilitation. Front. Hum. Neurosci. 6, 60 (2012)

    Article  Google Scholar 

  • Kagerer, F.A., Contreras-Vidal, J.L.: Adaptation of sound localization induced by rotated visual feedback in reaching movements. Exp. Brain Res. 193(2), 315–321 (2009)

    Article  Google Scholar 

  • Keller, P.E., Knoblich, G., Repp, B.H.: Pianists duet better when they play with themselves: on the possible role of action simulation in synchronization. Conscious. Cogn. 16(1), 102–111 (2007)

    Article  Google Scholar 

  • Keller, P.E., Novembre, G., Hove, M.J.: Rhythm in joint action: psychological and neurophysiological mechanisms for real-time interpersonal coordination. Philos. Trans. Roy. Soc. B: Biol. Sci. 369(1658), 20130394 (2014)

    Article  Google Scholar 

  • Keysers, C., Kohler, E., Umilta, M.A., Nanetti, L., Fogassi, L., Gallese, V.: Audiovisual mirror neurons and action recognition. Exp. Brain Res. 153, 628–636 (2003)

    Article  Google Scholar 

  • Khoramshahi, M., Shukla, A., Raffard, S., Bardy, B.G., Billard, A.: Role of gaze cues in interpersonal motor coordination: towards higher affiliation in human-robot interaction. PLoS ONE 11(6), e0156874 (2016). https://doi.org/10.1371/journal.pone.0156874

    Article  Google Scholar 

  • Kirk, A.G., O’Brien, J.F., Forsyth, D.A.: Skeletal parameter estimation from optical motion capture data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 782–788. IEEE (2005)

    Google Scholar 

  • Knoblich, G., Butterfill, S., Sebanz, N.: Psychological research on joint action: theory and data. In: Psychology of Learning and Motivation-Advances in Research and Theory, vol. 54, p. 59 (2011)

    Chapter  Google Scholar 

  • Kohler, E., Keysers, C., Umilta, M.A., Fogassi, L., Gallese, V., Rizzolati, G.: Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846–848 (2002)

    Article  Google Scholar 

  • Kohler, I.: Ãœber Aufbau und Wandlungen der Wahrnehmungswelt, insbesondere über bedingte Empfindungen.’: In: Kommission bei RM Rohrer (1951)

    Google Scholar 

  • Kohler, I.: The formation and transformation of the perceptual world. Psychological Issues (1963)

    Google Scholar 

  • Lacquaniti, F., et al.: Multisensory integration and internal models for sensing gravity effects in primates. In: BioMed Research International (2014)

    Google Scholar 

  • Lahav, A., Saltzman, E., Schlaug, G.: Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J. Neurosci. 27(2), 308–314 (2007)

    Article  Google Scholar 

  • Lee, J.Y., Schweighofer, N.: Dual adaptation supports a parallel architecture of motor memory. J. Neurosci. 29(33), 10396–10404 (2009)

    Article  Google Scholar 

  • Lewald, J.M., Getzmann, S.: Horizontal and vertical effects of eye-position on sound localization. Hear. Res. 213, 99–106 (2006)

    Article  Google Scholar 

  • Lohnes, C.A., Earhart, G.M.: The impact of attentional, auditory, and combined cues on walking during single and cognitive dual tasks in Parkinson disease. Gait Posture 33(3), 478–483 (2011)

    Article  Google Scholar 

  • Magescas, F., Prablanc, C.: Automatic drive of limb motor plasticity. J. Cogn. Neurosci. 18(1), 75–83 (2006)

    Article  Google Scholar 

  • Magill, R.A., Anderson, D.I.: Motor Learning and Control: Concepts and Applications, vol. 11. McGraw-Hill, New York (2007)

    Google Scholar 

  • Martin, L.M., Newman, C.V.: Simultaneous right- and left-hand adaptation in opposite lateral directions following bidirectional optical displacement. Bull. Psychon. Soc. 16(6), 432–434 (1980)

    Article  Google Scholar 

  • Michel, C., Pisella, L., Prablanc, C., Rode, G., Rossetti, Y.: Enhancing visuomotor adaptation by reducing error signals: single-step (Aware) versus multiplestep (Unaware) exposure to wedge prisms. J. Cogn. Neurosci. 19(2), 341–350 (2007)

    Article  Google Scholar 

  • Mikaelian, H.: Lack of bilateral generalization of adaptation to auditory rearrangement. Percept. Psychophys. 11(3), 222–224 (1972)

    Article  Google Scholar 

  • Mikaelian, H.: Adaptation to displaced hearing: a nonproprioceptive change. J. Exp. Psychol. 103, 326–330 (1974)

    Article  Google Scholar 

  • Miyata, K., Varlet, M., Miura, A., Kudo, K., Keller, P.E.: Modulation of individual auditory-motor coordination dynamics through interpersonal visual coupling. Sci. Rep. 7, 16220 (2017). https://doi.org/10.1038/s41598-017-16151-5

    Article  Google Scholar 

  • Moeslund, T.B., Hilton, A., Krueger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104(2–3), 90–126 (2006)

    Article  Google Scholar 

  • Morton, S.M., Bastian, A.J.: Prism adaptation during walking generalizes to reaching and requires the cerebellum. J. Neurosci. 92, 2497–2509 (2004)

    Google Scholar 

  • Mueller, G., Moeser, M. (eds.): Handbook of Engineering Acoustics. Springer Science and Business Media, Berlin (2012). https://doi.org/10.1007/978-3-540-69460-1

    Book  Google Scholar 

  • Murgia, M., et al.: Ecological sounds affect breath duration more than artificial sounds. Psychol. Res. 80(1), 76–81 (2016)

    Article  Google Scholar 

  • Nieuwboer, A., et al.: The short-term effects of different cueing modalities on turn speed in people with Parkinson’s disease. Neurorehabil. Neural Repair 23(8), 831–836 (2009)

    Article  Google Scholar 

  • Nieuwboer, A., et al.: Cueing training in the home improves gait-related mobility in Parkinson’s disease: the RESCUE trial. J. Neurol. Neurosurg. Psychiatry 78(2), 134–140 (2007)

    Article  Google Scholar 

  • Nombela, C., Hughes, L.E., Owen, A.M., Grahn, J.A.: Into the groove: can rhythm influence Parkinson’s disease? Neurosci. Biobehav. Rev. 37(10), 2564–2570 (2013)

    Article  Google Scholar 

  • Nunes, M.E., Souza, M.G., Basso, L., Monteiro, C., Corrêa, U.C., Santos, S.: Frequency of provision of knowledge of performance on skill acquisition in older persons. Front. Psychol. 5, 1454 (2014)

    Article  Google Scholar 

  • Oostwoud Wijdenes, L., Medendorp, W.P.: State estimation for early feedback responses in reaching: intramodal or multimodal? Front. Integr. Neurosci. 11, 38 (2017). https://doi.org/10.3389/fnint.2017.00038

    Article  Google Scholar 

  • Oppenheim, A.V.: Discrete-Time Signal Processing. Pearson Education India, Bangalore (1999)

    Google Scholar 

  • Oscari, F., Secoli, R., Avanzini, F., Rosati, G., Reinkensmeyer, D.J.: Substituting auditory for visual feedback to adapt to altered dynamic and kinematic environments during reaching. Exp. Brain Res. 221, 33–41 (2012)

    Article  Google Scholar 

  • Oullier, O., De Guzman, G.C., Jantzen, K.J., Lagarde, J., Scott Kelso, J.: Social coordination dynamics: measuring human bonding. Soc. Neurosci. 3(2), 178–192 (2008)

    Article  Google Scholar 

  • Prablanc, C., Tzavaras, A., Jeannerod, M.: Adaptation of the two arms to opposite prism displacements. Q. J. Exp. Psychol. 27(4), 667–671 (1975)

    Article  Google Scholar 

  • Rinehart, N.J., Bellgrove, M.A., Tonge, B.J., Brereton, A.V., Howells-Rankin, D., Bradshaw, J.L.: An examination of movement kinematics in young people with high-functioning autism and Asperger’s disorder: further evidence for a motor planning deficit. J. Autism Dev. Disord. 36(6), 757–767 (2006)

    Article  Google Scholar 

  • Reh, J., Hwang, T.H., Michalke, V., Effenberg, A.O.: Instruction and real-time sonification for gait rehabilitation after unilateral hip arthroplasty. In: 11th Joint Conference on Motor Control Learning Biomechanics Training, pp. 1–2. DVS (2016)

    Google Scholar 

  • Rios, J.A., White, E.: Fusion filter algorithm enhancements for a MEMS GPS/IMU, pp. 1–12. Crossbow Technology, Inc. (2002)

    Google Scholar 

  • Rueterbories, J., Spaich, E.G., Larsen, B., Andersen, O.K.: Methods for gait event detection and analysis in ambulatory systems. Med. Eng. Phys. 32(6), 545–552 (2010)

    Article  Google Scholar 

  • Sainburg, R., Wang, J.: Interlimb transfer of visuomotor rotations: independence of direction and final position information. Exp. Brain Res. 145, 437–447 (2002)

    Article  Google Scholar 

  • Schaefer, R.S.: Auditory rhythmic cueing in movement rehabilitation: findings and possible mechanisms. Philos. Trans. Roy. Soc. B: Biol. Sci. 369, 20130402 (2014)

    Article  Google Scholar 

  • Schmitz, G., Bock, O.: Properties of intermodal transfer after dual visuo- and auditory-motor adaptation. Hum. Mov. Sci. 55, 108–120 (2017)

    Article  Google Scholar 

  • Schmitz, G.: Visuo- und Audiomotorische Adaptation. Hofmann-Verlag, Schorndorf (2014). ISBN 978-3-7780-4850-4

    Google Scholar 

  • Schmitz, G.: Interference between adaptation to double steps and adaptation to rotated feedback in spite of differences in directional selectivity. Exp. Brain Res. 234, 1491–1504 (2016). https://doi.org/10.1007/s00221-016-4559-y

    Article  Google Scholar 

  • Schmitz, G., Effenberg, A.O.: Sound joint actions in rowing and swimming. In: Meyer, C., Wedelstaedt, U.V. (eds.) Moving Bodies in Interaction - Interacting Bodies in Motion. John Benjamins Publishing Company, Amsterdam (2016)

    Google Scholar 

  • Schmitz, G., Bergmann, J., Effenberg, A.O., Krewer, C., Hwang, T.H., Mueller, F.: Movement sonification in stroke rehabilition. Front. Neurol. 9, 389 (2018)

    Article  Google Scholar 

  • Schmitz, G., Bock, O.: A comparison of sensorimotor adaptation in the visual and in the auditory modality. PLoS ONE 9(9), e107834 (2014)

    Article  Google Scholar 

  • Schmitz, G., Effenberg, A.O.: Perceptual effects of auditory information about own and other movements. In: 18th International Conference on Auditory Display, Atlanta, GA, USA (2012)

    Google Scholar 

  • Schmitz, G., Effenberg, A.O.: Schlagmann 2.0 – Bewegungsakustische Dimensionen interpersonaler Koordination im Mannschaftssport. Ger. J. Exerc. Sport Res. 47(3), 232–245 (2017)

    Article  Google Scholar 

  • Schmitz, G., Grigorova, V.: Alternating adaptation of eye and hand movements to opposite directed double steps. J. Mot. Behav. 49(3), 255–264 (2017). https://doi.org/10.1080/00222895.2016.1191419

    Article  Google Scholar 

  • Schmitz, G., Kroeger, D., Effenberg, A.O.: A mobile sonification system for stroke rehabilitation. In: The 20th International Conference on Auditory Display, New York (2014)

    Google Scholar 

  • Schmitz, G., et al.: Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci. 14, 32 (2013). https://doi.org/10.1186/1471-2202-14-32

    Article  Google Scholar 

  • Sebanz, N., Knoblich, G.: Prediction in joint action: what, when, and where. Top. Cogn. Sci. 1(2), 353–367 (2009)

    Article  Google Scholar 

  • Sebanz, N., Bekkering, H., Knoblich, G.: Joint action: bodies and minds moving together. Trends Cogn. Sci. 10(2), 70–76 (2006)

    Article  Google Scholar 

  • Seitz, A.R., Kim, R., Shams, L.: Sound facilitates visual learning. Curr. Biol. 16(14), 1422–1427 (2006)

    Article  Google Scholar 

  • Sengpielaudio homepage: http://www.sengpielaudio.com/calculator-notenames.htm

  • Shadmehr, R., Smith, R.A., Krakauer, J.W.: Error correction, sensory prediction, and adaptation in motor control. Ann. Rev. Neurosci. 33, 89–108 (2010)

    Article  Google Scholar 

  • Shams, L., Seitz, A.R.: Benefits of multisensory learning. Trends Cogn. Sci. 12(11), 411–417 (2008)

    Article  Google Scholar 

  • Sharma, D.A., Chevidikunnan, M.F., Khan, F.R., Gaowgzeh, R.A.: Effectiveness of knowledge of result and knowledge of performance in the learning of a skilled motor activity by healthy young adults. J. Phys. Ther. Sci. 28(5), 1482–1486 (2016)

    Article  Google Scholar 

  • Silaghi, M.-C., Plänkers, R., Boulic, R., Fua, P., Thalmann, D.: Local and global skeleton fitting techniques for optical motion capture. In: Magnenat-Thalmann, N., Thalmann, D. (eds.) CAPTECH 1998. LNCS (LNAI), vol. 1537, pp. 26–40. Springer, Heidelberg (1998). https://doi.org/10.1007/3-540-49384-0_3

    Chapter  Google Scholar 

  • Smith, M.A., Ghazizadeh, A., Shadmehr, R.: Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol. 4(6), e179 (2006). https://doi.org/10.1371/journal.pbio.0040179

    Article  Google Scholar 

  • Spence, C.: Cross-modal perceptual organization. In: Wagemans, J. (ed.) The Oxford Handbook of Perceptual Organization. Oxford University Press, Oxford (2015)

    Google Scholar 

  • Spence, C., Driver, J. (eds.): Crossmodal Space and Crossmodal Attention. Oxford University Press, Oxford (2004)

    Google Scholar 

  • Stein, B.E., Meredith, M.A.: The Merging of the Senses. MIT Press, Cambridge (1993)

    Google Scholar 

  • Stein, B.E., Stanford, T.R.: Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9(4), 255–266 (2008)

    Article  Google Scholar 

  • Stoffregen, T.A., Bardy, B.G.: On specification and the senses. Behav. Brain Sci. 24, 195–213 (2001). Discussion 213-161

    Article  Google Scholar 

  • Suh, J.H., et al.: Effect of rhythmic auditory stimulation on gait and balance in hemiplegic stroke patients. NeuroRehabilitation 34(1), 193–199 (2014)

    MathSciNet  Google Scholar 

  • Tagliabue, M., McIntyre, J.: A modular theory of multisensory integration for motor control. Front. Comput. Neurosci. 8, 1 (2014). https://doi.org/10.3389/fncom.2014.00001

    Article  Google Scholar 

  • Tecchio, F., Salustri, C., Thaut, M.H., Pasqualetti, P., Rossini, P.: Conscious and preconscious adaptation to rhythmic auditory stimuli: a magnetoencephalographic study of human brain responses. Exp. Brain Res. 135(2), 222–230 (2000)

    Article  Google Scholar 

  • Thaut, M.H.: Neural basis of rhythmic timing networks in the human brain. Ann. N. Y. Acad. Sci. 999(1), 364–373 (2003)

    Article  Google Scholar 

  • Thaut, M.H.: Rhythm, Music, and the Brain: Scientific Foundations and Clinical Applications, vol. 7. Routledge, Abingdon (2005)

    Google Scholar 

  • Thaut, M.H., et al.: Neurologic music therapy improves executive function and emotional adjustment in traumatic brain injury rehabilitation. Ann. N. Y. Acad. Sci. 1169(1), 406–416 (2009)

    Article  Google Scholar 

  • Thaut, M.H., Leins, A.K., Rice, R.R., Argstatter, H., Kenyon, G.P., McIntosh, G.C., Fetter, M., et al.: Rhythmic auditory stimulation improves gait more than NDT/Bobath training in near-ambulatory patients early poststroke: a single-blind, randomized trial. Neurorehabil. Neural Repair 21(5), 455–459 (2007)

    Article  Google Scholar 

  • Thaut, M., Schleiffers, S., Davis, W.: Analysis of EMG activity in biceps and triceps muscle in an upper extremity gross motor task under the influence of auditory rhythm. J. Music Ther. 28(2), 64–88 (1991)

    Article  Google Scholar 

  • Tierney, A., Kraus, N.: The ability to move to a beat is linked to the consistency of neural responses to sound. J. Neurosci. 33(38), 14981–14988 (2013)

    Article  Google Scholar 

  • Torres, E.B., Heilman, K.M., Poizner, H.: Impaired endogenously evoked automated reaching in Parkinson’s disease. J. Neurosci. 31(49), 17848–17863 (2011)

    Article  Google Scholar 

  • Vesper, C., Butterfill, S., Knoblich, G., Sebanz, N.: A minimal architecture for joint action. Neural Netw. 23(8), 998–1003 (2010)

    Article  Google Scholar 

  • Weeks, D.L., Kordus, R.N.: Relative frequency of knowledge of performance and motor skill learning. Res. Q. Exerc. Sport 69(3), 224–230 (1998)

    Article  Google Scholar 

  • Welch, R.B.: Perceptual Modification. Adapting to Altered Sensory Environments. Academic Press, Cambridge (1978)

    Google Scholar 

  • Whitall, J., et al.: Bilateral and unilateral arm training improve motor function through differing neuroplastic mechanisms a single-blinded randomized controlled trial. Neurorehabil. Neural Repair 25(2), 118–129 (2011)

    Article  Google Scholar 

  • Wigmore, V., Tong, C., Flanagan, J.R.: Visuomotor rotations of varying size and direction compete for single internal model in working memory. J. Exp. Psychol. Hum. Percept. Perform. 28, 447–457 (2002)

    Article  Google Scholar 

  • Winstein, C.J.: Knowledge of results and motor learning—implications for physical therapy. Phys. Ther. 71(2), 140–149 (1991)

    Article  Google Scholar 

  • Wolpert, D.M., Kawato, M.: Multiple paired forward and inverse models for motor control. Neural Netw. 11(7–8), 1317–1329 (1998)

    Article  Google Scholar 

  • Wolpert, D.M., Diedrichsen, J., Flanagan, J.R.: Principles of sensorimotor learning. Nat. Rev. Neurosci. 12, 739–751 (2011)

    Article  Google Scholar 

  • Wolpert, D.M., Ghahramani, Z., Jordan, M.I.: An internal model for sensorimotor integration. Science 269, 1880–1882 (1995)

    Article  Google Scholar 

  • Young, W.R., Rodger, M.W., Craig, C.M.: Auditory observation of stepping actions can cue both spatial and temporal components of gait in Parkinson’s disease patients. Neuropsychologia 57, 140–153 (2014)

    Article  Google Scholar 

  • Zmigrod, S., Hommel, B.: Feature Integration across multimodal perception and action: a review. Multisensory Res. 26, 143–157 (2013)

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by European Commission H2020-FETPROACT-2014 No. 641321.

Author information

Authors and Affiliations

  1. Leibniz University Hannover, 30167, Hannover, Germany

    Alfred O. Effenberg, Tong-Hun Hwang, Shashank Ghai & Gerd Schmitz

Authors
  1. Alfred O. Effenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tong-Hun Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shashank Ghai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gerd Schmitz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alfred O. Effenberg .

Editor information

Editors and Affiliations

  1. Laboratoire PRISM, AMU-CNRS, Marseille, France

    Mitsuko Aramaki

  2. INESC TEC, Porto, Portugal

    Matthew E. P. Davies

  3. Laboratoire PRISM, AMU-CNRS, Marseille, France

    Richard Kronland-Martinet

  4. Laboratoire PRISM, AMU-CNRS, Marseille, France

    Sølvi Ystad

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Effenberg, A.O., Hwang, TH., Ghai, S., Schmitz, G. (2018). Auditory Modulation of Multisensory Representations. In: Aramaki, M., Davies , M., Kronland-Martinet, R., Ystad, S. (eds) Music Technology with Swing. CMMR 2017. Lecture Notes in Computer Science(), vol 11265. Springer, Cham. https://doi.org/10.1007/978-3-030-01692-0_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01692-0_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01691-3

  • Online ISBN: 978-3-030-01692-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation