Advertisement

Auditory Modulation of Multisensory Representations

  • Alfred O. EffenbergEmail author
  • Tong-Hun Hwang
  • Shashank Ghai
  • Gerd Schmitz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11265)

Abstract

Motor control and motor learning as well as interpersonal coordination are based on motor perception and emergent perceptuomotor representations. At least in early stages motor learning and interpersonal coordination are emerging heavily on visual information in terms of observing others and transforming the information into internal representations to guide owns behavior. With progressing learning, also other perceptual modalities are added when a new motor pattern is established by repeated physical exercises. In contrast to the vast majority of publications on motor learning and interpersonal coordination referring to a certain perceptual modality here we look at the perceptual system as a unitary system coordinating and unifying the information of all involved perceptual modalities. The relation between perceptual streams of different modalities, the intermodal processing and multisensory integration of information as a basis for motor control and learning will be the main focus of this contribution.

Multi-/intermodal processing of perceptual streams results in multimodal representations and opens up new approaches to support motor learning and interpersonal coordination: Creating an additional perceptual stream adequately auditory movement information can be generated suitable to be integrated with information of other modalities and thereby modulating the resulting perceptuomotor representations without the need of attention and higher cognition. Here, the concept of a movement defined real-time acoustics is used to serve the auditory system in terms of an additional movement-auditory stream. Before the computational approach of kinematic real-time sonification is finally described, a special focus is directed to the level of adaptation modules of the internal models. Furthermore, this concept is compared with different approaches of additional acoustic movement information. Moreover, a perspective of this approach is given in a broad spectrum of new applications of supporting motor control and learning in sports and motor rehabilitation as well as a broad spectrum of joint action and interpersonal coordination between humans but also concerning human-robot-interaction.

Keywords

Interpersonal coordination Motor control Motor learning Movement sonification Multimodal integration Multimodal perception Perceptuomotor representation 

Notes

Acknowledgments

This research was supported by European Commission H2020-FETPROACT-2014 No. 641321.

References

  1. Asanuma, H., Keller, A.: Neuronal mechanisms of motor learning in mammals. NeuroReport 2(5), 217–224 (1991)CrossRefGoogle Scholar
  2. Atallah, L., Aziz, O., Lo, B., Yang, G.Z.: Detecting walking gait impairment with an ear-worn sensor. In: Sixth International Workshop on Wearable and Implantable Body Sensor Networks, BSN 2009, pp. 175–180. IEEE (2009)Google Scholar
  3. Bastian, A.J.: Understanding sensorimotor adaptation and learning for rehabilitation. Curr. Opin. Neurol. 21(6), 628–633 (2008)CrossRefGoogle Scholar
  4. Baumann, O., Greenlee, M.W.: Neural correlates of coherent audiovisual motion perception. Cereb. Cortex 17, 1433–1443 (2006)CrossRefGoogle Scholar
  5. Bernier, P., Gauthier, G., Blouin, J.: Evidence for distinct, differentially adaptable sensorimotor transformations for reaches to visual and proprioceptive targets. J. Neurophysiol. 98, 1815–1819 (2007)CrossRefGoogle Scholar
  6. Berthoz, A., Viaud-Delmon, I.: Multisensory integration in spatial orientation. Curr. Opin. Neurobiol. 9(6), 708–712 (1999)CrossRefGoogle Scholar
  7. Bialkowski, S.E.: Real-time digital filters: infinite impulse response filters. Anal. Chem. 60(6), 403A–413A (1988)CrossRefGoogle Scholar
  8. Bishop, L., Goebl, W.: Beating time: how ensemble musicians’ cueing gestures communicate beat position and tempo. Psychol. Music 46(1), 84–106 (2018).  https://doi.org/10.1177/0305735617702971CrossRefGoogle Scholar
  9. Bock, O.: Basic principles of sensorimotor adaptation to different distortions with different effectors and movement types: a review and synthesis of behavioral findings. Front. Hum. Neurosci. 7, 81 (2013)CrossRefGoogle Scholar
  10. Bock, O., Schmitz, G.: Transfer of visuomotor adaptation to unpractised hands and sensory modalities. Psychology 4(12), 1004–1007 (2013).  https://doi.org/10.4236/psych.2013.412145CrossRefGoogle Scholar
  11. Bock, O., Schmitz, G., Grigorova, V.: Transfer of adaptation between ocular saccades and arm movements. Hum. Mov. Sci. 27, 383–395 (2008)CrossRefGoogle Scholar
  12. Brodie, M., Walmsley, A., Page, W.: Fusion motion capture: a prototype system using inertial measurement units and GPS for the biomechanical analysis of ski racing. Sports Technol. 1(1), 17–28 (2008)CrossRefGoogle Scholar
  13. Buchecker, M., Wegenkittl, S., Stöggl, T., Müller, E.: Unstable footwear affects magnitude and structure of variability in postural control. Motor Control 22(1), 1–35 (2017)CrossRefGoogle Scholar
  14. Calvert, G.A., Spence, C., Stein, B.E. (eds.): The Handbook of Multisensory Processes. MIT Press, Cambridge (2004)Google Scholar
  15. Campos-Sousa, I.S., Campos-Sousa, R.N., Ataide Jr., L., Soares, M.M., Almeida, K.J.: Executive dysfunction and motor symptoms in Parkinson’s disease. Arq. Neuropsiquiatr. 68(2), 246–251 (2010)CrossRefGoogle Scholar
  16. Chen, L., Vroomen, Y.: Intersensory binding across space and time: a tutorial review. Atten. Percept. Psychophys. 75, 790–811 (2013)CrossRefGoogle Scholar
  17. Cohen, M.: Changes in auditory localization following prismatic exposure under continuous and terminal visual feedback. Percept. Mot. Skills 38, 1202 (1974)CrossRefGoogle Scholar
  18. Craske, B.: Intermodal transfer of adaptation to displacement. Nature 5037, 765 (1966)CrossRefGoogle Scholar
  19. Cromwell, R., Wellmon, R.: Sagittal plane head stabilization during level walking and ambulation on stairs. Physiotherapy Res. Int. 6(3), 179–192 (2001)CrossRefGoogle Scholar
  20. Cunnington, R., Iansek, R., Bradshaw, J.L., Phillips, J.G.: Movement-related potentials in Parkinson’s disease. Brain 118(4), 935–950 (1995)CrossRefGoogle Scholar
  21. D’Ausilio, A., Badino, L., Li, Y., Tokay, S., Craighero, L., Canto, R., Aloimonos, Y., Fadiga, L.: Leadership in orchestra emerges from the causal relationships of movement kinematics. PLoS ONE 7(5), e35757 (2012)CrossRefGoogle Scholar
  22. D’Ausilio, A., Novembre, G., Fadiga, L., Keller, P.E.: What can music tell us about social interaction? Trends Cogn. Sci. 19(3), 111–114 (2015)CrossRefGoogle Scholar
  23. Debaere, F., Wenderoth, N., Sunaert, S., Van Hecke, P., Swinnen, S.P.: Internal vs external generation of movements: differential neural pathways involved in bimanual coordination performed in the presence or absence of augmented visual feedback. Neuroimage 19(3), 764–776 (2003)CrossRefGoogle Scholar
  24. Delignières, D., Torre, K.: Fractal dynamics of human gait: a reassessment of the 1996 data of Hausdorff et al. J. Appl. Physiol. 106(4), 1272–1279 (2009)CrossRefGoogle Scholar
  25. Demos, A.P., Chaffin, R., Begosh, K.T., Daniels, J.R., Marsh, K.L.: Rocking to the beat: effects of music and partner’s movements on spontaneous interpersonal coordination. J. Exp. Psychol. Gen. 141(1), 49 (2012)CrossRefGoogle Scholar
  26. Dijkerman, H.C., McIntosh, R.D., Anema, H.A., de Haan, E.H., Kappelle, L.J., Milner, A.D.: Reaching errors in optic ataxia are linked to eye position rather than head or body position. Neuropsychologia 44(13), 2766–2773 (2006)CrossRefGoogle Scholar
  27. Dubus, G., Bresin, R.: A systematic review of mapping strategies for the sonification of physical quantities. PLoS ONE 8(12), e82491 (2013)CrossRefGoogle Scholar
  28. Effenberg, A.O.: Movement sonification: effects on perception and action. IEEE Multimedia 12(2), 53–59 (2005)CrossRefGoogle Scholar
  29. Effenberg, A.O., Fehse, U., Schmitz, G., Krueger, B., Mechling, H.: Movement sonification: effects on motor learning beyond rhythmic adjustments. Front. Neurosci. 10 (2016).  https://doi.org/10.3389/fnins.2016.00219
  30. Effenberg, A.O., Schmitz, G., Baumann, F., Rosenhahn, B., Kroeger, D.: Soundscript–supporting the acquisition of character writing by multisensory integration. Open Psychol. J. 8(3), 230–237 (2015).  https://doi.org/10.2174/1874350101508010230CrossRefGoogle Scholar
  31. Effenberg, A.O., Schmitz, G.: Acceleration and deceleration at constant speed: systematic modulation of motion perception by kinematic sonification. Ann. N. Y. Acad. Sci. (2018).  https://doi.org/10.1111/nyas.13693CrossRefGoogle Scholar
  32. El-Tamawy, M.S., Darwish, M.H., Khallaf, M.E.: Effects of augmented proprioceptive cues on the parameters of gait of individuals with Parkinson’s disease. Ann. Indian Acad. Neurol. 15(4), 267–272 (2012)CrossRefGoogle Scholar
  33. Felix, R.A., Fridberger, A., Leijon, S., Berrebi, A.S., Magnusson, A.K.: Sound rhythms are encoded by postinhibitory rebound spiking in the superior paraolivary nucleus. J. Neurosci. 31(35), 12566–12578 (2011)CrossRefGoogle Scholar
  34. Flannigan, J.C., Posthuma, R.J., Lombardo, J.N., Murray, C., Cressmann, E.K.: Adaptation to proprioceptive targets following visuomotor adaptation. Exp. Brain Res. 236, 419–432 (2018)CrossRefGoogle Scholar
  35. Ford, M.P., Malone, L.A., Nyikos, I., Yelisetty, R., Bickel, C.S.: Gait training with progressive external auditory cueing in persons with Parkinson’s disease. Arch. Phys. Med. Rehabil. 91(8), 1255–1261 (2010)CrossRefGoogle Scholar
  36. Fujioka, T., Trainor, L.J., Large, E.W., Ross, B.: Internalized timing of isochronous sounds is represented in neuromagnetic beta oscillations. J. Neurosci. 32(5), 1791–1802 (2012)CrossRefGoogle Scholar
  37. Galea, J., Miall, R.: Concurrent adaptation to opposing visual displacements during an alternating movement. Exp. Brain Res. 175, 676–688 (2006)CrossRefGoogle Scholar
  38. Ghai, S., Ghai, I., Effenberg, A.O.: Effect of rhythmic auditory cueing on aging gait: a systematic review and meta-analysis. Aging Dis. 131–200 (2017a)Google Scholar
  39. Ghai, S., Ghai, I., Effenberg, A.O.: Effects of dual-task training and dual-tasks on postural stability: a systematic review and meta-analysis. Clin. Interv. Aging 12, 557–577 (2017b)Google Scholar
  40. Ghai, S., Ghai, I., Effenberg, A.O.: Effect of rhythmic auditory cueing on gait in cerebral palsy: a systematic review and meta-analysis. Neuropsychiatric Dis. Treat. 14, 43–59 (2018a)CrossRefGoogle Scholar
  41. Ghai, S., Ghai, I., Schmitz, G., Effenberg, A.O.: Effect of rhythmic auditory cueing on Parkinsonian gait: a systematic review and meta-analysis. Sci. Rep. 8(1), 506 (2018b)CrossRefGoogle Scholar
  42. Ghai, S., Schmitz, G., Hwang, T.-H., Effenberg, A.O.: Auditory proprioceptive integration: effects of real-time kinematic auditory feedback on knee proprioception. Front. Neurosci. 12, 142 (2018c)CrossRefGoogle Scholar
  43. Gibson, J.J.: The Senses Considered as Perceptual Systems. Houghton-Mifflin, Boston (1966)Google Scholar
  44. Gibson, J.J.: The Ecological Approach to Visual Perception. Houghton-Mifflin, Boston (1979)Google Scholar
  45. Goodman, J.R., Isenhower, R.W., Marsh, K., Schmidt, R., Richardson, M.: The interpersonal phase entrainment of rocking chair movements. In: Heft, H., Marsh, K.L. (eds.) Studies in Perception and Action VIII: Thirteenth International Conference on Perception and Action (2005)Google Scholar
  46. Haar, S., Donchin, O., Dinstein, I.: Dissociating visual and motor directional selectivity using visuomotor adaptation. J. Neurosci. 35(17), 6813–6821 (2015)CrossRefGoogle Scholar
  47. Hameed, S., Ferris, T., Jayaraman, S., Sarter, N.: Using informative peripheral visual and tactile cues to support task and interruption management. Hum. Factors 51(2), 126–135 (2009)CrossRefGoogle Scholar
  48. Harris, C.S.: Adaptation to displaced vision: visual, motor, or proprioceptive change? Science 140, 812–813 (1963)CrossRefGoogle Scholar
  49. Harris, C.S.: Perceptual adaptation to inverted, reversed, and displaced vision. Psychol. Rev. 72(6), 419–444 (1965)CrossRefGoogle Scholar
  50. Hatada, Y., Miall, R.C., Rossetti, Y.: Two waves of a long-lasting aftereffect of prism adaptation measured over 7 days. Exp. Brain Res. 169(3), 417–426 (2006)CrossRefGoogle Scholar
  51. Hausdorff, J.M., Lowenthal, J., Herman, T., Gruendlinger, L., Peretz, C., Giladi, N.: Rhythmic auditory stimulation modulates gait variability in Parkinson’s disease. Eur. J. Neurosci. 26(8), 2369–2375 (2007)CrossRefGoogle Scholar
  52. Hausdorff, J.M., Purdon, P.L., Peng, C., Ladin, Z., Wei, J.Y., Goldberger, A.L.: Fractal dynamics of human gait: stability of long-range correlations in stride interval fluctuations. J. Appl. Physiol. 80(5), 1448–1457 (1996)CrossRefGoogle Scholar
  53. Hellmers, H., Norrdine, A., Blankenbach, J., Eichhorn, A.: An IMU/magnetometer-based indoor positioning system using Kalman filtering. In: 2013 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–9. IEEE (2013)Google Scholar
  54. Hesch, J.A., Kottas, D.G., Bowman, S.L., Roumeliotis, S.I.: Camera-IMU-based localization: observability analysis and consistency improvement. Int. J. Robot. Res. 33(1), 182–201 (2014)CrossRefGoogle Scholar
  55. Hopkins, K., Kass, S.J., Blalock, L.D., Brill, J.C.: Effectiveness of auditory and tactile crossmodal cues in a dual-task visual and auditory scenario. Ergonomics 60(5), 692–700 (2017)CrossRefGoogle Scholar
  56. Hove, M.J., Suzuki, K., Uchitomi, H., Orimo, S., Miyake, Y.: Interactive rhythmic auditory stimulation reinstates natural 1/f timing in gait of Parkinson’s patients. PLoS ONE 7(3), e32600 (2012)CrossRefGoogle Scholar
  57. Hwang, T.H., Reh, J., Effenberg, A.O., Blume, H.: Real-time gait analysis using a single head-worn inertial measurement unit. IEEE Trans. Consum. Electron. 64(2), 240–248 (2018a).  https://doi.org/10.1109/tce.2018.2843289CrossRefGoogle Scholar
  58. Hwang, T.-H., et al.: Effect and performance-based auditory feedback on interpersonal coordination. Front. Psychol. 9, 404 (2018b).  https://doi.org/10.3389/fpsyg.2018.00404CrossRefGoogle Scholar
  59. Imamizu, H., Shimojo, S.: The locus of visual-motor learning at the task or manipulator level: implications from intermanual transfer. J. Exp. Psychol. Hum. Percept. Perform. 21, 719–733 (1995)CrossRefGoogle Scholar
  60. Imamizu, H., et al.: Explicit contextual information selectively contributes to predictive switching of internal models. Exp. Brain Res. 181(3), 395–408 (2007)CrossRefGoogle Scholar
  61. Johansson, B.B.: Multisensory stimulation in stroke rehabilitation. Front. Hum. Neurosci. 6, 60 (2012)CrossRefGoogle Scholar
  62. Kagerer, F.A., Contreras-Vidal, J.L.: Adaptation of sound localization induced by rotated visual feedback in reaching movements. Exp. Brain Res. 193(2), 315–321 (2009)CrossRefGoogle Scholar
  63. Keller, P.E., Knoblich, G., Repp, B.H.: Pianists duet better when they play with themselves: on the possible role of action simulation in synchronization. Conscious. Cogn. 16(1), 102–111 (2007)CrossRefGoogle Scholar
  64. Keller, P.E., Novembre, G., Hove, M.J.: Rhythm in joint action: psychological and neurophysiological mechanisms for real-time interpersonal coordination. Philos. Trans. Roy. Soc. B: Biol. Sci. 369(1658), 20130394 (2014)CrossRefGoogle Scholar
  65. Keysers, C., Kohler, E., Umilta, M.A., Nanetti, L., Fogassi, L., Gallese, V.: Audiovisual mirror neurons and action recognition. Exp. Brain Res. 153, 628–636 (2003)CrossRefGoogle Scholar
  66. Khoramshahi, M., Shukla, A., Raffard, S., Bardy, B.G., Billard, A.: Role of gaze cues in interpersonal motor coordination: towards higher affiliation in human-robot interaction. PLoS ONE 11(6), e0156874 (2016).  https://doi.org/10.1371/journal.pone.0156874CrossRefGoogle Scholar
  67. Kirk, A.G., O’Brien, J.F., Forsyth, D.A.: Skeletal parameter estimation from optical motion capture data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 782–788. IEEE (2005)Google Scholar
  68. Knoblich, G., Butterfill, S., Sebanz, N.: Psychological research on joint action: theory and data. In: Psychology of Learning and Motivation-Advances in Research and Theory, vol. 54, p. 59 (2011)CrossRefGoogle Scholar
  69. Kohler, E., Keysers, C., Umilta, M.A., Fogassi, L., Gallese, V., Rizzolati, G.: Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846–848 (2002)CrossRefGoogle Scholar
  70. Kohler, I.: Über Aufbau und Wandlungen der Wahrnehmungswelt, insbesondere über bedingte Empfindungen.’: In: Kommission bei RM Rohrer (1951)Google Scholar
  71. Kohler, I.: The formation and transformation of the perceptual world. Psychological Issues (1963)Google Scholar
  72. Lacquaniti, F., et al.: Multisensory integration and internal models for sensing gravity effects in primates. In: BioMed Research International (2014)Google Scholar
  73. Lahav, A., Saltzman, E., Schlaug, G.: Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J. Neurosci. 27(2), 308–314 (2007)CrossRefGoogle Scholar
  74. Lee, J.Y., Schweighofer, N.: Dual adaptation supports a parallel architecture of motor memory. J. Neurosci. 29(33), 10396–10404 (2009)CrossRefGoogle Scholar
  75. Lewald, J.M., Getzmann, S.: Horizontal and vertical effects of eye-position on sound localization. Hear. Res. 213, 99–106 (2006)CrossRefGoogle Scholar
  76. Lohnes, C.A., Earhart, G.M.: The impact of attentional, auditory, and combined cues on walking during single and cognitive dual tasks in Parkinson disease. Gait Posture 33(3), 478–483 (2011)CrossRefGoogle Scholar
  77. Magescas, F., Prablanc, C.: Automatic drive of limb motor plasticity. J. Cogn. Neurosci. 18(1), 75–83 (2006)CrossRefGoogle Scholar
  78. Magill, R.A., Anderson, D.I.: Motor Learning and Control: Concepts and Applications, vol. 11. McGraw-Hill, New York (2007)Google Scholar
  79. Martin, L.M., Newman, C.V.: Simultaneous right- and left-hand adaptation in opposite lateral directions following bidirectional optical displacement. Bull. Psychon. Soc. 16(6), 432–434 (1980)CrossRefGoogle Scholar
  80. Michel, C., Pisella, L., Prablanc, C., Rode, G., Rossetti, Y.: Enhancing visuomotor adaptation by reducing error signals: single-step (Aware) versus multiplestep (Unaware) exposure to wedge prisms. J. Cogn. Neurosci. 19(2), 341–350 (2007)CrossRefGoogle Scholar
  81. Mikaelian, H.: Lack of bilateral generalization of adaptation to auditory rearrangement. Percept. Psychophys. 11(3), 222–224 (1972)CrossRefGoogle Scholar
  82. Mikaelian, H.: Adaptation to displaced hearing: a nonproprioceptive change. J. Exp. Psychol. 103, 326–330 (1974)CrossRefGoogle Scholar
  83. Miyata, K., Varlet, M., Miura, A., Kudo, K., Keller, P.E.: Modulation of individual auditory-motor coordination dynamics through interpersonal visual coupling. Sci. Rep. 7, 16220 (2017).  https://doi.org/10.1038/s41598-017-16151-5CrossRefGoogle Scholar
  84. Moeslund, T.B., Hilton, A., Krueger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104(2–3), 90–126 (2006)CrossRefGoogle Scholar
  85. Morton, S.M., Bastian, A.J.: Prism adaptation during walking generalizes to reaching and requires the cerebellum. J. Neurosci. 92, 2497–2509 (2004)Google Scholar
  86. Mueller, G., Moeser, M. (eds.): Handbook of Engineering Acoustics. Springer Science and Business Media, Berlin (2012).  https://doi.org/10.1007/978-3-540-69460-1CrossRefGoogle Scholar
  87. Murgia, M., et al.: Ecological sounds affect breath duration more than artificial sounds. Psychol. Res. 80(1), 76–81 (2016)CrossRefGoogle Scholar
  88. Nieuwboer, A., et al.: The short-term effects of different cueing modalities on turn speed in people with Parkinson’s disease. Neurorehabil. Neural Repair 23(8), 831–836 (2009)CrossRefGoogle Scholar
  89. Nieuwboer, A., et al.: Cueing training in the home improves gait-related mobility in Parkinson’s disease: the RESCUE trial. J. Neurol. Neurosurg. Psychiatry 78(2), 134–140 (2007)CrossRefGoogle Scholar
  90. Nombela, C., Hughes, L.E., Owen, A.M., Grahn, J.A.: Into the groove: can rhythm influence Parkinson’s disease? Neurosci. Biobehav. Rev. 37(10), 2564–2570 (2013)CrossRefGoogle Scholar
  91. Nunes, M.E., Souza, M.G., Basso, L., Monteiro, C., Corrêa, U.C., Santos, S.: Frequency of provision of knowledge of performance on skill acquisition in older persons. Front. Psychol. 5, 1454 (2014)CrossRefGoogle Scholar
  92. Oostwoud Wijdenes, L., Medendorp, W.P.: State estimation for early feedback responses in reaching: intramodal or multimodal? Front. Integr. Neurosci. 11, 38 (2017).  https://doi.org/10.3389/fnint.2017.00038CrossRefGoogle Scholar
  93. Oppenheim, A.V.: Discrete-Time Signal Processing. Pearson Education India, Bangalore (1999)Google Scholar
  94. Oscari, F., Secoli, R., Avanzini, F., Rosati, G., Reinkensmeyer, D.J.: Substituting auditory for visual feedback to adapt to altered dynamic and kinematic environments during reaching. Exp. Brain Res. 221, 33–41 (2012)CrossRefGoogle Scholar
  95. Oullier, O., De Guzman, G.C., Jantzen, K.J., Lagarde, J., Scott Kelso, J.: Social coordination dynamics: measuring human bonding. Soc. Neurosci. 3(2), 178–192 (2008)CrossRefGoogle Scholar
  96. Prablanc, C., Tzavaras, A., Jeannerod, M.: Adaptation of the two arms to opposite prism displacements. Q. J. Exp. Psychol. 27(4), 667–671 (1975)CrossRefGoogle Scholar
  97. Rinehart, N.J., Bellgrove, M.A., Tonge, B.J., Brereton, A.V., Howells-Rankin, D., Bradshaw, J.L.: An examination of movement kinematics in young people with high-functioning autism and Asperger’s disorder: further evidence for a motor planning deficit. J. Autism Dev. Disord. 36(6), 757–767 (2006)CrossRefGoogle Scholar
  98. Reh, J., Hwang, T.H., Michalke, V., Effenberg, A.O.: Instruction and real-time sonification for gait rehabilitation after unilateral hip arthroplasty. In: 11th Joint Conference on Motor Control Learning Biomechanics Training, pp. 1–2. DVS (2016)Google Scholar
  99. Rios, J.A., White, E.: Fusion filter algorithm enhancements for a MEMS GPS/IMU, pp. 1–12. Crossbow Technology, Inc. (2002)Google Scholar
  100. Rueterbories, J., Spaich, E.G., Larsen, B., Andersen, O.K.: Methods for gait event detection and analysis in ambulatory systems. Med. Eng. Phys. 32(6), 545–552 (2010)CrossRefGoogle Scholar
  101. Sainburg, R., Wang, J.: Interlimb transfer of visuomotor rotations: independence of direction and final position information. Exp. Brain Res. 145, 437–447 (2002)CrossRefGoogle Scholar
  102. Schaefer, R.S.: Auditory rhythmic cueing in movement rehabilitation: findings and possible mechanisms. Philos. Trans. Roy. Soc. B: Biol. Sci. 369, 20130402 (2014)CrossRefGoogle Scholar
  103. Schmitz, G., Bock, O.: Properties of intermodal transfer after dual visuo- and auditory-motor adaptation. Hum. Mov. Sci. 55, 108–120 (2017)CrossRefGoogle Scholar
  104. Schmitz, G.: Visuo- und Audiomotorische Adaptation. Hofmann-Verlag, Schorndorf (2014). ISBN 978-3-7780-4850-4Google Scholar
  105. Schmitz, G.: Interference between adaptation to double steps and adaptation to rotated feedback in spite of differences in directional selectivity. Exp. Brain Res. 234, 1491–1504 (2016).  https://doi.org/10.1007/s00221-016-4559-yCrossRefGoogle Scholar
  106. Schmitz, G., Effenberg, A.O.: Sound joint actions in rowing and swimming. In: Meyer, C., Wedelstaedt, U.V. (eds.) Moving Bodies in Interaction - Interacting Bodies in Motion. John Benjamins Publishing Company, Amsterdam (2016)Google Scholar
  107. Schmitz, G., Bergmann, J., Effenberg, A.O., Krewer, C., Hwang, T.H., Mueller, F.: Movement sonification in stroke rehabilition. Front. Neurol. 9, 389 (2018)CrossRefGoogle Scholar
  108. Schmitz, G., Bock, O.: A comparison of sensorimotor adaptation in the visual and in the auditory modality. PLoS ONE 9(9), e107834 (2014)CrossRefGoogle Scholar
  109. Schmitz, G., Effenberg, A.O.: Perceptual effects of auditory information about own and other movements. In: 18th International Conference on Auditory Display, Atlanta, GA, USA (2012)Google Scholar
  110. Schmitz, G., Effenberg, A.O.: Schlagmann 2.0 – Bewegungsakustische Dimensionen interpersonaler Koordination im Mannschaftssport. Ger. J. Exerc. Sport Res. 47(3), 232–245 (2017)CrossRefGoogle Scholar
  111. Schmitz, G., Grigorova, V.: Alternating adaptation of eye and hand movements to opposite directed double steps. J. Mot. Behav. 49(3), 255–264 (2017).  https://doi.org/10.1080/00222895.2016.1191419CrossRefGoogle Scholar
  112. Schmitz, G., Kroeger, D., Effenberg, A.O.: A mobile sonification system for stroke rehabilitation. In: The 20th International Conference on Auditory Display, New York (2014)Google Scholar
  113. Schmitz, G., et al.: Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci. 14, 32 (2013).  https://doi.org/10.1186/1471-2202-14-32CrossRefGoogle Scholar
  114. Sebanz, N., Knoblich, G.: Prediction in joint action: what, when, and where. Top. Cogn. Sci. 1(2), 353–367 (2009)CrossRefGoogle Scholar
  115. Sebanz, N., Bekkering, H., Knoblich, G.: Joint action: bodies and minds moving together. Trends Cogn. Sci. 10(2), 70–76 (2006)CrossRefGoogle Scholar
  116. Seitz, A.R., Kim, R., Shams, L.: Sound facilitates visual learning. Curr. Biol. 16(14), 1422–1427 (2006)CrossRefGoogle Scholar
  117. Shadmehr, R., Smith, R.A., Krakauer, J.W.: Error correction, sensory prediction, and adaptation in motor control. Ann. Rev. Neurosci. 33, 89–108 (2010)CrossRefGoogle Scholar
  118. Shams, L., Seitz, A.R.: Benefits of multisensory learning. Trends Cogn. Sci. 12(11), 411–417 (2008)CrossRefGoogle Scholar
  119. Sharma, D.A., Chevidikunnan, M.F., Khan, F.R., Gaowgzeh, R.A.: Effectiveness of knowledge of result and knowledge of performance in the learning of a skilled motor activity by healthy young adults. J. Phys. Ther. Sci. 28(5), 1482–1486 (2016)CrossRefGoogle Scholar
  120. Silaghi, M.-C., Plänkers, R., Boulic, R., Fua, P., Thalmann, D.: Local and global skeleton fitting techniques for optical motion capture. In: Magnenat-Thalmann, N., Thalmann, D. (eds.) CAPTECH 1998. LNCS (LNAI), vol. 1537, pp. 26–40. Springer, Heidelberg (1998).  https://doi.org/10.1007/3-540-49384-0_3CrossRefGoogle Scholar
  121. Smith, M.A., Ghazizadeh, A., Shadmehr, R.: Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol. 4(6), e179 (2006).  https://doi.org/10.1371/journal.pbio.0040179CrossRefGoogle Scholar
  122. Spence, C.: Cross-modal perceptual organization. In: Wagemans, J. (ed.) The Oxford Handbook of Perceptual Organization. Oxford University Press, Oxford (2015)Google Scholar
  123. Spence, C., Driver, J. (eds.): Crossmodal Space and Crossmodal Attention. Oxford University Press, Oxford (2004)Google Scholar
  124. Stein, B.E., Meredith, M.A.: The Merging of the Senses. MIT Press, Cambridge (1993)Google Scholar
  125. Stein, B.E., Stanford, T.R.: Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9(4), 255–266 (2008)CrossRefGoogle Scholar
  126. Stoffregen, T.A., Bardy, B.G.: On specification and the senses. Behav. Brain Sci. 24, 195–213 (2001). Discussion 213-161CrossRefGoogle Scholar
  127. Suh, J.H., et al.: Effect of rhythmic auditory stimulation on gait and balance in hemiplegic stroke patients. NeuroRehabilitation 34(1), 193–199 (2014)MathSciNetGoogle Scholar
  128. Tagliabue, M., McIntyre, J.: A modular theory of multisensory integration for motor control. Front. Comput. Neurosci. 8, 1 (2014).  https://doi.org/10.3389/fncom.2014.00001CrossRefGoogle Scholar
  129. Tecchio, F., Salustri, C., Thaut, M.H., Pasqualetti, P., Rossini, P.: Conscious and preconscious adaptation to rhythmic auditory stimuli: a magnetoencephalographic study of human brain responses. Exp. Brain Res. 135(2), 222–230 (2000)CrossRefGoogle Scholar
  130. Thaut, M.H.: Neural basis of rhythmic timing networks in the human brain. Ann. N. Y. Acad. Sci. 999(1), 364–373 (2003)CrossRefGoogle Scholar
  131. Thaut, M.H.: Rhythm, Music, and the Brain: Scientific Foundations and Clinical Applications, vol. 7. Routledge, Abingdon (2005)Google Scholar
  132. Thaut, M.H., et al.: Neurologic music therapy improves executive function and emotional adjustment in traumatic brain injury rehabilitation. Ann. N. Y. Acad. Sci. 1169(1), 406–416 (2009)CrossRefGoogle Scholar
  133. Thaut, M.H., Leins, A.K., Rice, R.R., Argstatter, H., Kenyon, G.P., McIntosh, G.C., Fetter, M., et al.: Rhythmic auditory stimulation improves gait more than NDT/Bobath training in near-ambulatory patients early poststroke: a single-blind, randomized trial. Neurorehabil. Neural Repair 21(5), 455–459 (2007)CrossRefGoogle Scholar
  134. Thaut, M., Schleiffers, S., Davis, W.: Analysis of EMG activity in biceps and triceps muscle in an upper extremity gross motor task under the influence of auditory rhythm. J. Music Ther. 28(2), 64–88 (1991)CrossRefGoogle Scholar
  135. Tierney, A., Kraus, N.: The ability to move to a beat is linked to the consistency of neural responses to sound. J. Neurosci. 33(38), 14981–14988 (2013)CrossRefGoogle Scholar
  136. Torres, E.B., Heilman, K.M., Poizner, H.: Impaired endogenously evoked automated reaching in Parkinson’s disease. J. Neurosci. 31(49), 17848–17863 (2011)CrossRefGoogle Scholar
  137. Vesper, C., Butterfill, S., Knoblich, G., Sebanz, N.: A minimal architecture for joint action. Neural Netw. 23(8), 998–1003 (2010)CrossRefGoogle Scholar
  138. Weeks, D.L., Kordus, R.N.: Relative frequency of knowledge of performance and motor skill learning. Res. Q. Exerc. Sport 69(3), 224–230 (1998)CrossRefGoogle Scholar
  139. Welch, R.B.: Perceptual Modification. Adapting to Altered Sensory Environments. Academic Press, Cambridge (1978)Google Scholar
  140. Whitall, J., et al.: Bilateral and unilateral arm training improve motor function through differing neuroplastic mechanisms a single-blinded randomized controlled trial. Neurorehabil. Neural Repair 25(2), 118–129 (2011)CrossRefGoogle Scholar
  141. Wigmore, V., Tong, C., Flanagan, J.R.: Visuomotor rotations of varying size and direction compete for single internal model in working memory. J. Exp. Psychol. Hum. Percept. Perform. 28, 447–457 (2002)CrossRefGoogle Scholar
  142. Winstein, C.J.: Knowledge of results and motor learning—implications for physical therapy. Phys. Ther. 71(2), 140–149 (1991)CrossRefGoogle Scholar
  143. Wolpert, D.M., Kawato, M.: Multiple paired forward and inverse models for motor control. Neural Netw. 11(7–8), 1317–1329 (1998)CrossRefGoogle Scholar
  144. Wolpert, D.M., Diedrichsen, J., Flanagan, J.R.: Principles of sensorimotor learning. Nat. Rev. Neurosci. 12, 739–751 (2011)CrossRefGoogle Scholar
  145. Wolpert, D.M., Ghahramani, Z., Jordan, M.I.: An internal model for sensorimotor integration. Science 269, 1880–1882 (1995)CrossRefGoogle Scholar
  146. Young, W.R., Rodger, M.W., Craig, C.M.: Auditory observation of stepping actions can cue both spatial and temporal components of gait in Parkinson’s disease patients. Neuropsychologia 57, 140–153 (2014)CrossRefGoogle Scholar
  147. Zmigrod, S., Hommel, B.: Feature Integration across multimodal perception and action: a review. Multisensory Res. 26, 143–157 (2013)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Alfred O. Effenberg
    • 1
    Email author
  • Tong-Hun Hwang
    • 1
  • Shashank Ghai
    • 1
  • Gerd Schmitz
    • 1
  1. 1.Leibniz University HannoverHannoverGermany

Personalised recommendations