Experimental Brain Research

, Volume 221, Issue 1, pp 33–41 | Cite as

Substituting auditory for visual feedback to adapt to altered dynamic and kinematic environments during reaching

  • Fabio Oscari
  • Riccardo Secoli
  • Federico Avanzini
  • Giulio Rosati
  • David J. Reinkensmeyer
Research Article


The arm movement control system often relies on visual feedback to drive motor adaptation and to help specify desired trajectories. Here we studied whether kinematic errors that were indicated with auditory feedback could be used to control reaching in a way comparable with when vision was available. We randomized twenty healthy adult subjects to receive either visual or auditory feedback of their movement trajectory error with respect to a line as they performed timed reaching movements while holding a robotic joystick. We delivered auditory feedback using spatialized pink noise, the loudness and location of which reflected kinematic error. After a baseline period, we unexpectedly perturbed the reaching trajectories using a perpendicular viscous force field applied by the joystick. Subjects adapted to the force field as well with auditory feedback as they did with visual feedback and exhibited comparable after effects when the force field was removed. When we changed the reference trajectory to be a trapezoid instead of a line, subjects shifted their trajectories by about the same amount with either auditory or visual feedback of error. These results indicate that arm motor networks can readily incorporate auditory feedback to alter internal models and desired trajectories, a finding with implications for the organization of the arm motor control adaptation system as well as sensory substitution and motor training technologies.


Auditory feedback Visual feedback Sensory substitution Motor adaptation Motor training 


  1. Auvray M, Hanneton S, O’Regan JK (2007) Learning to perceive with a visuo-auditory substitution system: localisation and object recognition with ‘the voice’. Percept 36(3):416–430CrossRefGoogle Scholar
  2. Bach-y-Rita P, Kercel SW (2003) Sensory substitution and the human-machine interface. Trends Cogn Sci 7(12):541–546PubMedCrossRefGoogle Scholar
  3. Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L (1969) Vision substitution by tactile image projection. Trans Pac Coast Otoophthalmol Soc Annu Meet 221(5184):83–91Google Scholar
  4. Brown D, Macpherson T, Ward J (2011) Seeing with sound? Exploring different characteristics of a visual-to-auditory sensory substitution device. Percept 40(9):1120–1135CrossRefGoogle Scholar
  5. Burge J, Ernst MO, Banks MS (2008) The statistical determinants of adaptation rate in human reaching. J Vis 8(4):1–19PubMedCrossRefGoogle Scholar
  6. Collignon O, Champoux F, Voss P, Lepore F (2011) Sensory rehabilitation in the plastic brain. Prog Brain Res 191:211–231PubMedCrossRefGoogle Scholar
  7. Criscimagna-Hemminger SE, Bastian AJ, Shadmehr R (2010) Size of error affects cerebellar contributions to motor learning. J Neurophysiol 103(4):2275–2284PubMedCrossRefGoogle Scholar
  8. Fu KMG, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE (2003) Auditory cortical neurons respond to somatosensory stimulation. J Neurosci 23(20):7510–7515PubMedGoogle Scholar
  9. Ghahramani Z, Wolpert DM, Jordan MI (1997) Computational models for sensorimotor integration. In: Morasso PG, Sanguineti V (eds) Self-organization, computational maps and motor control. Elsevier, Amsterdam, pp 117–147CrossRefGoogle Scholar
  10. Hyvarinen J, Shelepin Y (1979) Distribution of visual and somatic functions in the parietal associative area 7 of the monkey. Brain Res 169(3):561–564PubMedCrossRefGoogle Scholar
  11. Kagerer F, Contreras-Vidal J (2009) Adaptation of sound localization induced by rotated visual feedback in reaching movements. Exp Brain Res 193:315–321PubMedCrossRefGoogle Scholar
  12. Klassen J, Tong C, Flanagan JR (2005) Learning and recall of incremental kinematic and dynamic sensorimotor transformations. Exp Brain Res 164(2):250–259PubMedCrossRefGoogle Scholar
  13. Molier BI, Prange GB, Buurke JH (2011) The role of visual feedback in conventional therapy and future research. In: Proceedings of ICVR2011, Zurich, SwitzerlandGoogle Scholar
  14. Orban de Xivry JJ, Criscimagna-Hemminger SE, Shadmehr R (2011) Contributions of the motor cortex to adaptive control of reaching depend on the perturbation schedule. Cereb Cortex 21(7):1475–1484PubMedCrossRefGoogle Scholar
  15. Pouget A, Deneve S, Duhamel J (2002) A computational perspective on the neural basis of multisensory spatial representations. Nat Rev Neurosci 3(9):741–747PubMedCrossRefGoogle Scholar
  16. Proulx MJ, Stoerig P, Ludowig E, Knoll I (2008) Seeing ‘where’ through the ears: effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. PLoS ONE 3(3):e1840PubMedCrossRefGoogle Scholar
  17. Rath M, Rocchesso D (2005) Continuous sonic feedback from a rolling ball. IEEE Multimedia 12(2):60–69CrossRefGoogle Scholar
  18. Reich L, Maidenbaum S, Amedi A (2012) The brain as a flexible task machine: implications for visual rehabilitation using noninvasive vs. invasive approaches. Curr Opin Neurol 25(1):86–95PubMedCrossRefGoogle Scholar
  19. Reuschel J, Rosler F, Henriques DYP, Fiehler K (2011) Testing the limits of optimal integration of visual and proprioceptive information of path trajectory. Exp Brain Res 209(4):619–630PubMedCrossRefGoogle Scholar
  20. Robertson JVG, Hoellinger T, Lindberg P, Bensmail D, Hanneton S, Roby-Brami A (2009) Effect of auditory feedback differs according to side of hemiparesis: a comparative pilot study. J NeuroEng Rehabil 6:45–56PubMedCrossRefGoogle Scholar
  21. Scheidt RA, Reinkensmeyer DJ, Conditt MA, Rymer WZ, Mussa-Ivaldi FA (2000) Persistence of motor adaptation during constrained, multi-joint, arm movements. J Neurophysiol 84(2):853–862PubMedGoogle Scholar
  22. Scheidt RA, Conditt MA, Secco EL, Mussa-Ivaldi FA (2005) Interaction of visual and proprioceptive feedback during adaptation of human reaching movements. J Neurophysiol 93(6):3200–3213PubMedCrossRefGoogle Scholar
  23. Schroeder CE, Smiley J, Fu KG, Mcginnis T (2003) Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. Int J Psychophysiol 50(1–2):5–17PubMedCrossRefGoogle Scholar
  24. Secoli R, Rosati G, Reinkensmeyer DJ (2009) Using sound feedback to counteract visual distractor during robot-assisted movement training. In: Proceedings of HAVE2009, Lecco, ItalyGoogle Scholar
  25. Secoli R, Milot MH, Rosati G, Reinkensmeyer DJ (2011) Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke. J NeuroEng Rehabil 8(1):21–30Google Scholar
  26. Shadmehr R, Mussa-Ivaldi FA (1994) Adaptive representation of dynamics during learning of a motor task. J Neurosci 14(5 pt 2):3208–3224PubMedGoogle Scholar
  27. Takahashi CD, Nemet D, Rose-Gottron CM, Larson JK, Cooper DM, Reinkensmeyer DJ (2006) Effect of muscle fatigue on internal model formation and retention during reaching with the arm. J Appl Physiol 100(2):695–706PubMedCrossRefGoogle Scholar
  28. Taylor JA, Thoroughman KA (2007) Divided attention impairs human motor adaptation but not feedback control. J Neurophysiol 98(1):317–326PubMedCrossRefGoogle Scholar
  29. Thielman G (2010) Rehabilitation of reaching poststroke: a randomized pilot investigation of tactile versus auditory feedback for trunk control. J Neurol Phys Ther 34(3):138–144PubMedGoogle Scholar
  30. Thoroughman KA, Shadmehr R (2000) Learning of action through adaptive combination of motor primitives. Stroke 407:742–747Google Scholar
  31. Wolpert DM, Ghahramani Z, Jordan M (1995) Are arm trajectories planned in kinematic or dynamic coordinates? An adaptation study. Exp Brain Res 103(3):460–470PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Fabio Oscari
    • 1
  • Riccardo Secoli
    • 2
  • Federico Avanzini
    • 3
  • Giulio Rosati
    • 1
  • David J. Reinkensmeyer
    • 2
  1. 1.Department of Management and EngineeringUniversity of PaduaPaduaItaly
  2. 2.Department of Mechanical and Aerospace EngineeringUniversity of CaliforniaIrvineUSA
  3. 3.Department of Information EngineeringUniversity of PaduaPaduaItaly

Personalised recommendations