Experimental Brain Research

, Volume 233, Issue 3, pp 909–925 | Cite as

Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning

  • Roland Sigrist
  • Georg Rauter
  • Laura Marchal-Crespo
  • Robert Riener
  • Peter Wolf
Research Article


Concurrent augmented feedback has been shown to be less effective for learning simple motor tasks than for complex tasks. However, as mostly artificial tasks have been investigated, transfer of results to tasks in sports and rehabilitation remains unknown. Therefore, in this study, the effect of different concurrent feedback was evaluated in trunk-arm rowing. It was then investigated whether multimodal audiovisual and visuohaptic feedback are more effective for learning than visual feedback only. Naïve subjects (N = 24) trained in three groups on a highly realistic virtual reality-based rowing simulator. In the visual feedback group, the subject’s oar was superimposed to the target oar, which continuously became more transparent when the deviation between the oars decreased. Moreover, a trace of the subject’s trajectory emerged if deviations exceeded a threshold. The audiovisual feedback group trained with oar movement sonification in addition to visual feedback to facilitate learning of the velocity profile. In the visuohaptic group, the oar movement was inhibited by path deviation-dependent braking forces to enhance learning of spatial aspects. All groups significantly decreased the spatial error (tendency in visual group) and velocity error from baseline to the retention tests. Audiovisual feedback fostered learning of the velocity profile significantly more than visuohaptic feedback. The study revealed that well-designed concurrent feedback fosters complex task learning, especially if the advantages of different modalities are exploited. Further studies should analyze the impact of within-feedback design parameters and the transferability of the results to other tasks in sports and rehabilitation.


Augmented feedback Movement sonification Haptic guidance Multimodal feedback Robot-assisted learning Rowing simulator 


  1. Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262PubMedCrossRefGoogle Scholar
  2. Blandin Y, Toussaint L, Shea CH (2008) Specificity of practice: interaction between concurrent sensory information and terminal feedback. J Exp Psychol Learn Mem Cogn 34(4):994–1000PubMedCrossRefGoogle Scholar
  3. Braun DA, Mehring C, Wolpert DM (2010) Structure learning in action. Behav Brain Res 206(2):157–165PubMedCentralPubMedCrossRefGoogle Scholar
  4. Brown R, Palmer C (2012) Auditory–motor learning influences auditory memory for music. Mem Cognit 40(4):567–578PubMedCrossRefGoogle Scholar
  5. Buchanan J, Wang C (2012) Overcoming the guidance effect in motor skill learning: feedback all the time can be beneficial. Exp Brain Res 219(2):305–320PubMedCrossRefGoogle Scholar
  6. Burke JL, Prewett MS, Gray AA, Yang L, Stilson FRB, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In: Proceedings of the 8th international conference on multimodal interfaces, New York, NY, USA, pp 108–117Google Scholar
  7. Carson RG, Kelso JAS (2004) Governing coordination: behavioural principles and neural correlates. Exp Brain Res 154(3):267–274PubMedCrossRefGoogle Scholar
  8. Cesqui B, Aliboni S, Mazzoleni S, Carrozza M, Posteraro F, Micera S (2008) On the use of divergent force fields in robot-mediated neurorehabilitation. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 854–861Google Scholar
  9. Chen X, Agrawal S (2013) Assisting versus repelling force-feedback for learning of a line following task in a wheelchair. IEEE Trans Neural Syst Rehabil Eng 21(6):959–968PubMedCrossRefGoogle Scholar
  10. Chen JL, Penhune VB, Zatorre RJ (2008) Listening to musical rhythms recruits motor regions of the brain. Cereb Cortex 18(12):2844–2854PubMedCrossRefGoogle Scholar
  11. Chiviacowsky S, Wulf G (2007) Feedback after good trials enhances learning. Res Q Exerc Sport 78:40–47PubMedCrossRefGoogle Scholar
  12. Chollet D, Madani M, Micallef JP (1992) Effects of two types of biomechanical bio-feedback on crawl performance. In: MacLaren D, Reilly T, Lees A (eds) Biomechanics and medicine in swimming. E & FN Spon, London, pp 57–62Google Scholar
  13. Effenberg AO (2005) Movement sonification: effects on perception and action. IEEE Multimedia 12(2):53–59CrossRefGoogle Scholar
  14. Emken J, Reinkensmeyer DJ (2005) Robot-enhanced motor learning: accelerating internal model formation during locomotion by transient dynamic amplification. IEEE Trans Neural Syst Rehabil Eng 13(1):33–39PubMedCrossRefGoogle Scholar
  15. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433PubMedCrossRefGoogle Scholar
  16. Freides D (1974) Human information processing and sensory modality: cross-modal functions information complexity memory and deficit. Psychol Bull 81(5):284–310PubMedCrossRefGoogle Scholar
  17. Giese MA, Poggio T (2000) Morphable models for the analysis and synthesis of complex motion patterns. Int J Comput Vis 38(1):59–73CrossRefGoogle Scholar
  18. Guadagnoli M, Kohl R (2001) Knowledge of results for motor learning: relationship between error estimation and knowledge of results frequency. J Mot Behav 33(2):217–224PubMedCrossRefGoogle Scholar
  19. Guadagnoli MA, Lee TD (2004) Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. J Mot Behav 36(2):212–224PubMedCrossRefGoogle Scholar
  20. Hale K, Stanney K (2004) Deriving haptic design guidelines from human physiological psychophysical and neurological foundations. IEEE Comput Graph Appl 24(2):33–39PubMedCrossRefGoogle Scholar
  21. Holden MK (2005) Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8(3):187–211PubMedCrossRefGoogle Scholar
  22. Huang F, Patton J (2013) Augmented dynamics and motor exploration as training for stroke. IEEE Trans Biomed Eng 60(3):838–844PubMedCrossRefGoogle Scholar
  23. Huang H, Ingalls T, Olson L, Ganley K, Rikakis T, He J (2005) Interactive multimodal biofeedback for task-oriented neural rehabilitation. In: 27th annual international conference of the engineering in medicine and biology society 2005 IEEE-EMBS 2005, Shanghai, pp 2547–2550Google Scholar
  24. Hubbard T (2013) Auditory imagery contains more than audition. In: Lacey S, Lawson R (eds) Multisensory imagery. Springer, New York, pp 221–247CrossRefGoogle Scholar
  25. Huegel J, O’Malley MK (2010) Progressive haptic and visual guidance for training in a virtual dynamic task. In: haptics symposium 2010 IEEE, pp 343–350Google Scholar
  26. Israel J, Campbell D, Kahn J, Hornby T (2006) Metabolic costs and muscle activity patterns during robotic-and therapist-assisted treadmill walking in individuals with incomplete spinal cord injury. Phys Ther 86(11):1466–1478PubMedCrossRefGoogle Scholar
  27. Kapur A, Tzanetakis G, Virji-Babul N, Wang G, Cook PR (2005) A framework for sonification of vicon motion capture data. In: Proceedings of the 8th conference on digital audio effects, Madrid, SpainGoogle Scholar
  28. Kennedy D, Boyle J, Shea C (2013) The role of auditory and visual models in the production of bimanual tapping patterns. Exp Brain Res 224(4):507–518PubMedCrossRefGoogle Scholar
  29. Keysers C, Gazzola V (2010) Social neuroscience: mirror neurons recorded in humans. Curr Biol 20(8):R353–R354PubMedCrossRefGoogle Scholar
  30. Kim RS, Seitz AR, Shams L (2008) Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One 3(1):e1532PubMedCentralPubMedCrossRefGoogle Scholar
  31. Kovacs AJ, Shea CH (2011) The learning of 90° continuous relative phase with and without lissajous feedback: external and internally generated bimanual coordination. Acta Psychol 136(3):311–320CrossRefGoogle Scholar
  32. Krakauer J, Mazzoni P (2011) Human sensorimotor learning: adaptation skill and beyond. Curr Opin Neurobiol 21(4):636–644PubMedCrossRefGoogle Scholar
  33. Kramer G (1994) Auditory display: sonification audification and auditory interfaces. Addison-Wesley, Reading MAGoogle Scholar
  34. Krebs HI, Palazzolo JJ, Dipietro L, Ferraro M, Krol J, Rannekleiv K, Volpe BT, Hogan N (2003) Rehabilitation robotics: performance-based progressive robot-assisted therapy. Auton Robots 15(1):7–20CrossRefGoogle Scholar
  35. Lahav A, Saltzman E, Schlaug G (2007) Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J Neurosci 27(2):308–314PubMedCrossRefGoogle Scholar
  36. Liebermann DG, Katz L, Hughes MD, Bartlett RM, McClements J, Franks IM (2002) Advances in the application of information technology to sport performance. J Sports Sci 20(10):755–769PubMedCrossRefGoogle Scholar
  37. Liu D, Todorov E (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27(35):9354–9368PubMedCrossRefGoogle Scholar
  38. Liu J, Wrisberg CA (1997) The effect of knowledge of results delay and the subjective estimation of movement form on the acquisition and retention of a motor skill. Res Q Exerc Sport 68(2):145–151PubMedCrossRefGoogle Scholar
  39. Lüttgen J, Heuer H (2012a) The influence of haptic guidance on the production of spatio-temporal patterns. Hum Mov Sci 31(3):519–528PubMedCrossRefGoogle Scholar
  40. Lüttgen J, Heuer H (2012b) Robotic guidance benefits the learning of dynamic but not of spatial movement characteristics. Exp Brain Res 222(1–2):1–9PubMedCrossRefGoogle Scholar
  41. Lüttgen J, Heuer H (2013) The influence of robotic guidance on different types of motor timing. J Mot Behav 45(3):249–258PubMedCrossRefGoogle Scholar
  42. Marchal-Crespo L, Reinkensmeyer DJ (2008a) Effect of robotic guidance on motor learning of a timing task. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 199–204Google Scholar
  43. Marchal-Crespo L, Reinkensmeyer DJ (2008b) Haptic guidance can enhance motor learning of a steering task. J Mot Behav 40(6):545–557PubMedCrossRefGoogle Scholar
  44. Marchal-Crespo L, Reinkensmeyer DJ (2009) Review of control strategies for robotic movement training after neurologic injury. J Neuroeng Rehabil 6(1):20PubMedCentralPubMedCrossRefGoogle Scholar
  45. Marchal-Crespo L, Furumasu J, Reinkensmeyer DJ (2010) A robotic wheelchair trainer: design overview and a feasibility study. J Neuroeng Rehabil 7(1):40–51PubMedCentralPubMedCrossRefGoogle Scholar
  46. Marchal-Crespo L, Raai M, Rauter G, Wolf P, Riener R (2013) The effect of haptic guidance and visual feedback on learning a complex tennis task. Exp Brain Res 231(3):277–291PubMedCrossRefGoogle Scholar
  47. Marschall F, Bund A, Wiemeyer J (2007) Does frequent feedback really degrade learning? A meta analysis. E-Journal Bewegung und Training 1:75–86Google Scholar
  48. Milot MH, Marchal-Crespo L, Green CS, Cramer SC, Reinkensmeyer DJ (2010) Comparison of error-amplification and haptic-guidance training techniques for learning of a timing-based motor task by healthy individuals. Exp Brain Res 201(2):119–131PubMedCrossRefGoogle Scholar
  49. Minogue J, Jones MG (2006) Haptics in Education: exploring an untapped sensory modality. Rev Educ Res 76(3):3–17CrossRefGoogle Scholar
  50. Patton JL, Stoykov M, Kovic M, Mussa-Ivaldi F (2006) Evaluation of robotic training forces that either enhance or reduce error in chronic hemiparetic stroke survivors. Exp Brain Res 168(3):368–383PubMedCrossRefGoogle Scholar
  51. Patton JL, Wei YJ, Bajaj P, Scheidt RA (2013) Visuomotor learning enhanced by augmenting instantaneous trajectory error feedback during reaching. PLoS One 8(1):e46466PubMedCentralPubMedCrossRefGoogle Scholar
  52. Proteau L (1992) On the specificity of learning and the role of visual information for movement control. In: Proteau L, Elliott D (eds) Vision and motor control, vol 85. North-Holland, Amsterdam, pp 67–103CrossRefGoogle Scholar
  53. Rauter G, von Zitzewitz J, Duschau-Wicke A, Vallery H, Riener R (2010) A tendon based parallel robot applied to motor learning in sports. In: 3rd IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics (BioRob) 2010, Tokyo, Japan, pp 82–87Google Scholar
  54. Rauter G, Sigrist R, Marchal-Crespo L, Vallery H, Riener R, Wolf P (2011) Assistance or challenge? Filling a gap in user-cooperative control. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 3068–3073Google Scholar
  55. Rauter G, Sigrist R, Koch C, Crivelli F, van Raai M, Riener R, Wolf P (2013) Transfer of complex skill learning from virtual to real rowing. PLoS One 8(12):1–18CrossRefGoogle Scholar
  56. Reinkensmeyer DJ, Akoner O, Ferris D, Gordon K (2009) Slacking by the human motor system: computational models and implications for robotic orthoses. In: Engineering in medicine and biology society 2009 EMBC 2009 annual international conference of the IEEE, pp 2129–2132Google Scholar
  57. Ribeiro DC, Sole G, Abbott JH, Milosavljevic S (2011) Extrinsic feedback and management of low back pain: a critical review of the literature. Man Ther 16(3):231–239PubMedCrossRefGoogle Scholar
  58. Robin C, Toussaint L, Blandin Y, Proteau L (2005) Specificity of learning in a video-aiming task: modifying the salience of dynamic visual cues. J Mot Behav 37(5):367–376PubMedCrossRefGoogle Scholar
  59. Ronsse R, Puttemans V, Coxon JP, Goble DJ, Wagemans J, Wenderoth N, Swinnen SP (2011) Motor learning with augmented feedback: modality-dependent behavioral and neural consequences. Cereb Cortex 21(6):1283–1294PubMedCrossRefGoogle Scholar
  60. Salmoni S (1984) Knowledge of results and motor learning A review and critical reappraisal. Psychol Bull 95(3):355–386PubMedCrossRefGoogle Scholar
  61. Schaffert N, Mattes K, Effenberg AO (2011) An investigation of online acoustic information for elite rowers in on-water training conditions. J Hum Sport Exerc 6(2):392–405CrossRefGoogle Scholar
  62. Schmidt RA (1991) Frequent augmented feedback can degrade learning: evidence and interpretations. Tutor Motor Neurosci 62:59–75CrossRefGoogle Scholar
  63. Schmidt RA, Wrisberg C (2008) Motor learning and performance: a situation-based learning approach. Human Kinetics, Champaign, ILGoogle Scholar
  64. Schmidt RA, Wulf G (1997) Continuous concurrent feedback degrades skill learning: implications for training and simulation. Hum Factors 39(4):509–525PubMedCrossRefGoogle Scholar
  65. Schmidt RA, Young DE, Swinnen S, Shapiro DC (1989) Summary knowledge of results for skill acquisition: support for the guidance hypothesis. J Exp Psychol Learn Mem Cogn 15(2):352–359PubMedCrossRefGoogle Scholar
  66. Schmitz G, Mohammadi B, Hammer A, Heldmann M, Samii A, Munte T, Effenberg A (2013) Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci 14(1):1–11CrossRefGoogle Scholar
  67. Secoli R, Milot M, Rosati G, Reinkensmeyer DJ (2011) Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke. J Neuroeng Rehabil 8(1):1–10CrossRefGoogle Scholar
  68. Seitz AR, Dinse HR (2007) A common framework for perceptual learning. Curr Opin Neurobiol 17(2):148–153PubMedCrossRefGoogle Scholar
  69. Seitz AR, Kim R, Shams R (2006) Sound facilitates visual learning. Curr Biol 16(14):1422–1427PubMedCrossRefGoogle Scholar
  70. Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417PubMedCrossRefGoogle Scholar
  71. Sigrist R, Schellenberg J, Rauter G, Broggi S, Riener R, Wolf P (2011) Visual and auditory augmented concurrent feedback in a complex motor task. Presence Teleop Virt 20(1):15–32CrossRefGoogle Scholar
  72. Sigrist R, Rauter G, Riener R, Wolf P (2013a) Augmented visual auditory haptic and multimodal feedback in motor learning: a review. Psychon Bull Rev 20:21–53PubMedCrossRefGoogle Scholar
  73. Sigrist R, Rauter G, Riener R, Wolf P (2013b) Terminal feedback outperforms concurrent visual auditory and haptic feedback in learning a complex rowing-type task. J Mot Behav 45(6):455–472PubMedCrossRefGoogle Scholar
  74. Snodgrass SJ, Rivett DA, Robertson VJ, Stojanovski E (2010) Real-time feedback improves accuracy of manually applied forces during cervical spine mobilization. Man Ther 15:19–25PubMedCrossRefGoogle Scholar
  75. Swinnen SP, Schmidt RA, Nicholson DE, Shapiro DC (1990) Information feedback for skill acquisition: instantaneous knowledge of results degrades learning. J Exp Psychol Learn Mem Cogn 16(4):706–716CrossRefGoogle Scholar
  76. Swinnen SP, Lee TD, Verschueren S, Serrien DJ, Bogaerds H (1997) Interlimb coordination: learning and transfer under different feedback conditions. Hum Mov Sci 16(6):749–785CrossRefGoogle Scholar
  77. Thoroughman K, Shadmehr R (2000) Learning of action through adaptive combination of motor primitives. Nature 407(6805):742PubMedCentralPubMedCrossRefGoogle Scholar
  78. Timmermans AAA, Seelen HAM, Willmann RD, Kingma H (2009) Technology-assisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroeng Rehabil 6:1PubMedCentralPubMedCrossRefGoogle Scholar
  79. Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7:907–915PubMedCentralPubMedCrossRefGoogle Scholar
  80. Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5:1226–1235PubMedCrossRefGoogle Scholar
  81. Todorov E, Shadmehr R, Bizzi E (1997) Augmented feedback presented in a virtual environment accelerates learning of a difficult motor task. J Mot Behav 29(2):147–158PubMedCrossRefGoogle Scholar
  82. van Beers RJ (2009) Motor learning is optimally tuned to the properties of motor noise. Neuron 63(3):406–417PubMedCrossRefGoogle Scholar
  83. van Beers RJ, Sittig AC, Gon JJ (1999) Integration of proprioceptive and visual position-information: an experimentally supported model. J Neurophysiol 81(3):1355PubMedGoogle Scholar
  84. van der Linden DW, Cauraugh JH, Greene TA (1993) The effect of frequency of kinetic feedback on learning an isometric force production task in nondisabled subjects. Phys Ther 73(2):79–87Google Scholar
  85. Vlachos M, Hadjieleftheriou M, Gunopulos D, Keogh E (2003) Indexing multi-dimensional time-series with support for multiple distance measures. In: proceedings of the ninth acm sigkdd international conference on knowledge discovery and data mining KDD’03, New York, NY, USA, pp 216–225Google Scholar
  86. von Zitzewitz J, Wolf P, Novakovic V, Wellner M, Rauter G, Brunschweiler A, Riener R (2008) Real-time rowing simulator with multimodal feedback. Sports Technol 1(6):257–266CrossRefGoogle Scholar
  87. Wei K, Körding K (2009) Relevance of error: what drives motor adaptation? J Neurophysiol 101(2):655–664PubMedCentralPubMedCrossRefGoogle Scholar
  88. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667PubMedCrossRefGoogle Scholar
  89. Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3(2):159–177CrossRefGoogle Scholar
  90. Winstein CJ (1991) Knowledge of results and motor learning—implications for physical therapy. Phys Ther 71(2):140–149PubMedGoogle Scholar
  91. Winstein CJ, Pohl PS, Cardinale C, Green A, Scholtz L, Waters CS (1996) Learning a partial-weight-bearing skill: effectiveness of two forms of feedback. Phys Ther 76(9):985–993PubMedGoogle Scholar
  92. Wishart LR, Lee TD, Cunningham SJ, Murdoch JE (2002) Age-related differences and the role of augmented visual feedback in learning a bimanual coordination pattern. Acta Psychol (Amst) 110(2–3):247–263CrossRefGoogle Scholar
  93. Wolpert D, Flanagan J (2010) Motor learning. Curr Biol 20(11):R467–R472PubMedCrossRefGoogle Scholar
  94. Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12:739–749PubMedGoogle Scholar
  95. Wulf G (2007) Self-controlled practice enhances motor learning: implications for physiotherapy. Physiotherapy 93(2):96–101CrossRefGoogle Scholar
  96. Wulf G, Shea CH (2002) Principles derived from the study of simple skills do not generalize to complex skill learning. Psychon Bull Rev 9(2):185–211PubMedCrossRefGoogle Scholar
  97. Wulf G, Shea CH, Matschiner S (1998) Frequent feedback enhances complex motor skill learning. J Mot Behav 30(2):180–192PubMedCrossRefGoogle Scholar
  98. Wulf G, Hörger M, Shea CH (1999) Benefits of blocked over serial feedback on complex motor skill learning. J Mot Behav 31(1):95–103PubMedCrossRefGoogle Scholar
  99. Zatorre RJ, Chen JL, Penhune VB (2007) When the brain plays music: auditory–motor interactions in music perception and production. Nat Rev Neurosci 8(7):547–558PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Roland Sigrist
    • 1
    • 2
  • Georg Rauter
    • 1
    • 2
  • Laura Marchal-Crespo
    • 1
    • 2
  • Robert Riener
    • 1
    • 2
  • Peter Wolf
    • 1
    • 2
  1. 1.Sensory-Motor Systems (SMS) Lab, Institute of Robotics and Intelligent Systems (IRIS)ETH ZurichZurichSwitzerland
  2. 2.Medical FacultyUniversity of ZurichZurichSwitzerland

Personalised recommendations