Effects of Experience, Training and Expertise on Multisensory Perception: Investigating the Link between Brain and Behavior

  • Scott A. Love
  • Frank E. Pollick
  • Karin Petrini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7403)


The ability to successfully integrate information from different senses is of paramount importance for perceiving the world and has been shown to change with experience. We first review how experience, in particular musical experience, brings about changes in our ability to fuse together sensory information about the world. We next discuss evidence from drumming studies that demonstrate how the perception of audiovisual synchrony depends on experience. These studies show that drummers are more robust than novices to perturbations of the audiovisual signals and appear to use different neural mechanisms in fusing sight and sound. Finally, we examine how experience influences audiovisual speech perception. We present an experiment investigating how perceiving an unfamiliar language influences judgments of temporal synchrony of the audiovisual speech signal. These results highlight the influence of both the listener’s experience with hearing an unfamiliar language as well as the speaker’s experience with producing non-native words.


multisensory audiovisual perception expertise drumming 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Landy, M.S., et al.: Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35(3), 389–412 (1995)CrossRefGoogle Scholar
  2. 2.
    Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870), 429–433 (2002)CrossRefGoogle Scholar
  3. 3.
    Green, C.S., Bavelier, D.: Action-video-game experience alters the spatial resolution of vision. Psychol. Sci. 18(1), 88–94 (2007)CrossRefGoogle Scholar
  4. 4.
    Simmons, R.W., Locher, P.J.: Role of extended perceptual experience upon haptic perception of nonrepresentational shapes. Percept. Mot. Skills 48(3 Pt. 1), 987–991 (1979)CrossRefGoogle Scholar
  5. 5.
    Kisilevsky, B.S., et al.: Effects of experience on fetal voice recognition. Psychol. Sci. 14(3), 220–224 (2003)CrossRefGoogle Scholar
  6. 6.
    Atkins, J.E., Fiser, J., Jacobs, R.A.: Experience-dependent visual cue integration based on consistencies between visual and haptic percepts. Vision Res. 41(4), 449–461 (2001)CrossRefGoogle Scholar
  7. 7.
    Jacobs, R.A., Fine, I.: Experience-dependent integration of texture and motion cues to depth. Vision Res. 39(24), 4062–4075 (1999)CrossRefGoogle Scholar
  8. 8.
    Powers III, A.R., Hillock, A.R., Wallace, M.T.: Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29(39), 12265–12674 (2009)CrossRefGoogle Scholar
  9. 9.
    Mamassian, P., Goutcher, R.: Prior knowledge on the illumination position. Cognition 81(1), B1-B9 (2001)Google Scholar
  10. 10.
    Mamassian, P., Landy, M.S.: Interaction of visual prior constraints. Vision Res. 41(20), 2653–2668 (2001)CrossRefGoogle Scholar
  11. 11.
    Mondloch, C.J., et al.: Face perception during early infancy. Psychol. Sci. 10(5), 419–422 (1999)CrossRefGoogle Scholar
  12. 12.
    Turati, C.: Why faces are not special to newborns: An alternative account of the face preference. Current Directions in Psychological Science 13(1), 5–8 (2004)CrossRefGoogle Scholar
  13. 13.
    Hershber, W.: Attached-Shadow Orientation Perceived as Depth by Chickens Reared in an Environment Illuminated from Below. Journal of Comparative and Physiological Psychology 73(3), 407-&. (1970)Google Scholar
  14. 14.
    Gregory, R.L.: Knowledge in perception and illusion. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences 352(1358), 1121–1127 (1997)CrossRefGoogle Scholar
  15. 15.
    Adams, W.J., Graf, E.W., Ernst, M.O.: Experience can change the ’light-from-above’ prior. Nat. Neurosci. 7(10), 1057–1058 (2004)CrossRefGoogle Scholar
  16. 16.
    Dayan, E., Cohen, L.G.: Neuroplasticity subserving motor skill learning. Neuron 72(3), 443–454 (2011)CrossRefGoogle Scholar
  17. 17.
    May, A.: Experience-dependent structural plasticity in the adult human brain. Trends Cogn. Sci. 15(10), 475–482 (2011)CrossRefGoogle Scholar
  18. 18.
    Pascual-Leone, A., et al.: The plastic human brain cortex. Annu. Rev. Neurosci. 28, 377–401 (2005)CrossRefGoogle Scholar
  19. 19.
    Latinus, M., Crabbe, F., Belin, P.: Learning-induced changes in the cerebral processing of voice identity. Cereb Cortex 21(12), 2820–2828 (2011)CrossRefGoogle Scholar
  20. 20.
    Bentin, S., et al.: Electrophysiological Studies of Face Perception in Humans. J. Cogn. Neurosci. 8(6), 551–565 (1996)CrossRefGoogle Scholar
  21. 21.
    Rossion, B., et al.: Expertise training with novel objects leads to left-lateralized facelike electrophysiological responses. Psychol. Sci. 13(3), 250–257 (2002)CrossRefGoogle Scholar
  22. 22.
    Bukach, C.M., et al.: Does acquisition of Greeble expertise in prosopagnosia rule out a domain-general deficit? Neuropsychologia 50(2), 289–304 (2012)CrossRefGoogle Scholar
  23. 23.
    Tanaka, J.W., Curran, T.: A neural basis for expert object recognition. Psychol. Sci. 12(1), 43–47 (2001)CrossRefGoogle Scholar
  24. 24.
    Busey, T.A., Vanderkolk, J.R.: Behavioral and electrophysiological evidence for configural processing in fingerprint experts. Vision Res. 45(4), 431–448 (2005)CrossRefGoogle Scholar
  25. 25.
    Busey, T.A., Parada, F.J.: The nature of expertise in fingerprint examiners. Psychon. Bull. Rev. 17(2), 155–160 (2010)CrossRefGoogle Scholar
  26. 26.
    Calvo-Merino, B., et al.: Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex 15(8), 1243–1249 (2005)CrossRefGoogle Scholar
  27. 27.
    Calvo-Merino, B., et al.: Seeing or doing? Influence of visual and motor familiarity in action observation. Curr. Biol. 16(19), 1905–1910 (2006)CrossRefGoogle Scholar
  28. 28.
    Cross, E.S., Hamilton, A.F., Grafton, S.T.: Building a motor simulation de novo: observation of dance by dancers. Neuroimage 31(3), 1257–1267 (2006)CrossRefGoogle Scholar
  29. 29.
    Maguire, E.A., et al.: Navigation-related structural change in the hippocampi of taxi drivers. Proc. Natl. Acad. Sci. U S A 97(8), 4398–4403 (2000)CrossRefGoogle Scholar
  30. 30.
    Woollett, K., Maguire, E.A.: Acquiring "the Knowledge" of London’s layout drives structural brain changes. Curr. Biol. 21(24), 2109–2114 (2011)CrossRefGoogle Scholar
  31. 31.
    Hufner, K., et al.: Structural and functional plasticity of the hippocampal formation in professional dancers and slackliners. Hippocampus 21(8), 855–865 (2011)Google Scholar
  32. 32.
    Bezzola, L., et al.: Training-induced neural plasticity in golf novices. J. Neurosci. 31(35), 12444–12448 (2011)CrossRefGoogle Scholar
  33. 33.
    Johansen-Berg, H., et al.: Integrity of white matter in the corpus callosum correlates with bimanual co-ordination skills. Neuroimage 36(suppl. 2), T16–T21 (2007)CrossRefGoogle Scholar
  34. 34.
    Tuch, D.S., et al.: Choice reaction time performance correlates with diffusion anisotropy in white matter pathways supporting visuospatial attention. Proc. Natl. Acad. Sci. U S A 102(34), 12212–12217 (2005)CrossRefGoogle Scholar
  35. 35.
    Draganski, B., et al.: Neuroplasticity: changes in grey matter induced by training. Nature 427(6972), 311–312 (2004)CrossRefGoogle Scholar
  36. 36.
    Scholz, J., et al.: Training induces changes in white-matter architecture. Nat. Neurosci. 12(11), 1370–1371 (2009)CrossRefGoogle Scholar
  37. 37.
    Bengtsson, S.L., et al.: Extensive piano practicing has regionally specific effects on white matter development. Nat. Neurosci. 8(9), 1148–1150 (2005)CrossRefGoogle Scholar
  38. 38.
    Bermudez, P., et al.: Neuroanatomical correlates of musicianship as revealed by cortical thickness and voxel-based morphometry. Cereb Cortex 19(7), 1583–1596 (2009)CrossRefGoogle Scholar
  39. 39.
    Gaser, C., Schlaug, G.: Brain structures differ between musicians and non-musicians. J. Neurosci. 23(27), 9240–9245 (2003)Google Scholar
  40. 40.
    Hutchinson, S., et al.: Cerebellar volume of musicians. Cereb Cortex 13(9), 943–949 (2003)CrossRefGoogle Scholar
  41. 41.
    Halwani, G.F., et al.: Effects of practice and experience on the arcuate fasciculus: comparing singers, instrumentalists, and non-musicians. Front Psychol. 2, 156 (2011)CrossRefGoogle Scholar
  42. 42.
    Imfeld, A., et al.: White matter plasticity in the corticospinal tract of musicians: a diffusion tensor imaging study. Neuroimage 46(3), 600–607 (2009)CrossRefGoogle Scholar
  43. 43.
    Schmithorst, V.J., Wilke, M.: Differences in white matter architecture between musicians and non-musicians: a diffusion tensor imaging study. Neurosci. Lett. 321(1-2), 57–60 (2002)CrossRefGoogle Scholar
  44. 44.
    Schlaug, G., et al.: In vivo evidence of structural brain asymmetry in musicians. Science 267(5198), 699–701 (1995)CrossRefGoogle Scholar
  45. 45.
    Ozturk, A.H., et al.: Morphometric comparison of the human corpus callosum in professional musicians and non-musicians by using in vivo magnetic resonance imaging. J. Neuroradiol. 29(1), 29–34 (2002)Google Scholar
  46. 46.
    Tervaniemi, M.: Musicians–same or different? Ann. N Y Acad. Sci. 1169, 151–156 (2009)CrossRefGoogle Scholar
  47. 47.
    Aydin, K., et al.: Quantitative proton MR spectroscopic findings of cortical reorganization in the auditory cortex of musicians. AJNR Am. J. Neuroradiol. 26(1), 128–136 (2005)Google Scholar
  48. 48.
    Elbert, T., et al.: Increased cortical representation of the fingers of the left hand in string players. Science 270(5234), 305–357 (1995)CrossRefGoogle Scholar
  49. 49.
    Hyde, K.L., et al.: Musical training shapes structural brain development. J. Neurosci. 29(10), 3019–3025 (2009)CrossRefGoogle Scholar
  50. 50.
    Hyde, K.L., et al.: The effects of musical training on structural brain development: a longitudinal study. Ann. N Y Acad. Sci. 1169, 182–186 (2009)CrossRefGoogle Scholar
  51. 51.
    Kraus, N., Chandrasekaran, B.: Music training for the development of auditory skills. Nat. Rev. Neurosci. 11(8), 599–605 (2010)CrossRefGoogle Scholar
  52. 52.
    Munte, T.F., Altenmuller, E., Jancke, L.: The musician’s brain as a model of neuroplasticity. Nat. Rev. Neurosci. 3(6), 473–478 (2002)Google Scholar
  53. 53.
    Bangert, M., Altenmuller, E.O.: Mapping perception to action in piano practice: a longitudinal DC-EEG study. BMC Neurosci. 4, 26 (2003)CrossRefGoogle Scholar
  54. 54.
    Magne, C., Schon, D., Besson, M.: Musician children detect pitch violations in both music and language better than nonmusician children: behavioral and electrophysiological approaches. J. Cogn. Neurosci. 18(2), 199–211 (2006)CrossRefGoogle Scholar
  55. 55.
    Milovanov, R., Tervaniemi, M.: The Interplay between Musical and Linguistic Aptitudes: A Review. Front Psychol. 2, 321 (2011)CrossRefGoogle Scholar
  56. 56.
    Chan, A.S., Ho, Y.C., Cheung, M.C.: Music training improves verbal memory. Nature 396(6707), 128 (1998)CrossRefGoogle Scholar
  57. 57.
    Aleman, A., et al.: Music training and mental imagery ability. Neuropsychologia 38(12), 1664–1668 (2000)CrossRefGoogle Scholar
  58. 58.
    Brochard, R., Dufour, A., Despres, O.: Effect of musical expertise on visuospatial abilities: evidence from reaction times and mental imagery. Brain Cogn. 54(2), 103–109 (2004)CrossRefGoogle Scholar
  59. 59.
    Schmithorst, V.J., Holland, S.K.: The effect of musical training on the neural correlates of math processing: a functional magnetic resonance imaging study in humans. Neurosci. Lett. 354(3), 193–196 (2004)CrossRefGoogle Scholar
  60. 60.
    Besson, M., Chobert, J., Marie, C.: Transfer of Training between Music and Speech: Common Processing, Attention, and Memory. Front Psychol. 2, 94 (2011)CrossRefGoogle Scholar
  61. 61.
    Patel, A.D.: Why would Musical Training Benefit the Neural Encoding of Speech? The OPERA Hypothesis. Front Psychol. 2, 142 (2011)CrossRefGoogle Scholar
  62. 62.
    Wallace, M.T., Stein, B.E.: Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21(22), 8886–8894 (2001)Google Scholar
  63. 63.
    Wallace, M.T., Stein, B.E.: Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17(7), 2429–2444 (1997)Google Scholar
  64. 64.
    Wallace, M.T., Stein, B.E.: Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71(1), 429–4232 (1994)Google Scholar
  65. 65.
    Jiang, W., Jiang, H., Stein, B.E.: Neonatal cortical ablation disrupts multisensory development in superior colliculus. J. Neurophysiol. 95(3), 1380–1396 (2006)CrossRefGoogle Scholar
  66. 66.
    Haslinger, B., et al.: Transmodal sensorimotor networks during action observation in professional pianists. J. Cogn. Neurosci. 17(2), 282–293 (2005)CrossRefGoogle Scholar
  67. 67.
    Hodges, D.A., Hairston, W.D., Burdette, J.H.: Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann. N Y Acad. Sci. 1060, 175–185 (2005)CrossRefGoogle Scholar
  68. 68.
    Petrini, K., et al.: Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Exp. Brain Res. 198(2-3), 339–352 (2009)MathSciNetCrossRefGoogle Scholar
  69. 69.
    Petrini, K., Russell, M., Pollick, F.: When knowing can replace seeing in audiovisual integration of actions. Cognition 110(3), 432–439 (2009)CrossRefGoogle Scholar
  70. 70.
    Petrini, K., Holt, S.P., Pollick, F.: Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. J. Vis. 10(5), 2 (2010)CrossRefGoogle Scholar
  71. 71.
    Petrini, K., et al.: Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56, 1480–1492 (2011)CrossRefGoogle Scholar
  72. 72.
    Musacchia, G., et al.: Musicians have enhanced subcortical auditory and audiovisual processing of speech and music. Proc. Natl. Acad. Sci. U S A 104(40), 15894–15898 (2007)CrossRefGoogle Scholar
  73. 73.
    Schulz, M., Ross, B., Pantev, C.: Evidence for training-induced crossmodal reorganization of cortical functions in trumpet players. Neuroreport 14(1), 157–161 (2003)CrossRefGoogle Scholar
  74. 74.
    Lappe, C., et al.: Cortical plasticity induced by short-term unimodal and multimodal musical training. J. Neurosci. 28(39), 9632–9639 (2008)CrossRefGoogle Scholar
  75. 75.
    Schlaug, G., et al.: Effects of music training on the child’s brain and cognitive development. Ann. N Y Acad. Sci. 1060, 219–230 (2005)CrossRefGoogle Scholar
  76. 76.
    Spence, C., Squire, S.: Multisensory integration: maintaining the perception of synchrony. Curr. Biol. 13(13), R519–R521 (2003)CrossRefGoogle Scholar
  77. 77.
    Arrighi, R., Alais, D., Burr, D.: Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. J. Vis. 6(3), 260–268 (2006)CrossRefGoogle Scholar
  78. 78.
    van Wassenhove, V., Grant, K.W., Poeppel, D.: Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3), 598–607 (2007)CrossRefGoogle Scholar
  79. 79.
    Vatakis, A., Spence, C.: Audiovisual synchrony perception for music, speech, and object actions. Brain Res. 1111(1), 134–142 (2006)CrossRefGoogle Scholar
  80. 80.
    Vatakis, A., Spence, C.: Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neurosci. Lett. 393(1), 40–44 (2006)CrossRefGoogle Scholar
  81. 81.
    Dixon, N.F., Spitz, L.: The detection of auditory visual desynchrony. Perception 9(6), 719–721 (1980)CrossRefGoogle Scholar
  82. 82.
    Lee, H., Noppeney, U.: Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc. Natl. Acad. Sci. U S A 108(51), E1441–E1450 (2011)CrossRefGoogle Scholar
  83. 83.
    Petrini, K., et al.: The music of your emotions: neural substrates involved in detection of emotional correspondence between auditory and visual music actions. PLoS One 6(4), e19165 (2011)Google Scholar
  84. 84.
    Petrini, K., McAleer, P., Pollick, F.: Audiovisual integration of emotional signals from music improvisation does not depend on temporal correspondence. Brain Res. 1323, 139–148 (2010)CrossRefGoogle Scholar
  85. 85.
    Hein, G., et al.: Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J. Neurosci. 27(30), 7881–7887 (2007)CrossRefGoogle Scholar
  86. 86.
    Kim, R.S., Seitz, A.R., Shams, L.: Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One 3(1), e1532 (2008)Google Scholar
  87. 87.
    Munhall, K.G., et al.: Temporal constraints on the McGurk effect. Percept. Psychophys 58(3), 351–362 (1996)CrossRefGoogle Scholar
  88. 88.
    Vatakis, A., et al.: Temporal recalibration during asynchronous audiovisual speech perception. Exp. Brain Res. 181(1), 173–181 (2007)CrossRefGoogle Scholar
  89. 89.
    Fain, G.L.: Sensory transduction, 340 p. Sinauer Associates, Sunderland (2003)Google Scholar
  90. 90.
    King, A.J.: Multisensory integration: strategies for synchronization. Curr. Biol. 15(9), R339–R3941 (2005)CrossRefGoogle Scholar
  91. 91.
    King, A.J., Palmer, A.R.: Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60(3), 492–500 (1985)CrossRefGoogle Scholar
  92. 92.
    Waadeland, C.H.: Strategies in empirical studies of swing groove. Musicologia Norvegica 32, 169–191 (2006)Google Scholar
  93. 93.
    Jansson, G., Johansson, G.: Visual perception of bending motion. Perception 2(3), 321–326 (1973)CrossRefGoogle Scholar
  94. 94.
    McGurk, H.M., Macdonald, J.: Hearing lips and seeing voices. Nature 264((5588), 746–748 (1976)CrossRefGoogle Scholar
  95. 95.
    Vatakis, A., Spence, C.: Crossmodal binding: evaluating the "unity assumption" using audiovisual speech stimuli. Percept. Psychophys 69(5), 744–756 (2007)CrossRefGoogle Scholar
  96. 96.
    Drake, C., Jones, M.R., Baruch, C.: The development of rhythmic attending in auditory sequences: attunement, referent period, focal attending. Cognition 77(3), 251–288 (2000)CrossRefGoogle Scholar
  97. 97.
    Saygin, A.P., Driver, J., de Sa, V.R.: In the footsteps of biological motion and multisensory perception: judgments of audiovisual temporal relations are enhanced for upright walkers. Psychol. Sci. 19(5), 469–475 (2008)CrossRefGoogle Scholar
  98. 98.
    Aschersleben, G., Prinz, W.: Synchronizing actions with events: the role of sensory information. Percept. Psychophys 57(3), 305–317 (1995)CrossRefGoogle Scholar
  99. 99.
    Miyake, Y., Onishi, Y., Poppel, E.: Two types of anticipation in synchronization tapping. Acta Neurobiol Exp. (Wars), 64(3), 415–426 (2004)Google Scholar
  100. 100.
    Navarra, J., et al.: Perception of audiovisual speech synchrony for native and non-native language. Brain Res. 1323, 84–93 (2010)CrossRefGoogle Scholar
  101. 101.
    Petrini, K., et al.: Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Experimental Brain Research 198(2-3), 339–352 (2009)MathSciNetCrossRefGoogle Scholar
  102. 102.
    Vatakis, A., Spence, C.: Evaluating the influence of the ’unity assumption’ on the temporal perception of realistic audiovisual stimuli. Acta Psychol. (Amst) 127(1), 12–23 (2008)CrossRefGoogle Scholar
  103. 103.
    Vatakis, A., Ghazanfar, A.A., Spence, C.: Facilitation of multisensory integration by the "unity effect" reveals that speech is special. J. Vis. 8(9),14. 1–14.11 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Scott A. Love
    • 1
  • Frank E. Pollick
    • 2
  • Karin Petrini
    • 3
  1. 1.Department of Psychological and Brain SciencesIndiana UniversityBloomingtonUSA
  2. 2.School of PsychologyUniversity of GlasgowGlasgowUK
  3. 3.Institute of OphthalmologyUniversity College LondonUK

Personalised recommendations