Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Long-term music training modulates the recalibration of audiovisual simultaneity

Abstract

To overcome differences in physical transmission time and neural processing, the brain adaptively recalibrates the point of simultaneity between auditory and visual signals by adapting to audiovisual asynchronies. Here, we examine whether the prolonged recalibration process of passively sensed visual and auditory signals is affected by naturally occurring multisensory training known to enhance audiovisual perceptual accuracy. Hence, we asked a group of drummers, of non-drummer musicians and of non-musicians to judge the audiovisual simultaneity of musical and non-musical audiovisual events, before and after adaptation with two fixed audiovisual asynchronies. We found that the recalibration for the musicians and drummers was in the opposite direction (sound leading vision) to that of non-musicians (vision leading sound), and change together with both increased music training and increased perceptual accuracy (i.e. ability to detect asynchrony). Our findings demonstrate that long-term musical training reshapes the way humans adaptively recalibrate simultaneity between auditory and visual signals.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Alais D, Carlile S (2005) Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proc Natl Acad Sci USA 102(6):2244–2247

  2. Arnold DH, Johnston A, Nishida S (2005) Timing sight and sound. Vision Res 45(10):1275–1284

  3. Aschersleben G, Prinz W (1995) Synchronizing actions with events: the role of sensory information. Percept Psychophys 57(3):305–317

  4. Bigand E, Poulin-Charronnat B (2006) Are we “experienced listeners”? A review of the musical capacities that do not depend on formal musical training. Cognition 100(1):100–130

  5. Bishop L, Goebl W (2014) Context-specific effects of musical expertise on audiovisual integration. Frontiers in Psychology 5:1123

  6. Botella L (2008) Timekeeping is everything 1: rhythm and the construction of meaning. J Constructivist Psychol 21(4):309–320

  7. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436

  8. Bruns P, Röder B (2015) Sensory recalibration integrates information from the immediate and the cumulative past. Sci Rep 5:12739

  9. Calvo-Merino B, Glaser DE, Grezes J, Passingham RE, Haggard P (2005) Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex 15(8):1243–1249

  10. Calvo-Merino B, Grèzes J, Glaser DE, Passingham RE, Haggard P (2006) Seeing or doing? Influence of visual and motor familiarity in action observation. Curr Biol 16(19):1905–1910

  11. De Niear MA, Noel JP, Wallace MT (2017) The impact of feedback on the different time courses of multisensory temporal recalibration. Neural Plasticity

  12. Desantis A, Haggard P (2016) Action-outcome learning and prediction shape the window of simultaneity of audiovisual outcomes. Cognition 153:33–42

  13. Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vision 9(12):7–7

  14. Faul F, Erdfelder E, Lang AG, Buchner A (2007) G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 39(2):175–191

  15. Flatischler R (1992) The forgotten power of rhythm. Life Rhythm

  16. Fontana F, Avanzini F, Rocchesso D (2004) Computation of nonlinear filter networks containing delay-free paths. In: Proceedings of the 7th International Conference on Digital Audio Effects (DAFX-04), Naples, Italy, pp 113–118

  17. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389

  18. Foxe JJ, Molholm S, Bene D, Frey VA, Russo HP, Blanco D, Ross LA (2013) Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cerebral Cortex, bht213

  19. Fujisaki W, Shimojo S, Kashino M, Nishida SY (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778

  20. García-Pérez MA, Alcalá-Quintana R (2012) On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model. Psychon Bull Rev 19(5):820–846

  21. Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524

  22. Hodges DA, Hairston WD, Burdette JH (2005) Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann N Y Acad Sci 1060(1):175–185

  23. Keetels M, Vroomen J (2007) No effect of auditory–visual spatial disparity on temporal recalibration. Exp Brain Res 182(4):559–565

  24. King AJ (2005) Multisensory integration: strategies for synchronization. Curr Biol 15(9):R339–R341

  25. Lee H, Noppeney U (2011) Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc Natl Acad Sci 108(51):E1441–E1450

  26. Love SA, Petrini K, Cheng A, Pollick FE (2013) A psychophysical investigation of differences between synchrony and temporal order judgments. PloS One, 8(1), e54798

  27. Navarra J, García-Morera J, Spence C (2012) Temporal adaptation to audiovisual asynchrony generalizes across different sound frequencies. Front Psychol 3:152

  28. Nichols J (2012) Music education in homeschooling: Jamie’s story. Narrative soundings: an anthology of narrative inquiry in music education. Springer Netherlands, pp 115–125

  29. Noel JP, Łukowska M, Wallace M, Serino A (2016a) Multisensory simultaneity judgment and proximity to the body. J Vision 16(3):21–21

  30. Noel JP, De Niear M, Van der Burg E, Wallace MT (2016b) Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PloS One, 11(8):e0161698

  31. Noel JP, De Niear MA, Stevenson R, Alais D, Wallace MT (2017) Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Res 10(1):121–129

  32. Oberman LM, Ramachandran VS (2008) Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: the mirror neuron hypothesis. Soc Neurosci 3(3–4):348–355

  33. Pelli DG (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis 10(4):437–442

  34. Petrini K, Dahl S, Rocchesso D, Waadeland CH, Avanzini F, Puce A, Pollick FE (2009a) Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Exp Brain Res 198(2–3):339–352

  35. Petrini K, Russell M, Pollick F (2009b) When knowing can replace seeing in audiovisual integration of actions. Cognition 110(3):432–439

  36. Petrini K, Holt SP, Pollick F (2010) Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. J Vision 10(5):2–2

  37. Petrini K, Pollick FE, Dahl S, McAleer P, McKay L, Rocchesso D, … Puce A (2011) Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56(3):1480–1492

  38. Proverbio AM, Attardo L, Cozzi M, Zani A (2015) The effect of musical practice on gesture/sound pairing. Front Psychol 6

  39. Repp BH, Su YH (2013) Sensorimotor synchronization: a review of recent research (2006–2012) Psychon Bull Rev 20(3):403–452

  40. Roach NW, Heron J, Whitaker D, McGraw PV (2010) Asynchrony adaptation reveals neural population code for audio-visual timing. Proc R Soc Lond B: Biol Sci rspb20101737

  41. Rohde M, Ernst MO (2013) To lead and to lag–forward and backward recalibration of perceived visuo-motor simultaneity. Front Psychol 3:599

  42. Rohde M, Ernst MO (2016) Time, agency, and sensory feedback delays during action. Curr Opin Behav Sci 8:193–199

  43. Rohde M, Scheller M, Ernst MO (2014a) Effects can precede their cause in the sense of agency. Neuropsychologia 65:191–196

  44. Rohde M, van Dam LC, Ernst MO (2014b) Predictability is necessary for closed-loop visual feedback delay adaptation. J Vision 14(3):4–4

  45. Schroeder CE, Foxe JJ (2004) 18 multisensory convergence in early cortical processing. The handbook of multisensory processes, p 295

  46. Shams L, Kamitani Y, Shimojo S (2000) Illusions: what you see is what you hear. Nature 408(6814):788

  47. Simon DM, Noel JP, Wallace MT (2017) Event related potentials index rapid recalibration to audiovisual temporal asynchrony. Front Integr Neurosci, 11

  48. Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519-R521

  49. Stein BE, Meredith MA, Wallace MT (1993) The visually responsive neuron and beyond: multisensory integration in cat and monkey. Progress in Brain Research, vol. 95. Elsevier, pp 79–90

  50. Stevenson RA, Segers M, Ferber S, Barense MD, Wallace MT (2015) The impact of multisensory integration deficits on speech perception in children with autism spectrum disorders. Multisensory Sensorimotor Interactions In Speech Perception, 249 Studia Musicologica Norvegica, 32:169–191

  51. Turi M, Karaminis T, Pellicano E, Burr D (2016) No rapid audiovisual recalibration in adults on the autism spectrum. Sci Rep 6:21756

  52. Van der Burg E, Goodbourn PT (2015) Rapid, generalized adaptation to asynchronous audiovisual speech. Proc R Soc Lond B: Biol Sci 282(1804):20143083

  53. Van der Burg E, Alais D, Cass J (2013) Rapid recalibration to audiovisual asynchrony. J Neurosci 33(37):14633–14637

  54. Van der Burg E, Alais D, Cass J (2015a) Audiovisual temporal recalibration occurs independently at two different time scales. Sci Rep, 5

  55. Van der Burg E, Orchard-Mills E, Alais D (2015b) Rapid temporal recalibration is unique to audiovisual stimuli. Exp Brain Res 233(1):53–59

  56. Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142

  57. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007) Temporal recalibration during asynchronous audiovisual speech perception. Exp Brain Res 181(1):173–181

  58. Vines BW, Krumhansl CL, Wanderley MM, Levitin DJ (2006) Cross-modal interactions in the perception of musical performance. Cognition 101(1):80–113

  59. Vroomen J, Keetels M, De Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22(1):32–35

Download references

Acknowledgements

We would like to thank Dr Chris Bevan and Eliot Farmer for engaging and useful discussion on what it means to be a drummer.

Author information

KP designed the study, KP and FEP piloted the study, CJ conducted the experiment, KP and MJP supervised CJ during the experiment conduction, KP and CJ analyzed the data, KP, CJ, FEP and MJP wrote the manuscript.

Correspondence to Crescent Jicol.

Ethics declarations

Conflict of interest

The authors declare no competing financial interests.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jicol, C., Proulx, M.J., Pollick, F.E. et al. Long-term music training modulates the recalibration of audiovisual simultaneity. Exp Brain Res 236, 1869–1880 (2018). https://doi.org/10.1007/s00221-018-5269-4

Download citation

Keywords

  • Recalibration
  • Multisensory perception
  • Simultaneity judgement