Skip to main content

Long-term music training modulates the recalibration of audiovisual simultaneity

Experimental Brain Research volume 236, pages 1869–1880 (2018)Cite this article

Abstract

To overcome differences in physical transmission time and neural processing, the brain adaptively recalibrates the point of simultaneity between auditory and visual signals by adapting to audiovisual asynchronies. Here, we examine whether the prolonged recalibration process of passively sensed visual and auditory signals is affected by naturally occurring multisensory training known to enhance audiovisual perceptual accuracy. Hence, we asked a group of drummers, of non-drummer musicians and of non-musicians to judge the audiovisual simultaneity of musical and non-musical audiovisual events, before and after adaptation with two fixed audiovisual asynchronies. We found that the recalibration for the musicians and drummers was in the opposite direction (sound leading vision) to that of non-musicians (vision leading sound), and change together with both increased music training and increased perceptual accuracy (i.e. ability to detect asynchrony). Our findings demonstrate that long-term musical training reshapes the way humans adaptively recalibrate simultaneity between auditory and visual signals.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  • Alais D, Carlile S (2005) Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proc Natl Acad Sci USA 102(6):2244–2247

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Arnold DH, Johnston A, Nishida S (2005) Timing sight and sound. Vision Res 45(10):1275–1284

    Article  PubMed  Google Scholar 

  • Aschersleben G, Prinz W (1995) Synchronizing actions with events: the role of sensory information. Percept Psychophys 57(3):305–317

    Article  PubMed  CAS  Google Scholar 

  • Bigand E, Poulin-Charronnat B (2006) Are we “experienced listeners”? A review of the musical capacities that do not depend on formal musical training. Cognition 100(1):100–130

    Article  PubMed  CAS  Google Scholar 

  • Bishop L, Goebl W (2014) Context-specific effects of musical expertise on audiovisual integration. Frontiers in Psychology 5:1123

    Article  PubMed  PubMed Central  Google Scholar 

  • Botella L (2008) Timekeeping is everything 1: rhythm and the construction of meaning. J Constructivist Psychol 21(4):309–320

    Article  Google Scholar 

  • Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436

    Article  PubMed  CAS  Google Scholar 

  • Bruns P, Röder B (2015) Sensory recalibration integrates information from the immediate and the cumulative past. Sci Rep 5:12739

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Calvo-Merino B, Glaser DE, Grezes J, Passingham RE, Haggard P (2005) Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex 15(8):1243–1249

    Article  PubMed  CAS  Google Scholar 

  • Calvo-Merino B, Grèzes J, Glaser DE, Passingham RE, Haggard P (2006) Seeing or doing? Influence of visual and motor familiarity in action observation. Curr Biol 16(19):1905–1910

    Article  PubMed  CAS  Google Scholar 

  • De Niear MA, Noel JP, Wallace MT (2017) The impact of feedback on the different time courses of multisensory temporal recalibration. Neural Plasticity

  • Desantis A, Haggard P (2016) Action-outcome learning and prediction shape the window of simultaneity of audiovisual outcomes. Cognition 153:33–42

    Article  PubMed  Google Scholar 

  • Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vision 9(12):7–7

    Article  Google Scholar 

  • Faul F, Erdfelder E, Lang AG, Buchner A (2007) G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 39(2):175–191

    Article  PubMed  Google Scholar 

  • Flatischler R (1992) The forgotten power of rhythm. Life Rhythm

  • Fontana F, Avanzini F, Rocchesso D (2004) Computation of nonlinear filter networks containing delay-free paths. In: Proceedings of the 7th International Conference on Digital Audio Effects (DAFX-04), Naples, Italy, pp 113–118

  • Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389

    Article  PubMed  PubMed Central  Google Scholar 

  • Foxe JJ, Molholm S, Bene D, Frey VA, Russo HP, Blanco D, Ross LA (2013) Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cerebral Cortex, bht213

  • Fujisaki W, Shimojo S, Kashino M, Nishida SY (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778

    Article  PubMed  CAS  Google Scholar 

  • García-Pérez MA, Alcalá-Quintana R (2012) On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model. Psychon Bull Rev 19(5):820–846

    Article  PubMed  Google Scholar 

  • Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524

    Article  PubMed  Google Scholar 

  • Hodges DA, Hairston WD, Burdette JH (2005) Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann N Y Acad Sci 1060(1):175–185

    Article  PubMed  Google Scholar 

  • Keetels M, Vroomen J (2007) No effect of auditory–visual spatial disparity on temporal recalibration. Exp Brain Res 182(4):559–565

    Article  PubMed  PubMed Central  Google Scholar 

  • King AJ (2005) Multisensory integration: strategies for synchronization. Curr Biol 15(9):R339–R341

    Article  PubMed  CAS  Google Scholar 

  • Lee H, Noppeney U (2011) Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc Natl Acad Sci 108(51):E1441–E1450

    Article  Google Scholar 

  • Love SA, Petrini K, Cheng A, Pollick FE (2013) A psychophysical investigation of differences between synchrony and temporal order judgments. PloS One, 8(1), e54798

  • Navarra J, García-Morera J, Spence C (2012) Temporal adaptation to audiovisual asynchrony generalizes across different sound frequencies. Front Psychol 3:152

    Article  PubMed  PubMed Central  Google Scholar 

  • Nichols J (2012) Music education in homeschooling: Jamie’s story. Narrative soundings: an anthology of narrative inquiry in music education. Springer Netherlands, pp 115–125

  • Noel JP, Łukowska M, Wallace M, Serino A (2016a) Multisensory simultaneity judgment and proximity to the body. J Vision 16(3):21–21

    Article  Google Scholar 

  • Noel JP, De Niear M, Van der Burg E, Wallace MT (2016b) Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PloS One, 11(8):e0161698

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Noel JP, De Niear MA, Stevenson R, Alais D, Wallace MT (2017) Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Res 10(1):121–129

    Article  PubMed  Google Scholar 

  • Oberman LM, Ramachandran VS (2008) Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: the mirror neuron hypothesis. Soc Neurosci 3(3–4):348–355

    Article  PubMed  Google Scholar 

  • Pelli DG (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis 10(4):437–442

    Article  PubMed  CAS  Google Scholar 

  • Petrini K, Dahl S, Rocchesso D, Waadeland CH, Avanzini F, Puce A, Pollick FE (2009a) Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Exp Brain Res 198(2–3):339–352

    Article  PubMed  Google Scholar 

  • Petrini K, Russell M, Pollick F (2009b) When knowing can replace seeing in audiovisual integration of actions. Cognition 110(3):432–439

    Article  PubMed  Google Scholar 

  • Petrini K, Holt SP, Pollick F (2010) Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. J Vision 10(5):2–2

    Article  Google Scholar 

  • Petrini K, Pollick FE, Dahl S, McAleer P, McKay L, Rocchesso D, … Puce A (2011) Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56(3):1480–1492

    Article  PubMed  Google Scholar 

  • Proverbio AM, Attardo L, Cozzi M, Zani A (2015) The effect of musical practice on gesture/sound pairing. Front Psychol 6

  • Repp BH, Su YH (2013) Sensorimotor synchronization: a review of recent research (2006–2012) Psychon Bull Rev 20(3):403–452

    Article  PubMed  Google Scholar 

  • Roach NW, Heron J, Whitaker D, McGraw PV (2010) Asynchrony adaptation reveals neural population code for audio-visual timing. Proc R Soc Lond B: Biol Sci rspb20101737

  • Rohde M, Ernst MO (2013) To lead and to lag–forward and backward recalibration of perceived visuo-motor simultaneity. Front Psychol 3:599

    Article  PubMed  PubMed Central  Google Scholar 

  • Rohde M, Ernst MO (2016) Time, agency, and sensory feedback delays during action. Curr Opin Behav Sci 8:193–199

    Article  Google Scholar 

  • Rohde M, Scheller M, Ernst MO (2014a) Effects can precede their cause in the sense of agency. Neuropsychologia 65:191–196

    Article  PubMed  Google Scholar 

  • Rohde M, van Dam LC, Ernst MO (2014b) Predictability is necessary for closed-loop visual feedback delay adaptation. J Vision 14(3):4–4

    Article  Google Scholar 

  • Schroeder CE, Foxe JJ (2004) 18 multisensory convergence in early cortical processing. The handbook of multisensory processes, p 295

  • Shams L, Kamitani Y, Shimojo S (2000) Illusions: what you see is what you hear. Nature 408(6814):788

    Article  PubMed  CAS  Google Scholar 

  • Simon DM, Noel JP, Wallace MT (2017) Event related potentials index rapid recalibration to audiovisual temporal asynchrony. Front Integr Neurosci, 11

  • Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519-R521

    Article  CAS  Google Scholar 

  • Stein BE, Meredith MA, Wallace MT (1993) The visually responsive neuron and beyond: multisensory integration in cat and monkey. Progress in Brain Research, vol. 95. Elsevier, pp 79–90

  • Stevenson RA, Segers M, Ferber S, Barense MD, Wallace MT (2015) The impact of multisensory integration deficits on speech perception in children with autism spectrum disorders. Multisensory Sensorimotor Interactions In Speech Perception, 249 Studia Musicologica Norvegica, 32:169–191

  • Turi M, Karaminis T, Pellicano E, Burr D (2016) No rapid audiovisual recalibration in adults on the autism spectrum. Sci Rep 6:21756

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Van der Burg E, Goodbourn PT (2015) Rapid, generalized adaptation to asynchronous audiovisual speech. Proc R Soc Lond B: Biol Sci 282(1804):20143083

    Article  Google Scholar 

  • Van der Burg E, Alais D, Cass J (2013) Rapid recalibration to audiovisual asynchrony. J Neurosci 33(37):14633–14637

    Article  PubMed  CAS  Google Scholar 

  • Van der Burg E, Alais D, Cass J (2015a) Audiovisual temporal recalibration occurs independently at two different time scales. Sci Rep, 5

  • Van der Burg E, Orchard-Mills E, Alais D (2015b) Rapid temporal recalibration is unique to audiovisual stimuli. Exp Brain Res 233(1):53–59

    Article  PubMed  Google Scholar 

  • Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142

    Article  PubMed  CAS  Google Scholar 

  • Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007) Temporal recalibration during asynchronous audiovisual speech perception. Exp Brain Res 181(1):173–181

    Article  PubMed  Google Scholar 

  • Vines BW, Krumhansl CL, Wanderley MM, Levitin DJ (2006) Cross-modal interactions in the perception of musical performance. Cognition 101(1):80–113

    Article  PubMed  Google Scholar 

  • Vroomen J, Keetels M, De Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22(1):32–35

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank Dr Chris Bevan and Eliot Farmer for engaging and useful discussion on what it means to be a drummer.

Author information

Authors and Affiliations

  1. Department of Psychology, University of Bath, Bath, UK

    Crescent Jicol, Michael J. Proulx & Karin Petrini

  2. School of Psychology, University of Glasgow, Glasgow, UK

    Frank E. Pollick

  3. Department of Computer Science, University of Bath, Claverton Down, Bath, BA2 7AY, UK

    Crescent Jicol

Authors
  1. Crescent Jicol
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael J. Proulx
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frank E. Pollick
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Karin Petrini
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

KP designed the study, KP and FEP piloted the study, CJ conducted the experiment, KP and MJP supervised CJ during the experiment conduction, KP and CJ analyzed the data, KP, CJ, FEP and MJP wrote the manuscript.

Corresponding author

Correspondence to Crescent Jicol.

Ethics declarations

Conflict of interest

The authors declare no competing financial interests.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jicol, C., Proulx, M.J., Pollick, F.E. et al. Long-term music training modulates the recalibration of audiovisual simultaneity. Exp Brain Res 236, 1869–1880 (2018). https://doi.org/10.1007/s00221-018-5269-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-018-5269-4

Keywords

  • Recalibration
  • Multisensory perception
  • Simultaneity judgement