Advertisement

Experimental Brain Research

, Volume 236, Issue 7, pp 1869–1880 | Cite as

Long-term music training modulates the recalibration of audiovisual simultaneity

  • Crescent Jicol
  • Michael J. Proulx
  • Frank E. Pollick
  • Karin Petrini
Research Article

Abstract

To overcome differences in physical transmission time and neural processing, the brain adaptively recalibrates the point of simultaneity between auditory and visual signals by adapting to audiovisual asynchronies. Here, we examine whether the prolonged recalibration process of passively sensed visual and auditory signals is affected by naturally occurring multisensory training known to enhance audiovisual perceptual accuracy. Hence, we asked a group of drummers, of non-drummer musicians and of non-musicians to judge the audiovisual simultaneity of musical and non-musical audiovisual events, before and after adaptation with two fixed audiovisual asynchronies. We found that the recalibration for the musicians and drummers was in the opposite direction (sound leading vision) to that of non-musicians (vision leading sound), and change together with both increased music training and increased perceptual accuracy (i.e. ability to detect asynchrony). Our findings demonstrate that long-term musical training reshapes the way humans adaptively recalibrate simultaneity between auditory and visual signals.

Keywords

Recalibration Multisensory perception Simultaneity judgement 

Notes

Acknowledgements

We would like to thank Dr Chris Bevan and Eliot Farmer for engaging and useful discussion on what it means to be a drummer.

Author contributions

KP designed the study, KP and FEP piloted the study, CJ conducted the experiment, KP and MJP supervised CJ during the experiment conduction, KP and CJ analyzed the data, KP, CJ, FEP and MJP wrote the manuscript.

Compliance with ethical standards

Conflict of interest

The authors declare no competing financial interests.

Supplementary material

221_2018_5269_MOESM1_ESM.docx (351 kb)
Supplementary material 1 (DOCX 351 KB)
221_2018_5269_MOESM2_ESM.mov (1.4 mb)
Supplementary material 2 (MOV 1477 KB)
221_2018_5269_MOESM3_ESM.mov (1.4 mb)
Supplementary material 3 (MOV 1477 KB)
221_2018_5269_MOESM4_ESM.mov (1.4 mb)
Supplementary material 4 (MOV 1477 KB)
221_2018_5269_MOESM5_ESM.mov (178 kb)
Supplementary material 5 (MOV 177 KB)
221_2018_5269_MOESM6_ESM.mov (177 kb)
Supplementary material 6 (MOV 176 KB)
221_2018_5269_MOESM7_ESM.mov (177 kb)
Supplementary material 7 (MOV 177 KB)

References

  1. Alais D, Carlile S (2005) Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proc Natl Acad Sci USA 102(6):2244–2247CrossRefPubMedPubMedCentralGoogle Scholar
  2. Arnold DH, Johnston A, Nishida S (2005) Timing sight and sound. Vision Res 45(10):1275–1284CrossRefPubMedGoogle Scholar
  3. Aschersleben G, Prinz W (1995) Synchronizing actions with events: the role of sensory information. Percept Psychophys 57(3):305–317CrossRefPubMedGoogle Scholar
  4. Bigand E, Poulin-Charronnat B (2006) Are we “experienced listeners”? A review of the musical capacities that do not depend on formal musical training. Cognition 100(1):100–130CrossRefPubMedGoogle Scholar
  5. Bishop L, Goebl W (2014) Context-specific effects of musical expertise on audiovisual integration. Frontiers in Psychology 5:1123CrossRefPubMedPubMedCentralGoogle Scholar
  6. Botella L (2008) Timekeeping is everything 1: rhythm and the construction of meaning. J Constructivist Psychol 21(4):309–320CrossRefGoogle Scholar
  7. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436CrossRefPubMedGoogle Scholar
  8. Bruns P, Röder B (2015) Sensory recalibration integrates information from the immediate and the cumulative past. Sci Rep 5:12739CrossRefPubMedPubMedCentralGoogle Scholar
  9. Calvo-Merino B, Glaser DE, Grezes J, Passingham RE, Haggard P (2005) Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex 15(8):1243–1249CrossRefPubMedGoogle Scholar
  10. Calvo-Merino B, Grèzes J, Glaser DE, Passingham RE, Haggard P (2006) Seeing or doing? Influence of visual and motor familiarity in action observation. Curr Biol 16(19):1905–1910CrossRefPubMedGoogle Scholar
  11. De Niear MA, Noel JP, Wallace MT (2017) The impact of feedback on the different time courses of multisensory temporal recalibration. Neural PlasticityGoogle Scholar
  12. Desantis A, Haggard P (2016) Action-outcome learning and prediction shape the window of simultaneity of audiovisual outcomes. Cognition 153:33–42CrossRefPubMedGoogle Scholar
  13. Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vision 9(12):7–7CrossRefGoogle Scholar
  14. Faul F, Erdfelder E, Lang AG, Buchner A (2007) G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 39(2):175–191CrossRefPubMedGoogle Scholar
  15. Flatischler R (1992) The forgotten power of rhythm. Life RhythmGoogle Scholar
  16. Fontana F, Avanzini F, Rocchesso D (2004) Computation of nonlinear filter networks containing delay-free paths. In: Proceedings of the 7th International Conference on Digital Audio Effects (DAFX-04), Naples, Italy, pp 113–118Google Scholar
  17. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389CrossRefPubMedPubMedCentralGoogle Scholar
  18. Foxe JJ, Molholm S, Bene D, Frey VA, Russo HP, Blanco D, Ross LA (2013) Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cerebral Cortex, bht213Google Scholar
  19. Fujisaki W, Shimojo S, Kashino M, Nishida SY (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778CrossRefPubMedGoogle Scholar
  20. García-Pérez MA, Alcalá-Quintana R (2012) On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model. Psychon Bull Rev 19(5):820–846CrossRefPubMedGoogle Scholar
  21. Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524CrossRefPubMedGoogle Scholar
  22. Hodges DA, Hairston WD, Burdette JH (2005) Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann N Y Acad Sci 1060(1):175–185CrossRefPubMedGoogle Scholar
  23. Keetels M, Vroomen J (2007) No effect of auditory–visual spatial disparity on temporal recalibration. Exp Brain Res 182(4):559–565CrossRefPubMedPubMedCentralGoogle Scholar
  24. King AJ (2005) Multisensory integration: strategies for synchronization. Curr Biol 15(9):R339–R341CrossRefPubMedGoogle Scholar
  25. Lee H, Noppeney U (2011) Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc Natl Acad Sci 108(51):E1441–E1450CrossRefGoogle Scholar
  26. Love SA, Petrini K, Cheng A, Pollick FE (2013) A psychophysical investigation of differences between synchrony and temporal order judgments. PloS One, 8(1), e54798Google Scholar
  27. Navarra J, García-Morera J, Spence C (2012) Temporal adaptation to audiovisual asynchrony generalizes across different sound frequencies. Front Psychol 3:152CrossRefPubMedPubMedCentralGoogle Scholar
  28. Nichols J (2012) Music education in homeschooling: Jamie’s story. Narrative soundings: an anthology of narrative inquiry in music education. Springer Netherlands, pp 115–125Google Scholar
  29. Noel JP, Łukowska M, Wallace M, Serino A (2016a) Multisensory simultaneity judgment and proximity to the body. J Vision 16(3):21–21CrossRefGoogle Scholar
  30. Noel JP, De Niear M, Van der Burg E, Wallace MT (2016b) Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PloS One, 11(8):e0161698CrossRefPubMedPubMedCentralGoogle Scholar
  31. Noel JP, De Niear MA, Stevenson R, Alais D, Wallace MT (2017) Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Res 10(1):121–129CrossRefPubMedGoogle Scholar
  32. Oberman LM, Ramachandran VS (2008) Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: the mirror neuron hypothesis. Soc Neurosci 3(3–4):348–355CrossRefPubMedGoogle Scholar
  33. Pelli DG (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis 10(4):437–442CrossRefPubMedGoogle Scholar
  34. Petrini K, Dahl S, Rocchesso D, Waadeland CH, Avanzini F, Puce A, Pollick FE (2009a) Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Exp Brain Res 198(2–3):339–352CrossRefPubMedGoogle Scholar
  35. Petrini K, Russell M, Pollick F (2009b) When knowing can replace seeing in audiovisual integration of actions. Cognition 110(3):432–439CrossRefPubMedGoogle Scholar
  36. Petrini K, Holt SP, Pollick F (2010) Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. J Vision 10(5):2–2CrossRefGoogle Scholar
  37. Petrini K, Pollick FE, Dahl S, McAleer P, McKay L, Rocchesso D, … Puce A (2011) Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56(3):1480–1492CrossRefPubMedGoogle Scholar
  38. Proverbio AM, Attardo L, Cozzi M, Zani A (2015) The effect of musical practice on gesture/sound pairing. Front Psychol 6Google Scholar
  39. Repp BH, Su YH (2013) Sensorimotor synchronization: a review of recent research (2006–2012) Psychon Bull Rev 20(3):403–452CrossRefPubMedGoogle Scholar
  40. Roach NW, Heron J, Whitaker D, McGraw PV (2010) Asynchrony adaptation reveals neural population code for audio-visual timing. Proc R Soc Lond B: Biol Sci rspb20101737Google Scholar
  41. Rohde M, Ernst MO (2013) To lead and to lag–forward and backward recalibration of perceived visuo-motor simultaneity. Front Psychol 3:599CrossRefPubMedPubMedCentralGoogle Scholar
  42. Rohde M, Ernst MO (2016) Time, agency, and sensory feedback delays during action. Curr Opin Behav Sci 8:193–199CrossRefGoogle Scholar
  43. Rohde M, Scheller M, Ernst MO (2014a) Effects can precede their cause in the sense of agency. Neuropsychologia 65:191–196CrossRefPubMedGoogle Scholar
  44. Rohde M, van Dam LC, Ernst MO (2014b) Predictability is necessary for closed-loop visual feedback delay adaptation. J Vision 14(3):4–4CrossRefGoogle Scholar
  45. Schroeder CE, Foxe JJ (2004) 18 multisensory convergence in early cortical processing. The handbook of multisensory processes, p 295Google Scholar
  46. Shams L, Kamitani Y, Shimojo S (2000) Illusions: what you see is what you hear. Nature 408(6814):788CrossRefPubMedGoogle Scholar
  47. Simon DM, Noel JP, Wallace MT (2017) Event related potentials index rapid recalibration to audiovisual temporal asynchrony. Front Integr Neurosci, 11Google Scholar
  48. Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519-R521CrossRefGoogle Scholar
  49. Stein BE, Meredith MA, Wallace MT (1993) The visually responsive neuron and beyond: multisensory integration in cat and monkey. Progress in Brain Research, vol. 95. Elsevier, pp 79–90Google Scholar
  50. Stevenson RA, Segers M, Ferber S, Barense MD, Wallace MT (2015) The impact of multisensory integration deficits on speech perception in children with autism spectrum disorders. Multisensory Sensorimotor Interactions In Speech Perception, 249 Studia Musicologica Norvegica, 32:169–191Google Scholar
  51. Turi M, Karaminis T, Pellicano E, Burr D (2016) No rapid audiovisual recalibration in adults on the autism spectrum. Sci Rep 6:21756CrossRefPubMedPubMedCentralGoogle Scholar
  52. Van der Burg E, Goodbourn PT (2015) Rapid, generalized adaptation to asynchronous audiovisual speech. Proc R Soc Lond B: Biol Sci 282(1804):20143083CrossRefGoogle Scholar
  53. Van der Burg E, Alais D, Cass J (2013) Rapid recalibration to audiovisual asynchrony. J Neurosci 33(37):14633–14637CrossRefPubMedGoogle Scholar
  54. Van der Burg E, Alais D, Cass J (2015a) Audiovisual temporal recalibration occurs independently at two different time scales. Sci Rep, 5Google Scholar
  55. Van der Burg E, Orchard-Mills E, Alais D (2015b) Rapid temporal recalibration is unique to audiovisual stimuli. Exp Brain Res 233(1):53–59CrossRefPubMedGoogle Scholar
  56. Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142CrossRefPubMedGoogle Scholar
  57. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007) Temporal recalibration during asynchronous audiovisual speech perception. Exp Brain Res 181(1):173–181CrossRefPubMedGoogle Scholar
  58. Vines BW, Krumhansl CL, Wanderley MM, Levitin DJ (2006) Cross-modal interactions in the perception of musical performance. Cognition 101(1):80–113CrossRefPubMedGoogle Scholar
  59. Vroomen J, Keetels M, De Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22(1):32–35CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of BathBathUK
  2. 2.School of PsychologyUniversity of GlasgowGlasgowUK
  3. 3.Department of Computer ScienceUniversity of BathBathUK

Personalised recommendations