Multisensory integration is independent of perceived simultaneity

Abstract

The importance of multisensory integration for perception and action has long been recognised. Integrating information from individual senses increases the chance of survival by reducing the variability in the incoming signals, thus allowing us to respond more rapidly. Reaction times (RTs) are fastest when the components of the multisensory signals are simultaneous. This response facilitation is traditionally attributed to multisensory integration. However, it is unclear if facilitation of RTs occurs when stimuli are perceived as synchronous or are actually physically synchronous. Repeated exposure to audiovisual asynchrony can change the delay at which multisensory stimuli are perceived as simultaneous, thus changing the delay at which the stimuli are integrated—perceptually. Here we set out to determine how such changes in multisensory integration for perception affect our ability to respond to multisensory events. If stimuli perceived as simultaneous were reacted to most rapidly, it would suggest a common system for multisensory integration for perception and action. If not, it would suggest separate systems. We measured RTs to auditory, visual, and audiovisual stimuli following exposure to audiovisual asynchrony. Exposure affected the variability of the unisensory RT distributions; in particular, the slowest RTs were either speed up or slowed down (in the direction predicted from shifts in perceived simultaneity). Additionally, the multisensory facilitation of RTs (beyond statistical summation) only occurred when audiovisual onsets were physically synchronous, rather than when they appeared simultaneous. We conclude that the perception of synchrony is therefore independent of multisensory integration and suggest a division between multisensory processes that are fast (automatic and unaffected by temporal adaptation) and those that are slow (perceptually driven and adaptable).

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    The GEE is similar to the one described in “Data analysis” with a few minor differences. The within-subjects variables were SOA and “week” since in order to have enough data at a given SOA, multisensory RT data needed to be combined across sessions.

  2. 2.

    Using an uncorrected alpha-rate, there were several additional SOAs that produced RTs that were significantly slower than the race model demonstrating a trend towards response inhibition: SOA = −80 ms, mean response inhibition () = 2.6 ms ± (SE)1.7 ms, t 29 = 2.09, p = .045; SOA = −60 ms, = 3.9 ± 1.6 ms, t 29 = 2.52, p = .018; SOA = −40 ms, = 3.5 ± 1.6 ms, t 29 = 2.14, p = .041; SOA = 60 ms, = 4.0 ± 1.6 ms, t 29 = 2.59, p = .015; SOA = 100 ms, = 4.1 ± 1.9 ms, t 29 = 2.18, p = .038.

References

  1. Balota DA, Spieler DH (1999) Word frequency, repetition, and lexicality effects in word recognition tasks: beyond measures of central tendency. J Exp Psychol Gen 128(1):32

    CAS  Article  PubMed  Google Scholar 

  2. Berman AL (1961) Interaction of cortical responses to somatic and auditory stimuli in anterior ectosylvian gyrus of cat. J Neurophysiol 24(6):608–620

    CAS  PubMed  Google Scholar 

  3. Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009. doi:10.1162/0898929041502733

    Article  PubMed  Google Scholar 

  4. Colonius H, Diederich A (2006) The race model inequality: interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154. doi:10.1037/0033-295X.113.1.148

    Article  PubMed  Google Scholar 

  5. Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vis 9(12):7.1–716. doi:10.1167/9.12.7

    Google Scholar 

  6. Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404

    Article  PubMed  Google Scholar 

  7. Diederich A, Colonius H (2015) The time window of multisensory integration: relating reaction times and judgments of temporal order. Psychol Rev 122(2):232–241. doi:10.1037/a0038696

    Article  PubMed  Google Scholar 

  8. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    CAS  Article  PubMed  Google Scholar 

  9. Donders FC (1868) /1969). Die Schnelligkeit psychischer Processe [On the speed of mental processes]. Acta Psychol 30:412–431

    Article  Google Scholar 

  10. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433. doi:10.1038/415429a

    CAS  Article  PubMed  Google Scholar 

  11. Farmer ME, Klein RM (1995) The evidence for a temporal processing deficit linked to dyslexia: a review. Psychon Bull Rev 2(4):460–493. doi:10.3758/Bf03210983

    CAS  Article  PubMed  Google Scholar 

  12. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389. doi:10.1007/s00221-010-2240-4

    Article  PubMed  PubMed Central  Google Scholar 

  13. Foucher JR, Lacambre M, Pham BT, Giersch A, Elliott MA (2007) Low time resolution in schizophrenia Lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophr Res 97(1–3):118–127. doi:10.1016/j.schres.2007.08.013

    CAS  Article  PubMed  Google Scholar 

  14. Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778. doi:10.1038/nn1268

    CAS  Article  PubMed  Google Scholar 

  15. Gibbon J, Rutschmann R (1969) Temporal order judgement and reaction time. Science 165(3891):413–415

    CAS  Article  PubMed  Google Scholar 

  16. Goodale MA, Milner AD (1992) Separate visual pathways for perception and action. Trends Neurosci 15(1):20–25

    CAS  Article  PubMed  Google Scholar 

  17. Hanley JA, Negassa A, Forrester JE (2003) Statistical analysis of correlated data using generalized estimating equations: an orientation. Am J Epidemiol 157(4):364–375

    Article  PubMed  Google Scholar 

  18. Hanson JV, Heron J, Whitaker D (2008) Recalibration of perceived time across sensory modalities. Exp Brain Res 185(2):347–352. doi:10.1007/s00221-008-1282-3

    Article  PubMed  Google Scholar 

  19. Harrar V, Harris LR (2005) Simultaneity constancy: detecting events with touch and vision. Exp Brain Res 166(3–4):465–473. doi:10.1007/s00221-005-2386-7

    Article  PubMed  Google Scholar 

  20. Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524. doi:10.1007/s00221-007-1253-0

    Article  PubMed  Google Scholar 

  21. Harrar V, Tammam J, Perez-Bellido A, Pitt A, Stein J, Spence C (2014) Multisensory integration and attention in developmental dyslexia. Curr Biol 24(5):531–535. doi:10.1016/j.cub.2014.01.029

    CAS  Article  PubMed  Google Scholar 

  22. Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293

    CAS  Article  PubMed  Google Scholar 

  23. Jaekl P, Soto-Faraco S, Harris LR (2012) Perceived size change induced by audiovisual temporal delays. Exp Brain Res 216(3):457–462. doi:10.1007/s00221-011-2948-9

    Article  PubMed  Google Scholar 

  24. Jaekl P, Seidlitz J, Harris LR, Tadin D (2015) Audiovisual delay as a novel cue to visual distance. PLoS ONE 10(10):e0141125. doi:10.1371/journal.pone.0141125

    Article  PubMed  PubMed Central  Google Scholar 

  25. Jaskowski P (1992) Temporal-order judgment and reaction time for short and long stimuli. Psychol Res 54(3):141–145

    CAS  Article  PubMed  Google Scholar 

  26. Jiang H, Lepore F, Ptito M, Guillemot J-P (1994) Sensory interactions in the anterior ectosylvian cortex of cats. Exp Brain Res 101(3):385–396

    CAS  Article  PubMed  Google Scholar 

  27. Keetels M, Vroomen J (2012) Exposure to delayed visual feedback of the hand changes motor-sensory synchrony perception. Exp Brain Res 219(4):431–440. doi:10.1007/s00221-012-3081-0

    Article  PubMed  PubMed Central  Google Scholar 

  28. Kopinska A, Harris LR (2004) Simultaneity constancy. Perception 33(9):1049–1060

    Article  PubMed  Google Scholar 

  29. Lacouture Y, Cousineau D (2008) How to use MATLAB to fit the ex-Gaussian and other probability functions to a distribution of response times. Tutor Quant Methods Psychol 4(1):35–45

    Article  Google Scholar 

  30. Leone LM, McCourt ME (2012) The question of simultaneity in multisensory integration. Human Vis Electron Imaging Xvii. doi:10.1117/12.912183

    Google Scholar 

  31. Leone LM, McCourt ME (2013) The roles of physical and physiological simultaneity in audiovisual multisensory facilitation. i-Perception 4(4):213–228. doi:10.1068/i0532

    Article  PubMed  PubMed Central  Google Scholar 

  32. Leone LM, McCourt ME (2015) Dissociation of perception and action in audiovisual multisensory integration. Eur J Neurosci 42(11):2915–2922

    Article  PubMed  PubMed Central  Google Scholar 

  33. Leth-Steensen C, Elbaz ZK, Douglas VI (2000) Mean response times, variability, and skew in the responding of ADHD children: a response time distributional approach. Acta Psychol (Amst) 104(2):167–190

    CAS  Article  Google Scholar 

  34. Luce RD (1986) Response times. Oxford University Press, New York

    Google Scholar 

  35. Maier JX, Di Luca M, Noppeney U (2011) Audiovisual asynchrony detection in human speech. J Exp Psychol Hum Percept Perform 37(1):245

    Article  PubMed  Google Scholar 

  36. Matzke D, Wagenmakers EJ (2009) Psychological interpretation of the ex-Gaussian and shifted Wald parameters: a diffusion model analysis. Psychon Bull Rev 16(5):798–817. doi:10.3758/PBR.16.5.798

    Article  PubMed  Google Scholar 

  37. Megevand P, Molholm S, Nayak A, Foxe JJ (2013) Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS ONE 8(8):e71608. doi:10.1371/journal.pone.0071608

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  38. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temp Factors. J Neurosci 7(10):3215–3229

    CAS  Google Scholar 

  39. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14(2):247–279

    CAS  Article  PubMed  Google Scholar 

  40. Miller J (1986) Timecourse of coactivation in bimodal divided attention. Percept Psychophys 40(5):331–343

    CAS  Article  PubMed  Google Scholar 

  41. Miller RL, Pluta SR, Stein BE, Rowland BA (2015) Relative unisensory strength and timing predict their multisensory product. J Neurosci 35(13):5213–5220. doi:10.1523/JNEUROSCI.4771-14.2015

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  42. Miyazaki M, Yamamoto S, Uchida S, Kitazawa S (2006) Bayesian calibration of simultaneity in tactile temporal order judgment. Nat Neurosci 9(7):875–877. doi:10.1038/nn1712

    CAS  Article  PubMed  Google Scholar 

  43. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14(1):115–128

    Article  PubMed  Google Scholar 

  44. Navarra J, Hartcher-O’Brien J, Piazza E, Spence C (2009) Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proc Natl Acad Sci U S A 106(23):9169–9173. doi:10.1073/pnas.0810486106

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  45. Otto TU, Mamassian P (2012) Noise and correlations in parallel perceptual decision making. Curr Biol 22(15):1391–1396. doi:10.1016/j.cub.2012.05.031

    CAS  Article  PubMed  Google Scholar 

  46. Otto TU, Dassy B, Mamassian P (2013) Principles of multisensory behavior. J Neurosci 33(17):7463–7474. doi:10.1523/JNEUROSCI.4678-12.2013

    CAS  Article  PubMed  Google Scholar 

  47. Pöppel E, Schill K, von Steinbuchel N (1990) Sensory integration within temporally neutral systems states: a hypothesis. Naturwissenschaften 77(2):89–91

    Article  PubMed  Google Scholar 

  48. Ratcliff R, Murdock BB (1976) Retrieval processes in recognition memory. Psychol Rev 83(3):190

    Article  Google Scholar 

  49. Roach NW, Heron J, Whitaker D, McGraw PV (2011) Asynchrony adaptation reveals neural population code for audio–visual timing. Proc Biol Sci 278(1710):1314–1322. doi:10.1098/rspb.2010.1737

    Article  PubMed  Google Scholar 

  50. Roseboom W, Arnold DH (2011) Twice upon a time multiple concurrent temporal recalibrations of audiovisual speech. Psychol Sci 22(7):872–877

    Article  PubMed  Google Scholar 

  51. Roseboom W, Linares D, Nishida S (2015) Sensory adaptation for timing perception. Proc R Soc Lon B: Biol Sci 282:20142833. doi:10.1098/rspb.2014.2833

    Article  Google Scholar 

  52. Sanders A (1983) Towards a model of stress and human performance. Acta Psychol (Amst) 53(1):61–97

    CAS  Article  Google Scholar 

  53. Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519–R521

    CAS  Article  PubMed  Google Scholar 

  54. Spence C, Nicholls ME, Driver J (2001a) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63(2):330–336

    CAS  Article  PubMed  Google Scholar 

  55. Spence C, Shore DI, Klein RM (2001b) Multisensory prior entry. J Exp Psychol-Gen 130(4):799–832. doi:10.1037//0096-3445.130.4.799

    CAS  Article  PubMed  Google Scholar 

  56. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, Mass

    Google Scholar 

  57. Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266. doi:10.1038/nrn2331

    CAS  Article  PubMed  Google Scholar 

  58. Sternberg S, Knoll RL (1973) The perception of temporal order: Fundamental issues and a general model. Attention and Performance IV:629–685

    Google Scholar 

  59. Tünnermann J, Petersen A, Scharlau I (2015) Does attention speed up processing? Decreases and increases of processing rates in visual prior entry. J Vis 15(3):1–27

    Article  PubMed  Google Scholar 

  60. Ulrich R (1987) Threshold models of temporal-order judgments evaluated by a ternary response task. Percept Psychophys 42(3):224–239

    CAS  Article  PubMed  Google Scholar 

  61. Ulrich R, Miller J, Schroter H (2007) Testing the race model inequality: an algorithm and computer programs. Behav Res Methods 39(2):291–302

    Article  PubMed  Google Scholar 

  62. Van der Burg E, Olivers CN, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053–1065. doi:10.1037/0096-1523.34.5.1053

    Article  PubMed  Google Scholar 

  63. Vibell J, Klinge C, Zampini M, Spence C, Nobre AC (2007) Temporal order is coded temporally in the brain: early ERP latency shifts underlying prior entry in a crossmodal temporal order judgment task. J Cogn Neurosci 19:109–120

    CAS  Article  PubMed  Google Scholar 

  64. Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72(4):871–884. doi:10.3758/APP.72.4.871

    Article  PubMed  Google Scholar 

  65. Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Brain Res Cogn Brain Res 22(1):32–35. doi:10.1016/j.cogbrainres.2004.07.003

    Article  PubMed  Google Scholar 

  66. Wallace MT, Stevenson RA (2014) The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia 64:105–123. doi:10.1016/j.neuropsychologia.2014.08.005

    Article  PubMed  PubMed Central  Google Scholar 

  67. Woodworth RS, Schlosberg H (1938) Experimental psychology. Holt, New York, pp 327–328

    Google Scholar 

  68. Yarrow K, Jahn N, Durant S, Arnold DH (2011) Shifts of criteria or neural timing? The assumptions underlying timing perception studies. Conscious Cogn 20(4):1518–1531

    Article  PubMed  Google Scholar 

  69. Yuan X, Bi C, Yin H, Li B, Huang X (2014) The recalibration patterns of perceptual synchrony and multisensory integration after exposure to asynchronous speech. Neurosci Lett 569:148–152

    CAS  Article  PubMed  Google Scholar 

Download references

Acknowledgements

Thanks to Alice Newton-Fenner for help with data collection and Phil Jaekl for excellent manuscript revision suggestions. Vanessa Harrar was supported by the Mary Somerville Junior Research Fellowship (Somerville College, Oxford University, UK), and by the Banting Fellowship (Canadian Institute of Health Research, Canada). LRH was supported by the Natural Sciences and Engineering Research Council of Canada.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Vanessa Harrar.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 262 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Harrar, V., Harris, L.R. & Spence, C. Multisensory integration is independent of perceived simultaneity. Exp Brain Res 235, 763–775 (2017). https://doi.org/10.1007/s00221-016-4822-2

Download citation

Keywords

  • Multisensory integration
  • Race model
  • Reaction time
  • Miller’s inequality
  • Stimulus onset asynchrony
  • Adaptation
  • Time
  • Crossmodal
  • Ex-Gaussian