Experimental Brain Research

, Volume 235, Issue 3, pp 763–775 | Cite as

Multisensory integration is independent of perceived simultaneity

  • Vanessa Harrar
  • Laurence R. Harris
  • Charles Spence
Research Article

Abstract

The importance of multisensory integration for perception and action has long been recognised. Integrating information from individual senses increases the chance of survival by reducing the variability in the incoming signals, thus allowing us to respond more rapidly. Reaction times (RTs) are fastest when the components of the multisensory signals are simultaneous. This response facilitation is traditionally attributed to multisensory integration. However, it is unclear if facilitation of RTs occurs when stimuli are perceived as synchronous or are actually physically synchronous. Repeated exposure to audiovisual asynchrony can change the delay at which multisensory stimuli are perceived as simultaneous, thus changing the delay at which the stimuli are integrated—perceptually. Here we set out to determine how such changes in multisensory integration for perception affect our ability to respond to multisensory events. If stimuli perceived as simultaneous were reacted to most rapidly, it would suggest a common system for multisensory integration for perception and action. If not, it would suggest separate systems. We measured RTs to auditory, visual, and audiovisual stimuli following exposure to audiovisual asynchrony. Exposure affected the variability of the unisensory RT distributions; in particular, the slowest RTs were either speed up or slowed down (in the direction predicted from shifts in perceived simultaneity). Additionally, the multisensory facilitation of RTs (beyond statistical summation) only occurred when audiovisual onsets were physically synchronous, rather than when they appeared simultaneous. We conclude that the perception of synchrony is therefore independent of multisensory integration and suggest a division between multisensory processes that are fast (automatic and unaffected by temporal adaptation) and those that are slow (perceptually driven and adaptable).

Keywords

Multisensory integration Race model Reaction time Miller’s inequality Stimulus onset asynchrony Adaptation Time Crossmodal Ex-Gaussian 

Notes

Acknowledgements

Thanks to Alice Newton-Fenner for help with data collection and Phil Jaekl for excellent manuscript revision suggestions. Vanessa Harrar was supported by the Mary Somerville Junior Research Fellowship (Somerville College, Oxford University, UK), and by the Banting Fellowship (Canadian Institute of Health Research, Canada). LRH was supported by the Natural Sciences and Engineering Research Council of Canada.

Supplementary material

221_2016_4822_MOESM1_ESM.docx (263 kb)
Supplementary material 1 (DOCX 262 kb)

References

  1. Balota DA, Spieler DH (1999) Word frequency, repetition, and lexicality effects in word recognition tasks: beyond measures of central tendency. J Exp Psychol Gen 128(1):32CrossRefPubMedGoogle Scholar
  2. Berman AL (1961) Interaction of cortical responses to somatic and auditory stimuli in anterior ectosylvian gyrus of cat. J Neurophysiol 24(6):608–620PubMedGoogle Scholar
  3. Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009. doi: 10.1162/0898929041502733 CrossRefPubMedGoogle Scholar
  4. Colonius H, Diederich A (2006) The race model inequality: interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154. doi: 10.1037/0033-295X.113.1.148 CrossRefPubMedGoogle Scholar
  5. Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vis 9(12):7.1–716. doi: 10.1167/9.12.7 Google Scholar
  6. Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404CrossRefPubMedGoogle Scholar
  7. Diederich A, Colonius H (2015) The time window of multisensory integration: relating reaction times and judgments of temporal order. Psychol Rev 122(2):232–241. doi: 10.1037/a0038696 CrossRefPubMedGoogle Scholar
  8. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721CrossRefPubMedGoogle Scholar
  9. Donders FC (1868) /1969). Die Schnelligkeit psychischer Processe [On the speed of mental processes]. Acta Psychol 30:412–431CrossRefGoogle Scholar
  10. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433. doi: 10.1038/415429a CrossRefPubMedGoogle Scholar
  11. Farmer ME, Klein RM (1995) The evidence for a temporal processing deficit linked to dyslexia: a review. Psychon Bull Rev 2(4):460–493. doi: 10.3758/Bf03210983 CrossRefPubMedGoogle Scholar
  12. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389. doi: 10.1007/s00221-010-2240-4 CrossRefPubMedPubMedCentralGoogle Scholar
  13. Foucher JR, Lacambre M, Pham BT, Giersch A, Elliott MA (2007) Low time resolution in schizophrenia Lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophr Res 97(1–3):118–127. doi: 10.1016/j.schres.2007.08.013 CrossRefPubMedGoogle Scholar
  14. Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778. doi: 10.1038/nn1268 CrossRefPubMedGoogle Scholar
  15. Gibbon J, Rutschmann R (1969) Temporal order judgement and reaction time. Science 165(3891):413–415CrossRefPubMedGoogle Scholar
  16. Goodale MA, Milner AD (1992) Separate visual pathways for perception and action. Trends Neurosci 15(1):20–25CrossRefPubMedGoogle Scholar
  17. Hanley JA, Negassa A, Forrester JE (2003) Statistical analysis of correlated data using generalized estimating equations: an orientation. Am J Epidemiol 157(4):364–375CrossRefPubMedGoogle Scholar
  18. Hanson JV, Heron J, Whitaker D (2008) Recalibration of perceived time across sensory modalities. Exp Brain Res 185(2):347–352. doi: 10.1007/s00221-008-1282-3 CrossRefPubMedGoogle Scholar
  19. Harrar V, Harris LR (2005) Simultaneity constancy: detecting events with touch and vision. Exp Brain Res 166(3–4):465–473. doi: 10.1007/s00221-005-2386-7 CrossRefPubMedGoogle Scholar
  20. Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524. doi: 10.1007/s00221-007-1253-0 CrossRefPubMedGoogle Scholar
  21. Harrar V, Tammam J, Perez-Bellido A, Pitt A, Stein J, Spence C (2014) Multisensory integration and attention in developmental dyslexia. Curr Biol 24(5):531–535. doi: 10.1016/j.cub.2014.01.029 CrossRefPubMedGoogle Scholar
  22. Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293CrossRefPubMedGoogle Scholar
  23. Jaekl P, Soto-Faraco S, Harris LR (2012) Perceived size change induced by audiovisual temporal delays. Exp Brain Res 216(3):457–462. doi: 10.1007/s00221-011-2948-9 CrossRefPubMedGoogle Scholar
  24. Jaekl P, Seidlitz J, Harris LR, Tadin D (2015) Audiovisual delay as a novel cue to visual distance. PLoS ONE 10(10):e0141125. doi: 10.1371/journal.pone.0141125 CrossRefPubMedPubMedCentralGoogle Scholar
  25. Jaskowski P (1992) Temporal-order judgment and reaction time for short and long stimuli. Psychol Res 54(3):141–145CrossRefPubMedGoogle Scholar
  26. Jiang H, Lepore F, Ptito M, Guillemot J-P (1994) Sensory interactions in the anterior ectosylvian cortex of cats. Exp Brain Res 101(3):385–396CrossRefPubMedGoogle Scholar
  27. Keetels M, Vroomen J (2012) Exposure to delayed visual feedback of the hand changes motor-sensory synchrony perception. Exp Brain Res 219(4):431–440. doi: 10.1007/s00221-012-3081-0 CrossRefPubMedPubMedCentralGoogle Scholar
  28. Kopinska A, Harris LR (2004) Simultaneity constancy. Perception 33(9):1049–1060CrossRefPubMedGoogle Scholar
  29. Lacouture Y, Cousineau D (2008) How to use MATLAB to fit the ex-Gaussian and other probability functions to a distribution of response times. Tutor Quant Methods Psychol 4(1):35–45CrossRefGoogle Scholar
  30. Leone LM, McCourt ME (2012) The question of simultaneity in multisensory integration. Human Vis Electron Imaging Xvii. doi: 10.1117/12.912183 Google Scholar
  31. Leone LM, McCourt ME (2013) The roles of physical and physiological simultaneity in audiovisual multisensory facilitation. i-Perception 4(4):213–228. doi: 10.1068/i0532 CrossRefPubMedPubMedCentralGoogle Scholar
  32. Leone LM, McCourt ME (2015) Dissociation of perception and action in audiovisual multisensory integration. Eur J Neurosci 42(11):2915–2922CrossRefPubMedPubMedCentralGoogle Scholar
  33. Leth-Steensen C, Elbaz ZK, Douglas VI (2000) Mean response times, variability, and skew in the responding of ADHD children: a response time distributional approach. Acta Psychol (Amst) 104(2):167–190CrossRefGoogle Scholar
  34. Luce RD (1986) Response times. Oxford University Press, New YorkGoogle Scholar
  35. Maier JX, Di Luca M, Noppeney U (2011) Audiovisual asynchrony detection in human speech. J Exp Psychol Hum Percept Perform 37(1):245CrossRefPubMedGoogle Scholar
  36. Matzke D, Wagenmakers EJ (2009) Psychological interpretation of the ex-Gaussian and shifted Wald parameters: a diffusion model analysis. Psychon Bull Rev 16(5):798–817. doi: 10.3758/PBR.16.5.798 CrossRefPubMedGoogle Scholar
  37. Megevand P, Molholm S, Nayak A, Foxe JJ (2013) Recalibration of the multisensory temporal window of integration results from changing task demands. PLoS ONE 8(8):e71608. doi: 10.1371/journal.pone.0071608 CrossRefPubMedPubMedCentralGoogle Scholar
  38. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temp Factors. J Neurosci 7(10):3215–3229Google Scholar
  39. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14(2):247–279CrossRefPubMedGoogle Scholar
  40. Miller J (1986) Timecourse of coactivation in bimodal divided attention. Percept Psychophys 40(5):331–343CrossRefPubMedGoogle Scholar
  41. Miller RL, Pluta SR, Stein BE, Rowland BA (2015) Relative unisensory strength and timing predict their multisensory product. J Neurosci 35(13):5213–5220. doi: 10.1523/JNEUROSCI.4771-14.2015 CrossRefPubMedPubMedCentralGoogle Scholar
  42. Miyazaki M, Yamamoto S, Uchida S, Kitazawa S (2006) Bayesian calibration of simultaneity in tactile temporal order judgment. Nat Neurosci 9(7):875–877. doi: 10.1038/nn1712 CrossRefPubMedGoogle Scholar
  43. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14(1):115–128CrossRefPubMedGoogle Scholar
  44. Navarra J, Hartcher-O’Brien J, Piazza E, Spence C (2009) Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proc Natl Acad Sci U S A 106(23):9169–9173. doi: 10.1073/pnas.0810486106 CrossRefPubMedPubMedCentralGoogle Scholar
  45. Otto TU, Mamassian P (2012) Noise and correlations in parallel perceptual decision making. Curr Biol 22(15):1391–1396. doi: 10.1016/j.cub.2012.05.031 CrossRefPubMedGoogle Scholar
  46. Otto TU, Dassy B, Mamassian P (2013) Principles of multisensory behavior. J Neurosci 33(17):7463–7474. doi: 10.1523/JNEUROSCI.4678-12.2013 CrossRefPubMedGoogle Scholar
  47. Pöppel E, Schill K, von Steinbuchel N (1990) Sensory integration within temporally neutral systems states: a hypothesis. Naturwissenschaften 77(2):89–91CrossRefPubMedGoogle Scholar
  48. Ratcliff R, Murdock BB (1976) Retrieval processes in recognition memory. Psychol Rev 83(3):190CrossRefGoogle Scholar
  49. Roach NW, Heron J, Whitaker D, McGraw PV (2011) Asynchrony adaptation reveals neural population code for audio–visual timing. Proc Biol Sci 278(1710):1314–1322. doi: 10.1098/rspb.2010.1737 CrossRefPubMedGoogle Scholar
  50. Roseboom W, Arnold DH (2011) Twice upon a time multiple concurrent temporal recalibrations of audiovisual speech. Psychol Sci 22(7):872–877CrossRefPubMedGoogle Scholar
  51. Roseboom W, Linares D, Nishida S (2015) Sensory adaptation for timing perception. Proc R Soc Lon B: Biol Sci 282:20142833. doi: 10.1098/rspb.2014.2833 CrossRefGoogle Scholar
  52. Sanders A (1983) Towards a model of stress and human performance. Acta Psychol (Amst) 53(1):61–97CrossRefGoogle Scholar
  53. Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13(13):R519–R521CrossRefPubMedGoogle Scholar
  54. Spence C, Nicholls ME, Driver J (2001a) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63(2):330–336CrossRefPubMedGoogle Scholar
  55. Spence C, Shore DI, Klein RM (2001b) Multisensory prior entry. J Exp Psychol-Gen 130(4):799–832. doi: 10.1037//0096-3445.130.4.799 CrossRefPubMedGoogle Scholar
  56. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MassGoogle Scholar
  57. Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266. doi: 10.1038/nrn2331 CrossRefPubMedGoogle Scholar
  58. Sternberg S, Knoll RL (1973) The perception of temporal order: Fundamental issues and a general model. Attention and Performance IV:629–685Google Scholar
  59. Tünnermann J, Petersen A, Scharlau I (2015) Does attention speed up processing? Decreases and increases of processing rates in visual prior entry. J Vis 15(3):1–27CrossRefPubMedGoogle Scholar
  60. Ulrich R (1987) Threshold models of temporal-order judgments evaluated by a ternary response task. Percept Psychophys 42(3):224–239CrossRefPubMedGoogle Scholar
  61. Ulrich R, Miller J, Schroter H (2007) Testing the race model inequality: an algorithm and computer programs. Behav Res Methods 39(2):291–302CrossRefPubMedGoogle Scholar
  62. Van der Burg E, Olivers CN, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053–1065. doi: 10.1037/0096-1523.34.5.1053 CrossRefPubMedGoogle Scholar
  63. Vibell J, Klinge C, Zampini M, Spence C, Nobre AC (2007) Temporal order is coded temporally in the brain: early ERP latency shifts underlying prior entry in a crossmodal temporal order judgment task. J Cogn Neurosci 19:109–120CrossRefPubMedGoogle Scholar
  64. Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72(4):871–884. doi: 10.3758/APP.72.4.871 CrossRefPubMedGoogle Scholar
  65. Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Brain Res Cogn Brain Res 22(1):32–35. doi: 10.1016/j.cogbrainres.2004.07.003 CrossRefPubMedGoogle Scholar
  66. Wallace MT, Stevenson RA (2014) The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia 64:105–123. doi: 10.1016/j.neuropsychologia.2014.08.005 CrossRefPubMedPubMedCentralGoogle Scholar
  67. Woodworth RS, Schlosberg H (1938) Experimental psychology. Holt, New York, pp 327–328Google Scholar
  68. Yarrow K, Jahn N, Durant S, Arnold DH (2011) Shifts of criteria or neural timing? The assumptions underlying timing perception studies. Conscious Cogn 20(4):1518–1531CrossRefPubMedGoogle Scholar
  69. Yuan X, Bi C, Yin H, Li B, Huang X (2014) The recalibration patterns of perceptual synchrony and multisensory integration after exposure to asynchronous speech. Neurosci Lett 569:148–152CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.School of OptometryUniversity of MontrealMontrealCanada
  2. 2.Department of Experimental PsychologyUniversity of OxfordOxfordUK
  3. 3.Centre for Vision ResearchYork UniversityTorontoCanada

Personalised recommendations