Advertisement

Experimental Brain Research

, Volume 227, Issue 2, pp 249–261 | Cite as

Multisensory temporal integration: task and stimulus dependencies

  • Ryan A. StevensonEmail author
  • Mark T. Wallace
Research Article

Abstract

The ability of human sensory systems to integrate information across the different modalities provides a wide range of behavioral and perceptual benefits. This integration process is dependent upon the temporal relationship of the different sensory signals, with stimuli occurring close together in time typically resulting in the largest behavior changes. The range of temporal intervals over which such benefits are seen is typically referred to as the temporal binding window (TBW). Given the importance of temporal factors in multisensory integration under both normal and atypical circumstances such as autism and dyslexia, the TBW has been measured with a variety of experimental protocols that differ according to criterion, task, and stimulus type, making comparisons across experiments difficult. In the current study, we attempt to elucidate the role that these various factors play in the measurement of this important construct. The results show a strong effect of stimulus type, with the TBW assessed with speech stimuli being both larger and more symmetrical than that seen using simple and complex non-speech stimuli. These effects are robust across task and statistical criteria and are highly consistent within individuals, suggesting substantial overlap in the neural and cognitive operations that govern multisensory temporal processes.

Keywords

Audiovisual Multisensory integration Temporal binding window Psychophysics Temporal perception Speech perception Synchrony Multimodal 

Notes

Acknowledgments

This research was funded in part through a grant from NIDCD awarded to Mark Wallace and Stephen Camarata, NIH # R34 DC010927, as well as an NIDCD grant awarded to Ryan Stevenson, NIH 1F32 DC011993. We also acknowledge the help of Raquel Zemtsov, Juliane Kreuger Fister, and Justin Siemann with assistance running subjects, Zachary Barnett for technical assistance, and Lena Quinto for the speech stimuli.

References

  1. Beauchamp MS, Nath AR, Pasalar S (2010) fMRI-Guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J Neurosci 30:2414–2417. doi: 10.1523/JNEUROSCI.4865-09.2010 PubMedCrossRefGoogle Scholar
  2. Boenke LT, Deliano M, Ohl FW (2009) Stimulus duration influences perceived simultaneity in audiovisual temporal-order judgment. Exp Brain Res 198:233–244. doi: 10.1007/s00221-009-1917-z PubMedCrossRefGoogle Scholar
  3. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436PubMedCrossRefGoogle Scholar
  4. Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119:4065–4073PubMedCrossRefGoogle Scholar
  5. de Boer-Schellekens L, Eussen M, Vroomen J (2013) Diminished sensitivity of audiovisual temporal order in autism spectrum disorder. Front Integr Neurosci 7:8. doi: 10.3389/fnint.2013.00008 PubMedCrossRefGoogle Scholar
  6. Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66:1388–1404PubMedCrossRefGoogle Scholar
  7. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9:719–721PubMedCrossRefGoogle Scholar
  8. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2009) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res. doi: 10.1007/s00221-010-2240-4 Google Scholar
  9. Hairston WD, Burdette JH, Flowers DL, Wood FB, Wallace MT (2005) Altered temporal profile of visual-auditory multisensory interactions in dyslexia. Exp Brain Res 166:474–480. doi: 10.1007/s00221-005-2387-6 PubMedCrossRefGoogle Scholar
  10. Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293PubMedCrossRefGoogle Scholar
  11. Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49:461–467. doi: 10.1016/j.neuropsychologia.2010.11.041 PubMedCrossRefGoogle Scholar
  12. Hillock-Dunn A, Wallace MT (2012) Developmental changes in the multisensory temporal binding window persist into adolescence. Dev Sci 15:688–696. doi: 10.1111/j.1467-7687.2012.01171.x PubMedCrossRefGoogle Scholar
  13. Hirsh IJ, Sherrick CE Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432PubMedCrossRefGoogle Scholar
  14. James TW, Stevenson RA, Kim S (2009) Assessing multisensory integration with additive factors and functional MRI. The International Society for Psychophysics, DublinGoogle Scholar
  15. James TW, VanDerKlok RM, Stevenson RA, James KH (2011) Multisensory perception of action in posterior temporal and parietal cortices. Neuropsychologia 49:108–114. doi: 10.1016/j.neuropsychologia.2010.10.030 PubMedCrossRefGoogle Scholar
  16. James TW, Stevenson RA, Kim S (2012) Inverse effectiveness in multisensory processing. In: Stein BE (ed) The new handbook of multisensory processes. MIT Press, Cambridge, MAGoogle Scholar
  17. Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167:635–640. doi: 10.1007/s00221-005-0067-1 PubMedCrossRefGoogle Scholar
  18. Kuling IA, van Eijk RL, Juola JF, Kohlrausch A (2012) Effects of stimulus duration on audio-visual synchrony perception. Exp Brain Res 221:403–412. doi: 10.1007/s00221-012-3182-9 PubMedCrossRefGoogle Scholar
  19. Kwakye LD, Foss-Feig JH, Cascio CJ, Stone WL, Wallace MT (2011) Altered auditory and multisensory temporal processing in autism spectrum disorders. Front Integr Neurosci 4:129. doi: 10.3389/fnint.2010.00129 PubMedCrossRefGoogle Scholar
  20. Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17:447–453. doi: 10.1016/S0926-6410(03)00160-5 PubMedCrossRefGoogle Scholar
  21. Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21:725–732PubMedCrossRefGoogle Scholar
  22. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7:3215–3229PubMedGoogle Scholar
  23. Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88:181–186PubMedCrossRefGoogle Scholar
  24. Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884–5893PubMedCrossRefGoogle Scholar
  25. Nath AR, Beauchamp MS (2011) A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage. doi: 10.1016/j.neuroimage.2011.07.024 Google Scholar
  26. Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley RL (1998) Effects of localized auditory information on visual target detection performance using a helmet-mounted display. Hum Factors 40:452–460PubMedCrossRefGoogle Scholar
  27. Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10:437–442PubMedCrossRefGoogle Scholar
  28. Pelphrey KA, Carter EJ (2008) Charting the typical and atypical development of the social brain. Dev Psychopathol 20:1081–1102. doi: 10.1017/S0954579408000515 PubMedCrossRefGoogle Scholar
  29. Pöppel E, Schill K, von Steinbüchel N (1990) Sensory integration within temporally neutral systems states: a hypothesis. Naturwissenschaften 77:89–91PubMedCrossRefGoogle Scholar
  30. Powers AR 3rd, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29:12265–12274. doi: 10.1523/JNEUROSCI.3501-09.2009 PubMedCrossRefGoogle Scholar
  31. Quinto L, Thompson WF, Russo FA, Trehub SE (2010) A comparison of the McGurk effect for spoken and sung syllables. Atten Percept Psychophys 72:1450–1454. doi: 10.3758/APP.72.6.1450 PubMedCrossRefGoogle Scholar
  32. Royal DW, Carriere BN, Wallace MT (2009) Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Exp Brain Res 198:127–136. doi: 10.1007/s00221-009-1772-y PubMedCrossRefGoogle Scholar
  33. Schall JD, Hanes DP (1993) Neural basis of saccade target selection in frontal eye field during visual search. Nature 366:467–469. doi: 10.1038/366467a0 PubMedCrossRefGoogle Scholar
  34. Schall S, Quigley C, Onat S, Konig P (2009) Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli. Exp Brain Res 198:137–151. doi: 10.1007/s00221-009-1867-5 PubMedCrossRefGoogle Scholar
  35. Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG (2007) Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45:561–571PubMedCrossRefGoogle Scholar
  36. Stein BE, Wallace MT (1996) Comparisons of cross-modality integration in midbrain and cortex. Prog Brain Res 112:289–299PubMedCrossRefGoogle Scholar
  37. Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44:1210–1223. doi: 10.1016/j.neuroimage.2008.09.034 PubMedCrossRefGoogle Scholar
  38. Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179:85–95PubMedCrossRefGoogle Scholar
  39. Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198:183–194. doi: 10.1007/s00221-009-1783-8 PubMedCrossRefGoogle Scholar
  40. Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. Neuroimage 49:3308–3318. doi: 10.1016/j.neuroimage.2009.12.001 PubMedCrossRefGoogle Scholar
  41. Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. Neuroimage 55:1339–1345. doi: 10.1016/j.neuroimage.2010.12.063 PubMedCrossRefGoogle Scholar
  42. Stevenson RA, Wilson MM, Powers AR, Wallace MT (2013) The effects of visual training on multisensory temporal processing. Exp Brain Res. doi: 10.1007/s00221-012-3387-y Google Scholar
  43. van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17:962–974. doi: 10.1093/cercor/bhl007 PubMedCrossRefGoogle Scholar
  44. van Eijk RL, Kohlrausch A, Juola JF, van de Par S (2008) Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type. Percept Psychophys 70:955–968PubMedCrossRefGoogle Scholar
  45. van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45:598–607. doi: 10.1016/j.neuropsychologia.2006.01.001 PubMedCrossRefGoogle Scholar
  46. Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111:134–142. doi: 10.1016/j.brainres.2006.05.078 PubMedCrossRefGoogle Scholar
  47. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2008) Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments. Exp Brain Res 185:521–529. doi: 10.1007/s00221-007-1168-9 PubMedCrossRefGoogle Scholar
  48. Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72:871–884. doi: 10.3758/APP.72.4.871 PubMedCrossRefGoogle Scholar
  49. Wallace MH, Murray MM (eds) (2011) Frontiers in the neural basis of multisensory processes. Taylor & Francis, LondonGoogle Scholar
  50. Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158:252–258. doi: 10.1007/s00221-004-1899-9 PubMedCrossRefGoogle Scholar
  51. Wilkinson LK, Meredith MA, Stein BE (1996) The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res 112:1–10PubMedCrossRefGoogle Scholar
  52. Woynaroski TG, Kwakye LD, Foss-Feig JH, Stevenson RA, Stone WL, Wallace MT (2013) Multisensory speech perception in high-functioning children with autism spectrum disorders. J Autism Dev Disord. doi: JADD-D-12-00456R1
  53. Zampini M, Guest S, Shore DI, Spence C (2005) Audio-visual simultaneity judgments. Percept Psychophys 67:531–544PubMedCrossRefGoogle Scholar
  54. Zilbovicius M, Meresse I, Chabane N, Brunelle F, Samson Y, Boddaert N (2006) Autism, the superior temporal sulcus and social perception. Trends Neurosci 29:359–366. doi: 10.1016/j.tins.2006.06.004 PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Department of Hearing and Speech SciencesVanderbilt University Medical CenterNashvilleUSA
  2. 2.Vanderbilt Kennedy CenterNashvilleUSA
  3. 3.Vanderbilt Brain InstituteNashvilleUSA
  4. 4.Department of PsychologyVanderbilt UniversityNashvilleUSA
  5. 5.Department of PsychiatryVanderbilt UniversityNashvilleUSA

Personalised recommendations