Skip to main content

Advertisement

SpringerLink
Log in

Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

In natural environments, human sensory systems work in a coordinated and integrated manner to perceive and respond to external events. Previous research has shown that the spatial and temporal relationships of sensory signals are paramount in determining how information is integrated across sensory modalities, but in ecologically plausible settings, these factors are not independent. In the current study, we provide a novel exploration of the impact on behavioral performance for systematic manipulations of the spatial location and temporal synchrony of a visual-auditory stimulus pair. Simple auditory and visual stimuli were presented across a range of spatial locations and stimulus onset asynchronies (SOAs), and participants performed both a spatial localization and simultaneity judgment task. Response times in localizing paired visual-auditory stimuli were slower in the periphery and at larger SOAs, but most importantly, an interaction was found between the two factors, in which the effect of SOA was greater in peripheral as opposed to central locations. Simultaneity judgments also revealed a novel interaction between space and time: individuals were more likely to judge stimuli as synchronous when occurring in the periphery at large SOAs. The results of this study provide novel insights into (a) how the speed of spatial localization of an audiovisual stimulus is affected by location and temporal coincidence and the interaction between these two factors and (b) how the location of a multisensory stimulus impacts judgments concerning the temporal relationship of the paired stimuli. These findings provide strong evidence for a complex interdependency between spatial location and temporal structure in determining the ultimate behavioral and perceptual outcome associated with a paired multisensory (i.e., visual-auditory) stimulus.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Blauert J (1997) Spatial hearing: the psychophysics of human sound localization. MIT Press, Cambridge

    Google Scholar 

  • Carriere BN, Royal DW, Wallace MT (2008) Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions. J Neurophysiol 99(5):2357–2368. doi:10.1152/jn.01386.2007

    Article  PubMed  Google Scholar 

  • Colavita F (1974) Human sensory dominance. Atten Percept Psychophys 16(2):409–412. doi:10.3758/bf03203962

    Google Scholar 

  • Colonius H, Diederich A (2006) The race model inequality: interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154. doi:10.1037/0033-295X.113.1.148

    Article  PubMed  Google Scholar 

  • Conrey BL, Pisoni DB (2004) Detection of auditory-visual asynchrony in speech and nonspeech signals. Research on Spoken Language Processing, vol 26. Indiana University, Bloomington

  • Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119(6):4065–4073

    Article  PubMed  Google Scholar 

  • Corey DP, Hudspeth AJ (1979) Response latency of vertebrate hair cells. Biophys J 26(3):499–506. doi:10.1016/S0006-3495(79)85267-4

    Article  PubMed  CAS  Google Scholar 

  • Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404

    Article  PubMed  Google Scholar 

  • Diederich A, Colonius H (2008) When a high-intensity “distractor” is better then a low-intensity one: modeling the effect of an auditory or tactile nontarget stimulus on visual saccadic reaction time. Brain Res 1242:219–230. doi:10.1016/j.brainres.2008.05.081

    Article  PubMed  CAS  Google Scholar 

  • Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    Article  PubMed  CAS  Google Scholar 

  • Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389. doi:10.1007/s00221-010-2240-4

    Article  PubMed  Google Scholar 

  • Hairston WD, Laurienti PJ, Mishra G, Burdette JH, Wallace MT (2003) Multisensory enhancement of localization under conditions of induced myopia. Exp Brain Res 152(3):404–408

    Article  PubMed  Google Scholar 

  • Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189(2):133–143. doi:10.1007/s00221-008-1410-0

    Article  PubMed  Google Scholar 

  • Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293

    Article  PubMed  CAS  Google Scholar 

  • Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49(3):461–467. doi:10.1016/j.neuropsychologia.2010.11.041

    Article  PubMed  Google Scholar 

  • Hirsh IJ, Sherrick CE Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432

    Article  PubMed  CAS  Google Scholar 

  • Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167(4):635–640. doi:10.1007/s00221-005-0067-1

    Article  PubMed  Google Scholar 

  • Kim S, James TW (2010) Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience. Hum Brain Mapp 31(5):678–693. doi:10.1002/hbm.20897

    Article  PubMed  CAS  Google Scholar 

  • Kim S, Stevenson RA, James TW (in press) Visuo-haptic neuronal convergence demonstrated with an inversely effective pattern of BOLD activation. J Cognit Neurosci. doi:10.1162/jocn_a_00176

  • King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60(3):492–500

    Article  PubMed  CAS  Google Scholar 

  • Krueger J, Royal DW, Fister MC, Wallace MT (2009) Spatial receptive field organization of multisensory neurons and its impact on multisensory interactions. Hear Res 258(1–2):47–54. doi:10.1016/j.heares.2009.08.003

    Article  PubMed  Google Scholar 

  • Lamb TD, Pugh EN Jr (1992) A quantitative account of the activation steps involved in phototransduction in amphibian photoreceptors. J Physiol 449:719–758

    PubMed  CAS  Google Scholar 

  • Lennie P (1981) The physiological basis of variations in visual latency. Vision Res 21(6):815–824

    Article  PubMed  CAS  Google Scholar 

  • Lewald J, Guski R (2003) Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Brain Res Cogn Brain Res 16(3):468–478

    Article  PubMed  Google Scholar 

  • Lewald J, Guski R (2004) Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neurosci Lett 357(2):119–122. doi:10.1016/j.neulet.2003.12.045

    Article  PubMed  CAS  Google Scholar 

  • Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453

    Article  PubMed  Google Scholar 

  • Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. NeuroImage 21(2):725–732

    Article  PubMed  CAS  Google Scholar 

  • Matin L (1986) Visual localization and eye movements. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance, vol 1. Wiley-Interscience, New York, pp 20.21–20.45

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746–748

    Article  PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354

    Article  PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662

    PubMed  CAS  Google Scholar 

  • Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7(10):3215–3229

    PubMed  CAS  Google Scholar 

  • Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88(1):181–186

    Article  PubMed  CAS  Google Scholar 

  • Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14(2):247–279

    Article  PubMed  CAS  Google Scholar 

  • Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25(25):5884–5893

    Article  PubMed  CAS  Google Scholar 

  • Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley RL (1998) Effects of localized auditory information on visual target detection performance using a helmet-mounted display. Hum Factors 40(3):452–460

    Article  PubMed  CAS  Google Scholar 

  • Nordlund B (1962) Physical factors in angular localization. Acta Otolaryngol 54:75–93

    Article  PubMed  CAS  Google Scholar 

  • Pöppel E, Schill K, von Steinbüchel N (1990) Sensory integration within temporally neutral systems states: a hypothesis. Naturwissenschaften 77:89–91

    Article  PubMed  Google Scholar 

  • Powers AR III, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29(39):12265–12274. doi:10.1523/JNEUROSCI.3501-09.2009

    Article  PubMed  CAS  Google Scholar 

  • Raab DH (1962) Statistical facilitation of simple reaction times. Trans NY Acad Sci 24:574–590

    CAS  Google Scholar 

  • Roach NW, Heron J, Whitaker D, McGraw PV (2011) Asynchrony adaption reveals neural population code for audio-visual timing. Proc R Soc 278:9

    Article  Google Scholar 

  • Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007a) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17(5):1147–1153. doi:10.1093/cercor/bhl024

    Article  PubMed  Google Scholar 

  • Ross LA, Saint-Amour D, Leavitt VM, Molholm S, Javitt DC, Foxe JJ (2007b) Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions. Schizophr Res 97(1–3):173–183. doi:10.1016/j.schres.2007.08.008

    Article  PubMed  Google Scholar 

  • Royal DW, Carriere BN, Wallace MT (2009) Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Exp Brain Res 198(2–3):127–136. doi:10.1007/s00221-009-1772-y

    Article  PubMed  Google Scholar 

  • Schall S, Quigley C, Onat S, Konig P (2009) Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli. Exp Brain Res 198(2–3):137–151. doi:10.1007/s00221-009-1867-5

    Article  PubMed  Google Scholar 

  • Seitz AR Sr, Nanez JE, Holloway SR, Watanabe T (2006) Perceptual learning of motion leads to faster flicker perception. PLoS ONE 1(1):e28. doi:10.1371/journal.pone.0000028

    Article  PubMed  Google Scholar 

  • Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG (2007) Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45(3):561–571

    Article  PubMed  Google Scholar 

  • Shams L, Kamitani Y, Shimojo S (2000) Illusions. What you see is what you hear. Nature 408(6814):788

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Wallace MT (1996) Comparisons of cross-modality integration in midbrain and cortex. Prog Brain Res 112:289–299

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448(2):355–358

    Article  PubMed  CAS  Google Scholar 

  • Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. NeuroImage 44(3):1210–1223. doi:10.1016/j.neuroimage.2008.09.034

    Article  PubMed  Google Scholar 

  • Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95

    Article  PubMed  Google Scholar 

  • Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198(2–3):183–194. doi:10.1007/s00221-009-1783-8

    Article  PubMed  Google Scholar 

  • Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. NeuroImage 49(4):3308–3318. doi:10.1016/j.neuroimage.2009.12.001

    Article  PubMed  Google Scholar 

  • Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. NeuroImage 55(3):1339–1345. doi:10.1016/j.neuroimage.2010.12.063

    Article  PubMed  Google Scholar 

  • Sugita Y, Suzuki Y (2003) Audiovisual perception: implicit estimation of sound-arrival time. Nature 421(6926):911. doi:10.1038/421911a

    Article  PubMed  CAS  Google Scholar 

  • Talsma D, Senkowski D, Woldorff MG (2009) Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp Brain Res 198(2–3):313–328. doi:10.1007/s00221-009-1858-6

    Article  PubMed  Google Scholar 

  • van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974. doi:10.1093/cercor/bhl007

    Article  PubMed  Google Scholar 

  • van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607. doi:10.1016/j.neuropsychologia.2006.01.001

    Article  PubMed  Google Scholar 

  • Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142. doi:10.1016/j.brainres.2006.05.078

    Article  PubMed  CAS  Google Scholar 

  • Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72(4):871–884. doi:10.3758/APP.72.4.871

    Article  PubMed  Google Scholar 

  • Wallace MH, Murray MM (eds) (2011) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London

    Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91(3):484–488

    Article  PubMed  CAS  Google Scholar 

  • Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258. doi:10.1007/s00221-004-1899-9

    Article  PubMed  CAS  Google Scholar 

  • Wilkinson LK, Meredith MA, Stein BE (1996) The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res 112(1):1–10

    Article  PubMed  CAS  Google Scholar 

  • Zampini M, Shore DI, Spence C (2003) Audiovisual temporal order judgments. Exp Brain Res 152(2):198–210. doi:10.1007/s00221-003-1536-z

    Article  PubMed  Google Scholar 

  • Zampini M, Guest S, Shore DI, Spence C (2005a) Audio-visual simultaneity judgments. Percept Psychophys 67(3):531–544

    Article  PubMed  Google Scholar 

  • Zampini M, Shore DI, Spence C (2005b) Audiovisual prior entry. Neurosci Lett 381(3):217–222. doi:10.1016/j.neulet.2005.01.085

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

This research was funded in part through a grant from NIDCD awarded to Mark Wallace and Stephen Camarata, NIH # R34 DC010927, as well as an NIDCD grant awarded to Ryan Stevenson, NIH 1F32 DC011993. We would also like to acknowledge the support of the Vanderbilt Kennedy Center and the Vanderbilt Brain Institute.

Author information

Authors and Affiliations

  1. Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, TN, USA

    Ryan A. Stevenson, Juliane Krueger Fister, Zachary P. Barnett, Aaron R. Nidiffer & Mark T. Wallace

  2. Neuroscience Graduate Program, Vanderbilt University Medical Center, Nashville, TN, USA

    Juliane Krueger Fister

  3. Vanderbilt Kennedy Center, Nashville, TN, USA

    Ryan A. Stevenson, Zachary P. Barnett & Mark T. Wallace

  4. Vanderbilt Brain Institute, Nashville, TN, USA

    Mark T. Wallace

  5. Department of Psychology, Vanderbilt University, Nashville, TN, USA

    Mark T. Wallace

  6. Department of Psychiatry, Vanderbilt University, Nashville, TN, USA

    Mark T. Wallace

Authors
  1. Ryan A. Stevenson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juliane Krueger Fister
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zachary P. Barnett
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aaron R. Nidiffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mark T. Wallace
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryan A. Stevenson.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 330 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stevenson, R.A., Fister, J.K., Barnett, Z.P. et al. Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp Brain Res 219, 121–137 (2012). https://doi.org/10.1007/s00221-012-3072-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-012-3072-1

Keywords

Search

Navigation