Advertisement

Experimental Brain Research

, Volume 219, Issue 1, pp 121–137 | Cite as

Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance

  • Ryan A. StevensonEmail author
  • Juliane Krueger Fister
  • Zachary P. Barnett
  • Aaron R. Nidiffer
  • Mark T. Wallace
Research Article

Abstract

In natural environments, human sensory systems work in a coordinated and integrated manner to perceive and respond to external events. Previous research has shown that the spatial and temporal relationships of sensory signals are paramount in determining how information is integrated across sensory modalities, but in ecologically plausible settings, these factors are not independent. In the current study, we provide a novel exploration of the impact on behavioral performance for systematic manipulations of the spatial location and temporal synchrony of a visual-auditory stimulus pair. Simple auditory and visual stimuli were presented across a range of spatial locations and stimulus onset asynchronies (SOAs), and participants performed both a spatial localization and simultaneity judgment task. Response times in localizing paired visual-auditory stimuli were slower in the periphery and at larger SOAs, but most importantly, an interaction was found between the two factors, in which the effect of SOA was greater in peripheral as opposed to central locations. Simultaneity judgments also revealed a novel interaction between space and time: individuals were more likely to judge stimuli as synchronous when occurring in the periphery at large SOAs. The results of this study provide novel insights into (a) how the speed of spatial localization of an audiovisual stimulus is affected by location and temporal coincidence and the interaction between these two factors and (b) how the location of a multisensory stimulus impacts judgments concerning the temporal relationship of the paired stimuli. These findings provide strong evidence for a complex interdependency between spatial location and temporal structure in determining the ultimate behavioral and perceptual outcome associated with a paired multisensory (i.e., visual-auditory) stimulus.

Keywords

Audiovisual Inverse effectiveness Response time Race model Multisensory 

Notes

Acknowledgments

This research was funded in part through a grant from NIDCD awarded to Mark Wallace and Stephen Camarata, NIH # R34 DC010927, as well as an NIDCD grant awarded to Ryan Stevenson, NIH 1F32 DC011993. We would also like to acknowledge the support of the Vanderbilt Kennedy Center and the Vanderbilt Brain Institute.

Supplementary material

221_2012_3072_MOESM1_ESM.docx (330 kb)
Supplementary material 1 (DOCX 330 kb)

References

  1. Blauert J (1997) Spatial hearing: the psychophysics of human sound localization. MIT Press, CambridgeGoogle Scholar
  2. Carriere BN, Royal DW, Wallace MT (2008) Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions. J Neurophysiol 99(5):2357–2368. doi: 10.1152/jn.01386.2007 PubMedCrossRefGoogle Scholar
  3. Colavita F (1974) Human sensory dominance. Atten Percept Psychophys 16(2):409–412. doi: 10.3758/bf03203962 Google Scholar
  4. Colonius H, Diederich A (2006) The race model inequality: interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154. doi: 10.1037/0033-295X.113.1.148 PubMedCrossRefGoogle Scholar
  5. Conrey BL, Pisoni DB (2004) Detection of auditory-visual asynchrony in speech and nonspeech signals. Research on Spoken Language Processing, vol 26. Indiana University, BloomingtonGoogle Scholar
  6. Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119(6):4065–4073PubMedCrossRefGoogle Scholar
  7. Corey DP, Hudspeth AJ (1979) Response latency of vertebrate hair cells. Biophys J 26(3):499–506. doi: 10.1016/S0006-3495(79)85267-4 PubMedCrossRefGoogle Scholar
  8. Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404PubMedCrossRefGoogle Scholar
  9. Diederich A, Colonius H (2008) When a high-intensity “distractor” is better then a low-intensity one: modeling the effect of an auditory or tactile nontarget stimulus on visual saccadic reaction time. Brain Res 1242:219–230. doi: 10.1016/j.brainres.2008.05.081 PubMedCrossRefGoogle Scholar
  10. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721PubMedCrossRefGoogle Scholar
  11. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H, Stone WL, Wallace MT (2010) An extended multisensory temporal binding window in autism spectrum disorders. Exp Brain Res 203(2):381–389. doi: 10.1007/s00221-010-2240-4 PubMedCrossRefGoogle Scholar
  12. Hairston WD, Laurienti PJ, Mishra G, Burdette JH, Wallace MT (2003) Multisensory enhancement of localization under conditions of induced myopia. Exp Brain Res 152(3):404–408PubMedCrossRefGoogle Scholar
  13. Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189(2):133–143. doi: 10.1007/s00221-008-1410-0 PubMedCrossRefGoogle Scholar
  14. Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293PubMedCrossRefGoogle Scholar
  15. Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49(3):461–467. doi: 10.1016/j.neuropsychologia.2010.11.041 PubMedCrossRefGoogle Scholar
  16. Hirsh IJ, Sherrick CE Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432PubMedCrossRefGoogle Scholar
  17. Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167(4):635–640. doi: 10.1007/s00221-005-0067-1 PubMedCrossRefGoogle Scholar
  18. Kim S, James TW (2010) Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience. Hum Brain Mapp 31(5):678–693. doi: 10.1002/hbm.20897 PubMedCrossRefGoogle Scholar
  19. Kim S, Stevenson RA, James TW (in press) Visuo-haptic neuronal convergence demonstrated with an inversely effective pattern of BOLD activation. J Cognit Neurosci. doi: 10.1162/jocn_a_00176
  20. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60(3):492–500PubMedCrossRefGoogle Scholar
  21. Krueger J, Royal DW, Fister MC, Wallace MT (2009) Spatial receptive field organization of multisensory neurons and its impact on multisensory interactions. Hear Res 258(1–2):47–54. doi: 10.1016/j.heares.2009.08.003 PubMedCrossRefGoogle Scholar
  22. Lamb TD, Pugh EN Jr (1992) A quantitative account of the activation steps involved in phototransduction in amphibian photoreceptors. J Physiol 449:719–758PubMedGoogle Scholar
  23. Lennie P (1981) The physiological basis of variations in visual latency. Vision Res 21(6):815–824PubMedCrossRefGoogle Scholar
  24. Lewald J, Guski R (2003) Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Brain Res Cogn Brain Res 16(3):468–478PubMedCrossRefGoogle Scholar
  25. Lewald J, Guski R (2004) Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neurosci Lett 357(2):119–122. doi: 10.1016/j.neulet.2003.12.045 PubMedCrossRefGoogle Scholar
  26. Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453PubMedCrossRefGoogle Scholar
  27. Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. NeuroImage 21(2):725–732PubMedCrossRefGoogle Scholar
  28. Matin L (1986) Visual localization and eye movements. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance, vol 1. Wiley-Interscience, New York, pp 20.21–20.45Google Scholar
  29. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746–748PubMedCrossRefGoogle Scholar
  30. Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354PubMedCrossRefGoogle Scholar
  31. Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662PubMedGoogle Scholar
  32. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7(10):3215–3229PubMedGoogle Scholar
  33. Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88(1):181–186PubMedCrossRefGoogle Scholar
  34. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14(2):247–279PubMedCrossRefGoogle Scholar
  35. Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25(25):5884–5893PubMedCrossRefGoogle Scholar
  36. Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley RL (1998) Effects of localized auditory information on visual target detection performance using a helmet-mounted display. Hum Factors 40(3):452–460PubMedCrossRefGoogle Scholar
  37. Nordlund B (1962) Physical factors in angular localization. Acta Otolaryngol 54:75–93PubMedCrossRefGoogle Scholar
  38. Pöppel E, Schill K, von Steinbüchel N (1990) Sensory integration within temporally neutral systems states: a hypothesis. Naturwissenschaften 77:89–91PubMedCrossRefGoogle Scholar
  39. Powers AR III, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29(39):12265–12274. doi: 10.1523/JNEUROSCI.3501-09.2009 PubMedCrossRefGoogle Scholar
  40. Raab DH (1962) Statistical facilitation of simple reaction times. Trans NY Acad Sci 24:574–590Google Scholar
  41. Roach NW, Heron J, Whitaker D, McGraw PV (2011) Asynchrony adaption reveals neural population code for audio-visual timing. Proc R Soc 278:9CrossRefGoogle Scholar
  42. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007a) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17(5):1147–1153. doi: 10.1093/cercor/bhl024 PubMedCrossRefGoogle Scholar
  43. Ross LA, Saint-Amour D, Leavitt VM, Molholm S, Javitt DC, Foxe JJ (2007b) Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions. Schizophr Res 97(1–3):173–183. doi: 10.1016/j.schres.2007.08.008 PubMedCrossRefGoogle Scholar
  44. Royal DW, Carriere BN, Wallace MT (2009) Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Exp Brain Res 198(2–3):127–136. doi: 10.1007/s00221-009-1772-y PubMedCrossRefGoogle Scholar
  45. Schall S, Quigley C, Onat S, Konig P (2009) Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli. Exp Brain Res 198(2–3):137–151. doi: 10.1007/s00221-009-1867-5 PubMedCrossRefGoogle Scholar
  46. Seitz AR Sr, Nanez JE, Holloway SR, Watanabe T (2006) Perceptual learning of motion leads to faster flicker perception. PLoS ONE 1(1):e28. doi: 10.1371/journal.pone.0000028 PubMedCrossRefGoogle Scholar
  47. Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG (2007) Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45(3):561–571PubMedCrossRefGoogle Scholar
  48. Shams L, Kamitani Y, Shimojo S (2000) Illusions. What you see is what you hear. Nature 408(6814):788PubMedCrossRefGoogle Scholar
  49. Stein BE, Wallace MT (1996) Comparisons of cross-modality integration in midbrain and cortex. Prog Brain Res 112:289–299PubMedCrossRefGoogle Scholar
  50. Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448(2):355–358PubMedCrossRefGoogle Scholar
  51. Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. NeuroImage 44(3):1210–1223. doi: 10.1016/j.neuroimage.2008.09.034 PubMedCrossRefGoogle Scholar
  52. Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95PubMedCrossRefGoogle Scholar
  53. Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198(2–3):183–194. doi: 10.1007/s00221-009-1783-8 PubMedCrossRefGoogle Scholar
  54. Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. NeuroImage 49(4):3308–3318. doi: 10.1016/j.neuroimage.2009.12.001 PubMedCrossRefGoogle Scholar
  55. Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. NeuroImage 55(3):1339–1345. doi: 10.1016/j.neuroimage.2010.12.063 PubMedCrossRefGoogle Scholar
  56. Sugita Y, Suzuki Y (2003) Audiovisual perception: implicit estimation of sound-arrival time. Nature 421(6926):911. doi: 10.1038/421911a PubMedCrossRefGoogle Scholar
  57. Talsma D, Senkowski D, Woldorff MG (2009) Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp Brain Res 198(2–3):313–328. doi: 10.1007/s00221-009-1858-6 PubMedCrossRefGoogle Scholar
  58. van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974. doi: 10.1093/cercor/bhl007 PubMedCrossRefGoogle Scholar
  59. van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607. doi: 10.1016/j.neuropsychologia.2006.01.001 PubMedCrossRefGoogle Scholar
  60. Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142. doi: 10.1016/j.brainres.2006.05.078 PubMedCrossRefGoogle Scholar
  61. Vroomen J, Keetels M (2010) Perception of intersensory synchrony: a tutorial review. Atten Percept Psychophys 72(4):871–884. doi: 10.3758/APP.72.4.871 PubMedCrossRefGoogle Scholar
  62. Wallace MH, Murray MM (eds) (2011) Frontiers in the neural basis of multisensory processes. Taylor & Francis, LondonGoogle Scholar
  63. Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91(3):484–488PubMedCrossRefGoogle Scholar
  64. Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258. doi: 10.1007/s00221-004-1899-9 PubMedCrossRefGoogle Scholar
  65. Wilkinson LK, Meredith MA, Stein BE (1996) The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res 112(1):1–10PubMedCrossRefGoogle Scholar
  66. Zampini M, Shore DI, Spence C (2003) Audiovisual temporal order judgments. Exp Brain Res 152(2):198–210. doi: 10.1007/s00221-003-1536-z PubMedCrossRefGoogle Scholar
  67. Zampini M, Guest S, Shore DI, Spence C (2005a) Audio-visual simultaneity judgments. Percept Psychophys 67(3):531–544PubMedCrossRefGoogle Scholar
  68. Zampini M, Shore DI, Spence C (2005b) Audiovisual prior entry. Neurosci Lett 381(3):217–222. doi: 10.1016/j.neulet.2005.01.085 PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Ryan A. Stevenson
    • 1
    • 3
    Email author
  • Juliane Krueger Fister
    • 1
    • 2
  • Zachary P. Barnett
    • 1
    • 3
  • Aaron R. Nidiffer
    • 1
  • Mark T. Wallace
    • 1
    • 3
    • 4
    • 5
    • 6
  1. 1.Department of Hearing and Speech SciencesVanderbilt University Medical CenterNashvilleUSA
  2. 2.Neuroscience Graduate ProgramVanderbilt University Medical CenterNashvilleUSA
  3. 3.Vanderbilt Kennedy CenterNashvilleUSA
  4. 4.Vanderbilt Brain InstituteNashvilleUSA
  5. 5.Department of PsychologyVanderbilt UniversityNashvilleUSA
  6. 6.Department of PsychiatryVanderbilt UniversityNashvilleUSA

Personalised recommendations