Experimental Brain Research

, Volume 158, Issue 2, pp 252–258 | Cite as

Unifying multisensory signals across time and space

  • M. T. WallaceEmail author
  • G. E. Roberson
  • W. D. Hairston
  • B. E. Stein
  • J. W. Vaughan
  • J. A. Schirillo
Research Article


The brain integrates information from multiple sensory modalities and, through this process, generates a coherent and apparently seamless percept of the external world. Although multisensory integration typically binds information that is derived from the same event, when multisensory cues are somewhat discordant they can result in illusory percepts such as the “ventriloquism effect.” These biases in stimulus localization are generally accompanied by the perceptual unification of the two stimuli. In the current study, we sought to further elucidate the relationship between localization biases, perceptual unification and measures of a participant’s uncertainty in target localization (i.e., variability). Participants performed an auditory localization task in which they were also asked to report on whether they perceived the auditory and visual stimuli to be perceptually unified. The auditory and visual stimuli were delivered at a variety of spatial (0°, 5°, 10°, 15°) and temporal (200, 500, 800 ms) disparities. Localization bias and reports of perceptual unity occurred even with substantial spatial (i.e., 15°) and temporal (i.e., 800 ms) disparities. Trial-by-trial comparison of these measures revealed a striking correlation: regardless of their disparity, whenever the auditory and visual stimuli were perceived as unified, they were localized at or very near the light. In contrast, when the stimuli were perceived as not unified, auditory localization was often biased away from the visual stimulus. Furthermore, localization variability was significantly less when the stimuli were perceived as unified. Intriguingly, on non-unity trials such variability increased with decreasing disparity. Together, these results suggest strong and potentially mechanistic links between the multiple facets of multisensory integration that contribute to our perceptual Gestalt.


Cross-modal Ventriloquism Sensorimotor Visual Auditory 



We thank Julie Edelson for editorial assistance. The study was supported in part by US National Institutes of Health grants MH63861, NS22543 and NS36916.


  1. Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 42:487–493PubMedGoogle Scholar
  2. Bertelson P, Aschersleben G (1998) Automatic visual bias of perceived auditory location. Psychon Bull Rev 5:482–489Google Scholar
  3. Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29:578–584PubMedGoogle Scholar
  4. Choe CS, Welch RB, Gilford RM, Juola JF (1975) The “ventriloquist effect”: visual dominance or response bias? Percept Psychophys 18:55–60Google Scholar
  5. Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA (2003) Visual localization ability influences cross-modal bias. J Cogn Neurosci 15:20–29CrossRefPubMedGoogle Scholar
  6. Hartigan PM (1985) Statistical algorithms: algorithm AS 217. Computation of the dip statistic to test for unimodality. Appl Stat 34:320–325Google Scholar
  7. Hartigan JA, Hartigan PM (1985) The dip test of unimodality. Ann Stat 13:70–84Google Scholar
  8. Howard IP, Templeton WB (1966) Human Spatial orientation. Wiley, New YorkGoogle Scholar
  9. Jack CE, Thurlow WR (1973) Effects of degree of visual association and angle of displacement on the “ventriloquism” effect. Percept Mot Skills 37:967–979PubMedGoogle Scholar
  10. Lewald J, Guski R (2003) Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cogn Brain Res 16:468–478CrossRefGoogle Scholar
  11. Lewald J, Ehrenstein WH, Guski R (2001) Spatio-temporal constraints for auditory—visual integration. Behav Brain Res 121:69–79CrossRefPubMedGoogle Scholar
  12. Pick HL, Warren DH, Hay JC (1969) Sensory conflict in judgements of spatial direction. Percept Psychophys 6:203–205Google Scholar
  13. Radeau M, Bertelson P (1987) Auditory-visual interaction and the timing of inputs: Thomas (1941) revisited. Psychol Res 49:17–22PubMedGoogle Scholar
  14. Slutsky DA, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. Neuroreport 12:7–10PubMedGoogle Scholar
  15. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge MAGoogle Scholar
  16. Stein BE, Wallace MT (1996) Comparisons of cross-modality integration in midbrain and cortex. Prog Brain Res 112:289–299PubMedGoogle Scholar
  17. Stein BE, Laurienti PJ, Wallace MT, Stanford TR (2002) Multisensory integration. In: Ramachandran V (ed) Encyclopedia of the human brain. Elsevier, Amsterdam, pp 227–241Google Scholar
  18. Thomas G (1941) Experimental study of the influence of vision on sound localisation. J Exp Psychol 28:167–177Google Scholar
  19. Thurlow WR, Jack CE (1973) Certain determinants of the “ventriloquism effect”. Percept Mot Skills 36:1171–1184PubMedGoogle Scholar
  20. Wallace MT, Stein BE (1996) Sensory organization of the superior colliculus in cat and monkey. Prog Brain Res 112:301–311PubMedGoogle Scholar
  21. Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488PubMedGoogle Scholar
  22. Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246–1266PubMedGoogle Scholar
  23. Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010PubMedGoogle Scholar
  24. Warren DH, Welch RB, McCarthy TJ (1981) The role of visual-auditory compellingness in the ventriloquism effect: implications for transitivity among the spatial senses. Percept Psychophys 30:557–564PubMedGoogle Scholar
  25. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638–667PubMedGoogle Scholar

Copyright information

© Springer-Verlag 2004

Authors and Affiliations

  • M. T. Wallace
    • 1
    Email author
  • G. E. Roberson
    • 2
  • W. D. Hairston
    • 1
  • B. E. Stein
    • 1
  • J. W. Vaughan
    • 1
  • J. A. Schirillo
    • 2
  1. 1.Department of Neurobiology and AnatomyWake Forest University School of MedicineWinston-SalemUSA
  2. 2.Psychology DepartmentWake Forest UniversityWinston-SalemUSA

Personalised recommendations