How are multiple, multi-sensory stimuli combined for use in segmenting spatio-temporal events? For an answer, we measured the effect of various auditory or visual stimuli, in isolation or in combination, on a bistable percept of visual motion (“bouncing” vs. “streaming”). To minimize individual differences, the physical properties of stimuli were adjusted to reflect individual subjects’ sensitivity to each cue in isolation. When put into combination, perceptual influences that had been equipotent in isolation were substantially altered. Specifically, auditory cues that had been strong when presented alone were greatly reduced in combination. Evaluation of alternative models of sensory integration showed that the state of the visual bistable percept could not be accounted for by probability summation among cues, as might occur at the level of decision processes. Instead, the state of the bistable percept was well predicted from a weighted sum of cues, with visual cues strongly dominating auditory cues. Finally, when cue weights were compared for individual subjects, it was found that subjects differ somewhat in the strategy they use for integrating multi-sensory information.
Root Mean Square Difference Psychometric Function Multisensory Integration Weibull Function Auditory Motion
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.
We thank Larry Abbott, Yuko Yotsumoto, Takeo Watanabe, Allison B. Sekuler, and Kristina Visscher for excellent suggestions. Feng Zhou is now at the Department of Psychological and Brain Sciences, the Johns Hopkins University. Victoria Wong was supported by an NSF IGERT fellowship, and by an undergraduate research grant; she is currently at the School of Medicine, University of Hawaii. Research supported by AFOSR grant F49620-03-1-0376 and National Institutes of Health grant MH-55687. e-mail: firstname.lastname@example.org.
Rui Y, Anandan P (2000) Segmenting visual actions based on spatio-temporal motion patterns. In: IEEE proceedings of conference on computer vision and pattern recognition. Hilton Head, p. 1111–1118Google Scholar
Rui Y, Gupta A, Acero A (2000) Automatically extracting highlights for TV baseball programs. In: Proceedings of ACM Multimedia. Los Angeles, p. 105–115Google Scholar
Sakurai K, Grove PM (2006) Auditory induced bounce perception when visual trajectories are inconsistent with motion reversal. In: European conference on visual perception. St. PetersburgGoogle Scholar
Sanabria D, Correa A, Lupianez J, Spence C (2004) Bouncing or streaming? Exploring the infuence of auditory cues on the interpretation of ambiguous visual motion. Exp Brain Res 157:537–541PubMedCrossRefGoogle Scholar
Sanabria D, Spence C, Soto-Faraco S (2007) Perceptual and decisional contributions to audiovisual interactions in the perception of apparent motion: a signal detection study. Cognition 102:299–310PubMedCrossRefGoogle Scholar
Scheier C, Lewkowicz DJ, Shimojo S (2003) Sound induces perceptual reorganization of an ambiguous motion display in human infants. Dev Sci 6:233–241CrossRefGoogle Scholar
Sekuler AB, Sekuler R (1999) Collisions between moving visual targets: what controls alternative ways of seeing an ambiguous display? Perception 28:415–432PubMedCrossRefGoogle Scholar
Shams L, Kamitani Y, Shimojo S (2002) Visual illusion induced by sound. Cogn Brain Res 14:147–152CrossRefGoogle Scholar
Shimojo S, Shams L (2001) Sensory modalities are not separate modalities: plasticity and interactions. Curr Opin Neurobiol 11:505–509PubMedCrossRefGoogle Scholar
Soto-Faraco S, Lyons J, Gazzaniga M, Spence C, Kingstone A (2002) The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. Cogn Brain Res 14:139–146CrossRefGoogle Scholar
Soto-Faraco S, Spence C, Kingstone A (2004a) Cross-modal dynamic capture: congruency effects in the perception of motion across sensory modalities. J Exp Psychol Hum Percept Perform 30:330–345CrossRefGoogle Scholar
Soto-Faraco S, Spence C, Lloyd D, Kingstone A (2004b) Moving multisensory research along: motion perception across sensory modalities. Curr Dir Psychol Sci 13:29–32CrossRefGoogle Scholar
Stanford T, Quessy S, Stein B (2005) Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25:6499–6508PubMedCrossRefGoogle Scholar