Abstract
The relative spatiotemporal correspondence between sensory events affects multisensory integration across a variety of species; integration is maximal when stimuli in different sensory modalities are presented from approximately the same position at about the same time. In the present study, we investigated the influence of spatial and temporal factors on audio-visual simultaneity perception in humans. Participants made unspeeded simultaneous versus successive discrimination responses to pairs of auditory and visual stimuli presented at varying stimulus onset asynchronies from either the same or different spatial positions using either the method of constant stimuli (Experiments 1 and 2) or psychophysical staircases (Experiment 3). The participants in all three experiments were more likely to report the stimuli as being simultaneous when they originated from the same spatial position than when they came from different positions, demonstrating that the apparent perception of multisensory simultaneity is dependent on the relative spatial position from which stimuli are presented.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Allan, L. G. (1975). The relationship between judgments of successiveness and judgments of order.Perception & Psychophysics,18, 29–36.
Allen, P. G., &Kolers, P. A. (1981). Sensory specificity of apparent motion.Journal of Experimental Psychology: Human Perception & Performance,7, 1318–1326.
Bertelson, P. (1998). Starting from the ventriloquist: The perception of multimodal events. In M. Sabourin, F. I. M. Craik, & M. Robert (Eds.),Advances in psychological science: Vol. 2. Biological and cognitive aspects (pp. 419–439). Hove, U.K.: Psychology Press.
Bertelson, P. (1999). Ventriloquism: A case of cross-modal perceptual grouping. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 347–362). Amsterdam: Elsevier.
Bertelson, P., &Aschersleben, G. (1998). Automatic visual bias of perceived auditory location.Psychonomic Bulletin & Review,5, 482–489.
Bertelson, P., &Aschersleben, G. (2003). Temporal ventriloquism. Crossmodal interaction on the time dimension: 1. Evidence from time order judgments.International Journal of Psychophysiology,50, 147–155.
Bertelson, P., &de Gelder, B. (2004). The psychology of multimodal perception. In C. Spence & J. Driver (Eds.),Crossmodal space and crossmodal attention (pp. 141–177). Oxford: Oxford University Press.
Bertelson, P., Vroomen, J., & De Gelder, B. (1997). Auditory-visual interaction in voice localization and in bimodal speech recognition: The effects of desynchronization. In C. Benoit & gnR. Campbell (Eds.),Proceedings of the Workshop on Audio-Visual Speech Processing (pp. 97-100). Rhodes.
Bushara, K. O., Grafman, J., &Hallett, M. (2001). Neural correlates of auditory-visual stimulus onset asynchrony detection.Journal of Neuroscience,21, 300–304.
Caclin, A., Soto-Faraco, S., Kingstone, A., &Spence, C. (2002). Tactile “capture” of audition.Perception & Psychophysics,64, 616–630.
Calvert, G. A., Brammer, M. J., &Iversen, S. D. (1998). Crossmodal identification.Trends in Cognitive Sciences,2, 247–253.
Campbell, R., &Dodd, B. (1980). Hearing by eye.Quarterly Journal of Experimental Psychology,32, 85–99.
Choe, C. S., Welch, R. B., Gilford, R. M., &Juola, J. F. (1975). The “ventriloquist effect”: Visual dominance or response bias?Perception & Psychophysics,18, 55–60.
Dixon, N. N. F., &Spitz, L. (1980). The detection of auditory visual desynchrony.Perception,9, 719–721.
Driver, J., &Spence, C. (2000). Multisensory perception: Beyond modularity and convergence.Current Biology,10, R731-R735.
Efron, B., &Tibshirani, R. (1993).An introduction to the bootstrap. London: Chapman & Hall.
Engel, G. R., &Dougherty, W. G. (1971). Visual-auditory distance constancy.Nature,234, 308.
Erlebacher, A., &Sekuler, R. (1971). Response frequency equalization: A bias model for psychophysics.Perception & Psychophysics,9, 315–320.
Exner, S. (1875). Experimentelle Untersuchung der einfachsten psychischen Processe [Experimental study of the simplest psychological processes].Archiv für die Geschichte Physiologie [Pflüger’s Archive],11, 403–432.
Fraisse, P. (1964).The psychology of time. London: Eyre & Spottiswoode.
Girin, L., Schwartz, J. L., &Feng, G. (2001) Audiovisual enhancement of speech in noise.Journal of the Acoustical Society of America,109, 3007–3020.
Guinzberg, R. L. (1928). È possibile l’apprendimento di sensazioni eterogenee come perfettamente simultanee?[Is it possible to learn that heterogeneous sensations are perfectly simultaneous?] Archivi Italiani di Psicologia,6, 103–114.
Guski, R., &Troje, N. F. (2003). Audiovisual phenomenal causality.Perception & Psychophysics,65, 789–800.
Hirsh, I. J., &Fraisse, P. (1964). Simultanéité et succession de stimuli hétérogènes [Simultaneity and succession of heterogeneous stimuli].L’Année Psychologique,64, 1–19.
Hirsh, I. J., &Sherrick, C. E., Jr. (1961). Perceived order in different sense modalities.Journal of Experimental Psychology,62, 423–432.
ITU-T (1990).Television and sound transmission: Tolerances for transmission time differences between the vision and sound components of a television signal. International Telecommunication Union, Telecommunication standardization sector of ITU, Recommendation J.100.
Jack, C. E., &Thurlow, W. R. (1973). Effects of degree of visual association and angle of displacement on the “ventriloquism” effect.Perceptual & Motor Skills,37, 967–979.
Jaśkowski, P. (1999). Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 265–282). Amsterdam: Elsevier, North-Holland.
Jaśkowski, P., Jaroszyk, F., &Hojan-Jesierska, D. (1990). Temporalorder judgments and reaction time for stimuli of different modalities.Psychological Research,52, 35–38.
Jones, J. A., &Munhall, K. G. (1997). The effects of separating auditory and visual sources on audiovisual integration of speech.Canadian Acoustics,25, 13–19.
Kristofferson, A. B. (1967). Attention and psychophysical time.Acta Psychologica,27, 93–100.
Lewald, J., Ehrenstein, W. H., &Guski, R. (2001). Spatio-temporal constraints for auditory—visual integration.Behavioural Brain Research,121, 69–79.
Macaluso, E., George, N., Dolan, R., Spence, C., &Driver, J. (2004). Spatiotemporal contributions to audiovisual speech perception: A PET study.NeuroImage,21, 725–732.
Massaro, D. W., Cohen, M. M., &Smeele, P. M. (1996). Perception of asynchronous and conflicting visual and auditory speech.Journal of the Acoustical Society of America,100, 1777–1786.
McGrath, M., &Summerfield, Q. (1985). Intermodal timing relations and audio—visual speech recognition by normal-hearing adults.Journal of the Acoustical Society of America,77, 678–685.
Michotte, A. (1912). Nouvelles recherches sur la simultanéité apparente d’impressions disparates périodiques (expérience de complication) [New research on the apparent simultaneity of periodic heterogeneous stimuli (complication situation)].Annales de l’Institut Supérieur de Philosophie,1, 571–663.
Miller, E. A. (1972). Interactions of vision and touch in conflict and nonconflict form perception tasks.Journal of Experimental Psychology,96, 114–123.
Mollon, J. D., &Perkins, A. J. (1996). Errors of judgement at Greenwich in 1796.Nature,380, 101–102.
Morein-Zamir, S., Soto-Faraco, S., &Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism.Cognitive Brain Research,17, 154–163.
Munhall, K., &Vatikiotis-Bateson, E. (2004). Specialized spatiotemporal integration constraints of speech. In G. Calvert, C. Spence, & B. E. Stein (Eds.),The handbook of multisensory processing (pp. 177–188). Cambridge, MA: MIT Press.
Myung, I. J. (2003). Tutorial on maximum likelihood estimation.Journal of Mathematical Psychology,47, 90–100.
Navarra, J., Vatakis, A., Zampini, M., Soto-Faraco, S., Humphreys, W., & Spence, C. (2005).Exposure to asynchronous audiovisual speech increases the temporal window for audiovisual integration of non-speech stimuli. Manuscript submitted for publication.
Pandev, P. C., Kunov, H., &Abel, S. M. (1986). Disruptive effects of auditory signal delay on speech perception with lipreading.Journal of Auditory Research,26, 27–41.
Parducci, A., &Haugen, R. (1967). The frequency principle for comparative judgments.Perception & Psychophysics,2, 81–82.
Piéron, H. (1952).The sensations: Their functions, processes and mechanisms. London: Muller.
Radeau, M., &Bertelson, P. (1977). Adaptation to auditory—visual discordance and ventriloquism in semirealistic situations.Perception & Psychophysics,22, 137–146.
Radeau, M., &Bertelson, P. (1987). Auditory—visual interaction and the timing of inputs: Thomas (1941) revisited.Psychological Research,49, 17–22.
Raizada, R. D. S., &Poldrack, R. A. (2001). Event-related fMRI of audio—visual simultaneity perception.Journal of Cognitive Neuroscience,14 (Suppl.), 172.
Rao, S. M., Mayer, A. R., &Harrington, D. L. (2001). The evolution of brain activation during temporal processing.Nature Neuroscience,4, 317–323.
Rhodes, G. (1987). Auditory attention and the representation of spatial information.Perception & Psychophysics,42, 1–14.
Rihs, S. (1995). The influence of audio on perceived picture quality and subjective audio—visual delay tolerance. In R. Hamberg & H. de Ridder (Eds.),Proceedings of the MOSAIC workshop: Advanced methods for the evaluation of television picture quality (pp. 133-137). Eindhoven.
Scheier, C. R., Nijhawan, R., &Shimojo, S. (1999). Sound alters visual temporal resolution.Investigative Ophthalmology & Visual Science,40, S792.
Sekuler, R., &Erlebacher, A. (1971). The invalidity of “invalid results from the method of constant stimuli”: A common artifact in the methods of psychophysics.Perception & Psychophysics,9, 309–311.
Shore, D. I., Spry, E., &Spence, C. (2002). Confusing the mind by crossing the hands.Cognitive Brain Research,14, 153–163.
Shulman, G. L., Remington, R.W., &McLean, J. P. (1979). Moving attention through visual space.Journal of Experimental Psychology: Human Perception & Performance,5, 522–526.
Slutsky, D. A., &Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect.NeuroReport,12, 7–10.
Spence, C. (2001). Crossmodal attentional capture: A controversy resolved? In C. Folk & B. Gibson (Eds.),Attention, distraction and action: Multiple perspectives on attentional capture (pp. 231–262). Amsterdam: Elsevier.
Spence, C., Baddeley, R., Zampini, M., James, R., &Shore, D. I. (2003). Multisensory temporal order judgments: When two locations are better than one.Perception & Psychophysics,65, 318–328.
Spence, C., &Driver, J. (Eds.) (2004).Crossmodal space and crossmodal attention. Oxford: Oxford University Press.
Spence, C., Nicholls, M. E. R., &Driver, J. (2001). The cost of expecting events in the wrong sensory modality.Perception & Psychophysics,63, 330–336.
Spence, C., Shore, D. I., &Klein, R. M. (2001). Multisensory prior entry.Journal of Experimental Psychology: General,130, 799–832.
Spence, C., &Squire, S. B. (2003). Multisensory integration: Maintaining the perception of synchrony.Current Biology,13, R519-R521.
Stein, B. E., &Meredith, M. A. (1993).The merging of the senses. Cambridge, MA: MIT Press.
Stone, J.V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M., &Porter, N. R. (2001). When is now? Perception of simultaneity.Proceedings of the Royal Society of London: Series B,268, 31–38.
Strybel, T. Z., Manligas, C. L., Chan, O., &Perrott, D. R. (1990). A comparison of the effects of spatial separation on apparent motion in the auditory and visual modalities.Perception & Psychophysics,47, 439–448.
Sugita, Y., &Suzuki, Y. (2003). Implicit estimation of sound-arrival time.Nature,421, 911.
Taylor, M. M., &Creelman, C. D. (1967). PEST: Efficiency estimates on probability functions.Journal of the Acoustical Society of America,41, 782–787.
Van de Par, S., Kohlrausch, A., & Juola, J. F. (1999, November).Judged synchrony/asynchrony for light-tone pairs. Poster presented at the 40th annual meeting of the Psychonomic Society, Los Angeles.
Van de Par, S., Kohlrausch, A., & Juola, J. F. (2004).Synchrony judgments and temporal discrimination thresholds for audio—visual stimulus pairs. Manuscript submitted for publication.
Vroomen, J. (1999). Ventriloquism and the nature of the unity assumption. In G. Ashersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 389–393). Amsterdam: Elsevier.
Welch, R. B. (1999a). The advantages and limitations of the psychophysical staircase procedure in the study of intersensory bias: Commentary on Bertelson. In G. Ashersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 363–369). Amsterdam: Elsevier.
Welch, R. B. (1999b). Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 371–387). Amsterdam: Elsevier.
Welch, R. B., &Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy.Psychological Bulletin,3, 638–667.
Whipple, G. M., Sanford, E. C., &Colgrove, F. W. (1899). Minor studies from the psychological laboratory of Clark University: On nearly simultaneous clicks and flashes. The time required for recognition: Notes on mental standards of length.American Journal of Psychology,10, 280–286.
Zampini, M., Shore, D. I., &Spence, C. (2003a). Audiovisual temporal order judgments.Experimental Brain Research,152, 198–210.
Zampini, M., Shore, D. I., &Spence, C. (2003b). Multisensory temporal order judgments: The role of hemispheric redundancy.International Journal of Psychophysiology,50, 165–180.
Author information
Authors and Affiliations
Corresponding author
Additional information
C.S. and D.I.S. were funded by a Network grant from the McDonnell-Pew Centre for Cognitive Neuroscience, University of Oxford. D.I.S. was also funded by an operating grant from the Natural Science and Engineering Research Council of Canada. M.Z. was funded by a grant from the MIUR, Italy.
Rights and permissions
About this article
Cite this article
Zampini, M., Guest, S., Shore, D.I. et al. Audio-visual simultaneity judgments. Perception & Psychophysics 67, 531–544 (2005). https://doi.org/10.3758/BF03193329
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03193329