Perception & Psychophysics

, Volume 67, Issue 3, pp 531–544 | Cite as

Audio-visual simultaneity judgments

  • Massimiliano Zampini
  • Steve Guest
  • David I. Shore
  • Charles Spence
Article
  • 864 Downloads

Abstract

The relative spatiotemporal correspondence between sensory events affects multisensory integration across a variety of species; integration is maximal when stimuli in different sensory modalities are presented from approximately the same position at about the same time. In the present study, we investigated the influence of spatial and temporal factors on audio-visual simultaneity perception in humans. Participants made unspeeded simultaneous versus successive discrimination responses to pairs of auditory and visual stimuli presented at varying stimulus onset asynchronies from either the same or different spatial positions using either the method of constant stimuli (Experiments 1 and 2) or psychophysical staircases (Experiment 3). The participants in all three experiments were more likely to report the stimuli as being simultaneous when they originated from the same spatial position than when they came from different positions, demonstrating that the apparent perception of multisensory simultaneity is dependent on the relative spatial position from which stimuli are presented.

References

  1. Allan, L. G. (1975). The relationship between judgments of successiveness and judgments of order.Perception & Psychophysics,18, 29–36.CrossRefGoogle Scholar
  2. Allen, P. G., &Kolers, P. A. (1981). Sensory specificity of apparent motion.Journal of Experimental Psychology: Human Perception & Performance,7, 1318–1326.CrossRefGoogle Scholar
  3. Bertelson, P. (1998). Starting from the ventriloquist: The perception of multimodal events. In M. Sabourin, F. I. M. Craik, & M. Robert (Eds.),Advances in psychological science: Vol. 2. Biological and cognitive aspects (pp. 419–439). Hove, U.K.: Psychology Press.Google Scholar
  4. Bertelson, P. (1999). Ventriloquism: A case of cross-modal perceptual grouping. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 347–362). Amsterdam: Elsevier.CrossRefGoogle Scholar
  5. Bertelson, P., &Aschersleben, G. (1998). Automatic visual bias of perceived auditory location.Psychonomic Bulletin & Review,5, 482–489.Google Scholar
  6. Bertelson, P., &Aschersleben, G. (2003). Temporal ventriloquism. Crossmodal interaction on the time dimension: 1. Evidence from time order judgments.International Journal of Psychophysiology,50, 147–155.PubMedCrossRefGoogle Scholar
  7. Bertelson, P., &de Gelder, B. (2004). The psychology of multimodal perception. In C. Spence & J. Driver (Eds.),Crossmodal space and crossmodal attention (pp. 141–177). Oxford: Oxford University Press.Google Scholar
  8. Bertelson, P., Vroomen, J., & De Gelder, B. (1997). Auditory-visual interaction in voice localization and in bimodal speech recognition: The effects of desynchronization. In C. Benoit & gnR. Campbell (Eds.),Proceedings of the Workshop on Audio-Visual Speech Processing (pp. 97-100). Rhodes.Google Scholar
  9. Bushara, K. O., Grafman, J., &Hallett, M. (2001). Neural correlates of auditory-visual stimulus onset asynchrony detection.Journal of Neuroscience,21, 300–304.PubMedGoogle Scholar
  10. Caclin, A., Soto-Faraco, S., Kingstone, A., &Spence, C. (2002). Tactile “capture” of audition.Perception & Psychophysics,64, 616–630.CrossRefGoogle Scholar
  11. Calvert, G. A., Brammer, M. J., &Iversen, S. D. (1998). Crossmodal identification.Trends in Cognitive Sciences,2, 247–253.CrossRefGoogle Scholar
  12. Campbell, R., &Dodd, B. (1980). Hearing by eye.Quarterly Journal of Experimental Psychology,32, 85–99.PubMedCrossRefGoogle Scholar
  13. Choe, C. S., Welch, R. B., Gilford, R. M., &Juola, J. F. (1975). The “ventriloquist effect”: Visual dominance or response bias?Perception & Psychophysics,18, 55–60.CrossRefGoogle Scholar
  14. Dixon, N. N. F., &Spitz, L. (1980). The detection of auditory visual desynchrony.Perception,9, 719–721.PubMedCrossRefGoogle Scholar
  15. Driver, J., &Spence, C. (2000). Multisensory perception: Beyond modularity and convergence.Current Biology,10, R731-R735.PubMedCrossRefGoogle Scholar
  16. Efron, B., &Tibshirani, R. (1993).An introduction to the bootstrap. London: Chapman & Hall.Google Scholar
  17. Engel, G. R., &Dougherty, W. G. (1971). Visual-auditory distance constancy.Nature,234, 308.PubMedCrossRefGoogle Scholar
  18. Erlebacher, A., &Sekuler, R. (1971). Response frequency equalization: A bias model for psychophysics.Perception & Psychophysics,9, 315–320.Google Scholar
  19. Exner, S. (1875). Experimentelle Untersuchung der einfachsten psychischen Processe [Experimental study of the simplest psychological processes].Archiv für die Geschichte Physiologie [Pflüger’s Archive],11, 403–432.CrossRefGoogle Scholar
  20. Fraisse, P. (1964).The psychology of time. London: Eyre & Spottiswoode.Google Scholar
  21. Girin, L., Schwartz, J. L., &Feng, G. (2001) Audiovisual enhancement of speech in noise.Journal of the Acoustical Society of America,109, 3007–3020.PubMedCrossRefGoogle Scholar
  22. Guinzberg, R. L. (1928). È possibile l’apprendimento di sensazioni eterogenee come perfettamente simultanee?[Is it possible to learn that heterogeneous sensations are perfectly simultaneous?] Archivi Italiani di Psicologia,6, 103–114.Google Scholar
  23. Guski, R., &Troje, N. F. (2003). Audiovisual phenomenal causality.Perception & Psychophysics,65, 789–800.CrossRefGoogle Scholar
  24. Hirsh, I. J., &Fraisse, P. (1964). Simultanéité et succession de stimuli hétérogènes [Simultaneity and succession of heterogeneous stimuli].L’Année Psychologique,64, 1–19.PubMedCrossRefGoogle Scholar
  25. Hirsh, I. J., &Sherrick, C. E., Jr. (1961). Perceived order in different sense modalities.Journal of Experimental Psychology,62, 423–432.PubMedCrossRefGoogle Scholar
  26. ITU-T (1990).Television and sound transmission: Tolerances for transmission time differences between the vision and sound components of a television signal. International Telecommunication Union, Telecommunication standardization sector of ITU, Recommendation J.100.Google Scholar
  27. Jack, C. E., &Thurlow, W. R. (1973). Effects of degree of visual association and angle of displacement on the “ventriloquism” effect.Perceptual & Motor Skills,37, 967–979.Google Scholar
  28. Jaśkowski, P. (1999). Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 265–282). Amsterdam: Elsevier, North-Holland.CrossRefGoogle Scholar
  29. Jaśkowski, P., Jaroszyk, F., &Hojan-Jesierska, D. (1990). Temporalorder judgments and reaction time for stimuli of different modalities.Psychological Research,52, 35–38.PubMedCrossRefGoogle Scholar
  30. Jones, J. A., &Munhall, K. G. (1997). The effects of separating auditory and visual sources on audiovisual integration of speech.Canadian Acoustics,25, 13–19.Google Scholar
  31. Kristofferson, A. B. (1967). Attention and psychophysical time.Acta Psychologica,27, 93–100.PubMedCrossRefGoogle Scholar
  32. Lewald, J., Ehrenstein, W. H., &Guski, R. (2001). Spatio-temporal constraints for auditory—visual integration.Behavioural Brain Research,121, 69–79.PubMedCrossRefGoogle Scholar
  33. Macaluso, E., George, N., Dolan, R., Spence, C., &Driver, J. (2004). Spatiotemporal contributions to audiovisual speech perception: A PET study.NeuroImage,21, 725–732.PubMedCrossRefGoogle Scholar
  34. Massaro, D. W., Cohen, M. M., &Smeele, P. M. (1996). Perception of asynchronous and conflicting visual and auditory speech.Journal of the Acoustical Society of America,100, 1777–1786.PubMedCrossRefGoogle Scholar
  35. McGrath, M., &Summerfield, Q. (1985). Intermodal timing relations and audio—visual speech recognition by normal-hearing adults.Journal of the Acoustical Society of America,77, 678–685.PubMedCrossRefGoogle Scholar
  36. Michotte, A. (1912). Nouvelles recherches sur la simultanéité apparente d’impressions disparates périodiques (expérience de complication) [New research on the apparent simultaneity of periodic heterogeneous stimuli (complication situation)].Annales de l’Institut Supérieur de Philosophie,1, 571–663.Google Scholar
  37. Miller, E. A. (1972). Interactions of vision and touch in conflict and nonconflict form perception tasks.Journal of Experimental Psychology,96, 114–123.PubMedCrossRefGoogle Scholar
  38. Mollon, J. D., &Perkins, A. J. (1996). Errors of judgement at Greenwich in 1796.Nature,380, 101–102.PubMedCrossRefGoogle Scholar
  39. Morein-Zamir, S., Soto-Faraco, S., &Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism.Cognitive Brain Research,17, 154–163.PubMedCrossRefGoogle Scholar
  40. Munhall, K., &Vatikiotis-Bateson, E. (2004). Specialized spatiotemporal integration constraints of speech. In G. Calvert, C. Spence, & B. E. Stein (Eds.),The handbook of multisensory processing (pp. 177–188). Cambridge, MA: MIT Press.Google Scholar
  41. Myung, I. J. (2003). Tutorial on maximum likelihood estimation.Journal of Mathematical Psychology,47, 90–100.CrossRefGoogle Scholar
  42. Navarra, J., Vatakis, A., Zampini, M., Soto-Faraco, S., Humphreys, W., & Spence, C. (2005).Exposure to asynchronous audiovisual speech increases the temporal window for audiovisual integration of non-speech stimuli. Manuscript submitted for publication.Google Scholar
  43. Pandev, P. C., Kunov, H., &Abel, S. M. (1986). Disruptive effects of auditory signal delay on speech perception with lipreading.Journal of Auditory Research,26, 27–41.Google Scholar
  44. Parducci, A., &Haugen, R. (1967). The frequency principle for comparative judgments.Perception & Psychophysics,2, 81–82.Google Scholar
  45. Piéron, H. (1952).The sensations: Their functions, processes and mechanisms. London: Muller.Google Scholar
  46. Radeau, M., &Bertelson, P. (1977). Adaptation to auditory—visual discordance and ventriloquism in semirealistic situations.Perception & Psychophysics,22, 137–146.CrossRefGoogle Scholar
  47. Radeau, M., &Bertelson, P. (1987). Auditory—visual interaction and the timing of inputs: Thomas (1941) revisited.Psychological Research,49, 17–22.PubMedCrossRefGoogle Scholar
  48. Raizada, R. D. S., &Poldrack, R. A. (2001). Event-related fMRI of audio—visual simultaneity perception.Journal of Cognitive Neuroscience,14 (Suppl.), 172.Google Scholar
  49. Rao, S. M., Mayer, A. R., &Harrington, D. L. (2001). The evolution of brain activation during temporal processing.Nature Neuroscience,4, 317–323.PubMedCrossRefGoogle Scholar
  50. Rhodes, G. (1987). Auditory attention and the representation of spatial information.Perception & Psychophysics,42, 1–14.Google Scholar
  51. Rihs, S. (1995). The influence of audio on perceived picture quality and subjective audio—visual delay tolerance. In R. Hamberg & H. de Ridder (Eds.),Proceedings of the MOSAIC workshop: Advanced methods for the evaluation of television picture quality (pp. 133-137). Eindhoven.Google Scholar
  52. Scheier, C. R., Nijhawan, R., &Shimojo, S. (1999). Sound alters visual temporal resolution.Investigative Ophthalmology & Visual Science,40, S792.Google Scholar
  53. Sekuler, R., &Erlebacher, A. (1971). The invalidity of “invalid results from the method of constant stimuli”: A common artifact in the methods of psychophysics.Perception & Psychophysics,9, 309–311.Google Scholar
  54. Shore, D. I., Spry, E., &Spence, C. (2002). Confusing the mind by crossing the hands.Cognitive Brain Research,14, 153–163.PubMedCrossRefGoogle Scholar
  55. Shulman, G. L., Remington, R.W., &McLean, J. P. (1979). Moving attention through visual space.Journal of Experimental Psychology: Human Perception & Performance,5, 522–526.CrossRefGoogle Scholar
  56. Slutsky, D. A., &Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect.NeuroReport,12, 7–10.PubMedCrossRefGoogle Scholar
  57. Spence, C. (2001). Crossmodal attentional capture: A controversy resolved? In C. Folk & B. Gibson (Eds.),Attention, distraction and action: Multiple perspectives on attentional capture (pp. 231–262). Amsterdam: Elsevier.CrossRefGoogle Scholar
  58. Spence, C., Baddeley, R., Zampini, M., James, R., &Shore, D. I. (2003). Multisensory temporal order judgments: When two locations are better than one.Perception & Psychophysics,65, 318–328.CrossRefGoogle Scholar
  59. Spence, C., &Driver, J. (Eds.) (2004).Crossmodal space and crossmodal attention. Oxford: Oxford University Press.Google Scholar
  60. Spence, C., Nicholls, M. E. R., &Driver, J. (2001). The cost of expecting events in the wrong sensory modality.Perception & Psychophysics,63, 330–336.Google Scholar
  61. Spence, C., Shore, D. I., &Klein, R. M. (2001). Multisensory prior entry.Journal of Experimental Psychology: General,130, 799–832.CrossRefGoogle Scholar
  62. Spence, C., &Squire, S. B. (2003). Multisensory integration: Maintaining the perception of synchrony.Current Biology,13, R519-R521.PubMedCrossRefGoogle Scholar
  63. Stein, B. E., &Meredith, M. A. (1993).The merging of the senses. Cambridge, MA: MIT Press.Google Scholar
  64. Stone, J.V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M., &Porter, N. R. (2001). When is now? Perception of simultaneity.Proceedings of the Royal Society of London: Series B,268, 31–38.CrossRefGoogle Scholar
  65. Strybel, T. Z., Manligas, C. L., Chan, O., &Perrott, D. R. (1990). A comparison of the effects of spatial separation on apparent motion in the auditory and visual modalities.Perception & Psychophysics,47, 439–448.Google Scholar
  66. Sugita, Y., &Suzuki, Y. (2003). Implicit estimation of sound-arrival time.Nature,421, 911.PubMedCrossRefGoogle Scholar
  67. Taylor, M. M., &Creelman, C. D. (1967). PEST: Efficiency estimates on probability functions.Journal of the Acoustical Society of America,41, 782–787.CrossRefGoogle Scholar
  68. Van de Par, S., Kohlrausch, A., & Juola, J. F. (1999, November).Judged synchrony/asynchrony for light-tone pairs. Poster presented at the 40th annual meeting of the Psychonomic Society, Los Angeles.Google Scholar
  69. Van de Par, S., Kohlrausch, A., & Juola, J. F. (2004).Synchrony judgments and temporal discrimination thresholds for audio—visual stimulus pairs. Manuscript submitted for publication.Google Scholar
  70. Vroomen, J. (1999). Ventriloquism and the nature of the unity assumption. In G. Ashersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 389–393). Amsterdam: Elsevier.CrossRefGoogle Scholar
  71. Welch, R. B. (1999a). The advantages and limitations of the psychophysical staircase procedure in the study of intersensory bias: Commentary on Bertelson. In G. Ashersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 363–369). Amsterdam: Elsevier.CrossRefGoogle Scholar
  72. Welch, R. B. (1999b). Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.),Cognitive contributions to the perception of spatial and temporal events (pp. 371–387). Amsterdam: Elsevier.CrossRefGoogle Scholar
  73. Welch, R. B., &Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy.Psychological Bulletin,3, 638–667.CrossRefGoogle Scholar
  74. Whipple, G. M., Sanford, E. C., &Colgrove, F. W. (1899). Minor studies from the psychological laboratory of Clark University: On nearly simultaneous clicks and flashes. The time required for recognition: Notes on mental standards of length.American Journal of Psychology,10, 280–286.CrossRefGoogle Scholar
  75. Zampini, M., Shore, D. I., &Spence, C. (2003a). Audiovisual temporal order judgments.Experimental Brain Research,152, 198–210.CrossRefGoogle Scholar
  76. Zampini, M., Shore, D. I., &Spence, C. (2003b). Multisensory temporal order judgments: The role of hemispheric redundancy.International Journal of Psychophysiology,50, 165–180.PubMedCrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2005

Authors and Affiliations

  • Massimiliano Zampini
    • 1
    • 2
  • Steve Guest
    • 1
  • David I. Shore
    • 3
  • Charles Spence
    • 1
  1. 1.University of OxfordOxfordEngland
  2. 2.Department of Cognitive Sciences and EducationUniversity of TrentoRoveretoItaly
  3. 3.McMaster UniversityHamiltonCanada

Personalised recommendations