Experimental Brain Research

, Volume 152, Issue 2, pp 198–210 | Cite as

Audiovisual temporal order judgments

  • Massimiliano ZampiniEmail author
  • David I. Shore
  • Charles Spence
Research Article


In two experiments, we examined the extent to which audiovisual temporal order judgments (TOJs) were affected by spatial factors and by the dimension along which TOJs were made. Pairs of auditory and visual stimuli were presented from either the left and/or right of fixation at varying stimulus onset asynchronies (SOAs), and participants made unspeeded TOJs regarding either "Which modality was presented first?" (experiment 1), or "Which side was presented first?" (experiment 2). Modality TOJs were more accurate (i.e. just-noticeable differences, JNDs, were smaller) when the auditory and visual stimuli were presented from different spatial positions rather than from the same position, highlighting an important potential confound inherent in previous research. By contrast, spatial TOJs were unaffected by whether or not the two stimuli were presented in different modalities. A between-experiments comparison revealed more accurate performance (i.e. smaller JNDs) when people reported which modality came first than when they reported which side came first for identical bimodal stimulus pairs. These results demonstrate that multisensory TOJs are critically dependent on both the relative spatial position from which stimuli are presented and on the particular dimension being judged.


Multisensory perception Temporal order judgment Auditory Visual 



Charles Spence and David I. Shore were funded by a Network Grant from the McDonnell-Pew Centre for Cognitive Neuroscience, University of Oxford. David I. Shore was also funded by an operating grant from the Natural Science and Engineering Research Council of Canada.


  1. Allan LG (1975) The relationship between judgments of successiveness and judgments of order. Percept Psychophys 18:29–36Google Scholar
  2. Bald L, Berrien FK, Price JB, Sprague RO (1942) Errors in perceiving the temporal order of auditory and visual stimuli. J Appl Psychol 26:382–388Google Scholar
  3. Banich MT (1998) The missing link: the role of interhemispheric interaction in attentional processing. Brain Cogn 36:128–157CrossRefPubMedGoogle Scholar
  4. Bergenheim M, Johansson H, Granlund B, Pedersen J (1996) Experimental evidence for a sensory synchronization of sensory information to conscious experience. In: Hameroff SR, Kaszniak AW, Scott AC (eds) Toward a science of consciousness: the first Tucson discussions and debates. MIT Press, London, UK, pp 303–310Google Scholar
  5. Bertelson P (1998) Starting from the ventriloquist: the perception of multimodal events. In: Sabourin M, Craik FIM, Robert M (eds) Biological and cognitive aspects. (Advances in psychological science, Vol 2) Psychological, Hove, UK, pp 419–439Google Scholar
  6. Bertelson P, Gelder B de (2003) The psychology of multimodal perception. In: Spence C, Driver J (eds) Crossmodal space and crossmodal attention. Oxford University Press, Oxford, UKGoogle Scholar
  7. Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21:300–304PubMedGoogle Scholar
  8. Calvert GA, Brammer MJ, Iversen SD (1998) Crossmodal identification. Trends Cognit Sci 2:247–253CrossRefGoogle Scholar
  9. Dinnerstein AJ, Zlotogura P (1968) Intermodal perception of temporal order and motor skills: effects of age. Percept Mot Skill 26:987–1000Google Scholar
  10. Drew F (1896) Attention: experimental and critical. Am J Psychol 7:533–573Google Scholar
  11. Driver J, Spence C (1998) Crossmodal links in spatial attention. Philos Trans R Soc B 353:1319–1331Google Scholar
  12. Driver J, Spence C (2000) Multisensory perception: beyond modularity and convergence. Curr Biol 10:731–735PubMedGoogle Scholar
  13. Exner S (1875) Experimentelle Untersuchung der einfachsten psychischen Processe (Experimental study of the most simple psychological processes). Pflugers Arch 11:403–432Google Scholar
  14. Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophys 63:719–725PubMedGoogle Scholar
  15. Finger R, Davis AW (2001) Measuring video quality in videoconferencing systems. Technical Report SN187-D. Pixel Instrument Corporation, Los Gatos, CAGoogle Scholar
  16. Finney DJ (1964) Probit analysis: statistical treatment of the sigmoid response curve. Cambridge University Press, London, UKGoogle Scholar
  17. Friedman A, Polson MC (1981) Hemispheres as independent resource systems: limited-capacity processing and cerebral specialization. J Exp Psychol Hum Percept Perform 7:1031–1058CrossRefPubMedGoogle Scholar
  18. Gebhardt JW, Mowbray GH (1959) On discriminating the rate of visual flicker and auditory flutter. Am J Psychol 72:521–528Google Scholar
  19. Gengel RW, Hirsh IJ (1970) Temporal order: the effect of single versus repeated presentations, practice, and verbal feedback. Percept Psychophys 7:209–211Google Scholar
  20. Grossberg S, Grunewald A (1997) Cortical synchronization and perceptual framing. J Cognit Neurosci 9:117–132Google Scholar
  21. Hamlin AJ (1895) On the least observable interval between stimuli addressed to disparate senses and to different organs of the same sense. Am J Psychol 6:564–575Google Scholar
  22. Herdman CM, Friedman A (1985) Multiple resources in divided attention: a cross modal test of independence of hemispheric resources. J Exp Psychol Hum Percept Perform 11:40–49CrossRefPubMedGoogle Scholar
  23. Hirsh IJ (1959) Auditory perception of temporal order. J Acoust Soc Am 31:759–767Google Scholar
  24. Hirsh IJ, Sherrick CE Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432Google Scholar
  25. ITU-T (1990) Television and sound transmission: tolerances for transmission time differences between the vision and sound components of a television signal. Telecommunication standardization sector of ITU, Recommendation J100, CMTT 717 in CCIR Recommendations 12. International Telecommunication Union, Düsseldorf, GermanyGoogle Scholar
  26. Jaskowski P, Jaroszyk F, Hojan-Jerierska D (1990) Temporal-order judgments and reaction time for stimuli of different modalities. Psychol Res 52:35–38PubMedGoogle Scholar
  27. Kahneman D, Treisman A, Gibbs BJ (1992) The reviewing of object files: object-specific integration of information. Cognit Psychol 24:175–219PubMedGoogle Scholar
  28. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500PubMedGoogle Scholar
  29. Kinsbourne M, Cook J (1971) Generalized and lateralized effects of concurrent verbalization on a unimanual skill. Q J Exp Psychol 23:341–345Google Scholar
  30. Kinsbourne M, Hicks RE (1978) Functional cerebral space: a model for overflow, transfer and interference effects in human performance: a tutorial review. In: Requin J (ed) Attention and performance, VII. Erlbaum, Hillsdale, NJ, pp 345–362Google Scholar
  31. Kristofferson AB (1967) Attention and psychophysical time. Acta Psychol 27:93–100CrossRefGoogle Scholar
  32. Lewald J, Ehrenstein WH, Guski R (2001) Spatio-temporal constraints for auditory-visual integration. Behav Brain Res 121:69–79CrossRefPubMedGoogle Scholar
  33. McFarland DJ, Cacace AT, Setzen G (1998) Temporal-order discrimination for selected auditory and visual stimulus dimensions. J Speech, Lang Hearing Res 41:300–314Google Scholar
  34. McGrath M, Summerfield Q (1985) Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. J Acoust Soc Am 77:678–685PubMedGoogle Scholar
  35. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391PubMedGoogle Scholar
  36. Mitrani L, Shekerdjiski S, Yakimoff N (1986) Mechanisms and asymmetries in visual perception of simultaneity and temporal order. Biol Cybern 54:159–165PubMedGoogle Scholar
  37. Morein-Zamir S, Soto-Faraco S, Kingstone AF (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17:154–163Google Scholar
  38. Mortlock AN, Machin D, McConnell S, Sheppard P (1997) Virtual conferencing. BT Technol J 15:120–129CrossRefGoogle Scholar
  39. Moutoussis K, Zeki S (1997) A direct demonstration of perceptual asynchrony in vision. Proc R Soc Lond B Biol 264:393–399CrossRefGoogle Scholar
  40. Neumann O, Niepel M (2003) Timing of "perception" and perception of "time". In: Kaernbach C, Schröger E, Müller H (eds) Psychophysics beyond sensation: laws and invariants of human cognition. (Scientific Psychology Series) Erlbaum, Hove, UKGoogle Scholar
  41. Pandev PC, Kunov H, Abel SM (1986) Disruptive effects of auditory signal delay on speech perception with lip-reading. J Aud Res 26:27–41PubMedGoogle Scholar
  42. Pöppel E, Schill K, Steinbüchel N von (1990) Sensory integration within temporally neutral system states: a hypothesis. Naturwissenschaften 77:89–91PubMedGoogle Scholar
  43. Posner MI, Nissen MJ, Klein RM (1976) Visual dominance: an information-processing account of its origins and significance. Psychol Rev 83:157–171PubMedGoogle Scholar
  44. Rains JD (1963) Signal luminance and position effects in human reaction time. Vision Res 3:239–251CrossRefGoogle Scholar
  45. Raizada RDS, Poldrack RA (2001) Event-related fMRI of audio-visual simultaneity perception. Soc Neurosci Abstr 27:511.14Google Scholar
  46. Rao SM, Mayer AR Harrington DL (2001) The evolution of brain activation during temporal processing. Nat Neurosci 4:317–323CrossRefPubMedGoogle Scholar
  47. Reeves B, Voelker D (1993) Effects of audio-video asynchrony on viewer's memory, evaluation of content and detection ability. Research report. Pixel Instruments, Los Gatos, CAGoogle Scholar
  48. Regan D (1982) Visual information channeling in normal and disordered vision. Psychol Rev 89:407–444CrossRefPubMedGoogle Scholar
  49. Regan D, Spekreijse H (1977) Auditory-visual interactions and the correspondence between perceived auditory space and perceived visual space. Perception 6:133–138PubMedGoogle Scholar
  50. Rihs S (1995) The influence of audio on perceived picture quality and subjective audio-visual delay tolerance. In: Hamberg R, Ridder H de (eds) Report 1071. Proc MOSAIC workshop, Eindhoven, 18–19 Sept: Advanced methods for the evaluation of television picture quality. Inst Percept Res, pp 133–137Google Scholar
  51. Roberson GE, Hairston WD, Wallace MT, Stein BE, Laurienti PJ, Schirillo JA (2001) Unifying multisensory signals across time and space. Soc Neurosci Abstr 27:511.19Google Scholar
  52. Rutschmann J, Link R (1964) Perception of temporal order of stimuli differing in sense mode and simple reaction time. Percept Mot Skills 18:345–352Google Scholar
  53. Sade A, Spitzer H (1998) The effects of attentional spread and attentional effort on orientation discrimination. Spatial Vision 11:367–383PubMedGoogle Scholar
  54. Scheier CR, Nijhawan R, Shimojo S (1999) Sound alters visual temporal resolution. Invest Ophthalmol Vis Sci 40:S792Google Scholar
  55. Shore DI, Spence C, Klein RM (2001) Visual prior entry. Psychol Sci 12:205–212CrossRefPubMedGoogle Scholar
  56. Shore DI, Spry E, Spence C (2002) Confusing the mind by crossing the hands. Cogn Brain Res 14:153–163CrossRefGoogle Scholar
  57. Slutsky DA, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. Neuroreport 12:7–10PubMedGoogle Scholar
  58. Smith WF (1933) The relative quickness of visual and auditory perception. J Exp Psychol 16:239–257Google Scholar
  59. Spence C (2001) Crossmodal attentional capture: a controversy resolved? In: Folk C, Gibson B (eds) Attention, distraction and action: multiple perspectives on attentional capture. Elsevier Science, Amsterdam, pp 231–262Google Scholar
  60. Spence C, Driver J (1999) A new approach to the design of multimodal warning signals. In: Harris D (ed) Job design, product design and human-computer interaction. (Engineering psychology and cognitive ergonomics, Vol 4) Ashgate: Hampshire, pp 455-461Google Scholar
  61. Spence C, Squire SB (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13:R519–R521Google Scholar
  62. Spence C, Nicholls MER, Driver J (2001a) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63:330–336PubMedGoogle Scholar
  63. Spence C, Shore DI, Klein RM (2001b) Multisensory prior entry. J Exp Psychol Gen 130:799–832PubMedGoogle Scholar
  64. Spence C, Baddeley R, Zampini M, James R, Shore DI (2003) Crossmodal temporal order judgments: when two locations are better than one. Percept PsychophysGoogle Scholar
  65. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MAGoogle Scholar
  66. Sternberg S, Knoll RL, Gates BA (1971) Prior entry reexamined: effect of attentional bias on order perception. Presented, meeting Psychon Soc, November, St. Louis, MOGoogle Scholar
  67. Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M, Port M, Porter NR (2001) When is now? Perception of simultaneity. Proc R Soc B 268:31–38CrossRefGoogle Scholar
  68. Stone SA (1926) Prior entry in the auditory-tactual complication. Am J Psychol 37:284–287Google Scholar
  69. Sugita Y, Suzuki Y (2003) Implicit estimation of sound-arrival time. Nature 421:911CrossRefGoogle Scholar
  70. Teatini G, Farnè M, Verzella F, Berruecos P Jr (1976) Perception of temporal order: visual and auditory stimuli. G Ital Psicol 3:157–164Google Scholar
  71. Ulrich R (1987) Threshold models of temporal-order judgments evaluated by a ternary response task. Percept Psychophys 42:224–239PubMedGoogle Scholar
  72. Urbach D, Spitzer H (1995) Attentional effort modulated by task difficulty. Vision Res 35:2169–2177CrossRefPubMedGoogle Scholar
  73. Van de Par S, Juola JF, Kohlrausch A (1999) Judged synchrony/asynchrony for light-tone pairs. Presented 40th Ann meeting Psychon Soc, Los Angeles, CAGoogle Scholar
  74. Welch RB, DuttonHurt LD, Warren DH (1986) Contributions of audition and vision to temporal rate perception. Percept Psychophys 39:294–300Google Scholar
  75. Whipple GM, Sanford EC, Colgrove FW (1899) Minor studies from the psychological laboratory of Clark University: on nearly simultaneous clicks and flashes: the time required for recognition: notes on mental standards of length. Am J Psychol 10:280–295Google Scholar
  76. Zampini M, Shore DI, Spence C (2003) Multisensory temporal order judgments: the role of hemispheric redundancy. Int J Psychophysiol (in press)Google Scholar

Copyright information

© Springer-Verlag 2003

Authors and Affiliations

  • Massimiliano Zampini
    • 1
    • 2
    Email author
  • David I. Shore
    • 3
  • Charles Spence
    • 1
  1. 1.Department of Experimental PsychologyUniversity of OxfordOxfordUK
  2. 2.University of VeronaItaly
  3. 3.Department of PsychologyMcMaster UniversityHamiltonCanada

Personalised recommendations