Experimental Brain Research

, Volume 185, Issue 3, pp 521–529 | Cite as

Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments

  • Argiro Vatakis
  • Jordi Navarra
  • Salvador Soto-Faraco
  • Charles Spence
Research Article

Abstract

The temporal perception of simple auditory and visual stimuli can be modulated by exposure to asynchronous audiovisual speech. For instance, research using the temporal order judgment (TOJ) task has shown that exposure to temporally misaligned audiovisual speech signals can induce temporal adaptation that will influence the TOJs of other (simpler) audiovisual events (Navarra et al. (2005) Cognit Brain Res 25:499–507). Given that TOJ and simultaneity judgment (SJ) tasks appear to reflect different underlying mechanisms, we investigated whether adaptation to asynchronous speech inputs would also influence SJ task performance. Participants judged whether a light flash and a noise burst, presented at varying stimulus onset asynchronies, were simultaneous or not, or else they discriminated which of the two sensory events appeared to have occurred first. While performing these tasks, participants monitored a continuous speech stream for target words that were either presented in synchrony, or with the audio channel lagging 300 ms behind the video channel. We found that the sensitivity of participant’s TOJ and SJ responses was reduced when the background speech stream was desynchronized. A significant modulation of the point of subjective simultaneity (PSS) was also observed in the SJ task but, interestingly, not in the TOJ task, thus supporting previous claims that TOJ and SJ tasks may tap somewhat different aspects of temporal perception.

Keywords

Simultaneity judgment Temporal order judgment Speech Temporal adaptation Audition Vision 

Notes

Acknowledgments

A.V. was supported by a Newton Abraham Studentship from the Medical Sciences Division, University of Oxford. S.S.-F., J.N., and C.S. were supported by grants from the McDowell-Pew Foundation, University of Oxford and the Ministerio de Educación y Ciencia from Spain (TIN2004-04363-C03-02).

References

  1. Alsius A, Navarra J, Campbell R, Soto-Faraco S (2005) Audiovisual integration of speech falters under high attention demands. Curr Biol 15:1–5CrossRefGoogle Scholar
  2. Axelrod S, Thompson LW, Cohen LD (1968) Effects of senescence on the temporal resolution of somesthetic stimuli presented to one hand or both. J Gerontol 23:191–195PubMedGoogle Scholar
  3. Baron J (1969) Temporal ROC curves and the psychological moment. Psychon Sci 15:299–300Google Scholar
  4. Coren S, Ward LM, Enns JT (2004) Sensation and perception, 6th edn. Harcourt Brace, Fort WorthGoogle Scholar
  5. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9:719–721PubMedCrossRefGoogle Scholar
  6. Durgin FH, Sternberg S (2002) The time of consciousness and vice versa. Conscious Cogn 11:248–290Google Scholar
  7. Finney DJ (1964) Probit analysis: statistical treatment of the sigmoid response curve. Cambridge University Press, LondonGoogle Scholar
  8. Frey RD (1990) Selective attention, event perception and the criterion of acceptability principle: evidence supporting and rejecting the doctrine of prior entry. Hum Mov Sci 9:481–530CrossRefGoogle Scholar
  9. Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7:773–778PubMedCrossRefGoogle Scholar
  10. Grant KW, Seitz PF (1998) The use of visible speech cues (speechreading) for directing auditory attention: reducing temporal and spectral uncertainty in auditory detection of spoken sentences. In: Kuhl PK, Crum LA (Eds) Proceedings of the 16th international congress on acoustics and the 135th meeting of the acoustical society of America, vol 3. ASA, New York, pp 2335–2336Google Scholar
  11. Hirsh IJ (1959) Auditory perception of temporal order. J Acoust Soc Am 31:759–767CrossRefGoogle Scholar
  12. Hirsh IJ, Fraisse P (1964) Simultanéité et succession de stimuli hétérogènes [Simultaneity and succession of heterogeneous stimuli]. L’Année Psychologique 64:1–19CrossRefGoogle Scholar
  13. Hirsh IJ, Sherrick Jr CE (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432PubMedCrossRefGoogle Scholar
  14. Jaskowski P (1991) Two-stage modal for order discrimination. Percept Psychophys 50:76–82PubMedGoogle Scholar
  15. Juola JF, van Eijk RLJ, Kohlrausch A, van der Par S (2007) Synchrony detection and temporal order judgments for bimodal stimuli. Poster presented at the joint meeting of the Experimental Psychology Society and The Psychonomics Society. Edinburgh, 4–7 JulyGoogle Scholar
  16. Kohlrausch A, van de Par S (2005) Audio–visual interactions in the context of multi-media applications. In: Blauert J (Ed) Communication acoustics. Springer, Berlin, pp 109–138CrossRefGoogle Scholar
  17. Lavie N (2005) Distracted and confused? Selective attention under load. Trends Cognit Sci 9:75–82CrossRefGoogle Scholar
  18. Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech perception: a PET study. Neuroimage 21:725–732PubMedCrossRefGoogle Scholar
  19. Mitrani L, Shekerdjiiski S, Yakimoff N (1986) Mechanisms and asymmetries in visual perception of simultaneity and temporal order. Biol Cybern 54:159–165PubMedCrossRefGoogle Scholar
  20. Myung IJ (2003) Tutorial on maximum likelihood estimation. J of Math Psychol 47:90–100CrossRefGoogle Scholar
  21. Navarra J, Vatakis A, Zampini M, Humphreys W, Soto-Faraco S, Spence C (2005) Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognit Brain Res 25:499–507CrossRefGoogle Scholar
  22. Navarra J, Soto-Faraco S, Spence C (2007) Adaptation to audiotactile asynchrony. Neurosci Lett 413:72–76PubMedCrossRefGoogle Scholar
  23. Schneider KA, Bavelier D (2003) Components of visual prior entry. Cognit Psychol 47:333–366PubMedCrossRefGoogle Scholar
  24. Shore DI, Spence C, Klein RM (2001) Visual prior entry. Psychol Sci 12:205–212PubMedCrossRefGoogle Scholar
  25. Shore DI, Spry E, Spence C (2002) Confusing the mind by crossing the hands. Cognit Brain Res 14:153–163CrossRefGoogle Scholar
  26. Shore DI, Gray K, Spry E, Spence C (2005) Spatial modulation of tactile temporal-order judgments. Perception 34:1251–1262PubMedCrossRefGoogle Scholar
  27. Soto-Farco S, Alsius A (2007) Conscious access to the unisensory components of a cross-modal illusion. Neuroreport 18:347–350CrossRefGoogle Scholar
  28. Spence C (2007) Audiovisual multisensory integration. J Acoust Soc Japan Acoust Sci Technol 28:61–70Google Scholar
  29. Spence C, Shore DI, Klein RM (2001) Multisensory prior entry. J Exp Psychol Gen 130:799–832PubMedCrossRefGoogle Scholar
  30. Stelmach LB, Herdman CM (1991) Directed attention and perception of temporal order. J Exp Psychol Human Percept Perform 17:539–550CrossRefGoogle Scholar
  31. Tuomainen J, Andersen TS, Tiippana K, Sams M (2005) Audio-visual speech perception is special. Cognition 96:B13–B22PubMedCrossRefGoogle Scholar
  32. Van de Par S, Kohlrausch A, Juola JF (2002) Some methodological aspects for measuring asynchrony detection in audio–visual stimuli. In: Calvo Manzano A, Perez Lopez A, Santiago JS (eds) Proceedings of the Forum acusticum, Sevilla, Spain, 16–20 September 2002 (on CD-rom). Forum Acusticum, Sevilla, SpainGoogle Scholar
  33. Van Eijk PLJ, Kohlrausch A, van de Par S, Juola JF (2006) The influence of psychophysical procedure and stimulus type on estimates of human performance in detecting audio–visual asynchrony. Poster presented at the 7th Annual Meeting of the International Multisensory Research Forum, Dublin, IrelandGoogle Scholar
  34. Van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186PubMedCrossRefGoogle Scholar
  35. Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111:134–142PubMedCrossRefGoogle Scholar
  36. Vatakis A, Spence C (2007) Crossmodal binding: evaluating the ‘unity assumption’ using audiovisual speech stimuli. Percept Psychophys 69:744–756PubMedGoogle Scholar
  37. Vatakis A, Navarra J, Soto-Faraco S, Spence C (2007) Temporal recalibration during asynchronous audiovisual speech perception. Exp Brain Res 181:173–181PubMedCrossRefGoogle Scholar
  38. Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio–visual asynchrony. Cognit Brain Res 22:32–35CrossRefGoogle Scholar
  39. Welch RB (1999) Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. In: Ashersleben G, Bachmann T, Müsseler J (Eds) Cognitive contributions to the perception of spatial and temporal events. Elsevier Science, B.V, Amsterdam, pp 371–387CrossRefGoogle Scholar
  40. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 3:638–667CrossRefGoogle Scholar
  41. Zampini M, Shore DI, Spence C (2003) Audiovisual temporal order judgments. Exp Brain Res 152:198–210PubMedCrossRefGoogle Scholar
  42. Zampini M, Guest S, Shore DI, Spence C (2005a) Audio–visual simultaneity judgments. Percept Psychophys 67:531–544PubMedGoogle Scholar
  43. Zampini M, Shore DI, Spence C (2005b) Audiovisual prior entry. Neurosci Lett 381:217–222PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  • Argiro Vatakis
    • 1
  • Jordi Navarra
    • 1
  • Salvador Soto-Faraco
    • 2
  • Charles Spence
    • 1
  1. 1.Crossmodal Research Laboratory, Department of Experimental PsychologyUniversity of OxfordOxfordUK
  2. 2.ICREA and Parc Científic de BarcelonaUniversitat de BarcelonaBarcelonaSpain

Personalised recommendations