Advertisement

Perception & Psychophysics

, Volume 70, Issue 6, pp 955–968 | Cite as

Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type

  • Rob L. J. van EijkEmail author
  • Armin Kohlrausch
  • James F. Juola
  • Steven van de Par
Article

Abstract

When an audio—visual event is perceived in the natural environment, a physical delay will always occur between the arrival of the leading visual component and that of the trailing auditory component. This natural timing relationship suggests that the point of subjective simultaneity (PSS) should occur at an auditory delay greater than or equal to 0 msec. A review of the literature suggests that PSS estimates derived from a temporal order judgment (TOJ) task differ from those derived from a synchrony judgment (SJ) task, with (unnatural) auditory-leading PSS values reported mainly for the TOJ task. We report data from two stimulus types that differed in terms of complexity— namely, (1) a flash and a click and (2) a bouncing ball and an impact sound. The same participants judged the temporal order and synchrony of both stimulus types, using three experimental methods: (1) a TOJ task with two response categories (“audio first” or “video first”), (2) an SJ task with two response categories (“synchronous” or “asynchronous”; SJ2), and (3) an SJ task with three response categories (“audio first,” “synchronous,” or “video first”; SJ3). Both stimulus types produced correlated PSS estimates with the SJ tasks, but the estimates from the TOJ procedure were uncorrelated with those obtained from the SJ tasks. These results suggest that the SJ task should be preferred over the TOJ task when the primary interest is in perceived audio—visual synchrony.

Keywords

Stimulus Type Temporal Order Judgment Visual Speech Response Proportion Auditory Component 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Allan, L. G. (1975). The relationship between judgments of successiveness and judgments of order. Perception & Psychophysics, 18, 29–36.CrossRefGoogle Scholar
  2. Allan, L. G., & Kristofferson, A. B. (1974). Successiveness discrimination: Two models. Perception & Psychophysics, 15, 37–46.CrossRefGoogle Scholar
  3. Arrighi, R., Alais, D., & Burr, D. (2006). Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. Journal of Vision, 6, 260–268.PubMedCrossRefGoogle Scholar
  4. Aschersleben, G. (1999). On the asymmetry of the temporal contiguity window: Commentary on Lewkowicz. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 421–424). Amsterdam: Elsevier.CrossRefGoogle Scholar
  5. Aschersleben, G., & Müsseler, J. (1999). Dissociations in the timing of stationary and moving stimuli. Journal of Experimental Psychology: Human Perception & Performance, 25, 1709–1720.CrossRefGoogle Scholar
  6. Bald, L., Berrien, F. K., Price, J. B., & Sprague, R. O. (1942). Errors in perceiving the temporal order of auditory and visual stimuli. Journal of Applied Psychology, 26, 382–388.CrossRefGoogle Scholar
  7. Dinnerstein, A. J., & Zlotogura, P. (1968). Intermodal perception of temporal order and motor skills: Effects of age. Perceptual & Motor Skills, 26, 987–1000.Google Scholar
  8. Dixon, N. F., & Spitz, L. (1980). The detection of auditory visual desynchrony. Perception, 9, 719–721.PubMedCrossRefGoogle Scholar
  9. Enoki, K., Washikita, K., & Yamada, M. (2006). Detection threshold of asynchrony between auditory and visual stimuli for various motion patterns of a ball. In Proceedings of the 9th Western Pacific Acoustics Conference (WESPAC IX) [CD-ROM]. Seoul: Acoustical Society of Korea.Google Scholar
  10. Erlebacher, A., & Sekuler, R. (1971). Response frequency equalization: A bias model for psychophysics. Perception & Psychophysics, 9, 315–320.Google Scholar
  11. Exner, S. (1875). Experimentelle Untersuchung der einfachsten psychischen Processe: III. Der persönlichen Gleichung zweiter Theil. Archiv für die gesammte Physiologie des Menschen & der Thiere, 11, 403–432.CrossRefGoogle Scholar
  12. Fujisaki, W., Shimojo, S., Kashino, M., & Nishida, S. (2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7, 773–778.PubMedCrossRefGoogle Scholar
  13. Hamlin, A. J. (1895). On the least observable interval between stimuli addressed to disparate senses and to different organs of the same sense. American Journal of Psychology, 6, 564–575.CrossRefGoogle Scholar
  14. Hirsh, I. J., & Fraisse, P. (1964). Simultanéité et succession de stimuli hétérogènes. L'Année Psychologique, 64, 1–19.PubMedCrossRefGoogle Scholar
  15. Hirsh, I. J., & Sherrick, C. E., Jr. (1961). Perceived order in different sense modalities. Journal of Experimental Psychology, 62, 423–432.PubMedCrossRefGoogle Scholar
  16. Hollier, M. P., & Rimell, A. N. (1998, September). An experimental investigation into multi-modal synchronization sensitivity for perceptual model development. Paper presented at the 105th Convention of the Audio Engineering Society, San Francisco, CA.Google Scholar
  17. Jaśkowski, P. (1993). Selective attention and temporal-order judgment. Perception, 22, 681–689.PubMedCrossRefGoogle Scholar
  18. Jaśkowski, P., Jaroszyk, F., & Hojan-Jezierska, D. (1990). Temporal-order judgments and reaction time for stimuli of different modalities. Psychological Research, 52, 35–38.PubMedCrossRefGoogle Scholar
  19. Juola, J. F., van Eijk, R. L. J., Kohlrausch, A., & van de Par, S. (2007, July). Synchrony detection and temporal order judgments for bimodal stimuli. Paper presented at the joint meeting of the Experimental Psychology (U.K.) & Psychonomic (U.S.A.) Societies, Edinburgh, Scotland.Google Scholar
  20. Keetels, M., & Vroomen, J. (2005). The role of spatial disparity and hemifields in audio—visual temporal order judgments. Experimental Brain Research, 167, 635–640.CrossRefGoogle Scholar
  21. King, A. J., & Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Experimental Brain Research, 60, 492–500.CrossRefGoogle Scholar
  22. Lewkowicz, D. J. (1996). Perception of auditory—visual temporal synchrony in human infants. Journal of Experimental Psychology: Human Perception & Performance, 22, 1094–1106.CrossRefGoogle Scholar
  23. Ludbrook, J. (1998). Multiple comparison procedures updated. Clinical & Experimental Pharmacology & Physiology, 25, 1032–1037.CrossRefGoogle Scholar
  24. McGrath, M., & Summerfield, Q. (1985). Intermodal timing relations and audio—visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America, 77, 678–685.PubMedCrossRefGoogle Scholar
  25. Mitrani, L., Shekerdjiiski, S., & Yakimoff, N. (1986). Mechanisms and asymmetries in visual perception of simultaneity and temporal order. Biological Cybernetics, 54, 159–165.PubMedCrossRefGoogle Scholar
  26. Myung, I. J. (2003). Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology, 47, 90–100.CrossRefGoogle Scholar
  27. Neumann, O., & Niepel, M. (2004). Timing of “perception” and perception of “time.” In C. Kaernbach, E. Schröger, & H. Müller (Eds.), Psychophysics beyond sensation: Laws and invariants of human cognition (pp. 245–269). Mahwah, NJ: Erlbaum.Google Scholar
  28. Parducci, A., & Haugen, R. (1967). The frequency principle for comparative judgments. Perception & Psychophysics, 2, 81–82.Google Scholar
  29. Pöppel, E. (1988). Mindworks: Time and conscious experience [Trans. of Grenzen des Bewuβtseins by T. Artin]. Boston: Harcourt Brace Jovanovich.Google Scholar
  30. Rihs, S. (1995). The influence of audio on perceived picture quality and subjective audio—video delay tolerance. In R. Hamberg & H. de Ridder (Eds.), Advanced methods for the evaluation of television picture quality: Proceedings of the MOSAIC Workshop (pp. 133–137). Eindhoven, The Netherlands: Institute for Perception Research.Google Scholar
  31. Rutschmann, J., & Link, R. (1964). Perception of temporal order of stimuli differing in sense mode and simple reaction time. Perceptual & Motor Skills, 18, 345–352.Google Scholar
  32. Schneider, K. A., & Bavelier, D. (2003). Components of visual prior entry. Cognitive Psychology, 47, 333–366.PubMedCrossRefGoogle Scholar
  33. Sekuler, R., & Erlebacher, A. (1971). The invalidity of “invalid results from the method of constant stimuli”: A common artifact in the methods of psychophysics. Perception & Psychophysics, 9, 309–311.Google Scholar
  34. Shore, D. I., Gray, K., Spry, E., & Spence, C. (2005). Spatial modulation of tactile temporal-order judgments. Perception, 34, 1251–1262.PubMedCrossRefGoogle Scholar
  35. Shore, D. I., Spry, E., & Spence, C. (2002). Confusing the mind by crossing the hands. Cognitive Brain Research, 14, 153–163.PubMedCrossRefGoogle Scholar
  36. Smeele, P. M. T. (1994). Perceiving speech: Integrating auditory and visual speech. Unpublished PhD thesis, Technische Universiteit Delft, The Netherlands. Retrieved April 11, 2007, from repository.tudelft.nl/ file/257315/201176.Google Scholar
  37. Smith, W. F. (1933). The relative quickness of visual and auditory perception. Journal of Experimental Psychology, 16, 239–257.CrossRefGoogle Scholar
  38. Spence, C., Baddeley, R., Zampini, M., James, R., & Shore, D. I. (2003). Multisensory temporal order judgments: When two locations are better than one. Perception & Psychophysics, 65, 318–328.CrossRefGoogle Scholar
  39. Spence, C., Shore, D. I., & Klein, R. M. (2001). Multisensory prior entry. Journal of Experimental Psychology: General, 130, 799–832.CrossRefGoogle Scholar
  40. Stekelenburg, J. J., & Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience, 19, 1964–1973.PubMedCrossRefGoogle Scholar
  41. Stelmach, L. B., & Herdman, C. M. (1991). Directed attention and perception of temporal order. Journal of Experimental Psychology: Human Perception & Performance, 17, 539–550.CrossRefGoogle Scholar
  42. Sternberg, S., & Knoll, R. L. (1973). The perception of temporal order: Fundamental issues and a general model. In S. Kornblum (Ed.), Attention and performance IV (pp. 629–685). New York: Academic Press.Google Scholar
  43. Stone, J. V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., et al. (2001). When is now? Perception of simultaneity. Proceedings of the Royal Society B, 268, 31–38.PubMedCrossRefGoogle Scholar
  44. Teatini, G., Farnè, M., Verzella, F., & Berruecos, P., Jr. (1976). Perception of temporal order: Visual and auditory stimuli. Giornale Italiano di Psicologia, 3, 157–164.Google Scholar
  45. Tune, G. S. (1964). Response preferences: A review of some relevant literature. Psychological Bulletin, 61, 286–302.PubMedCrossRefGoogle Scholar
  46. Van de Par, S., Kohlrausch, A., & Juola, J. F. (2002). Some methodological aspects for measuring asynchrony detection in audio—visual stimuli. In A. Calvo-Manzano, A. Perez-Lopez, & J. S. Santiago (Eds.), Proceedings of the Forum Acusticum 2002 [CD-ROM]. Porto, Portugal: Universidade do Porto, Faculdade de Engenharia.Google Scholar
  47. Van Eijk, R. L. J., Kohlrausch, A., van de Par, S., & Juola, J. F. (2006, June). The influence of psychophysical procedure and stimulus type on estimates of human performance in detecting audio—visual asynchrony. Paper presented at the 7th International Multisensory Research Forum (IMRF), University of Dublin, Trinity College, Dublin, Ireland.Google Scholar
  48. Van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neuropsychologia, 45, 598–607.PubMedCrossRefGoogle Scholar
  49. Vatakis, A., Navarra, J., Soto-Faraco, S., & Spence, C. (2008). Audiovisual temporal adaptation of speech: Temporal order versus simultaneity judgments. Experimental Brain Research, 185, 521–529.CrossRefGoogle Scholar
  50. Vatakis, A., & Spence, C. (2006a). Audiovisual synchrony perception for music, speech, and object actions. Brain Research, 1111, 134–142.PubMedCrossRefGoogle Scholar
  51. Vatakis, A., & Spence, C. (2006b). Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neuroscience Letters, 393, 40–44.PubMedCrossRefGoogle Scholar
  52. Vatakis, A., & Spence, C. (2007). Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & Psychophysics, 69, 744–756.CrossRefGoogle Scholar
  53. Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audio— visual asynchrony. Cognitive Brain Research, 22, 32–35.PubMedCrossRefGoogle Scholar
  54. Whipple, G. M. (1899). On nearly simultaneous clicks and flashes. American Journal of Psychology, 10, 280–286.CrossRefGoogle Scholar
  55. Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fit. Perception & Psychophysics, 63, 1293–1313.CrossRefGoogle Scholar
  56. Zampini, M., Brown, T., Shore, D. I., Maravita, A., Röder, B., & Spence, C. (2005). Audiotactile temporal order judgments. Acta Psychologica, 118, 277–291.PubMedCrossRefGoogle Scholar
  57. Zampini, M., Guest, S., Shore, D. I., & Spence, C. (2005). Audio— visual simultaneity judgments. Perception & Psychophysics, 67, 531–544.CrossRefGoogle Scholar
  58. Zampini, M., Shore, D. I., & Spence, C. (2003a). Audiovisual temporal order judgments. Experimental Brain Research, 152, 198–210.CrossRefGoogle Scholar
  59. Zampini, M., Shore, D. I., & Spence, C. (2003b). Multisensory temporal order judgments: The role of hemispheric redundancy. International Journal of Psychophysiology, 50, 165–180.PubMedCrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2008

Authors and Affiliations

  • Rob L. J. van Eijk
    • 1
    Email author
  • Armin Kohlrausch
    • 1
    • 2
  • James F. Juola
    • 1
    • 3
  • Steven van de Par
    • 2
  1. 1.Eindhoven University of TechnologyEindhovenThe Netherlands
  2. 2.Philips Research LaboratoriesEindhovenThe Netherlands
  3. 3.University of KansasLawrence

Personalised recommendations