Experimental Brain Research

, Volume 203, Issue 4, pp 723–735 | Cite as

Auditory temporal modulation of the visual Ternus effect: the influence of time interval

Research Article

Abstract

Research on multisensory interactions has shown that the perceived timing of a visual event can be captured by a temporally proximal sound. This effect has been termed ‘temporal ventriloquism effect.’ Using the Ternus display, we systematically investigated how auditory configurations modulate the visual apparent-motion percepts. The Ternus display involves a multielement stimulus that can induce either of two different percepts of apparent motion: ‘element motion’ or ‘group motion’. We found that two sounds presented in temporal proximity to, or synchronously with, the two visual frames, respectively, can shift the transitional threshold for visual apparent motion (Experiments 1 and 3). However, such effects were not evident with single-sound configurations (Experiment 2). A further experiment (Experiment 4) provided evidence that time interval information is an important factor for crossmodal interaction of audiovisual Ternus effect. The auditory interval was perceived as longer than the same physical visual interval in the sub-second range. Furthermore, the perceived audiovisual interval could be predicted by optimal integration of the visual and auditory intervals.

Keywords

Time perception Vision Audition Temporal ventriloquism effect Ternus display 

References

  1. Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262PubMedGoogle Scholar
  2. Allen PG, Kolers PA (1981) Sensory specificity of apparent motion. J Exp Psychol Hum Percept Perform 7(6):1318–1328CrossRefPubMedGoogle Scholar
  3. Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis 20(7):1391–1397CrossRefPubMedGoogle Scholar
  4. Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 42(43):487–493PubMedGoogle Scholar
  5. Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29(6):578–584PubMedGoogle Scholar
  6. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10(4):433–436CrossRefPubMedGoogle Scholar
  7. Bruns P, Getzmann S (2008) Audiovisual influences on the perception of visual apparent motion: exploring the effect of a single sound. Acta Psychol 129(2):273–283CrossRefGoogle Scholar
  8. Burr D, Banks MS, Morrone MC (2009) Auditory dominance over vision in the perception of interval duration. Exp Brain Res 198(1):49–57CrossRefPubMedGoogle Scholar
  9. Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, CambridgeGoogle Scholar
  10. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721CrossRefPubMedGoogle Scholar
  11. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433CrossRefPubMedGoogle Scholar
  12. Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8(4):162–169CrossRefPubMedGoogle Scholar
  13. Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophy 63(4):719–725Google Scholar
  14. Fraisse P (1984) Perception and estimation of time. Annu Rev Psychol 35:1–36CrossRefPubMedGoogle Scholar
  15. Freeman E, Driver J (2008) Direction of visual apparent motion driven by timing of a static sound. Curr Biol 18:1262–1266CrossRefPubMedGoogle Scholar
  16. Getzmann S (2007) The effect of brief auditory stimuli on visual apparent motion. Perception 36(7):1089–1103CrossRefPubMedGoogle Scholar
  17. Goldstone S, Goldfarb JL (1964a) Auditory and visual time judgment. J Gen Psychol 70:369–387PubMedGoogle Scholar
  18. Goldstone S, Goldfarb JL (1964b) Direct comparison of auditory and visual durations. J Exp Psychol 67:483–485CrossRefPubMedGoogle Scholar
  19. Goldstone S, Lhamon WT (1974) Studies of auditory-visual differences in human time judgment. 1. Sounds are judged longer than lights. Percept Mot Skills 39(1):63–82PubMedGoogle Scholar
  20. Harrar V, Harris LR (2007) Multimodal Ternus: visual, tactile, and multisensory grouping in apparent motion. Perception 36:1455–1464CrossRefPubMedGoogle Scholar
  21. Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, New YorkGoogle Scholar
  22. Jaekl PM, Harris LR (2007) Auditory-visual temporal integration measured by shifts in perceived temporal location. Neurosci Lett 417(3):219–224CrossRefPubMedGoogle Scholar
  23. Keetels M, Stekelenburg J, Vroomen J (2007) Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Exp Brain Res 180(3):449–456CrossRefPubMedGoogle Scholar
  24. Kramer P, Yantis S (1997) Perceptual grouping in space and time: evidence from the Ternus display. Percept Psychophys 59(1):87–99PubMedGoogle Scholar
  25. Levitin DJ, MacLean K, Mathews M, Chu L (2000) The perception of cross-modal simultaneity. Int J Comput Anticip Syst 323–329Google Scholar
  26. Ley I, Haggard P, yarrow K (2009) Optimal integration of auditory and vibrotactile information for judgments of temporal order. J Exp Psychol Hum Percept Perform 35(4):1005–1019CrossRefPubMedGoogle Scholar
  27. Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17(1):154–163CrossRefGoogle Scholar
  28. Pantle AJ, Petersik JT (1980) Effects of spatial parameters on the perceptual organization of a bistable motion display. Percept Psychophys 27(4):307–312PubMedGoogle Scholar
  29. Pantle AJ, Picciano L (1976) A multistable movement display: evidence for two separate motion systems in human vision. Science 193(4252):500–502CrossRefPubMedGoogle Scholar
  30. Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10:437–442CrossRefPubMedGoogle Scholar
  31. Petersik JT, Rice CM (2006) The evolution of explanations of a perceptual phenomenon: a case history using the Ternus effect. Perception 35(6):807–821CrossRefPubMedGoogle Scholar
  32. Pick HL, Warren DH, Hay JC (1969) Sensory conflict in judgments of spatial direction. Percept Psychophys 6(4):203–205Google Scholar
  33. Posner MI, Nissen MJ, Klein RM (1976) Visual dominance: an information-processing account of its origins and significance. Psychol Rev 83(2):157–171CrossRefPubMedGoogle Scholar
  34. Radeau M, Bertelson P (1987) Auditory-visual interaction, the timing of inputs. Thomas (1941) revisited. Psychol Res 49(1):17–22CrossRefPubMedGoogle Scholar
  35. Sanabria D, Soto-Faraco S, Chan JS, Spence C (2004) When does visual perceptual grouping affect multisensory integration? Cogn Affect Behav Neurosci 4(2):218–229CrossRefPubMedGoogle Scholar
  36. Scheier C, Nijhawan R, Shimojo S (1999) Sound alters visual temporal resolution. Invest Ophthalmol Vis Sci 40(Suppl. 4):4169Google Scholar
  37. Shimojo S, Scheier C, Nijhawan R, Shams L, Kamitani Y, Watanabe K (2001) Beyond perceptual modality: auditory effects on visual perception. Acoust Sci Technol 22(2):61–67CrossRefGoogle Scholar
  38. Soto-Faraco S, Kingstone A (2004) Multisensory integration of dynamic information. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, CambridgeGoogle Scholar
  39. Spence C (2007) Audiovisual multisensory integration. Acoust Sci Technol 28:61–70CrossRefGoogle Scholar
  40. Spence C, Sanabria D, Soto-Faraco S (2007) Intersensory Gestalten and crossmodal scene perception. In: Noguchi K (ed) Psychology of beauty and Kansei: new horizons of Gestalt perception. Fuzanbo International, Tokyo, pp 519–579Google Scholar
  41. Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M et al (2001) When is now? Perception of simultaneity. Proc R Soc Lond B Biol Sci 268(1462):31–38CrossRefGoogle Scholar
  42. Ternus J (1926) Experimentelle Untersuchungen über phänomenale Identität. Psychologische Forschung 7:81–136CrossRefGoogle Scholar
  43. Treutwein B, Strasburger H (1999) Fitting the psychometric function. Percept Psychophys 61(1):87–106PubMedGoogle Scholar
  44. van Erp JBF, Werkhoven PJ (2004) Vibro-tactile and visual asynchronies: sensitivity and consistency. Perception 33:103–111CrossRefPubMedGoogle Scholar
  45. Vroomen J, de Gelder B (2004) Temporal ventriloquism: sound modulates the flash-lag effect. J Exp Psychol Hum Percept Perform 30:513–518CrossRefPubMedGoogle Scholar
  46. Vroomen J, Keetels M (2006) The spatial constraint in intersensory pairing: no role in temporal ventriloquism. J Exp Psychol Hum Percept Perform 32(4):1063–1071CrossRefPubMedGoogle Scholar
  47. Walker JT, Scott KJ (1981) Auditory-visual conflicts in the perceived duration of lights, tones and gaps. J Exp Psychol Hum Percept Perform 7(6):1327–1339CrossRefPubMedGoogle Scholar
  48. Wearden JH (2006) When do auditory/visual differences in duration judgments occur? Q J Exp Psychol 59(10):1709–1724CrossRefGoogle Scholar
  49. Wearden JH, Edwards H, Fakhri M, Percival A (1998) Why “sounds are judged longer than lights”: application of a model of the internal clock in humans. Q J Exp Psychol 51(2):97–120Google Scholar
  50. Welch RB (1999) Meaning, attention, and the ‘unity assumption’ in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G, Bachmann T, Müsseler J (eds) Cognitive contribution to the perception of spatial and temporal events. Elsevier, Amsterdam, pp 317–387Google Scholar
  51. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667CrossRefPubMedGoogle Scholar
  52. Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance: sensory processes and perception, vol 1. Wiley, New York, pp 1–36Google Scholar
  53. Witten IB, Knudsen EI (2005) Why seeing is believing: merging auditory and visual worlds. Neuron 48(3):489–496CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Zhuanghua Shi
    • 1
  • Lihan Chen
    • 1
    • 2
  • Hermann J. Müller
    • 1
    • 3
  1. 1.Department PsychologieLudwig-Maximilians-Universität MünchenMunichGermany
  2. 2.Department of PsychologyPeking UniversityBeijingPeople’s Republic of China
  3. 3.School of PsychologyBirkbeck College, University of LondonLondonUK

Personalised recommendations