Auditory temporal modulation of the visual Ternus effect: the influence of time interval

Abstract

Research on multisensory interactions has shown that the perceived timing of a visual event can be captured by a temporally proximal sound. This effect has been termed ‘temporal ventriloquism effect.’ Using the Ternus display, we systematically investigated how auditory configurations modulate the visual apparent-motion percepts. The Ternus display involves a multielement stimulus that can induce either of two different percepts of apparent motion: ‘element motion’ or ‘group motion’. We found that two sounds presented in temporal proximity to, or synchronously with, the two visual frames, respectively, can shift the transitional threshold for visual apparent motion (Experiments 1 and 3). However, such effects were not evident with single-sound configurations (Experiment 2). A further experiment (Experiment 4) provided evidence that time interval information is an important factor for crossmodal interaction of audiovisual Ternus effect. The auditory interval was perceived as longer than the same physical visual interval in the sub-second range. Furthermore, the perceived audiovisual interval could be predicted by optimal integration of the visual and auditory intervals.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262

    CAS  PubMed  Google Scholar 

  2. Allen PG, Kolers PA (1981) Sensory specificity of apparent motion. J Exp Psychol Hum Percept Perform 7(6):1318–1328

    Article  CAS  PubMed  Google Scholar 

  3. Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis 20(7):1391–1397

    Article  PubMed  Google Scholar 

  4. Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 42(43):487–493

    CAS  PubMed  Google Scholar 

  5. Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29(6):578–584

    CAS  PubMed  Google Scholar 

  6. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10(4):433–436

    Article  CAS  PubMed  Google Scholar 

  7. Bruns P, Getzmann S (2008) Audiovisual influences on the perception of visual apparent motion: exploring the effect of a single sound. Acta Psychol 129(2):273–283

    Article  Google Scholar 

  8. Burr D, Banks MS, Morrone MC (2009) Auditory dominance over vision in the perception of interval duration. Exp Brain Res 198(1):49–57

    Article  PubMed  Google Scholar 

  9. Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  10. Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    Article  CAS  PubMed  Google Scholar 

  11. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433

    Article  CAS  PubMed  Google Scholar 

  12. Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8(4):162–169

    Article  PubMed  Google Scholar 

  13. Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophy 63(4):719–725

    CAS  Google Scholar 

  14. Fraisse P (1984) Perception and estimation of time. Annu Rev Psychol 35:1–36

    Article  CAS  PubMed  Google Scholar 

  15. Freeman E, Driver J (2008) Direction of visual apparent motion driven by timing of a static sound. Curr Biol 18:1262–1266

    Article  CAS  PubMed  Google Scholar 

  16. Getzmann S (2007) The effect of brief auditory stimuli on visual apparent motion. Perception 36(7):1089–1103

    Article  PubMed  Google Scholar 

  17. Goldstone S, Goldfarb JL (1964a) Auditory and visual time judgment. J Gen Psychol 70:369–387

    CAS  PubMed  Google Scholar 

  18. Goldstone S, Goldfarb JL (1964b) Direct comparison of auditory and visual durations. J Exp Psychol 67:483–485

    Article  CAS  PubMed  Google Scholar 

  19. Goldstone S, Lhamon WT (1974) Studies of auditory-visual differences in human time judgment. 1. Sounds are judged longer than lights. Percept Mot Skills 39(1):63–82

    CAS  PubMed  Google Scholar 

  20. Harrar V, Harris LR (2007) Multimodal Ternus: visual, tactile, and multisensory grouping in apparent motion. Perception 36:1455–1464

    Article  PubMed  Google Scholar 

  21. Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, New York

    Google Scholar 

  22. Jaekl PM, Harris LR (2007) Auditory-visual temporal integration measured by shifts in perceived temporal location. Neurosci Lett 417(3):219–224

    Article  CAS  PubMed  Google Scholar 

  23. Keetels M, Stekelenburg J, Vroomen J (2007) Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Exp Brain Res 180(3):449–456

    Article  PubMed  Google Scholar 

  24. Kramer P, Yantis S (1997) Perceptual grouping in space and time: evidence from the Ternus display. Percept Psychophys 59(1):87–99

    CAS  PubMed  Google Scholar 

  25. Levitin DJ, MacLean K, Mathews M, Chu L (2000) The perception of cross-modal simultaneity. Int J Comput Anticip Syst 323–329

  26. Ley I, Haggard P, yarrow K (2009) Optimal integration of auditory and vibrotactile information for judgments of temporal order. J Exp Psychol Hum Percept Perform 35(4):1005–1019

    Article  PubMed  Google Scholar 

  27. Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17(1):154–163

    Article  Google Scholar 

  28. Pantle AJ, Petersik JT (1980) Effects of spatial parameters on the perceptual organization of a bistable motion display. Percept Psychophys 27(4):307–312

    CAS  PubMed  Google Scholar 

  29. Pantle AJ, Picciano L (1976) A multistable movement display: evidence for two separate motion systems in human vision. Science 193(4252):500–502

    Article  CAS  PubMed  Google Scholar 

  30. Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10:437–442

    Article  CAS  PubMed  Google Scholar 

  31. Petersik JT, Rice CM (2006) The evolution of explanations of a perceptual phenomenon: a case history using the Ternus effect. Perception 35(6):807–821

    Article  PubMed  Google Scholar 

  32. Pick HL, Warren DH, Hay JC (1969) Sensory conflict in judgments of spatial direction. Percept Psychophys 6(4):203–205

    Google Scholar 

  33. Posner MI, Nissen MJ, Klein RM (1976) Visual dominance: an information-processing account of its origins and significance. Psychol Rev 83(2):157–171

    Article  CAS  PubMed  Google Scholar 

  34. Radeau M, Bertelson P (1987) Auditory-visual interaction, the timing of inputs. Thomas (1941) revisited. Psychol Res 49(1):17–22

    Article  CAS  PubMed  Google Scholar 

  35. Sanabria D, Soto-Faraco S, Chan JS, Spence C (2004) When does visual perceptual grouping affect multisensory integration? Cogn Affect Behav Neurosci 4(2):218–229

    Article  PubMed  Google Scholar 

  36. Scheier C, Nijhawan R, Shimojo S (1999) Sound alters visual temporal resolution. Invest Ophthalmol Vis Sci 40(Suppl. 4):4169

    Google Scholar 

  37. Shimojo S, Scheier C, Nijhawan R, Shams L, Kamitani Y, Watanabe K (2001) Beyond perceptual modality: auditory effects on visual perception. Acoust Sci Technol 22(2):61–67

    Article  Google Scholar 

  38. Soto-Faraco S, Kingstone A (2004) Multisensory integration of dynamic information. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  39. Spence C (2007) Audiovisual multisensory integration. Acoust Sci Technol 28:61–70

    Article  Google Scholar 

  40. Spence C, Sanabria D, Soto-Faraco S (2007) Intersensory Gestalten and crossmodal scene perception. In: Noguchi K (ed) Psychology of beauty and Kansei: new horizons of Gestalt perception. Fuzanbo International, Tokyo, pp 519–579

    Google Scholar 

  41. Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M et al (2001) When is now? Perception of simultaneity. Proc R Soc Lond B Biol Sci 268(1462):31–38

    Article  CAS  Google Scholar 

  42. Ternus J (1926) Experimentelle Untersuchungen über phänomenale Identität. Psychologische Forschung 7:81–136

    Article  Google Scholar 

  43. Treutwein B, Strasburger H (1999) Fitting the psychometric function. Percept Psychophys 61(1):87–106

    CAS  PubMed  Google Scholar 

  44. van Erp JBF, Werkhoven PJ (2004) Vibro-tactile and visual asynchronies: sensitivity and consistency. Perception 33:103–111

    Article  PubMed  Google Scholar 

  45. Vroomen J, de Gelder B (2004) Temporal ventriloquism: sound modulates the flash-lag effect. J Exp Psychol Hum Percept Perform 30:513–518

    Article  PubMed  Google Scholar 

  46. Vroomen J, Keetels M (2006) The spatial constraint in intersensory pairing: no role in temporal ventriloquism. J Exp Psychol Hum Percept Perform 32(4):1063–1071

    Article  PubMed  Google Scholar 

  47. Walker JT, Scott KJ (1981) Auditory-visual conflicts in the perceived duration of lights, tones and gaps. J Exp Psychol Hum Percept Perform 7(6):1327–1339

    Article  CAS  PubMed  Google Scholar 

  48. Wearden JH (2006) When do auditory/visual differences in duration judgments occur? Q J Exp Psychol 59(10):1709–1724

    Article  CAS  Google Scholar 

  49. Wearden JH, Edwards H, Fakhri M, Percival A (1998) Why “sounds are judged longer than lights”: application of a model of the internal clock in humans. Q J Exp Psychol 51(2):97–120

    CAS  Google Scholar 

  50. Welch RB (1999) Meaning, attention, and the ‘unity assumption’ in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G, Bachmann T, Müsseler J (eds) Cognitive contribution to the perception of spatial and temporal events. Elsevier, Amsterdam, pp 317–387

    Google Scholar 

  51. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667

    Article  CAS  PubMed  Google Scholar 

  52. Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance: sensory processes and perception, vol 1. Wiley, New York, pp 1–36

    Google Scholar 

  53. Witten IB, Knudsen EI (2005) Why seeing is believing: merging auditory and visual worlds. Neuron 48(3):489–496

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

This research was supported in part by a grant from the German Research Foundation (DFG) within the Collaborative Research Centre SFB 453. We thank Hans Strasburger for stimulating comments and the reviewers for insightful suggestions.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Zhuanghua Shi.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Shi, Z., Chen, L. & Müller, H.J. Auditory temporal modulation of the visual Ternus effect: the influence of time interval. Exp Brain Res 203, 723–735 (2010). https://doi.org/10.1007/s00221-010-2286-3

Download citation

Keywords

  • Time perception
  • Vision
  • Audition
  • Temporal ventriloquism effect
  • Ternus display