Skip to main content
SpringerLink
Log in

Auditory temporal modulation of the visual Ternus effect: the influence of time interval

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Research on multisensory interactions has shown that the perceived timing of a visual event can be captured by a temporally proximal sound. This effect has been termed ‘temporal ventriloquism effect.’ Using the Ternus display, we systematically investigated how auditory configurations modulate the visual apparent-motion percepts. The Ternus display involves a multielement stimulus that can induce either of two different percepts of apparent motion: ‘element motion’ or ‘group motion’. We found that two sounds presented in temporal proximity to, or synchronously with, the two visual frames, respectively, can shift the transitional threshold for visual apparent motion (Experiments 1 and 3). However, such effects were not evident with single-sound configurations (Experiment 2). A further experiment (Experiment 4) provided evidence that time interval information is an important factor for crossmodal interaction of audiovisual Ternus effect. The auditory interval was perceived as longer than the same physical visual interval in the sub-second range. Furthermore, the perceived audiovisual interval could be predicted by optimal integration of the visual and auditory intervals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262

    CAS  PubMed  Google Scholar 

  • Allen PG, Kolers PA (1981) Sensory specificity of apparent motion. J Exp Psychol Hum Percept Perform 7(6):1318–1328

    Article  CAS  PubMed  Google Scholar 

  • Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis 20(7):1391–1397

    Article  PubMed  Google Scholar 

  • Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 42(43):487–493

    CAS  PubMed  Google Scholar 

  • Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29(6):578–584

    CAS  PubMed  Google Scholar 

  • Brainard DH (1997) The psychophysics toolbox. Spat Vis 10(4):433–436

    Article  CAS  PubMed  Google Scholar 

  • Bruns P, Getzmann S (2008) Audiovisual influences on the perception of visual apparent motion: exploring the effect of a single sound. Acta Psychol 129(2):273–283

    Article  Google Scholar 

  • Burr D, Banks MS, Morrone MC (2009) Auditory dominance over vision in the perception of interval duration. Exp Brain Res 198(1):49–57

    Article  PubMed  Google Scholar 

  • Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  • Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    Article  CAS  PubMed  Google Scholar 

  • Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433

    Article  CAS  PubMed  Google Scholar 

  • Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8(4):162–169

    Article  PubMed  Google Scholar 

  • Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophy 63(4):719–725

    CAS  Google Scholar 

  • Fraisse P (1984) Perception and estimation of time. Annu Rev Psychol 35:1–36

    Article  CAS  PubMed  Google Scholar 

  • Freeman E, Driver J (2008) Direction of visual apparent motion driven by timing of a static sound. Curr Biol 18:1262–1266

    Article  CAS  PubMed  Google Scholar 

  • Getzmann S (2007) The effect of brief auditory stimuli on visual apparent motion. Perception 36(7):1089–1103

    Article  PubMed  Google Scholar 

  • Goldstone S, Goldfarb JL (1964a) Auditory and visual time judgment. J Gen Psychol 70:369–387

    CAS  PubMed  Google Scholar 

  • Goldstone S, Goldfarb JL (1964b) Direct comparison of auditory and visual durations. J Exp Psychol 67:483–485

    Article  CAS  PubMed  Google Scholar 

  • Goldstone S, Lhamon WT (1974) Studies of auditory-visual differences in human time judgment. 1. Sounds are judged longer than lights. Percept Mot Skills 39(1):63–82

    CAS  PubMed  Google Scholar 

  • Harrar V, Harris LR (2007) Multimodal Ternus: visual, tactile, and multisensory grouping in apparent motion. Perception 36:1455–1464

    Article  PubMed  Google Scholar 

  • Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, New York

    Google Scholar 

  • Jaekl PM, Harris LR (2007) Auditory-visual temporal integration measured by shifts in perceived temporal location. Neurosci Lett 417(3):219–224

    Article  CAS  PubMed  Google Scholar 

  • Keetels M, Stekelenburg J, Vroomen J (2007) Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Exp Brain Res 180(3):449–456

    Article  PubMed  Google Scholar 

  • Kramer P, Yantis S (1997) Perceptual grouping in space and time: evidence from the Ternus display. Percept Psychophys 59(1):87–99

    CAS  PubMed  Google Scholar 

  • Levitin DJ, MacLean K, Mathews M, Chu L (2000) The perception of cross-modal simultaneity. Int J Comput Anticip Syst 323–329

  • Ley I, Haggard P, yarrow K (2009) Optimal integration of auditory and vibrotactile information for judgments of temporal order. J Exp Psychol Hum Percept Perform 35(4):1005–1019

    Article  PubMed  Google Scholar 

  • Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17(1):154–163

    Article  Google Scholar 

  • Pantle AJ, Petersik JT (1980) Effects of spatial parameters on the perceptual organization of a bistable motion display. Percept Psychophys 27(4):307–312

    CAS  PubMed  Google Scholar 

  • Pantle AJ, Picciano L (1976) A multistable movement display: evidence for two separate motion systems in human vision. Science 193(4252):500–502

    Article  CAS  PubMed  Google Scholar 

  • Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10:437–442

    Article  CAS  PubMed  Google Scholar 

  • Petersik JT, Rice CM (2006) The evolution of explanations of a perceptual phenomenon: a case history using the Ternus effect. Perception 35(6):807–821

    Article  PubMed  Google Scholar 

  • Pick HL, Warren DH, Hay JC (1969) Sensory conflict in judgments of spatial direction. Percept Psychophys 6(4):203–205

    Google Scholar 

  • Posner MI, Nissen MJ, Klein RM (1976) Visual dominance: an information-processing account of its origins and significance. Psychol Rev 83(2):157–171

    Article  CAS  PubMed  Google Scholar 

  • Radeau M, Bertelson P (1987) Auditory-visual interaction, the timing of inputs. Thomas (1941) revisited. Psychol Res 49(1):17–22

    Article  CAS  PubMed  Google Scholar 

  • Sanabria D, Soto-Faraco S, Chan JS, Spence C (2004) When does visual perceptual grouping affect multisensory integration? Cogn Affect Behav Neurosci 4(2):218–229

    Article  PubMed  Google Scholar 

  • Scheier C, Nijhawan R, Shimojo S (1999) Sound alters visual temporal resolution. Invest Ophthalmol Vis Sci 40(Suppl. 4):4169

    Google Scholar 

  • Shimojo S, Scheier C, Nijhawan R, Shams L, Kamitani Y, Watanabe K (2001) Beyond perceptual modality: auditory effects on visual perception. Acoust Sci Technol 22(2):61–67

    Article  Google Scholar 

  • Soto-Faraco S, Kingstone A (2004) Multisensory integration of dynamic information. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  • Spence C (2007) Audiovisual multisensory integration. Acoust Sci Technol 28:61–70

    Article  Google Scholar 

  • Spence C, Sanabria D, Soto-Faraco S (2007) Intersensory Gestalten and crossmodal scene perception. In: Noguchi K (ed) Psychology of beauty and Kansei: new horizons of Gestalt perception. Fuzanbo International, Tokyo, pp 519–579

    Google Scholar 

  • Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M et al (2001) When is now? Perception of simultaneity. Proc R Soc Lond B Biol Sci 268(1462):31–38

    Article  CAS  Google Scholar 

  • Ternus J (1926) Experimentelle Untersuchungen über phänomenale Identität. Psychologische Forschung 7:81–136

    Article  Google Scholar 

  • Treutwein B, Strasburger H (1999) Fitting the psychometric function. Percept Psychophys 61(1):87–106

    CAS  PubMed  Google Scholar 

  • van Erp JBF, Werkhoven PJ (2004) Vibro-tactile and visual asynchronies: sensitivity and consistency. Perception 33:103–111

    Article  PubMed  Google Scholar 

  • Vroomen J, de Gelder B (2004) Temporal ventriloquism: sound modulates the flash-lag effect. J Exp Psychol Hum Percept Perform 30:513–518

    Article  PubMed  Google Scholar 

  • Vroomen J, Keetels M (2006) The spatial constraint in intersensory pairing: no role in temporal ventriloquism. J Exp Psychol Hum Percept Perform 32(4):1063–1071

    Article  PubMed  Google Scholar 

  • Walker JT, Scott KJ (1981) Auditory-visual conflicts in the perceived duration of lights, tones and gaps. J Exp Psychol Hum Percept Perform 7(6):1327–1339

    Article  CAS  PubMed  Google Scholar 

  • Wearden JH (2006) When do auditory/visual differences in duration judgments occur? Q J Exp Psychol 59(10):1709–1724

    Article  CAS  Google Scholar 

  • Wearden JH, Edwards H, Fakhri M, Percival A (1998) Why “sounds are judged longer than lights”: application of a model of the internal clock in humans. Q J Exp Psychol 51(2):97–120

    CAS  Google Scholar 

  • Welch RB (1999) Meaning, attention, and the ‘unity assumption’ in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G, Bachmann T, Müsseler J (eds) Cognitive contribution to the perception of spatial and temporal events. Elsevier, Amsterdam, pp 317–387

    Google Scholar 

  • Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667

    Article  CAS  PubMed  Google Scholar 

  • Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance: sensory processes and perception, vol 1. Wiley, New York, pp 1–36

    Google Scholar 

  • Witten IB, Knudsen EI (2005) Why seeing is believing: merging auditory and visual worlds. Neuron 48(3):489–496

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

This research was supported in part by a grant from the German Research Foundation (DFG) within the Collaborative Research Centre SFB 453. We thank Hans Strasburger for stimulating comments and the reviewers for insightful suggestions.

Author information

Authors and Affiliations

  1. Department Psychologie, Ludwig-Maximilians-Universität München, Leopoldstrasse 13, 80802, Munich, Germany

    Zhuanghua Shi, Lihan Chen & Hermann J. Müller

  2. Department of Psychology, Peking University, 100871, Beijing, People’s Republic of China

    Lihan Chen

  3. School of Psychology, Birkbeck College, University of London, Malet Street, London, WC1E 7HX, UK

    Hermann J. Müller

Authors
  1. Zhuanghua Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lihan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hermann J. Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhuanghua Shi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shi, Z., Chen, L. & Müller, H.J. Auditory temporal modulation of the visual Ternus effect: the influence of time interval. Exp Brain Res 203, 723–735 (2010). https://doi.org/10.1007/s00221-010-2286-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-010-2286-3

Keywords

Search

Navigation