Skip to main content
SpringerLink
Log in

Catch the moment: multisensory enhancement of rapid visual events by sound

  • Research article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Repetition blindness (RB) is a visual deficit, wherein observers fail to perceive the second occurrence of a repeated item in a rapid serial visual presentation stream. Chen and Yeh (Psychon Bull Rev 15:404–408, 2008) recently observed a reduction of the RB effect when the repeated items were accompanied by two sounds. The current study further manipulated the pitch of the two sounds (same versus different) in order to examine whether this cross-modal facilitation effect is caused by the multisensory enhancement of the visual event by sound, or multisensory Gestalt (perceptual grouping) of a new representation formed by combining the visual and auditory inputs. The results showed robust facilitatory effects of sound on RB regardless of the pitch of the sounds (Experiment 1), despite an effort to further increase the difference in pitch (Experiment 2). Experiment 3 revealed a close link between participants’ awareness of pitch and the effect of pitch on the RB effect. We conclude that the facilitatory effect of sound on RB results from multisensory enhancement of the perception of visual events by auditory signals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Arieh Y, Marks LE (2008) Cross-modal interaction between vision and hearing: a speed-accuracy analysis. Percept Psychophys 70:412–421

    Article  PubMed  Google Scholar 

  • Arnell KM, Larson JM (2002) Cross-modality attentional blinks without preparatory task-set switching. Psychon Bull Rev 9:497–506

    PubMed  Google Scholar 

  • Barlow HB (1990) A theory about the functional role and synaptic mechanism of visual after-effects. In: Blakemore C (ed) Vision: coding and efficiency. Cambridge University Press, Cambridge, pp 363–375

    Google Scholar 

  • Bavelier D (1994) Repetition blindness between visually different items: the case of pictures and words. Cognition 51:199–236

    Article  PubMed  CAS  Google Scholar 

  • Bolognini N, Frassinetti F, Serino A, Ladavas E (2005) “Acoustical vision” of below threshold stimuli: interaction among spatially converging audiovisual inputs. Exp Brain Res 160:273–282

    Article  PubMed  Google Scholar 

  • Bowman H, Wyble B (2007) The simultaneous type, serial token model of temporal attention and working memory. Psychol Rev 114:38–70

    Article  PubMed  Google Scholar 

  • Brand-D’Abrescia M, Lavie N (2008) Task coordination between and within sensory modalities: effects on distraction. Percept Psychophys 70:508–515

    Article  PubMed  Google Scholar 

  • Bregman AS (1990) Auditory scene analysis: the perceptual organization of sound. MIT Press, Cambridge

    Google Scholar 

  • Chen Y-C, Yeh S-L (2008) Visual events modulated by sound in repetition blindness. Psychon Bull Rev 15:404–408

    Article  PubMed  Google Scholar 

  • Chun MM (1997) Types and tokens in visual processing: a double dissociation between the attentional blink and repetition blindness. J Exp Psychol Hum Percept Perform 23:738–755

    Article  PubMed  CAS  Google Scholar 

  • Colavita FB (1974) Human sensory dominance. Percept Psychophys 16:409–412

    Google Scholar 

  • Duncan J, Martens S, Ward R (1997) Restricted attentional capacity within but not between sensory modalities. Nature 387:808–810

    Article  PubMed  CAS  Google Scholar 

  • Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophys 63:719–725

    PubMed  CAS  Google Scholar 

  • Frassinetti F, Pavani F, Lavadas E (2002) Acoustical vision of neglected stimuli: Interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci 14:62–69

    Article  PubMed  Google Scholar 

  • Fujisaki W, Nishida S (2005) Temporal frequency characteristics of synchrony-asynchrony discrimination of audio-visual signals. Exp Brain Res 166:455–464

    Article  PubMed  Google Scholar 

  • Gebhard JW, Mowbray GH (1959) On discriminating the rate of visual flicker and auditory flutter. Am J Psychol 72:521–528

    Article  PubMed  CAS  Google Scholar 

  • Geldard FA, Sherrick CE (1972) The cutaneous “rabbit”: a perceptual illusion. Science 178:178–179

    Article  PubMed  CAS  Google Scholar 

  • Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    Article  PubMed  CAS  Google Scholar 

  • Harris CL, Morris AL (2000) Orthographic repetition blindness. Q J Exp Psychol 53A:1039–1060

    Article  Google Scholar 

  • Ho C, Mason O, Spence C (2007) An investigation into the temporal dimension of the Mozart effect: evidence from the attentional blink task. Acta Psychol 125:117–128

    Article  Google Scholar 

  • Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, New York

    Google Scholar 

  • Huddleston WE, Lewis JW, Phinney RE, DeYoe EA (2008) Auditory and visual attention-based apparent motion share functional parallels. Percept Psychophys 70:1207–1216

    Article  PubMed  Google Scholar 

  • Humphreys GW, Besner D, Quinlan PT (1988) Event perception and the word repetition effect. J Exp Psychol Gen 117:51–67

    Article  Google Scholar 

  • Kanwisher N (1987) Repetition blindness: type recognition without token individuation. Cognition 27:117–143

    Article  PubMed  CAS  Google Scholar 

  • Kanwisher N (1991) Repetition blindness and illusory conjunctions: errors in binding visual types with visual tokens. J Exp Psychol Hum Percept Perform 17:402–421

    Google Scholar 

  • Keetels M, Stekelenburg J, Vroomen J (2007) Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Exp Brain Res 180:449–456

    Article  PubMed  Google Scholar 

  • Kirk RE (1995) Fundamental assumptions in analysis of variance. In: Experimental design: procedures for the behavioral sciences. Brooks/Cole Publishing Company, Pacific Frove, CA, pp 72–112

  • Koppen C, Spence C (2007) Seeing the light: exploring the Colavita visual dominance effect. Exp Brain Res 180:737–754

    Article  PubMed  Google Scholar 

  • Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414

    Article  PubMed  Google Scholar 

  • Lewald J, Ehrenstein WH, Guski R (2001) Spatio-temporal constraints for auditory-visual integration. Behav Brain Res 121:69–79

    Article  PubMed  CAS  Google Scholar 

  • Lippert M, Logothetis NK, Kayser C (2007) Improvement of visual contrast detection by a simultaneous sound. Brain Res 1173:102–109

    Article  PubMed  CAS  Google Scholar 

  • Luo CR, Caramazza A (1996) Temporal and spatial repetition blindness: effects of presentation mode and repetition lag on the perception of repeated items. J Exp Psychol Hum Percept Perform 22:95–113

    Article  PubMed  CAS  Google Scholar 

  • Martens S, Munneke J, Smid H, Johnson A (2006) Quick minds don’t blink: electrophysiological correlates of individual differences in attentional selection. J Cogn Neurosci 18:1423–1438

    Article  PubMed  Google Scholar 

  • Miller J (1982) Divided attention evidence for coactivation with redundant signals. Cogn Psychol 14:247–279

    Article  PubMed  CAS  Google Scholar 

  • Mitroff SR, Scholl BJ, Noles NS (2007) Object files can be purely episodic. Perception 36:1730–1735

    Article  PubMed  Google Scholar 

  • Morris AL, Still ML, Caldwell-Harris CL (2009) Repetition blindness: an emergent property of inter-item competition. Cogn Psychol 58:338–375

    Article  PubMed  Google Scholar 

  • Odgaard EC, Arieh Y, Marks LE (2003) Cross-modal enhancement of perceived brightness: sensory interaction versus response bias. Percept Psychophys 65:123–132

    PubMed  Google Scholar 

  • Olivers CNL, Van der Burg E (2008) Bleeping you out of the blink: sound saves vision from oblivion. Brain Res 1242:191–199

    Article  PubMed  CAS  Google Scholar 

  • Park J, Kanwisher N (1994) Determinants of repetition blindness. J Exp Psychol Hum Percept Perform 20:500–519

    Article  PubMed  CAS  Google Scholar 

  • Partan S, Marler P (1999) Communication goes multimodal. Science 283:1272–1273

    Article  PubMed  CAS  Google Scholar 

  • Pick HL, Warren DH, Hay JC (1969) Sensory conflict in judgment of spatial direction. Percept Psychophys 6:203–205

    Google Scholar 

  • Raymond JE, Shapiro KL, Arnell KM (1992) Temporary suppression of visual processing in an RSVP task: an attentional blink? J Exp Psychol Hum Percept Perform 18:849–860

    Article  PubMed  CAS  Google Scholar 

  • Robertson IH, Mattingley JB, Rorden C, Driver J (1998) Phasic alerting of neglect patients overcomes their spatial deficit in visual awareness. Nature 395:169–172

    Article  PubMed  CAS  Google Scholar 

  • Santangelo V, Belardinelli MO, Spence C (2007) The suppression of reflexive visual and auditory orienting when attention is otherwise engaged. J Exp Psychol Hum Percept Perform 33:137–148

    Article  PubMed  Google Scholar 

  • Sheth BR, Shimojo S (2004) Sound-aided recovery from and persistence against visual filling-in. Vis Res 44:1907–1917

    Article  PubMed  Google Scholar 

  • Shimojo S, Watanabe K, Scheier C (2001) The resolution of ambiguous motion: attentional modulation and development. In: Braun J, Koch C, Davis JL (eds) Visual attention and control circuits. MIT Press, Cambridge, pp 243–264

    Google Scholar 

  • Sinnett S, Soto-Faraco S, Spence C (2008) The co-occurrence of multisensory competition and facilitation. Acta Psychol 128:153–161

    Article  Google Scholar 

  • Soto-Faraco S, Spence C (2002) Modality-specific auditory and visual temporal processing of deficits. Q J Exp Psychol A 55:23–40

    PubMed  Google Scholar 

  • Spence C, Soto-Faraco S (2009) Auditory perception: interactions with vision. In: Plack C (ed) Auditory perception. Oxford University Press, Oxford

  • Spence C, Nicholls MER, Driver J (2001) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63:330–336

    PubMed  CAS  Google Scholar 

  • Spence C, Sanabria D, Soto-Faraco S (2007) Intersensory Gestalten and crossmodal scene perception. In: Noguchi K (ed) The psychology of beauty and Kansei: new horizons of Gestalt perception. Nihon University, Tokyo, pp 519–579

    Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge

    Google Scholar 

  • Stein BE, London N, Wilkinson LK, Price DD (1996) Enhancement of perceived visual intensity by auditory stimuli: a psychophysical analysis. J Cogn Sci 8:497–506

    Google Scholar 

  • Tsai CH (1996) Frequency and stroke counts of Chinese characters. Available via http://technology.chtsai.org/charfreq/. Accessed 31 Aug 1999

  • Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2008) Pip and pop: non-spatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34:1053–1065

    Article  PubMed  Google Scholar 

  • Van Vleet TM, Robertson LC (2006) Cross-modal interactions in time and space: auditory influence on visual attention in hemispatial neglect. J Cogn Neurosci 18:1368–1379

    Article  PubMed  Google Scholar 

  • Vroomen J, de Gelder B (2000) Sound enhances visual perception: crossmodal effects of auditory organization on vision. J Exp Psychol Hum Percept Perform 26:1583–1590

    Article  PubMed  CAS  Google Scholar 

  • Vroomen J, de Gelder B (2004) Temporal ventriloquism: sound modulates the flash-lag effect. J Exp Psychol Hum Percept Perform 30:513–518

    Article  PubMed  Google Scholar 

  • Watanabe K, Shimojo S (2001) When sound affects vision: effects of auditory grouping on visual motion perception. Psychol Sci 12:109–116

    Article  PubMed  CAS  Google Scholar 

  • Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638–667

    Article  PubMed  CAS  Google Scholar 

  • Welch RB, DuttonHurt LD, Warren DH (1986) Contributions of audition and vision to temporal rate perception. Percept Psychophys 39:294–300

    PubMed  CAS  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Science Council of Taiwan (NSC96-2413-H-002-009-MY3 & NSC 97-2911-I-002-038). We thank Charles Spence for his suggestions and editing of this article.

Author information

Author notes

  1. Yi-Chuan Chen

    Present address: Department of Experimental Psychology, University of Oxford, South Park Road, Oxford, OX1 3UD, UK

Authors and Affiliations

  1. Department of Psychology, National Taiwan University, Taipei, Taiwan

    Yi-Chuan Chen & Su-Ling Yeh

Authors
  1. Yi-Chuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Su-Ling Yeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Su-Ling Yeh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chen, YC., Yeh, SL. Catch the moment: multisensory enhancement of rapid visual events by sound. Exp Brain Res 198, 209–219 (2009). https://doi.org/10.1007/s00221-009-1831-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-009-1831-4

Keywords

Search

Navigation