Experimental Brain Research

, Volume 166, Issue 3–4, pp 337–344

Is the auditory sensory memory sensitive to visual information?

  • Julien Besle
  • Alexandra Fort
  • Marie-Hélène Giard
Research Article


The mismatch negativity (MMN) component of auditory event-related brain potentials can be used as a probe to study the representation of sounds in auditory sensory memory (ASM). Yet it has been shown that an auditory MMN can also be elicited by an illusory auditory deviance induced by visual changes. This suggests that some visual information may be encoded in ASM and is accessible to the auditory MMN process. It is not known, however, whether visual information affects ASM representation for any audiovisual event or whether this phenomenon is limited to specific domains in which strong audiovisual illusions occur. To highlight this issue, we have compared the topographies of MMNs elicited by non-speech audiovisual stimuli deviating from audiovisual standards on the visual, the auditory, or both dimensions. Contrary to what occurs with audiovisual illusions, each unimodal deviant elicited sensory-specific MMNs, and the MMN to audiovisual deviants included both sensory components. The visual MMN was, however, different from a genuine visual MMN obtained in a visual-only control oddball paradigm, suggesting that auditory and visual information interacts before the MMN process occurs. Furthermore, the MMN to audiovisual deviants was significantly different from the sum of the two sensory-specific MMNs, showing that the processes of visual and auditory change detection are not completely independent.


Electrophysiology Audiovisual MMN Multisensory integration Memory 


  1. Alain C, Woods DL, Knight RT (1998) A distributed cortical network for auditory sensory memory in humans. Brain Res 812:23–37PubMedGoogle Scholar
  2. Bental E, Dafny N, Feldman S (1968) Convergence of auditory and visual stimuli on single cells in the primary visual cortex of unanesthetized unrestrained cats. Exp Neurol 20:341–351PubMedGoogle Scholar
  3. Bertelson P, Aschersleben G (1998) Automatic visual bias of perceived auditory location. Psychon Bull Rev 5:482–489Google Scholar
  4. Berti S, Schroger E (2004) Distraction effects in vision: behavioral and event-related potential indices. Neuroreport 15:665–669PubMedGoogle Scholar
  5. Besle J, Fort A, Delpuech C, Giard M-H (2004) Bimodal speech: early suppressive visual effects in the human auditory cortex. Eur J Neurosci 20:2225–2234PubMedGoogle Scholar
  6. Bruneau N, Roux S, Garreau B, Martineau J, Lelord G (1990) Cortical evoked potentials as indicators of Auditory-Visual Cross-Modal Association in young adults. Pavlov J Biol Sci 25:189–204Google Scholar
  7. Cahill L, Ohl F, Scheich H (1996) Alteration of auditory cortex activity with a visual stimulus through conditioning: a 2-deoxyglucose analysis. Neurobiol Learn Mem 65:213–222PubMedGoogle Scholar
  8. Colin C, Radeau M, Soquet A, Demolin D, Colin F, Deltenre P (2002b) Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory. Clin Neurophysiol 113:495–506PubMedGoogle Scholar
  9. Colin C, Radeau M, Soquet A, Dachy B, Deltenre P (2002a) Electrophysiology of spatial scene analysis: the mismatch negativity (MMN) is sensitive to the ventriloquism illusion. Clin Neurophysiol 113:507–518PubMedGoogle Scholar
  10. Colin C, Radeau M, Soquet A, Deltenre P (2004) Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants. Clin Neurophysiol 115:1989–2000PubMedGoogle Scholar
  11. Czigler I, Balazs L, Winkler I (2002) Memory-based detection of task-irrelevant visual changes. Psychophysiology 39:869–873PubMedGoogle Scholar
  12. Fort A, Giard M-H (2004) Multiple electrophysiological mechanisms of audio-visual integration in human perception. In: Calvert G, Spence C, Stein B (eds) The handbook of multisensory processes. MIT Press, CambridgeGoogle Scholar
  13. Fort A, Delpuech C, Pernier J, Giard MH (2002a) Dynamics of cortico-subcortical crossmodal operations involved in audio-visual object detection in humans. Cereb Cortex 12:1031–1039PubMedGoogle Scholar
  14. Fort A, Delpuech C, Pernier J, Giard MH (2002b) Early auditory-visual interactions in human cortex during nonredundant target identification. Brain Res Cogn Brain Res 14:20–30PubMedGoogle Scholar
  15. Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490PubMedGoogle Scholar
  16. Giard MH, Perrin F, Pernier J (1990) Brain generators implicated in processing of auditory stimulus deviance A topographic ERP study. Psychophysiology 27:627–640PubMedGoogle Scholar
  17. Heslenfeld DJ (2003) Visual mismatch negativity. In: Polich J (ed) Detection of change: event-related potential and fMRI findings. Kluwer Academic Publishers, Dordrecht, pp 41–60Google Scholar
  18. Kenemans JL, Jong TG, Verbaten MN (2003) Detection of visual change: mismatch or rareness?. Neuroreport 14:1239–1242PubMedGoogle Scholar
  19. Kropotov JD, Näätänen R, Sevostianov AV, Alho K, Reinikainen K, Kropotova OV (1995) Mismatch negativity to auditory stimulus change recorded directly from the human temporal cortex. Psychophysiology 32:418–422PubMedGoogle Scholar
  20. Lebib R, Papo D, de Bode S, Baudonniere PM (2003) Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation. Neurosci Lett 341:185–188PubMedGoogle Scholar
  21. McGurk H, McDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748PubMedGoogle Scholar
  22. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128PubMedGoogle Scholar
  23. Möttönen R, Krause CM, Tiippana K, Sams M (2002) Processing of changes in visual speech in the human auditory cortex. Brain Res Cogn Brain Res 13:417–425PubMedGoogle Scholar
  24. Näätänen R (1992) Attention and Brain Function. Hillsdale, NJ, USAGoogle Scholar
  25. Näätänen R, Winkler I (1999) The concept of auditory stimulus representation in cognitive neuroscience. Psychol Bull 125:826–859PubMedGoogle Scholar
  26. Nyman G, Alho K, laurinen P, Paavilainen P, Radil T, Rainikainen K, Sams M, Näätänen R (1990) Mismatch negativity (MMN) for sequences of auditory and visual stimuli: evidence for a mechanism specific to the auditory modality. Electroencephalogr Clin Neurophysiol 77:436–444PubMedGoogle Scholar
  27. Pazo-Alvarez P, Cadaveira F, Amenedo E (2003) MMN in the visual modality: a review. Biol Psychol 63:199–236PubMedGoogle Scholar
  28. Pernier J, Perrin F, Bertrand O (1988) Scalp current density fields: concept and properties. Electroencephalogr Clin Neurophysiol 69:385–389PubMedGoogle Scholar
  29. Perrin F, Pernier J, Bertrand O, Giard M-H (1987) Mapping of scalp potentials by surface spline interpolation. Electroencephalogr Clin Neurophysiol 66:75–81PubMedGoogle Scholar
  30. Perrin F, Pernier J, Bertrand O, Echallier JF (1989) Spherical splines for scalp potential and current density mapping. Electroencephalogr Clin Neurophysiol 72:184–187PubMedGoogle Scholar
  31. Ritter W, Deacon D, Gomes H, Javitt DC, Vaughan HG Jr (1995) The mismatch negativity of event-related potentials as a probe of transient auditory memory: a review. Ear Hear 16:52–67PubMedGoogle Scholar
  32. Rosenblum LD, Fowler CA (1991) Audiovisual investigation of the loudness-effort effect for speech and nonspeech events. J Exp Psychol Hum Percept Perform 17:976–985PubMedGoogle Scholar
  33. Saldana HM, Rosenblum LD (1993) Visual influences on auditory pluck and bow judgments. Percept Psychophys 54:406–416PubMedGoogle Scholar
  34. Sams M, Aulanko R, Hamalainen H, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141–145PubMedGoogle Scholar
  35. Soto-Faraco S, Navarra J, Alsius A (2004) Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition 92:B13–B23PubMedGoogle Scholar
  36. Stagg C, Hindley P, Tales A, Butler S (2004) Visual mismatch negativity: the detection of stimulus change. Neuroreport 15:659–663PubMedGoogle Scholar
  37. Stekelenburg JJ, Vroomen J, de Gelder B (2004) Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neurosci Lett 357:163–166PubMedGoogle Scholar
  38. Vroomen J, Bertelson P, de Gelder B (2001) The ventriloquist effect does not depend on the direction of automatic visual attention. Percept Psychophys 63:651–659PubMedGoogle Scholar

Copyright information

© Springer-Verlag 2005

Authors and Affiliations

  • Julien Besle
    • 1
    • 2
    • 3
    • 4
  • Alexandra Fort
    • 2
    • 3
    • 4
  • Marie-Hélène Giard
    • 2
    • 3
    • 4
  1. 1.Univ. Lyon 2LyonFrance
  2. 2.Univ. Lyon 1LyonFrance
  3. 3.IFNL IFR19LyonFrance
  4. 4.INSERM U280Mental Processes and Brain ActivationBron CedexFrance

Personalised recommendations