Skip to main content

Mismatch Negativity with Visual-only and Audiovisual Speech

Brain Topography volume 21, pages 207–215 (2009)Cite this article

Abstract

The functional organization of cortical speech processing is thought to be hierarchical, increasing in complexity and proceeding from primary sensory areas centrifugally. The current study used the mismatch negativity (MMN) obtained with electrophysiology (EEG) to investigate the early latency period of visual speech processing under both visual-only (VO) and audiovisual (AV) conditions. Current density reconstruction (CDR) methods were used to model the cortical MMN generator locations. MMNs were obtained with VO and AV speech stimuli at early latencies (approximately 82–87 ms peak in time waveforms relative to the acoustic onset) and in regions of the right lateral temporal and parietal cortices. Latencies were consistent with bottom-up processing of the visible stimuli. We suggest that a visual pathway extracts phonetic cues from visible speech, and that previously reported effects of AV speech in classical early auditory areas, given later reported latencies, could be attributable to modulatory feedback from visual phonetic processing.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. Note: Auditory conditions were also tested, including “da” and “ba”, but they are not reported here.

  2. In an fMRI study (Bernstein et al., Visual phonetic processing localized using speech and non-speech face gestures in video and point-light displays, in revision for publication), to isolate cortical sites with responsibility for processing visible speech features, speech and non-speech face gestures were presented in natural video and point-light displays during fMRI scanning at 3.0T. Participants with normal hearing and varied lipreading ability viewed the stimuli. Independent of stimulus media (i.e., point-light versus video), bilateral regions of the superior temporal sulcus, the superior temporal gyrii, and the middle temporal gyrii were activated by speech gestures. These regions were more activated in good versus poor lipreaders, consistent with an interpretation that they are important areas in the processing of visible speech.

References

  • Astikainen P, Lillstrang E, Ruusuvirta T (2008) Visual mismatch negativity for changes in orientation—a sensory memory-dependent response. Eur J NeuroSci 28:2319–2324

    PubMed  Article  Google Scholar 

  • Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008a) Spatio-temporal dynamics of audiovisual speech processing. NeuroImage 39:423–435

    PubMed  Article  Google Scholar 

  • Bernstein LE, Lu ZL, Jiang J (2008b) Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing. Brain Res 1242:172–184

    PubMed  Article  CAS  Google Scholar 

  • Calvert GA, Campbell R (2003) Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 15:57–70

    PubMed  Article  Google Scholar 

  • Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport 10:2619–2623

    PubMed  Article  CAS  Google Scholar 

  • Capek CM, MacSweeney M, Woll B, Waters D, McGuire PK, David AS, Brammer MJ, Campbell R (2008) Cortical circuits for silent speechreading in deaf and hearing people. Neuropsychologia 46:1233–1241

    PubMed  Article  Google Scholar 

  • Colin C, Radeau M, Soquet A, Demolin D, Colin F, Deltenre P (2002) Mismatch negativity evoked by the McGurk-MacDonald effect: a phonetic representation within short-term memory. Clin Neurophysiol 113:495–506

    PubMed  Article  CAS  Google Scholar 

  • Colin C, Radeau M, Soquet A, Deltenre P (2004) Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants. Clin Neurophysiol 115:1989–2000

    PubMed  Article  CAS  Google Scholar 

  • Czigler I, Weisz J, Winkler I (2007) Backward masking and visual mismatch negativity: electrophysiological evidence for memory-based detection of deviant stimuli. Psychophysiology 44:610–619

    PubMed  Article  Google Scholar 

  • Darvas F, Schmitt U, Louis AK, Fuchs M, Knoll G, Buchner H (2001) Spatio-temporal current density reconstruction (stCDR) from EEG/MEG-data. Brain Topogr 13:195–207

    PubMed  Article  CAS  Google Scholar 

  • Fuchs M, Wagner M, Kohler T, Wischmann H-A (1999) Linear and nonlinear current density reconstruction. J Clin Neurophysiol 16:267–295

    PubMed  Article  CAS  Google Scholar 

  • Giard MH, Perrin F, Pernier J, Bouchet P (1990) Brain generators implicated in the processing of auditory stimulus deviance: a topographic event-related potential study. Psychophysiology 27:627–640

    PubMed  Article  CAS  Google Scholar 

  • Grossman E, Donnelly M, Price R, Pickens D, Morgan V, Neighbor G, Blake R (2000) Brain areas involved in perception of biological motion. J Cogn Neurosci 12:711–720

    PubMed  Article  CAS  Google Scholar 

  • Horvath J, Czigler I, Jacobsen T, Maess B, Schroger E, Winkler I (2008) MMN or no MMN: no magnitude of deviance effect on the MMN amplitude. Psychophysiology 45:60–69

    PubMed  Google Scholar 

  • Jasper HH (1958) The ten-twenty electrode system of the international federation. Electroencephalogr Clin Neurophysiol 10:371–375

    Google Scholar 

  • Kislyuk DS, Möttönen R, Sams M (2008) Visual processing affects the neural basis of auditory discrimination. J Cogn Neurosci 20:2175–2184

    PubMed  Article  Google Scholar 

  • Lebib R, Papo D, Douiri A, de Bode S, Dowens MG, Baudonniere P-M (2004) Modulations of ‘late’ event-related brain potentials in humans by dynamic audiovisual speech stimuli. Neurosci Lett 372:74–79

    PubMed  Article  CAS  Google Scholar 

  • Lehmann D, Skrandies W (1980) Reference-free identification of components of checkerboard-evoked multichannel potential fields. Electroencephalogr Clin Neurophysiol 48:609–621

    PubMed  Article  CAS  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    PubMed  Article  CAS  Google Scholar 

  • Molholm S, Martinez A, Ritter W, Javitt DC, Foxe JJ (2005) The neural circuitry of pre-attentive auditory change-detection: an fMRI study of pitch and duration mismatch negativity generators. Cereb Cortex 15:545–551

    PubMed  Article  Google Scholar 

  • Möttönen R, Krause CM, Tiippana K, Sams M (2002) Processing of changes in visual speech in the human auditory cortex. Cogn Brain Res 13:417–425

    Article  Google Scholar 

  • Näätänen R, Gaillard AWK, Mäntysalo S (1978) Early selective-attention effect on evoked potential reinterpreted. Acta Psychol 42:313–329

    Article  Google Scholar 

  • Näätänen R, Jacobsen T, Winkler I (2005) Memory-based or afferent processes in mismatch negativity (MMN): a review of the evidence. Psychophysiology 42:25–32

    PubMed  Article  Google Scholar 

  • Näätänen R, Paavilainen P, Rinne T, Alho K (2007) The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin Neurophysiol 118:2544–2590

    PubMed  Article  Google Scholar 

  • Pascual-Marqui R (2002) Standardized low resolution brain electromagnetic tomography (sLORETA): technical details. Methods Find Exp Clin Pharmacol 24D:5–12

    Google Scholar 

  • Pazo-Alvarez O, Cadaveira F, Amenedo E (2003) MMN in the visual modality: a review. Biol Psychol 63:199–236

    PubMed  Article  CAS  Google Scholar 

  • Ponton CW, Don M, Eggermont JJ, Kwong B (1997) Integrated mismatch negativity (MMNi): a noise-free representation of evoked responses allowing single-point distribution-free statistical tests. Electroencephalogr Clin Neurophysiol 104:143–150

    PubMed  Article  CAS  Google Scholar 

  • Ponton CW, Auer ET Jr, Bernstein LE (2002) Neurocognitive basis for audiovisual speech perception: evidence from event-related potentials. In: 7th international congress of spoken language processing, Denver, CO, pp 1697–1700

  • Puce A, Perrett D (2003) Electrophysiology and brain imaging of biological motion. Philos Trans R Soc B-Biol Sci 358:435–445

    Article  Google Scholar 

  • Puce A, Allison T, Bentin S, Gore JC, McCarthy G (1998) Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci 18:2188–2199

    PubMed  CAS  Google Scholar 

  • Puce A, Smith A, Allison T (2000) ERPs evoked by viewing facial movements. Cogn Neuropsychol 17:221–239

    Article  Google Scholar 

  • Reale RA, Calvert GA, Thesen T, Jenison RL, Kawasaki H, Oya H, Howard MA, Brugge JF (2007) Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 145:162–184

    PubMed  Article  CAS  Google Scholar 

  • Saint-Amour D, De Sanctis P, Molholm S, Ritter W, Foxe JJ (2007) Seeing voices: high-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia 45:587–597

    PubMed  Article  Google Scholar 

  • Sams M, Aulanko R, Hamalainen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141–145

    PubMed  Article  CAS  Google Scholar 

  • Santi A, Servos P, Vatikiotis-Bateson E, Kuratate T, Munhall K (2003) Perceiving biological motion: dissociating visible speech from walking. J Cogn Neurosci 15:800–809

    PubMed  Article  Google Scholar 

  • Scherg M (1990) Fundamentals of dipole source analysis. In: Grandori F, Hoke M, Romani GL (eds) Auditory evoked magnetic fields and electric potentials. Karger, Basel, pp 40–69

    Google Scholar 

  • Scott SK, Johnsrude IS (2003) The neuroanatomical and functional organization of speech perception. Trends Neurosci 26:100–107

    PubMed  Article  CAS  Google Scholar 

  • Wagner M, Fuchs M, Wischmann H-A, Ottenberg K, Dössel O (1995) Cortex segmentation from 3D MR images for MEG reconstructions. In: Baumgartner C, Deecke L, Stroink G, Williamson SJ (eds) Biomagnetism: fundamental research and clinical applications, vol 7 studies in applied electromagnetics and mechanics. Elsevier Science IOS Press, Amsterdam, The Netherlands, pp 433–438

    Google Scholar 

  • Yvert B, Bertrand O, Thevenet M, Echallier JF, Pernier J (1997) A systematic evaluation of the spherical model accuracy in EEG dipole localization. Electroencephalogr Clin Neurophysiol 102:452–459

    PubMed  Article  CAS  Google Scholar 

Download references

Acknowledgments

Ewen Chao and Betty Kwong are thanked for their assistance. This research was supported by a grant from NIH/NIDCD DC006035 and DC008308, and by NSF IIS9996088.

Author information

Affiliations

  1. Compumedics/Neuroscan, Inc., 6605W W.T. Harris Blvd Suite F, Charlotte, NC, 28269, USA

    Curtis W. Ponton

  2. Division of Communication and Auditory Neuroscience, House Ear Institute, 2100 W Third Street, Los Angeles, CA, 90057, USA

    Lynne E. Bernstein

  3. Psychology Department, University of Southern California, Los Angeles, CA, 90089, USA

    Lynne E. Bernstein

  4. Department of Speech-Language-Hearing, Dole Human Development Center, University of Kansas, 1000 Sunnyside Avenue, Room 3001, Lawrence, KS, 66045-7555, USA

    Edward T. Auer Jr.

Authors
  1. Curtis W. Ponton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lynne E. Bernstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Edward T. Auer Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lynne E. Bernstein.

Additional information

This article is published as part of the Special Issue on Multisensory Integration.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ponton, C.W., Bernstein, L.E. & Auer, E.T. Mismatch Negativity with Visual-only and Audiovisual Speech. Brain Topogr 21, 207–215 (2009). https://doi.org/10.1007/s10548-009-0094-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10548-009-0094-5

Keywords

  • EEG
  • MMN
  • Audiovisual speech processing
  • Visual speech processing