Skip to main content
SpringerLink
Log in

Seeing speech affects acoustic information processing in the human brainstem

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Afferent auditory processing in the human brainstem is often assumed to be determined by acoustic stimulus features alone and immune to stimulation by other senses or cognitive factors. In contrast, we show that lipreading during speech perception influences early acoustic processing. Event-related brainstem potentials were recorded from ten healthy adults to concordant (acoustic-visual match), conflicting (acoustic-visual mismatch) and unimodal stimuli. Audiovisual (AV) interactions occurred as early as ∼11 ms post-acoustic stimulation and persisted for the first 30 ms of the response. Furthermore, the magnitude of interaction depended on AV pairings. These findings indicate considerable plasticity in early auditory processing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Binder JR, Frost JA, Hammeke TA, Bellgowan PS, Springer JA, Kaufman JN, Possing ET (2000) Human temporal lobe activation by speech and non-speech sounds. Cereb Cortex 10:512–528

    Article  PubMed  CAS  Google Scholar 

  • Bradlow AR, Kraus N, Nicol TG, Mcgee TJ, Cunningham J, Zecker SG, Carrell TD (1999) Effects of lengthened formant transition duration on discrimination and neural representation of synthetic CV syllables by normal and learning-disabled children. J Acoust Soc Am 106:2086–2096

    Article  PubMed  CAS  Google Scholar 

  • Burnett LR, Stein BE, Chaponis D, Wallace MT (2004) Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124:535–547

    Article  PubMed  CAS  Google Scholar 

  • Burton MW, Small SL, Blumstein SE (2000) The role of segmentation in phonological processing: an fMRI investigation. J Cogn Neurosci 12:679–690

    Article  PubMed  CAS  Google Scholar 

  • Bushara KO, Hanakawa T, Immisch I, Toma K, Kansaku K, Hallett M (2003) Neural correlates of cross-modal binding. Nat Neurosci 6:190–195

    Article  PubMed  CAS  Google Scholar 

  • Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E (2003) Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213–2218

    Article  PubMed  Google Scholar 

  • Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997) Activation of auditory cortex during silent lipreading. Science 276:593–596

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619–2623

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

    Article  PubMed  CAS  Google Scholar 

  • Campbell R, MacSweeney M, Surguladze S, Calvert G, McGuire P, Suckling J, Brammer MJ, David AS (2001) Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts. Cogn Brain Res 12:233–243

    Article  CAS  Google Scholar 

  • Celesia GG (1968) Auditory evoked responses. Intracranial and extracranial average evoked responses. Arch Neurol 19:430–437

    PubMed  CAS  Google Scholar 

  • Chomsky N (1985) The logical structure of linguistic theory. The University of Chicago Press, Chicago

    Google Scholar 

  • Cunningham J, Nicol T, Zecker SG, Bradlow A, Kraus N (2001) Neurobiologic responses to speech in noise in children with learning problems: deficits and strategies for improvement. Clin Neurophysiol 112:758–767

    Article  PubMed  CAS  Google Scholar 

  • Galbraith GC, Arbagey PW, Branski R, Comerci N, Rector PM (1995) Intelligible speech encoded in the human brain stem frequency-following response. Neuroreport 6:2363–2367

    Article  PubMed  CAS  Google Scholar 

  • Gallese V, Fadiga L, Fogassi L, Rizzolatti G (1996) Action recognition in the premotor cortex. Brain 119:593–609

    Article  PubMed  Google Scholar 

  • Gardi J, Merzenich M, McKean C (1979) Origins of the scalp recorded frequency-following response in the cat. Audiology 18:358–381

    PubMed  CAS  Google Scholar 

  • Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    Article  PubMed  CAS  Google Scholar 

  • Grant KW, Seitz PF (2000) The use of visible speech cues for improving auditory detection of spoken sentences. J Acoust Soc Am 108:1197–1208

    Article  PubMed  CAS  Google Scholar 

  • Green KP (1987) The perception of speaking rate using visual information from a talker’s face. Percept Psychophys 42:587–593

    PubMed  CAS  Google Scholar 

  • Hall JWI (1992) Handbook of auditory evoked responses. Allyn and Bacon, Needham Heights

    Google Scholar 

  • Hauser MD, Chomsky N, Fitch WT (2002) The faculty of language: what is it, who has it, and how did it evolve? Science 298:1569–1579

    Article  PubMed  CAS  Google Scholar 

  • Hayes EA, Warrier CM, Nicol TG, Zecker SG, Kraus N (2003) Neural plasticity following auditory training in children with learning problems. Clin Neurophysiol 114:673–684

    Article  PubMed  Google Scholar 

  • Hoormann J, Falkenstein M, Hohnsbein J (2000) Early attention effects in human auditory-evoked potentials. Psychophysiology 37:29–42

    Article  PubMed  CAS  Google Scholar 

  • Hyde PS, Knudsen EI (2001) A topographic instructive signal guides the adjustment of the auditory space map in the optic tectum. J Neurosci 21:8586–8593

    PubMed  CAS  Google Scholar 

  • Jacobson J (1991) The auditory brainstem response. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  • Jiang W, Stein BE (2003) Cortex controls multisensory depression in superior colliculus. J Neurophysiol 90:2123–2135

    Article  PubMed  Google Scholar 

  • Jiang W, Jiang H, Stein BE (2002) Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci Nov 15:1240–1255

    Article  Google Scholar 

  • Johnson KL, Nicol TG, Kraus N (2005) The brainstem response to speech. A biological marker of auditory processing Ear and Haring (in press)

  • Kent RD (1984) Psychobiology of speech development: coemergence of language and a movement system. Am J Physiol 246:R888–R894

    PubMed  CAS  Google Scholar 

  • King C, Warrier CM, Hayes E, Kraus N (2002) Deficits in auditory brainstem pathway encoding of speech sounds in children with learning problems. Neurosci Lett 319:111–115

    Article  PubMed  CAS  Google Scholar 

  • Klucharev V, Sams M (2004) Interaction of gaze direction and facial expressions processing: ERP study. Neuroreport 22:621–625

    Article  Google Scholar 

  • Klucharev V, Mottonen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65–75

    Article  Google Scholar 

  • Kraus N, Nicol T (2005) Brainstem origins for cortical ‘what’ and ‘where’ pathways in the auditory system. Trends Neurosci 28:176–181

    Article  PubMed  CAS  Google Scholar 

  • Linkenkaer-Hansen K, Palva JM, Sams M, Hietanen JK, Aronen HJ, Ilmoniemi RJ (1998) Face-selective processing in human extrastriate cortex around 120 ms after stimulus onset revealed by mag. Neurosci Lett 253:147–150

    Article  PubMed  CAS  Google Scholar 

  • Lu ST, Hämäläinen MS, Hari R, Ilmoniemi RJ, Lounasmaa OV, Sams M, Vilkman V (1991) Seeing faces activates three separate areas outside the occipital visual cortex in man. Neuroscience 43:287–290

    Article  PubMed  CAS  Google Scholar 

  • Møller AR (1999) Neural mechanisms of BAEP. Electroencephalogr Clin Neurophysiol Suppl 49:27–35

    PubMed  Google Scholar 

  • MacDonald J, McGurk H (1978) Visual influences on speech perception processes. Percept Psychophys 24:253–257

    PubMed  CAS  Google Scholar 

  • MacLeod A, Summerfield Q (1987) Quantifying the contribution of vision to speech perception in noise. Br J Audiol 21:131–141

    Article  PubMed  CAS  Google Scholar 

  • Marks LE (1982) Bright sneezes and dark coughs, loud sunlight and soft moonlight. J Exp Psychol Hum Percept Perform 8:177–193

    Article  PubMed  CAS  Google Scholar 

  • Marks LE (2004) Cross-modal interactions in speeded classification. In: Calvert GA, Spence C, Stein BE (eds) The handbook of mutisensory processes. MIT Press, Cambridge, pp 85–106

    Google Scholar 

  • Marsh JT, Brown WS, Smith JC (1975) Far-field recorded frequency-following responses: correlates of low pitch auditory perception in humans. Electroencephalogr Clin Neurophysiol 38:113–119

    Article  PubMed  CAS  Google Scholar 

  • Massaro DW (1998) Perceiving talking faces: from speech perception to a behavioral principle. MIT Press, Cambridge, MA

    Google Scholar 

  • Meredith MA, Stein BE (1986) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662

    PubMed  CAS  Google Scholar 

  • Möttönen R, Krause CM, Tiippana K, Sams M (2002) Processing of changes in visual speech in the human auditory cortex. Cogn Brain Res 13:417–425

    Article  Google Scholar 

  • Nishitani N, Hari R (2002) Viewing lip forms: cortical dynamics. Neuron 19:1211–1220

    Article  Google Scholar 

  • Oatman LC, Anderson BW (1977) Effects of visual attention on tone burst evoked auditory potentials. Exp Neurol 57:200–211

    Article  PubMed  CAS  Google Scholar 

  • Perrault TJ, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90:4022–4026

    Article  PubMed  Google Scholar 

  • Picton TW, Hillyard SA (1974) Human auditory evoked potentials. II. Effects of attention. Electroencephalogr Clin Neurophysiol 36:191–199

    Article  PubMed  CAS  Google Scholar 

  • Russo N, Nicol T, Musacchia G, Kraus N (2004) Brainstem responses to speech syllables. Clin Neurophysiol 115:2021–2030

    Article  PubMed  Google Scholar 

  • Russo NM, Nicol TG, Zecker SG, Hayes EA, Kraus N (2005) Auditory training improves neural timing in the human brainstem. Behav Brain Res 6:95–103

    Article  Google Scholar 

  • Saito DN, Yoshimura K, Kochiyama T, Okada T, Honda M, Sadato N (2005) Cross-modal binding and activated attentional networks during audio-visual speech integration: a functional MRI study. Cereb Cortex 16 (Epub ahead of print)

  • Sams M, Aulanko R, Hämäläinen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141–145

    Article  PubMed  CAS  Google Scholar 

  • Scott SK, Johnsrude IS (2003) The neuroanatomical and functional organization of speech perception. Trends Neurosci 26:100–107

    Article  PubMed  CAS  Google Scholar 

  • Sekiyama K, Tohkura Y (1991) McGurk effect in non-English listeners: few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. J Acoust Soc Am 90:1797–1805

    Article  PubMed  CAS  Google Scholar 

  • Stein BE (1998) Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Exp Brain Res 123:124–135

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Wallace MW, Stanford TR, Jiang W (2002) Cortex governs multisensory integration in the midbrain. Neuroscientist 8:306–314

    Article  PubMed  Google Scholar 

  • Suga N, Ma X (2003) Multiparametric corticofugal modulation and plasticity in the auditory system. Nat Rev Neurosci 4:783–794

    Article  PubMed  CAS  Google Scholar 

  • Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

    Article  Google Scholar 

  • Summerfield Q (1987) Hearing by eye. In Dodd B, Campbell R (eds) Lawrence Erlbaum Associates, Hillsdale, pp 3–51

  • Teder-Salejarvi WA, McDonald JJ, Di Russo F, Hillyard SA (2002) An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Brain Res Cogn Brain Res 14:106–114

    Article  PubMed  CAS  Google Scholar 

  • Tervaniemi M, Hugdahl K (2003) Lateralization of auditory-cortex functions. Brain Res Rev 43:231–246

    Article  PubMed  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior collicuclus. J Neurophysiol 69:1797–1809

    PubMed  CAS  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010

    PubMed  CAS  Google Scholar 

  • Wallace MT, Perrault TJ Jr, Hairston WD, Stein BE (2004) Visual experience is necessary for the development of multisensory integration. J Neurosci 27:9580–9584

    Article  CAS  Google Scholar 

  • Watkins K, Paus T (2004) Modulation of motor excitability during speech perception: the role of Broca’s area. J Cogn Neurosci 16:978–987

    Article  PubMed  Google Scholar 

  • Wible B, Nicol T, Kraus N (2004) Atypical brainstem representation of onset and formant structure of speech sounds in children with language-based learning problems. Biol Psychol 67:299–317

    Article  PubMed  Google Scholar 

  • Wible B, Nicol T, Kraus N (2005) Correlation between brainstem and cortical auditory processes in normal and language-impaired children. Brain 128:417–423

    Article  PubMed  Google Scholar 

  • Woldorff MG, Hillyard SA (1991) Modulation of early auditory processing during selective listening to rapidly presented tones. Electroencephalogr Clin Neurophysiol 79:170–191

    Article  PubMed  CAS  Google Scholar 

  • Woldorff M, Hansen JC, Hillyard SA (1987) Evidence for effects of selective attention in the mid-latency range of the human auditory event-related potential. Electroencephalogr Clin Neurophysiol Suppl 40:146–154

    PubMed  CAS  Google Scholar 

  • Yvert B, Fischer C, Guenot M, Krolak-Salmon P, Isnard J, Pernier J (2002) Simultaneous intracerebral EEG recordings of early auditory thalamic and cortical activity in human. Eur J Neurosci 16:1146–1150

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

NIH R01 DC01510 supported this work. The authors wish to thank their colleagues in the Auditory Neuroscience Laboratory at Northwestern University as well as Dan Zellner and the staff at Northwestern’s Digital Media Studio for their film editing expertise.

Author information

Authors and Affiliations

  1. Auditory Neuroscience Laboratory, Department of Communication Sciences, Northwestern University, 2240 Campus Dr., Evanston, IL, 60208, USA

    Gabriella Musacchia, Trent Nicol & Nina Kraus

  2. Laboratory of Computational Engineering, Helsinki University of Technology, Tekniikantie 14, Espoo (Innopoli II), P.O.Box 9203, 02015, Hut, Finland

    Mikko Sams

  3. Auditory Neuroscience Laboratory, Department of Communication Sciences; Neurobiology and Physiology; Otolaryngology, Northwestern University, 2240 Campus Dr., Evanston, IL, 60208, USA

    Nina Kraus

Authors
  1. Gabriella Musacchia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mikko Sams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Trent Nicol
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nina Kraus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gabriella Musacchia.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Musacchia, G., Sams, M., Nicol, T. et al. Seeing speech affects acoustic information processing in the human brainstem. Exp Brain Res 168, 1–10 (2006). https://doi.org/10.1007/s00221-005-0071-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-005-0071-5

Keywords

Search

Navigation