Skip to main content
SpringerLink
Log in

Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users

  • Original Article
  • Published:
Brain Structure and Function Aims and scope Submit manuscript

Abstract

While deafness-induced plasticity has been investigated in the visual and auditory domains, not much is known about language processing in audiovisual multimodal environments for patients with restored hearing via cochlear implant (CI) devices. Here, we examined the effect of agreeing or conflicting visual inputs on auditory processing in deaf patients equipped with degraded artificial hearing. Ten post-lingually deafened CI users with good performance, along with matched control subjects, underwent H 152 O-positron emission tomography scans while carrying out a behavioral task requiring the extraction of speech information from unimodal auditory stimuli, bimodal audiovisual congruent stimuli, and incongruent stimuli. Regardless of congruency, the control subjects demonstrated activation of the auditory and visual sensory cortices, as well as the superior temporal sulcus, the classical multisensory integration area, indicating a bottom-up multisensory processing strategy. Compared to CI users, the control subjects exhibited activation of the right ventral premotor-supramarginal pathway. In contrast, CI users activated primarily the visual cortices more in the congruent audiovisual condition than in the null condition. In addition, compared to controls, CI users displayed an activation focus in the right amygdala for congruent audiovisual stimuli. The most notable difference between the two groups was an activation focus in the left inferior frontal gyrus in CI users confronted with incongruent audiovisual stimuli, suggesting top-down cognitive modulation for audiovisual conflict. Correlation analysis revealed that good speech performance was positively correlated with right amygdala activity for the congruent condition, but negatively correlated with bilateral visual cortices regardless of congruency. Taken together these results suggest that for multimodal inputs, cochlear implant users are more vision-reliant when processing congruent stimuli and are disturbed more by visual distractors when confronted with incongruent audiovisual stimuli. To cope with this multimodal conflict, CI users activate the left inferior frontal gyrus to adopt a top-down cognitive modulation pathway, whereas normal hearing individuals primarily adopt a bottom-up strategy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

NH:

Normal hearing

AV:

Audiovisual

STS:

Superior temporal sulcus

IFG:

Inferior frontal gyrus

CI:

Cochlear implant

PET:

Positron emission tomography

MRI:

Magnetic resonance imaging

CAP:

Categories of auditory performance

MTG:

Middle temporal gyrus

ITG:

Inferior temporal gyrus

rCBF:

Regional cerebral blood flow

vPMC:

Ventral premotor cortex

SMG:

Supramarginal gyrus

SFG:

Superior frontal gyrus

MeFG:

Medial frontal gyrus

References

  • Aboitiz F, Garcia R (2009) Merging of phonological and gestural circuits in early language evolution. Rev Neurosci 20:71–84

    PubMed  Google Scholar 

  • Adam R, Noppeney U (2010) Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex. Neuroimage 52:1592–1602

    Article  PubMed  Google Scholar 

  • Archbold S, Lutman ME, Marshall DH (1995) Categories of auditory performance. Ann Otol Rhinol Laryngol Suppl 166:312–314

    CAS  PubMed  Google Scholar 

  • Aron AR, Fletcher PC, Bullmore ET, Sahakian BJ, Robbins TW (2003) Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans. Nat Neurosci 6:115–116

    Article  CAS  PubMed  Google Scholar 

  • Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113

    Article  PubMed Central  PubMed  Google Scholar 

  • Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192

    Article  CAS  PubMed  Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823

    Article  CAS  PubMed  Google Scholar 

  • Beauchamp MS, Nath AR, Pasalar S (2010) fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J Neurosci 30:2414–2417

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008) Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39:423–435

    Article  PubMed Central  PubMed  Google Scholar 

  • Bitan T, Lifshitz A, Breznitz Z, Booth JR (2010) Bidirectional connectivity between hemispheres occurs at multiple levels in language processing but depends on sex. J Neurosci 30:11576–11585

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Bunge SA, Dudukovic NM, Thomason ME, Vaidya CJ, Gabrieli JD (2002) Immature frontal lobe contributions to cognitive control in children: evidence from fMRI. Neuron 33:301–311

    Article  CAS  PubMed  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

    Article  CAS  PubMed  Google Scholar 

  • Champoux F, Lepore F, Gagne JP, Theoret H (2009) Visual stimuli can impair auditory processing in cochlear implant users. Neuropsychologia 47:17–22

    Article  PubMed  Google Scholar 

  • Corbetta M, Shulman GL (2002) Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci 3:201–215

    Article  CAS  PubMed  Google Scholar 

  • Doucet ME, Bergeron F, Lassonde M, Ferron P, Lepore F (2006) Cross-modal reorganization and speech perception in cochlear implant users. Brain 129:3376–3383

    Article  CAS  PubMed  Google Scholar 

  • Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K (2005) A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25:1325–1335

    Article  PubMed  Google Scholar 

  • Finney EM, Fine I, Dobkins KR (2001) Visual stimuli activate auditory cortex in the deaf. Nat Neurosci 4:1171–1173

    Article  CAS  PubMed  Google Scholar 

  • Fletcher PC, Henson RN (2001) Frontal lobes and human memory: insights from functional neuroimaging. Brain 124:849–881

    Article  CAS  PubMed  Google Scholar 

  • Garavan H, Ross TJ, Stein EA (1999) Right hemispheric dominance of inhibitory control: an event-related functional MRI study. Proc Natl Acad Sci USA 96:8301–8306

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Genovese CR, Lazar NA, Nichols T (2002) Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15:870–878

    Article  PubMed  Google Scholar 

  • Giraud AL, Lee HJ (2007) Predicting cochlear implant outcome from brain organisation in the deaf. Restor Neurol Neurosci 25:381–390

    PubMed  Google Scholar 

  • Giraud AL, Truy E (2002) The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects. Neuropsychologia 40:1562–1569

    Article  PubMed  Google Scholar 

  • Giraud AL, Price CJ, Graham JM, Truy E, Frackowiak RS (2001) Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron 30:657–663

    Article  CAS  PubMed  Google Scholar 

  • Goldberg RF, Perfetti CA, Fiez JA, Schneider W (2007) Selective retrieval of abstract semantic knowledge in left prefrontal cortex. J Neurosci 27:3790–3798

    Article  CAS  PubMed  Google Scholar 

  • Guenther FH, Ghosh SS, Tourville JA (2006) Neural modeling and imaging of the cortical interactions underlying syllable production. Brain Lang 96:280–301

    Article  PubMed Central  PubMed  Google Scholar 

  • Haddon JE, Killcross S (2006) Prefrontal cortex lesions disrupt the contextual control of response conflict. J Neurosci 26:2933–2940

    Article  CAS  PubMed  Google Scholar 

  • Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887

    Article  CAS  PubMed  Google Scholar 

  • Jones JA, Callan DE (2003) Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. Neuroreport 14:1129–1133

    Article  PubMed  Google Scholar 

  • Kempe V, Thoresen JC, Kirk NW, Schaeffler F, Brooks PJ (2012) Individual differences in the discrimination of novel speech sounds: effects of sex, temporal processing, musical and cognitive abilities. PLoS One 7:e48623

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Koechlin E, Hyafil A (2007) Anterior prefrontal function and the limits of human decision-making. Science 318:594–598

    Article  CAS  PubMed  Google Scholar 

  • Koechlin E, Ody C, Kouneiher F (2003) The architecture of cognitive control in the human prefrontal cortex. Science 302:1181–1185

    Article  CAS  PubMed  Google Scholar 

  • Koles ZJ, Lind JC, Flor-Henry P (2010) Gender differences in brain functional organization during verbal and spatial cognitive challenges. Brain Topogr 23:199–204

    Article  PubMed  Google Scholar 

  • Kral A, O’Donoghue GM (2010) Profound deafness in childhood. N Engl J Med 363:1438–1450

    Article  CAS  PubMed  Google Scholar 

  • Kreifelts B, Ethofer T, Huberle E, Grodd W, Wildgruber D (2010) Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Hum Brain Mapp 31:979–991

    Article  PubMed  Google Scholar 

  • Lazard DS, Lee HJ, Gaebler M, Kell CA, Truy E, Giraud AL (2010) Phonological processing in post-lingual deafness and cochlear implant outcome. Neuroimage 49:3443–3451

    Article  CAS  PubMed  Google Scholar 

  • Lazard DS, Giraud AL, Truy E, Lee HJ (2011) Evolution of non-speech sound memory in postlingual deafness: implications for cochlear implant rehabilitation. Neuropsychologia 49:2475–2482

    Article  CAS  PubMed  Google Scholar 

  • Lazard DS, Giraud AL, Gnansia D, Meyer B, Sterkers O (2012) Understanding the deafened brain: implications for cochlear implant rehabilitation. Eur Ann Otorhinolaryngol Head Neck Dis 129:98–103

    Article  CAS  PubMed  Google Scholar 

  • Lazard DS, Lee HJ, Truy E, Giraud AL (2013) Bilateral reorganization of posterior temporal cortices in post-lingual deafness and its relation to cochlear implant outcome. Hum Brain Mapp 34:1208–1219

    Google Scholar 

  • Lee DS, Lee JS, Oh SH, Kim SK, Kim JW, Chung JK, Lee MC, Kim CS (2001) Cross-modal plasticity and cochlear implants. Nature 409:149–150

    Article  CAS  PubMed  Google Scholar 

  • Lee JS, Lee DS, Oh SH, Kim CS, Kim JW, Hwang CH, Koo J, Kang E, Chung JK, Lee MC (2003) PET evidence of neuroplasticity in adult auditory cortex of postlingual deafness. J Nucl Med 44:1435–1439

    PubMed  Google Scholar 

  • Lee HJ, Giraud AL, Kang E, Oh SH, Kang H, Kim CS, Lee DS (2007a) Cortical activity at rest predicts cochlear implantation outcome. Cereb Cortex 17:909–917

    Article  PubMed  Google Scholar 

  • Lee HJ, Truy E, Mamou G, Sappey-Marinier D, Giraud AL (2007b) Visual speech circuits in profound acquired deafness: a possible role for latent multimodal connectivity. Brain 130:2929–2941

    Article  PubMed  Google Scholar 

  • Marquis JP, Killcross S, Haddon JE (2007) Inactivation of the prelimbic, but not infralimbic, prefrontal cortex impairs the contextual control of response conflict in rats. Eur J Neurosci 25:559–566

    Article  PubMed  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    Article  CAS  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391

    Article  CAS  PubMed  Google Scholar 

  • Mesulam MM (1998) From sensation to cognition. Brain 121(Pt 6):1013–1052

    Article  PubMed  Google Scholar 

  • Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884–5893

    Article  CAS  PubMed  Google Scholar 

  • Molenberghs P, Brander C, Mattingley JB, Cunnington R (2010) The role of the superior temporal sulcus and the mirror neuron system in imitation. Hum Brain Mapp 31:1316–1326

    Article  PubMed  Google Scholar 

  • Nath AR, Beauchamp MS (2011) Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. J Neurosci 31:1704–1714

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Nath AR, Beauchamp MS (2012) A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage 59:781–787

    Article  PubMed Central  PubMed  Google Scholar 

  • Nishitani N, Hari R (2002) Viewing lip forms: cortical dynamics. Neuron 36:1211–1220

    Article  CAS  PubMed  Google Scholar 

  • Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18:598–609

    Article  PubMed  Google Scholar 

  • Park H, Kang E, Kang H, Kim JS, Jensen O, Chung CK, Lee DS (2011) Cross-frequency power correlations reveal the right superior temporal gyrus as a hub region during working memory maintenance. Brain Connect 1:460–472

    Article  PubMed  Google Scholar 

  • Rauschecker JP (1999) Auditory cortical plasticity: a comparison with other sensory systems. Trends Neurosci 22:74–80

    Article  CAS  PubMed  Google Scholar 

  • Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147–1153

    Article  PubMed  Google Scholar 

  • Rouger J, Lagleyre S, Fraysse B, Deneve S, Deguine O, Barone P (2007) Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc Natl Acad Sci USA 104:7295–7300

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Rouger J, Fraysse B, Deguine O, Barone P (2008) McGurk effects in cochlear-implanted deaf subjects. Brain Res 1188:87–99

    Article  CAS  PubMed  Google Scholar 

  • Rouger J, Lagleyre S, Demonet JF, Fraysse B, Deguine O, Barone P (2012) Evolution of crossmodal reorganization of the voice area in cochlear-implanted deaf patients. Hum Brain Mapp 33:1929–1940

    Article  PubMed  Google Scholar 

  • Ruff DA, Marrett S, Heekeren HR, Bandettini PA, Ungerleider LG (2010) Complementary roles of systems representing sensory evidence and systems detecting task difficulty during perceptual decision making. Front Neurosci 4:190

    Article  PubMed Central  PubMed  Google Scholar 

  • Schaefer A, Braver TS, Reynolds JR, Burgess GC, Yarkoni T, Gray JR (2006) Individual differences in amygdala activity predict response speed during working memory. J Neurosci 26:10120–10128

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Skipper JI, Nusbaum HC, Small SL (2005) Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25:76–89

    Article  PubMed  Google Scholar 

  • Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2007) Speech-associated gestures, Broca’s area, and the human mirror system. Brain Lang 101:260–277

    Article  PubMed Central  PubMed  Google Scholar 

  • Song JJ, Choi HG, Oh SH, Chang SO, Kim CS, Lee JH (2009) Unilateral sensorineural hearing loss in children: the importance of temporal bone computed tomography and audiometric follow-up. Otol Neurotol 30:604–608

    Article  PubMed  Google Scholar 

  • Song JJ, De Ridder D, Van de Heyning P, Vanneste S (2012a) Mapping tinnitus-related brain activation: an activation-likelihood estimation metaanalysis of PET studies. J Nucl Med 53:1550–1557

    Article  PubMed  Google Scholar 

  • Song JJ, Yoo YT, An YH, Yoo JC, Kim JS, Koo JW (2012b) Comorbid benign paroxysmal positional vertigo in idiopathic sudden sensorineural hearing loss: an ominous sign for hearing recovery. Otol Neurotol 33:137–141

    Article  CAS  PubMed  Google Scholar 

  • Song JJ, Punte AK, De Ridder D, Vanneste S, Van de Heyning P (2013a) Neural substrates predicting improvement of tinnitus after cochlear implantation in patients with single-sided deafness. Hear Res 299:1–9

    Article  PubMed  Google Scholar 

  • Song JJ, Vanneste S, Schlee W, Van de Heyning P, De Ridder D (2013b) Onset-related differences in neural substrates of tinnitus-related distress: the anterior cingulate cortex in late-onset tinnitus, and the frontal cortex in early-onset tinnitus. Brain Struct Funct [Epub ahead of print]

  • Strelnikov K, Rosito M, Barone P (2011) Effect of audiovisual training on monaural spatial hearing in horizontal plane. PLoS One 6:e18344

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Strelnikov K, Rouger J, Demonet JF, Lagleyre S, Fraysse B, Deguine O, Barone P (2013) Visual activity predicts auditory recovery from deafness after adult cochlear implantation. Brain 136:3682–3695

    Google Scholar 

  • Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

    Article  Google Scholar 

  • Szycik GR, Jansma H, Munte TF (2009) Audiovisual integration during speech comprehension: an fMRI study comparing ROI-based and whole brain analyses. Hum Brain Mapp 30:1990–1999

    Article  PubMed  Google Scholar 

  • Talairach J, Tornoux P (1988) Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: An approach to cerebral imaging. Georg Thieme, Stuttgart

    Google Scholar 

  • Talsma D, Doty TJ, Woldorff MG (2007) Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb Cortex 17:679–690

    Article  PubMed  Google Scholar 

  • Tremblay C, Champoux F, Lepore F, Theoret H (2010) Audiovisual fusion and cochlear implant proficiency. Restor Neurol Neurosci 28:283–291

    PubMed  Google Scholar 

  • van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282

    Article  PubMed  Google Scholar 

  • van Atteveldt NM, Formisano E, Goebel R, Blomert L (2007) Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 36:1345–1360

    Article  PubMed  Google Scholar 

  • van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186

    Article  PubMed Central  PubMed  Google Scholar 

  • Vanneste S, Plazier M, der Loo E, de Heyning PV, Congedo M, De Ridder D (2010) The neural correlates of tinnitus-related distress. Neuroimage 52:470–480

    Article  PubMed  Google Scholar 

  • Vigneau M, Beaucousin V, Herve PY, Duffau H, Crivello F, Houde O, Mazoyer B, Tzourio-Mazoyer N (2006) Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. Neuroimage 30:1414–1432

    Article  CAS  PubMed  Google Scholar 

  • Werner S, Noppeney U (2010a) Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci 30:2662–2675

    Article  CAS  PubMed  Google Scholar 

  • Werner S, Noppeney U (2010b) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex 20:1829–1842

    Article  PubMed  Google Scholar 

  • Yun RJ, Krystal JH, Mathalon DH (2010) Working memory overload: fronto-limbic interactions and effects on subsequent working memory function. Brain Imaging Behav 4:96–108

    Article  PubMed Central  PubMed  Google Scholar 

Download references

Acknowledgments

The authors thank Dr. Yong-Hwi Ahn for his support on the manuscript. Also, the first author thanks Dr. DY Yoon for giving precious support to the study. This work was supported by Korean government (MOST) [Korea Science and Engineering Foundation (KOSEF) (no. 2012-0030102)].

Author information

Authors and Affiliations

  1. Department of Otorhinolaryngology-Head and Neck Surgery, Seoul National University Bundang Hospital, Seongnam, Korea

    Jae-Jin Song

  2. Department of Otorhinolaryngology-Head and Neck Surgery, Hallym University College of Medicine, Chun-Cheon, Korea

    Hyo-Jeong Lee

  3. Sensory Organ Research Institute, Seoul National University Medical Research Center, Seoul, Korea

    Hyo-Jeong Lee, Sun O. Chang & Seung Ha Oh

  4. Department of Nuclear Medicine, Seoul National University Hospital, Seoul, Korea

    Hyejin Kang & Dong Soo Lee

  5. Department of Otorhinolaryngology-Head and Neck Surgery, Seoul National University Hospital, 101 Daehak-Ro Jongno-Gu, Seoul, 110-744, Korea

    Sun O. Chang & Seung Ha Oh

Authors
  1. Jae-Jin Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyo-Jeong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hyejin Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dong Soo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sun O. Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Seung Ha Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seung Ha Oh.

Additional information

Jae-Jin Song and Hyo-Jeong Lee contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Song, JJ., Lee, HJ., Kang, H. et al. Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users. Brain Struct Funct 220, 1109–1125 (2015). https://doi.org/10.1007/s00429-013-0704-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00429-013-0704-6

Keywords

Search

Navigation