Brain Structure and Function

, Volume 220, Issue 2, pp 1109–1125

Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users

  • Jae-Jin Song
  • Hyo-Jeong Lee
  • Hyejin Kang
  • Dong Soo Lee
  • Sun O. Chang
  • Seung Ha Oh
Original Article

DOI: 10.1007/s00429-013-0704-6

Cite this article as:
Song, JJ., Lee, HJ., Kang, H. et al. Brain Struct Funct (2015) 220: 1109. doi:10.1007/s00429-013-0704-6


While deafness-induced plasticity has been investigated in the visual and auditory domains, not much is known about language processing in audiovisual multimodal environments for patients with restored hearing via cochlear implant (CI) devices. Here, we examined the effect of agreeing or conflicting visual inputs on auditory processing in deaf patients equipped with degraded artificial hearing. Ten post-lingually deafened CI users with good performance, along with matched control subjects, underwent H215O-positron emission tomography scans while carrying out a behavioral task requiring the extraction of speech information from unimodal auditory stimuli, bimodal audiovisual congruent stimuli, and incongruent stimuli. Regardless of congruency, the control subjects demonstrated activation of the auditory and visual sensory cortices, as well as the superior temporal sulcus, the classical multisensory integration area, indicating a bottom-up multisensory processing strategy. Compared to CI users, the control subjects exhibited activation of the right ventral premotor-supramarginal pathway. In contrast, CI users activated primarily the visual cortices more in the congruent audiovisual condition than in the null condition. In addition, compared to controls, CI users displayed an activation focus in the right amygdala for congruent audiovisual stimuli. The most notable difference between the two groups was an activation focus in the left inferior frontal gyrus in CI users confronted with incongruent audiovisual stimuli, suggesting top-down cognitive modulation for audiovisual conflict. Correlation analysis revealed that good speech performance was positively correlated with right amygdala activity for the congruent condition, but negatively correlated with bilateral visual cortices regardless of congruency. Taken together these results suggest that for multimodal inputs, cochlear implant users are more vision-reliant when processing congruent stimuli and are disturbed more by visual distractors when confronted with incongruent audiovisual stimuli. To cope with this multimodal conflict, CI users activate the left inferior frontal gyrus to adopt a top-down cognitive modulation pathway, whereas normal hearing individuals primarily adopt a bottom-up strategy.


Cochlear implantDeafnessPositron emission tomographyAudiovisualPlasticity



Normal hearing




Superior temporal sulcus


Inferior frontal gyrus


Cochlear implant


Positron emission tomography


Magnetic resonance imaging


Categories of auditory performance


Middle temporal gyrus


Inferior temporal gyrus


Regional cerebral blood flow


Ventral premotor cortex


Supramarginal gyrus


Superior frontal gyrus


Medial frontal gyrus

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Jae-Jin Song
    • 1
  • Hyo-Jeong Lee
    • 2
    • 3
  • Hyejin Kang
    • 4
  • Dong Soo Lee
    • 4
  • Sun O. Chang
    • 3
    • 5
  • Seung Ha Oh
    • 3
    • 5
  1. 1.Department of Otorhinolaryngology-Head and Neck SurgerySeoul National University Bundang HospitalSeongnamKorea
  2. 2.Department of Otorhinolaryngology-Head and Neck SurgeryHallym University College of MedicineChun-CheonKorea
  3. 3.Sensory Organ Research Institute, Seoul National University Medical Research CenterSeoulKorea
  4. 4.Department of Nuclear MedicineSeoul National University HospitalSeoulKorea
  5. 5.Department of Otorhinolaryngology-Head and Neck SurgerySeoul National University HospitalSeoulKorea