Brain Structure and Function

, Volume 220, Issue 2, pp 1109–1125

Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users

  • Jae-Jin Song
  • Hyo-Jeong Lee
  • Hyejin Kang
  • Dong Soo Lee
  • Sun O. Chang
  • Seung Ha Oh
Original Article


While deafness-induced plasticity has been investigated in the visual and auditory domains, not much is known about language processing in audiovisual multimodal environments for patients with restored hearing via cochlear implant (CI) devices. Here, we examined the effect of agreeing or conflicting visual inputs on auditory processing in deaf patients equipped with degraded artificial hearing. Ten post-lingually deafened CI users with good performance, along with matched control subjects, underwent H215O-positron emission tomography scans while carrying out a behavioral task requiring the extraction of speech information from unimodal auditory stimuli, bimodal audiovisual congruent stimuli, and incongruent stimuli. Regardless of congruency, the control subjects demonstrated activation of the auditory and visual sensory cortices, as well as the superior temporal sulcus, the classical multisensory integration area, indicating a bottom-up multisensory processing strategy. Compared to CI users, the control subjects exhibited activation of the right ventral premotor-supramarginal pathway. In contrast, CI users activated primarily the visual cortices more in the congruent audiovisual condition than in the null condition. In addition, compared to controls, CI users displayed an activation focus in the right amygdala for congruent audiovisual stimuli. The most notable difference between the two groups was an activation focus in the left inferior frontal gyrus in CI users confronted with incongruent audiovisual stimuli, suggesting top-down cognitive modulation for audiovisual conflict. Correlation analysis revealed that good speech performance was positively correlated with right amygdala activity for the congruent condition, but negatively correlated with bilateral visual cortices regardless of congruency. Taken together these results suggest that for multimodal inputs, cochlear implant users are more vision-reliant when processing congruent stimuli and are disturbed more by visual distractors when confronted with incongruent audiovisual stimuli. To cope with this multimodal conflict, CI users activate the left inferior frontal gyrus to adopt a top-down cognitive modulation pathway, whereas normal hearing individuals primarily adopt a bottom-up strategy.


Cochlear implant Deafness Positron emission tomography Audiovisual Plasticity 



Normal hearing




Superior temporal sulcus


Inferior frontal gyrus


Cochlear implant


Positron emission tomography


Magnetic resonance imaging


Categories of auditory performance


Middle temporal gyrus


Inferior temporal gyrus


Regional cerebral blood flow


Ventral premotor cortex


Supramarginal gyrus


Superior frontal gyrus


Medial frontal gyrus


  1. Aboitiz F, Garcia R (2009) Merging of phonological and gestural circuits in early language evolution. Rev Neurosci 20:71–84PubMedGoogle Scholar
  2. Adam R, Noppeney U (2010) Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex. Neuroimage 52:1592–1602CrossRefPubMedGoogle Scholar
  3. Archbold S, Lutman ME, Marshall DH (1995) Categories of auditory performance. Ann Otol Rhinol Laryngol Suppl 166:312–314PubMedGoogle Scholar
  4. Aron AR, Fletcher PC, Bullmore ET, Sahakian BJ, Robbins TW (2003) Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans. Nat Neurosci 6:115–116CrossRefPubMedGoogle Scholar
  5. Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113CrossRefPubMedCentralPubMedGoogle Scholar
  6. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192CrossRefPubMedGoogle Scholar
  7. Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823CrossRefPubMedGoogle Scholar
  8. Beauchamp MS, Nath AR, Pasalar S (2010) fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J Neurosci 30:2414–2417CrossRefPubMedCentralPubMedGoogle Scholar
  9. Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008) Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39:423–435CrossRefPubMedCentralPubMedGoogle Scholar
  10. Bitan T, Lifshitz A, Breznitz Z, Booth JR (2010) Bidirectional connectivity between hemispheres occurs at multiple levels in language processing but depends on sex. J Neurosci 30:11576–11585CrossRefPubMedCentralPubMedGoogle Scholar
  11. Bunge SA, Dudukovic NM, Thomason ME, Vaidya CJ, Gabrieli JD (2002) Immature frontal lobe contributions to cognitive control in children: evidence from fMRI. Neuron 33:301–311CrossRefPubMedGoogle Scholar
  12. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657CrossRefPubMedGoogle Scholar
  13. Champoux F, Lepore F, Gagne JP, Theoret H (2009) Visual stimuli can impair auditory processing in cochlear implant users. Neuropsychologia 47:17–22CrossRefPubMedGoogle Scholar
  14. Corbetta M, Shulman GL (2002) Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci 3:201–215CrossRefPubMedGoogle Scholar
  15. Doucet ME, Bergeron F, Lassonde M, Ferron P, Lepore F (2006) Cross-modal reorganization and speech perception in cochlear implant users. Brain 129:3376–3383CrossRefPubMedGoogle Scholar
  16. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K (2005) A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25:1325–1335CrossRefPubMedGoogle Scholar
  17. Finney EM, Fine I, Dobkins KR (2001) Visual stimuli activate auditory cortex in the deaf. Nat Neurosci 4:1171–1173CrossRefPubMedGoogle Scholar
  18. Fletcher PC, Henson RN (2001) Frontal lobes and human memory: insights from functional neuroimaging. Brain 124:849–881CrossRefPubMedGoogle Scholar
  19. Garavan H, Ross TJ, Stein EA (1999) Right hemispheric dominance of inhibitory control: an event-related functional MRI study. Proc Natl Acad Sci USA 96:8301–8306CrossRefPubMedCentralPubMedGoogle Scholar
  20. Genovese CR, Lazar NA, Nichols T (2002) Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15:870–878CrossRefPubMedGoogle Scholar
  21. Giraud AL, Lee HJ (2007) Predicting cochlear implant outcome from brain organisation in the deaf. Restor Neurol Neurosci 25:381–390PubMedGoogle Scholar
  22. Giraud AL, Truy E (2002) The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects. Neuropsychologia 40:1562–1569CrossRefPubMedGoogle Scholar
  23. Giraud AL, Price CJ, Graham JM, Truy E, Frackowiak RS (2001) Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron 30:657–663CrossRefPubMedGoogle Scholar
  24. Goldberg RF, Perfetti CA, Fiez JA, Schneider W (2007) Selective retrieval of abstract semantic knowledge in left prefrontal cortex. J Neurosci 27:3790–3798CrossRefPubMedGoogle Scholar
  25. Guenther FH, Ghosh SS, Tourville JA (2006) Neural modeling and imaging of the cortical interactions underlying syllable production. Brain Lang 96:280–301CrossRefPubMedCentralPubMedGoogle Scholar
  26. Haddon JE, Killcross S (2006) Prefrontal cortex lesions disrupt the contextual control of response conflict. J Neurosci 26:2933–2940CrossRefPubMedGoogle Scholar
  27. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887CrossRefPubMedGoogle Scholar
  28. Jones JA, Callan DE (2003) Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. Neuroreport 14:1129–1133CrossRefPubMedGoogle Scholar
  29. Kempe V, Thoresen JC, Kirk NW, Schaeffler F, Brooks PJ (2012) Individual differences in the discrimination of novel speech sounds: effects of sex, temporal processing, musical and cognitive abilities. PLoS One 7:e48623CrossRefPubMedCentralPubMedGoogle Scholar
  30. Koechlin E, Hyafil A (2007) Anterior prefrontal function and the limits of human decision-making. Science 318:594–598CrossRefPubMedGoogle Scholar
  31. Koechlin E, Ody C, Kouneiher F (2003) The architecture of cognitive control in the human prefrontal cortex. Science 302:1181–1185CrossRefPubMedGoogle Scholar
  32. Koles ZJ, Lind JC, Flor-Henry P (2010) Gender differences in brain functional organization during verbal and spatial cognitive challenges. Brain Topogr 23:199–204CrossRefPubMedGoogle Scholar
  33. Kral A, O’Donoghue GM (2010) Profound deafness in childhood. N Engl J Med 363:1438–1450CrossRefPubMedGoogle Scholar
  34. Kreifelts B, Ethofer T, Huberle E, Grodd W, Wildgruber D (2010) Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Hum Brain Mapp 31:979–991CrossRefPubMedGoogle Scholar
  35. Lazard DS, Lee HJ, Gaebler M, Kell CA, Truy E, Giraud AL (2010) Phonological processing in post-lingual deafness and cochlear implant outcome. Neuroimage 49:3443–3451CrossRefPubMedGoogle Scholar
  36. Lazard DS, Giraud AL, Truy E, Lee HJ (2011) Evolution of non-speech sound memory in postlingual deafness: implications for cochlear implant rehabilitation. Neuropsychologia 49:2475–2482CrossRefPubMedGoogle Scholar
  37. Lazard DS, Giraud AL, Gnansia D, Meyer B, Sterkers O (2012) Understanding the deafened brain: implications for cochlear implant rehabilitation. Eur Ann Otorhinolaryngol Head Neck Dis 129:98–103CrossRefPubMedGoogle Scholar
  38. Lazard DS, Lee HJ, Truy E, Giraud AL (2013) Bilateral reorganization of posterior temporal cortices in post-lingual deafness and its relation to cochlear implant outcome. Hum Brain Mapp 34:1208–1219Google Scholar
  39. Lee DS, Lee JS, Oh SH, Kim SK, Kim JW, Chung JK, Lee MC, Kim CS (2001) Cross-modal plasticity and cochlear implants. Nature 409:149–150CrossRefPubMedGoogle Scholar
  40. Lee JS, Lee DS, Oh SH, Kim CS, Kim JW, Hwang CH, Koo J, Kang E, Chung JK, Lee MC (2003) PET evidence of neuroplasticity in adult auditory cortex of postlingual deafness. J Nucl Med 44:1435–1439PubMedGoogle Scholar
  41. Lee HJ, Giraud AL, Kang E, Oh SH, Kang H, Kim CS, Lee DS (2007a) Cortical activity at rest predicts cochlear implantation outcome. Cereb Cortex 17:909–917CrossRefPubMedGoogle Scholar
  42. Lee HJ, Truy E, Mamou G, Sappey-Marinier D, Giraud AL (2007b) Visual speech circuits in profound acquired deafness: a possible role for latent multimodal connectivity. Brain 130:2929–2941CrossRefPubMedGoogle Scholar
  43. Marquis JP, Killcross S, Haddon JE (2007) Inactivation of the prelimbic, but not infralimbic, prefrontal cortex impairs the contextual control of response conflict in rats. Eur J Neurosci 25:559–566CrossRefPubMedGoogle Scholar
  44. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748CrossRefPubMedGoogle Scholar
  45. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391CrossRefPubMedGoogle Scholar
  46. Mesulam MM (1998) From sensation to cognition. Brain 121(Pt 6):1013–1052CrossRefPubMedGoogle Scholar
  47. Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884–5893CrossRefPubMedGoogle Scholar
  48. Molenberghs P, Brander C, Mattingley JB, Cunnington R (2010) The role of the superior temporal sulcus and the mirror neuron system in imitation. Hum Brain Mapp 31:1316–1326CrossRefPubMedGoogle Scholar
  49. Nath AR, Beauchamp MS (2011) Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. J Neurosci 31:1704–1714CrossRefPubMedCentralPubMedGoogle Scholar
  50. Nath AR, Beauchamp MS (2012) A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage 59:781–787CrossRefPubMedCentralPubMedGoogle Scholar
  51. Nishitani N, Hari R (2002) Viewing lip forms: cortical dynamics. Neuron 36:1211–1220CrossRefPubMedGoogle Scholar
  52. Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18:598–609CrossRefPubMedGoogle Scholar
  53. Park H, Kang E, Kang H, Kim JS, Jensen O, Chung CK, Lee DS (2011) Cross-frequency power correlations reveal the right superior temporal gyrus as a hub region during working memory maintenance. Brain Connect 1:460–472CrossRefPubMedGoogle Scholar
  54. Rauschecker JP (1999) Auditory cortical plasticity: a comparison with other sensory systems. Trends Neurosci 22:74–80CrossRefPubMedGoogle Scholar
  55. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147–1153CrossRefPubMedGoogle Scholar
  56. Rouger J, Lagleyre S, Fraysse B, Deneve S, Deguine O, Barone P (2007) Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc Natl Acad Sci USA 104:7295–7300CrossRefPubMedCentralPubMedGoogle Scholar
  57. Rouger J, Fraysse B, Deguine O, Barone P (2008) McGurk effects in cochlear-implanted deaf subjects. Brain Res 1188:87–99CrossRefPubMedGoogle Scholar
  58. Rouger J, Lagleyre S, Demonet JF, Fraysse B, Deguine O, Barone P (2012) Evolution of crossmodal reorganization of the voice area in cochlear-implanted deaf patients. Hum Brain Mapp 33:1929–1940CrossRefPubMedGoogle Scholar
  59. Ruff DA, Marrett S, Heekeren HR, Bandettini PA, Ungerleider LG (2010) Complementary roles of systems representing sensory evidence and systems detecting task difficulty during perceptual decision making. Front Neurosci 4:190CrossRefPubMedCentralPubMedGoogle Scholar
  60. Schaefer A, Braver TS, Reynolds JR, Burgess GC, Yarkoni T, Gray JR (2006) Individual differences in amygdala activity predict response speed during working memory. J Neurosci 26:10120–10128CrossRefPubMedCentralPubMedGoogle Scholar
  61. Skipper JI, Nusbaum HC, Small SL (2005) Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25:76–89CrossRefPubMedGoogle Scholar
  62. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2007) Speech-associated gestures, Broca’s area, and the human mirror system. Brain Lang 101:260–277CrossRefPubMedCentralPubMedGoogle Scholar
  63. Song JJ, Choi HG, Oh SH, Chang SO, Kim CS, Lee JH (2009) Unilateral sensorineural hearing loss in children: the importance of temporal bone computed tomography and audiometric follow-up. Otol Neurotol 30:604–608CrossRefPubMedGoogle Scholar
  64. Song JJ, De Ridder D, Van de Heyning P, Vanneste S (2012a) Mapping tinnitus-related brain activation: an activation-likelihood estimation metaanalysis of PET studies. J Nucl Med 53:1550–1557CrossRefPubMedGoogle Scholar
  65. Song JJ, Yoo YT, An YH, Yoo JC, Kim JS, Koo JW (2012b) Comorbid benign paroxysmal positional vertigo in idiopathic sudden sensorineural hearing loss: an ominous sign for hearing recovery. Otol Neurotol 33:137–141CrossRefPubMedGoogle Scholar
  66. Song JJ, Punte AK, De Ridder D, Vanneste S, Van de Heyning P (2013a) Neural substrates predicting improvement of tinnitus after cochlear implantation in patients with single-sided deafness. Hear Res 299:1–9CrossRefPubMedGoogle Scholar
  67. Song JJ, Vanneste S, Schlee W, Van de Heyning P, De Ridder D (2013b) Onset-related differences in neural substrates of tinnitus-related distress: the anterior cingulate cortex in late-onset tinnitus, and the frontal cortex in early-onset tinnitus. Brain Struct Funct [Epub ahead of print]Google Scholar
  68. Strelnikov K, Rosito M, Barone P (2011) Effect of audiovisual training on monaural spatial hearing in horizontal plane. PLoS One 6:e18344CrossRefPubMedCentralPubMedGoogle Scholar
  69. Strelnikov K, Rouger J, Demonet JF, Lagleyre S, Fraysse B, Deguine O, Barone P (2013) Visual activity predicts auditory recovery from deafness after adult cochlear implantation. Brain 136:3682–3695Google Scholar
  70. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215CrossRefGoogle Scholar
  71. Szycik GR, Jansma H, Munte TF (2009) Audiovisual integration during speech comprehension: an fMRI study comparing ROI-based and whole brain analyses. Hum Brain Mapp 30:1990–1999CrossRefPubMedGoogle Scholar
  72. Talairach J, Tornoux P (1988) Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: An approach to cerebral imaging. Georg Thieme, StuttgartGoogle Scholar
  73. Talsma D, Doty TJ, Woldorff MG (2007) Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb Cortex 17:679–690CrossRefPubMedGoogle Scholar
  74. Tremblay C, Champoux F, Lepore F, Theoret H (2010) Audiovisual fusion and cochlear implant proficiency. Restor Neurol Neurosci 28:283–291PubMedGoogle Scholar
  75. van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282CrossRefPubMedGoogle Scholar
  76. van Atteveldt NM, Formisano E, Goebel R, Blomert L (2007) Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 36:1345–1360CrossRefPubMedGoogle Scholar
  77. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186CrossRefPubMedCentralPubMedGoogle Scholar
  78. Vanneste S, Plazier M, der Loo E, de Heyning PV, Congedo M, De Ridder D (2010) The neural correlates of tinnitus-related distress. Neuroimage 52:470–480CrossRefPubMedGoogle Scholar
  79. Vigneau M, Beaucousin V, Herve PY, Duffau H, Crivello F, Houde O, Mazoyer B, Tzourio-Mazoyer N (2006) Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. Neuroimage 30:1414–1432CrossRefPubMedGoogle Scholar
  80. Werner S, Noppeney U (2010a) Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci 30:2662–2675CrossRefPubMedGoogle Scholar
  81. Werner S, Noppeney U (2010b) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex 20:1829–1842CrossRefPubMedGoogle Scholar
  82. Yun RJ, Krystal JH, Mathalon DH (2010) Working memory overload: fronto-limbic interactions and effects on subsequent working memory function. Brain Imaging Behav 4:96–108CrossRefPubMedCentralPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Jae-Jin Song
    • 1
  • Hyo-Jeong Lee
    • 2
    • 3
  • Hyejin Kang
    • 4
  • Dong Soo Lee
    • 4
  • Sun O. Chang
    • 3
    • 5
  • Seung Ha Oh
    • 3
    • 5
  1. 1.Department of Otorhinolaryngology-Head and Neck SurgerySeoul National University Bundang HospitalSeongnamKorea
  2. 2.Department of Otorhinolaryngology-Head and Neck SurgeryHallym University College of MedicineChun-CheonKorea
  3. 3.Sensory Organ Research Institute, Seoul National University Medical Research CenterSeoulKorea
  4. 4.Department of Nuclear MedicineSeoul National University HospitalSeoulKorea
  5. 5.Department of Otorhinolaryngology-Head and Neck SurgerySeoul National University HospitalSeoulKorea

Personalised recommendations