Integrating information from different senses in the auditory cortex

Abstract

Multisensory integration was once thought to be the domain of brain areas high in the cortical hierarchy, with early sensory cortical fields devoted to unisensory processing of inputs from their given set of sensory receptors. More recently, a wealth of evidence documenting visual and somatosensory responses in auditory cortex, even as early as the primary fields, has changed this view of cortical processing. These multisensory inputs may serve to enhance responses to sounds that are accompanied by other sensory cues, effectively making them easier to hear, but may also act more selectively to shape the receptive field properties of auditory cortical neurons to the location or identity of these events. We discuss the new, converging evidence that multiplexing of neural signals may play a key role in informatively encoding and integrating signals in auditory cortex across multiple sensory modalities. We highlight some of the many open research questions that exist about the neural mechanisms that give rise to multisensory integration in auditory cortex, which should be addressed in future experimental and theoretical studies.

References

  1. Ahveninen J, Jääskeläinen IP, Raij T, Bonmassar G, Devore S, Hämäläinen M, Levänen S, Lin FH, Sams M, Shinn-Cunningham BG, Witzel T, Belliveau JW (2006) Task-modulated “what” and “where” pathways in human auditory cortex. Proc Natl Acad Sci USA 103: 14608–14613

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  2. Alais D, Newell FN, Mamassian P (2010) Multisensory processing in review: from physiology to behaviour. Seeing Perceiving 23: 3–38

    PubMed  Article  Google Scholar 

  3. Bendor D, Wang X (2008) Neural response properties of primary, rostral, and rostrotemporal core fields in the auditory cortex of marmoset monkeys. J Neurophysiol 100: 888–906

    PubMed Central  PubMed  Article  Google Scholar 

  4. Benedek G, Perény J, Kovács G, Fischer-Szátmári L, Katoh YY (1997) Visual, somatosensory, auditory and nociceptive modality properties in the feline suprageniculate nucleus. Neuroscience 78: 179–189

    CAS  PubMed  Article  Google Scholar 

  5. Besle J, Bertrand O, Giard MH (2009) Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 258: 143–151

    PubMed  Article  Google Scholar 

  6. Bizley JK, King AJ (2008) Visual-auditory spatial processing in auditory cortical neurons. Brain Res 1242: 24–36

    CAS  PubMed  Article  Google Scholar 

  7. Bizley JK, Nodal FR, Bajo VM, Nelken I, King AJ (2007) Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb Cortex 17: 2172–2189

    PubMed  Article  Google Scholar 

  8. Bizley JK, Walker KMM, Silverman BW, King AJ, Schnupp JWH (2009) Interdependent encoding of pitch, timbre, and spatial location in auditory cortex. J Neurosci 29: 2064–2075

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  9. Blackburn CC, Sachs MB (1989) Classification of unit types in the anteroventral cochlear nucleus: PST histograms and regularity analysis. J Neurophysiol 62: 1303–1329

    CAS  PubMed  Google Scholar 

  10. Blumhagen F, Zhu P, Shum J, Scharer YP, Yaksi E, Deisseroth K, Friedrich RW (2011) Neuronal filtering of multiplexed odour representations. Nature 479: 493–498

    CAS  PubMed  Article  Google Scholar 

  11. Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA (2007) Neural basis of the ventriloquist illusion. Curr Biol 17: 1697–1703

    CAS  PubMed  Article  Google Scholar 

  12. Brosch M, Selezneva E, Scheich H (2005) Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys. J Neurosci 25: 6797–6806

    CAS  PubMed  Article  Google Scholar 

  13. Budinger E, Scheich H (2009) Anatomical connections suitable for the direct processing of neuronal information of different modalities via the rodent primary auditory cortex. Hear Res 258: 16–27

    PubMed  Article  Google Scholar 

  14. Butt AE, Chavez CM, Flesher MM, Kinney-Hurd BL, Araujo GC, Miasnikov AA, Weinberger NM (2009) Association learning- dependent increases in acetylcholine release in the rat auditory cortex during auditory classical conditioning. Neurobiol Learn Mem 92: 400–409

    CAS  PubMed  Article  Google Scholar 

  15. Campi KL, Bales KL, Grunewald R, Krubitzer L (2010) Connections of auditory and visual cortex in the prairie vole (Microtus ochrogaster): evidence for multisensory processing in primary sensory areas. Cereb Cortex 20: 89–108

    PubMed Central  PubMed  Article  Google Scholar 

  16. Chen YC, Spence C (2010) When hearing the bark helps to identify the dog: semantically-congruent sounds modulate the identification of masked pictures. Cognition 114: 389–404

    PubMed  Article  Google Scholar 

  17. Clarke S, Bellmann A, Meuli RA, Assal G, Steck AJ (2000) Auditory agnosia and auditory spatial deficits following left hemispheric lesions: evidence for distinct processing pathways. Neuropsychologia 38: 797–807

    CAS  PubMed  Article  Google Scholar 

  18. Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242: 136–150

    CAS  PubMed  Article  Google Scholar 

  19. Falchier A, Schroeder CE, Hackett TA, Lakatos P, Nascimento-Silva S, Ulbert I, Karmos G, Smiley JF (2010) Projection from visual areas V2 and prostriata to caudal auditory cortex in the monkey. Cereb Cortex 20: 1529–1538

    PubMed Central  PubMed  Article  Google Scholar 

  20. Foote SL, Freedman R, Oliver AP (1975) Effects of putative neurotransmitters on neuronal activity in monkey auditory cortex. Brain Res 86: 229–242

    CAS  PubMed  Article  Google Scholar 

  21. Ghazanfar AA, Chandrasekaran C, Logothetis NK (2008) Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. J Neurosci 28: 4457–4469

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  22. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25: 5004–5012

    CAS  PubMed  Article  Google Scholar 

  23. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory. Trends Cogn Sci 10: 278–285

    PubMed  Article  Google Scholar 

  24. Goard M, Dan Y (2009) Basal forebrain activation enhances cortical coding of natural scenes. Nat Neurosci 12: 1444–1449

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  25. Gutig R, Sompolinsky H (2006) The tempotron: a neuron that learns spike timing-based decisions. Nat Neurosci 9: 420–428

    PubMed  Article  Google Scholar 

  26. Harrington IA, Heffner RS, Heffner HE (2001) An investigation of sensory deficits underlying the aphasia-like behaviour of macaques with auditory cortex lesions. Neuroreport 12: 1217–1221

    CAS  PubMed  Article  Google Scholar 

  27. Heffner HE, Heffner RS (1986) Effect of unilateral and bilateral auditory cortex lesions on the discrimination of vocalizations by Japanese macaques. J Neurophysiol 56: 683–701

    CAS  PubMed  Google Scholar 

  28. Heffner HE, Heffner RS (1990) Effect of bilateral auditory cortex lesions on sound localization in Japanese macaques. J Neurophysiol 64: 915–931

    CAS  PubMed  Google Scholar 

  29. Hurley LM, Hall IC (2011) Context-dependent modulation of auditory processing by serotonin. Hear Res 279: 74–84

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  30. Jenkins WM, Masterton RB (1982) Sound localization: effects of unilateral lesions in central auditory system. J Neurophysiol 47:987–1016

    CAS  PubMed  Google Scholar 

  31. Kayser C, Logothetis NK, Panzeri S (2010) Visual enhancement of the information representation in auditory cortex. Curr Biol 20: 19–24

    CAS  PubMed  Article  Google Scholar 

  32. Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. J Neurosci 27: 1824–1835

    CAS  PubMed  Article  Google Scholar 

  33. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18: 1560–1574

    PubMed  Article  Google Scholar 

  34. Kayser C, Petkov CI, Logothetis NK (2009a) Multisensory interactions in primate auditory cortex: fMRI and electrophysiology. Hear Res 258: 80–88

    PubMed  Article  Google Scholar 

  35. Kayser C, Montemurro MA, Logothetis NK, Panzeri S (2009b) Spike-phase coding boosts and stabilizes information carried by spatial and temporal spike patterns. Neuron 61: 597–608

    CAS  PubMed  Article  Google Scholar 

  36. King AJ (2004) The superior colliculus. Curr Biol 14: R335–338

    CAS  PubMed  Article  Google Scholar 

  37. King AJ (2005) Multisensory integration: strategies for synchronization. Curr Biol 15: R339–341

    CAS  PubMed  Article  Google Scholar 

  38. Kvasnák E, Popelár J, Syka J (2000) Discharge properties of neurons in subdivisions of the medial geniculate body of the guinea pig. Physiol Res 49: 369–378

    PubMed  Google Scholar 

  39. Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53: 279–292

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  40. Lakatos P, Pincze Z, Fu KM, Javitt DC, Karmos G, Schroeder CE (2005) Timing of pure tone and noise-evoked responses in macaque auditory cortex. Neuroreport 16: 933–937

    PubMed  Article  Google Scholar 

  41. Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412: 150–157

    CAS  PubMed  Article  Google Scholar 

  42. Lomber SG, Malhotra S (2008) Double dissociation of ’what’ and ’where’ processing in auditory cortex. Nat Neurosci 11: 609–616

    CAS  PubMed  Article  Google Scholar 

  43. Lomber SG, Meredith MA, Kral A (2010) Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nat Neurosci 13: 1421–1427

    CAS  PubMed  Article  Google Scholar 

  44. Luo H, Liu Z, Poeppel D (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8: e1000445

    PubMed Central  PubMed  Article  Google Scholar 

  45. Maunsell JH, Gibson JR (1992) Visual response latencies in striate cortex of the macaque monkey. J Neurophysiol 68: 1332–1344

    CAS  PubMed  Google Scholar 

  46. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264: 746–748

    CAS  PubMed  Article  Google Scholar 

  47. Meredith MA, Allman BL (2009) Subthreshold multisensory processing in cat auditory cortex. Neuroreport 20: 126–131

    PubMed Central  PubMed  Article  Google Scholar 

  48. Meredith MA, Allman BL, Keniston LP, Clemo HR (2009) Auditory influences on non-auditory cortices. Hear Res 258: 64–71

    PubMed Central  PubMed  Article  Google Scholar 

  49. Montemurro MA, Rasch MJ, Murayama Y, Logothetis NK, Panzeri S (2008) Phase-of-firing coding of natural visual stimuli in primary visual cortex. Curr Biol 18: 375–380

    CAS  PubMed  Article  Google Scholar 

  50. Nelken I, Chechik G, Mrsic-Flogel T, King AJ, Schnupp JW (2005) Encoding stimulus information by spike numbers and mean response time in primary auditory cortex. J Comput Neurosci 19: 199–221

    PubMed  Article  Google Scholar 

  51. Nodal FR, Kacelnik O, Bajo VM, Bizley JK, Moore DR, King AJ (2010) Lesions of the auditory cortex impair azimuthal sound localization and its recalibration in ferrets. J Neurophysiol 103: 1209–1225

    PubMed Central  PubMed  Article  Google Scholar 

  52. Panzeri S, Brunel N, Logothetis NK, Kayser C (2010) Sensory neural codes using multiplexed temporal scales. Trends Neurosci 33: 111–120

    CAS  PubMed  Article  Google Scholar 

  53. Pekkola J, Ojanen V, Autti T, Jääskeläinen IP, Möttönen R, Tarkiainen A, Sams M (2005) Primary auditory cortex activation by visual speech: an fMRI study at 3 T. Neuroreport 16: 125–128

    PubMed  Article  Google Scholar 

  54. Pouille F, Scanziani M (2001) Enforcement of temporal fidelity in pyramidal cells by somatic feed-forward inhibition. Science 293: 1159–1163

    CAS  PubMed  Article  Google Scholar 

  55. Powers AR, Hevey MA, Wallace MT (2012) Neural correlates of multisensory perceptual learning. J Neurosci 32: 6263– 6274

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  56. Powers AR, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29: 12265–12274

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  57. Recanzone GH, Guard DC, Phan ML (2000) Frequency and intensity response properties of single neurons in the auditory cortex of the behaving macaque monkey. J Neurophysiol 83: 2315–2331

    CAS  PubMed  Google Scholar 

  58. Recanzone GH (2008) Representation of con-specific vocalizations in the core and belt areas of the auditory cortex in the alert macaque monkey. J Neurosci 28: 13184–13193

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  59. Schroeder CE, Mehta AD, Givre SJ (1998) A spatiotemporal profile of visual system activation revealed by current source density analysis in the awake macaque. Cereb Cortex 8: 575–592

    CAS  PubMed  Article  Google Scholar 

  60. Schürmann M, Caetano G, Hlushchuk Y, Jousmäki V, Hari R (2006) Touch activates human auditory cortex. Neuroimage 30: 1325–1331

    PubMed  Article  Google Scholar 

  61. Sekuler R, Sekuler AB, Lau R (1997) Sound alters visual motion perception. Nature 385: 308

    CAS  PubMed  Article  Google Scholar 

  62. Seltzer B, Pandya DN (1994) Parietal, temporal, and occipital projections to cortex of the superior temporal sulcus in the rhesus monkey: a retrograde tracer study. J Comp Neurol 343: 445–463

    CAS  PubMed  Article  Google Scholar 

  63. Slutsky DA, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. Neuroreport 12: 7–10

    CAS  PubMed  Article  Google Scholar 

  64. Soto-Faraco S, Lyons J, Gazzaniga M, Spence C, Kingstone A (2002) The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. Brain Res Cogn Brain Res 14: 139–146

    PubMed  Article  Google Scholar 

  65. Stark H, Scheich H (1997) Dopaminergic and serotonergic neurotransmission systems are differentially involved in auditory cortex learning: a long-term microdialysis study of metabolites. J Neurochem 68: 691–697

    CAS  PubMed  Article  Google Scholar 

  66. Stecker GC, Harrington IA, Macpherson EA, Middlebrooks JC (2005) Spatial sensitivity in the dorsal zone (area DZ) of cat auditory cortex. J Neurophysiol 94: 1267–1280

    PubMed  Article  Google Scholar 

  67. Stecker GC, Middlebrooks JC (2003) Distributed coding of sound locations in the auditory cortex. Biol Cybern 89: 341–349

    PubMed  Article  Google Scholar 

  68. Stein, BE (ed) (2012) The new handbook of multisensory processing. MIT Press, Cambridge

    Google Scholar 

  69. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26: 212–215

    Article  Google Scholar 

  70. Thorne JD, De Vos M, Viola FC, Debener S (2011) Cross-modal phase reset predicts auditory task performance in humans. J Neurosci 31: 3853–3861

    CAS  PubMed  Article  Google Scholar 

  71. Tian B, Reser D, Durham A, Kustov A, Rauschecker JP (2001) Functional specialization in rhesus monkey auditory cortex. Science 292: 290–293

    CAS  PubMed  Article  Google Scholar 

  72. Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45: 598–607

    PubMed  Article  Google Scholar 

  73. Vatakis A, Spence C (2001) Audiovisual temporal integration for complex speech, object-action, animal call, and musical stimuli. In: Naumer MJ, Kaiser J (eds) Multisensory object perception in the primate brain. Springer, Berlin

    Google Scholar 

  74. Walker KM, Bizley JK, King AJ, Schnupp JW (2011) Multiplexed and robust representations of sound features in auditory cortex. J Neurosci 31: 14565–14576

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  75. Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci USA 101: 2167–2172

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  76. Warren JD, Griffiths TD (2003) Distinct mechanisms for processing spatial sequences and pitch sequences in the human auditory brain. J Neurosci 23: 5799–5804

    CAS  PubMed  Google Scholar 

  77. Zampini M, Guest S, Shore DI, Spence C (2005) Audio-visual simultaneity judgments. Percept Psychophys 67:531–544

    PubMed  Article  Google Scholar 

Download references

Acknowledgments

Our research is supported by the Wellcome Trust through a Principal Research Fellowship to A. J.King (WT076508AIA).

Open Access

This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Andrew J. King.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

King, A.J., Walker, K.M.M. Integrating information from different senses in the auditory cortex. Biol Cybern 106, 617–625 (2012). https://doi.org/10.1007/s00422-012-0502-x

Download citation

Keywords

  • Multisensory
  • Auditory cortex
  • Perception
  • Multiplex
  • Information
  • Neural code
  • Visual
  • Somatosensory