Audio-Visual Perception of Everyday Natural Objects – Hemodynamic Studies in Humans

  • James W. Lewis


Our ability to perceive and recognize objects, people, and meaningful action events is a cognitive function of prime importance, which is characterized by an interplay of visual, auditory, and sensory-motor processing. One goal of sensory neuroscience is to better understand multisensory perception, including how information from auditory and visual systems may merge to create stable, unified representations of objects and actions in our environment. This chapter summarizes and compares results from 49 paradigms published over the past decade that have explicitly examined human brain regions associated with audio-visual interactions. A series of meta-analyses compare and contrast distinct cortical networks preferentially activated under five major types of audio-visual interactions: (1) matching spatial and/or temporal features of nonnatural objects, (2–3) matching crossmodal features characteristic of natural objects (moving versus static images), (4) associating artificial audio-visual pairings (e.g., written/spoken language), and (5) an examination of networks activated when auditory and visual stimuli are incongruent. These meta-analysis results are discussed in the context of cognitive theories regarding how object knowledge representations may mesh with the multiple parallel pathways that appear to mediate audio-visual perception.


Superior Parietal Lobule Primary Auditory Cortex Inferior Frontal Cortex Action Sound Early Visual Area 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Thanks to Mr. Chris Frum for assistance with preparation of figures. Thanks also to Dr. David Van Essen, Donna Hanlon, and John Harwell for continual development of cortical data analysis and presentation with CARET software, and William J. Talkington, Mary Pettit, and two anonymous reviewers for helpful comments on earlier versions of the text. This work was supported by the NCRR/NIH COBRE grant P20 RR15574 (to the Sensory Neuroscience Research Center of West Virginia University) and subproject to JWL.


  1. Adams RB, Janata P (2002) A comparison of neural circuits underlying auditory and visual object categorization. Neuroimage 16:361–377PubMedCrossRefGoogle Scholar
  2. Aglioti SM, Cesari P, Romani M, Urgesi C (2008) Action anticipation and motor resonance in elite basketball players. Nat Neurosci 11:1109–1116PubMedCrossRefGoogle Scholar
  3. Alink A, Singer W, Muckli L (2008) Capture of auditory motion by vision is represented by an activation shift from auditory to visual motion cortex. J Neurosci 28:2690–2697PubMedCrossRefGoogle Scholar
  4. Allison T, McCarthy G, Nobre A, Puce A, Belger A (1994) Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cereb Cortex 5:544–554CrossRefGoogle Scholar
  5. Altmann CF, Doehrmann O, Kaiser J (2007) Selectivity for animal vocalizations in the human auditory cortex. Cereb Cortex 17:2601–2608PubMedCrossRefGoogle Scholar
  6. Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp MS, Naumer MJ (2005) Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166:559–571PubMedCrossRefGoogle Scholar
  7. Amedi A, Stern WM, Camprodon JA, Bermpohl F, Merabet L, Rotman S, Hemond C, Meijer P, Pascual-Leone A (2007) Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat Neurosci 10:687–689PubMedCrossRefGoogle Scholar
  8. Andersen RA (1997) Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annu Rev Neurosci 20:303–330PubMedCrossRefGoogle Scholar
  9. Andersen RA, Zipser D (1988) The role of the posterior parietal cortex in coordinate transformations for visual-motor integration. Can J Physiol Pharmacol 66:488–501PubMedCrossRefGoogle Scholar
  10. Arbib MA (2005) From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics. Behav Brain Sci 28:105–124; discussion 125–167PubMedGoogle Scholar
  11. Arnott SR, Binns MA, Grady CL, Alain C (2004) Assessing the auditory dual-pathway model in humans. Neuroimage 22:401–408PubMedCrossRefGoogle Scholar
  12. Avillac M, Deneve S, Olivier E, Pouget A, Duhamel JR (2005) Reference frames for representing visual and tactile locations in parietal cortex. Nat Neurosci 8:941–949PubMedCrossRefGoogle Scholar
  13. Aziz-Zadeh L, Iacoboni M, Zaidel E, Wilson S, Mazziotta J (2004) Left hemisphere motor facilitation in response to manual action sounds. Eur J Neurosci 19:2609–2612PubMedCrossRefGoogle Scholar
  14. Barsalou LW (2008) Grounded cognition. Annu Rev Psychol 59:617–645PubMedCrossRefGoogle Scholar
  15. Barsalou LW, Kyle Simmons W, Barbey AK, Wilson CD (2003) Grounding conceptual knowledge in modality-specific systems. Trends Cogn Sci 7:84–91PubMedCrossRefGoogle Scholar
  16. Bates E, Dick F (2002) Language, gesture, and the developing brain. Dev Psychobiol 40:293–310PubMedCrossRefGoogle Scholar
  17. Baumann O, Greenlee MW (2007) Neural correlates of coherent audiovisual motion perception. Cereb Cortex 17:1433–1443PubMedCrossRefGoogle Scholar
  18. Beauchamp M, Lee K, Haxby J, Martin A (2002) Parallel visual motion processing streams for manipulable objects and human movements. Neuron 34:149–159PubMedCrossRefGoogle Scholar
  19. Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113PubMedCrossRefGoogle Scholar
  20. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192PubMedCrossRefGoogle Scholar
  21. Beauchamp MS, Lee KM, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823PubMedCrossRefGoogle Scholar
  22. Belardinelli M, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, Ferretti A, Caulo M, Tartaro A, Romani G (2004) Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn Process 5:167–174Google Scholar
  23. Belin P, Zatorre R (2000) ‘What’, ‘where’ and ‘how’ in auditory cortex. Nat Neurosci 3:965–966PubMedCrossRefGoogle Scholar
  24. Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B (2000) Voice-selective areas in human auditory cortex. Nature 403:309–312PubMedCrossRefGoogle Scholar
  25. Bidet-Caulet A, Voisin J, Bertrand O, Fonlupt P (2005) Listening to a walking human activates the temporal biological motion area. Neuroimage 28:132–139PubMedCrossRefGoogle Scholar
  26. Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund H-J (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13:400–404PubMedGoogle Scholar
  27. Bulkin DA, Groh JM (2006) Seeing sounds: visual and auditory interactions in the brain. Curr Opin Neurobiol 16:415–419PubMedCrossRefGoogle Scholar
  28. Burton H, Snyder AZ, Raichle ME (2004) Default brain functionality in blind people. Proc Natl Acad Sci USA 101:15500–15505PubMedCrossRefGoogle Scholar
  29. Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21:300–304PubMedGoogle Scholar
  30. Bushara KO, Hanakawa T, Immisch I, Toma K, Kansaku K, Hallett M (2003) Neural correlates of cross-modal binding. Nat Neurosci 6:190–195PubMedCrossRefGoogle Scholar
  31. Calvert GA, Brammer MJ (1999) FMRI evidence of a multimodal response in human superior temporal sulcus. Neuroimage 9:(S1038)Google Scholar
  32. Calvert GA, Campbell R (2003) Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci 15:57–70PubMedCrossRefGoogle Scholar
  33. Calvert GA, Lewis JW (2004) Hemodynamic studies of audio-visual interactions. In: Calvert GA, Spence C, Stein B (eds) Handbook of multisensory processing. MIT Press, Cambridge, MA, pp 483–502Google Scholar
  34. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657PubMedCrossRefGoogle Scholar
  35. Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438PubMedCrossRefGoogle Scholar
  36. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619–2623PubMedCrossRefGoogle Scholar
  37. Campanella S, Belin P (2007) Integrating face and voice in person perception. Trends Cogn Sci 11:535–543PubMedCrossRefGoogle Scholar
  38. Campbell R (2008) The processing of audio-visual speech: empirical and neural bases. Philos Trans R Soc Lond B Biol Sci 363:1001–1010PubMedCrossRefGoogle Scholar
  39. Capek CM, Macsweeney M, Woll B, Waters D, McGuire PK, David AS, Brammer MJ, Campbell R (2008) Cortical circuits for silent speechreading in deaf and hearing people. Neuropsychologia 46:1233–1241PubMedCrossRefGoogle Scholar
  40. Caramazza A, Shelton JR (1998) Domain-specific knowledge systems in the brain the animate-inanimate distinction. J Cogn Neurosci 10:1–34PubMedCrossRefGoogle Scholar
  41. Caramazza A, Mahon BZ (2003) The organization of conceptual knowledge: the evidence from category-specific semantic deficits. Trends Cogn Sci 7:354–361PubMedCrossRefGoogle Scholar
  42. Castro-Caldas A, Petersson KM, Reis A, Stone-Elander S, Ingvar M (1998) The illiterate brain. Learning to read and write during childhood influences the functional organization of the adult brain. Brain 121 (Pt 6):1053–1063PubMedCrossRefGoogle Scholar
  43. Catani M, Jones DK, Donato R, Ffytche DH (2003) Occipito-temporal connections in the human brain. Brain 126:2093–2107PubMedCrossRefGoogle Scholar
  44. Chao LL, Martin A (2000) Representation of manipulable man-made objects in the dorsal stream. Neuroimage 12:478–484PubMedCrossRefGoogle Scholar
  45. Corballis MC (1992) On the evolution of language and generativity. Cognition 44:197–126PubMedCrossRefGoogle Scholar
  46. Corballis MC (2003) From mouth to hand: gesture, speech, and the evolution of right-handedness. Behav Brain Sci 26:199–208; discussion 208–160PubMedGoogle Scholar
  47. Corbetta M, Miezin FM, Shulman GL, Petersen SE (1993) A PET study of visuospatial attention. J Neurosci 13:1202–1226PubMedGoogle Scholar
  48. Cox RW (1996) AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29:162–173PubMedCrossRefGoogle Scholar
  49. Craig AD (2009) How do you feel--now? The anterior insula and human awareness. Nat Rev Neurosci 10:59–70PubMedCrossRefGoogle Scholar
  50. Creem SH, Proffitt DR (2001) Defining the cortical visual systems: “what”, “where”, and “how”. Acta Psychol (Amst) 107:43–68CrossRefGoogle Scholar
  51. Crick FC, Koch C (2005) What is the function of the claustrum? Philos Trans R Soc Lond B Biol Sci 360:1271–1279PubMedCrossRefGoogle Scholar
  52. Cross ES, Kraemer DJ, Hamilton AF, Kelley WM, Grafton ST (2008) Sensitivity of the action observation network to physical and observational learning. Cereb Cortex 19(2):315–326Google Scholar
  53. Damasio H, Grabowski TJ, Tranel D, Hichwa RD, Damasio RD (1996) A neural basis for lexical retrieval. Nature 380:499–505PubMedCrossRefGoogle Scholar
  54. Davachi L (2006) Item, context and relational episodic encoding in humans. Curr Opin Neurobiol 16:693–700PubMedCrossRefGoogle Scholar
  55. De Renzi E, Faglioni P, Previdi P (1977) Spatial memory and hemispheric locus of lesion. Cortex 13:424–433PubMedGoogle Scholar
  56. Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150PubMedCrossRefGoogle Scholar
  57. Doehrmann O, Naumer MJ, Volz S, Kaiser J, Altmann CF (2008) Probing category selectivity for environmental sounds in the human auditory brain. Neuropsychologia 46:2776–2786PubMedCrossRefGoogle Scholar
  58. Downing PE, Jiang Y, Shuman M, Kanwisher N (2001) A cortical area selective for visual processing of the human body. Science 293:2470–2473PubMedCrossRefGoogle Scholar
  59. Driver J (1996) Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature 381:66–68PubMedCrossRefGoogle Scholar
  60. Engel LR, Frum C, Puce A, Walker NA, Lewis JW (2009) Different categories of living and non-living sound-sources activate distinct cortical networks. Neuroimage 47:1778–1791PubMedCrossRefGoogle Scholar
  61. Epstein R, Kanwisher N (1998) A cortical representation of the local visual environment. Nature 392:598–601PubMedCrossRefGoogle Scholar
  62. Ethofer T, Pourtois G, Wildgruber D (2006) Investigating audiovisual integration of emotional signals in the human brain. Prog Brain Res 156:345–361PubMedCrossRefGoogle Scholar
  63. Ettlinger G, Wilson WA (1990) Cross-modal performance: behavioural processes, phylogenetic considerations and neural mechanisms. Behav Brain Res 40:169–192PubMedCrossRefGoogle Scholar
  64. Falchier A, Clavagnier S, Barone P, Kennedy H (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22:5749–5759PubMedGoogle Scholar
  65. Fecteau S, Armony JL, Joanette Y, Belin P (2004) Is voice processing species-specific in human auditory cortex? An fMRI study. Neuroimage 23:840–848PubMedCrossRefGoogle Scholar
  66. Felleman DJ, van Essen DC (1991) Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex 1:1–47PubMedCrossRefGoogle Scholar
  67. Formisano E, Kim DS, Di Salle F, van de Moortele PF, Ugurbil K, Goebel R (2003) Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 40:859–869PubMedCrossRefGoogle Scholar
  68. Galati G, Committeri G, Spitoni G, Aprile T, Di Russo F, Pitzalis S, Pizzamiglio L (2008) A selective representation of the meaning of actions in the auditory mirror system. Neuroimage 40:1274–1286PubMedCrossRefGoogle Scholar
  69. Gale TM, Done DJ, Frank RJ (2001) Visual crowding and category specific deficits for pictorial stimuli: a neural network model. Cogn Neuropsychol 18:509–550PubMedGoogle Scholar
  70. Gauthier I, Skudlarski P, Gore JC, Anderson AW (2000) Expertise for cars and birds recruits brain areas involved in face recognition. Nat Neurosci 3:191–197PubMedCrossRefGoogle Scholar
  71. Gazzola V, Aziz-Zadeh L, Keysers C (2006) Empathy and the somatotopic auditory mirror system in humans. Curr Biol 16:1824–1829PubMedCrossRefGoogle Scholar
  72. Gonzalo D, Shallice T, Dolan R (2000) Time-dependent changes in learning audiovisual associations: a single-trial fMRI study. Neuroimage 11:243–255PubMedCrossRefGoogle Scholar
  73. Goodale MA, Meenan JP, Bulthoff HH, Nicolle DA, Murphy KJ, Racicot CI (1994) Separate neural pathways for the visual analysis of object shape in perception and prehension. Curr Biol 4:604–610PubMedCrossRefGoogle Scholar
  74. Gray CM, Konig P, Engel AK, Singer W (1989) Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338:334–337PubMedCrossRefGoogle Scholar
  75. Grefkes C, Ritzl A, Zilles K, Fink GR (2004) Human medial intraparietal cortex subserves visuomotor coordinate transformation. Neuroimage 23:1494–1506PubMedCrossRefGoogle Scholar
  76. Grill-Spector K, Malach R (2004) The human visual cortex. Annu Rev Neurosci 27:649–677PubMedCrossRefGoogle Scholar
  77. Gron G, Wunderlich AP, Spitzer M, Tomczak R, Riepe MW (2000) Brain activation during human navigation: gender-different neural networks as substrate of performance. Nat Neurosci 3:404–408PubMedCrossRefGoogle Scholar
  78. Grossman ED, Blake R (2002) Brain areas active during visual perception of biological motion. Neuron 35:1167–1175PubMedCrossRefGoogle Scholar
  79. Hadjikhani N, Roland PE (1998) Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J Neurosci 18:1072–1084PubMedGoogle Scholar
  80. Hashimoto R, Sakai KL (2004) Learning letters in adulthood: direct visualization of cortical plasticity for forming a new link between orthography and phonology. Neuron 42:311–322PubMedCrossRefGoogle Scholar
  81. Hashimoto T, Usui N, Taira M, Nose I, Haji T, Kojima S (2006) The neural mechanism associated with the processing of onomatopoeic sounds. Neuroimage 31:1762–1770PubMedCrossRefGoogle Scholar
  82. Hasson U, Harel M, Levy I, Malach R (2003) Large-scale mirror-symmetry organization of human occipito-temporal object areas. Neuron 37:1027–1041PubMedCrossRefGoogle Scholar
  83. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430PubMedCrossRefGoogle Scholar
  84. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887PubMedCrossRefGoogle Scholar
  85. Hocking J, Price CJ (2008) The role of the posterior superior temporal sulcus in audiovisual processing. Cereb Cortex 18:2439–2449PubMedCrossRefGoogle Scholar
  86. Hubbard EM, Piazza M, Pinel P, Dehaene S (2005) Interactions between number and space in parietal cortex. Nat Rev Neurosci 6:435–448PubMedCrossRefGoogle Scholar
  87. Iacoboni M, Molnar-Szakacs I, Gallese V, Buccino G, Mazziotta JC, Rizzolatti G (2005) Grasping the intentions of others with one’s own mirror neuron system. PLoS Biol 3:529–535CrossRefGoogle Scholar
  88. Irving-Bell L, Small M, Cowey A (1999) A distortion of perceived space in patients with right-hemisphere lesions and visual hemineglect. Neuropsychologia 37:919–925PubMedCrossRefGoogle Scholar
  89. James TW, Gauthier I (2003) Auditory and action semantic features activate sensory-specific perceptual brain regions. Curr Biol 13:1792–1796PubMedCrossRefGoogle Scholar
  90. Jellema T, Perrett DI (2006) Neural representations of perceived bodily actions using a categorical frame of reference. Neuropsychologia 44:1535–1546PubMedCrossRefGoogle Scholar
  91. Johansson G (1973) Visual perception of biological motion and a model for its analysis. Percept Psychophys 14:201–211CrossRefGoogle Scholar
  92. Kaas JH, Hackett TA (1999) ‘What’ and ‘where’ processing in auditory cortex. Nat Neurosci 2:1045–1047PubMedCrossRefGoogle Scholar
  93. Kanwisher N, McDermott J, Chun MM (1997) The fusiform face area: a module in human extrastriate cortex specialized for face perception. J Neurosci 17:4302–4311PubMedGoogle Scholar
  94. Kanwisher N, Downing P, Epstein R, Kourtzi Z (2001) Functional neuroimaging of visual recognition. In: Cabeza R, Kingstone A (eds) Handbook of functional neuroimaging of cognition. MIT Press, Cambridge, MA, pp 109–152Google Scholar
  95. Kayser C, Logothetis NK (2007) Do early sensory cortices integrate cross-modal information? Brain Struct Funct 212:121–132PubMedCrossRefGoogle Scholar
  96. Kellenbach ML, Brett M, Patterson K (2003) Actions speak louder than functions: the importance of manipulability and action in tool representation. J Cogn Neurosci 15:30–46PubMedCrossRefGoogle Scholar
  97. Keysers C, Kohler E, Umilta A, Nanetti L, Fogassi L, Gallese V (2003) Audiovisual mirror neurons and action recognition. Exp Brain Res 153:628–636PubMedCrossRefGoogle Scholar
  98. Kiefer M, Sim EJ, Herrnberger B, Grothe J, Hoenig K (2008) The sound of concepts: four markers for a link between auditory and conceptual brain systems. J Neurosci 28:12224–12230PubMedCrossRefGoogle Scholar
  99. Kilner JM, Vargas C, Duval S, Blakemore SJ, Sirigu A (2004) Motor activation prior to observation of a predicted movement. Nat Neurosci 7:1299–1301PubMedCrossRefGoogle Scholar
  100. King AJ, Nelken I (2009) Unraveling the principles of auditory cortical processing: can we learn from the visual system? Nat Neurosci 12:698–701PubMedCrossRefGoogle Scholar
  101. Kohler E, Keysers C, Umilta A, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848PubMedCrossRefGoogle Scholar
  102. Kreifelts B, Ethofer T, Grodd W, Erb M, Wildgruber D (2007) Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 37:1445–1456PubMedCrossRefGoogle Scholar
  103. Krumbholz K, Schonwiesner M, von Cramon DY, Rubsamen R, Shah NJ, Zilles K, Fink GR (2005) Representation of interaural temporal information from left and right auditory space in the human planum temporale and inferior parietal lobe. Cereb Cortex 15:317–324PubMedCrossRefGoogle Scholar
  104. Lavenex P, Amaral DG (2000) Hippocampal-neocortical interaction: a hierarchy of associativity. Hippocampus 10:420–430PubMedCrossRefGoogle Scholar
  105. Lewis JW (2006) Cortical networks related to human use of tools. Neuroscientist 12:211–231PubMedCrossRefGoogle Scholar
  106. Lewis JW, Beauchamp MS, DeYoe EA (2000) A comparison of visual and auditory motion processing in human cerebral cortex. Cereb Cortex 10:873–888PubMedCrossRefGoogle Scholar
  107. Lewis JW, Phinney RE, Brefczynski-Lewis JA, DeYoe EA (2006) Lefties get it “right” when hearing tool sounds. J Cogn Neurosci 18(8):1314–1330PubMedCrossRefGoogle Scholar
  108. Lewis JW, Brefczynski JA, Phinney RE, Janik JJ, DeYoe EA (2005) Distinct cortical pathways for processing tool versus animal sounds. J Neurosci 25:5148–5158PubMedCrossRefGoogle Scholar
  109. Lewis JW, Wightman FL, Brefczynski JA, Phinney RE, Binder JR, DeYoe EA (2004) Human brain regions involved in recognizing environmental sounds. Cereb Cortex 14:1008–1021PubMedCrossRefGoogle Scholar
  110. Lewis JW, Talkington WJ, Walker NA, Spirou GA, Jajosky A, Frum C, Brefczynski-Lewis JA (2009) Human cortical organization for processing vocalizations indicates representation of harmonic structure as a signal attribute. J Neurosci 29:2283–2296PubMedCrossRefGoogle Scholar
  111. Lewkowicz DJ (2000) The development of intersensory temporal perception: an epigenetic systems/limitations view. Psycholog Bull 126:281–308CrossRefGoogle Scholar
  112. Liberman AM, Mattingly IG (1985) The motor theory of speech perception revised. Cognition 21:1–36PubMedCrossRefGoogle Scholar
  113. Lissauer H (1890/1988) A case of visual agnosia with a contribution to theory. Cogn Neuropsychol 5:157–192CrossRefGoogle Scholar
  114. Macaluso E (2006) Multisensory processing in sensory-specific cortical areas. Neuroscientist 12:327–338PubMedCrossRefGoogle Scholar
  115. Mahon BZ, Caramazza A (2005) The orchestration of the sensory-motor systems: clues from neuropsychology. Cogn Neuropsychol 22:480–494PubMedCrossRefGoogle Scholar
  116. Mahon BZ, Anzellotti S, Schwarzbach J, Zampini M, Caramazza A (2009) Category-specific organization in the human brain does not require visual experience. Neuron 63:397–405PubMedCrossRefGoogle Scholar
  117. Martin A (2001) Functional neuroimaging of semantic memory. In: Cabeza R, Kingstone A (eds) Handbook of functional neuroimaging of cognition. The MIT Press, Cambridge, MA, pp 153–186Google Scholar
  118. Martin A (2007) The representation of object concepts in the brain. Annu Rev Psychol 58:25–45PubMedCrossRefGoogle Scholar
  119. Mayberry RI, Lock E, Kazmi H (2002) Linguistic ability and early language exposure. Nature 417:38PubMedCrossRefGoogle Scholar
  120. McCandliss BD, Cohen L, Dehaene S (2003) The visual word form area: expertise for reading in the fusiform gyrus. Trends Cogn Sci 7:293–299PubMedCrossRefGoogle Scholar
  121. McCarthy G, Puce A, Gore JC, Allison T (1997) Face-specific processing in the human fusiform gyrus. J Cogn Neurosci 9:605–610CrossRefGoogle Scholar
  122. McClelland JL, Rogers TT (2003) The parallel distributed processing approach to semantic cognition. Nat Rev Neurosci 4:310–322PubMedCrossRefGoogle Scholar
  123. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748PubMedCrossRefGoogle Scholar
  124. McNamara A, Buccino G, Menz MM, Glascher J, Wolbers T, Baumgartner A, Binkofski F (2008) Neural dynamics of learning sound-action associations. PLoS ONE 3:e3845PubMedCrossRefGoogle Scholar
  125. Mesulam MM (1998) From sensation to cognition. Brain 121:1013–1052PubMedCrossRefGoogle Scholar
  126. Meyer M, Baumann S, Marchina S, Jancke L (2007) Hemodynamic responses in human multisensory and auditory association cortex to purely visual stimulation. BMC Neurosci 8:14PubMedCrossRefGoogle Scholar
  127. Miller EK, Nieder A, Freedman DJ, Wallis JD (2003) Neural correlates of categories and concepts. Curr Opin Neurobiol 13:198–203PubMedCrossRefGoogle Scholar
  128. Moss HE, Rodd JM, Stamatakis EA, Bright P, Tyler LK (2005) Anteromedial temporal cortex supports fine-grained differentiation among objects. Cereb Cortex 15:616–627PubMedCrossRefGoogle Scholar
  129. Murray EA, Richmond BJ (2001) Role of perirhinal cortex in object perception, memory, and associations. Curr Opin Neurobiol 11:188–193PubMedCrossRefGoogle Scholar
  130. Murray EA, Bussey TJ, Hampton RR, Saksida LM (2000) The parahippocampal region and object identification. Ann N Y Acad Sci 911:166–174PubMedCrossRefGoogle Scholar
  131. Naghavi HR, Eriksson J, Larsson A, Nyberg L (2007) The claustrum/insula region integrates conceptually related sounds and pictures. Neurosci Lett 422:77–80PubMedCrossRefGoogle Scholar
  132. Naumer MJ, Doehrmann O, Muller NG, Muckli L, Kaiser J, Hein G (2008) Cortical plasticity of audio-visual object representations. Cereb Cortex 19:1641–1653Google Scholar
  133. Neal JW, Pearson RCA, Powell TPS (1988) The cortico-cortical connections within the parieto-temporal lobe of area PG,7a in the monkey. Brain Res 438:343–350PubMedCrossRefGoogle Scholar
  134. Neville H, Bavelier D, Corina D, Rauschecker J, Karni A, Lalwani A, Braun A, Clark V, Jezzard P, Turner R (1998) Cerebral organization for language in deaf and hearing subjects: biological constraints and effects of experience. Proc Natl Acad Sci USA 95:922–929PubMedCrossRefGoogle Scholar
  135. Nieder A, Dehaene S (2009) Representation of number in the brain. Annu Rev Neurosci 32:185–208PubMedCrossRefGoogle Scholar
  136. Nishitani N, Hari R (2000) Temporal dynamics of cortical representation for action. Proc Natl Acad Sci USA 97:913–918PubMedCrossRefGoogle Scholar
  137. Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18:598–609PubMedCrossRefGoogle Scholar
  138. Norman D, Shallice T (1986) Attention to action: willed and automatic control of behaviour. In: Davidson RJ, Schwartz GE, Shapiro D (eds) Consciousness and self regulation. Plenum, New York, pp 1–18CrossRefGoogle Scholar
  139. Olson IR, Gatenby JC, Gore JC (2002) A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. Brain Res Cogn Brain Res 14:129–138PubMedCrossRefGoogle Scholar
  140. Pascual-Leone A, Hamilton R (2001) The metamodal organization of the brain. Prog Brain Res 134:427–445PubMedCrossRefGoogle Scholar
  141. Patterson K, Nestor PJ, Rogers TT (2007) Where do you know what you know? The representation of semantic knowledge in the human brain. Nat Rev Neurosci 8:976–987PubMedCrossRefGoogle Scholar
  142. Pelphrey KA, Morris JP, McCarthy G (2004) Grasping the intentions of others: the perceived intentionality of an action influences activity in the superior temporal sulcus during social perception. J Cogn Neurosci 16:1706–1716PubMedCrossRefGoogle Scholar
  143. Petacchi A, Laird AR, Fox PT, Bower JM (2005) Cerebellum and auditory function: an ALE meta-analysis of functional neuroimaging studies. Hum Brain Mapp 25:118–128PubMedCrossRefGoogle Scholar
  144. Pizzamiglio L, Aprile T, Spitoni G, Pitzalis S, Bates E, D’Amico S, Di Russo F (2005) Separate neural systems for processing action- or non-action-related sounds. Neuroimage 24:852–861PubMedCrossRefGoogle Scholar
  145. Polk TA, Stallcup M, Aguirre GK, Alsop DC, D’Esposito M, Detre JA, Farah MJ (2002) Neural specialization for letter recognition. J Cogn Neurosci 14:145–159PubMedCrossRefGoogle Scholar
  146. Rademacher J, Morosan P, Schormann T, Schleicher A, Werner C, Freund HJ, Zilles K (2001) Probabilistic mapping and volume measurement of human primary auditory cortex. Neuroimage 13:669–683PubMedCrossRefGoogle Scholar
  147. Rauschecker JP, Tian B (2000) Mechanisms and streams for processing of “what” and “where” in auditory cortex. Proc Natl Acad Sci USA 97:11800–11806PubMedCrossRefGoogle Scholar
  148. Rauschecker JP, Scott SK (2009) Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing. Nat Neurosci 12:718–724PubMedCrossRefGoogle Scholar
  149. Reale RA, Calvert GA, Thesen T, Jenison RL, Kawasaki H, Oya H, Howard MA, Brugge JF (2007) Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 145:162–184PubMedCrossRefGoogle Scholar
  150. Recanzone GH, Sutter ML (2008) The biological basis of audition. Annu Rev Psychol 59:119–142PubMedCrossRefGoogle Scholar
  151. Recanzone GH, Cohen YE (2009) Serial and parallel processing in the primate auditory cortex revisited. Behav Brain Res 206:1–7Google Scholar
  152. Rilling JK (2008) Neuroscientific approaches and applications within anthropology. Am J Phys Anthropol Suppl 47:2–32CrossRefGoogle Scholar
  153. Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192PubMedCrossRefGoogle Scholar
  154. Robins DL, Hunyadi E, Schultz RT (2009) Superior temporal activation in response to dynamic audio-visual emotional cues. Brain Cogn 69:269–278PubMedCrossRefGoogle Scholar
  155. Rockland KS, Ojima H (2003) Multisensory convergence in calcarine visual areas in macaque monkey. Int J Psychophysiol 50:19–26PubMedCrossRefGoogle Scholar
  156. Roder B, Stock O, Bien S, Neville H, Rosler F (2002) Speech processing activates visual cortex in congenitally blind humans. Eur J Neurosci 16:930–936PubMedCrossRefGoogle Scholar
  157. Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP (1999) Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci 2:1131–1136PubMedCrossRefGoogle Scholar
  158. Rosch EH (1973) Natural categories. Cogn Psychol 4:328–350CrossRefGoogle Scholar
  159. Scheef L, Boecker H, Daamen M, Fehse U, Landsberg MW, Granath DO, Mechling H, Effenberg AO (2009) Multimodal motion processing in area V5/MT: evidence from an artificial class of audio-visual events. Brain Res 1252:94–104PubMedCrossRefGoogle Scholar
  160. Sestieri C, Di Matteo R, Ferretti A, Del Gratta C, Caulo M, Tartaro A, Olivetti Belardinelli M, Romani GL (2006) “What” versus “where” in the audiovisual domain: an fMRI study. Neuroimage 33:672–680PubMedCrossRefGoogle Scholar
  161. Skipper JI, van Wassenhove V, Nusbaum HC, Small SL (2007) Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb Cortex 17:2387–2399PubMedCrossRefGoogle Scholar
  162. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MAGoogle Scholar
  163. Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44:1210–1223PubMedCrossRefGoogle Scholar
  164. Talavage TM, Sereno MI, Melcher JR, Ledden PJ, Rosen BR, Dale AM (2004) Tonotopic organization in human auditory cortex revealed by progressions of frequency sensitivity. J Neurophysiol 91:1282–1296PubMedCrossRefGoogle Scholar
  165. Tanabe HC, Honda M, Sadato N (2005) Functionally segregated neural substrates for arbitrary audiovisual paired-association learning. J Neurosci 25:6409–6418PubMedCrossRefGoogle Scholar
  166. Taylor KI, Stamatakis EA, Tyler LK (2009) Crossmodal integration of object features: voxel-based correlations in brain-damaged patients. Brain 132:671–683PubMedCrossRefGoogle Scholar
  167. Taylor KI, Moss HE, Stamatakis EA, Tyler LK (2006) Binding crossmodal object features in perirhinal cortex. Proc Natl Acad Sci USA 103:8239–8244PubMedCrossRefGoogle Scholar
  168. Tettamanti M, Buccino G, Saccuman MC, Gallese V, Danna M, Scifo P, Fazio F, Rizzolatti G, Cappa SF, Perani D (2005) Listening to action-related sentences activates fronto-parietal motor circuits. J Cogn Neurosci 17:273–281PubMedCrossRefGoogle Scholar
  169. Thompson JC, Clarke M, Stewart T, Puce A (2005) Configural processing of biological motion in human superior temporal sulcus. J Neurosci 25:9059–9066PubMedCrossRefGoogle Scholar
  170. Tootell RB, Tsao D, Vanduffel W (2003) Neuroimaging weighs in: humans meet macaques in “primate” visual cortex. J Neurosci 23:3981–3989PubMedGoogle Scholar
  171. Tranel D, Logan CG, Frank RJ, Damasio AR (1997) Explaining category-related effects in the retrieval of conceptual and lexical knowledge for concrete entities: operationalization and analysis of factors. Neuropsychologia 35:1329–1339PubMedCrossRefGoogle Scholar
  172. Tranel D, Damasio H, Eichhorn GR, Grabowski TJ, Ponto LLB, Hichwa RD (2003) Neural correlates of naming animals from their characteristic sounds. Neuropsychologia 41:847–854PubMedCrossRefGoogle Scholar
  173. Tyler LK, Moss HE (2001) Towards a distributed account of conceptual knowledge. Trends Cogn Sci 5:244–252PubMedCrossRefGoogle Scholar
  174. Tyler LK, Stamatakis EA, Bright P, Acres K, Abdallah S, Rodd JM, Moss HE (2004) Processing objects at different levels of specificity. J Cogn Neurosci 16:351–362PubMedCrossRefGoogle Scholar
  175. Ungerleider LG, Haxby JV (1994) ‘What’ and ‘where’ in the human brain. Curr Opin Neurobiol 4:157–165PubMedCrossRefGoogle Scholar
  176. Ungerleider LG, Mishkin M, Goodale MA, Mansfield RJW (1982) Two cortical visual systems. In: Ingle DJ (ed.) Analysis of visual behavior. MIT Press, Cambridge, MA, pp 549–586Google Scholar
  177. Valenza N, Murray MM, Ptak R, Vuilleumier P (2004) The space of senses: impaired crossmodal interactions in a patient with Balint syndrome after bilateral parietal damage. Neuropsychologia 42:1737–1748PubMedCrossRefGoogle Scholar
  178. van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282PubMedCrossRefGoogle Scholar
  179. van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17:962–974PubMedCrossRefGoogle Scholar
  180. van Atteveldt NM, Formisano E, Goebel R, Blomert L (2007) Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 36:1345–1360PubMedCrossRefGoogle Scholar
  181. van Essen DC (1997) A tension-based theory of morphogenesis and compact wiring in the central nervous system. Nature 385:313–318PubMedCrossRefGoogle Scholar
  182. van Essen DC (2005) A population-average, landmark- and surface-based (PALS) atlas of human cerebral cortex. Neuroimage 28:635–662PubMedCrossRefGoogle Scholar
  183. van Essen DC, Drury HA, Dickson J, Harwell J, Hanlon D, Anderson CH (2001) An integrated software suite for surface-based analyses of cerebral cortex. J Am Med Inform Assoc 8:443–459PubMedCrossRefGoogle Scholar
  184. von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4:e326CrossRefGoogle Scholar
  185. Vygotsky L (1978) Mind in society: the development of higher psychological processes. Harvard University Press, Cambridge, MAGoogle Scholar
  186. Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79PubMedCrossRefGoogle Scholar
  187. Warren J, Zielinski B, Green G, Rauschecker J, Griffiths T (2002) Perception of sound-source motion by the human brain. Neuron 34:139–148PubMedCrossRefGoogle Scholar
  188. Watkins S, Shams L, Josephs O, Rees G (2007) Activity in human V1 follows multisensory perception. Neuroimage 37:572–578PubMedCrossRefGoogle Scholar
  189. Watkins S, Shams L, Tanaka S, Haynes JD, Rees G (2006) Sound alters activity in human V1 in association with illusory visual perception. Neuroimage 31:1247–1256PubMedCrossRefGoogle Scholar
  190. Wheaton KJ, Pipingas A, Silberstein RB, Puce A (2001) Human neural responses elicited to observing the actions of others. Vis Neurosci 18:401–406PubMedCrossRefGoogle Scholar
  191. Wheeler L, Griffin HC (1997) A movement-based approach to language development in children who are deaf-blind. Am Ann Deaf 142:387–390PubMedCrossRefGoogle Scholar
  192. Whiten A, Horner V, Litchfield CA, Marshall-Pescini S (2004) How do apes ape? Learn Behav 32:36–52PubMedCrossRefGoogle Scholar
  193. Zilbovicius M, Meresse I, Chabane N, Brunelle F, Samson Y, Boddaert N (2006) Autism, the superior temporal sulcus and social perception. Trends Neurosci 29:359–366PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media, LLC 2010

Authors and Affiliations

  1. 1.Department of Physiology and PharmacologySensory Neuroscience Research Center, and Center for Advanced Imaging, West Virginia UniversityMorgantownUSA

Personalised recommendations