Cognitive Processing

, Volume 5, Issue 2, pp 69–83 | Cite as

The development of multisensory processes

  • Mark T. WallaceEmail author


To understand the development of sensory processes, it is necessary not only to look at the maturation of each of the sensory systems in isolation, but also to study the development of the nervous system’s capacity to integrate information across the different senses. It is through such multisensory integration that a coherent perceptual gestalt of the world comes to be generated. In the adult brain, multisensory convergence and integration take place at a number of brainstem and cortical sites, where individual neurons have been found that respond to multisensory stimuli with patterns of activation that depend on the nature of the stimulus complex and the intrinsic properties of the neuron. Parallels between the responses of these neurons and multisensory behavior and perception suggest that they are the substrates that underlie these cognitive processes. In both cat and monkey models, the development of these multisensory neurons and the appearance of their integrative capacity is a gradual postnatal process. For subcortical structures (i.e., the superior colliculus) this maturational process appears to be gated by the appearance of functional projections from regions of association cortex. The slow postnatal maturation of multisensory processes, coupled with its dependency on functional corticotectal connections, suggested that the development of multisensory integration may be tied to sensory experiences acquired during postnatal life. In support of this, eliminating experience in one sensory modality (i.e., vision) during postnatal development severely compromises the integration of multisensory cues. Research is ongoing to better elucidate the critical development antecedents for the emergence of normal multisensory capacity.


Cross-modal Intersensory Cortex Superior colliculus Postnatal 


  1. Amlot R, Walker R, Driver J, Spence C (2003) Multimodal visual-somatosensory integration in saccade generation. Neuropsychologia 41:1–15CrossRefPubMedGoogle Scholar
  2. Andersen RA, Buneo CA (2002) Intentional maps in posterior parietal cortex. Annu Rev Neurosci 25:189–220CrossRefPubMedGoogle Scholar
  3. Andreassi JL, Greco JR (1975) Effects of bisensory stimulation on reaction time and the evoked cortical potential. Physiol Psychol 3:189–194Google Scholar
  4. Birch H, Lefford A (1963) Intersensory development in children. Monogr Soc Res Child Dev 28:1–47Google Scholar
  5. Birch H, Lefford A (1967) Visual differentiation, intersensory integration, and voluntary motor control. Monogr Soc Res Child Dev 32:1–82Google Scholar
  6. Bower TGR (1974) Development in infancy. Freeman, San FranciscoGoogle Scholar
  7. Brett-Green B, Fifkova E, Larue DT, Winer JA, Barth DS (2003) A multisensory zone in rat parietotemporal cortex: intra- and extracellular physiology and thalamocortical connections. J Comp Neurol 460:223–237CrossRefPubMedGoogle Scholar
  8. Burnett LR, Stein BE, Chaponis D, Wallace MT (2004) Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124:535–547CrossRefPubMedGoogle Scholar
  9. Chalupa LM, Rhoades RW (1977) Responses of visual, somatosensory, and auditory neurones in the golden hamster’s superior colliculus. J Physiol 270:595–626PubMedGoogle Scholar
  10. Clarey JC, Irvine DR (1986) Auditory response properties of neurons in the anterior ectosylvian sulcus of the cat. Brain Res 386:12–19CrossRefPubMedGoogle Scholar
  11. Clemo HR, Stein BE (1982) Somatosensory cortex: a ‘new’ somatotopic representation. Brain Res 235:162–168CrossRefPubMedGoogle Scholar
  12. Clemo HR, Stein BE (1983) Organization of a fourth somatosensory area of cortex in cat. J Neurophysiol 50:910–925PubMedGoogle Scholar
  13. Colonius H, Arndt P (2001) A two-stage model for visual–auditory interaction in saccadic latencies. Percept Psychophys 63:126–147Google Scholar
  14. Corneil BD, Van Wanrooij M, Munoz DP, Van Opstal AJ (2002) Auditory-visual interactions subserving goal-directed saccades in a complex scene. J Neurophysiol 88:438–454PubMedGoogle Scholar
  15. Cynader M, Berman N (1972) Receptive-field organization of monkey superior colliculus. J Neurophysiol 35:187–201PubMedGoogle Scholar
  16. Diederich A, Colonius H, Bockhorst D, Tabeling S (2003) Visual-tactile spatial interaction in saccade generation. Exp Brain Res 148:328–337PubMedGoogle Scholar
  17. Drager UC, Hubel DH (1975) Responses to visual stimulation and relationship between visual, auditory, and somatosensory inputs in mouse superior colliculus. J Neurophysiol 38:690–713PubMedGoogle Scholar
  18. Duhamel JR, Colby CL, Goldberg ME (1998) Ventral intraparietal area of the macaque: congruent visual and somatic response properties. J Neurophysiol 79:126–136PubMedGoogle Scholar
  19. Finlay BL, Schneps SE, Wilson KG, Schneider GE (1978) Topography of visual and somatosensory projections to the superior colliculus of the golden hamster. Brain Res 142:223–235CrossRefPubMedGoogle Scholar
  20. Fogassi L, Gallese V, Fadiga L, Luppino G, Matelli M, Rizzolatti G (1996) Coding of peripersonal space in inferior premotor cortex (area F4). J Neurophysiol 76:141–157PubMedGoogle Scholar
  21. Forster B, Cavina-Pratesi C, Aglioti SM, Berlucchi G (2002) Redundant target effect and intersensory facilitation from visual–tactile interactions in simple reaction time. Exp Brain Res 143:480–487CrossRefPubMedGoogle Scholar
  22. Frassinetti F, Bolognini N, Ladavas E (2002) Enhancement of visual perception by crossmodal visuo-auditory interaction. Exp Brain Res 147:332–343CrossRefPubMedGoogle Scholar
  23. Frens MA, Van Opstal AJ, Van der Willigen RF (1995) Spatial and temporal factors determine auditory–visual interactions in human saccadic eye movements. Percept Psychophys 57:802–816Google Scholar
  24. Fu KM, Johnston TA, Shah AS, Arnold L, Smiley J, Hackett TA, Garraghty PE, Schroeder CE (2003) Auditory cortical neurons respond to somatosensory stimulation. J Neurosci 23:7510–7515PubMedGoogle Scholar
  25. Fuster JM, Bodner M, Kroger JK (2000) Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405:347–351CrossRefPubMedGoogle Scholar
  26. Gibson JJ (1966) The senses considered as perceptual systems. Houghton Mifflin, BostonGoogle Scholar
  27. Goldberg ME, Wurtz RH (1972) Activity of superior colliculus in behaving monkey. I. Visual receptive fields of single neurons. J Neurophysiol 35:542–559PubMedGoogle Scholar
  28. Gordon B (1973) Receptive fields in deep layers of cat superior colliculus. J Neurophysiol 36:157–178PubMedGoogle Scholar
  29. Graybiel A (1952) Oculogravic illusion. Ama Arch Opthalmol 48:605–615PubMedGoogle Scholar
  30. Graziano MS, Hu XT, Gross CG (1997) Visuospatial properties of ventral premotor cortex. J Neurophysiol 77:2268–2292PubMedGoogle Scholar
  31. Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA (2003) Visual localization ability influences cross-modal bias. J Cogn Neurosci 15:20–29CrossRefPubMedGoogle Scholar
  32. Harrington LK, Peck CK (1998) Spatial disparity affects visual-auditory interactions in human sensorimotor processing. Exp Brain Res 122:247–252PubMedGoogle Scholar
  33. Harris LR, Blakemore C, Donaghy M (1980) Integration of visual and auditory space in the mammalian superior colliculus. Nature 288:56–59PubMedGoogle Scholar
  34. Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293Google Scholar
  35. Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, New YorkGoogle Scholar
  36. Hubel DH, Wiesel TN (1998) Early exploration of the visual cortex. Neuron 20:401–412PubMedGoogle Scholar
  37. Huerta M, Harting J (1984) The mammalian superior colliculus: studies of its morphology and connections. In: Vanegas H (ed) Comparative neurology of the optic tectum. Plenum, New York, pp 687–773Google Scholar
  38. Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R (1994) Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J Exp Psychol Hum Percept Perform 20:131–153PubMedGoogle Scholar
  39. Jassik-Gerschenfeld D (1965) Somesthetic and visual responses of superior colliculus neurones. Nature 208:898–900PubMedGoogle Scholar
  40. Jiang H, Lepore F, Ptito M, Guillemot JP (1994a) Sensory modality distribution in the anterior ectosylvian cortex (AEC) of cats. Exp Brain Res 97:404–414PubMedGoogle Scholar
  41. Jiang H, Lepore F, Ptito M, Guillemot JP (1994b) Sensory interactions in the anterior ectosylvian cortex of cats. Exp Brain Res 101:385–396PubMedGoogle Scholar
  42. Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85:506–522PubMedGoogle Scholar
  43. Jiang W, Jiang H, Stein BE (2002) Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci 14:1240–1255CrossRefPubMedGoogle Scholar
  44. Jousmaki V, Hari R (1998) Parchment-skin illusion: sound biased touch. Curr Biol 8:R190Google Scholar
  45. Kao CQ, McHaffie JG, Meredith MA, Stein BE (1994) Functional development of a central visual map in cat. J Neurophysiol 72:266–272PubMedGoogle Scholar
  46. King AJ, Hutchings ME, Moore DR, Blakemore C (1988) Developmental plasticity in the visual and auditory representations in the mammalian superior colliculus. Nature 332:73–76PubMedGoogle Scholar
  47. King AJ, Palmer AR (1983) Cells responsive to free-field auditory stimuli in guinea-pig superior colliculus: distribution and response properties. J Physiol 342:361–381PubMedGoogle Scholar
  48. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500PubMedGoogle Scholar
  49. Knudsen EI, Brainard MS (1991) Visual instruction of the neural map of auditory space in the developing optic tectum. Science 253:85–87PubMedGoogle Scholar
  50. Knudsen EI, Brainard MS (1995) Creating a unified representation of visual and auditory space in the brain. Annu Rev Neurosci 18:19–43PubMedGoogle Scholar
  51. Lewkowicz DJ (1994) Development of intersensory perception in human infants. In: Lewkowicz DJ, Lickliter R (eds) The development of intersensory perception: comparative perspectives. Erlbaum, Hillsdale, N.J., pp 165–203Google Scholar
  52. Lovelace CT, Stein BE, Wallace MT (2003) An irrelevent light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Cogn Brain 17:447–453CrossRefGoogle Scholar
  53. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748PubMedGoogle Scholar
  54. McHaffie JG, Stein BE (1982) Eye movements evoked by electrical stimulation in the superior colliculus of rats and hamsters. Brain Res 247:243–253CrossRefPubMedGoogle Scholar
  55. Meredith MA, Clemo HR (1989) Auditory cortical projection from the anterior ectosylvian sulcus (Field AES) to the superior colliculus in the cat: an anatomical and electrophysiological study. J Comp Neurol 289:687–707PubMedGoogle Scholar
  56. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391PubMedGoogle Scholar
  57. Meredith MA, Stein BE (1986a) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662PubMedGoogle Scholar
  58. Meredith MA, Stein BE (1986b) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365:350–354PubMedGoogle Scholar
  59. Meredith MA, Stein BE (1990) The visuotopic component of the multisensory map in the deep laminae of the cat superior colliculus. Neuroscience 10:3727–3742PubMedGoogle Scholar
  60. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. 1 Temporal factors. J Neurosci 7:3215–3229PubMedGoogle Scholar
  61. Meredith MA, Clemo HR, Stein BE (1991) Somatotopic component of the multisensory map in the deep laminae of the cat superior colliculus. J Comp Neurol 312:353–370PubMedGoogle Scholar
  62. Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88:181–186PubMedGoogle Scholar
  63. Middlebrooks JC, Knudsen EI (1984) A neural code for auditory space in the cat’s superior colliculus. J Neurosci 4:2621–2634PubMedGoogle Scholar
  64. Mucke L, Norita M, Benedek G, Creutzfeldt O (1982) Physiologic and anatomic investigation of a visual cortical area situated in the ventral bank of the anterior ectosylvian sulcus of the cat. Exp Brain Res 46:1–11PubMedGoogle Scholar
  65. Norita M, Mucke L, Benedek G, Albowitz B, Katoh Y, Creutzfeldt OD (1986) Connections of the anterior ectosylvian visual area (AEV). Exp Brain Res 62:225–240Google Scholar
  66. Norton TT (1974) Receptive-field properties of superior colliculus cells and development of visual behavior in kittens. J Neurophysiol 37:674–690PubMedGoogle Scholar
  67. Olson CR, Graybiel AM (1987) Ectosylvian visual area of the cat: location, retinotopic organization, and connections. J Comp Neurol 261:277–294Google Scholar
  68. Palmer AR, King AJ (1985) A monaural space map in the guinea-pig superior colliculus. Hear Res 17:267–280CrossRefPubMedGoogle Scholar
  69. Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90:4022–4026PubMedGoogle Scholar
  70. Piaget J (1952) The origins of intelligence in children. International Universities Press, New YorkGoogle Scholar
  71. Robinson DA (1972) Eye movements evoked by collicular stimulation in the alert monkey. Vis Res 12:1795–1808CrossRefPubMedGoogle Scholar
  72. Schiller PH, Stryker M (1972) Single-unit recording and stimulation in superior colliculus of the alert rhesus monkey. J Neurophysiol 35:915–924PubMedGoogle Scholar
  73. Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85:1322–1327PubMedGoogle Scholar
  74. Sekuler R, Sekuler AB, Lau R (1997) Sound alters visual motion perception. Nature 385:308CrossRefPubMedGoogle Scholar
  75. Shams L, Kamitani Y, Shimojo S (2000) Illusions. What you see is what you hear. Nature 408:788CrossRefPubMedGoogle Scholar
  76. Shams L, Kamitani Y, Shimojo S (2002) Visual illusion induced by sound. Brain Res Cogn Brain Res 14:147–152CrossRefPubMedGoogle Scholar
  77. Sparks DL (1986) Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol Rev 66:118–171PubMedGoogle Scholar
  78. Sprague JM, Meikle TH Jr (1965) The role of the superior colliculus in visually guided behavior. Exp Neurol 11:115–146Google Scholar
  79. Stein BE (1978) Nonequivalent visual, auditory, and somatic corticotectal influences in cat. J Neurophysiol 41:55–64PubMedGoogle Scholar
  80. Stein BE, Arigbede MO (1972) Unimodal and multimodal response properties of neurons in the cat superior colliculus. Exp Neurol 36:179–196Google Scholar
  81. Stein BE, Clamann HP (1981) Control of pinna movements and sensorimotor register in cat superior colliculus. Brain Behav Evol 19:180–192PubMedGoogle Scholar
  82. Stein BE, Dixon JP (1979) Properties of superior colliculus neurons in the golden hamster. J Comp Neurol 183:269–284PubMedGoogle Scholar
  83. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, Mass.Google Scholar
  84. Stein BE, Labos E, Kruger L (1973) Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J Neurophysiol 36:667–679PubMedGoogle Scholar
  85. Stein BE, Goldberg SJ, Clamann HP (1976a) The control of eye movements by the superior colliculus in the alert cat. Brain Res 118:469–474CrossRefPubMedGoogle Scholar
  86. Stein BE, Magalhaes-Castro B, Kruger L (1976b) Relationship between visual and tactile representations in cat superior colliculus. J Neurophysiol 39:401–419PubMedGoogle Scholar
  87. Stein BE, Spencer RF, Edwards SB (1983) Corticotectal and corticothalamic efferent projections of SIV somatosensory cortex in cat. J Neurophysiol 50:896–909PubMedGoogle Scholar
  88. Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448:355–358CrossRefPubMedGoogle Scholar
  89. Stein B, Meredith M, Huneycutt W, McDade L (1989) Behavioral indices of multisensory integration: orientation to visual cues is affected by auditory stimuli. J Cogn Neurosci 1:12–24Google Scholar
  90. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215Google Scholar
  91. Taylor-Clarke M, Kennett S, Haggard P (2004) Persistence of visual-tactile enhancement in humans. Neurosci Lett 354:22–25CrossRefPubMedGoogle Scholar
  92. Tiao YC, Blakemore C (1976) Functional organization in the superior colliculus of the golden hamster. J Comp Neurol 168:483–503PubMedGoogle Scholar
  93. Tortelly A, Reinoso-Suarez F, Llamas A (1980) Projections from non-visual cortical areas to the superior colliculus demonstrated by retrograde transport of HRP in the cat. Brain Res 188:543–549CrossRefPubMedGoogle Scholar
  94. Wallace MT, Stein BE (1994) Cross-modal synthesis in the midbrain depends on input from cortex. J Neurophysiol 71:429–432PubMedGoogle Scholar
  95. Wallace MT, Stein BE (1996) Sensory organization of the superior colliculus in cat and monkey. Prog Brain Res 112:301–311PubMedGoogle Scholar
  96. Wallace MT, Stein BE (1997) Development of multisensory neurons and multisensory integration in cat superior colliculus. J Neurosci 17:2429–2444PubMedGoogle Scholar
  97. Wallace MT, Stein BE (2000) Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J Neurophysiol 83:3578–3582PubMedGoogle Scholar
  98. Wallace MT, Stein BE (2001) Sensory and multisensory responses in the newborn monkey superior colliculus. J Neurosci 21:8886–8894PubMedGoogle Scholar
  99. Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488PubMedGoogle Scholar
  100. Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J Neurophysiol 69:1797–1809PubMedGoogle Scholar
  101. Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246–1266PubMedGoogle Scholar
  102. Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010PubMedGoogle Scholar
  103. Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci 101:2167–2172CrossRefPubMedGoogle Scholar
  104. Werner H (1973) Comparative psychology of mental development. International Universities Press, New YorkGoogle Scholar
  105. Wickelgren BG (1971) Superior colliculus: some receptive field properties of bimodally responsive cells. Science 173:69–72PubMedGoogle Scholar
  106. Xing J, Andersen RA (2000) Models of the posterior parietal cortex which perform multimodal integration and represent space in several coordinate frames. J Cogn Neurosci 12:601–614PubMedGoogle Scholar

Copyright information

© Marta Olivetti Belardinelli and Springer-Verlag 2004

Authors and Affiliations

  1. 1.Department of Neurobiology and AnatomyWake Forest University School of MedicineWinston-SalemUSA

Personalised recommendations