Skip to main content

Neuroimaging of multisensory processing in vision, audition, touch, and olfaction

Abstract

The development of neuroimaging methods has had a significant impact on the study of the human brain. Functional MRI, with its high spatial resolution, provides investigators with a method to localize the neuronal correlates of many sensory and cognitive processes. Magneto- and electroencephalography, in turn, offer excellent temporal resolution allowing the exact time course of neuronal processes to be investigated. Applying these methods to multisensory processing, many research laboratories have been successful in describing cross-sensory interactions and their spatio-temporal dynamics in the human brain. Here, we review data from selected neuroimaging investigations showing how vision can influence and interact with other senses, namely audition, touch, and olfaction. We highlight some of the similarities and differences in the cross-processing of the different sensory modalities and discuss how different neuroimaging methods can be applied to answer specific questions about multisensory processing.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3a,b

References

  1. Anderson AK, Christoff K, Stappen I, Panitz D, Ghahremani DG, Glover G et al (2003) Dissociated neural representations of intensity and valence in human olfaction. Nat Neurosci 6:196–202

    Article  CAS  PubMed  Google Scholar 

  2. Barbas H (1988) Anatomic organization of basoventral and mediodorsal visual recipient prefrontal regions in the rhesus monkey. J Comput Neurol 276:313–342

    CAS  Google Scholar 

  3. Barbas H (1993) Organization of cortical afferent input to orbitofrontal areas in the rhesus monkey. Neuroscience 56:841–864

    Article  CAS  PubMed  Google Scholar 

  4. Benevento LA, Fallon J, Davis BJ, Rezak M (1977) Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Exp Neurol 57:849–872

    CAS  PubMed  Google Scholar 

  5. Bruce C, Desimone R, Gross CG (1981) Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. J Neurophysiol 46:369–384

    CAS  PubMed  Google Scholar 

  6. Calvert GA, Thesen T (2004) Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol Paris (in press)

    Google Scholar 

  7. Calvert GA, Bullmore ET, Brammer MJ, Campbell R et al (1997) Activation of auditory cortex during silent lipreading. Science 276:593–596

    Article  CAS  PubMed  Google Scholar 

  8. Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio–visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438

    Article  CAS  PubMed  Google Scholar 

  9. Christensen CM (1985) Effect of color on judgments of food aroma and flavour intensity in young and elderly adults. Perception 14:755–762

    CAS  PubMed  Google Scholar 

  10. Clark VP, Fan S, Hillyard SA (1995) Identification of early visual evoked potential generators by retinotopic and topographic analyses. Hum Brain Mapp 2:170–187

    Google Scholar 

  11. Cooper HM, Parvopassu F, Herbin M, Magnin M (1994) Neuroanatomical pathways linking vision and olfaction in mammals. Psychoneuroendocrinology 19:623–639

    Article  CAS  PubMed  Google Scholar 

  12. Driver J, Spence C (1998) Attention and the crossmodal construction of space. Trends Cogn Sci 2:254–262

    Article  Google Scholar 

  13. Duhamel JR, Colby CL, Goldberg ME (1991) Brain and space. In: Paillard J (ed). Oxford University Press, New York, pp 223–236

  14. Engen T (1972) The effect of expectation on judgments of odor. Acta Psychol 36:450–458

    Article  CAS  Google Scholar 

  15. Falchier A, Clavagnier S, Barone P, Kennedy H (2002) Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci 22:5749–5759

    CAS  PubMed  Google Scholar 

  16. Fort A, Delpuech C, Pernier J, Giard MH (2002) Dynamics of cortico-subcortical cross-modal operations involved in audio–visual object detection in humans. Cereb Cortex 12:1031–1039

    Article  PubMed  Google Scholar 

  17. Giard MH, Peronnet F (1999) Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    CAS  PubMed  Google Scholar 

  18. Gottfried JA, O’Doherty J, Dolan RJ (2002) Appetitive and aversive olfactory learning in humans tudied using event-related functional magnetic resonance imaging. J Neurosci 22:10829–10837

    CAS  PubMed  Google Scholar 

  19. Grigor J (1995) Do the eyes see what the nose knows? An investigation of the effects of olfactory priming on visual event-related potentials. Chem Senses 20:163

    Google Scholar 

  20. Grigor J, Van TS, Behan J, Richardson A (1999) The effect of odour priming on long latency visual evoked potentials of matching and mismatching objects. Chem Senses 24:137–144

    Article  CAS  PubMed  Google Scholar 

  21. Haxby JV, Grady CL, Ungerleider LG, Horwitz B (1991) Mapping the functional neuroanatomy of the intact human brain with brain work imaging. Neuropsychologia 29:539–555

    Article  CAS  PubMed  Google Scholar 

  22. Jezzard P, Matthews P, Smith S (2001) Functional MRI: an introduction to methods. Oxford University Press, Oxford

    Google Scholar 

  23. Kennett S, Eimer M, Spence C, Driver J (2001a) Tactile–visual links in exogenous spatial attention under different postures: convergent evidence from psychophysics and ERPs. J Cogn Neurosci 13:462–478

    CAS  PubMed  Google Scholar 

  24. Kennett S, Taylor Clarke M, Haggard P (2001b) Noninformative vision improves the spatial resolution of touch in humans. Curr Biol 11:1188–1191

    Article  CAS  PubMed  Google Scholar 

  25. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500

    CAS  PubMed  Google Scholar 

  26. Lloyd DM, Shore DI, Spence C, Calvert GA (2003) Multisensory representation of limb position in human premotor cortex. Nat Neurosci 6:17–18

    Article  CAS  PubMed  Google Scholar 

  27. Macaluso E, Driver J (2001) Spatial attention and crossmodal interactions between vision and touch. Neuropsychologia 39:1304–1316

    Article  CAS  PubMed  Google Scholar 

  28. Macaluso E, Frith CD, Driver J (2000) Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206–1208

    CAS  PubMed  Google Scholar 

  29. MacLeod A, Summerfield Q (1987) Quantifying the contribution of vision to speech perception on noise. Br J Audiol 21:131–141

    CAS  PubMed  Google Scholar 

  30. MacSweeney M, Amaro E, Calvert GA, Campbell R, David AS, McGuire P et al (2000) Silent speechreading in the absence of scanner noise: an event-related fMRI study. Neuroreport 11:1729–1733

    CAS  PubMed  Google Scholar 

  31. Maravita A, Spence C, Sergent C, Driver J (2002) Seeing your own touched hands in a mirror modulates cross-modal interactions. Psychol Sci 13:350–355

    Article  PubMed  Google Scholar 

  32. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    CAS  PubMed  Google Scholar 

  33. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cognit Psychol 14:247–279

    CAS  PubMed  Google Scholar 

  34. Moir HC (1936) Some observations on the appreciation of flavour in foodstuffs. J Soc Chem Industry Industry Industry Rev 14:145–148

    Google Scholar 

  35. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res 14:115–128

    Article  Google Scholar 

  36. Morrot G, Brochet F, Dubourdieu D (2001) The color of odors. Brain Lang 79:309–320

    Article  CAS  PubMed  Google Scholar 

  37. Ongur D, Price JL (2000) The organization of networks within the orbital and medial prefrontal cortex of rats, monkeys and humans. Cereb Cortex 10:206–219

    CAS  PubMed  Google Scholar 

  38. Osterbauer RA, Matthews PM, Calvert GA (2002) Crossmodal interaction of color and smell. In: Proceedings of the international society for magnetic resonance imaging in medicine.

  39. Pavani F, Spence C, Driver J (2000) Visual capture of touch: out-of-the-body experiences with rubber gloves. Psychol Sci 11:353–359

    CAS  PubMed  Google Scholar 

  40. Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625

    CAS  PubMed  Google Scholar 

  41. Regan D (1989) Human brain electrophysiology. Elsevier, New York

  42. Rockland K, Ojima H (2001) Calcarine area V1 as a multimodal convergence area. Abstract Soc Neurosci 27:511.20

    Google Scholar 

  43. Rockland KS, Ojima H (2002) Multimodal convergence in calcarine visual areas in macaque monkey. In: International multisensory research forum. Geneva, Switzerland

  44. Rolls BJ, Rowe EA, Rolls ET (1982) How sensory properties of foods affect human feeding behavior. Physiol Behav 29:409–417

    Article  CAS  PubMed  Google Scholar 

  45. Rolls ET (2001) The rules of formation of the olfactory representations found in the orbitofrontal cortex olfactory areas in primates. Chem Senses 26:595–604

    Article  CAS  PubMed  Google Scholar 

  46. Rolls ET, Baylis LL (1994) Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex. J Neurosci 14:5437–5452

    CAS  PubMed  Google Scholar 

  47. Royet JP, Zald D, Versace R, Costes N, Lavenne F, Koenig O et al (2000) Emotional responses to pleasant and unpleasant olfactory, visual, and auditory stimuli: a positron emission tomography study. J Neurosci 20:7752–7759

    CAS  PubMed  Google Scholar 

  48. Sams M, Aulanko R, Hamalainen M, Hari R, Lounasmaa OV, Lu ST et al (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141–145

    Article  CAS  PubMed  Google Scholar 

  49. Shams L, Kamitani Y, Shimojo S (2000) Illusions. What you see is what you hear. Nature 408:788

    Article  CAS  PubMed  Google Scholar 

  50. Shams L, Kamitani Y, Thompson S, Shimojo S (2001) Sound alters visual evoked potentials in humans. Neuroreport 12:3849–3852

    Article  CAS  PubMed  Google Scholar 

  51. Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586

    CAS  PubMed  Google Scholar 

  52. Spence C, Shore DI, Klein RM (2001) Multisensory prior entry. J Exp Psychol 130:799–832

    CAS  PubMed  Google Scholar 

  53. Stein BE, Meredith, MA (1993) Merging of the senses. MIT Press, Cambridge, Mass.

  54. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

    Google Scholar 

  55. Taylor Clarke M, Kennett S, Haggard P (2002) Vision modulates somatosensory cortical processing. Curr Biol 12:233–236

    Article  CAS  PubMed  Google Scholar 

  56. Tipper SP, Lloyd D, Shorland B, Dancer C, Howard LA, McGlone F (1998) Vision influences tactile perception without proprioceptive orienting. Neuroreport 9:1741–1744

    CAS  PubMed  Google Scholar 

  57. Van Riper C (1935) An experimental study of the Japanese illusion. Am J Psychol 47:252–263

    Google Scholar 

  58. Welch RB, Warren DH (1986) Intersensory interactions. In: Kaufman KR, Thomas JP (eds) Handbook of perception and human performance, vol 1. Sensory processes and perception. Wiley, New York, pp 1–36

  59. Wheately J (1973) Putting color into marketing. Marketing 67:24–29

    Google Scholar 

  60. Zald DH (2003) The human amygdala and the emotional evaluation of sensory stimuli. Brain Res Brain Res Rev 41:88–123

    Article  PubMed  Google Scholar 

  61. Zald DH, Donndelinger MJ, Pardo JV (1998) Elucidating dynamic brain interactions with across-subjects correlational analyses of positron emission tomographic data: the functional connectivity of the amygdala and orbitofrontal cortex during olfactory tasks. J Cereb Blood Flow Metab 18:896–905

    Article  CAS  PubMed  Google Scholar 

  62. Zellner DA, Kautz MA (1990) Color affects perceived odor intensity. J Exp Psychol Hum Percept Perform 16:391–397

    Article  CAS  PubMed  Google Scholar 

  63. Zellner DA, Bartoli AM, Eckard R (1991) Influence of color on odor identification and liking ratings. Am J Psychol 104:547–561

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

G. A. Calvert and T. Thesen are supported by The Wellcome Trust, R. Osterbauer by the J. S. McDonnell Foundation, and J. F. Vibell by the Medical Research Council UK. We thank A. Fort for helpful comments on the manuscript and for Fig. 2. We thank M. Kringelbach for help with Fig. 1.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Thomas Thesen.

Additional information

Edited by: Marie-Hélène Giard and Mark Wallace

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Thesen, T., Vibell, J.F., Calvert, G.A. et al. Neuroimaging of multisensory processing in vision, audition, touch, and olfaction. Cogn Process 5, 84–93 (2004). https://doi.org/10.1007/s10339-004-0012-4

Download citation

Keywords

  • Multisensory
  • Auditory
  • Visual
  • Tactile
  • Smell