Experimental Brain Research

, 198:113 | Cite as

Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness

  • Barry E. Stein
  • Terrence R. Stanford
  • Ramnarayan Ramachandran
  • Thomas J. PerraultJr.
  • Benjamin A. Rowland
Research Article


Single-neuron studies provide a foundation for understanding many facets of multisensory integration. These studies have used a variety of criteria for identifying and quantifying multisensory integration. While a number of techniques have been used, an explicit discussion of the assumptions, criteria, and analytical methods traditionally used to define the principles of multisensory integration is lacking. This was not problematic when the field was small, but with rapid growth a number of alternative techniques and models have been introduced, each with its own criteria and sets of implicit assumptions to define and characterize what is thought to be the same phenomenon. The potential for misconception prompted this reexamination of traditional approaches in order to clarify their underlying assumptions and analytic techniques. The objective here is to review and discuss traditional quantitative methods advanced in the study of single-neuron physiology in order to appreciate the process of multisensory integration and its impact.


Sensory Cross-modal Computation Vision Auditory Somatosensory 



We thank Nancy London for technical assistance. Part of the research discussed here was supported by NIH grants EY016716 and NS36916.


  1. Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14:257–262PubMedGoogle Scholar
  2. Alvarado JC, Stanford TR, Vaughan JW, Stein BE (2007a) Cortex mediates multisensory but not unisensory integration in superior colliculus. J Neurosci 27:12775–12786PubMedCrossRefGoogle Scholar
  3. Alvarado JC, Vaughan JW, Stanford TR, Stein BE (2007b) Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J Neurophysiol 97:3193–3205PubMedCrossRefGoogle Scholar
  4. Anastasio TJ, Patton PE (2003) A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system. J Neurosci 23:6713–6727PubMedGoogle Scholar
  5. Anastasio TJ, Patton PE, Belkacem-Boussaid K (2000) Using Bayes’ rule to model multisensory enhancement in the superior colliculus. Neural Comput 12:1165–1187PubMedCrossRefGoogle Scholar
  6. Arieh Y, Marks LE (2008) Cross-modal interaction between vision and hearing: a speed-accuracy analysis. Percept Psychophys 70:412–421PubMedCrossRefGoogle Scholar
  7. Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113PubMedCrossRefGoogle Scholar
  8. Bolognini N, Leo F, Passamonti C, Stein BE, Ladavas E (2007) Multisensory-mediated auditory localization. Perception 36:1477–1485PubMedCrossRefGoogle Scholar
  9. Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA (2007) Neural basis of the ventriloquist illusion. Curr Biol 17:1697–1703PubMedCrossRefGoogle Scholar
  10. Burnett LR, Stein BE, Chaponis D, Wallace MT (2004) Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience 124:535–547PubMedCrossRefGoogle Scholar
  11. Burnett LR, Stein BE, Perrault TJ Jr, Wallace MT (2007) Excitotoxic lesions of the superior colliculus preferentially impact multisensory neurons and multisensory integration. Exp Brain Res 179:325–338PubMedCrossRefGoogle Scholar
  12. Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123PubMedCrossRefGoogle Scholar
  13. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657PubMedCrossRefGoogle Scholar
  14. Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438PubMedCrossRefGoogle Scholar
  15. Carriere BN, Royal DW, Perrault TJ, Morrison SP, Vaughan JW, Stein BE, Wallace MT (2007) Visual deprivation alters the development of cortical multisensory integration. J Neurophysiol 98:2858–2867PubMedCrossRefGoogle Scholar
  16. Deneve S, Pouget A (2004) Bayesian multisensory integration and cross-modal spatial links. J Physiol Paris 98:249–258PubMedCrossRefGoogle Scholar
  17. Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgements. Neuron 57:11–23PubMedCrossRefGoogle Scholar
  18. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433PubMedCrossRefGoogle Scholar
  19. Foxe JJ, Schroeder CE (2005) The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16:419–423PubMedCrossRefGoogle Scholar
  20. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25:5004–5012PubMedCrossRefGoogle Scholar
  21. Gillmeister H, Eimer M (2007) Tactile enhancement of auditory detection and perceived loudness. Brain Res 1160:58–68PubMedCrossRefGoogle Scholar
  22. Gingras G, Rowland BA, Stein BE (2006) Unisensory versus multisensory integration: computational distinctions in behavior. Soc Neurosci Abstr (36th Annual Meeting) (Abstract)Google Scholar
  23. Hecht D, Reiner M, Karni A (2008a) Enhancement of response times to bi- and tri-modal sensory stimuli during active movements. Exp Brain Res 185:655–665PubMedCrossRefGoogle Scholar
  24. Hecht D, Reiner M, Karni A (2008b) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189:133–143PubMedCrossRefGoogle Scholar
  25. Holmes NP (2007) The law of inverse effectiveness in neurons and behaviour: multisensory integration versus normal variability. Neuropsychologia 45:3340–3345PubMedCrossRefGoogle Scholar
  26. Holmes NP (2008) The seemingly inviolable principle of inverse effectiveness: in search of a null hypothesis. International Multisensory Research Forum, Hamburg, Germany, 2008-07-17Google Scholar
  27. Holt GR, Softky WR, Koch C, Douglas RJ (1996) Comparison of discharge variability in vitro and in vivo in cat visual cortex neurons. J Neurophysiol 75:1806–1814PubMedGoogle Scholar
  28. Jiang W, Stein BE (2003) Cortex controls multisensory depression in superior colliculus. J Neurophysiol 90:2123–2135PubMedCrossRefGoogle Scholar
  29. Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85:506–522PubMedGoogle Scholar
  30. Jiang W, Jiang H, Stein BE (2002) Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci 14:1240–1255PubMedCrossRefGoogle Scholar
  31. Kadunce DC, Vaughan JW, Wallace MT, Benedek G, Stein BE (1997) Mechanisms of within- and cross-modality suppression in the superior colliculus. J Neurophysiol 78:2834–2847PubMedGoogle Scholar
  32. Khanna SM, Teich MC (1989) Spectral characteristics of the responses of primary auditory-nerve fibers to amplitude-modulated signals. Hear Res 39:143–157PubMedCrossRefGoogle Scholar
  33. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500PubMedCrossRefGoogle Scholar
  34. Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–292PubMedCrossRefGoogle Scholar
  35. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166:289–297PubMedCrossRefGoogle Scholar
  36. Lippert M, Logothetis NK, Kayser C (2007) Improvement of visual contrast detection by a simultaneous sound. Brain Res 1173:102–109PubMedCrossRefGoogle Scholar
  37. Logothetis NK, Pfeuffer J (2004) On the nature of the BOLD fMRI contrast mechanism. Magn Reson Imaging 22:1517–1531PubMedCrossRefGoogle Scholar
  38. Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17:447–453PubMedCrossRefGoogle Scholar
  39. Ma WJ, Pouget A (2008) Linking neurons to behavior in multisensory perception: a computational review. Brain Res 1242:4–12PubMedCrossRefGoogle Scholar
  40. Magosso E, Cuppini C, Serino A, di Pellegrino G, Ursino M (2008) A theoretical study of multisensory integration in the superior colliculus by a neural network model. Neural Netw 21:817–829PubMedCrossRefGoogle Scholar
  41. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391PubMedCrossRefGoogle Scholar
  42. Meredith MA, Stein BE (1986) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662PubMedGoogle Scholar
  43. Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857PubMedGoogle Scholar
  44. Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884–5893PubMedCrossRefGoogle Scholar
  45. Odgaard EC, Arieh Y, Marks LE (2003) Cross-modal enhancement of perceived brightness: sensory interaction versus response bias. Percept Psychophys 65:123–132PubMedGoogle Scholar
  46. Patton P, Belkacem-Boussaid K, Anastasio TJ (2002) Multimodality in the superior colliculus: an information theoretic analysis. Brain Res Cogn Brain Res 14:10–19PubMedCrossRefGoogle Scholar
  47. Peck CK (1996) Visual-auditory integration in cat superior colliculus: implications for neuronal control of the orienting response. Prog Brain Res 112:167–177PubMedCrossRefGoogle Scholar
  48. Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575–2586PubMedCrossRefGoogle Scholar
  49. Populin LC, Yin TC (2002) Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci 22:2826–2834PubMedGoogle Scholar
  50. Pouget A, Deneve S, Duhamel JR (2002) A computational perspective on the neural basis of multisensory spatial representations. Nat Rev Neurosci 3:741–747PubMedCrossRefGoogle Scholar
  51. Rach S, Diederich A (2006) Visual–tactile integration: does stimulus duration influence the relative amount of response enhancement? Exp Brain Res 173:514–520PubMedCrossRefGoogle Scholar
  52. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147–1153PubMedCrossRefGoogle Scholar
  53. Rowland BA, Stein BE (2007) Multisensory integration produces an initial response enhancement. Front Integr Neurosci 1:4PubMedCrossRefGoogle Scholar
  54. Rowland B, Stanford T, Stein B (2007a) A Bayesian model unifies multisensory spatial localization with the physiological properties of the superior colliculus. Exp Brain Res 180:153–161PubMedCrossRefGoogle Scholar
  55. Rowland BA, Quessy S, Stanford TR, Stein BE (2007b) Multisensory integration shortens physiological response latencies. J Neurosci 27:5879–5884PubMedCrossRefGoogle Scholar
  56. Schroeder CE (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85:1322–1327PubMedGoogle Scholar
  57. Softky WR, Koch C (1993) The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs. J Neurosci 13:334–350PubMedGoogle Scholar
  58. Stanford TR, Stein BE (2007) Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18:787–792PubMedCrossRefGoogle Scholar
  59. Stanford TR, Quessy S, Stein BE (2005) Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci 25:6499–6508PubMedCrossRefGoogle Scholar
  60. Stein BE (2005) The development of a dialogue between cortex and midbrain to integrate multisensory information. Cortex 168:305–315Google Scholar
  61. Stein BE, Meredith MA (1990) Multisensory integration. Neural and behavioral solutions for dealing with stimuli from different sensory modalities. Ann N Y Acad Sci 608:51–65PubMedCrossRefGoogle Scholar
  62. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MAGoogle Scholar
  63. Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9:255–266PubMedCrossRefGoogle Scholar
  64. Stein BE, Huneycutt WS, Meredith MA (1988) Neurons and behavior: the same rules of multisensory integration apply. Brain Res 448:355–358PubMedCrossRefGoogle Scholar
  65. Stein BE, Meredith MA, Huneycutt WS, McDade L (1989) Behavioral indices of multisensory integration: orientation to visual cues is affected by visual stimuli. J Cogn Neurosci 1:12–24CrossRefGoogle Scholar
  66. Stein BE, London N, Wilkinson LK, Price DD (1996) Enhancement of perceived visual intensity by auditory stimuli: a psychophysical analysis. J Cogn Neurosci 8:497–506CrossRefGoogle Scholar
  67. Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179:85–95PubMedCrossRefGoogle Scholar
  68. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215CrossRefGoogle Scholar
  69. Teich MC, Heneghan C, Lowen SB, Ozaki T, Kaplan E (1997) Fractal character of the neural spike train in the visual system of the cat. J Opt Soc Am A Opt Image Sci Vis 14:529–546PubMedCrossRefGoogle Scholar
  70. Ursino M, Cuppini C, Magosso E, Serino A, di Pellegrino G (2008) Multisensory integration in the superior colliculus: a neural network model. J Comput Neurosci 26:55–73PubMedCrossRefGoogle Scholar
  71. Wallace MT, Stein BE (1994) Cross-modal synthesis in the midbrain depends on input from cortex. J Neurophysiol 71:429–432PubMedGoogle Scholar
  72. Wallace MT, Stein BE (1997) Development of multisensory neurons and multisensory integration in cat superior colliculus. J Neurosci 17:2429–2444PubMedGoogle Scholar
  73. Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488PubMedCrossRefGoogle Scholar
  74. Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246–1266PubMedGoogle Scholar
  75. Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010PubMedGoogle Scholar
  76. Wallace MT, Perrault TJ Jr, Hairston WD, Stein BE (2004) Visual experience is necessary for the development of multisensory integration. J Neurosci 24:9580–9584PubMedCrossRefGoogle Scholar
  77. Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79PubMedCrossRefGoogle Scholar
  78. Wilkinson LK, Meredith MA, Stein BE (1996) The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res 112:1–10PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Barry E. Stein
    • 1
  • Terrence R. Stanford
    • 1
  • Ramnarayan Ramachandran
    • 1
  • Thomas J. PerraultJr.
    • 1
  • Benjamin A. Rowland
    • 1
  1. 1.Department of Neurobiology and AnatomyWake Forest University School of MedicineWinston-SalemUSA

Personalised recommendations