Brain Topography

, Volume 21, Issue 3–4, pp 168–176 | Cite as

The Principle of Inverse Effectiveness in Multisensory Integration: Some Statistical Considerations

Original Paper

Abstract

The principle of inverse effectiveness (PoIE) in multisensory integration states that, as the responsiveness to individual sensory stimuli decreases, the strength of multisensory integration increases. I discuss three potential problems in the analysis of multisensory data with regard to the PoIE. First, due to ‘regression towards the mean,’ the PoIE may often be observed in datasets that are analysed post-hoc (i.e., when sorting the data by the unisensory responses). The solution is to design discrete levels of stimulus intensity a priori. Second, due to neurophysiological or methodological constraints on responsiveness, the PoIE may be, in part, a consequence of ‘floor’ and ‘ceiling’ effects. The solution is to avoid analysing or interpreting data that are too close to the limits of responsiveness, enabling both enhancement and suppression to be reliably observed. Third, the choice of units of measurement may affect whether the PoIE is observed in a given dataset. Both relative (%) and absolute (raw) measurements have advantages, but the interpretation of both is affected by systematic changes in response variability with changes in response mean, an issue that may be addressed by using measures of discriminability or effect-size such as Cohen’s d. Most importantly, randomising or permuting a dataset to construct a null distribution of a test parameter may best indicate whether any observed inverse effectiveness specifically characterises multisensory integration. When these considerations are taken into account, the PoIE may disappear or even reverse in a given dataset. I conclude that caution should be exercised when interpreting data that appear to follow the PoIE.

Keywords

Regression towards the mean Floor effect Ceiling effect Effect-size Coefficient of variation RT Congruency effect EPSP Superior colliculus Visual Auditory Somatosensory 

Supplementary material

10548_2009_97_MOESM1_ESM.doc (62 kb)
(DOC 61 kb)
10548_2009_97_MOESM2_ESM.tif (1.6 mb)
Supplementary Fig. 1aInverse effectiveness in randomly-generated datasets when using a relative (%) measurement of multisensory integration. Multisensory datasets were generated randomly from a unisensory dataset of 500 points drawn from a Gaussian distribution (with a mean and SD of 0.5 and 0.15 respectively, x~N(0.5,0.15)), by adding one of five levels of Gaussian noise (i.e., Y = x+ae, where e~N(0,1), and a = 0, 0.025, 0.050, 0.075, 0.100). The resulting datasets were then rescaled to fit a range of (0≤Y≤1) (by subtracting the minimum value and dividing by the range). To generate different standard deviations of the multisensory dataset with respect to the unisensory dataset (and thereby the slope of the relationship between multisensory and unisensory data), the multisensory data were multiplied by one of seven constants (0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, plotted as separate panels across the horizontal axis of the figure). To generate different means of the multisensory dataset with respect to the unisensory dataset (and thereby the intercept of the relationship between multisensory and unisensory data), one of seven constants was added to the multisensory data (-0.15, -0.10, -0.05, 0, 0.05, 0.10, 0.15, plotted as separate panels up the vertical axis of the figure). The five resulting multisensory datasets in each of the 49 separate simulations were highly correlated with the unisensory datasets (r≈1, 0.99, 0.95, 0.90, 0.83, plot as differently-coloured data points in each panel, respectively dark magenta, red, green, cyan, and blue). The data were converted into a relative (%) index of multisensory interaction (MSI=100*[Multisensory response– Unisensory response]/Unisensory response), separately for multisensory enhancements (MSI≥0, shown on a logarithmic scale in the upper sub-panel of each plot)) and suppressions (MSI<0, on a linear scale in the lower sub-panel of each data plot). For each dataset, multisensory integration is plot on the y-axis, and the maximum unisensory response on the x-axis. A power regression (enhancement) and logarithmic regression (suppression) lines and associated r2 values are also shown. Linear regression on the suppression data showed the same trends, but tended to provide worse fits (lower r2 values) to the data. In general, as the correlation between unisensory and multisensory datasets decreases (from magenta to red, green, cyan, and blue data points), the slope and intercept of the regression of multisensory integration on the maximum unisensory response increases. The data indicate that inverse effectiveness is a common feature of datasets with a linear relationship between unisensory and multisensory responses, but that as the correlation between these variables decreases, inverse effectiveness increases. Critical 1-tailed R2 value = 0.005 (p<.05). The raw data and other files used to produce these figures are freely available for download from the author's website. (TIFF 1669 kb)
10548_2009_97_MOESM3_ESM.tif (1.4 mb)
Supplementary Fig. 1bAs for Supplementary Figure 1A, except that the index of multisensory integration was calculated as an absolute (raw) difference (MSI=Multisensory response – Unisensory response). All data are plot on linear axes, and linear regressions are shown. Inverse effectiveness is less commonly observed for absolute (raw) than for the relative (%) measurements, but the effects of increasing variability and decreasing correlation between unisensory and multisensory data are the same – as the correlation decreases, inverse effectiveness increases. In contrast to the relative (%) measurements, multisensory suppression with absolute (raw) data generally shows negative, not positive slopes. Critical 1-tailed R2 value = 0.005 (p<.05). (TIFF 1388 kb)
10548_2009_97_MOESM4_ESM.tif (153 kb)
Supplementary Fig. 2Six commonly-used equations constrain the possible values that multisensory integration can take, given absolute floor (0.0) and ceiling (1.0) effects. max & min: maximum & minimum possible multisensory integration, given ceiling and floor effects. a) Multisensory interaction, MSI=100%*[(Multisensory-Maximum Unisensory)/Maximum Unisensory], e.g., Meredith and Stein 1983; b) Multisensory interaction, MSI=100%*[(Multisensory-Sum of Unisensory)/(Sum of Unisensory)], e.g., King and Palmer 1985; c) Multisensory contrast, MSC=100%*[(Multisensory-Maximum Unisensory)/(Multisensory+Maximum Unisensory)], e.g., Avillac et al. 2007; d) Multisensory additivity MSA=100%*[(Multisensory-Sum of Unisensory)/(Multisensory+Sum of Unisensory)], e.g., Avillac et al. 2007; e) Multisensory gain, MSG=100%*[(Multisensory-Maximum Unisensory)/Multisensory], e.g., Ross et al. 2007); f) Multisensory gain relative to ceiling effects, MS(r)G=100*[(Multisensory-Maximum Unisensory)/(Ceiling-Maximum Unisensory)], e.g., Ross et al. 2007; Sumby and Pollack 1954. (TIFF 153 kb)
10548_2009_97_MOESM5_ESM.tif (568 kb)
Supplementary Fig. 3ABehavioural data from 180 healthy human participants performing 12 conditions of a multisensory spatial congruency task (Holmes et al. 2004, 2007). Participants responded as quickly as possible using one of two foot pedals to the elevation (upper vs. lower) of vibrotactile target stimuli presented on either thumb or forefinger, while trying to ignore the elevation (upper vs. lower, i.e., congruent or incongruent with the vibrotactile target elevation) of simultaneous visual distractor stimuli, presented near to or far from the vibrotactile targets. The variability (SD) of RTs for all 12 conditions (total n = 2160) is plot against the mean RT. a) The SD changes systematically with the mean. b) Plotting the coefficient of variation (SD/mean) results in 'inverse effectiveness' – RTs are relatively more variable when the mean RT is longer (i.e., when 'responsiveness' is lower and participants perform worse on the task). Critical 1-tailed R2 value ≈ 0.001 (p<.05). (TIFF 567 kb)
10548_2009_97_MOESM6_ESM.tif (363 kb)
Supplementary Fig. 3BElectrophysiological data from whole-cell patch clamp recordings made from neurons in the intermediate and deep (multisensory) layers of the neonatal rat superior colliculus. Excitatory inputs were simulated by microstimulation of the superficial layers of the superior colliculus ('visual' inputs), and the nucleus of the brachium of the inferior colliculus ('auditory' inputs). Monosynaptic excitatory post-synaptic potentials (EPSPs) were recorded from putative multisensory cells in the intermediate and deeper layers of the superior colliculus. Sixteen responses where both the mean and SD of EPSP amplitude were available, are presented. a) The variability (SD) of EPSP amplitude increases (marginally significantly) with the mean. b) The coefficient of variation (SD/mean) of EPSP amplitude decreases significantly with the mean, showing inverse effectiveness – greater variability when the responses are weaker. Critical 1-tailed R2 value = 0.181 (p<.05). (TIFF 363 kb)
10548_2009_97_MOESM7_ESM.tif (544 kb)
Supplementary Fig. 4AInverse effectiveness in behavioural choice RT data depends on how multisensory integration is calculated. The difference in RT between incongruent and congruent trials (the crossmodal congruency effect) is plot against the mean RT, separately for stimuli presented on the same-hand (same-location) and on different-hands (different locations), for 180 healthy human participants. Dark coloured symbols and regression lines, and the upper regression equation show trials with bimodal stimuli in the same location. Light coloured symbols and lines and the lower regression equation show trials with bimodal stimuli in different locations. a) Relative (%) measures show significant inverse effectiveness for same-location, and a significant reverse trend for different-location bimodal stimuli (congruency effect (%) = 100%*[(Multisensory congruency effect-Mean RT)/Mean RT]. b) Absolute (ms) measures show significant inverse effectiveness for same-location bimodal stimuli, but no significant trend for different-location stimuli (congruency effect (ms) = Multisensory congruency effect). c) Expressed as a percentage of the maximum possible change (i.e., the congruency effect relative to the imposed ceiling of 1500ms), the data show significant inverse effectiveness for same-location, but no trend for different-location stimuli (congruency effect (% of max) = 100%*[congruency effect/(1500-congruent RT)]. d) Effect-size measurements (the data shown in panel b, divided by the pooled SD of congruent and incongruent trials) show no significant trend for same-location, and a significant reverse trend for different-location stimuli. In summary, effect size measurements, which take into account the systematic changes of response variability with response mean show no inverse effectiveness. Other measurements, which do not take changes in variability into account, all show significant inverse effectiveness for same-location bimodal stimuli. Critical 1-tailed R2 value = 0.016 (p<.05). (TIFF 544 kb)
10548_2009_97_MOESM8_ESM.tif (322 kb)
Supplementary Fig. 4BInverse effectiveness in whole cell patch-clamp recordings of EPSPs in cells from intermediate layers of the superior colliculus depends on how multisensory interactions are calculated. 12 cells showing multisensory enhancement (Multisensory response > Maximum unisensory response) are shown. a) Relative (%) enhancement (MSI = 100*[(Multisensory-Maximum Unisensory)/Maximum Unisensory]). b) Absolute (raw) enhancement (MSI (mV) = [Multisensory – Maximum Unisensory]. c) Enhancement relative to the maximum possible change (MSI (% of max) = 100%*[(Multisensory – Maximum unisensory)/(Maximum response across all cells and conditions – Maximum unisensory)]). d) Multisensory effect size = (MSI (d) = [(Multisensory-Maximum unisensory)/estimated pooled SD]. Only the relative percentage measurement showed significant inverse effectiveness. The meaning of this significant inverse effectiveness (if it is not simply artefactual) in these data is hard to interpret – these data were recorded from slices of brain of neonatal animals (12-15 days old) which had not yet opened their eyes, and from cells which could not specifically be identified as responding to naturalistic multisensory stimuli (see Wallace and Stein 1997, for the typical time-course of development of multisensory integration in postnatal cats). Critical 1-tailed R2 value = 0.247 (p<.05). (TIFF 321 kb)
10548_2009_97_MOESM9_ESM.tif (341 kb)
Supplementary Fig. 5ARandomisation of the data shown in Figure 3 illustrates the statistical dependence between measurements of multisensory interactions and the maximal unisensory response. Three randomisation procedures were run: Randomisation 1 (left column): The relationship between unisensory and multisensory responses was randomised – i.e., the cell labels for the unisensory data were randomised. Randomisation 2 (middle column): The unisensory and multisensory data were randomly resampled with replacement separately from the observed unisensory and multisensory distributions. Randomisation 3 (right column): The unisensory and multisensory data were randomly resampled with replacement from the combined unisensory and multisensory distributions. In each randomisation procedure, 1178 data points were randomised or sampled for unisensory and multisensory responses. Multisensory integration was calculated in one of 4 ways: a) Relative % MSI = 100*[(Multisensory-Unisensory)/Unisensory]; b) Absolute (raw) MSI = [Multisensory – Unisensory]; c) Relative (% of max) MSI = 100%*[(Multisensory – Unisensory)/(Maximum observed response – Unisensory)]; d) Effect-size (d) MSI = [(Multisensory – Unisensory)/estimated pooled SD]. The SD of each mean response was estimated using the best-fit equation from Figure 2: SD = 0.416mean + 1.27. The multisensory enhancements were identified (MSI>0), the linear and log slopes and correlations of multisensory enhancement on the unisensory response was calculated. The randomisation was repeated 10000 times, and the resulting distributions of the log slopes are shown in the each figure panel. Above each distribution, the upward-pointing black triangles show the lower (5%) and upper (95%) limits for the tails of the distributions. The downward-pointing green triangles show the observed log slopes for each measurement unit. For all randomisation procedures, and all units of measurement, the observed slope is substantially (and highly significantly) less negative or more positive (i.e., less inversely-effective) than the null distribution of slopes. The raw data and program scripts used to perform these randomisations are freely available for download from the author's website. (TIFF 340 kb)
10548_2009_97_MOESM10_ESM.tif (366 kb)
Supplementary Fig. 5BThe procedures outlined in Supplementary Figure 5A were applied to the RT dataset shown in Supplementary Figure 4A (same-location data only, n=180). All the randomisation procedures and all the measurement units resulted in distributions of positive slopes (i.e., in the direction predicted by inverse effectiveness), but the obtained slopes (green triangles) were substantially (and highly significantly) less positive (i.e., less inversely-effective) than the null distributions. (TIFF 366 kb)
10548_2009_97_MOESM11_ESM.tif (380 kb)
Supplementary Fig. 5CThe procedures outlined in Supplementary Figure 5A were applied to the EPSP dataset (n=14) shown in Supplementary Figure 4B. Since two unisensory responses were available, the maximum unisensory response in each randomisation was compared to the multisensory response. All the randomisation procedures and all the measurement units resulted in distributions of slopes shifted towards negative values (i.e., in the direction predicted by inverse effectiveness), but the obtained slopes (green triangles) were all less negative or more positive (i.e., less inversely-effective) than the mean of the null distributions. For Randomisation 1, two of the four obtained slopes (a: relative (%), and b: absolute (raw) measurements) were less inversely effective than the 95% cut-off of the null distributions, and 2 were within the cut-off. For Randomisation 2, only the relative (%) measurement was outside the 95% cut-off. For Randomisation 3, none of the four measures resulted in a slope different from the null distributions in either direction. (TIFF 379 kb)
10548_2009_97_MOESM12_ESM.tif (867 kb)
Supplementary Fig. 6ABinned randomisation of the data shown in Figure 3. In order to preserve the correlation between raw unisensory and multisensory datasets, the three randomisation procedures detailed in Supplementary Figure 5A were repeated, each time with a different bin width. The raw data were first sorted by the unisensory response. Randomisation or resampling was then performed only within restricted bins of unisensory responses. 10,000 randomisations were performed, with each of the 26 bin widths: 2, 3, 4, 5, 6, 8, 10, 12, 15, 19, 24, 31, 34, 38, 49, 62, 79, 98, 118, 147, 196, 236, 295, 393, 589, & 1178 responses. Each figure panel shows the relationship between two parameters of the resulting distributions: The correlation between raw unisensory and multisensory responses (x-axis), and the resulting slope of multisensory integration on unisensory responsiveness (y-axis). The horizontal axis of each ellipse covers 90% of the resulting correlations. The vertical axis of each ellipse covers 90% of the resulting slopes. The red dot and circle show the actual values obtained from the reconstructed dataset. The panels a-d and the three randomisation procedures are as shown in Supplementary Figure 5A. The raw data and program scripts used to perform these randomisations are freely available for download from the author's website. (TIFF 867 kb)
10548_2009_97_MOESM13_ESM.tif (1.2 mb)
Supplementary Fig. 6BBinned randomisation of the data shown in Supplementary Figure 4A. In order to preserve the correlation between raw unisensory and multisensory datasets, the three randomisation procedures detailed in Supplementary Figure 5B were repeated, each time with a different bin width. The raw data were first sorted by the unisensory response. Randomisation or resampling was then performed only within restricted bins of unisensory responses. 10,000 randomisations were performed, with each of the 25 bin widths: 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 26, 30, 36, 45, 60, 90, & 180 responses. Each panel shows the relationship between two parameters of the resulting distributions: the correlation between raw unisensory and multisensory responses (x-axis), and the resulting slope of multisensory integration on unisensory responsiveness (y-axis). The horizontal axis of each ellipse covers 90% of the resulting correlations. The vertical axis of each ellipse covers 90% of the resulting slopes. The red dot and circle show the values obtained from the real dataset. The panels a-d and the three randomisation procedures are as shown in Supplementary Figure 5B. The raw data and program scripts used to perform these randomisations are freely available for download from the author's website. (TIFF 1232 kb)

References

  1. Alvarado JC, Stanford TR, Vaughan JW, Stein BE (2007a) Cortex mediates multisensory but not unisensory integration in superior colliculus. J Neurosci 27:12775–12786PubMedCrossRefGoogle Scholar
  2. Alvarado JC, Vaughan JW, Stanford TR, Stein BE (2007b) Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J Neurophysiol 97:3193–3205PubMedCrossRefGoogle Scholar
  3. Beauchamp MS (2005) Statistical criteria in fMRI studies of multisensory integration. Neuroinformatics 3:93–113PubMedCrossRefGoogle Scholar
  4. Diederich A, Colonius H (2008) When a high-intensity “distractor” is better then a low-intensity one: modeling the effect of an auditory or tactile nontarget stimulus on visual saccadic reaction time. Brain Res 1242:219–230PubMedCrossRefGoogle Scholar
  5. Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57:11–23PubMedCrossRefGoogle Scholar
  6. Gilmeister H, Eimer M (2007) Tactile enhancement of auditory detection and perceived loudness. Brain Res 1160:58–68CrossRefGoogle Scholar
  7. Gondan M, Niederhaus B, Rösler F (2005) Multisensory processing in the redundant-target effect: a behavioral and event-related potential study. Percept Psychophys 67:713–726PubMedGoogle Scholar
  8. Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189:133–143PubMedCrossRefGoogle Scholar
  9. Holmes NP (2007) The law of inverse effectiveness in neurons and behaviour: multisensory integration versus normal variability. Neuropsychologia 45:3340–3345PubMedCrossRefGoogle Scholar
  10. Holmes NP, Calvert GA, Spence C (2004) Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools. Neurosci Lett 372:62–67PubMedCrossRefGoogle Scholar
  11. Holmes NP, Calvert GA, Spence C (2007) Tool use changes multisensory interactions in seconds: evidence from the crossmodal congruency task. Exp Brain Res 183:465–476PubMedCrossRefGoogle Scholar
  12. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18:1560–1574PubMedCrossRefGoogle Scholar
  13. King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500PubMedCrossRefGoogle Scholar
  14. Lakatos P, Chen C-M, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–292PubMedCrossRefGoogle Scholar
  15. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166:289–297PubMedCrossRefGoogle Scholar
  16. Longo MR, Cardozo S, Haggard P (2008) Visual enhancement of touch and the bodily self. Conscious Cogn 17:1181–1191PubMedCrossRefGoogle Scholar
  17. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391PubMedCrossRefGoogle Scholar
  18. Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365:350–354PubMedCrossRefGoogle Scholar
  19. Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662PubMedGoogle Scholar
  20. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7:3215–3229PubMedGoogle Scholar
  21. Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90:4022–4026PubMedCrossRefGoogle Scholar
  22. Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575–2586PubMedCrossRefGoogle Scholar
  23. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147–1153PubMedCrossRefGoogle Scholar
  24. Serino A, Farnè A, Rinaldesi ML, Haggard P, Làdavas E (2007) Can vision of the body ameliorate impaired somatosensory function? Neuropsychologia 45:1101–1107PubMedCrossRefGoogle Scholar
  25. Skaliora I, Doubell TP, Holmes NP, Nodal FR, King AJ (2004) Functional topography of converging visual and auditory inputs to neurons in the rat superior colliculus. J Neurophysiol 92:2933–2946PubMedCrossRefGoogle Scholar
  26. Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. NeuroImage 44:1210–1223PubMedCrossRefGoogle Scholar
  27. Stevenson RA, Kim S, James T (in press) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res. doi:10.1007/s00221-009-1783-8
  28. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Department of Psychology & Interdisciplinary Center for Neural ComputationHebrew University of JerusalemJerusalemIsrael

Personalised recommendations