Inverse Effectiveness and Multisensory Interactions in Visual Event-Related Potentials with Audiovisual Speech

Abstract

In recent years, it has become evident that neural responses previously considered to be unisensory can be modulated by sensory input from other modalities. In this regard, visual neural activity elicited to viewing a face is strongly influenced by concurrent incoming auditory information, particularly speech. Here, we applied an additive-factors paradigm aimed at quantifying the impact that auditory speech has on visual event-related potentials (ERPs) elicited to visual speech. These multisensory interactions were measured across parametrically varied stimulus salience, quantified in terms of signal to noise, to provide novel insights into the neural mechanisms of audiovisual speech perception. First, we measured a monotonic increase of the amplitude of the visual P1-N1-P2 ERP complex during a spoken-word recognition task with increases in stimulus salience. ERP component amplitudes varied directly with stimulus salience for visual, audiovisual, and summed unisensory recordings. Second, we measured changes in multisensory gain across salience levels. During audiovisual speech, the P1 and P1-N1 components exhibited less multisensory gain relative to the summed unisensory components with reduced salience, while N1-P2 amplitude exhibited greater multisensory gain as salience was reduced, consistent with the principle of inverse effectiveness. The amplitude interactions were correlated with behavioral measures of multisensory gain across salience levels as measured by response times, suggesting that change in multisensory gain associated with unisensory salience modulations reflects an increased efficiency of visual speech processing.

Appendix

The most commonly used metric of multisensory interactions in ERP research is the additive metric, or model (Barth et al. 1995; Berman 1961; Besle et al. 2004b). The additive model’s null hypothesis asserts that the multisensory response should equal the summed responses measured with both unisensory presentations in isolation, an assertion based on the law of superposition of electrical fields (Besle et al. 2004b, 2009; Giard and Besle 2010). While this metric is useful, there are a number of situations where it may produce spurious superadditive, or subadditive results. In particular, variations in attention across sensory modality (including variations in difficulty across sensory modalities or divided attention in bi-sensory conditions relative to unisensory conditions) and common activity create serious concerns about the use of the additive metric in multisensory paradigms (for an in-depth discussion, see Besle et al. 2004b; Giard and Besle 2010; Gondan and Röder 2006). While the additive metric calculates interactions as:

$$ {\text{AV}} \ne {\text{A}} + {\text{V}}. $$

In terms of sensory specific activations, it is more accurately written as:

$$ {\text{AV}} + {\text{CA}} \ne \left( {{\text{A}} + {\text{CA}}} \right) + \left( {{\text{V}} + {\text{CA}}} \right), $$

where CA refers to common activation, i.e., activation of processes that commonly occur regardless of sensory input. In this case, the common activation is accounted for twice on the right side of the equation, but only once on the left, producing spurious findings of multisensory interactions (Besle et al. 2004b; Giard and Besle 2010). Importantly, the additive factors equation reduces the impact of the common activation, measuring a change in each sensory modality:

$$ {\text{AV}}_{\text{H}} - {\text{AV}}_{\text{L}} \ne \left( {{\text{A}}_{\text{H}} - {\text{A}}_{\text{L}} } \right) + \left( {{\text{V}}_{\text{H}} - {\text{V}}_{\text{L}} } \right) .$$

In terms of specific sensory activations, again, this equation is more accurately written as:

$$ \left( {{\text{AV}}_{\text{H}} + {\text{CA}}} \right) - \left( {{\text{AV}}_{\text{L}} + {\text{CA}}} \right) \ne \left[ {\left( {{\text{A}}_{\text{H}} + {\text{CA}}} \right) - \left( {{\text{A}}_{\text{L}} + {\text{CA}}} \right)} \right] + \left[ {\left( {{\text{V}}_{\text{H}} + {\text{CA}}} \right) - \left( {{\text{V}}_{\text{L}} + {\text{CA}}} \right)} \right]. $$

Here, it should be noted that the impact of common activations, or CA, are diminished relative to the classic additive metric equation. Each component of the additive-factors equation includes two common activations that are subtracted from one another, leaving only the difference between common activations associated with levels of the added factor.

Finally, it should also be noted that there are a number of other approaches that have been used to circumvent the issues associated with the additive criterion. One such example is the application of electrical neuroimaging analyses to ERPs that includes assessing not only the response amplitude and timing of responses, but also utilizes response topography. This analysis, in addition to bypassing issues of associated with the additive metric also allows the experimenter to differentiate effects cause by changes in response strength from a given set of generators from effects caused by changes in the configuration of these generators Furthermore, the use of global field power can allow for the identification of the directionality of those interactions (Cappe et al. 2010; Murray et al. 2005, 2008).