Semantic congruence is a critical factor in multisensory behavioral performance

Abstract

It has repeatedly been demonstrated that the presence of multiple cues in different sensory modalities can enhance behavioral performance by speeding responses, increasing accuracy, and/or improving stimulus detection. Despite an extensive knowledge base as to how the spatial, temporal, and physical (e.g., intensity) characteristics of multisensory stimuli influence such enhancements, little is known about the role of semantic or contextual congruence. Our hypothesis was that semantically congruent multisensory stimuli would result in enhanced behavioral performance, and that semantically incongruent multisensory stimuli would result in either no enhancement or a decrement in behavioral performance. The results from a redundant cue feature discrimination task clearly demonstrate that congruent cross-modal stimulation improves behavioral performance. This effect is specific to the multisensory stimuli, as no improvements are seen in the presence of redundant unimodal stimulus pairs. In contrast, incongruent stimulus pairs result in behavioral decrements for both multisensory and paired unimodal stimuli. These results highlight that in addition to such simple stimulus features as space, time and relative effectiveness, the semantic content of a multisensory stimulus plays a critical role in determining how it is processed by the nervous system.

Appendix

Vincent transformation

Each subject’s response times (n) were divided into (q) quantiles (q=21 representing from 0 to 100% of responses in increments of 5). Each response time was then replaced by q equal response times. For example, if the first response time was 350 ms and 21 quantiles were desired then the first 21 values would all be 350 ms. Repeating this procedure for all response times generates a list that contains n*q values. The response time for the first quantile was then calculated by summing the first n response times and dividing by n. The value for the second quantile was calculated by averaging the next n response times and dividing by n. This procedure was repeated q times to generate the Vincent transformed CDF. The cumulative probability curves were then averaged across subjects to generate group CDFs. The CDFs were statistically compared across conditions by performing paired t-tests between the response times at each quantile (p values were corrected for multiple comparisons using the Bonferroni method).

After applying the Vincent Averaging Method to generate group CDFs, common data points were transformed from the x axis (response time) to the y axis (quantile or probability). Thus, each CDF had a data point at every quantile and the response time for that quantile was known. However, there was not a data point for every response time, so the quantile for any one time bin may or may not have been known. Because the Miller Inequality Model requires that probabilities at each time bin be used to calculate the race model, the data had to be fit to generate common data points for each time bin. In addition, the error had to be transformed from the response time axis to the probability axis to perform statistics at each time bin.

Curve fitting

The Vincent transformed CDF for each subject was fit to the CDF of the gamma distribution,

$$CDF_\gamma \left( {x,\alpha ,\lambda ,x_0 } \right) = {{\gamma \left( {\alpha ,{{\left( {x - x_0 } \right)} \over \lambda }} \right)} \over {\Gamma \left( \alpha \right)}}$$

(1)

where α and λ are the shape and scale parameters, respectively. The shift parameter, x₀, was added to the account for the different starting times of each Vincent transformed CDF. The incomplete gamma function,γ(a,z), is defined as

$$ \gamma {\left( {a,z} \right)} = {\int_z^\infty {t\,^{{a - 1}} } }e^{{ - t}} dt. $$

(2)

Γ(α) is the complete gamma function, which is equal to the incomplete gamma function, γ(α,z), with z equal to 0 (Snedecor and Cochran 1989). Nonlinear regression using the Levenberg Marquardt algorithm provided in Mathematica (Wolfram Inc., Champaign, IL, USA) was used to fit the CDF_γ function to the Vincent transformed CDFs. The fitting of the Vincent transformed CDFs was simplified from a three parameter fitting problem to a two parameter fitting problem by setting the shape parameter, α, equal to 4, which was chosen heuristically. However, our experience with selecting different values of α revealed little difference on the curve fits with the scale parameter, λ, and shift parameter, x₀, compensating for different values of α. The adequacy of the fits was assessed using the Chi Squared goodness-of-fit test. We were unable to reject the null hypothesis (p>0.05) for any of the curve fits, indicating that the gamma distribution was a reasonable model for the data.

Propagation of error

Error propagation for the CDF_γ(x,α,λ,x₀) was determined from the error propagation formula

$$\sigma _{CDF_\gamma } = \left\{ {\left( {{{\partial CDF_\gamma } \over {dx}}} \right)^2 \sigma _x^2 + \left( {{{\partial CDF_\gamma } \over {d\lambda }}} \right)^2 \sigma _\lambda ^2 + \left( {{{\partial CDF_\gamma } \over {dx_0 }}} \right)^2 \sigma _{x_0 }^2 } \right.\left. { + \left( {{{\partial CDF_\gamma } \over {d\lambda }}} \right)\left( {{{\partial CDF_\gamma } \over {dx_0 }}} \right)\sigma _{\lambda ,x_0 }^2 } \right\}^{{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-\nulldelimiterspace}\!\lower0.7ex\hbox{$2$}}} $$

(3)

where $\frac{{\partial CDF_{\gamma } }} {{\partial n}}$ is the partial derivative of the CDF_γfunction with respect to n, σ_nis the standard deviation of the parameter n, where n is equal to the parameters, x, λ, and x₀. $\sigma ^{2}_{{\lambda ,x_{0} }}$ is the estimated covariance between the scaling parameter, λ, and the shifting parameter, x₀. The partial derivatives of the CDF function were calculated with Mathematica and are as follows

$$ \frac{{\partial CDF_{\gamma } }} {{\partial x}} = \frac{{{\left( {e^{{\frac{{ - {\left( {x - x_{0} } \right)}}} {\lambda }}} } \right)}{\left( {\frac{{x - x_{0} }} {\lambda }} \right)}^{{\alpha - 1}} }} {{\lambda \,\,\Gamma {\left( \alpha \right)}}}, $$

(4)

$$ \frac{{\partial CDF_{\gamma } }} {{\partial x_{0} }} = - \frac{{\partial CDF_{\gamma } }} {{\partial x}}, $$

(5)

and

$$ \frac{{\partial CDF_{\gamma } }} {{\partial \lambda }} = \frac{{{\left( {e^{{\frac{{ - {\left( {x - x_{0} } \right)}}} {\lambda }}} } \right)}{\left( {x - x_{0} } \right)}{\left( {\frac{{x - x_{0} }} {\lambda }} \right)}^{{\alpha - 1}} }} {{\lambda ^{2} \,\,\Gamma {\left( \alpha \right)}}}, $$

(6)

where Γ(α) is the complete gamma function. The error for each bin was calculated and used to determine significance.

Miller inequality calculations

The fitted data for each subject were used to calculate the race model prediction using the Miller Inequality Equation ((P_{stimulus 1}+ P _{stimulus 2})−(P _{stimulus 1}*P _{stimulus 2})) where P is the probability of response for any give time bin. This equation was solved for each time bin to generate predicted response time CDFs if the dual stimulus conditions were processed independently.