Skip to main content
SpringerLink
Log in

Semantic incongruity influences response caution in audio-visual integration

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Multisensory stimulus combinations trigger shorter reaction times (RTs) than individual single-modality stimuli. It has been suggested that this inter-sensory facilitation effect is found exclusively for semantically congruent stimuli, because incongruity would prevent multisensory integration. Here we provide evidence that the effect of incongruity is due to a change in response caution rather than prevention of stimulus integration. In two experiments, participants performed two-alternative forced-choice decision tasks in which they categorized auditory stimuli, visual stimuli or audio-visual stimulus pairs. The pairs were either semantically congruent (e.g. ambulance image and horn sound) or incongruent (e.g. ambulance image and bell sound). Shorter RTs and violations of the race model inequality on congruent trials are in accordance with previous studies. However, Bayesian hierarchical drift diffusion analyses contradict former co-activation-based explanations of the effects of congruency. Instead, they show that longer RTs on incongruent compared to congruent trials are most likely the result of an incongruity caution effect—more cautious response behaviour in face of semantically incongruent sensory input. Further, they show that response caution can be adjusted on a trial-by-trial basis depending on incoming information. Finally, stimulus modality influenced non-cognitive components of the response. We suggest that the combined stimulus energy from simultaneously presented stimuli reduces encoding time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Alais D, Burr D, (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262

  • Assink N, Lubbe R, Fox J-P, Wang Y, Pierre BE, Rudas I (2015) Does time pressure induce tunnel vision? An examination with the Eriksen Flanker Task by applying the Hierarchical Drift Diffusion Model. Proceedings of the international conference on neural networks-fuzzy systems, pp 30–40

  • Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436

    Article  CAS  PubMed  Google Scholar 

  • Chen YC, Spence C (2011) Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. J Exp Psychol Hum Percept Perform 37:1554–1568. doi:10.1037/a0024329

    Article  PubMed  Google Scholar 

  • Colonius H, Diederich A (2006) The race model inequality: Interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154

  • Domenech P, Dreher JC (2010) Decision threshold modulation in the human brain. J Neurosci 30:14305–14317. doi:10.1523/JNEUROSCI.2371-10.2010

    Article  CAS  PubMed  Google Scholar 

  • Diederich A (1995) Intersensory Facilitation of Reaction Time: evaluation of counter and diffusion coactivation models. J Math Psychol 39(2):197–215

  • Gondan M, Heckel A (2008) Testing the race inequality: A simple correction procedure for fast guesses. J Math Psychol 52(5):322–325

  • Gondan M, Blurton SP, Hughes F, Greenlee MW (2011) Effects of spatial and selective attention on basic multisensory integration. J Exp Psychol: Hum Percept Perform 37(6):1887–1897

  • Hackley SA, Valle-Inclán F (1998) Automatic alerting does not speed late motoric processes in a reaction-time task. Nature 391(6669):786–788

  • Horn SS, Bayen UJ, Smith RE (2011) What can the diffusion model tell us about prospective memory? Can J Exp Psychol 65:69–75. doi:10.1037/a0022808

    Article  PubMed  PubMed Central  Google Scholar 

  • Hu Z, Zhang R, Zhang Q, Liu Q, Li H (2012) Neural correlates of audiovisual integration of semantic category information. Brain Lang 121:70–75. doi:10.1016/j.bandl.2012.01.002

    Article  PubMed  Google Scholar 

  • Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R (1994) Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J Exp Psychol: Hum Percept Perform 20(1):131–153

  • Jepma M, Wagenmakers EJ, Band GP, Nieuwenhuis S (2009) The effects of accessory stimuli on information processing: evidence from electrophysiology and a diffusion model analysis. J Cogn Neurosci 21:847–864. doi:10.1162/jocn.2009.21063

    Article  PubMed  Google Scholar 

  • Koppen C, Alsius A, Spence C (2008) Semantic congruency and the Colavita visual dominance effect. Exp Brain Res 184:533–546. doi:10.1007/s00221-007-1120-z

    Article  PubMed  Google Scholar 

  • Krummenacher J, Müller HJ, Heller D (2002) Visual search for dimensionally redundant pop-out targets: parallel-coactive processing of dimensions is location specific. J Exp Psychol Hum Percept Perform 28:1303–1322. doi:10.1037//0096-1523.28.6.1303

    Article  PubMed  Google Scholar 

  • Laurienti P, Kraft R, Maldjian J, Burdette J, Wallace M (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414. doi:10.1007/s00221-004-1913-2

    Article  PubMed  Google Scholar 

  • Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14:247–279. doi:10.1016/0010-0285(82)90010-x

    Article  CAS  PubMed  Google Scholar 

  • Miller J (1991) Channel interaction and the redundant-targets effect in bimodal divided attention. J Exp Psychol Hum Percept Perform 17:160–169. doi:10.1037/0096-1523.17.1.160

    Article  CAS  PubMed  Google Scholar 

  • Miller J (2016) Statistical facilitation and the redundant signals effect: what are race and coactivation models? Atten, Percept, Psychophys 78(2):516–519

  • Miller J, Franz V, Ulrich R (1999) Effects of auditory stimulus intensity on response force in simple, go/no-go, and choice RT tasks. Percept Psychophys 61(1):107–119

  • Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465. doi:10.1093/cercor/bhh007

    Article  PubMed  Google Scholar 

  • Mordkoff JT, Yantis S (1991) An interactive race model of divided attention. J Exp Psychol Hum Percept Perform 17:520–538. doi:10.1037/0096-1523.17.2.520

    Article  CAS  PubMed  Google Scholar 

  • Noppeney U, Ostwald D, Werner S (2010) Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci 30:7434–7446. doi:10.1523/JNEUROSCI.0455-10.2010

    Article  CAS  PubMed  Google Scholar 

  • Otto T, Mamassian P (2012) Noise and correlations in parallel perceptual decision making. Curr Biol 22:1391–1396. doi:10.1016/j.cub.2012.05.031

    Article  CAS  PubMed  Google Scholar 

  • Parise CV, Harrar V, Ernst MO, Spence C (2013) Cross-correlation between auditory and visual signals promotes multisensory integration. Multisens Res 26:307–316. doi:10.1163/22134808-00002417

    Article  PubMed  Google Scholar 

  • Raab DH (1962) Statistical facilitation of simple reaction times. Trans N Y Acad Sci 24:574–590. doi:10.1111/j.2164-0947.1962.tb01433.x

    Article  CAS  PubMed  Google Scholar 

  • Ratcliff R, McKoon G (2008) The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput 20:873–922. doi:10.1162/neco.2008.12-06-420

    Article  PubMed  PubMed Central  Google Scholar 

  • Ratcliff R, Gomez P, McKoon G (2004) A diffusion model account of the lexical decision task. Psychol Rev 111:159–182. doi:10.1037/0033-295X.111.1.159

    Article  PubMed  PubMed Central  Google Scholar 

  • Schwarz W (1994) Diffusion, superposition, and the redundant-targets effect. J Math Psychol 38(4):504–520

  • Spiegelhalter DJ, Best NG, Carlin BP, van der Linde A (2002) Bayesian measures of model complexity and fit. J R Stat Soc Ser B (Stat Methodol) 64:583–639. doi:10.2307/3088806

    Article  Google Scholar 

  • Stahl J, Rammsayer TH (2005) Accessory stimulation in the time course of visuomotor information processing: Stimulus intensity effects on reaction time and response force. Acta Psychologica 120(1):1–18

  • Starns JJ, Ratcliff R, White CN (2012) Diffusion model drift rates can be influenced by decision processes: an analysis of the strength-based mirror effect. J Exp Psychol Learn Mem Cogn 38:1137–1151. doi:10.1037/a0028151

    Article  PubMed  PubMed Central  Google Scholar 

  • Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266

  • Suied C, Bonneel N, Viaud-Delmon I (2009) Integration of auditory and visual information in the recognition of realistic objects. Exp Brain Res 194:91–102. doi:10.1007/s00221-008-1672-6

    Article  PubMed  Google Scholar 

  • Thelen A, Talsma D, Murray MM (2015) Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138:148–160. doi:10.1016/j.cognition.2015.02.003

    Article  PubMed  Google Scholar 

  • Todd JW (1912) Reaction to multiple stimuli. Archives of psychology. The Science Press, Lancaster. doi:10.1037/13053-000

  • Voss A, Nagler M, Lerche V (2013) Diffusion models in experimental psychology. Exp Psychol 60(6):385–402

  • Wagenmakers E-J (2009) Methodological and empirical developments for the Ratcliff diffusion model of response times and accuracy. Eur J Cogn Psychol 21:641–671. doi:10.1080/09541440802205067

    Article  Google Scholar 

  • Wiecki TV, Sofer I, Frank MJ (2013) HDDM: hierarchical Bayesian estimation of the drift-diffusion model in Python. Front Neuroinform 7:14. doi:10.3389/fninf.2013.00014

    Article  PubMed  PubMed Central  Google Scholar 

  • Wille C, Ebersbach M (2016) Semantic congruency and the (reversed) Colavita effect in children and adults. J Exp Child Psychol 141:23–33. doi:10.1016/j.jecp.2015.07.015

    Article  PubMed  Google Scholar 

  • Yuval-Greenberg S, Deouell L (2009) The dogs meow: asymmetrical interaction in cross-modal object recognition. Exp Brain Res 193:603–614. doi:10.1007/s00221-008-1664-6

    Article  PubMed  Google Scholar 

  • Zhang J, Rowe JB (2014) Dissociable mechanisms of speed-accuracy tradeoff during visual perceptual learning are revealed by a hierarchical drift-diffusion model. Front Neurosci 8:69. doi:10.3389/fnins.2014.00069

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

Thomas Otto and Michael Herzog provided helpful comments on an earlier version of this manuscript and valuable advice for the data analysis. We also would like to thank Halina Sutter and Marina Wunderlin for their help in carrying out the experiments.

Author information

Authors and Affiliations

  1. Department of Psychology, University of Bern, Fabrikstrasse 8, 3012, Bern, Switzerland

    Benjamin Steinweg & Fred W. Mast

  2. Center for Cognition, Learning and Memory, University of Bern, Bern, Switzerland

    Benjamin Steinweg & Fred W. Mast

Authors
  1. Benjamin Steinweg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fred W. Mast
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin Steinweg.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Appendices

Appendix 1

See Tables 1 and 2.

Table 1 RMI test for congruent bimodal stimuli in Experiment 1
Full size table
Table 2 RMI test for incongruent bimodal stimuli in Experiment 1
Full size table

Appendix 2

See Tables 3 and 4.

Table 3 RMI test for congruent bimodal stimuli in Experiment 2
Full size table
Table 4 RMI test for incongruent bimodal stimuli in Experiment 2
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Steinweg, B., Mast, F.W. Semantic incongruity influences response caution in audio-visual integration. Exp Brain Res 235, 349–363 (2017). https://doi.org/10.1007/s00221-016-4796-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-016-4796-0

Keywords

Search

Navigation