Experimental Brain Research

, Volume 235, Issue 1, pp 349–363 | Cite as

Semantic incongruity influences response caution in audio-visual integration

  • Benjamin SteinwegEmail author
  • Fred W. Mast
Research Article


Multisensory stimulus combinations trigger shorter reaction times (RTs) than individual single-modality stimuli. It has been suggested that this inter-sensory facilitation effect is found exclusively for semantically congruent stimuli, because incongruity would prevent multisensory integration. Here we provide evidence that the effect of incongruity is due to a change in response caution rather than prevention of stimulus integration. In two experiments, participants performed two-alternative forced-choice decision tasks in which they categorized auditory stimuli, visual stimuli or audio-visual stimulus pairs. The pairs were either semantically congruent (e.g. ambulance image and horn sound) or incongruent (e.g. ambulance image and bell sound). Shorter RTs and violations of the race model inequality on congruent trials are in accordance with previous studies. However, Bayesian hierarchical drift diffusion analyses contradict former co-activation-based explanations of the effects of congruency. Instead, they show that longer RTs on incongruent compared to congruent trials are most likely the result of an incongruity caution effect—more cautious response behaviour in face of semantically incongruent sensory input. Further, they show that response caution can be adjusted on a trial-by-trial basis depending on incoming information. Finally, stimulus modality influenced non-cognitive components of the response. We suggest that the combined stimulus energy from simultaneously presented stimuli reduces encoding time.


Audio-visual integration Multisensory Drift diffusion model Semantic congruency Reaction times 



Thomas Otto and Michael Herzog provided helpful comments on an earlier version of this manuscript and valuable advice for the data analysis. We also would like to thank Halina Sutter and Marina Wunderlin for their help in carrying out the experiments.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Alais D, Burr D, (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262Google Scholar
  2. Assink N, Lubbe R, Fox J-P, Wang Y, Pierre BE, Rudas I (2015) Does time pressure induce tunnel vision? An examination with the Eriksen Flanker Task by applying the Hierarchical Drift Diffusion Model. Proceedings of the international conference on neural networks-fuzzy systems, pp 30–40Google Scholar
  3. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436CrossRefPubMedGoogle Scholar
  4. Chen YC, Spence C (2011) Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. J Exp Psychol Hum Percept Perform 37:1554–1568. doi: 10.1037/a0024329 CrossRefPubMedGoogle Scholar
  5. Colonius H, Diederich A (2006) The race model inequality: Interpreting a geometric measure of the amount of violation. Psychol Rev 113(1):148–154Google Scholar
  6. Domenech P, Dreher JC (2010) Decision threshold modulation in the human brain. J Neurosci 30:14305–14317. doi: 10.1523/JNEUROSCI.2371-10.2010 CrossRefPubMedGoogle Scholar
  7. Diederich A (1995) Intersensory Facilitation of Reaction Time: evaluation of counter and diffusion coactivation models. J Math Psychol 39(2):197–215Google Scholar
  8. Gondan M, Heckel A (2008) Testing the race inequality: A simple correction procedure for fast guesses. J Math Psychol 52(5):322–325Google Scholar
  9. Gondan M, Blurton SP, Hughes F, Greenlee MW (2011) Effects of spatial and selective attention on basic multisensory integration. J Exp Psychol: Hum Percept Perform 37(6):1887–1897Google Scholar
  10. Hackley SA, Valle-Inclán F (1998) Automatic alerting does not speed late motoric processes in a reaction-time task. Nature 391(6669):786–788Google Scholar
  11. Horn SS, Bayen UJ, Smith RE (2011) What can the diffusion model tell us about prospective memory? Can J Exp Psychol 65:69–75. doi: 10.1037/a0022808 CrossRefPubMedPubMedCentralGoogle Scholar
  12. Hu Z, Zhang R, Zhang Q, Liu Q, Li H (2012) Neural correlates of audiovisual integration of semantic category information. Brain Lang 121:70–75. doi: 10.1016/j.bandl.2012.01.002 CrossRefPubMedGoogle Scholar
  13. Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R (1994) Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. J Exp Psychol: Hum Percept Perform 20(1):131–153Google Scholar
  14. Jepma M, Wagenmakers EJ, Band GP, Nieuwenhuis S (2009) The effects of accessory stimuli on information processing: evidence from electrophysiology and a diffusion model analysis. J Cogn Neurosci 21:847–864. doi: 10.1162/jocn.2009.21063 CrossRefPubMedGoogle Scholar
  15. Koppen C, Alsius A, Spence C (2008) Semantic congruency and the Colavita visual dominance effect. Exp Brain Res 184:533–546. doi: 10.1007/s00221-007-1120-z CrossRefPubMedGoogle Scholar
  16. Krummenacher J, Müller HJ, Heller D (2002) Visual search for dimensionally redundant pop-out targets: parallel-coactive processing of dimensions is location specific. J Exp Psychol Hum Percept Perform 28:1303–1322. doi: 10.1037//0096-1523.28.6.1303 CrossRefPubMedGoogle Scholar
  17. Laurienti P, Kraft R, Maldjian J, Burdette J, Wallace M (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414. doi: 10.1007/s00221-004-1913-2 CrossRefPubMedGoogle Scholar
  18. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14:247–279. doi: 10.1016/0010-0285(82)90010-x CrossRefPubMedGoogle Scholar
  19. Miller J (1991) Channel interaction and the redundant-targets effect in bimodal divided attention. J Exp Psychol Hum Percept Perform 17:160–169. doi: 10.1037/0096-1523.17.1.160 CrossRefPubMedGoogle Scholar
  20. Miller J (2016) Statistical facilitation and the redundant signals effect: what are race and coactivation models? Atten, Percept, Psychophys 78(2):516–519Google Scholar
  21. Miller J, Franz V, Ulrich R (1999) Effects of auditory stimulus intensity on response force in simple, go/no-go, and choice RT tasks. Percept Psychophys 61(1):107–119Google Scholar
  22. Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465. doi: 10.1093/cercor/bhh007 CrossRefPubMedGoogle Scholar
  23. Mordkoff JT, Yantis S (1991) An interactive race model of divided attention. J Exp Psychol Hum Percept Perform 17:520–538. doi: 10.1037/0096-1523.17.2.520 CrossRefPubMedGoogle Scholar
  24. Noppeney U, Ostwald D, Werner S (2010) Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci 30:7434–7446. doi: 10.1523/JNEUROSCI.0455-10.2010 CrossRefPubMedGoogle Scholar
  25. Otto T, Mamassian P (2012) Noise and correlations in parallel perceptual decision making. Curr Biol 22:1391–1396. doi: 10.1016/j.cub.2012.05.031 CrossRefPubMedGoogle Scholar
  26. Parise CV, Harrar V, Ernst MO, Spence C (2013) Cross-correlation between auditory and visual signals promotes multisensory integration. Multisens Res 26:307–316. doi: 10.1163/22134808-00002417 CrossRefPubMedGoogle Scholar
  27. Raab DH (1962) Statistical facilitation of simple reaction times. Trans N Y Acad Sci 24:574–590. doi: 10.1111/j.2164-0947.1962.tb01433.x CrossRefPubMedGoogle Scholar
  28. Ratcliff R, McKoon G (2008) The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput 20:873–922. doi: 10.1162/neco.2008.12-06-420 CrossRefPubMedPubMedCentralGoogle Scholar
  29. Ratcliff R, Gomez P, McKoon G (2004) A diffusion model account of the lexical decision task. Psychol Rev 111:159–182. doi: 10.1037/0033-295X.111.1.159 CrossRefPubMedPubMedCentralGoogle Scholar
  30. Schwarz W (1994) Diffusion, superposition, and the redundant-targets effect. J Math Psychol 38(4):504–520Google Scholar
  31. Spiegelhalter DJ, Best NG, Carlin BP, van der Linde A (2002) Bayesian measures of model complexity and fit. J R Stat Soc Ser B (Stat Methodol) 64:583–639. doi: 10.2307/3088806 CrossRefGoogle Scholar
  32. Stahl J, Rammsayer TH (2005) Accessory stimulation in the time course of visuomotor information processing: Stimulus intensity effects on reaction time and response force. Acta Psychologica 120(1):1–18Google Scholar
  33. Starns JJ, Ratcliff R, White CN (2012) Diffusion model drift rates can be influenced by decision processes: an analysis of the strength-based mirror effect. J Exp Psychol Learn Mem Cogn 38:1137–1151. doi: 10.1037/a0028151 CrossRefPubMedPubMedCentralGoogle Scholar
  34. Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266Google Scholar
  35. Suied C, Bonneel N, Viaud-Delmon I (2009) Integration of auditory and visual information in the recognition of realistic objects. Exp Brain Res 194:91–102. doi: 10.1007/s00221-008-1672-6 CrossRefPubMedGoogle Scholar
  36. Thelen A, Talsma D, Murray MM (2015) Single-trial multisensory memories affect later auditory and visual object discrimination. Cognition 138:148–160. doi: 10.1016/j.cognition.2015.02.003 CrossRefPubMedGoogle Scholar
  37. Todd JW (1912) Reaction to multiple stimuli. Archives of psychology. The Science Press, Lancaster. doi: 10.1037/13053-000
  38. Voss A, Nagler M, Lerche V (2013) Diffusion models in experimental psychology. Exp Psychol 60(6):385–402Google Scholar
  39. Wagenmakers E-J (2009) Methodological and empirical developments for the Ratcliff diffusion model of response times and accuracy. Eur J Cogn Psychol 21:641–671. doi: 10.1080/09541440802205067 CrossRefGoogle Scholar
  40. Wiecki TV, Sofer I, Frank MJ (2013) HDDM: hierarchical Bayesian estimation of the drift-diffusion model in Python. Front Neuroinform 7:14. doi: 10.3389/fninf.2013.00014 CrossRefPubMedPubMedCentralGoogle Scholar
  41. Wille C, Ebersbach M (2016) Semantic congruency and the (reversed) Colavita effect in children and adults. J Exp Child Psychol 141:23–33. doi: 10.1016/j.jecp.2015.07.015 CrossRefPubMedGoogle Scholar
  42. Yuval-Greenberg S, Deouell L (2009) The dogs meow: asymmetrical interaction in cross-modal object recognition. Exp Brain Res 193:603–614. doi: 10.1007/s00221-008-1664-6 CrossRefPubMedGoogle Scholar
  43. Zhang J, Rowe JB (2014) Dissociable mechanisms of speed-accuracy tradeoff during visual perceptual learning are revealed by a hierarchical drift-diffusion model. Front Neurosci 8:69. doi: 10.3389/fnins.2014.00069 CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of BernBernSwitzerland
  2. 2.Center for Cognition, Learning and MemoryUniversity of BernBernSwitzerland

Personalised recommendations