Skip to main content
Log in

Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms

  • Original Paper
  • Published:
Brain Topography Aims and scope Submit manuscript

Abstract

The ability to effectively combine sensory inputs across modalities is vital for acquiring a unified percept of events. For example, watching a hammer hit a nail while simultaneously identifying the sound as originating from the event requires the ability to identify spatio-temporal congruencies and statistical regularities. In this study, we applied a reaction time and hazard function measure known as capacity (e.g., Townsend and AshbyCognitive Theory 200–239, 1978) to quantify the extent to which observers learn paired associations between simple auditory and visual patterns in a model theoretic manner. As expected, results showed that learning was associated with an increase in accuracy, but more significantly, an increase in capacity. The aim of this study was to associate capacity measures of multisensory learning, with neural based measures, namely mean global field power (GFP). We observed a co-variation between an increase in capacity, and a decrease in GFP amplitude as learning occurred. This suggests that capacity constitutes a reliable behavioral index of efficient energy expenditure in the neural domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. It is worth noting that these principles are not necessarily associated with stimulus features per se, but rather neural features including receptive field properties, effectiveness in eliciting action potentials, etc.

  2. Contingencies, or probabilistic stimulus configurations that could facilitate/inhibit audiovisual target (i.e., matched) trial responses (Mordkoff and Yantis 1991), were reduced by appropriately balancing redundant, single, and target-absent trials. Although this yielded a difference in the number of matched and mismatched audiovisual trials, the proportion remained constant across training days. Hence, holding the proportion of audiovisual matched versus mismatched trials constant across days provides a valid way to examine the effects of training on changes in capacity and neural signals.

References

  • Allison PD (1996) Fixed-effects partial likelihood for repeated events. Sociol Methods Res 25:2207–2222

    Article  Google Scholar 

  • Altieri N, Townsend JT (2011) An assessment of behavioral dynamic information processing measures in audiovisual speech perception. Front Psychol 2(238):1–15

    Google Scholar 

  • Atkinson RC, Shiffrin RM (1968) Human memory: a proposed system and its control processes. In: Spence KW, Spence JT (eds) The psychology of learning and motivation, vol 2. Academic Press, New York, pp 89–195

    Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823

    Article  CAS  PubMed  Google Scholar 

  • Belardinelli MO, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, Ferretti A et al (2004) Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn Process 5(3):167–174

    Article  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657

    Article  CAS  PubMed  Google Scholar 

  • Cappe C, Thut G, Romei V, Murray MM (2010) Auditory-visual multisensory interactions in humans: timing, topography, directionality, and sources. J Neurosci 30(38):12572–12580

    Article  CAS  PubMed  Google Scholar 

  • Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009

    Article  PubMed  Google Scholar 

  • Colonius H, Diederich A (2010) The optimal time window of visual-auditory integration: a reaction time analysis. Front Integr Neurosci 4:11

    PubMed Central  PubMed  Google Scholar 

  • Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119(6):4065–4073

    Article  PubMed Central  PubMed  Google Scholar 

  • Cowan N, Elliott EM, Saults JS, Morey CC, Mattox S, Hismjatullina A, Conway ARA (2005) On the capacity of attention: its estimation and its role in working memory and cognitive aptitudes. Cogn Psychol 51:42–100

    Article  PubMed Central  PubMed  Google Scholar 

  • Cox DR (1972) Regresion models and life tables (with discussion). J Roy Stat Soc B 34:187–220

    Google Scholar 

  • Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404

    Article  PubMed  Google Scholar 

  • Diederich A, Colonius H (2009) Crossmodal interaction in speeded responses: time window of integration model. Prog Brain Res 174:119–135

    Article  PubMed  Google Scholar 

  • Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    Article  CAS  PubMed  Google Scholar 

  • Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150

    Article  CAS  PubMed  Google Scholar 

  • Dosher B, Lu ZL (1999) Mechanisms of perceptual learning. Vision Res 39:3197–3221

    Article  CAS  PubMed  Google Scholar 

  • Erber NP (2003) Use of hearing aids by older people: influence of non-auditory factors (vision, manual dexterity). Int J Audiol 42:2S21–2S25

    Article  PubMed  Google Scholar 

  • Fiebelkorn IC, Foxe JJ, Schwartz TH, Molholm S (2010) Staying within the lines: the formation of visuospatial boundaries influences multisensory feature integration. Eur J Neurosci 31(10):1737–1743

    Article  PubMed  Google Scholar 

  • Frens MA, Van Opstal AJ, Van der Willigen RF (1995) Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept Psychophys 57(6):802–816

    Article  CAS  PubMed  Google Scholar 

  • Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189(2):133–143

    Article  PubMed  Google Scholar 

  • Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27(30):7881–7887

    Article  CAS  PubMed  Google Scholar 

  • Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293

    Article  CAS  PubMed  Google Scholar 

  • Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49(3):461–467

    Article  PubMed Central  PubMed  Google Scholar 

  • James W (1890) The principles of psychology. Harvard University Press, Boston

    Book  Google Scholar 

  • James TW, Stevenson RA (2012) The use of fMRI to assess multisensory integration. In: Wallace MH, Murray MM (eds) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London

    Google Scholar 

  • James TW, VanDerKlok RM, Stevenson RA, James KH (2011) Multisensory perception of action in posterior temporal and parietal cortices. Neuropsychologia 49(1):108–114

    Article  PubMed Central  PubMed  Google Scholar 

  • James TW, Stevenson RA, Kim S (2012) Inverse effectiveness in multisensory processing. In: Stein BE (ed) The new handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  • Johnson SA, Blaha LM, Houpt JW, Townsend JT (2010) Systems Factorial Technology provides new insights on global-local information processing in autism spectrum disorders. J Math Psychol 54:53–72

    Article  PubMed Central  PubMed  Google Scholar 

  • Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167(4):635–640

    Article  PubMed  Google Scholar 

  • Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19(4):213–223

    Article  PubMed  Google Scholar 

  • Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453

    Article  PubMed  Google Scholar 

  • Luce RD (1986) Response times: their role in inferring elementary mental organization. Oxford University Press, New York

    Google Scholar 

  • Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21(2):725–732

    Article  CAS  PubMed  Google Scholar 

  • McKee SP, Westheimer G (1978) Improvement in vernier acuity with practice. Percept Psychophys 24:258–262

    Article  CAS  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354

    Article  CAS  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662

    CAS  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75(5):1843–1857

    CAS  PubMed  Google Scholar 

  • Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I. Temporal factors. J Neurosci 7(10):3215–3229

    CAS  PubMed  Google Scholar 

  • Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25(25):5884–5893

    Article  CAS  PubMed  Google Scholar 

  • Molholm S, Ritter W, Murray M, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res 14:115–128

    Article  Google Scholar 

  • Mordkoff JT, Yantis S (1991) An interactive race model of divided attention. J Exp Psychol Hum Percept Perform 17:520–538

    Article  CAS  PubMed  Google Scholar 

  • Munhall K, Vatikiotis-Bateson E (2004) Spatial and Temporal Constraints on Audiovisual Speech Perception. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 177–188

    Google Scholar 

  • Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135

    Article  PubMed  Google Scholar 

  • Naue N, Rach S, Strüber D, Huster RJ, Zaehle T, Körner U, Herrmann CS (2011) Auditory event-related responses in visual cortex modulates subsequent visual responses in humans. J Neurosci 31(21):7729–7736

    Article  CAS  PubMed  Google Scholar 

  • Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley, RL (1998) Effects of localized auditory information on visual target detection performance using helmet-mounted display. Hum Factors 40(3):452–460

    Article  CAS  PubMed  Google Scholar 

  • Neufeld RWJ, Townsend JT, Jetté J (2007) Quantitative response time technology for measuring cognitive-processing capacity in clinical studies. In: Neufeld RWJ (ed) Advances in clinical cognitive science: formal modeling and assessment of processes and symptoms. American Psychological Association, Washington DC

    Chapter  Google Scholar 

  • Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18(3):598–609

    Article  PubMed  Google Scholar 

  • Pilling M (2009) Auditory event-related potentials (ERPs) in audiovisual speech perception. J Speech Lang Hearing Res 52:1073–1081

    Article  Google Scholar 

  • Powers AR, Hevey MA, Wallace MT (2012) Neural correlates of multisensory perceptual learning. J Neurosci 32(18):6263–6274

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Powers AR 3rd, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29(39):12265–12274

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Saffran JR, Johnson EK, Aslin RN, Newport EL (1999) Statistical learning of tone sequences by human infants and adults. Cognition 70:27–52

    Article  CAS  PubMed  Google Scholar 

  • Seitz A, Kim, Shams M (2006) Sound Facilitates Visual Learning. Curr Biol 16(14):1422–1427

    Article  CAS  PubMed  Google Scholar 

  • Seitz A, Kim, Van Wassenhove V, Shams M (2007) Simultaneous and Independent Acquisition of Multisensory and Unisensory Associations. Perception 36:1445–1453

    Article  PubMed  Google Scholar 

  • Shams M, Seitz A (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417

    Article  PubMed  Google Scholar 

  • Skrandies W (1990) Global field power and topographic similarity. Brain Topogr 3:147–541

    Article  Google Scholar 

  • Stein BE, Stanford TR, Ramachandran R, Perrault TJ Jr, Rowland BA (2009) Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198(2–3):113–126

    Google Scholar 

  • Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44(3):1210–1223

    Article  PubMed  Google Scholar 

  • Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95

    Article  PubMed  Google Scholar 

  • Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198(2–3):183–194

    Article  PubMed  Google Scholar 

  • Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. Neuroimage 49(4):3308–3318

    Article  PubMed Central  PubMed  Google Scholar 

  • Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. Neuroimage 55(3):1339–1345

    Article  PubMed Central  PubMed  Google Scholar 

  • Stevenson RA, Krueger Fister J, Barnett ZP, Nidiffer AR, Wallace MT (2012) Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp Brain Res 219:121–137

    Article  PubMed Central  PubMed  Google Scholar 

  • Stevenson RA, Wilson MM, Powers AR, Wallace MT (2013) The effects of visual training on multisensory temporal processing. Exp Brain Res. doi:10.1007/s00221-012-3387-y

    Google Scholar 

  • Talsma D, Senkowski D, Woldorff MG (2009) Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp Brain Res 198(2–3):313–328

    Article  PubMed Central  PubMed  Google Scholar 

  • Tanabe S, Doi T, Umeda K, Fujita I (2005) Disparity-tuning characteristics of neuronal responses to dynamic random-dot stereograms in macaque visual area V4. J Neurophysiol 94(4):2683–2699

    Article  PubMed  Google Scholar 

  • Taylor KI, Moss HE, Stamatakis EA, Tyler LK (2006) Binding crossmodal object features in perirhinal cortex. Proc Natl Acad Sci U S A 103(21):8239–8244

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  • Thelen A, Murray MM (2013) The efficacy of single-trial multisensory memories. Multisens Res. doi:10.1163/22134808-00002426

    PubMed  Google Scholar 

  • Thelen A, Cappe C, Murray MM (2012) Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62(3):1478–1488

    Article  PubMed  Google Scholar 

  • Townsend JT (1990) The truth and consequences of ordinal differences in statistical distributions: toward a theory of hierarchical inference. Psychol Bull 108:551–567

    Article  CAS  PubMed  Google Scholar 

  • Townsend JT, Altieri NA (2012) An accuracy-response time capacity assessment function that measures performance against standard parallel predictions. Psychol Rev. doi:10.1037/a0028448

    PubMed  Google Scholar 

  • Townsend JT, Ashby FG (1978) Methods of modeling capacity in simple processing systems. In: Castellan J, Restle F (eds) Cognitive theory, vol III. Erlbaum Associates, Hillsdale, pp 200–239

    Google Scholar 

  • Townsend JT, Nozawa G (1995) Spatio-temporal properties of elementary perception: an investigation of parallel, serial and coactive theories. J Math Psychol 39:321–360

    Article  Google Scholar 

  • Townsend JT, Wenger MJ (2004) A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. Psychol Rev 111(4):1003–1035

    Article  PubMed  Google Scholar 

  • van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974

    Article  PubMed  Google Scholar 

  • van Atteveldt N, Formisano E, Goebel R, Blomert, L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282

    Article  PubMed  Google Scholar 

  • van Wassenhove V, Grant K, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186

    Article  PubMed Central  PubMed  Google Scholar 

  • van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607

    Article  PubMed  Google Scholar 

  • Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22:32–35

    Article  Google Scholar 

  • Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76(2):1246–1266

    CAS  PubMed  Google Scholar 

  • Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258

    Article  CAS  PubMed  Google Scholar 

  • Wallace MT, Carriere BN, Perrault TJ Jr, Vaughan JW, Stein BE (2006) The development of cortical multisensory integration. J Neurosci 26(46):11844–11849

    Article  CAS  PubMed  Google Scholar 

  • Wenger MJ, Gibson BS (2004) Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. J Exp Psychol Hum Percept Perform 30:708–719

    Article  PubMed  Google Scholar 

  • Wenger MJ, Negash S, Petersen RC, Petersen L (2010) Modeling and estimating recall processing capacity: sensitivity and diagnostic utility in application to mild cognitive impairment. J Math Psychol 54:73–89

    Article  PubMed Central  PubMed  Google Scholar 

  • Zampini M, Guest S, Shore DI, Spence C (2005) Audio-visual simultaneity judgments. Percept Psychophys 67(3):531–544

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nicholas Altieri.

Additional information

This is one of several papers published together in Brain Topography on the “Special Issue: Auditory Cortex 2012”.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Altieri, N., Stevenson, R.A., Wallace, M.T. et al. Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms. Brain Topogr 28, 479–493 (2015). https://doi.org/10.1007/s10548-013-0333-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10548-013-0333-7

Keywords

Navigation