Abstract
The ability to effectively combine sensory inputs across modalities is vital for acquiring a unified percept of events. For example, watching a hammer hit a nail while simultaneously identifying the sound as originating from the event requires the ability to identify spatio-temporal congruencies and statistical regularities. In this study, we applied a reaction time and hazard function measure known as capacity (e.g., Townsend and AshbyCognitive Theory 200–239, 1978) to quantify the extent to which observers learn paired associations between simple auditory and visual patterns in a model theoretic manner. As expected, results showed that learning was associated with an increase in accuracy, but more significantly, an increase in capacity. The aim of this study was to associate capacity measures of multisensory learning, with neural based measures, namely mean global field power (GFP). We observed a co-variation between an increase in capacity, and a decrease in GFP amplitude as learning occurred. This suggests that capacity constitutes a reliable behavioral index of efficient energy expenditure in the neural domain.
Similar content being viewed by others
Notes
It is worth noting that these principles are not necessarily associated with stimulus features per se, but rather neural features including receptive field properties, effectiveness in eliciting action potentials, etc.
Contingencies, or probabilistic stimulus configurations that could facilitate/inhibit audiovisual target (i.e., matched) trial responses (Mordkoff and Yantis 1991), were reduced by appropriately balancing redundant, single, and target-absent trials. Although this yielded a difference in the number of matched and mismatched audiovisual trials, the proportion remained constant across training days. Hence, holding the proportion of audiovisual matched versus mismatched trials constant across days provides a valid way to examine the effects of training on changes in capacity and neural signals.
References
Allison PD (1996) Fixed-effects partial likelihood for repeated events. Sociol Methods Res 25:2207–2222
Altieri N, Townsend JT (2011) An assessment of behavioral dynamic information processing measures in audiovisual speech perception. Front Psychol 2(238):1–15
Atkinson RC, Shiffrin RM (1968) Human memory: a proposed system and its control processes. In: Spence KW, Spence JT (eds) The psychology of learning and motivation, vol 2. Academic Press, New York, pp 89–195
Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823
Belardinelli MO, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, Ferretti A et al (2004) Audio-visual crossmodal interactions in environmental perception: an fMRI investigation. Cogn Process 5(3):167–174
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657
Cappe C, Thut G, Romei V, Murray MM (2010) Auditory-visual multisensory interactions in humans: timing, topography, directionality, and sources. J Neurosci 30(38):12572–12580
Colonius H, Diederich A (2004) Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci 16(6):1000–1009
Colonius H, Diederich A (2010) The optimal time window of visual-auditory integration: a reaction time analysis. Front Integr Neurosci 4:11
Conrey B, Pisoni DB (2006) Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am 119(6):4065–4073
Cowan N, Elliott EM, Saults JS, Morey CC, Mattox S, Hismjatullina A, Conway ARA (2005) On the capacity of attention: its estimation and its role in working memory and cognitive aptitudes. Cogn Psychol 51:42–100
Cox DR (1972) Regresion models and life tables (with discussion). J Roy Stat Soc B 34:187–220
Diederich A, Colonius H (2004) Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept Psychophys 66(8):1388–1404
Diederich A, Colonius H (2009) Crossmodal interaction in speeded responses: time window of integration model. Prog Brain Res 174:119–135
Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721
Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150
Dosher B, Lu ZL (1999) Mechanisms of perceptual learning. Vision Res 39:3197–3221
Erber NP (2003) Use of hearing aids by older people: influence of non-auditory factors (vision, manual dexterity). Int J Audiol 42:2S21–2S25
Fiebelkorn IC, Foxe JJ, Schwartz TH, Molholm S (2010) Staying within the lines: the formation of visuospatial boundaries influences multisensory feature integration. Eur J Neurosci 31(10):1737–1743
Frens MA, Van Opstal AJ, Van der Willigen RF (1995) Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept Psychophys 57(6):802–816
Hecht D, Reiner M, Karni A (2008) Multisensory enhancement: gains in choice and in simple response times. Exp Brain Res 189(2):133–143
Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27(30):7881–7887
Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293
Hillock AR, Powers AR, Wallace MT (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia 49(3):461–467
James W (1890) The principles of psychology. Harvard University Press, Boston
James TW, Stevenson RA (2012) The use of fMRI to assess multisensory integration. In: Wallace MH, Murray MM (eds) Frontiers in the neural basis of multisensory processes. Taylor & Francis, London
James TW, VanDerKlok RM, Stevenson RA, James KH (2011) Multisensory perception of action in posterior temporal and parietal cortices. Neuropsychologia 49(1):108–114
James TW, Stevenson RA, Kim S (2012) Inverse effectiveness in multisensory processing. In: Stein BE (ed) The new handbook of multisensory processes. MIT Press, Cambridge
Johnson SA, Blaha LM, Houpt JW, Townsend JT (2010) Systems Factorial Technology provides new insights on global-local information processing in autism spectrum disorders. J Math Psychol 54:53–72
Keetels M, Vroomen J (2005) The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp Brain Res 167(4):635–640
Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19(4):213–223
Lovelace CT, Stein BE, Wallace MT (2003) An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Brain Res Cogn Brain Res 17(2):447–453
Luce RD (1986) Response times: their role in inferring elementary mental organization. Oxford University Press, New York
Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21(2):725–732
McKee SP, Westheimer G (1978) Improvement in vernier acuity with practice. Percept Psychophys 24:258–262
Meredith MA, Stein BE (1986a) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365(2):350–354
Meredith MA, Stein BE (1986b) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56(3):640–662
Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75(5):1843–1857
Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I. Temporal factors. J Neurosci 7(10):3215–3229
Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25(25):5884–5893
Molholm S, Ritter W, Murray M, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res 14:115–128
Mordkoff JT, Yantis S (1991) An interactive race model of divided attention. J Exp Psychol Hum Percept Perform 17:520–538
Munhall K, Vatikiotis-Bateson E (2004) Spatial and Temporal Constraints on Audiovisual Speech Perception. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 177–188
Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135
Naue N, Rach S, Strüber D, Huster RJ, Zaehle T, Körner U, Herrmann CS (2011) Auditory event-related responses in visual cortex modulates subsequent visual responses in humans. J Neurosci 31(21):7729–7736
Nelson WT, Hettinger LJ, Cunningham JA, Brickman BJ, Haas MW, McKinley, RL (1998) Effects of localized auditory information on visual target detection performance using helmet-mounted display. Hum Factors 40(3):452–460
Neufeld RWJ, Townsend JT, Jetté J (2007) Quantitative response time technology for measuring cognitive-processing capacity in clinical studies. In: Neufeld RWJ (ed) Advances in clinical cognitive science: formal modeling and assessment of processes and symptoms. American Psychological Association, Washington DC
Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18(3):598–609
Pilling M (2009) Auditory event-related potentials (ERPs) in audiovisual speech perception. J Speech Lang Hearing Res 52:1073–1081
Powers AR, Hevey MA, Wallace MT (2012) Neural correlates of multisensory perceptual learning. J Neurosci 32(18):6263–6274
Powers AR 3rd, Hillock AR, Wallace MT (2009) Perceptual training narrows the temporal window of multisensory binding. J Neurosci 29(39):12265–12274
Saffran JR, Johnson EK, Aslin RN, Newport EL (1999) Statistical learning of tone sequences by human infants and adults. Cognition 70:27–52
Seitz A, Kim, Shams M (2006) Sound Facilitates Visual Learning. Curr Biol 16(14):1422–1427
Seitz A, Kim, Van Wassenhove V, Shams M (2007) Simultaneous and Independent Acquisition of Multisensory and Unisensory Associations. Perception 36:1445–1453
Shams M, Seitz A (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417
Skrandies W (1990) Global field power and topographic similarity. Brain Topogr 3:147–541
Stein BE, Stanford TR, Ramachandran R, Perrault TJ Jr, Rowland BA (2009) Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198(2–3):113–126
Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44(3):1210–1223
Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179(1):85–95
Stevenson RA, Kim S, James TW (2009) An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198(2–3):183–194
Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW (2010) Neural processing of asynchronous audiovisual speech perception. Neuroimage 49(4):3308–3318
Stevenson RA, VanDerKlok RM, Pisoni DB, James TW (2011) Discrete neural substrates underlie complementary audiovisual speech integration processes. Neuroimage 55(3):1339–1345
Stevenson RA, Krueger Fister J, Barnett ZP, Nidiffer AR, Wallace MT (2012) Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Exp Brain Res 219:121–137
Stevenson RA, Wilson MM, Powers AR, Wallace MT (2013) The effects of visual training on multisensory temporal processing. Exp Brain Res. doi:10.1007/s00221-012-3387-y
Talsma D, Senkowski D, Woldorff MG (2009) Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp Brain Res 198(2–3):313–328
Tanabe S, Doi T, Umeda K, Fujita I (2005) Disparity-tuning characteristics of neuronal responses to dynamic random-dot stereograms in macaque visual area V4. J Neurophysiol 94(4):2683–2699
Taylor KI, Moss HE, Stamatakis EA, Tyler LK (2006) Binding crossmodal object features in perirhinal cortex. Proc Natl Acad Sci U S A 103(21):8239–8244
Thelen A, Murray MM (2013) The efficacy of single-trial multisensory memories. Multisens Res. doi:10.1163/22134808-00002426
Thelen A, Cappe C, Murray MM (2012) Electrical neuroimaging of memory discrimination based on single-trial multisensory learning. Neuroimage 62(3):1478–1488
Townsend JT (1990) The truth and consequences of ordinal differences in statistical distributions: toward a theory of hierarchical inference. Psychol Bull 108:551–567
Townsend JT, Altieri NA (2012) An accuracy-response time capacity assessment function that measures performance against standard parallel predictions. Psychol Rev. doi:10.1037/a0028448
Townsend JT, Ashby FG (1978) Methods of modeling capacity in simple processing systems. In: Castellan J, Restle F (eds) Cognitive theory, vol III. Erlbaum Associates, Hillsdale, pp 200–239
Townsend JT, Nozawa G (1995) Spatio-temporal properties of elementary perception: an investigation of parallel, serial and coactive theories. J Math Psychol 39:321–360
Townsend JT, Wenger MJ (2004) A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. Psychol Rev 111(4):1003–1035
van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974
van Atteveldt N, Formisano E, Goebel R, Blomert, L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282
van Wassenhove V, Grant K, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186
van Wassenhove V, Grant KW, Poeppel D (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3):598–607
Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 22:32–35
Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76(2):1246–1266
Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258
Wallace MT, Carriere BN, Perrault TJ Jr, Vaughan JW, Stein BE (2006) The development of cortical multisensory integration. J Neurosci 26(46):11844–11849
Wenger MJ, Gibson BS (2004) Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. J Exp Psychol Hum Percept Perform 30:708–719
Wenger MJ, Negash S, Petersen RC, Petersen L (2010) Modeling and estimating recall processing capacity: sensitivity and diagnostic utility in application to mild cognitive impairment. J Math Psychol 54:73–89
Zampini M, Guest S, Shore DI, Spence C (2005) Audio-visual simultaneity judgments. Percept Psychophys 67(3):531–544
Author information
Authors and Affiliations
Corresponding author
Additional information
This is one of several papers published together in Brain Topography on the “Special Issue: Auditory Cortex 2012”.
Rights and permissions
About this article
Cite this article
Altieri, N., Stevenson, R.A., Wallace, M.T. et al. Learning to Associate Auditory and Visual Stimuli: Behavioral and Neural Mechanisms. Brain Topogr 28, 479–493 (2015). https://doi.org/10.1007/s10548-013-0333-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10548-013-0333-7