Skip to main content
SpringerLink
Log in

Single-Trial Multisensory Learning and Memory Retrieval

  • Chapter
  • First Online:
Book cover Multisensory Object Perception in the Primate Brain

Abstract

This chapter reviews our research investigating how the processing of unisensory visual objects is impacted by past experiences with multisensory auditory–visual versions of these objects. Single-trial exposure to multisensory objects during an orthogonal old/new discrimination task can incidentally modulate memory performance as well as brain activity. Performance was enhanced when the preceding multisensory stimulus had been a semantically congruent pairing, whereas performance was impaired by pairings with meaningless pure tones. Importantly, these opposing effects occurred despite identical performance patterns with the initial exposure to the objects (and presumably encoding thereof). In terms of brain activity, responses were enhanced to visual stimuli that had been previously encountered in a multisensory context within regions of the lateral occipital cortex beginning just 60 ms post-stimulus onset. Our collective findings indicate that single-trial learning of multisensory object associations occurs incidentally, is distinguishable from encoding processes, persists despite many intervening stimuli, and manifests as differential brain activity at early processing stages within visual cortices. The consequences of multisensory interactions thus persist over time to impact memory retrieval and object discrimination.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    An alternative viewpoint is that the V+ condition is not a pure repetition, but rather a novel context (see Section 2.3 for discussion).

References

  • Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp MS, Naumer MJ (2005) Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166:559–71

    Article  PubMed  CAS  Google Scholar 

  • Amedi A, Jacobson G, Hendler T, Malach R, Zohary E (2002) Convergence of visual and tactile shape processing in the human lateral occipital complex. Cereb Cortex 12:1202–1212

    Article  PubMed  Google Scholar 

  • Amedi A, Malach R, Hendler T, Peled S, Zohary E (2001) Visuohaptic object-related activation in the ventral visual pathway. Nat Neurosci 4:324–330

    Article  PubMed  CAS  Google Scholar 

  • Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192

    Article  PubMed  CAS  Google Scholar 

  • Besle J, Bertrand O, Giard MH (2009) Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 258(1–2):143–151

    Google Scholar 

  • Braeutigam S, Bailey AJ, Swithenby SJ (2001) Task-dependent early latency (30–60 ms) visual processing of human faces and other objects. Neuroreport 12:1531–36

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during cross-modal binding. Neuroreport 10:2619–23

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–57

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio–visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–38

    Article  PubMed  CAS  Google Scholar 

  • Cappe C, Morel A, Barone P, Rouiller EM (2009) The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor integrations. Cereb Cortex. 19(9):2025–2037, doi:10.1093/cercor/bhn228

    Google Scholar 

  • Chen YC, Yeh SL (2008) Visual events modulated by sound in repetition blindness. Psychonom Bull Rev 15:404–408

    Article  Google Scholar 

  • Colombo M, Gross CG (1994) Responses of inferior temporal cortex and hippocampal neurons during delayed matching to sample in monkeys (Macaca fascicularis). Behav Neurosci 108:443–55

    Article  PubMed  CAS  Google Scholar 

  • Debruille JB, Guillem F, Renault B (1998) ERPs and chronometry of face recognition: following-up Seeck et al. and George et al. Neuroreport 9:3349–53

    Article  PubMed  CAS  Google Scholar 

  • Deibert E, Kraut M, Kremen S, Hart JJ (1999) Neural pathways in tactile object recognition. Neurology 52:1413–7

    Article  PubMed  CAS  Google Scholar 

  • Desimone, R (1996) Neural mechanisms for visual memory and their role in attention. Proc Natl Acad Sci USA 93:13494–9

    Article  PubMed  CAS  Google Scholar 

  • Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150

    Article  PubMed  CAS  Google Scholar 

  • Doniger GM, Foxe JJ, Murray MM, Higgins BA, Snodgrass JG, Schroeder CE, Javitt DC (2000) Activation timecourse of ventral visual stream object-recognition areas: high density electrical mapping of perceptual closure processes. J Cogn Neurosci 12:615–621

    Article  PubMed  CAS  Google Scholar 

  • Doniger GM, Foxe JJ, Schroeder CE, Murray MM, Higgins BA, Javitt DC (2001) Visual perceptual learning in human object recognition areas: a repetition priming study using high-density electrical mapping. Neuroimage 13:305–313

    Article  PubMed  CAS  Google Scholar 

  • Fort A, Delpuech C, Pernier J, Giard MH (2002a) Dynamics of cortico-subcortical cross-modal operations involved in audio–visual object detection in humans. Cereb Cortex 12:1031–9

    Article  PubMed  Google Scholar 

  • Fort A, Delpuech C, Pernier J, Giard MH (2002b) Early auditory–visual interactions in human cortex during nonredundant target identification. Brain Res Cogn Brain Res 14:20–30

    Article  PubMed  Google Scholar 

  • Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77–83

    Article  PubMed  CAS  Google Scholar 

  • George N, Jemel B, Fiore N, Renault B (1997) Face and shape repetition effects in humans: a spatio-temporal ERP study. Neuroreport 8:1417–23

    Article  PubMed  CAS  Google Scholar 

  • Giard MH, Peronnet, F (1999) Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–90

    Article  PubMed  CAS  Google Scholar 

  • Gibson JR, Maunsell JHR (1997) Sensory modality specificity of neural activity related to memory in visual cortex. J Neurophysiol 78:1263–75

    PubMed  CAS  Google Scholar 

  • Gottfried JA, Smith APR, Rugg MD, Dolan RJ (2004) Remembrance of odors past: human olfactory cortex in cross-modal recognition memory. Neuron 42:687–95

    Article  PubMed  CAS  Google Scholar 

  • Goulet S, Murray EA (2001) Neural substrates of crossmodal association memory in monkeys: the amygdala versus the anterior rhinal cortex. Behav Neurosci 115:271–284

    Article  PubMed  CAS  Google Scholar 

  • Grill-Spector K, Henson RN, Martin A (2006) Repetition and the brain, neural models of stimulus specific effects. Trends Cogn Sci 10:14–23

    Article  PubMed  Google Scholar 

  • Guo J, Guo A (2005) Crossmodal interactions between olfactory and visual learning in Drosophila. Science 309:307–310

    Article  PubMed  CAS  Google Scholar 

  • Haenny PE, Maunsell JHR, Schiler PH (1988) State dependent activity in monkey visual cortex: II. Retinal and extraretinal factors in V4. Exp Brain Res 69:245–259

    Article  PubMed  CAS  Google Scholar 

  • Halgren E, Raij T, Marinkovic K, Jousmaki V, Hari R (2000) Cognitive response profile of the human fusiform face area as determined by MEG. Cereb Cortex 10:69–81

    Article  PubMed  CAS  Google Scholar 

  • Hamilton W (1859) Lectures on Metaphysics and Logic. Gould & Lincoln, Boston

    Book  Google Scholar 

  • Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430

    Article  PubMed  CAS  Google Scholar 

  • Hein G, Doehrmann O, Müller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887

    Article  PubMed  CAS  Google Scholar 

  • Henson RN, Rugg MD (2003) Neural response suppression, haemodynamic repetition effects, and behavioural priming. Neuropsychologia 41:263–270

    Article  PubMed  CAS  Google Scholar 

  • Holmes NP (2009) The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topogr 21(3–4):168–176

    Google Scholar 

  • James TW, Humphrey GK, Gati JS, Servos P, Menon RS, Goodale MA (2002) Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia 40:1706–1714

    Article  PubMed  Google Scholar 

  • Landis T, Lehmann D, Mita T, Skrandies W (1984) Evoked potential correlates of figure and ground Int J Psychophysiol 1:345–348

    Article  CAS  Google Scholar 

  • Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414

    Article  PubMed  Google Scholar 

  • Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223

    Article  PubMed  Google Scholar 

  • Lehmann S, Murray MM (2005) The role of multisensory memories in unisensory object discrimination. Brain Res Cogn Brain Res 24:326–334

    Article  PubMed  Google Scholar 

  • MacSweeney M, Woll B, Campbell R, McGuire PK, David AS, Williams SC, Suckling J, Calvert GA, Brammer MJ (2002) Neural systems underlying British Sign Language and audio–visual English processing in native users. Brain 125:1583–93

    Article  PubMed  Google Scholar 

  • Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP, Clarke S, Meuli RA (2007) Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb Cortex 17:1672–1779

    Article  PubMed  Google Scholar 

  • Maunsell JHR, Sclar G, Nealey TA, Depriest DD (1991) Extraretinal representations in area V4 in the macaque monkey. Vis Neurosci 7:561–573

    Article  PubMed  CAS  Google Scholar 

  • Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, Grave de Peralta R (2004) EEG source imaging. Clin Neurophysiol 115:2195–222

    Article  PubMed  Google Scholar 

  • Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465

    Article  PubMed  Google Scholar 

  • Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128

    Article  PubMed  Google Scholar 

  • Mouchetant-Rostaing Y, Giard MH, Delpuech C, Echallier JF, Pernier J (2000a) Early signs of visual categorization for biological and non-biological stimuli in humans. Neuroreport 11:2521–2525

    Article  PubMed  CAS  Google Scholar 

  • Mouchetant-Rostaing Y, Giard MH, Bentin S, Aguera PE, Pernier J (2000b) Neurophysiological correlates of face gender processing in humans. Eur J Neurosci 12:303–310

    Article  PubMed  CAS  Google Scholar 

  • Murray MM, Camen C, Spierer L, Clarke S (2008a) Plasticity in representations of environmental sounds revealed by electrical neuroimaging. Neuroimage 39:847–856

    Article  PubMed  Google Scholar 

  • Murray MM, Brunet D, Michel CM (2008b) Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr 20:249–264

    Article  PubMed  Google Scholar 

  • Murray MM, Camen C, Gonzalez Andino SL, Bovet P, Clarke S (2006) Rapid brain discrimination of sounds of objects. J Neurosci 26:1293–1302

    Article  PubMed  CAS  Google Scholar 

  • Murray MM, Foxe JJ, Wylie GR (2005a) The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage 27:473–478

    Article  PubMed  Google Scholar 

  • Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ (2005b) Grabbing your ear: rapid auditory–somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15:963–974

    Article  PubMed  Google Scholar 

  • Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135

    Article  PubMed  Google Scholar 

  • Naumer MJ, Doehrmann O, Müller NG, Muckli L, Kaiser J, Hein G (2009) Cortical plasticity of audio-visual object representations. Cereb Cortex. 19(7):1641–1653, doi: 10.1093/cercor/ bhn200

    Google Scholar 

  • Nyberg L, Habib R, McIntosh AR, Tulving E (2000) Reactivation of encoding-related brain activity during memory retrieval. Proc Natl Acad Sci USA 97:11120–11124

    Article  PubMed  CAS  Google Scholar 

  • Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625

    Article  PubMed  CAS  Google Scholar 

  • Ranganath C, Rainer G (2003) Neural mechanisms for detecting and remembering novel events. Nat Rev Neurosci 4:193–202

    Article  PubMed  CAS  Google Scholar 

  • Ritter W, Simson R, Vaughan Jr HG, Macht M (1982) Manipulation of event-related potential manifestations of information processing stages. Science 218:909–911

    Article  PubMed  CAS  Google Scholar 

  • Rolls ET, Treves A, Tovee MJ (1997) The representational capacity of the distributed encoding of information provided by populations of neurons in primate temporal visual cortex. Exp Brain Res 114:149–162

    Article  PubMed  CAS  Google Scholar 

  • Romei V, Murray MM, Merabet LB, Thut G (2007) Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J Neurosci 27:11465–11472

    Article  PubMed  CAS  Google Scholar 

  • Rousselet GA, Fabre-Thorpe M, Thorpe SJ (2002) Parallel processing in high-level categorization of natural images. Nat Neurosci 5:629–630

    PubMed  CAS  Google Scholar 

  • Rugg MD, Mark RE, Walla P, Schloerscheidt AM, Birch CS, Allan K (1998) Dissociation of the neural correlates of implicit and explicit memory. Nature 392:595–598

    Article  PubMed  CAS  Google Scholar 

  • Schnider A, Valenza N, Morand S, Michel CM (2002) Early cortical distinction between memories that pertain to ongoing reality and memories that don’t. Cereb Cortex 12:54–61

    Article  PubMed  Google Scholar 

  • Seeck M, Michel CM, Mainwaring N, Cosgrove R, Blume H, Ives J, Landis T, Schomer DL (1997) Evidence for rapid face recognition from human scalp and intracranial electrodes. Neuroreport 8:2749–2754

    Article  PubMed  CAS  Google Scholar 

  • Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12:411–417

    Article  PubMed  Google Scholar 

  • Spence C, Nicholls ME, Driver J (2001) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63:330–336

    Article  PubMed  CAS  Google Scholar 

  • Sperdin HF, Cappe C, Foxe JJ, Murray MM (2009) Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front Integr Neurosci 3:2. doi:10.3389/neuro.07.002.2009

    Article  PubMed  Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA

    Google Scholar 

  • Stoesz MR, Zhang M, Weisser VD, Prather SC, Mao H, Sathian K (2003) Neural networks active during tactile form perception: common and differential activity during macrospatial and microspatial tasks. Int J Psychophysiol 50:41–49

    Article  PubMed  Google Scholar 

  • Thierry G, Martin CD, Downing P, Pegna AJ (2007) Controlling for interstimulus perceptual variance abolishes N170 face selectivity. Nat Neurosci 10:505–511

    Article  PubMed  CAS  Google Scholar 

  • Tsivilis D, Otten LJ, Rugg MD (2001) Context effects on the neural correlates of recognition memory: an electrophysiological study. Neuron 31:497–505

    Article  PubMed  CAS  Google Scholar 

  • Tulving E, Schacter DL (1990) Priming and human memory systems. Science 247:301–306

    Article  PubMed  CAS  Google Scholar 

  • Vanni S, Revonsuo A, Saarinen J, Hari R (1996) Visual awareness of objects correlates with activity of right occipital cortex. Neuroreport 8:183–186

    Article  PubMed  CAS  Google Scholar 

  • Von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4:e326

    Article  Google Scholar 

  • Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79

    Article  PubMed  Google Scholar 

  • Wheeler ME, Petersen SE, Buckner RL (2000) Memory’s echo: vivid remembering reactivates sensory-specific cortex. Proc Natl Acad Sci USA 97:11125–11129

    Article  PubMed  CAS  Google Scholar 

  • Wiggs CL, Martin A (1998) Properties and mechanisms of perceptual priming. Curr Opin Neurobiol 8:227–233

    Article  PubMed  CAS  Google Scholar 

  • Zangaladze A, Epstein CM, Grafton ST, Sathian K (1999) Involvement of visual cortex in tactile discrimination of orientation. Nature 401:587–590

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Biomedical Imaging, Department of Clinical Neurosciences, Department of Radiology, Vaudois University Hospital Centre and University of Lausanne, BH08.078, Rue du Bugnon 46, 1011, Lausanne, Switzerland

    Micah M. Murray

  2. Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, Tennessee, USA

    Micah M. Murray

  3. Neuropsychology and Neurorehabilitation Service, Centre Hospitalier Universitaire Vaudois and University of Lausanne, Hôpital Nestlé, 5 av. Pierre Decker, 1011, Lausanne, Switzerland

    Holger F. Sperdin

Authors
  1. Micah M. Murray
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Holger F. Sperdin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Micah M. Murray .

Editor information

Editors and Affiliations

  1. Inst. Medizinische Psychologie (IMP), Universität Frankfurt, Heinrich-Hoffmann-Str. 10, Frankfurt, 60528, Germany

    Jochen Kaiser

  2. Abt. Neurophysiologie, MPI für Hirnforschung, Deutschordenstr. 46, Frankfurt/Main, 60528, Germany

    Marcus Johannes Naumer

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer Science + Business Media, LLC

About this chapter

Cite this chapter

Murray, M.M., Sperdin, H.F. (2010). Single-Trial Multisensory Learning and Memory Retrieval. In: Kaiser, J., Naumer, M. (eds) Multisensory Object Perception in the Primate Brain. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-5615-6_11

Download citation

Publish with us

Policies and ethics

Search

Navigation