Single-Trial Multisensory Learning and Memory Retrieval



This chapter reviews our research investigating how the processing of unisensory visual objects is impacted by past experiences with multisensory auditory–visual versions of these objects. Single-trial exposure to multisensory objects during an orthogonal old/new discrimination task can incidentally modulate memory performance as well as brain activity. Performance was enhanced when the preceding multisensory stimulus had been a semantically congruent pairing, whereas performance was impaired by pairings with meaningless pure tones. Importantly, these opposing effects occurred despite identical performance patterns with the initial exposure to the objects (and presumably encoding thereof). In terms of brain activity, responses were enhanced to visual stimuli that had been previously encountered in a multisensory context within regions of the lateral occipital cortex beginning just 60 ms post-stimulus onset. Our collective findings indicate that single-trial learning of multisensory object associations occurs incidentally, is distinguishable from encoding processes, persists despite many intervening stimuli, and manifests as differential brain activity at early processing stages within visual cortices. The consequences of multisensory interactions thus persist over time to impact memory retrieval and object discrimination.


Auditory Cortex Object Representation Rapid Serial Visual Presentation Repeated Presentation Repetition Blindness 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp MS, Naumer MJ (2005) Functional imaging of human crossmodal identification and object recognition. Exp Brain Res 166:559–71PubMedCrossRefGoogle Scholar
  2. Amedi A, Jacobson G, Hendler T, Malach R, Zohary E (2002) Convergence of visual and tactile shape processing in the human lateral occipital complex. Cereb Cortex 12:1202–1212PubMedCrossRefGoogle Scholar
  3. Amedi A, Malach R, Hendler T, Peled S, Zohary E (2001) Visuohaptic object-related activation in the ventral visual pathway. Nat Neurosci 4:324–330PubMedCrossRefGoogle Scholar
  4. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192PubMedCrossRefGoogle Scholar
  5. Besle J, Bertrand O, Giard MH (2009) Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 258(1–2):143–151Google Scholar
  6. Braeutigam S, Bailey AJ, Swithenby SJ (2001) Task-dependent early latency (30–60 ms) visual processing of human faces and other objects. Neuroreport 12:1531–36PubMedCrossRefGoogle Scholar
  7. Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during cross-modal binding. Neuroreport 10:2619–23PubMedCrossRefGoogle Scholar
  8. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–57PubMedCrossRefGoogle Scholar
  9. Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio–visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–38PubMedCrossRefGoogle Scholar
  10. Cappe C, Morel A, Barone P, Rouiller EM (2009) The thalamocortical projection systems in primate: an anatomical support for multisensory and sensorimotor integrations. Cereb Cortex. 19(9):2025–2037, doi:10.1093/cercor/bhn228Google Scholar
  11. Chen YC, Yeh SL (2008) Visual events modulated by sound in repetition blindness. Psychonom Bull Rev 15:404–408CrossRefGoogle Scholar
  12. Colombo M, Gross CG (1994) Responses of inferior temporal cortex and hippocampal neurons during delayed matching to sample in monkeys (Macaca fascicularis). Behav Neurosci 108:443–55PubMedCrossRefGoogle Scholar
  13. Debruille JB, Guillem F, Renault B (1998) ERPs and chronometry of face recognition: following-up Seeck et al. and George et al. Neuroreport 9:3349–53PubMedCrossRefGoogle Scholar
  14. Deibert E, Kraut M, Kremen S, Hart JJ (1999) Neural pathways in tactile object recognition. Neurology 52:1413–7PubMedCrossRefGoogle Scholar
  15. Desimone, R (1996) Neural mechanisms for visual memory and their role in attention. Proc Natl Acad Sci USA 93:13494–9PubMedCrossRefGoogle Scholar
  16. Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150PubMedCrossRefGoogle Scholar
  17. Doniger GM, Foxe JJ, Murray MM, Higgins BA, Snodgrass JG, Schroeder CE, Javitt DC (2000) Activation timecourse of ventral visual stream object-recognition areas: high density electrical mapping of perceptual closure processes. J Cogn Neurosci 12:615–621PubMedCrossRefGoogle Scholar
  18. Doniger GM, Foxe JJ, Schroeder CE, Murray MM, Higgins BA, Javitt DC (2001) Visual perceptual learning in human object recognition areas: a repetition priming study using high-density electrical mapping. Neuroimage 13:305–313PubMedCrossRefGoogle Scholar
  19. Fort A, Delpuech C, Pernier J, Giard MH (2002a) Dynamics of cortico-subcortical cross-modal operations involved in audio–visual object detection in humans. Cereb Cortex 12:1031–9PubMedCrossRefGoogle Scholar
  20. Fort A, Delpuech C, Pernier J, Giard MH (2002b) Early auditory–visual interactions in human cortex during nonredundant target identification. Brain Res Cogn Brain Res 14:20–30PubMedCrossRefGoogle Scholar
  21. Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77–83PubMedCrossRefGoogle Scholar
  22. George N, Jemel B, Fiore N, Renault B (1997) Face and shape repetition effects in humans: a spatio-temporal ERP study. Neuroreport 8:1417–23PubMedCrossRefGoogle Scholar
  23. Giard MH, Peronnet, F (1999) Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–90PubMedCrossRefGoogle Scholar
  24. Gibson JR, Maunsell JHR (1997) Sensory modality specificity of neural activity related to memory in visual cortex. J Neurophysiol 78:1263–75PubMedGoogle Scholar
  25. Gottfried JA, Smith APR, Rugg MD, Dolan RJ (2004) Remembrance of odors past: human olfactory cortex in cross-modal recognition memory. Neuron 42:687–95PubMedCrossRefGoogle Scholar
  26. Goulet S, Murray EA (2001) Neural substrates of crossmodal association memory in monkeys: the amygdala versus the anterior rhinal cortex. Behav Neurosci 115:271–284PubMedCrossRefGoogle Scholar
  27. Grill-Spector K, Henson RN, Martin A (2006) Repetition and the brain, neural models of stimulus specific effects. Trends Cogn Sci 10:14–23PubMedCrossRefGoogle Scholar
  28. Guo J, Guo A (2005) Crossmodal interactions between olfactory and visual learning in Drosophila. Science 309:307–310PubMedCrossRefGoogle Scholar
  29. Haenny PE, Maunsell JHR, Schiler PH (1988) State dependent activity in monkey visual cortex: II. Retinal and extraretinal factors in V4. Exp Brain Res 69:245–259PubMedCrossRefGoogle Scholar
  30. Halgren E, Raij T, Marinkovic K, Jousmaki V, Hari R (2000) Cognitive response profile of the human fusiform face area as determined by MEG. Cereb Cortex 10:69–81PubMedCrossRefGoogle Scholar
  31. Hamilton W (1859) Lectures on Metaphysics and Logic. Gould & Lincoln, BostonCrossRefGoogle Scholar
  32. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–2430PubMedCrossRefGoogle Scholar
  33. Hein G, Doehrmann O, Müller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887PubMedCrossRefGoogle Scholar
  34. Henson RN, Rugg MD (2003) Neural response suppression, haemodynamic repetition effects, and behavioural priming. Neuropsychologia 41:263–270PubMedCrossRefGoogle Scholar
  35. Holmes NP (2009) The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topogr 21(3–4):168–176Google Scholar
  36. James TW, Humphrey GK, Gati JS, Servos P, Menon RS, Goodale MA (2002) Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia 40:1706–1714PubMedCrossRefGoogle Scholar
  37. Landis T, Lehmann D, Mita T, Skrandies W (1984) Evoked potential correlates of figure and ground Int J Psychophysiol 1:345–348CrossRefGoogle Scholar
  38. Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414PubMedCrossRefGoogle Scholar
  39. Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223PubMedCrossRefGoogle Scholar
  40. Lehmann S, Murray MM (2005) The role of multisensory memories in unisensory object discrimination. Brain Res Cogn Brain Res 24:326–334PubMedCrossRefGoogle Scholar
  41. MacSweeney M, Woll B, Campbell R, McGuire PK, David AS, Williams SC, Suckling J, Calvert GA, Brammer MJ (2002) Neural systems underlying British Sign Language and audio–visual English processing in native users. Brain 125:1583–93PubMedCrossRefGoogle Scholar
  42. Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP, Clarke S, Meuli RA (2007) Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb Cortex 17:1672–1779PubMedCrossRefGoogle Scholar
  43. Maunsell JHR, Sclar G, Nealey TA, Depriest DD (1991) Extraretinal representations in area V4 in the macaque monkey. Vis Neurosci 7:561–573PubMedCrossRefGoogle Scholar
  44. Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, Grave de Peralta R (2004) EEG source imaging. Clin Neurophysiol 115:2195–222PubMedCrossRefGoogle Scholar
  45. Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465PubMedCrossRefGoogle Scholar
  46. Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128PubMedCrossRefGoogle Scholar
  47. Mouchetant-Rostaing Y, Giard MH, Delpuech C, Echallier JF, Pernier J (2000a) Early signs of visual categorization for biological and non-biological stimuli in humans. Neuroreport 11:2521–2525PubMedCrossRefGoogle Scholar
  48. Mouchetant-Rostaing Y, Giard MH, Bentin S, Aguera PE, Pernier J (2000b) Neurophysiological correlates of face gender processing in humans. Eur J Neurosci 12:303–310PubMedCrossRefGoogle Scholar
  49. Murray MM, Camen C, Spierer L, Clarke S (2008a) Plasticity in representations of environmental sounds revealed by electrical neuroimaging. Neuroimage 39:847–856PubMedCrossRefGoogle Scholar
  50. Murray MM, Brunet D, Michel CM (2008b) Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr 20:249–264PubMedCrossRefGoogle Scholar
  51. Murray MM, Camen C, Gonzalez Andino SL, Bovet P, Clarke S (2006) Rapid brain discrimination of sounds of objects. J Neurosci 26:1293–1302PubMedCrossRefGoogle Scholar
  52. Murray MM, Foxe JJ, Wylie GR (2005a) The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage 27:473–478PubMedCrossRefGoogle Scholar
  53. Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Schroeder CE, Foxe JJ (2005b) Grabbing your ear: rapid auditory–somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15:963–974PubMedCrossRefGoogle Scholar
  54. Murray MM, Michel CM, Grave de Peralta R, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. Neuroimage 21:125–135PubMedCrossRefGoogle Scholar
  55. Naumer MJ, Doehrmann O, Müller NG, Muckli L, Kaiser J, Hein G (2009) Cortical plasticity of audio-visual object representations. Cereb Cortex. 19(7):1641–1653, doi: 10.1093/cercor/ bhn200Google Scholar
  56. Nyberg L, Habib R, McIntosh AR, Tulving E (2000) Reactivation of encoding-related brain activity during memory retrieval. Proc Natl Acad Sci USA 97:11120–11124PubMedCrossRefGoogle Scholar
  57. Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625PubMedCrossRefGoogle Scholar
  58. Ranganath C, Rainer G (2003) Neural mechanisms for detecting and remembering novel events. Nat Rev Neurosci 4:193–202PubMedCrossRefGoogle Scholar
  59. Ritter W, Simson R, Vaughan Jr HG, Macht M (1982) Manipulation of event-related potential manifestations of information processing stages. Science 218:909–911PubMedCrossRefGoogle Scholar
  60. Rolls ET, Treves A, Tovee MJ (1997) The representational capacity of the distributed encoding of information provided by populations of neurons in primate temporal visual cortex. Exp Brain Res 114:149–162PubMedCrossRefGoogle Scholar
  61. Romei V, Murray MM, Merabet LB, Thut G (2007) Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. J Neurosci 27:11465–11472PubMedCrossRefGoogle Scholar
  62. Rousselet GA, Fabre-Thorpe M, Thorpe SJ (2002) Parallel processing in high-level categorization of natural images. Nat Neurosci 5:629–630PubMedGoogle Scholar
  63. Rugg MD, Mark RE, Walla P, Schloerscheidt AM, Birch CS, Allan K (1998) Dissociation of the neural correlates of implicit and explicit memory. Nature 392:595–598PubMedCrossRefGoogle Scholar
  64. Schnider A, Valenza N, Morand S, Michel CM (2002) Early cortical distinction between memories that pertain to ongoing reality and memories that don’t. Cereb Cortex 12:54–61PubMedCrossRefGoogle Scholar
  65. Seeck M, Michel CM, Mainwaring N, Cosgrove R, Blume H, Ives J, Landis T, Schomer DL (1997) Evidence for rapid face recognition from human scalp and intracranial electrodes. Neuroreport 8:2749–2754PubMedCrossRefGoogle Scholar
  66. Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12:411–417PubMedCrossRefGoogle Scholar
  67. Spence C, Nicholls ME, Driver J (2001) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63:330–336PubMedCrossRefGoogle Scholar
  68. Sperdin HF, Cappe C, Foxe JJ, Murray MM (2009) Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed. Front Integr Neurosci 3:2. doi:10.3389/neuro.07.002.2009PubMedCrossRefGoogle Scholar
  69. Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MAGoogle Scholar
  70. Stoesz MR, Zhang M, Weisser VD, Prather SC, Mao H, Sathian K (2003) Neural networks active during tactile form perception: common and differential activity during macrospatial and microspatial tasks. Int J Psychophysiol 50:41–49PubMedCrossRefGoogle Scholar
  71. Thierry G, Martin CD, Downing P, Pegna AJ (2007) Controlling for interstimulus perceptual variance abolishes N170 face selectivity. Nat Neurosci 10:505–511PubMedCrossRefGoogle Scholar
  72. Tsivilis D, Otten LJ, Rugg MD (2001) Context effects on the neural correlates of recognition memory: an electrophysiological study. Neuron 31:497–505PubMedCrossRefGoogle Scholar
  73. Tulving E, Schacter DL (1990) Priming and human memory systems. Science 247:301–306PubMedCrossRefGoogle Scholar
  74. Vanni S, Revonsuo A, Saarinen J, Hari R (1996) Visual awareness of objects correlates with activity of right occipital cortex. Neuroreport 8:183–186PubMedCrossRefGoogle Scholar
  75. Von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biol 4:e326CrossRefGoogle Scholar
  76. Wang Y, Celebrini S, Trotter Y, Barone P (2008) Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence. BMC Neurosci 9:79PubMedCrossRefGoogle Scholar
  77. Wheeler ME, Petersen SE, Buckner RL (2000) Memory’s echo: vivid remembering reactivates sensory-specific cortex. Proc Natl Acad Sci USA 97:11125–11129PubMedCrossRefGoogle Scholar
  78. Wiggs CL, Martin A (1998) Properties and mechanisms of perceptual priming. Curr Opin Neurobiol 8:227–233PubMedCrossRefGoogle Scholar
  79. Zangaladze A, Epstein CM, Grafton ST, Sathian K (1999) Involvement of visual cortex in tactile discrimination of orientation. Nature 401:587–590PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media, LLC 2010

Authors and Affiliations

  1. 1.Centre for Biomedical Imaging, Department of Clinical Neurosciences, Department of RadiologyVaudois University Hospital Centre and University of LausanneLausanneSwitzerland
  2. 2.Department of Hearing and Speech SciencesVanderbilt University Medical CenterNashvilleUSA
  3. 3.Neuropsychology and Neurorehabilitation ServiceCentre Hospitalier Universitaire Vaudois and University of LausanneLausanneSwitzerland

Personalised recommendations