Skip to main content

Advertisement

Log in

Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simians

Experimental Brain Research Aims and scope Submit manuscript

Abstract

Stimulus-related oscillations are known to be closely linked to integrative processing in the brain. One research domain within which there has been tremendous interest in oscillatory mechanisms is in the integration of inputs across the widely separated sensory systems. Under the standard approach of assessing multisensory interactions in electrophysiological datasets, the event-related response to a multisensory stimulus is directly compared with the sum of the responses to its unisensory constituents when presented alone. When using methods like wavelet transformation or fast Fourier transformation to derive induced oscillatory signals, however, such linear operations are not appropriate. Here we introduce a simple bootstrapping procedure wherein the linear summation of single unisensory trials forms a distribution against which multisensory trials may be statistically compared, an approach that circumvents the issue of non-linearity when combining unisensory oscillatory responses. To test this approach we applied it to datasets from intracranial recordings in non-human primates and human scalp-recorded EEG, both derived from a simple audio-visual integration paradigm. Significant multisensory interactions were revealed in oscillatory activity centered at 15 and 20 Hz (the so-called beta band). Simulations of different levels of background noise further validated the results obtained by this method. By demonstrating super- and sub-additive effects, our analyses showed that this approach is a valuable metric for studying multisensory interactions reflected in induced oscillatory responses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. By convention, multisensory responses that are smaller than the sum of the unisensory responses are referred to as sub-additive while multisensory responses that are larger than the sum are referred to as super-additive (e.g., Calvert 2001a, b). However, it should be noted that a multisensory AV stimulus could evoke larger responses than either of the respective unisensory responses while still being classified as sub-additive relative to the summed unisensory responses.

  2. If a decrease in oscillatory responses relative to baseline is observed, then the dependent variable may be taken as the most negative value in the selected time interval.

References

  • Alegre M, Gurtubay IG, Labarga A, Iriarte J, Valencia M, Artieda J (2004) Frontal and central oscillatory changes related to different aspects of the motor process: a study in go/no-go paradigms. Exp Brain Res 159:14–22

    PubMed  CAS  Google Scholar 

  • Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192

    Article  PubMed  CAS  Google Scholar 

  • Bertrand O, Tallon-Baudry C (2000) Oscillatory gamma activity in humans: a possible role for object representation. Int J Psychophysiol 38:211–223

    Article  PubMed  CAS  Google Scholar 

  • Besle J, Fort A, Delpuech C, Giard MH (2004) Bimodal speech: early suppressive visual effects in human auditory cortex. Eur J Neurosci 20:2225–2234

    Article  PubMed  Google Scholar 

  • Bhattacharya J, Shams L, Shimojo S (2002) Sound-induced illusory flash perception: role of gamma band responses. Neuroreport 13: 1727–1730

    Article  PubMed  Google Scholar 

  • Brovelli A, Ding M, Ledberg A, Chen Y, Nakamura R, Bressler SL (2004) Beta oscillations in a large-scale sensorimotor cortical network: directional influences revealed by Granger causality. Proc Natl Acad Sci USA 101:9849–9854

    Article  PubMed  CAS  Google Scholar 

  • Busch NA, Herrmann CS (2003) Object-load and feature-load modulate EEG in a short-term memory task. Neuroreport 14: 1721–1724

    Article  PubMed  Google Scholar 

  • Bushara KO, Hanakawa T, Immisch I, Toma K, Kansaku K, Hallett M (2003) Neural correlates of cross-modal binding. Nat Neurosci 6:190–195

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA (2001a) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123

    Article  CAS  Google Scholar 

  • Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619–2623

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA (2001b) Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cereb Cortex 11: 1110–1123

    Article  CAS  Google Scholar 

  • Castelo-Branco M, Neuenschwander S, Singer W (1998) Synchronization of visual responses between the cortex, lateral geniculate nucleus, and retina in the anesthetized cat. J Neurosci 18:6395–6410

    PubMed  CAS  Google Scholar 

  • Classen J, Gerloff C, Honda M, Hallett M (1998) Integrative visuomotor behavior is associated with interregionally coherent oscillations in the human brain. J Neurophysiol 79:1567–1573

    PubMed  CAS  Google Scholar 

  • Comerchero MD, Polich J (1999) P3a and P3b from typical auditory and visual stimuli. Clin Neurophysiol 110:24–30

    Article  PubMed  CAS  Google Scholar 

  • Engel AK, Konig P, Singer W (1991) Direct physiological evidence for scene segmentation by temporal coding. Proc Natl Acad Sci USA 88:9136–9140

    Article  PubMed  CAS  Google Scholar 

  • Fort A, Delpuech C, Pernier J, Giard MH (2002a) Early auditory–visual interactions in human cortex during nonredundant target identification. Brain Res Cogn Brain Res. 14:20–30

    Article  Google Scholar 

  • Fort A, Depuech C, Pernier J, Giard MH (2002b). Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. Cereb Cortex 12: 1031–1039

    Article  Google Scholar 

  • Foxe JJ, Schroeder CE (2005) The case for feedforward multisensory convergence during early cortical processing. Neuroreport 16:419–423

    Article  PubMed  Google Scholar 

  • Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol 88:540–543

    PubMed  Google Scholar 

  • Foxe JJ, Morocz IA, Higgins BA, Murray MM, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing. Brain Res Cogn Brain Res. 10:77–83

    Article  PubMed  CAS  Google Scholar 

  • Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    Article  PubMed  CAS  Google Scholar 

  • Gonzalez Andino SL, Murray MM, Foxe JJ, Grave de Peralta Menendez R (2005) How single-trial electrical neuroimaging contributes to multisensory research? Exp Brain Res 166:298–304

    Article  PubMed  Google Scholar 

  • Gray CM (1999) The temporal correlation hypothesis of visual feature integration: still alive and well. Neuron 24:31–25

    Article  PubMed  CAS  Google Scholar 

  • Gray CM, Konig P, Engel AK, Singer W (1989) Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338:334–337

    Article  PubMed  CAS  Google Scholar 

  • Gruber T, Muller MM (2002) Effects of picture repetition on induced gamma band responses, evoked potentials, and phase synchrony in the human EEG. Brain Res Cogn Brain Res. 13:377–392

    Article  PubMed  Google Scholar 

  • Gruber T, Muller MM, Keil A, Elbert T (1999) Selective visual-spatial attention alters induced gamma band responses in the human EEG. Clin Neurophysiol 110:2074–2085

    Article  PubMed  CAS  Google Scholar 

  • Herrmann CS, Mecklinger A (2001) Gamma activity in human EEG is related to high-speed memory comparisons during object selective attention. Vis Cogn 8:593–608

    Article  Google Scholar 

  • Herrmann CS, Mecklinger A, Pfeifer E (1999) Gamma responses and ERPs in a visual classification task. Clin Neurophysiol 110:636–642

    Article  PubMed  CAS  Google Scholar 

  • Herrmann CS, Senkowski D, Rottger S (2004) Phase-locking and amplitude modulations of EEG alpha: two measures reflect different cognitive processes in a working memory task. Exp Psychol 51:311–318

    PubMed  Google Scholar 

  • Kaiser J, Buhler M, Lutzenberger W (2004) Magnetoencephalographic gamma-band responses to illusory triangles in humans. Neuroimage 23:551–560

    Article  PubMed  Google Scholar 

  • Kaiser J, Hertrich I, Ackermann H, Mathiak K, Lutzenberger W (2005) Hearing Lips: Gamma-band Activity During Audiovisual Speech Perception. Cereb Cortex 15:646–653

    Article  PubMed  Google Scholar 

  • Kaiser J, Lutzenberger W (2003) Induced gamma-band activity and human brain function. Neuroscientist 9:475–484

    Article  PubMed  Google Scholar 

  • Kaiser J, Lutzenberger W, Ackermann H, Birbaumer N (2002) Dynamics of gamma-band activity induced by auditory pattern changes in humans. Cereb Cortex 12:212–221

    Article  PubMed  Google Scholar 

  • Klimesch W (1999) EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res Brain Res Rev 29:169–195

    Article  PubMed  CAS  Google Scholar 

  • Konig P, Engel AK, Singer W (1995) Relation between oscillatory activity and long-range synchronization in cat visual cortex. Proc Natl Acad Sci USA 92:290–294

    Article  PubMed  CAS  Google Scholar 

  • Lakatos P, Chen CM, Rajkai C, Karmos G, Schroeder CE (2005a) The role of oscillations in multisensory enhancement of auditory stimulus processing. Abstract at the Society for Neuroscience

  • Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE (2005b) An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. J Neurophysiol 94:1904–1911

    Article  Google Scholar 

  • Liang H, Bressler SL, Ding M, Truccolo WA, Nakamura R (2002) Synchronized activity in prefrontal cortex during anticipation of visuomotor processing. Neuroreport 13:2011–2015

    Article  PubMed  Google Scholar 

  • Logothetis NK (2002) The neural basis of the blood-oxygen-level-dependent functional magnetic resonance imaging signal. Philos Trans R Soc Lond B Biol Sci 357:1003–1037

    Article  PubMed  Google Scholar 

  • Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual–auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465

    Article  PubMed  Google Scholar 

  • Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128

    Article  PubMed  Google Scholar 

  • Muller MM, Bosch J, Elbert T, Kreiter A, Sosa MV, Sosa PV, Rockstroh B (1996) Visually induced gamma-band responses in human electroencephalographic activity—a link to animal studies. Exp Brain Res 112:96–102

    Article  PubMed  CAS  Google Scholar 

  • Murray MM, Molholm S, Michel CM, Ritter W, Heslenfeld DJ, Schroeder CE, Javitt DC, Foxe JJ (2005a) Grabbing your ear: Rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15: 963–974

    Article  Google Scholar 

  • Murray MM, Foxe JJ, Wylie GR (2005b) The brain uses single-trial multisensory memories to discriminate without awareness. Neuroimage 27:473–478

    Article  Google Scholar 

  • Niessing J, Ebisch B, Schmidt KE, Niessing M, Singer W, Galuske RA (2005) Hemodynamic signals correlate tightly with synchronized gamma oscillations. Science 309:948–951

    Article  PubMed  CAS  Google Scholar 

  • Pfurtscheller G (1992) Event-related synchronization (ERS): an electrophysiological correlate of cortical areas at rest. Electroencephalogr Clin Neurophysiol 83:62–69

    Article  PubMed  CAS  Google Scholar 

  • Polich J, Alexander JE, Bauer LO, Kuperman S, Morzorati S, O’Connor SJ, Porjesz B, Rohrbaugh J, Begleiter H (1997) P300 topography of amplitude/latency correlations. Brain Topogr. 9:275–282

    Article  PubMed  CAS  Google Scholar 

  • Sakowitz O, Quian Quiroga R., Schurmann M, Basar E (2005) Spatio-temporal frequency characteristics of intersensory components in audiovisual evoked potentials. Brain Res Cogn Brain Res 23:316–326

    Article  PubMed  Google Scholar 

  • Sakowitz OW, Quian Quiroga R, Schurmann M, Basar E (2001) Bisensory stimulation increases gamma-responses over multiple cortical regions. Brain Res Cogn Brain Res 11:267–279

    Article  PubMed  CAS  Google Scholar 

  • Sauseng P, Klimesch W, Doppelmayr M, Hanslmayr S, Schabus M, Gruber WR (2004) Theta coupling in the human electroencephalogram during a working memory task. Neurosci Lett 354:123–126

    Article  PubMed  CAS  Google Scholar 

  • Schroeder CE, Foxe JJ (2002) The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res 14: 187–198

    Article  PubMed  Google Scholar 

  • Schroeder CE, Foxe JJ (2004) Multisensory convergence in early cortical processing. In: Calvert GA, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge, pp 295–309

    Google Scholar 

  • Schroeder CE, Molholm S, Lakatos P, Ritter W, Foxe JJ (2004) Human–Simian correspondence in the early cortical processing of multisensory cues. Cogn Process 5:140–151

    Article  Google Scholar 

  • Schroeder CE, Foxe JJ (2005) Multisensory contributions to low-level, ‘unisensory’ processing. Curr Opin Neurobiol 15:454–458

    Article  PubMed  CAS  Google Scholar 

  • Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85:1322–1327

    PubMed  CAS  Google Scholar 

  • Schroeder CE, Mehta AD, Givre SJ (1998) A spatiotemporal profile of visual system activation revealed by current source density analysis in the awake macaque. Cereb Cortex 8:575–592

    Article  PubMed  CAS  Google Scholar 

  • Senkowski D, Herrmann CS (2002) Effects of task difficulty on evoked gamma activity and ERPs in a visual discrimination task. Clin Neurophysiol 113:1742–1753

    Article  PubMed  Google Scholar 

  • Senkowski D, Talsma D, Herrmann CS, Woldorff MG (2005) Multisensory processing and oscillatory gamma responses: Effects of spatial selective attention. Exp Brain Res 166:411–426

    Article  PubMed  Google Scholar 

  • Senkowski D, Molholm, S, Gomez-Ramirez M, Foxe JJ (2006a) Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: a high-density electrical mapping study. Cereb Cortex 16:1556–1565

    Article  Google Scholar 

  • Senkowski D, Talsma D, Grigutsch M, Herrmann CS, and Woldorff MG (2006b) Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia (in press)

  • Singer W, Gray CM (1995) Visual feature integration and the temporal correlation hypothesis. Annu Rev Neurosci 18:555–586

    Article  PubMed  CAS  Google Scholar 

  • Tallon-Baudry C, Bertrand O (1999) Oscillatory gamma activity in humans and its role in object representation. Trends Cogn Sci 3:151–162

    Article  PubMed  Google Scholar 

  • Tallon-Baudry C, Bertrand O, Delpuech C, Permier J (1997) Oscillatory gamma-band (30–70 Hz) activity induced by a visual search task in humans. J Neurosci 17:722–734

    PubMed  CAS  Google Scholar 

  • Tallon-Baudry C, Bertrand O, Delpuech C, Pernier J (1996) Stimulus specificity of phase-locked and non-phase-locked 40 Hz visual responses in human. J Neurosci 16:4240–4249

    PubMed  CAS  Google Scholar 

  • Tallon-Baudry C, Bertrand O, Peronnet F, Pernier J (1998) Induced gamma-band activity during the delay of a visual short-term memory task in humans. J Neurosci 18:4244–4254

    PubMed  CAS  Google Scholar 

  • Talsma D, Woldorff MG (2005) Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. J.Cogn Neurosci 17:1098–1114

    Article  PubMed  Google Scholar 

  • von Stein A, Chiang C, Konig P (2000) Top-down processing mediated by interareal synchronization. Proc Natl Acad Sci USA 97:14748–14753

    Article  Google Scholar 

Download references

Acknowledgements

This work supported by grants from the National Institute of Mental Health (MH65350) and the National Institute on Aging (AG22696) to Dr. J. J. Foxe. We would also like to express our sincere thanks to Dr. Simon Kelly for his insightful comments on earlier drafts and to Dr. Scott Makeig who provided advice regarding the bootstrap technique used here.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to John J. Foxe.

Appendix: Materials and paradigm

Appendix: Materials and paradigm

Intracranial field potentials in macaques

To evaluate the application of this technique to intracranial field potential data, we used data recorded during the course of a multisensory investigation in auditory cortex of an awake behaving macaque monkey (see Lakatos et al. 2005a). All procedures were approved in advance by the Animal Care and Use Committee of the Nathan Kline Institute. The subject, a Rhesus monkey (Maccaca mulatta) was adapted to a custom fitted primate chair and to the recording chamber and then surgically prepared for chronic awake electrophysiological recording. Surgery was performed under anesthesia (1–2% isoflourane), using standard aseptic surgical methods (see Schroeder et al. 2001). To allow electrode access to the brain, and to promote an orderly pattern of sampling across the surface of the auditory cortices, matrices of 18 gauge stainless steel guide tubes were placed over auditory cortex. These matrices were angled so that the electrode track would be perpendicular to the plane of auditory cortex, as determined by pre-implant MRI (Schroeder et al. 1998). They were placed within small, appropriately shaped craniotomies, to rest against the intact dura. The matrices, along with a titanium head post (permitting painless head restraint), were embedded in dental acrylic and secured to the skull with titanium orthopaedic screws.

Recordings were made in an electrically shielded, sound-attenuated chamber with SONEX ProSPEC Composite™ sound absorbing foam. Laminar profiles of field potentials (EEG) were recorded using a linear array multi-contact electrode (24 contacts, 100 μm intercontact spacing) positioned to sample from all the layers simultaneously (Lakatos et al. 2005b). Signals were impedance matched with a pre-amplifier (10× gain, bandpass dc-10 kHz) situated on the electrode, and after further amplification (500×) the signal was band pass filtered by analogue filtering (0.1–500 Hz) to extract the field potential sampled at 2 kHz/16 bit precision.

Eye position was monitored using an ISCAN ETL-200 eye tracking system, and stimuli were presented only when the monkey’s gaze was held within a 10° degree window surrounding the fixation point in the middle of the monitor. Auditory stimuli consisted of broadband Gaussian noise bursts (16 ms duration; 70 dB SPL, 1 ms rise/fall times). The visual stimuli consisted of a bright white monitor flash (16 ms duration). Unisensory-auditory, unisensory-visual, and multisensory audiovisual stimuli were presented in separate blocks (SOA = 767 ms), each consisting of 100 stimuli. Trial blocks were separated by brief breaks in which the monkey was checked and fed dried fruits and other preferred treats.

In accordance with the time–frequency representations (Fig. 2), for the analysis of multisensory interactions a 15 Hz WT was performed. The wavelet had a duration (2σ t ) of 67 ms and a spectral bandwidth (2σ f ) of 4.8 Hz. As a measure of 15 Hz activity, the maximum value of the modulus of the complex transform coefficient was computed in a time interval between 10 and 110 ms for each epoch.

Human EEG recordings

Induced oscillatory beta responses for one subject (female, 27 years, right handed) from one right-frontal channel were analyzed (Fig. 5). This channel was selected because a right-frontal maximum was found at this electrode site. The nearest neighboring site within the 10–20 system of this electrode is the F4 electrode. The EEG was recorded in an electrically shielded chamber from 2 EOG and 128 scalp electrodes (impedances < 5 kΩ), referenced to the nose at a sample rate of 500 Hz. Epochs for EEG beta activity lasted from 500 ms before to 1000 ms after stimulus onset. Baselines were computed from −300 to −100 ms pre-stimulus. For artifact suppression, trials were automatically excluded from averaging if the standard deviation within a moving 200 ms time interval exceeded 30 μV in any one of the EEG channels and 40 μV at the EOG channels in a time interval between −300 and 500 ms.

Auditory stimuli in the experiment consisted of a 1,000 Hz tone (60 ms duration; 75 dB SPL) presented from a single speaker located atop the monitor on which the visual stimuli were presented. The visual stimulus consisted of a red circular disk subtending 1.2° in diameter presented on a black background. Visual stimuli were presented 1.56° lateral left above a central fixation cross for 60 ms. Unisensory-auditory (A), unisensory-visual (V), and multisensory audiovisual (AV) were presented at inter-stimulus-intervals ranging between 750 and 3,000 ms. The subject was instructed to maintain central fixation at all times and to make a speeded button response with their right index finger when a stimulus in either modality was detected. A total number of 533 auditory, 547 auditory and 530 audiovisual trials were submitted to the analysis.

For the analysis of beta responses, the maximum value of the modulus of the complex 20 Hz transformed coefficient was computed in a time interval between 80 and 170 ms for each epoch. The duration (2σ t ) of the wavelet was 100 ms with a spectral bandwidth (2σ f ) of 6.4 Hz. The subject gave written informed consent to participate in the study.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Senkowski, D., Gomez-Ramirez, M., Lakatos, P. et al. Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simians. Exp Brain Res 177, 184–195 (2007). https://doi.org/10.1007/s00221-006-0664-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-006-0664-7

Keywords

Navigation