Information-Theoretic Characterization of the Neural Mechanisms of Active Multisensory Decision Making

  • Ioannis DelisEmail author
  • Robin A. A. Ince
  • Paul Sajda
  • Qi Wang
Conference paper
Part of the Biosystems & Biorobotics book series (BIOSYSROB, volume 21)


The signals delivered by different sensory modalities provide us with complementary information about the environment. A key component of interacting with the world is how to direct ones’ sensors so as to extract task-relevant information in order to optimize subsequent perceptual decisions. This process is often referred to as active sensing. Importantly, the processing of multisensory information acquired actively from multiple sensory modalities requires the interaction of multiple brain areas over time. Here we investigated the neural underpinnings of active visual-haptic integration during performance of a two-alternative forced choice (2AFC) reaction time (RT) task. We asked human subjects to discriminate the amplitude of two texture stimuli (a) using only visual (V) information, (b) using only haptic (H) information and (c) combining the two sensory cues (VH), while electroencephalograms (EEG) were recorded. To quantify multivariate interactions between EEG signals and active sensory experience in the three sensory conditions, we employed a novel information-theoretic methodology. This approach provides a principled way to quantify the contribution of each one of the sensory modalities to the perception of the stimulus and assess whether the respective neural representations may interact to form a percept of the stimulus and ultimately drive perceptual decisions. Application of this method to our data identified (a) an EEG component (comprising frontal and occipital electrodes) carrying behavioral information that is common to the two sensory inputs and (b) another EEG component (mainly motor) reflecting a synergistic representational interaction between the two sensory inputs. We suggest that the proposed approach can be used to elucidate the neural mechanisms underlying cross-modal interactions in active multisensory processing and decision-making.


  1. 1.
    Campion, G., Wang, Q., Hayward, V.: The pantograph Mk-II: a haptic instrument. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 1–4, pp. 723–728 (2005)Google Scholar
  2. 2.
    Delis, I., Dmochowski, J.P., Sajda, P., Wang, Q.: Correlation of neural activity with behavioral kinematics reveals distinct sensory encoding and evidence accumulation processes during active tactile sensing. NeuroImage 175, 12–21 (2018)CrossRefGoogle Scholar
  3. 3.
    Ince, R.A.A., Giordano, B.L., Kayser, C., Rousselet, G.A., Gross, J., Schyns, P.G.: A statistical framework for neuroimaging data analysis based on mutual information estimated via a Gaussian copula. Hum. Brain Mapp. 38(3), 1541–1573 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ioannis Delis
    • 1
    Email author
  • Robin A. A. Ince
    • 2
  • Paul Sajda
    • 3
  • Qi Wang
    • 4
  1. 1.School of Biomedical SciencesUniversity of LeedsLeedsUK
  2. 2.Institute of Neuroscience and PsychologyUniversity of GlasgowGlasgowUK
  3. 3.Biomedical Engineering Department, Data Science InstituteColumbia UniversityNew YorkUSA
  4. 4.Biomedical Engineering DepartmentColumbia UniversityNew YorkUSA

Personalised recommendations