Brain Structure and Function

, Volume 223, Issue 9, pp 4087–4098 | Cite as

Cross-decoding supramodal information in the human brain

  • Seth M. LevineEmail author
  • Jens V. SchwarzbachEmail author
Original Article


Perceptual decision making is the cognitive process wherein the brain classifies stimuli into abstract categories for more efficient downstream processing. A system that, during categorization, can process information regardless of the information’s original sensory modality (i.e., a supramodal system) would have a substantial advantage over a system with dedicated processes for specific sensory modalities. While many studies have probed decision processes through the lens of one sensory modality, it remains unclear whether there are such supramodal brain areas that can flexibly process task-relevant information regardless of the original “format” of the information. To investigate supramodality, one must ensure that supramodal information exists somewhere within the functional architecture by rendering information from multiple sensory systems necessary but insufficient for categorization. To this aim, we tasked participants with categorizing auditory and tactile frequency-modulated sweeps according to learned, supramodal categories in a delayed match-to-category paradigm while we measured their blood-oxygen-level dependent signal with functional MRI. To detect supramodal information, we implemented a set of cross-modality pattern classification analyses, which demonstrated that the left caudate nucleus encodes category-level information but not stimulus-specific information (such as spatial directions and stimulus modalities), while the right inferior frontal gyrus, showing the opposite pattern, encodes stimulus-specific information but not category-level information. Given our paradigm, these results reveal abstract representations in the brain that are independent of motor, semantic, and sensory-specific processing, instead reflecting supramodal, categorical information, which points to the caudate nucleus as a locus of cognitive processes involved in complex behavior.


Categorization Cross-decoding fMRI Perceptual decision making Supramodal 


Author contributions

SML and JVS designed the experiment and acquired data. SML analyzed the data. SML and JVS wrote the manuscript.

Compliance with ethical standards

Conflict of interest

The authors declare no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Supplementary material

429_2018_1740_MOESM1_ESM.pdf (1.3 mb)
Supplementary material 1 (PDF 1321 KB)
429_2018_1740_MOESM2_ESM.pdf (737 kb)
Supplementary material 2 (PDF 736 KB)
429_2018_1740_MOESM3_ESM.pdf (1.1 mb)
Supplementary material 3 (PDF 1123 KB)
429_2018_1740_MOESM4_ESM.pdf (2 mb)
Supplementary material 4 (PDF 2029 KB)


  1. Aron AR, Fletcher PC, Bullmore ET et al (2003) Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans. Nat Neurosci 6:115–116. CrossRefPubMedGoogle Scholar
  2. Ashby FG, Alfonso-Reese LA, Turken AU, Waldron EM (1998) A neuropsychological theory of multiple systems in category learning. Psychol Rev 105:442–481. CrossRefPubMedGoogle Scholar
  3. Ashby FG, Noble S, Filoteo JV et al (2003) Category learning deficits in Parkinson’s disease. Neuropsychology 17:115–124. CrossRefPubMedGoogle Scholar
  4. Ballard I, Miller EM, Piantadosi ST et al (2017) Beyond reward prediction errors: human striatum updates rule values during learning. Cereb Cortex. CrossRefPubMedGoogle Scholar
  5. Binder JR (2016) In defense of abstract conceptual representations. Psychon Bull Rev 23:1096–1108. CrossRefPubMedGoogle Scholar
  6. Binder JR, Liebenthal E, Possing ET et al (2004) Neural correlates of sensory and decision processes in auditory object identification. Nat Neurosci 7:295–301. CrossRefPubMedGoogle Scholar
  7. Bishop CW, Miller LM (2009) A multisensory cortical network for understanding speech in noise. J Cogn Neurosci 21:1790–1804. CrossRefPubMedPubMedCentralGoogle Scholar
  8. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436. CrossRefPubMedGoogle Scholar
  9. Crone EA, Wendelken C, Donohue SE, Bunge SA (2006) Neural evidence for dissociable components of task-switching. Cereb Cortex 16:475–486. CrossRefPubMedGoogle Scholar
  10. Dippel G, Beste C (2015) A causal role of the right inferior frontal cortex in implementing strategies for multi-component behaviour. Nat Commun 6:6587. CrossRefPubMedGoogle Scholar
  11. Filimon F, Philiastides MG, Nelson JD et al (2013) How embodied is perceptual decision making? Evidence for separate processing of perceptual and motor decisions. J Neurosci 33:2121–2136. CrossRefPubMedGoogle Scholar
  12. Freedman DJ, Assad JA (2006) Experience-dependent representation of visual categories in parietal cortex. Nature 443:85–88. CrossRefPubMedGoogle Scholar
  13. Freedman DJ, Assad JA (2011) A proposed common neural mechanism for categorization and perceptual decisions. Nat Neurosci 14:143–146. CrossRefPubMedGoogle Scholar
  14. Goebel R, Van Atteveldt N (2009) Multisensory functional magnetic resonance imaging: a future perspective. Exp Brain Res 198:153–164CrossRefGoogle Scholar
  15. Gold JI, Shadlen MN (2007) The neural basis of decision making. Annu Rev Neurosci 30:535–574. CrossRefPubMedGoogle Scholar
  16. Grinband J, Hirsch J, Ferrera VP (2006) A neural representation of categorization uncertainty in the human brain. Neuron 49:757–763. CrossRefPubMedGoogle Scholar
  17. Hampshire A, Thompson R, Duncan J, Owen AM (2009) Selective tuning of the right inferior frontal gyrus during target detection. Cogn Affect Behav Neurosci 9:103–112. CrossRefPubMedPubMedCentralGoogle Scholar
  18. Handjaras G, Leo A, Cecchetti L et al (2017) Modality-independent encoding of individual concepts in the left parietal cortex. Neuropsychologia 105:39–49. CrossRefPubMedGoogle Scholar
  19. Harnad S (1990) The symbol grounding problem. Phys D 42:335–346. CrossRefGoogle Scholar
  20. Haxby JV, Gobbini MI, Furey ML et al (2001) Distrubuted and overlapping representations of face and objects in ventral temporal cortex. Science 293:2425–2430. CrossRefGoogle Scholar
  21. Heekeren HR, Marrett S, Bandettini PA, Ungerleider LG (2004) A general mechanism for perceptual decision-making in the human brain. Nature 431:859–862. CrossRefPubMedGoogle Scholar
  22. Heekeren HR, Marrett S, Ungerleider LG (2008) The neural systems that mediate human perceptual decision making. Nat Rev Neurosci 9:467–479. CrossRefPubMedGoogle Scholar
  23. Helie S, Roeder JL, Ashby FG (2010) Evidence for cortical automaticity in rule-based categorization. J Neurosci 30:14225–14234. CrossRefPubMedGoogle Scholar
  24. Hikosaka O, Sakamoto M, Usui S (1989) Functional properties of monkey caudate neurons. II. Visual and auditory responses. J Neurophysiol 61:799–813CrossRefGoogle Scholar
  25. Ho TC, Brown S, Serences JT (2009) Domain general mechanisms of perceptual decision making in human cortex. J Neurosci 29:8675–8687. CrossRefPubMedPubMedCentralGoogle Scholar
  26. Ivanoff J, Branning P, Marois R (2009) Mapping the pathways of information processing from sensation to action in four distinct sensorimotor tasks. Hum Brain Mapp 30:4167–4186. CrossRefPubMedPubMedCentralGoogle Scholar
  27. Jenkinson M, Bannister P, Brady M, Smith S (2002) Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17:825–841. CrossRefPubMedGoogle Scholar
  28. Kaiser J, Lennert T, Lutzenberger W (2007) Dynamics of oscillatory activity during auditory decision making. Cereb Cortex 17:2258–2267. CrossRefPubMedGoogle Scholar
  29. Kaplan JT, Man K, Greening SG (2015) Multivariate cross-classification: applying machine learning techniques to characterize abstraction in neural representations. Front Hum Neurosci 9:151. CrossRefPubMedPubMedCentralGoogle Scholar
  30. Klemen J, Chambers CD (2012) Current perspectives and methods in studying neural mechanisms of multisensory interactions. Neurosci Biobehav Rev 36:111–133CrossRefGoogle Scholar
  31. Kriegeskorte N, Goebel R, Bandettini P (2006) Information-based functional brain mapping. Proc Natl Acad Sci USA 103:3863–3868. CrossRefPubMedGoogle Scholar
  32. Lee Y-S, Turkeltaub P, Granger R, Raizada RDS (2012) Categorical speech processing in Broca’s area: an fMRI study using multivariate pattern-based analysis. J Neurosci 32:3942–3948. CrossRefPubMedGoogle Scholar
  33. Levine SM, Schwarzbach J (2017) Decoding of auditory and tactile perceptual decisions in parietal cortex. Neuroimage 162:297–305. CrossRefPubMedGoogle Scholar
  34. Macmillan NA, Creelman CD (2005) Detection theory: a user’s guide, 2nd edn. Psychology, ErlbaumGoogle Scholar
  35. Menon V, Adleman NE, White CD et al (2001) Error-related brain activation during a Go/NoGo response inhibition task. Hum Brain Mapp 12:131–143.;2-C CrossRefPubMedGoogle Scholar
  36. Misaki M, Kim Y, Bandettini PA, Kriegeskorte N (2010) Comparison of multivariate classifiers and response normalizations for MVPA. Neuroimage 53:103–118. CrossRefPubMedPubMedCentralGoogle Scholar
  37. Muhammad R, Wallis JD, Miller EK (2006) A comparison of abstract rules in the prefrontal cortex, premotor cortex, inferior temporal cortex, and striatum. J Cogn Neurosci 18:974–989. CrossRefPubMedGoogle Scholar
  38. Myers EB, Blumstein SE, Walsh E, Eliassen J (2009) Inferior frontal regions underlie the perception of phonetic category invariance. Psychol Sci 20:895–903. CrossRefPubMedPubMedCentralGoogle Scholar
  39. Nagy A, Paróczy Z, Norita M, Benedek G (2005) Multisensory responses and receptive field properties of neurons in the substantia nigra and in the caudate nucleus. Eur J Neurosci 22:419–424. CrossRefPubMedGoogle Scholar
  40. Nagy A, Eördegh G, Paróczy Z et al (2006) Multisensory integration in the basal ganglia. Eur J Neurosci 24:917–924. CrossRefPubMedGoogle Scholar
  41. Nichols TE, Holmes AP (2001) Nonparametric permutation tests for functional neuroimaging: a primer with examples. Hum Brain Mapp 15:1–25. CrossRefGoogle Scholar
  42. Nichols T, Brett M, Andersson J et al (2005) Valid conjunction inference with the minimum statistic. Neuroimage 25:653–660. CrossRefPubMedGoogle Scholar
  43. Noppeney U, Ostwald D, Werner S (2010) Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci 30:7434–7446. CrossRefPubMedGoogle Scholar
  44. O’Connell RG, Dockree PM, Kelly SP (2012) A supramodal accumulation-to-bound signal that determines perceptual decisions in humans. Nat Neurosci 15:1729–1735. CrossRefPubMedGoogle Scholar
  45. Oosterhof NN, Connolly AC, Haxby JV (2016) CoSMoMVPA: multi-modal multivariate pattern analysis of neuroimaging data in Matlab/GNU Octave. Front Neuroinform 10:047118. CrossRefGoogle Scholar
  46. Pleger B, Ruff CC, Blankenburg F et al (2006) Neural coding of tactile decisions in the human prefrontal cortex. J Neurosci 26:12596–12601. CrossRefPubMedPubMedCentralGoogle Scholar
  47. Ploran EJ, Tremel JJ, Nelson SM, Wheeler ME (2011) High quality but limited quantity perceptual evidence produces neural accumulation in frontal and parietal cortex. Cereb Cortex 21:2650–2662. CrossRefPubMedPubMedCentralGoogle Scholar
  48. Preuschhof C, Heekeren HR, Taskin B et al (2006) Neural correlates of vibrotactile working memory in the human brain. J Neurosci 26:13231–13239. CrossRefPubMedGoogle Scholar
  49. Reig R, Silberberg G (2014) Multisensory integration in the mouse striatum. Neuron 83:1200–1212. CrossRefPubMedPubMedCentralGoogle Scholar
  50. Robinson S, Jovicich J (2011) B0 mapping with multi-channel RF coils at high field. Magn Reson Med 66:976–988. CrossRefPubMedGoogle Scholar
  51. Schwarzbach J (2011) A simple framework (ASF) for behavioral and neuroimaging experiments based on the psychophysics toolbox for MATLAB. Behav Res Methods 43:1194–1201. CrossRefPubMedGoogle Scholar
  52. Simanova I, Hagoort P, Oostenveld R, Van Gerven MAJ (2014) Modality-independent decoding of semantic information from the human brain. Cereb Cortex 24:426–434. CrossRefPubMedGoogle Scholar
  53. Smith SM, Nichols TE (2009) Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference. Neuroimage 44:83–98. CrossRefPubMedGoogle Scholar
  54. Smith SM, Jenkinson M, Woolrich MW et al (2004) Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23:S208–S219. CrossRefPubMedGoogle Scholar
  55. Stocco A, Anderson JR (2008) Endogenous control and task representation: an fMRI study in algebraic problem-solving. J Cogn Neurosci 20:1300–1314. CrossRefPubMedGoogle Scholar
  56. Sugihara T, Diltz MD, Averbeck BB, Romanski LM (2006) Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J Neurosci 26:11138–11147. CrossRefPubMedPubMedCentralGoogle Scholar
  57. Tamber-Rosenau BJ, Dux PE, Tombu MN et al (2013) Amodal processing in human prefrontal cortex. J Neurosci 33:11573–11587. CrossRefPubMedPubMedCentralGoogle Scholar
  58. Van Kemenade BM, Seymour K, Wacker E et al (2014) Tactile and visual motion direction processing in hMT+/V5. Neuroimage 84:420–427. CrossRefPubMedGoogle Scholar
  59. Vettel JM, Green JR, Heller L, Tarr MJ (2016) Temporal and semantic effects on multisensory integration. arXiv:1606.05004
  60. Watanabe K (2005) Immediate changes in anticipatory activity of caudate neurons associated with reversal of position-reward contingency. J Neurophysiol 94:1879–1887. CrossRefPubMedGoogle Scholar
  61. Wilson JS, Hull CD, Buchwald NA (1983) Intracellular studies of the convergence of sensory input on caudate neurons of cat. Brain Res 270:197–208. CrossRefPubMedGoogle Scholar
  62. Zaitsev M, Hennig J, Speck O (2004) Point spread function mapping with parallel imaging techniques and high acceleration factors: fast, robust, and flexible method for echo-planar imaging distortion correction. Magn Reson Med 52:1156–1166. CrossRefPubMedGoogle Scholar
  63. Zou L, Ding G, Abutalebi J et al (2012) Structural plasticity of the left caudate in bimodal bilinguals. Cortex 48:1197–1206. CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Psychiatry and PsychotherapyUniversity of RegensburgRegensburgGermany

Personalised recommendations