Cognitive Computation

, Volume 10, Issue 2, pp 284–295 | Cite as

Anatomical Pattern Analysis for Decoding Visual Stimuli in Human Brains

  • Muhammad YousefnezhadEmail author
  • Daoqiang Zhang


A universal unanswered question in neuroscience and machine learning is whether computers can decode the patterns of the human brain. Multi-Voxel Pattern Analysis (MVPA) is a critical tool for addressing this question. However, there are two challenges in the previous MVPA methods, which include decreasing sparsity and noise in the extracted features and increasing the performance of prediction. In overcoming mentioned challenges, this paper proposes Anatomical Pattern Analysis (APA) for decoding visual stimuli in the human brain. This framework develops a novel anatomical feature extraction method and a new imbalance AdaBoost algorithm for binary classification. Further, it utilizes an Error-Correcting Output Codes (ECOC) method for multiclass prediction. APA can automatically detect active regions for each category of the visual stimuli. Moreover, it enables us to combine homogeneous datasets for applying advanced classification. Experimental studies on four visual categories (words, consonants, objects, and scrambled photos) demonstrate that the proposed approach achieves superior performance to state-of-the-art methods.


Brain decoding Multi-voxel pattern analysis Anatomical feature extraction Visual object recognition 



This work was supported in part by the National Natural Science Foundation of China (61422204 and 61473149) and NUAA Fundamental Research Funds (NE2013105).

Compliance with Ethical Standards

Conflict of Interests

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. 1.
    Anderson M, Oates T. A critique of multi-voxel pattern analysis. In: Proceedings of the cognitive science society; 2010.Google Scholar
  2. 2.
    Carlson TA, Schrater P, He S. Patterns of activity in the categorical representations of objects. J Cogn Neurosci 2003;15(5):704–717.CrossRefPubMedGoogle Scholar
  3. 3.
    Carroll MK, Cecchi GA, Rish I, Garg R, Rao AR. Prediction and interpretation of distributed neural activity with sparse models. NeuroImage 2009;44(1):112–122.CrossRefPubMedGoogle Scholar
  4. 4.
    Chen PH, Chen J, Yeshurun Y, Hasson U, Haxby J, Ramadge PJ. A reduced-dimension fMRI shared response model. In: 28Th advances in neural information processing systems (NIPS-15), p. 460–468. Advances in neural information processing systems (NIPS), december/7–12, Montral; 2015.Google Scholar
  5. 5.
    Chen PH, Zhu X, Zhang H, Turek JS, Chen J, Willke TL, Hasson U, Ramadge PJ. A convolutional autoencoder for multi-subject fmri data aggregation. In: 29th workshop of representation learning in artificial and biological neural networks. Advances in neural information processing systems (NIPS), december/5–10, barcelona; 2016.Google Scholar
  6. 6.
    Cohen L, Dehaene S, Naccache L, Lehéricy S, Dehaene-Lambertz G, Hénaff MA, Michel F. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain 2000;123(2):291–307.CrossRefPubMedGoogle Scholar
  7. 7.
    Connolly A, Gobbini M, Haxby J. 2012. Three virtues of similarity-based multi-voxel pattern analysis.Google Scholar
  8. 8.
    Connolly AC, Guntupalli JS, Gors J, Hanke M, Halchenko YO, Wu YC, Abdi H, Haxby JV. The representation of biological classes in the human brain. J Neurosci 2012;32(8):2608–2618.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Cox DD, Savoy RL. Functional magnetic resonance imaging (fMRI) brain reading: detecting and classifying distributed patterns of fmri activity in human visual cortex. NeuroImage 2003;19(2):261–270.CrossRefPubMedGoogle Scholar
  10. 10.
    Duncan KJ, Pattamadilok C, Knierim I, Devlin JT. Consistency and variability in functional localisers. NeuroImage 2009;46(4):1018–1026.CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Escalera S, Pujol O, Radeva P. Error-correcting output codes library. J Mach Learn Res 2010;11(Feb): 661–664.Google Scholar
  12. 12.
    Friston KJ. Statistical parametric mapping. In: Neuroscience databases. Berlin: Springer; 2003. p. 237–250.Google Scholar
  13. 13.
    Haxby JV, Connolly AC, Guntupalli JS. Decoding neural representational spaces using multivariate pattern analysis. Ann Rev Neurosci 2014;37:435–456.CrossRefPubMedGoogle Scholar
  14. 14.
    Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 2001;293(5539):2425–2430.CrossRefPubMedGoogle Scholar
  15. 15.
    Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, Ramadge PJ. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 2011;72(2):404–416.CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Haynes JD, Rees G. Decoding mental states from brain activity in humans. Nat Rev Neurosci 2006;7(7): 523.CrossRefPubMedGoogle Scholar
  17. 17.
    Haynes JD, Sakai K, Rees G, Gilbert S, Frith C, Passingham RE. Reading hidden intentions in the human brain. Curr Biol 2007;17(4):323–328.CrossRefPubMedGoogle Scholar
  18. 18.
    Jenkinson M, Bannister P, Brady M, Smith S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 2002;17(2):825–841.CrossRefPubMedGoogle Scholar
  19. 19.
    Kamitani Y, Tong F. Decoding the visual and subjective contents of the human brain. Nat Neurosci 2005;8 (5):679–685.CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Kanwisher N, McDermott J, Chun MM. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J Neurosci 1997;17(11):4302–4311.CrossRefPubMedGoogle Scholar
  21. 21.
    Kay KN, Naselaris T, Prenger RJ, Gallant JL. Identifying natural images from human brain activity. Nature 2008;452(7185):352.CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Kriegeskorte N, Mur M, Bandettini P. Representational similarity analysis–connecting the branches of systems neuroscience. Front Syst Neurosci 2008;2:1–28.Google Scholar
  23. 23.
    Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI. Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 2009;12(5):535–540.CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Liesegang TJ. A cortical area selective for visual processing of the human body. downing pe, 1 school of psychology, centre for cognitive neuroscience, University of Wales, bangor, ll57 2as, United Kingdom. e-mail: p. downing@ bangor. ac. uk jiang y, shuman m, kanwisher n. science 2001; 293: 2470–2473. Am J Ophthalmol 2002;133(4):598.Google Scholar
  25. 25.
    Liu XY, Wu J, Zhou ZH. Exploratory undersampling for class-imbalance learning. IEEE Trans Syst Man Cybern B Cybern 2009;39(2):539–550.CrossRefPubMedGoogle Scholar
  26. 26.
    Malach R, Reppas J, Benson R, Kwong K, Jiang H, Kennedy W, Ledden P, Brady T, Rosen B, Tootell R. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences (PNAS) 1995;92(18):8135–8139.CrossRefGoogle Scholar
  27. 27.
    McMenamin BW, Deason RG, Steele VR, Koutstaal W, Marsolek CJ. Separability of abstract-category and specific-exemplar visual object subsystems: Evidence from fMRI pattern analysis. Brain Cogn 2015;93:54–63.CrossRefPubMedGoogle Scholar
  28. 28.
    Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA. Predicting human brain activity associated with the meanings of nouns. Science 2008;320(5880):1191–1195.CrossRefPubMedGoogle Scholar
  29. 29.
    Miyawaki Y, Uchida H, Yamashita O, Sato Ma, Morito Y, Tanabe HC, Sadato N, Kamitani Y. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 2008;60(5):915–929.CrossRefPubMedGoogle Scholar
  30. 30.
    Mohr H, Wolfensteller U, Frimmel S, Ruge H. Sparse regularization techniques provide novel insights into outcome integration processes. NeuroImage 2015;104:163–176.CrossRefPubMedGoogle Scholar
  31. 31.
    Norman KA, Polyn SM, Detre GJ, Haxby JV. Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends Cogn Sci 2006;10(9):424–430.CrossRefPubMedGoogle Scholar
  32. 32.
    Osher DE, Saxe RR, Koldewyn K, Gabrieli JD, Kanwisher N, Saygin ZM. Structural connectivity fingerprints predict cortical selectivity for multiple visual categories across cortex. Cereb Cortex 2015;26 (4):1668–1683.CrossRefPubMedPubMedCentralGoogle Scholar
  33. 33.
    O’toole AJ, Jiang F, Abdi H, Haxby JV. Partially distributed representations of objects and faces in ventral temporal cortex. J Cogn Neurosci 2005;17(4):580–590.CrossRefPubMedGoogle Scholar
  34. 34.
    Rice GE, Watson DM, Hartley T, Andrews TJ. Low-level image properties of visual objects predict patterns of neural response across category-selective regions of the ventral visual pathway. J Neurosci 2014;34(26): 8837–8844.CrossRefPubMedPubMedCentralGoogle Scholar
  35. 35.
    Varoquaux G, Gramfort A, Thirion B. Small-sample brain mapping: sparse recovery on spatially correlated designs with randomization and clustering. In: Proceedings of the 29th international conference on machine learning (ICML-12); 2012. p. 1375–1382.Google Scholar
  36. 36.
    Wakeman DG, Henson RN. A multi-subject, multi-modal human neuroimaging dataset. Scientific Data 2015; 2:1–10.CrossRefGoogle Scholar
  37. 37.
    Xu J, Potenza MN, Calhoun VD. Spatial ICA reveals functional activity hidden from traditional fMRI GLM-based analyses. Front Neurosci 2013;7:1–4.CrossRefGoogle Scholar
  38. 38.
    Yamashita O, Sato Ma, Yoshioka T, Tong F, Kamitani Y. Sparse estimation automatically selects voxels relevant for the decoding of fmri activity patterns. NeuroImage 2008;42(4):1414– 1429.CrossRefPubMedPubMedCentralGoogle Scholar
  39. 39.
    Yousefnezhad M, Zhang D. Decoding visual stimuli in human brain by using anatomical pattern analysis on fMRI images. In: 8Th international conference on brain inspired cognitive systems (BICS’16), p. 47–57. Springer, November/28–30, Beijing; 2016.Google Scholar
  40. 40.
    Yousefnezhad M, Zhang D. Local discriminant hyperalignment for multi-subject fmri data alignment. In: 34Th AAAI conference on artificial intelligence (AAAI-17), pp. 59–65. Association for the advancement of artificial intelligence (AAAI), february/4–9, san francisco; 2017.Google Scholar
  41. 41.
    Yousefnezhad M, Zhang D. Multi-region neural representation: a novel model for decoding visual stimuli in human brains. In: 17Th SIAM international conference on data mininig (SDM-17), pp. 54–62. Society for industrial and applied mathematics (SIAM), april/27–29, houston; 2017.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.College of Computer Science and TechnologyNanjing University of Aeronautics and AstronauticsNanjingChina

Personalised recommendations