Encoding the local connectivity patterns of fMRI for cognitive task and state classification


In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5


  1. Abbas, A., Deligiannis, N., Andreopoulos, Y. (2015). Vectors of locally aggregated centers for compact video representation. In International conference on multimedia and expo (ICME) (pp. 1–6). IEEE.

  2. Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., et al. (2013). Function in the human connectome: task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fMRI data. Journal of Integrative Neuroscience, 13(04), 645–667.

    Article  PubMed  Google Scholar 

  4. Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fMRI. Brain Imaging and Behavior, 9(2), 178–189.

    Article  PubMed  Google Scholar 

  5. Cai, S., Chong, T., Peng, Y., Shen, W., Li, J., von Deneen, K.M., Huang, L., Initiative, A.D.N., et al. (2017). Altered functional brain networks in amnestic mild cognitive impairment: a resting-state fMRI study. Brain Imaging and Behavior, 11(3), 619–631.

    Article  PubMed  Google Scholar 

  6. Carvajal, J, McCool, C, Lovell, B, Sanderson, C. (2016). Joint recognition and segmentation of actions via probabilistic integration of spatio-temporal fisher vectors. arXiv:160201601.

  7. Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fMRI: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.

    Article  PubMed  Google Scholar 

  9. Daliri, M.R. (2014). A hybrid method for the decoding of spatial attention using the meg brain signals. Biomedical Signal Processing and Control, 10, 308–312.

    Article  Google Scholar 

  10. Delhumeau, J, Gosselin, P.H., Jégou, H., Pérez, P. (2013). Revisiting the vlad image representation. In International conference on multimedia (pp. 653–656). ACM.

  11. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1–30.

    Google Scholar 

  12. Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425–2429.

    Article  CAS  PubMed  Google Scholar 

  13. Jaakkola, T.S., Haussler, D., et al. (1999). Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems, 487–493.

  14. Jégou, H, Douze, M., Schmid, C., Pérez, P. (2010). Aggregating local descriptors into a compact image representation. In Computer Vision and pattern recognition (CVPR) (pp. 3304–3311). IEEE.

  15. Khazaee, A., Ebrahimzadeh, A., Babajani-Feremi, A. (2016). Application of advanced machine learning methods on resting-state fMRI network for identification of mild cognitive impairment and alzheimer’s disease. Brain Imaging and Behavior, 10(3), 799– 817.

    Article  PubMed  Google Scholar 

  16. Lee, Y.S., Peelle, J.E., Kraemer, D., Lloyd, S., Granger, R. (2015). Multivariate sensitivity to voice during auditory categorization. Journal of Neurophysiology, 114(3), 1819–1826.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Liu, L, Shen, C, Wang, L, Van Den Hengel, A, Wang, C. (2014). Encoding high dimensional local features by sparse coding based fisher vectors. In Advances in neural information processing systems (pp. 1143–1151).

  18. Meriño, LM, Meng, J., Gordon, S., Lance, B.J., Johnson, T., Paul, V., Robbins, K., Vettel, J.M., Huang, Y. (2013). A bag-of-words model for task-load prediction from eeg in complex environments. In International Conference on acoustics, speech and signal processing (ICASSP) (pp. 1227–1231). IEEE.

  19. Mitchell, T.M., Hutchinson, R., Niculescu, R.S., Pereira, F., Wang, X., Just, M., Newman, S. (2004). Learning to decode cognitive states from brain images. Machine Learning, 57(1–2), 145–175.

    Article  Google Scholar 

  20. Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320(5880), 1191–1195.

    Article  CAS  PubMed  Google Scholar 

  21. Mızrak, E, Singmann, H, Öztekin, I. (2017). Forgetting emotional material in working memory. Social Cognitive and Affective Neuroscience, 1, 10.

    Google Scholar 

  22. Onal, I., Ozay, M., Vural, F.T.Y. (2015a). Functional mesh model with temporal measurements for brain decoding. In International Conference of the engineering in medicine and biology society (EMBC) (pp. 2624–2628). IEEE.

  23. Onal, I, Ozay, M, Vural, F.T.Y. (2015b). Modeling voxel connectivity for brain decoding. In: International Workshop on pattern recognition in NeuroImaging (pp. 5–8). IEEE.

  24. Onal, I., Ozay, M., Mizrak, E., Oztekin, I, Yarman-Vural, F. (2017). A new representation of fMRI signal by a set of local meshes for brain decoding. IEEE Transactions on Signal and Information Processing over Networks.

  25. Oneata, D, Verbeek, J, Schmid, C. (2013). Action and event recognition with fisher vectors on a compact feature set. In International Conference on computer vision (pp. 1817–1824). IEEE.

  26. Ozay, M., Ȯztekin, I, Ȯztekin, U, Yarman-Vural, F.T. (2012). Mesh learning for classifying cognitive processes. CoRR arXiv:1205.2382.

  27. Perronnin, F., & Dance, C. (2007). Fisher kernels on visual vocabularies for image categorization. In Computer Vision and pattern recognition (CVPR) (pp. 1–8). IEEE.

  28. Perronnin, F., Sánchez, J, Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. Computer Vision–ECCV, 2010, 143–156.

    Google Scholar 

  29. Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., Ville, D.V.D. (2011). Decoding brain states from fMRI connectivity graphs. NeuroImage, 56(2), 616–626.

    Article  PubMed  Google Scholar 

  30. Rissman, J., Chow, T.E., Reggente, N., Wagner, A.D. (2016). Decoding fMRI signatures of real-world autobiographical memory retrieval. Journal of Cognitive Neuroscience, 28(4), 604–620.

    Article  PubMed  Google Scholar 

  31. Saarimäki, H, Gotsopoulos, A., Jääskeläinen, I P, Lampinen, J., Vuilleumier, P., Hari, R., Sams, M., Nummenmaa, L. (2015). Discrete neural signatures of basic emotions. Cerebral Cortex, 26(6), 2563–2573.

    Article  PubMed  Google Scholar 

  32. Sánchez, J, & Redolfi, J. (2015). Exponential family fisher vector for image classification. Pattern Recognition Letters, 59, 26–32.

    Article  Google Scholar 

  33. Sánchez, J, Perronnin, F., Mensink, T., Verbeek, J. (2013). Image classification with the fisher vector: theory and practice. International Journal of Computer Vision, 105(3), 222–245.

    Article  Google Scholar 

  34. Savelonas, M.A., Pratikakis, I., Sfikas, K. (2016). Fisher encoding of differential fast point feature histograms for partial 3d object retrieval. Pattern Recognition, 55, 114–124.

    Article  Google Scholar 

  35. Sekma, M., Mejdoub, M., Amar, C.B. (2015). Human action recognition based on multi-layer fisher vector encoding method. Pattern Recognition Letters, 65, 37–43.

    Article  Google Scholar 

  36. Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex.

  37. Simonyan, K., Parkhi, O.M., Vedaldi, A., Zisserman, A. (2013). Fisher vector faces in the wild. In British Machine vision conference (Vol. 5, pp. 11).

  38. Solmaz, B, Dey, S, Rao, A.R., Shah, M. (2012). Adhd classification using bag of words approach on network features. In SPIE Medical Imaging (pp. 83,144T–83,144T).

  39. Sucu, G., Akbas, E., Oztekin, I., Mizrak, E., Vural, F.Y. (2016). Decoding cognitive states using the bag of words model on fMRI time series. In Signal Processing and communication application conference (SIU) (pp. 2245–2248). IEEE.

  40. Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., Joliot, M. (2002). Automated anatomical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain. NeuroImage, 15(1), 273–289.

    Article  CAS  Google Scholar 

  41. Wang, J., Liu, P., She, M.F., Nahavandi, S., Kouzani, A. (2013). Bag-of-words representation for biomedical time series classification. Biomedical Signal Processing and Control, 8(6), 634– 644.

    Article  Google Scholar 

  42. Xia, M., Wang, J., He, Y. (2013). Brainnet viewer: a network visualization tool for human brain connectomics. PloS one, 8(7), e68,910.

    Article  CAS  Google Scholar 

  43. Zhou, L., Wang, L., Liu, L., Ogunbona, P., Shen, D. (2016). Learning discriminative Bayesian networks from high-dimensional continuous neuroimaging data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(11), 2269–2283.

    Article  PubMed  Google Scholar 

Download references


This work was completed when Itir Onal Ertugrul was with the Department of Computer Engineering, METU.


This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.

Author information



Corresponding author

Correspondence to Itir Onal Ertugrul.

Ethics declarations

Conflict of interests

Itir Onal Ertugrul, Mete Ozay and Fatos T. Yarman Vural declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Onal Ertugrul, I., Ozay, M. & Yarman Vural, F.T. Encoding the local connectivity patterns of fMRI for cognitive task and state classification. Brain Imaging and Behavior 13, 893–904 (2019). https://doi.org/10.1007/s11682-018-9901-5

Download citation


  • fMRI
  • Brain decoding
  • Fisher vector encoding
  • Mesh arc descriptors