Encoding the local connectivity patterns of fMRI for cognitive task and state classification

  • Itir Onal ErtugrulEmail author
  • Mete Ozay
  • Fatos T. Yarman Vural
Original Research


In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.


fMRI Brain decoding Fisher vector encoding Mesh arc descriptors 



This work was completed when Itir Onal Ertugrul was with the Department of Computer Engineering, METU.


This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.

Compliance with Ethical Standards

Conflict of interests

Itir Onal Ertugrul, Mete Ozay and Fatos T. Yarman Vural declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected.


  1. Abbas, A., Deligiannis, N., Andreopoulos, Y. (2015). Vectors of locally aggregated centers for compact video representation. In International conference on multimedia and expo (ICME) (pp. 1–6). IEEE.Google Scholar
  2. Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., et al. (2013). Function in the human connectome: task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189.CrossRefPubMedPubMedCentralGoogle Scholar
  3. Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fMRI data. Journal of Integrative Neuroscience, 13(04), 645–667.CrossRefPubMedGoogle Scholar
  4. Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fMRI. Brain Imaging and Behavior, 9(2), 178–189.CrossRefPubMedGoogle Scholar
  5. Cai, S., Chong, T., Peng, Y., Shen, W., Li, J., von Deneen, K.M., Huang, L., Initiative, A.D.N., et al. (2017). Altered functional brain networks in amnestic mild cognitive impairment: a resting-state fMRI study. Brain Imaging and Behavior, 11(3), 619–631.CrossRefPubMedGoogle Scholar
  6. Carvajal, J, McCool, C, Lovell, B, Sanderson, C. (2016). Joint recognition and segmentation of actions via probabilistic integration of spatio-temporal fisher vectors. arXiv:160201601.
  7. Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fMRI: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.CrossRefPubMedGoogle Scholar
  9. Daliri, M.R. (2014). A hybrid method for the decoding of spatial attention using the meg brain signals. Biomedical Signal Processing and Control, 10, 308–312.CrossRefGoogle Scholar
  10. Delhumeau, J, Gosselin, P.H., Jégou, H., Pérez, P. (2013). Revisiting the vlad image representation. In International conference on multimedia (pp. 653–656). ACM.Google Scholar
  11. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1–30.Google Scholar
  12. Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425–2429.CrossRefPubMedGoogle Scholar
  13. Jaakkola, T.S., Haussler, D., et al. (1999). Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems, 487–493.Google Scholar
  14. Jégou, H, Douze, M., Schmid, C., Pérez, P. (2010). Aggregating local descriptors into a compact image representation. In Computer Vision and pattern recognition (CVPR) (pp. 3304–3311). IEEE.Google Scholar
  15. Khazaee, A., Ebrahimzadeh, A., Babajani-Feremi, A. (2016). Application of advanced machine learning methods on resting-state fMRI network for identification of mild cognitive impairment and alzheimer’s disease. Brain Imaging and Behavior, 10(3), 799– 817.CrossRefPubMedGoogle Scholar
  16. Lee, Y.S., Peelle, J.E., Kraemer, D., Lloyd, S., Granger, R. (2015). Multivariate sensitivity to voice during auditory categorization. Journal of Neurophysiology, 114(3), 1819–1826.CrossRefPubMedPubMedCentralGoogle Scholar
  17. Liu, L, Shen, C, Wang, L, Van Den Hengel, A, Wang, C. (2014). Encoding high dimensional local features by sparse coding based fisher vectors. In Advances in neural information processing systems (pp. 1143–1151).Google Scholar
  18. Meriño, LM, Meng, J., Gordon, S., Lance, B.J., Johnson, T., Paul, V., Robbins, K., Vettel, J.M., Huang, Y. (2013). A bag-of-words model for task-load prediction from eeg in complex environments. In International Conference on acoustics, speech and signal processing (ICASSP) (pp. 1227–1231). IEEE.Google Scholar
  19. Mitchell, T.M., Hutchinson, R., Niculescu, R.S., Pereira, F., Wang, X., Just, M., Newman, S. (2004). Learning to decode cognitive states from brain images. Machine Learning, 57(1–2), 145–175.CrossRefGoogle Scholar
  20. Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320(5880), 1191–1195.CrossRefPubMedGoogle Scholar
  21. Mızrak, E, Singmann, H, Öztekin, I. (2017). Forgetting emotional material in working memory. Social Cognitive and Affective Neuroscience, 1, 10.Google Scholar
  22. Onal, I., Ozay, M., Vural, F.T.Y. (2015a). Functional mesh model with temporal measurements for brain decoding. In International Conference of the engineering in medicine and biology society (EMBC) (pp. 2624–2628). IEEE.Google Scholar
  23. Onal, I, Ozay, M, Vural, F.T.Y. (2015b). Modeling voxel connectivity for brain decoding. In: International Workshop on pattern recognition in NeuroImaging (pp. 5–8). IEEE.Google Scholar
  24. Onal, I., Ozay, M., Mizrak, E., Oztekin, I, Yarman-Vural, F. (2017). A new representation of fMRI signal by a set of local meshes for brain decoding. IEEE Transactions on Signal and Information Processing over Networks.Google Scholar
  25. Oneata, D, Verbeek, J, Schmid, C. (2013). Action and event recognition with fisher vectors on a compact feature set. In International Conference on computer vision (pp. 1817–1824). IEEE.Google Scholar
  26. Ozay, M., Ȯztekin, I, Ȯztekin, U, Yarman-Vural, F.T. (2012). Mesh learning for classifying cognitive processes. CoRR arXiv:1205.2382.
  27. Perronnin, F., & Dance, C. (2007). Fisher kernels on visual vocabularies for image categorization. In Computer Vision and pattern recognition (CVPR) (pp. 1–8). IEEE.Google Scholar
  28. Perronnin, F., Sánchez, J, Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. Computer Vision–ECCV, 2010, 143–156.Google Scholar
  29. Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., Ville, D.V.D. (2011). Decoding brain states from fMRI connectivity graphs. NeuroImage, 56(2), 616–626.CrossRefPubMedGoogle Scholar
  30. Rissman, J., Chow, T.E., Reggente, N., Wagner, A.D. (2016). Decoding fMRI signatures of real-world autobiographical memory retrieval. Journal of Cognitive Neuroscience, 28(4), 604–620.CrossRefPubMedGoogle Scholar
  31. Saarimäki, H, Gotsopoulos, A., Jääskeläinen, I P, Lampinen, J., Vuilleumier, P., Hari, R., Sams, M., Nummenmaa, L. (2015). Discrete neural signatures of basic emotions. Cerebral Cortex, 26(6), 2563–2573.CrossRefPubMedGoogle Scholar
  32. Sánchez, J, & Redolfi, J. (2015). Exponential family fisher vector for image classification. Pattern Recognition Letters, 59, 26–32.CrossRefGoogle Scholar
  33. Sánchez, J, Perronnin, F., Mensink, T., Verbeek, J. (2013). Image classification with the fisher vector: theory and practice. International Journal of Computer Vision, 105(3), 222–245.CrossRefGoogle Scholar
  34. Savelonas, M.A., Pratikakis, I., Sfikas, K. (2016). Fisher encoding of differential fast point feature histograms for partial 3d object retrieval. Pattern Recognition, 55, 114–124.CrossRefGoogle Scholar
  35. Sekma, M., Mejdoub, M., Amar, C.B. (2015). Human action recognition based on multi-layer fisher vector encoding method. Pattern Recognition Letters, 65, 37–43.CrossRefGoogle Scholar
  36. Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex.Google Scholar
  37. Simonyan, K., Parkhi, O.M., Vedaldi, A., Zisserman, A. (2013). Fisher vector faces in the wild. In British Machine vision conference (Vol. 5, pp. 11).Google Scholar
  38. Solmaz, B, Dey, S, Rao, A.R., Shah, M. (2012). Adhd classification using bag of words approach on network features. In SPIE Medical Imaging (pp. 83,144T–83,144T).Google Scholar
  39. Sucu, G., Akbas, E., Oztekin, I., Mizrak, E., Vural, F.Y. (2016). Decoding cognitive states using the bag of words model on fMRI time series. In Signal Processing and communication application conference (SIU) (pp. 2245–2248). IEEE.Google Scholar
  40. Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., Joliot, M. (2002). Automated anatomical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain. NeuroImage, 15(1), 273–289.CrossRefPubMedGoogle Scholar
  41. Wang, J., Liu, P., She, M.F., Nahavandi, S., Kouzani, A. (2013). Bag-of-words representation for biomedical time series classification. Biomedical Signal Processing and Control, 8(6), 634– 644.CrossRefGoogle Scholar
  42. Xia, M., Wang, J., He, Y. (2013). Brainnet viewer: a network visualization tool for human brain connectomics. PloS one, 8(7), e68,910.CrossRefGoogle Scholar
  43. Zhou, L., Wang, L., Liu, L., Ogunbona, P., Shen, D. (2016). Learning discriminative Bayesian networks from high-dimensional continuous neuroimaging data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(11), 2269–2283.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.Graduate School of Information SciencesTohoku UniversitySendaiJapan
  3. 3.Department of Computer EngineeringMiddle East Technical UniversityAnkaraTurkey

Personalised recommendations