In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Abbas, A., Deligiannis, N., Andreopoulos, Y. (2015). Vectors of locally aggregated centers for compact video representation. In International conference on multimedia and expo (ICME) (pp. 1–6). IEEE.
Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., et al. (2013). Function in the human connectome: task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189.
Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fMRI data. Journal of Integrative Neuroscience, 13(04), 645–667.
Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fMRI. Brain Imaging and Behavior, 9(2), 178–189.
Cai, S., Chong, T., Peng, Y., Shen, W., Li, J., von Deneen, K.M., Huang, L., Initiative, A.D.N., et al. (2017). Altered functional brain networks in amnestic mild cognitive impairment: a resting-state fMRI study. Brain Imaging and Behavior, 11(3), 619–631.
Carvajal, J, McCool, C, Lovell, B, Sanderson, C. (2016). Joint recognition and segmentation of actions via probabilistic integration of spatio-temporal fisher vectors. arXiv:160201601.
Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fMRI: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.
Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.
Daliri, M.R. (2014). A hybrid method for the decoding of spatial attention using the meg brain signals. Biomedical Signal Processing and Control, 10, 308–312.
Delhumeau, J, Gosselin, P.H., Jégou, H., Pérez, P. (2013). Revisiting the vlad image representation. In International conference on multimedia (pp. 653–656). ACM.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1–30.
Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539), 2425–2429.
Jaakkola, T.S., Haussler, D., et al. (1999). Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems, 487–493.
Jégou, H, Douze, M., Schmid, C., Pérez, P. (2010). Aggregating local descriptors into a compact image representation. In Computer Vision and pattern recognition (CVPR) (pp. 3304–3311). IEEE.
Khazaee, A., Ebrahimzadeh, A., Babajani-Feremi, A. (2016). Application of advanced machine learning methods on resting-state fMRI network for identification of mild cognitive impairment and alzheimer’s disease. Brain Imaging and Behavior, 10(3), 799– 817.
Lee, Y.S., Peelle, J.E., Kraemer, D., Lloyd, S., Granger, R. (2015). Multivariate sensitivity to voice during auditory categorization. Journal of Neurophysiology, 114(3), 1819–1826.
Liu, L, Shen, C, Wang, L, Van Den Hengel, A, Wang, C. (2014). Encoding high dimensional local features by sparse coding based fisher vectors. In Advances in neural information processing systems (pp. 1143–1151).
Meriño, LM, Meng, J., Gordon, S., Lance, B.J., Johnson, T., Paul, V., Robbins, K., Vettel, J.M., Huang, Y. (2013). A bag-of-words model for task-load prediction from eeg in complex environments. In International Conference on acoustics, speech and signal processing (ICASSP) (pp. 1227–1231). IEEE.
Mitchell, T.M., Hutchinson, R., Niculescu, R.S., Pereira, F., Wang, X., Just, M., Newman, S. (2004). Learning to decode cognitive states from brain images. Machine Learning, 57(1–2), 145–175.
Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320(5880), 1191–1195.
Mızrak, E, Singmann, H, Öztekin, I. (2017). Forgetting emotional material in working memory. Social Cognitive and Affective Neuroscience, 1, 10.
Onal, I., Ozay, M., Vural, F.T.Y. (2015a). Functional mesh model with temporal measurements for brain decoding. In International Conference of the engineering in medicine and biology society (EMBC) (pp. 2624–2628). IEEE.
Onal, I, Ozay, M, Vural, F.T.Y. (2015b). Modeling voxel connectivity for brain decoding. In: International Workshop on pattern recognition in NeuroImaging (pp. 5–8). IEEE.
Onal, I., Ozay, M., Mizrak, E., Oztekin, I, Yarman-Vural, F. (2017). A new representation of fMRI signal by a set of local meshes for brain decoding. IEEE Transactions on Signal and Information Processing over Networks.
Oneata, D, Verbeek, J, Schmid, C. (2013). Action and event recognition with fisher vectors on a compact feature set. In International Conference on computer vision (pp. 1817–1824). IEEE.
Ozay, M., Ȯztekin, I, Ȯztekin, U, Yarman-Vural, F.T. (2012). Mesh learning for classifying cognitive processes. CoRR arXiv:1205.2382.
Perronnin, F., & Dance, C. (2007). Fisher kernels on visual vocabularies for image categorization. In Computer Vision and pattern recognition (CVPR) (pp. 1–8). IEEE.
Perronnin, F., Sánchez, J, Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. Computer Vision–ECCV, 2010, 143–156.
Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., Ville, D.V.D. (2011). Decoding brain states from fMRI connectivity graphs. NeuroImage, 56(2), 616–626.
Rissman, J., Chow, T.E., Reggente, N., Wagner, A.D. (2016). Decoding fMRI signatures of real-world autobiographical memory retrieval. Journal of Cognitive Neuroscience, 28(4), 604–620.
Saarimäki, H, Gotsopoulos, A., Jääskeläinen, I P, Lampinen, J., Vuilleumier, P., Hari, R., Sams, M., Nummenmaa, L. (2015). Discrete neural signatures of basic emotions. Cerebral Cortex, 26(6), 2563–2573.
Sánchez, J, & Redolfi, J. (2015). Exponential family fisher vector for image classification. Pattern Recognition Letters, 59, 26–32.
Sánchez, J, Perronnin, F., Mensink, T., Verbeek, J. (2013). Image classification with the fisher vector: theory and practice. International Journal of Computer Vision, 105(3), 222–245.
Savelonas, M.A., Pratikakis, I., Sfikas, K. (2016). Fisher encoding of differential fast point feature histograms for partial 3d object retrieval. Pattern Recognition, 55, 114–124.
Sekma, M., Mejdoub, M., Amar, C.B. (2015). Human action recognition based on multi-layer fisher vector encoding method. Pattern Recognition Letters, 65, 37–43.
Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex.
Simonyan, K., Parkhi, O.M., Vedaldi, A., Zisserman, A. (2013). Fisher vector faces in the wild. In British Machine vision conference (Vol. 5, pp. 11).
Solmaz, B, Dey, S, Rao, A.R., Shah, M. (2012). Adhd classification using bag of words approach on network features. In SPIE Medical Imaging (pp. 83,144T–83,144T).
Sucu, G., Akbas, E., Oztekin, I., Mizrak, E., Vural, F.Y. (2016). Decoding cognitive states using the bag of words model on fMRI time series. In Signal Processing and communication application conference (SIU) (pp. 2245–2248). IEEE.
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., Joliot, M. (2002). Automated anatomical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain. NeuroImage, 15(1), 273–289.
Wang, J., Liu, P., She, M.F., Nahavandi, S., Kouzani, A. (2013). Bag-of-words representation for biomedical time series classification. Biomedical Signal Processing and Control, 8(6), 634– 644.
Xia, M., Wang, J., He, Y. (2013). Brainnet viewer: a network visualization tool for human brain connectomics. PloS one, 8(7), e68,910.
Zhou, L., Wang, L., Liu, L., Ogunbona, P., Shen, D. (2016). Learning discriminative Bayesian networks from high-dimensional continuous neuroimaging data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(11), 2269–2283.
This work was completed when Itir Onal Ertugrul was with the Department of Computer Engineering, METU.
This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.
Conflict of interests
Itir Onal Ertugrul, Mete Ozay and Fatos T. Yarman Vural declare that they have no conflicts of interest.
This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected.
About this article
Cite this article
Onal Ertugrul, I., Ozay, M. & Yarman Vural, F.T. Encoding the local connectivity patterns of fMRI for cognitive task and state classification. Brain Imaging and Behavior 13, 893–904 (2019). https://doi.org/10.1007/s11682-018-9901-5
- Brain decoding
- Fisher vector encoding
- Mesh arc descriptors