Encoding the local connectivity patterns of fMRI for cognitive task and state classification
In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.
KeywordsfMRI Brain decoding Fisher vector encoding Mesh arc descriptors
This work was completed when Itir Onal Ertugrul was with the Department of Computer Engineering, METU.
This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.
Compliance with Ethical Standards
Conflict of interests
Itir Onal Ertugrul, Mete Ozay and Fatos T. Yarman Vural declare that they have no conflicts of interest.
This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected.
- Abbas, A., Deligiannis, N., Andreopoulos, Y. (2015). Vectors of locally aggregated centers for compact video representation. In International conference on multimedia and expo (ICME) (pp. 1–6). IEEE.Google Scholar
- Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., et al. (2013). Function in the human connectome: task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189.CrossRefPubMedPubMedCentralGoogle Scholar
- Carvajal, J, McCool, C, Lovell, B, Sanderson, C. (2016). Joint recognition and segmentation of actions via probabilistic integration of spatio-temporal fisher vectors. arXiv:160201601.
- Delhumeau, J, Gosselin, P.H., Jégou, H., Pérez, P. (2013). Revisiting the vlad image representation. In International conference on multimedia (pp. 653–656). ACM.Google Scholar
- Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1–30.Google Scholar
- Jaakkola, T.S., Haussler, D., et al. (1999). Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems, 487–493.Google Scholar
- Jégou, H, Douze, M., Schmid, C., Pérez, P. (2010). Aggregating local descriptors into a compact image representation. In Computer Vision and pattern recognition (CVPR) (pp. 3304–3311). IEEE.Google Scholar
- Liu, L, Shen, C, Wang, L, Van Den Hengel, A, Wang, C. (2014). Encoding high dimensional local features by sparse coding based fisher vectors. In Advances in neural information processing systems (pp. 1143–1151).Google Scholar
- Meriño, LM, Meng, J., Gordon, S., Lance, B.J., Johnson, T., Paul, V., Robbins, K., Vettel, J.M., Huang, Y. (2013). A bag-of-words model for task-load prediction from eeg in complex environments. In International Conference on acoustics, speech and signal processing (ICASSP) (pp. 1227–1231). IEEE.Google Scholar
- Mızrak, E, Singmann, H, Öztekin, I. (2017). Forgetting emotional material in working memory. Social Cognitive and Affective Neuroscience, 1, 10.Google Scholar
- Onal, I., Ozay, M., Vural, F.T.Y. (2015a). Functional mesh model with temporal measurements for brain decoding. In International Conference of the engineering in medicine and biology society (EMBC) (pp. 2624–2628). IEEE.Google Scholar
- Onal, I, Ozay, M, Vural, F.T.Y. (2015b). Modeling voxel connectivity for brain decoding. In: International Workshop on pattern recognition in NeuroImaging (pp. 5–8). IEEE.Google Scholar
- Onal, I., Ozay, M., Mizrak, E., Oztekin, I, Yarman-Vural, F. (2017). A new representation of fMRI signal by a set of local meshes for brain decoding. IEEE Transactions on Signal and Information Processing over Networks.Google Scholar
- Oneata, D, Verbeek, J, Schmid, C. (2013). Action and event recognition with fisher vectors on a compact feature set. In International Conference on computer vision (pp. 1817–1824). IEEE.Google Scholar
- Ozay, M., Ȯztekin, I, Ȯztekin, U, Yarman-Vural, F.T. (2012). Mesh learning for classifying cognitive processes. CoRR arXiv:1205.2382.
- Perronnin, F., & Dance, C. (2007). Fisher kernels on visual vocabularies for image categorization. In Computer Vision and pattern recognition (CVPR) (pp. 1–8). IEEE.Google Scholar
- Perronnin, F., Sánchez, J, Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. Computer Vision–ECCV, 2010, 143–156.Google Scholar
- Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex.Google Scholar
- Simonyan, K., Parkhi, O.M., Vedaldi, A., Zisserman, A. (2013). Fisher vector faces in the wild. In British Machine vision conference (Vol. 5, pp. 11).Google Scholar
- Solmaz, B, Dey, S, Rao, A.R., Shah, M. (2012). Adhd classification using bag of words approach on network features. In SPIE Medical Imaging (pp. 83,144T–83,144T).Google Scholar
- Sucu, G., Akbas, E., Oztekin, I., Mizrak, E., Vural, F.Y. (2016). Decoding cognitive states using the bag of words model on fMRI time series. In Signal Processing and communication application conference (SIU) (pp. 2245–2248). IEEE.Google Scholar
- Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., Joliot, M. (2002). Automated anatomical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain. NeuroImage, 15(1), 273–289.CrossRefPubMedGoogle Scholar