Brain Imaging and Behavior

, Volume 12, Issue 4, pp 1067–1083 | Cite as

Hierarchical multi-resolution mesh networks for brain decoding

  • Itir Onal Ertugrul
  • Mete Ozay
  • Fatos T. Yarman Vural
Original Research


Human brain is supposed to process information in multiple frequency bands. Therefore, we can extract diverse information from functional Magnetic Resonance Imaging (fMRI) data by processing it at multiple resolutions. We propose a framework, called Hierarchical Multi-resolution Mesh Networks (HMMNs), which establishes a set of brain networks at multiple resolutions of fMRI signal to represent the underlying cognitive process. Our framework, first, decomposes the fMRI signal into various frequency subbands using wavelet transform. Then, a brain network is formed at each subband by ensembling a set of local meshes. Arc weights of each local mesh are estimated by ridge regression. Finally, adjacency matrices of mesh networks obtained at different subbands are used to train classifiers in an ensemble learning architecture, called fuzzy stacked generalization (FSG). Our decoding performances on Human Connectome Project task-fMRI dataset reflect that HMMNs can successfully discriminate tasks with 99% accuracy, across 808 subjects. Diversity of information embedded in mesh networks of multiple subbands enables the ensemble of classifiers to collaborate with each other for brain decoding. The suggested HMMNs decode the cognitive tasks better than a single classifier applied to any subband. Also mesh networks have a better representation power compared to pairwise correlations or average voxel time series. Moreover, fusion of diverse information using FSG outperforms fusion with majority voting. We conclude that, fMRI data, recorded during a cognitive task, provide diverse information in multi-resolution mesh networks. Our framework fuses this complementary information and boosts the brain decoding performances obtained at individual subbands.


fMRI data analysis Brain decoding Wavelet decomposition Mesh networks Hierarchical models Ensemble models 



Itir Onal Ertugrul is funded by TUBITAK 2211E. This work was supported by CREST, JST, and TUBITAK Project No 116E091.

Compliance with Ethical Standards

Conflict of interests

Itir Onal Ertugrul, Mete Ozay and Fatos Yarman Vural declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected and archived in a data bank.


  1. Alkan, S., & Yarman-Vural, F.T. (2015). Ensembling brain regions for brain decoding. In 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 2948–2951).Google Scholar
  2. Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., Nolan, D., Bryant, E., Hartley, T., Footer, O., Bjork, J.M., Poldrack, R., Smith, S., Johansen-Berg, H., Snyder, A.Z., & Essen, D.C.V. (2013). Function in the human connectome: task-fmri and individual differences in behavior. NeuroImage, 80, 169–189.CrossRefPubMedPubMedCentralGoogle Scholar
  3. Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fmri data. Journal of Integrative Neuroscience, 13(04), 645–667.CrossRefPubMedGoogle Scholar
  4. Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fmri. Brain Imaging and Behavior, 9(2), 178–189.CrossRefPubMedGoogle Scholar
  5. Behroozi, M., Daliri, M.R., & Boyaci, H. (2011). Statistical analysis methods for the fmri data. Basic and Clinical Neuroscience, 2(4), 67–74.Google Scholar
  6. Binder, J.R., Gross, W.L., Allendorfer, J.B., Bonilha, L., Chapin, J., Edwards, J.C., Grabowski, T.J., Langfitt, J.T., Loring, D.W., Lowe, M.J., Koenig, K., Morgan, P.S., Ojemann, J.G., Rorden, C., Szaflarski, J.P., Tivarus, M.E., & Weaver, K.E. (2011). Mapping anterior temporal lobe language areas with fmri: a multicenter normative study. NeuroImage, 54(2), 1465–1475.CrossRefPubMedGoogle Scholar
  7. Buckner, R.L., Krienen, F.M., Castellanos, A., Diaz, J.C., & Yeo, B.T.T. (2011). The organization of the human cerebellum estimated by intrinsic functional connectivity. Journal of Neurophysiology, 106(5), 2322–2345.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Bullmore, E., Fadili, J., Maxim, V., Şendur, L., Whitcher, B., Suckling, J., Brammer, M., & Breakspear, M. (2004). Wavelets and functional magnetic resonance imaging of the human brain. NeuroImage, 23(Supplement 1), S234–S249.CrossRefPubMedGoogle Scholar
  9. Cabral, C., Silveira, M., & Figueiredo, P. (2012). Decoding visual brain states from fmri using an ensemble of classifiers. Pattern Recognition, 45(6), 2064–2074.CrossRefGoogle Scholar
  10. Castelli, F., Happé, F., Frith, U., & Frith, C. (2000). Movement and mind: a functional imaging study of perception and interpretation of complex intentional movement patterns. NeuroImage, 12(3), 314–325.CrossRefPubMedGoogle Scholar
  11. Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., & Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fmri: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.CrossRefPubMedPubMedCentralGoogle Scholar
  12. Cox, D.D., & Savoy, R.L. (2003). Functional magnetic resonance imaging (fmri) “brain reading”: detecting and classifying distributed patterns of fmri activity in human visual cortex. NeuroImage, 19(2), 261–270.CrossRefPubMedGoogle Scholar
  13. Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.CrossRefPubMedGoogle Scholar
  14. Delgado, M.R., Nystrom, L.E., Fissell, C., Noll, D.C., & Fiez, J.A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology, 84(6), 3072–3077.CrossRefPubMedGoogle Scholar
  15. Dinov, I.D., Boscardin, J.W., Mega, M.S., Sowell, E.L., & Toga, A.W. (2005). A wavelet-based statistical analysis of fmri data. Neuroinformatics, 3(4), 319–342.CrossRefPubMedGoogle Scholar
  16. Ekman, M., Derrfuss, J., Tittgemeyer, M., & Fiebach, C.J. (2012). Predicting errors from reconfiguration patterns in human brain networks. Proceedings of the National Academy of Sciences, 109(41), 16,714–16,719.CrossRefGoogle Scholar
  17. Fan, Y. (2003). On the approximate decorrelation property of the discrete wavelet transform for fractionally differenced processes. IEEE Transactions on Information Theory, 49(2), 516–521.CrossRefGoogle Scholar
  18. Fang, J., Hu, X., Han, J., Jiang, X., Zhu, D., Guo, L., & Liu, T. (2015). Data-driven analysis of functional brain interactions during free listening to music and speech. Brain Imaging and Behavior, 9(2), 162–177.CrossRefPubMedGoogle Scholar
  19. Fornito, A., Harrison, B.J., Zalesky, A., & Simons, J.S. (2012). Competitive and cooperative dynamics of large-scale brain functional networks supporting recollection. Proceedings of the National Academy of Sciences, 109 (31), 12,788–12,793.CrossRefGoogle Scholar
  20. Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al. (2013). The minimal preprocessing pipelines for the human connectome project. NeuroImage, 80, 105–124.CrossRefPubMedPubMedCentralGoogle Scholar
  21. Hariri, A.R., Tessitore, A., Mattay, V.S., Fera, F., & Weinberger, D.R. (2002). The amygdala response to emotional stimuli: a comparison of faces and scenes. NeuroImage, 17(1), 317–323.CrossRefPubMedGoogle Scholar
  22. Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679–685.CrossRefPubMedPubMedCentralGoogle Scholar
  23. Kauppi, J.P., Jääskeläinen, I.P., Sams, M., & Tohka, J. (2010). Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency. Frontiers in Neuroinformatics, 4(5).Google Scholar
  24. Kauppi, J.P., Pajula, J., & Tohka, J. (2014). A versatile software package for inter-subject correlation based analyses of fmri. Frontiers in Neuroinformatics, 8(2).Google Scholar
  25. Kuncheva, L.I. (2004). Combining pattern classifiers: methods and algorithms. Wiley.Google Scholar
  26. Kuncheva, L.I., Rodríguez, J J, Plumpton, C.O., Linden, D.E.J., & Johnston, S.J. (2010). Random subspace ensembles for fmri classification. IEEE Transactions on Medical Imaging, 29(2), 531–542.CrossRefPubMedGoogle Scholar
  27. Lindquist, M.A. (2008). The statistical analysis of fmri data. Statistical Science, 23(4), 439–464.CrossRefGoogle Scholar
  28. Mandelbrot, B.B. (1977). The fractal geometry of nature. New York: Springer.Google Scholar
  29. Onal, I., Ozay, M., & Yarman Vural, F. (2015a). Modeling voxel connectivity for brain decoding. In PRNI, Stanford, CA, USA (pp. 5–8).Google Scholar
  30. Onal, I., Ozay, M., & Yarman Vural, F.T. (2015b). Functional mesh model with temporal measurements for brain decoding. In Engineering in medicine and biology society (EMBC), 2015 37th annual international conference of the IEEE (pp. 2624–2628): IEEE.Google Scholar
  31. Ozay, M., & Yarman-Vural, F.T. (2016). Hierarchical distance learning by stacking nearest neighbor classifiers. Information Fusion, 29, 14–31.CrossRefGoogle Scholar
  32. Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., & Ville, D.V.D. (2011). Decoding brain states from fmri connectivity graphs. NeuroImage, 56(2), 616–626. Multivariate Decoding and Brain Reading.CrossRefPubMedGoogle Scholar
  33. Richiardi, J., Achard, S., Bunke, H., & De Ville, D.V. (2013). Machine learning with brain graphs: predictive modeling approaches for functional imaging in systems neuroscience. IEEE Signal Processing Magazine.Google Scholar
  34. Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., & Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex, 22(1), 158–165.CrossRefPubMedGoogle Scholar
  35. Smith, R., Keramatian, K., & Christoff, K. (2007). Localizing the rostrolateral prefrontal cortex at the individual level. NeuroImage, 36(4), 1387–1396.CrossRefPubMedGoogle Scholar
  36. Thompson, W.H., & Fransson, P. (2015). The frequency dimension of fmri dynamic connectivity: network connectivity, functional hubs and integration in the resting brain. NeuroImage, 121, 227–242.CrossRefPubMedGoogle Scholar
  37. Van De Ville, D., Blu, T., & Unser, M. (2006). Surfing the brain—an overview of wavelet-based techniques for fMRI data analysis. IEEE Engineering in Medicine and Biology Magazine, 25(2), 65–78.CrossRefGoogle Scholar
  38. Wheatley, T., Milleville, S.C., & Martin, A. (2007). Understanding animate agents: distinct roles for the social network and mirror system. Psychological Science, 18(6), 469–474.CrossRefPubMedGoogle Scholar
  39. Xu, Z., & Chan, A.K. (2002). Encoding with frames in mri and analysis of the signal-to-noise ratio. IEEE Transactions on Medical Imaging, 21(4), 332–342.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Department of Computer EngineeringMiddle East Technical UniversityAnkaraTurkey
  2. 2.Graduate School of Information SciencesTohoku UniversitySendaiJapan

Personalised recommendations