Skip to main content

Hierarchical multi-resolution mesh networks for brain decoding

Abstract

Human brain is supposed to process information in multiple frequency bands. Therefore, we can extract diverse information from functional Magnetic Resonance Imaging (fMRI) data by processing it at multiple resolutions. We propose a framework, called Hierarchical Multi-resolution Mesh Networks (HMMNs), which establishes a set of brain networks at multiple resolutions of fMRI signal to represent the underlying cognitive process. Our framework, first, decomposes the fMRI signal into various frequency subbands using wavelet transform. Then, a brain network is formed at each subband by ensembling a set of local meshes. Arc weights of each local mesh are estimated by ridge regression. Finally, adjacency matrices of mesh networks obtained at different subbands are used to train classifiers in an ensemble learning architecture, called fuzzy stacked generalization (FSG). Our decoding performances on Human Connectome Project task-fMRI dataset reflect that HMMNs can successfully discriminate tasks with 99% accuracy, across 808 subjects. Diversity of information embedded in mesh networks of multiple subbands enables the ensemble of classifiers to collaborate with each other for brain decoding. The suggested HMMNs decode the cognitive tasks better than a single classifier applied to any subband. Also mesh networks have a better representation power compared to pairwise correlations or average voxel time series. Moreover, fusion of diverse information using FSG outperforms fusion with majority voting. We conclude that, fMRI data, recorded during a cognitive task, provide diverse information in multi-resolution mesh networks. Our framework fuses this complementary information and boosts the brain decoding performances obtained at individual subbands.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    mathworks.com/matlabcentral/fileexchange/9554-a-numerical-tour-of-signal-processing

References

  1. Alkan, S., & Yarman-Vural, F.T. (2015). Ensembling brain regions for brain decoding. In 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 2948–2951).

  2. Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., Nolan, D., Bryant, E., Hartley, T., Footer, O., Bjork, J.M., Poldrack, R., Smith, S., Johansen-Berg, H., Snyder, A.Z., & Essen, D.C.V. (2013). Function in the human connectome: task-fmri and individual differences in behavior. NeuroImage, 80, 169–189.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fmri data. Journal of Integrative Neuroscience, 13(04), 645–667.

    Article  PubMed  Google Scholar 

  4. Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fmri. Brain Imaging and Behavior, 9(2), 178–189.

    Article  PubMed  Google Scholar 

  5. Behroozi, M., Daliri, M.R., & Boyaci, H. (2011). Statistical analysis methods for the fmri data. Basic and Clinical Neuroscience, 2(4), 67–74.

    Google Scholar 

  6. Binder, J.R., Gross, W.L., Allendorfer, J.B., Bonilha, L., Chapin, J., Edwards, J.C., Grabowski, T.J., Langfitt, J.T., Loring, D.W., Lowe, M.J., Koenig, K., Morgan, P.S., Ojemann, J.G., Rorden, C., Szaflarski, J.P., Tivarus, M.E., & Weaver, K.E. (2011). Mapping anterior temporal lobe language areas with fmri: a multicenter normative study. NeuroImage, 54(2), 1465–1475.

    Article  PubMed  Google Scholar 

  7. Buckner, R.L., Krienen, F.M., Castellanos, A., Diaz, J.C., & Yeo, B.T.T. (2011). The organization of the human cerebellum estimated by intrinsic functional connectivity. Journal of Neurophysiology, 106(5), 2322–2345.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Bullmore, E., Fadili, J., Maxim, V., Şendur, L., Whitcher, B., Suckling, J., Brammer, M., & Breakspear, M. (2004). Wavelets and functional magnetic resonance imaging of the human brain. NeuroImage, 23(Supplement 1), S234–S249.

    Article  PubMed  Google Scholar 

  9. Cabral, C., Silveira, M., & Figueiredo, P. (2012). Decoding visual brain states from fmri using an ensemble of classifiers. Pattern Recognition, 45(6), 2064–2074.

    Article  Google Scholar 

  10. Castelli, F., Happé, F., Frith, U., & Frith, C. (2000). Movement and mind: a functional imaging study of perception and interpretation of complex intentional movement patterns. NeuroImage, 12(3), 314–325.

    Article  PubMed  CAS  Google Scholar 

  11. Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., & Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fmri: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Cox, D.D., & Savoy, R.L. (2003). Functional magnetic resonance imaging (fmri) “brain reading”: detecting and classifying distributed patterns of fmri activity in human visual cortex. NeuroImage, 19(2), 261–270.

    Article  PubMed  Google Scholar 

  13. Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.

    Article  PubMed  Google Scholar 

  14. Delgado, M.R., Nystrom, L.E., Fissell, C., Noll, D.C., & Fiez, J.A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology, 84(6), 3072–3077.

    Article  PubMed  CAS  Google Scholar 

  15. Dinov, I.D., Boscardin, J.W., Mega, M.S., Sowell, E.L., & Toga, A.W. (2005). A wavelet-based statistical analysis of fmri data. Neuroinformatics, 3(4), 319–342.

    Article  PubMed  Google Scholar 

  16. Ekman, M., Derrfuss, J., Tittgemeyer, M., & Fiebach, C.J. (2012). Predicting errors from reconfiguration patterns in human brain networks. Proceedings of the National Academy of Sciences, 109(41), 16,714–16,719.

    Article  Google Scholar 

  17. Fan, Y. (2003). On the approximate decorrelation property of the discrete wavelet transform for fractionally differenced processes. IEEE Transactions on Information Theory, 49(2), 516–521.

    Article  Google Scholar 

  18. Fang, J., Hu, X., Han, J., Jiang, X., Zhu, D., Guo, L., & Liu, T. (2015). Data-driven analysis of functional brain interactions during free listening to music and speech. Brain Imaging and Behavior, 9(2), 162–177.

    Article  PubMed  Google Scholar 

  19. Fornito, A., Harrison, B.J., Zalesky, A., & Simons, J.S. (2012). Competitive and cooperative dynamics of large-scale brain functional networks supporting recollection. Proceedings of the National Academy of Sciences, 109 (31), 12,788–12,793.

    Article  Google Scholar 

  20. Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al. (2013). The minimal preprocessing pipelines for the human connectome project. NeuroImage, 80, 105–124.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Hariri, A.R., Tessitore, A., Mattay, V.S., Fera, F., & Weinberger, D.R. (2002). The amygdala response to emotional stimuli: a comparison of faces and scenes. NeuroImage, 17(1), 317–323.

    Article  PubMed  Google Scholar 

  22. Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679–685.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  23. Kauppi, J.P., Jääskeläinen, I.P., Sams, M., & Tohka, J. (2010). Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency. Frontiers in Neuroinformatics, 4(5).

  24. Kauppi, J.P., Pajula, J., & Tohka, J. (2014). A versatile software package for inter-subject correlation based analyses of fmri. Frontiers in Neuroinformatics, 8(2).

  25. Kuncheva, L.I. (2004). Combining pattern classifiers: methods and algorithms. Wiley.

  26. Kuncheva, L.I., Rodríguez, J J, Plumpton, C.O., Linden, D.E.J., & Johnston, S.J. (2010). Random subspace ensembles for fmri classification. IEEE Transactions on Medical Imaging, 29(2), 531–542.

    Article  PubMed  Google Scholar 

  27. Lindquist, M.A. (2008). The statistical analysis of fmri data. Statistical Science, 23(4), 439–464.

    Article  Google Scholar 

  28. Mandelbrot, B.B. (1977). The fractal geometry of nature. New York: Springer.

    Google Scholar 

  29. Onal, I., Ozay, M., & Yarman Vural, F. (2015a). Modeling voxel connectivity for brain decoding. In PRNI, Stanford, CA, USA (pp. 5–8).

  30. Onal, I., Ozay, M., & Yarman Vural, F.T. (2015b). Functional mesh model with temporal measurements for brain decoding. In Engineering in medicine and biology society (EMBC), 2015 37th annual international conference of the IEEE (pp. 2624–2628): IEEE.

  31. Ozay, M., & Yarman-Vural, F.T. (2016). Hierarchical distance learning by stacking nearest neighbor classifiers. Information Fusion, 29, 14–31.

    Article  Google Scholar 

  32. Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., & Ville, D.V.D. (2011). Decoding brain states from fmri connectivity graphs. NeuroImage, 56(2), 616–626. Multivariate Decoding and Brain Reading.

    Article  PubMed  Google Scholar 

  33. Richiardi, J., Achard, S., Bunke, H., & De Ville, D.V. (2013). Machine learning with brain graphs: predictive modeling approaches for functional imaging in systems neuroscience. IEEE Signal Processing Magazine.

  34. Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., & Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex, 22(1), 158–165.

    Article  PubMed  Google Scholar 

  35. Smith, R., Keramatian, K., & Christoff, K. (2007). Localizing the rostrolateral prefrontal cortex at the individual level. NeuroImage, 36(4), 1387–1396.

    Article  PubMed  Google Scholar 

  36. Thompson, W.H., & Fransson, P. (2015). The frequency dimension of fmri dynamic connectivity: network connectivity, functional hubs and integration in the resting brain. NeuroImage, 121, 227–242.

    Article  PubMed  Google Scholar 

  37. Van De Ville, D., Blu, T., & Unser, M. (2006). Surfing the brain—an overview of wavelet-based techniques for fMRI data analysis. IEEE Engineering in Medicine and Biology Magazine, 25(2), 65–78.

    Article  Google Scholar 

  38. Wheatley, T., Milleville, S.C., & Martin, A. (2007). Understanding animate agents: distinct roles for the social network and mirror system. Psychological Science, 18(6), 469–474.

    Article  PubMed  Google Scholar 

  39. Xu, Z., & Chan, A.K. (2002). Encoding with frames in mri and analysis of the signal-to-noise ratio. IEEE Transactions on Medical Imaging, 21(4), 332–342.

    Article  PubMed  Google Scholar 

Download references

Funding

Itir Onal Ertugrul is funded by TUBITAK 2211E. This work was supported by CREST, JST, and TUBITAK Project No 116E091.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Itir Onal Ertugrul.

Ethics declarations

Conflict of interests

Itir Onal Ertugrul, Mete Ozay and Fatos Yarman Vural declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected and archived in a data bank.

Additional information

This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Onal Ertugrul, I., Ozay, M. & Yarman Vural, F.T. Hierarchical multi-resolution mesh networks for brain decoding. Brain Imaging and Behavior 12, 1067–1083 (2018). https://doi.org/10.1007/s11682-017-9774-z

Download citation

Keywords

  • fMRI data analysis
  • Brain decoding
  • Wavelet decomposition
  • Mesh networks
  • Hierarchical models
  • Ensemble models