Abstract
Applying network science approaches to investigate the functions and anatomy of the human brain is prevalent in modern medical imaging analysis. Due to the complex network topology, for an individual brain, mining a discriminative network representation from the multimodal brain networks is non-trivial. The recent success of deep learning techniques on graph-structured data suggests a new way to model the non-linear cross-modality relationship. However, current deep brain network methods either ignore the intrinsic graph topology or require a network basis shared within a group. To address these challenges, we propose a novel end-to-end deep graph representation learning (Deep Multimodal Brain Networks - DMBN) to fuse multimodal brain networks. Specifically, we decipher the cross-modality relationship through a graph encoding and decoding process. The higher-order network mappings from brain structural networks to functional networks are learned in the node domain. The learned network representation is a set of node features that are informative to induce brain saliency maps in a supervised manner. We test our framework in both synthetic and real image data. The experimental results show the superiority of the proposed method over some other state-of-the-art deep brain network models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arslan, S., Ktena, S.I., Glocker, B., Rueckert, D.: Graph saliency maps through spectral convolutional networks: application to sex classification with brain connectivity. arXiv preprint arXiv:1806.01764 (2018)
Avena-Koenigsberger, A., Misic, B., Sporns, O.: Communication dynamics in complex brain networks. Nature Rev. Neurosci. 19(1), 17–33 (2018)
Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)
Bullmore, E., Sporns, O.: The economy of brain network organization. Nat. Rev. Neurosci. 13(5), 336 (2012)
Camicioli, R., Moore, M.M., Kinney, A., Corbridge, E., Glassberg, K., Kaye, J.A.: Parkinson’s disease is associated with hippocampal atrophy. Mov. Disord. 18(7), 784–790 (2003)
Cao, B., et al.: t-BNE: tensor-based brain network embedding. In: Proceedings of SIAM International Conference on Data Mining (SDM) (2017)
Deshpande, G., Wang, P., Rangaprakash, D., Wilamowski, B.: Fully connected cascade artificial neural network architecture for attention deficit hyperactivity disorder classification from functional magnetic resonance imaging data. IEEE Trans. Cybern. 45, 2668–2679 (2015)
Dyrba, M., Grothe, M., Kirste, T., Teipel, S.J.: Multimodal analysis of functional and structural disconnection in a Lzheimer’s disease using multiple kernel svm. Hum. Brain Mapp. 36(6), 2118–2131 (2015)
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: NIPS (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Honey, C., Sporns, O., Cammoun, L., Gigandet, X., Thiran, J.P., Meuli, R., Hagmann, P.: Predicting human resting-state functional connectivity from structural connectivity. Proc. Natl. Acad. Sci. 106(6), 2035–2040 (2009)
Jao, T., et al.: Functional brain network changes associated with clinical and biochemical measures of the severity of he patic encephalopathy. Neuroimage 122, 332–344 (2015)
Jenkinson, M., Beckmann, C.F., Behrens, T.E., Woolrich, M.W., Smith, S.M.: Fsl. Neuroimage 62(2), 782–790 (2012)
Kawahara, J., et al.: Brainnetcnn: convolutional neural networks for brain networks; towards predicting neurodevelopment. NeuroImage 146, 1038–1049 (2017)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)
Ktena, S.I., et al.: Metric learning with spectral graph convolutions on brain connectivity networks. NeuroImage 169, 431–442 (2018)
Li, G., Muller, M., Thabet, A., Ghanem, B.: DeepGCNs: Can GCNs go as deep as CNNs? In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9267–9276 (2019)
Marek, K., Jennings, D., Lasch, S., Siderowf, A., Tanner, C., Simuni, T., Coffey, C., Kieburtz, K., Flagg, E., Chowdhury, S., et al.: The parkinson progression marker initiative (PPMI). Prog. Neurobiol. 95(4), 629–635 (2011)
Obeso, J.A., Rodriguez-Oroz, M.C., Rodriguez, M., Lanciego, J.L., Artieda, J., Gonzalo, N., Olanow, C.W.: Pathophysiology of the basal ganglia in parkinson’s disease. Trends Neurosci. 23, S8–S19 (2000)
Osmanlıoğlu, Y., Tunç, B., Parker, D., Elliott, M.A., Baum, G.L., Ciric, R., Satterthwaite, T.D., Gur, R.E., Gur, R.C., Verma, R.: System-level matching of structural and functional connectomes in the human brain. NeuroImage 199, 93–104 (2019)
Plis, S.M., et al.: Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia. NeuroImage 181, 734–747 (2018)
Repovs, G., Csernansky, J.G., Barch, D.M.: Brain network connectivity in individuals with schizophrenia and their siblings. Biological psychiatry (2011)
Rijpkema, M., Everaerd, D., van der Pol, C., Franke, B., Tendolkar, I., Fernández, G.: Normal sexual dimorphism in the human basal ganglia. Hum. Brain Mapping 33(5), 1246–1252 (2012)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Ruigrok, A.N., et al.: A meta-analysis of sex differences in human brain structure. Neurosci. Biobehav. Rev. 39, 34–50 (2014)
Schwarz, A.J., McGonigle, J.: Negative edges and soft thresholding in complex network analysis of resting state functional connectivity data. Neuroimage 55(3), 1132–1146 (2011)
Stam, C., et al.: The relation between structural and functional connectivity patterns in complex brain networks. Int. J. Psychophysiol. 103, 149–160 (2016)
Sui, J., Pearlson, G., Caprihan, A., Adali, T., Kiehl, K.A., Liu, J., Yamamoto, J., Calhoun, V.D.: Discriminating schizophrenia and bipolar disorder by fusing FMRI and DTI in a multimodal CCA+ joint ICA model. Neuroimage 57(3), 839–855 (2011)
Suk, H.-I., Shen, D.: Deep learning-based feature representation for AD/MCI classification. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8150, pp. 583–590. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40763-5_72
Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K.: The WU-Minn human connectome project: an overview. Neuroimage 80, 62–79 (2013)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
Wang, D., Cui, P., Zhu, W.: Structural deep network embedding. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1225–1234. ACM (2016)
Wang, S., He, L., Cao, B., Lu, C.T., Yu, P.S., Ragin, A.B.: Structural deep brain network mining. In: ACM SIGKDD. ACM (2017)
Whitfield-Gabrieli, S., Nieto-Castanon, A.: Conn: a functional connectivity toolbox for correlated and anticorrelated brain networks. Brain Connectivity 2(3), 125–141 (2012)
Zhang, W., Shu, K., Wang, S., Liu, H., Wang, Y.: Multimodal fusion of brain networks with longitudinal couplings. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 3–11. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00931-1_1
Acknowledgments
This work was supported in part by NIH (RF1AG051710 and R01EB025032). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, W., Zhan, L., Thompson, P., Wang, Y. (2020). Deep Representation Learning for Multimodal Brain Networks. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12267. Springer, Cham. https://doi.org/10.1007/978-3-030-59728-3_60
Download citation
DOI: https://doi.org/10.1007/978-3-030-59728-3_60
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59727-6
Online ISBN: 978-3-030-59728-3
eBook Packages: Computer ScienceComputer Science (R0)
