Abstract
Diffusion MRI tractography is an advanced imaging technique for quantitative mapping of the brain’s structural connectivity. Whole brain tractography (WBT) data contains over hundreds of thousands of individual fiber streamlines (estimated brain connections), and this data is usually parcellated to create compact representations for data analysis applications such as disease classification. In this paper, we propose a novel parcellation-free WBT analysis framework, TractoFormer, that leverages tractography information at the level of individual fiber streamlines and provides a natural mechanism for interpretation of results using the attention mechanism of transformers. TractoFormer includes two main contributions. First, we propose a novel and simple 2D image representation of WBT, TractoEmbedding, to encode 3D fiber spatial relationships and any feature of interest that can be computed from individual fibers (such as FA or MD). Second, we design a network based on vision transformers (ViTs) that includes: 1) data augmentation to overcome model overfitting on small datasets, 2) identification of discriminative fibers for interpretation of results, and 3) ensemble learning to leverage fiber information from different brain regions. In a synthetic data experiment, TractoFormer successfully identifies discriminative fibers with simulated group differences. In a disease classification experiment comparing several methods, TractoFormer achieves the highest accuracy in classifying schizophrenia vs control. Discriminative fibers are identified in left hemispheric frontal and parietal superficial white matter regions, which have previously been shown to be affected in schizophrenia patients.
This work is supported by the following NIH grants: R01MH119222, R01MH125860, P41EB015902, R01MH074794.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
While embeddings from the first 3 dimensions can be used to generate 3D TractoEmbedding images, our unpublished results show that this decreases group classification performance potentially due to the data sparsity where many voxels on the 3D grid do not have any mapped fibers.
- 2.
Following instructions from: https://github.com/jeonsworld/ViT-pytorch.
- 3.
References
Basser, P.J., Pajevic, S., Pierpaoli, C., Duda, J., Aldroubi, A.: In vivo fiber tractography using DT-MRI data. Magn. Reson. Med. 44, 625–632 (2000)
Zhang, F., Daducci, A., He, Y., et al.: Quantitative mapping of the brain’s structural connectivity using diffusion MRI tractography: a review. Neuroimage 249, 118870 (2022)
Zhan, L., et al.: Comparison of nine tractography algorithms for detecting abnormal structural brain networks in Alzheimer’s disease. Front. Aging Neurosci. (2015). https://doi.org/10.3389/fnagi.2015.00048
Deng, Y., et al.: Tractography-based classification in distinguishing patients with first-episode schizophrenia from healthy individuals. Prog. Neuropsychopharmacol. Biol. Psychiatry 88, 66–73 (2019)
Hu, M., X., et al.: Structural and diffusion MRI based schizophrenia classification using 2D pretrained and 3D naive Convolutional Neural Networks. Schizophr. Res. (2021). https://doi.org/10.1016/j.schres.2021.06.011
Brown, C.J., Hamarneh, G.: Machine Learning on human connectome data from MRI. arXiv [cs.LG] (2016). http://arxiv.org/abs/1611.08699
Mansour, L.S., Tian, Y., Yeo, B.T.T., Cropley, V., Zalesky, A.: High-resolution connectomic fingerprints: mapping neural identity and behavior. Neuroimage 229, 117695 (2021)
Cole, M., Murray, K., et al.: Surface-Based Connectivity Integration: an atlas-free approach to jointly study functional and structural connectivity. Hum. Brain Mapp. 42, 3481–3499 (2021)
Barile, B., Marzullo, A., Stamile, C., Durand-Dubief, F., Sappey-Marinier, D.: Data augmentation using generative adversarial neural networks on brain structural connectivity in multiple sclerosis. Comput. Methods Programs Biomed. 206, 106113 (2021)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Dosovitskiy, A., Beyer, L., et al.: An image is worth 16 × 16 words: transformers for image recognition at scale. In: ICLR (2021)
Han, K., et al.: A survey on vision transformer. IEEE Trans. Pattern. Anal. Mach. Intell. (2022). https://doi.org/10.1109/TPAMI.2022.3152247
Steiner, A., Kolesnikov, A., et al.: How to train your ViT? Data, augmentation, and regularization in vision transformers. TMLR (2022)
Hofmann, S.M., Beyer, F., Lapuschkin, S., et al.: Towards the interpretability of deep learning models for human neuroimaging. bioRxiv, p. 2021.06.25.449906 (2021)
Zhang, Q.-S., Zhu, S.-C.: Visual interpretability for deep learning: a survey. Front. Inf. Technol. Electron. Eng. 19(1), 27–39 (2018). https://doi.org/10.1631/FITEE.1700808
Lombardi, A., Diacono, D., Amoroso, N., et al.: Explainable deep learning for personalized age prediction with brain morphology. Front. Neurosci. 15, 674055 (2021)
Chefer, H., Gur, S., Wolf, L.: Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In: ICCV, pp. 397–406 (2021)
Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. In: ACL 2020. pp. 4190–4197 (2020)
Van Essen, D.C., et al.: The WU-Minn Human Connectome Project: an overview. Neuroimage 80, 62–79 (2013)
Poldrack, R.A., et al.: A phenome-wide examination of neural and cognitive function. Sci. Data 3, 160110 (2016)
Malcolm, J.G., Shenton, M.E., Rathi, Y.: Filtered multitensor tractography. IEEE Trans. Med. Imaging 29, 1664–1675 (2010)
Reddy, C.P., Rathi, Y.: Joint multi-fiber NODDI parameter estimation and tractography using the unscented information filter. Front. Neurosci. 10, 166 (2016)
Norton, I., Essayed, W.I., Zhang, F., Pujol, S., et al.: SlicerDMRI: open source diffusion MRI software for brain cancer research. Cancer Res. 77, e101–e103 (2017)
Zhang, F., Noh, T., et al.: SlicerDMRI: diffusion MRI and tractography research software for brain cancer surgery planning and visualization. JCO Clin. Can. Inform. 4, 299–309 (2020)
Zhang, F., Savadjiev, P., Cai, W., et al.: Whole brain white matter connectivity analysis using machine learning: an application to autism. Neuroimage 172, 826–837 (2018)
Hamoda, H.M., et al.: Abnormalities in thalamo-cortical connections in patients with first-episode schizophrenia: a two-tensor tractography study. Brain Imaging Behav. 13(2), 472–481 (2018). https://doi.org/10.1007/s11682-018-9862-8
Zhang, F., Wu, Y., et al.: An anatomically curated fiber clustering white matter atlas for consistent white matter tract parcellation across the lifespan. Neuroimage 179, 429–447 (2018)
O’Donnell, L.J., Wells, W.M., Golby, A.J., Westin, C.-F.: Unbiased groupwise registration of white matter tractography. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7512, pp. 123–130. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33454-2_16
Vercruysse, D., Christiaens, D., Maes, F., Sunaert, S., Suetens, P.: Fiber bundle segmentation using spectral embedding and supervised learning. In: CDMRI, pp. 103–114 (2014)
O’Donnell, L.J., Westin, C.-F.: Automatic tractography segmentation using a high-dimensional white matter atlas. IEEE Trans. Med. Imaging. 26, 1562–1575 (2007)
Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the Nyström method. IEEE Trans. Pattern Anal. Mach. Intell. 26, 214–225 (2004)
Moberts, B., Vilanova, A., van Wijk, J.J.: Evaluation of fiber clustering methods for diffusion tensor imaging. In: IEEE Conference on Visualization, pp. 65–72 (2005)
Jianu, R., Demiralp, C., Laidlaw, D.H.: Exploring 3D DTI fiber tracts with linked 2D representations. IEEE Trans. Vis. Comput. Graph. 15, 1449–1456 (2009)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
Zhang, F., Wu, W., Ning, L., et al.: Suprathreshold fiber cluster statistics: Leveraging white matter geometry to enhance tractography statistical analysis. Neuroimage 171, 341–354 (2018)
Smith, S.M., Nichols, T.E.: Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference. Neuroimage 44, 83–98 (2009)
Kelly, S., et al.: Widespread white matter microstructural differences in schizophrenia across 4322 individuals: results from the ENIGMA Schizophrenia DTI Working Group. Mol. Psychiatry 23, 1261–1269 (2018)
He, H., Zhang, F., et al.: Model and predict age and sex in healthy subjects using brain white matter features: a deep learning approach. In: ISBI, pp. 1–5 (2022)
Calamante, F., Tournier, D., et al.: Track-density imaging (TDI): super-resolution white matter imaging using whole-brain track-density mapping. Neuroimage 53(4), 1233–1243 (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR, pp. 2921–2929 (2016)
Nazeri, A., Chakravarty, M., et al.: Alterations of superficial white matter in schizophrenia and relationship to cognitive performance. Neuropsychopharmacology 38, 1954–1962 (2013)
Makris, N., Seidman, L.J., Ahern, T., Kennedy, D.N., et al.: White matter volume abnormalities and associations with symptomatology in schizophrenia. Psychiatry Res. 183, 21–29 (2010)
Ji, E., Guevara, P., et al.: Increased and decreased superficial white matter structural connectivity in schizophrenia and bipolar disorder. Schizophr. Bull. 45, 1367–1378 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, F., Xue, T., Cai, W., Rathi, Y., Westin, CF., O’Donnell, L.J. (2022). TractoFormer: A Novel Fiber-Level Whole Brain Tractography Analysis Framework Using Spectral Embedding and Vision Transformers. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13431. Springer, Cham. https://doi.org/10.1007/978-3-031-16431-6_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-16431-6_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16430-9
Online ISBN: 978-3-031-16431-6
eBook Packages: Computer ScienceComputer Science (R0)
