Abstract
Benefiting from complementary information, multimodal brain imaging analysis has distinct advantages over single-modal methods for the diagnosis of neurodegenerative diseases such as Alzheimer’s disease. However, multi-modal brain images are often incomplete with missing data in clinical practice due to various issues such as motion, medical costs, and scanner availability. Most existing methods attempted to build machine learning models to directly estimate the missing images. However, since brain images are of high dimension, accurate and efficient estimation of missing data is quite challenging, and not all voxels in the brain images are associated with the disease. In this paper, we propose a multimodal feature-based transformer to impute multimodal brain features with missing data for the diagnosis of neurodegenerative disease. The proposed method consists of a feature regression subnetwork and a multimodal fusion subnetwork based on transformer, for completion of the features of missing data and also multimodal diagnosis of disease. Different from previous methods for the generation of missing images, our method imputes high-level and disease-related features for multimodal classification. Experiments on the ADNI database with 1,364 subjects show better performance of our method over the state-of-the-art methods in disease diagnosis with missing multimodal data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, D., Wang, Y., Zhou, L., Yuan, H., Shen, D.: Multimodal classification of Alzheimer’s disease and mild cognitive impairment. Neuroimage 55(3), 856–867 (2011)
Liu, M., Cheng, D., Wang, K., Wang, Y.: Multi-modality cascaded convolutional neural networks for Alzheimer’s disease diagnosis. Neuroinformatics 16(3–4), 295–308 (2018)
Cheng, B., Liu, M., Zhang, D., Shen, D.: Robust multi-label transfer feature learning for early diagnosis of Alzheimer’s disease. Brain Imaging Behav. 13(1), 138–153 (2019)
Zhou, T., Thung, K., Zhu, X., Shen, D.: Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Hum. Brain Mapp. 40(3), 1001–1016 (2019)
Xiang, S., Yuan, L., Fan, W., Wang, Y., Thompson, P.M., Ye, J.: Bi-level multi-source learning for heterogeneous block-wise missing data. Neuroimage 102, 192–206 (2014)
Liu, M., Gao, Y., Yap, P.-T., Shen, D.: Multi-hypergraph learning for incomplete multimodality data. IEEE J. Biomed. Health Inform. 22(4), 1197–1208 (2018)
Nie, D., Cao, X., Gao, Y., Wang, L., Shen, D.: Estimating CT image from MRI Data Using 3D Fully convolutional networks. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 170–178. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46976-8_18
Huynh, T., et al.: Estimating CT image From MRI data using structured random forest and auto-context model. IEEE Trans. Med. Imaging 35(1), 174–183 (2016)
Pan, Y., Liu, M., Lian, C., Xia, Y., Shen, D.: Disease-image specific generative adversarial network for brain disease diagnosis with incomplete multi-modal neuroimages. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 137–145. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_16
Liu, Y., et al.: Incomplete multi-modal representation learning for Alzheimer’s disease diagnosis. Med. Image Anal. 69, 101953 (2021)
Song, J., Zheng, J., Li, P., Lu, X., Zhu, G., Shen, P.: An effective multimodal image fusion method Using MRI and PET for Alzheimer’s disease diagnosis. Front. Digit. Health 3, 637386 (2021)
Zhang, T., Shi, M.: Multi-modal neuroimaging feature fusion for diagnosis of Alzheimer’s disease. J. Neurosci. Methods 341, 108795 (2020)
Hao, X., et al.: Multi-modal neuroimaging feature selection with consistent metric constraint for diagnosis of Alzheimer’s disease. Med. Image Anal. 60, 101625 (2020)
Pan, Y., Liu, M., Lian, C., Zhou, T., Xia, Y., Shen, D.: Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for alzheimer’s disease Diagnosis. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 455–463. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00931-1_52
Gao, X., Shi, F., Shen, D., Liu, M.: Task-induced pyramid and attention GAN for multimodal brain image imputation and classification in Alzheimer’s disease. IEEE J. Biomed. Health Inform. 26(1), 36–43 (2022)
Wee, C.-Y., Yap, P.-T., Zhang, D., Wang, L., Shen, D.: Group-constrained sparse fMRI connectivity modeling for mild cognitive impairment identification. Brain Struct. Funct. 219(2), 641–656 (2014)
Liu, M., Zhang, J., Nie, D., Yap, P.-T., Shen, D.: Anatomical landmark based deep feature representation for MR images in brain disease diagnosis. IEEE J. Biomed. Health Inform. 22(5), 1476–1485 (2018)
Acknowledgements
This study was supported in part by the National Natural Science Foundation of China (No.62171283), Natural Science Foundation of Shanghai (20ZR1426300), in part by the National Key R &D Program of China (National key research and development program of China) under Grant (2022YFC2503302/2022YFC2503305), by Shanghai Jiao Tong University Scientific and Technological Innovation Funds (No.2019QYB02), CAAI-Huawei MindSpore Open Fund, Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), the Fundamental Research Funds for the Central Universities, Supported by Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence (LCNBI) and ZJLab.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gao, X., Shi, F., Shen, D., Liu, M. (2023). Feature-Based Transformer with Incomplete Multimodal Brain Images for Diagnosis of Neurodegenerative Diseases. In: Rekik, I., Adeli, E., Park, S.H., Cintas, C., Zamzmi, G. (eds) Predictive Intelligence in Medicine. PRIME 2023. Lecture Notes in Computer Science, vol 14277. Springer, Cham. https://doi.org/10.1007/978-3-031-46005-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-46005-0_3
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46004-3
Online ISBN: 978-3-031-46005-0
eBook Packages: Computer ScienceComputer Science (R0)
