Abstract
Multimodal neuroimage can provide complementary information about the dementia, but small size of complete multimodal data limits the ability in representation learning. Moreover, the data distribution inconsistency from different modalities may lead to ineffective fusion, which fails to sufficiently explore the intra-modal and inter-modal interactions and compromises the disease diagnosis performance. To solve these problems, we proposed a novel multimodal representation learning and adversarial hypergraph fusion (MRL-AHF) framework for Alzheimer’s disease diagnosis using complete trimodal images. First, adversarial strategy and pre-trained model are incorporated into the MRL to extract latent representations from multimodal data. Then two hypergraphs are constructed from the latent representations and the adversarial network based on graph convolution is employed to narrow the distribution difference of hyperedge features. Finally, the hyperedge-invariant features are fused for disease prediction by hyperedge convolution. Experiments on the public Alzheimer’s Disease Neuroimaging Initiative(ADNI) database demonstrate that our model achieves superior performance on Alzheimer’s disease detection compared with other related models and provides a possible way to understand the underlying mechanisms of disorder’s progression by analyzing the abnormal brain connections.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alzheimer’s Association: 2019 Alzheimer’s disease facts and figures. Alzheimer’s Dementia. 15(3), 321–387 (2019)
Li, Y., Liu, J., Tang, Z., et al.: Deep spatial-temporal feature fusion from adaptive dynamic functional connectivity for MCI identification. IEEE Trans. Med. Imaging 39(9), 2818–2830 (2020)
Wang, S., Shen, Y., Shi, C., et al.: Skeletal maturity recognition using a fully automated system with convolutional neural networks. IEEE Access 6, 29979–29993 (2018)
Wang, S., Shen, Y., Zeng, D., et al.: Bone age assessment using convolutional neural networks. In: 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 175–178 (2018)
Wang, S., Hu, Y., Shen, Y., et al.: Classification of diffusion tensor metrics for the diagnosis of a myelopathic cord using machine learning. Int. J. Neural Syst. 28(02), 1750036 (2018)
Lei, B., Xia, Z., Jiang, F., et al.: Skin lesion segmentation via generative adversarial networks with dual discriminators. Med. Image Analy. 64, 101716 (2020)
Wang, S., Wang, X., Shen, Y., et al.: An ensemble-based densely-connected deep learning system for assessment of skeletal maturity. IEEE Trans. Syst. Man Cybern. Syst. (2020)
Zeng, D., Wang, S., Shen, Y., et al.: A GA-based feature selection and parameter optimization for support tucker machine. Procedia Comput. Sci. 111, 17–23 (2017)
Wu, K., Shen, Y., Wang, S.: 3D convolutional neural network for regional precipitation nowcasting. J. Image Signal Process. 7(4), 200–212 (2018)
Franzmeier, N., Dyrba, M.: Functional brain network architecture may route progression of Alzheimer’s disease pathology. Brain 140(12), 3077–3080 (2017)
Pereira, J.B., Van Westen, D., Stomrud, E., et al.: Abnormal structural brain connectome in individuals with preclinical Alzheimer’s disease. Cereb. Cortex 28(10), 3638–3649 (2018)
Schuff, N., Woerner, N., Boreta, L., et al.: MRI of hippocampal volume loss in early Alzheimer’s disease in relation to ApoE genotype and biomarkers. Brain 132(4), 1067–1077 (2009)
Huang, J., Zhou, L., Wang, L., et al.: Attention-diffusion-bilinear neural network for brain network analysis. IEEE Trans. Med. Imaging 39(7), 2541–2552 (2020)
Xing, X., Li, Q., Wei, H., et al.: Dynamic spectral graph convolution networks with assistant task training for early mci diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 639–646 (2019)
Yu, S., et al.: Multi-scale enhanced graph convolutional network for early mild cognitive impairment detection. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12267, pp. 228–237. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59728-3_23
Li, Y., Liu, J., Tang, Z., et al.: Graph convolution network with similarity awareness and adaptive calibration for disease-induced deterioration prediction. Medical Image Analysis. 69, 101947 (2021)
Yu, W., Lei, B., Michael, K., et al.: Tensorizing GAN with high-order pooling for Alzheimer’s disease assessment. IEEE Trans. Neural Netw. Learn. Syst. (2021). https://doi.org/10.1109/TNNLS.2021.3063516
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)
Mo, L.F., Wang, S.Q.: A variational approach to nonlinear two-point boundary value problems. Nonlinear Anal. Theory Methods Appl. 71(12), e834–e838 (2009)
Wang, S.Q.: A variational approach to nonlinear two-point boundary value problems. Comput. Math. Appl. 58(11–12), 2452–2455 (2009)
Wang, S.Q., He, J.H.: Variational iteration method for a nonlinear reaction-diffusion process. Int. J. Chem. Reactor Eng. 6(1) (2008)
Wang S.Q., Wang X., Hu Y., et al.: Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision. IEEE Trans. Autom. Sci. Eng. 18, 574–585 (2020)
Hu, S., Shen, Y., Wang, S., Lei, B.: Brain MR to PET synthesis via bidirectional generative adversarial network. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 698–707. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_67
Hu, S., Shen, Y., Wang, S., Lei, B.: Brain MR to PET synthesis via bidirectional generative adversarial network. In: MartelMartel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 698–707. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_67
Hu, S., Yu, W., Chen, Z., et al.: Medical image reconstruction using generative adversarial network for Alzheimer disease assessment with class-imbalance problem. In 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp. 1323–1327 (2020)
Dai, Q., Li, Q., Tang, J., et al.: Adversarial network embedding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32(1) (2018)
Pan, S., Hu, R., Long, G., et al.: Adversarially regularized graph autoencoder for graph embedding. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2609–2615 (2018)
Wen, J., Thibeau-Sutre, E., Diaz-Melo, M., et al.: Convolutional neural networks for classification of Alzheimer’s disease: overview and reproducible evaluation. Med. Image Anal. 63, 101694 (2020)
Wang, H., Shen, Y., Wang, S., et al.: Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer’s disease. Neurocomputing 333, 145–156 (2019)
Wang, S., Shen, Y., Chen, W., et al.: Automatic recognition of mild cognitive impairment from MRI images using expedited convolutional neural networks. In: International Conference on Artificial Neural Networks, pp. 373–380 (2017)
Wang, S., Wang, H., Shen, Y., et al.: Automatic recognition of mild cognitive impairment and Alzheimers disease using ensemble based 3D densely connected convolutional networks. In: 2018 17th IEEE International Conference on Machine Learning and Applications, pp. 517–523 (2018)
Hu, S., Yuan, J., Wang, S., et al.: Cross-modality synthesis from MRI to PET using adversarial U-net with different normalization. In 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), pp. 1–5 (2019)
Lei, B., Yang, M., Yang, P., et al.: Deep and joint learning of longitudinal data for Alzheimer’s disease prediction. Patt. Recogn. 102, 107247 (2020)
Wang, S., Wang, H., Cheung, A.C., et al.: Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer’s disease. Deep Learn. Appl. 1098, 53 (2020)
Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2018)
Makhzani, A., Shlens, J., Jaitly, N., et al.: Adversarial autoencoders. arXiv preprint arXiv:1511.05644. (2015)
Li, Y., Liu, J., Gao, X., et al.: Multimodal hyper-connectivity of functional networks using functionally-weighted LASSO for MCI classification. Med. Image Anal. 52, 80–96 (2019)
Feng, Y., You, H., Zhang, Z., et al.: Hypergraph neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3558–3565 (2019)
Kulesza, A., Taskar, B.: Fixed-size determinantal point processes. In: Proceedings of the 28th International Conference on Machine learning, pp. 1193–1200 (2011)
Suykens, J.A.K., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)
Atwood, J., Towsley, D.: Diffusion-convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1993–2001 (2016)
Zhu, Q., Yuan, N., Huang, J., et al.: Multi-modal AD classification via self-paced latent correlation analysis. Neurocomputing 255, 143–154 (2019)
Montembeault, M., Rouleau, I., Provost, J.S., et al.: Altered gray matter structural covariance networks in early stages of Alzheimer’s disease. Cereb. Cortex 26(6), 2650–2662 (2016)
Sun, Y., Dai, Z., Li, Y., et al.: Subjective cognitive decline: mapping functional and structural brain changes–a combined resting-state functional and structural MR imaging study. Radiology 281(1), 185–192 (2016)
Jin, D., Wang, P., Zalesky, A., et al.: Grab- AD: generalizability and reproducibility of altered brain activity and diagnostic classification in Alzheimer’s Disease. Hum. Brain Mapp. 41(12), 3379–3391 (2020)
Acknowledgment
This work was supported by the National Natural Science Foundations of China under Grant 61872351, the International Science and Technology Cooperation Projects of Guangdong under Grant 2019A050510030, the Distinguished Young Scholars Fund of Guangdong under Grant 2021B1515020019, the Excellent Young Scholars of Shenzhen under Grant RCYX20200714114641211 and Shenzhen Key Basic Research Project under Grant JCYJ20200109115641762.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zuo, Q., Lei, B., Shen, Y., Liu, Y., Feng, Z., Wang, S. (2021). Multimodal Representations Learning and Adversarial Hypergraph Fusion for Early Alzheimer’s Disease Prediction. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13021. Springer, Cham. https://doi.org/10.1007/978-3-030-88010-1_40
Download citation
DOI: https://doi.org/10.1007/978-3-030-88010-1_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88009-5
Online ISBN: 978-3-030-88010-1
eBook Packages: Computer ScienceComputer Science (R0)