Skip to main content

Advertisement

SpringerLink
Log in

Pairwise feature-based generative adversarial network for incomplete multi-modal Alzheimer’s disease diagnosis

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Magnetic resonance imaging (MRI) and positron emission tomography (PET) are widely used in diagnosis of Alzheimer’s disease (AD). In practice, incomplete modality problem is unavoidable due to the cost of data acquisition. Deep learning based models especially generative adversarial networks (GAN) are usually adopted to impute missing images. However, there are still some problems: (1) there are many regions unrelated to the disease and have little significance in the actual diagnosis in brain images, which are very cumbersome to generate. (2) The image generated by GAN would introduce noises causing the poor performance in the diagnostic model. To address these problems, a pairwise feature-based generation adversarial network is proposed. Specifically, features from the original brain images are extracted firstly. For the paired data without modality loss, the extracted MRI features are used as input to generate its corresponding PET features, which not only reduces the scale of the model, but also ensures the direct correlation between the generated features and the diagnosis. In addition, the available real PET features of the paired samples are added as label to constrain the generated ones. Finally, the attention mechanism is adopted in both the generator and discriminator, which can effectively retain the structural information of the feature itself. A large number of experiments have demonstrated that our proposed method has achieved promising results in the diagnosis of AD.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Weller, J., Budson, A.: Current understanding of Alzheimer’s disease diagnosis and treatment. F1000Res 7, 1161 (2018)

    Article  Google Scholar 

  2. Kirova, A.-M., Bays, R.B., Lagalwar, S.: Working memory and executive function decline across normal aging, mild cognitive impairment, and Alzheimer’s disease. Biomed. Res. Int. 2015, 1–9 (2015)

    Article  Google Scholar 

  3. Guarino, A., Favieri, F., Boncompagni, I., Agostini, F., Cantone, M., Casagrande, M.: Executive functions in Alzheimer disease: a systematic review. Front. Aging Neurosci. 10, 437 (2019)

    Article  Google Scholar 

  4. Kitamura, Y., Usami, R., Ichihara, S., Kida, H., Satoh, M., Tomimoto, H., Murata, M., Oikawa, S.: Plasma protein profiling for potential biomarkers in the early diagnosis of Alzheimer’s disease. Neurol. Res. 39(3), 231–238 (2017)

    Article  Google Scholar 

  5. Cummings, J., Lee, G., Ritter, A., Sabbagh, M., Zhong, K.: “Alzheimer’s disease drug development pipeline: 2019”, Alzheimer’s \& Dement. Transl. Res. Clin. Interv. 5, 272–293 (2019)

    Article  Google Scholar 

  6. Zhang, D., Wang, Y., Zhou, L., Yuan, H., Shen, D., Alzheimer's Disease Neuroimaging Initiative et al.: Multimodal classification of Alzheimer’s disease and mild cognitive impairment, Neuroimage, 55(3), 856–867, (2011)

  7. Lin, E., Lin, C.-H., Lane, H.-Y.: Deep learning with neuroimaging and genomics in Alzheimer’s disease. Int. J. Mol. Sci. 22(15), 7911 (2021)

    Article  Google Scholar 

  8. Zhang, D., Shen, D., Alzheimer's Disease Neuroimaging Initiative et al.: Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease. Neuroimage, 59(2): 895–907, (2012).

  9. Ieracitano, C., Mammone, N., Hussain, A., Morabito, F.C.: A novel multi-modal machine learning based approach for automatic classification of EEG recordings in dementia. Neural Netw. 123, 176–190 (2020)

    Article  Google Scholar 

  10. Zhou, T., Thung, K.-H., Liu, M., Shi, F., Zhang, C., Shen, D.: Multi-modal latent space inducing ensemble SVM classifier for early dementia diagnosis with neuroimaging data. Med. Image Anal. 60, 101630 (2020)

    Article  Google Scholar 

  11. Basavegowda, H.S., Dagnew, G.: Deep learning approach for microarray cancer data classification. CAAI Trans. Intell. Technol. 5(1), 22–33 (2020)

    Article  Google Scholar 

  12. Shao, W., He, L., Philip, S. Y.: Multiple incomplete views clustering via weighted nonnegative matrix factorization with $$ L_ {2, 1} $$ regularization, In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 318–334 (2015)

  13. Wen, J., Zhang, Z., Xu, Y., Zhang, B., Fei, L., Liu, H.: Unified embedding alignment with missing views inferring for incomplete multi-view clustering. Proc. AAAI Conf. Artif. Intell. 33(01), 5393–5400 (2019)

    Google Scholar 

  14. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks, (2014) arXiv Prepr. arXiv: 1406.2661.

  15. Zhu, C., Yan, W., Cai, X., Liu, S., Li, T.H., Li, G.: Neural saliency algorithm guide bi-directional visual perception style transfer. CAAI Trans. Intell. Technol. 5(1), 1–8 (2020)

    Article  Google Scholar 

  16. Pan, Y., Liu, M., Lian, C., Zhou, T., Xia, Y., Shen,D.: Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer’s disease diagnosis, In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 455–463 (2018).

  17. Pan, Y., Liu, M., Lian, C., Xia, Y., Shen, D.: Disease-image specific generative adversarial network for brain disease diagnosis with incomplete multi-modal neuroimages, In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 137–145 (2019).

  18. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. Adv. Neural Inf. Process. Syst. 29, 2234–2242 (2016)

    Google Scholar 

  19. Rensink, R.A.: The dynamic representation of scenes. Vis. cogn. 7(1–3), 17–42 (2000)

    Article  Google Scholar 

  20. Yang, J., Xing, D., Hu, Z., Yao, T.: A two-branch network with pyramid-based local and spatial attention global feature learning for vehicle re-identification. CAAI Trans. Intell. Technol. 6(1), 46–54 (2021)

    Article  Google Scholar 

  21. Mescheder, L., Geiger, A., Nowozin, S.: Which training methods for GANs do actually converge? In: International Conference on Machine Learning, pp. 3481–3490 (2018).

  22. Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., Joliot, M.: Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1), 273–289 (2002)

    Article  Google Scholar 

  23. Ashburner, J., Friston, K.J.: Voxel-based morphometry—the methods. Neuroimage 11(6), 805–821 (2000)

    Article  Google Scholar 

  24. Li, S.-Y., Jiang, Y., Zhou, Z.-H.: Partial multi-view clustering, In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28(1) (2014)

  25. Yoon, J., Jordon, J., Van Der Schaar, M.: GAIN: Missing data imputation using generative adversarial nets, In: 35th International Conference Machine Learning. ICML 2018, vol. 13, pp. 9042–9051, (2018).

Download references

Funding

This work was supported in part by National Natural Science Foundation of China (Nos. 62076129, 61501230, 61732006, 61876082 and 61861130366), National Science and Technology Major Project (No. 2018ZX10201002), and the National Key R&D Program of China (Grant Nos.: 2018YFC2001600, 2018YFC2001602).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Haizhou Ye, Qi Zhu, Yuan Yao & Daoqiang Zhang

  2. Indeed Singapore Pte Ltd, Singapore, 049315, Singapore

    Yichao Jin

Authors
  1. Haizhou Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuan Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yichao Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daoqiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

QZ and DZ received the idea, identified and coordinated the study. HY partially designed the methods. QZ contributed to the method. HY contributed to programing generated results, and wrote the initial draft. QZ, YY, YJ and DZ revised the manuscript. All authors read, revised and approved the final manuscript.

Corresponding author

Correspondence to Qi Zhu.

Ethics declarations

Conflicts of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Consent for publication

All the authors listed have approved the manuscript that is enclosed.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ye, H., Zhu, Q., Yao, Y. et al. Pairwise feature-based generative adversarial network for incomplete multi-modal Alzheimer’s disease diagnosis. Vis Comput 39, 2235–2244 (2023). https://doi.org/10.1007/s00371-021-02354-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02354-5

Keywords

Search

Navigation