Synthesizing Missing PET from MRI with Cycle-consistent Generative Adversarial Networks for Alzheimer’s Disease Diagnosis

  • Yongsheng Pan
  • Mingxia Liu
  • Chunfeng Lian
  • Tao Zhou
  • Yong XiaEmail author
  • Dinggang ShenEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11072)


Multi-modal neuroimages (e.g., MRI and PET) have been widely used for diagnosis of brain diseases such as Alzheimer’s disease (AD) by providing complementary information. However, in practice, it is unavoidable to have missing data, i.e., missing PET data for many subjects in the ADNI dataset. A straightforward strategy to tackle this challenge is to simply discard subjects with missing PET, but this will significantly reduce the number of training subjects for learning reliable diagnostic models. On the other hand, since different modalities (i.e., MRI and PET) were acquired from the same subject, there often exist underlying relevance between different modalities. Accordingly, we propose a two-stage deep learning framework for AD diagnosis using both MRI and PET data. Specifically, in the first stage, we impute missing PET data based on their corresponding MRI data by using 3D Cycle-consistent Generative Adversarial Networks (3D-cGAN) to capture their underlying relationship. In the second stage, with the complete MRI and PET (i.e., after imputation for the case of missing PET), we develop a deep multi-instance neural network for AD diagnosis and also mild cognitive impairment (MCI) conversion prediction. Experimental results on subjects from ADNI demonstrate that our synthesized PET images with 3D-cGAN are reasonable, and also our two-stage deep learning method outperforms the state-of-the-art methods in AD diagnosis.



This research was supported in part by the National Natural Science Foundation of China under Grants 61771397 and 61471297, in part by Innovation Foundation for Doctor Dissertation of NPU under Grants CX201835, and in part by NIH grants EB008374, AG041721, AG042599, EB022880. Data collection and sharing for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and DOD ADNI.


  1. 1.
    Calhoun, V.D., Sui, J.: Multimodal fusion of brain imaging data: a key to finding the missing link(s) in complex mental illness. Biol. Psychiatry: Cogn. Neurosci. Neuroimaging 1(3), 230–244 (2016)Google Scholar
  2. 2.
    Liu, M., Gao, Y., Yap, P.T., Shen, D.: Multi-hypergraph learning for incomplete multi-modality data. IEEE J. Biomed. Health Inform. 22(4), 1197–1208 (2017)CrossRefGoogle Scholar
  3. 3.
    Parker, R.: Missing Data Problems in Machine Learning. VDM Verlag, Saarbrücken (2010)Google Scholar
  4. 4.
    Liu, M., Zhang, J., Yap, P.T., Shen, D.: View-aligned hypergraph learning for Alzheimer’s disease diagnosis with incomplete multi-modality data. Med. Image Anal. 36, 123–134 (2017)CrossRefGoogle Scholar
  5. 5.
    Zhang, D., Shen, D.: Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease. NeuroImage 59(2), 895–907 (2012)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017)
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  8. 8.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  9. 9.
    Zhang, J., Gao, Y., Gao, Y.: Detecting anatomical landmarks for fast Alzheimer’s disease diagnosis. IEEE Trans. Med. Imaging 35(12), 2524–2533 (2016)CrossRefGoogle Scholar
  10. 10.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefGoogle Scholar
  11. 11.
    Jack, C., Bernstein, M., Fox, N.: The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. Magn. Reson. Imaging 27(4), 685–691 (2008)CrossRefGoogle Scholar
  12. 12.
    Matthews, B.: Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim. Biophys. Acta (BBA) - Protein Struct. 405(2), 442–451 (1975)CrossRefGoogle Scholar
  13. 13.
    Ashburner, J., Friston, K.J.: Voxel-based morphometry - the methods. NeuroImage 11(6), 805–821 (2000)CrossRefGoogle Scholar
  14. 14.
    Liu, M., Zhang, J., Adeli, E., Shen, D.: Landmark-based deep multi-instance learning for brain disease diagnosis. Med. Image Anal. 43, 157–168 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of Computer Science and EngineeringNorthwestern Polytechnical UniversityXi’anChina
  2. 2.Department of Radiology and BRICUniversity of North Carolina at Chapel HillChapel HillUSA

Personalised recommendations