XCAT-GAN for Synthesizing 3D Consistent Labeled Cardiac MR Images on Anatomically Variable XCAT Phantoms

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12264)


Generative adversarial networks (GANs) have provided promising data enrichment solutions by synthesizing high-fidelity images. However, generating large sets of labeled images with new anatomical variations remains unexplored. We propose a novel method for synthesizing cardiac magnetic resonance (CMR) images on a population of virtual subjects with a large anatomical variation, introduced using the 4D eXtended Cardiac and Torso (XCAT) computerized human phantom. We investigate two conditional image synthesis approaches grounded on a semantically-consistent mask-guided image generation technique: 4-class and 8-class XCAT-GANs. The 4-class technique relies on only the annotations of the heart; while the 8-class technique employs a predicted multi-tissue label map of the heart-surrounding organs and provides better guidance for our conditional image synthesis. For both techniques, we train our conditional XCAT-GAN with real images paired with corresponding labels and subsequently at the inference time, we substitute the labels with the XCAT derived ones. Therefore, the trained network accurately transfers the tissue-specific textures to the new label maps. By creating 33 virtual subjects of synthetic CMR images at the end-diastolic and end-systolic phases, we evaluate the usefulness of such data in the downstream cardiac cavity segmentation task under different augmentation strategies. Results demonstrate that even with only 20% of real images (40 volumes) seen during training, segmentation performance is retained with the addition of synthetic CMR images. Moreover, the improvement in utilizing synthetic images for augmenting the real data is evident through the reduction of Hausdorff distance up to 28% and an increase in the Dice score up to 5%, indicating a higher similarity to the ground truth in all dimensions.


Conditional image synthesis Cardiac Magnetic Resonance imaging XCAT anatomical phantom 



This research is a part of the openGTN project, supported by the European Union in the Marie Curie Innovative Training Networks (ITN) fellowship program under project No. 764465.

Supplementary material

Supplementary material 1 (mp4 355 KB)

Supplementary material 2 (mp4 356 KB)

505216_1_En_13_MOESM3_ESM.pdf (178 kb)
Supplementary material 3 (pdf 178 KB)


  1. 1.
    Abbasi-Sureshjani, S., Amirrajab, S., Lorenz, C., Weese, J., Pluim, J., Breeuwer, M.: 4D semantic cardiac magnetic resonance image synthesis on XCAT anatomical model. In: Medical Imaging with Deep Learning (2020)Google Scholar
  2. 2.
    Amirrajab, S., Al Khalil, Y., Lorenz, C., Weese, J., Breeuwer, M.: Towards generating realistic and hetrogeneous cardiac magnetic resonance simulated image database for deep learning based image segmentation algorithms. Proceedings of the 12th Annual Meeting ISMRM Benelux Chapter 2020, P-077 (2020)Google Scholar
  3. 3.
    Andreopoulos, A., Tsotsos, J.K.: Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI. Med. Image Anal. 12(3), 335–357 (2008)CrossRefGoogle Scholar
  4. 4.
    Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., et al.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018)CrossRefGoogle Scholar
  5. 5.
    Chaitanya, K., Karani, N., Baumgartner, C.F., Becker, A., Donati, O., Konukoglu, E.: Semi-supervised and task-driven data augmentation. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 29–41. Springer, Cham (2019). Scholar
  6. 6.
    Chartsias, A., Joyce, T., Dharmakumar, R., Tsaftaris, S.A.: Adversarial image synthesis for unpaired multi-modal cardiac data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 3–13. Springer, Cham (2017). Scholar
  7. 7.
    Chen, C., et al.: Unsupervised multi-modal style transfer for cardiac MR segmentation. arXiv e-prints arXiv:1908.07344 (Aug 2019)
  8. 8.
    Corral Acero, J., et al.: SMOD - data augmentation based on statistical models of deformation to enhance segmentation in 2D cine cardiac MRI. In: Coudière, Y., Ozenne, V., Vigmond, E., Zemzemi, N. (eds.) FIMH 2019. LNCS, vol. 11504, pp. 361–369. Springer, Cham (2019). Scholar
  9. 9.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates Inc., New York (2014)Google Scholar
  10. 10.
    Huang, X., Liu, M.-Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 179–196. Springer, Cham (2018). Scholar
  11. 11.
    Isensee, F., Petersen, J., Kohl, S.A.A., Jäger, P.F., Maier-Hein, K.: nnU-Net: breaking the spell on successful medical image segmentation. ArXiv abs/1904.08128 (2019)Google Scholar
  12. 12.
    Joyce, T., Kozerke, S.: 3D medical image synthesis by factorised representation and deformable model learning. In: Burgos, N., Gooya, A., Svoboda, D. (eds.) SASHIMI 2019. LNCS, vol. 11827, pp. 110–119. Springer, Cham (2019). Scholar
  13. 13.
    Kazeminia, S., et al.: Gans for medical image analysis (2018)Google Scholar
  14. 14.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv e-prints arXiv:1312.6114, December 2013
  15. 15.
    Ma, C., Ji, Z., Gao, M.: Neural style transfer improves 3D cardiovascular MR image segmentation on inconsistent data. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 128–136. Springer, Cham (2019). Scholar
  16. 16.
    Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, pp. 2332–2341. IEEE Computer Society, June 2019Google Scholar
  17. 17.
    Pfeiffer, M., et al.: Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 119–127. Springer, Cham (2019). Scholar
  18. 18.
    Radau, P., Lu, Y., Connelly, K., Paul, G., Dick, A., Wright, G.: Evaluation framework for algorithms segmenting short axis cardiac MRI, July 2009Google Scholar
  19. 19.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  20. 20.
    Segars, W., Sturgeon, G., Mendonca, S., Grimes, J., Tsui, B.M.: 4D XCAT phantom for multimodality imaging research. Med. Phys. 37(9), 4902–4915 (2010)CrossRefGoogle Scholar
  21. 21.
    Tang, Y.B., Oh, S., Tang, Y.X., Xiao, J., Summers, R.M.: CT-realistic data augmentation using generative adversarial network for robust lymph node segmentation. In: Mori, K., Hahn, H.K. (eds.) Medical Imaging 2019: Computer-Aided Diagnosis, vol. 10950, pp. 976–981. International Society for Optics and Photonics, SPIE (2019)Google Scholar
  22. 22.
    Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8798–8807, June 2018Google Scholar
  23. 23.
    Wissmann, L., Santelli, C., Segars, W.P., Kozerke, S.: MRXCAT: realistic numerical phantoms for cardiovascular magnetic resonance. J. Cardiovasc. Magn. Reson. 16(1), 63 (2014)CrossRefGoogle Scholar
  24. 24.
    Wu, Z., Wang, X., Gonzalez, J.E., Goldstein, T., Davis, L.S.: ACE: adapting to changing environments for semantic segmentation. CoRR abs/1904.06268 (2019)Google Scholar
  25. 25.
    Yasaka, K., Abe, O.: Deep learning and artificial intelligence in radiology: current applications and future directions. PLOS Med. 15(11), 1–4 (2018)CrossRefGoogle Scholar
  26. 26.
    Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: a review. Med. Image Anal. 58, 101552 (2019)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Eindhoven University of TechnologyEindhovenThe Netherlands
  2. 2.Philips Research LaboratoriesHamburgGermany
  3. 3.Philips Healthcare, MR R&D - Clinical ScienceBestThe Netherlands

Personalised recommendations