Multi-scale GANs for Memory-efficient Generation of High Resolution Medical Images

  • Hristina UzunovaEmail author
  • Jan Ehrhardt
  • Fabian Jacob
  • Alex Frydrychowicz
  • Heinz Handels
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11769)


Currently generative adversarial networks (GANs) are rarely applied to medical images of large sizes, especially 3D volumes, due to their large computational demand. We propose a novel multi-scale patch-based GAN approach to generate large high resolution 2D and 3D images. Our key idea is to first learn a low-resolution version of the image and then generate patches of successively growing resolutions conditioned on previous scales. In a domain translation use-case scenario, 3D thorax CTs of size \(512^3\) and thorax X-rays of size \(2048^2\) are generated and we show that, due to the constant GPU memory demand of our method, arbitrarily large images of high resolution can be generated. Moreover, compared to common patch-based approaches, our multi-resolution scheme enables better image quality and prevents patch artifacts.


Multi-scale GAN High resolution 3D images 

Supplementary material

490281_1_En_13_MOESM1_ESM.pdf (4.5 mb)
Supplementary material 1 (pdf 4589 KB)

Supplementary material 2 (mp4 51416 KB)


  1. 1.
    Chollet, F., et al.: Keras (2015). Accessed 15 Sept 2019
  2. 2.
    Denton, E.L., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Advances in Neural Information Processing Systems, pp. 1486–1494 (2015)Google Scholar
  3. 3.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  4. 4.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017)Google Scholar
  5. 5.
    Kamnitsas, K., et al.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation (2017)CrossRefGoogle Scholar
  6. 6.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)Google Scholar
  7. 7.
    Lei, Y., et al.: MRI-based synthetic CT generation using deep convolutional neural network. In: SPIE Medical Imaging, vol. 10949 (2019)Google Scholar
  8. 8.
    Paszke, A., et al.: Automatic differentiation in PyTorch (2017)Google Scholar
  9. 9.
    Shin, H.-C., et al.: Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In: Gooya, A., Goksel, O., Oguz, I., Burgos, N. (eds.) SASHIMI 2018. LNCS, vol. 11037, pp. 1–11. Springer, Cham (2018). Scholar
  10. 10.
    Wu, J., Zhang, C., Xue, T., Freeman, W.T., Tenenbaum, J.B.: Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In: Advances in Neural Information Processing Systems, pp. 82–90 (2016)Google Scholar
  11. 11.
    Yu, B., Zhou, L., Wang, L., Fripp, J., Bourgeat, P.: 3D cGAN based cross-modality MR image synthesis for brain tumor segmentation. In: IEEE International Symposium on Biomedical Imaging (ISBI), pp. 626–630 (2018)Google Scholar
  12. 12.
    Zhang, H., et al.: StackGAN++: realistic image synthesis with stacked generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 47, 1947–1962 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hristina Uzunova
    • 1
    Email author
  • Jan Ehrhardt
    • 1
  • Fabian Jacob
    • 2
  • Alex Frydrychowicz
    • 2
  • Heinz Handels
    • 1
  1. 1.Institute of Medical InformaticsUniversity of LübeckLübeckGermany
  2. 2.Department for Radiology and Nuclear MedicineUniversity Hospital of Schleswig-HolsteinLübeckGermany

Personalised recommendations