Mask Embedding for Realistic High-Resolution Medical Image Synthesis

  • Yinhao Ren
  • Zhe Zhu
  • Yingzhou Li
  • Dehan Kong
  • Rui Hou
  • Lars J. Grimm
  • Jeffery R. Marks
  • Joseph Y. LoEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11769)


Generative Adversarial Networks (GANs) have found applications in natural image synthesis and begin to show promises generating synthetic medical images. In many cases, the ability to perform controlled image synthesis using masked priors such as shape and size of organs is desired. However, mask-guided image synthesis is challenging due to the pixel level mask constraint. While the few existing mask-guided image generation approaches suffer from the lack of fine-grained texture details, we tackle the issue of mask-guided stochastic image synthesis via mask embedding. Our novel architecture first encodes the input mask as an embedding vector and then inject these embedding into the random latent vector input. The intuition is to classify semantic masks into partitions before feature up-sampling for improved sample space mapping stability. We validate our approach on a large dataset containing 39,778 patients with 443,556 negative screening Full Field Digital Mammography (FFDM) images. Experimental results show that our approach can generate realistic high-resolution (\(256\times 512\)) images with pixel-level mask constraints, and outperform other state-of-the-art approaches.


Generative Adversarial Networks Image synthesis Mask embedding Mammogram 



This work was supported in part by NIH/NCI U01-CA214183 and U2C-CA233254, and an equipment donation by NVIDIA Corporation.

Supplementary material

490281_1_En_47_MOESM1_ESM.pdf (9.2 mb)
Supplementary material 1 (pdf 9408 KB)


  1. 1.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 214–223 (2017)Google Scholar
  2. 2.
    Che, T., Li, Y., Jacob, A.P., Bengio, Y., Li, W.: Mode regularized generative adversarial networks. CoRR abs/1612.02136 (2016)Google Scholar
  3. 3.
    Costa, P., et al.: End-to-end adversarial retinal image synthesis. IEEE Trans. Med. Imaging 37(3), 781–791 (2018)CrossRefGoogle Scholar
  4. 4.
    Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014)Google Scholar
  5. 5.
    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)Google Scholar
  6. 6.
    Han, C., et al.: GAN-based synthetic brain MR image generation. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 734–738, April 2018.
  7. 7.
    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a Nash equilibrium. CoRR abs/1706.08500 (2017)Google Scholar
  8. 8.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017Google Scholar
  9. 9.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. CoRR abs/1710.10196 (2017)Google Scholar
  10. 10.
    Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. CoRR abs/1812.04948 (2018)Google Scholar
  11. 11.
    Korkinof, D., Rijken, T., O’Neill, M., Yearsley, J., Harvey, H., Glocker, B.: High-resolution mammogram synthesis using progressive generative adversarial networks. CoRR abs/1807.03401 (2018)Google Scholar
  12. 12.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv e-prints arXiv:1411.1784, November 2014
  13. 13.
    Moradi, M., Madani, A., Karargyris, A., Syeda-Mahmood, T.F.: Chest X-ray generation and data augmentation for cardiovascular abnormality classification, p. 57, March 2018Google Scholar
  14. 14.
    Nie, D., et al.: Medical image synthesis with deep convolutional adversarial networks. IEEE Trans. Biomed. Eng. 65(12), 2720–2730 (2018)CrossRefGoogle Scholar
  15. 15.
    Nie, D., et al.: Medical image synthesis with context-aware generative adversarial networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 417–425. Springer, Cham (2017). Scholar
  16. 16.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. CoRR abs/1505.04597 (2015)Google Scholar
  17. 17.
    Salimans, T., et al.: Improved techniques for training GANs. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 2234–2242 (2016)Google Scholar
  18. 18.
    Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: a review. CoRR abs/1809.07294 (2018)Google Scholar
  19. 19.
    Zhao, H., Li, H., Maurer-Stroh, S., Cheng, L.: Synthesizing retinal and neuronal images with generative adversarial nets. Med. Image Anal. 49, 14–26 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yinhao Ren
    • 1
  • Zhe Zhu
    • 2
  • Yingzhou Li
    • 3
  • Dehan Kong
    • 4
  • Rui Hou
    • 5
  • Lars J. Grimm
    • 2
  • Jeffery R. Marks
    • 6
  • Joseph Y. Lo
    • 1
    • 2
    • 5
    Email author
  1. 1.Department of Biomedical EngineeringDuke UniversityDurhamUSA
  2. 2.Department of RadiologyDuke University School of MedicineDurhamUSA
  3. 3.Department of MathematicsDuke UniversityDurhamUSA
  4. 4.Department of AutomationBeijing Institute of TechnologyBeijingChina
  5. 5.Department of Electrical EngineeringDuke UniversityDurhamUSA
  6. 6.Department of SurgeryDuke University School of MedicineDurhamUSA

Personalised recommendations