Unpaired Synthetic Image Generation in Radiology Using GANs

  • Denis ProkopenkoEmail author
  • Joël Valentin Stadelmann
  • Heinrich Schulz
  • Steffen Renisch
  • Dmitry V. Dylov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11850)


In this work, we investigate approaches to generating synthetic Computed Tomography (CT) images from the real Magnetic Resonance Imaging (MRI) data. Generating the radiological scans has grown in popularity in the recent years due to its promise to enable single-modality radiotherapy planning in clinical oncology, where the co-registration of the radiological modalities is cumbersome. We rely on the Generative Adversarial Network (GAN) models with cycle consistency which permit unpaired image-to-image translation between the modalities. We also introduce the perceptual loss function term and the coordinate convolutional layer to further enhance the quality of translated images. The Unsharp masking and the Super-Resolution GAN (SRGAN) were considered to improve the quality of synthetic images. The proposed architectures were trained on the unpaired MRI-CT data and then evaluated on the paired brain dataset. The resulting CT scans were generated with the mean absolute error (MAE), the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) scores of 60.83 HU, 17.21 dB, and 0.8, respectively. DualGAN with perceptual loss function term and coordinate convolutional layer proved to perform best. The MRI-CT translation approach holds potential to eliminate the need for the patients to undergo both examinations and to be clinically accepted as a new tool for radiotherapy planning.


Deep learning Image translation Radiotherapy 



Data used in this publication were generated by the National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC).


  1. 1.
    Battista, J.J., Rider, W.D., Van Dyk, J.: Computed tomography for radiotherapy planning. Int. J. Radiat. Oncol.* Biol.* Phys. 6(1), 99–107 (1980)CrossRefGoogle Scholar
  2. 2.
    Chen, L., et al.: MRI-based treatment planning for radiotherapy: dosimetric verification for prostate IMRT. Int. J. Radiat. Oncol.* Biol.* Phys. 60(2), 636–647 (2004)CrossRefGoogle Scholar
  3. 3.
    Clark, K., et al.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26(6), 1045–1057 (2013)CrossRefGoogle Scholar
  4. 4.
    Coy, P., Kennelly, G.: The role of curative radiotherapy in the treatment of lung cancer. Cancer 45(4), 698–702 (1980)CrossRefGoogle Scholar
  5. 5.
    (CPTAC), N.C.I.C.P.T.A.C.: Radiology Data from the Clinical Proteomic Tumor Analysis Consortium Glioblastoma Multiforme [CPTAC-GBM] collection [Data set]. The Cancer Imaging Archive (2018).
  6. 6.
    Gelband, H., Jha, P., Sankaranarayanan, R., Horton, S.: Disease Control Priorities: Cancer, vol. 3. The World Bank (2015)Google Scholar
  7. 7.
    Hofmann, M., et al.: MRI-based attenuation correction for PET/MRI: a novel approach combining pattern recognition and atlas registration. J. Nucl. Med. 49(11), 1875–1883 (2008)CrossRefGoogle Scholar
  8. 8.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). Scholar
  9. 9.
    Kapanen, M., Collan, J., Beule, A., Seppälä, T., Saarilahti, K., Tenhunen, M.: Commissioning of MRI-only based treatment planning procedure for external beam radiotherapy of prostate. Magn. Reson. Med. 70(1), 127–135 (2013)CrossRefGoogle Scholar
  10. 10.
    Karlsson, M., Karlsson, M.G., Nyholm, T., Amies, C., Zackrisson, B.: Dedicated magnetic resonance imaging in the radiotherapy clinic. Int. J. Radiat. Oncol.* Biol.* Phys. 74(2), 644–651 (2009)CrossRefGoogle Scholar
  11. 11.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. CoRR abs/1609.04802 (2016)Google Scholar
  12. 12.
    Lei, Y., et al.: MRI-based pseudo CT synthesis using anatomical signature and alternating random forest with iterative refinement model. J. Med. Imaging 5(4), 043504 (2018)CrossRefGoogle Scholar
  13. 13.
    Liu, M., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. CoRR abs/1703.00848 (2017).
  14. 14.
    Liu, R., et al.: An intriguing failing of convolutional neural networks and the coordconv solution. arXiv preprint arXiv:1807.03247 (2018)
  15. 15.
    Mph, R.L.S., Kimberly, D.: Cancer statistics, 2017. CA Cancer J. Clin. 67(1), 7–30 (2017)CrossRefGoogle Scholar
  16. 16.
    Polesel, A., Ramponi, G., Mathews, V.J.: Image enhancement via adaptive unsharp masking. IEEE Trans. Image Process. 9(3), 505–510 (2000)CrossRefGoogle Scholar
  17. 17.
    Prokopenko, D., Stadelmann, J.V., Schulz, H., Renisch, S., Dylov, D.V.: Synthetic CT generation from MRI using improved DualGAN (2019)Google Scholar
  18. 18.
    Ren, H.: SRGAN: A PyTorch implementation of SRGAN based on CVPR 2017 paper photo-realistic single image super-resolution using a generative adversarial networkGoogle Scholar
  19. 19.
    Vallières, M., et al.: Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci. Rep. 7(1), 10117 (2017)CrossRefGoogle Scholar
  20. 20.
    Vallières, M., et al.: Data from Head-Neck-PET-CT. The Cancer Imaging Archive (2017).
  21. 21.
    Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Isgum, I.: Deep MR to CT synthesis using unpaired data. CoRR abs/1708.01155 (2017).
  22. 22.
    Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. CoRR abs/1704.02510 (2017)Google Scholar
  23. 23.
    Zhang, R., Pfister, T., Li, J.: Harmonic unpaired image-to-image translation. In: ICLR (2019)Google Scholar
  24. 24.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Skolkovo Institute of Science and TechnologyMoscowRussian Federation
  2. 2.Philips Innovation Labs RUSMoscowRussian Federation
  3. 3.Philips GmbH Innovative TechnologiesHamburgGermany

Personalised recommendations