Deep MR to CT Synthesis Using Unpaired Data

  • Jelmer M. WolterinkEmail author
  • Anna M. Dinkla
  • Mark H. F. Savenije
  • Peter R. Seevinck
  • Cornelis A. T. van den Berg
  • Ivana Išgum
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10557)


MR-only radiotherapy treatment planning requires accurate MR-to-CT synthesis. Current deep learning methods for MR-to-CT synthesis depend on pairwise aligned MR and CT training images of the same patient. However, misalignment between paired images could lead to errors in synthesized CT images. To overcome this, we propose to train a generative adversarial network (GAN) with unpaired MR and CT images. A GAN consisting of two synthesis convolutional neural networks (CNNs) and two discriminator CNNs was trained with cycle consistency to transform 2D brain MR image slices into 2D brain CT image slices and vice versa. Brain MR and CT images of 24 patients were analyzed. A quantitative evaluation showed that the model was able to synthesize CT images that closely approximate reference CT images, and was able to outperform a GAN model trained with paired MR and CT images.


Deep learning Radiotherapy Treatment planning CT synthesis Generative adversarial networks 


  1. 1.
    Bahrami, K., Shi, F., Rekik, I., Shen, D.: Convolutional neural network for reconstruction of 7T-like Images from 3T MRI using appearance and anatomical features. In: Carneiro, G., Mateus, D., Peter, L., Bradley, A., Tavares, J.M.R.S., Belagiannis, V., Papa, J.P., Nascimento, J.C., Loog, M., Lu, Z., Cardoso, J.S., Cornebise, J. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 39–47. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_5 Google Scholar
  2. 2.
    Edmund, J.M., Nyholm, T.: A review of substitute CT generation for MRI-only radiation therapy. Radiat. Oncol. 12(1), 28 (2017). doi: 10.1186/s13014-016-0747 CrossRefGoogle Scholar
  3. 3.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  4. 4.
    Han, X.: MR-based synthetic CT generation using a deep convolutional neural network method. Med. Phys. 44(4), 1408–1419 (2017)CrossRefGoogle Scholar
  5. 5.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint (2016). arXiv:1611.07004
  6. 6.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). doi: 10.1007/978-3-319-46475-6_43 CrossRefGoogle Scholar
  7. 7.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  8. 8.
    Nie, D., Cao, X., Gao, Y., Wang, L., Shen, D.: Estimating CT image from MRI data using 3D fully convolutional networks. In: Carneiro, G., Mateus, D., Peter, L., Bradley, A., Tavares, J.M.R.S., Belagiannis, V., Papa, J.P., Nascimento, J.C., Loog, M., Lu, Z., Cardoso, J.S., Cornebise, J. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 170–178. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_18 Google Scholar
  9. 9.
    Nie, D., Trullo, R., Petitjean, C., Ruan, S., Shen, D.: Medical image synthesis with context-aware generative adversarial networks. arXiv preprint (2016). arXiv:1612.05362
  10. 10.
    Wolterink, J.M., Leiner, T., Viergever, M.A., Isgum, I.: Generative adversarial networks for noise reduction in low-dose CT. IEEE Trans. Med. Imaging (2017).
  11. 11.
    Yi, Z., Zhang, H., Gong, P.T., et al.: Dualgan: unsupervised dual learning for image-to-image translation. arXiv preprint (2017). arXiv:1704.02510
  12. 12.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint (2017). arXiv:1703.10593

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jelmer M. Wolterink
    • 1
    Email author
  • Anna M. Dinkla
    • 2
  • Mark H. F. Savenije
    • 2
  • Peter R. Seevinck
    • 1
  • Cornelis A. T. van den Berg
    • 2
  • Ivana Išgum
    • 1
  1. 1.Image Sciences Institute, University Medical Center UtrechtUtrechtThe Netherlands
  2. 2.Department of RadiotherapyUniversity Medical Center UtrechtUtrechtThe Netherlands

Personalised recommendations