Skip to main content
SpringerLink
Log in

Synthetic CT generation from weakly paired MR images using cycle-consistent GAN for MR-guided radiotherapy

  • Original Article
  • Published:
Biomedical Engineering Letters Aims and scope Submit manuscript

Abstract

Although MR-guided radiotherapy (MRgRT) is advancing rapidly, generating accurate synthetic CT (sCT) from MRI is still challenging. Previous approaches using deep neural networks require large dataset of precisely co-registered CT and MRI pairs that are difficult to obtain due to respiration and peristalsis. Here, we propose a method to generate sCT based on deep learning training with weakly paired CT and MR images acquired from an MRgRT system using a cycle-consistent GAN (CycleGAN) framework that allows the unpaired image-to-image translation in abdomen and thorax. Data from 90 cancer patients who underwent MRgRT were retrospectively used. CT images of the patients were aligned to the corresponding MR images using deformable registration, and the deformed CT (dCT) and MRI pairs were used for network training and testing. The 2.5D CycleGAN was constructed to generate sCT from the MRI input. To improve the sCT generation performance, a perceptual loss that explores the discrepancy between high-dimensional representations of images extracted from a well-trained classifier was incorporated into the CycleGAN. The CycleGAN with perceptual loss outperformed the U-net in terms of errors and similarities between sCT and dCT, and dose estimation for treatment planning of thorax, and abdomen. The sCT generated using CycleGAN produced virtually identical dose distribution maps and dose-volume histograms compared to dCT. CycleGAN with perceptual loss outperformed U-net in sCT generation when trained with weakly paired dCT-MRI for MRgRT. The proposed method will be useful to increase the treatment accuracy of MR-only or MR-guided adaptive radiotherapy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

Research data are not available at this time.

References

  1. Edmund JM, Nyholm T. A review of substitute CT generation for MRI-only radiation therapy. Radiat Oncol. 2017;12(1):28.

    Article  Google Scholar 

  2. Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys. 2017;44(4):1408–19.

    Article  Google Scholar 

  3. Chen S, et al. Technical Note: U-net-generated synthetic CT images for magnetic resonance imaging-only prostate intensity-modulated radiation therapy treatment planning. Med Phys. 2018;45(12):5659–65.

    Article  Google Scholar 

  4. Dinkla AM, et al. MR-only brain radiation therapy: dosimetric evaluation of synthetic CTs generated by a dilated convolutional neural network. Int J Radiat Oncol Biol Phys. 2018;102(4):801–12.

    Article  Google Scholar 

  5. Gupta D, et al. Generation of synthetic CT images From MRI for treatment planning and patient positioning using a 3-channel U-Net trained on sagittal images. Front Oncol. 2019;9:964.

    Article  Google Scholar 

  6. Neppl S, et al. Evaluation of proton and photon dose distributions recalculated on 2D and 3D Unet-generated pseudoCTs from T1-weighted MR head scans. Acta Oncol. 2019;58(10):1429–34.

    Article  Google Scholar 

  7. Fu J, et al. Deep learning approaches using 2D and 3D convolutional neural networks for generating male pelvic synthetic computed tomography from magnetic resonance imaging. Med Phys. 2019;46(9):3788–98.

    Article  Google Scholar 

  8. Alvarez Andres E, et al. Dosimetry-driven quality measure of brain pseudo computed tomography generated from deep learning for mri-only radiation therapy treatment planning. Int J Radiat Oncol Biol Phys. 2020;108(3):813–23.

    Article  Google Scholar 

  9. Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. Springer; 2015. pp. 234–41.

  10. Goodfellow I, et al. Generative adversarial networks. In: Advances in neural information processing systems. 2014. pp. 2672–80.

  11. Nie D, et al. Medical image synthesis with context-aware generative adversarial networks. Med Image Comput Comput Assist Interv. 2017;10435:417–25.

    Google Scholar 

  12. Isola P, et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. pp. 1125–34.

  13. Emami H, et al. Generating synthetic CTs from magnetic resonance images using generative adversarial networks. Med Phys. 2018;45(8):3627–36.

    Article  Google Scholar 

  14. Largent A, et al. Comparison of deep learning-based and patch-based methods for pseudo-CT generation in MRI-based prostate dose planning. Int J Radiat Oncol Biol Phys. 2019;105(5):1137–50.

    Article  Google Scholar 

  15. Olberg S, et al. Synthetic CT reconstruction using a deep spatial pyramid convolutional framework for MR-only breast radiotherapy. Med Phys. 2019;46(9):4135–47.

    Article  Google Scholar 

  16. Fu J, et al. Generation of abdominal synthetic CTs from 0.35 T MR images using generative adversarial networks for MR-only liver radiotherapy. Biomedical Physics & Engineering Express. 2020;6(1):015033.

    Article  Google Scholar 

  17. Zhu JY, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE international conference on computer vision. 2017. pp. 2223–32.

  18. Lei Y, et al. MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks. Med Phys. 2019;46(8):3565–81.

    Article  Google Scholar 

  19. Shafai-Erfani G, et al. Dose evaluation of MRI-based synthetic CT generated using a machine learning method for prostate cancer radiotherapy. Med Dosim. 2019;44(4):e64–70.

    Article  Google Scholar 

  20. Liu Y, et al. MRI-based treatment planning for liver stereotactic body radiotherapy: validation of a deep learning-based synthetic CT generation method. Br J Radiol. 2019;92(1100):20190067.

    Article  Google Scholar 

  21. Wolterink JM, et al. Deep MR to CT synthesis using unpaired data. In: International workshop on simulation and synthesis in medical imaging. Springer; 2017. pp. 14–23.

  22. Tustison NJ, et al. N4ITK: improved N3 bias correction. IEEE Trans Med Imaging. 2010;29(6):1310–20.

    Article  Google Scholar 

  23. Lehtinen J, et al. Noise2noise: Learning image restoration without clean data. arXiv preprint https://arxiv.org/abs/1803.04189. 2018.

  24. He K, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. pp. 770–8

  25. Miyato T, et al. Spectral normalization for generative adversarial networks. arXiv preprint https://arxiv.org/abs/1802.05957. 2018.

  26. Heusel M, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems. 2017. pp. 6629–40. https://proceedings.neurips.cc/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html.

  27. Park J, et al. Computed tomography super-resolution using deep convolutional neural network. Phys Med Biol. 2018;63(14):145011.

    Article  Google Scholar 

  28. Hwang D, et al. Improving the accuracy of simultaneously reconstructed activity and attenuation maps using deep learning. J Nucl Med. 2018;59(10):1624–9.

    Article  Google Scholar 

  29. Kang SK, et al. Adaptive template generation for amyloid PET using a deep learning approach. Hum Brain Mapp. 2018;39(9):3769–78.

    Article  Google Scholar 

  30. Lee MS, et al. Deep-dose: a voxel dose estimation method using deep convolutional neural network for personalized internal dosimetry. Sci Rep. 2019;9(1):1–9.

    Google Scholar 

  31. Hwang D, et al. Generation of PET attenuation map for whole-body time-of-flight 18F-FDG PET/MRI using a deep neural network trained with simultaneously reconstructed activity and attenuation maps. J Nucl Med. 2019;60(8):1183–9.

    Article  Google Scholar 

  32. Lee JS. A review of deep Learning-based approaches for attenuation correction in positron emission tomography. IEEE Trans Radiat Plasma Med Sci. 2020;5(2):160–84.

    Article  Google Scholar 

  33. Korb JP, Bryant RG. Magnetic field dependence of proton spin-lattice relaxation times. Magn Reson Med Off J Int Soc Magn Reson Med. 2002;48(1):21–6.

    Article  Google Scholar 

  34. Klüter S. Technical design and concept of a 0.35 T MR-Linac. Clin Transl Radiat Oncol. 2019;18:98–101.

    Article  Google Scholar 

  35. Park JM, et al. Commissioning experience of tri-cobalt-60 MRI-guided radiation therapy system. Prog Med Phys. 2015;26(4):193–200.

    Article  Google Scholar 

  36. Henke L, et al. Magnetic resonance image-guided radiotherapy (MRIgRT): a 4.5-year clinical experience. Clin Oncol. 2018;30(11):720–7.

    Article  Google Scholar 

  37. Hegazy MA, et al. U-net based metal segmentation on projection domain for metal artifact reduction in dental CT. Biomed Eng Lett. 2019;9(3):375–85.

    Article  Google Scholar 

  38. Comelli A, et al. Deep learning approach for the segmentation of aneurysmal ascending aorta. Biomed Eng Lett. 2020;11(1):1–10.

    Google Scholar 

  39. Park J, et al. Measurement of glomerular filtration rate using quantitative SPECT/CT and deep-learning-based kidney segmentation. Sci Rep. 2019;9(1):1–8.

    Google Scholar 

  40. Yoo J, Eom H, Choi YS. Image-to-image translation using a cross-domain auto-encoder and decoder. Appl Sci. 2019;9(22):4780.

    Article  Google Scholar 

  41. Wang C, et al. Perceptual adversarial networks for image-to-image transformation. IEEE Trans Image Process. 2018;27(8):4066–79.

    Article  MathSciNet  Google Scholar 

  42. Boldrini L, et al. Online adaptive magnetic resonance guided radiotherapy for pancreatic cancer: state of the art, pearls and pitfalls. Radiat Oncol. 2019;14(1):1–6.

    Article  Google Scholar 

  43. Rudra S, et al. Using adaptive magnetic resonance image-guided radiation therapy for treatment of inoperable pancreatic cancer. Cancer Med. 2019;8(5):2123–32.

    Article  Google Scholar 

  44. Placidi L, et al. On-line adaptive MR guided radiotherapy for locally advanced pancreatic cancer: Clinical and dosimetric considerations. Tech Innov Patient Support Radiat Oncol. 2020;15:15–21.

    Article  Google Scholar 

  45. Shinohara RT, et al. Statistical normalization techniques for magnetic resonance imaging. NeuroImage Clin. 2014;6:9–19.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Grants from the Radiation Technology R&D program through the National Research Foundation of Korea funded by the Ministry of Science and ICT (2017M2A2A7A02020641, 2019M2A2B4095126, and 2020M2D9A109398911).

Author information

Author notes

  1. Seung Kwan Kang and Hyun Joon An have contributed equally to this work.

Authors and Affiliations

  1. Department of Biomedical Sciences and Nuclear Medicine, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Seung Kwan Kang & Jae Sung Lee

  2. Department of Radiation Oncology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Hyun Joon An, Hyeongmin Jin, Jung-in Kim, Eui Kyu Chie & Jong Min Park

  3. Department of Nuclear Medicine, Seoul National University Hospital, Seoul, 03080, South Korea

    Jae Sung Lee

  4. Institute of Radiation Medicine, Medical Research Center, Seoul National University College of Medicine, Seoul, 03080, South Korea

    Seung Kwan Kang, Jung-in Kim, Eui Kyu Chie, Jong Min Park & Jae Sung Lee

Authors
  1. Seung Kwan Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyun Joon An
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hyeongmin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jung-in Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eui Kyu Chie
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jong Min Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jae Sung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jong Min Park or Jae Sung Lee.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 420 KB)

Supplementary file2 (PDF 540 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, S.K., An, H.J., Jin, H. et al. Synthetic CT generation from weakly paired MR images using cycle-consistent GAN for MR-guided radiotherapy. Biomed. Eng. Lett. 11, 263–271 (2021). https://doi.org/10.1007/s13534-021-00195-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13534-021-00195-8

Search

Navigation