Adversarial Optimization for Joint Registration and Segmentation in Prostate CT Radiotherapy

  • Mohamed S. ElmahdyEmail author
  • Jelmer M. Wolterink
  • Hessam Sokooti
  • Ivana Išgum
  • Marius Staring
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11769)


Joint image registration and segmentation has long been an active area of research in medical imaging. Here, we reformulate this problem in a deep learning setting using adversarial learning. We consider the case in which fixed and moving images as well as their segmentations are available for training, while segmentations are not available during testing; a common scenario in radiotherapy. The proposed framework consists of a 3D end-to-end generator network that estimates the deformation vector field (DVF) between fixed and moving images in an unsupervised fashion and applies this DVF to the moving image and its segmentation. A discriminator network is trained to evaluate how well the moving image and segmentation align with the fixed image and segmentation. The proposed network was trained and evaluated on follow-up prostate CT scans for image-guided radiotherapy, where the planning CT contours are propagated to the daily CT images using the estimated DVF. A quantitative comparison with conventional registration using elastix showed that the proposed method improved performance and substantially reduced computation time, thus enabling real-time contour propagation necessary for online-adaptive radiotherapy.


Deformable image registration Adversarial training Image segmentation Contour propagation Radiotherapy 



This study was financially supported by Varian Medical Systems and ZonMw, the Netherlands Organization for Health Research and Development, grant number 104003012. The dataset with contours were collected at Haukeland University Hospital, Bergen, Norway and were provided to us by responsible oncologist Svein Inge Helle and physicist Liv Bolstad Hysing; they are gratefully acknowledged.

Supplementary material

490281_1_En_41_MOESM1_ESM.pdf (17.5 mb)
Supplementary material 1 (pdf 17924 KB)


  1. 1.
    Lu, C., et al.: An integrated approach to segmentation and nonrigid registration for application in image-guided pelvic radiotherapy. Med. Image Anal. 15(5), 772–785 (2011)CrossRefGoogle Scholar
  2. 2.
    Yezzi, A., et al.: A variational framework for integrating segmentation and registration through active contours. Med. Image Anal. 7(2), 171–185 (2003)CrossRefGoogle Scholar
  3. 3.
    Unal, G., Slabaugh, G.: Coupled PDEs for non-rigid registration and segmentation. In: IEEE CVPR (2005)Google Scholar
  4. 4.
    Litjens, G., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRefGoogle Scholar
  5. 5.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014)Google Scholar
  6. 6.
    Kazeminia, S., et al.: GANs for Medical Image Analysis. arXiv:1809.06222v2 (2018)
  7. 7.
    Haskins, G., et al.: Deep Learning in Medical Image Registration: A Survey. arXiv:1903.02026v1 (2019)
  8. 8.
    Mahapatra, D., Ge, Z., Sedai, S., Chakravorty, R.: Joint registration and segmentation of Xray images using generative adversarial networks. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 73–80. Springer, Cham (2018). Scholar
  9. 9.
    de Vos, B.D., et al.: A deep learning framework for unsupervised affine and deformable image registration. In: Medical Image Analysis, pp. 204–212. Springer, Heidelberg (2019) Google Scholar
  10. 10.
    Klein, S., et al.: Elastix: a toolbox for intensity-based medical image registration. IEEE Trans. Med. Imaging. 29(1), 196–205 (2010)CrossRefGoogle Scholar
  11. 11.
    Arjovsky, M., et al.: Wasserstein GAN. arXiv:1701.07875v3 (2017)
  12. 12.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  13. 13.
    Gibson, E., et al.: NiftyNet: a deep-learning platform for medical imaging. Comput. Methods Programs Biomed. 158, 113–122 (2018)CrossRefGoogle Scholar
  14. 14.
    Muren, L.P., et al.: Intensity-modulated radiotherapy of pelvic lymph nodes in locally advanced prostate cancer: planning procedures and early experiences. Int. J. Radiat. Oncol. Biol. Phys. 71, 4, 1034–1041 (2008)CrossRefGoogle Scholar
  15. 15.
    Matin, et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv:1603.04467 (2017)
  16. 16.
    Isola, et al.: Image-to-Image Translation with Conditional Adversarial Networks. arXiv:1611.07004v3 (2016)
  17. 17.
    Qiao, Y.: Fast optimization methods for image registration in adaptive radiation therapy. Ph.D. thesis, Chapter 5, Leiden University Medical Center (2017).

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mohamed S. Elmahdy
    • 1
    Email author
  • Jelmer M. Wolterink
    • 2
  • Hessam Sokooti
    • 1
  • Ivana Išgum
    • 2
  • Marius Staring
    • 1
    • 3
  1. 1.Division of Image Processing, Department of RadiologyLeiden University Medical CenterLeidenThe Netherlands
  2. 2.Image Sciences InstituteUniversity Medical Center UtrechtUtrechtThe Netherlands
  3. 3.Department of Radiation OncologyLeiden University Medical CenterLeidenThe Netherlands

Personalised recommendations