Adversarial Optimization for Joint Registration and Segmentation in Prostate CT Radiotherapy
Joint image registration and segmentation has long been an active area of research in medical imaging. Here, we reformulate this problem in a deep learning setting using adversarial learning. We consider the case in which fixed and moving images as well as their segmentations are available for training, while segmentations are not available during testing; a common scenario in radiotherapy. The proposed framework consists of a 3D end-to-end generator network that estimates the deformation vector field (DVF) between fixed and moving images in an unsupervised fashion and applies this DVF to the moving image and its segmentation. A discriminator network is trained to evaluate how well the moving image and segmentation align with the fixed image and segmentation. The proposed network was trained and evaluated on follow-up prostate CT scans for image-guided radiotherapy, where the planning CT contours are propagated to the daily CT images using the estimated DVF. A quantitative comparison with conventional registration using elastix showed that the proposed method improved performance and substantially reduced computation time, thus enabling real-time contour propagation necessary for online-adaptive radiotherapy.
KeywordsDeformable image registration Adversarial training Image segmentation Contour propagation Radiotherapy
This study was financially supported by Varian Medical Systems and ZonMw, the Netherlands Organization for Health Research and Development, grant number 104003012. The dataset with contours were collected at Haukeland University Hospital, Bergen, Norway and were provided to us by responsible oncologist Svein Inge Helle and physicist Liv Bolstad Hysing; they are gratefully acknowledged.
- 3.Unal, G., Slabaugh, G.: Coupled PDEs for non-rigid registration and segmentation. In: IEEE CVPR (2005)Google Scholar
- 5.Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014)Google Scholar
- 6.Kazeminia, S., et al.: GANs for Medical Image Analysis. arXiv:1809.06222v2 (2018)
- 7.Haskins, G., et al.: Deep Learning in Medical Image Registration: A Survey. arXiv:1903.02026v1 (2019)
- 8.Mahapatra, D., Ge, Z., Sedai, S., Chakravorty, R.: Joint registration and segmentation of Xray images using generative adversarial networks. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 73–80. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00919-9_9CrossRefGoogle Scholar
- 9.de Vos, B.D., et al.: A deep learning framework for unsupervised affine and deformable image registration. In: Medical Image Analysis, pp. 204–212. Springer, Heidelberg (2019) Google Scholar
- 11.Arjovsky, M., et al.: Wasserstein GAN. arXiv:1701.07875v3 (2017)
- 12.Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
- 15.Matin, et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv:1603.04467 (2017)
- 16.Isola, et al.: Image-to-Image Translation with Conditional Adversarial Networks. arXiv:1611.07004v3 (2016)
- 17.Qiao, Y.: Fast optimization methods for image registration in adaptive radiation therapy. Ph.D. thesis, Chapter 5, Leiden University Medical Center (2017). http://elastix.isi.uu.nl/marius/downloads/2017_t_Qiao.pdf