Joint Registration And Segmentation Of Xray Images Using Generative Adversarial Networks

  • Dwarikanath MahapatraEmail author
  • Zongyuan Ge
  • Suman Sedai
  • Rajib Chakravorty
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11046)


Medical image registration and segmentation are complementary functions and combining them can improve each other’s performance. Conventional deep learning (DL) based approaches tackle the two problems separately without leveraging their mutually beneficial information. We propose a DL based approach for joint registration and segmentation (JRS) of chest Xray images. Generative adversarial networks (GANs) are trained to register a floating image to a reference image by combining their segmentation map similarity with conventional feature maps. Intermediate segmentation maps from the GAN’s convolution layers are used in the training stage to generate the final segmentation mask at test time. Experiments on chest Xray images show that JRS gives better registration and segmentation performance than when solving them separately.


  1. 1.
    Dosovitskiy, A., Fischer, P., et. al.: Flownet: Learning optical flow with convolutional networks. In: Proceedings of IEEE ICCV, pp. 2758–2766 (2015)Google Scholar
  2. 2.
    van Ginneken, B., Stegmann, M., Loog, M.: Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database. Med. Imag. Anal. 10(1), 19–40 (2006)CrossRefGoogle Scholar
  3. 3.
    Goodfellow, I., et al.: Generative adversarial nets. In: Proceedings of NIPS, pp. 2672–2680 (2014)Google Scholar
  4. 4.
    Harrison, A., Xu, Z., George, K., Lu, L.: Progressive and multi-path holistically nested neural networks for pathological lung segmentation from ct images. In: Proceedings of MICCAI, pp. 621–629 (2017)Google Scholar
  5. 5.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: arXiv preprint arXiv:1412.6980 (2014)
  6. 6.
    Klein, S., Staring, M., Murphy, K., Viergever, M., Pluim, J.: Elastix: a toolbox for intensity based medical image registration. IEEE Trans. Med. Imag. 29(1), 196–205 (2010)CrossRefGoogle Scholar
  7. 7.
    Liao, R., et al.: An artificial agent for robust image registration. In: AAAI, pp. 4168–4175 (2017)Google Scholar
  8. 8.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of CVPR, pp. 3431–3440 (2015)Google Scholar
  9. 9.
    Mahapatra, D., Sun, Y.: Joint registration and segmentation of dynamic cardiac perfusion images using mrfs. In: Proceedings of MICCAI, pp. 493–501 (2010)CrossRefGoogle Scholar
  10. 10.
    Miao, S., Z.J. Wang, Y.Z., Liao, R.: Real-time 2d/3d registration via cnn regression. In: IEEE ISBI, pp. 1430–1434 (2016)Google Scholar
  11. 11.
    Rohe, M., Datar, M., Heimann, T., Sermesant, M., Pennec, X.: SVF-Net: Learning deformable image registration using shape matching. In: Proceedings of MICCAI, pp. 266–274 (2017)CrossRefGoogle Scholar
  12. 12.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Proceedings of MICCAI, pp. 234–241 (2015)Google Scholar
  13. 13.
    Shen, H., Wang, R., Zhang, J., McKenna, S.: Boundary-aware fully convolutional network for brain tumor segmentation. In: Proceedings of MICCAI, pp. 433–441 (2017)Google Scholar
  14. 14.
    Simonyan, K., Zisserman., A.: Very deep convolutional networks for large-scale image recognition. CoRR absarXiv:1409.1556 (2014)
  15. 15.
    Sokooti, H., de Vos, B., Berendsen, F., Lelieveldt, B., Isgum, I., Staring, M., et al.: Nonrigid image registration using multiscale 3d convolutional neural networks. In: MICCAI, pp. 232–239 (2017)Google Scholar
  16. 16.
    de Vos, B., Berendsen, F., Viergever, M., Staring, M., Isgum, I.: End-to-end unsupervised deformable image registration with a convolutional neural network. In: arXiv preprint arXiv:1704.06065 (2017)
  17. 17.
    Wang, F., Vemuri, B., Eisenschenk, S.: Joint registration and segmentation of neuroanatomic structures from brain mri. J Acad. Radiol. 12(9), 1104–1111 (2006)CrossRefGoogle Scholar
  18. 18.
    Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R., et al.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of CVPR (2017)Google Scholar
  19. 19.
    Wang, Z., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Imag. Proc. 13(4), 600–612 (2004)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of CVPR, pp. 2921–2929 (2016)Google Scholar
  21. 21.
    Zhu, J., park, T., Isola, P., Efros, A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: arXiv preprint arXiv:1703.10593 (2017)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Dwarikanath Mahapatra
    • 1
    Email author
  • Zongyuan Ge
    • 2
  • Suman Sedai
    • 1
  • Rajib Chakravorty
    • 1
  1. 1.IBM Research AustraliaMelbourneAustralia
  2. 2.Monash UniversityMelbourneAustralia

Personalised recommendations