Advertisement

Deep Complementary Joint Model for Complex Scene Registration and Few-Shot Segmentation on Medical Images

Conference paper
  • 525 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12363)

Abstract

Deep learning-based medical image registration and segmentation joint models utilize the complementarity (augmentation data or weakly supervised data from registration, region constraints from segmentation) to bring mutual improvement in complex scene and few-shot situation. However, further adoption of the joint models are hindered: 1) the diversity of augmentation data is reduced limiting the further enhancement of segmentation, 2) misaligned regions in weakly supervised data disturb the training process, 3) lack of label-based region constraints in few-shot situation limits the registration performance. We propose a novel Deep Complementary Joint Model (DeepRS) for complex scene registration and few-shot segmentation. We embed a perturbation factor in the registration to increase the activity of deformation thus maintaining the augmentation data diversity. We take a pixel-wise discriminator to extract alignment confidence maps which highlight aligned regions in weakly supervised data so the misaligned regions’ disturbance will be suppressed via weighting. The outputs from segmentation model are utilized to implement deep-based region constraints thus relieving the label requirements and bringing fine registration. Extensive experiments on the CT dataset of MM-WHS 2017 Challenge [42] show great advantages of our DeepRS that outperforms the existing state-of-the-art models.

Notes

Acknowledgments

This research was supported by the National Natural Science Foundation under grants (61828101,31571001,31800825), the Short-Term Recruitment Program of Foreign Experts (WQ20163200398), and Southeast University-Nanjing Medical University Cooperative Research Project (2242019K3DN08). We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.

References

  1. 1.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017)
  2. 2.
    Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9252–9260 (2018)Google Scholar
  3. 3.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M., Unal, G., Wells, W. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016. Lecture Notes in Computer Science, vol. 9901, pp. 424–432. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46723-8_49CrossRefGoogle Scholar
  4. 4.
    Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: learning augmentation strategies from data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 113–123 (2019)Google Scholar
  5. 5.
    Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)CrossRefGoogle Scholar
  6. 6.
    Estienne, T., et al.: U-ReSNet: ultimate coupling of registration and segmentation with deep nets. In: Shen, D., et al. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, vol. 11766, pp. 310–319. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-32248-9_35CrossRefGoogle Scholar
  7. 7.
    Fan, J., Cao, X., Wang, Q., Yap, P.T., Shen, D.: Adversarial learning for mono-or multi-modal registration. Med. Image Anal. 48, 101545 (2019)CrossRefGoogle Scholar
  8. 8.
    Fan, J., Cao, X., Xue, Z., Yap, P.T.: Adversarial similarity network for evaluating image alignment in deep learning based registration. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science, vol. 11070, pp. 739–746. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00928-1_83CrossRefGoogle Scholar
  9. 9.
    Ge, R., Yang, G., Xu, C., Chen, Y., Luo, L., Li, S.: Stereo-correlation and noise-distribution aware ResVoxGAN for dense slices reconstruction and noise reduction in thick low-dose CT. In: Shen, D., et al. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Lecture Notes in Computer Science, vol. 11769, pp. 328–338. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-32226-7_37CrossRefGoogle Scholar
  10. 10.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  11. 11.
    Haskins, G., Kruger, U., Yan, P.: Deep learning in medical image registration: a survey. arXiv preprint arXiv:1903.02026 (2019)
  12. 12.
    Hauberg, S., Freifeld, O., Larsen, A.B.L., Fisher, J., Hansen, L.: Dreaming more data: class-dependent distributions over diffeomorphisms for learned data augmentation. Artif. Intell. Stat. 342–350 (2016)Google Scholar
  13. 13.
    Hu, Y., et al.: Adversarial deformation regularization for training image registration neural networks. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science, vol. 11070, pp. 774–782. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00928-1_87CrossRefGoogle Scholar
  14. 14.
    Hu, Y., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal. 49, 1–13 (2018)CrossRefGoogle Scholar
  15. 15.
    Hung, W.C., Tsai, Y.H., Liou, Y.T., Lin, Y.Y., Yang, M.H.: Adversarial learning for semi-supervised semantic segmentation. arXiv preprint arXiv:1802.07934 (2018)
  16. 16.
    Jackson, P.T., Atapour-Abarghouei, A., Bonner, S., Breckon, T., Obara, B.: Style augmentation: data augmentation via style randomization. arXiv preprint arXiv:1809.05375 (2018)
  17. 17.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  18. 18.
    Kolesnikov, A., Lampert, C.H.: Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. Lecture Notes in Computer Science, vol. 9908, pp. 695–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_42CrossRefGoogle Scholar
  19. 19.
    Lateef, F., Ruichek, Y.: Survey on semantic segmentation using deep learning techniques. Neurocomputing 338, 321–348 (2019)CrossRefGoogle Scholar
  20. 20.
    Learned-Miller, E.G.: Data driven image models through continuous joint alignment. IEEE Trans. Pattern Anal. Mach. Intell. 28(2), 236–250 (2005)CrossRefGoogle Scholar
  21. 21.
    Lemley, J., Bazrafkan, S., Corcoran, P.: Smart augmentation learning an optimal data augmentation strategy. IEEE Access 5, 5858–5869 (2017)CrossRefGoogle Scholar
  22. 22.
    Li, B., et al.: A hybrid deep learning framework for integrated segmentation and registration: evaluation on longitudinal white matter tract changes. In: Shen, D., et al. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Lecture Notes in Computer Science, pp. 645–653. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-32248-9_72CrossRefGoogle Scholar
  23. 23.
    Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRefGoogle Scholar
  24. 24.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  25. 25.
    Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. IEEE (2016)Google Scholar
  26. 26.
    Nie, D., Gao, Y., Wang, L., Shen, D.: ASDNet: attention based semi-supervised deep networks for medical image segmentation. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science, vol. 11073, pp. 370–378. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00937-3_43CrossRefGoogle Scholar
  27. 27.
    Nielsen, C., Okoniewski, M.: GAN data augmentation through active learning inspired sample acquisition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 109–112 (2019)Google Scholar
  28. 28.
    Papandreou, G., Chen, L.C., Murphy, K.P., Yuille, A.L.: Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1742–1750 (2015)Google Scholar
  29. 29.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  30. 30.
    Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 60 (2019)CrossRefGoogle Scholar
  31. 31.
    Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: The IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  32. 32.
    Tang, M., Djelouah, A., Perazzi, F., Boykov, Y., Schroers, C.: Normalized cut loss for weakly-supervised CNN segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1818–1827 (2018)Google Scholar
  33. 33.
    Tang, M., Perazzi, F., Djelouah, A., Ayed, I.B., Schroers, C., Boykov, Y.: On regularized losses for weakly-supervised CNN segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. Lecture Notes in Computer Science, vol. 11220, pp. 524–540. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01270-0_31CrossRefGoogle Scholar
  34. 34.
    Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop, coursera: Neural networks for machine learning. University of Toronto, Technical report (2012)Google Scholar
  35. 35.
    Vakalopoulou, M., et al.: ATLASNet: multi-atlas non-linear deep networks for medical image segmentation. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Lecture Notes in Computer Science, vol. 11073, pp. 658–666. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00937-3_75CrossRefGoogle Scholar
  36. 36.
    de Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Išgum, I.: A deep learning framework for unsupervised affine and deformable image registration. Med. Image Anal. 52, 128–143 (2019)CrossRefGoogle Scholar
  37. 37.
    Xu, Z., Niethammer, M.: DeepatLas: joint semi-supervised learning of image registration and segmentation. arXiv preprint arXiv:1904.08465 (2019)
  38. 38.
    Yan, P., Xu, S., Rastinehad, A.R., Wood, B.J.: Adversarial image registration with application for MR and TRUS image fusion. In: Shi, Y., Suk, H.I., Liu, M. (eds.) Machine Learning in Medical Imaging. MLMI 2018. Lecture Notes in Computer Science, vol. 11046, pp. 197–204. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00919-9_23CrossRefGoogle Scholar
  39. 39.
    Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: a review. Med. Image Anal. 58, 101552 (2019)CrossRefGoogle Scholar
  40. 40.
    Zhao, A., Balakrishnan, G., Durand, F., Guttag, J.V., Dalca, A.V.: Data augmentation using learned transformations for one-shot medical image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8543–8553 (2019)Google Scholar
  41. 41.
    Zhou, Z.H.: A brief introduction to weakly supervised learning. Nat. Sci. Rev. 5(1), 44–53 (2017)CrossRefGoogle Scholar
  42. 42.
    Zhuang, X., Shen, J.: Multi-scale patch and multi-modality atlases for whole heart segmentation of mri. Med. Image Anal. 31, 77–87 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Laboratory of Image Science and TechnologySoutheast UniversityNanjingChina
  2. 2.Centre de Recherche en Information Biomédicale Sino-Français (CRIBs)RennesFrance
  3. 3.Univ Rennes, Inserm, LTSI - UMR1099RennesFrance
  4. 4.Department of Medical BiophysicsUniversity of Western OntarioLondonCanada

Personalised recommendations