Skip to main content
SpringerLink
Log in

Incremental Learning Meets Transfer Learning: Application to Multi-site Prostate MRI Segmentation

  • Conference paper
  • First Online:
Distributed, Collaborative, and Federated Learning, and Affordable AI and Healthcare for Resource Diverse Global Health (DeCaF 2022, FAIR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13573))

Abstract

Many medical datasets have recently been created for medical image segmentation tasks, and it is natural to question whether we can use them to sequentially train a single model that (1) performs better on all these datasets, and (2) generalizes well and transfers better to the unknown target site domain. Prior works have achieved this goal by jointly training one model on multi-site datasets, which achieve competitive performance on average but such methods rely on the assumption about the availability of all training data, thus limiting its effectiveness in practical deployment. In this paper, we propose a novel multi-site segmentation framework called incremental-transfer learning (ITL), which learns a model from multi-site datasets in an end-to-end sequential fashion. Specifically, “incremental” refers to training sequentially constructed datasets, and “transfer” is achieved by leveraging useful information from the linear combination of embedding features on each dataset. In addition, we introduce our ITL framework, where we train the network including a site-agnostic encoder with pretrained weights and at most two segmentation decoder heads. We also design a novel site-level incremental loss in order to generalize well on the target domain. Second, we show for the first time that leveraging our ITL training scheme is able to alleviate challenging catastrophic forgetting problems in incremental learning. We conduct experiments using five challenging benchmark datasets to validate the effectiveness of our incremental-transfer learning approach. Our approach makes minimal assumptions on computation resources and domain-specific expertise, and hence constitutes a strong starting point in multi-site medical image segmentation.

C. You and J. Xiang—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aslani, S., Murino, V., Dayan, M., Tam, R., Sona, D., Hamarneh, G.: Scanner invariant multiple sclerosis lesion segmentation from MRI. In: ISBI. IEEE (2020)

    Google Scholar 

  2. Bloch, N., et al.: NCI-ISBI 2013 challenge: automated segmentation of prostate structures. Cancer Imaging Archive 370 (2015)

    Google Scholar 

  3. Chang, W.G., You, T., Seo, S., Kwak, S., Han, B.: Domain-specific batch normalization for unsupervised domain adaptation. In: CVPR (2019)

    Google Scholar 

  4. Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.: Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: ECCV (2018)

    Google Scholar 

  5. Davidson, G., Mozer, M.C.: Sequential mastery of multiple visual tasks: networks naturally learn to learn and forget to forget. In: CVPR (2020)

    Google Scholar 

  6. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR (2020)

    Google Scholar 

  7. Dou, Q., Liu, Q., Heng, P.A., Glocker, B.: Unpaired multi-modal segmentation via knowledge distillation. IEEE Trans. Med. Imaging 39(7), 2415–2425 (2020)

    Article  Google Scholar 

  8. Gibson, E., et al.: Inter-site variability in prostate segmentation accuracy using deep learning. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 506–514. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_58

    Chapter  Google Scholar 

  9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  10. Jia, H., Song, Y., Huang, H., Cai, W., Xia, Y.: HD-Net: hybrid discriminative network for prostate segmentation in MR images. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 110–118. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_13

    Chapter  Google Scholar 

  11. Karani, N., Chaitanya, K., Baumgartner, C., Konukoglu, E.: A lifelong learning approach to brain MR segmentation across scanners and protocols. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 476–484. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_54

    Chapter  Google Scholar 

  12. Lemaître, G., Martí, R., Freixenet, J., Vilanova, J.C., Walker, P.M., Meriaudeau, F.: Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric MRI: a review. CBM 60, 8–31 (2015)

    Google Scholar 

  13. Li, D., Zhang, J., Yang, Y., Liu, C., Song, Y.Z., Hospedales, T.M.: Episodic training for domain generalization. In: CVPR (2019)

    Google Scholar 

  14. Li, X., Yu, L., Chen, H., Fu, C.W., Heng, P.A.: Semi-supervised skin lesion segmentation via transformation consistent self-ensembling model. arXiv preprint arXiv:1808.03887 (2018)

  15. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)

    Article  Google Scholar 

  16. Litjens, G., et al.: Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. MIA 18(2), 359–373 (2014)

    Google Scholar 

  17. Liu, P., Xiao, L., Zhou, S.K.: Incremental learning for multi-organ segmentation with partially labeled datasets. In: MICCAI (2021)

    Google Scholar 

  18. Liu, Q., Dou, Q., Heng, P.-A.: Shape-aware meta-learning for generalizing prostate MRI segmentation to unseen domains. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 475–485. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_46

    Chapter  Google Scholar 

  19. Liu, Q., Dou, Q., Yu, L., Heng, P.A.: MS-net: multi-site network for improving prostate segmentation with heterogeneous MRI data. IEEE Trans. Med. Imaging 39(9), 2713–2724 (2020)

    Article  Google Scholar 

  20. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Elsevier (1989)

    Google Scholar 

  21. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 3DV, pp. 565–571. IEEE (2016)

    Google Scholar 

  22. Nie, D., Gao, Y., Wang, L., Shen, D.: ASDNet: attention based semi-supervised deep networks for medical image segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 370–378. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_43

    Chapter  Google Scholar 

  23. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: CVPR (2017)

    Google Scholar 

  24. Rundo, L., et al.: Use-net: incorporating squeeze-and-excitation blocks into u-net for prostate zonal segmentation of multi-institutional MRI datasets. Neurocomputing 365, 31–43 (2019)

    Article  Google Scholar 

  25. Rundo, L., et al.: CNN-based prostate zonal segmentation on T2-weighted MR images: a cross-dataset study. In: Esposito, A., Faundez-Zanuy, M., Morabito, F.C., Pasero, E. (eds.) Neural Approaches to Dynamics of Signal Exchanges. SIST, vol. 151, pp. 269–280. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-8950-4_25

    Chapter  Google Scholar 

  26. Shi, G., Xiao, L., Chen, Y., Zhou, S.K.: Marginal loss and exclusion loss for partially supervised multi-organ segmentation. Med. Image Anal. 70, 101979 (2021)

    Article  Google Scholar 

  27. Wu, Y., et al.: Large scale incremental learning. In: CVPR (2019)

    Google Scholar 

  28. Xiang, J., Shlizerman, E.: TKIL: tangent kernel approach for class balanced incremental learning. arXiv preprint arXiv:2206.08492 (2022)

  29. Yang, L., et al.: NuSeT: a deep learning tool for reliably separating and analyzing crowded cells. PLoS Comput. Biol. 16(9), e1008193 (2020)

    Article  Google Scholar 

  30. Yao, Q., Xiao, L., Liu, P., Zhou, S.K.: Label-free segmentation of COVID-19 lesions in lung CT. IEEE Trans. Med. Imaging 40(10), 2808–2819 (2021)

    Article  Google Scholar 

  31. You, C., Dai, W., Staib, L., Duncan, J.S.: Bootstrapping semi-supervised medical image segmentation with anatomical-aware contrastive distillation. arXiv preprint arXiv:2206.02307 (2022)

  32. You, C., Yang, J., Chapiro, J., Duncan, J.S.: Unsupervised Wasserstein distance guided domain adaptation for 3D multi-domain liver segmentation. In: Cardoso, J., et al. (eds.) IMIMIC/MIL3ID/LABELS -2020. LNCS, vol. 12446, pp. 155–163. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61166-8_17

    Chapter  Google Scholar 

  33. You, C., et al.: Class-aware generative adversarial transformers for medical image segmentation. arXiv preprint arXiv:2201.10737 (2022)

  34. You, C., Zhao, R., Staib, L., Duncan, J.S.: Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. arXiv preprint arXiv:2105.07059 (2021)

  35. You, C., Zhou, Y., Zhao, R., Staib, L., Duncan, J.S.: SimCVD: simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation. IEEE Trans. Med. Imaging (2022)

    Google Scholar 

  36. Yu, L., Yang, X., Chen, H., Qin, J., Heng, P.A.: Volumetric convnets with mixed residual connections for automated prostate segmentation from 3D MR images. In: AAAI (2017)

    Google Scholar 

  37. Zhang, X., et al.: Automatic spinal cord segmentation from axial-view MRI slices using CNN with grayscale regularized active contour propagation. Comput. Biol. Med. 132, 104345 (2021)

    Article  Google Scholar 

  38. Zhang, X., Martin, D.G., Noga, M., Punithakumar, K.: Fully automated left atrial segmentation from MR image sequences using deep convolutional neural network and unscented Kalman filter. In: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 2316–2323. IEEE (2018)

    Google Scholar 

  39. Zhang, X., Noga, M., Martin, D.G., Punithakumar, K.: Fully automated left atrium segmentation from anatomical cine long-axis MRI sequences using deep convolutional neural network with unscented Kalman filter. Med. Image Anal. 68, 101916 (2021)

    Article  Google Scholar 

  40. Zhang, X., Noga, M., Punithakumar, K.: Fully automated deep learning based segmentation of normal, infarcted and edema regions from multiple cardiac MRI sequences. In: Zhuang, X., Li, L. (eds.) MyoPS 2020. LNCS, vol. 12554, pp. 82–91. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65651-5_8

    Chapter  Google Scholar 

  41. Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., Chen, D.Z.: Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 408–416. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_47

    Chapter  Google Scholar 

  42. Zheng, Y., Xiang, J., Su, K., Shlizerman, E.: BI-MAML: balanced incremental approach for meta learning. arXiv preprint arXiv:2006.07412 (2020)

  43. Zhou, S.K., et al.: A review of deep learning in medical imaging: imaging traits, technology trends, case studies with progress highlights, and future promises. Proc. IEEE 109(5), 820–838 (2021)

    Article  Google Scholar 

  44. Zhu, J., Li, Y., Hu, Y., Ma, K., Zhou, S.K., Zheng, Y.: Rubik’s Cube+: a self-supervised feature learning framework for 3D medical image analysis. Med. Image Anal. 64, 101746 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering, Yale University, New Haven, CT, USA

    Chenyu You, Siyuan Dong, Lawrence Staib & James S. Duncan

  2. Electrical and Computer Engineering, The University of Washington, WA, USA

    Jinlin Xiang & Kun Su

  3. Biomedical Engineering, Yale University, New Haven, CT, USA

    Xiaoran Zhang, Lawrence Staib & James S. Duncan

  4. Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA

    John Onofrey, Lawrence Staib & James S. Duncan

Authors
  1. Chenyu You
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinlin Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kun Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoran Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Siyuan Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. John Onofrey
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lawrence Staib
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. James S. Duncan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chenyu You .

Editor information

Editors and Affiliations

  1. University of Bonn, Bonn, Germany

    Shadi Albarqouni

  2. University of Pennsylvania, Philadelphia, PA, USA

    Spyridon Bakas

  3. University College London, London, UK

    Sophia Bano

  4. King’s College London, London, UK

    M. Jorge Cardoso

  5. NepAl Applied Mathematics and Informatics, Kathmandu, Nepal

    Bishesh Khanal

  6. Vanderbilt University, Brentwood, TN, USA

    Bennett Landman

  7. University of British Columbia, Vancouver, BC, Canada

    Xiaoxiao Li

  8. University of Edinburgh, Edinburgh, UK

    Chen Qin

  9. Istanbul Technical University, Istanbul, Turkey

    Islem Rekik

  10. NVIDIA GmbH, Munich, Bayern, Germany

    Nicola Rieke

  11. NVIDIA Corporation, Santa Clara, CA, USA

    Holger Roth

  12. Indian Institute of Technology, Kharagpur, India

    Debdoot Sheet

  13. NVIDIA Corporation, Santa Clara, CA, USA

    Daguang Xu

Appendix

Appendix

Fig. 2.
figure 2

Visualization of segmentation results on five benchmarks using ResNet-18 as the encoder. Different site results are shown in different colors.

Full size image
Table 4. Segmentation decoder head architecture
Full size table
Table 5. Comparison of different ordering strategies using ResNet-18. We report mean and standard deviation across three random trials. Note that a larger DSC (\(\uparrow \)) and a smaller 95HD (\(\downarrow \)) indicate better performing ITL models.
Full size table
Table 6. Comparison of different training strategies using ResNet-18. We report mean and standard deviation across three random trials.
Full size table
Table 7. Ablation of each component in the proposed ITL when using ResNet-18 under 5% exemplar portion. We report mean and standard deviation across three random trials. Note that a larger DSC (\(\uparrow \)) and a smaller 95HD (\(\downarrow \)) indicate better performing ITL models. The best results are in bold.
Full size table
Fig. 3.
figure 3

Comparison of training from scratch against using pretraining. We use ResNet-18 on ImageNet as the encoder. Under 5% exemplar portion, we plot (a) training loss (Scratch), (b) training loss (Pretraining), (c) validation loss (Scratch), (d) validation loss (Pretraining)

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

You, C. et al. (2022). Incremental Learning Meets Transfer Learning: Application to Multi-site Prostate MRI Segmentation. In: Albarqouni, S., et al. Distributed, Collaborative, and Federated Learning, and Affordable AI and Healthcare for Resource Diverse Global Health. DeCaF FAIR 2022 2022. Lecture Notes in Computer Science, vol 13573. Springer, Cham. https://doi.org/10.1007/978-3-031-18523-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18523-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18522-9

  • Online ISBN: 978-3-031-18523-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation