Skip to main content
Log in

MVPCL: multi-view prototype consistency learning for semi-supervised medical image segmentation

  • Research
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Semi-supervised learning (SSL) methods show their powerful performance in dealing with the issue of data shortage in the field of medical image segmentation. However, most existing SSL methods neither fully utilize the multi-view information of medical images nor generate high-quality pseudo-labels to expand the training set. In this study, we propose a multi-view prototype consistency learning (MVPCL) framework for semi-supervised medical image segmentation. Specifically, the multi-view encoder and cross-view attention mechanism are employed to obtain high-level latent features of the original volumes, and the consistency constraints on the predictions of multi-view inputs enhance the performance of the networks. Moreover, the fused prototype representations are learned from the entire dataset by adopting an entropy-based uncertainty map. Eventually, the consistency constraints on prototype-based predictions result in a more representative prototype for each class, thereby optimizing the embedding space distribution. Substantial experimental results on three public benchmark datasets, including LiTS, LA, and ACDC, demonstrate the efficacy of the proposed method compared to the existing approaches. The source code is available at https://github.com/lixiafan/MVPCL-master.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability statement

The data that support the findings of this study are openly available in: https://www.kaggle.com/datasets/andrewmvd/liver-tumor-segmentationhttps://www.cardiacatlas.org/atriaseg2018-challenge/atria-seg-data/https://aistudio.baidu.com/datasetdetail/136629.

References

  1. Wang, Y., Wang, H., Shen, Y., Fei, J., Li, W., Jin, G., Wu, L., Zhao, R., Le, X.: Semi-supervised semantic segmentation using unreliable pseudo-labels. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18–24, 2022, pp. 4238–4247. IEEE (2022)

  2. Wu, L., Fang, L., He, X., He, M., Ma, J., Zhong, Z.: Querying labeled for unlabeled: cross-image semantic consistency guided semi-supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 45(7), 8827–8844 (2023)

    Google Scholar 

  3. Xu, Z., Wang, Y., Lu, D., Yu, L., Yan, J., Luo, J., Ma, K., Zheng, Y., Tong, R.K.: All-around real label supervision: cyclic prototype consistency learning for semi-supervised medical image segmentation. IEEE J. Biomed. Health Informatics 26(7), 3174–3184 (2022)

    Article  Google Scholar 

  4. Cheema, M.N., Nazir, A., Yang, P., Sheng, B., Li, P., Li, H., Wei, X., Qin, J., Kim, J., Feng, D.D.: Modified GAN-CAED to minimize risk of unintentional liver major vessels cutting by controlled segmentation using CTA/SPET-CT. IEEE Trans. Ind. Inform. 17(12), 7991–8002 (2021)

    Article  Google Scholar 

  5. Nazir, A., Cheema, M.N., Sheng, B., Li, H., Li, P., Yang, P., Jung, Y., Qin, J., Kim, J., Feng, D.D.: Off-enet: An optimally fused fully end-to-end network for automatic dense volumetric 3d intracranial blood vessels segmentation. IEEE Trans. Image Process. 29, 7192–7202 (2020)

    Article  Google Scholar 

  6. Li, J., Zhang, P., Wang, T., Zhu, L., Liu, R., Yang, X., Wang, K., Shen, D., Sheng, B.: Dsmt-net: dual self-supervised multi-operator transformation for multi-source endoscopic ultrasound diagnosis. IEEE Trans. Med. Imaging 43(1), 64–75 (2024)

    Article  Google Scholar 

  7. Al-Jebrni, A.H., Ali, S.G., Li, H., Lin, X., Li, P., Jung, Y., Kim, J., Feng, D.D., Sheng, B., Jiang, L., Du, J.: Sthy-net: a feature fusion-enhanced dense-branched modules network for small thyroid nodule classification from ultrasound images. Vis. Comput. 39(8), 3675–3689 (2023)

    Article  Google Scholar 

  8. Wang, D., Hu, G., Lyu, C.: Frnet: an end-to-end feature refinement neural network for medical image segmentation. Vis. Comput. 37(5), 1101–1112 (2021)

    Article  Google Scholar 

  9. Li, Y., Wang, Z., Yin, L., Zhu, Z., Qi, G., Liu, Y.: X-net: a dual encoding-decoding method in medical image segmentation. Vis. Comput. 39(6), 2223–2233 (2023)

    Article  Google Scholar 

  10. Qian, L., Huang, H., Xia, X., Li, Y., Zhou, X.: Automatic segmentation method using FCN with multi-scale dilated convolution for medical ultrasound image. Vis. Comput. 39(11), 5953–5969 (2023)

    Article  Google Scholar 

  11. Oliver, A., Odena, A., Raffel, C., Cubuk, E.D., Goodfellow, I.J.: Realistic evaluation of deep semi-supervised learning algorithms. In: Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3–8, 2018, Montréal, Canada, pp. 3239–3250 (2018)

  12. Dong, N., Xing, E.P.: Few-shot semantic segmentation with prototype learning. In: British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3–6, 2018, p. 79. BMVA Press (2018)

  13. Wang, K., Liew, J. H., Zou, Y., Zhou, D., Feng, J.: Panet: few-shot image semantic segmentation with prototype alignment. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27–November 2, 2019, pp. 9196–9205. IEEE (2019)

  14. Li, G., Jampani, V., Sevilla-Lara, L., Sun, D., Kim, J., Kim, J.: Adaptive prototype learning and allocation for few-shot segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19–25, 2021, pp. 8334–8343. Computer Vision Foundation/IEEE (2021)

  15. Lu, W., Lei, J., Qiu, P., Sheng, R., Zhou, J., Lu, X., Yang, Y.: Upcol: uncertainty-informed prototype consistency learning for semi-supervised medical image segmentation. In: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T.F., Taylor, R.H. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2023 - 26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part IV, vol. 14223 of Lecture Notes in Computer Science, pp. 662–672. Springer (2023)

  16. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: Transunet: transformers make strong encoders for medical image segmentation. CoRR (2021). arXiv:2102.04306

  17. Gao, Y., Zhou, M., Metaxas, D.N.: Utnet: a hybrid transformer architecture for medical image segmentation. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.)Medical Image Computing and Computer Assisted Intervention - MICCAI 2021—24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III, vol. 12903 of Lecture Notes in Computer Science, pp. 61–71. Springer (2021)

  18. Ji, Y., Zhang, R., Wang, H., Li, Z., Wu, L., Zhang, S., Luo, P.: Multi-compound transformer for accurate biomedical image segmentation. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2021-24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I, vol. 12901 of Lecture Notes in Computer Science, pp. 326–336. Springer (2021)

  19. Yu, L., Wang, S., Li, X., Fu, C., Heng, P.: Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P., Khan, A.R. (eds.) Medical Image Computing and Computer Assisted Intervention—MICCAI 2019-22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II, vol. 11765 of Lecture Notes in Computer Science, pp. 605–613. Springer (2019)

  20. Li, S., Zhang, C., He, X.: “Shape-aware semi-supervised 3d semantic segmentation for medical images,” in Medical Image Computing and Computer Assisted Intervention - MICCAI 2020 - 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part I (A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, and L. Joskowicz, eds.), vol. 12261 of Lecture Notes in Computer Science, pp. 552–561, Springer, (2020)

  21. Luo, X., Chen, J., Song, T., Wang, G.: “Semi-supervised medical image segmentation through dual-task consistency,” in Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 8801–8809, AAAI Press, 2021

  22. Wu, Y., Xu, M., Ge, Z., Cai, J., Zhang, L.: “Semi-supervised left atrium segmentation with mutual consistency training,” in Medical Image Computing and Computer Assisted Intervention - MICCAI 2021 - 24th International Conference, Strasbourg, France, September 27 - October 1, 2021, Proceedings, Part II (M. de Bruijne, P. C. Cattin, S. Cotin, N. Padoy, S. Speidel, Y. Zheng, and C. Essert, eds.), vol. 12902 of Lecture Notes in Computer Science, pp. 297–306, Springer, 2021

  23. Gao, S., Zhang, Z., Ma, J., Li, Z., Zhang, S.: Correlation-aware mutual learning for semi-supervised medical image segmentation. In: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T.F., Taylor, R.H. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2023-26th International Conference, Vancouver, BC, Canada, October 8–12, 2023, Proceedings, Part I, vol. 14220 of Lecture Notes in Computer Science, pp. 98–108. Springer (2023)

  24. Bai, Y., Chen, D., Li, Q., Shen, W., Wang, Y.: Bidirectional copy-paste for semi-supervised medical image segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17–24, 2023, pp. 11514–11524. IEEE (2023)

  25. Bilic, P., Christ, P.F., Li, H., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Adi Szeskin, et al.: The liver tumor segmentation benchmark (lits). Med. Image Anal. 84, 102680 (2023)

  26. Xiong, Z., Xia, Q., Hu, Z., Huang, N., Bian, C., Zheng, Y., Vesal, S., Ravikumar, N., Maier, A.K., Yang, X., Heng, P., Ni, D., Li, C., Tong, Q., Si, W., Puybareau, É., Khoudli, Y., Géraud, T., Zhao, J.: A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging. Med. Image Anal. 67, 101832 (2021)

    Article  Google Scholar 

  27. Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P., Cetin, I., Lekadir, K., Camara, O., Ballester, M.Á.G., Sanroma, G., Napel, S., Petersen, S.E., Tziritas, G., Grinias, E., Khened, M., Varghese, A., Krishnamurthi, G., Rohé, M., Pennec, X., Sermesant, M., Isensee, F., Jaeger, P., Maier-Hein, K.H., Full, P.M., Wolf, I., Engelhardt, S., Baumgartner, C.F., Koch, L.M., Wolterink, J.M., Isgum, I., Jang, Y., Hong, Y., Patravali, J., Jain, S., Humbert, O., Jodoin, P.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018)

    Article  Google Scholar 

  28. Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Workshop Track Proceedings. OpenReview.net (2017)

  29. Chen, J., Zhang, J., Debattista, K., Han, J.: Semi-supervised unpaired medical image segmentation through task-affinity consistency. IEEE Trans. Med. Imaging 42(3), 594–605 (2023)

    Article  Google Scholar 

  30. Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net (2017)

  31. Miyato, T., Maeda, S., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1979–1993 (2019)

    Article  Google Scholar 

  32. Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation. Medical Image Anal. 87, 102792 (2023)

    Article  Google Scholar 

  33. Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., Chen, D.Z.: Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: Descoteaux, M., Maier-Hein, L., Franz, A.M., Jannin, P., Collins, D.L., Duchesne, S. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2017 - 20th International Conference, Quebec City, QC, Canada, September 11–13, 2017, Proceedings, Part III, vol. 10435 of Lecture Notes in Computer Science, pp. 408–416. Springer (2017)

  34. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021, pp. 9992–10002. IEEE (2021)

  35. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18–24, 2022, pp. 15979–15988. IEEE (2022)

  36. Wang, Y., Li, Z., Mei, J., Wei, Z., Liu, L., Wang, C., Sang, S., Yuille, A.L., Xie, C., Zhou, Y.: Swinmm: masked multi-view with swin transformers for 3d medical image segmentation. In: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T.F., Taylor, R.H. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2023—26th International Conference, Vancouver, BC, Canada, October 8-12, 2023, Proceedings, Part III, vol. 14222 of Lecture Notes in Computer Science, pp. 486–496. Springer (2023)

  37. Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22–29, 2017, pp. 2999–3007. IEEE Computer Society (2017)

  38. Milletari, F., Navab, N., Ahmadi, S.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: Fourth International Conference on 3D Vision, 3DV 2016, Stanford, CA, USA, October 25–28, 2016, pp. 565–571. IEEE Computer Society (2016)

  39. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J.,W.M.W. III, Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015-18th International Conference Munich, Germany, October 5–9, 2015, Proceedings, Part III, vol. 9351 of Lecture Notes in Computer Science, pp. 234–241. Springer (2015)

  40. Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18–24, 2022, pp. 11966–11976. IEEE (2022)

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hongyan Quan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Quan, H. MVPCL: multi-view prototype consistency learning for semi-supervised medical image segmentation. Vis Comput (2024). https://doi.org/10.1007/s00371-024-03497-x

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-024-03497-x

Keywords

Navigation