Abstract
The unsupervised cross-modality image segmentation has gained much attention. Many methods attempt to align different modalities via adversarial learning. Recently, self-training with pseudo labels for the unsupervised target modality has also been widely used and achieved very promising results. The pseudo labels are usually obtained by selecting reliable predictions whose highest predicted probability is larger than an empirically set value. Such pseudo label generation inevitably has noise and training a segmentation model using incorrect pseudo labels could yield nontrivial errors for the target modality. In this paper, we propose a confidence-weighted mutual supervision on dual networks for unsupervised cross-modality image segmentation. Specifically, we independently initialize two networks with the same architecture, and propose a novel confidence-weighted Dice loss to mutually supervise the two networks using their predicted results for unlabeled data. In this way, we make full use of all predictions of unlabeled images and leverage the prediction confidence to alleviate the negative impact of noisy pseudo labels. Extensive experiments on three widely-used unsupervised cross-modality image segmentation datasets (i.e., MM-WHS 2017, Brats 2018, and Multi-organ segmentation) demonstrate that the proposed method achieves superior performance to some state-of-the-art methods.
Similar content being viewed by others
References
Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. 3431–3440
Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015. 234–241
Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2881–2890
Chen L C, Papandreou G, Kokkinos I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell, 2017, 40: 834–848
Wang J H, Liu B, Xu K. Semantic segmentation of high-resolution images. Sci China Inf Sci, 2017, 60: 123101
Geng Q C, Zhou Z, Cao X C. Survey of recent progress in semantic image segmentation with CNNs. Sci China Inf Sci, 2018, 61: 051101
Ma S, Pang Y W, Pan J, et al. Preserving details in semantics-aware context for scene parsing. Sci China Inf Sci, 2020, 63: 120106
Zhang Z J, Pang Y W. CGNet: cross-guidance network for semantic segmentation. Sci China Inf Sci, 2020, 63: 120104
Isensee F, Jaeger P F, Kohl S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods, 2021, 18: 203–211
Xu Q, Xi X M, Meng X J, et al. Difficulty-aware bi-network with spatial attention constrained graph for axillary lymph node segmentation. Sci China Inf Sci, 2022, 65: 192102
Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), 2017. 2223–2232
Zou D, Zhu Q, Yan P. Unsupervised domain adaptation with dual-scheme fusion network for medical image segmentation. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, 2020. 3291–3298
Chen J, Zhang Z, Xie X, et al. Beyond mutual information: generative adversarial network for domain adaptation using information bottleneck constraint. IEEE Trans Med Imag, 2021, 41: 595–607
Chen X, Lian C, Wang L, et al. Anatomy-regularized representation learning for cross-modality medical image segmentation. IEEE Trans Med Imag, 2020, 40: 274–285
Liu J, Liu H, Gong S, et al. Automated cardiac segmentation of cross-modal medical images using unsupervised multi-domain adaptation and spatial neural attention structure. Med Image Anal, 2021, 72: 102135
Jafari M, Francis S, Garibaldi J M, et al. LMISA: a lightweight multi-modality image segmentation network via domain adaptation using gradient magnitude and shape constraint. Med Image Anal, 2022, 81: 102536
Xie Q, Li Y, He N, et al. Unsupervised domain adaptation for medical image segmentation by disentanglement learning and self-training. IEEE Trans Med Imag, 2022. doi: https://doi.org/10.1109/TMI.2022.3192303
Zhou W, Wang Y, Chu J, et al. Affinity space adaptation for semantic segmentation across domains. IEEE Trans Image Process, 2020, 30: 2549–2561
Toldo M, Maracani A, Michieli U, et al. Unsupervised domain adaptation in semantic segmentation: a review. Technologies, 2020, 8: 35
French G, Mackiewicz M, Fisher M. Self-ensembling for visual domain adaptation. In: Proceedings of International Conference On Learning Representations, 2018
Li J, Zhou K, Qian S H, et al. Feature re-representation and reliable pseudo label retraining for cross-domain semantic segmentation. IEEE Trans Pattern Anal Mach Intell, 2022. doi: https://doi.org/10.1109/TPAMI.2022.3154933
Tranheden W, Olsson V, Pinto J, et al. DACS: domain adaptation via cross-domain mixed sampling. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), 2021. 1379–1389
Chen M, Xue H, Cai D. Domain adaptation for semantic segmentation with maximum squares loss. In: Proceedings of International Conference on Computer Vision, 2019. 2090–2099
Spadotto T, Toldo M, Michieli U, et al. Unsupervised domain adaptation with multiple domain discriminators and adaptive self-training. In: Proceedings of International Conference on Pattern Recognition, 2021. 2845–2852
Zou Y, Yu Z, Kumar B, et al. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of European Conference on Computer Vision, 2018. 289–305
Han B, Yao Q, Yu X, et al. Co-teaching: robust training of deep neural networks with extremely noisy labels. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018. 8536–8546
Ouali Y, Hudelot C, Tami M. Semi-supervised semantic segmentation with cross-consistency training. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 12674–12684
Sohn K, Berthelot D, Carlini N, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence. In: Proceedings of Advances in Neural Information Processing Systems, 2020. 33: 596–608
Liu S, Niles-Weed J, Razavian N, et al. Early-learning regularization prevents memorization of noisy labels. In: Proceedings of Advances in Neural Information Processing Systems, 2020. 33: 20331–20342
Liu S, Liu K, Zhu W, et al. Adaptive early-learning correction for segmentation from noisy annotations. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2606–2616
Zhang Y, Xiang T, Hospedales T M, et al. Deep mutual learning. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4320–4328
Chen X, Yuan Y, Zeng G, et al. Semi-supervised semantic segmentation with cross pseudo supervision. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2613–2622
Yang Y, Soatto S. FDA: Fourier domain adaptation for semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 4085–4095
Chen X, Kuang T, Deng H, et al. Dual adversarial attention mechanism for unsupervised domain adaptive medical image segmentation. IEEE Trans Med Imag, 2022, 41: 3445–3453
Vu T H, Jain H, Bucher M, et al. Advent: adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2517–2526
Truong T D, Duong C N, Le N, et al. BiMaL: Bijective maximum likelihood approach to domain adaptation in semantic scene segmentation. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 8548–8557
Han X, Qi L, Yu Q, et al. Deep symmetric adaptation network for cross-modality medical image segmentation. IEEE Trans Med Imag, 2021, 41: 121–132
Li C, Luo X, Chen W, et al. AttENT: domain-adaptive medical image segmentation via attention-aware translation and adversarial entropy minimization. In: Proceedings of IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2021. 952–959
Chen C, Dou Q, Chen H, et al. Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation. IEEE Trans Med Imag, 2020, 39: 2494–2505
Liu X, Xing F, Fakhri G E, et al. Self-semantic contour adaptation for cross modality brain tumor segmentation. In: Proceedings of IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2022. 1–5
Zeng G, Lerch T D, Schmaranzer F, et al. Semantic consistent unsupervised domain adaptation for cross-modality medical image segmentation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021. 201–210
Choi J, Kim T, Kim C. Self-ensembling with GAN-based data augmentation for domain adaptation in semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 6830–6840
Liu Z, Zhu Z, Zheng S, et al. Margin preserving self-paced contrastive learning towards domain adaptation for medical image segmentation. IEEE J Biomed Health Inform, 2022, 26: 638–647
Fan J, Gao B, Jin H, et al. UCC: uncertainty guided cross-head co-training for semi-supervised semantic segmentation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 9947–9956
Wang Y, Wang H, Shen Y, et al. Semi-supervised semantic segmentation using unreliable pseudo-labels. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4248–4257
Gu X, Sun J, Xu Z. Unsupervised and semi-supervised robust spherical space domain adaptation. IEEE Trans Pattern Anal Mach Intell, 2022. doi: https://doi.org/10.1109/TPAMI.2022.3158637
He N J, Fang L Y, Plaza A. Hybrid first and second order attention Unet for building segmentation in remote sensing images. Sci China Inf Sci, 2020, 63: 140305
Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. UNet+-+: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imag, 2020, 39: 1856–1867
Gut D, Tabor Z, Szymkowski M, et al. Benchmarking of deep architectures for segmentation of medical images. IEEE Trans Med Imag, 2022, 41: 3231–3241
Zhuang X, Shen J. Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal, 2016, 31: 77–87
Menze B H, Jakab A, Bauer S, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imag, 2014, 34: 1993–2024
Kavur A E, Gezer N S, Barış M, et al. CHAOS Challenge-combined (CT-MR) healthy abdominal organ segmentation. Med Image Anal, 2021, 69: 101950
Landman B, Xu Z, Igelsias J, et al. MICCAI multi-atlas labeling beyond the cranial vault-workshop and challenge. In: Proceedings of MICCAI Workshop, 2015. 12
He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 770–778
Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1125–1134
Kingma D P, Ba J. Adam: a method for stochastic optimization. In: Proceedings of International Conference on Learning Representations, 2015
Jiang J, Hu Y C, Tyagi N, et al. PSIGAN: joint probabilistic segmentation and image distribution matching for unpaired cross-modality adaptation-based MRI segmentation. IEEE Trans Med Imag, 2020, 39: 4071–4084
Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 248–255
Acknowledgements
This work was supported by National Natural Science Foundation of China (Grant Nos. 62061160490, 62122029, U20B2064).
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Chen, Y., Yang, X. & Bai, X. Confidence-weighted mutual supervision on dual networks for unsupervised cross-modality image segmentation. Sci. China Inf. Sci. 66, 210104 (2023). https://doi.org/10.1007/s11432-022-3871-0
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11432-022-3871-0