Skip to main content
SpringerLink
Log in

Self-training and Multi-level Adversarial Network for Domain Adaptive Remote Sensing Image Segmentation

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Unsupervised domain adaptive (UDA) image segmentation has received more and more attention in recent years. Domain adaptive methods can align the features of data in different domains, so that segmentation models can be migrated to data in other domains without incurring additional labeling costs. The traditional adversarial training uses the global alignment strategy to align the feature space, which may cause some categories to be incorrectly mapped. At the same time, in high-spatial resolution remote sensing images (RSI), the same category from different scenes (such as urban and rural areas) may have completely different distributions, which severely limits the accuracy of UDA. In order to solve these problems, in this paper: (1) a multi-level adversarial network at category-level is proposed, aiming at integrating feature information in different dimensions, studying the joint distribution at category-level, and aligning each category with adaptive adversarial loss in different dimensional spaces. (2) Use covariance regularization to optimize self-training. A method combining self-training with adversarial training is proposed, optimizes the domain adaptation effect, reduces the negative impact of false pseudo-label iteration caused by self-training. We demonstrated the latest performance of semantic segmentation on challenging LoveDA datasets. Experiments on “urban-to-rural” and “rural-to-urban” show that our method has better performance than the most advanced methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The datasets analysed during the current study are available in the zenodo repository, https://zenodo.org/record/5706578.

References

  1. Inglada J (2007) Automatic recognition of man-made objects in high resolution optical remote sensing images by svm classification of geometric image features. Isprs J Photogramm Remote Sens 62(3):236–248

    Article  Google Scholar 

  2. Maloof MA, Langley P, Binford TO et al (2003) Improved rooftop detection in aerial images with machine learning. Mach Learn 53(1–2):157–191

    Article  Google Scholar 

  3. Pal SK, Ghosh A, Shankar BU (2000) Segmentation of remotely sensed images with fuzzy thresholding, and quantitative evaluation. Int J Remote Sens 21(11):2269–2300

    Article  Google Scholar 

  4. Sirmaek B, Unsalan C (2009) Urban-area and building detection using sift keypoints and graph theory. IEEE Trans Geosci Remote Sens 47(4):1156–1167

    Article  Google Scholar 

  5. Trias-Sanz R, Stamon G, Louchet J (2008) Using colour, texture, and hierarchial segmentation for high-resolution remote sensing. Isprs J Photogramm Remote Sens 63(2):156–168

    Article  Google Scholar 

  6. Turker M, Koc-San D (2015) Building extraction from high-resolution optical spaceborne images using the integration of support vector machine (svm) classification, hough transformation and perceptual grouping. Int J Appl Earth Obs Geoinf 34:58–69

    Google Scholar 

  7. Deng Z, Sun H, Zhou S et al (2018) Multi-scale object detection in remote sensing imagery with convolutional neural networks. ISPRS J Photogramm Remote Sens 145:3–22. https://doi.org/10.1016/j.isprsjprs.2018.04.003

    Article  Google Scholar 

  8. Xia G, Bai X, Ding J, et al (2018) DOTA: a large-scale dataset for object detection in aerial images. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018. Computer Vision Foundation/IEEE Computer Society, pp 3974–3983

  9. Zheng Z, Zhong Y, Wang J, et al (2020) Foreground-aware relation network for geospatial object segmentation in high spatial resolution remote sensing imagery. In: 2020 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation/IEEE, pp 4095–4104, https://doi.org/10.1109/CVPR42600.2020.00415, https://openaccess.thecvf.com/content_CVPR_2020/html/Zheng_Foreground-Aware_Relation_Network_for_Geospatial_Object_Segmentation_in_High_Spatial_CVPR_2020_paper.html

  10. Pang J, Li C, Shi J et al (2019) \(\mathscr {R}\)\({}^{\text{2 }}\)-cnn: fast tiny object detection in large-scale remote sensing images. IEEE Trans Geosci Remote Sens 57(8):5512–5524. https://doi.org/10.1109/TGRS.2019.2899955

    Article  Google Scholar 

  11. Deng Z, Sun H, Zhou S et al (2019) Learning deep ship detector in SAR images from scratch. IEEE Trans Geosci Remote Sens 57(6):4021–4039. https://doi.org/10.1109/TGRS.2018.2889353

    Article  Google Scholar 

  12. Chen L, Yang Y, Wang J, et al (2016) Attention to scale: Scale-aware semantic image segmentation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016. IEEE Computer Society, pp 3640–3649

  13. Chen LC, Papandreou G, Kokkinos I et al (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  14. Chen L, Papandreou G, Schroff F, et al (2017) Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587

  15. Chen L, Zhu Y, Papandreou G, et al (2018a) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari V, Hebert M, Sminchisescu C, et al (eds) Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, Lecture Notes in Computer Science, vol 11211. Springer, pp 833–851

  16. Yang M, Yu K, Zhang C, et al (2018) Denseaspp for semantic segmentation in street scenes. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018. Computer Vision Foundation/IEEE Computer Society, pp 3684–3692

  17. Wang J, Zheng Z, Ma A, et al (2021a) LoveDA: a remote sensing land-cover dataset for domain adaptive semantic segmentation. In: Vanschoren J, Yeung S (eds) Proceedings of the neural information processing systems track on datasets and benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021

  18. Liu W, Luo Z, Cai Y et al (2021) Adversarial unsupervised domain adaptation for 3d semantic segmentation with multi-modal learning. ISPRS J Photogramm Remote Sens 176:211–221. https://doi.org/10.1016/j.isprsjprs.2021.04.012

    Article  Google Scholar 

  19. Huang L, Fu Q, He M et al (2021) Detection algorithm of safety helmet wearing based on deep learning. Concurr Comput Pract Exp 33(13):e6234. https://doi.org/10.1002/cpe.6234

    Article  Google Scholar 

  20. Huang L, Chen C, Yun J et al (2022) Multi-scale feature fusion convolutional neural network for indoor small target detection. Front Neurorobotics 16:881021. https://doi.org/10.3389/fnbot.2022.881021

    Article  Google Scholar 

  21. Jiang D, Li G, Tan C et al (2021) Semantic segmentation for multiscale target based on object recognition using the improved faster-rcnn model. Future Gener Comput Syst 123:94–104. https://doi.org/10.1016/j.future.2021.04.019

    Article  Google Scholar 

  22. Sun Y, Zhao Z, Jiang D et al (2022) Low-illumination image enhancement algorithm based on improved multi-scale retinex and abc algorithm optimization. Front Bioeng Biotechnol 10:865820. https://doi.org/10.3389/fbioe.2022.865820

    Article  Google Scholar 

  23. Yun J, Jiang D, Sun Y et al (2022) Grasping pose detection for loose stacked object based on convolutional neural network with multiple self-powered sensors information. IEEE Sens J. https://doi.org/10.1109/JSEN.2022.3190560

    Article  Google Scholar 

  24. Liu Y, Jiang D, Xu C et al (2022) Deep learning based 3d target detection for indoor scenes. Appl Intell 53(9):10218–10231

    Article  Google Scholar 

  25. Jiang D, Li G, Sun Y et al (2021) Manipulator grabbing position detection with information fusion of color image and depth image using deep learning. J Amb Intell Hum Comput 12(12):10809–10822

    Article  Google Scholar 

  26. Liu Y, Jiang D, Duan H et al (2021) Dynamic gesture recognition algorithm based on 3d convolutional neural network. Comput Intell Neurosci 12:1–12

    Google Scholar 

  27. Zhou ZH, Li M (2005) Tri-training: exploiting unlabeled data using three classifiers. IEEE Trans Knowl Data Eng 17(11):1529–1541

    Article  Google Scholar 

  28. Goodfellow I, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial nets. In: Neural Information Processing Systems, pp 2672–2680

  29. Yu W, Bai J, Jiao L (2020) Background subtraction based on gan and domain adaptation for vhr optical remote sensing videos. IEEE Access 8:119144–119157. https://doi.org/10.1109/ACCESS.2020.3004495

    Article  Google Scholar 

  30. Li X, Du Z, Huang Y et al (2021) A deep translation (gan) based change detection network for optical and sar remote sensing images. ISPRS J Photogramm Remote Sens 179:14–34. https://doi.org/10.1016/j.isprsjprs.2021.07.007

    Article  Google Scholar 

  31. Hoffman J, Tzeng E, Park T, et al (2018) Cycada: Cycle-consistent adversarial domain adaptation. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, Proceedings of Machine Learning Research, vol 80. PMLR, pp 1994–2003

  32. Tsai YH, Hung WC, Schulter S, et al (2018) Learning to adapt structured output space for semantic segmentation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR) pp 7472–7481

  33. Luo Y, Zheng L, Guan T, et al (2019) Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp 2507–2516

  34. Wang H, Shen T, Zhang W, et al (2020) Classes matter: a fine-grained adversarial approach to cross-domain semantic segmentation. In: The European conference on computer vision (ECCV), pp 642–659

  35. Wang X, Jin Y, Long M, et al (2019) Transferable normalization: towards improving transferability of deep neural networks. In: Neural information processing systems, pp 1951–1961

  36. Zhao Y, Zhong Z, Zhao N, et al (2022) Style-hallucinated dual consistency learning for domain generalized semantic segmentation. In: Computer Vision—ECCV 2022—17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, Lecture Notes in Computer Science, vol 13688. Springer, pp 535–552

  37. Huang J, Guan D, Xiao A et al (2022) Multi-level adversarial network for domain adaptive semantic segmentation. Pattern Recognit 123:108384. https://doi.org/10.1016/j.patcog.2021.108384

    Article  Google Scholar 

  38. Ning M, Lu D, Wei D, et al (2021) Multi-anchor active domain adaptation for semantic segmentation. In: 2021 IEEE/CVF international conference on computer vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021. IEEE, pp 9092–9102

  39. Cheng Y, Wei F, Bao J, et al (2021) Dual path learning for domain adaptation of semantic segmentation. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021. IEEE, pp 9062–9071

  40. Lai X, Tian Z, Xu X, et al (2022) Decouplenet: decoupled network for domain adaptive semantic segmentation. In: Computer Vision—ECCV 2022—17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, Lecture Notes in Computer Science, vol 13693. Springer, pp 369–387

  41. Lian Q, Lv F, Duan L, et al (2019) Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: a non-adversarial approach. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE, pp 6757–6766

  42. Zou Y, Yu Z, Kumar BVKV, et al (2018) Domain adaptation for semantic segmentation via class-balanced self-training. arXiv:1810.07911

  43. Mei K, Zhu C, Zou J et al (2020) Instance adaptive self-training for unsupervised domain adaptation. In: Vedaldi A, Bischof H, Brox T et al (eds) Computer Vision–ECCV 2020, vol 12371. Springer, Cham, pp 415–430

    Chapter  Google Scholar 

  44. Liu Y, Zhang S, Li Y, et al (2021d) Learning to adapt via latent domains for adaptive semantic segmentation. In: Beygelzimer A, Dauphin Y, Liang P, et al (eds) Advances in neural information processing systems 34: annual conference on neural information processing systems 2021, NeurIPS 2021, December 6–14, 2021, virtual, pp 1167–1178

  45. Liu W, Liu J, Luo Z et al (2022) Weakly supervised high spatial resolution land cover mapping based on self-training with weighted pseudo-labels. Int J Appl Earth Obs Geoinf 112:102931. https://doi.org/10.1016/j.jag.2022.102931

    Article  Google Scholar 

  46. Hoyer L, Dai D, Van Gool L (2022b) HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. In: Computer Vision—ECCV 2022—17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX, Lecture Notes in Computer Science, vol 13690. Springer, pp 372–391

  47. Hoyer L, Dai D, Van Gool L (2022a) DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, pp 9914–9925

  48. Liu Y, Deng J, Gao X, et al (2021b) Bapa-net: boundary adaptation and prototype alignment for cross-domain semantic segmentation. In: 2021 IEEE/CVF international conference on computer vision, ICCV 2021, Montreal, QC, Canada, October 10–17, 2021. IEEE, pp 8781–8791

  49. Wang W, Ma L, Chen M et al (2021) Joint correlation alignment-based graph neural network for domain adaptation of multitemporal hyperspectral remote sensing images. IEEE J Sel Top Appl Earth Obs Remote Sens 14:3170–3184. https://doi.org/10.1109/JSTARS.2021.3063460

    Article  Google Scholar 

  50. Pan F, Shin I, Rameau F et al (2020) Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. IEEE conference on computer vision and pattern recoginition (CVPR). Computer Vision Foundation, IEEE, pp 3763–3772

    Google Scholar 

  51. Shen W, Wang Q, Jiang H, et al (2021) Unsupervised domain adaptation for semantic segmentation via self-supervision. In: IEEE international geoscience and remote sensing symposium, IGARSS 2021, Brussels, Belgium, July 11–16, 2021. IEEE, pp 2747–2750

  52. Deng X, Yang HL, Makkar N, et al (2019a) Large scale unsupervised domain adaptation of segmentation networks with adversarial learning. In: IGARSS 2019—2019 IEEE International Geoscience and Remote Sensing Symposium, pp 4955–4958

  53. Wu L, Lu M, Fang L (2022) Deep covariance alignment for domain adaptive remote sensing image segmentation. IEEE Trans Geosci Remote Sens 60:1–11. https://doi.org/10.1109/TGRS.2022.3163278

    Article  Google Scholar 

  54. Richter SR, Vineet V, Roth S et al (2016) Playing for data: Ground truth from computer games. In: Part II (ed) Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings. Springer, pp 102–118

    Chapter  Google Scholar 

  55. Cordts M, Omran M, Ramos S, et al (2016) The cityscapes dataset for semantic urban scene understanding. IEEE, pp 3213–3223

  56. He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: CVPR. IEEE Computer Society, pp 770–778

  57. Deng J, Dong W, Socher R et al (2009) Imagenet: a large-scale hierarchical image database. In: Florida USA (ed) 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20–25 June 2009, Miami. IEEE Computer Society, pp 248–255

    Google Scholar 

  58. Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions. In: 4th International conference on learning representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, conference track proceedings, arXiv:1511.07122

  59. Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. Comput Sci. arXiv:1511.06434

  60. Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. Int Conf Mach Learn 30(1):3

    Google Scholar 

  61. Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Lechevallier Y, Saporta G (eds) 19th International conference on computational statistics, COMPSTAT 2010, Paris, France, August 22–27, 2010–keynote, invited and contributed papers. Physica-Verlag, pp 177–186

    Google Scholar 

  62. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. Computer Science http://arxiv.org/abs/arXiv:1412.6980

  63. Maaten LV, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(2605):2579–2605

    MATH  Google Scholar 

  64. Bi X, Chen D, Huang H et al (2023) Combining pixel-level and structure-level adaptation for semantic segmentation. Neural Process Lett. https://doi.org/10.1007/s11063-023-11220-5

    Article  Google Scholar 

  65. Li W, Yang X, Li Z (2023) Mlcb-net: a multi-level class balancing network for domain adaptive semantic segmentation. Multimed Syst. https://doi.org/10.1007/s00530-023-01055-4

    Article  Google Scholar 

  66. Zhu S, Tian Y (2023) Shape robustness in style enhanced cross domain semantic segmentation. Pattern Recognit 135:109143. https://doi.org/10.1016/j.patcog.2022.109143

    Article  Google Scholar 

  67. Zhang Y, Tian S, Liao M et al (2023) A hybrid domain learning framework for unsupervised semantic segmentation. Neurocomputing 516:133–145. https://doi.org/10.1016/j.neucom.2022.10.005

    Article  Google Scholar 

Download references

Funding

The authors did not receive support from any organization for the submitted work. No funding was received to assist with the preparation of this manuscript. No funding was received for conducting this study. No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. School of Information Engineering, China Jiliang University, No. 258 Xueyuan Street, Hangzhou, 310018, Zhejiang, China

    Yilin Zheng, Lingmin He, Xiangping Wu & Chen Pan

Authors
  1. Yilin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingmin He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiangping Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chen Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yilin Zheng.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, Y., He, L., Wu, X. et al. Self-training and Multi-level Adversarial Network for Domain Adaptive Remote Sensing Image Segmentation. Neural Process Lett 55, 10613–10638 (2023). https://doi.org/10.1007/s11063-023-11341-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11341-x

Keywords

Search

Navigation