Skip to main content
SpringerLink
Log in

Explored seeds generation for weakly supervised semantic segmentation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Weakly supervised semantic segmentation with only image-level labels is an essential application since it reduces the considerable human effort to fully annotate image. Most state-of-the-art methods take into account discriminative regions or prior knowledge (called seeds) by applying the recent Classification Activation Maps (CAM) methods. As seeds used in the CAM methods could be considered as segmentation knowledge, increasing seeds will naturally improve the performance of the CAM methods. We treat such task as a seed exploration with clustering learning, which is different from the current methods in which region mining or seeds learning are performed in an end-to-end learning manner. In this paper, we propose a progressive framework containing autoencoders to explore numerous and accurate seeds to further improve the performance of CAM methods. Our explored seeds can replace the initial seeds, and thus, they can be integrated with current weakly supervised semantic segmentation methods for performance improvement. We show experimentally that our provided seeds lead to better model training. As a result, we obtain performance improvement from corresponding counterparts to reach 68.1 and 69.2% mIoU on PASCAL VOC 2012 validation and test dataset, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability statement

The images in this study were taken from the publicly dataset, The PASCAL Visual Object Classes Challenge 2012, http://host.robots.ox.ac.uk/pascal/VOC/voc2012/.

Notes

  1. The source code is available for reviewers and will be made public after publication http://www.yiminyang.com/weakly_supervised.html.

  2. https://github.com/js-fan/CIANl.

  3. https://github.com/halbielee/SEC_pytorchl.

  4. https://github.com/xtudbxk/DSRG-tensorflow.

References

  1. Dai J, He K, Sun J (2015) Boxsup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 1635–1643

  2. Lin D, Dai J, Jia J, He K, Sun J (2016) Scribblesup: scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3159–3167

  3. Bearman A, Russakovsky O, Ferrari V, Fei-Fei L (2016) What’s the point: semantic segmentation with point supervision. In: European conference on computer vision, pp 549–565, Springer

  4. Kolesnikov A, Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: European conference on computer vision, pp 695–711, Springer

  5. Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7014–7023

  6. Fan J, Zhang Z, Tan T, Song C, Xiao J (2020) Cian: cross-image affinity net for weakly supervised semantic segmentation. In: Proceedings of the AAAI conference on artificial intelligence 34:10762–10769

    Article  Google Scholar 

  7. Sun G, Wang W, Dai J, Van Gool L (2020) Mining cross-image semantics for weakly supervised semantic segmentation. In: European conference on computer vision, pp 347–365. Springer

  8. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929

  9. Singh KK, Lee YJ (2017) Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization. In: 2017 IEEE international conference on computer vision (ICCV), pp 3544–3553, IEEE

  10. Wei Y, Feng J, Liang X, Cheng M-M, Zhao Y, Yan S (2017) Object region mining with adversarial erasing: a simple classification to semantic segmentation approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1568–1576

  11. Li K, Wu Z, Peng K-C, Ernst J, Fu Y (2018) Tell me where to look: guided attention inference network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 9215–9223

  12. Chaudhry A, Dokania PK, Torr PH (2017) Discovering class-specific pixels for weakly-supervised semantic segmentation. arXiv preprint arXiv:1707.05821

  13. Hou Q, Jiang P-T, Wei Y, Cheng M-M (2018) Self-erasing network for integral object attention. arXiv preprint arXiv:1810.09821

  14. Zhang X, Wei Y, Feng J, Yang Y, Huang TS (2018) Adversarial complementary learning for weakly supervised object localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1325–1334

  15. Oh SJ, Benenson R, Khoreva A, Akata Z, Fritz M, Schiele B (2017) Exploiting saliency for object segmentation from image level labels. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5038–5047, IEEE

  16. Ahn J, Kwak S (2018) Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4981–4990

  17. Lee J, Choi J, Mok J, Yoon S (2021) Reducing information bottleneck for weakly supervised semantic segmentation. arXiv:2110.06530 [cs]

  18. Zhang F, Gu C, Zhang C, Dai Y (2021) Complementary patch for weakly supervised semantic segmentation. arXiv:2108.03852 [cs]

  19. Chen Q, Yang L, Lai J, Xie X (2022) Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation. arXiv:2203.02909 [cs]

  20. Chen Z, Wang T, Wu X, Hua X-S, Zhang H, Sun Q (2022) Class re-activation maps for weakly-supervised semantic segmentation. arXiv:2203.00962 [cs]

  21. Lee M, Kim D, Shim H (2022) Threshold matters in WSSS: manipulating the activation for the Robust and accurate segmentation model against thresholds. arXiv:2203.16045 [cs]

  22. Rumelhart DE, Hinton GE, Williams RJ (1985) Learning internal representations by error propagation. Tech. rep. California Univ San Diego La Jolla Inst for Cognitive Science

  23. Yang Y, Wang Y, Jonathan Wu QM, Lin X, Liu M (2015) Progressive learning machine: a new approach for general hybrid system approximation. IEEE Trans Neural Netw Learn Syst 26(9):1855–1874

    Article  MathSciNet  Google Scholar 

  24. MacQueen J et al (1967) Some methods for classification and analysis of multivariate observations. In: Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, vol 1, pp 281–297, Oakland, CA, USA

  25. Fukunaga K, Hostetler L (1975) The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans Inf Theory 21(1):32–40

    Article  MathSciNet  Google Scholar 

  26. Everingham M, Eslami SA, Van Gool L, Williams CK, Winn J, Zisserman A (2015) The pascal visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136

    Article  Google Scholar 

  27. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  28. Yang Y, Wu QJ, Wang Y (2016) Autoencoder with invertible functions for dimension reduction and image reconstruction. IEEE Trans Syst Man Cybernet Syst 48(7):1065–1079

    Article  Google Scholar 

  29. Miyato T, Maeda S-I, Koyama M, Ishii S (2018) Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans Pattern Anal Mach Intell 41(8):1979–1993

    Article  Google Scholar 

  30. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  31. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2014) Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062

  32. Hariharan B, Arbeláez P, Bourdev L, Maji S, Malik J (2011) Semantic contours from inverse detectors. In: 2011 International Conference on Computer Vision, pp 991–998, IEEE

  33. Chan L, Hosseini MS, Plataniotis KN (2021) A comprehensive analysis of weakly-supervised semantic segmentation in different image domains.Int J Comput Vis 129(2):361–384

    Article  Google Scholar 

  34. Lee J, Kim E, Lee S, Lee J, Yoon S (2019) Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5267–5276

  35. Xu L, Xue H, Bennamoun M, Boussaid F, Sohel F (2021) Atrous convolutional feature network for weakly supervised semantic segmentation. Neurocomputing 421:115–126

    Article  Google Scholar 

  36. Wei Y, Liang X, Chen Y, Shen X, Cheng M-M, Feng J, Zhao Y, Yan S (2016) Stc: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(11):2314–2320

    Article  Google Scholar 

  37. Hong S, Yeo D, Kwak S, Lee H, Han B (2017) Weakly supervised semantic segmentation using web-crawled videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7322–7330

  38. Pinheiro PO,  Collobert R (2015) From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1713–1721

  39. Papandreou G, Chen L-C, Murphy KP, Yuille AL (2015) Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 1742–1750

  40. Wang X, You S, Li X, Ma H (2018) Weakly-supervised semantic segmentation by iteratively mining common object features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1354–1362

  41. Liu J, Yu C, Yang B, Gao C, Sang N (2020) Csenet: cascade semantic erasing network for weakly-supervised semantic segmentation. Neurocomputing

  42. Li Y, Liu Y, Liu G, Guo M (2020) Weakly supervised semantic segmentation by iterative superpixel-crf refinement with initial clues guiding. Neurocomputing 391:25–41

    Article  Google Scholar 

  43. Wang Y, Zhang J, Kan M, Shan S, Chen X (2020) Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12275–12284

  44. Chang Y-T, Wang Q, Hung W-C, Piramuthu R, Tsai Y-H, Yang M-H (2020) Weakly-supervised semantic segmentation via sub-category exploration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8991–9000

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Lakehead University, Orillia, Canada

    Terence Chow

  2. Department of Electrical and Computer Engineering, Western University, London, Canada

    Haojin Deng & Yimin Yang

  3. Vector Institute, Toronto , Canada

    Yimin Yang

  4. School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Singapore

    Zhiping Lin

  5. Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangdong, China

    Huiping Zhuang

  6. Department of Computer Science, Mathematics, Physics and Statistics, The University of British Columbia, Kelowna, Canada

    Shan Du

Authors
  1. Terence Chow
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haojin Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yimin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiping Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Huiping Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shan Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiping Lin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chow, T., Deng, H., Yang, Y. et al. Explored seeds generation for weakly supervised semantic segmentation. Neural Comput & Applic 36, 1007–1022 (2024). https://doi.org/10.1007/s00521-023-09073-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09073-z

Keywords

Search

Navigation