Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 pp 606–623Cite as

Active Pointly-Supervised Instance Segmentation

  • 1803 Accesses

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13688)

Abstract

The requirement of expensive annotations is a major burden for training a well-performed instance segmentation model. In this paper, we present an economic active learning setting, named active pointly-supervised instance segmentation (APIS), which starts with box-level annotations and iteratively samples a point within the box and asks if it falls on the object. The key of APIS is to find the most desirable points to maximize the segmentation accuracy with limited annotation budgets. We formulate this setting and propose several uncertainty-based sampling strategies. The model developed with these strategies yields consistent performance gain on the challenging MS-COCO dataset, compared against other learning strategies. The results suggest that APIS, integrating the advantages of active learning and point-based supervision, is an effective learning paradigm for label-efficient instance segmentation.

Keywords

  • Instance segmentation
  • Active learning
  • Point-based supervision
  • Label-efficient learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Agarwal, S., Arora, H., Anand, S., Arora, C.: Contextual diversity for active learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 137–153. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_9

    CrossRef  Google Scholar 

  2. Aghdam, H.H., Gonzalez-Garcia, A., Weijer, J.V.D., López, A.M.: Active learning for deep detection neural networks. In: International Conference on Computer Vision, pp. 3672–3680 (2019)

    Google Scholar 

  3. Arun, A., Jawahar, C.V., Kumar, M.P.: Weakly supervised instance segmentation by learning annotation consistent instances. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12373, pp. 254–270. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58604-1_16

    CrossRef  Google Scholar 

  4. Bearman, A., Russakovsky, O., Ferrari, V., Fei-Fei, L.: What’s the point: semantic segmentation with point supervision. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 549–565. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_34

    CrossRef  Google Scholar 

  5. Beluch, W.H., Genewein, T., Nürnberger, A., Köhler, J.M.: The power of ensembles for active learning in image classification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9368–9377 (2018)

    Google Scholar 

  6. Benenson, R., Popov, S., Ferrari, V.: Large-scale interactive object segmentation with human annotators. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11700–11709 (2019)

    Google Scholar 

  7. Chen, K., et al.: Hybrid task cascade for instance segmentation. In: Conference on Computer Vision and Pattern Recognition, pp. 4974–4983 (2019)

    Google Scholar 

  8. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290–1299 (2022)

    Google Scholar 

  9. Cheng, B., Parkhi, O., Kirillov, A.: Pointly-supervised instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2617–2626 (2022)

    Google Scholar 

  10. Choi, J., Elezi, I., Lee, H.J., Farabet, C., Alvarez, J.M.: Active learning for deep object detection via probabilistic modeling. In: International Conference on Computer Vision (2021)

    Google Scholar 

  11. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)

    Google Scholar 

  12. Desai, S.V., Balasubramanian, V.N.: Towards fine-grained sampling for active learning in object detection. In: Conference on Computer Vision and Pattern Recognition and Workshops, pp. 924–925 (2020)

    Google Scholar 

  13. Desai, S.V., Chandra, A.L., Guo, W., Ninomiya, S., Balasubramanian, V.N.: An adaptive supervision framework for active learning in object detection. In: British Machine Vision Conference (2019)

    Google Scholar 

  14. Dong, B., Zeng, F., Wang, T., Zhang, X., Wei, Y.: SOLQ: segmenting objects by learning queries. In: Advances in Neural Information Processing Systems (2021)

    Google Scholar 

  15. Fang, Y., et al.: Instances as queries. In: International Conference on Computer Vision, pp. 6910–6919 (2021)

    Google Scholar 

  16. Gal, Y., Islam, R., Ghahramani, Z.: Deep Bayesian active learning with image data. In: International Conference on Machine Learning, pp. 1183–1192 (2017)

    Google Scholar 

  17. Gupta, A., Dollar, P., Girshick, R.: Lvis: a dataset for large vocabulary instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5356–5364 (2019)

    Google Scholar 

  18. Hasan, M., Roy-Chowdhury, A.K.: Context aware active learning of activity recognition models. In: International Conference on Computer Vision, pp. 4543–4551 (2015)

    Google Scholar 

  19. Haussmann, E., et al.: Scalable active learning for object detection. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1430–1435 (2020)

    Google Scholar 

  20. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

  21. Hsu, C.C., Hsu, K.J., Tsai, C.C., Lin, Y.Y., Chuang, Y.Y.: Weakly supervised instance segmentation using the bounding box tightness prior. In: Advances in Neural Information Processing Systems, pp. 6586–6597 (2019)

    Google Scholar 

  22. Huang, Z., Huang, L., Gong, Y., Huang, C., Wang, X.: Mask scoring R-CNN. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6409–6418 (2019)

    Google Scholar 

  23. Jang, W.D., Kim, C.S.: Interactive image segmentation via backpropagating refinement scheme. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5292–5301 (2019)

    Google Scholar 

  24. Joshi, A.J., Porikli, F., Papanikolopoulos, N.: Multi-class active learning for image classification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2372–2379 (2009)

    Google Scholar 

  25. Kao, C.C., Lee, T.Y., Sen, P., Liu, M.Y.: Localization-aware active learning for object detection. In: Asian Conference on Computer Vision, pp. 506–522 (2018)

    Google Scholar 

  26. Khoreva, A., Benenson, R., Hosang, J., Hein, M., Schiele, B.: Simple does it: weakly supervised instance and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 876–885 (2017)

    Google Scholar 

  27. Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in Neural Information Processing Systems (2017)

    Google Scholar 

  28. Lan, S., et al.: Discobox: weakly supervised instance segmentation and semantic correspondence from box supervision. In: International Conference on Computer Vision (2021)

    Google Scholar 

  29. Laradji, I.H., Rostamzadeh, N., Pinheiro, P.O., Vazquez, D., Schmidt, M.: Proposal-based instance segmentation with point supervision. In: IEEE International Conference on Image Processing, pp. 2126–2130 (2020)

    Google Scholar 

  30. Lewis, D.D., Catlett, J.: Heterogeneous uncertainty sampling for supervised learning. In: International Conference on Machine Learning (1994)

    Google Scholar 

  31. Li, Y., et al.: Fully convolutional networks for panoptic segmentation with point-based supervision. arXiv preprint arXiv:2108.07682 (2021)

  32. Li, Z., Chen, Q., Koltun, V.: Interactive image segmentation with latent diversity. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 577–585 (2018)

    Google Scholar 

  33. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    CrossRef  Google Scholar 

  34. Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8759–8768 (2018)

    Google Scholar 

  35. Liu, Z., Ding, H., Zhong, H., Li, W., Dai, J., He, C.: Influence selection for active learning. In: International Conference on Computer Vision, pp. 9274–9283 (2021)

    Google Scholar 

  36. Maninis, K.K., Caelles, S., Pont-Tuset, J., Van Gool, L.: Deep extreme cut: From extreme points to object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 616–625 (2018)

    Google Scholar 

  37. Melville, P., Mooney, R.J.: Diverse ensembles for active learning. In: International Conference on Machine Learning (2004)

    Google Scholar 

  38. Papadopoulos, D.P., Uijlings, J.R., Keller, F., Ferrari, V.: Extreme clicking for efficient object annotation. In: International Conference on Computer Vision, pp. 4930–4939 (2017)

    Google Scholar 

  39. Pardo, A., Xu, M., Thabet, A., Arbelaez, P., Ghanem, B.: Baod: budget-aware object detection. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1247–1256 (2021)

    Google Scholar 

  40. Pont-Tuset, J., Arbelaez, P., Barron, J.T., Marques, F., Malik, J.: Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 128–140 (2016)

    CrossRef  Google Scholar 

  41. Qian, R., Wei, Y., Shi, H., Li, J., Liu, J., Huang, T.: Weakly supervised scene parsing with point-based distance metric learning. In: AAAI, pp. 8843–8850 (2019)

    Google Scholar 

  42. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 658–666 (2019)

    Google Scholar 

  43. Roy, S., Unmesh, A., Namboodiri, V.P.: Deep active learning for object detection. In: British Machine Vision Conference (2018)

    Google Scholar 

  44. Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. arXiv preprint arXiv:1708.00489 (2017)

  45. Shannon, C.E.: A mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Commun. Rev. 5(1), 3–55 (2001)

    CrossRef  MathSciNet  Google Scholar 

  46. Shin, G., Xie, W., Albanie, S.: All you need are a few pixels: Semantic segmentation with pixelpick. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1687–1697 (2021)

    Google Scholar 

  47. Sinha, S., Ebrahimi, S., Darrell, T.: Variational adversarial active learning. In: International Conference on Computer Vision, pp. 5972–5981 (2019)

    Google Scholar 

  48. Tang, C., Chen, H., Li, X., Li, J., Zhang, Z., Hu, X.: Look closer to segment better: Boundary patch refinement for instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 13926–13935 (2021)

    Google Scholar 

  49. Tian, Z., Chen, H., Wang, X., Liu, Y., Shen, C.: AdelaiDet: a toolbox for instance-level recognition tasks. https://git.io/adelaidet (2019)

  50. Tian, Z., Shen, C., Chen, H.: Conditional convolutions for instance segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 282–298. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_17

    CrossRef  Google Scholar 

  51. Tian, Z., Shen, C., Wang, X., Chen, H.: Boxinst: high-performance instance segmentation with box annotations. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5443–5452 (2021)

    Google Scholar 

  52. Wang, J., et al.: Semi-supervised active learning for instance segmentation via scoring predictions. In: British Machine Vision Conference (2020)

    Google Scholar 

  53. Wang, K., Zhang, D., Li, Y., Zhang, R., Lin, L.: Cost-effective active learning for deep image classification. IEEE Trans. Circuits Syst. Video Technol. 27(12), 2591–2600 (2016)

    CrossRef  Google Scholar 

  54. Wang, X., Zhang, R., Shen, C., Kong, T., Li, L.: Solo: a simple framework for instance segmentation. IEEE Trans. Pattern Anal. Mach Intell. 4, 8587–8601 (2021)

    Google Scholar 

  55. Wu, T.H., et al.: Redal: region-based and diversity-aware active learning for point cloud semantic segmentation. In: International Conference on Computer Vision, pp. 15510–15519 (2021)

    Google Scholar 

  56. Xu, N., Price, B., Cohen, S., Yang, J., Huang, T.S.: Deep interactive object selection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 373–381 (2016)

    Google Scholar 

  57. Yang, L., Zhang, Y., Chen, J., Zhang, S., Chen, D.Z.: Suggestive annotation: a deep active learning framework for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 399–407 (2017)

    Google Scholar 

  58. Yuan, T., et al.: Multiple instance active learning for object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5330–5339 (2021)

    Google Scholar 

  59. Zhang, G., et al.: Refinemask: towards high-quality instance segmentation with fine-grained features. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6861–6869 (2021)

    Google Scholar 

  60. Zhu, B., et al.: Autoassign: differentiable label assignment for dense object detection. arXiv preprint arXiv:2007.03496 (2020)

  61. Zhu, Y., Zhou, Y., Xu, H., Ye, Q., Doermann, D., Jiao, J.: Learning instance activation maps for weakly supervised instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3116–3125 (2019)

    Google Scholar 

Download references

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (Nos. U19B2034, 62061136001 and 61836014).

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Institute for AI, BNRist, State Key Laboratory of Intelligent Technology and Systems, Tsinghua University, Beijing, China

    Chufeng Tang, Gang Zhang & Xiaolin Hu

  2. Huawei Inc., Shenzhen, China

    Lingxi Xie, Xiaopeng Zhang & Qi Tian

  3. Chinese Institute for Brain Research (CIBR), Beijing, China

    Xiaolin Hu

  4. IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China

    Xiaolin Hu

Authors
  1. Chufeng Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingxi Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaopeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qi Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaolin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Qi Tian or Xiaolin Hu .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7711 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tang, C., Xie, L., Zhang, G., Zhang, X., Tian, Q., Hu, X. (2022). Active Pointly-Supervised Instance Segmentation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13688. Springer, Cham. https://doi.org/10.1007/978-3-031-19815-1_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19815-1_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19814-4

  • Online ISBN: 978-3-031-19815-1

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation