Advertisement

Part-Aware Prototype Network for Few-Shot Semantic Segmentation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12354)

Abstract

Few-shot semantic segmentation aims to learn to segment new object classes with only a few annotated examples, which has a wide range of real-world applications. Most existing methods either focus on the restrictive setting of one-way few-shot segmentation or suffer from incomplete coverage of object regions. In this paper, we propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes, capable of capturing diverse and fine-grained object features. In addition, we propose to leverage unlabeled data to enrich our part-aware prototypes, resulting in better modeling of intra-class variations of semantic objects. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes based on labeled and unlabeled images. Extensive experimental evaluations on two benchmarks show that our method outperforms the prior art with a sizable margin (Code is available at: https://github.com/Xiangyi1996/PPNet-PyTorch).

Supplementary material

504446_1_En_9_MOESM1_ESM.pdf (13.9 mb)
Supplementary material 1 (pdf 14271 KB)

References

  1. 1.
    Ayyad, A., Navab, N., Elhoseiny, M., Albarqouni, S.: Semi-supervised few-shot learning with local and global consistency. arXiv preprint arXiv (2019)Google Scholar
  2. 2.
    Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)CrossRefGoogle Scholar
  3. 3.
    Boots, Z.L.I.E.B., Shaban, A., Bansal, S.: One-shot learning for semantic segmentation. In: British Machine Vision Conference (BMVC) (2017)Google Scholar
  4. 4.
    Brabandere, B.D., Neven, D., Gool, L.V.: Semantic instance segmentation for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR) (2017)Google Scholar
  5. 5.
    Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv (2017)Google Scholar
  6. 6.
    Chung, Y.A., Weng, W.H.: Learning deep representations of medical images using siamese CNNs with application to content-based image retrieval. In: NIPS Machine Learning for Health Workshop (NIPS Workshop) (2017)Google Scholar
  7. 7.
    Dong, N., Xing, E.: Few-shot semantic segmentation with prototype learning. In: British Machine Vision Conference (BMVC) (2018)Google Scholar
  8. 8.
    Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML) (2017)Google Scholar
  9. 9.
    Garcia, V., Bruna, J.: Few-shot learning with graph neural networks. arXiv preprint arXiv (2017)Google Scholar
  10. 10.
    Gori, M., Monfardini, G., Scarselli, F.: A new model for learning in graph domains. In: Proceedings 2005 IEEE International Joint Conference on Neural Networks. IEEE (2005)Google Scholar
  11. 11.
    Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse detectors. In: 2011 International Conference on Computer Vision (ICCV) (2011)Google Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  13. 13.
  14. 14.
    Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  15. 15.
    Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)Google Scholar
  16. 16.
    Li, X., et al.: Learning to self-train for semi-supervised few-shot classification. In: Advances in Neural Information Processing Systems (NIPS) (2019)Google Scholar
  17. 17.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_48CrossRefGoogle Scholar
  18. 18.
    Liu, Y., et al.: Learning to propagate labels: transductive propagation network for few-shot learning. arXiv preprint arXiv (2018)Google Scholar
  19. 19.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  20. 20.
    Nguyen, K., Todorovic, S.: Feature weighting and boosting for few-shot segmentation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  21. 21.
    Rakelly, K., Shelhamer, E., Darrell, T., Efros, A.A., Levine, S.: Few-shot segmentation propagation with guided networks. arXiv preprint (2018)Google Scholar
  22. 22.
    Rakelly, K., Shelhamer, E., Darrell, T., Efros, A., Levine, S.: Conditional networks for few-shot semantic segmentation (2018)Google Scholar
  23. 23.
    Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning (2016)Google Scholar
  24. 24.
    Ren, M., et al.: Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv (2018)Google Scholar
  25. 25.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision (2015)Google Scholar
  26. 26.
    Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008)CrossRefGoogle Scholar
  27. 27.
    Siam, M., Oreshkin, B.: Adaptive masked weight imprinting for few-shot segmentation. arXiv preprint arXiv (2019)Google Scholar
  28. 28.
    Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems (NIPS) (2017)Google Scholar
  29. 29.
    Tian, P., Wu, Z., Qi, L., Wang, L., Shi, Y., Gao, Y.: Differentiable meta-learning model for few-shot semantic segmentation. arXiv preprint arXiv (2019)Google Scholar
  30. 30.
    Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems (NIPS) (2017)Google Scholar
  31. 31.
    Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
  32. 32.
    Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems (NIPS) (2016)Google Scholar
  33. 33.
    Wang, K., Liew, J.H., Zou, Y., Zhou, D., Feng, J.: Panet: few-shot image semantic segmentation with prototype alignment. arXiv preprint arXiv (2019)Google Scholar
  34. 34.
    Yan, S., Zhang, S., He, X., et al.: A dual attention network with semantic embedding for few-shot learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2019)Google Scholar
  35. 35.
    Zhang, C., Lin, G., Liu, F., Guo, J., Wu, Q., Yao, R.: Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  36. 36.
    Zhang, C., Lin, G., Liu, F., Yao, R., Shen, C.: Canet: class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  37. 37.
    Zhang, X., Wei, Y., Yang, Y., Huang, T.: SG-one: similarity guidance network for one-shot semantic segmentation. arXiv preprint arXiv (2018)Google Scholar
  38. 38.
    Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Information Science and TechnologyShanghaiTech UniversityShanghaiChina
  2. 2.Shanghai Engineering Research Center of Intelligent Vision and ImagingShanghaiChina

Personalised recommendations