Advertisement

Associative Alignment for Few-Shot Image Classification

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12350)

Abstract

Few-shot image classification aims at training a model from only a few examples for each of the “novel” classes. This paper proposes the idea of associative alignment for leveraging part of the base data by aligning the novel training instances to the closely related ones in the base training set. This expands the size of the effective novel training set by adding extra “related base” instances to the few novel ones, thereby allowing a constructive fine-tuning. We propose two associative alignment strategies: 1) a metric-learning loss for minimizing the distance between related base samples and the centroid of novel instances in the feature space, and 2) a conditional adversarial alignment loss based on the Wasserstein distance. Experiments on four standard datasets and three backbones demonstrate that combining our centroid-based alignment loss results in absolute accuracy improvements of 4.4%, 1.2%, and 6.2% in 5-shot learning over the state of the art for object recognition, fine-grained classification, and cross-domain adaptation, respectively .

Keywords

Associative alignment Few-shot image classification 

Notes

Acknowledgement

This project was supported by funding from NSERC-Canada, Mitacs, Prompt-Québec, and E Machine Learning. We thank Ihsen Hedhli, Saed Moradi, Marc-André Gardner, and Annette Schwerdtfeger for proofreading of the manuscript.

Supplementary material

504441_1_En_2_MOESM1_ESM.pdf (355 kb)
Supplementary material 1 (pdf 354 KB)

References

  1. 1.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv preprint arXiv:1701.07875 (2017)
  2. 2.
    Bertinetto, L., Henriques, J.F., Torr, P., Vedaldi, A.: Meta-learning with differentiable closed-form solvers. In: The International Conference on Learning Representations (2019)Google Scholar
  3. 3.
    Chen, W.Y., Liu, Y.C., Kira, Z., Wang, Y.C.F., Huang, J.B.: A closer look at few-shot classification. arXiv preprint arXiv:1904.04232 (2019)
  4. 4.
    Chen, Z., Fu, Y., Wang, Y.X., Ma, L., Liu, W., Hebert, M.: Image deformation meta-networks for one-shot learning. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  5. 5.
    Chu, W.H., Li, Y.J., Chang, J.C., Wang, Y.C.F.: Spot and learn: a maximum-entropy patch sampler for few-shot image classification. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  6. 6.
    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: additive angular margin loss for deep face recognition. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  7. 7.
    Dhillon, G.S., Chaudhari, P., Ravichandran, A., Soatto, S.: A baseline for few-shot image classification. arXiv preprint arXiv:1909.02729 (2019)
  8. 8.
    Dvornik, N., Schmid, C., Mairal, J.: Diversity with cooperation: ensemble methods for few-shot classification. In: The International Conference on Computer Vision (2019)Google Scholar
  9. 9.
    Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: The International Conference on Machine Learning (2017)Google Scholar
  10. 10.
    Finn, C., Xu, K., Levine, S.: Probabilistic model-agnostic meta-learning. In: Advances in Neural Information Processing Systems (2018)Google Scholar
  11. 11.
    Gao, H., Shou, Z., Zareian, A., Zhang, H., Chang, S.F.: Low-shot learning via covariance-preserving adversarial augmentation networks. In: Advances in Neural Information Processing Systems (2018)Google Scholar
  12. 12.
    Garcia, V., Bruna, J.: Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043 (2017)
  13. 13.
    Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P., Cord, M.: Boosting few-shot visual learning with self-supervision. In: The International Conference on Computer Vision (2019)Google Scholar
  14. 14.
    Gidaris, S., Komodakis, N.: Dynamic few-shot visual learning without forgetting. In: The Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  15. 15.
    Gidaris, S., Komodakis, N.: Generating classification weights with GNN denoising autoencoders for few-shot learning. arXiv preprint arXiv:1905.01102 (2019)
  16. 16.
    Gui, L.-Y., Wang, Y.-X., Ramanan, D., Moura, J.M.F.: Few-shot human motion prediction via meta-learning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 441–459. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01237-3_27CrossRefGoogle Scholar
  17. 17.
    Hariharan, B., Girshick, R.: Low-shot visual recognition by shrinking and hallucinating features. In: The International Conference on Computer Vision (2017)Google Scholar
  18. 18.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  19. 19.
    Hilliard, N., Phillips, L., Howland, S., Yankov, A., Corley, C.D., Hodas, N.O.: Few-shot learning with metric-agnostic conditional embeddings. arXiv preprint arXiv:1802.04376 (2018)
  20. 20.
    Jiang, H., Wang, R., Shan, S., Chen, X.: Learning class prototypes via structure alignment for zero-shot recognition. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 121–138. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01249-6_8CrossRefGoogle Scholar
  21. 21.
    Kim, J., Oh, T.H., Lee, S., Pan, F., Kweon, I.S.: Variational prototyping-encoder: one-shot learning with prototypical images. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  22. 22.
    Kim, T., Yoon, J., Dia, O., Kim, S., Bengio, Y., Ahn, S.: Bayesian model-agnostic meta-learning. arXiv preprint arXiv:1806.03836 (2018)
  23. 23.
    Krizhevsky, A., Nair, V., Hinton, G.: CIFAR-10 and CIFAR-100 datasets (2009). https://www.cs.toronto.edu/kriz/cifar. html
  24. 24.
    Lee, K., Maji, S., Ravichandran, A., Soatto, S.: Meta-learning with differentiable convex optimization. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  25. 25.
    Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., Luo, J.: Revisiting local descriptor based image-to-class measure for few-shot learning. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  26. 26.
    Li, X., et al.: Learning to self-train for semi-supervised few-shot classification. In: Advances in Neural Information Processing Systems (2019)Google Scholar
  27. 27.
    Lifchitz, Y., Avrithis, Y., Picard, S., Bursuc, A.: Dense classification and implanting for few-shot learning. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  28. 28.
    Lim, J.J., Salakhutdinov, R.R., Torralba, A.: Transfer learning by borrowing examples for multiclass object detection. In: Advances in Neural Information Processing Systems (2011)Google Scholar
  29. 29.
    Liu, B., Wu, Z., Hu, H., Lin, S.: Deep metric transfer for label propagation with limited annotated data. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2019Google Scholar
  30. 30.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579–2605 (2008)zbMATHGoogle Scholar
  31. 31.
    Mehrotra, A., Dukkipati, A.: Generative adversarial residual pairwise networks for one shot learning. arXiv preprint arXiv:1703.08033 (2017)
  32. 32.
    Mishra, N., Rohaninejad, M., Chen, X., Abbeel, P.: A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141 (2017)
  33. 33.
    Oreshkin, B., López, P.R., Lacoste, A.: Tadam: task dependent adaptive metric for improved few-shot learning. In: Advances in Neural Information Processing Systems (2018)Google Scholar
  34. 34.
    Qi, H., Brown, M., Lowe, D.G.: Low-shot learning with imprinted weights. In: The Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  35. 35.
    Qiao, S., Liu, C., Shen, W., Yuille, A.L.: Few-shot image recognition by predicting parameters from activations. In: The Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  36. 36.
    Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning (2016)Google Scholar
  37. 37.
    Ren, M., et al.: Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803.00676 (2018)
  38. 38.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. (2015)Google Scholar
  39. 39.
    Rusu, A.A., et al.: Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 (2018)
  40. 40.
    Schwartz, E., et al.: Delta-encoder: an effective sample synthesis method for few-shot object recognition. In: Advances in Neural Information Processing Systems (2018)Google Scholar
  41. 41.
    Sergey, Z., Nikos, K.: Wide residual networks. In: British Machine Vision Conference (2016)Google Scholar
  42. 42.
    Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems (2017)Google Scholar
  43. 43.
    Sun, Q., Liu, Y., Chua, T.S., Schiele, B.: Meta-transfer learning for few-shot learning. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  44. 44.
    Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: relation network for few-shot learning. In: The Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  45. 45.
    Tseng, H.Y., Lee, H.Y., Huang, J.B., Yang, M.H.: Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprint arXiv:2001.08735 (2020)
  46. 46.
    Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. (2002)Google Scholar
  47. 47.
    Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems (2016)Google Scholar
  48. 48.
    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-UCSD birds-200-2011 dataset (2011)Google Scholar
  49. 49.
    Wang, Y.X., Girshick, R., Hebert, M., Hariharan, B.: Low-shot learning from imaginary data. In: The Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  50. 50.
    Wang, Y.X., Hebert, M.: Learning from small sample sets by combining unsupervised meta-training with CNNs. In: Advances in Neural Information Processing Systems (2016)Google Scholar
  51. 51.
    Wang, Y.-X., Hebert, M.: Learning to learn: model regression networks for easy small sample learning. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 616–634. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46466-4_37CrossRefGoogle Scholar
  52. 52.
    Wang, Y.X., Ramanan, D., Hebert, M.: Meta-learning to detect rare objects. In: The International Conference on Computer Vision (2019)Google Scholar
  53. 53.
    Wertheimer, D., Hariharan, B.: Few-shot learning with localization in realistic settings. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  54. 54.
    Yoon, S.W., Seo, J., Moon, J.: Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. arXiv preprint arXiv:1905.06549 (2019)
  55. 55.
    Zhang, H., Zhang, J., Koniusz, P.: Few-shot learning via saliency-guided hallucination of samples. In: The Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  56. 56.
    Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  57. 57.
    Zhang, J., Zhao, C., Ni, B., Xu, M., Yang, X.: Variational few-shot learning. In: The International Conference on Computer Vision (2019)Google Scholar
  58. 58.
    Zhao, F., Zhao, J., Yan, S., Feng, J.: Dynamic conditional networks for few-shot learning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 20–36. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01267-0_2CrossRefGoogle Scholar
  59. 59.
    Zhu, L., Yang, Y.: Compound memory networks for few-shot video classification. In: The European Conference on Computer Vision (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Université LavalQuebec CityCanada
  2. 2.Canada CIFAR AI Chair, MilaMontrealCanada

Personalised recommendations