Skip to main content
SpringerLink
Log in

A transfer-based few-shot classification approach via masked manifold mixup and fuzzy memory contrastive learning

  • S.I.: Interpretation of Deep Learning
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Few-shot learning studies the problem of classifying unseen images by learning only a small number of samples in these categories with the assistance of a large amount of data in other classes. In recent studies, the idea of transfer learning is an effective method to solve the problem of few-shot classification. However, the insufficient generalization ability of the model still restricts the performance of these transfer-based methods. This paper proposes a masked manifold mixup and fuzzy memory contrastive learning (M3FM) method for transfer-based few-shot learning to improve the generalization ability. We design a regularization technique that enhances the model’s learning of local features by masking and mixing the data manifold in the hidden states of neural networks. Then, a momentum updated fuzzy memory is adopted in contrastive learning with the masked mixup manifold to help the model learn the specific distinctions of different categories. Experimental results show that the proposed method outperforms previous baseline methods on miniImageNet, CUB-200, and CIFAR-FS benchmarks. Further adaptation research demonstrates that our method can be generalized to complex few-shot classification tasks and cross-domain scenarios. Ablation studies verify the effectiveness of masked manifold mixup and fuzzy memory contrastive learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  2. Li X-J, Yang G-H (2018) Neural-network-based adaptive decentralized fault-tolerant control for a class of interconnected nonlinear systems. IEEE Trans Neural Netw Learn Syst 29(1):144–155

    Article  MathSciNet  Google Scholar 

  3. Li X-J, Yang G-H (2016) Fls-based adaptive synchronization control of complex dynamical networks with nonlinear couplings and state-dependent uncertainties. IEEE Trans Cybern 46(1):171–180

    Article  MathSciNet  Google Scholar 

  4. Kowsher M, Hossen I, Tahabilder A, Prottasha N, Habib K, Azmi Z (2021) Support directional shifting vector: a direction based machine learning classifier. Emerg Sci J 5:700–713

    Article  Google Scholar 

  5. Dai Z, Liu H, Le QV, Tan M (2021) CoAtNet: marrying convolution and attention for all data sizes. In: Advances in neural information processing systems

  6. Pham H, Dai Z, Xie Q, Le QV (2021) Meta pseudo labels. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11557–11568

  7. Foret P, Kleiner A, Mobahi H, Neyshabur B (2021) Sharpness-aware minimization for efficiently improving generalization. In: International conference on learning representations

  8. Yeruva S, Varalakshmi M, Gowtham B, Chandana Y, Prasad P (2021) Identification of sickle cell anemia using deep neural networks. Emerg Sci J 5:200–210

    Article  Google Scholar 

  9. Ziafati Bagherzadeh H, Toosizadeh S (2022) Eye tracking algorithm based on multi model Kalman filter. HighTech Innov J 3:15–27

    Article  Google Scholar 

  10. Huisman M, van Rijn JN, Plaat A (2021) A survey of deep meta-learning. Artif Intell Rev 54(6):4483–4541

    Article  Google Scholar 

  11. Li X, Sun Z, Xue J-H, Ma Z (2021) A concise review of recent few-shot meta-learning methods. Neurocomputing 456:463–468

    Article  Google Scholar 

  12. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th international conference on machine learning, vol 70, pp 1126–1135

  13. Rusu AA, Rao D, Sygnowski J, Vinyals O, Pascanu R, Osindero S, Hadsell R (2019) Meta-learning with latent embedding optimization. In: International conference on learning representations

  14. Raghu A, Raghu M, Bengio S, Vinyals O (2020) Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In: International conference on learning representations

  15. Wang Y-X, Girshick R, Hebert M, Hariharan B (2018) Low-shot learning from imaginary data. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  16. Schwartz E, Karlinsky L, Shtok J, Harary S, Marder M, Kumar A, Feris R, Giryes R, Bronstein A (2018) Delta-encoder: an effective sample synthesis method for few-shot object recognition. In: Advances in neural information processing systems, vol 31

  17. Zhang R, Che T, Ghahramani Z, Bengio Y, Song Y (2018) MetaGAN: an adversarial approach to few-shot learning. In: Advances in neural information processing systems, vol 31

  18. Zhang H, Zhang J, Koniusz P (2019) Few-shot learning via saliency-guided hallucination of samples. In: 2019 IEEE/CVF conference on computer vision and pattern recognition, pp 2765–2774

  19. Chen Z, Fu Y, Wang Y-X, Ma L, Liu W, Hebert M (2019) Image deformation meta-networks for one-shot learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  20. Chen W-Y, Liu Y-C, Kira Z, Wang Y-CF, Huang J-B (2019) A closer look at few-shot classification. In: International conference on learning representations

  21. Mangla P, Kumari N, Sinha A, Singh M, Krishnamurthy B, Balasubramanian VN (2020) Charting the right manifold: manifold mixup for few-shot learning. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision

  22. Hu Y, Gripon V, Pateux S (2021) Leveraging the feature distribution in transfer-based few-shot learning. In: Artificial neural networks and machine learning—ICANN 2021, pp 487–499

  23. Verma V, Lamb A, Beckham C, Najafi A, Mitliagkas I, Lopez-Paz D, Bengio Y (2019) Manifold mixup: better representations by interpolating hidden states. In: Proceedings of the 36th international conference on machine learning, vol 97, pp 6438–6447

  24. Vinyals O, Blundell C, Lillicrap T, kavukcuoglu k, Wierstra D (2016) Matching networks for one shot learning. In: Advances in neural information processing systems, vol 29

  25. Snell J, Swersky K, Zemel R (2017) Prototypical networks for few-shot learning. In: Advances in neural information processing systems, vol 30

  26. Sung F, Yang Y, Zhang L, Xiang T, Torr PHS, Hospedales TM (2018) Learning to compare: relation network for few-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  27. Tian Y, Wang Y, Krishnan D, Tenenbaum JB, Isola P (2020) Rethinking few-shot image classification: a good embedding is all you need? In: Computer vision—ECCV 2020, pp 266–282

  28. Gidaris S, Bursuc A, Komodakis N, Perez P, Cord M (2019) Boosting few-shot visual learning with self-supervision. In: Proceedings of the IEEE/CVF international conference on computer vision

  29. Gidaris S, Singh P, Komodakis N (2018) Unsupervised representation learning by predicting image rotations. In: International conference on learning representations

  30. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(56):1929–1958

    MATH  MathSciNet  Google Scholar 

  31. Ghiasi G, Lin T-Y, Le QV (2018) DropBlock: a regularization method for convolutional networks. In: Advances in neural information processing systems, vol 31

  32. Zhong Z, Zheng L, Kang G, Li S, Yang Y (2020) Random erasing data augmentation. In: Proceedings of the AAAI conference on artificial intelligence (AAAI)

  33. Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2018) mixup: beyond empirical risk minimization. In: International conference on learning representations

  34. Hendrycks D, Mu N, Cubuk ED, Zoph B, Gilmer J, Lakshminarayanan B (2020) AugMix: a simple method to improve robustness and uncertainty under data shift. In: International conference on learning representations

  35. Yun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y (2019) Cutmix: regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV)

  36. Cascante-Bonilla P, Sekhon A, Qi Y, Ordonez V (2021) Evolving image compositions for feature representation learning. In: British machine vision conference (BMVC)

  37. Ye M, Zhang X, Yuen PC, Chang S-F (2019) Unsupervised embedding learning via invariant and spreading instance feature. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  38. Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th international conference on machine learning. proceedings of machine learning research, vol 119, pp 1597–1607

  39. Wu Z, Xiong Y, Yu SX, Lin D (2018) Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  40. Caron M, Misra I, Mairal J, Goyal P, Bojanowski P, Joulin A (2020) Unsupervised learning of visual features by contrasting cluster assignments. In: Advances in neural information processing systems, vol 33, pp 9912–9924

  41. He K, Fan H, Wu Y, Xie S, Girshick R (2020) Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  42. Zagoruyko S, Komodakis N (2016) Wide residual networks. In: Proceedings of the British machine vision conference (BMVC), pp 1–12

  43. Wah C, Branson S, Welinder P, Perona P, Belongie S (2011) The Caltech-UCSD Birds-200-2011 Dataset. Technical report CNS-TR-2011-001, California Institute of Technology

  44. Bertinetto L, Henriques JF, Torr PHS, Vedaldi A (2019) Meta learning with differentiable closed-form solvers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  45. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations

  46. Das D, Lee CSG (2020) A two-stage approach to few-shot learning for image recognition. IEEE Trans Image Process 29:3336–3350

    Article  MATH  Google Scholar 

  47. Mishra N, Rohaninejad M, Chen X, Abbeel P (2018) A simple neural attentive meta-learner. In: International conference on learning representations

  48. Dhillon GS, Chaudhari P, Ravichandran A, Soatto S (2020) A baseline for few-shot image classification. In: International conference on learning representations

  49. Lee K, Maji S, Ravichandran A, Soatto S (2019) Meta-learning with differentiable convex optimization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  50. Chen D, Chen Y, Li Y, Mao F, He Y, Xue H (2021) Self-supervised learning for few-shot image classification. In: ICASSP 2021–2021 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1745–1749

  51. Simon C, Koniusz P, Nock R, Harandi M (2020) Adaptive subspaces for few-shot learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  52. Ma Y, Bai S, An S, Liu W, Liu A, Zhen X, Liu X (2020) Transductive relation-propagation network for few-shot learning. In: Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI-20, pp 804–810

  53. Hu SX, Moreno PG, Xiao Y, Shen X, Obozinski G, Lawrence N, Damianou A (2020) Empirical bayes transductive meta-learning with synthetic gradients. In: International conference on learning representations

  54. Liu J, Song L, Qin Y (2020) Prototype rectification for few-shot learning. In: Computer vision—ECCV 2020, pp 741–756

  55. Ma Y, Bai S, Liu W, Wang S, Yu Y, Bai X, Liu X, Wang M (2021) Transductive relation-propagation with decoupling training for few-shot learning. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3082928

    Article  Google Scholar 

  56. Ziko I, Dolz J, Granger E, Ayed IB (2020) Laplacian regularized few-shot learning. In: Proceedings of the 37th international conference on machine learning, vol 119, pp 11660–11670

  57. Hu Y, Gripon V, Pateux S (2021) Graph-based interpolation of feature vectors for accurate few-shot classification. In: 2020 25th international conference on pattern recognition, pp 8164–8171

  58. Lichtenstein M, Sattigeri P, Feris R, Giryes R, Karlinsky L (2020) TAFSSL: task-adaptive feature sub-space learning for few-shot classification. In: Computer vision—ECCV 2020, pp 522–539

Download references

Funding

This work was supported by the National Natural Science Foundation of China under Grant Nos. 52072026 and 62076022.

Author information

Authors and Affiliations

  1. School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Beijing, 100044, People’s Republic of China

    Runliang Tian & Hongmei Shi

  2. Key Laboratory of Vehicle Advanced Manufacturing, Measuring and Control Technology (Beijing Jiaotong University), Ministry of Education, Beijing, 100044, People’s Republic of China

    Hongmei Shi

Authors
  1. Runliang Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongmei Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongmei Shi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tian, R., Shi, H. A transfer-based few-shot classification approach via masked manifold mixup and fuzzy memory contrastive learning. Neural Comput & Applic 35, 10069–10082 (2023). https://doi.org/10.1007/s00521-022-07607-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07607-5

Keywords

Search

Navigation