Abstract
Various few-shot image classification methods indicate that transferring knowledge from other sources can improve the accuracy of the classification. However, most of these methods work with one single source or use only closely correlated knowledge sources. In this paper, we propose a novel weakly correlated knowledge integration (WCKI) framework to address these issues. More specifically, we propose a unified knowledge graph (UKG) to integrate knowledge transferred from different sources (i.e., visual domain and textual domain). Moreover, a graph attention module is proposed to sample the subgraph from the UKG with low complexity. To avoid explicitly aligning the visual features to the potentially biased and weakly correlated knowledge space, we sample a task-specific subgraph from UKG and append it as latent variables. Our framework demonstrates significant improvements on multiple few-shot image classification datasets.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
J. Q. Gu, H. F. Hu, H. X. Li. Local robust sparse representation for face recognition with single sample per person. IEEE/CAA Journal of Automatica Sinica, vol.5, no. 2, pp. 547–554, 2018. DOI: https://doi.org/10.1109/JAS.2017.7510658.
D. Y. Liu, J. Xu, P. Y. Zhang, Y. H. Yan. Investigation of knowledge transfer approaches to improve the acoustic modeling of Vietnamese ASR system. IEEE/CAA Journal of Automatica Sinica, vol.6, no.5, pp. 1187–1195, 2019. DOI: https://doi.org/10.1109/JAS.2019.1911693.
E. F. Ohata, G. M. Bezerra, J. V. S. das Chagas, A. V. L. Neto, A. B. Albuquerque, V. H. C. de Albuquerque, P. P. R. Filho. Automatic detection of COVID-19 infection using chest X-ray images through transfer learning. IEEE/CAA Journal of Automatica Sinica, vol.8, no. 1, pp. 239–248, 2021. DOI: https://doi.org/10.1109/JAS.2020.1003393.
Y. Li, D. Xu. Skill learning for robotic insertion based on one-shot demonstration and reinforcement learning. International Journal of Automation and Computing, vol. 18, no. 3, pp. 457–467, 2021. DOI: https://doi.org/10.1007/s11633-021-1290-3.
Y. Q. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele. Latent embeddings for zero-shot classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 69–77, 2016. DOI: https://doi.org/10.1109/CVPR.2016.15.
E. Schönfeld, S. Ebrahimi, S. Sinha, T. Darrell, Z. Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 8239–8247, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00844.
S. Changpinyo, W. L. Chao, B. Q. Gong, F. Sha. Synthesized classifiers for zero-shot learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 5327–5336, 2016. DOI: https://doi.org/10.1109/CVPR.2016.575.
Y. H. H. Tsai, L. K. Huang, R. Salakhutdinov. Learning robust visual-semantic embeddings. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 3591–3600, 2017. DOI: https://doi.org/10.1109/ICCV.2017.386.
A. X. Li, T. G. Luo, Z. W. Lu, T. Xiang, L. W. Wang. Large-scale few-shot learning: Knowledge transfer with class hierarchy. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 7205–7213, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00738.
A. X. Li, K. X. Zhang, L. W. Wang. Zero-shot fine-grained classification by deep feature learning with semantics. International Journal of Automation and Computing, vol. 16, no. 5, pp. 563–574, 2019. DOI: https://doi.org/10.1007/s11633-019-1177-8.
Y. Q. Xian, C. H. Lampert, B. Schiele, Z. Akata. Zero-shot learning — A comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.41, no. 9, pp. 2251–2265, 2019. DOI: https://doi.org/10.1109/TPAMI.2018.2857768.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, P. Perona. Caltech-UCSD Birds 200, Technical Report CNS-TR-2010–001, California Institute of Technology, USA, 2010.
S. C. Li, D. P. Chen, B. Liu, M. H. Yu, R. Zhao. Memory-based neighbourhood embedding for visual recognition. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Korea, pp. 6101–6110, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00620.
H. X. Yao, X. Wu, Z. Q. Tao, Y. L. Li, B. L. Ding, R. R. Li, Z. H. Li. Automated relational meta-learning. In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
C. Xing, N. Rostamzadeh, B. N. Oreshkin, P. O. Pinheiro. Adaptive cross-modal few-shot learning. In Proceedings of the 33rd Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 4848–4858, 2019.
Z. M. Peng, Z. C. Li, J. G. Zhang, Y. Li, G. J. Qi, J. H. Tang. Few-shot image recognition with knowledge transfer. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Korea, pp. 441–449, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00053.
D. Debasmit, C. S. George Lee. A two-stage approach to few-shot learning for image recognition. IEEE Transactions on Image Processing, 2020, vol. 29, pp.3336–3350. DOI: https://doi.org/10.1109/TIP.2019.2959254.
V. G. Satorras, J. B. Estrach. Few-shot learning with graph neural networks. In Proceedings of the 6th International Conference on Learning Representation, Vancouver, Canada, 2018.
J. Kim, T. Kim, S. Kim, C. D. Yoo. Edge-labeling graph neural network for few-shot learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 11–20, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00010.
Y. B. Liu, J. Lee, M. Park, S. Kim, E. Yang, S. J. Hwang, Y. Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, USA, 2019.
X. K. Zhou, W. Liang, S. Shimizu, J. H. Ma, Q. Jin. Siamese neural network based few-shot learning for anomaly detection in industrial cyber-physical systems. IEEE Transactions on Industrial Informatics, vol. 17, no. 8, pp 5790–5798, 2021. DOI: https://doi.org/10.1109/TII.2020.3047675.
H. J. Ye, H. X. Hu, D. C. Zhan. Learning adaptive classifiers synthesis for generalized few-shot learning. International Journal of Computer Vision, vol.129, no. 6, pp. 1930–1953, 2021. DOI: https://doi.org/10.1007/s11263-020-01381-4.
M. A. Jamal, G. J. Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 11711–11719, 2019. DOI: https://doi.org/10.1109/CVPR.2019.01199.
A. Obamuyide, A. Vlachos. Model-agnostic meta-learning for relation classification with limited supervision. In Proceedings of the 57th Conference of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, pp. 5873–5879, 2019.
S. P. Yan, S. Y. Zhang, X. M. He. A dual attention network with semantic embedding for few-shot learning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI, Honolulu, USA pp. 9079–9086, 2019. DOI: https://doi.org/10.1609/aaai.v33i01.33019079.
S. Ravi, H. Larochelle. Optimization as a model for few-shot learning. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
H. X. Yao, X. Wu, Z. Q. Tao, Y. L. Li, B. L. Ding, R. R. Li, Z. H. Li. Automated relational meta-learning. In Proceedings of 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
A. Nichol, J. Achiam, J. Schulman. On first-order meta-learning algorithms. [Online], Available: https://arxiv.org/abs/1803.02999, 2018.
D. P. Kingma, J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA, 2015.
O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, D. Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp.3637–3645, 2016.
K. R. Allen, E. Shelhamer, H. Shin, J. B. Tenenbaum. Infinite mixture prototypes for few-shot learning. In Proceedings of 36th International Conference on Machine Learning, Long Beach, USA, pp. 232–241, 2019.
J. Snell, K. Swersky, R. Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 4080–4090, 2017.
F. Sung, Y. X. Yang, L. Zhang, T. Xiang, P. H. S. Torr, T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1199–1208, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00131.
L. Bertinetto, J. F. Henriques, P. H. S. Torr, A. Vedaldi. Meta-learning with differentiable closed-form solvers. In Proceedings of 7th International Conference on Learning Representations, New Orleans, USA, 2019.
F. S. Hao, F. X. He, J. Cheng, L. Wang, J. Z. Cao, D. C. Tao. Collect and select: Semantic alignment metric learning for few-shot learning. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Korea, pp. 8459–8468, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00855.
A. X. Li, T. G. Luo, T. Xiang, W. R. Huang, L. W. Wang. Few-shot learning with global class representations. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Korea, pp. 9714–9723, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00981.
Z. Y. Wu, Y. W. Li, L. H. Guo, K. Jia. PARN: Position-aware relation networks for few-shot learning. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Korea, pp. 6658–6666, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00676.
N. Mishra, M. Rohaninejad, X. Chen, P. Abbeel. A simple neural attentive meta-learner. In Proceedings of 6th International Conference on Learning Representations, Vancouver, Canada, 2018.
T. Munkhdalai, X. D. Yuan, S. Mehri, A. Trischler. Rapid adaptation with conditionally shifted neurons. In Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholm, Sweden, pp. 3661–3670, 2018.
L. M. Qiao, Y. M. Shi, J. Li, Y. H. Tian, T. J. Huang, Y. W. Wang. Transductive episodic-wise adaptive metric for few-shot learning. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Korea, pp. 3602–3611, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00370.
W. Y. Chen, Y. C. Liu, Z. Kira, Y. C. F. Wang, J. B. Huang. A closer look at few-shot classification. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, USA, 2019.
A. Antoniou, A. J. Storkey. Learning to learn by self-critique. In Proceedings of the 33rd Conference on Neural Information Processing Systems, Vancouver, Canada, pp.9936–9946, 2019.
Acknowledgements
The research was supported by National Key Research and Development Program of China (No. 2020AAA 09701), and National Natural Science Foundation of China (Nos. 62076024 and 62006018).
Author information
Authors and Affiliations
Corresponding author
Additional information
Colored figures are available in the online version at https://link.springer.com/journal/11633
Chun Yang received the B. Sc. and Ph. D. degrees in computer science from University of Science and Technology Beijing, China in 2011 and 2018, respectively. He is currently a faculty member with School of Computer and Communication Engineering, University of Science and Technology Beijing, China.
His research interests include pattern recognition, classifier ensemble, and document analysis and recognition. E-mail: chunyang@ustb.edu.cn ORCID iD: 0000-0002-6297-4500
Chang Liu received the B. Sc. degree in computer science from University of Science and Technology Beijing, China in 2016, where he is a Ph. D. degree candidate with Department of Computer Science and Technology.
His research interests include text detection, few-shot learning, and text recognition. E-mail: lassercat@gmx.us ORCID iD: 0000-0002-7353-0251
Xu-Cheng Yin received the B. Sc. and M. Sc. degrees in computer science from University of Science and Technology Beijing, China in 1999 and 2002, respectively, and the Ph.D. degree in pattern recognition and intelligent systems from Institute of Automation, Chinese Academy of Sciences, China in 2006. He is a full professor, the director of Pattern Recognition and Information Retrieval Lab, Department of Computer Science and Technology, University of Science and Technology Beijing, China. He was a visiting professor in College of Information and Computer Sciences, University of Massachusetts Amherst, USA, for three times (January 2013 to January 2014, July 2014 to August 2014, and July 2016 to September 2016).
His research interests include pattern recognition and machine learning, document analysis and recognition, information retrieval, computer vision, multimedia understanding, and data mining. E-mail: xuchengyin@ustb.edu.cn (Corresponding author) ORCID iD: 0000-0003-0023-0220
Rights and permissions
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Yang, C., Liu, C. & Yin, XC. Weakly Correlated Knowledge Integration for Few-shot Image Classification. Mach. Intell. Res. 19, 24–37 (2022). https://doi.org/10.1007/s11633-022-1320-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-022-1320-9