Skip to main content
SpringerLink
Log in

Fast and accurate image retrieval using knowledge distillation from multiple deep pre-trained networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The content retrieval systems aim to retrieve images similar to a query image from a large data set. A feature extractor and similarity measure play a key role in these systems. Hand-crafted feature descriptors like SURF, SIFT, and GIST find a suitable pattern for measuring the similarity between images. Recently deep learning in this field has been given much attention, which performs feature extraction and similarity learning simultaneously. Various research shows that the feature vector extracted from pre-trained networks contains richer information than class labels in classifying or retrieving information. This paper presents an effective method, Deep Muti-teacher Transfer Hash (DMTH), which uses knowledge from several complex models to teach a simple one. Due to the variety of available pre-trained models and the diversity among their extracted features, we utilize an attention mechanism to obtain richer features from them to teach a simple model via an appropriate knowledge distillation loss. We test our method on widely used datasets Cifar10 & Cifar100 and compare our method with other state-of-the-art methods. The experimental results show that DMTH can improve the image retrieval performance by learning better features obtained through an attention mechanism from multiple teachers without increasing evaluation time. Specifically, the proposed multi-teacher model surpasses the best individual teacher by 2% in terms of accuracy on Cifar10. Meanwhile, it boosts the performance of the student model by more than 4% using our knowledge transfer mechanism.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Algorithm 2
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The datasets analyzed during the current study is available in the following repository, https://www.cs.toronto.edu/~kriz/cifar.html

References

  1. Al-Kaabi K, Monsefi R, Zabihzadeh D (2021) "A Framework to Enhance Generalization of Deep Metric Learning Methods Using General Discriminative Feature Learning and Class Adversarial Neural Networks." arXiv preprint arXiv:2106.06420

  2. Babenko A, Slesarev A, Chigorin A, Lempitsky V" (2014) Neural codes for image retrieval." Paper presented at the European conference on computer vision

  3. Berthelot D, Roelofs R, Sohn K, Carlini N, Kurakin A (2021) "Adamatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation." arXiv preprint arXiv:2106.04732

  4. Cao Z, Long M, Wang J, and Yu PS (2017) "Hashnet: Deep Learning to Hash by Continuation." Paper presented at the Proceedings of the IEEE international conference on computer vision

  5. Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) "Imagenet: A Large-Scale Hierarchical Image Database." Paper presented at the 2009 IEEE conference on computer vision and pattern recognition.

  6. Dey S, Riba P, Dutta A, Llados J, Song Y-Z (2019) "Doodle to search: practical zero-shot sketch-based image retrieval." Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  7. Dubey SR (2021) A Decade Survey of Content Based Image Retrieval Using Deep Learning. IEEE Transact Circuits Syst Video Technol 32(5):2687–2704

    Article  Google Scholar 

  8. Dubey SR, Singh SK, Singh RK (2014) Rotation and Illumination Invariant Interleaved Intensity Order-Based Local Descriptor. IEEE Trans Image Process 23(12):5323–5333

    Article  MathSciNet  MATH  Google Scholar 

  9. Dubey SR, Singh SK, Singh RK (2015) Local Diagonal Extrema Pattern: A New and Efficient Feature Descriptor for Ct Image Retrieval. IEEE Sig Proc Lett 22(9):1215–1219

    Article  Google Scholar 

  10. Dubey SR, Singh SK, Singh RK (2016) Multichannel decoded local binary patterns for content-based image retrieval. IEEE Trans Image Process 25(9):4018–4032

    Article  MathSciNet  MATH  Google Scholar 

  11. Ganin, Y, Lempitsky V (2015) "Unsupervised domain adaptation by backpropagation." Paper presented at the International conference on machine learning

  12. Gou J, Yu B, Maybank SJ, Tao D (2021) Knowledge Distillation: A Survey. Int J Comput Vis:1–31

  13. Gu G, Ko B, Kim H-G (2021) "Proxy synthesis: learning with synthetic classes for deep metric learning." Paper presented at the Proceedings of the AAAI Conference on Artificial Intelligence

  14. He K, Zhang X, Ren S, Sun J (2016) "Deep residual learning for image recognition." Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

  15. Hsu K-J, Lin Y-Y, Chuang Y-Y (2015) "Robust image alignment with multiple feature descriptors and matching-guided neighborhoods". Paper presented at the CVPR

  16. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) "Densely Connected Convolutional Networks." Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

  17. Jacob IJ, Srinivasagan KG, Jayapriya K (2014) Local Oppugnant color texture pattern for image retrieval system. Pattern Recogn Lett 42:72–78

    Article  Google Scholar 

  18. Jegou H, Douze M, Schmid C (2008) "Hamming embedding and weak geometric consistency for large scale image search". Paper presented at the European conference on computer vision

  19. Jegou H, Perronnin F, Douze M, Sánchez J, Perez P, Schmid C (2011) Aggregating local image descriptors into compact codes. IEEE Trans Pattern Anal Mach Intell 34(9):1704–1716

    Article  Google Scholar 

  20. Jiang Q-Y, Li W-J (2018) "Asymmetric deep supervised hashing." Paper presented at the Proceedings of the AAAI conference on artificial intelligence

  21. Kim S, Kim D, Cho M, Kwak S (2020) "Proxy anchor loss for deep metric learning." Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

  22. Krizhevsky A, Hinton GE (2011) "Using Very Deep Autoencoders for Content-Based Image Retrieval." Paper presented at the ESANN

  23. Krizhevsky A, Sutskever I, Hinton GE (2012) "Imagenet Classification with Deep Convolutional Neural Networks". Advances in neural information processing systems 25

  24. Lai H, Pan Y, Liu Y, Yan S (2015) "Simultaneous feature learning and hash coding with deep neural networks." Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

  25. Li Z, Tang J (2015) Weakly supervised deep metric learning for community-contributed image retrieval. IEEE Transac Multimed 17(11):1989–1999

    Article  Google Scholar 

  26. Li Z, Tang J, Mei T (2018) Deep collaborative embedding for social image understanding. IEEE Trans Pattern Anal Mach Intell 41(9):2070–2083

    Article  Google Scholar 

  27. Li J, Ng WY, Tian X, Kwong S, Wang H (2020) Weighted Multi-Deep Ranking Supervised Hashing for Efficient Image Retrieval. Int J Mach Learn Cybern 11(4):883–897

    Article  Google Scholar 

  28. Liu H, Wang R, Shan S, Chen X (2016) "Deep supervised hashing for fast image retrieval." Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

  29. Lowe DG (2004) Distinctive Image Features from Scale-Invariant Keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  30. Movshovitz-Attias Y, Toshev A, Leung TK, Ioffe S, Singh S (2017) "No Fuss Distance Metric Learning Using Proxies." Paper presented at the Proceedings of the IEEE International Conference on Computer Vision

  31. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  32. Peng Z, Li Z, Zhang J, Li Y, Qi G-J, Tang J (2019) "Few-shot image recognition with knowledge transfer." Paper presented at the Proceedings of the IEEE/CVF International Conference on Computer Vision

  33. Qian Q, Shang L, Sun B, Hu J, Li H, Jin R (2019) "Softtriple loss: deep metric learning without triplet sampling." Paper presented at the Proceedings of the IEEE/CVF International Conference on Computer Vision

  34. Simonyan K, Zisserman A (2014) "Very Deep Convolutional Networks for Large-Scale Image Recognition." arXiv preprint arXiv:1409.1556

  35. Sohn K (2016) "Improved Deep Metric Learning with Multi-Class N-Pair Loss Objective." Paper presented at the Proceedings of the 30th International Conference on Neural Information Processing Systems

  36. Song O, Hyun YX, Jegelka S, Savarese S (2016) "Deep Metric Learning Via Lifted Structured Feature Embedding." Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition

  37. Wang J, Yang S, Leung T, Rosenberg C, Wang J, Philbin J, Chen B, Ying W (2014) "Learning fine-grained image similarity with deep ranking." Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  38. Wei S, Liao L, Li J, Zheng Q, Yang F, Zhao Y (2019) Saliency inside: learning attentive Cnns for content-based image retrieval. IEEE Trans Image Process 28(9):4580–4593

    Article  MathSciNet  MATH  Google Scholar 

  39. Wu D, Dai Q, Liu J, Li B, Wang W (2019) "Deep incremental hashing network for efficient image retrieval." Paper presented at the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  40. Xia R, Pan Y, Lai H, Liu C, Yan S (2014) "Supervised hashing for image retrieval via image representation learning." Paper presented at the AAAI

  41. Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) "Hierarchical Attention Networks for Document Classification." Paper presented at the Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies

  42. Yao T, Long F, Mei T, Rui Y (2016) "Deep semantic-preserving and ranking-based hashing for image retrieval". Paper presented at the IJCAI

  43. Zhai H, Lai S, Jin H, Qian X, Mei T (2020) "Deep transfer hashing for image retrieval." IEEE Transactions on Circuits and Systems for Video Technology

  44. Zhu C, Han S, Mao H, Dally WJ (2016) "Trained Ternary Quantization." arXiv preprint arXiv:1612.01064

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, Faculty of Engineering, Ferdowsi University of Mashhad, Mashhad, Iran

    Hasan Salman & Amir Hossein Taherinia

  2. Department of Computer Engineering, Hakim Sabzevari University, Sabzevar, Iran

    Davood Zabihzadeh

Authors
  1. Hasan Salman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amir Hossein Taherinia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Davood Zabihzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amir Hossein Taherinia.

Ethics declarations

Competing interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Salman, H., Taherinia, A.H. & Zabihzadeh, D. Fast and accurate image retrieval using knowledge distillation from multiple deep pre-trained networks. Multimed Tools Appl 82, 33937–33959 (2023). https://doi.org/10.1007/s11042-023-14761-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14761-y

Keywords

Search

Navigation