Skip to main content
SpringerLink
Log in

Content-based image retrieval using handcraft feature fusion in semantic pyramid

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

The main challenge of image retrieval systems is to retrieve similar samples in a way that can be interpreted in a semantic relationship with the user's query image. In recent years, deep neural networks, due to their remarkable role in extracting the content and semantic features of the image, have been at the center of attention for image retrieval. Processing occurs in deep neural networks, at multiple levels, and with a pyramid approach. This characteristic allows the extraction of semantic and high-level features from the image. On the other hand, the image content features can be extracted with high interpretability using handcraft features. Therefore, in the proposed approach, by fusing features and adding extra information sources, handcraft features are semantically enhanced. In this approach, handcraft features including color and texture are extracted from the semantic pyramid of the deep neural network. The semantic pyramid is the result of the fusion of feature maps in different levels of deep neural networks. Additionally, in this approach, feature vector interpretability is also considered. The t-SNE technique has been used to interpret the discriminability of the feature vector between the classes of the database. Also, the silhouette criterion has been introduced to study the degree of intra-class compatibility and inter-class dataset samples discriminability with feature vector.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

The public dataset that supports the findings of this study are available in reference [47].

References

  1. Thusnavis Bella MI, Vasuki A (2019) An efficient image retrieval framework using fused information feature. Comput Electr Eng 75:46–60. https://doi.org/10.1016/J.COMPELECENG.2019.01.022

    Article  Google Scholar 

  2. Alemu LT, Pelillo M (2020) Multi-feature fusion for image retrieval using constrained dominant sets. Image Vis Comput 94:103862. https://doi.org/10.1016/J.IMAVIS.2019.103862

    Article  Google Scholar 

  3. Barbhuiya AA, Karsh RK, Jain R (2022) A convolutional neural network and classical moments-based feature fusion model for gesture recognition. Multimedia Syst 28(5):1779–1792. https://doi.org/10.1007/S00530-022-00951-5/METRICS

    Article  Google Scholar 

  4. Hassanin M, Radwan I, Khan S, Tahtali M (2022) Learning discriminative representations for multi-label image recognition. J Vis Commun Image Represent 83:103448. https://doi.org/10.1016/J.JVCIR.2022.103448

    Article  Google Scholar 

  5. Yang F, Ma Z, Xie M (2021) Image classification with superpixels and feature fusion method. J Electron Sci Technol 19(1):100096. https://doi.org/10.1016/J.JNLEST.2021.100096

    Article  Google Scholar 

  6. Zhang L, He Z, Yang Y, Wang L, Gao X (2022) Tasks integrated networks: joint detection and retrieval for image search. IEEE Trans Pattern Anal Mach Intell 44(1):456–473. https://doi.org/10.1109/TPAMI.2020.3009758

    Article  Google Scholar 

  7. Xu L, Zeng X, Zheng B, Li W (2022) Multi-manifold deep discriminative cross-modal hashing for medical image retrieval. IEEE Trans Image Process 31:3371–3385. https://doi.org/10.1109/TIP.2022.3171081

    Article  Google Scholar 

  8. Li X, Yang J, Ma J (2021) Recent developments of content-based image retrieval (CBIR). Neurocomputing 452:675–689. https://doi.org/10.1016/J.NEUCOM.2020.07.139

    Article  Google Scholar 

  9. Liu Y, Peng Y, Lim K, Ling N (2018) A novel image retrieval algorithm based on transfer learning and fusion features. World Wide Web 22(3):1313–1324. https://doi.org/10.1007/S11280-018-0585-Y

    Article  Google Scholar 

  10. Mistry Y, Ingole DT, Ingole MD (2018) Content based image retrieval using hybrid features and various distance metric. J Electr Syst Inform Technol 5(3):874–888. https://doi.org/10.1016/j.jesit.2016.12.009

    Article  Google Scholar 

  11. Vasudevan S, Chauhan N, Sarobin V, Geetha S (2021) Image-based recommendation engine using VGG model. Lect Notes Electr Eng 668:257–265. https://doi.org/10.1007/978-981-15-5341-7_21/COVER

    Article  Google Scholar 

  12. Simonyan K, Zisserman A (2015). Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December, 770–778. doi:https://doi.org/10.1109/CVPR.2016.90

  14. Ahmed KT, Jaffar S, Hussain MG, Fareed S, Mehmood A, Choi GS (2021) Maximum response deep learning using markov, retinal primitive patch binding with GoogLeNet VGG-19 for large image retrieval. IEEE Access 9:41934–41957. https://doi.org/10.1109/ACCESS.2021.3063545

    Article  Google Scholar 

  15. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 07-12-June-2015 doi:https://doi.org/10.1109/CVPR.2015.7298594

  16. Wei W, Wang W, Yang Y, Wang Y (2021) A novel color image retrieval method based on texture and deep features. Multimed Tools Appl 81(1):659–679. https://doi.org/10.1007/S11042-021-11198-Z

    Article  Google Scholar 

  17. Zhou Z, Wu QMJ, Wan S, Sun W, Sun X (2020) Integrating SIFT and CNN feature matching for partial-duplicate image detection. IEEE Trans Emerg Topics Comput Intel 4(5):593–604. https://doi.org/10.1109/TETCI.2019.2909936

    Article  Google Scholar 

  18. Phadikar BS, Phadikar A, Thakur SS (2021) A comprehensive assessment of content-based image retrieval using selected full reference image quality assessment algorithms. Multimed Tools Appl 80(10):15619–15646. https://doi.org/10.1007/S11042-021-10573-0

    Article  Google Scholar 

  19. Devulapalli S, Potti A, Krishnan R, Khan MS (2021) Experimental evaluation of unsupervised image retrieval application using hybrid feature extraction by integrating deep learning and handcrafted techniques. Mater Today Proc. https://doi.org/10.1016/J.MATPR.2021.04.326

    Article  Google Scholar 

  20. Kanaparthi SK, Raju USN, Shanmukhi P, Aneesha GK, Rahman MEU (2019) Image retrieval by integrating global correlation of color and intensity histograms with local texture features. Multimed Tools Appl 79(47):34875–34911. https://doi.org/10.1007/S11042-019-08029-7

    Article  Google Scholar 

  21. Varish N (2022) A modified similarity measurement for image retrieval scheme using fusion of color, texture and shape moments. Multimed Tools Appl 81(15):20373–20405. https://doi.org/10.1007/S11042-022-12289-1

    Article  Google Scholar 

  22. Taheri F, Rahbar K, Salimi P (2022) Effective features in content-based image retrieval from a combination of low-level features and deep Boltzmann machine. Multimed Tools Appl 2022:1–24. https://doi.org/10.1007/S11042-022-13670-W

    Article  Google Scholar 

  23. Chavda S, Goyani M (2021) Robust image retrieval using CCV, GCH, and MS-LBP descriptors. Multimed Tools Appl 81(3):4039–4072. https://doi.org/10.1007/S11042-021-11698-Y

    Article  Google Scholar 

  24. Khan UA, Javed A (2022) A hybrid CBIR system using novel local tetra angle patterns and color moment features. J King Saud Univ Comput Inform Sci 34(10):7856–7873. https://doi.org/10.1016/J.JKSUCI.2022.07.005

    Article  Google Scholar 

  25. Sunitha T, Sivarani TS (2022) Novel content based medical image retrieval based on BoVW classification method. Biomed Signal Process Control 77:103678. https://doi.org/10.1016/J.BSPC.2022.103678

    Article  Google Scholar 

  26. Liu GH, Yang JY (2022) Exploiting deep textures for image retrieval. Int J Mach Learn Cybernet. https://doi.org/10.1007/S13042-022-01645-0/METRICS

    Article  Google Scholar 

  27. Salih SF, Abdulla AA (2022) An effective bi-layer content-based image retrieval technique. J Supercomput. https://doi.org/10.1007/S11227-022-04748-1/METRICS

    Article  Google Scholar 

  28. Sudha SK, Aji S (2022) An active learning method with entropy weighting subspace clustering for remote sensing image retrieval. Appl Soft Comput 125:109107. https://doi.org/10.1016/J.ASOC.2022.109107

    Article  Google Scholar 

  29. Kenchappa YD, Kwadiki K (2022) Content-based image retrieval using integrated features and multi-subspace randomization and collaboration. Int J Syst Assur Eng Manag 13(5):2540–2550. https://doi.org/10.1007/S13198-022-01663-9/METRICS

    Article  Google Scholar 

  30. Bedi AK, Sunkaria RK (2021) Mean distance local binary pattern: a novel technique for color and texture image retrieval for liver ultrasound images. Multimed Tools Appl 80(14):20773–20802. https://doi.org/10.1007/S11042-021-10758-7/METRICS

    Article  Google Scholar 

  31. Lai WC, Srividhya SR (2022) A modified LBP operator-based optimized fuzzy art map medical image retrieval system for disease diagnosis and prediction. Biomedicines 10(10):2438. https://doi.org/10.3390/BIOMEDICINES10102438

    Article  Google Scholar 

  32. Gonzalez-Garcia A, Modolo D, Ferrari V (2018) Do Semantic Parts Emerge in Convolutional Neural Networks? Int J Comput Vision 126(5):476–494. https://doi.org/10.1007/s11263-017-1048-0

    Article  MathSciNet  Google Scholar 

  33. Khan S, Rahmani H, Shah SAA, Bennamoun M (2018) A guide to convolutional neural networks for computer vision. Synth Lect Comput Vis 8(1):1–207. https://doi.org/10.2200/S00822ED1V01Y201712COV015

    Article  Google Scholar 

  34. Dong R, Liu M, Li F (2019) Multilayer Convolutional Feature Aggregation Algorithm for Image Retrieval. Math Probl Eng. https://doi.org/10.1155/2019/9794202

    Article  Google Scholar 

  35. Zhan Z, Zhou G, Yang X (2020) A method of hierarchical image retrieval for real-time photogrammetry based on multiple features. IEEE Access 8:21524–21533. https://doi.org/10.1109/ACCESS.2020.2969287

    Article  Google Scholar 

  36. Dubey SR, Roy SK, Chakraborty S, Mukherjee S, Chaudhuri BB (2019) Local bit-plane decoded convolutional neural network features for biomedical image retrieval. Neurals Comput Appl 32(11):7539–7551. https://doi.org/10.1007/S00521-019-04279-6

    Article  Google Scholar 

  37. Mohite NB, Gonde AB (2022) Deep features based medical image retrieval. Multimed Tools Appl 81(8):11379–11392. https://doi.org/10.1007/S11042-022-12085-X

    Article  Google Scholar 

  38. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks arXiv:1311.2901v3 [cs.CV] 28 Nov 2013. Computer Vision–ECCV 2014, 8689(PART 1), 818–833. doi:https://doi.org/10.1007/978-3-319-10590-1_53

  39. Sezavar A, Farsi H, Mohamadzadeh S (2019) Content-based image retrieval by combining convolutional neural networks and sparse representation. Multimed Tools Appl 78(15):20895–20912. https://doi.org/10.1007/S11042-019-7321-1

    Article  Google Scholar 

  40. Magesh PR, Myloth RD, Tom RJ (2020) An explainable machine learning model for early detection of parkinson’s disease using LIME on DaTSCAN imagery. Comput Biol Med 126:104041. https://doi.org/10.1016/J.COMPBIOMED.2020.104041

    Article  Google Scholar 

  41. Rodriguez-Martinez I, Lafuente J, Santiago RHN, Dimuro GP, Herrera F, Bustince H (2022) Replacing pooling functions in convolutional neural networks by linear combinations of increasing functions. Neural Netw 152:380–393. https://doi.org/10.1016/J.NEUNET.2022.04.028

    Article  Google Scholar 

  42. Bengio Y, Lecun Y, Hinton G (2021) Deep learning for AI. Commun ACM 64(7):58–65. https://doi.org/10.1145/3448250

    Article  Google Scholar 

  43. Zafar A, Aamir M, Mohd Nawi N, Arshad A, Riaz S, Alruban A, Almotairi S (2022) A comparison of pooling methods for convolutional neural networks. Appl Sci 12(17):8643. https://doi.org/10.3390/APP12178643

    Article  Google Scholar 

  44. Li H, Wu XJ (2017) Multi-focus image fusion using dictionary learning and low-rank representation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10666 LNCS 675–686. doi:https://doi.org/10.1007/978-3-319-71607-7_59/TABLES/2

  45. Bu HH, Kim NC, Park KW, Kim SH (2019) Content-based image retrieval using combined texture and color features based on multi-resolution multi-direction filtering and color autocorrelogram. J Ambient Intell Humaniz Comput 2019:1–9. https://doi.org/10.1007/S12652-019-01466-0

    Article  Google Scholar 

  46. Marcílio WE, Eler DM (2021) Explaining dimensionality reduction results using Shapley values. Expert Syst Appl 178:115020. https://doi.org/10.1016/J.ESWA.2021.115020

    Article  Google Scholar 

  47. Content Based Image Retrieval / Image Database Search Engine (SIMPLIcity, WIPE, Virtual Microscope). online available: http://wang.ist.psu.edu/docs/related/

  48. Kan S, Cen Y, Li Y, Vladimir M, He Z (2022) Local semantic correlation modeling over graph neural networks for deep feature embedding and image retrieval. IEEE Trans Image Process 31:2988–3003. https://doi.org/10.1109/TIP.2022.3163571

    Article  Google Scholar 

  49. Kalsekar A, Khade R, Jariwala K, Chattopadhyay C (2022) RISC-Net : rotation invariant siamese convolution network for floor plan image retrieval. Multimed Tools Appl 81(28):41199–41223. https://doi.org/10.1007/S11042-022-13124-3/METRICS

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, South Tehran Branch, Islamic Azad University, Tehran, Iran

    Fatemeh Taheri, Kambiz Rahbar & Ziaeddin Beheshtifard

Authors
  1. Fatemeh Taheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kambiz Rahbar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziaeddin Beheshtifard
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The contribution of the first author is conceptualization, design and research methodology, developing programs, analyzing outcomes and writing the manuscript. For Second Author the contribution is conceptualization, design and research methodology, analyzing outcomes and reviewing manuscripts. For Third Author the contribution is analyzing outcomes and reviewing manuscripts.

Corresponding author

Correspondence to Kambiz Rahbar.

Ethics declarations

Conflict of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Taheri, F., Rahbar, K. & Beheshtifard, Z. Content-based image retrieval using handcraft feature fusion in semantic pyramid. Int J Multimed Info Retr 12, 21 (2023). https://doi.org/10.1007/s13735-023-00292-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13735-023-00292-7

Keywords

Search

Navigation