Skip to main content
SpringerLink
Log in

Ornament Image Retrieval Using Multimodal Fusion

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Search-by-example, i.e. finding images that are similar to a query image, is an indispensable function for various modern image search engines. The applications of such systems are manifold. The primary application of search-by-example is in recommending fashion materials based on user interests. There are various challenges in this area of research such as a large volume of the product database, similar visual appearances, and a large variety of products. The problem becomes more difficult to solve when the product is complex in design such as ornaments. In this paper, we have proposed a fusion-based retrieval model. The method uses weighted average of multiple similarity measures. We have used four different methods namely hash-based, histogram-based, deep feature comparison, and feature cross correlation to find the similarity. A dataset of ornaments (golden earrings) has been prepared and made available to the research community. We achieve 81% top-1 and 89% top-5 accuracy using the proposed method. The dataset and the code is available publicly in https://github.com/skarifahmed/RingFIR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Abdel-Nabi H, Al-Naymat G, Awajan A. Content based image retrieval approach using deep learning. In: 2019 2nd international conference on new trends in computing sciences (ICTCS), IEEE; 2019. p. 1–8.

  2. Datta R, Joshi D, Li J, Wang JZ. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput Surv. 2008;40(2):5.

    Article  Google Scholar 

  3. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, IEEE; 2009. p. 248–255

  4. Dey M, Raman B, Verma M (2016) A novel colour-and texture-based image retrieval technique using multi-resolution local extrema peak valley pattern and rgb colour histogram. Pattern Analysis and Applications 19(4), 1159–1179.

    Article  MathSciNet  Google Scholar 

  5. Dey S, Dutta A, Ghosh SK, Valveny E, Lladós J, Pal U. Learning cross-modal deep embeddings for multi-object image retrieval using text and sketch. In: International conference on pattern recognition, IEEE; 2018. p. 916–921.

  6. Ge Y, Zhang R, Wu L, Wang X, Tang X, Luo P. Deepfashion2: a versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. arXiv preprint arXiv:1901079732019.

  7. Gonde AB, Maheshwari R, Balasubramanian R (2013) Modified curvelet transform with vocabulary tree for content based image retrieval. Digital Signal Processing 23(1), 142–150.

    Article  MathSciNet  Google Scholar 

  8. Grace S, Annadurai S (2008) Content based image retrieval for medical images using generic fourier descriptor. Journal of computational Intelligence in Bioinformatics 1(1), 65–72.

    Google Scholar 

  9. Guo JM, Prasetyo H, Su HS. Image indexing using the color and bit pattern feature fusion. J Vis Commun Image Represent. 2013;24(8):1360–79.

    Article  Google Scholar 

  10. Ha I, Kim H, Park S, Kim H (2018) Image retrieval using bim and features from pretrained vgg network for indoor localization. Building and Environment 140:23–31.

    Article  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition; 2016. p. 770–778.

  12. Höschl IV C, Flusser J (2016) Robust histogram-based image retrieval. Pattern Recognition Letters 69:72–81.

    Article  Google Scholar 

  13. Huang J, Feris RS, Chen Q, Yan S. Cross-domain image retrieval with a dual attribute-aware ranking network. In: IEEE international conference on computer vision; 2015. p. 1062–1070.

  14. Jenitta A, Ravindran RS (2017) Image retrieval based on local mesh vector co-occurrence pattern for medical diagnosis from mri brain images. Journal of Medical Systems 41(10):157.

    Article  Google Scholar 

  15. Jeong D, Kim BG. Dong SY (2020) Deep joint spatiotemporal network (djstn) for efficient facial expression recognition. Sensors 20(7):1936.

    Article  Google Scholar 

  16. Jetchev N, Bergmann U. The conditional analogy gan: Swapping fashion articles on people images. In: IEEE international conference on computer vision, 2017. p. 2287–2292.

  17. Jhansi Y, Reddy ES. An efficient sketch based image retrieval using cross-correlation. International Journal of Computer Science and Information Security. 2016;14(12):445.

    Google Scholar 

  18. Ji X, Wang W, Zhang M, Yang Y. Cross-domain image retrieval with attention modeling. In: ACM international conference on multimedia, ACM; 2017. p. 1654–1662.

  19. Jiang W, Er G, Dai Q, Gu J (2006) Similarity-based online feature selection in content-based image retrieval. IEEE Transactions on Image Processing 15(3), 702–712.

    Article  Google Scholar 

  20. Jiang YG, Wang J, Xue X, Chang SF (2012) Query-adaptive image search with hash codes. IEEE Transactions on Multimedia 15(2), 442–453.

    Article  Google Scholar 

  21. Kailath T. The divergence and bhattacharyya distance measures in signal selection. IEEE Transactions on Communication Technology. 1967;15(1):52–60.

    Article  Google Scholar 

  22. Kalantidis Y, Mellina C, Osindero S (2016) Cross-dimensional weighting for aggregated deep convolutional features. In: European conference on computer vision. Springer, Berlin, pp 685–701.

    Google Scholar 

  23. Kekre H, Thepade SD, Banura VK. Amelioration of walsh-hadamard texture patterns based image retrieval using hsv color space. International Journal of Computer Science and Information Security. 2011;9(3):64.

    Google Scholar 

  24. Kim JH, Hong GS, Kim BG, Dogra DP. deepgesture: Deep learning-based gesture recognition scheme using motion sensors. Displays. 2018;55:38–45.

    Article  Google Scholar 

  25. Kim JH, Kim BG, Roy PP, Jeong DM (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285.

    Article  Google Scholar 

  26. Kong B, Supan J, Ramanan D, Fowlkes CC. Cross-domain image matching with deep feature maps. Int J Comput Vis. 2019;1–13.

  27. Liao L, He X, Zhao B, Ngo CW, Chua TS. Interpretable multimodal retrieval for fashion products. In: 2018 ACM multimedia conference on multimedia conference, ACM; 2018. p. 1571–1579.

  28. Lin K, Yang HF, Hsiao JH, Chen CS. Deep learning of binary hash codes for fast image retrieval. In: IEEE conference on computer vision and pattern recognition workshops; 2015a. p. 27–35.

  29. Lin K, Yang HF, Liu KH, Hsiao JH, Chen CS. Rapid clothing retrieval via deep learning of binary codes and hierarchical search. In: ACM international conference on multimedia retrieval, ACM; 2015b. p. 499–502.

  30. Liu GH, Yang JY (2013) Content-based image retrieval using color difference histogram. Pattern Recognition 46(1), 188–198.

    Article  Google Scholar 

  31. Liu H, Wang R, Shan S, Chen X. Deep supervised hashing for fast image retrieval. In: IEEE conference on computer vision and pattern recognition; 2016a. p. 2064–2072.

  32. Liu Z, Luo P, Qiu S, Wang X, Tang X. Deepfashion: powering robust clothes recognition and retrieval with rich annotations. In: IEEE conference on computer vision and pattern recognition; 2016b. p. 1096–1104.

  33. Liu L, Shen F, Shen Y, Liu X, Shao L. Deep sketch hashing: Fast free-hand sketch-based image retrieval. In: IEEE conference on computer vision and pattern recognition; 2017a. p. 2862–2871.

  34. Liu P, Guo JM, Chamnongthai K, Prasetyo H (2017b) Fusion of color histogram and lbp-based features for texture image retrieval and classification. Information Sciences 390:95–111.

    Article  Google Scholar 

  35. Luo Y, Wang Z, Huang Z, Yang Y, Lu H. Snap and find: Deep discrete cross-domain garment image retrieval. arXiv preprint arXiv:190402887, 2019.

  36. Manfredi M, Grana C, Calderara S, Cucchiara R (2014) A complete system for garment segmentation and color classification. Machine Vision and Applications 25(4), 955–969.

    Article  Google Scholar 

  37. Mistry Y, Ingole D, Ingole M. Content based image retrieval using hybrid features and various distance metric. J Electr Syst Inf Technol; 2017.

  38. Murala S, Maheshwari R, Balasubramanian R (2012) Local tetra patterns: a new feature descriptor for content-based image retrieval. IEEE Transactions on Image Processing 21(5), 2874–2886.

    Article  MathSciNet  MATH  Google Scholar 

  39. Nan B, Xu Y, Mu Z, Chen L. Content-based image retrieval using local texture-based color histogram. In: 2015 IEEE 2nd international conference on cybernetics, IEEE; 2015. p. 399–405.

  40. Nodari A, Ghiringhelli M, Zamberletti A, Vanetti M, Albertini S, Gallo I. A mobile visual search application for content based image retrieval in the fashion domain. In: International workshop on content-based multimedia indexing, IEEE; 2012. p. 1–6.

  41. Pal N, Kilaru A, Savaria Y, Lakhssassi A (2018) Hybrid features of tamura texture and shape-based image retrieval. In: Recent findings in intelligent computing techniques, Springer, Berlin, p. 587–597.

    Chapter  Google Scholar 

  42. Papushoy A, Bors AG (2015) Image retrieval based on query by saliency content. Digital Signal Processing 36:156–173.

    Article  MathSciNet  Google Scholar 

  43. Pelka O, Nensa F, Friedrich CM. Annotation of enhanced radiographs for medical image retrieval with deep convolutional neural networks. PloS one. 2018;13(11): e0206229.

    Article  Google Scholar 

  44. Peng HQL. Research of content-based image retrieval technology micro-computer. Manag Control Integr. 2011;23:158–67.

    Google Scholar 

  45. Piras L, Giacinto G. Information fusion in content based image retrieval: A comprehensive overview. Information Fusion. 2017;37:50–60.

    Article  Google Scholar 

  46. Qayyum A, Anwar SM, Awais M, Majid M (2017) Medical image retrieval using deep convolutional neural network. Neurocomputing 266:8–20.

    Article  Google Scholar 

  47. Rahimi M, Moghaddam ME (2015) A content-based image retrieval system based on color ton distribution descriptors. Signal, Image and Video Processing 9(3), 691–704.

    Article  Google Scholar 

  48. Reta C, Solis-Moreno I, Cantoral-Ceballos JA, Alvarez-Vargas R, Townend P. Improving content-based image retrieval for heterogeneous datasets using histogram-based descriptors. Multimedia Tools and Applications. 2018;77(7):8163–93.

    Article  Google Scholar 

  49. Sabahi F, Ahmad MO, Swamy M. Content-based image retrieval using perceptual image hashing and hopfield neural network. In: IEEE international midwest symposium on circuits and systems, IEEE; 2018. p. 352–355.

  50. Snoek J, Larochelle H, Adams RP. Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems; 2012. p. 2951–2959.

  51. Suhasini PS, Krishna KSR, Krishna IM. Content based image retrieval based on different global and local color histogram methods: a survey. J Inst Eng India Ser B. 2017; 98(1):129–135.

    Article  Google Scholar 

  52. Valem LP, Pedronette DCG (2020) Unsupervised selective rank fusion for image retrieval tasks. Neurocomputing 377:182–199.

    Article  Google Scholar 

  53. Vasileva MI, Plummer BA, Dusad K, Rajpal S, Kumar R, Forsyth D. Learning type-aware embeddings for fashion compatibility. In: European conference on computer vision; 2018. p. 390–405.

  54. Walia E, Pal A (2014) Fusion framework for effective color image retrieval. Journal of Visual Communication and Image Representation 25(6), 1335–1348.

    Article  Google Scholar 

  55. Wan J, Wang D, Hoi SCH, Wu P, Zhu J, Zhang Y, Li J. Deep learning for content-based image retrieval: A comprehensive study. In: Proceedings of the 22nd ACM international conference on multimedia; 2014. p. 157–166.

  56. Wang XY, Yu YJ, Yang HY. An effective image retrieval scheme using color, texture and shape features. Computer Standards & Interfaces. 2011;33(1):59–68.

    Article  Google Scholar 

  57. Wang W, Xu Y, Shen J, Zhu SC. Attentive fashion grammar network for fashion landmark detection and clothing category classification. In: IEEE conference on computer vision and pattern recognition; 2018. p. 4271–4280.

  58. Xu J, Shi C, Qi C, Wang C, Xiao B. Unsupervised part-based weighting aggregation of deep convolutional features for image retrieval. In: AAAI conference on artificial intelligence; 2018.

  59. Yager RR. On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans Syst Man Cybern. 1988;18(1):183–90.

    Article  MATH  Google Scholar 

  60. Yang X, He X, Wang X, Ma Y, Feng F, Wang M, Chua TS. Interpretable fashion matching with rich attributes. In: Special interest group on information retrieval; 2019.

  61. Yasmin M, Mohsin S, Sharif M. Intelligent image retrieval techniques: a survey. Journal of applied research and technology. 2014;12(1):87–103.

    Article  Google Scholar 

  62. Younus ZS, Mohamad D, Saba T, Alkawaz MH, Rehman A, Al-Rodhaan M, Al-Dhelaan A (2015) Content-based image retrieval using pso and k-means clustering algorithm. Arabian Journal of Geosciences 8(8), 6211–6224.

    Article  Google Scholar 

  63. Zhang J, Peng Y (2018) Query-adaptive image retrieval by deep-weighted hashing. IEEE Transactions on Multimedia 20(9), 2400–2414.

    Article  Google Scholar 

  64. Zhang J, Lu C, Li X, Kim HJ, Wang J (2019) A full convolutional network based on densenet for remote sensing scene classification. Math Biosci Eng 16(5), 3345–3367.

    Article  Google Scholar 

  65. Zhao F, Huang Y, Wang L, Tan T. Deep semantic ranking based hashing for multi-label image retrieval. In: IEEE conference on computer vision and pattern recognition; 2015. p. 1556–1564.

  66. Zhao B, Feng J, Wu X, Yan S. Memory-augmented attribute manipulation networks for interactive fashion search. In: IEEE conference on computer vision and pattern recognition; 2017. p. 1520–1528.

  67. Zheng S, Yang F, Kiapour MH, Piramuthu R. Modanet: a large-scale street fashion dataset with polygon annotations. In: ACM multimedia conference on multimedia conference, ACM, 2018. p. 1670–1678.

  68. Zhou W, Mok P, Zhou Y, Zhou Y, Shen J, Qu Q, Chau K. Fashion recommendations through cross-media information retrieval. Journal of Visual Communication and Image Representation. 2019;61:112–20.

    Article  Google Scholar 

  69. Zhu L, Shen J, Xie L, Cheng Z. Unsupervised visual hashing with semantic assistant for content-based image retrieval. IEEE Transactions on Knowledge and Data Engineering. 2016;29(2):472–86.

    Article  Google Scholar 

Download references

Funding

This study is not funded from anywhere.

Author information

Authors and Affiliations

  1. Global Institute of Science and Technology, Purba Medinipur, India

    Sk Maidul Islam

  2. Haldia Institute of Technology, Purba Medinipur, India

    Subhankar Joardar

  3. Indian Institute of Technology Bhubaneswar, Bhubaneswar, India

    Debi Prosad Dogra

  4. UiT The Arctic University of Norway, Troms, Norway

    Arif Ahmed Sekh

Authors
  1. Sk Maidul Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Subhankar Joardar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Debi Prosad Dogra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arif Ahmed Sekh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sk Maidul Islam.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Islam, S.M., Joardar, S., Dogra, D.P. et al. Ornament Image Retrieval Using Multimodal Fusion. SN COMPUT. SCI. 2, 336 (2021). https://doi.org/10.1007/s42979-021-00734-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-021-00734-1

Keywords

Search

Navigation