Skip to main content
SpringerLink
Log in

A Deep Convolutional Neural Network-Based Approach for Visual Search & Recommendation of Grocery Products

  • Published:
Annals of Data Science Aims and scope Submit manuscript

Abstract

Search and recommendation are two essential features of any e-commerce website for finding and purchasing a specific product. Visual Search is a promising and quick method in comparison to a textual-based search method. Hence, the objective of this research is to propose a conceptual framework for developing a visual search and recommendation system for grocery products using Ensemble Learning with CNN models. Traditional Deep learning and Ensemble Learning techniques were implemented with a publicly available and a self-made data set containing 3174 and 3162 images respectively. Various combinations of the suitable models found from research findings were used to find the best-fitted model for both the search and recommendation functionalities. All the models were evaluated using suitable performance metrics and the Ensemble Learning approach performed better. The best-performed results for visual searching are obtained by incorporating VGG16 and MobileNet with an accuracy of 99.8% for classification and in the case of product recommendation, the combination of MobileNET and ResNET50 performs better than other techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

Both of the prepared dataset and Kaggle dataset are openly available in GitHub at https://github.com/The-Talking-Head/Sourcecode- and-Dataset, under the folder of Dataset and Kaggle Dataset respectively.

Code Availability

The codes for this study are openly available in GitHub at https://github.com/The-Talking-Head/Source-code-and-Dataset, under the folder of Source Code.

References

  1. Imran (2020) On-site search statistics for ecommerce and others. https://www.addsearch.com/blog/site-search-statistics. Accessed 26 June 2023

  2. pixyle.ai (2020) Is visual search making text-based search obsolete? https://www.pixyle.ai/product-discovery/is-visual-search-making-text-based-search-obsolete. Accessed 28 July 2023

  3. Shi Y (2022) Advances in big data analytics: theory, algorithm and practice. Springer, Singapore

    Book  Google Scholar 

  4. Olson DL, Shi Y (2007) Introduction to business data mining. McGraw-Hill/Irwin, New York

    Google Scholar 

  5. Shi Y, Tian YJ, Kou G, Peng Y, Li JP (2011) Optimization based data mining: theory and applications. Springer, Berlin

    Book  Google Scholar 

  6. Tien JM (2017) Internet of things, real-time decision making, and artificial intelligence. Ann Data Sci 4(2):149–178

    Article  Google Scholar 

  7. Felfernig A, Jeran M, Ninaus G, Reinfrank F and Reiterer S, Stettinger M (2014) Basic approaches in recommendation systems

  8. Umer S, Mohanta PP, Rout RK, Pandey HM (2021) Machine learning method for cosmetic product recognition: a visual searching approach. Multimed Tools Appl 80:34997–35023. https://doi.org/10.1007/s11042-020-09079-y

    Article  Google Scholar 

  9. Bishop CM, Nasrabadi NM (2006) Pattern recognition and machine learning. Springer, New York

    Google Scholar 

  10. Chang CC, Lin CJ (2011) A library for Support Vector Machines. ACM Trans Intell Syst Technol 2(3):2157–6904. https://doi.org/10.1145/1961189.1961199

    Article  Google Scholar 

  11. Mullick SS, Datta S, Das S (2018) Adaptive learning-based k-nearest neighbor classifiers with resilience to class imbalance. IEEE Trans Neural Netw Learn Syst 29(11):5713–5725. https://doi.org/10.1109/TNNLS.2018.2812279

    Article  Google Scholar 

  12. Raschka S (2015) Python machine learning. Packt Publishing Ltd, Birmingham

    Google Scholar 

  13. Safavian SR, Landgrebe D (1991) A survey of decision tree classifier methodology. IEEE Trans Syst Man Cybern 21(3):660–674. https://doi.org/10.1109/21.97458

    Article  Google Scholar 

  14. Rinaldi AM, Russo C, Tommasino C (2021) Visual query posing in multimedia web document retrieval. In: Proceedings of the 15th international conference on semantic computing (ICSC). IEEE, Laguna Hills. https://doi.org/10.1109/ICSC50631.2021.00086

  15. Smelyakov K, Sandrkin D, Ruban I, Martovytskyi V, Romanenkov Y (2018) Search by image. New search engine service model. In: Proceedings of the international scientific-practical conference problems of infocommunications. Science and Technology (PIC S &T). IEEE, Kharkiv. https://doi.org/10.1109/INFOCOMMST.2018.8632117

  16. Hu C, Li Q, Zhang Z, Chang Kh, Zhang R (2020) A multimodal fusion framework for brand recognition from product image and context. In: Proceedings of the international conference on multimedia & expo workshops (ICMEW), pp 1–4. https://doi.org/10.1109/ICMEW46912.2020.9105947

  17. Wei Z, Song X and Lin Y (2020) Image feature recognition algorithm for rural revitalization product design based on visual attention model. In: Proceedings of the 4th international conference on inventive systems and control (ICISC). IEEE, Coimbatore. https://doi.org/10.1109/ICISC47916.2020.9171207

  18. Sogi N, Souza LS, Gatto BB, Fukui K (2020) Metric learning with a-based scalar product for image-set recognition. In: Proceedings of the conference on computer vision and pattern recognition workshops (CVPRW). IEEE, Seattle. https://doi.org/10.1109/CVPRW50498.2020.00433

  19. Boriya A, Malla SS, Manjunath R, Velicheti V, Eirinaki, M (2019) ViSeR: A visual search engine for e-retail. In: Proceedings of the first international conference on transdisciplinary AI (TransAI). IEEE, Laguna Hills. https://doi.org/10.1109/TransAI46475.2019.00021

  20. Dakhare BS, Khatu A, Singh HR, Gupta A (2020) Visual E-commerce application using deep learning. Int Res J Eng Technol (IRJET) 7(3):1–9

    Google Scholar 

  21. Kim Y (2021) A study on the efficient product search service for the damaged image information. arXiv preprint arXiv:2111.07346

  22. Zhang Ya, Pan P, Zheng Y, Zhao K, Zhang Y, Ren X and Jin R (2018) Visual search at Alibaba. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 993–1001. https://doi.org/10.1145/3219819.3219820

  23. Szegedy C, Wei L, Yangqing J, Sermanet P, Reed S, Anguelov D, Erhan, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  24. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg A, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115:211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  Google Scholar 

  25. Li J, Liu H, Gui C, Chen J, Ni Z, Wang N, Chen Y (2018) The Design and implementation of a real time visual search system on JD E-commerce platform. In: Proceedings of the 19th international middleware conference industry. https://doi.org/10.1145/3284028.3284030

  26. Hu H, Wang Y, Yang L, Komlev P, Huang L, Chen X, Huang J, Wu Y, Merchant M, Sacheti A (2018) Web-scale responsive visual search at bing. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 359–367. https://doi.org/10.1145/3219819.3219843

  27. Smelyakov K, Chupryna A, Sandrkin D, Kolisnyk M (2020) Search by image engine for big data warehouse. In: Proceedings of the 2020 IEEE open conference of electrical, electronic and information sciences (eStream), pp 1–4

  28. Mu C, Zhao J, Yang G, Zhang J, Yan Z (2018) Towards practical visual search engine within elasticsearch. arXiv preprint arXiv:1806.08896

  29. Karmakar C, Datcu M (2020) A fast search system for remote sensing imagery based on bag of visual words and latent Dirichlet allocation. In: Proceeding of the international geoscience and remote sensing symposium (IGARSS). IEEE, Waikoloa. https://doi.org/10.1109/IGARSS39084.2020.9324359

  30. Krizhevsky A, Sutskever I, Hinton GE (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60:84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  31. He K, Zhang X, Ren A, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37:1904–1916. https://doi.org/10.1109/TPAMI.2015.2389824

    Article  Google Scholar 

  32. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv. https://doi.org/10.48550/arXiv.1502.03167

  33. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas. https://doi.org/10.1109/CVPR.2016.90

  34. Marichal X, De VC, Macq B (1997) Towards visual search engine based on fuzzy logic

  35. Sumaiya, Armanuzzaman M (2020) Enhancement of resulting image search engine (ERISE) by content-based image retrieval system. In: Proceedings of the IEEE region 10 symposium (TENSYMP). IEEE, Dhaka. https://doi.org/10.1109/TENSYMP50017.2020.9230653

  36. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. arXiv. https://doi.org/10.48550/arXiv.1409.1556

  37. Hur C, Hyun C, Park H (2020) Automatic image recommendation for economic topics using visual and semantic information. In: Proceedings of the 14th international conference on semantic computing (ICSC). IEEE, San Diego. https://doi.org/10.1109/ICSC.2020.00037

  38. Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Honolulu. https://doi.org/10.1109/CVPR.2017.690

  39. Tautkute I, Trzciński T, Skorup AP, Brocki L, Marasek K (2019) DeepStyle: multimodal search engine for fashion and interior design. IEEE Access 7:84613–84628. https://doi.org/10.1109/ACCESS.2019.2923552

    Article  Google Scholar 

  40. Eswaran A, Varshini E (2022) Reverse image search engine for garment industry. In: Proceedings of the 2022 8th international conference on advanced computing and communication systems (ICACCS). IEEE, Coimbatore, pp. 414–418

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Military Institute of Science and Technology (MIST), Dhaka, Bangladesh

    Nawreen Anan Khandaker, Amrin Rahman, Amrin Akter Pinky & Tasmiah Tamzid Anannya

Authors
  1. Nawreen Anan Khandaker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amrin Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amrin Akter Pinky
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tasmiah Tamzid Anannya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tasmiah Tamzid Anannya.

Ethics declarations

Conflict of interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khandaker, N.A., Rahman, A., Pinky, A.A. et al. A Deep Convolutional Neural Network-Based Approach for Visual Search & Recommendation of Grocery Products. Ann. Data. Sci. (2024). https://doi.org/10.1007/s40745-024-00540-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s40745-024-00540-5

Keywords

Search

Navigation