Extraction of Visual Features for Recommendation of Products via Deep Learning

  • Elena AndreevaEmail author
  • Dmitry I. Ignatov
  • Artem Grachev
  • Andrey V. Savchenko
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11179)


In this paper (The first author is the 1st place winner of the Open HSE Student Research Paper Competition (NIRS) in 2017, Computer Science nomination, with the topic “Extraction of Visual Features for Recommendation of Products”, as alumni of 2017 “Data Science” master program at Computer Science Faculty, HSE, Moscow), we describe a special recommender approach based on features extracted from the clothes’ images. The method of feature extraction relies on pre-trained deep neural network that follows transfer learning on the dataset. Recommendations are generated by the neural network as well. All the experiments are based on the items of category Clothing, Shoes and Jewelry from Amazon product dataset. It is demonstrated that the proposed approach outperforms the baseline collaborative filtering method.


Visual feature Recommender system Fine tuning Collaborative filtering 



This study is supported by Russian Federation President grant MD-306.2017.9. Andrey V. Savchenko is supported by the Laboratory of Algorithms and Technologies for Network Analysis, National Research University Higher School of Economics. The work of Dmitry I. Ignatov (contributed to Sects. 1, 2, 3 and 6) was supported by the Russian Science Foundation under grant 17-11-01294 and performed at National Research University Higher School of Economics, Russia. The authors would like to thank Dmitry Soloviev from and Gleb Gusev from Yandex team for their piece of advice and relevant literature on the problem.


  1. 1.
    Bell, S., Bala, K.: Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. (TOG) 34(4), 98 (2015)CrossRefGoogle Scholar
  2. 2.
    Corbière, C., Ben-Younes, H., Ramé, A., Ollion, C.: Leveraging weakly annotated data for fashion image retrieval and label prediction. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 2268–2274 (2017)Google Scholar
  3. 3.
    Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the Fourth ACM Conference on Recommender Systems, pp. 39–46. ACM (2010)Google Scholar
  4. 4.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
  5. 5.
    Jiang, S., Wu, Y., Fu, Y.: Deep bi-directional cross-triplet embedding for cross-domain clothing retrieval. In: Proceedings of the 2016 ACM on Multimedia Conference, MM 2016, pp. 52–56, New York, NY, USA. ACM (2016)Google Scholar
  6. 6.
    Kang, W.C., Fang, C., Wang, Z., McAuley, J.: Visually-aware fashion recommendation and design with generative image models. In ICDM, pp. 207–216. IEEE Computer Society (2017)Google Scholar
  7. 7.
    Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: Deepfashion: powering robust clothes recognition and retrieval with rich annotations. In: CVPR, pp. 1096–1104. IEEE Computer Society (2016)Google Scholar
  8. 8.
    McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommendations on styles and substitutes. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43–52. ACM (2015)Google Scholar
  9. 9.
    Shankar, D., Narumanchi, S., Ananya, H.A., Kompalli, P., Chaudhury, K.: Deep learning based large scale visual recommendation and search for e-commerce. arXiv preprint arXiv:1703.02344 (2017)
  10. 10.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)Google Scholar
  11. 11.
    Veit, A., Kovacs, B., Bell, S., McAuley, J., Bala, K., Belongie, S.: Learning visual clothing style with heterogeneous dyadic co-occurrences. In: International Conference on Computer Vision (ICCV), Santiago, Chile (2015)Google Scholar
  12. 12.
    Zhai, A., et al.: Visual discovery at pinterest. In: Proceedings of the 26th International Conference on World Wide Web Companion, pp. 515–524. International World Wide Web Conferences Steering Committee (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Elena Andreeva
    • 1
    • 3
    Email author
  • Dmitry I. Ignatov
    • 1
  • Artem Grachev
    • 1
  • Andrey V. Savchenko
    • 2
  1. 1.National Research University Higher School of EconomicsMoscowRussia
  2. 2.National Research University Higher School of EconomicsNizhniy NovgorodRussia
  3. 3.ex Mail.Ru GroupMoscowRussia

Personalised recommendations