Interactive Product Search Based on Global and Local Visual-Semantic Features

  • Tomáš Skopal
  • Ladislav PeškaEmail author
  • Tomáš Grošup
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11223)


In this paper, we present a prototype web application of a product search engine of a fashion e-shop. Today, e-shop product metadata consist of text description, simple attributes (price, size, color, fabric, etc.) and visual information (product photo). Search engines used in e-shops mostly provide text and attribute/category interface for product filtering. In our model, we focus on the visual information applied in an interactive query-by-example scenario. The global visual descriptors may be often ambiguous and may not correspond well with the intended mental query of the user. Therefore, we proposed and evaluated model and GUI allowing user to guide the query process by selecting image regions (patches) of interest within the query. In the demo evaluation, we show that allowing user to specify relevant image patches led to a significant improvement of the results’ relevance in the vast majority of tested queries.



This research has been supported by Czech Science Foundation (GAČR) project Nr. 17-22224S.


  1. 1.
    Hsiao, J.H., Li, L.J.: On visual similarity based interactive product recommendation for online shopping. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 3038–3041, October 2014Google Scholar
  2. 2.
    Skopal, T., Peška, L., Kovalčík, G., Grosup, T., Lokoč, J.: Product exploration based on latent visual attributes. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, pp. 2531–2534. ACM (2017)Google Scholar
  3. 3.
    Fischer, P., Dosovitskiy, A., Brox, T.: Descriptor matching with convolutional neural networks: a comparison to sift. CoRR, vol. abs/1405.5769 (2014)Google Scholar
  4. 4.
    Long, J., Zhang, N., Darrell, T.: Do convnets learn correspondence? CoRR, vol. abs/1411.1091 (2014)Google Scholar
  5. 5.
    Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3279–3286, June 2015Google Scholar
  6. 6.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. CoRR, vol. abs/1504.03641 (2015)Google Scholar
  7. 7.
    Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: DeepFashion: powering robust clothes recognition and retrieval with rich annotations. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 1096–1104 (2016)Google Scholar
  8. 8.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). Scholar
  9. 9.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates Inc. (2012)Google Scholar
  10. 10.
    Čech, P., Maroušek, J., Lokoč, J., Silva, Y.N., Starks, J.: Comparing MapReduce-based k-NN similarity joins on Hadoop for high-dimensional data. In: Cong, G., Peng, W.-C., Zhang, W.E., Li, C., Sun, A. (eds.) ADMA 2017. LNCS (LNAI), vol. 10604, pp. 63–75. Springer, Cham (2017). Scholar
  11. 11.
    Peska, L., Trojanova, H.: Towards recommender systems for police photo lineup. In: Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems, DLRS 2017, pp. 19–23. ACM (2017)Google Scholar
  12. 12.
    Lokoc, J., Bailer, W., Schoeffmann, K., Muenzer, B., Awad, G.: On influential trends in interactive video retrieval: video browser showdown 2015–2017. IEEE Trans. Multimed. (2018).

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Tomáš Skopal
    • 1
  • Ladislav Peška
    • 1
    Email author
  • Tomáš Grošup
    • 1
  1. 1.Faculty of Mathematics and Physics, SIRET Research GroupCharles UniversityPragueCzech Republic

Personalised recommendations