Dynamic Exploratory Search in Content-Based Image Retrieval

  • Joel PyykköEmail author
  • Dorota Głowacka
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10269)


With the increase of digital media databases, the need for methods that can allow the user to efficiently peruse them has risen dramatically. This paper studies how to explore image datasets more efficiently in online content-based image retrieval (CBIR). We present a new approach for exploratory CBIR that is dynamic, robust and gives a good coverage of the search space, while maintaining a high retrieval precision. Our method uses deep similarity-based learning to find a new representation of the image space. With this metric, it finds the central point of interest and clusters its local region to present the user with representative images within the vicinity of their target search. This clustering provides a more varied training set for the next iteration, allowing the location of relevant features faster. Additionally, relearning a representation of the user’s search interest in each round enables the system to find other non-local regions of interest in the search space, thus preventing the user from getting stuck in a context trap. We test our method in a simulated online setting, taking into consideration the accuracy, coverage and flexibility of adapting to changes in the user’s interest.


Content based image retrieval (CBIR) Deep neural networks Vector space models Interactive information retrieval Exploratory search 


  1. 1.
    Berg, T.L., Berg, A.C.: Finding iconic images. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8, June 2009Google Scholar
  2. 2.
    Camargo, J.E., Caicedo, J.C., Gonzalez, F.A.: A kernel-based framework for image collection exploration. J. Vis. Lang. Comput. 24(1), 53–67 (2013)CrossRefGoogle Scholar
  3. 3.
    Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. J. Mach. Learn. Res. 11, 1109–1135 (2010)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Chen, Y., Wang, J.Z., Krovetz, R.: Clue: cluster-based retrieval of images by unsupervised learning. IEEE Trans. Image Process. 14(8), 1187–1201 (2005)CrossRefGoogle Scholar
  5. 5.
    Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of CVPR (2005)Google Scholar
  6. 6.
    Dimitriadou, K., Papaemmanouil, O., Diao, Y.: Explore-by-example: an automatic query steering framework for interactive data exploration. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SIGMOD 2014), pp. 517–528, New York, NY, USA. ACM (2014)Google Scholar
  7. 7.
    Ester, M., Kriegel, H.-P., Sander, J., Xiaowei, X.: A density-based algorithm for discovering clusters in large spatial databases with noise, pp. 226–231. AAAI Press (1996)Google Scholar
  8. 8.
    Głowacka, D., Shawe-Taylor, J.: Content-based image retrieval with multinomial relevance feedback. In: Proceedings of ACML, pp. 111–125 (2010)Google Scholar
  9. 9.
    Głowacka, D., Hore, S.: Balancing exploration-exploitation in image retrieval. In: Proceedings of UMAP (2014)Google Scholar
  10. 10.
    Glowacka, D., Teh, Y.W., Shawe-Taylor, J.: Image retrieval with a Bayesian model of relevance feedback. arXiv preprint (2016). arXiv:1603.09522
  11. 11.
    Heesch, D.: A survey of browsing models for content based image retrieval. Multimed. Tools Appl. 40(2), 261–284 (2008)CrossRefGoogle Scholar
  12. 12.
    Hoque, E., Hoeber, O., Gong, M.: CIDER: concept-based image diversification, exploration, and retrieval. Inf. Process. Manage. 49(5), 1122–1138 (2013)CrossRefGoogle Scholar
  13. 13.
    Hore, S., Glowacka, D., Kosunen, I., Athukorala, K., Jacucci, G.: Futureview: enhancing exploratory image search. In: IntRS@RecSys, pp. 37–40 (2015)Google Scholar
  14. 14.
    Hore, S., Tyrvainen, L., Pyykko, J., Glowacka, D.: A reinforcement learning approach to query-less image retrieval. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 121–126. Springer, Cham (2014). doi: 10.1007/978-3-319-13500-7_10 Google Scholar
  15. 15.
    Huiskes, M.J., Lew, M.S.: The MIR flickr retrieval evaluation. In: Proceedings of MIR (2008)Google Scholar
  16. 16.
    Kato, T., Kurita, T., Otsu, N., Hirata, K.: A sketch retrieval method for full color image database-query by visual example. In: Pattern Recognition. Computer Vision and Applications, pp. 530–533 (1992)Google Scholar
  17. 17.
    Kelly, D., Xin, F.: Elicitation of term relevance feedback: an investigation of term source and context. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), pp. 453–460, New York, NY, USA. ACM (2006)Google Scholar
  18. 18.
    Konyushkova, K., Glowacka, D.: Content-based image retrieval with hierarchical gaussian process bandits with self-organizing maps. In: ESANN (2013)Google Scholar
  19. 19.
    Krizhevsky, A., Hinton, G.E.: Using very deep autoencoders for content-based image retrieval. In: Proceedings of ESANN (2011)Google Scholar
  20. 20.
    Lowe, D.G.: Object recognition from local scale-invariant features. In: ICCV, pp. 1150–1157 (1999)Google Scholar
  21. 21.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3), 145–175 (2001)CrossRefzbMATHGoogle Scholar
  22. 22.
    Pyykko, J., Glowacka, D.: Interactive content-based image retrieval with deep neural networks. In: Gamberini, L., et al. (eds.) Symbiotic 2016. LNCS, vol. 9961. Springer, Heidelberg (2017)Google Scholar
  23. 23.
    Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. CoRR, abs/1403.6382 (2014)Google Scholar
  24. 24.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2014)Google Scholar
  25. 25.
    Salton, G.: The SMART Retrieval System-Experiments in Automatic Document Processing. Prentice-Hall Inc., Upper Saddle River (1971)Google Scholar
  26. 26.
    Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: integrated recognition, localization and detection using convolutional networks. In: Proceedings of ICLR (2014)Google Scholar
  27. 27.
    Suditu, N., Fleuret, F.: Iterative relevance feedback with adaptive exploration/exploitation trade-off. In: Proceedings of CIKM (2012)Google Scholar
  28. 28.
    Wan, J., Wang, D., Hoi, S.C.H., Wu, P., Zhu, J., Zhang, Y., Li, J.: Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of MM (2014)Google Scholar
  29. 29.
    Wan, J., Wang, D., Hoi, S.C., Wu, P., Zhu, J., Zhang, Y., Li, J.: Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the 22nd ACM International Conference on Multimedia (MM 2014), pp. 157–166, New York, NY, USA. ACM (2014)Google Scholar
  30. 30.
    Watkins, C.J.C.H.: Learning from delayed rewards. Ph.D. thesis, King’s College, Cambridge, UK, May 1989Google Scholar
  31. 31.
    Yang, J., Jiang, Y.-G., Hauptmann, A.G., Ngo, C.-W.: Evaluating bag-of-visual-words representations in scene classification. In: Multimedia, Information Retrieval, pp. 197–206 (2007)Google Scholar
  32. 32.
    Yee, K.-P., Swearingen, K., Li, K., Hearst, M.: Faceted metadata for image search and browsing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2003), pp. 401–408, New York, NY, USA. ACM (2003)Google Scholar
  33. 33.
    Zhao, F., Huang, Y., Wang, L., Tan, T.: Deep semantic ranking based hashing for multi-label image retrieval. ArXiv e-prints, January 2015Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of HelsinkiHelsinkiFinland

Personalised recommendations