Two Step Relevance Feedback for Semantic Disambiguation in Image Retrieval

  • Daniel Heesch
  • Stefan Rüger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5188)


This paper presents a new approach to the problem of feature weighting for content based image retrieval. If a query image admits to multiple interpretations, user feedback on the set of returned images can be an effective tool to improve retrieval performance in subsequent rounds. For this to work, however, the first results set has to include representatives of the semantic facet of interest. We will argue that relevance feedback techniques that fix the distance metric for the first retrieval round are semantically biased and may fail to distil relevant semantic facets thus limiting the scope of relevance feedback. Our approach is based on the notion of the NN k of a query image, defined as the set of images that are nearest neighbours of the query under some instantiation of a parametrised distance metric. Different neighbours may be viewed as representing different meanings of the query. By associating each NN k with the parameters for which it was ranked closest to the query, the selection of relevant NN k by a user provides us with parameters for the second retrieval round. We evaluate this two step relevance feedback technique on two collections and compare it to an alternative relevance feedback method and to an oracle for which the optimal parameter values are known.


Weight Vector Image Retrieval Average Precision Query Image Relevance Feedback 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, G., Ashwin, T., Ghosal, S.: An image retrieval system with automatic query modification. IEEE Trans Multimedia, 4(2), 201–213 (2002)CrossRefGoogle Scholar
  2. 2.
    Aslam, J., Montague, M.: Models for metasearch. In: Proc. Int’l ACM SIGIR, pp. 276–284 (2001)Google Scholar
  3. 3.
    Ghahramani, Z., Heller, K.: Bayesian sets. In: Proc. NIPS (2005)Google Scholar
  4. 4.
    Goldberger, J., Roweis, S., Hinton, G., Salakhutdinov, R.: Neighbourhood component analysis. In: Proc. NIPS (2005)Google Scholar
  5. 5.
    Gosselin, P., Cord, M.: Active learning methods for interactive image retrieval. IEEE Trans. Image Processing 17(7), 1200–1211 (2008)CrossRefGoogle Scholar
  6. 6.
    Heesch, D.: The Open image in new window technique for image searching and browsing. PhD thesis, Imperial College London (2005)Google Scholar
  7. 7.
    Heesch, D., Rüger, S.: Interaction models and relevance feedback in content-based image retrieval. In: Zhang, Y.-J. (ed.) Semantic-Based Visual Information Retrieval, pp. 160–186. Idea-Group (2006)Google Scholar
  8. 8.
    Heller, K., Ghahramani, Z.: A simple Bayesian framework for content-based image retrieval. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 2110–2117 (2006)Google Scholar
  9. 9.
    Kim, D.-H., Chung, C.-W.: Qcluster: relevance feedback using adaptive clustering for content-based image retrieval. In: Proc. ACM SIGMOD Int’l Conf. Management of Data, pp. 599–610 (2003)Google Scholar
  10. 10.
    Nguyen, G., Worring, M., Smeulders, A.: Interactive search by direct manipulation of dissimilarity space. IEEE Trans. Multimedia 9(7), 1404–1415 (2007)CrossRefGoogle Scholar
  11. 11.
    Qin, T., Zhang, X.-D., Liu, T.-Y., Wang, D.-S., Ma, W.-Y., Zhang, H.-J.: An active feedback framework for image retrieval. Pattern Recognition Letters 29, 637–646 (2008)CrossRefGoogle Scholar
  12. 12.
    Rui, Y., Huang, T.: Optimizing learning in image retrieval. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 236–243 (2000)Google Scholar
  13. 13.
    Rui, Y., Huang, T., Ortega, M., Mehrotra, S.: Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans. Circuits, Systems and Video Technology 8(5), 644–655 (1998)CrossRefGoogle Scholar
  14. 14.
    Sclaroff, S., Taycher, L., La Cascia, M.: ImageRover: A content-based image browser for the WWW. Technical report, Boston University (1997)Google Scholar
  15. 15.
    Urban, J., Jose, J.: Evidence combination for multi-point query learning in content-based image retrieval. In: Proc. IEEE Int’l Symposium Multimedia Software Engineering, pp. 583–586 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Daniel Heesch
    • 1
  • Stefan Rüger
    • 2
  1. 1.Pixsta ResearchLondonUnited Kingdom
  2. 2.Knowledge Media InstituteThe Open UniversityMilton KeynesUnited Kingdom

Personalised recommendations