Relevance Feedback Models for Content-Based Image Retrieval

  • Peter Auer
  • Alex Po Leung
Part of the Studies in Computational Intelligence book series (SCI, volume 346)


We investigate models for content-based image retrieval with relevance feedback, in particular focusing on the exploration-exploitation dilemma. We propose quantitative models for the user behavior and investigate implications of these models. Three search algorithms for efficient searches based on the user models are proposed and evaluated. In the first model a user queries a database for the most (or a sufficiently) relevant image. The user gives feedback to the system by selecting the most relevant image from a number of images presented by the system. In the second model we consider a filtering task where relevant images should be extracted from a database and presented to the user. The feedback of the user is a binary classification of each presented image as relevant or irrelevant. While these models are related, they differ significantly in the kind of feedback provided by the user. This requires very different mechanisms to trade off exploration (finding out what the user wants) and exploitation (serving images which the system believes relevant for the user).


Image Retrieval Discount Factor User Model Relevance Feedback User Feedback 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Auer, P.: Using Confidence Bounds for Exploitation-Exploration Trade-offs. Journal of Machine Learning Research 3, 397–422 (2002)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Leung, A.P., Auer, P.: An Efficient Search Algorithm for Content-Based Image Retrieval with User Feedback. In: 1st Int. Workshop on Video Mining (VM 2008) in association with IEEE International Conference on Data Mining, ICDM 2008 (2008)Google Scholar
  3. 3.
    Chang, E., Tong, S., Goh, K., Chang, C.: Support Vector Machine Concept-Dependent Active Learning for Image Retrieval. IEEE Transactions on Multimedia (2005)Google Scholar
  4. 4.
    Chen, Y., Zhou, X.S., Huang, T.S.: One-class SVM for learning in image retrieval. In: Proc. ICIP (1), pp. 34–37 (2001)Google Scholar
  5. 5.
    Crucianu, M., Ferecatu, M., Boujemaa, N.: Relevance feedback for image retrieval: a short survey. State of the Art in Audiovisual Content-Based Retrieval. Information Universal Access and Interaction, Including Datamodels and Languages, report of the DELOS2 European Network of Excellence, FP6, 20 (2004)Google Scholar
  6. 6.
    Dani, V., Hayes, T.P., Kakade, S.M.: Stochastic Linear Optimization under Bandit Feedback. In: Proc. 21st Ann. Conf. on Learning Theory, pp. 355–366 (2008)Google Scholar
  7. 7.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. (2008)Google Scholar
  8. 8.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2007 Results (2007),
  9. 9.
    Fournier, J., Cord, M.: Long-term similarity learning in content-based image retrieval. In: Proc. ICIP (1), pp. 441–444 (2002)Google Scholar
  10. 10.
    Gosselin, P.-H., Cord, M., Philipp-Foliguet, S.: Active learning methods for Interactive Image Retrieval. IEEE Transactions on Image Processing (2008)Google Scholar
  11. 11.
    He, X., King, O., Ma, W., Li, M., Zhang, H.: Learning a semantic space from user’s relevance feedback for image retrieval. IEEE Trans. Circuits Syst. Video Techn., 39–48 (2003)Google Scholar
  12. 12.
    Jing, F., Li, M., Zhang, H., Zhang, B.: A unified framework for image retrieval using keyword and visual features. IEEE Transactions on Image Processing, 979–989 (2005)Google Scholar
  13. 13.
    Karp, R.M., Kleinberg, R.: Noisy binary search and its applications. In: SODA 2007: Proc. 18th Symp. on Discrete Algorithms, pp. 881–890 (2007)Google Scholar
  14. 14.
    Koskela, M., Laaksonen, J.: Using Long-Term Learning to Improve Efficiency of Content-Based Image Retrieval. In: Proc. PRIS, pp. 72–79 (2003)Google Scholar
  15. 15.
    Koskela, M., Laaksonen, J., Oja, E.: Inter-Query Relevance Learning in PicSOM for Content-Based Image Retrieval. In: Kaynak, O., Alpaydın, E., Oja, E., Xu, L. (eds.) ICANN 2003 and ICONIP 2003. LNCS, vol. 2714, Springer, Heidelberg (2003)Google Scholar
  16. 16.
    Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(7), 1088–1099 (2006)CrossRefGoogle Scholar
  17. 17.
    Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: State of the art and challenges. TOMCCAP, 1–19 (2006)Google Scholar
  18. 18.
    Linenthal, J., Qi, X.: An Effective Noise-Resilient Long-Term Semantic Learning Approach to Content-Based Image Retrieval. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2008), Las Vegas, Nevada, USA, March 30-April 4 (2008)Google Scholar
  19. 19.
    Tao, D., Li, X., Maybank, S.J.: Negative Samples Analysis in Relevance Feedback. IEEE Trans. Knowl. Data Eng. 19(4), 568–580 (2007)CrossRefGoogle Scholar
  20. 20.
    Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. Information and Computation, 212–216 (1994)Google Scholar
  21. 21.
    Pelc, A.: Searching games with errors–fifty years of coping with liars. Theoretical Computer Science, 71-109 (2002)Google Scholar
  22. 22.
    Tao, D., Tang, X.: Nonparametric Discriminant Analysis in Relevance Feedback for Content-based Image Retrieval. In: IEEE International Conference on Pattern Recognition (ICPR), pp. 1013–1016 (2004)Google Scholar
  23. 23.
    Rocchio, J.: Relevance Feedback in Information Retrieval. In: Salton: The SMART Retrieval System: Experiments in Automatic Document Processing, ch. 14, pp. 313–323. Prentice-Hall, Englewood Cliffs (1971)Google Scholar
  24. 24.
    Rui, Y., Huang, T.S.: Optimizing Learning in Image Retrieval. In: Proc. CVPR, pp. 1236–1236 (2000)Google Scholar
  25. 25.
    Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-Based Image Retrieval at the End of the Early Years. IEEE Trans. Pattern Anal. Mach. Intell., 1349–1380 (2000)Google Scholar
  26. 26.
    Tong, S., Chang, E.Y.: Support vector machine active learning for image retrieval. In: Proc. ACM Multimedia, pp. 107–118 (2001)Google Scholar
  27. 27.
    Veltkamp, R.C., Tanase, M.: Content-based Image Retrieval Systems: a Survey. State-of-the-Art in Content-Based Image and Video Retrieval, 97–124 (1999)Google Scholar
  28. 28.
    Wacht, M., Shan, J., Qi, X.: A Short-Term and Long-Term Learning Approach for Content-Based Image Retrieval. In: Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2006), Toulouse, France, May 14-19, pp. 389–392 (2006)Google Scholar
  29. 29.
    Zhang, C., Chen, T.: An active learning framework for content-based information retrieval. IEEE Transactions on Multimedia, 260–268 (2002)Google Scholar
  30. 30.
    Zhou, X.S., Huang, T.S.: Unifying Keywords and Visual Contents in Image Retrieval. In: IEEE MultiMedia, pp. 23–33 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Peter Auer
    • 1
  • Alex Po Leung
    • 1
  1. 1.Department Mathematik und InformationstechnologieMontanuniversität at LeobenLeobenAustria

Personalised recommendations