Relevance Feedback in Content-Based Image Retrieval: A Survey

  • Jing Li
  • Nigel M. Allinson
Part of the Intelligent Systems Reference Library book series (ISRL, volume 49)


In content-based image retrieval, relevance feedback is an interactive process, which builds a bridge to connect users with a search engine. It leads to much improved retrieval performance by updating a query and similarity measures according to a user’s preference; and recently techniques have matured to some extent. Most previous relevance feedback approaches exploit short-term learning (intraquery learning) that deals with the current feedback session but ignoring historical data from other users, which potentially results in a great loss of useful information. In the last few years, long-term learning (inter-query learning), by recording and collecting feedback knowledge from different users over a variety of query sessions has played an increasingly important role in multimedia information searching. It can further improve the retrieval performance in terms of effectiveness and efficiency. In the published literature, no comprehensive survey of both short-term learning and long-term learning RF techniques has been conducted. To this end, the goal of this chapter is to address this omission and offer suggestions for future work.


Support Vector Machine Image Retrieval Relevance Feedback Scale Invariant Feature Transform Semantic Concept 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anick, P.: Using Terminological Feedback for Web Search Refinement - A Log-based Study. In: Proc. Int’l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 88–95 (2003)Google Scholar
  2. 2.
    Bian, W., Tao, D.: Biased Discriminant Euclidean Embedding for Content-Based Image Retrieval. IEEE Transactions on Image Processing 19(2), 545–554 (2010)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Blum, A., Mitchell, T.: Combining Labeled and Unlabeled Data with Cotraining. In: COLT: Proc. Workshop on Computational Learning Theory (1998)Google Scholar
  4. 4.
    Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1995)Google Scholar
  5. 5.
    Burges, J.C.: A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery 2(2), 121–167 (1998)CrossRefGoogle Scholar
  6. 6.
    Carson, C., Belongie, S., Greenspan, H., Malik, J.: Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying. IEEE Trans. Pattern Analysis and Machine Intelligence 24(8), 1026–1038 (2002)CrossRefGoogle Scholar
  7. 7.
    Chang, T., Kuo, C.: Texture Analysis and Classification with Tree-Structured Wavelet Transform. IEEE Trans. Image Processing 2(4), 429–441 (1993)CrossRefGoogle Scholar
  8. 8.
    Chang, H., Yeung, D.Y.: Stepwise Metric Adaptation Based on Semi-Supervised Learning for Boosting Image Retrieval Performance. In: British Machine Vision Conference (2005)Google Scholar
  9. 9.
    Chen, X., Zhang, C., Chen, S.C., Chen, M.: A Latent Semantic Indexing Based Method for Solving Multiple Instance Learning Problem in Region-Based Image Retrieval. In: Proc. IEEE Int’l Symposium Multimedia, pp. 37–45 (2005)Google Scholar
  10. 10.
    Chen, Y., Zhou, X., Huang, T.S.: One-class SVM for Learning in Image Retrieval. In: Proc. IEEE Int’l Conf. Image Processing, vol. 1, pp. 34–37 (2001)Google Scholar
  11. 11.
    Cord, M., Gosselin, P.H.: Image Retrieval using Long-Term Semantic Learning. In: IEEE Int’l Conf. Image Processing, pp. 2,909–2,912 (2006)Google Scholar
  12. 12.
    Cox, I.J., Miller, M.L., Minka, T.P., Papathomas, T.V., Yianilos, P.N.: The Bayesian Image Retrieval System, PicHunter: Theory, Implementation and Psychophysical Experiments. IEEE Trans. Image Processing 9(1), 20–37 (2000)CrossRefGoogle Scholar
  13. 13.
    Cui, H., Wen, J., Nie, J., Ma, W.: Query Expansion by Mining User Logs. IEEE Trans. Knowledge and Data Engineering 15(4), 829–939 (2003)CrossRefGoogle Scholar
  14. 14.
    Daubechies, I.: The Wavelet Transform, Time-frequency Localization and Signal Analysis. IEEE Trans. Information Theory 36(5), 961–1005 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Daugman, J.G.: Two-Dimensional Spectral Analysis of Cortical Receptive Field Profile. Vision Research 20, 847–856 (1980)CrossRefGoogle Scholar
  16. 16.
    Daugman, J.G.: Uncertainty Relation for Resolution in Space, Spatial Frequency, and Orientation Optimized by Two–Dimensional Visual Cortical Filters. J. Optical Society of America 2(7), 1,160–1,169 (1985)Google Scholar
  17. 17.
    Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. American Society for Information Science 41, 391–407 (1990)CrossRefGoogle Scholar
  18. 18.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet: A Large-scale Hierarchical Image Database. In: IEEE Conf. Computer Vision and Pattern Recognition (2009)Google Scholar
  19. 19.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. Wiley, John & Sons, Incorporated (1999)Google Scholar
  20. 20.
    Fahlman, S.E., Lebiere, C.: The Cascade-Correlation Learning Architecture. In: Advances in Neural Information Processing Systems 2, pp. 524–532 (1990)Google Scholar
  21. 21.
    Freund, Y., Schapire, R.E.: Experiments with a New Boosting Algorithm. In: Proc. Int’l Conf. Machine Learning, pp. 148–156 (1996)Google Scholar
  22. 22.
    Fournier, J., Cord, M.: Long-term Similarity Learning in Content-based Image Retrieval. In: Proc. IEEE Int’l Conf. Image Processing, vol. 1, pp. 1,441–1,444 (2002)Google Scholar
  23. 23.
    Gevers, T., Smeulders, A.: Pictoseek: Combining color and shape invariant features for image retrieval. IEEE Trans. Image Processing 9(1), 102–119 (2000)CrossRefGoogle Scholar
  24. 24.
    Giacinto, G., Roli, F.: Instance-Based Relevance Feedback for Image Retrieval. In: Advances Neural Information Processing Systems, pp. 489–496 (2004)Google Scholar
  25. 25.
    Gosselin, P.H., Cord, M.: Semantic Kernel Learning for Interactive Image Retrieval. In: IEEE Int’l Conf. Image Processing, vol. 1, 1,177–1,180 (2005)Google Scholar
  26. 26.
    Guo, G., Jain, A.K., Ma, W., Zhang, H.: Learning Similarity Measure for Natural Image Retrieval with Relevance Feedback. IEEE Trans. Neural Networks 12(4), 811–820 (2002)Google Scholar
  27. 27.
    Han, J., Ngan, K.N., Li, M., Zhang, H.J.: A Memory Learning Framework for Effective Image Retrieval. IEEE Trans. Image Processing 14(4), 511–524 (2005)CrossRefGoogle Scholar
  28. 28.
    He, X., King, O., Ma, W., Li, M., Zhang, H.: Learning a Semantic Space from User’s Relevance Feedback for Image Retrieval. IEEE Trans. Circuits and Systems for Video Technology 13(1), 39–48 (2003)CrossRefGoogle Scholar
  29. 29.
    He, X., Ma, W.Y., Zhang, H.J.: Learning an Image Manifold for Retrieval. In: Proc. ACM Int’l Conf. Multimedia, pp. 17–23 (2004)Google Scholar
  30. 30.
    Heisterkamp, D.R.: Building a Latent Semantic Index of an Image Database from Patterns of Relevance Feedback. In: Proc. Int’l Conf. Pattern Recognition, vol. 4, pp. 134–137 (2002)Google Scholar
  31. 31.
    Ho, T.K.: The Random Subspace Method for Constructing Decision Forests. IEEE Trans. Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)CrossRefGoogle Scholar
  32. 32.
    Hoi, C.H., Chan, C.H., Huang, K., Lyu, M.R., King, I.: Biased Support Vector Machine for Relevance Feedback in Image Retrieval. In: Proc. Int’l Joint Conf. Neural Networks, pp. 3189–3194 (2004)Google Scholar
  33. 33.
    Hoi, C.H., Lyu, M.R.: Group-based Relevance Feedback with Support Vector Machine Ensembles. In: Int’l Conf. Pattern Recognition, vol. 3, pp. 874–877 (2004)Google Scholar
  34. 34.
    Hoi, S.C.H., Lyu, M.R., Jin, R.: A Unified Log-based Relevance Feedback Scheme for Image Retrieval. IEEE Trans. Knowledge and Data Engineering 18(4), 509–524 (2006)CrossRefGoogle Scholar
  35. 35.
    Hsu, C.T., Li, C.Y.: Relevance Feedback Using Generalized Bayesian Framework With Region-Based Optimization Learning. IEEE Trans. Image Processing 14(10), 1,617–1,631 (2005)Google Scholar
  36. 36.
    Jain, A., Vailaya, A.: Image Retrieval Using Color and Shape. Pattern Recognition 29(8), 1233–1244 (1996)CrossRefGoogle Scholar
  37. 37.
    Jain, A.K.: Fundamentals of Digital Image Processing. Prentice Hall Inc., New Jersey (1989)zbMATHGoogle Scholar
  38. 38.
    Jain, A.K., Lee, J.E., Jin, R., Gregg, N.: Content-based Image Retrieval: An Application to Tattoo Images. In: IEEE Int’l Conf. Image Processing, vol. 2, pp. 745–742 (2009)Google Scholar
  39. 39.
    Jiang, W., Er, G., Dai, Q., Gu, J.: Hidden Annotation for Image Retrieval with Long-Term Relevance Feedback Learning. Pattern Recognition 38(11), 2,007–2,021 (2005)Google Scholar
  40. 40.
    Jing, F., Li, M., Zhang, H., Zhang, B.: Relevance Feedback in Region-Based Image Retrieval. IEEE Trans. Circuits and Systems for Video Technology 14(5), 672–681 (2004)CrossRefGoogle Scholar
  41. 41.
    Koskela, M., Laaksonen, J.: Using Long-Term Learning to Improve Efficiency of Content-Based Image Retrieval. In: Proc. Int’l Workshop on Pattern Recognition in Information Systems, pp. 72–79 (2003)Google Scholar
  42. 42.
    Kushki, A., Androutsos, P., Plataniotis, K.N., Venetsanopoulos, A.N.: Query Feedback for Interactive Image Retrieval. IEEE Trans. Circuits and Systems for Video Technology 14, 644–655 (2004)CrossRefGoogle Scholar
  43. 43.
    Laaksonen, J., Koskela, M., Laakso, S., Oja, E.: PicSOM—Content-based Image Retrieval with Self-Organizing Maps. Pattern Recognition Letters 21(13-14), 1,199–1,207 (2000)Google Scholar
  44. 44.
    LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation 1(4), 541–551 (1989)CrossRefGoogle Scholar
  45. 45.
    Leung, A.P., Auer, P.: An Efficient Search Algorithm for Content-Based Image Retrieval with User Feedback. In: IEEE Int’l Conf. Data Mining Workshops, pp. 884–890 (2008)Google Scholar
  46. 46.
    Li, J., Allinson, N.M.: A Comprehensive Review of Current Local Features for Computer Vision. Neurocomputing 71(10-12), 1771–1787 (2008)CrossRefGoogle Scholar
  47. 47.
    Li, J., Allinson, N.M.: Long-term Learning in Content-based Image Retrieval. Int’l J. Imaging Systems and Technology 18(2-3), 160–169 (2008)CrossRefGoogle Scholar
  48. 48.
    Li, J., Allinson, N., Tao, D., Li, X.: Multitraining Support Vector Machine for Image Retrieval. IEEE Trans. Image Processing 15(11), 3,597–3,601 (2006)Google Scholar
  49. 49.
    Long, F., Zhang, H., Feng, D.: Fundamentals of Content-based Image Retrieval. In: Multimedia Information Retrieval and Management, pp. 1–12. Springer (2003)Google Scholar
  50. 50.
    Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. Int’l J. Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  51. 51.
    Lu, Y., Zhang, H., Liu, W., Hu, C.: Joint Semantics and Feature Based Image Retrieval Using Relevance Feedback. IEEE Trans. Multimedia 5(3), 339–347 (2003)CrossRefGoogle Scholar
  52. 52.
    Manjunath, B., Ma, W.: Texture Features for Browsing and Retrieval of Image Data. IEEE Trans. Pattern Analysis and Machine Intelligence 18(8), 837–842 (1996)CrossRefGoogle Scholar
  53. 53.
    Manjunath, B., Ohm, J., Vasudevan, V., Yamada, A.: Color and Texture Descriptors. IEEE Trans. Circuits and Systems for Video Technology 11(6), 703–715 (2001)CrossRefGoogle Scholar
  54. 54.
    Mikolajczyk, K., Schmid, C.: A Performance Evaluation of Local Descriptors. IEEE Trans. Pattern Analysis and Machine Intelligence 27(10), 1,615–1,630 (2005)Google Scholar
  55. 55.
    Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Van Gool, L.: A Comparison of Affine Region Detectors. Int’l J. Computer Vision 65(1/2), 43–72 (2005)CrossRefGoogle Scholar
  56. 56.
    Minka, T.P., Picard, R.W.: Interactive Learning with a “Society of Models”. Pattern Recognition 30(4), 565–581 (1997)CrossRefGoogle Scholar
  57. 57.
    Müller, H., Müller, W., Squire, D.M., Marchand-Maillet, S., Pun, T.: Long-Term Learning from User Behavior in Content-Based Image Retrieval. Technical report, University of Geneva (2000)Google Scholar
  58. 58.
    Nakazato, M., Dagli, C., Huang, T.S.: Evaluating Group-based Relevance Feedback for Content-based Image Retrieval. In: Proc. IEEE Int’l. Conf. on Image Processing, pp. 599–602 (2003)Google Scholar
  59. 59.
    Nedovic, V., Marques, O.: A Collaborative, Long-term Learning Approach to Using Relevance Feedback in Content-based Image Retrieval Systems. In: Int’l Symposium ELMAR, pp. 143–146 (2005)Google Scholar
  60. 60.
    Niblack, W., Barber, R., Equitz, W., Flickner, M., Glasman, E., Petkovic, D., Yanker, P., Faloutsos, C., Taubin, G.: The QBIC project: Quering Images by Content using Color, Texture, and Shape. In: Proc. SPIE Storage and Retrieval for Image and Video Databases, pp. 173–187 (1993)Google Scholar
  61. 61.
    Ortega, M., Rui, Y., Chakrabarti, K., Mehrotra, S., Huang, T.S.: Supporting similarity queries in MARS. In: Proc. ACM Int’l Conf. Multimedia, pp. 403–413 (1997)Google Scholar
  62. 62.
    Pass, G., Zabih, R., Miller, J.: Comparing Images Using Color Coherence Vectors. In: Proc. ACM Int’l Conf. Multimedia, pp. 65–73 (1996)Google Scholar
  63. 63.
    Rocchio, J.J.: Document Retrieval System: Optimization and Evaluation. PhD dissertation, Harvard Computational Lab, Harvard University, Cambridge, MA (1996)Google Scholar
  64. 64.
    Rubner, Y., Tomasi, C., Guibas, L.J.: The Earth Mover’s Distance as a Metric for Image Retrieval. Int’l J. Computer Vision 40(2), 99–121 (2000)zbMATHCrossRefGoogle Scholar
  65. 65.
    Rui, Y., Huang, T.S., Ortega, M., Mehrotra, S.: Relevance Feedback: A Power Tool in Interactive Content-based Image Retrieval. IEEE Trans. Circuits and Systems for Video Technology 8(5), 644–655 (1998)CrossRefGoogle Scholar
  66. 66.
    Sav, S., O’Connor, N., Smeaton, A., Murphy, N.: Associating Low-level Features with Semantic Concepts using Video Objects and Relevance Feedback. In: Int’l Workshop on Image Analysis for Multimedia Interactive Services (2005)Google Scholar
  67. 67.
    Schapire, R.E.: The Strength of Weak Learnability. Machine Learning 5(2), 197–227 (1990)Google Scholar
  68. 68.
    Schmid, C., Mohr, R.: Local Grayvalue Invariants for Image Retrieval. IEEE Trans. Pattern Analysis and Machine Intelligence 19(5), 530–534 (1997)CrossRefGoogle Scholar
  69. 69.
    Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Analysis and Machine Intelligence 22(12), 1,349–1,380 (2000)Google Scholar
  70. 70.
    Su, Z., Li, S., Zhang, H.: Extraction of Feature Subspaces for Content-Based Retrieval Using Relevance Feedback. In: Proc. of ACM Multimedia, pp. 98–106 (2001)Google Scholar
  71. 71.
    Su, Z., Zhang, H., Li, S., Ma, S.: Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces, and Progressive Learning. IEEE Trans. Image Processing 12(8), 924–937 (2003)CrossRefGoogle Scholar
  72. 72.
    Swain, M., Ballard, D.: Color Indexing. Int’l J. Computer Vision 7(1), 11–32 (1991)CrossRefGoogle Scholar
  73. 73.
    Tai, X., Ren, F., Kita, K.: Long-Term Relevance Feedback Using Simple PCA and Linear Transformation. In: Proc. Int’l Workshop on Database and Expert Systems Applications, pp. 261–268 (2002)Google Scholar
  74. 74.
    Tamura, H., Mori, S., Yamawaki, T.: Texture Features Corresponding to Visual Perception. IEEE Trans. Systems, Man, and Cybernetics 8(6), 460–473 (1978)CrossRefGoogle Scholar
  75. 75.
    Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric Bagging and Random Subspace for Support Vector Machines-based Relevance Feedback in Image Retrieval. IEEE Trans. Pattern Analysis and Machine Intelligence 28(7), 1,088–1,099 (2006)Google Scholar
  76. 76.
    Tao, D., Tang, X., Li, X., Rui, Y.: Kernel Direct Biased Discriminant Analysis: A New Content-based Image Retrieval Relevance Feedback Algorithm. IEEE Trans. Multimedia 8(4), 716–727 (2006)zbMATHCrossRefGoogle Scholar
  77. 77.
    Tao, D., Li, X., Maybank, S.J.: Negative Samples Analysis in Relevance Feedback. IEEE Trans. Knowledge and Data Engineering 19(4), 568–580 (2007)CrossRefGoogle Scholar
  78. 78.
    Tong, S., Chang, E.: Support Vector Machine Active Learning for Image Retrieval. In: Proc. ACM Int’l Conf. Multimedia, pp. 107–118 (2001)Google Scholar
  79. 79.
    Torralba, A., Fergus, R., Freeman, W.T.: 80 Million Tiny Images: A Large Dataset for Non-parametric Object and Scene Recognition. IEEE Trans. Pattern Analysis and Machine Intelligence 30(11), 1958–1970 (2008)CrossRefGoogle Scholar
  80. 80.
    Torre, F.D.L., Kanade, T.: Multimodal Oriented Discriminant Analysis. In: Int’l Conf. Machine Learning (2005)Google Scholar
  81. 81.
    Tversky, A.: Features of Similarity. Psychological Review 84(4), 327–352 (1977)CrossRefGoogle Scholar
  82. 82.
    Urban, J., Jose, J.M.: Adaptive Image Retrieval using a Graph Model for Semantic Feature Integration. In: Proc. ACM Int’l Workshop on Multimedia Information Retrieval, pp. 117–126 (2006)Google Scholar
  83. 83.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer (1995)Google Scholar
  84. 84.
    Vasconcelos, N., Kunt, M.: Content-based Retrieval from Image Databases: Current Solutions and Future Directions. In: Proc. IEEE Int’l Conf. Image Processing, vol. 3, pp. 6–9 (2001)Google Scholar
  85. 85.
    Wacht, M., Shan, J., Qi, X.: A Short-Term and Long-Term Learning Approach for Content-Based Image Retrieval. In: Proc. IEEE Int’l Conf. Acoustics, Speech, and Signal Processing, vol. 2, pp. 389–392 (2006)Google Scholar
  86. 86.
    Wang, J.Z., Li, J., Wiederhold, G.: SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries. IEEE Trans. Pattern Analysis and Machine Intelligence 23(9), 947–963 (2001)CrossRefGoogle Scholar
  87. 87.
    Yang, J., Li, Q., Zhuang, Y.: Image Retrieval and Relevance Feedback using Peer Indexing. In: Proc. IEEE Int’l Conf. Multimedia and Expo, vol. 2, pp. 409–412 (2002)Google Scholar
  88. 88.
    Yin, P.Y., Bhanu, B., Chang, K.C., Dong, A.: Integrating Relevance Feedback Techniques for Image Retrieval Using Reinforcement Learning. IEEE Trans. Pattern Analysis and Machine Intelligence 27(10), 1,536–1,551 (2005)Google Scholar
  89. 89.
    Yoshizawa, T., Schweitzer, H.: Long-term Learning of Semantic Grouping from Relevance-feedback. In: Proc. ACM SIGMM Int’l Workshop on Multimedia Information Retrieval, pp. 165–172 (2004)Google Scholar
  90. 90.
    Zhang, L., Lin, F., Zhang, B.: Support Vector Machine Learning for Image Retrieval. In: Proc. IEEE Int. Conf. Image Processing, vol. 2, pp. 21–724 (2001)Google Scholar
  91. 91.
    Zhou, X.S., Huang, T.S.: Comparing Discriminating Transformations and SVM for Learning during Multimedia Retrieval. In: Proc. ACM International Conference on Multimedia (2001)Google Scholar
  92. 92.
    Zhou, X.S., Huang, T.S.: Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems 8(6), 536–544 (2003)CrossRefGoogle Scholar
  93. 93.
    Zhou, X.S., Huang, T.S.: Small Sample Learning during Multimedia Retrieval using Biasmap. In: Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 11–17 (2001)Google Scholar
  94. 94.
    Zhou, X.D., Zhang, L., Liu, L., Zhang, Q., Shi, B.: A Relevance Feedback Method in Image Retrieval by Analyzing Feedback Log File. In: Proc. Int’l Conf. Machine Learning and Cybernetics, vol. 3, pp. 1,641–1,646 (2002)Google Scholar
  95. 95.
    Zhou, Z.H.: Ensemble Learning. In: Encyclopedia of Biometrics, pp. 270–273 (2009)Google Scholar
  96. 96.
    Zhuang, Y., Yang, J., Li, Q., Pan, Y.: A Graphic-theoretic Model for Incremental Relevance Feedback in Image Retrieval. In: Proc. Int’l Conf. Image Processing, vol. 1, pp. 413–416 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.School of Information EngineeringNanchang UniversityNanchangPeoples Republic of China
  2. 2.School of Computer ScienceUniversity of LincolnLincolnUK

Personalised recommendations