Ten Years of Relevance Score for Content Based Image Retrieval

  • Lorenzo PutzuEmail author
  • Luca Piras
  • Giorgio Giacinto
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10935)


After more than 20 years of research on Content-Based Image Retrieval (CBIR), the community is still facing many challenges to improve the retrieval results by filling the semantic gap between the user needs and the automatic image description provided by different image representations. Including the human in the loop through Relevance Feedback (RF) mechanisms turned out to help improving the retrieval results in CBIR. In this paper, we claim that Nearest Neighbour approaches still provide an effective method to assign a Relevance Score to images, after the user labels a small set of images as being relevant or not to a given query. Although many other approaches to relevance feedback have been proposed in the past ten years, we show that the Relevance Score, while simple in its implementation, allows attaining superior results with respect to more complex approaches, can be easily adopted with any feature representations. Reported results on different real-world datasets with a large number of classes, characterised by different degrees of semantic and visual intra- e inter-class variability, clearly show the current challenges faced by CBIR system in reaching acceptable retrieval performances, and the effectiveness of Nearest neighbour approaches to exploit Relevance Feedback.


Image retrieval Image description Relevance feedback Nearest neighbour 



This work has been supported by the Regional Administration of Sardinia (RAS), Italy, within the project BS2R - Beyond Social Semantic Recommendation (POR FESR 2007/2013 - PIA 2013).


  1. 1.
    Arevalillo-Herráez, M., Domingo, J., Ferri, F.J.: Combining similarity measures in content-based image retrieval. Pattern Recogn. Lett. 29(16), 2174–2181 (2008)CrossRefGoogle Scholar
  2. 2.
    Arevalillo-Herráez, M., Ferri, F.J., Domingo, J.: A naive relevance feedback model for content-based image retrieval using multiple similarity measures. Pattern Recogn. 43(3), 619–629 (2010)CrossRefGoogle Scholar
  3. 3.
    Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 584–599. Springer, Cham (2014). Scholar
  4. 4.
    Rosdi, B.A., Shing, C.W., Suandi, S.A.: Finger vein recognition using local line binary pattern. Sensors 11, 11357–11371 (2011)CrossRefGoogle Scholar
  5. 5.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  6. 6.
    Chatzichristofis, S.A., Boutalis, Y.S.: CEDD: color and edge directivity descriptor: a compact descriptor for image indexing and retrieval. In: Gasteratos, A., Vincze, M., Tsotsos, J.K. (eds.) ICVS 2008. LNCS, vol. 5008, pp. 312–322. Springer, Heidelberg (2008). Scholar
  7. 7.
    Chen, H.: A socio-technical perspective of museum practitioners’ image-using behaviors. The Electron. Libr. 25(1), 18–35 (2007)CrossRefGoogle Scholar
  8. 8.
    Chen, Y., Zhou, X.S., Huang, T.: One-class SVM for learning in image retrieval. ICIP 1, 34–37 (2001)Google Scholar
  9. 9.
    Cohn, D.A., Atlas, L.E., Ladner, R.E.: Improving generalization with active learning. Mach. Learn. 15(2), 201–221 (1994)Google Scholar
  10. 10.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of CVPR, pp. 886–893 (2005)Google Scholar
  11. 11.
    Dang-Nguyen, D.T., Piras, L., Giacinto, G., Boato, G., De Natale, F.G.B.: Multimodal retrieval with diversification and relevance feedback for tourist attraction images. ACM Trans. Multimedia Comput. Commun. Appl. 13(4), 49:1–49:24 (2017)CrossRefGoogle Scholar
  12. 12.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 1–60 (2008)CrossRefGoogle Scholar
  13. 13.
    Escalante, H.J., Hérnadez, C.A., Sucar, L.E., Montes, M.: Late fusion of heterogeneous methods for multimedia image retrieval. In: Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, pp. 172–179 (2008)Google Scholar
  14. 14.
    Giacinto, G.: A nearest-neighbor approach to relevance feedback in content based image retrieval. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, CIVR 2007, pp. 456–463. ACM, New York (2007)Google Scholar
  15. 15.
    Giacinto, G., Roli, F.: Bayesian relevance feedback for content-based image retrieval. Pattern Recogn. 37(7), 1499–1508 (2004)CrossRefGoogle Scholar
  16. 16.
    Giacinto, G., Roli, F.: Nearest-prototype relevance feedback for content based image retrieval. In: ICPR, vol. 2, pp. 989–992 (2004)Google Scholar
  17. 17.
    Graps, A.: An introduction to wavelets. IEEE Comput. Sci. Eng. 2(2), 50–61 (1995)CrossRefGoogle Scholar
  18. 18.
    Hoi, S.C.H., Jin, R., Zhu, J., Lyu, M.R.: Semisupervised SVM batch mode active learning with applications to image retrieval. ACM Trans. Inf. Syst. 27(3), 16:1–16:29 (2009)CrossRefGoogle Scholar
  19. 19.
    Jégou, H., Zisserman, A.: Triangulation embedding and democratic aggregation for image search. In: Proceedings of CVPR, pp. 3310–3317. IEEE Computer Society (2014)Google Scholar
  20. 20.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: 26th Annual Conference on Advances in Neural Information Processing Systems, pp. 1106–1114 (2012)Google Scholar
  21. 21.
    Laaksonen, J., Koskela, M., Oja, E.: PicSOM-self-organizing image retrieval with MPEG-7 content descriptors. IEEE Trans. Neural Netw. 13(4), 841–853 (2002)CrossRefGoogle Scholar
  22. 22.
    Lee, T.S.: Image representation using 2D Gabor wavelets. IEEE Trans. Pattern Anal. Mach. Intell. 18(10), 959–971 (1996)CrossRefGoogle Scholar
  23. 23.
    van Leuken, R.H., Garcia, L., Olivares, X., van Zwol, R.: Visual diversification of image search results. In: ACM International Conference on World Wide Web, pp. 341–350 (2009)Google Scholar
  24. 24.
    Liang, S., Sun, Z.: Sketch retrieval and relevance feedback with biased SVM classification. Pattern Recogn. Lett. 29(12), 1733–1741 (2008)CrossRefGoogle Scholar
  25. 25.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  26. 26.
    Mitro, J.: Content-based image retrieval tutorial. ArXiv e-prints (2016)Google Scholar
  27. 27.
    Müller, H., Clough, P.D., Deselaers, T., Caputo, B. (eds.): ImageCLEF: Experimental Evaluation in Visual Information Retrieval. Springer, Heidelberg (2010). Scholar
  28. 28.
    Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefGoogle Scholar
  29. 29.
    Passerini, A., Pontil, M., Frasconi, P.: New results on error correcting output codes of kernel machines. IEEE Trans. Neural Netw. 15(1), 45–54 (2004)CrossRefGoogle Scholar
  30. 30.
    Pavlidis, T.: Limitations of content-based image retrieval. Technical report, Stony Brook University (2008)Google Scholar
  31. 31.
    Piras, L., Giacinto, G.: Neighborhood-based feature weighting for relevance feedback in content-based retrieval. In: WIAMIS, pp. 238–241. IEEE Computer Society (2009)Google Scholar
  32. 32.
    Piras, L., Giacinto, G.: Information fusion in content based image retrieval: a comprehensive overview. Inf. Fusion 37, 50–60 (2017)CrossRefGoogle Scholar
  33. 33.
    Piras, L., Giacinto, G., Paredes, R.: Enhancing image retrieval by an exploration-exploitation approach. In: Perner, P. (ed.) MLDM 2012. LNCS (LNAI), vol. 7376, pp. 355–365. Springer, Heidelberg (2012). Scholar
  34. 34.
    Piras, L., Giacinto, G., Paredes, R.: Passive-aggressive online learning for relevance feedback in content based image retrieval. In: Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods, pp. 182–187 (2013)Google Scholar
  35. 35.
    Piras, L., Tronci, R., Giacinto, G.: Diversity in ensembles of codebooks for visual concept detection. In: Petrosino, A. (ed.) ICIAP 2013. LNCS, vol. 8157, pp. 399–408. Springer, Heidelberg (2013). Scholar
  36. 36.
    Rocchio, J.J.: Relevance feedback in information retrieval, pp. 313–323. Prentice Hall, Englewood Cliffs (1971)Google Scholar
  37. 37.
    Rui, Y., Huang, T.S., Mehrotra, S.: Content-based image retrieval with relevance feedback in MARS. In: International Conference on Image Processing Proceedings, pp. 815–818, October 1997Google Scholar
  38. 38.
    Rui, Y., Huang, T.S., Mehrotra, S.: Relevance feedback: a power tool in interactive content-based image retrieval. IEEE Trans. Circuits Syst. Video Technol. 8(5), 644–655 (1998)CrossRefGoogle Scholar
  39. 39.
    da Torres, R.S., Falcão, A.X., Gonçalves, M.A., Papa, J.P., Zhang, B., Fan, W., Fox, E.A.: A genetic programming framework for content-based image retrieval. Pattern Recogn. 42(2), 283–292 (2009)CrossRefGoogle Scholar
  40. 40.
    Sivic, J., Zisserman, A.: Efficient visual search for objects in videos. Proc. IEEE 96(4), 548–566 (2008)CrossRefGoogle Scholar
  41. 41.
    Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)CrossRefGoogle Scholar
  42. 42.
    Tong, S., Chang, E.Y.: Support vector machine active learning for image retrieval. In: ACM Multimedia, pp. 107–118 (2001)Google Scholar
  43. 43.
    Tronci, R., Murgia, G., Pili, M., Piras, L., Giacinto, G.: ImageHunter: a novel tool for relevance feedback in content based image retrieval. In: Lai, C., Semeraro, G., Vargiu, E. (eds.) New Challenges in Distributed Information Filtering and Retrieval. SCI, vol. 439, pp. 53–70. Springer, Heidelberg (2013). Scholar
  44. 44.
    Tsai, C.M., Qamra, A., Chang, E., Wang, Y.F.: Extent: interring image metadata from context and content. In: IEEE International Conference on Multimedia and Expo, pp. 1270–1273 (2006)Google Scholar
  45. 45.
    Yu, J., Qin, Z., Wan, T., Zhang, X.: Feature integration analysis of bag-of-features model for image retrieval. Neurocomputing 120, 355–364 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Pluribus OneCagliariItaly
  2. 2.Department of Electrical and Electronic EngineeringUniversity of CagliariCagliariItaly

Personalised recommendations