Semantic Image Retrieval Using Region-Based Relevance Feedback

  • José Manuel Torres
  • David Hutchison
  • Luís Paulo Reis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4398)


A structured vocabulary of terms, such as a textual thesaurus, provides a way to conceptually describe visual information. The retrieval model described in this paper combines a conceptual and a visual layer as a first step towards the integration of ontologies and content-based image retrieval. Terms are related with image regions through a weighted association. This model allows the execution of concept-level queries, fulfilling user expectations and reducing the so-called semantic gap. Region-based relevance feedback is used to improve the quality of results in each query session and to help in the discovery of associations between text and image. The learning mechanism, whose function is to discover existing term-region associations, is based on a clustering algorithm applied over the features space and on propagation functions, which acts in each cluster where new information is available from user interaction. This approach is validated with the presentation of promising results obtained using the VOIR - Visual Object Information Retrieval system.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barthes, R.: Rhetoric of the Image (trans. by Stephen Heath). In: Barthes, R. (ed.) Image, music, text, pp. 32–51. Fontana, London (1977)Google Scholar
  2. 2.
    Carson, C., et al.: Blobworld: Image segmentation using Expectation-Maximization and its application to image querying. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(8), 1026–1038 (2002)CrossRefGoogle Scholar
  3. 3.
    Duygulu, P.: Translating images to words: A novel approach for object recognition. PhD Middle East Technical University, Dept. of Computer Engineering (2003)Google Scholar
  4. 4.
    Dwork, C., et al.: Rank aggregation methods for the Web. In: Proc. of tenth international conference on World Wide Web, Hong Kong, pp. 613–622 (2001)Google Scholar
  5. 5.
    Eakins, J.P.: Towards intelligent image retrieval. Pattern Recognition 35, 3–14 (2002)CrossRefMATHGoogle Scholar
  6. 6.
    Jain, A.K., Murty, M.N., Flynn, P.J.: Data Clustering: A Review. ACM Computing Surveys 31(3), 264–323 (1999)CrossRefGoogle Scholar
  7. 7.
    Jing, F., et al.: Relevance Feedback in Region-Based Image Retrieval. IEEE Transactions on Circuits and Systems for Video Technology 14(5), 672–681 (2004)CrossRefGoogle Scholar
  8. 8.
    Kingscote, A.: The Australian Pictorial Thesaurus 2 years on. In: DC-ANZ Metadata Conference, Australian National University, Canberra (2003)Google Scholar
  9. 9.
    Lee, C.S., Ma, W.Y., Zhang, H.J.: Information Embedding Based on User’s Relevance Feedback for Image Retrieval. SPIE Photonic East, Boston, USA (1999)Google Scholar
  10. 10.
    Li, J., Wang, J., Wiederhold, G.: IRM: Integrated Region Matching for Image Retrieval. In: Proc. ACM Multimedia 2000, Los Angeles, CA, USA, pp. 147–156. ACM, New York (2000)CrossRefGoogle Scholar
  11. 11.
    Lu, Y., et al.: A unified framework for semantics and feature based relevance feedback in image retrieval systems. In: Proc. of ACM Multimedia 2000, Los Angeles, USA, pp. 31–38. ACM Press, New York (2000)CrossRefGoogle Scholar
  12. 12.
    Ma, W.Y., Manjunath, B.S.: NeTra: A toolbox for navigating large image databases. Multimedia Systems 7(3), 184–198 (1999)CrossRefGoogle Scholar
  13. 13.
    Manjunath, B.S., Salembier, P., Sikora, T.: Introduction to MPEG-7, Multimedia Content Description Interface. John Wiley & Sons, Chichester (2002)Google Scholar
  14. 14.
    Martin, D., et al.: A database of human segmented natural images and its applications to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. IEEE 8th Int. Conf. Computer Vision, Vancouver, Canada, pp. 416–423. IEEE, Los Alamitos (2001)Google Scholar
  15. 15.
    Mezaris, V., Kompatsiaris, I., Strintzis, M.G.: Region-based Image Retrieval using an Object Ontology and Relevance Feedback. EURASIP Journal on Applied Signal Processing 2004(6), 886–901 (2004)CrossRefGoogle Scholar
  16. 16.
    Panofsky, E.: Iconography and Iconology: An Introduction to the Study of Renaissance Art. In: Meaning in the visual arts, pp. 26–54. Penguin Books (1970)Google Scholar
  17. 17.
    Rui, Y., Huang, T.S.: A Novel Relevance Feedback Technique in Image Retrieval. In: Proc.of ACM Multimedia, pp. 67–70. ACM Press, New York (1999)Google Scholar
  18. 18.
    Rui, Y., et al.: Relevance Feedback: A Power Tool for Interactive Content-Based Image Retrieval. IEEE Transactions on Circuits and Systems for Video Technology 8(5), 644–655 (1998)CrossRefGoogle Scholar
  19. 19.
    Torres, J., Parkes, A., Corte-Real, L.: Region-Based Relevance Feedback in Concept-Based Image Retrieval. In: Proc. of the 5th International Workshop on Image Analysis for Multimedia Interactive Services, Lisboa, Portugal (2004)Google Scholar
  20. 20.
    Wang, T., Rui, Y., Sun, J.-G.: Constraint Based Region Matching for Image Retrieval. International Journal of Computer Vision 56(1/2), 37–45 (2004)CrossRefGoogle Scholar
  21. 21.
    Wu, L., et al.: FALCON Feedback Adaptive Loop for Content-Based Retrieval. In: VLDB 2000 (2000)Google Scholar
  22. 22.
    Zhang, R., Zhang, Z.: Hidden Semantic Concept Discovery in Region Based Image Retrieval. In: Proc. of the 2004 IEEE Conf. on Computer Vision and Pattern Recognition, vol. 2, pp. II-996–II-1001. IEEE Computer Society Press, Los Alamitos (2004)CrossRefGoogle Scholar
  23. 23.
    Zhuang, Y., Yang, J., Li, Q.: A Graphic-Theoretic Model for Incremental Relevance Feedback in Image Retrieval. In: Proc. IEEE Int. Conf. on Image Processing 2002, New York, USA, IEEE Computer Society Press, Los Alamitos (2002)Google Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • José Manuel Torres
    • 1
  • David Hutchison
    • 2
  • Luís Paulo Reis
    • 3
  1. 1.University Fernando Pessoa / INESC – PortoPortugal
  2. 2.Lancaster UniversityUK
  3. 3.FEUP/LIACC – Faculty of Engineering of the University of PortoPortugal

Personalised recommendations