Query by Semantic Example

  • Nikhil Rasiwasia
  • Nuno Vasconcelos
  • Pedro J. Moreno
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4071)


A solution to the problem of image retrieval based on query-by-semantic-example (QBSE) is presented. QBSE extends the idea of query-by-example to the domain of semantic image representations. A semantic vocabulary is first defined, and a semantic retrieval system is trained to label each image with the posterior probability of appearance of each concept in the vocabulary. The resulting vector is interpreted as the projection of the image onto a semantic probability simplex, where a suitable similarity function is defined. Queries are specified by example images, which are projected onto the probability simplex. The database images whose projections on the simplex are closer to that of the query are declared its closest neighbors. Experimental evaluation indicates that 1) QBSE significantly outperforms the traditional query-by-visual-example paradigm when the concepts in the query image are known to the retrieval system, and 2) has equivalent performance even in the worst case scenario of queries composed by unknown concepts.


Discrete Cosine Transform Retrieval System Image Retrieval Query Image Semantic Concept 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barnard, K., Duygulu, P., de Freitas, N., Forsyth, D., Blei, D., Jordan, M.I.: Matching words and pictures. JMLR 3, 1107–1135 (2003)MATHCrossRefGoogle Scholar
  2. 2.
    Blei, D., Jordan, M.I.: Modeling annotated data. In: Proceedings of the 26th Intl. ACM SIGIR Conf., pp. 127–134 (2003)Google Scholar
  3. 3.
    Lavrenko, V., Manmatha, R., Jeon, J.: A model for learning the semantics of pictures. In: NIPS (2003)Google Scholar
  4. 4.
    Picard, R.: Digital Libraries: Meeting Place for High-Level and Low-Level Vision. In: Li, S., Teoh, E.-K., Mital, D., Wang, H. (eds.) ACCV 1995. LNCS, vol. 1035. Springer, Heidelberg (1996)Google Scholar
  5. 5.
    Feng, S.L., Manmatha, R., Lavrenko, V.: Multiple bernoulli relevance models for image and video annotation. In: CVPR (2004)Google Scholar
  6. 6.
    Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval: the end of the early years. PAMI 22(12), 1349–1380 (2000)Google Scholar
  7. 7.
    Vasconcelos, N.: Image Indexing with Mixture Hierarchies. In: CVPR, Kawai, Hawaii (2001)Google Scholar
  8. 8.
    Sclaroff, S., Cascia, M.L., Sethi, S., Taycher, L.: Unifying textual and visual cues for content-based image retrieval on the world wide web. Computer Vision and Image Understanding 75(1-2), 86–98 (1999)CrossRefGoogle Scholar
  9. 9.
    Carneiro, G., Vasconcelos, N.: Formulating Semantics Image Annotation as a Supervised Learning Problem. In: CVPR, San Diego (2005)Google Scholar
  10. 10.
    Vasconcelos, N.: Minimum Probability of Error Image Retrieval. IEEE Transactions on Signal Processing 52(8) (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Nikhil Rasiwasia
    • 1
  • Nuno Vasconcelos
    • 1
  • Pedro J. Moreno
    • 2
  1. 1.Statistical Visual Computing LabUniversity of CaliforniaSan Diego
  2. 2.Google, Inc.New YorkUSA

Personalised recommendations