A Linear-Algebraic Technique with an Application in Semantic Image Retrieval

  • Jonathon S. Hare
  • Paul H. Lewis
  • Peter G. B. Enser
  • Christine J. Sandom
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4071)


This paper presents a novel technique for learning the underlying structure that links visual observations with semantics. The technique, inspired by a text-retrieval technique known as cross-language latent semantic indexing uses linear algebra to learn the semantic structure linking image features and keywords from a training set of annotated images. This structure can then be applied to unannotated images, thus providing the ability to search the unannotated images based on keyword. This factorisation approach is shown to perform well, even when using only simple global image features.


Training Image Average Precision Query Term Mean Average Precision Factorisation Approach 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hare, J.S., Lewis, P.H., Enser, P.G.B., Sandom, C.J.: Mind the gap. In: Chang, E.Y., Hanjalic, A., Sebe, N. (eds.) Multimedia Content Analysis, Management, and Retrieval 2006, San Jose, California, USA, vol. 6073, pp. 607309–1–607309–12. SPIE (2006) Google Scholar
  2. 2.
    Enser, P.G.B., Sandom, C.J., Lewis, P.H.: Surveying the reality of semantic image retrieval. In: Bres, S., Laurini, R. (eds.) VISUAL 2005. LNCS, vol. 3736, pp. 177–188. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society of Information Science 41, 391–407 (1990)CrossRefGoogle Scholar
  4. 4.
    Landauer, T.K., Littman, M.L.: Fully automatic cross-language document retrieval using latent semantic indexing. In: Proceedings of the Sixth Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research, Waterloo, Ontario, Canada, pp. 31–38 (1990)Google Scholar
  5. 5.
    Monay, F., Gatica-Perez, D.: On image auto-annotation with latent space models. In: MULTIMEDIA 2003: Proceedings of the eleventh ACM international conference on Multimedia, pp. 275–278. ACM Press, New York (2003)CrossRefGoogle Scholar
  6. 6.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. IJCV 9, 137–154 (1992)CrossRefGoogle Scholar
  7. 7.
    University of Washington: Ground truth image database (2004), http://www.cs.washington.edu/research/imagedatabase/groundtruth/
  8. 8.
    Duygulu, P., Barnard, K., de Freitas, J.F.G., Forsyth, D.A.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Hare, J.S., Lewis, P.H.: Saliency-based models of image content and their application to auto-annotation by semantic propagation. In: Proceedings of the Second European Semantic Web Conference (ESWC 2005), Heraklion, Crete (2005)Google Scholar
  10. 10.
    Hare, J.S., Lewis, P.H.: On image retrieval using salient regions with vector-spaces and latent semantics. In: Leow, W.-K., Lew, M., Chua, T.-S., Ma, W.-Y., Chaisorn, L., Bakker, E.M. (eds.) CIVR 2005. LNCS, vol. 3568, pp. 540–549. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  11. 11.
    Yavlinsky, A., Schofield, E., Rüger, S.: Automated Image Annotation Using Global Features and Robust Nonparametric Density Estimation. In: Leow, W.-K., Lew, M., Chua, T.-S., Ma, W.-Y., Chaisorn, L., Bakker, E.M. (eds.) CIVR 2005. LNCS, vol. 3568, pp. 507–517. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jonathon S. Hare
    • 1
  • Paul H. Lewis
    • 1
  • Peter G. B. Enser
    • 2
  • Christine J. Sandom
    • 2
  1. 1.School of Electronics and Computer ScienceUniversity of SouthamptonUK
  2. 2.School of Computing, Mathematical and Information SciencesUniversity of BrightonUK

Personalised recommendations