Automatic Image Annotation with Relevance Feedback and Latent Semantic Analysis

  • Donn Morrison
  • Stéphane Marchand-Maillet
  • Eric Bruno
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4918)


The goal of this paper is to study the image-concept relationship as it pertains to image annotation. We demonstrate how automatic annotation of images can be implemented on partially annotated databases by learning image-concept relationships from positive examples via inter-query learning. Latent semantic analysis (LSA), a method originally designed for text retrieval, is applied to an image/session matrix where relevance feedback examples are collected from a large number of artificial queries (sessions). Singular value decomposition (SVD) is exploited during LSA to propagate image annotations using only relevance feedback information. We will show how SVD can be used to filter a noisy image/session matrix and reconstruct missing values.


Singular Value Decomposition Image Retrieval Latent Dirichlet Allocation Relevance Feedback Latent Semantic Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: CHI 2004: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 319–326. ACM Press, New York, NY, USA (2004)CrossRefGoogle Scholar
  2. 2.
    Grira, N., Crucianu, M., Boujemaa, N.: Active semi-supervised fuzzy clustering for image database categorization. In: MIR 2005: Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, pp. 9–16. ACM Press, New York, NY, USA (2005)CrossRefGoogle Scholar
  3. 3.
    Mueller, H., Mueller, W., Squire, D.M., Marchand-Maillet, S., Pun, T.: Long-term learning from user behavior in content-based image retrieval. Technical report, Université de Genève (2000)Google Scholar
  4. 4.
    Wenyin, L., Dumais, S., Sun, Y., Zhang, H., Czerwinski, M., Field, B.: Semi-automatic image annotation (2001)Google Scholar
  5. 5.
    Fournier, J., Cord, M.: Long-term similarity learning in content-based image retrieval (2002)Google Scholar
  6. 6.
    Li, M., Chen, Z., Zhang, H.: Statistical correlation analysis in image retrieval (2002)Google Scholar
  7. 7.
    Heisterkamp, D.: Building a latent-semantic index of an image database from patterns of relevance feedback (2002)Google Scholar
  8. 8.
    Koskela, M., Laaksonen, J.: Using long-term learning to improve efficiency of content-based image retrieval (2003)Google Scholar
  9. 9.
    Cord, M., Gosselin, P.H.: Image retrieval using long-term semantic learning. In: IEEE International Conference on Image Processing (2006)Google Scholar
  10. 10.
    Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society of Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
  11. 11.
    Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. IEEE Trans. on PAMI 25 (2000)Google Scholar
  12. 12.
    Monay, F., Gatica-Perez, D.: On image auto-annotation with latent space models. In: Proc. ACM Int. Conf. on Multimedia (ACM MM), Berkeley (2003)Google Scholar
  13. 13.
    Monay, F., Gatica-Perez, D.: Plsa-based image auto-annotation: Constraining the latent space. In: MULTIMEDIA 2004: Proceedings of the 12th annual ACM international conference on Multimedia, pp. 348–351. ACM Press, New York, NY, USA (2004)CrossRefGoogle Scholar
  14. 14.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. Journal of Machine Learning Research 3, 993–1022 (2003)zbMATHGoogle Scholar
  15. 15.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D., Jordan, M.: Matching words and pictures. Machine Learning Research 3, 1107–1135 (2003)zbMATHGoogle Scholar
  16. 16.
    Blei, D.M., Jordan, M.I.: Modeling annotated data. In: SIGIR 2003: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pp. 127–134. ACM Press, New York, NY, USA (2003)CrossRefGoogle Scholar
  17. 17.
    Wang, J.Z., Li, J.: Learning-based linguistic indexing of pictures with 2-d mhmms. In: MULTIMEDIA 2002: Proceedings of the tenth ACM international conference on Multimedia, pp. 436–445. ACM Press, New York, NY, USA (2002)CrossRefGoogle Scholar
  18. 18.
    Goh, K.S., Chang, E.Y., Li, B.: Using one-class and two-class svms for multiclass image annotation. IEEE Transactions on Knowledge and Data Engineering 17(10), 1333–1346 (2005)CrossRefGoogle Scholar
  19. 19.
    Tang, J., Hare, J.S., Lewis, P.H.: Image auto-annotation using a statistical model with salient regions. In: Proceedings of IEEE International Conference on Multimedia & Expo (ICME), Hilton Toronto, Toronto, Ontario, Canada (2006)Google Scholar
  20. 20.
    Gondra, I., Heisterkamp, D.R.: Incremental semantic clustering summarizing inter-query learning in content-based image retrieval via. In: Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC 2004) (2004)Google Scholar
  21. 21.
    Kosinov, S., Marchand-Maillet, S.: Multimedia autoannotation via hierarchical semantic ensembles. In: Proceedings of the Int. Workshop on Learning for Adaptable Visual Systems (LAVS 2004), Cambridge, UK (2004)Google Scholar
  22. 22.
    Kosinov, S., Marchand-Maillet, S.: Hierarchical ensemble learning for multimedia categorization and autoannotation. In: Proceedings of the 2004 IEEE Signal Processing Society Workshop (MLSP 2004), São Luís, Brazil, pp. 645–654 (2004)Google Scholar
  23. 23.
    Jin, Y., Khan, L., Wang, L., Awad, M.: Image annotations by combining multiple evidence & wordnet. In: MULTIMEDIA 2005: Proceedings of the 13th annual ACM international conference on Multimedia, pp. 706–715. ACM Press, New York, NY, USA (2005)CrossRefGoogle Scholar
  24. 24.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Donn Morrison
    • 1
  • Stéphane Marchand-Maillet
    • 1
  • Eric Bruno
    • 1
  1. 1.Centre Universitaire InformatiqueUniversité de GenèveGenèveSwitzerland

Personalised recommendations