Automatic Image Annotation Using Maximum Entropy Model

  • Wei Li
  • Maosong Sun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3651)


Automatic image annotation is a newly developed and promising technique to provide semantic image retrieval via text descriptions. It concerns a process of automatically labeling the image contents with a pre-defined set of keywords which are exploited to represent the image semantics. A Maximum Entropy Model-based approach to the task of automatic image annotation is proposed in this paper. In the phase of training, a basic visual vocabulary consisting of blob-tokens to describe the image content is generated at first; then the statistical relationship is modeled between the blob-tokens and keywords by a Maximum Entropy Model constructed from the training set of labeled images. In the phase of annotation, for an unlabeled image, the most likely associated keywords are predicted in terms of the blob-token set extracted from the given image. We carried out experiments on a medium-sized image collection with about 5000 images from Corel Photo CDs. The experimental results demonstrated that the annotation performance of this method outperforms some traditional annotation methods by about 8% in mean precision, showing a potential of the Maximum Entropy Model in the task of automatic image annotation.


Maximum Entropy Image Retrieval Training Image Machine Translation Image Region 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barnard, K., Dyugulu, P., de Freitas, N., Forsyth, D., Blei, D., Jordan, M.I.: Matching words and pictures. Journal of Machine Learning Research 3, 1107–1135 (2003)zbMATHCrossRefGoogle Scholar
  2. 2.
    Duygulu, P., Barnard, K., de Freitas, N., Forsyth, D.: Ojbect recognition as machine translation: Learning a lexicon fro a fixed image vocabulary. In: Seventh European Conf. on Computer Vision, pp. 97–112 (2002)Google Scholar
  3. 3.
    Jeon, J., Lavrenko, V., Manmatha, R.: Automatic image annotation and retrieval using cross-media relevance models. In: Proceedings of the 26th intl. SIGIR Conf, pp. 119–126 (2003)Google Scholar
  4. 4.
    Mori, Y., Takahashi, H., Oka, R.: Image-to-word transformation based on dividing and vector quantizing images with words. In: First International Workshop on Multimedia Intelligent Storage and Retrieval Management (1999)Google Scholar
  5. 5.
    Chang, E., Goh, K., Sychay, G., Wu, G.: CBSA: Content-based soft annotation for multimodal image retrieval using bayes point machines. IEEE Transactions on Circuts and Systems for Video Technology Special Issue on Conceptual and Dynamical Aspects of Multimedia Content Descriptions 13(1), 26–38 (2003)Google Scholar
  6. 6.
    shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions On Pattern Analysis and Machine Intelligence 22(8), 888–905 (2000)CrossRefGoogle Scholar
  7. 7.
    Berger, A., Pietra, S., Pietra, V.: A maximum entropy approach to natural language processing. In: Computational Linguistics, pp. 39–71 (1996)Google Scholar
  8. 8.
    Li, J., Wang, J.A.: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on PAMI 25(10), 1075–1088 (2003)Google Scholar
  9. 9.
    Jeon, J., Manmatha, R.: Using maximum entropy for automatic image annotation. In: Proceedings of third international conference on image and video retrieval, pp. 24–31 (2004)Google Scholar
  10. 10.
    Malouf, R.: A comparison of algorithms for maximum entropy parameter estimation. In: Proceedings of the 6th Workshop on Computational Language Learning (2003)Google Scholar
  11. 11.
    Lavrenko, V., Manmatha, R., Jeon, J.: A model for learning the semantics of pictures. In: Proceedings of the 16th Annual Conference on Neural Information Processing Systems (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Wei Li
    • 1
  • Maosong Sun
    • 1
  1. 1.State Key Lab of Intelligent Technology and Systems, Department of Computer Science and TechnologyTsinghua UniversityBeijingChina

Personalised recommendations