Bayesian Learning of Hierarchical Multinomial Mixture Models of Concepts for Automatic Image Annotation

  • Rui Shi
  • Tat-Seng Chua
  • Chin-Hui Lee
  • Sheng Gao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4071)


We propose a novel Bayesian learning framework of hierarchical mixture model by incorporating prior hierarchical knowledge into concept representations of multi-level concept structures in images. Characterizing image concepts by mixture models is one of the most effective techniques in automatic image annotation (AIA) for concept-based image retrieval. However it also poses problems when large-scale models are needed to cover the wide variations in image samples. To alleviate the potential difficulties arising in estimating too many parameters with insufficient training images, we treat the mixture model parameters as random variables characterized by a joint conjugate prior density of the mixture model parameters. This facilitates a statistical combination of the likelihood function of the available training data and the prior density of the concept parameters into a well-defined posterior density whose parameters can now be estimated via a maximum a posteriori criterion. Experimental results on the Corel image dataset with a set of 371 concepts indicate that the proposed Bayesian approach achieved a maximum F1 measure of 0.169, which outperforms many state-of-the-art AIA algorithms.


Mixture Model Prior Density Bayesian Learn Concept Hierarchy Primitive Concept 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barnard, K., Duygulu, P., Forsyth, D.: Clustering Art. In: Proceedings of CVPR (2001)Google Scholar
  2. 2.
    Carneiro, G., Vasconcelos, N.: Formulating Semantic Image Annotation as a Supervised Learning Problem. In: Proceedings of CVPR (2005)Google Scholar
  3. 3.
    Duyulu, P., Barnard, K., de Freitas, N., Forsyth, D.: Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  4. 4.
    Fan, J.P., Luo, H.Z., Gao, Y.L.: Learning the Semantics of Images by Using Unlabeled Samples. In: Proceedings of CVPR (2005)Google Scholar
  5. 5.
    Gao, S., Wang, D.-H., Lee, C.-H.: Automatic Image Annotation through Multi-Topic Text Categorization. In: Proceedings of ICASSP, Toulouse, France (May 2006)Google Scholar
  6. 6.
    Huo, Q., Chan, C., Lee, C.-H.: Bayesian Adaptive Learning of the Parameters of Hidden Markov Model for Speech Recognition. IEEE Trans. Speech Audio Processing 3, 334–345 (1995)CrossRefGoogle Scholar
  7. 7.
    Jeon, J., Lavrenko, V., Manmatha, R.: Automatic Image Annotation and Retrieval Using Cross-Media Relevance Models. In: Proceedings of the 26th ACM SIGIR (2003)Google Scholar
  8. 8.
    Lavrenko, V., Manmatha, R., Jeon, J.: A Model for Learning the Semantics of Pictures. In: Proceedings of the 16th Conference on NIPS (2003)Google Scholar
  9. 9.
    Lee, C.-H., Huo, Q.: On Adaptive Decision Rules and Decision Parameter Adaptation for Automatic Speech Recognition. Proceedings of the IEEE 88(8) (August 2000)Google Scholar
  10. 10.
    Minka, T.: Estimating a Dirichlet Distribution (2003),
  11. 11.
    Miller, G.A., Beckwith, R., Fellbaum, C., Gross, D., Miller, K.J.: Introduction to WordNet: an on-line lexical database. Intl. Jour. of Lexicography 3, 235–244 (1990)CrossRefGoogle Scholar
  12. 12.
    Mori, Y., Takahashi, H., Oka, R.: Image-to-Word Transformation Based on Dividing and Vector Quantizing Images with Words. In: Proceedings of MISRM (1999)Google Scholar
  13. 13.
    Novovicova, J., Malik, A.: Application of Multinomial Mixture Model to Text Classification. In: Perales, F.J., Campilho, A.C., Pérez, N., Sanfeliu, A. (eds.) IbPRIA 2003. LNCS, vol. 2652, pp. 646–653. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  14. 14.
    Srikanth, M., Varner, J., Bowden, M., Moldovan, D.: Exploiting Ontologies for Automatic Image Annotation. In: Proceedings of the 28th ACM SIGIR (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Rui Shi
    • 1
  • Tat-Seng Chua
    • 1
  • Chin-Hui Lee
    • 2
  • Sheng Gao
    • 3
  1. 1.School of ComputingNational University of SingaporeSingapore
  2. 2.School of ECEGeorgia Institute of TechnologyAtlantaUSA
  3. 3.Institute for Infocomm ResearchSingapore

Personalised recommendations