Classification and Automatic Annotation Extension of Images Using Bayesian Network

  • Sabine Barrat
  • Salvatore Tabbone
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5342)


In many vision problems, instead of having fully annotated training data, it is easier to obtain just a subset of data with annotations, because it is less restrictive for the user. For this reason, in this paper, we consider especially the problem of classifying weakly-annotated images, where just a small subset of the database is annotated with keywords. In this paper we present and evaluate a new method which improves the effectiveness of content-based image classification, by integrating semantic concepts extracted from text, and by automatically extending annotations to the images with missing keywords. Our model is inspired from the probabilistic graphical model theory: we propose a hierarchical mixture model which enables to handle missing values. Results of visual-textual classification, reported on a database of images collected from the Web, partially and manually annotated, show an improvement by 32.3% in terms of recognition rate against only visual information classification. Besides the automatic annotation extension with our model for images with missing keywords outperforms the visual-textual classification by 6.8%. Finally the proposed method is experimentally competitive with the state-of-art classifiers.


probabilistic graphical models Bayesian networks image classification image annotation 


  1. 1.
    Barnard, K., Duygulu, P., Forsyth, D., De Freitas, N., Blei, D.M., Jordan, M.I.: Matching words and pictures. Journal of Machine Learning Research 3(6), 1107–1135 (2003)zbMATHGoogle Scholar
  2. 2.
    Benitez, A., Chang, S.F.: Perceptual knowledge construction from annotated image collections. ICME 2002 1, 189–192 (2002)Google Scholar
  3. 3.
    Grosky, W.I., Zhao, R.: Negotiating the semantic gap: From feature maps to semantic landscapes. In: Pacholski, L., Ružička, P. (eds.) SOFSEM 2001. LNCS, vol. 2234, pp. 33–52. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  4. 4.
    Kherfi, M.L., Brahmi, D., Ziou, D.: Combining visual features with semantics for a more effective image retrieval. In: ICPR 2004, vol. 2, pp. 961–964 (2004)Google Scholar
  5. 5.
    Blei, D.M., Jordan, M.I.: Modeling annotated data. In: SIGIR 2003, pp. 127–134 (2003)Google Scholar
  6. 6.
    Gao, Y., Fan, J., Xue, X., Jain, R.: Automatic image annotation by incorporating feature hierarchy and boosting to scale up svm classifiers. In: ACM MULTIMEDIA 2006, pp. 901–910 (2006)Google Scholar
  7. 7.
    Yang, C., Dong, M., Hua, J.: Region-based image annotation using asymmetrical support vector machine-based multiple-instance learning. In: CVPR 2006, pp. 2057–2063 (2006)Google Scholar
  8. 8.
    Wang, C., Jing, F., Zhang, L., Zhang, H.J.: Image annotation refinement using random walk with restarts. In: ACM MULTIMEDIA 2006, pp. 647–650 (2006)Google Scholar
  9. 9.
    Rui, X., Li, M., Li, Z., Ma, W.Y., Yu, N.: Bipartite graph reinforcement model for web image annotation. In: ACM MULTIMEDIA 2007, pp. 585–594 (2007)Google Scholar
  10. 10.
    Feng, S., Manmatha, R., Lavrenko, V.: Multiple bernoulli relevance models for image and video annotation. CVPR 2004 2, 1002–1009 (2004)Google Scholar
  11. 11.
    Swain, M.J., Ballard, D.H.: Color indexing. International Journal of Computer Vision 7(1), 11–32 (1991)CrossRefGoogle Scholar
  12. 12.
    Tabbone, S., Wendling, L.: Technical Symbols Recognition Using the Two-dimensional Radon Transform. In: ICPR 2002, vol. 2, pp. 200–203 (2002)Google Scholar
  13. 13.
    Robert, C.: A decision-Theoretic Motivation. Springer, Heidelberg (1997)Google Scholar
  14. 14.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Kim, J.H., Pearl, J.: A computational model for combined causal and diagnostic reasoning in inference systems. In: IJCAI 1983, pp. 190–193 (1983)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Sabine Barrat
    • 1
  • Salvatore Tabbone
    • 1
  1. 1.LORIA-UMR 7503University of Nancy 2, BP 239Vandœuvre-lés-NancyFrance

Personalised recommendations