Building Semantic Hierarchies Faithful to Image Semantics

  • Hichem Bannour
  • Céline Hudelot
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7131)


This paper proposes a new image-semantic measure, named ”Semantico-Visual Relatedness of Concepts” (SVRC), to estimate the semantic similarity between concepts. The proposed measure incorporates visual, conceptual and contextual information to provide a measure which is more meaningful and more representative of image semantics. We also propose a new methodology to automatically build a semantic hierarchy suitable for the purpose of image annotation and/or classification. The building is based on the previously proposed measure SVRC and on a new heuristic, named TRUST-ME, to connect concepts with higher relatedness till the building of the final hierarchy. The built hierarchy explicitly encodes a general to specific concepts relationship and therefore provides a semantic structure to concepts which facilitates the semantic interpretation of images. Our experiments showed that the use of the constructed semantic hierarchies as a hierarchical classification framework provides a better image annotation.


Visual Word Semantic Similarity Semantic Relatedness Visual Similarity Contextual Similarity 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Banerjee, S., Pedersen, T.: Extended gloss overlaps as a measure of semantic relatedness. In: IJCAI (2003)Google Scholar
  2. 2.
    Bannour, H., Hudelot, C.: Towards ontologies for image interpretation and annotation. In: CBMI (2011)Google Scholar
  3. 3.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D.M., Jordan, M.I.: Matching words and pictures. JMLR 3, 1107–1135 (2003)zbMATHGoogle Scholar
  4. 4.
    Bart, E., Porteous, I., Perona, P., Welling, M.: Unsupervised learning of visual taxonomies. In: CVPR (2008)Google Scholar
  5. 5.
    Budanitsky, A., Hirst, G.: Evaluating wordnet-based measures of lexical semantic relatedness. Comput. Linguist. 32, 13–47 (2006)CrossRefzbMATHGoogle Scholar
  6. 6.
    Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning, 20 (1995)Google Scholar
  7. 7.
    Deng, J., Berg, A.C., Li, K., Fei-Fei, L.: What Does Classifying More Than 10,000 Image Categories Tell Us? In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 71–84. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  9. 9.
    Fan, J., Gao, Y., Luo, H.: Hierarchical classification for automatic image annotation. In: SIGIR (2007)Google Scholar
  10. 10.
    Fan, J., Luo, H., Shen, Y., Yang, C.: Integrating visual and semantic contexts for topic network generation and word sense disambiguation. In: CIVR (2009)Google Scholar
  11. 11.
    Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press (1998)Google Scholar
  12. 12.
    Griffin, G., Perona, P.: Learning and using taxonomies for fast visual categorization. In: CVPR (2008)Google Scholar
  13. 13.
    Hauptmann, A., Yan, R., Lin, W.-H.: How many high-level concepts will fill the semantic gap in news video retrieval? In: CIVR (2007)Google Scholar
  14. 14.
    Lavrenko, V., Manmatha, R., Jeon, J.: A model for learning the semantics of pictures. In: NIPS. MIT Press (2003)Google Scholar
  15. 15.
    Li, L.-J., Wang, C., Lim, Y., Blei, D., Fei-Fei, L.: Building and using a semantivisual image hierarchy. In: CVPR (2010)Google Scholar
  16. 16.
    Liu, Y., Zhang, D., Lu, G., Ma, W.-Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recognition 40(1), 262–282 (2007)CrossRefzbMATHGoogle Scholar
  17. 17.
    Lowe, D.G.: Object recognition from local scale-invariant features. In: ICCV (1999)Google Scholar
  18. 18.
    Marszalek, M., Schmid, C.: Semantic hierarchies for visual object recognition. In: CVPR (2007)Google Scholar
  19. 19.
    Naphade, M., Smith, J.R., Tesic, J., Hsu, W., Kennedy, L., Hauptmann, A., Curtis, J.: Large-scale concept ontology for multimedia. IEEE MultiMedia (2006)Google Scholar
  20. 20.
    Patwardhan, S., Pedersen, T.: Using wordnet-based context vectors to estimate the semantic relatedness of concepts. In: EACL (2006)Google Scholar
  21. 21.
    Resnik, P.: Using information content to evaluate semantic similarity in a taxonomy. In: IJCAI (1995)Google Scholar
  22. 22.
    Sivic, J., Russell, B.C., Zisserman, A., Freeman, W.T., Efros, A.A.: Unsupervised discovery of visual object class hierarchies. In: CVPR (2008)Google Scholar
  23. 23.
    Smeulders, A.W.M., Member, S., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE PAMI, 22 (2000)Google Scholar
  24. 24.
    Wei, X.-Y., Ngo, C.-W.: Ontology-enriched semantic space for video search. In: MULTIMEDIA, pp. 981–990 (2007)Google Scholar
  25. 25.
    Wu, L., Hua, X.-S., Yu, N., Ma, W.-Y., Li, S.: Flickr distance. In: MM (2008)Google Scholar
  26. 26.
    Yao, B., Yang, X., Lin, L., Lee, M.W., Zhu, S.C.: I2t: Image parsing to text description. Proceedings of IEEE (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Hichem Bannour
    • 1
  • Céline Hudelot
    • 1
  1. 1.Applied Mathematics and Systems DepartmentEcole Centrale ParisChâtenay-MalabryFrance

Personalised recommendations