Context-Based Support Vector Machines for Interconnected Image Annotation

  • Hichem Sahbi
  • Xi Li
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6492)


We introduce in this paper a novel image annotation approach based on support vector machines (SVMs) and a new class of kernels referred to as context-dependent. The method goes beyond the naive use of the intrinsic low level features (such as color, texture, shape, etc.) and context-free kernels, in order to design a kernel function applicable to interconnected databases such as social networks. The main contribution of our method includes (i) a variational approach which helps designing this function using both intrinsic features and the underlying contextual information resulting from different links and (ii) the proof of convergence of our kernel to a positive definite fixed-point, usable for SVM training and other kernel methods. When plugged in SVMs, our context-dependent kernel consistently improves the performance of image annotation, compared to context-free kernels, on hundreds of thousands of Flickr images.


Support Vector Machine Visual Feature Reproduce Kernel Hilbert Space Image Annotation Equal Error Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Carneiro, G., Vasconcelos, N.: Formulating semantic image annotation as a supervised learning problem. In: Proc. of CVPR (2005)Google Scholar
  2. 2.
    Li, J., Wang, J.Z.: Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Trans. on PAMI 25(9), 1075–1088 (2003)CrossRefGoogle Scholar
  3. 3.
    Barnard, K., Duygululu, P., Forsyth, D., Blei, D., Jordan, M.: Matching words and pictures. The Journal of Machine Learning Research (2003)Google Scholar
  4. 4.
    Monay, F., GaticaPerez, D.: PLSA-based Image AutoAnnotation: Constraining the Latent Space. In: Proc. of ACM International Conference on Multimedia (2004)Google Scholar
  5. 5.
    Gao, Y., Fan, J., Xue, X., Jain, R.: Automatic Image Annotation by Incorporating Feature Hierarchy and Boosting to Scale up SVM Classifiers. In: Proc. of ACM MULTIMEDIA (2006)Google Scholar
  6. 6.
    Jeon, J., Lavrenko, V., Manmatha, R.: Automatic image annotation and retrieval using cross-media relevance models. In: Proc. of ACM SIGIR, pp. 119–126 (2003)Google Scholar
  7. 7.
    Lavrenko, V., Manmatha, R., Jeon, J.: A model for learning the semantics of pictures. In: Proc. of NIPS (2004)Google Scholar
  8. 8.
    Feng, S., Manmatha, R., Lavrenko, V.: Multiple Bernoulli relevance models for image and video annotation. In: Proc. of ICCV, pp. 1002–1009 (2004)Google Scholar
  9. 9.
    Liu, J., Wang, B., Li, M., Li, Z., Ma, W., Lu, H., Ma, S.: Dual cross-media relevance model for image annotation. In: Proc. of ACM MULTIMEDIA, pp. 605–614 (2007)Google Scholar
  10. 10.
    Wan, X., Yang, J., Xiao, J.: Manifold-ranking based topic-focused multi-document summarization. In: Proc. of IJCAI, pp. 2903–2908 (2007)Google Scholar
  11. 11.
    Zhou, D., Weston, J., Gretton, A., Bousquet, O., Schölkopf, B.: Ranking on data manifolds. In: Proc. of NIPS (2004)Google Scholar
  12. 12.
    Liu, J., Li, M., Liu, Q., Lu, H., Ma, S.: Image annotation via graph learning. Pattern Recognition 42(2), 218–228 (2009)CrossRefzbMATHGoogle Scholar
  13. 13.
    Liu, J., Li, M., Ma, W., Liu, Q., Lu, H.: An adaptive graph model for automatic image annotation. In: Proc. of ACM International Workshop on Multimedia Information Retrieval, pp. 61–70 (2006)Google Scholar
  14. 14.
    Srikanth, M., Varner, J., Bowden, M., Moldovan, D.: Exploiting ontologies for automatic image annotation. In: Proc. of SIGIR, pp. 552–558 (2005)Google Scholar
  15. 15.
    Wu, Y., Chang, E.Y., Tseng, B.L.: Multimodal metadata fusion using causal strength. In: Proc. of ACM MULTIMEDIA, pp. 872–881 (2005)Google Scholar
  16. 16.
    Miller, G.A.: Wordnet: a lexical database for English. ACM Commun. 38(11), 39–41 (1995)CrossRefGoogle Scholar
  17. 17.
    Wang, C., Jing, F., Zhang, L., Zhang, H.J.: Image annotation refinement using random walk with restarts. In: Proc. of ACM MULTIMEDIA, pp. 647–650 (2006)Google Scholar
  18. 18.
    Jin, Y., Khan, L., Wang, L., Awad, M.: Image annotations by combining multiple evidence & wordNet. In: Proc. of ACM MULTIMEDIA, pp. 706–715 (2005)Google Scholar
  19. 19.
    Cilibrasi, R., Vitanyi, P.M.B.: The google similarity distance. IEEE Transactions on Knowledge and Data Engineering (2007)Google Scholar
  20. 20.
    Wu, L., Hua, X., Yu, N., Ma, W., Li, S.: Flickr distance. In: Proc. of ACM MULTIMEDIA (2008)Google Scholar
  21. 21.
    Wang, Y., Gong, S.: Translating Topics to Words for Image Annotation. In: Proc. of ACM CIKM (2007)Google Scholar
  22. 22.
    Lu, Z., Ip, H.H.S., He, Q.: Context-Based Multi-Label Image Annotation. In: Proc. of ACM CIVR (2009)Google Scholar
  23. 23.
    Boser, B., Guyon, I., Vapnik, V.: An training algorithm for optimal margin classifiers. In: Fifth Annual ACM Workshop on Computational Learning Theory, Pittsburgh (1992)Google Scholar
  24. 24.
    Vapnik, V.: Statistical Learning Theory. A Wiley-Interscience Publication, Hoboken (1998)zbMATHGoogle Scholar
  25. 25.
    Wang, C., Yan, S., Zhang, L., Zhang, H.: Multi-Label Sparse Coding for Automatic Image Annotation. In: Proc. of CVPR (2009)Google Scholar
  26. 26.
    Duygulu, P., Barnard, K., de Freitas, J.F.G., Forsyth, D.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  27. 27.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. In: Proc. of CVPR (2006)Google Scholar
  28. 28.
    Gallagher, A.C., Neustaedter, C.G., Cao, L., Luo, J., Chen, T.: Image Annotation Using Personal Calendars as Context. In: Proc. of ACM Multimedia (2008)Google Scholar
  29. 29.
    Cao, L., Luo, J., Huang, T.S.: Annotating Photo Collection by Label Propagation According to Multiple Similarity Cues. In: Proc. of ACM Multimedia (2008)Google Scholar
  30. 30.
    Sahbi, H., Audibert, J.-Y.: Social network kernels for image ranking and retrieval. In Technical Report, N 2009D009, TELECOM ParisTech (March 2009)Google Scholar
  31. 31.
    Shawe-Taylor, J., Cristianini, N.: Support Vector Machines and other kernel-based learning methods. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
  32. 32.
    Yang, Y.H., Wu, P.T., Lee, C.W., Lin, K.H., Hsu, W.H., Chen, H.: ContextSeer: Context Search and Recommendation at Query Time for Shared Consumer Photos. In: Proc. of ACM Multimedia (2008)Google Scholar
  33. 33.
    Haussler, D.: Convolution Kernels on Discrete Structures. In Technical Report UCSC-CRL-99-10, University of California in Santa Cruz, Computer Science Department (July 1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Hichem Sahbi
    • 1
  • Xi Li
    • 1
    • 2
    • 3
  1. 1.CNRS Telecom ParisTechParisFrance
  2. 2.School of Computer ScienceThe University of AdelaideAustralia
  3. 3.NLPR, CASIABeijingChina

Personalised recommendations