Image Region Labeling by Exploring Contextual Information of Visual Spatial and Semantic Concepts

  • Kai He
  • Wen Chan
  • Guangtang Zhu
  • Lan Lin
  • Xiangdong Zhou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8879)


Region Labeling is to automatically assign semantic labels to the corresponding image regions. Most of the previous works focus on exploiting low level visual features, particularly visual spatial contextual information, to address the problem. However, very few work explore high level semantic information of the whole image to deal with the problem. In this paper, we propose a new region labeling approach by integrating both visual spatial and semantic contextual information into a unified model. In our method, region labeling is regarded as a multi-class classification problem. For each semantic concept, we train a Conditional Random Field (CRF) model respectively. It consists of both the region grid sub-graph and the co-occurred semantic label sub-graph. In our model, the integration of the two kinds of contextual information brings reinforcement effect on the improvement of region labeling. The experiments are conducted on two commonly used benchmark datasets and the experimental results show that our method achieves the best performance compared with the strong baselines and the state-of-the-art methods.


Image Region Labeling Region Graph Concept Graph Conditional Random Field 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Athanasiadis, T., Mylonas, P., Avrithis, Y.: A context-based region labeling approach for semantic image segmentation. In: Avrithis, Y., Kompatsiaris, Y., Staab, S., O’Connor, N.E. (eds.) SAMT 2006. LNCS, vol. 4306, pp. 212–225. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Corso, J.J., Yuille, A., Tu, Z.: Natural image labeling by dynamic hierarchical computing. In: CVPR (2008)Google Scholar
  3. 3.
    Doretto, G., Yao, Y.: Region moments: Fast invariant descriptors for detecting samll image strctures. In: CVPR (2010)Google Scholar
  4. 4.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. IJCV 59(2), 167–181 (2004)CrossRefGoogle Scholar
  5. 5.
    Feng, S., Manmatha, R., Lavrenko, V.: Multiple bernoulli relevance models for image and vedio annotation. In: CVPR (2004)Google Scholar
  6. 6.
    Han, Y., Wu, F., Shao, J., Tian, Q., Zhuang, Y.: Graph-guided sparse reconstruction for region tagging. In: CVPR (2012)Google Scholar
  7. 7.
    Ji, C., Zhou, X., Lin, L., Yang, W.: Labeling images by integrating sparse multiple distance learning and semantic context modeling. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 688–701. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: ICML (2001)Google Scholar
  9. 9.
    Lin, Z., Chan, W., He, K., Zhou, X., Wang, M.: Conditional random fileds for image region labeling with global observation. In: Huet, B., Ngo, C.-W., Tang, J., Zhou, Z.-H., Hauptmann, A.G., Yan, S. (eds.) PCM 2013. LNCS, vol. 8294, pp. 586–597. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  10. 10.
    Liu, X., Cheng, B., Yan, S.: Label to region by bi-layer sparsity priors. In: MM (2009)Google Scholar
  11. 11.
    Llorente, A., Motta, E., Rüger, S.: Exploring the semantics behind a collection to improve automated image annotation. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 307–314. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  13. 13.
    Rubinstein, M., Liu, C., Freeman, W.T.: Annnotaion propagation in large image databases via dense image correspondence. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 85–99. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  14. 14.
    Shotton, J., Winn, J.M., Rother, C., Criminisi, A.: Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 1–15. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  15. 15.
    Sutton, C., McCallum, A.: An introduction to conditional random fields. Foundations and Trends in Machine Learning 4, 267–373 (2011)CrossRefzbMATHGoogle Scholar
  16. 16.
    Tang, J., Hong, R., Yan, S., Chua, T., Qi, G., Jain, R.: Image annotation by knn-sparse graph-based label propagation over noisily-tagged web images. ACM Transactions on Intelligent Systems and Techology 2 (2011)Google Scholar
  17. 17.
    Toyota, T., Hasegawa, O.: Random field model for integration of local information and global information. IEEE Tranaction on PAMI 30(8), 1483–1489 (2008)CrossRefGoogle Scholar
  18. 18.
    Wang, H., Huang, H., Ding, C.: Image annotation using bi-relational graph of images and semantic labels. In: CVPR (2011)Google Scholar
  19. 19.
    Xiang, Y., Zhou, X., Chua, T., Ngo, C.: A revisit of generative model for automatic image annotation using markov random fields. In: CVPR (2009)Google Scholar
  20. 20.
    Xiang, Y., Zhou, X., Chua, T., Ngo, C.: Semantic context modeling with maximal margin conditional random fields for automatic image annotation. In: CVPR (2010)Google Scholar
  21. 21.
    Zheng, J., Jiang, Z.: Tag taxonomy aware dictionary learning for region tagging. In: CVPR (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Kai He
    • 1
  • Wen Chan
    • 1
  • Guangtang Zhu
    • 1
  • Lan Lin
    • 2
  • Xiangdong Zhou
    • 1
  1. 1.School of Computer ScienceFudan UniversityChina
  2. 2.School of Electronics and Information EngineeringTongji UniversityChina

Personalised recommendations