Stable multi-label boosting for image annotation with structural feature selection

  • 114 Accesses

  • 5 Citations


Automatic annotating images with appropriate multiple tags are very important to image retrieval and image understanding. We can obtain high-dimensional heterogenous visual features from real-world images to describe their various aspects of visual characteristics, such as color, texture, and shape. Different kinds of heterogenous features have different intrinsic discriminative power for image understanding. The selection of groups of discriminative features for certain semantics is hence crucial to make the image understanding more interpretable. This paper proposes an approach, called stable multi-label boosting with structural feature selection (S-MtBFS), for image annotation. S-MtBFS comprises two steps, namely structural feature selection for each label and stable multi-label boosting by curds and whey. In the first step, a (structural) sparse selection model is learned to identify subgroups of homogenous features for the purpose of predicting a certain label. Moreover, a stable method of multi-label boosting with a re-sampling policy is employed in the second step to utilize the correlations among multiple tags. Extensive experiments on public image datasets show that the proposed approach has better and stable performance of multi-label image annotation and leads to a quite interpretable model for image understanding.

This is a preview of subscription content, log in to check access.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.


  1. 1

    Grangier D, Bengio S. A discriminative kernel-based approach to rank images from text queries. IEEE Trans Patt Anal Mach Intel, 2008, 30: 1371–1384

  2. 2

    Chen Y, Wang J Z, Geman D. Image categorization by learning and reasoning with regions. J Mach Learn Res, 2004, 5: 913–939

  3. 3

    Maron O, Ratan A L. Multiple-instance learning for natural scene classification. In: Proceedings of the 15th International Conference on Machine Learning, Madison, Wisconsin, USA, 1998. 341–349

  4. 4

    Wang C, Yan S, Zhang L, et al. Multi-label sparse coding for automatic image annotation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009. 1643–1650

  5. 5

    Han Y, Wu F, Jia J, et al. Multi-task sparse discriminant analysis (MTSDA) with overlapping categories. In: Proceedings of the 24th AAAI Conference on Artificial Intelligence, Atlanta, Georgia, USA, 2010. 469–474

  6. 6

    Cao L, Luo J, Liang F, et al. Heterogeneous feature machines for visual recognition. In: Proceedings of the 12th IEEE International Conference on Computer Vision, Kyoto, Japan, 2009

  7. 7

    Tibshirani R. Regression shrinkage and selection via the lasso. J Royal Stat Soc Ser B, 1996, 58: 267–288

  8. 8

    Yuan M, Lin Y. Model selection and estimation in regression with grouped variables. J Royal Stat Soc Ser B, 2006, 68: 49–67

  9. 9

    Zou H, Hastie T. Regularization and variable selection via the elastic net. J Royal Stat Soc Ser B, 2005, 67: 301–320

  10. 10

    Friedman J, Hastie T, Tibshirani R. A note on the group lasso and a sparse group lasso. Arxiv preprint, arXiv: 1001.0736, 2010

  11. 11

    Shen X, Huang H. Grouping pursuit in regression. J Am Stat Assoc, 2010, 105: 727–739

  12. 12

    Breiman L, Friedman J. Predicting multivariate responses in multiple linear regression. J Royal Stat Soc Ser B, 1997, 59: 3–54

  13. 13

    Hotelling H. Relations between two sets of variates. Biometrika, 1936, 28: 321–377

  14. 14

    Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the International Joint Conference on Artificial Intelligence, Montréal, Québec, Canada, 1995. 1137–1145

  15. 15

    Miller R G. The jackknife—a review. Biometrika, 1974, 61: 1–15

  16. 16

    Efron B. Bootstrap methods: another look at the jackknife. Ann Stat, 1979, 7: 1–26

  17. 17

    Beck A, Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imag Sci, 2009, 2: 183–202

  18. 18

    Wu F, Han Y, Tian Q, et al. Multi-label boosting for image annotation by structural grouping sparsity. In: Proceedings of the ACM International Conference on Multimedia, Firenze, Italy, 2010. 15–24

  19. 19

    Bishop C M. Pattern Recognition and Machine Learning, Volume 4. New York: Springer, 2006

  20. 20

    Bousquet O, Elisseeff A. Stability and generalization. J Mach Learn Res, 2002, 2: 499–526

  21. 21

    Zhou Z H, Zhang M L. Multi-instance multi-label learning with application to scene classification. In: Proceedings of Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, 2007

  22. 22

    Chua T S, Tang J, Hong R, et al. Nus-wide: A real-world web image database from National University of Singapore. In: Proceedings of the ACM International Conference on Image and Video Retrieval, Island of Santorini, Greece, 2009. 1–9

  23. 23

    Lewis D D. Evaluating text categorization. In: Proceedings of Speech and Natural Language Workshop, Pacific Grove, California, USA, 1991. 312–318

  24. 24

    Ji S, Tang L, Yu S, et al. Extracting shared subspace for multi-label classification. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, 2008. 381–389

Download references

Author information

Correspondence to YaHong Han.

Additional information

ZHUANG YueTing was born in 1965. He received the B.S., M.S., and Ph.D. degrees from Zhejiang University, Hangzhou, China in 1986, 1989, and 1998, respectively. Currently, he is a Professor and Ph.D. Supervisor with the College of Computer Science, Zhejiang University. His current research interests include multimedia databases, artificial intelligence, and video-based animation.

WU Fei was born in 1973. He received the B.S. degree from Lanzhou University, Lanzhou, Gansu, China, the M.S. degree from Macao University, Taipa, Macau, and the Ph.D. degree from Zhejiang University, Hangzhou, China. He is currently a Professor with the College of Computer Science, Zhejiang University. His current research interests include multimedia analysis, retrieval, statistic learning, and pattern recognition.

HAN YaHong was born in 1977. He received the B.S. degree from Zhengzhou University, Zhengzhou, Henan, China in 2000, and the M.S. degree from Hohai University, Nanjing, Jiangsu, China in 2003. He is currently pursuing the Ph.D. degree from the College of Computer Science, Zhejiang University, Hangzhou, China. His current research interests include multimedia analysis, retrieval, and machine learning.

Electronic supplementary material

Supplementary material, approximately 9.71 MB.

Supplementary material, approximately 9.71 MB.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Zhuang, Y., Han, Y., Wu, F. et al. Stable multi-label boosting for image annotation with structural feature selection. Sci. China Inf. Sci. 54, 2508–2521 (2011).

Download citation


  • image annotation
  • structural feature selection
  • multi-label boosting
  • stability