HFS: Hierarchical Feature Selection for Efficient Image Segmentation

  • Ming-Ming Cheng
  • Yun Liu
  • Qibin Hou
  • Jiawang Bian
  • Philip Torr
  • Shi-Min Hu
  • Zhuowen Tu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9907)


In this paper, we propose a real-time system, Hierarchical Feature Selection (HFS), that performs image segmentation at a speed of 50 frames-per-second. We make an attempt to improve the performance of previous image segmentation systems by focusing on two aspects: (1) a careful system implementation on modern GPUs for efficient feature computation; and (2) an effective hierarchical feature selection and fusion strategy with learning. Compared with classic segmentation algorithms, our system demonstrates its particular advantage in speed, with comparable results in segmentation quality. Adopting HFS in applications like salient object detection and object proposal generation results in a significant performance boost. Our proposed HFS system (will be open-sourced) can be used in a variety computer vision tasks that are built on top of image segmentation and superpixel extraction.


Image segmentation Superpixel Grouping 



We would like to thank the anonymous reviewers for their useful feedbacks. This research was sponsored by NSFC (NO. 61572264), Huawei Innovation Research Program (HIRP), and CAST young talents plan.


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE TPAMI 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE TPAMI 34(11), 2189–2202 (2012)CrossRefGoogle Scholar
  3. 3.
    Alpert, S., Galun, M., Brandt, A., Basri, R.: Image segmentation by probabilistic bottom-up aggregation and cue integration. IEEE TPAMI 34(2), 315–327 (2012)CrossRefGoogle Scholar
  4. 4.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE TPAMI 33(5), 898–916 (2011)CrossRefGoogle Scholar
  5. 5.
    Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., Malik, J.: Multiscale combinatorial grouping. In: IEEE CVPR, pp. 328–335 (2014)Google Scholar
  6. 6.
    Chang, J., Wei, D., Fisher, J.W.: A video representation using temporal superpixels. In: IEEE CVPR, pp. 2051–2058 (2013)Google Scholar
  7. 7.
    Chen, T., Cheng, M.M., Tan, P., Shamir, A., Hu, S.M.: Sketch2Photo: internet image montage. ACM TOG 28(5), 124 (2009)Google Scholar
  8. 8.
    Chen, X., Ma, H., Wang, X., Zhao, Z.: Improving object proposals with multi-thresholding straddling expansion. In: IEEE CVPR (2015)Google Scholar
  9. 9.
    Cheng, J., Liu, J., Xu, Y., Yin, F., Wong, D.W.K., Tan, N.M., Tao, D., Cheng, C.Y., Aung, T., Wong, T.Y.: Superpixel classification based optic disc and optic cup segmentation for glaucoma screening. IEEE Trans. Med. Imaging 32(6), 1019–1032 (2013)CrossRefGoogle Scholar
  10. 10.
    Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.: Global contrast based salient region detection. IEEE TPAMI 37(3), 569–582 (2015)CrossRefGoogle Scholar
  11. 11.
    Cheng, M.M., Warrell, J., Lin, W.Y., Zheng, S., Vineet, V., Crook, N.: Efficient salient region detection with soft image abstraction. In: IEEE ICCV, pp. 1529–1536 (2013)Google Scholar
  12. 12.
    Cheng, M.M., Zhang, Z., Lin, W.Y., Torr, P.: Bing: binarized normed gradients for objectness estimation at 300fps. In: IEEE CVPR, pp. 3286–3293 (2014)Google Scholar
  13. 13.
    Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE TPAMI 24(5), 603–619 (2002)CrossRefGoogle Scholar
  14. 14.
    Cour, T., Benezit, F., Shi, J.: Spectral segmentation with multiscale graph decomposition. In: IEEE CVPR, vol. 2, pp. 1124–1131 (2005)Google Scholar
  15. 15.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE CVPR, pp. 248–255 (2009)Google Scholar
  16. 16.
    Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. IEEE ICCV 88(2), 303–338 (2010)Google Scholar
  17. 17.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. IJCV 59(2), 167–181 (2004)CrossRefGoogle Scholar
  18. 18.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE CVPR, pp. 580–587 (2014)Google Scholar
  19. 19.
    Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. IEEE TPAMI 34(10), 1915–1926 (2012)CrossRefGoogle Scholar
  20. 20.
    Hoiem, D., Efros, A., Hebert, M., et al.: Geometric context from a single image. IEEE ICCV, vol. 1, pp. 654–661 (2005)Google Scholar
  21. 21.
    Hu, S.M., Zhang, F.L., Wang, M., Martin, R.R., Wang, J.: PatchNet: a patch-based image representation for interactive library-driven image editing. ACM TOG 32(6), 196 (2013)Google Scholar
  22. 22.
    Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., Li, S.: Salient object detection: a discriminative regional feature integration approach. In: IEEE CVPR, pp. 2083–2090 (2013)Google Scholar
  23. 23.
    Jiang, Z., Davis, L.S.: Submodular salient region detection. In: IEEE CVPR, pp. 2043–2050 (2013)Google Scholar
  24. 24.
    Juneja, M., Vedaldi, A., Jawahar, C., Zisserman, A.: Blocks that shout: distinctive parts for scene classification. In: IEEE CVPR, pp. 923–930 (2013)Google Scholar
  25. 25.
    Kohli, P., Torr, P.H., et al.: Robust higher order potentials for enforcing label consistency. IJCV 82(3), 302–324 (2009)CrossRefGoogle Scholar
  26. 26.
    Li, K., Zhu, Y., Yang, J., Jiang, J.: Video super-resolution using an adaptive superpixel-guided auto-regressive model. Pattern Recogn. 51, 59–71 (2016)CrossRefGoogle Scholar
  27. 27.
    Li, Y., Hou, X., Koch, C., Rehg, J., Yuille, A.: The secrets of salient object segmentation. In: IEEE CVPR, pp. 280–287 (2014)Google Scholar
  28. 28.
    Maire, M., Yu, S.X.: Progressive multigrid eigensolvers for multiscale spectral segmentation. In: IEEE ICCV, pp. 2184–2191 (2013)Google Scholar
  29. 29.
    Margolin, R., Tal, A., Zelnik-Manor, L.: What makes a patch distinct? In: IEEE CVPR, pp. 1139–1146 (2013)Google Scholar
  30. 30.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE TPAMI 26(5), 530–549 (2004)CrossRefGoogle Scholar
  31. 31.
    Nguyen, R.M., Brown, M.S.: Fast and effective l0 gradient minimization by region fusion. In: IEEE ICCV, pp. 208–216 (2015)Google Scholar
  32. 32.
    Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: IEEE CVPR, pp. 733–740 (2012)Google Scholar
  33. 33.
    Qi, W., Cheng, M.M., Borji, A., Lu, H., Bai, L.F.: SaliencyRank: two-stage manifold ranking for salient object detection. Comput. Vis. Media 1(4), 309–320 (2015)CrossRefGoogle Scholar
  34. 34.
    Ren, C.Y., Prisacariu, V.A., Reid, I.D.: gSLICr: SLIC superpixels at over 250Hz. arXiv preprint arXiv:1509.04232 (2015)
  35. 35.
    Ren, Z., Shakhnarovich, G.: Image segmentation by cascaded region agglomeration. In: IEEE CVPR, pp. 2011–2018 (2013)Google Scholar
  36. 36.
    Rivest, J., Cabanagh, P.: Localizing contours defined by more than one attribute. Vis. Res. 36(1), 53–66 (1996)CrossRefGoogle Scholar
  37. 37.
    Russell, C., Kohli, P., Torr, P.H., et al.: Associative hierarchical CRFs for object class image segmentation. In: IEEE ICCV, pp. 739–746 (2009)Google Scholar
  38. 38.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE TPAMI 22(8), 888–905 (2000)CrossRefGoogle Scholar
  39. 39.
    Song, X., Zhang, J., Han, Y., Jiang, J.: Semi-supervised feature selection via hierarchical regression for web image classification. Multimedia Syst. 22(1), 41–49 (2016)CrossRefGoogle Scholar
  40. 40.
    Storath, M., Weinmann, A.: Fast partitioning of vector-valued images. SIAM J. Imaging Sci. 7(3), 1826–1852 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Sun, J., Ponce, J.: Learning discriminative part detectors for image classification and cosegmentation. In: IEEE ICCV, pp. 3400–3407 (2013)Google Scholar
  42. 42.
    Taylor, C.J.: Towards fast and accurate segmentation. In: IEEE CVPR, pp. 1916–1922 (2013)Google Scholar
  43. 43.
    Uijlings, J.R., van de Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. IJCV 104(2), 154–171 (2013)CrossRefGoogle Scholar
  44. 44.
    Wang, S., Lu, H., Yang, F., Yang, M.H.: Superpixel tracking. In: IEEE ICCV, pp. 1323–1330 (2011)Google Scholar
  45. 45.
    Wang, X., Yang, M., Zhu, S., Lin, Y.: Regionlets for generic object detection. In: IEEE ICCV, pp. 17–24 (2013)Google Scholar
  46. 46.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: IEEE ICCV, pp. 1395–1403 (2015)Google Scholar
  47. 47.
    Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: IEEE CVPR, pp. 1155–1162 (2013)Google Scholar
  48. 48.
    Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.: Saliency detection via graph-based manifold ranking. In: IEEE CVPR, pp. 3166–3173 (2013)Google Scholar
  49. 49.
    Zhang, L., Gao, Y., Xia, Y., Lu, K., Shen, J., Ji, R.: Representative discovery of structure cues for weakly-supervised image segmentation. IEEE Trans. Multimedia 16(2), 470–479 (2014)CrossRefGoogle Scholar
  50. 50.
    Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 391–405. Springer, Heidelberg (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Ming-Ming Cheng
    • 1
  • Yun Liu
    • 1
  • Qibin Hou
    • 1
  • Jiawang Bian
    • 1
  • Philip Torr
    • 3
  • Shi-Min Hu
    • 2
  • Zhuowen Tu
    • 4
  1. 1.CCCE & CSNankai UniversityTianjinChina
  2. 2.Tsinghua UniversityBeijingChina
  3. 3.Oxford UniversityOxfordUK
  4. 4.UCSDSan DiegoUSA

Personalised recommendations