International Journal of Computer Vision

, Volume 123, Issue 2, pp 251–268 | Cite as

Salient Object Detection: A Discriminative Regional Feature Integration Approach

  • Jingdong WangEmail author
  • Huaizu Jiang
  • Zejian Yuan
  • Ming-Ming Cheng
  • Xiaowei Hu
  • Nanning Zheng


Feature integration provides a computational framework for saliency detection, and a lot of hand-crafted integration rules have been developed. In this paper, we present a principled extension, supervised feature integration, which learns a random forest regressor to discriminatively integrate the saliency features for saliency computation. In addition to contrast features, we introduce regional object-sensitive descriptors: the objectness descriptor characterizing the common spatial and appearance property of the salient object, and the image-specific backgroundness descriptor characterizing the appearance of the background of a specific image, which are shown more important for estimating the saliency. To the best of our knowledge, our supervised feature integration framework is the first successful approach to perform the integration over the saliency features for salient object detection, and outperforms the integration approach over the saliency maps. Together with fusing the multi-level regional saliency maps to impose the spatial saliency consistency, our approach significantly outperforms state-of-the-art methods on seven benchmark datasets. We also discuss several followup works which jointly learn the representation and the saliency map using deep learning.


Salient object detection Data-driven 



This work was supported in part by the National Basic Research Program of China under Grant No. 2015CB351703 and 2012CB316400, and the National Natural Science Foundation of China under Grant No. 91120006 and NSFC (No. 61572264), Huawei Innovation Research Program (HIRP), and CAST young talents plan.


  1. Achanta, R., Hemami, S. S., Estrada, F. J., & Süsstrunk S. (2009). Frequency-tuned salient region detection. In CVPR.Google Scholar
  2. Alexe, B., Deselaers, T., & Ferrari, V. (2012). Measuring the objectness of image windows. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2189–2202.CrossRefGoogle Scholar
  3. Alpert, S., Galun, M., Basri, R., & Brandt, A. (2007). Image segmentation by probabilistic bottom-up aggregation and cue integration. In CVPR.Google Scholar
  4. Batra, D., Kowdle, A., Parikh, D., Luo, J., & Chen, T. (2010). iCoseg: Interactive co-segmentation with intelligent scribble guidance. In IEEE CVPR (pp. 3169–3176).Google Scholar
  5. Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2014). Salient object detection: A survey. CoRR, arXiv:1411.5878.
  6. Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015). Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12), 5706–5722.MathSciNetCrossRefGoogle Scholar
  7. Borji, A., & Itti, L. (2012). Exploiting local and global patch rarities for saliency detection. In CVPR (pp. 478–485).Google Scholar
  8. Borji, A., & Itti, L. (2013). State-of-the-art in visual attention modeling. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 185–207.CrossRefGoogle Scholar
  9. Borji, A., Sihite, D. N., & Itti, L. (2012). Salient object detection: A benchmark. ECCV, 2, 414–429.Google Scholar
  10. Chang, K.-Y., Liu, T.-L., Chen, H.-T., Lai, S.-H. (2011). Fusing generic objectness and visual saliency for salient object detection. In ICCV (pp. 914–921).Google Scholar
  11. Chen, T., Lin, L., Liu, L., Luo, X., & Li, X. (2015). DISC: Deep image saliency computing via progressive representation learning. CoRR, arXiv:1511.04192.
  12. Cheng, M.-M., Mitra, N. J., Huang, X., Torr, P. H. S., & Hu, S.-M. (2014). Global contrast based salient region detection. In IEEE TPAMI.Google Scholar
  13. Cheng, M.-M., Warrell, J., Lin, W.-Y., Zheng, S., Vineet, V., & Crook, N. (2013). Efficient salient region detection with soft image abstraction. In ICCV (pp. 1529–1536).Google Scholar
  14. Desingh, K., Krishna, K. M., Rajan, D., & Jawahar, C. V. (2013). Depth really matters: Improving visual salient region detection with depth. In BMVC.Google Scholar
  15. Elazary, L., & Itti, L. (2008). Interesting objects are visually salient. Journal of Vision, 8(3), 3.1–3.15.Google Scholar
  16. Felzenszwalb, P. F., & Huttenlocher, D. P. (2004). Efficient graph-based image segmentation. International Journal of Computer Vision, 59(2), 167–181.CrossRefGoogle Scholar
  17. Gao, D., Mahadevan, V., Vasconcelos, N. (2007). The discriminant center-surround hypothesis for bottom-up saliency. In NIPS.Google Scholar
  18. Gao, D., & Vasconcelos, N. (2007). Bottom-up saliency is a discriminant process. In ICCV (pp. 1–6).Google Scholar
  19. Goferman, S., Tal, A., & Zelnik-Manor, L. (2010). Puzzle-like collage. Computer Graphics Forum, 29(2), 459–468.CrossRefGoogle Scholar
  20. Goferman, S., Zelnik-Manor, L., & Tal, A. (2010). Context-aware saliency detection. In CVPR (pp. 2376–2383).Google Scholar
  21. Hoiem, D., Efros, A. A., & Hebert, M. (2005). Geometric context from a single image. In ICCV (pp. 654–661).Google Scholar
  22. Itti, L. (2004). Automatic foveation for video compression using a neurobiological model of visual attention. IEEE TIP.Google Scholar
  23. Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE TPAMI.Google Scholar
  24. Jia, Y., & Han, M. (2013). Category-independent object-level saliency detection. In ICCV.Google Scholar
  25. Jiang, B., Zhang, L., Lu, H., Yang, C., & Yang, M.-H. (2013). Saliency detection via absorbing markov chain. In ICCV.Google Scholar
  26. Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., & Li, S. (2011). Automatic salient object segmentation based on context and shape prior. In BMVC.Google Scholar
  27. Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., & Li, S. (2013). Salient object detection: A discriminative regional feature integration approach. In IEEE CVPR (pp. 2083–2090).Google Scholar
  28. Jiang, P., Ling, H., Jingyi, Y. & Peng, J. (2013). Salient region detection by UFO: Uniqueness, focusness and objectness. In ICCV.Google Scholar
  29. Jiang, Z., & Davis, L. S. (2013). Submodular salient region detection. In CVPR (pp. 2043–2050).Google Scholar
  30. Kanan, C., & Cottrell, G. W. (2010). Robust classification of objects, faces, and flowers using natural image statistics. In CVPR (pp. 2472–2479).Google Scholar
  31. Khuwuthyakorn, P., Robles-Kelly, A., & Zhou, J. (2010). Object of interest detection by saliency learning. In ECCV.Google Scholar
  32. Kim, J., Han, D., Tai, Y.-W., Kim, J. (2014). Salient region detection via high-dimensional color transform. In CVPR.Google Scholar
  33. Kimchi, R., & Peterson, M. A. (2008). Figure-ground segmentation can occur without attention. Psychological Science, 19(7), 660–668.Google Scholar
  34. Klein, D. A., & Frintrop, S. (2011). Center-surround divergence of feature statistics for salient object detection. In ICCV.Google Scholar
  35. Koffka, K. (1935). Principles of Gestalt Psychology. Brace: Harcourt.Google Scholar
  36. Küttel, D., & Ferrari, V. (2012). Figure-ground segmentation by transferring window masks. In CVPR (pp. 558–565).Google Scholar
  37. Li, G., & Yu, Y. (2015). Visual saliency based on multiscale deep features. In CVPR (pp. 5455–5463).Google Scholar
  38. Li, G., & Yizhou, Y. (2016). Deep contrast learning for salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
  39. Li, N., Ye, J., Ji, Y., Ling, H., & Yu, J. (2014). Saliency detection on light fields. In CVPR.Google Scholar
  40. Li, X., Li, Y., Shen, C., Dick, A. R., & van den Hengel, A. (2013). Contextual hypergraph modeling for salient object detection. In ICCV (pp. 3328–3335).Google Scholar
  41. Li, X., Zhao, L., Wei, L., Yang, M., Wu, F., Zhuang, Y., Ling, H., & Wang, J. (2015). Deepsaliency: Multi-task deep neural network model for salient object detection. CoRR, arXiv:1510.05484.
  42. Li, X., Lu, H., Zhang, L., Ruan, X., & Yang, M.-H. (2013). Saliency detection via dense and sparse reconstruction. In ICCV.Google Scholar
  43. Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014). The secrets of salient object segmentation. In CVPR.Google Scholar
  44. Liu, F., & Gleicher, M. (2006). Region enhanced scale-invariant saliency detection. In ICME (pp. 1477–1480).Google Scholar
  45. Liu, N., Han, J., Zhang, D., Wen, S., & Liu, T. (2015). Predicting eye fixations using convolutional neural networks. In CVPR (pp. 362–370).Google Scholar
  46. Liu, R., Cao, J., Zhong, G., Lin, Z., Shan, S., & Su, Z. (2014). Adaptive partial differential equation learning for visual saliency detection. In CVPR.Google Scholar
  47. Liu, T., Sun, J., Zheng, N.-N., Tang, X., & Shum, H.-Y. (2007). Learning to detect a salient object. CVPR (pp. 1–8).Google Scholar
  48. Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., et al. (2011). Learning to detect a salient object. IEEE TPAMI, 33(2), 353–367.CrossRefGoogle Scholar
  49. Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In CVPR (pp. 3431–3440).Google Scholar
  50. Lu, S., Mahadevan, V., & Vasconcelos, N. (2014). Learning optimal seeds for diffusion-based salient object detection. In CVPR.Google Scholar
  51. Lu, Y., Zhang, W., Jin, C., & Xue, X. (2012). Learning attention map from images. In CVPR (pp. 1067–1074).Google Scholar
  52. Lu, Y., Zhang, W., Lu, H., & Xue, X. (2011) Salient object detection using concavity context. In ICCV (pp. 233–240).Google Scholar
  53. Ma, Y.-F., & Zhang, H.-J. (2003). Contrast-based image attention analysis by using fuzzy growing. In ACM Multimedia.Google Scholar
  54. Marchesotti, L., Cifarelli, C., & Csurka, G. (2009). A framework for visual saliency detection with applications to image thumbnailing. In ICCV (pp. 2232–2239).Google Scholar
  55. Margolin, R., Tal, A., & Zelnik-Manor, L. (2013). What makes a patch distinct? In CVPR.Google Scholar
  56. Mehrani, P., & Veksler, O. (2010). Saliency segmentation based on learning and graph cut refinement. In BMVC.Google Scholar
  57. Moosmann, F., Larlus, D., & Jurie, F. (2006). Learning saliency maps for object categorization. In EECVW.Google Scholar
  58. Niu, Y., Geng, Y., Li, X., & Liu, F. (2012) Leveraging stereopsis for saliency analysis. In CVPR (pp. 454–461).Google Scholar
  59. Peng, H., Li, B., Ji, R., Hu, W., Xiong, W., & Lang, C. (2013). Salient object detection via low-rank and structured sparse matrix decomposition. In AAAI.Google Scholar
  60. Perazzi, F., Krähenbühl, P., Pritch, Y., & Hornung, A. (2012). Saliency filters: Contrast based filtering for salient region detection. In CVPR (pp. 733–740).Google Scholar
  61. Rahtu, E., Kannala, J., Salo, M., & Heikkilä, J. (2010). Segmenting salient objects from images and videos. ECCV, 5, 366–379.Google Scholar
  62. Ren, X., Fowlkes, C. C., & Malik, J. (2006). Figure/ground assignment in natural images. ECCV, Part II (pp. 614–627).Google Scholar
  63. Rubin, E. (1958). Figure and ground. In Readings in Perception (pp. 194–203). Princeton, NJ: Van Nostrand.Google Scholar
  64. Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173.CrossRefGoogle Scholar
  65. Scharfenberger, C., Wong, A., Fergani, K., Zelek, J. S., & Clausi, D. A. (2013). Statistical textural distinctiveness for salient region detection in natural images. In CVPR (pp. 979–986).Google Scholar
  66. Schroff, F., Criminisi, A., & Zisserman, A. (2008). Object class segmentation using random forests. In BMVC (pp. 1–10).Google Scholar
  67. Shen, X., & Wu, Y. (2012). A unified approach to salient object detection via low rank matrix recovery. In CVPR.Google Scholar
  68. Shi, K., Wang, K., Lu, J., & Lin, L. (2013). Pisa: Pixelwise image saliency by aggregating complementary appearance contrast measures with spatial priors. In CVPR (pp. 2115–2122).Google Scholar
  69. Treisman, A., & Gelad, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.CrossRefGoogle Scholar
  70. Vicente, S., Kolmogorov, V., & Rother, C. (2008). Graph cut based image segmentation with connectivity priors. In CVPR.Google Scholar
  71. Walther, D., & Koch, C. (2006). Modeling attention to salient proto-objects. Neural Networks, 19(9), 1395–1407.CrossRefzbMATHGoogle Scholar
  72. Wang, J., Quan, L., Sun, J., Tang, X., & Shum, H.-Y. (2006). Picture collage. CVPR, 1, 347–354.Google Scholar
  73. Wang, L., Xue, J., Zheng, N., & Hua, G. (2011). Automatic salient object extraction with contextual cue. In ICCV.Google Scholar
  74. Wang, M., Konrad, J., Ishwar, P., Jing, K., & Rowley, H. A. (2011). Image saliency: From intrinsic to extrinsic context. In CVPR (pp. 417–424).Google Scholar
  75. Wang, P., Wang, J., Zeng, G., Feng, J., Zha, H., & Li, S. (2012). Salient object detection for searched web images via global saliency. In CVPR (pp. 3194–3201).Google Scholar
  76. Wang, P., Zhang, D., Wang, J., Wu, Z., Hua, X.-S. & Li, S. (2012). Color filter for image search. In ACM Multimedia.Google Scholar
  77. Wang, P., Zhang, D., Zeng, G., & Wang, J. (2012). Contextual dominant color name extraction for web image search. In ICME Workshops (pp. 319–324).Google Scholar
  78. Wei, Y., Wen, F., Zhu, W., & Sun, J. (2012). Geodesic saliency using background priors. ECCV, 3, 29–42.Google Scholar
  79. Xie, Y., Huchuan, L., & Yang, M.-H. (2013). Bayesian saliency via low and mid level cues. IEEE TIP, 22(5), 1689–1698.MathSciNetGoogle Scholar
  80. Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical saliency detection. In CVPR (pp. 1155–1162).Google Scholar
  81. Yang, C., Zhang, L., Lu, H., Ruan, X., & Yang, M.-H. (2013). Saliency detection via graph-based manifold ranking. In CVPR.Google Scholar
  82. Yu, S. X., & Shi, J. (2003). Object-specific figure-ground segregation. In CVPR (pp. 39–45).Google Scholar
  83. Zhang, J., & Sclaroff, S. (2013). Saliency detection: A boolean map approach. In ICCV (pp. 153–160).Google Scholar
  84. Zhu, W., Liang, S., Wei, Y., & Sun, J. (2014). Saliency optimization from robust background detection. In CVPR.Google Scholar
  85. Zou, W., Kpalma, K., Liu, Z., Ronsin, J., et al. (2013). Segmentation driven low-rank matrix recovery for saliency detection. In BMVC (pp. 1–13).Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Microsoft ResearchBeijingPeople’s Republic of China
  2. 2.University of MassachusettsAmherstUSA
  3. 3.Xi’an Jiaotong UniversityXi’anPeople’s Republic of China
  4. 4.CCCE & CSNankai UniversityTianjinPeople’s Republic of China

Personalised recommendations