Salient Object Detection: A Discriminative Regional Feature Integration Approach

Abstract

Feature integration provides a computational framework for saliency detection, and a lot of hand-crafted integration rules have been developed. In this paper, we present a principled extension, supervised feature integration, which learns a random forest regressor to discriminatively integrate the saliency features for saliency computation. In addition to contrast features, we introduce regional object-sensitive descriptors: the objectness descriptor characterizing the common spatial and appearance property of the salient object, and the image-specific backgroundness descriptor characterizing the appearance of the background of a specific image, which are shown more important for estimating the saliency. To the best of our knowledge, our supervised feature integration framework is the first successful approach to perform the integration over the saliency features for salient object detection, and outperforms the integration approach over the saliency maps. Together with fusing the multi-level regional saliency maps to impose the spatial saliency consistency, our approach significantly outperforms state-of-the-art methods on seven benchmark datasets. We also discuss several followup works which jointly learn the representation and the saliency map using deep learning.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Notes

  1. 1.

    They are empirically set to range from 5 to 1800 with increasingly larger gaps. Check our code at https://github.com/playerkk/drfi_matlab for more details.

  2. 2.

    http://supermoe.cs.umass.edu/~hzjiang/drfi/

  3. 3.

    Objectness is a feature vector in this paper and different from the concept objectness in Alexe et al. (2012) where objectness is used to quantify how likely it is for an image window to contain an object of any class.

  4. 4.

    http://research.microsoft.com/en-us/um/people/jiansun/

  5. 5.

    http://chenlab.ece.cornell.edu/projects/touch-coseg

  6. 6.

    http://www.wisdom.weizmann.ac.il/~vision/Seg_Evaluation_DB/

  7. 7.

    http://www.cse.cuhk.edu.hk/leojia/projects/hsaliency

  8. 8.

    http://ice.dlut.edu.cn/lu/dut-omron/homepage.htm

  9. 9.

    https://sites.google.com/site/ligb86/mdfsaliency/

  10. 10.

    We tested the performance of random forests with 5, 10, 50, 100, 150, 200, and 300 trees.

  11. 11.

    We tested the performance for 3, 5, 10, 15 and 20 features.

  12. 12.

    Note that this observation holds for salient object detection. It might not hold for other saliency detection task, e.g., eye-fixation prediction, in which the contrast feature is perhaps more important.

References

  1. Achanta, R., Hemami, S. S., Estrada, F. J., & Süsstrunk S. (2009). Frequency-tuned salient region detection. In CVPR.

  2. Alexe, B., Deselaers, T., & Ferrari, V. (2012). Measuring the objectness of image windows. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2189–2202.

    Article  Google Scholar 

  3. Alpert, S., Galun, M., Basri, R., & Brandt, A. (2007). Image segmentation by probabilistic bottom-up aggregation and cue integration. In CVPR.

  4. Batra, D., Kowdle, A., Parikh, D., Luo, J., & Chen, T. (2010). iCoseg: Interactive co-segmentation with intelligent scribble guidance. In IEEE CVPR (pp. 3169–3176).

  5. Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2014). Salient object detection: A survey. CoRR, arXiv:1411.5878.

  6. Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015). Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12), 5706–5722.

    MathSciNet  Article  Google Scholar 

  7. Borji, A., & Itti, L. (2012). Exploiting local and global patch rarities for saliency detection. In CVPR (pp. 478–485).

  8. Borji, A., & Itti, L. (2013). State-of-the-art in visual attention modeling. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 185–207.

    Article  Google Scholar 

  9. Borji, A., Sihite, D. N., & Itti, L. (2012). Salient object detection: A benchmark. ECCV, 2, 414–429.

    Google Scholar 

  10. Chang, K.-Y., Liu, T.-L., Chen, H.-T., Lai, S.-H. (2011). Fusing generic objectness and visual saliency for salient object detection. In ICCV (pp. 914–921).

  11. Chen, T., Lin, L., Liu, L., Luo, X., & Li, X. (2015). DISC: Deep image saliency computing via progressive representation learning. CoRR, arXiv:1511.04192.

  12. Cheng, M.-M., Mitra, N. J., Huang, X., Torr, P. H. S., & Hu, S.-M. (2014). Global contrast based salient region detection. In IEEE TPAMI.

  13. Cheng, M.-M., Warrell, J., Lin, W.-Y., Zheng, S., Vineet, V., & Crook, N. (2013). Efficient salient region detection with soft image abstraction. In ICCV (pp. 1529–1536).

  14. Desingh, K., Krishna, K. M., Rajan, D., & Jawahar, C. V. (2013). Depth really matters: Improving visual salient region detection with depth. In BMVC.

  15. Elazary, L., & Itti, L. (2008). Interesting objects are visually salient. Journal of Vision, 8(3), 3.1–3.15.

  16. Felzenszwalb, P. F., & Huttenlocher, D. P. (2004). Efficient graph-based image segmentation. International Journal of Computer Vision, 59(2), 167–181.

    Article  Google Scholar 

  17. Gao, D., Mahadevan, V., Vasconcelos, N. (2007). The discriminant center-surround hypothesis for bottom-up saliency. In NIPS.

  18. Gao, D., & Vasconcelos, N. (2007). Bottom-up saliency is a discriminant process. In ICCV (pp. 1–6).

  19. Goferman, S., Tal, A., & Zelnik-Manor, L. (2010). Puzzle-like collage. Computer Graphics Forum, 29(2), 459–468.

    Article  Google Scholar 

  20. Goferman, S., Zelnik-Manor, L., & Tal, A. (2010). Context-aware saliency detection. In CVPR (pp. 2376–2383).

  21. Hoiem, D., Efros, A. A., & Hebert, M. (2005). Geometric context from a single image. In ICCV (pp. 654–661).

  22. Itti, L. (2004). Automatic foveation for video compression using a neurobiological model of visual attention. IEEE TIP.

  23. Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE TPAMI.

  24. Jia, Y., & Han, M. (2013). Category-independent object-level saliency detection. In ICCV.

  25. Jiang, B., Zhang, L., Lu, H., Yang, C., & Yang, M.-H. (2013). Saliency detection via absorbing markov chain. In ICCV.

  26. Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., & Li, S. (2011). Automatic salient object segmentation based on context and shape prior. In BMVC.

  27. Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., & Li, S. (2013). Salient object detection: A discriminative regional feature integration approach. In IEEE CVPR (pp. 2083–2090).

  28. Jiang, P., Ling, H., Jingyi, Y. & Peng, J. (2013). Salient region detection by UFO: Uniqueness, focusness and objectness. In ICCV.

  29. Jiang, Z., & Davis, L. S. (2013). Submodular salient region detection. In CVPR (pp. 2043–2050).

  30. Kanan, C., & Cottrell, G. W. (2010). Robust classification of objects, faces, and flowers using natural image statistics. In CVPR (pp. 2472–2479).

  31. Khuwuthyakorn, P., Robles-Kelly, A., & Zhou, J. (2010). Object of interest detection by saliency learning. In ECCV.

  32. Kim, J., Han, D., Tai, Y.-W., Kim, J. (2014). Salient region detection via high-dimensional color transform. In CVPR.

  33. Kimchi, R., & Peterson, M. A. (2008). Figure-ground segmentation can occur without attention. Psychological Science, 19(7), 660–668.

  34. Klein, D. A., & Frintrop, S. (2011). Center-surround divergence of feature statistics for salient object detection. In ICCV.

  35. Koffka, K. (1935). Principles of Gestalt Psychology. Brace: Harcourt.

    Google Scholar 

  36. Küttel, D., & Ferrari, V. (2012). Figure-ground segmentation by transferring window masks. In CVPR (pp. 558–565).

  37. Li, G., & Yu, Y. (2015). Visual saliency based on multiscale deep features. In CVPR (pp. 5455–5463).

  38. Li, G., & Yizhou, Y. (2016). Deep contrast learning for salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  39. Li, N., Ye, J., Ji, Y., Ling, H., & Yu, J. (2014). Saliency detection on light fields. In CVPR.

  40. Li, X., Li, Y., Shen, C., Dick, A. R., & van den Hengel, A. (2013). Contextual hypergraph modeling for salient object detection. In ICCV (pp. 3328–3335).

  41. Li, X., Zhao, L., Wei, L., Yang, M., Wu, F., Zhuang, Y., Ling, H., & Wang, J. (2015). Deepsaliency: Multi-task deep neural network model for salient object detection. CoRR, arXiv:1510.05484.

  42. Li, X., Lu, H., Zhang, L., Ruan, X., & Yang, M.-H. (2013). Saliency detection via dense and sparse reconstruction. In ICCV.

  43. Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014). The secrets of salient object segmentation. In CVPR.

  44. Liu, F., & Gleicher, M. (2006). Region enhanced scale-invariant saliency detection. In ICME (pp. 1477–1480).

  45. Liu, N., Han, J., Zhang, D., Wen, S., & Liu, T. (2015). Predicting eye fixations using convolutional neural networks. In CVPR (pp. 362–370).

  46. Liu, R., Cao, J., Zhong, G., Lin, Z., Shan, S., & Su, Z. (2014). Adaptive partial differential equation learning for visual saliency detection. In CVPR.

  47. Liu, T., Sun, J., Zheng, N.-N., Tang, X., & Shum, H.-Y. (2007). Learning to detect a salient object. CVPR (pp. 1–8).

  48. Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., et al. (2011). Learning to detect a salient object. IEEE TPAMI, 33(2), 353–367.

    Article  Google Scholar 

  49. Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In CVPR (pp. 3431–3440).

  50. Lu, S., Mahadevan, V., & Vasconcelos, N. (2014). Learning optimal seeds for diffusion-based salient object detection. In CVPR.

  51. Lu, Y., Zhang, W., Jin, C., & Xue, X. (2012). Learning attention map from images. In CVPR (pp. 1067–1074).

  52. Lu, Y., Zhang, W., Lu, H., & Xue, X. (2011) Salient object detection using concavity context. In ICCV (pp. 233–240).

  53. Ma, Y.-F., & Zhang, H.-J. (2003). Contrast-based image attention analysis by using fuzzy growing. In ACM Multimedia.

  54. Marchesotti, L., Cifarelli, C., & Csurka, G. (2009). A framework for visual saliency detection with applications to image thumbnailing. In ICCV (pp. 2232–2239).

  55. Margolin, R., Tal, A., & Zelnik-Manor, L. (2013). What makes a patch distinct? In CVPR.

  56. Mehrani, P., & Veksler, O. (2010). Saliency segmentation based on learning and graph cut refinement. In BMVC.

  57. Moosmann, F., Larlus, D., & Jurie, F. (2006). Learning saliency maps for object categorization. In EECVW.

  58. Niu, Y., Geng, Y., Li, X., & Liu, F. (2012) Leveraging stereopsis for saliency analysis. In CVPR (pp. 454–461).

  59. Peng, H., Li, B., Ji, R., Hu, W., Xiong, W., & Lang, C. (2013). Salient object detection via low-rank and structured sparse matrix decomposition. In AAAI.

  60. Perazzi, F., Krähenbühl, P., Pritch, Y., & Hornung, A. (2012). Saliency filters: Contrast based filtering for salient region detection. In CVPR (pp. 733–740).

  61. Rahtu, E., Kannala, J., Salo, M., & Heikkilä, J. (2010). Segmenting salient objects from images and videos. ECCV, 5, 366–379.

    Google Scholar 

  62. Ren, X., Fowlkes, C. C., & Malik, J. (2006). Figure/ground assignment in natural images. ECCV, Part II (pp. 614–627).

  63. Rubin, E. (1958). Figure and ground. In Readings in Perception (pp. 194–203). Princeton, NJ: Van Nostrand.

  64. Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173.

    Article  Google Scholar 

  65. Scharfenberger, C., Wong, A., Fergani, K., Zelek, J. S., & Clausi, D. A. (2013). Statistical textural distinctiveness for salient region detection in natural images. In CVPR (pp. 979–986).

  66. Schroff, F., Criminisi, A., & Zisserman, A. (2008). Object class segmentation using random forests. In BMVC (pp. 1–10).

  67. Shen, X., & Wu, Y. (2012). A unified approach to salient object detection via low rank matrix recovery. In CVPR.

  68. Shi, K., Wang, K., Lu, J., & Lin, L. (2013). Pisa: Pixelwise image saliency by aggregating complementary appearance contrast measures with spatial priors. In CVPR (pp. 2115–2122).

  69. Treisman, A., & Gelad, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.

    Article  Google Scholar 

  70. Vicente, S., Kolmogorov, V., & Rother, C. (2008). Graph cut based image segmentation with connectivity priors. In CVPR.

  71. Walther, D., & Koch, C. (2006). Modeling attention to salient proto-objects. Neural Networks, 19(9), 1395–1407.

    Article  MATH  Google Scholar 

  72. Wang, J., Quan, L., Sun, J., Tang, X., & Shum, H.-Y. (2006). Picture collage. CVPR, 1, 347–354.

    Google Scholar 

  73. Wang, L., Xue, J., Zheng, N., & Hua, G. (2011). Automatic salient object extraction with contextual cue. In ICCV.

  74. Wang, M., Konrad, J., Ishwar, P., Jing, K., & Rowley, H. A. (2011). Image saliency: From intrinsic to extrinsic context. In CVPR (pp. 417–424).

  75. Wang, P., Wang, J., Zeng, G., Feng, J., Zha, H., & Li, S. (2012). Salient object detection for searched web images via global saliency. In CVPR (pp. 3194–3201).

  76. Wang, P., Zhang, D., Wang, J., Wu, Z., Hua, X.-S. & Li, S. (2012). Color filter for image search. In ACM Multimedia.

  77. Wang, P., Zhang, D., Zeng, G., & Wang, J. (2012). Contextual dominant color name extraction for web image search. In ICME Workshops (pp. 319–324).

  78. Wei, Y., Wen, F., Zhu, W., & Sun, J. (2012). Geodesic saliency using background priors. ECCV, 3, 29–42.

    Google Scholar 

  79. Xie, Y., Huchuan, L., & Yang, M.-H. (2013). Bayesian saliency via low and mid level cues. IEEE TIP, 22(5), 1689–1698.

    MathSciNet  Google Scholar 

  80. Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical saliency detection. In CVPR (pp. 1155–1162).

  81. Yang, C., Zhang, L., Lu, H., Ruan, X., & Yang, M.-H. (2013). Saliency detection via graph-based manifold ranking. In CVPR.

  82. Yu, S. X., & Shi, J. (2003). Object-specific figure-ground segregation. In CVPR (pp. 39–45).

  83. Zhang, J., & Sclaroff, S. (2013). Saliency detection: A boolean map approach. In ICCV (pp. 153–160).

  84. Zhu, W., Liang, S., Wei, Y., & Sun, J. (2014). Saliency optimization from robust background detection. In CVPR.

  85. Zou, W., Kpalma, K., Liu, Z., Ronsin, J., et al. (2013). Segmentation driven low-rank matrix recovery for saliency detection. In BMVC (pp. 1–13).

Download references

Acknowledgements

This work was supported in part by the National Basic Research Program of China under Grant No. 2015CB351703 and 2012CB316400, and the National Natural Science Foundation of China under Grant No. 91120006 and NSFC (No. 61572264), Huawei Innovation Research Program (HIRP), and CAST young talents plan.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jingdong Wang.

Additional information

Communicated by Cordelia Schmid.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Jiang, H., Yuan, Z. et al. Salient Object Detection: A Discriminative Regional Feature Integration Approach. Int J Comput Vis 123, 251–268 (2017). https://doi.org/10.1007/s11263-016-0977-3

Download citation

Keywords

  • Salient object detection
  • Data-driven