Skip to main content
SpringerLink
Log in

Salient Object Detection: A Discriminative Regional Feature Integration Approach

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Feature integration provides a computational framework for saliency detection, and a lot of hand-crafted integration rules have been developed. In this paper, we present a principled extension, supervised feature integration, which learns a random forest regressor to discriminatively integrate the saliency features for saliency computation. In addition to contrast features, we introduce regional object-sensitive descriptors: the objectness descriptor characterizing the common spatial and appearance property of the salient object, and the image-specific backgroundness descriptor characterizing the appearance of the background of a specific image, which are shown more important for estimating the saliency. To the best of our knowledge, our supervised feature integration framework is the first successful approach to perform the integration over the saliency features for salient object detection, and outperforms the integration approach over the saliency maps. Together with fusing the multi-level regional saliency maps to impose the spatial saliency consistency, our approach significantly outperforms state-of-the-art methods on seven benchmark datasets. We also discuss several followup works which jointly learn the representation and the saliency map using deep learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. They are empirically set to range from 5 to 1800 with increasingly larger gaps. Check our code at https://github.com/playerkk/drfi_matlab for more details.

  2. http://supermoe.cs.umass.edu/~hzjiang/drfi/

  3. Objectness is a feature vector in this paper and different from the concept objectness in Alexe et al. (2012) where objectness is used to quantify how likely it is for an image window to contain an object of any class.

  4. http://research.microsoft.com/en-us/um/people/jiansun/

  5. http://chenlab.ece.cornell.edu/projects/touch-coseg

  6. http://www.wisdom.weizmann.ac.il/~vision/Seg_Evaluation_DB/

  7. http://www.cse.cuhk.edu.hk/leojia/projects/hsaliency

  8. http://ice.dlut.edu.cn/lu/dut-omron/homepage.htm

  9. https://sites.google.com/site/ligb86/mdfsaliency/

  10. We tested the performance of random forests with 5, 10, 50, 100, 150, 200, and 300 trees.

  11. We tested the performance for 3, 5, 10, 15 and 20 features.

  12. Note that this observation holds for salient object detection. It might not hold for other saliency detection task, e.g., eye-fixation prediction, in which the contrast feature is perhaps more important.

References

  • Achanta, R., Hemami, S. S., Estrada, F. J., & Süsstrunk S. (2009). Frequency-tuned salient region detection. In CVPR.

  • Alexe, B., Deselaers, T., & Ferrari, V. (2012). Measuring the objectness of image windows. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2189–2202.

    Article  Google Scholar 

  • Alpert, S., Galun, M., Basri, R., & Brandt, A. (2007). Image segmentation by probabilistic bottom-up aggregation and cue integration. In CVPR.

  • Batra, D., Kowdle, A., Parikh, D., Luo, J., & Chen, T. (2010). iCoseg: Interactive co-segmentation with intelligent scribble guidance. In IEEE CVPR (pp. 3169–3176).

  • Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2014). Salient object detection: A survey. CoRR, arXiv:1411.5878.

  • Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015). Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12), 5706–5722.

    Article  MathSciNet  Google Scholar 

  • Borji, A., & Itti, L. (2012). Exploiting local and global patch rarities for saliency detection. In CVPR (pp. 478–485).

  • Borji, A., & Itti, L. (2013). State-of-the-art in visual attention modeling. The IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 185–207.

    Article  Google Scholar 

  • Borji, A., Sihite, D. N., & Itti, L. (2012). Salient object detection: A benchmark. ECCV, 2, 414–429.

    Google Scholar 

  • Chang, K.-Y., Liu, T.-L., Chen, H.-T., Lai, S.-H. (2011). Fusing generic objectness and visual saliency for salient object detection. In ICCV (pp. 914–921).

  • Chen, T., Lin, L., Liu, L., Luo, X., & Li, X. (2015). DISC: Deep image saliency computing via progressive representation learning. CoRR, arXiv:1511.04192.

  • Cheng, M.-M., Mitra, N. J., Huang, X., Torr, P. H. S., & Hu, S.-M. (2014). Global contrast based salient region detection. In IEEE TPAMI.

  • Cheng, M.-M., Warrell, J., Lin, W.-Y., Zheng, S., Vineet, V., & Crook, N. (2013). Efficient salient region detection with soft image abstraction. In ICCV (pp. 1529–1536).

  • Desingh, K., Krishna, K. M., Rajan, D., & Jawahar, C. V. (2013). Depth really matters: Improving visual salient region detection with depth. In BMVC.

  • Elazary, L., & Itti, L. (2008). Interesting objects are visually salient. Journal of Vision, 8(3), 3.1–3.15.

  • Felzenszwalb, P. F., & Huttenlocher, D. P. (2004). Efficient graph-based image segmentation. International Journal of Computer Vision, 59(2), 167–181.

    Article  Google Scholar 

  • Gao, D., Mahadevan, V., Vasconcelos, N. (2007). The discriminant center-surround hypothesis for bottom-up saliency. In NIPS.

  • Gao, D., & Vasconcelos, N. (2007). Bottom-up saliency is a discriminant process. In ICCV (pp. 1–6).

  • Goferman, S., Tal, A., & Zelnik-Manor, L. (2010). Puzzle-like collage. Computer Graphics Forum, 29(2), 459–468.

    Article  Google Scholar 

  • Goferman, S., Zelnik-Manor, L., & Tal, A. (2010). Context-aware saliency detection. In CVPR (pp. 2376–2383).

  • Hoiem, D., Efros, A. A., & Hebert, M. (2005). Geometric context from a single image. In ICCV (pp. 654–661).

  • Itti, L. (2004). Automatic foveation for video compression using a neurobiological model of visual attention. IEEE TIP.

  • Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE TPAMI.

  • Jia, Y., & Han, M. (2013). Category-independent object-level saliency detection. In ICCV.

  • Jiang, B., Zhang, L., Lu, H., Yang, C., & Yang, M.-H. (2013). Saliency detection via absorbing markov chain. In ICCV.

  • Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N., & Li, S. (2011). Automatic salient object segmentation based on context and shape prior. In BMVC.

  • Jiang, H., Wang, J., Yuan, Z., Wu, Y., Zheng, N., & Li, S. (2013). Salient object detection: A discriminative regional feature integration approach. In IEEE CVPR (pp. 2083–2090).

  • Jiang, P., Ling, H., Jingyi, Y. & Peng, J. (2013). Salient region detection by UFO: Uniqueness, focusness and objectness. In ICCV.

  • Jiang, Z., & Davis, L. S. (2013). Submodular salient region detection. In CVPR (pp. 2043–2050).

  • Kanan, C., & Cottrell, G. W. (2010). Robust classification of objects, faces, and flowers using natural image statistics. In CVPR (pp. 2472–2479).

  • Khuwuthyakorn, P., Robles-Kelly, A., & Zhou, J. (2010). Object of interest detection by saliency learning. In ECCV.

  • Kim, J., Han, D., Tai, Y.-W., Kim, J. (2014). Salient region detection via high-dimensional color transform. In CVPR.

  • Kimchi, R., & Peterson, M. A. (2008). Figure-ground segmentation can occur without attention. Psychological Science, 19(7), 660–668.

  • Klein, D. A., & Frintrop, S. (2011). Center-surround divergence of feature statistics for salient object detection. In ICCV.

  • Koffka, K. (1935). Principles of Gestalt Psychology. Brace: Harcourt.

    Google Scholar 

  • Küttel, D., & Ferrari, V. (2012). Figure-ground segmentation by transferring window masks. In CVPR (pp. 558–565).

  • Li, G., & Yu, Y. (2015). Visual saliency based on multiscale deep features. In CVPR (pp. 5455–5463).

  • Li, G., & Yizhou, Y. (2016). Deep contrast learning for salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Li, N., Ye, J., Ji, Y., Ling, H., & Yu, J. (2014). Saliency detection on light fields. In CVPR.

  • Li, X., Li, Y., Shen, C., Dick, A. R., & van den Hengel, A. (2013). Contextual hypergraph modeling for salient object detection. In ICCV (pp. 3328–3335).

  • Li, X., Zhao, L., Wei, L., Yang, M., Wu, F., Zhuang, Y., Ling, H., & Wang, J. (2015). Deepsaliency: Multi-task deep neural network model for salient object detection. CoRR, arXiv:1510.05484.

  • Li, X., Lu, H., Zhang, L., Ruan, X., & Yang, M.-H. (2013). Saliency detection via dense and sparse reconstruction. In ICCV.

  • Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014). The secrets of salient object segmentation. In CVPR.

  • Liu, F., & Gleicher, M. (2006). Region enhanced scale-invariant saliency detection. In ICME (pp. 1477–1480).

  • Liu, N., Han, J., Zhang, D., Wen, S., & Liu, T. (2015). Predicting eye fixations using convolutional neural networks. In CVPR (pp. 362–370).

  • Liu, R., Cao, J., Zhong, G., Lin, Z., Shan, S., & Su, Z. (2014). Adaptive partial differential equation learning for visual saliency detection. In CVPR.

  • Liu, T., Sun, J., Zheng, N.-N., Tang, X., & Shum, H.-Y. (2007). Learning to detect a salient object. CVPR (pp. 1–8).

  • Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., et al. (2011). Learning to detect a salient object. IEEE TPAMI, 33(2), 353–367.

    Article  Google Scholar 

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In CVPR (pp. 3431–3440).

  • Lu, S., Mahadevan, V., & Vasconcelos, N. (2014). Learning optimal seeds for diffusion-based salient object detection. In CVPR.

  • Lu, Y., Zhang, W., Jin, C., & Xue, X. (2012). Learning attention map from images. In CVPR (pp. 1067–1074).

  • Lu, Y., Zhang, W., Lu, H., & Xue, X. (2011) Salient object detection using concavity context. In ICCV (pp. 233–240).

  • Ma, Y.-F., & Zhang, H.-J. (2003). Contrast-based image attention analysis by using fuzzy growing. In ACM Multimedia.

  • Marchesotti, L., Cifarelli, C., & Csurka, G. (2009). A framework for visual saliency detection with applications to image thumbnailing. In ICCV (pp. 2232–2239).

  • Margolin, R., Tal, A., & Zelnik-Manor, L. (2013). What makes a patch distinct? In CVPR.

  • Mehrani, P., & Veksler, O. (2010). Saliency segmentation based on learning and graph cut refinement. In BMVC.

  • Moosmann, F., Larlus, D., & Jurie, F. (2006). Learning saliency maps for object categorization. In EECVW.

  • Niu, Y., Geng, Y., Li, X., & Liu, F. (2012) Leveraging stereopsis for saliency analysis. In CVPR (pp. 454–461).

  • Peng, H., Li, B., Ji, R., Hu, W., Xiong, W., & Lang, C. (2013). Salient object detection via low-rank and structured sparse matrix decomposition. In AAAI.

  • Perazzi, F., Krähenbühl, P., Pritch, Y., & Hornung, A. (2012). Saliency filters: Contrast based filtering for salient region detection. In CVPR (pp. 733–740).

  • Rahtu, E., Kannala, J., Salo, M., & Heikkilä, J. (2010). Segmenting salient objects from images and videos. ECCV, 5, 366–379.

    Google Scholar 

  • Ren, X., Fowlkes, C. C., & Malik, J. (2006). Figure/ground assignment in natural images. ECCV, Part II (pp. 614–627).

  • Rubin, E. (1958). Figure and ground. In Readings in Perception (pp. 194–203). Princeton, NJ: Van Nostrand.

  • Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173.

    Article  Google Scholar 

  • Scharfenberger, C., Wong, A., Fergani, K., Zelek, J. S., & Clausi, D. A. (2013). Statistical textural distinctiveness for salient region detection in natural images. In CVPR (pp. 979–986).

  • Schroff, F., Criminisi, A., & Zisserman, A. (2008). Object class segmentation using random forests. In BMVC (pp. 1–10).

  • Shen, X., & Wu, Y. (2012). A unified approach to salient object detection via low rank matrix recovery. In CVPR.

  • Shi, K., Wang, K., Lu, J., & Lin, L. (2013). Pisa: Pixelwise image saliency by aggregating complementary appearance contrast measures with spatial priors. In CVPR (pp. 2115–2122).

  • Treisman, A., & Gelad, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.

    Article  Google Scholar 

  • Vicente, S., Kolmogorov, V., & Rother, C. (2008). Graph cut based image segmentation with connectivity priors. In CVPR.

  • Walther, D., & Koch, C. (2006). Modeling attention to salient proto-objects. Neural Networks, 19(9), 1395–1407.

    Article  MATH  Google Scholar 

  • Wang, J., Quan, L., Sun, J., Tang, X., & Shum, H.-Y. (2006). Picture collage. CVPR, 1, 347–354.

    Google Scholar 

  • Wang, L., Xue, J., Zheng, N., & Hua, G. (2011). Automatic salient object extraction with contextual cue. In ICCV.

  • Wang, M., Konrad, J., Ishwar, P., Jing, K., & Rowley, H. A. (2011). Image saliency: From intrinsic to extrinsic context. In CVPR (pp. 417–424).

  • Wang, P., Wang, J., Zeng, G., Feng, J., Zha, H., & Li, S. (2012). Salient object detection for searched web images via global saliency. In CVPR (pp. 3194–3201).

  • Wang, P., Zhang, D., Wang, J., Wu, Z., Hua, X.-S. & Li, S. (2012). Color filter for image search. In ACM Multimedia.

  • Wang, P., Zhang, D., Zeng, G., & Wang, J. (2012). Contextual dominant color name extraction for web image search. In ICME Workshops (pp. 319–324).

  • Wei, Y., Wen, F., Zhu, W., & Sun, J. (2012). Geodesic saliency using background priors. ECCV, 3, 29–42.

    Google Scholar 

  • Xie, Y., Huchuan, L., & Yang, M.-H. (2013). Bayesian saliency via low and mid level cues. IEEE TIP, 22(5), 1689–1698.

    MathSciNet  Google Scholar 

  • Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical saliency detection. In CVPR (pp. 1155–1162).

  • Yang, C., Zhang, L., Lu, H., Ruan, X., & Yang, M.-H. (2013). Saliency detection via graph-based manifold ranking. In CVPR.

  • Yu, S. X., & Shi, J. (2003). Object-specific figure-ground segregation. In CVPR (pp. 39–45).

  • Zhang, J., & Sclaroff, S. (2013). Saliency detection: A boolean map approach. In ICCV (pp. 153–160).

  • Zhu, W., Liang, S., Wei, Y., & Sun, J. (2014). Saliency optimization from robust background detection. In CVPR.

  • Zou, W., Kpalma, K., Liu, Z., Ronsin, J., et al. (2013). Segmentation driven low-rank matrix recovery for saliency detection. In BMVC (pp. 1–13).

Download references

Acknowledgements

This work was supported in part by the National Basic Research Program of China under Grant No. 2015CB351703 and 2012CB316400, and the National Natural Science Foundation of China under Grant No. 91120006 and NSFC (No. 61572264), Huawei Innovation Research Program (HIRP), and CAST young talents plan.

Author information

Authors and Affiliations

  1. Microsoft Research, Beijing, People’s Republic of China

    Jingdong Wang

  2. University of Massachusetts, Amherst, MA, USA

    Huaizu Jiang

  3. Xi’an Jiaotong University, Xi’an, Shaanxi, People’s Republic of China

    Zejian Yuan & Nanning Zheng

  4. CCCE & CS, Nankai University, Tianjin, People’s Republic of China

    Ming-Ming Cheng & Xiaowei Hu

Authors
  1. Jingdong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huaizu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zejian Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming-Ming Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaowei Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nanning Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingdong Wang.

Additional information

Communicated by Cordelia Schmid.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Jiang, H., Yuan, Z. et al. Salient Object Detection: A Discriminative Regional Feature Integration Approach. Int J Comput Vis 123, 251–268 (2017). https://doi.org/10.1007/s11263-016-0977-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-016-0977-3

Keywords

Search

Navigation