Residual feature pyramid networks for salient object detection


Existing fully convolutional networks-based salient object detection (SOD) methods are still struggling to detect salient objects of an image in challenging cases due to the incompetent convolutional features, such as complex background, low contrast, multi-tiny objects. To address it, we propose novel residual feature pyramid networks with richer convolutional features for accurate SOD. Specially, we first introduce richer convolutional features to fully exploit multi-scale and multi-level information of objects, which makes it more discriminative for challenging cases. Secondly, based on the powerful stage-wise convolutional features, we further propose residual feature pyramid networks by focusing on the non-predicted regions to learn residual details more effectively and efficiently, which resulted in high-resolution prediction. Experimental results on five standard datasets demonstrate that our model outperforms 17 recent state-of-the-art methods.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    Wei, Y., Liang, X., Chen, Y., Shen, X., Cheng, M.M., Feng, J., Zhao, Y., Yan, S.: Stc: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2314–2320 (2017)

    Article  Google Scholar 

  2. 2.

    Choi, J., Chang, H.J., Fischer, T., Yun, S., Lee, K., Jeong, J., Demiris, Y., Choi, J.Y.: Context-aware deep feature compression for high-speed visual tracking. In: CVPR, pp. 479–488 (2018)

  3. 3.

    Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.-Y.: Learning to detect a salient object. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)

    Article  Google Scholar 

  4. 4.

    Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.-H.: Saliency detection via graph-based manifold ranking. In: CVPR (2013)

  5. 5.

    Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background detection. In: CVPR (2014)

  6. 6.

    Cheng, M.-M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.-M.: Global proceedings of the twenty-seventh international joint conference on artificial intelligence (IJCAI-18)689 contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Article  Google Scholar 

  7. 7.

    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015)

  8. 8.

    Dai, J., He, K., Li, Y., Ren, S., Sun, J.: Instance-sensitive fully convolutional networks. In: ECCV, pp. 534–549 (2016)

  9. 9.

    Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV, pp. 1395–1403 (2015)

  10. 10.

    Spurr, A., Song, J., Park, S., Hilliges, O.: Cross-modal deep variational hand pose estimation. In: CVPR, pp. 89–98 (2018)

  11. 11.

    Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: CVPR, pp. 1265–1274 (2015)

  12. 12.

    Wang, T., Borji, A., Zhang, L., Zhang, P., Lu, H.: A stagewise refinement model for detecting salient objects in images. In: CVPR, pp. 4019–4028 (2017)

  13. 13.

    Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: ICCV, pp. 212–221 (2017)

  14. 14.

    Li, G., and Yu, Y.: Deep contrast learning for salient object detection. In: CVPR, pp. 478–487 (2016)

  15. 15.

    Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., Torr, P.: Deeply supervised salient object detection with short connections. In: CVPR, pp. 5300–5309 (2017)

  16. 16.

    Ke, W., Chen, J., Jiao, J., Zhao, G., Ye, Q.: Srn: Side-output residual network for object symmetry detection in the wild. In: CVPR, pp. 302–310 (2017)

  17. 17.

    Hu, P., Shuai, B., Liu, J., Wang, G.: Deep level sets for salient object detection. In: CVPR, pp. 2300–2309 (2017)

  18. 18.

    Liu, Z., Tang, J., Zhao, P.: Salient object detection via hybrid up sampling and hybrid loss computing. Vis. Comput. (2019).

    Article  Google Scholar 

  19. 19.

    Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: CVPR, pp. 1741–1750 (2018)

  20. 20.

    Zhang, X., Wang, T., Qi, J., Lu, H., Wang, G.: Progressive attention guided recurrent network for salient object detection. In: CVPR, pp. 714–722 (2018)

  21. 21.

    Wang, T., Zhang, L., Wang, S., Huchuan, L., Yang, G., Ruan, X., Borji, A.: Detect globally. A novel approach to saliency detection. In: CVPR, Refine Locally, pp. 3127–3135 (2018)

  22. 22.

    Wang, W., Shen, J., Shao, L., Porikli, F.: Correspondence driven saliency transfer. IEEE TIP 25(11), 5025–5034 (2016)

    MathSciNet  MATH  Google Scholar 

  23. 23.

    Wenguan, W., Shen, J.: Deep visual attention prediction. IEEE TIP 27(5), 2368–2378 (2018)

    MathSciNet  Google Scholar 

  24. 24.

    Wang, W., Shen, J., Cheng, M.-M, Borji, A.: Revisiting video saliency: a large-scale benchmark and a new model. In: CVPR (2018)

  25. 25.

    Wang, W., Shen, J., Yizhou, Y., Ma, K.-L.: Stereoscopic thumbnail creation via efficient stereo saliency detection. IEEE TVCG 23(8), 2014–2027 (2017)

    Google Scholar 

  26. 26.

    Wang, W., Shen, J., Dong, X., Borji, A.: Salient object detection driven by fixation prediction. In: CVPR (2018)

  27. 27.

    Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: \(R^{3}\)Net: recurrent residual refinement network for saliency detection. In: IJCAI, pp. 684–690

  28. 28.

    Zhu, L., Hu, X., Fu, C.-W., Qin, J., Heng, P.-A..: Saliency-aware texture smoothing. In: IEEE TVCG

  29. 29.

    Li, X., Yang, F., Cheng, H., Liu, W., Shen, D.: Contour knowledge transfer for salient object detection. In: ECCV (2018) 355-370

  30. 30.

    Wu, R., Feng, M., Guan, W., Wang, D., Lu, H., Ding, E.: A mutual learning method for salient object detection with intertwined multi-supervision. In: CVPR, pp. 8150–8159 (2019)

  31. 31.

    Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M. Boundary-aware salient object detection. In: CVPR, BASNet, pp. 7479–7489 (2019)

  32. 32.

    Liu, J.-J., Hou, Q., Cheng, M.-M., Feng, J., Jiang, J.: A simple pooling-based design for real-time salient object detection. In: CVPR, pp. 3917–3926 (2019)

  33. 33.

    Hu, X., Fu, C.-W., Zhu, L., Heng, P.-A.: SAC-Net: spatial attenuation context for salient object detection

  34. 34.

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

  35. 35.

    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp 1646–1654 (2016)

  36. 36.

    Wang, W., Shen, J., Lin, H.: A deep network solution for attention and aesthetics aware photo cropping. IEEE Trans. Image Process. 41(7), 1531–1544 (2019)

    Google Scholar 

  37. 37.

    Wang, W., Shen, J., Porikli, F.: Saliency-aware geodesic video object segmentation. In: CVPR, pp. 3395–3402 (2015)

  38. 38.

    Shen, J., Peng, J., Shao, L.: Submodular trajectories for better motion segmentation in videos. IEEE Trans. Image Process. 27(6), 2688–2700 (2018)

    MathSciNet  Article  Google Scholar 

  39. 39.

    Wang, W., Shen, J., Shao, L.: Video salient object detection via fully convolutional networks. IEEE Trans. Image Process. 27(1), 38–49 (2018)

    MathSciNet  Article  Google Scholar 

  40. 40.

    Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection (2017)

  41. 41.

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ArXiv e-prints (2014)

  42. 42.

    Liu, Y., Cheng, M.-M. , Hu, Xiaowei, Wang, K, Bai, X.: Richer convolutional features for edge detection. In: CVPR, pp. 3000–3009 (2017)

  43. 43.

    Chen, S., Tan, X., Wang, B., Hu, X.: Reverse attention for salient object detection. In: ECCV, pp. 234–250 (2018)

  44. 44.

    Li, G., Yu, Y.: Visual saliency detection based on multiscale deep CNN features. IEEE Trans. Image Proc. 25(11), 5012–5024 (2016)

    MathSciNet  Article  Google Scholar 

  45. 45.

    Shi, J., Yan, Q., Xu, L., Jia, J.: Hierarchical image saliency detection on extended CSSD. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 717–729 (2016)

    Article  Google Scholar 

  46. 46.

    Li, Y., Hou, X., Koch, C., Rehg, J.M., Yuille, A.L.: The secrets of salient object segmentation. In: CVPR, pp. 280–287 (2014)

  47. 47.

    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, pp. 416–423 (2001)

  48. 48.

    Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.: Saliency detection via graphbased manifold ranking. In: CVPR, pp. 3166–3173 (2013)

  49. 49.

    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: ACM multimedia, pp. 675–678 (2014)

  50. 50.

    Liu, N., Han, J.: Dhsnet: deep hierarchical saliency network for salient object detection. In: CVPR, pp. 678–686 (2016)

  51. 51.

    Kim, J., Pavlovic, V.: A shape-based approach for salient object detection using deep learning. In: ECCV, pp. 455–470 (2016)

  52. 52.

    Wang, L., Wang, L., Lu, H., Zhang, P., Ruan, X.: Saliency detection with recurrent fully convolutional networks. In: ECCV, pp. 825–841 (2016)

  53. 53.

    Luo, Z., Mishra, A., Achkar, A., Eichel, J., Li, S., Jodoin, P.M.: Non-local deep features for salient object detection. In: CVPR, pp. 6593–660 (2017)

  54. 54.

    Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multi-level convolutional features for salient object detection. In: ICCV, pp. 202–211 (2017)

  55. 55.

    Hu, X., Zhu, L., Qin, J., Fu, C.-W., Heng, P.-A.: Recurrently aggregating deep features for salient object detection. In: AAAI, pp. 6943–6950 (2018)

  56. 56.

    Liu, N., Han, J., Yang, M.-H.: Learning pixel-wise contextual attention for saliency detection. In CVPR, PiCANet, pp. 3089–3098 (2018)

  57. 57.

    Liu, Y., Cheng, M.-M., Zhang, X.-Y., Fan, D.-P., Nie, G.-Y., Wang, M.: Deeply-supervised nonlinear aggregation for salient object detection. arXiv preprint DNA (2019)

  58. 58.

    Jiang, H.,Wang, J., Yuan, Z.,Wu, Y., Zheng, N., Li, S.: Salient object detection: a discriminative regional feature integration approach. In: CVPR, pp. 2083–2090 (2013)

  59. 59.

    Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.Y.: Learning to detect a salient object. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 353–367 (2011)

    Article  Google Scholar 

  60. 60.

    Borji, A., Cheng, M.-M., Jiang, H., Li, J.: Salient object detection: a benchmark. IEEE TIP 24(12), 5706–5722 (2015)

    MathSciNet  MATH  Google Scholar 

Download references


This work was supported by the Natural Science Foundation of China (No. 61802336), Jiangsu Province 7th Projects for Summit Talents in Six Main Industries, Electronic Information Industry (DZXX-149, No. 110), Foundation of Yangzhou University (No. 2017CXJ026).

Author information



Corresponding author

Correspondence to Shuhan Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, B., Chen, S., Wang, J. et al. Residual feature pyramid networks for salient object detection. Vis Comput 36, 1897–1908 (2020).

Download citation


  • Saliency detection
  • Richer convolutional features
  • Residual feature pyramid networks