Skip to main content

Light field salient object detection: A review and benchmark

Abstract

Salient object detection (SOD) is a long-standing research topic in computer vision with increasing interest in the past decade. Since light fields record comprehensive information of natural scenes that benefit SOD in a number of ways, using light field inputs to improve saliency detection over conventional RGB inputs is an emerging trend. This paper provides the first comprehensive review and a benchmark for light field SOD, which has long been lacking in the saliency community. Firstly, we introduce light fields, including theory and data forms, and then review existing studies on light field SOD, covering ten traditional models, seven deep learning-based models, a comparative study, and a brief review. Existing datasets for light field SOD are also summarized. Secondly, we benchmark nine representative light field SOD models together with several cutting-edge RGB-D SOD models on four widely used light field datasets, providing insightful discussions and analyses, including a comparison between light field SOD and RGB-D SOD models. Due to the inconsistency of current datasets, we further generate complete data and supplement focal stacks, depth maps, and multi-view images for them, making them consistent and uniform. Our supplemental data make a universal benchmark possible. Lastly, light field SOD is a specialised problem, because of its diverse data representations and high dependency on acquisition hardware, so it differs greatly from other saliency detection tasks. We provide nine observations on challenges and future directions, and outline several open issues. All the materials including models, datasets, benchmarking results, and supplemented light field datasets are publicly available at https://github.com/kerenfu/LFSOD-Survey.

References

  1. Wang, S. Z.; Liao, W. J.; Surman, P.; Tu, Z. G.; Zheng, Y. J.; Yuan, J. S. Salience guided depth calibration for perceptually optimized compressive light field 3D display. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2031–2040, 2018.

  2. Cheng, M. M.; Zhang, G. X.; Mitra, N. J.; Huang, X. L.; Hu, S. M. Global contrast based salient region detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 409–416, 2011.

  3. Borji, A.; Cheng, M. M.; Jiang, H. Z.; Li, J. Salient object detection: A benchmark. IEEE Transactions on Image Processing Vol. 24, No. 12, 5706–5722, 2015.

    MathSciNet  MATH  Article  Google Scholar 

  4. Borji, A.; Cheng, M. M.; Hou, Q. B.; Jiang, H. Z.; Li, J. Salient object detection: A survey. Computational Visual Media Vol. 5, No. 2, 117–150, 2019.

    Article  Google Scholar 

  5. Li, N. Y.; Ye, J. W.; Ji, Y.; Ling, H. B.; Yu, J. Y. Saliency detection on light field. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2806–2813, 2014.

  6. Li, N. Y.; Ye, J. W.; Ji, Y.; Ling, H. B.; Yu, J. Y. Saliency detection on light field. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 8, 1605–1616, 2017.

    Article  Google Scholar 

  7. Ren, Z. X.; Gao, S. H.; Chia, L. T.; Tsang, I. W. H. Region-based saliency detection and its application in object recognition. IEEE Transactions on Circuits and Systems for Video Technology Vol. 24, No. 5, 769–779, 2014.

    Article  Google Scholar 

  8. Zhang, D.; Meng, D.; Zhao, L.; Han, J. Bridging saliency detection to weakly supervised object detection based on self-paced curriculum learning. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence, 3538–3544, 2016.

  9. Rutishauser, U.; Walther, D.; Koch, C.; Perona, P. Is bottom-up attention useful for object recognition? In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, II, 2004.

  10. Moosmann, F.; Larlus, D.; Jurie, F. Learning saliency maps for object categorization. In: Proceedings of the ECCV’06 Workshop on the Representation and Use of Prior Knowledge in Vision, 2006.

  11. Cheng, M. M.; Liu, Y.; Lin, W. Y.; Zhang, Z. M.; Rosin, P. L.; Torr, P. H. S. BING: Binarized normed gradients for objectness estimation at 300fps. Computational Visual Media Vol. 5, No. 1, 3–20, 2019.

    Article  Google Scholar 

  12. Wei, Y. C.; Feng, J. S.; Liang, X. D.; Cheng, M. M.; Zhao, Y.; Yan, S. C. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6488–6496, 2017.

  13. Wei, Y. C.; Liang, X. D.; Chen, Y. P.; Shen, X. H.; Cheng, M. M.; Feng, J. S.; Zhao, Y.; Yan, S. STC: A simple to complex framework for weakly-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 11, 2314–2320, 2017.

    Article  Google Scholar 

  14. Wang, X.; You, S. D.; Li, X.; Ma, H. M. Weakly-supervised semantic segmentation by iteratively mining common object features. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1354–1362, 2018.

  15. Wang, W.; Shen, J.; Yang, R.; Porikli, F. Saliency-aware video object segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 40, No. 1, 20–33, 2018.

    Article  Google Scholar 

  16. Song, H.; Wang, W.; Zhao, S.; Shen, J.; Lam, K.-M. Pyramid dilated deeper ConvLSTM for video salient object detection. In: Computer Vision — ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, Vol. 11215. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 744–760, 2018.

    Chapter  Google Scholar 

  17. Itti, L. Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Transactions on Image Processing Vol. 13, No. 10, 1304–1318, 2004.

    Article  Google Scholar 

  18. Ma, Y. F.; Hua, X. S.; Lu, L.; Zhang, H. J. A generic framework of user attention model and its application in video summarization. IEEE Transactions on Multimedia Vol. 7, No. 5, 907–919, 2005.

    Article  Google Scholar 

  19. Ma, Y. F.; Lu, L.; Zhang, H. J.; Li, M. J. A user attention model for video summarization. In: Proceedings of the 10th ACM International Conference on Multimedia, 533–542, 2002.

  20. Ouerhani, N.; Bracamonte, J.; Hugli, H.; Ansorge, M.; Pellandini, F. Adaptive color image compression based on visual attention. In: Proceedings of the 11th International Conference on Image Analysis and Processing, 416–421, 2001.

  21. Han, J. G.; Pauwels, E. J.; de Zeeuw, P. Fast saliency-aware multi-modality image fusion. Neurocomputing Vol. 111, 70–80, 2013.

    Article  Google Scholar 

  22. Jin, S.; Ling, H. B. Scale and object aware image retargeting for thumbnail browsing. In: Proceedings of the International Conference on Computer Vision, 1511–1518, 2011.

  23. Sugano, Y.; Matsushita, Y.; Sato, Y. Calibration-free gaze sensing using saliency maps. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2667–2674, 2010.

  24. Borji, A.; Itti, L. Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision Vol. 14, No. 3, 29, 2014.

    Article  Google Scholar 

  25. Fu, K. R.; Zhao, Q. J.; Yu-Hua Gu, I.; Yang, J. Deepside: A general deep framework for salient object detection. Neurocomputing Vol. 356, 69–82, 2019.

    Article  Google Scholar 

  26. Wang, W. G.; Shen, J. B.; Shao, L.; Porikli, F. Correspondence driven saliency transfer. IEEE Transactions on Image Processing Vol. 25, No. 11, 5025–5034, 2016.

    MathSciNet  MATH  Article  Google Scholar 

  27. Zhang, P. P.; Wang, D.; Lu, H. C.; Wang, H. Y.; Ruan, X. Amulet: Aggregating multi-level convolutional features for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision, 202–211, 2017.

  28. Feng, M. Y.; Lu, H. C.; Ding, E. R. Attentive feedback network for boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1623–1632, 2019.

  29. Zhang, P. P.; Wang, D.; Lu, H. C.; Wang, H. Y.; Yin, B. C. Learning uncertain convolutional features for accurate saliency detection. In: Proceedings of the IEEE International Conference on Computer Vision, 212–221, 2017.

  30. Zhang, J.; Wang, M.; Gao, J.; Wang, Y.; Zhang, X.; Wu, X. Saliency detection with a deeper investigation of light field. In: Proceedings of the 24th International Conference on Artificial Intelligence, 2212–2218, 2015.

  31. Zhang, M.; Li, J.; Ji, W.; Piao, Y.; Lu, H. Memory-oriented decoder for light field salient object detection. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 898–908, 2019.

  32. Piao, Y. R.; Rong, Z. K.; Zhang, M.; Lu, H. C. Exploit and replace: An asymmetrical two-stream architecture for versatile light field saliency detection. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 11865–11873, 2020.

    Article  Google Scholar 

  33. Fu, K. R.; Fan, D. P.; Ji, G. P.; Zhao, Q. J. JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3049–3059, 2020.

  34. Fu, K. R.; Fan, D. P.; Ji, G. P.; Zhao, Q. J.; Shen, J. B.; Zhu, C. Siamese network for RGB-D salient object detection and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence doi:https://doi.org/10.1109/TPAMI.2021.3073689, 2021.

  35. Fan, D. P.; Zhai, Y.; Borji, A.; Yang, J.; Shao, L. BBS-Net: RGB-D salient object detection with a bifurcated backbone strategy network. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12357. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 275–292, 2020.

    Chapter  Google Scholar 

  36. Zhang, M.; Fei, S. X.; Liu, J.; Xu, S.; Piao, Y. R.; Lu, H. C. Asymmetric two-stream architecture for accurate RGB-D saliency detection. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12373. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 374–390, 2020.

    Chapter  Google Scholar 

  37. Gershun, A. The light field. Studies in Applied Mathematics Vol. 18, Nos. 1–4, 51–151, 1939.

    MATH  Google Scholar 

  38. Jeon, H. G.; Park, J.; Choe, G.; Park, J.; Bok, Y.; Tai, Y. W.; Kweon, I.-S. Accurate depth map estimation from a lenslet light field camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1547–1555, 2015.

  39. Tao, M. W.; Hadap, S.; Malik, J.; Ramamoorthi, R. Depth from combining defocus and correspondence using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision, 673–680, 2013.

  40. Tao, M. W.; Srinivasan, P. P.; Malik, J.; Rusinkiewicz, S.; Ramamoorthi, R. Depth from shading, defocus, and correspondence using light-field angular coherence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1940–1948, 2015.

  41. Wang, T. C.; Efros, A. A.; Ramamoorthi, R. Occlusion-aware depth estimation using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision, 3487–3495, 2015.

  42. Ng, R.; Levoy, M.; Brédif, M.; Duval, G.; Horowitz, M.; Hanrahan, P. Light field photography with a handheld plenopic camera. Stanford Tech Report CTSR 2005-02, 2005.

  43. Piao, Y.; Zhang, Y.; Zhang, M.; Ji, X. Dynamic fusion network for light field depth estimation. arXiv preprint arXiv:2104.05969, 2021.

  44. Piao, Y.; Ji, X.; Zhang, M.; Zhang, Y. Learning multi-modal information for robust light field depth estimation. arXiv preprint arXiv:2104.05971, 2021.

  45. Piao, Y.; Rong, Z.; Zhang, M.; Li, X.; Lu, H. Deep light-field-driven saliency detection from a single view. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, 904–911, 2019.

  46. Jiang, P.; Ling, H. B.; Yu, J. Y.; Peng, J. L. Salient region detection by UFO: Uniqueness, focusness and objectness. In: Proceedings of the IEEE International Conference on Computer Vision, 1976–1983, 2013.

  47. Buehler, C.; Bosse, M.; McMillan, L.; Gortler, S.; Cohen, M. Unstructured lumigraph rendering. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 425–432, 2001.

  48. Zhang, X. D.; Wang, Y.; Zhang, J.; Hu, L. M.; Wang, M. Light field saliency vs. 2D saliency: A comparative study. Neurocomputing Vol. 166, 389–396, 2015.

    Article  Google Scholar 

  49. Zhou, T.; Fan, D. P.; Cheng, M. M.; Shen, J. B.; Shao, L. RGB-D salient object detection: A survey. Computational Visual Media Vol. 7, No. 1, 37–69, 2021.

    Article  Google Scholar 

  50. Li, N. Y.; Sun, B. L.; Yu, J. Y. A weighted sparse coding framework for saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5216–5223, 2015.

  51. Sheng, H.; Zhang, S.; Liu, X. Y.; Xiong, Z. Relative location for light field saliency detection. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 1631–1635, 2016.

  52. Wang, A. Z.; Wang, M. H.; Li, X. Y.; Mi, Z. T.; Zhou, H. A two-stage Bayesian integration framework for salient object detection on light field. Neural Processing Letters Vol. 46, No. 3, 1083–1094, 2017.

    Article  Google Scholar 

  53. Zhang, J.; Wang, M.; Lin, L.; Yang, X.; Gao, J.; Rui, Y. Saliency detection on light field. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 13, No. 3, Article No. 32, 2017.

  54. Wang, X.; Dong, Y. Y.; Zhang, Q.; Wang, Q. Region-based depth feature descriptor for saliency detection on light field. Multimedia Tools and Applications Vol. 80, No. 11, 16329–16346, 2021.

    Article  Google Scholar 

  55. Piao, Y. R.; Li, X.; Zhang, M.; Yu, J. Y.; Lu, H. C. Saliency detection via depth-induced cellular automata on light field. IEEE Transactions on Image Processing Vol. 29, 1879–1889, 2020.

    MathSciNet  Article  Google Scholar 

  56. Wang, H. Q.; Yan, B.; Wang, X. Z.; Zhang, Y. B.; Yang, Y. Accurate saliency detection based on depth feature of 3D images. Multimedia Tools and Applications Vol. 77, No. 12, 14655–14672, 2018.

    Article  Google Scholar 

  57. Wang, T. T.; Piao, Y. R.; Lu, H. C.; Li, X.; Zhang, L. H. Deep learning for light field saliency detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 8837–8847, 2019.

  58. Zhang, M.; Ji, W.; Piao, Y. R.; Li, J. J.; Zhang, Y.; Xu, S.; Lu, H. LFNet: Light field fusion network for salient object detection. IEEE Transactions on Image Processing Vol. 29, 6276–6287, 2020.

    Article  Google Scholar 

  59. Zhang, J.; Liu, Y. M.; Zhang, S. P.; Poppe, R.; Wang, M. Light field saliency detection with deep convolutional networks. IEEE Transactions on Image Processing Vol. 29, 4421–4434, 2020.

    Article  Google Scholar 

  60. Zhang, Q. D.; Wang, S. Q.; Wang, X.; Sun, Z. H.; Kwong, S.; Jiang, J. M. A multi-task collaborative network for light field salient object detection. IEEE Transactions on Circuits and Systems for Video Technology Vol. 31, No. 5, 1849–1861, 2021.

    Article  Google Scholar 

  61. Zhang, M.; Ren, W. S.; Piao, Y. R.; Rong, Z. K.; Lu, H. C. Select, supplement and focus for RGB-D saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3469–3478, 2020.

  62. Zhang, J.; Fan, D. P.; Dai, Y. C.; Anwar, S.; Saleh, F. S.; Zhang, T.; Barnes, N. UC-net: Uncertainty inspired RGB-D saliency detection via conditional variational autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8579–8588, 2020.

  63. Fan, D. P.; Lin, Z.; Zhang, Z.; Zhu, M. L.; Cheng, M. M. Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks. IEEE Transactions on Neural Networks and Learning Systems Vol. 32, No. 5, 2075–2089, 2021.

    Article  Google Scholar 

  64. Liu, N.; Zhang, N.; Han, J. W. Learning selective self-mutual attention for RGB-D saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13753–13762, 2020.

  65. Li, C. Y.; Cong, R. M.; Piao, Y. R.; Xu, Q. Q.; Loy, C. C. RGB-D salient object detection with cross-modality modulation and selection. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12353. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 225–241, 2020.

    Chapter  Google Scholar 

  66. Pang, Y.; Zhang, L.; Zhao, X.; Lu, H. Hierarchical dynamic filtering network for RGB-D salient object detection. In: Computer Vision — ECCV 2020. Lecture Notes in Computer Science, Vol. 12370. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 235–252, 2020.

    Chapter  Google Scholar 

  67. Adelson, E.; Bergen, J. The Plenoptic function and the elements of early vision. In: Computational Models of Visual Processing. MIT Press, 3–20, 1991.

  68. Levoy, M.; Hanrahan, P. Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, 31–42, 1996.

  69. Agarwala, A.; Dontcheva, M.; Agrawala, M.; Drucker, S.; Colburn, A.; Curless, B.; Salesin, D.; Cohen, M. Interactive digital photomontage. ACM Transactions on Graphics Vol. 23, No. 3, 294–302, 2004.

    Article  Google Scholar 

  70. Kuthirummal, S.; Nagahara, H.; Zhou, C. Y.; Nayar, S. K. Flexible depth of field photography. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 1, 58–71, 2011.

    Article  Google Scholar 

  71. Zhu, W. J.; Liang, S.; Wei, Y. C.; Sun, J. Saliency optimization from robust background detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2814–2821, 2014.

  72. He, K. M.; Sun, J.; Tang, X. O. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 12, 2341–2353, 2011.

    Article  Google Scholar 

  73. Shelhamer, E.; Long, J.; Darrell, T. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 4, 640–651, 2017.

    Article  Google Scholar 

  74. Shi, X.; Chen, Z.; Wang, H.; Yeung, D.; Wong, W.; Woo, W. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 802–810, 2015.

  75. Chen, L. C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 40, No. 4, 834–848, 2018.

    Article  Google Scholar 

  76. Yang, C.; Zhang, L. H.; Lu, H. C.; Ruan, X.; Yang, M. H. Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3166–3173, 2013.

  77. Wang, L. J.; Lu, H. C.; Wang, Y. F.; Feng, M. Y.; Wang, D.; Yin, B. C.; Ruan, X. Learning to detect salient objects with image-level supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3796–3805, 2017.

  78. Achanta, R.; Hemami, S.; Estrada, F.; Susstrunk, S. Frequency-tuned salient region detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1597–1604, 2009.

  79. Perazzi, F.; Krähenbühl, P.; Pritch, Y.; Hornung, A. Saliency filters: Contrast based filtering for salient region detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 733–740, 2012.

  80. Fan, D. P.; Cheng, M. M.; Liu, Y.; Li, T.; Borji, A. Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE International Conference on Computer Vision, 4558–4567, 2017.

  81. Zhao, J. X.; Cao, Y.; Fan, D. P.; Cheng, M. M.; Li, X. Y.; Zhang, L. Contrast prior and fluid pyramid integration for RGBD salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3922–3931, 2019.

  82. Fan, D. P.; Gong, C.; Cao, Y.; Ren, B.; Cheng, M. M.; Borji, A. Enhanced-alignment measure for binary foreground map evaluation. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, 698–704, 2018.

  83. Wu, Z.; Su, L.; Huang, Q. M. Stacked cross refinement network for edge-aware salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7263–7272, 2019.

  84. Qin, X. B.; Zhang, Z. C.; Huang, C. Y.; Gao, C.; Dehghan, M.; Jagersand, M. BASNet: Boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7471–7481, 2019.

  85. Li, G. Y.; Liu, Z.; Ling, H. B. ICNet: Information conversion network for RGB-D based salient object detection. IEEE Transactions on Image Processing Vol. 29, 4873–4884, 2020.

    Article  Google Scholar 

  86. Tsiami, A.; Koutras, P.; Maragos, P. STAViS: Spatio-temporal AudioVisual saliency network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4765–4775, 2020.

  87. Fan, D. P.; Wang, W. G.; Cheng, M. M.; Shen, J. B. Shifting more attention to video salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8546–8556, 2019.

  88. Zhao, W. D.; Zhao, F.; Wang, D.; Lu, H. C. Defocus blur detection via multi-stream bottom-top-bottom network. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 42, No. 8, 1884–1897, 2020.

    Article  Google Scholar 

  89. Park, J.; Tai, Y. W.; Cho, D.; Kweon, I. S. A unified approach of multi-scale deep and hand-crafted features for defocus estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2760–2769, 2017.

  90. Zeng, Y.; Zhuge, Y. Z.; Lu, H. C.; Zhang, L. H.; Qian, M. Y.; Yu, Y. Z. Multi-source weak supervision for saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6067–6076, 2019.

  91. Qian, M. Y.; Qi, J. Q.; Zhang, L. H.; Feng, M. Y.; Lu, H. C. Language-aware weak supervision for salient object detection. Pattern Recognition Vol. 96, 106955, 2019.

    Article  Google Scholar 

  92. Chen, T. L.; Liu, S. J.; Chang, S. Y.; Cheng, Y.; Amini, L.; Wang, Z. Y. Adversarial robustness: From self-supervised pre-training to fine-tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 696–705, 2020.

  93. Dai, A.; Diller, C.; Niessner, M. SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 846–855, 2020.

  94. Zeng, Y.; Zhang, P. P.; Lin, Z.; Zhang, J. M.; Lu, H. C. Towards high-resolution salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7233–7242, 2019.

  95. Cai, Z. W.; Vasconcelos, N. Cascade R-CNN: High quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 5, 1483–1498, 2021.

    Article  Google Scholar 

  96. Chen, K.; Pang, J. M.; Wang, J. Q.; Xiong, Y.; Li, X. X.; Sun, S. Y.; Feng, W.; Liu, Z.; Shi, J.; Ouyang, W.; et al. Hybrid task cascade for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4969–4978, 2019.

  97. Liu, S.; Qi, L.; Qin, H. F.; Shi, J. P.; Jia, J. Y. Path aggregation network for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8759–8768, 2018.

  98. Li, G.; Xie, Y.; Lin, L.; Yu, Y. Instance-level salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 247–256, 2017.

  99. Fan, R. C.; Cheng, M. M.; Hou, Q. B.; Mu, T. J.; Wang, J. D.; Hu, S. M. S4Net: Single stage salient-instance segmentation. Computational Visual Media Vol. 6, No. 2, 191–204, 2020.

    Article  Google Scholar 

  100. Karpathy, A.; Li, F. F. Deep visual-semantic alignments for generating image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 4, 664–676, 2017.

    Article  Google Scholar 

  101. Wei, Y.; Xia, W.; Huang, J.; Ni, B.; Dong, J.; Zhao, Y.; Yan, S. CNN: Single-label to multi-label. arXiv preprint arXiv:1406.5726, 2014.

  102. Chen, X. L.; Gupta, A. Webly supervised learning of convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, 1431–1439, 2015.

  103. Lai, B. S.; Gong, X. J. Saliency guided dictionary learning for weakly-supervised image parsing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3630–3639, 2016.

  104. Tian, X.; Xu, K.; Yang, X.; Yin, B.; Lau, R. Weakly-supervised salient instance detection. In: Proceedings of the Conference on British Machine Vision Conference, 2020.

  105. Borji, A. Saliency prediction in the deep learning era: Successes and limitations. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 43, No. 2, 679–700, 2021.

    Article  Google Scholar 

  106. Borji, A.; Itti, L. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 35, No. 1, 185–207, 2013.

    Article  Google Scholar 

  107. Fan, D. P.; Ji, G. P.; Sun, G. L.; Cheng, M. M.; Shen, J. B.; Shao, L. Camouflaged object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2774–2784, 2020.

  108. Xu, Y. C.; Nagahara, H.; Shimada, A.; Taniguchi, R. I. TransCut: Transparent object segmentation from a light-field image. In: Proceedings of the IEEE International Conference on Computer Vision, 3442–3450, 2015.

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Nos. 62176169, 62172228, 61703077, 61773270), and SCU-Luzhou Municipal People’s Government Strategic Cooperation Project (No. 2020CDLZ-10).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tao Zhou.

Ethics declarations

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Keren Fu received his dual Ph.D. degrees from Shanghai Jiao Tong University, China, and Chalmers University of Technology, Sweden, under the joint supervision of Prof. Jie Yang and Prof. Irene Yu-Hua Gu. He is currently a research associate professor with the College of Computer Science, Sichuan University, China. His current research interests include visual computing, saliency analysis, and machine learning.

Yao Jiang is currently pursuing his master degree in the College of Computer Science, Sichuan University under the supervision of Dr. Keren Fu. His research interests include machine learning and computer vision.

Ge-Peng Ji received his master degree in communication and information systems from the School of Computer Science, Wuhan University, China. He is currently a research intern at the Inception Institute of Artificial Intelligence (IIAI), Abu Dhabi, United Arab Emirates. His research interests lie in designing deep neural networks and applying deep learning to various fields of low-level vision, such as camouflaged/salient object detection, video salient object detection, and medical image segmentation.

Tao Zhou received his Ph.D. degree in pattern recognition and intelligent systems from the Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, in 2016. From 2016 to 2018, he was a postdoctoral fellow in the BRIC and IDEA Lab, University of North Carolina at Chapel Hill. From 2018 to 2020, he was a research scientist at the Inception Institute of Artificial Intelligence (IIAI), Abu Dhabi. He is currently a professor in the School of Computer Science and Engineering, Nanjing University of Science and Technology, China. His research interests include machine learning, computer vision, and medical image analysis.

Qijun Zhao is currently a professor in the College of Computer Science at Sichuan University. He is also a visiting professor in the School of Information Science and Technology at Tibet University. He obtained his B.Sc. and M.Sc. degrees in computer science both from Shanghai Jiao Tong University, and his Ph.D. degree in computer science from Hong Kong Polytechnic University. He worked as a post-doctoral research fellow in the Pattern Recognition and Image Processing Lab at Michigan State University from 2010 to 2012. His research is in the fields of pattern recognition, image processing, and computer vision.

Deng-Ping Fan received his Ph.D. degree from Nankai University in 2019. He joined the Inception Institute of Artificial Intelligence (IIAI) in 2019. He has published over 30 top journal and conference papers. His research interests include computer vision, deep learning, and saliency detection, especially human vision for co-salient object detection, RGB salient object detection, RGB-D salient object detection, and video salient object detection.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fu, K., Jiang, Y., Ji, GP. et al. Light field salient object detection: A review and benchmark. Comp. Visual Media 8, 509–534 (2022). https://doi.org/10.1007/s41095-021-0256-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41095-021-0256-2

Keywords

  • light field
  • salient object detection (SOD)
  • deep learning
  • benchmarking