Advertisement

Multimedia Tools and Applications

, Volume 78, Issue 6, pp 6787–6804 | Cite as

RGB-D image saliency detection from 3D perspective

  • Zhengyi LiuEmail author
  • Tengfei Song
  • Feng Xie
Article
  • 143 Downloads

Abstract

With the advent of stereo camera saliency object detection for RGB-D image is attracting more and more interest. Most existing algorithms treat RGB-D image as one RGB image and one depth map, then measure saliency map independently, and last fuse them. They disregard the fact that human visual system operates in real 3D environments. The paper proposed saliency object detection for RGB-D image from 3D perspective. It regards object as three dimensional structures, and redefines boundary conception in RGB-D image, and regards space boundary including top, down, left, right, front, back plane in real 3D environment as background. It incorporates 3D compactness feature, in which salient objects typically have 3D compact spatial distributions, into color and depth feature to express similarity among supervoxels and applies manifold ranking by six boundary planes to generate six saliency maps, and then integrates them to get the RGB-D saliency map from background view. In the end it refines saliency map by high confident salient seeds from foreground view. Experiment results show that six planes of RGB-D image are superior to four sides of RGB image as background seeds and 3D compactness plays an important role in saliency measurement. Our approach outperforms other state-of-the-art algorithms on NLPR RGBD 1000 benchmark.

Keywords

RGB-D image saliency 3D boundary prior 3D compactness Manifold ranking 

Notes

Acknowledgements

We thank Prof. Jianguo Wu from Anhui University for helping with acquisition of funding. We also thank all anonymous reviewers for their valuable comments. This research is supported by National Key Technology Research and Development Program of the Ministry of Science and Technology of China (2015BAK24B01), Key Program of Natural Science Project of Educational Commission of Anhui Province, China (KJ2015A009), Open issues on Co-Innovation Center for Information Supply & Assurance Technology, Anhui University (ADXXBZ201610).

References

  1. 1.
    Achanta R, Shaji A, Smith K et al (2012) SLIC superpixels compared to state-of-the-art superpixel methods [J]. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282CrossRefGoogle Scholar
  2. 2.
    Ahmad J, Sajjad M, Mehmood I et al (2015) Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems[J]. J Real-Time Image Proc 13(3):431–447CrossRefGoogle Scholar
  3. 3.
    Chaib S, Gu Y, Yao H et al (2016) A VHR scene classification method integrating sparse PCA and saliency computing[C]. Geoscience and Remote Sensing Symposium (IGARSS). 2016 IEEE International, pp 2742–2745Google Scholar
  4. 4.
    Cheng Y, Fu H, Wei X et al (2014) Depth enhanced saliency detection method[C]. Proceedings of international conference on internet multimedia computing and service. ACM, pp 23Google Scholar
  5. 5.
    Desingh K, Madhava K K, Rajan D et al (2013) Depth really matters: improving visual salient region detection with depth[C]. British machine vision conference (BMVC), pp 98.1–98.11Google Scholar
  6. 6.
    Feng D, Barnes N, You S et al (2016) Local background enclosure for RGB-D salient object detection[C]. Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on. IEEE, pp 2343–2350Google Scholar
  7. 7.
    Guo J, Ren T, Bei J et al (2015) Salient object detection in RGB-D image based on saliency fusion and propagation[C]. Proceedings of the 7th international conference on internet multimedia computing and service. ACM, pp 59Google Scholar
  8. 8.
    Hong S, You T, Kwak S et al (2015) Online tracking by learning discriminative saliency map with convolutional neural network[C]. International Conference on Machine Learning (ICML), pp 597–606Google Scholar
  9. 9.
    Hussain CA, Rao DV, Masthani SA (2016) Robust pre-processing technique based on saliency detection for content based image retrieval systems[J]. Procedia Computer Science 85:571–580CrossRefGoogle Scholar
  10. 10.
    Hussain CA, Rao DV, Masthani SA (2016) Image retrieval using saliency content[C]. Electrical, electronics, and optimization techniques (ICEEOT), international conference on. IEEE, pp 840–844Google Scholar
  11. 11.
    Jiang L, Koch A, Zell A (2015) Salient regions detection for indoor robots using RGB-D data[C]. 2015 IEEE International conference on robotics and automation (ICRA). IEEE, pp 1323–1328Google Scholar
  12. 12.
    Ju R, Ge L, Geng W et al (2014) Depth saliency based on anisotropic center-surround difference[C]. 2014 IEEE international conference on image processing (ICIP). IEEE, pp 1115–1119Google Scholar
  13. 13.
    Ju R, Liu Y, Ren T et al (2015) Depth-aware salient object detection using anisotropic center-surround difference[J]. Signal Process Image Commun 38:115–126CrossRefGoogle Scholar
  14. 14.
    Khanna MT, Rai K, Chaudhury S et al (2015) Perceptual depth preserving saliency based image compression[C]. Proceedings of the 2nd international conference on perception and machine intelligence. ACM, pp 218–223Google Scholar
  15. 15.
    Krause EA, Zillich M, Williams TE et al (2014) Learning to recognize novel objects in one shot through human-robot interactions in natural language dialogues[C]. AAAI, pp 2796–2802Google Scholar
  16. 16.
    Lang C, Nguyen T V, Katti H et al (2012) Depth matters: influence of depth cues on visual saliency[M]. Computer Vision–ECCV 2012. Springer Berlin Heidelberg, pp 101–115CrossRefGoogle Scholar
  17. 17.
    Li Z, Liu G, Zhang D et al (2016) Robust single-object image segmentation based on salient transition region[J]. Pattern Recogn 52:317–331CrossRefGoogle Scholar
  18. 18.
    Liu GH, Yang JY, Li ZY (2015) Content-based image retrieval using computational visual attention model[J]. Pattern Recogn 48(8):2554–2566CrossRefGoogle Scholar
  19. 19.
    Margolin R, Zelnik-Manor L, Tal A (2014) How to evaluate foreground maps?[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 248–255Google Scholar
  20. 20.
    Oh SJ, Benenson R, Khoreva A et al (2017) Exploiting saliency for object segmentation from image level labels[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4410–4419Google Scholar
  21. 21.
    Otsu N (1979) A threshold selection method from gray-level histograms[J]. IEEE Trans Syst Man Cybern 9(1):62–66CrossRefGoogle Scholar
  22. 22.
    Papushoy A, Bors AG (2015) Image retrieval based on query by saliency content[J]. Digital Signal Processing 36:156–173MathSciNetCrossRefGoogle Scholar
  23. 23.
    Peng H, Li B, Xiong W et al (2014) Rgbd salient object detection: a benchmark and algorithms[C]. European conference on computer vision. Springer, Cham, pp 92–109Google Scholar
  24. 24.
    Qi M, Wang Y (2016) DEEP-CSSR: scene classification using category-specific salient region with deep features[C]. 2016 IEEE international conference on image processing (ICIP). IEEE, pp 1047-1051Google Scholar
  25. 25.
    Quo J, Ren T, Bei J (2016) Salient object detection for RGB-D image via saliency evolution[C]. 2016 IEEE international conference on multimedia and expo (ICME). IEEE, pp 1–6Google Scholar
  26. 26.
    Ren J, Gong X, Yu L et al (2015) Exploiting global priors for RGB-D saliency detection[C]. 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 25–32Google Scholar
  27. 27.
    Song H, Liu Z, Du H et al (2015) Saliency detection for RGBD images[C]. Proceedings of the 7th international conference on internet multimedia computing and service. ACM, pp 72Google Scholar
  28. 28.
    Srivastava S, Mukherjee P, Lall B (2016) Adaptive image compression using saliency and KAZE features[C]. International conference on signal processing and communications, pp:1–5Google Scholar
  29. 29.
    Sun J, Wu J, Yu H et al (2016) Boosting image retrieval framework with salient objects[C]. Audio, language and image processing (ICALIP), 2016 international conference on. IEEE, pp 241–245Google Scholar
  30. 30.
    Tamura Y, Akashi T, Yano S et al (2016) Human visual attention model based on analysis of magic for smooth human–robot interaction[J]. Int J Soc Robot 8(5):685–694CrossRefGoogle Scholar
  31. 31.
    Tang Y, Tong R, Tang M et al (2016) Depth incorporating with color improves salient object detection[J]. Vis Comput 32(1):111–121CrossRefGoogle Scholar
  32. 32.
    Wu P, Duan L, Kong L (2015) RGB-D salient object detection via feature fusion and multi-scale enhancement[C]. CCF Chinese Conference on Computer vision. Springer Berlin Heidelberg, pp 359–368Google Scholar
  33. 33.
    Xue H, Gu Y, Li Y et al (2015) RGB-D saliency detection via mutual guided manifold ranking[C]. Image Processing (ICIP), 2015 IEEE international conference on. IEEE, pp 666–670Google Scholar
  34. 34.
    Yang C, Zhang L, Lu H et al (2013) Saliency detection via graph-based manifold ranking[C]. Computer vision and pattern recognition (CVPR), 2013 IEEE Conference on. IEEE, pp 3166–3173Google Scholar
  35. 35.
    Zhang X, Chen X (2016) Robust sketch-based image retrieval by saliency detection[C]. International conference on multimedia modeling. Springer, Cham, pp 515–526CrossRefGoogle Scholar
  36. 36.
    Zhang F, Du B, Zhang L (2015) Saliency-guided unsupervised feature learning for scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing 53(4):2175–2184CrossRefGoogle Scholar
  37. 37.
    Zhang L, Chen J, Qiu B (2016) Region-of-interest coding based on saliency detection and directional wavelet for remote sensing images[J]. IEEE Geosci Remote Sens Lett 14(1):23–27CrossRefGoogle Scholar
  38. 38.
    Zhou D, Weston J, Gretton A et al (2004) Ranking on data manifolds[C]. Adv Neural Inf Proces Syst 16:169–176Google Scholar
  39. 39.
    Zhou L, Yang Z, Yuan Q et al (2015) Salient region detection via integrating diffusion-based compactness and local contrast[J]. IEEE Trans Image Process 24(11):3308–3320MathSciNetCrossRefGoogle Scholar
  40. 40.
    Zhu S, Bo Y, He L (2016) Robust multi-feature visual tracking with a saliency-based target descriptor[C]. Control conference (CCC), 2016 35th Chinese. IEEE, pp 5008–5013Google Scholar
  41. 41.
    Zhu G, Wang J, Wu Y et al (2016) MC-HOG correlation tracking with saliency proposal[C]. Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, pp 3690–3696Google Scholar
  42. 42.
    Zhu W, Liang S, Wei Y et al Saliency optimization from robust background detection[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2814–2821Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Key Laboratory of Intelligent Computing & Signal Processing, Ministry of EducationAnhui UniversityHefeiChina
  2. 2.Co-Innovation Center for Information Supply & Assurance TechnologyAnhui UniversityHefeiChina

Personalised recommendations