Advertisement

Saliency Boosting: a novel framework to refine salient object detection

  • Vivek Kumar SinghEmail author
  • Nitin Kumar
  • Suresh Madhavan
Article
  • 27 Downloads

Abstract

Salient object detection is a challenging research area and various methods have been proposed in literature. However, these methods usually focus on detecting salient objects in particular type of images only and fail when exposed to a variety of images. Here, we address this problem by proposing a novel framework called Saliency Boosting for refining saliency maps. In particular, the framework trains an Artificial Neural Network Regressor to refine initial saliency measures which are obtained from existing saliency methods. Extensive experiments on seven publicly available datasets viz. MSRA10K-test, DUT-OMRON-test, ECSSD, PASCAL-S, SED2, THUR15K, and HKU-IS have been performed to determine the effectiveness of the proposed framework. The performance of the proposed framework is measured in terms of Precision, Recall, F-Measure, Precision–Recall curve, Overlapping Ratio, Area Under the Curve and Receiver Operating Characteristic curve. The proposed framework is compared with 20 state-of-the-art-methods including best performing methods in the last decade. Further, performance of the proposed framework is better than each individual saliency detection method used in the framework. The proposed framework outperforms or is comparable with 20 state-of-the-art methods in terms of the aforementioned performance measures on all datasets.

Keywords

Saliency Boosting Artificial Neural Network Regressor (ANN-R) Saliency map Learning 

Notes

Compliance with ethical standards

Conflict of interest

All the authors declare that they have no conflict of interest.

References

  1. Achanta R, Hemami, S, Estrada, F, Susstrunk S (2009) Frequency-tuned salient region detection. In: IEEE conference on computer vision and pattern recognition. CVPR 2009. IEEE, pp 1597–1604Google Scholar
  2. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S et al (2010) Slic superpixels. Ecole Polytechnique Fédéral de Lausssanne (EPFL). Technical Report, vol 149300, pp 155–162Google Scholar
  3. Alpert S, Galun M, Basri R, Brandt A (2007) Image segmentation by probabilistic bottom-up aggregation and cue integration. In: IEEE conference on computer vision and pattern recognition. CVPR\(\backslash \)’07. IEEE, pp 1–8Google Scholar
  4. Borji A, Itti L (2013) State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 35(1):185–207CrossRefGoogle Scholar
  5. Borji A, Cheng MM, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Trans Image Process 12(24):5706–5722MathSciNetzbMATHCrossRefGoogle Scholar
  6. Chen T, Cheng MM, Tan P, Shamir A, Hu SM (2009) Sketch2photo: internet image montage. ACM Trans Graph (TOG) 28(5):124Google Scholar
  7. Chen BC, Tao X, Yang MR, Yu C, Pan WM, Leung VC (2018) A saliency map fusion method based on weighted ds evidence theory. IEEE Access 6:27346–27355CrossRefGoogle Scholar
  8. Cheng MM, Zhang GX, Mitra NJ, Huang X, Hu SM (2011) Global contrast based salient region detection. CVPR 2011Google Scholar
  9. Cheng MM, Mitra NJ, Huang X, Hu SM (2014) Salientshape: group saliency in image collections. Vis Comput 30(4):443–453CrossRefGoogle Scholar
  10. Cheng MM, Mitra NJ, Huang X, Torr PH, Hu SM (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582CrossRefGoogle Scholar
  11. Ding Y, Xiao J, Yu J (2011) Importance filtering for image retargeting. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 89–96Google Scholar
  12. Dong X, Shen J (2018) Triplet loss in siamese network for object tracking. In: Proceedings of the European conference on computer vision (ECCV), pp 459–474CrossRefGoogle Scholar
  13. Dong X, Shen J, Wu D, Guo K, Jin X, Porikli F (2019) Quadruplet network with one-shot learning for fast visual object tracking. IEEE Trans Image Process 28(7):3516–3527MathSciNetzbMATHCrossRefGoogle Scholar
  14. Donoser M, Urschler M, Hirzer M, Bischof H (2009) Saliency driven total variation segmentation. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 817–824Google Scholar
  15. Duan L, Wu C, Miao J, Qing L, Fu Y (2011) Visual saliency detection by spatially weighted dissimilarity. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 473–480Google Scholar
  16. Erdem E, Erdem A (2013) Visual saliency estimation by nonlinearly integrating features using region covariances. J Vis 13(4):11–11CrossRefGoogle Scholar
  17. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338CrossRefGoogle Scholar
  18. Gao Y, Shi M, Tao D, Xu C (2015) Database saliency for fast image retrieval. IEEE Trans Multimed 17(3):359–369CrossRefGoogle Scholar
  19. Goceri E (2016a) Automatic labeling of portal and hepatic veins from mr images prior to liver transplantation. Int J Comput Assist Radiol Surg 11(12):2153–2161CrossRefGoogle Scholar
  20. Göçeri E (2016b) Fully automated liver segmentation using sobolev gradient-based level set evolution. Int J Numer Methods Biomed Eng 32(11):e02765CrossRefGoogle Scholar
  21. Goceri E (2017) Intensity normalization in brain MR images using spatially varying distribution matching. In: 11th international conference on computer graphics, visualization, computer vision and image processing, pp 300–304Google Scholar
  22. Goceri E (2018a) Formulas behind deep learning success. In: International conference on applied analysis and mathematical modeling. Istanbul, TurkeyGoogle Scholar
  23. Goceri E (2018b) Fully automated and adaptive intensity normalization using statistical features for brain mr images. Celal Bayar Univ J Sci 14(1):125–134.  https://doi.org/10.18466/cbayarfbe.384729 CrossRefGoogle Scholar
  24. Goceri N, Goceri E (2015) A neural network based kidney segmentation from mr images. In: 2015 IEEE 14th international conference on machine learning and applications (ICMLA). IEEE, pp 1195–1198Google Scholar
  25. Goceri E, Goceri N (2017) Deep learning in medical image analysis: recent advances and future trends. In: International conferences computer graphics, visualization, computer vision and image processing, pp 305–311Google Scholar
  26. Goceri E, Gooya A (2018) On the importance of batch size for deep learning. In: International conference on mathematics. Istanbul, TurkeyGoogle Scholar
  27. Goceri E, Martinez E (2015) Artificial neural network based abdominal organ segmentations: a review. In: 2015 IEEE 14th international conference on machine learning and applications (ICMLA). IEEE, pp 1191–1194Google Scholar
  28. Göçeri E, Gürcan MN, Dicle O (2014) Fully automated liver segmentation from spir image series. Comput Biol Med 53:265–278CrossRefGoogle Scholar
  29. Göçeri E, Ünlü MZ, Dicle O (2015) A comparative performance evaluation of various approaches for liver segmentation from spir images. Turk J Electr Eng Comput Sci 23(3):741–768CrossRefGoogle Scholar
  30. Goceri E, Goksel B, Elder JB, Puduvalli VK, Otero JJ, Gurcan MN (2017) Quantitative validation of anti-ptbp1 antibody for diagnostic neuropathology use: image analysis approach. Int J Numer Methods Biomed Eng 33(11):e2862.  https://doi.org/10.1002/cnm.2862 MathSciNetCrossRefGoogle Scholar
  31. Goferman S (2010) Context-aware saliency detection. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 2376–2383Google Scholar
  32. Guo F, Wang W, Shen J, Shao L, Yang J, Tao D, Tang YY (2017) Video saliency detection using object proposals. IEEE Trans Cybern 48(11):3159–3170CrossRefGoogle Scholar
  33. Harel J, Koch C, Perona P (2007) Graph-based visual saliency. In: Advances in neural information processing systems, pp 545–552Google Scholar
  34. Hou X, Zhang L (2007) Saliency detection: a spectral residual approach. In: IEEE conference on computer vision and pattern recognition. CVPR’07. IEEE, pp 1–8Google Scholar
  35. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259CrossRefGoogle Scholar
  36. Jiang H, Wang J, Yuan Z, Wu Y, Zheng N, Li S (2013a) Salient object detection: a discriminative regional feature integration approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2083–2090Google Scholar
  37. Jiang B, Zhang L, Lu H, Yang C, Yang MH (2013b) Saliency detection via absorbing markov chain. In: Proceedings of the IEEE international conference on computer vision, pp 1665–1672Google Scholar
  38. Judd T, Durand F, Torralba A (2012) A benchmark of computational models of saliency to predict human fixations. Technical report, MIT tech reportGoogle Scholar
  39. Kaya B, Goceri E, Becker A, Elder B, Puduvalli V, Winter J, Gurcan M, Otero JJ (2017) Automated fluorescent miscroscopic image analysis of ptbp1 expression in glioma. PLoS ONE 12(3):e0170991CrossRefGoogle Scholar
  40. Kim J, Han D, Tai YW, Kim J (2014) Salient region detection via high-dimensional color transform. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 883–890Google Scholar
  41. Le Meur O, Le Callet P, Barba D, Thoreau D (2006) A coherent computational approach to model the bottom-up visual attention. IEEE Trans Pattern Ana Mach Intell 28:802–817CrossRefGoogle Scholar
  42. Lei J, Wang B, Fang Y, Lin W, Le Callet P, Ling N, Hou C (2016) A universal framework for salient object detection. IEEE Trans Multimed 18(9):1783–1795CrossRefGoogle Scholar
  43. Li G, Yu Y (2015) Visual saliency based on multiscale deep features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5455–5463Google Scholar
  44. Li X, Lu H, Zhang L, Ruan X, Yang MH (2013) Saliency detection via dense and sparse reconstruction. In: Proceedings of the IEEE international conference on computer vision, pp 2976–2983Google Scholar
  45. Li Y, Hou X, Koch C, Rehg JM, Yuille AL (2014) The secrets of salient object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 280–287Google Scholar
  46. Li H, Lu H, Lin Z, Shen X, Price B (2015) Inner and inter label propagation: salient object detection in the wild. IEEE Trans Image Process 24(10):3176–3186MathSciNetzbMATHCrossRefGoogle Scholar
  47. Liu T (2007) Learning to detect a salient object. In: Proceeings of IEEE conference on computer vision and pattern recognition, pp 353–367Google Scholar
  48. Liu GH, Yang JY (2019) Exploiting color volume and color difference for salient region detection. IEEE Trans Image Process 28(1):6–16MathSciNetzbMATHCrossRefGoogle Scholar
  49. Liu T, Yuan Z, Sun J, Wang J, Zheng N, Tang X, Shum HY (2011) Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell 33(2):353–367CrossRefGoogle Scholar
  50. Lu S, Mahadevan V, Vasconcelos N (2014) Learning optimal seeds for diffusion-based salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2790–2797Google Scholar
  51. Ma YF, Zhang HJ (2003) Contrast-based image attention analysis by using fuzzy growing. In: Proceedings of the eleventh ACM international conference on Multimedia. ACM, pp 374–381Google Scholar
  52. Marchesotti L, Cifarelli C, Csurka G (2009) A framework for visual saliency detection with applications to image thumbnailing. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 2232–2239Google Scholar
  53. Margolin R, Tal A, Zelnik-Manor L (2013) What makes a patch distinct? In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1139–1146Google Scholar
  54. Murray N, Vanrell M, Otazu X, Parraga CA (2011) Saliency estimation using a non-parametric low-level vision model. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 433–440Google Scholar
  55. Rahtu E, Kannala J, Salo M, Heikkilä J (2010) Segmenting salient objects from images and videos. In: Computer vision–ECCV 2010. Springer, pp 366–379Google Scholar
  56. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533zbMATHCrossRefGoogle Scholar
  57. Rutishauser U, Walther D, Koch C, Perona P (2004) Is bottom-up attention useful for object recognition? In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition. CVPR 2004, vol 2. IEEE, pp II–IIGoogle Scholar
  58. Seo HJ, Milanfar P (2009) Static and space–time visual saliency detection by self-resemblance. J Vis 9(12):15–15CrossRefGoogle Scholar
  59. Shen J, Du Y, Wang W, Li X (2014) Lazy random walks for superpixel segmentation. IEEE Trans Image Process 23(4):1451–1462MathSciNetzbMATHCrossRefGoogle Scholar
  60. Shen J, Hao X, Liang Z, Liu Y, Wang W, Shao L (2016) Real-time superpixel segmentation by dbscan clustering algorithm. IEEE Trans Image Process 25(12):5933–5942MathSciNetzbMATHCrossRefGoogle Scholar
  61. Shen J, Peng J, Dong X, Shao L, Porikli F (2017) Higher order energies for image segmentation. IEEE Trans Image Process 26(10):4911–4922MathSciNetzbMATHCrossRefGoogle Scholar
  62. Shi J, Yan Q, Xu L, Jia J (2016) Hierarchical image saliency detection on extended cssd. IEEE Trans Pattern Anal Mach Intell 38(4):717–729CrossRefGoogle Scholar
  63. Sun J, Ling H (2011) Scale and object aware image retargeting for thumbnail browsing. In: 2011 IEEE international conference on computer vision (ICCV). IEEE, pp 1511–1518Google Scholar
  64. Tang L, Li H, Wu Q, Ngan KN (2018) Boundary-guided optimization framework for saliency refinement. IEEE Signal Process Lett 25(4):491–495CrossRefGoogle Scholar
  65. Tavakoli HR, Rahtu E, Heikkilä J (2011) Fast and efficient saliency detection using sparse sampling and kernel density estimation. In: Scandinavian conference on image analysis. Springer, pp 666–675Google Scholar
  66. Wang W, Shen J (2017) Deep visual attention prediction. IEEE Trans Image Process 27(5):2368–2378MathSciNetCrossRefGoogle Scholar
  67. Wang W, Shen J, Shao L (2015) Consistent video saliency using local gradient flow optimization and global refinement. IEEE Trans Image Process 24(11):4185–4196MathSciNetzbMATHCrossRefGoogle Scholar
  68. Wang W, Shen J, Shao L, Porikli F (2016a) Correspondence driven saliency transfer. IEEE Trans Image Process 25(11):5025–5034MathSciNetzbMATHCrossRefGoogle Scholar
  69. Wang W, Shen J, Yu Y, Ma KL (2016b) Stereoscopic thumbnail creation via efficient stereo saliency detection. IEEE Trans Vis Comput Graph 23(8):2014–2027CrossRefGoogle Scholar
  70. Wang W, Shen J, Shao L (2017a) Video salient object detection via fully convolutional networks. IEEE Trans Image Process 27(1):38–49MathSciNetzbMATHCrossRefGoogle Scholar
  71. Wang W, Shen J, Yang R, Porikli F (2017b) Saliency-aware video object segmentation. IEEE Trans Pattern Anal Mach Intell 40(1):20–33CrossRefGoogle Scholar
  72. Wang W, Shen J, Ling H (2018) A deep network solution for attention and aesthetics aware photo cropping. IEEE Trans Pattern Anal Mac Intell 41(7):1531–1544CrossRefGoogle Scholar
  73. Yang C, Zhang L, Lu H, Ruan X, Yang MH (2013) Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3166–3173Google Scholar
  74. Yan Q, Xu L, Shi J, Jia J (2013) Hierarchical saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1155–1162Google Scholar
  75. Yuan Y, Li C, Kim J, Cai W, Feng DD (2018) Reversion correction and regularized random walk ranking for saliency detection. IEEE Trans Image Process 27(3):1311–1322MathSciNetzbMATHCrossRefGoogle Scholar
  76. Zeng Y, Feng M, Lu H, Yang G, Borji A (2018) An unsupervised game-theoretic approach to saliency detection. IEEE Trans Image Process 27(9):4545–4554MathSciNetzbMATHCrossRefGoogle Scholar
  77. Zhang J, Sclaroff S (2016) Exploiting surroundedness for saliency detection: a boolean map approach. IEEE Trans Pattern Anal Mach Intell 38(5):889–902CrossRefGoogle Scholar
  78. Zhang L, Tong MH, Marks TK, Shan H, Cottrell GW (2008) Sun: a bayesian framework for saliency using natural statistics. J Vis 8(7):32–32CrossRefGoogle Scholar
  79. Zhang W, Wu QJ, Wang G, Yin H (2010) An adaptive computational model for salient object detection. IEEE Trans Multimed 12(4):300–316CrossRefGoogle Scholar
  80. Zhang J, Sclaroff S, Lin Z, Shen X, Price B, Mech R (2015) Minimum barrier salient object detection at 80 fps. In: Proceedings of the IEEE international conference on computer vision, pp 1404–1412Google Scholar
  81. Zhou X, Liu Z, Sun G, Ye L, Wang X (2016) Improving saliency detection via multiple kernel boosting and adaptive fusion. IEEE Signal Process Lett 23(4):517–521CrossRefGoogle Scholar
  82. Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2814–2821Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Vivek Kumar Singh
    • 1
    Email author
  • Nitin Kumar
    • 1
  • Suresh Madhavan
    • 1
  1. 1.National Institute of Technology UttarakhandSrinagarIndia

Personalised recommendations