Skip to main content
SpringerLink
Log in

A novel combination of second-order statistical features and segmentation using multi-layer superpixels for salient object detection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Salient object detection is one of the outstanding capabilities of the human visual system (HVS). The researcher community aims at developing a salient object detection model that matches the detection accuracy as well as computation time taken by the humans. These models can be developed in either spatial domain or frequency domain. Spatial domain models provide good detection accuracy at the cost of high computational time while frequency domain models offer fast computational speed to meet real-time requirements at the cost of poor detection accuracy. In order to induce a trade-off between computational time and accuracy, we propose a model which provides high detection accuracy without taking much of computation time. To detect the salient object with an accurate shape, we first segment the given image by utilizing a bipartite graph partitioning approach which aggregates multi-layer superpixels in a principled and effective manner. Second, the saliency of each segmented region is computed based on a hypercomplex Fourier transform (HFT) saliency map reconstructed using amplitude spectrum, filtered at an appropriate scale chosen using statistical features extracted from grey- level co-occurrence matrix and original phase spectrum. Finally, a saliency map is generated by taking average of the HFT coefficients of each region in the segmented image and then using the average HFT intensity value of the entire image as a threshold to clearly separate salient object from the background. The performance of the proposed model is evaluated in terms of F–measure, area under curve (AUC), and computation time using six publicly available image datasets. Both qualitative and quantitative evaluations on six publicly available datasets demonstrate the robustness and efficiency of the proposed model against twenty popular state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://www.research.microsoft.com/enus/um/people/jiansun/salientobject/salient_object.htm.

  2. http://ivrgwww.epfl.ch/supplementary_material/RK_CVPR09/GroundTruth/binarymasks.zip.

  3. E-mail at “rinki.arya89@gmail.com” or “navjot.singh.09@gmail. com”.

  4. http://elderlab.yorku.ca/~vida/SOD/.

  5. http://www.wisdom.weizmann.ac.il/~vision/Seg_Evaluation_DB.

  6. http://www.wisdom.weizmann.ac.il/~vision/Seg_Evaluation_DB.

References

  1. Borji A, Itti L (2013) State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 35(1):185–207

    Article  Google Scholar 

  2. Borji A, Cheng M-M, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE Trans Image Process 24(12):5706–5722

    Article  MathSciNet  Google Scholar 

  3. Borji A, Cheng Ming-Ming, Jiang H, Li J (2014) Salient object detection: a survey. arXiv:1411.5878

  4. Hadizadeh H, Bajic IV (2014) Saliency-aware video compression. IEEE Trans Image Process 23(1):19–33

    Article  MathSciNet  Google Scholar 

  5. Wang Y-S, Tai C-L, Sorkine O, Lee T-Y (2008) Optimized scale-and-stretch for image resizing. ACM Trans Graph (TOG) 27(5):118–118

    Article  Google Scholar 

  6. Marchesotti L, Cifarelli C, Csurka G (2009) A framework for visual saliency detection with applications to image thumbnailing. In: IEEE 12th international conference on computer vision, pp 2232–2239

  7. Boykov YY, Jolly Marie-Pierre (2001) Interactive graph cuts for optimal boundary & region segmentation of objects in ND images. In: Eighth IEEE international conference on computer vision, vol 1, pp 105–112

  8. Chul B, Ko B, Nam J-Y (2006) Object-of-interest image segmentation based on human attention and semantic region clustering. J Opt Soc Am A (JOSA A) 23(10):2462–2470

    Article  Google Scholar 

  9. Gopalakrishnan V, Hu Y, Rajan D (2010) Random walks on graphs for salient object detection in images. IEEE Trans Image Process 19(12):3232–3242

    Article  MathSciNet  Google Scholar 

  10. Ueli, Walther D, Koch C, Perona PR (2004) Is bottom-up attention useful for object recognition?. In: IEEE computer society conference on computer vision and pattern recognition, vol 2, pp II–37

  11. Navalpakkam V, Itti L (2006) An integrated model of top-down and bottom-up attention for optimizing detection speed. In: IEEE computer society conference on computer vision and pattern recognition, vol 2, pp 2049–2056

  12. Achanta R, Ssstrunk S (2009) Saliency detection for content-aware image resizing. In: IEEE international conference on image processing (ICIP), pp 1005–1008

  13. Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: IEEE conference on Computer vision and pattern recognition, pp 1597–1604

  14. Alpert S, Galun M, Brandt A, Basri R (2012) Image segmentation by probabilistic bottom-up aggregation and cue integration. IEEE Trans Pattern Anal Mach Intell 34(2):315–327

    Article  Google Scholar 

  15. Jian MW, Dong JY, Ma J (2011) Image retrieval using wavelet-based salient regions. The Imaging Science Journal 59(4):219–231

    Article  Google Scholar 

  16. Huang K, Tao D, Yuan Y, Li X, Tan T (2011) Biologically inspired features for scene classification in video surveillance. IEEE Trans Syst Man Cybern Part B Cybern 41(1):307–313

    Article  Google Scholar 

  17. Park J, Lee J-Y, Tai Y-W, Kweon IS (2012) Modeling photo composition and its application to photo re-arrangement. In: IEEE international conference on image processing (ICIP), pp 2741–2744

  18. Ninassi A, Le Meur O, Le Callet P, Barbba D (2007) Does where you gaze on an image affect your perception of quality? Applying visual attention to image quality metric. In: IEEE international conference on image processing, vol 2, pp II–169

  19. Li Z, Itti L (2011) Saliency and gist features for target detection in satellite images. IEEE Trans Image Process 20(7):2017–2029

    Article  MathSciNet  Google Scholar 

  20. Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M (2006) Gaze-based interaction for semi-automatic photo cropping. In: SIGCHI conference on human factors in computing systems, pp 771–780

  21. Chen L-Q, et al. (2003) A visual attention model for adapting images on small displays. Multimedia Systems 9(4):353–364

    Article  Google Scholar 

  22. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  23. Itti L (2000) Models of bottom-up and top-down visual attention. California Institute of Technology, Doctoral dissertation

  24. Karssemeijer N, te Brake GM (1996) Detection of stellate distortions in mammograms. IEEE Trans Med Imaging 15(5):611–619

    Article  Google Scholar 

  25. Rother C, Bordeaux L, Hamadi Y, Blake A (2006) Autocollage. ACM Trans Graph (TOG) 25 (3):847–852

    Article  Google Scholar 

  26. Gasparini F, Corchs S, Schettini R (2007) Low-quality image enhancement using visual attention. Opt Eng 46(4):040502–040502

    Article  Google Scholar 

  27. Kim J, Han D, Tai Y-W, Kim J (2016) Salient region detection via high-dimensional color transform and local spatial support. IEEE Trans Image Process 25(1):9–23

    Article  MathSciNet  Google Scholar 

  28. Zhu L, Klein DA, Frintrop S, Cao Z, Cremers AB (2014) A multisize superpixel approach for salient object detection based on multivariate normal distribution estimation. IEEE Trans Image Process 23(12):5094–5107

    Article  MathSciNet  Google Scholar 

  29. Itti L, Koch C (2001) Computational modelling of visual attention. Nat Rev Neurosci 2(3):194–203

    Article  Google Scholar 

  30. Borji A (2012) Boosting bottom-up and top-down visual features for saliency estimatio. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 438–445

  31. Cheung Y-M, Peng Q (2012) Salient region detection using local and global saliency. In: 21st international conference on pattern recognition (ICPR), pp 210–213

  32. Jia C, Hou F, Duan L (2013) Visual saliency based on local and global features in the spatial domain. Int J Comput Sci 10(3):713–719

    Google Scholar 

  33. Borji A, Itti L (2012) Exploiting local and global patch rarities for saliency detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 478–485

  34. Zhang L, Yang L, Luo T (2016) Unified saliency detection model using color and texture features. Plos one 11(2):e0149328

    Article  Google Scholar 

  35. Itti L (2005) Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes. Vis Cogn 12(6):1093–1123

    Article  Google Scholar 

  36. Corbetta M, Shulman GL (2002) Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci 3(3):201–215

    Article  Google Scholar 

  37. Wang K, Lin L, Lu J, Li C, Shi K (2015) PISA: Pixelwise image saliency by aggregating complementary appearance contrast measures with edge-preserving coherence. IEEE Trans Image Process 24(10):3019–3033

    Article  MathSciNet  Google Scholar 

  38. Cheng M, Mitra NJ, Huang X, Torr PHS, Hu S (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582

    Article  Google Scholar 

  39. Naqvi SS, Browne WN, Hollitt C (2016) Salient object detection via spectral matting. Pattern Recogn 51:209–224

    Article  Google Scholar 

  40. Huang X, Su Y, Liu Y (2016) Iteratively parsing contour fragments for object detection. Neurocomputing 175:585–598

    Article  Google Scholar 

  41. Huo L, Jiao L, Wang S, Yang S (2016) Object-level saliency detection with color attributes. Pattern Recogn 49:162–173

    Article  Google Scholar 

  42. Levine MD, An X, He H (2011) Saliency detection based on frequency and spatial domain analysis. In: British machine vision conference (BMVC), pp 86.1–86.11

  43. Li Z, Wu X-M, Chang S-F (2012) Segmentation using superpixels: a bipartite graph partitioning approach. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 789–796

  44. Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619

    Article  Google Scholar 

  45. Felzenszwalb PF, Huttenlocher DP (2004) Efficient graph-based image segmentation. Int J Comput Vis 59(2):167–181

    Article  Google Scholar 

  46. Han J, Ngan KN, Li M, Zhang H-J (2006) Unsupervised extraction of visual attention objects in color images. IEEE Trans Circuits Syst Video Technol 16(1):141–145

    Article  Google Scholar 

  47. Bruce N, Tsotsos J (2005) Saliency based on information maximization. Adv Neural Inf Proces Syst:155–162

  48. Harel J, Koch C, Perona P (2006) Graph-based visual saliency. In: Advances in neural information processing systems, pp 545–552

  49. Hou X, Zhang L (2007) Saliency detection: a spectral residual approach. In: IEEE conference on computer vision and pattern recognition, pp 1–8

  50. Liu T, et al (2011) Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell 33(2):353–367

    Article  Google Scholar 

  51. Liu T, et al (2007) Learning to detect a salient object. In: IEEE conference on computer vision and pattern recognition, pp 1–8

  52. Guo C, Qi M a, Zhang L (2008) Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1–8

  53. Yu Y, Wang B, Zhang L (2009) Pulse discrete cosine transform for saliency-based visual attention. In: IEEE 8th international conference on development and learning, pp 1–6

  54. Bian P, Zhang L (2008) Biological plausibility of spectral domain approach for spatiotemporal visual saliency. In: International conference on neural information processing, pp 251–258

  55. Bian P, Zhang L (2010) Visual saliency: a biologically plausible contourlet-like frequency domain approach. Cogn Neurodyn 4(3):189–198

    Article  Google Scholar 

  56. Bian P, Zhang L (2010) Piecewise frequency domain visual saliency detection. In: IEEE third international conference on information and computing (ICIC), vol 3, pp 269–272

  57. Achanta R, Susstrunk S (2010) Saliency detection using maximum symmetric surround. In: 17th IEEE international conference on image processing (ICIP), pp 2653–2656

  58. Guo C, Zhang L (2010) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):185–198

    Article  MathSciNet  Google Scholar 

  59. Goferman S, Zelnik-Manor L, Tal A (2012) Context-aware saliency detection. IEEE Trans Pattern Anal Mach Intell 34(10):1915–1926

    Article  Google Scholar 

  60. Shen X, Ying W u (2012) A unified approach to salient object detection via low rank matrix recovery. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 853–860

  61. Fang Y, et al. (2012) Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Trans Multimedia 14(1):187–198

    Article  MathSciNet  Google Scholar 

  62. Fang Y, Chen Z, Lin W, Lin C-W (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process 21(9):3888–3901

    Article  MathSciNet  Google Scholar 

  63. Li J, Levine MD, An X, Xu X, He H (2013) Visual saliency based on scale-space analysis in the frequency domain. IEEE Trans Pattern Anal Mach Intell 35(4):996–1010

    Article  Google Scholar 

  64. Li J, Duan L-Y, Chen X, Huang T, Tian Y (2015) Finding the secret of image saliency in the frequency domain. IEEE Trans Pattern Anal Mach Intell 37(12):2428–2440

    Article  Google Scholar 

  65. Arya R, Singh N, Agrawal RK (2015) A novel hybrid approach for salient object detection using local and global saliency in frequency domain. Multimedia Tools and Applications:1–21

  66. Huaizu, Yuan, Zejian, Cheng, Jiang M-M, Gong Y, Zheng N, Wang J (2014) Salient object detection: a discriminative regional feature integration approach. arXiv:1410.5926

  67. Zou W, Komodakis N (2015) HARF: Hierarchy-associatedrichfeaturesforsalientobjectdetection. In: IEEE international conference on computer vision, pp 406–414

  68. Sun J, Lu H, Liu X (2015) Saliency region detection based on Markov absorption probabilities. IEEE Trans Image Process 24(5):1639–1649

    Article  MathSciNet  Google Scholar 

  69. Perazzi F, Philipp, Krahenbuhl, Yael, Pritch, Hornung A (2012) Saliency filters: contrast based filtering for salient region detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 733–740

  70. Yan Q, Xu L, Jianping, Jia JS (2013) Hierarchical saliency detection. In: IEEE conference on computer vision and pattern recognition, pp 1155–1162

  71. Zhi, Zou, Wenbin, Le Meur, Liu O (2014) Saliency tree: a novel saliency detection framework. IEEE Trans Image Process 23(5):1937–1952

    Article  MathSciNet  Google Scholar 

  72. Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. In: IEEE conference on computer vision and pattern recognition, pp 2814–2821

  73. Li X, Lu H, Zhang L, Ruan X, Yang M-H (2013) Saliency detection via dense and sparse reconstruction. In: IEEE international conference on computer vision, pp 2976–2983

  74. Chang K-Y, Liu T-L, Chen H-T, Lai S-H (2011) Fusing generic objectness and visual saliency for salient object detection. In: IEEE international conference on computer vision (ICCV), pp 914–921

  75. Jiang H, et al (2011) Automatic salient object segmentation based on context and shape prior. In: BMVC, vol 6, p 9

  76. Yang C, Zhang L, Lu H, Ruan X, Yang M-H (2013) Saliency detection via graph-based manifold ranking. In: IEEE conference on computer vision and pattern recognition, pp 3166–3173

  77. Margolin R, Tal A, Zelnik-Manor L (2013) What makes a patch distinct?. In: IEEE conference on computer vision and pattern recognition, pp 1139–1146

  78. Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Transaction on Pattern and Machine Intelligence 22(8):888–905

    Article  Google Scholar 

  79. Zhang L, Lin W (2013) Selective visual attention: computational models and applications. Wiley

  80. Castleman Kenneth R (1996) Digital image processing. Prentice Hall Press, Upper Saddle River

    Google Scholar 

  81. Sangwine SJ (1996) Fourier transforms of colour images using quaternion or hypercomplex, numbers. Electron Lett 32(21):1979–1980

    Article  Google Scholar 

  82. Ell TA (1992) Hypercomplex spectral transformations. University of Minnesota

  83. Ell TA (1993) Quaternion-fourier transforms for analysis of two-dimensional linear time-invariant partial differential systems. In: 32nd IEEE conference on decision and control, pp 1830–1841

  84. Pei S-C, Ding J-J, Chang J-H (2001) Efficient implementation of quaternion Fourier transform, convolution, and correlation by 2-D complex FFT. IEEE Trans Signal Process 49(11):2783–2797

    Article  MathSciNet  Google Scholar 

  85. Sangwine SJ, Ell TA (2000) The discrete Fourier transform of a colour image. Image Processing II Mathematical Methods, Algorithms and Applications:430–441

  86. Ell TA, Sangwine SJ (2007) Hypercomplex Fourier transforms of color images. IEEE Trans Image Process 16(1):22–35

    Article  MathSciNet  MATH  Google Scholar 

  87. Itti L, Baldi PF (2005) Bayesian surprise attracts human attention. Adv Neural Inf Proces Syst:547–554

  88. Engel S, Zhang X, Wandell B (1997) Colour tuning in human visual cortex measured with functional magnetic resonance imaging. Nature 388(6637):68–71

    Article  Google Scholar 

  89. Haralick RM, Shanmugam K, Dinstein IH (1973) Textural features for image classification. IEEE Trans Syst Man Cybern:610–621

  90. Li J, Duan L-Y, Chen X, Huang T, Tian Y (2015) Finding the secret of image saliency in the frequency domain. IEEE Trans Pattern Anal Mach Intell 37(12):2428–2440

    Article  Google Scholar 

  91. Singh N, Arya R, Agrawal RK (2014) A novel approach to combine features for salient object detection using constrained particle swarm optimization. Pattern Recogn 47(4):1731–1739

    Article  Google Scholar 

  92. Han J, Ngan KN, Li M, Zhang H-J (2006) Unsupervised extraction of visual attention objects in color images. IEEE Trans Circuits Syst Video Technol 16(1):141–145

    Article  Google Scholar 

Download references

Acknowledgments

The first author expresses her sincere and reverential gratitude to University Grant Commission (UGC), India, for the obtained financial support during this research.

Author information

Authors and Affiliations

  1. School of Computer and Systems Sciences, Jawaharlal Nehru University, New Delhi, 110067, India

    Rinki Arya, Navjot Singh & R. K. Agrawal

  2. National Institute of Technology, Srinagar, Pauri Garhwal, Uttarakhand, 246174, India

    Navjot Singh

Authors
  1. Rinki Arya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Navjot Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. K. Agrawal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rinki Arya.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arya, R., Singh, N. & Agrawal, R.K. A novel combination of second-order statistical features and segmentation using multi-layer superpixels for salient object detection. Appl Intell 46, 254–271 (2017). https://doi.org/10.1007/s10489-016-0819-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-016-0819-6

Keywords

Search

Navigation