Skip to main content

Advertisement

SpringerLink
Log in

Off-the-Shelf Deep Features for Saliency Detection

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Computational saliency refers to the ability to highlight the salient visual information for processing. The mechanism has proven to be helpful for human as well as computer vision. Computational saliency focuses on designing algorithms which, similarly to human vision, predict which regions in a scene are salient. Recently, salient object segmentation has introduced the use of object proposals. Object proposal methods provide image segments as proposals which can be used for saliency estimation. We propose several saliency features which are computed from different networks and different levels with the aim to define which optimal network and layer for the task of saliency detection. Also much recently, convolutional neural networks breakthroughs computer vision with the extraction of the much powerful features which are based on deep CNN. In this paper, we develop a saliency approach based on the computation of deep whitened features combined with shape features from object proposals. We train an SVM to predict the saliency of every object proposal. Experimental results shows that we outperform other state-of-the-art methods in PASCAL-S, FT, ECSSD, MSRA-B and ImgSal data sets in terms of F-score, PR curves. Furthermore, experiments show that applying whitening improve performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Hemami S, Estrada F, Achanta R, Susstrunk S (2009) Frequency-tuned salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pages 1597–1604. IEEE.

  2. Arbelaez P, Pont-Tuset J, Barron J, Marques F, Malik J (2014) Multiscale combinatorial grouping. In: IEEE conference on computer vision and pattern recognition, pages 328–335. IEEE

  3. Azaza Aymen, van de Weijer Joost, Douik Ali, Masana Marc. Context proposals for saliency detection. Comput Vis Image Understanding. 2018;174:1–11.

    Article  Google Scholar 

  4. Borji A, Itti L. State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell. 2013;35(1):185–207.

    Article  Google Scholar 

  5. Borji A (2012) Boosting bottom-up and top-down visual features for saliency estimation. In: Computer vision and pattern recognition (CVPR), 2012 IEEE conference on, pages 438–445. IEEE, 2012.

  6. Neil Bruce and John Tsotsos. Saliency based on information maximization. In: Advances in neural information processing systems, pages 155–162, 2005.

  7. K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In: British machine vision conference, 2014.

  8. Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. In: IEEE transactions on pattern analysis and machine intelligence, 37(3):569–582, 2015.

  9. Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi. Deep filter banks for texture recognition and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3828–3836, 2015.

  10. Dan Cireşan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. arXiv preprint arXiv:1202.2745, 2012.

  11. J. Deng, Socher R. Li L.-J. Li K. Dong, W., and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.

  12. Chen Z, Lin W, Fang Y, Lin C-W (2011) Saliency-based image retargeting in the compressed domain. In: Proceedings of the 19th ACM international conference on Multimedia, pages 1049–1052. ACM

  13. Cooperative effects of top-down & bottom-up visual attention. R Gaborski, Vishal S Vaingankar, and RL Canosa. Goal directed visual search based on color cues. Proc Artificial Neural Netw Eng Rolla Missouri. 2003;13:613–8.

    Google Scholar 

  14. Fdez-Vidal X-R, Pardo XM, Dosil R, Garcia-Diaz A. Saliency from hierarchical adaptation through decorrelation and variance normalization. Image Vis Comput. 2012;30(1):51–64.

    Article  Google Scholar 

  15. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pages 1440–1448

  16. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587

  17. Zelnik-Manor L, Goferman S, Tal A (2012) Context-aware saliency detection. In: IEEE Transactions on pattern analysis and machine intelligence, 34(10):1915–1926

  18. Harel J, Koch C, Perona P (2006) Graph-based visual saliency. In: Advances in neural information processing systems, pages 545–552

  19. Ren S, He K, Zhang X, Sun J (2015) Deep residual learning for image recognition. In arXiv prepring arXiv:1506.01497

  20. Hong G-S, Kim B-G, Prosad Dogra D, Pratim Roy P (2018) A survey of real-time road detection techniques using visual color sensor. J Multimedia Inform Syst. 5(1):9–14, 2018.

  21. Koch C, Itti L, Niebur E, et al. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell. 1998;20(11):1254–9.

    Article  Google Scholar 

  22. Jeong Dami, Kim Byung-Gyu, Dong Suh-Yeon. Deep joint spatiotemporal network (djstn) for efficient facial expression recognition. Sensors. 2020;20(7):1936.

    Article  Google Scholar 

  23. Jiang H, Wang J, Yuan Z, Liu T, Zheng N, Li S. Automatic salient object segmentation based on context and shape prior. BMVC. 2011;6:9.

    Google Scholar 

  24. Wang J. Yuan Z.-Wu Y. Zheng N. Jiang, H. and S. Li. Salient object detection: A discriminative regional feature integration approach. In: IEEE conference on computer vision and pattern recognition, pages 2083–2090. IEEE, 2013.

  25. Judd T, Ehinger K, Durand F, Torralba A (2009) Learning to predict where humans look. In: 2009 IEEE 12th international conference on computer vision, pages 2106–2113. IEEE

  26. Deep learning-based gesture recognition scheme using motion sensors. Ji-Hae Kim, Gwang-Soo Hong, Byung-Gyu Kim, and Debi P Dogra. deepgesture. Displays. 2018;55:38–45.

    Article  Google Scholar 

  27. Ji-Hae Kim, Byung-Gyu Kim, Partha Pratim Roy, and Da-Mi Jeong. Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access, 7:41273–41285, 2019.

  28. Christof Koch and Shimon Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. In: Matters of intelligence, pages 115–141. Springer, 1987.

  29. Sutskever I. Krizhevsky, A. and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pages 1097–1105, 2012.

  30. Gautam Kumar, Prateek Keserwani, Partha Pratim Roy, and Debi Prosad Dogra. Logo detection using weakly supervised saliency map. Multimedia Tools and Applications, pages 1–25, 2020.

  31. LeCun Y, Boser B, Denker JS, D Henderson, RE Howard, W Hubbard, LD Jackel (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4):541–551

  32. G. Li and Y. Yu. Visual saliency based on multiscale deep features. In: IEEE conference on computer vision and pattern recognition. IEEE, 2015.

  33. G. Li and Y. Yu. Deep contrast learning for salient object detection. arXiv preprint arXiv:1603.01976, 2016.

  34. Levine M. D. An-X. Xu X. Li, J. and H. He. Visual saliency based on scale-space analysis in the frequency domain. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4):996–1010, 2013.

  35. Y Li, X Hou, C Koch, JM Rehg, AL Yuille (2014) The secrets of salient object segmentation. In: IEEE conference on computer vision and pattern recognition, pages 280–287. IEEE

  36. Zhao L. Wei L.-Yang M. Wu F. Zhuang Y. Ling H. Li, X. and J. Wang. Deepsaliency: Multi-task deep neural network model for salient object detection. arXiv preprint arXiv:1510.05484, 2015.

  37. Koch C.-Rehg J. Li Y., Hou X. and Y. A. L. The secrets of salient object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pages 280–287. IEEE, 2014.

  38. Liu Tie, Yuan Zejian, Sun Jian, Wang Jingdong, Zheng Nanning, Tang Xiaoou, Shum Heung-Yeung. Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell. 2011;33(2):353–67.

    Article  Google Scholar 

  39. Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.

  40. Ren M, Lou J, Wang H. Regional principal color based saliency detection. PLoS One. 2014;9(11):e112475.

    Article  Google Scholar 

  41. Cifarelli C, Marchesotti L, G Csurka (2009) A framework for visual saliency detection with applications to image thumbnailing. In: 12th international conference on computer vision, pages 2232–2239. IEEE

  42. Fowlkes C.C. Martin, D. R and J. Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans Pattern Anal Mach Intell 26(5):530–549, 2004.

  43. Mukherjee S, Kumar P, Saini R, Pratim Roy P, Prosad Dogra D, Kim B-G. Plant disease identification using deep neural networks. J Multimedia Inform Syst. 2017;4(4):233–8.

    Google Scholar 

  44. Subham Mukherjee, Rajkumar Saini, Pradeep Kumar, Partha Pratim Roy, Debi Prosad Dogra, Byung-Gyu Kim, et al. Fight detection in hockey videos using deep network. Journal of Multimedia Information System, 4(4):225–232, 2017.

  45. Murray N, Vanrell M, Otazu X, Alejandro Parraga C. Low-level spatiochromatic grouping for saliency estimation. IEEE Trans Pattern Anal Mach Intell. 2013;35(11):2810–6.

    Article  Google Scholar 

  46. Oquab M, Bottou L, Laptev I, Sivic J (2014) Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717–1724

  47. Yao Qin, Huchuan Lu, Yiqun Xu, and He Wang. Saliency detection via cellular automata. In IEEE Conference on Computer Vision and Pattern Recognition, pages 110–119, 2015.

  48. Ramanishka V, Das A, Zhang J, Saenko K (2017) Top-down visual saliency guided by captions. In: IEEE international conference on computer vision and pattern recognition

  49. A. Azizpour H. Sullivan-J. S., Razavian and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014.

  50. Avishek Saha, Young-Woon Lee, Young-Sup Hwang, Kostas E Psannis, and Byung-Gyu Kim. Context-aware block-based motion estimation algorithm for multimedia internet of things (iot) platform. Personal and Ubiquitous Computing, 22(1):163–172, 2018.

  51. Scharfenberger C, Wong A, Fergani K, Zelek JS, Clausi DA (2013) Statistical textural distinctiveness for salient region detection in natural images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 979–986

  52. Wong A, Fergani K, Zelek JS, Clausi D, Scharfenberger C et al. (2013) Statistical textural distinctiveness for salient region detection in natural images. In: IEEE Conference on computer vision and pattern recognition, pages 979–986. IEEE

  53. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.

  54. X. Y. Stella and D. A. Lisin. Image compression based on visual saliency at individual scales. In: Advances in Visual Computing, pages 157–166. Springer, 2009.

  55. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9

  56. Treisman AM, Gelade G. A feature-integration theory of attention. Cognitive Psychol. 1980;12(1):97–136.

    Article  Google Scholar 

  57. Gevers Th. Van De Weijer J. and Bagdanov A.D. Boosting color saliency in image feature detection. IEEE TRANS. PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 28:150–156, 2005.

  58. Jin P, Wan S, L Yue (2009) An approach for image retrieval based on visual saliency. In: International conference on image analysis and signal processing, pages 172–175. IEEE

  59. Lu H, Ruan X, Wang L, Yang M-H (2015) Deep networks for saliency detection via local estimation and global search. In: IEEE conference on computer vision and pattern recognition, pages 3183–3192

  60. Wang S, Jiang M, Morin Duchesne X, Laugeson EA , Kennedy DP , Adolphs R, Zhao Q (2015) Atypical visual saliency in autism spectrum disorder quantified through model-based eye tracking. Neuron, 88(3):604–616

  61. Xu P, Ehinger KA, Zhang Y, Finkelstein A, Kulkarni SR, Xiao J (2015) Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755

  62. Xu L,Shi J, Yan Q, Jia J (2013) Hierarchical saliency detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pages 1155–1162. IEEE,

  63. Yang MH, Yang J. Top-down visual saliency via joint crf and dictionary learning. IEEE Trans Pattern Anal Mach Intell. 2016;39(3):576–88.

    Article  Google Scholar 

  64. Zhang L, Lu H, Ruan X, Yang C, Yang M-H Saliency detection via graph-based manifold ranking. In: IEEE conference on computer vision and pattern recognition, pages 3166–3173. IEEE, 2013.

  65. Yuan Yue, Jun Chu Lu, Leng Jun Miao, Kim Byung-Gyu. A scale-adaptive object-tracking algorithm with occlusion detection. EURASIP J Image Video Process. 2020;2020(1):1–15.

    Article  Google Scholar 

  66. Zhang L, Gu Z, Li H (2013) Sdsp: A novel saliency detection method by combining simple priors. In: 20th international conference on image processing, pages 171–175. IEEE

  67. Zhao R, Ouyang LHW, Wang X (2015) Saliency detection by multi-context deep learning. In: IEEE conference on computer vision and pattern recognition, pages 1265–1274

Download references

Author information

Authors and Affiliations

  1. Eniso, University of Sousse, Sousse, Tunisia

    Aymen Azaza & Ali Douik

  2. University of Kairaouan, Kairouan, Tunisia

    Mehrez Abdellaoui

Authors
  1. Aymen Azaza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehrez Abdellaoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ali Douik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aymen Azaza.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest.

Data availability

The data used to support the findings of this study are available from the corresponding author upon request.

Funding Statement

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Azaza, A., Abdellaoui, M. & Douik, A. Off-the-Shelf Deep Features for Saliency Detection. SN COMPUT. SCI. 2, 127 (2021). https://doi.org/10.1007/s42979-021-00499-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-021-00499-7

Keywords

Search

Navigation