Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 20, pp 26351–26369 | Cite as

Meta-metric for saliency detection evaluation metrics based on application preference

  • Yuzhen Niu
  • Jianer Chen
  • Wenzhong GuoEmail author
Article
  • 198 Downloads

Abstract

Existing saliency detection evaluation metrics often produce inconsistent evaluation results. Because of the widespread application of image saliency detection, we propose a meta-metric to evaluate the performance of these metrics based on the preference of an application that uses saliency maps as weighting maps. This study uses content-based image retrieval (CBIR) as the representative application. First, we perform CBIR using image features extracted from deep convolutional layers of convolutional neural networks as well as saliency maps computed by various saliency detection algorithms as the weighting maps over queries. Second, we establish the preference order of the saliency detection algorithms in the CBIR application by sorting the mean average precision. Third, we determine the preference order of these algorithms using existing saliency detection evaluation metrics. Finally, our meta-metric evaluates these metrics by correlating the preference order in the CBIR application with that determined by each evaluation metric. Experiments on three publicly available datasets show that, of 24 evaluation metrics, the traditional metric: area under receiver operating characteristic curve is the best metric for a CBIR application.

Keywords

Saliency detection Traditional evaluation metrics Image quality assessment Content-based image retrieval 

Notes

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61672158, 61672159, 61502105, and 61300102, in part by the Fujian Natural Science Funds for Distinguished Young Scholar under Grant 2015J06014, in part by the Technology Guidance Project of Fujian Province under Grant 2017H0015, in part by the Industry-Academy Cooperation Project under Grant 2017H6008, and the Fujian Collaborative Innovation Center for Big Data Application in Governments.

References

  1. 1.
    Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency tuned salient region detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1597–1604Google Scholar
  2. 2.
    An J, Lee SH, Cho NI (2014) Content-based image retrieval using color features of salient regions, IEEE International Conference on Image Processing, pp. 3042–3046Google Scholar
  3. 3.
    Balanov A, Schwartz A, Moshe Y, Peleg N (2015) Image quality assessment based on DCT subband similarity. In: IEEE International Conference on Image Processing, pp. 2105–2109Google Scholar
  4. 4.
    Chang H, Zhang Q, Wu Q, Gan Y (2015) Perceptual image quality assessment by independent feature detector. Neurocomputing 151:1142–1152CrossRefGoogle Scholar
  5. 5.
    Chen T, Cheng M, Tan P, Shamir A, Hu S (2009) Sketch2photo: internet image montage. ACM Transactions on Graphics 28(5):124:1–124:10.  https://doi.org/10.1145/1661412.1618470
  6. 6.
    Cheng MM, Mitra NJ, Huang X, Hu SM (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 37(3):569–582CrossRefGoogle Scholar
  7. 7.
    Christian S, Wong A, John KF, David A (2013) Statistical textural distinctiveness for salient region detection in natural images. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern RecognitionGoogle Scholar
  8. 8.
    Christopoulos C, Skodras A, Koike A, Ebrahimi T (2000) The JPEG2000 still image coding system: an overview. IEEE Trans Consum Electron 46(4):1103–1127CrossRefGoogle Scholar
  9. 9.
    Dohyoung L, Konstantinos NP (2015) Towards a full-reference quality assessment for color images using directional statistics. IEEE Trans Image Process 24(11):3950–3965Google Scholar
  10. 10.
    Egiazarian K, Astola J, Ponomarenko N, Lukin V, Battisti F, Carli M (2006) New full-reference quality metrics based on HVS, Proceedings of the Second International Workshop on Video Processing and Quality MetricsGoogle Scholar
  11. 11.
    Eric L, Damon C (2010) Most apparent distortion: full-reference image quality assessment and the role of strategy. J Electron Imaging 19(1):011006CrossRefGoogle Scholar
  12. 12.
    Gao Y, Shi M, Tao D, Xu C (2015) Database saliency for fast image retrieval. IEEE Trans Multimed 17(3):359–369CrossRefGoogle Scholar
  13. 13.
    Goferman S, Manor LZ, Tal A (2010) Context-aware saliency detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1915–1926Google Scholar
  14. 14.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, pp 770–778Google Scholar
  15. 15.
    Hiremath P, Pujari J (2008) Content based image retrieval using color boosted salient points and shape features of an image. Int J Image Process 2(1):10–17Google Scholar
  16. 16.
    Hongliang L, Ngan KN (2011) A co-saliency model of image pairs. IEEE Trans Image Process 20(12):3365–3375MathSciNetCrossRefGoogle Scholar
  17. 17.
    Hu SM, Chen T, Xu K, Cheng MM, Martin RR (2013) Internet visual media processing: a survey with graphics and vision applications. The Visual Computer 29(5):393–405CrossRefGoogle Scholar
  18. 18.
    Jianping S, Qiong Y, Li X, Jiaya J (2016) Hierarchical image saliency detection on extended CSSD. IEEE Trans Pattern Anal Mach Intell 38(4):717–729Google Scholar
  19. 19.
    Juneja K, Verma A, Goel S, Goel S (2015) A survey on recent image indexing and retrieval techniques for low-level feature extraction in CBIR systems, IEEE International Conference on Computational Intelligence & Communication Technology, pp. 67–72Google Scholar
  20. 20.
    Kanan C, Cottrell GW (2010) Robust classification of objects, faces, and flowers using natural image statistics. IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, pp 2472–2479Google Scholar
  21. 21.
    Lin Z, Lei Z, Xuanqin M, David Z (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Processing 20(8):2378–2386Google Scholar
  22. 22.
    Margolin R, Zelnikmanor L, Tal A (2014) How to evaluate foreground maps. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 248–255Google Scholar
  23. 23.
    Mitsa T, Varkur KL (1993) Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in half toning algorithms, Proceedings of the IEEE International Conference on Acoustics, pp. 301–304Google Scholar
  24. 24.
    Peng H, Li B, Ling H, Hu W, Xiong W, Stephen J (2017) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):818–832CrossRefGoogle Scholar
  25. 25.
    Chunbiao Z, Ge L, Wenmin W, Ronggang W (2017) An innovative salient object detection using centerdark channel prior. IEEE International Conference on Computer Vision Workshops, Mountain View, pp 1509–1515Google Scholar
  26. 26.
    Perazzi F, Krahenbuhl P, Pritch Y, Hornung A (2012) Saliency filters: contrast based filtering for salient region detection, IEEE Conference on Computer Vision and Pattern Recognition, Providence, pp 733–740Google Scholar
  27. 27.
    Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern RecognitionGoogle Scholar
  28. 28.
    Ponomarenko N, Silvestri F, Egiazarian K, Carli M, Astola J, Lukin V (2007) On between-coefficient contrast masking of DCT basis functions, Proceedings of the Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics, pp. 25–26Google Scholar
  29. 29.
    Ponomarenko N, Eremeev O, Lukin V, Egiazarian K, Carli M (2011) Modified image visual quality metrics for contrast change and mean shift accounting. In: International Conference The Experience of Designing and Application of CAD Systems in Microelectronics, Polyana-Svalyava, pp 305–311Google Scholar
  30. 30.
    Robertson S (2009) Evaluation in information retrieval. Cambridge University Press, CambridgeGoogle Scholar
  31. 31.
    Rubinstein M, Joulin A, Kopf J, Liu C (2013) Unsupervised joint object discovery and segmentation in internet images. CVPR, pp. 1939–1946Google Scholar
  32. 32.
    Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451CrossRefGoogle Scholar
  33. 33.
    Shotton J, Winn J, Rother C, Criminisi A (2006) Textonboost: joint appearance, shape and context modeling for multi-class object recognition and segmentation. ECCV, pp. 1–15Google Scholar
  34. 34.
    Venkata ND, Kite T, Geisler W, Evans B, Bovik A (2000) Image quality assessment based on a degradation model. IEEE Trans Image Processing 9(4):636–650Google Scholar
  35. 35.
    Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84Google Scholar
  36. 36.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRefGoogle Scholar
  37. 37.
    Wang P, Wang J, Zeng G, Feng J, Zha H, Li S (2012) Salient object detection for searched web images via global saliency. IEEE Conference on Computer Vision and Pattern Recognition, Providence, pp 3194–3201Google Scholar
  38. 38.
    Weichi T, Shengfeng H, Qingxiong Y, Shaoyi C (2016) Real-time salient object detection with a minimum spanning tree. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2334–2342Google Scholar
  39. 39.
    Wu R, Yu Y, Wang W (2013) SCaLE: supervised and cascaded laplacian eigenmaps for visual object recognition based on nearest neighbors, IEEE Conference on Computer Vision and Pattern Recognition, Portland, pp 867–874Google Scholar
  40. 40.
    Wu H, Li G, Luo X (2014) Weighted attentional blocks for probabilistic object tracking. Vis Comput 30(2):229–243CrossRefGoogle Scholar
  41. 41.
    Xu X, Geng W, Ju R, Yang Y, Ren T, Wu G (2014) OBSIR: object-based stereo image retrieval, IEEE International Conference on Multimedia and Expo, Chengdu, pp 1–6Google Scholar
  42. 42.
    Yandex AB, Lempitsky V (2015) Aggregating local deep features for image retrieval, IEEE International Conference on Computer Vision, pp. 1269–1277Google Scholar
  43. 43.
    Yang Y, Yang L, Wu G, Li S (2012) A bag-of-objects retrieval model for web image search, Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 49–58Google Scholar
  44. 44.
    Yang C, Zhang L, Lu H, Ruan X, Yang MH (2013) Saliency detection via graph-based manifold ranking, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3166–3173Google Scholar
  45. 45.
    Yao Q, Huchuan L, Yiqun X, He W (2015) Saliency detection via cellular automata, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 110–119Google Scholar
  46. 46.
    Yujiao Z (2016) Image retrieval algorithm based on harris-laplace corners and SVM relevance feedback, IEEE International Conference on Software Engineering and Service Science, pp. 337–340Google Scholar
  47. 47.
    Zheng L, Yang Y, Tian Q (2017) SIFT meets CNN: a decade survey of instance retrieval, IEEE Transactions on Pattern Analysis and Machine IntelligenceGoogle Scholar
  48. 48.
    Zhu W, Liang S, Wei Y, Sun J (2014) Saliency optimization from robust background detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 814–2821Google Scholar
  49. 49.
    Zhu W, Liang S, Wei Y (2014) Saliency optimization from robust background detection, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2814–2821Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Mathematics and Computer ScienceFuzhou UniversityFuzhouChina
  2. 2.Fujian Key Laboratory of Network Computing and Intelligent Information ProcessingFuzhou UniversityFuzhouChina
  3. 3.Key Laboratory of Spatial Data Mining & Information SharingMinistry of EducationFuzhouChina

Personalised recommendations