Uncovering the Effect of Visual Saliency on Image Retrieval

Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 772)


Visual saliency modeling has achieved impressive performance for boosting vision-related systems. Intuitively, it should be beneficial to content-based image retrieval task, since the users’ query attention is heavily related to the region of interests (ROI) in query image. Although some approaches have been proposed to combine image retrieval systems with visual saliency models, no a comprehensive and systematic study is made to discover the effect of different saliency models on image retrieval in a qualitative and quantitative manner. In this paper, we attempt to concretely investigate the diversity of visual saliency models on image retrieval by making extensive experiments based on nine popular saliency models. To cooperatively mining the complementary information from different models, we also propose a novel approach to effectively involve visual saliency into image retrieval systems by a learning process. Extensive experiments on a generally used image benchmark demonstrate that the new image retrieval system remarkably outperforms the original one and other traditional ones.


Visual saliency Image retrieval Evaluation 



This work was supported in part by National Natural Science Foundation of China (No. 61572065, No. 61532005, No. 61370113), National Key Research and Development of China (No. 2016YFB0800404), Joint Fund of Ministry of Education of China and China Mobile (No. MCM20160102).


  1. 1.
    Acharya, S., Devi, M.R.V.: Image retrieval based on visual attention model. Procedia Eng. 30, 542–545 (2012)CrossRefGoogle Scholar
  2. 2.
    Babenko, A., Lempitsky, V.: Efficient indexing of billion-scale datasets of deep descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2055–2063 (2016)Google Scholar
  3. 3.
    Boato, G., Dang-Nguyen, D.T., Muratov, O., Alajlan, N., Natale, F.G.B.D.: Exploiting visual saliency for increasing diversity of image retrieval results. Multimedia Tools Appl. 75(10), 1–22 (2015)Google Scholar
  4. 4.
    Chiappino, S., Mazzu, A., Marcenaro, L., Regazzoni, C.S.: A bio-inspired logical process for saliency detections in cognitive crowd monitoring. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2110–2114 (2015)Google Scholar
  5. 5.
    Dai, Q., Li, J., Wang, J., Jiang, Y.G.: Binary optimized hashing. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 1247–1256. ACM (2016)Google Scholar
  6. 6.
    Duan, L., Ma, W., Miao, J., Zhang, X.: Visual saliency based bag of phrases for image retrival. In: The ACM Siggraph International Conference, pp. 243–246 (2014)Google Scholar
  7. 7.
    Fang, S., Li, J., Tian, Y., Huang, T., Chen, X.: Learning discriminative subspaces on random contrasts for image saliency analysis. IEEE Trans. Neural Netw. Learn. Syst. PP(99), 1–14 (2016)Google Scholar
  8. 8.
    Garciadiaz, A., Leborn, V., Fdezvidal, X.R., Pardo, X.M.: On the relationship between optical variability, visual saliency, and eye fixations: a computational approach. J. Vis. 12(6), 17 (2012)CrossRefGoogle Scholar
  9. 9.
    Giouvanakis, E., Kotropoulos, C.: Saliency map driven image retrieval combining the bag-of-words model and PLSA. In: 19th International Conference on Digital Signal Processing, pp. 280–285 (2014)Google Scholar
  10. 10.
    Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In: Advances in Neural Information Processing Systems, pp. 545–552 (2007)Google Scholar
  11. 11.
    Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Patt. Anal. Mach. Intell. 20(11), 1254–1259 (1998)CrossRefGoogle Scholar
  12. 12.
    Jegou, H., Douze, M., Schmid, C.: Hamming embedding and weak geometric consistency for large scale image search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 304–317. Springer, Heidelberg (2008). CrossRefGoogle Scholar
  13. 13.
    Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. ArXiv Preprint ArXiv:1702.08734 (2017)
  14. 14.
    Lee, G., Tai, Y.W., Kim, J.: Deep saliency with encoded low level distance map and high level features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 660–668 (2016)Google Scholar
  15. 15.
    Li, J., Duan, L.Y., Chen, X., Huang, T., Tian, Y.: Finding the secret of image saliency in the frequency domain. IEEE Trans. Patt. Anal. Mach. Intell. 37(12), 1 (2015)CrossRefGoogle Scholar
  16. 16.
    Li, J., Tian, Y., Huang, T.: Visual saliency with statistical priors. Int. J. Comput. Vis. 107(3), 239–253 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Li, J., Levine, M.D., An, X., Xu, X., He, H.: Visual saliency based on scale-space analysis in the frequency domain. IEEE Trans. Patt. Anal. Mach. Intell. 35(4), 996–1010 (2013)CrossRefGoogle Scholar
  18. 18.
    Liang, X., Xu, C., Shen, X., Yang, J., Tang, J., Lin, L., Yan, S.: Human parsing with contextualized convolutional neural network. IEEE Trans. Patt. Anal. Mach. Intell. 39(1), 115–127 (2017)CrossRefGoogle Scholar
  19. 19.
    Liao, L., Wei, S., Zhao, Y., Gu, G.: Improving the similarity estimation via score distribution. In: IEEE International Conference on Multimedia and Expo, pp. 1–6 (2016)Google Scholar
  20. 20.
    Liu, G.H., Yang, J.Y., Li, Z.Y.: Content-based image retrieval using computational visual attention model. Patt. Recogn. 48(8), 2554–2566 (2015)CrossRefGoogle Scholar
  21. 21.
    Liu, R., Zhao, Y., Wei, S., Zhu, Z., Liao, L., Qiu, S.: Indexing of CNN features for large scale image search. ArXiv Preprint ArXiv:1508.00217 (2015)
  22. 22.
    Liu, Y.J., Yu, C.C., Yu, M.J., He, Y.: Manifold SLIC: a fast method to compute content-sensitive superpixels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 651–659 (2016)Google Scholar
  23. 23.
    Papushoy, A., Bors, A.G.: Image retrieval based on query by saliency content. Dig. Sig. Process. 36, 156–173 (2015)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Riche, N., Mancas, M., Gosselin, B., Dutoit, T.: Rare: a new bottom-up saliency model. In: 19th IEEE International Conference on Image Processing, pp. 641–644. IEEE (2012)Google Scholar
  25. 25.
    Wan, S., Jin, P., Yue, L.: An approach for image retrieval based on visual saliency. In: International Conference on Image Analysis and Signal Processing, pp. 172–175 (2009)Google Scholar
  26. 26.
    Wei, S., Xu, D., Li, X., Zhao, Y.: Joint optimization toward effective and efficient image search. IEEE Trans. Cybern. 43(6), 2216–2227 (2013)CrossRefGoogle Scholar
  27. 27.
    Wen, Z., Gao, J., Luo, R., Wu, H.: Image Retrieval Based on Saliency Attention. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  28. 28.
    Zhang, D., Han, J., Li, C., Wang, J., Li, X.: Detection of co-salient objects by looking deep and wide. Int. J. Comput. Vis. 120(2), 215–232 (2016)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Zhang, J., Sclaroff, S.: Exploiting surroundedness for saliency detection: a boolean map approach. IEEE Trans. Patt. Anal. Mach. Intell. 38(5), 889–902 (2016)CrossRefGoogle Scholar
  30. 30.
    Zheng, L., Wang, S., Wang, J., Tian, Q.: Accurate image search with multi-scale contextual evidences. Int. J. Comput. Vis. 120(1), 1–13 (2016)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Zhou, W., Li, H., Sun, J., Tian, Q.: Collaborative index embedding for image retrieval. IEEE Trans. Patt. Anal. Mach. Intell. PP, 1 (2017)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2017

Authors and Affiliations

  1. 1.Institute of Information ScienceBeijing Jiaotong UniversityBeijingChina
  2. 2.Beijing Key Laboratory of Advanced Information Science and Network TechnologyBeijing Jiaotong UniversityBeijingChina
  3. 3.State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and EngineeringBeihang UniversityBeijingChina

Personalised recommendations