Robust Sketch-Based Image Retrieval by Saliency Detection

  • Xiao Zhang
  • Xuejin ChenEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9516)


Sketch-based image retrieval (SBIR) has been extensively studied for decades because sketch is one of the most intuitive ways to describe ideas. However, the large expressional gap between hand-drawn sketches and natural images with small-scale complex structures is the fundamental challenge for SBIR systems. We present a novel framework to efficiently retrieve images with a query sketch based on saliency detection. In order to extract primary contours of the scene and depress textures, a hierarchical saliency map is computed for each image. Object contours are extracted from the saliency map instead of the original natural image. Histograms of oriented gradients (HOG) are extracted at multiple scales on a dense gradient field. Using a bag-of-visual-words representation and an inverted index structure, our system efficiently retrieves images by sketches. The experimental results conducted on a dataset of 15 k photographs demonstrate that our method performs well for a wide range of natural scenes.


Sketch Image retrieval Saliency Multi-scale 



We would like to thank the anonymous reviewers. This work was supported by the National Natural Science Foundation of China (NSFC) under Nos. 61472377 and 61331017, and the Fundamental Research Funds for the Central Universities under No. WK2100060011.


  1. 1.
    Canny, J.: A computational approach to edge detection. IEEE TPAMI 6, 679–698 (1986)CrossRefGoogle Scholar
  2. 2.
    Cao, Y., Wang, H., Wang, C., Li, Z., Zhang, L., Zhang, L.: Mindfinder: interactive sketch-based image search on millions of images. In: ACMMM (2010)Google Scholar
  3. 3.
    Cao, Y., Wang, C., Zhang, L., Zhang, L.: Edgel index for large-scale sketch-based image search. In: IEEE CVPR, pp. 761–768 (2011)Google Scholar
  4. 4.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, pp. 886–893 (2005)Google Scholar
  5. 5.
    Dance, C., Willamowski, J., Fan, L., Bray, C., Csurka, G.: Visual categorization with bags of keypoints. In: ECCV International Workshop on Statistical Learning in Computer Vision (2004)Google Scholar
  6. 6.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: A descriptor for large scale image retrieval based on sketched feature lines. In: Eurographics Symposium on Sketch-Based Interfaces and Modeling, pp. 29–38 (2009)Google Scholar
  7. 7.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Comput. Graph. 34(5), 482–498 (2010)CrossRefGoogle Scholar
  8. 8.
    Hu, R., Barnard, M., Collomosse, J.: Gradient field descriptor for sketch based retrieval and localization. ICIP 10, 1025–1028 (2010)Google Scholar
  9. 9.
    Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. CVIU 117(7), 790–806 (2013)Google Scholar
  10. 10.
    Jacobs, C.E., Finkelstein, A.: Fast multiresolution image querying. In: ACM Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 277–286 (1995)Google Scholar
  11. 11.
    Parui, S., Mittal, A.: Similarity-Invariant sketch-based image retrieval in large databases. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 398–414. Springer, Heidelberg (2014) Google Scholar
  12. 12.
    Saavedra, J.M., Bustos, B.: Sketch-based image retrieval using keyshapes. Multimedia Tools Appl. 73(3), 2033–2062 (2014)CrossRefGoogle Scholar
  13. 13.
    Smith, J.R., Chang, S.F.: Visualseek: a fully automated content-based image query system. In: Proceedings of the Fourth ACM International Conference on Multimedia, pp. 87–98 (1997)Google Scholar
  14. 14.
    Tencer, L., Renakova, M., Cheriet, M.: Sketch-based retrieval of document illustrations and regions of interest. In: 2013 12th International Conference on Document Analysis and Recognition (ICDAR), pp. 728–732 (2013)Google Scholar
  15. 15.
    Wang, S., Zhang, J., Xu, T., Miao, Z.: Sketch-based image retrieval through hypothesis driven object boundary selection with HLR descriptor. IEEE Trans. Multimedia 17, 1045–1057 (2015)CrossRefGoogle Scholar
  16. 16.
    Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: CVPR (2013)Google Scholar
  17. 17.
    Zhou, R., Chen, L., Zhang, L.: Sketch-based image retrieval on a large scale database. In: Proceedings of the 20th ACM international conference on Multimedia, pp. 973–976 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application SystemUniversity of Science and Technology of ChinaHefeiChina

Personalised recommendations