Multimedia Tools and Applications

, Volume 76, Issue 21, pp 22019–22042 | Cite as

Combining pixel domain and compressed domain index for sketch based image retrieval

  • Carlos Alberto Fraga Pimentel Filho
  • Benjamin Bustos
  • Arnaldo de Albuquerque Araújo
  • Silvio Jamil Ferzoli Guimarães


Sketch-based image retrieval (SBIR) lets one express a precise visual query with simple and widespread means. In the SBIR approaches, the challenge consists in representing the image dataset features in a structure that allows one to efficiently and effectively retrieve images in a scalable system. We put forward a sketch-based image retrieval solution where sketches and natural image contours are represented and compared, in both, the compressed-domain of wavelet and in the pixel domain. The query is efficiently performed in the wavelet domain, while effectiveness refinements are achieved using the pixel domain to verify the spatial consistency between the sketch strokes and the natural image contours. Also, we present an efficient scheme of inverted lists for sketch-based image retrieval using the compressed-domain of wavelets. Our proposal of indexing presents two main advantages, the amount of the data to compute the query is smaller than the traditional method while it presents a better effectiveness.


Sketch-based image retrieval Multimedia indexing Scalability 



The authors are grateful to CAPES/COFECUB, FAPEMIG (PPM-006-16), CNPq (307062/2016-3) and PUC Minas for the financial support to this work.


  1. 1.
    Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916. doi: 10.1109/TPAMI.2010.161 CrossRefGoogle Scholar
  2. 2.
    Brogefors G (1988) Hierarchical chamfer matching: a parametric edge matching algorithm. IEEE Trans Pattern Anal Mach Intell 10(6):849–865. doi: 10.1109/34.9107 CrossRefGoogle Scholar
  3. 3.
    Bui T, Collomosse J (2015) Scalable sketch-based image retrieval using color gradient features. In: The IEEE international conference on computer vision (ICCV) workshopsGoogle Scholar
  4. 4.
    Cao Y, Wang C, Zhang L, Zhang L (2011) Edgel index for large-scale sketch-based image search. In: CVPR, pp 761–768Google Scholar
  5. 5.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) - Volume 1 - Volume 01, CVPR ’05. IEEE Computer Society, Washington, pp 886–893. doi: 10.1109/CVPR.2005.177
  6. 6.
    Davis J, Goadrich M (2006) The relationship between precision-recall and roc curves. ACM, New YorkCrossRefGoogle Scholar
  7. 7.
    Del Bimbo A, Pala P (1997) Visual image retrieval by elastic matching of user sketches. IEEE Trans Pattern Anal Mach Intell 19(2):121–132. doi: 10.1109/34.574790 CrossRefGoogle Scholar
  8. 8.
    DeVore RA, Jawerth BD, Lucier BJ (1992) Image compression through wavelet transform coding. IEEE Trans Inf Theory 38(2):719–746MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?. ACM Trans Graph 31(4):44:1–44:10. doi: 10.1145/2185520.2185540 Google Scholar
  10. 10.
    Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636CrossRefGoogle Scholar
  11. 11.
    Filho CAF, Araujó AA, Crucianu M, Gouet-Brunet V (2013) Sketch-finder: efficient and effective sketch-based retrieval for large image collections. In: Proceedings of the 2013 XXVI conference on graphics, patterns and Images, SIBGRAPI ’13. IEEE Computer Society, Washington, pp 234–241. doi: 10.1109/SIBGRAPI.2013.40
  12. 12.
    Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P (1995) Query by image and video content: the qbic system. Computer 28(9):23–32. doi: 10.1109/2.410146 CrossRefGoogle Scholar
  13. 13.
    Hu R, Collomosse J (2013) A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput Vis Image Underst 117(7):790–806. doi: 10.1016/j.cviu.2013.02.005 CrossRefGoogle Scholar
  14. 14.
    Jacobs CE, Finkelstein A, Salesin DH (1995) Fast multiresolution image querying. In: Proceedings of SIGGRAPH 95, pp 277–286Google Scholar
  15. 15.
    Jain R (2008) The art of computer systems performance analysis:. Wiley India pvt limitedGoogle Scholar
  16. 16.
    Kurita T, Otsu N, Hirata K (1992) A sketch retrieval method for full color image database-query by visual example. In: 1992 IEEE Computer society conference on computer vision and applications (IAPR 1992). IEEE, pp 530–533Google Scholar
  17. 17.
    Lee YJ, Grauman K (2009) Shape discovery from unlabeled image collections. 2013 IEEE Conference on Computer Vision and Pattern Recognition 0:2254–2261. doi: 10.1109/CVPRW.2009.5206698 Google Scholar
  18. 18.
    Liu MY, Tuzel O, Veeraraghavan A, Chellappa R (2010) Fast directional chamfer matching. In: CVPR. IEEE Computer societyGoogle Scholar
  19. 19.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110. doi: 10.1023/B:VISI.0000029664.99615.94 CrossRefGoogle Scholar
  20. 20.
    Ma C, Yang X, Zhang C, Ruan X, Yang MH (2016) Sketch retrieval via local dense stroke features. Image Vis Comput 46:64–73. doi: 10.1016/j.imavis.2015.11.007. CrossRefGoogle Scholar
  21. 21.
    Polsley S, Ray J, Hammond T (2017) Sketchseeker: Finding similar sketches. IEEE Transactions on Human-Machine Systems 47(2):194–205. doi: 10.1109/THMS.2017.2649684 CrossRefGoogle Scholar
  22. 22.
    Qi Y, Song Y, Zhang H, Liu J (2016) Sketch-based image retrieval via siamese convolutional neural network. In: 2016 IEEE international conference on image processing, ICIP 2016, Phoenix, AZ, USA, September 25-28, 2016, pp 2460–2464. doi: 10.1109/ICIP.2016.7532801
  23. 23.
    Qian X, Tan X, Zhang Y, Hong R, Wang M (2016) Enhancing sketch-based image retrieval by re-ranking and relevance feedback. IEEE Trans Image Process 25(1):195–208. doi: 10.1109/TIP.2015.2497145 MathSciNetCrossRefGoogle Scholar
  24. 24.
    Robertson S (2004) Understanding inverse document frequency: on theoretical arguments for idf. J Doc 60:2004CrossRefGoogle Scholar
  25. 25.
    Robertson S, Zaragoza H, Taylor M (2004) Simple bm25 extension to multiple weighted fields. In: Proceedings of the thirteenth ACMinternational conference on information and knowledge management, CIKM ’04. ACM, New York, pp 42–49, doi: 10.1145/1031171.1031181
  26. 26.
    Saavedra J, Bustos B (2010) An improved histogram of edge local orientations for sketch-based image retrieval. In: Goesele M, Roth S, Kuijper A, Schiele B, Schindler K (eds) Pattern recognition, lecture notes in computer science, vol 6376. Springer, Berlin Heidelberg, pp 432–441Google Scholar
  27. 27.
    Saavedra JM (2014) Sketch based image retrieval using a soft computation of the histogram of edge local orientations (s-HELO). In: 2014 IEEE International conference on image processing, ICIP 2014, Paris, France, October 27-30, 2014, pp 2998–3002. doi: 10.1109/ICIP.2014.7025606
  28. 28.
    Saavedra JM (2015) Rst-shelo: sketch-based image retrieval using sketch tokens and square root normalization. Multimedia Tools Appl 1–21Google Scholar
  29. 29.
    Saavedra JM, Barrios JM (2015) Sketch based image retrieval using learned keyshapes (LKS). In: Proceedings of the british machine vision conference 2015, BMVC, 2015, Swansea, UK, September 7-10, 2015, pp 164.1–164.11. doi: 10.5244/C.29.164
  30. 30.
    Salton G, McGill MJ (1986) Introduction to modern information retrieval. McGraw-Hill, Inc., New YorkMATHGoogle Scholar
  31. 31.
    Shannon C (1948) A mathematical theory of communication. Bell Syst Tech J 27(379-423):623–656MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: Proceedings of the 9th IEEE international conference on computer vision - vol. 2, ICCV ’03. IEEE Computer Society, Washington, p 1470Google Scholar
  33. 33.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380. doi: 10.1109/34.895972 CrossRefGoogle Scholar
  34. 34.
    Srinivas M, Patnaik LM (1994) Adaptive probabilities of crossover and mutation in genetic algorithms. IEEE Trans Syst Man Cybern 24(4):656–667CrossRefGoogle Scholar
  35. 35.
    Stenger BDR (2004) Model-based hand tracking using a hierarchical bayesian filterGoogle Scholar
  36. 36.
    Sun X, Wang C, Xu C, Zhang L (2013) Indexing billions of images for sketch-based retrieval. In: Proceedings of the 21st ACM international conference on multimedia, MM ’13. ACM, New York, pp 233-242. doi: 10.1145/2502081.2502281
  37. 37.
    Tseng KY, Lin YL, Chen YH, Hsu WH (2012) Sketch-based image retrieval on mobile devices using compact hash bits. In: Proceedings of the 20th ACM International Conference on Multimedia, MM ’12. ACM, New York, pp 913–916Google Scholar
  38. 38.
    Venters CC, Hartley RJ, Hewitt WT (2005) Content-based image retrieval query paradigms. In: Encyclopedia of information science and technology (i), pp 556–563Google Scholar
  39. 39.
    Xiao C, Wang C, Zhang L, Zhang L (2015) Sketch-based image retrieval via shape words. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, Shanghai, China, June 23-26, 2015, pp 571–574. doi: 10.1145/2671188.2749360

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Carlos Alberto Fraga Pimentel Filho
    • 1
    • 3
  • Benjamin Bustos
    • 2
  • Arnaldo de Albuquerque Araújo
    • 3
  • Silvio Jamil Ferzoli Guimarães
    • 1
  1. 1.Department of Computer Science/VIPLABPontifical Catholic University of Minas Gerais (PUC Minas)Belo HorizonteBrazil
  2. 2.Department of Computer ScienceUniversity of ChileSantiagoChile
  3. 3.Department of Computer Science/NPDIFederal University of Minas Gerais (UFMG)Belo HorizonteBrazil

Personalised recommendations