Sketch-based image retrieval (SBIR) is an emergent research area with a variety of applications, specially when an example image is not available for querying. Moreover, making a sketch has become a very attractive and simple task due to the already ubiquitous touch-screen and mobile technologies. Although a sketch is a natural way for representing the structure of a thought object, it may easily get confused in a dataset with high variability turning the retrieval task a quite challenging problem. Indeed, the state-of-the-art methods still show low performance on diverse evaluation datasets. Thereby, a robust sketch descriptor together with a better strategy for representing regular images as sketches are demanded. In this work, we present RST-SHELO, and improved version of SHELO (Soft Histogram of Edge Logal Orientations), an efficient state-of-the-art method for describing sketches. The proposed improvements comes from two aspects: a better technique for obtaining sketch-like representations and a better normalization strategy of SHELO. For the first case, we propose to use the sketch token approach , aiming to detect image contours by means of mid-level features. For the second case, we demonstrate that a square root normalization positively affect the effectiveness on the retrieval task. Based on our improvements, we present new state-of-the-art performance. To validate our achievements, we have conducted diverse experiments using two public datasets, Flickr15K and Saavedra’s. Our results show an effectiveness gain of 62 % in the first and 5 % in the second dataset.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
Arandjelovic R, Zisserman A (2012) Three things everyone should know to improve object retrieval. In: Proceedings of the 2012 IEEE conference on computer vision and pattern recognition (CVPR), CVPR ’12, pp 2911–2918
Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(5):898–916
Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(4):509–522
Borgefors G (1988) Hierarchical chamfer matching: a parametric edge matching algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence 10(6):849–865
Cao Y, Wang C, Zhang L, Zhang L (2011) Edgel index for large-scale sketch-based image search. In: Proceedings of the 2011 IEEE conference on computer vision and pattern recognition. IEEE Computer Society, pp 761–768
Chalechale A, Naghdy G, Mertins A (2005) Sketch-based image matching using angular partitioning. IEEE Trans Syst Man Cybern Syst Hum 35(1):28–41
Chen T, Cheng MM, Tan P, Shamir A, Hu SM (2009) Sketch2photo: internet image montage. ACM Transactions on Graphics 28(5):124:1–124:10
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) - Volume 1 - Volume 01. IEEE Computer Society, pp 886–893
Del Bimbo A, Pala P (1997) Visual image retrieval by elastic matching of user sketches. IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (2):121–132
Eitz M, Hildebrand K, Boubekeur T, Alexa M (2009) A descriptor for large scale image retrieval based on sketched feature lines. In: Proceedings of the 6th Eurographics symposium on sketch-based interfaces and modeling, pp 29–36
Eitz M, Hildebrand K, Boubekeur T, Alexa M (2009) Photosketch: a sketch based image query and compositing system. In: SIGGRAPH 2009: Talks, SIGGRAPH ’09, pp 60:1–60:1
Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636
Fei-Fei L, Fergus R, Perona P (2004) Lening generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories
Felzenszwalb P, David M, Deva R (2008) A discriminatively trained, multiscale, deformable part model. In: International conference on computer vision and pattern recognition
Gonzalez R, Woods R (2008) Digital Image Processing, 3rd edn. Pearson Prentice Hall, New Jersey
Hu R, Collomosse J (2013) A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comp Vision Image Underst 117(7):790–806
Hu R, Barnard M, Collomosse J (2010) Gradient field descriptor for sketch based retrieval and localization. In: 17th IEEE International Conference on Image Processing (ICIP), pp 1025–1028
Hu R, Wang T, Collomosse J (2011) A bag-of-regions approach to sketch-based image retrieval. In: 18th IEEE international conference on image processing (ICIP), pp 3661–3664
Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 2, pp 2169–2178
Li B, Schreck T, Godil A, Alexa M, Boubekeur T, Bustos B, Chen J, Eitz M, Furuya T, Hildebrand K, Huang S, Johan H, Kuijper A, Ohbuchi R, Richter R, Saavedra JM, Scherer M, Yanagimachi T, Yoon GJ, Yoon SM (2012) SHREC’12 track: sketch-based 3D shape retrieval. In: Eurographics workshop on 3D object retrieval, pp 109–118
Lim J, Zitnick C, Dollar P (2013) Sketch tokens: a learned mid-level representation for contour and object detection. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 3158–3165
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(10):1615–1630
Saavedra JM (2014) Sketch based image retrieval using a soft computation of the histogram of edge local orientations (s-helo). In: International conference on image processing, ICIP’2014 (To appear)
Saavedra J, Bustos B (2010) An improved histogram of edge local orientations for sketch-based image retrieval. In: Pattern recognition, lecture notes in computer science, vol 6376, pp 432–441
Saavedra JM, Bustos B (2013) Sketch-based image retrieval using keyshapes
Sun Won C, Kwon Park D, Park SJ (2002) Efficient use of MPEG-7 edge histogram descriptor. Electronic and Telecomunications Research Institute Journal 24:23–30
Tola E, Lepetit V, Fua P (2010) Daisy: an efficient dense descriptor applied to wide-baseline stereo. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 32
We are grateful for financial support from two chilean institutions: CONICYT, through the projects PAI-781204025 and 14STIC-01, and CORFO-INNOVA, through the project 15ITE2-38948.
About this article
Cite this article
Saavedra, J.M. RST-SHELO: sketch-based image retrieval using sketch tokens and square root normalization. Multimed Tools Appl 76, 931–951 (2017). https://doi.org/10.1007/s11042-015-3076-5
- Sketch based image retrieval
- Histogram of orientations
- Sketch tokens