Skip to main content
SpringerLink
Log in

Discovering discriminative patches for free-hand sketch analysis

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Since the ancient times, free-hand sketch has been widely used as an effective and convenient intermediate means to express human thoughts and highly diverse objects in reality. In recent years, a great quantity of researchers realized the significance of sketch and gradually focused on sketch-related problems, such as sketch-based image retrieval and recognition. Despite so many achievements, very few works concentrate on exploring the intrinsic factors which potentially influence the vivid degree of sketch. In this paper, we propose a weak supervised approach to discover the most discriminative patches for different categories of sketches, which perhaps grasp the key to a good free-hand sketch. In the beginning, we randomly extract tens of thousands of patches at multiple scales. After that, pyramid histogram of oriented gradient is calculated to represent these patches as an effective and uniform feature representation. To find the most discriminative patches for each class of sketches, we design an iterative detection process which combines cluster merging and discriminative ranking. The experimental results on the TU-Berlin sketch benchmark dataset demonstrate the effectiveness of the proposed method, as compared to other available approaches. Moreover, a reasonable analysis and discussion about good and bad sketches is provided based on the visual results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Battiato, S., Farinella, G.M., Giudice, O., Puglisi, G.: Aligning shapes for symbol classification and retrieval. Multim. Tools Appl., pp. 1–19 (2015) (in press)

  2. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–522 (2002)

    Article  Google Scholar 

  3. Borji, A., Itti, L.: Human vs. computer in scene and object recognition. IEEE Conf. Comput. Vis. Pattern Recogn., pp.113–120 (2014)

  4. Bozas, K., Izquierdo, E.: Large scale sketch based image retrieval using patch hashing. Adv. Vis. Comput. pp. 210–219 (2012)

  5. Cao, X., Zhang, H., Liu, S., Guo, X., Lin, L.: Sym-fish: A symmetry-aware flip invariant sketch histogram shape descriptor. IEEE Int. Conf. Comput. Vis., pp. 313–320 (2013)

  6. Cao, Y., Wang, C., Zhang, L., Zhang, L.: Edgel index for large-scale sketch-based image search. IEEE Conf. Comput. Vis. Pattern Recogn., pp. 761–768 (2011)

  7. Cao, Y., Wang, H., Wang, C., Li, Z., Zhang, L., Zhang, L.: Mindfinder: interactive sketch-based image search on millions of images. ACM Int. Conf. Multim., pp. 1605–1608 (2010)

  8. Chalechale, A., Naghdy, G., Mertins, A.: Sketch-based image matching using angular partitioning. IEEE Trans. Syst. Man Cybernet. Part A Syst. Humans 35(1), 28–41 (2005)

    Article  Google Scholar 

  9. Chen, T., Cheng, M.M., Tan, P., Shamir, A., Hu, S.M.: Sketch2photo: internet image montage. ACM Trans. Graph. 28(5), 124 (2009)

    Google Scholar 

  10. Chen, T., Tan, P., Ma, L.Q., Cheng, M.M., Shamir, A., Hu, S.M.: Poseshop: human image database construction and personalized content synthesis. IEEE Trans. Vis. Comput. Graph. 19(5), 824–837 (2013)

    Article  Google Scholar 

  11. Collomosse, J.P., McNeill, G., Qian, Y.: Storyboard sketches for content based video retrieval. IEEE Int. Conf. Comput. Vis., pp. 245–252 (2009)

  12. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. IEEE Conf. Comput. Vis. Pattern Recognit. 1, 886–893 (2005)

    Google Scholar 

  13. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 5 (2008)

    Article  Google Scholar 

  14. Dixon, D., Prasad, M., Hammond, T.: icandraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. ACM SIGCHI Conf. Human Factors Comput. Syst., pp. 897–906 (2010)

  15. Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes paris look like paris? ACM Trans. Graph. 31(4) (2012)

  16. Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. 31(4), 44 (2012)

    Google Scholar 

  17. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: A descriptor for large scale image retrieval based on sketched feature lines. EUROGRAPHICS Sympos. Sketch-Based Interfa. Model., pp. 29–36 (2009)

  18. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans. Vis. Comput.Graph.17(11), 1624–1636 (2011)

    Article  Google Scholar 

  19. Eitz, M., Richter, R., Hildebrand, K., Boubekeur, T., Alexa, M.: Photosketcher:interactive sketch-based image synthesis. IEEE Comput. Graph. Appl. 31(6), 56–66 (2011)

    Article  Google Scholar 

  20. Guo, C.E., Zhu, S.C., Wu, Y.N.: Primal sketch: Integrating structure and texture. Comput. Vis. Image Underst. 106(1), 5–19 (2007)

  21. Guo, J., Wang, C., Chao, H.: Building effective representations for sketch recognition.In: AAAI Conference on Artificial Intelligence pp. 3776–3782 (2015)

  22. Hammond, T., Davis, R.: Ladder, a sketching language for user interface developers. Comput. Graph. 29(4), 518–532 (2005)

    Article  Google Scholar 

  23. Herot, C.F.: Graphical input through machine recognition of sketches. ACM SIGGRAPH Comput. Graph. 10, 97–102 (1976)

    Article  Google Scholar 

  24. Hu, R.: Free-hand sketch based image and video retrieval. ACM Int. Conf. Multim., pp. 1697–1698 (2010)

  25. Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117(7), 790–806 (2013)

    Article  Google Scholar 

  26. Hu, R., James, S., Wang, T., Collomosse, J.: Markov random fields for sketch based video retrieval. ACM Int. Conf. Multim. Retriev., pp. 279–286 (2013)

  27. Huang, Z., Fu, H., Lau, R.W.: Data-driven segmentation and labeling of freehand sketches. ACM Trans. Graph. 33(6), 175 (2014)

    Article  Google Scholar 

  28. James, S., Collomosse, J.: Interactive video asset retrieval using sketched queries. ACM Eur. Conf. Vis. Media Prod., p. 11 (2014)

  29. Jayasumana, S., Hartley, R., Salzmann, M., Li, H., Harandi, M.: Optimizing over radial kernels on compact manifolds. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 3802–3809 (2014)

  30. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. IEEE Conf. Comput. Vis. Pattern Recognit. 2, 2169–2178 (2006)

    Google Scholar 

  31. Lee, Y.J., Zitnick, C.L., Cohen, M.F.: Shadowdraw: real-time user guidance for freehand drawing. ACM Trans. Graph. 30(4), 27 (2011)

    Article  Google Scholar 

  32. Li, B., Lu, Y., Godil, A., Schreck, T., Bustos, B., Ferreira, A., Furuya, T., Fonseca, M.J., Johan, H., Matsuda, T., et al.: A comparison of methods for sketch-based 3d shape retrieval. Comput. Vis. Image Underst. 119, 57–80 (2014)

    Article  Google Scholar 

  33. Li, Y., Hospedales, T.M., Song, Y.Z., Gong, S.: Free-hand sketch recognition by multi-kernel feature learning. Comput. Vis. Image Underst. 137, 1–11 (2015)

    Article  Google Scholar 

  34. Li, Y., Song, Y.Z., Gong, S.: Sketch recognition by ensemble matching of structured features. Br. Mach. Vis. Conf. (2013)

  35. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  36. Ma, C., Yang, X., Zhang, C., Ruan, X., Yang, M.H., Coporation, O.: Sketch retrieval via dense stroke features. Br. Mach. Vis. Conf. 2, 3 (2013)

    Google Scholar 

  37. Ma, C.X., Liu, Y.J., Wang, H.A., Teng, D.X., Dai, G.Z.: Sketch-based annotation and visualization in video authoring. IEEE Trans. Multim. 14(4), 1153–1165 (2012)

    Article  Google Scholar 

  38. Marvaniya, S., Bhattacharjee, S., Manickavasagam, V., Mittal, A.: Drawing an automatic sketch of deformable objects using only a few images. Eur. Conf. Comput. Vis. Workshops Demonstr., pp. 63–72 (2012)

  39. Matusiak, S., Daoudi, M., Blu, T., Avaro, O.: Sketch-based images database retrieval. Adv. Multim. Inf. Syst., pp. 185–191 (1998)

  40. Paulson, B., Hammond, T.: Paleosketch: accurate primitive sketch recognition and beautification. ACM Int. Conf. Intel. User Interf., pp. 1–10 (2008)

  41. Pu, J., Gur, D.: Automated freehand sketch segmentation using radial basis functions. Comput. Aided Des. 41(12), 857–864 (2009)

    Article  Google Scholar 

  42. Qi, Y., Guo, J., Song, Y.Z., Xiang, T., Zhang, H., Tan, Z.H.: Im2sketch: Sketch generation by unconflicted perceptual grouping. Neurocomputing. 165, 338–349 (2015)

    Article  Google Scholar 

  43. Saavedra, J.M., Bustos, B.: An improved histogram of edge local orientations for sketch-based image retrieval. In: Pattern Recognition, pp. 432–441 (2010)

  44. Schneider, R.G., Tuytelaars, T.: Sketch classification and classification-driven analysis using fisher vectors. ACM Trans. Graph. 33(6), 174 (2014)

    Article  Google Scholar 

  45. Sezgin, T.M., Davis, R.: Hmm-based efficient sketch recognition. ACM Int. Conf. Intell. User Interf., pp. 281–283 (2005)

  46. Shao, T., Xu, W., Yin, K., Wang, J., Zhou, K., Guo, B.: Discriminative sketch-based 3d model retrieval via robust shape matching. Comput. Graph. Forum. 30(7), 2011–2020 (2011)

    Article  Google Scholar 

  47. Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1–8 (2007)

  48. Shilane, P., Min, P., Kazhdan, M., Funkhouser, T.: The princeton shape benchmark. IEEE Shape Model. Appl., pp. 167–178 (2004)

  49. Singh, S., Gupta, A., Efros, A.: Unsupervised discovery of mid-level discriminative patches. Eur. Conf. Comput. Vis., pp. 73–86 (2012)

  50. Singhai, N., Shandilya, S.K.: A survey on: content based image retrieval systems. Int. J. Comput. Appl. 4(2), 22–26(2010)

  51. Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)

    Article  Google Scholar 

  52. Sun, X., Wang, C., Xu, C., Zhang, L.: Indexing billions of images for sketch-based retrieval. ACM Int. Conf. Multim., pp. 233–242 (2013)

  53. Sun, Z., Wang, C., Zhang, L., Zhang, L.: Free hand-drawn sketch segmentation. Eur. Conf. Comput. Vis., pp. 626–639 (2012)

  54. Sun, Z., Wang, C., Zhang, L., Zhang, L.: Query-adaptive shape topic mining for hand-drawn sketch recognition. ACM Int. Conf. Multim., pp. 519–528 (2012)

  55. Sutherland, I.E.: Sketch pad a man-machine graphical communication system. ACM SHARE Des. Autom. Workshop, pp. 6–329 (1964)

  56. Tseng, K.Y., Lin, Y.L., Chen, Y.H., Hsu, W.H.: Sketch-based image retrieval on mobile devices using compact hash bits. ACM Int. Conf. Multim., pp. 913–916 (2012)

  57. Xiao, B., Gao, X., Tao, D., Yuan, Y., Li, J.: Photo-sketch synthesis and recognition based on subspace learning. Neurocomputing 73(4), 840–852 (2010)

    Article  Google Scholar 

  58. Xiao, C., Wang, C., Zhang, L., Zhang, L.: Sketch-based image retrieval via shape words. ACM Int. Conf. Multim. Retriev., pp. 571–574 (2015)

  59. Xu, K., Chen, K., Fu, H., Sun, W.L., Hu, S.M.: Sketch2scene: sketch-based co-retrieval and co-placement of 3d models. ACM Trans. Graph. 32(4), 123 (2013)

    Article  Google Scholar 

  60. Yoon, S.M., Scherer, M., Schreck, T., Kuijper, A.: Sketch-based 3d model retrieval using diffusion tensor fields of suggestive contours. ACM Int. Conf. Multim., pp. 193–200 (2010)

  61. Zhao, S., Chen, L., Yao, H., Zhang, Y., Sun, X.: Strategy for dynamic 3d depth data matching towards robust action retrieval. Neurocomputing 151, 533–543 (2015)

    Article  Google Scholar 

  62. Zheng, Y., Yao, H., Sun, X., Zhao, S.: Distinctive action sketch. IEEE Int. Conf. Image Process., pp. 576–580 (2015)

  63. Zhou, R., Chen, L., Zhang, L.: Sketch-based image retrieval on a large scale database. ACM Int. Conf. Multim., pp. 973–976 (2012)

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 61472103) and Key Program (No. 61133003).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

    Ying Zheng, Hongxun Yao, Sicheng Zhao & Yasi Wang

Authors
  1. Ying Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongxun Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sicheng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yasi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongxun Yao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, Y., Yao, H., Zhao, S. et al. Discovering discriminative patches for free-hand sketch analysis. Multimedia Systems 23, 691–701 (2017). https://doi.org/10.1007/s00530-016-0507-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-016-0507-8

Keywords

Search

Navigation