Skip to main content
SpringerLink
Log in

A sketch recognition method based on transfer deep learning with the fusion of multi-granular sketches

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Most of existing sketch recognition methods focus on the contour/shape of whole sketches. They ignore different granularities of sketches during sketching. Stroke sequences of sketches often demonstrate the change of various granularities. In the progress of sketching, a coarser-grained contour gradually changes to a finer-grained object. Different granularities of sketch imply different levels of semantic information and play different roles in sketch recognition. In this paper, a transfer-deep-learning-based sketch recognition method--“sketch-transfer-net” is proposed. Sketch-transfer-net designs a novel fine-tuning strategy to use different granular sketches to fine-tune different layers of neural network. The extensive comparative experiments show that the proposed sketch-transfer-net can capture descriptive information of various granular sketches and therefore improve the performance of sketch recognition. In addition, the novel fine-turning strategy could weaken the negative effect in transfer learning and enable CNNs to be well trained on small sketch datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Alex K, Ilya S, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  2. Bonilla EV, Chai KMA, Williams CKI (2007) Multi-task Gaussian process prediction[C]. In: Conference on neural information processing systems. Vancouver. British Columbia, Canada

  3. Cao X, Zhang H, Liu S, Sym-fish GX (2013) A symmetry-aware flip invariant sketch histogram shape descriptor. In: Computer vision (ICCV), 2013 IEEE international conference on. IEEE, pp 313–320

  4. Carneiro G, Jepson AD (2004) Pruning local feature correspondences using shape context[C]. In: Proceedings of the 17th international conference on pattern recognition. IEEE Computer Society Press, Los Alamitos, pp 16–19

  5. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection[C]. In: 2005 IEEE computer society conference on computer vision and pattern recognition. San Diego, CA, USA, pp 886–893

  6. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors[J]. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  7. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?[J]. ACM Trans Graph 31(4):44–54

    Google Scholar 

  8. Evgeniou T, Pontil M (2004) Regularized multi-task learning[C]. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining. Seattle, Washington, USA, pp 109–117

  9. Finkel JR, Manning CD (2010) Hierarchical Bayesian domain adaptation[C]. In: Human language technologies: conference of the north American chapter of the Association of Computational Linguistics. Boulder, Colorado, pp 602–610

  10. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks[C]. In: Proceedings of the 14th international conference on artificial intelligence and statistics. MIT Press, Nevada, pp 315–323

  11. Goodfellow IJ, Wardefarley D, Mirza M, Courville A, Bengio Y (2017) Maxout networks[OL]. http://arxiv.org/abs/1302.4389v4

  12. Klare B, Li Z, Jain AK (2011) Matching forensic sketches to mug shot photos[J]. IEEE Trans Pattern Anal Mach Intell 33(3):639–646

    Article  Google Scholar 

  13. Lecun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition[J]. Neural Comput 1(4):541–551

    Article  Google Scholar 

  14. Li Y, Song YZ, Gong S (2013) Sketch recognition by ensemble matching of structured features. In: British machine vision conference (BMVC), pp 35.1–35.11

  15. Li Y, Hospedales TM, Song YZ, Gong S (2015) Free-hand sketch recognition by multi-kernel feature learning[J]. Comput Vis Image Underst 137(8):1–11

    Google Scholar 

  16. Long M, Wang J, Ding G, Cheng W, Zhang X, Wang W (2012) Dual transfer learning. In: Proceedings of SIAM international conference on data mining (SDM), Anaheim, USA, pp 1947–1961. (SDM 2012)

  17. Long M, Wang J, Ding G, Shen D, Yang Q (2012) Transfer learning with graph co-regularization. Twenty-sixth AAAI conference on artificial intelligence, pp 1033–1039

  18. Lowe DG (2004) Distinctive image features from scale-invariant keypoints[J]. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  19. Schneider RG, Tuytelaars T (2014) Sketch classification and classification-driven analysis using fisher vectors[J]. ACM Trans Graph 33(6):174–183

    Article  Google Scholar 

  20. Seddati O, Dupont S, Mahmoudi S (2015) Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: Content-based multimedia indexing (CBMI), 2015 13th international workshop on. IEEE, pp 1–6

  21. Simonyan K, Zisserman A (2017) Very deep convolutional networks for large-scale image recognition[OL]. http://arxiv.org/abs/1409.1556

  22. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting[J]. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  23. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. arXiv preprint arXiv:1409.4842

    Google Scholar 

  24. Wang F, Kang L, Li Y (2015) Sketch-based 3d shape retrieval using convolutional neural networks[C]. In: Proceedings of the 2015 IEEE conference on computer vision and pattern recognition. IEEE Computer Society Press, Los Alamitos, pp 1875–1883

  25. Weiss K, Khoshgoftaar TM, Wang DD (2016) A survey of transfer learning[J]. Journal of Big Data 3(1):9

    Article  Google Scholar 

  26. Yang Y, Hospedales TM (2015) Deep neural networks for sketch recognition. arXiv preprint arXiv:1501.07873v1

    Google Scholar 

  27. Yesilbek KT, Sen C, Cakmak S, Sezgin TM (2015) SVM-based sketch recognition: which hyperparameter interval to try? In: Proceedings of the workshop on sketch-based interfaces and modeling. Eurographics Association, pp 117–121

  28. Yu Q, Yang Y, Song YZ, Xiang T, Hospedales TM (2017) Sketch-a-net that beats humans[OL]. http://arxiv.org/abs/1501.07873v3

Download references

Acknowledgments

This work is supported by National Natural Science Foundation of China (Grant Nos. 61602004) to Dr. Zhao, Natural Science Foundation of the Education Department of Anhui Province (Grant No. KJ2016A041, KJ2017A011), and in part supported by NSF CNS-1305302 and Texas State Research Enhancement Program grant to Dr. Lu.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Anhui University, Hefei, 230039, China

    Peng Zhao, Yang Liu & Benpeng Xu

  2. Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, Anhui University, Hefei, 230601, China

    Peng Zhao

  3. Department of Computer Science, Texas State University, San Marcos, TX, 78666, USA

    Yijuan Lu

Authors
  1. Peng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yijuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Benpeng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peng Zhao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, P., Liu, Y., Lu, Y. et al. A sketch recognition method based on transfer deep learning with the fusion of multi-granular sketches. Multimed Tools Appl 78, 35179–35193 (2019). https://doi.org/10.1007/s11042-019-08216-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08216-6

Keywords

Search

Navigation