Skip to main content
SpringerLink
Log in

An extensive survey on traditional and deep learning-based face sketch synthesis models

  • Original Research
  • Published:
International Journal of Information Technology Aims and scope Submit manuscript

Abstract

In recent days, Face sketch synthesis (FSS) attracts various researchers for sketching the images to retrieve faces and in multimedia applications. The intention of FSS is to create a sketch for the image provided from a collection of sketch and photo images as the training set. Presently, the rise of deep learning (DL) models becomes useful in FSS because of its diverse benefits. As the FSS is employed in various applications, detailed experimentation to analyze the state of the art approaches methods is nontrivial. Though numerous FSS approaches are available, there is no review paper exist regarding the hierarchical classification of DL based FSS. Keeping this in mind, in this paper, we provide an extensive review of the available DL as well as conventional FSS techniques. We made a clear classification of the FSS techniques, and these are categorized into data-driven and model-driven methods. A comparative analysis of the reviewed techniques is made based on various aspects such as the objective, algorithms used, benefits, and performance measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Domingos P (2012) A few useful things to know about machine learning. Commun ACM 55(10):78. https://doi.org/10.1145/2347736.2347755

    Article  Google Scholar 

  2. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). San Diego, CA, USA, 20-26 June 2005: IEEE, pp 886–893. https://doi.org/10.1109/CVPR.2005.177

  3. Lowe D (2004) Distinctive image features from scale-invariant keypoints, cascade filtering approach. IJCV 60:91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  4. Bengio Y, LeCun Y (2007) Scaling learning algorithms towards, AI. In: Large-Scale Kernel Machines. MIT Press. http://www.iro.umontreal.ca/~lisa/pointeurs/bengio+lecun_chapter2007.pdf. Accessed 24 Jan 2019

  5. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. Pattern Anal Mach Intell IEEE Trans 35(8):1798–1828. https://doi.org/10.1109/TPAMI.2013.50

    Article  Google Scholar 

  6. Arel I, Rose DC, Karnowski TP (2010) Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE Comput Intell 5:13–18

    Article  Google Scholar 

  7. Hinton GE, Osindero S, The Y-W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  Google Scholar 

  8. Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. Proc Adv Neural Inf Proc 19:153–160. https://doi.org/10.7551/mitpress/7503.003.0024

    Article  Google Scholar 

  9. Larochelle H, Bengio Y, Louradour J, Lamblin P (2009) Exploring strategies for training deep neural networks. J Mach Learn Res 10:1–40

    MATH  Google Scholar 

  10. Salakhutdinov R, Hinton G (2009) Deep Boltzmann machines. In: Proceedings of International conference on artificial intelligence and statistics, pp 448–455

  11. Goodfellow I, Lee H, Le QV, Saxe A, Ng AY (2009) Measuring invariances in deep networks. In: Proceedings of Neural Information and Processing System, pp 646–654

  12. Dahl GE, Ranzato M, Mohamed A, Hinton GE (2010) Phone recognition with the mean-covariance restricted Boltzmann machine. In: Advances in Neural Information Processing Systems 23, pp 469–477

  13. Hinton G, Deng L, Yu D, Mohamed A-R, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath T, Dahl G, Kingsbury B (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. Signal Process Mag IEEE 29(6):82–97

    Article  Google Scholar 

  14. Seide F, Li G, Yu D (2011) Conversational speech transcription using context-dependent deep neural networks. In: Proceedings of Conference Int'l Speech Communication Association, pp 437–440

  15. Mohamed A-R, Dahl GE, Hinton G (2012) Acoustic modeling using deep belief networks. Audio Speech Lang Process IEEE Trans 20(1):14–22

    Article  Google Scholar 

  16. Dahl GE, Yu D, Deng L, Acero A (2012) Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. Audio Speech Lang Process IEEE Trans 20(1):30–42

    Article  Google Scholar 

  17. Krichevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25, pp 1106–1114

  18. Mikolov T et al (2011) Empirical evaluation and combination of advanced language modeling techniques. In: Twelfth Annual Conference of the International Speech Communication Association

  19. Socher R et al (2011) Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Advances in neural information processing systems, pp 801–809

  20. Bordes A et al (2012) Joint learning of words and meaning representations for open-text semantic parsing. In: Artificial Intelligence and Statistics

  21. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inform Process Syst. https://doi.org/10.1145/3065386

  22. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, Cham. https://doi.org/10.1007/978-3-319-46475-6_43

  23. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2016) Photo-realistic single image super-resolution using a generative adversarial network. Technical Report. arXiv preprint arXiv: 1609.04802v3

  24. Zhao W et al (2003) Face recognition: a literature survey. J ACM Comput Surv (CSUR) 35(4):399–458. https://doi.org/10.1145/954339.954342

    Article  MathSciNet  Google Scholar 

  25. Wang N, Tao D, Gao X, Li J (2014) A comprehensive survey to face hallucination. Int J Comput Vis 31:9–30

    Article  Google Scholar 

  26. Ding C, Tao D (2017) Trunk-branch ensemble convolutional neural networks for video-based face recognition. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2017.2700390

    Article  Google Scholar 

  27. Ouyang S, Hospedales T, Song YZ, Li X (2014) A survey on Heterogeneous face recognition: sketch, infra-red, 3D and low-resolution. arXiv:1409.5114

  28. Tang X, Wang X (2002) Face photo recognition using sketch. In: Proceedings of International Conference on Image Processing, vol 1. IEEE. https://doi.org/10.1109/ICIP.2002.1038008

  29. Tang X, Wang X (2003) Face sketch synthesis and recognition.In: Proceedings Ninth IEEE International Conference on Computer Vision. IEEE. https://doi.org/10.1109/ICCV.2003.1238414

  30. Liu Q et al (2005) A nonlinear approach for face sketch a synthesis and recognition. In: 2005 IEEE Computer Society conference on computer vision and pattern recognition (CVPR'05), vol 1. IEEE. https://doi.org/10.1109/CVPR.2005.39

  31. Klare B, Jain AK (2010) Sketch-to-photo matching: a feature-based approach. In: Biometric Technology for Human Identification VII, vol. 7667. International Society for Optics and Photonics. https://doi.org/10.1117/12.849821

  32. Zhou H, Kuang Z, Wong KYK (2012) Markov weight fields for face sketch synthesis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. https://doi.org/10.1109/CVPR.2012.6247788

  33. Zhang S, Gao X, Wang N, Li J, Zhang M (2015) Face sketch synthesis via sparse representation-based greedy search. IEEE Trans Image Process 24(8):2466–2477

    Article  MathSciNet  Google Scholar 

  34. Chang L et al (2011) Face sketch synthesis via multivariate output regression. In: International Conference on Human-Computer Interaction. Springer Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21602-2_60

  35. Zhang J et al (2011) Face sketch-photo synthesis based on support vector regression. In: 2011 18th IEEE International Conference on Image Processing IEEE. https://doi.org/10.1109/ICIP.2011.6115625

  36. Isola P, Zhu JY, Zhou T, Efros AA (2016) Image-to-image translation with conditional adversarial networks. https://arxiv.org/abs/1611.07004

  37. Zhong J, Gao X, Tian C (2007) Face sketch synthesis using e-hmm and selective ensemble. Speech Signal Process (ICASSP), Acoustics, pp 485–488

    Google Scholar 

  38. Zhang S, Gao X, Wang N, Li J, Zhang M (2015) Face sketch synthesis via Sparse Representation based greedy search. IEEE Trans Image Process 24:2466–2477

    Article  MathSciNet  Google Scholar 

  39. Lin D, Tang X (2006) Inter-modality face recognition. In: European conference on computer vision. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11744085_2

  40. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  41. Liu Q et al (2005) A nonlinear approach for face sketch synthesis and recognition. In: Computer vision and pattern recognition. 2005 IEEE Computer Society conference on computer vision and pattern recognition (CVPR'05), vol 1. IEEE

  42. Gao X, Wang N, Tao D, Li X (2012) Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans Circuits Syst Video Technol 22(8):1213–1226

    Article  Google Scholar 

  43. Zhou H, Kuang Z, Wong KYK (2012) Markov weight fields for face sketch synthesis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE

  44. Zhu M, Wang N (2016) A simple and fast method for face sketch synthesis. In: Proceedings of the International Conference on Internet Multimedia Computing and Service. ACM. https://doi.org/10.1145/3007669.3007679

  45. Wang N, Tao D, Gao X, Li X, Li J (2013) Transductive face sketch-photo synthesis. IEEE Trans Neural Netw Learn Syst 24(9):1364–1376

    Article  Google Scholar 

  46. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, Cham, pp 694–711

  47. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision. Springer, Cham, pp 391–407

  48. Zhang L, Lin L, Wu X, Ding S, Zhang L (2015) End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, pp 627–634

  49. Chen X et al (2016) Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems

  50. Dosovitskiy A, Brox T (2016) Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems

  51. Ledig C et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  52. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. arXiv preprint

  53. Zhang M et al (2018) Dual-transfer face sketch–photo synthesis. IEEE Trans Image Process 28(2):642–657. https://doi.org/10.1109/tip.2018.2869688

    Article  MathSciNet  MATH  Google Scholar 

  54. Jiang J et al (2018) Graph-regularized locality-constrained joint dictionary and residual learning for face sketch synthesis. IEEE Trans Image Process 28(2):628–641. https://doi.org/10.1109/TIP.2018.2870936

    Article  MathSciNet  MATH  Google Scholar 

  55. Zhang S et al (2018) Face sketch synthesis by multidomain adversarial learning. IEEE Trans Neural Netw Learn Syst 30(5):1419–1428. https://doi.org/10.1109/TNNLS.2018.2869574

    Article  Google Scholar 

  56. Zhang S et al (2015) Face sketch synthesis from a single photo-sketch pair. IEEE Trans Circuits Syst Video Technol 27(2):275–287. https://doi.org/10.1109/TCSVT.2015.2511482

    Article  Google Scholar 

  57. Wan W, Lee HJ (2019) A joint training model for face sketch synthesis. Appl Sci 9(9):1731. https://doi.org/10.3390/app9091731

    Article  Google Scholar 

  58. Galea C, Farrugia RA (2017) forensic face photo-sketch recognition using a deep learning-based architecture. IEEE Signal Process Lett 24(11):1586–1590. https://doi.org/10.1109/LSP.2017.2749266

    Article  Google Scholar 

  59. Peng C, Gao X, Wang N, Li J (2017) Superpixel-based face sketch-photo synthesis. IEEE Trans Circ Syst Video Technol 27(2):288–299. https://doi.org/10.1109/TCSVT.2015.2502861

    Article  Google Scholar 

  60. Wang N, Gao X, Sun L, Li J (2017) Bayesian face sketch synthesis. IEEE Trans Image Process Publ IEEE Signal Process Soc 26(3):1264–1274. https://doi.org/10.1109/TIP.2017.2651375

    Article  MathSciNet  MATH  Google Scholar 

  61. Zhang D, Lin L, Chen T, Wu X, Tan W, Izquierdo E (2017) Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans Image Process Publ IEEE Signal Process Soc 26(1):328–339. https://doi.org/10.1109/TIP.2016.2623485

    Article  MathSciNet  MATH  Google Scholar 

  62. Wang N, Gao X, Sun L, Li J (2018) Anchored neighborhood index for face sketch synthesis. IEEE Trans Circuits Syst Video Technol 28(9):2154–2163. https://doi.org/10.1109/TCSVT.2017.2709465

    Article  Google Scholar 

  63. Zhang M, Li J, Wang N, Gao X (2018) Compositional model-based sketch generator in facial entertainment. IEEE Trans Cybern 48(3):904–915. https://doi.org/10.1109/TCYB.2017.2664499

    Article  Google Scholar 

  64. Bae S, Din NU, Javed K, Yi J (2019) Efficient generation of multiple sketch styles using a single network. IEEE Access 7:100666–100674. https://doi.org/10.1109/access.2019.2931544

    Article  Google Scholar 

  65. Zhang M, Zhang J, Chi Y, Li Y, Wang N, Gao X (2019) Cross-domain face sketch synthesis. IEEE Access 7:98866–98874. https://doi.org/10.1109/ACCESS.2019.2931012

    Article  Google Scholar 

  66. Zhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas DN (2019) StackGAN++: realistic image synthesis with stacked generative adversarial networks. IEEE Trans Pattern Anal Mach Intell 41(8):1947–1962. https://doi.org/10.1109/TPAMI.2018.2856256

    Article  Google Scholar 

  67. Zhang M, Wang R, Gao X, Li J, Tao D (2019) Dual-transfer face sketch-photo synthesis. IEEE Trans Image Process 28(2):642–657. https://doi.org/10.1109/TIP.2018.2869688

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Vignan’s Foundation for Science, Technology & Research, Guntur, Andhra Pradesh, India

    Narasimhula Balayesu & Hemantha Kumar Kalluri

Authors
  1. Narasimhula Balayesu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hemantha Kumar Kalluri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Narasimhula Balayesu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Balayesu, N., Kalluri, H.K. An extensive survey on traditional and deep learning-based face sketch synthesis models. Int. j. inf. tecnol. 12, 995–1004 (2020). https://doi.org/10.1007/s41870-019-00386-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41870-019-00386-8

Keywords

Search

Navigation