Face Sketch Synthesis Based on Adaptive Similarity Regularization

  • Songze TangEmail author
  • Mingyue Qiu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11935)


Face sketch synthesis plays an important role in public security and digital entertainment. In this paper, we present a novel face sketch synthesis method via local similarity and nonlocal similarity regularization terms. The local similarity can overcome the technological bottlenecks of the patch representation scheme in traditional patch-based face sketch synthesis methods. It improves the quality of synthesized sketches by penalizing the dissimilar training patches (thus have very small weights or are discarded). In addition, taking the redundancy of image patches into account, a global nonlocal similarity regularization is employed to restrain the generation of the noise and maintain primitive facial features during the synthesized process. More robust synthesized results can be obtained. Extensive experiments on the public databases are carried out to validate the generality, effectiveness, and robustness of the proposed algorithm.


Face sketch synthesis Local similarity Nonlocal similarity 



This research was supported in part by the National Natural Science Foundation of China under Grant 61702269, and Grant 61671339, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20171074. The Fundamental Research Funds for the Central Universities at Nanjing Forest Police College under Grant No. LGZD201702.


  1. 1.
    Uhl, R.G. da Vitoria Lobo Jr., N.: A framework for recognizing a facial image from a police sketch. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, pp. 586–593 (1996)Google Scholar
  2. 2.
    Wang, N.N., Tao, D.C., Gao, X.B., Li, X., Li, J.: Transductive face sketch photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 24(9), 1–13 (2013)CrossRefGoogle Scholar
  3. 3.
    Wang, J., Bao, H., Zhou, W., Peng, Q., Xu, Y.Q.: Automatic image-based pencil sketch rendering. J. Comput. Sci. Technol. 17, 347–355 (2002)CrossRefGoogle Scholar
  4. 4.
    Li, X., Cao, X.: A simple framework for face photo-sketch synthesis. Math. Probl. Eng. 2012 (2012)MathSciNetzbMATHGoogle Scholar
  5. 5.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)CrossRefGoogle Scholar
  6. 6.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). Scholar
  7. 7.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint (2017)Google Scholar
  8. 8.
    Ledig, C., Theis, L., Huszár, F., Caballero, J., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, USA, p. 4 (2017)Google Scholar
  9. 9.
    Huang, C., Li, Y., Loy, C., Tang, X.: Learning deep representation for imbalanced classification. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, pp. 5375–5384 (2016)Google Scholar
  10. 10.
    Dong, C., Loy, C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  11. 11.
    Zhang, L., Lin, L., Wu, X., Ding, S., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, pp. 627–634 (2015)Google Scholar
  12. 12.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, Montreal, Canada, pp. 2672–2680 (2014)Google Scholar
  13. 13.
    Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Advances in Neural Information Processing Systems, Barcelona, Spain, pp. 658–666 (2016)Google Scholar
  14. 14.
    Wang, N.N., Zha, W., Li, J., Gao, X.B.: Back projection: an effective: postprocessing method for GAN-based face sketch synthesis. Pattern Recognit. Lett. 107, 59–65 (2018)CrossRefGoogle Scholar
  15. 15.
    Tang, X., Wang, X.: Face photo recognition using sketch. In: Proceedings of IEEE International Conference on Image Processing, New York, USA, pp. 257–260 (2002)Google Scholar
  16. 16.
    Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Diego, USA, pp. 1005–1010 (2005)Google Scholar
  17. 17.
    Roweis, S.T., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)CrossRefGoogle Scholar
  18. 18.
    Song, Y., Bao, L., Yang, Q., Yang, M.-H.: Real-time exemplar-based face sketch synthesis. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 800–813. Springer, Cham (2014). Scholar
  19. 19.
    Wang, N.N., Gao, X., Li, J.: Random sampling for fast face sketch synthesis. Pattern Recognit. 76, 215–227 (2018)CrossRefGoogle Scholar
  20. 20.
    Tang, S., Xiao, L., Liu, P., Huang, L., Zhou, N., Xu, Y.: Pansharpening via sparse regression. Opt. Eng. 56, 093105-1–093105-13 (2017)CrossRefGoogle Scholar
  21. 21.
    Wright, J., Ma, Y., Mairal, J., Sapiro, G., Huang, T., Yan, S.: Sparse representation for computer vision pattern recognition. Proc. IEEE 98(6), 1031–1044 (2010)CrossRefGoogle Scholar
  22. 22.
    Chang, L., Zhou, M., Han, Y., Deng, X.: Face sketch synthesis via sparse representation. In: Proceedings of International Conference on Pattern Recognition, Istanbul, Turkey, pp. 2146–2149 (2010)Google Scholar
  23. 23.
    Gao, X.B., Zhong, J., Li, J., Tian, C.: Face sketch synthesis algorithm using E-HMM and selective ensemble. IEEE Trans. Circuits Syst. Video Technol. 18(4), 487–496 (2008)CrossRefGoogle Scholar
  24. 24.
    Zhou, H., Kuang, Z., Wong, K.: Markov weight fields for face sketch synthesis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, pp. 1091–1097 (2012)Google Scholar
  25. 25.
    Peng, C.L., Gao, X.B., Wang, N.N., Tao, D.C., Li, X., Li, J.: Multiple representations-based face sketch-photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 27(11), 2201–2215 (2016)CrossRefGoogle Scholar
  26. 26.
    Li, C., Zhao, S., Xiao, K., Wang, Y.: Face recognition based on the combination of enhanced local texture feature and DBN under complex illumination conditions. J. Inf. Process. Syst. 14, 191–204 (2018)Google Scholar
  27. 27.
    Muntasa, A.: Homogeneous and non-homogeneous polynomial based eigenspaces to extract the features on facial images. J. Inf. Process. Syst. 12, 591–611 (2016)Google Scholar
  28. 28.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Diego, USA, pp. 60–65 (2005)Google Scholar
  29. 29.
    Zhang, L., Zhang, L., Mou, X., Zhang, D.: FSIM: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Criminal Science and TechnologyNanjing Forest Police CollegeNanjingChina
  2. 2.Department of Information and TechnologyNanjing Forest Police CollegeNanjingChina

Personalised recommendations