Convolutional Sketch Inversion

  • Yağmur GüçlütürkEmail author
  • Umut Güçlü
  • Rob van Lier
  • Marcel A. J. van Gerven
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9913)


In this paper, we use deep neural networks for inverting face sketches to synthesize photorealistic face images. We first construct a semi-simulated dataset containing a very large number of computer-generated face sketches with different styles and corresponding face images by expanding existing unconstrained face data sets. We then train models achieving state-of-the-art results on both computer-generated sketches and hand-drawn sketches by leveraging recent advances in deep learning such as batch normalization, deep residual learning, perceptual losses and stochastic optimization in combination with our new dataset. We finally demonstrate potential applications of our models in fine arts and forensic arts. In contrast to existing patch-based approaches, our deep-neural-network-based approach can be used for synthesizing photorealistic face images by inverting face sketches in the wild.


Deep neural network Face synthesis Face recognition Fine arts Forensic arts Sketch inversion Sketch recognition 


  1. 1.
    Anstis, S., Vergeer, M., Lier, R.V.: Looking at two paintings at once: luminance edges can gate colors. i-Perception 3(8), 515–518 (2012). CrossRefGoogle Scholar
  2. 2.
    Beyeler, M.: OpenCV with Python Blueprints. Packt Publishing, Birmingham (2015)Google Scholar
  3. 3.
    Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: International Conference on Computer Vision. Institute of Electrical and Electronics Engineers (IEEE), December 2015.
  4. 4.
    Cowen, A.S., Chun, M.M., Kuhl, B.A.: Neural portraits of perception: reconstructing face images from evoked brain activity. NeuroImage 94, 12–22 (2014). CrossRefGoogle Scholar
  5. 5.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (2014)Google Scholar
  6. 6.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). CrossRefGoogle Scholar
  7. 7.
    Gao, X., Wang, N., Tao, D., Li, X.: Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans. Circ. Syst. Video Technol. 22(8), 1213–1226 (2012). CrossRefGoogle Scholar
  8. 8.
    Gastal, E.S.L., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30(4), 1 (2011). CrossRefGoogle Scholar
  9. 9.
    Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. CoRR abs/1508.06576 (2015)Google Scholar
  10. 10.
    Güçlü, U., van Gerven, M.A.J.: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35(27), 10005–10014 (2015). CrossRefGoogle Scholar
  11. 11.
    Güçlü, U., van Gerven, M.A.J.: Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage (2015).
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015)Google Scholar
  13. 13.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 110 (2016)CrossRefGoogle Scholar
  14. 14.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167 (2015)Google Scholar
  15. 15.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. CoRR abs/1603.08155 (2016)Google Scholar
  16. 16.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  17. 17.
    Learned-Miller, E., Huang, G.B., RoyChowdhury, A., Li, H., Hua, G.: Labeled faces in the wild: a survey. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 189–248. Springer, Heidelberg (2016). CrossRefGoogle Scholar
  18. 18.
    Li, Y.h., Savvides, M., Bhagavatula, V.: Illumination tolerant face recognition using a novel face from sketch synthesis approach and advanced correlation filters. In: International Conference on Acoustics, Speech, and Signal Processing. Institute of Electrical and Electronics Engineers (IEEE) (2006).
  19. 19.
    Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: Conference on Computer Vision and Pattern Recognition. Institute of Electrical and Electronics Engineers (IEEE) (2005).
  20. 20.
    Liu, W., Tang, X., Liu, J.: Bayesian tensor inference for sketch-based facial photo hallucination. In: International Joint Conference on Artificial Intelligence (2007)Google Scholar
  21. 21.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: International Conference on Computer Vision (2015)Google Scholar
  22. 22.
    Martinez, A.M., Benavente, R.: The AR-face database. CVC Technical report 24 (1998)Google Scholar
  23. 23.
    Messer, K., Matas, J., Kittler, J., Jonsson, K.: XM2VTSDB: The extended M2VTS database. In: Audio and Video-based Biometric Person Authentication (1999)Google Scholar
  24. 24.
    Simo-Serra, E., Iizuka, S., Sasaki, K., Ishikawa, H.: Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans. Graph. 35(4), 121 (2016)CrossRefGoogle Scholar
  25. 25.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  26. 26.
    Tang, X., Wang, X.: Face sketch synthesis and recognition. In: International Conference on Computer Vision. Institute of Electrical and Electronics Engineers (IEEE) (2003).
  27. 27.
    Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: a next-generation open source framework for deep learning. In: Workshop on Machine Learning Systems at Neural Information Processing Systems (2015)Google Scholar
  28. 28.
    Vergeer, M., Anstis, S., van Lier, R.: Flexible color perception depending on the shape and positioning of achromatic contours. Front. Psychol. 6 (2015).
  29. 29.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106(1), 9–30 (2013). CrossRefGoogle Scholar
  30. 30.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: Transductive face sketch-photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 24(9), 1364–1376 (2013). CrossRefGoogle Scholar
  31. 31.
    Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009). CrossRefGoogle Scholar
  32. 32.
    Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). CrossRefGoogle Scholar
  33. 33.
    Xiao, B., Gao, X., Tao, D., Li, X.: A new approach for face recognition by sketches in photos. Sig. Process. 89(8), 1576–1588 (2009). CrossRefzbMATHGoogle Scholar
  34. 34.
    Zhang, L., Lin, L., Wu, X., Ding, S., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. In: International Conference on Multimedia Retrieval. Association for Computing Machinery (ACM) (2015).
  35. 35.
    Zhang, S., Gao, X., Wang, N., Li, J.: Robust face sketch style synthesis. IEEE Trans. Image Process. 25(1), 220–232 (2016). MathSciNetCrossRefGoogle Scholar
  36. 36.
    Zhang, W., Wang, X., Tang, X.: Lighting and pose robust face sketch synthesis. In: European Conference on Computer Vision (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Yağmur Güçlütürk
    • 1
    Email author
  • Umut Güçlü
    • 1
  • Rob van Lier
    • 1
  • Marcel A. J. van Gerven
    • 1
  1. 1.Donders Institute for Brain, Cognition and BehaviourRadboud UniversityNijmegenThe Netherlands

Personalised recommendations