Convolutional Sketch Inversion

  • Yağmur Güçlütürk
  • Umut Güçlü
  • Rob van Lier
  • Marcel A. J. van Gerven
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9913)

Abstract

In this paper, we use deep neural networks for inverting face sketches to synthesize photorealistic face images. We first construct a semi-simulated dataset containing a very large number of computer-generated face sketches with different styles and corresponding face images by expanding existing unconstrained face data sets. We then train models achieving state-of-the-art results on both computer-generated sketches and hand-drawn sketches by leveraging recent advances in deep learning such as batch normalization, deep residual learning, perceptual losses and stochastic optimization in combination with our new dataset. We finally demonstrate potential applications of our models in fine arts and forensic arts. In contrast to existing patch-based approaches, our deep-neural-network-based approach can be used for synthesizing photorealistic face images by inverting face sketches in the wild.

Keywords

Deep neural network Face synthesis Face recognition Fine arts Forensic arts Sketch inversion Sketch recognition 

References

  1. 1.
    Anstis, S., Vergeer, M., Lier, R.V.: Looking at two paintings at once: luminance edges can gate colors. i-Perception 3(8), 515–518 (2012). http://dx.doi.org/10.1068/i0537sas CrossRefGoogle Scholar
  2. 2.
    Beyeler, M.: OpenCV with Python Blueprints. Packt Publishing, Birmingham (2015)Google Scholar
  3. 3.
    Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: International Conference on Computer Vision. Institute of Electrical and Electronics Engineers (IEEE), December 2015. http://dx.doi.org/10.1109/ICCV.2015.55
  4. 4.
    Cowen, A.S., Chun, M.M., Kuhl, B.A.: Neural portraits of perception: reconstructing face images from evoked brain activity. NeuroImage 94, 12–22 (2014). http://dx.doi.org/10.1016/j.neuroimage.2014.03.018 CrossRefGoogle Scholar
  5. 5.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (2014)Google Scholar
  6. 6.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). http://dx.doi.org/10.1109/TPAMI.2015.2439281 CrossRefGoogle Scholar
  7. 7.
    Gao, X., Wang, N., Tao, D., Li, X.: Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans. Circ. Syst. Video Technol. 22(8), 1213–1226 (2012). http://dx.doi.org/10.1109/TCSVT.2012.2198090 CrossRefGoogle Scholar
  8. 8.
    Gastal, E.S.L., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30(4), 1 (2011). http://dx.doi.org/10.1145/2010324.1964964 CrossRefGoogle Scholar
  9. 9.
    Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. CoRR abs/1508.06576 (2015)Google Scholar
  10. 10.
    Güçlü, U., van Gerven, M.A.J.: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35(27), 10005–10014 (2015). http://dx.doi.org/10.1523/JNEUROSCI.5023-14.2015 CrossRefGoogle Scholar
  11. 11.
    Güçlü, U., van Gerven, M.A.J.: Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage (2015). http://dx.doi.org/10.1016/j.neuroimage.2015.12.036
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015)Google Scholar
  13. 13.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 110 (2016)CrossRefGoogle Scholar
  14. 14.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167 (2015)Google Scholar
  15. 15.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. CoRR abs/1603.08155 (2016)Google Scholar
  16. 16.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  17. 17.
    Learned-Miller, E., Huang, G.B., RoyChowdhury, A., Li, H., Hua, G.: Labeled faces in the wild: a survey. In: Kawulok, M., Celebi, M.E., Smolka, B. (eds.) Advances in Face Detection and Facial Image Analysis, pp. 189–248. Springer, Heidelberg (2016). http://dx.doi.org/10.1007/978-3-319-25958-1_8 CrossRefGoogle Scholar
  18. 18.
    Li, Y.h., Savvides, M., Bhagavatula, V.: Illumination tolerant face recognition using a novel face from sketch synthesis approach and advanced correlation filters. In: International Conference on Acoustics, Speech, and Signal Processing. Institute of Electrical and Electronics Engineers (IEEE) (2006). http://dx.doi.org/10.1109/ICASSP.2006.1660353
  19. 19.
    Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: Conference on Computer Vision and Pattern Recognition. Institute of Electrical and Electronics Engineers (IEEE) (2005). http://dx.doi.org/10.1109/CVPR.2005.39
  20. 20.
    Liu, W., Tang, X., Liu, J.: Bayesian tensor inference for sketch-based facial photo hallucination. In: International Joint Conference on Artificial Intelligence (2007)Google Scholar
  21. 21.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: International Conference on Computer Vision (2015)Google Scholar
  22. 22.
    Martinez, A.M., Benavente, R.: The AR-face database. CVC Technical report 24 (1998)Google Scholar
  23. 23.
    Messer, K., Matas, J., Kittler, J., Jonsson, K.: XM2VTSDB: The extended M2VTS database. In: Audio and Video-based Biometric Person Authentication (1999)Google Scholar
  24. 24.
    Simo-Serra, E., Iizuka, S., Sasaki, K., Ishikawa, H.: Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans. Graph. 35(4), 121 (2016)CrossRefGoogle Scholar
  25. 25.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  26. 26.
    Tang, X., Wang, X.: Face sketch synthesis and recognition. In: International Conference on Computer Vision. Institute of Electrical and Electronics Engineers (IEEE) (2003). http://dx.doi.org/10.1109/ICCV.2003.1238414
  27. 27.
    Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: a next-generation open source framework for deep learning. In: Workshop on Machine Learning Systems at Neural Information Processing Systems (2015)Google Scholar
  28. 28.
    Vergeer, M., Anstis, S., van Lier, R.: Flexible color perception depending on the shape and positioning of achromatic contours. Front. Psychol. 6 (2015). http://dx.doi.org/10.3389/fpsyg.2015.00620
  29. 29.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106(1), 9–30 (2013). http://dx.doi.org/10.1007/s11263-013-0645-9 CrossRefGoogle Scholar
  30. 30.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: Transductive face sketch-photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 24(9), 1364–1376 (2013). http://dx.doi.org/10.1109/TNNLS.2013.2258174 CrossRefGoogle Scholar
  31. 31.
    Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009). http://dx.doi.org/10.1109/TPAMI.2008.222 CrossRefGoogle Scholar
  32. 32.
    Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). http://dx.doi.org/10.1109/TIP.2003.819861 CrossRefGoogle Scholar
  33. 33.
    Xiao, B., Gao, X., Tao, D., Li, X.: A new approach for face recognition by sketches in photos. Sig. Process. 89(8), 1576–1588 (2009). http://dx.doi.org/10.1016/j.sigpro.2009.02.008 CrossRefMATHGoogle Scholar
  34. 34.
    Zhang, L., Lin, L., Wu, X., Ding, S., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. In: International Conference on Multimedia Retrieval. Association for Computing Machinery (ACM) (2015). http://dx.doi.org/10.1145/2671188.2749321
  35. 35.
    Zhang, S., Gao, X., Wang, N., Li, J.: Robust face sketch style synthesis. IEEE Trans. Image Process. 25(1), 220–232 (2016). http://dx.doi.org/10.1109/TIP.2015.2501755 MathSciNetCrossRefGoogle Scholar
  36. 36.
    Zhang, W., Wang, X., Tang, X.: Lighting and pose robust face sketch synthesis. In: European Conference on Computer Vision (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Yağmur Güçlütürk
    • 1
  • Umut Güçlü
    • 1
  • Rob van Lier
    • 1
  • Marcel A. J. van Gerven
    • 1
  1. 1.Donders Institute for Brain, Cognition and BehaviourRadboud UniversityNijmegenThe Netherlands

Personalised recommendations