Lighting and Pose Robust Face Sketch Synthesis

  • Wei Zhang
  • Xiaogang Wang
  • Xiaoou Tang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6316)


Automatic face sketch synthesis has important applications in law enforcement and digital entertainment. Although great progress has been made in recent years, previous methods only work under well controlled conditions and often fail when there are variations of lighting and pose. In this paper, we propose a robust algorithm for synthesizing a face sketch from a face photo taken under a different lighting condition and in a different pose than the training set. It synthesizes local sketch patches using a multiscale Markov Random Field (MRF) model. The robustness to lighting and pose variations is achieved in three steps. Firstly, shape priors specific to facial components are introduced to reduce artifacts and distortions caused by variations of lighting and pose. Secondly, new patch descriptors and metrics which are more robust to lighting variations are used to find candidates of sketch patches given a photo patch. Lastly, a smoothing term measuring both intensity compatibility and gradient compatibility is used to match neighboring sketch patches on the MRF network more effectively. The proposed approach significantly improves the performance of the state-of-the-art method. Its effectiveness is shown through experiments on the CUHK face sketch database and celebrity photos collected from the web.


Local Binary Pattern Markov Random Field Test Photo Lighting Variation Face Photo 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tang, X., Wang, X.: Face sketch synthesis and recognition. In: ICCV (2003)Google Scholar
  2. 2.
    Tang, X., Wang, X.: Face sketch recognition. IEEE Trans. CSVT 14, 50–57 (2004)MathSciNetGoogle Scholar
  3. 3.
    Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: CVPR (2005)Google Scholar
  4. 4.
    Gao, X., Zhong, J., Li, J., Tian, C.: Face sketch synthesis algorithm based on E-HMM and selective ensemble. IEEE Trans. CSVT 18, 487–496 (2008)Google Scholar
  5. 5.
    Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. PAMI 31, 1955–1967 (2009)Google Scholar
  6. 6.
    Hertzmann, A., Jacobs, C., Oliver, N., Curless, B., Salesin, D.: Image analogies. In: SIGGRAPH (2001)Google Scholar
  7. 7.
    Koshimizu, H., Tominaga, M., Fujiwara, T., Murakami, K.: On KANSEI facial image processing for computerized facialcaricaturing system PICASSO. In: Proc. IEEE Int’l. Conf. on Systems, Man, and Cybernetics (1999)Google Scholar
  8. 8.
    Freeman, W.T., Tenenbaum, J.B., Pasztor, E.C.: Learning style translation for the lines of a drawing. ACM Trans. Graphics 22, 33–46 (2003)CrossRefGoogle Scholar
  9. 9.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Trans. PAMI 28, 2037 (2006)Google Scholar
  10. 10.
    Freeman, W., Pasztor, E., Carmichael, O.: Learning low-level vision. IJCV 40, 25–47 (2000)zbMATHCrossRefGoogle Scholar
  11. 11.
    Yedidia, J., Freeman, W., Weiss, Y.: Understanding belief propagation and its generalizations. Exploring artificial intelligence in the new millennium 8, 236–239 (2003)Google Scholar
  12. 12.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 60, 91–110 (2004)CrossRefGoogle Scholar
  13. 13.
    Liang, L., Xiao, R., Wen, F., Sun, J.: Face alignment via component-based discriminative search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 72–85. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Pizer, S., Amburn, E., Austin, J., Cromartie, R., Geselowitz, A., Greer, T., Romeny, B., Zimmerman, J., Zuiderveld, K.: Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing 39, 355–368 (1987)CrossRefGoogle Scholar
  15. 15.
    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: SIGGRAPH (2001)Google Scholar
  16. 16.
    Yang, J., Zhang, D., Yang, J.: Is ICA significantly better than PCA for face recognition? In: ICCV (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Wei Zhang
    • 1
  • Xiaogang Wang
    • 2
  • Xiaoou Tang
    • 1
    • 3
  1. 1.Department of Information EngineeringThe Chinese University of Hong Kong 
  2. 2.Department of Electronic EngineeringThe Chinese University of Hong Kong 
  3. 3.Shenzhen Institutes of Advanced TechnologyChinese Academy of SciencesChina

Personalised recommendations