Face Recognition from Multiple Stylistic Sketches: Scenarios, Datasets, and Evaluation

  • Chunlei Peng
  • Nannan Wang
  • Xinbo Gao
  • Jie Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9913)


Matching a face sketch against mug shots, which plays an important role in law enforcement and security, is an interesting and challenging topic in face recognition community. Although great progress has been made in recent years, main focus is the face recognition based on SINGLE sketch in existing studies. In this paper, we present a fundamental study of face recognition from multiple stylistic sketches. Three specific scenarios with corresponding datasets are carefully introduced to mimic real-world situations: (1) recognition from multiple hand-drawn sketches; (2) recognition from hand-drawn sketch and composite sketches; (3) recognition from multiple composite sketches. We further provide the evaluation protocols and several benchmarks on these proposed scenarios. Finally, we discuss the plenty of challenges and possible future directions that worth to be further investigated. All the materials will be publicly available online (Available at for comparisons and further study of this problem.


Face recognition Viewed sketch Composite sketch Fusion 



This work was supported in part by the National Natural Science Foundation of China (under Grant 61432014, 61501339, and 61671339), in part by the Fundamental Research Funds for the Central Universities (under Grant XJS15049, and JB160104), in part by the China Post-Doctoral Science Foundation under Grant 2015M580818 and Grant 2016T90893 and in part by the Shaanxi Province Post-Doctoral Science Foundation.


  1. 1.
  2. 2.
  3. 3.
    Belhumeur, P., Hespanda, J., Kiregeman, D.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE TPAMI 19(7), 711–720 (1997)CrossRefGoogle Scholar
  4. 4.
    Best-Rowden, L., Han, H., Otto, C., Klare, B., Jain, A.: Unconstrained face recognition: identifying a person of interest from a media collection. IEEE TIFS 9(12), 2144–2157 (2014)Google Scholar
  5. 5.
    Bhatt, H., Bharadwaj, S., Singh, R., Vatsa, M.: Memetically optimized MCWLD for matching sketches with digital face images. IEEE TIFS 7(5), 1522–1535 (2012)Google Scholar
  6. 6.
    Frowd, C.: Eyewitnesses and the use and application of cognitive theory. Introduction to applied psychology (2011)Google Scholar
  7. 7.
    Galoogahi, H., Sim, T.: Face sketch recognition by local radon binary pattern. In: ICIP, pp. 1837–1840 (2012)Google Scholar
  8. 8.
    Gao, X., Wang, N., Tao, D., Li, X.: Face sketch-photo synthesis and retrieval using sparse representation. IEEE TCSVT 22, 1213–1226 (2012)Google Scholar
  9. 9.
    Gibson, L.: Forensic Art Essentials: A Manual for Law Enforcement Artists. Academic Press, Waltham (2010)Google Scholar
  10. 10.
    Han, H., Klare, B., Bonnen, K., Jain, A.: Matching composite sketches to face photos: a component-based approach. IEEE TIFS 8(1), 191–204 (2013)Google Scholar
  11. 11.
    Ho, T., Hull, J., Srihari, S.: Decision combination in multiple classifier systems. IEEE TPAMI 16(1), 66–75 (1994)CrossRefGoogle Scholar
  12. 12.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report. 07–49, University of Massachusetts, Amherst (2007)Google Scholar
  13. 13.
    Uhl Jr., R.G., da Victoria Lobo., N.: A framework for recognizing a facial imagefrom a police sketch. In: CVPR, pp. 586–593 (1996)Google Scholar
  14. 14.
    Kan, M., Shan, S., Zhang, H., Lao, S., Chen, X.: Multi-view discriminant analysis. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 808–821. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33718-5_58 CrossRefGoogle Scholar
  15. 15.
    Klare, B., Jain, A.: Heterogeneous face recognition using kernel prototype similarities. IEEE TPAMI 35(6), 1410–1422 (2013)CrossRefGoogle Scholar
  16. 16.
    Klare, B., Li, Z., Jain, A.: Matching forensic sketches to mug shot photos. IEEE TPAMI 33, 639–646 (2011)CrossRefGoogle Scholar
  17. 17.
    Klum, S., Han, H., Klare, B., Jain, A.K.: The FaceSketchID system: matching facial composites to mugshots. IEEE TIFS 9(12), 2248–2263 (2014)Google Scholar
  18. 18.
    Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: CVPR, pp. 1005–1010 (2005)Google Scholar
  19. 19.
    Lowe, D.: Distinctive image features from scale-invariant key-points. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  20. 20.
    Martinez, A., Benavente, R.: The AR face database. Technical report 24, CVC, Barcelona, Spain, June 1998Google Scholar
  21. 21.
    Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: the extended M2VTS database. In: AVBPA, Washington, DC, USA, pp. 72–77, April 1999Google Scholar
  22. 22.
    Mittal, P., Jain, A., Goswami, G., Vatsa, M., Singh, R.: Composite sketch recognition using saliency and attribute feedback. Inf. Fusion 33, 86–99 (2017)CrossRefGoogle Scholar
  23. 23.
    Mittal, P., Vatsa, M., Singh, R.: Composite sketch recognition via deep network-A transfer learning approach. In: ICB (2015)Google Scholar
  24. 24.
    Nejati, H., Zhang, L., Sim, T.: Eyewitness face sketch recognition based on two-step bias modeling. In: Wilson, R., Hancock, E., Bors, A., Smith, W. (eds.) CAIP 2013. LNCS, vol. 8048, pp. 26–33. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40246-3_4 CrossRefGoogle Scholar
  25. 25.
    Ouyang, S., Hospedales, T., Song, Y., Li, X.: ForgetMeNot: memory-aware forensic facial sketch matching. In: CVPRGoogle Scholar
  26. 26.
    Ouyang, S., Hospedales, T., Song, Y.-Z., Li, X.: Cross-modal face matching: beyond viewed sketches. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9004, pp. 210–225. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-16808-1_15 Google Scholar
  27. 27.
    Ouyang, S., Hospedales, T., Song, Y.Z., Li, X.: A survey on heterogeneous face recognition: Sketch, infra-red, 3d and low-resolution (2014).
  28. 28.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: BMVC (2015)Google Scholar
  29. 29.
    Peng, C., Gao, X., Wang, N., Li, J.: Superpixel-based face sketch-photo synthesis. IEEE TCSVT (2015)Google Scholar
  30. 30.
    Peng, C., Gao, X., Wang, N., Li, J.: Graphical representation for heterogeneous face recognition. IEEE TPAMI (2016)Google Scholar
  31. 31.
    Peng, C., Gao, X., Wang, N., Tao, D., Li, X., Li, J.: Multiple representations-based face sketch-photo synthesis. IEEE TNNLS (2015)Google Scholar
  32. 32.
    Phillips, P., Moon, H., Rizvi, S., Rauss, P.: The FERET evaluation methodology for face recognition algorithms. IEEE TPAMI 22(10), 1090–1104 (2000)CrossRefGoogle Scholar
  33. 33.
    Serrano, A., de Diego, I., Conde, C., Cabello, E., Shen, L., Bai, L.: Influence of wavelet frequency and orientation in an SVM based parallel Gabor PCA face verification system. In: IDEAL, pp. 219–228 (2007)Google Scholar
  34. 34.
    Sharma, A., Jacobs, D.: Bypass synthesis: PLS for face recognition with pose, low-resolution and sketch. In: CVPR, pp. 593–600 (2011)Google Scholar
  35. 35.
    Ahonen, T., Hadid, A., Pietikäinen, M.: Face description with local binarypatterns: application to face recognition. IEEE TPAMI 28(12), 2037–2041 (2006)CrossRefGoogle Scholar
  36. 36.
    Tang, X., Wang, X.: Face photo recognition using sketch. In: ICIP, pp. 257–260, September 2002Google Scholar
  37. 37.
    Taylor, K.: Forensic Art and Illustration. CRC Press, Boca Raton (2001)Google Scholar
  38. 38.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: Transductive face sketch-photo synthesis. IEEE TNNLS 24(9), 1364–1376 (2013)Google Scholar
  39. 39.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: A comprehensive survey to face hallucination. IJCV 31(1), 9–30 (2014)CrossRefGoogle Scholar
  40. 40.
    Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE TPAMI 31(11), 1955–1967 (2009)CrossRefGoogle Scholar
  41. 41.
    Zhang, S., Gao, X., Wang, N., Li, J., Zhang, M.: Face sketch synthesis via sparse representation-based greedy search. IEEE TIP 24(8), 2466–2477 (2015)MathSciNetGoogle Scholar
  42. 42.
    Zhang, W., Wang, X., Tang, X.: Coupled information-theoretic encoding for face photo-sketch recognition. In: CVPR, pp. 513–520 (2011)Google Scholar
  43. 43.
    Zhang, Y., McCullough, C., Sullins, J., Ross, C.: Hand-drawn face sketch recognition by humans and a pca-based algorithm for forensic applications. IEEE TSMC-Part A 40(3), 475–485 (2010)Google Scholar
  44. 44.
    Zhou, H., Kuang, Z., Wong, K.: Markov weight fields for face sketch synthesis. In: CVPR, pp. 1091–1097 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.State Key Laboratory of Integrated Services NetworksXidian UniversityXi’anChina

Personalised recommendations