Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 5, pp 5417–5441 | Cite as

Learning to sketch human facial portraits using personal styles by case-based reasoning

  • Bingwen Jin
  • Songhua Xu
  • Weidong Geng
Article

Abstract

This paper employs case-based reasoning (CBR) to capture the personal styles of individual artists and generate the human facial portraits from photos accordingly. For each human artist to be mimicked, a series of cases are firstly built-up from her/his exemplars of source facial photo and hand-drawn sketch, and then its stylization for facial photo is transformed as a style-transferring process of iterative refinement by looking-for and applying best-fit cases in a sense of style optimization. Two models, fitness evaluation model and parameter estimation model, are learned for case retrieval and adaptation respectively from these cases. The fitness evaluation model is to decide which case is best-fitted to the sketching of current interest, and the parameter estimation model is to automate case adaptation. The resultant sketch is synthesized progressively with an iterative loop of retrieval and adaptation of candidate cases until the desired aesthetic style is achieved. To explore the effectiveness and advantages of the novel approach, we experimentally compare the sketch portraits generated by the proposed method with that of a state-of-the-art example-based facial sketch generation algorithm as well as a couple commercial software packages. The comparisons reveal that our CBR based synthesis method for facial portraits is superior both in capturing and reproducing artists’ personal illustration styles to the peer methods.

Keywords

Facial portraits Sketch generation Style-transferring Artistic rendering Personalized exaggeration Case-based reasoning 

Notes

Acknowledgments

This work was supported by a grant from National Program on Key Basic Research Project of China (973 Program, 2013CB329504), National High Technology Research and Development Program of China (863 Program, 2013AA013705), National Natural Science Foundation of China (N0. 61379067), and National Key Technology R & D Program of the Ministry of Science and Technology (2012BAH03F03).

References

  1. 1.
    Aamodt A, Plaza E (1994) Case-based reasoning: foundational issues, methodological variations, and system approaches. AI Commun 7(1):39–59Google Scholar
  2. 2.
    AKVIS: Akvis sketch v.13.0 (2012) http://akvis.com/en/sketch/index.php
  3. 3.
    Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. ECCV. Springer, pp 404–417Google Scholar
  4. 4.
    Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. PAMI 24(4):509–522CrossRefGoogle Scholar
  5. 5.
    Berger I, Shamir A, Mahler M, Carter E, Hodgins J (2013) Style and abstraction in portrait sketching. TOG 32(4):55CrossRefGoogle Scholar
  6. 6.
    Bradski G (2000) The opencv library. Dr Dobb’s Journal of Software ToolsGoogle Scholar
  7. 7.
    Breiman L (1996) Bagging predictors. Mach Learn 24:123–140MathSciNetzbMATHGoogle Scholar
  8. 8.
    Burgess CM (2005) Cosmetic dermatology. SpringerGoogle Scholar
  9. 9.
    Chen CT (1998) Linear system theory and design, 3rd edn. Oxford University Press Inc, OxfordGoogle Scholar
  10. 10.
    Chen H, Liu Z, Rose C, Xu Y, Shum HY, Salesin D (2004) Example-based composite sketching of human portraits. In: NPAR. ACM, pp 95–153Google Scholar
  11. 11.
    Gonzalez RC, Woods RE (2002) Digital image processing, 2nd edn. Prentice Hall, SLGoogle Scholar
  12. 12.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explor Newsl 11(1):10–18CrossRefGoogle Scholar
  13. 13.
    Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH (2001) Image analogies. In: SIGGRAPH. ACM, pp 327–340Google Scholar
  14. 14.
    Kalogerakis E, Nowrouzezahrai D, Breslav S, Hertzmann A (2012) Learning hatching for pen-and-ink illustration of surfaces. ACM Trans Graph 31(1):1:1–1:17CrossRefGoogle Scholar
  15. 15.
    Kim SY, Maciejewski R, Isenberg T, Andrews WM, Chen W, Sousa MC, Ebert DS (2009) Stippling by example. In: NPAR. ACM, pp 41–50Google Scholar
  16. 16.
    Kohavi R, John GH (1997) Wrappers for feature subset selection. Artif Intell 97(1):273–324CrossRefzbMATHGoogle Scholar
  17. 17.
    Kyprianidis J, Collomosse J, Wang T, Isenberg T (2013) State of the art: a taxonomy of artistic stylization techniques for images and video. TVCG 19(5):866–885Google Scholar
  18. 18.
    Leake DB, Kinley A, Wilson D (1995) Learning to improve case adaptation by introspective reasoning and cbr. In: Case-based reasoning research and development. Springer, pp 229–240Google Scholar
  19. 19.
    Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: NPAR. ACM, pp 43–48Google Scholar
  20. 20.
    Lee SY, Chwa KY, Hahn J, Shin SY (1996) Image morphing using deformation techniques. JVCA 7(1):3–23Google Scholar
  21. 21.
    Liang L, Chen H, Xu YQ, Shum HY (2002) Example-based caricature generation with exaggeration. In: PG. IEEE Computer Society, p 386Google Scholar
  22. 22.
    Liu Q, Tang X, Jin H, Lu H, Ma S (2005) A nonlinear approach for face sketch synthesis and recognition. In: CVPR. IEEE Computer Society, pp 1005–1010Google Scholar
  23. 23.
    Lu C, Xu L, Jia J (2012) Combining sketch and tone for pencil drawing production. In: NPAR. Eurographics Association, pp 65–73Google Scholar
  24. 24.
    Lu J, Sander PV, Finkelstein A (2010) Interactive painterly stylization of images, videos and 3d animations. In: Proceedings of the 2010 ACM SIGGRAPH symposium on interactive 3d graphics and games. ACM, pp 127–134Google Scholar
  25. 25.
    Maes F, Collignon A, Vandermeulen D, Marchal G, Suetens P (1997) Multimodality image registration by maximization of mutual information. T-MI 16 (2):187–198Google Scholar
  26. 26.
    Microsoft: Microsoft office homestyle + trial edition (2003). http://www.microsoft.com/zh-tw/download/details.aspx?id=4851
  27. 27.
    Milborrow S, Nicolls F (2008) Locating facial features with an extended active shape model ECCVGoogle Scholar
  28. 28.
    Min F, Suo JL, Zhu SC, Sang N (2007) An automatic portrait system based on and-or graph representation. In: EMMCVPR. Springer, pp 184–197Google Scholar
  29. 29.
    Mish FC (1994) Merriam webster’s collegiate dictionary. Merriam Webster IncGoogle Scholar
  30. 30.
    Modat M, Ridgway GR, Taylor ZA, Lehmann M, Barnes J, Hawkes DJ, Fox NC, Ourselin S (2010) Fast free-form deformation using graphics processing units. Comput Methods Prog Biomed 98:278– 284CrossRefGoogle Scholar
  31. 31.
    Papari G, Petkov N (2009) Continuous glass patterns for painterly rendering. IEEE Trans Image Process 18(3):652–664MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Papari G, Petkov N, Campisi P (2007) Artistic edge and corner enhancing smoothing. IEEE Trans Image Process 16(10):2449–2462MathSciNetCrossRefGoogle Scholar
  33. 33.
    Parke FI (1972) Computer generated animation of faces. In: Proceedings of the ACM annual conference, vol 1. ACM, pp 451–457Google Scholar
  34. 34.
    Peng H, Long F, Ding C (2005) Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. PAMI 27(8):1226–1238CrossRefGoogle Scholar
  35. 35.
    Powell MJD (1998) Direct search algorithms for optimization calculations. Acta Numerica:287–336Google Scholar
  36. 36.
    Redman L (2012) How to draw caricatures. McGraw Hill ProfessionalGoogle Scholar
  37. 37.
    Reinhard E, Ashikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Graph Appl 21(5):34–41CrossRefGoogle Scholar
  38. 38.
    Salesin DH (2002) Non-Photorealistic animation & rendering: 7 grand challenges. In: Keynote talk at second international symposium on non-photorealistic animation and rendering (NPAR 2002, Annecy, France, June 3–5, 2002). http://www.research.microsoft.com/salesin/NPAR.ppt
  39. 39.
    Schaffer C (1993) Selecting a classification method by cross-validation. Mach Learn 13(1):135–143Google Scholar
  40. 40.
    Schaefer S, McPhail T, Warren J (2006) Image deformation using moving least squares. TOG 25:533–540CrossRefGoogle Scholar
  41. 41.
    Semillon (2013) Semillon’s homepage on sina weibo. http://weibo.com/semillon
  42. 42.
    Song Y, Bao L, Yang Q, Yang MH (2014) Real-time exemplar-based face sketch synthesis. Springer, pp 800–813Google Scholar
  43. 43.
    Sonka M, Hlavac V, Boyle R (1993) Image processing, analysis and machine vision. Chapman and Hall, LondonCrossRefGoogle Scholar
  44. 44.
    Thévenaz P, Unser M (2000) Optimization of mutual information for multiresolution image registration. IEEE Trans Image Process 9(12):2083–2099CrossRefzbMATHGoogle Scholar
  45. 45.
    Tu CT, Lien JJJ (2010) Automatic location of facial feature points and synthesis of facial sketches using direct combined model. Trans Sys Man Cyber Part B 40:1158–1169CrossRefGoogle Scholar
  46. 46.
    Wang N (2013) Results on cuhk face sketch database. https://nannanwang.github.io/Result_CUFS.htm
  47. 47.
    Wang N, Tao D, Gao X, Li X, Li J (2013) Transductive face sketch-photo synthesis. IEEE Transactions on Neural Networks and Learning Systems 24(9):1364–1376CrossRefGoogle Scholar
  48. 48.
    Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. PAMI 31:1955–1967CrossRefGoogle Scholar
  49. 49.
    Wei G, Rui M, Lei W, Yi Z, Zhenyun P, Yaohui Z (2009) Template-based portrait caricature generation with facial components analysis. In: ICIS, pp 219–223Google Scholar
  50. 50.
    Wu C, Liu C, Shum HY, Xy YQ, Zhang Z (2004) Automatic eyeglasses removal from face images. IEEE Trans Pattern Anal Mach Intell 26(3):322–336CrossRefGoogle Scholar
  51. 51.
    Zhang Y, Dong W, Deussen O, Huang F, Li K, Hu BG (2014) Data-driven face cartoon stylization. In: SIGGRAPH Asia 2014 technical briefs. ACM, pp 14:1–14:4Google Scholar
  52. 52.
    Zhao M, Zhu SC (2010) Sisley the abstract painter. In: NPAR. ACM, pp 99–107Google Scholar
  53. 53.
    Zhao M, Zhu SC (2011) Portrait painting using active templates. In: NPAR. ACM, pp 117–124Google Scholar
  54. 54.
    Zhou H, Kuang Z, Wong KYK (2012) Markov weight fields for face sketch synthesis. In: IEEE conference on computer vision and pattern recognition (CVPR), 2012, pp 1091–1097Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Computer Science and TechnologyZhejiang UniversityHangzhouChina
  2. 2.Department of Information SystemsNew Jersey Institute of TechnologyNewarkUSA

Personalised recommendations