The Visual Computer

, Volume 35, Issue 5, pp 653–666 | Cite as

Caricature synthesis with feature deviation matching under example-based framework

  • Honglin Li
  • Masahiro Toyoura
  • Xiaoyang MaoEmail author
Original Article


Example-based caricature synthesis techniques have been attracting large attentions for being able to generate attractive caricatures of various styles. This paper proposes a new example-based caricature synthesis system using a feature deviation matching method as a cross-modal distance metric. It employs the deviation values from average features across different feature spaces rather than the values of features themselves to search for similar components from caricature examples directly. Compared with traditional example-based systems, the proposed system can generate various styles of caricatures without requiring paired photograph–caricature example databases. The newly designed features can effectively capture visual characteristics of the hairstyles and facial components in input portrait images. In addition, this system can control the exaggeration of individual facial components and provide several similarity-based candidates to satisfy users’ different preferences. Experiments are conducted to prove the above ideas.


Caricature synthesis Example-based Cross-modal distance metric Feature deviation matching 



This study was funded by JSPS Grants-in-Aid for Scientific Research (Grant Nos. 26560006 and 26240015) and (Grant No. KAKENHI 17H00737).


  1. 1.
    Sadimon, S.B., Sunar, M.S., Mohamad, D., Haron, H.: Computer generated caricature: a survey. In: Proceedings of the International Conference on Cyberworlds, pp. 383–390 (2010)Google Scholar
  2. 2.
    Chen, H., Zheng, N.N., Liang, L., Li, Y., Xu, Y.Q., Shum, H.Y.: PicToon: a personalized image-based cartoon system. In: Proceedings of the tenth ACM international conference on Multimedia, pp. 171–178 (2002)Google Scholar
  3. 3.
    Wang, N.N., Tao, D.C., Gao, X.B., Li, X.L., Li, J.: Transductive face sketch-photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 24(9), 1364–1376 (2013)Google Scholar
  4. 4.
    Min, F., Suo, J.L., Zhu, S.C., Sang, N.: An automatic portrait system based on and-or graph representation. In: International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), pp. 184–197 (2007)Google Scholar
  5. 5.
    Xu, Z.J., Chen, H., Zhu, S.C., Luo, J.B.: A hierarchical compositional model for face representation and sketching. IEEE Trans. Pattern Anal. Mach. Intell. 30(6), 955–969 (2008)Google Scholar
  6. 6.
    Gooch, B., Reinhard, E., Gooch, A.: Human facial illustrations: creation and psychophysical evaluation. ACM Trans. Graph. 23(1), 27–44 (2004)Google Scholar
  7. 7.
    Wang, X.G., Tang, X.O.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009)Google Scholar
  8. 8.
    Song, Y.B., Bao, L.C., Yang, Q.X., Yang, M.H.: Real-time exemplar based face sketch synthesis. Comput. Vis. ECCV 2014, 800–813 (2014)Google Scholar
  9. 9.
    Zhang, D.Y., Lin, L., Chen, T.S., Wu, X., Tan, W.W., Izquierdo, E.: Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans. Image Process. 26(1), 328–339 (2017)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Chen, H., Liu, Z.Q., Rose, C., Xu, Y.Q., Shum, H.Y., Salesin, D.: Example-based composite sketching of human portraits. In: Proceedings of the 3rd International Symposium on Non-photorealistic Animation and Rendering, pp. 7–9 (2004)Google Scholar
  11. 11.
    Chen, H., Xu, Y.Q., Shum, H.Y., Zhu, S.C., Zheng, N.N.: Example-based facial sketch generation with non-parametric sampling. Proc. Int. Conf. Comput. Vis. 2, 433–438 (2001)Google Scholar
  12. 12.
    Chen, W.J., Yu, H.C., Zhang, J.J: Example based caricature synthesis. Adv. Comput. Sci. Eng. 5(1) (2010)Google Scholar
  13. 13.
    Liang, L., Chen, H., Xu, Y.Q., Shum, H.Y.: Example-based caricature generation with exaggeration. In: Computer Graphics and Applications, Pacific Conference, pp. 386–393 (2002)Google Scholar
  14. 14.
    Yang, W., Tajima, K., Xu, J.Y., Toyoura, M., Mao, X.X.: Example-based automatic caricature generation. In: Cyberworlds, pp. 237–244 (2014)Google Scholar
  15. 15.
    Yu, L.F., Yeung, S.K., Terzopoulos, D., Chan, T.F.: DressUp! Outfit synthesis through automatic optimization. ACM Trans. Graph. 31(6), 134:1–134:14 (2012)Google Scholar
  16. 16.
    Kalogerakis, E., Chaudhuri, S., Koller, D., Koltun, V.: A probabilistic model for component-based shape synthesis. ACM Trans. Graph. 31(4), 55:1–55:11 (2012)Google Scholar
  17. 17.
    Yang, W., Toyoura, M., Xu, J.J., Ohnuma, F., Mao, X.Y.: Example-based caricature generation with exaggeration control. Vis. Comput. 32(3), 383–392 (2016)Google Scholar
  18. 18.
    Zhang, Y., Dong, W.M., Ma, C.Y., Mei, X., Li, K., Huang, F.Y., Hu, B.G., Deuusen, O.: Data-driven synthesis of cartoon faces using different styles. IEEE Trans. Image Process. 26(1), 464–478 (2017)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Shih, Y.C., Paris, S., Barnes, C., Freeman, W.T., Durand, Frédo: Style transfer for headshot portraits. ACM Trans. Graph. 33(4), 148 (2014)Google Scholar
  20. 20.
    Selim, A., Elgharib, M., Doyle, L.: Painting style transfer for head portraits using convolutional neural networks. ACM Trans. Graph. 35(4), 129 (2016)Google Scholar
  21. 21.
    Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR (2016)Google Scholar
  22. 22.
    Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual Attribute Transfer Through Deep Image Analogy (2017). arXiv:1705.01088 [cs.CV]
  23. 23.
    Fišer, J., Jamriška, O., Simons, D., Shechtman, E., Lu, J.W., Asente, P., Luká, M., Sýkora, D.: Example-based synthesis of stylized facial animations. ACM Trans. Graph. 36(4), 155 (2017)Google Scholar
  24. 24.
    Taigman, Y., Polyak, A., Wolf, L.: Unsupervised cross-domain image generation. In: ICLR (2017)Google Scholar
  25. 25.
    Harmon, L.D.: The recognition of faces. Sci. Am. 229(5), 71–82 (1973)Google Scholar
  26. 26.
    Brennan, S.E.: Caricature generator: the dynamic exaggeration of faces by computer. Leonardo 18(3), 170–178 (1985)Google Scholar
  27. 27.
    Koshimizu, H., Tominaga, M., Fujiwara, T., Murakami, K.: On KANSEI facial image processing for computerized facial caricaturing system Picasso. In: IEEE International Conference on Systems, pp. 294–299 (1999)Google Scholar
  28. 28.
    Mo, Z.Y., Lewis, J.P., Neumann, U.: Improved automatic caricature by feature normalization and exaggeration. In: ACM SIGGRAPH Sketches, p. 57 (2004)Google Scholar
  29. 29.
    Xu, J.Y., Yang, W., Mao, X.Y., Toyoura, M., Jin, X.G.: A study on perceived similarity between photograph and shape exaggerated caricature. In: International Conference on Cyberworlds, pp. 213–220 (2014)Google Scholar
  30. 30.
    Cosker, D., Roy, S., Rosin, P.L., Marshall, D.: Re-mapping animation parameters between multiple types of facial model. In: International Conference on Computer Vision/Computer Graphics Collaboration Techniques, pp. 365–276 (2007)Google Scholar
  31. 31.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)zbMATHGoogle Scholar
  32. 32.
    Rasiwasia, N., Pereira, J.C., Coviello, E., Doyle, G., Lanckriet, G.R.G., Levy, R., Vasconcelos, N.: A new approach to cross-modal multimedia retrieval. In: ACM International Conference on Multimedia, pp. 251–260 (2010)Google Scholar
  33. 33.
    Cao, Y., Long, M.S., Wang, J.M., Liu, S.C.: Collective deep quantization for efficient cross-modal retrieval. In: IEEE International Symposium on Multimedia, pp. 3974–3980 (2017)Google Scholar
  34. 34.
    Zhong, C.L., Yu, Y., Tang, S.H., Satoh, S., Xing, K.: Deep multi-label hashing for large-scale visual search based on semantic graph. In: Spring International Publishing, pp. 169–184 (2017)Google Scholar
  35. 35.
    Yan, F., Mikolajczyk, K.: Deep correlation for matching images and text. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3441–3450 (2015)Google Scholar
  36. 36.
    Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: International Conference on International Conference on Machine Learning, pp. –1247–1255 (2013)Google Scholar
  37. 37.
    Yu, Y., Tang, S.H., Raposo, F., Chen, L.: Deep Cross-modal Correlation Learning for Audio and Lyrics in Music Retrieval (2017). arXiv:1711.08976 [cs.IR]
  38. 38.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)Google Scholar
  39. 39.
    Li, H.L., Yang, W., Sun, H.C., Toyoura, M., Mao, X.Y.: Example-based caricature synthesis via feature deviation matching. In: CGI’16 Proceedings of the 33rd Computer Graphics International, pp. 81–84 (2016)Google Scholar
  40. 40.
    Yacoob, Y., Davis, L.S.: Detection and analysis of hair. IEEE Trans. Pattern Anal. Mach. Intell. 28(7), 1164–1169 (2006)Google Scholar
  41. 41.
    Luc, V., Pierre, S.: Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans. Pattern Anal. Mach. Intell. 13(6), 583–598 (1991)Google Scholar
  42. 42.
    Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11(6), 567–585 (1989)zbMATHGoogle Scholar
  43. 43.
    Li, H.L., Toyoura, M., Shimizu, K., Yang, W., Mao, X.Y.: Retrieval of clothing images based on relevance feedback with focus on collar designs. Vis. Comput. 32(10), 1351–1363 (2016)Google Scholar
  44. 44.
    Yang, W., Toyoura, M., Mao, X.Y.: Hairstyle suggestion using statistical learning. In: International Conference on Multimedia Modeling, pp. 277–287 (2012)Google Scholar
  45. 45.
    Sunhem, W., Pasupa, K.: An approach to face shape classification for hairstyle recommendation. In: Eighth International Conference on Advanced Computational Intelligence, pp. 390–394 (2016)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of YamanashiKofuJapan

Personalised recommendations