Multimedia Tools and Applications

, Volume 76, Issue 6, pp 7989–8010 | Cite as

Image palette: painting style transfer via brushstroke control synthesis

  • Zheng Miao
  • Yan Zhang
  • Zhibin Zheng
  • Zhengxing Sun
Article
  • 211 Downloads

Abstract

As one kind of technology of style transfer, painting style transfer can be used to render the sample images with a specific art style. With this technology, we can render the target images in the same style as the samples after some computation. We present a new approach of painting style transfer in which such a style transfer artwork is done by simulating the process of creation. We take the sample as a palette where users can select arbitrary contours or textures as the input brush strokes. We then analyze the style feature of the brush strokes and use this feature for synthesis and style transfer along the stroke curves learned from the specified area in target images or target 3D models to get the same painting style as the samples. Based on this approach, we also design and realize the corresponding painting system. The results show that the users can get a style-transferred personalized target image just by using the given sample images with the least interactions.

Keywords

Style transfer Brush stroke synthesis Painting based on the samples Non-photorealistic rendering 

References

  1. 1.
    Ashikhmin M (2001) Synthesizing natural textures. In: Proceedings of the 2001 symposium on interactive 3D graphics, ser. I3D ’01. New York, NY, USA: ACM, pp 217–226. doi:10.1145/364338.364405
  2. 2.
    Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3):24:1–24:11. doi:10.1145/1531326.1531330 CrossRefGoogle Scholar
  3. 3.
    Bellman R (1956) Dynamic programming and lagrange multipliers. Proc Natl Acad Sci USA 42(10):767MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11):1222–1239. doi:10.1109/34.969114 CrossRefGoogle Scholar
  5. 5.
    Brito A (2008) Blender 3D: architecture, buildings, and scenery: create photorealistic 3D architectural visualizations of buildings, interiors, and environmental scenery. Packt Publishing LtdGoogle Scholar
  6. 6.
    DiVerdi S, Krishnaswamy A, Mech R, Ito D (2013) Painting with polygons: a procedural watercolor engine. IEEE Trans Vis Comput Graph 19(5):723–735CrossRefGoogle Scholar
  7. 7.
    Gooch B, Coombe G, Shirley P (2002) Artistic vision: painterly rendering using computer vision techniques. In: Proceedings of the 2Nd international symposium on non-photorealistic animation and rendering, ser. NPAR ’02. New York, NY, USA: ACM, pp 83–ff. doi:10.1145/508530.508545
  8. 8.
    Hays J, Essa I (2004) Image and video based painterly animation. In: Proceedings of the 3rd international symposium on non-photorealistic animation and rendering, ser. NPAR ’04. New York, NY, USA: ACM, pp 113–120. doi:10.1145/987657.987676
  9. 9.
    He K, Sun J (2012) Statistics of patch offsets for image completion. In: Proceedings of the 12th European Conference on Computer Vision - Volume Part II, ser. ECCV’12. Springer, Berlin, Heidelberg, pp 16–29. doi:10.1007/978-3-642-33709-3_2
  10. 10.
    Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques, ser. SIGGRAPH ’98. New York, NY, USA: ACM, pp 453–460. doi:10.1145/280814.280951
  11. 11.
    Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH Image analogies. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques, ser. SIGGRAPH ’01. New York, NY, USA: ACM, pp 327–340. doi:10.1145/383259.3832952001
  12. 12.
    Huang H, Zang Y, Li C-F (2010) Example-based painting guided by color features. Vis. Comput. 26(6-8):933–942. doi:10.1007/s00371-010-0498-y CrossRefGoogle Scholar
  13. 13.
    Kang H, Lee S, Chui CK (2007) Coherent line drawing. In: Proceedings of the 5th international symposium on non-photorealistic animation and rendering. ACM, pp 43–50Google Scholar
  14. 14.
    Lasram A, Lefebvre S (2012) Parallel patch-based texture synthesis. In: Proceedings of the Fourth ACM SIGGRAPH / Eurographics Conference on High-Performance Graphics, ser. EGGH-HPG’12, pp 115–124. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association. doi:10.2312/EGGH/HPG12/115-124
  15. 15.
    Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, ser. NPAR ’10. New York, NY, USA: ACM, pp 43–48. doi:10.1145/1809939.1809945
  16. 16.
    Lefebvre S, Hoppe H (2006) Appearance-space texture synthesis. In: ACM SIGGRAPH 2006 Papers, ser. SIGGRAPH ’06. New York, NY, USA: ACM, pp 541–548. doi:10.1145/1179352.1141921
  17. 17.
    Lim JJ, Zitnick CL, Dollr P (2013) Sketch tokens: a learned mid-level representation for contour and object detection. In: CVPR. IEEE, pp 3158–3165. http://dblp.uni-trier.de/db/conf/cvpr/cvpr2013.html
  18. 18.
    Lu J, Barnes C, DiVerdi S, Finkelstein A (2013) Realbrush: Painting with examples of physical media. ACM Trans. Graph. 32(4):117:1–117:12. doi:10.1145/2461912.2461998 CrossRefGoogle Scholar
  19. 19.
    Lu J, Sander PV, Finkelstein A (2010) Interactive painterly stylization of images, videos and 3D animations. In: Proceedings of i3d 2010Google Scholar
  20. 20.
    Lukáč M, Fišer J, Bazin J-C, Jamriška O, Sorkine-Hornung A, Sýkora D (2013) Painting by feature: texture boundaries for example-based image creation. ACM Trans. Graph. 32(4):116:1–116:8. doi:10.1145/2461912.2461956 Google Scholar
  21. 21.
    Miao Z, Zhang Y, Zheng Z, Sun Z Image palette: brushstroke synthesis-based style transfer, pp 55–60. http://diglib.eg.org/EG/DL/PE/PG/PG2014short/055-060.pdf
  22. 22.
    Nehab D, Velho L (2002) Multiscale moment-based painterly rendering. In: Proceedings of the 15th Brazilian symposium on computer graphics and image processing, ser. SIBGRAPI ’02. Washington, DC, USA: IEEE Computer Society, pp 244–251. http://dl.acm.org/citation.cfm?id=646016.677978
  23. 23.
    NPAR ’02 (2002) Proceedings of the 2Nd international symposium on non-photorealistic animation and rendering. ACM, New York, pp 434–020Google Scholar
  24. 24.
    Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7):971–987. doi:10.1109/TPAMI.2002.1017623 MATHCrossRefGoogle Scholar
  25. 25.
    Ojala T, Pietikainen M, Harwood D (1994) Performance evaluation of texture measures with classification based on kullback discrimination of distributions, pp A:582–585Google Scholar
  26. 26.
    Olsen SC, Maxwell BA, Gooch B (2005) Interactive vector fields for painterly rendering. In: GI ’05: Proceedings of the 2005 conference on graphics interface. School of computer science, University of Waterloo, Waterloo, Ontario, Canada: Canadian Human-Computer Communications Society, pp 241–247Google Scholar
  27. 27.
    Reinhard E, Ashikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput. Graph. Appl. 21(5):34–41. doi:10.1109/38.946629 CrossRefGoogle Scholar
  28. 28.
    Ritter L, Li W, Curless B, Agrawala M, Salesin D (2006) Painting with texture. In: Proceedings of the 17th eurographics conference on rendering techniques, ser. EGSR’06. Aire-la-Ville, Switzerland, Switzerland: Eurographics Association, pp 371–376. doi:10.2312/EGWR/EGSR06/371-376
  29. 29.
    Simakov D, Caspi Y, Shechtman E, Irani M (2008) Summarizing visual data using bidirectional similarity. In: CVPR. IEEE Computer Society. http://dblp.uni-trier.de/db/conf/cvpr/cvpr2008.html
  30. 30.
    Sun J, Yuan L, Jia J, Shum H-Y (2005) Image completion with structure propagation. ACM Trans. Graph. 24(3):861–868. doi:10.1145/1073204.1073274 CrossRefGoogle Scholar
  31. 31.
    Wang B, Wang W, Yang H, Sun J-G (2004) Efficient example-based painting and synthesis of 2d directional texture. IEEE Trans. Vis. Comput. Graph. 10 (3):266–277. http://dblp.uni-trier.de/db/journals/tvcg/tvcg10.html CrossRefGoogle Scholar
  32. 32.
    Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3):463–476. doi:10.1109/TPAMI.2007.60 CrossRefGoogle Scholar
  33. 33.
    Zang Y, Huang H, Li C-F (2013) Stroke style analysis for painterly rendering. J. Comput. Sci. Technol.:762–775Google Scholar
  34. 34.
    Zeng K, Zhao M, Xiong C, Zhu S-C (2009) From image parsing to painterly rendering. ACM Trans. Graph. 29(1):2:1–2:11. doi:10.1145/1640443.1640445 CrossRefGoogle Scholar
  35. 35.
    Zhang E, Hays J, Turk G (2007) Interactive tensor field design and visualization on surfaces. IEEE Trans Vis Comput Graph 13(1):94–107CrossRefGoogle Scholar
  36. 36.
    Zhao M, Zhu S-C (2011) Customizing painterly rendering styles using stroke processes. In: NPAR ’11: Proceedings Of the ACM SIGGRAPH/eurographics symposium on non-photorealistic animation and rendering. New York, NY, USA: ACM, pp 137–146Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Zheng Miao
    • 1
  • Yan Zhang
    • 1
  • Zhibin Zheng
    • 1
  • Zhengxing Sun
    • 1
  1. 1.Nanjing UniversityNanjingChina

Personalised recommendations