Web-based SBLR method of multimedia tools for computer-aided drawing

Abstract

As one of the most successful multimedia tools for digital media and creative industry, computer-aided drawing system assists users to convert the input real photos into painterly style images. Nowadays, it is widely developed as cloud brush engine service in many creative software tools and applications of artistic rendering such as Prisma, Photoshop Cloud, and Meitu, because the machine learning server has more powerful than the stand-alone version. In this paper, we propose a web collaborative Stroke-based Learning and Rendering (WebSBLR) system. Different from the existing methods that are mainly focused on the artistic filters, we concentrate on the stroke realistic rendering engine for browser on client using WebGL and HTML5. Moreover, we implement the learning-based stroke drawing path generation module on the server. By this way, we enable to achieve the computer-supported cooperative work (CSCW), especially for multi-screen synchronous interaction. The experiments demonstrated our method are efficient to web-based multi-screen painting simulation.It can successfully learn artists’ styles and render pictures with consistent and smooth brush strokes.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. 1.

    Abbeel P, Ng AY (2004) Apprenticeship learning via inverse reinforcement learning. In: Machine learning, proceedings of the twenty-first international conference(ICML 2004). Banff, Alberta, Canada, July 4-8 2004

  2. 2.

    Baran I, Lehtinen J, Popovic J (2010) Sketching clothoid splines using shortest paths. Comput Graph Forum 29(2):655–664

    Article  Google Scholar 

  3. 3.

    Bousseau A, Neyret F, Thollot J, Salesin D (2007) Video watercolorization using bidirectional texture advection. ACM Trans Graph 26(3):104

    Article  Google Scholar 

  4. 4.

    Chu NS, Tai C (2004) Real-time painting with an expressive virtual chinese brush. IEEE Comput Graph Appl 24(5):76–85

    Article  Google Scholar 

  5. 5.

    Chu NS, Tai C (2005) Moxi: real-time ink dispersion in absorbent paper. ACM Trans Graph 24(3):504–511

    Article  Google Scholar 

  6. 6.

    Chu N, Baxter B, Wei LY, Govindaraju NK (2010) Detail-preserving paint modeling for 3D brushes Proceedings of the 8th international symposium on NPAR. ACM, New York, pp 27–34

  7. 7.

    Davies E (2005) Machine vision: theory algorithms, practicalities, 3rd edn. Elsevier

  8. 8.

    Eric KJ, Kang H (2011) Image and video abstraction by coherence enhancing filtering. Comput Graph Forum, 593–602

  9. 9.

    Hertzmann A (2003) A survey of stroke-based rendering. IEEE Comput Graph Appl 23:70–81

    Article  Google Scholar 

  10. 10.

    Hu MQ, Yang Y, Shen F, Xie N, Shen HT (2018) Hashing with angular reconstructive embeddings. IEEE Trans Image Process 27(2):545–555

    MathSciNet  Article  Google Scholar 

  11. 11.

    Huang Z, Han L, Gong GH (2015) A local adaptive catmull-rom to reduce numerical dissipation of semi-lagrangian advection. J Visual Comp Animat 26 (2):141–146

    Google Scholar 

  12. 12.

    Jolliffe IT (2002) Principal component analysis. Springer Verlag, New York

    Google Scholar 

  13. 13.

    Kalogerakis E, Nowrouzezahrai D, Breslav S, Hertzmann A (2012) Learning hatching for pen-and-ink illustration of surfaces. ACM Trans Graph 31(1):1,1–1,17

    Article  Google Scholar 

  14. 14.

    Kang H, Lee S (2008) Shape-simplifying image abstraction. Comput Graph Forum 27(7):1773–1780

    Article  Google Scholar 

  15. 15.

    Kyprianidis JE, Kang H, Döllner J (2009) Image and video abstraction by anisotropic kuwahara filtering. Comput Graph Forum 28(7):1955–1963

    Article  Google Scholar 

  16. 16.

    Kyprianidis JE, Semmo A, Kang H, Döllner J (2010) Anisotropic kuwahara filtering with polynomial weighting functions. In: Theory and practice of computer graphics. Sheffield, United Kingdom, 2010. Proceedings, pp 25–30

  17. 17.

    Lu JW, Yu F, Finkelstein A, DiVerdi S (2012) Helpinghand: example-based stroke stylization. ACM Trans Graph 31(4):46,1–46,10

    Article  Google Scholar 

  18. 18.

    Lu J, Barnes C, Diverdi S, Finkelstein A (2013) Realbrush: painting with examples of physical media. Acm Trans Graph 32(4):117

    Article  Google Scholar 

  19. 19.

    Orbay G, Kara LB (2011) Beautification of design sketches using trainable stroke clustering and curve fitting. IEEE Trans Vis Comput Graph 17(5):694–708

    Article  Google Scholar 

  20. 20.

    Pham TQ, van Vliet LJ (2005) Separable bilateral filtering for fast video preprocessing. In: Proceedings of the 2005 IEEE international conference on multimedia and expo, ICME 2005, July 6-9, 2005. Amsterdam, pp 454–457

  21. 21.

    Sehnke F, Osendorfer C, Ru̇ckstieß T, Graves A, Peters J, Schmidhuber J (2010) Parameter-exploring policy gradients. Neur Netw 23(4):551–559

    Article  Google Scholar 

  22. 22.

    Semmo A, Trapp M, Dürschmid T, Döllner J, Pasewaldt S (2016) Interactive multi-scale oil paint filtering on mobile devices. In: SIGGRAPH ’16. Anaheim, CA, USA, July 24-28, 2016, pp 42:1– 42:2

  23. 23.

    Shen F, Xu Y, Liu L, Yang Y, Huang Z, Shen HT (2018) Unsupervised deep hashing with similarity-adaptive and discrete optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

  24. 24.

    Sýkora D, Buriánek J, ára J (2005) Colorization of black-and-white cartoons. Image Vis Comput 23(9):767–782

    Article  Google Scholar 

  25. 25.

    Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8:229–256

    MATH  Google Scholar 

  26. 26.

    Xie N, Laga H, Saito S, Nakajima M (2010) IR2s: interactive real photo to sumi-e. In: Proceedings of the 8th international symposium on non-photorealistic animation and rendering 2010. Annecy, France, June 7-10, 2010, pp 63–71

  27. 27.

    Xie N, Hachiya H, Sugiyama M (2012) Artist agent: a reinforcement learning approach to automatic stroke generation in oriental ink painting. In: Proceedings of the 29th international conference on machine learning, ICML 2012. Edinburgh, Scotland, UK, June 26 - July 1 2012

  28. 28.

    Xu S, Xu Y, Kang SB, Salesin D, Pan Y, Shum H (2006) Animating Chinese paintings through stroke-based decomposition. ACM Trans Graph 25(2):239–267

    Article  Google Scholar 

  29. 29.

    Zeng K, Zhao M, Xiong C, Zhu SC (2010) From image parsing to painterly rendering. ACM Trans Graph 29(1):2,1–2,11

    Google Scholar 

  30. 30.

    Zhao T, Hachiya H, Niu G, Sugiyama M (2011) Analysis and improvement of policy gradient estimation. Adv Neural Inf Process Syst 24:262–270

    MATH  Google Scholar 

  31. 31.

    Zhao Y, Jin X, Xu Y, Zhao H, Ai M, Zhou K (2015) Parallel style-aware image cloning for artworks. IEEE Trans Visual Comput Graph 21(2):229

    Article  Google Scholar 

  32. 32.

    Zitnick CL (2013) Handwriting beautification using token means. ACM Trans Graph 32(4):53,1–53,8

    Article  Google Scholar 

Download references

Acknowledgments

We firstly thank anonymous reviewers for their helpful comments. This work was supported in part by the National Natural Science Foundation of China under Project 61602088, Project 61572108, Project 61632007, and Project 61502339, the National Thousand-Young-Talents Program of China, the Fundamental Research Funds for the Central Universities under Project ZYGX2014Z007, Project ZYGX2015J055 and Project ZYGX2016J212.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Heng Tao Shen.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Xie, N., Zhao, T., Yang, Y. et al. Web-based SBLR method of multimedia tools for computer-aided drawing. Multimed Tools Appl 78, 799–816 (2019). https://doi.org/10.1007/s11042-018-5949-x

Download citation

Keywords

  • Multimedia tools
  • CSCW
  • SBR
  • Artistic stylization
  • PGPE