Skip to main content
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 pp 685–701Cite as

Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13676)

Abstract

Stroke based rendering methods have recently become a popular solution for the generation of stylized paintings. However, the current research in this direction is focused mainly on the improvement of final canvas quality, and thus often fails to consider the intelligibility of the generated painting sequences to actual human users. In this work, we motivate the need to learn more human-intelligible painting sequences in order to facilitate the use of autonomous painting systems in a more interactive context (e.g. as a painting assistant tool for human users or for robotic painting applications). To this end, we propose a novel painting approach which learns to generate output canvases while exhibiting a painting style which is more relatable to human users. The proposed painting pipeline Intelli-Paint consists of 1) a progressive layering strategy which allows the agent to first paint a natural background scene before adding in each of the foreground objects in a progressive fashion. 2) We also introduce a novel sequential brushstroke guidance strategy which helps the painting agent to shift its attention between different image regions in a semantic-aware manner. 3) Finally, we propose a brushstroke regularization strategy which allows for \(\sim \)60–80% reduction in the total number of required brushstrokes without any perceivable differences in the quality of generated canvases. Through both quantitative and qualitative results, we show that the resulting agents not only show enhanced efficiency in output canvas generation but also exhibit a more natural-looking painting style which would better assist human users express their ideas through digital artwork.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-19787-1_39
  • Chapter length: 17 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-031-19787-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

Notes

  1. 1.

    For simplicity, we primarily use \(L=2\) in the main paper. Further details on extending progressive layering to \(L>2\) are provided in Appendix A.2.

References

  1. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223. PMLR (2017)

    Google Scholar 

  2. Bangham, J.A., Gibson, S.E., Harvey, R.W.: The art of scale-space. In: BMVC. pp. 1–10. Citeseer (2003)

    Google Scholar 

  3. Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)

  4. Chen, W., Hays, J.: SketchyGAN: towards diverse and realistic sketch to image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9416–9425 (2018)

    Google Scholar 

  5. Collomosse, J.P., Hall, P.M.: Genetic paint: a search for salient paintings. In: Rothlauf, F., et al. (eds.) EvoWorkshops 2005. LNCS, vol. 3449, pp. 437–447. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-32003-6_44

    CrossRef  Google Scholar 

  6. Collomosse, J.P., Hall, P.M.: Salience-adaptive painterly rendering using genetic search. Int. J. Artif. Intell. Tools 15(04), 551–575 (2006)

    CrossRef  Google Scholar 

  7. Collomosse, J., Hall, P.: Painterly rendering using image salience. In: Proceedings 20th Eurographics UK Conference, pp. 122–128. IEEE (2002)

    Google Scholar 

  8. Frans, K., Cheng, C.Y.: Unsupervised image to sequence translation with canvas-drawer networks. arXiv preprint arXiv:1809.08340 (2018)

  9. Frans, K., Soros, L., Witkowski, O.: Clipdraw: exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843 (2021)

  10. Gagniuc, P.A.: Markov Chains: From Theory to Implementation and Experimentation. John Wiley & Sons, Hoboken (2017)

    Google Scholar 

  11. Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118 (2018)

  12. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)

    Google Scholar 

  13. Graves, A.: Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013)

  14. Ha, D., Eck, D.: A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477 (2017)

  15. Haeberli, P.: Paint by numbers: abstract image representations. In: Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, pp. 207–214 (1990)

    Google Scholar 

  16. Hertzmann, A.: Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, pp. 453–460 (1998)

    Google Scholar 

  17. Hertzmann, A., et al.: A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 23, 70–81. Institute of Electrical and Electronics Engineers (2003)

    Google Scholar 

  18. Huang, Z., Heng, W., Zhou, S.: Learning to paint with model-based deep reinforcement learning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8709–8718 (2019)

    Google Scholar 

  19. Jia, B., Brandt, J., Mech, R., Kim, B., Manocha, D.: LPaintB: learning to paint from self-supervision arXiv preprint arXiv:1906.06841 (2019)

  20. Jocher, G., et al.: ultralytics/yolov5: v6.0 - YOLOv5n ’Nano’ models, Roboflow integration, TensorFlow export, OpenCV DNN support, October 2021. https://doi.org/10.5281/zenodo.5563715

  21. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time Style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    CrossRef  Google Scholar 

  22. Kite-Powell, J.: This AI robot will paint a canvas at sxsw 2021 (March 2021), https://www.forbes.com/sites/jenniferhicks/2021/03/10/this-ai-robot-will-paint-a-canvas-at-sxsw-2021/?sh=5b1f0d1ab449

  23. Kotovenko, D., Wright, M., Heimbrecht, A., Ommer, B.: Rethinking style transfer: from pixels to parameterized brushstrokes. arXiv preprint arXiv:2103.17185 (2021)

  24. Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13). Sydney, Australia (2013)

    Google Scholar 

  25. Lee, C.H., Liu, Z., Wu, L., Luo, P.: MaskGAN: towards diverse and interactive facial image manipulation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  26. Litwinowicz, P.: Processing images and video for an impressionist effect. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 407–414 (1997)

    Google Scholar 

  27. Liu, B., Gould, S., Koller, D.: Single image depth estimation from predicted semantic labels. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1253–1260. IEEE (2010)

    Google Scholar 

  28. Liu, S., et al.: Paint transformer: feed forward neural painting with stroke prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6598–6607 (2021)

    Google Scholar 

  29. Mellor, J.F., et al.: Unsupervised doodling and painting with improved spiral. arXiv preprint arXiv:1910.01007 (2019)

  30. Nakano, R.: Neural painters: a learned differentiable constraint for generating brushstroke paintings. arXiv preprint arXiv:1904.08410 (2019)

  31. Nemire, B.: Ai painting robot, May 2017. https://developer.nvidia.com/blog/ai-painting-robot/

  32. Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zaiane, O., Jagersand, M.: U2-Net: going deeper with nested u-structure for salient object detection. Pattern Recogn. 106, p. 107404 (2020)

    Google Scholar 

  33. Reyner, N.: How to paint with layers - in acrylic & oil, December 2017. https://nancyreyner.com/2017/12/25/what-is-layering-for-painting/

  34. Shiraishi, M., Yamaguchi, Y.: An algorithm for automatic painterly rendering based on local source image approximation. In: Proceedings of the 1st International Symposium on Non-photorealistic Animation and Rendering, pp. 53–58 (2000)

    Google Scholar 

  35. Singh, J., Zheng, L.: Combining semantic guidance and deep reinforcement learning for generating human level paintings. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)

    Google Scholar 

  36. Singh, J., Zheng, L., Smith, C., Echevarria, J.: Paint2pix: interactive painting based progressive image synthesis and editing. In: European Conference on Computer Vision. Springer, Cham (2022)

    Google Scholar 

  37. Sochorová, Š, Jamriška, O.: Practical pigment mixing for digital painting. ACM Trans. Graph. 40(6), 1–11 (2021)

    CrossRef  Google Scholar 

  38. Teece, D.: 3D painting for non-photorealistic rendering. In: ACM SIGGRAPH 98 Conference Abstracts and Applications, p. 248 (1998)

    Google Scholar 

  39. Treavett, S., Chen, M.: Statistical techniques for the automated synthesis of non-photorealistic images. In: Proceedings of 15th Eurographics UK Conference, pp. 201–210 (1997)

    Google Scholar 

  40. Turk, G., Banks, D.: Image-guided streamline placement. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 453–460 (1996)

    Google Scholar 

  41. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)

    Google Scholar 

  42. Wikipedia contributors: Ai-da (robot) – Wikipedia, the free encyclopedia (2022). https://en.wikipedia.org/w/index.php?title=Ai-Da_(robot) &oldid=1070639724. Accessed 7 Mar 2022

  43. Xiang, X., Liu, D., Yang, X., Zhu, Y., Shen, X., Allebach, J.P.: Adversarial open domain adaptation for sketch-to-photo synthesis. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1434–1444 (2022)

    Google Scholar 

  44. Xie, N., Hachiya, H., Sugiyama, M.: Artist agent: a reinforcement learning approach to automatic stroke generation in oriental ink painting. IEICE Trans. Inf. Syst. 96(5), 1134–1144 (2013)

    CrossRef  Google Scholar 

  45. Yang, S., Wang, Z., Liu, J., Guo, Z.: Controllable sketch-to-image translation for robust face synthesis. IEEE Trans. Image Process. 30, 8797–8810 (2021)

    CrossRef  Google Scholar 

  46. Zeng, K., Zhao, M., Xiong, C., Zhu, S.C.: From image parsing to painterly rendering. ACM Trans. Graph. 29(1), 1–2 (2009)

    CrossRef  Google Scholar 

  47. Zhao, A., Balakrishnan, G., Lewis, K.M., Durand, F., Guttag, J.V., Dalca, A.V.: Painting many pasts: Synthesizing time lapse videos of paintings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8435–8445 (2020)

    Google Scholar 

  48. Zheng, N., Jiang, Y., Huang, D.: Strokenet: A neural painting environment. In: International Conference on Learning Representations (2018)

    Google Scholar 

  49. Zou, Z., Shi, T., Qiu, S., Yuan, Y., Shi, Z.: Stylized neural painting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15689–15698 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Australian National University, Canberra, Australia

    Jaskirat Singh & Liang Zheng

  2. Adobe Research, San Jose, USA

    Jaskirat Singh, Cameron Smith & Jose Echevarria

Authors
  1. Jaskirat Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cameron Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jose Echevarria
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jaskirat Singh .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 5151 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Singh, J., Smith, C., Echevarria, J., Zheng, L. (2022). Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13676. Springer, Cham. https://doi.org/10.1007/978-3-031-19787-1_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19787-1_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19786-4

  • Online ISBN: 978-3-031-19787-1

  • eBook Packages: Computer ScienceComputer Science (R0)