Abstract
In this paper, we present an active learning enabled robotic painting system, called ShadowPainter, which acquires artist-specific painting information from the artwork creating process and achieves robotic reproduction of the artwork. The artist’s painting process information, including interactive trajectories of paintbrushes with the environment and states of the canvas, is collected by a novel Visual Measurement System (VMS). A Robotic Painting System (RPS), accompanied by the VSM, is developed to reproduce human paintings by actively imitating the measured painting process. The critical factors that influence the final painting performance of the robot are revealed. At the end of this paper, the reproduced artworks and the painting ability of the RPS are evaluated by local and global criteria and metrics. The experimental results show that our ShadowPainter can reproduce human-level brush strokes, painting techniques, and overall paintings. Compared with the existing work, our system produces natural strokes and painting details that are closer to human artworks.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Gombrich, E.H.: The story of art. J. Aesthet. Art Crit. 9(4) (1951)
Gülzow, J.M., Paetzold, P., Deussen, O.: Recent developments regarding painting robots for research in automatic painting, artificial creativity, and machine learning. Appl. Sci. 10(10), 3396 (2020)
Hertzmann, A.: A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 23(4), 70–81 (2003). https://doi.org/10.1109/MCG.2003.1210867
Hertzmann, A.: Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. SIGGRAPH ’98, pp 453–460. https://doi.org/10.1145/280814.280951 (1998)
Cohen, H.: The further exploits of AARON, painter. Stanf. Humanit. Rev. 4(2), 141–158 (1995)
The Fusion of Art and Science: https://www.alproductions.org (2020)
Mueller, S., Huebel, N., Waibel, M., D’Andrea, R.: Robotic calligraphy - learning how to write single strokes of Chinese and Japanese character. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 1734–1739. https://doi.org/10.1109/IROS.2013.6696583 (2013)
Tresset, P., Leymarie, F.F.: Portrait drawing by paul the robot. Comput. Graph. 37(5), 348–363 (2013)
Calinon, S., Epiney, J., Billard, A.: A humanoid robot drawing human portraits. In: 5Th IEEE-RAS International Conference on Humanoid Robots, 2005, pp 161–166. https://doi.org/10.1109/ICHR.2005.1573562 (2005)
Robotart: https://robotart.org/ (2022)
PIX 18: https://www.pix18.com/ (2022)
Luo, R.C., Hong, M., Chung, P.: Robot artist for colorful picture painting with visual control system. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 2998–3003. https://doi.org/10.1109/IROS.2016.7759464 (2016)
Scalera, L., Seriani, S., Gasparetto, A., Gallina, P.: Watercolour robotic painting: a novel automatic system for artistic rendering. J Intell Robot Syst 95(3), 871–886 (2019). https://doi.org/10.1007/s10846-018-0937-y
Scalera, L., Seriani, S., Gasparetto, A., Gallina, P.: Non-photorealistic rendering techniques for artistic robotic painting. Robotics 8(1), 10 (2019). https://doi.org/10.3390/robotics8010010
Karimov, A.I., Kopets, E.E., Rybin, V.G., Leonov, S.V., Voroshilova, A.I., Butusov, D.N.: Advanced tone rendition technique for a painting robot. Robot. Auton. Syst. 115, 17–27 (2019). https://doi.org/10.1016/j.robot.2019.02.009
Thomas, L.: E-david: Non-photorealistic rendering using a robot and visual feedback. PhD thesis (2018)
Gülzow, J. M., Grayver, L., Deussen, O.: Self-improving robotic brushstroke replication. Arts 7(4), 84 (2018). https://doi.org/10.3390/arts7040084
Lindemeier, T., Spicker, M., Deussen, O.: Artistic composition for painterly rendering. In: VMV 2016: 21Th International Symposium on Vision, Modeling and Visualization (2016)
Lindemeier, T., Metzner, J., Pollak, L., Deussen, O.: Hardware-based non-photorealistic rendering using a painting robot. Computer Graphics Forum 34(2), 311–323 (2015). https://doi.org/10.1111/cgf.12562
Lindemeier, T., Gülzow, J.M., Deussen, O.: Painterly rendering using limited paint color palettes. In: Proceedings of the Conference on Vision, Modeling, and Visualization. EG VMV ’18, pp 135–145. https://doi.org/10.2312/vmv.20181263 (2018)
Bidgoli, A., De Guevara, M.L., Hsiung, C., Oh, J., Kang, E.: Artistic style in robotic painting; a machine learning approach to learning brushstroke from human artists. arXiv:2007.03647 [cs] (2020)
Huang, Z., Heng, W., Zhou, S.: Learning to Paint with Model-Based Deep Reinforcement Learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 8709–8718 (2019)
Schaldenbrand, P., Oh, J.: Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent. Proc. AAAI Conf. Artif. Intell. 35(1), 505–512 (2021)
Li, J., Yao, L., Hendriks, E., Wang, J.Z.: Rhythmic brushstrokes distinguish van gogh from his contemporaries: Findings via automated brushstroke extraction. IEEE Trans. Pattern Anal. Mach. Intell. 34(6), 1159–1176 (2012). https://doi.org/10.1109/TPAMI.2011.203
Zeng, K., Zhao, M., Xiong, C., Zhu, S.-C.: From image parsing to painterly rendering. ACM Trans. Graph. 29(1), 2–1211 (2009). https://doi.org/10.1145/1640443.1640445
Zhao, M., Zhu, S.-C.: Portrait painting using active templates. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-Photorealistic Animation And Rendering. NPAR ’11. https://doi.org/10.1145/2024676.2024696, pp 117–124. Association for Computing Machinery, USA (2011)
Zou, Z., Shi, T., Qiu, S., Yuan, Y., Shi, Z.: Stylized neural painting. arXiv:2011.08114 (2020)
Liu, S., Lin, T., He, D., Li, F., Deng, R., Li, X., Ding, E., Wang, H.: Paint transformer: Feed forward neural painting with stroke prediction. arXiv:2108.03798 [cs] (2021)
Joanne, H: https://joannehastie.com/ (2022)
Cloudpainter: An Artificially Intelligent Painting Robot. http://www.cloudpainter.com (2022)
Wang, Q., Li, R., Wang, Q., Chen, S.: Non-Fungible Token (NFT): Overview, Evaluation, Opportunities and Challenges. arXiv:2105.07447 [cs] (2021)
Mace, M.-A., Ward, T.: Modeling the creative process: A grounded theory analysis of creativity in the domain of art making. Creat. Res. J. 14(2), 179–192 (2002). https://doi.org/10.1207/S15326934CRJ1402_5
Rusu, R.B., Marton, Z.C., Blodow, N., Dolha, M., Beetz, M.: Towards 3D Point cloud based object maps for household environments. Robot. Auton. Syst. 56(11), 927–941 (2008). https://doi.org/10.1016/j.robot.2008.08.005
Mokrzycki, W.S., Tatol, M.: Colour difference delta e - a survey. MG&V 20(4), 383–411 (2011)
Görner, M., Haschke, R., Ritter, H., Zhang, J.: Moveit! Task Constructor for Task-Level Motion Planning. In: 2019 International Conference on Robotics and Automation (ICRA), pp 190–196. https://doi.org/10.1109/ICRA.2019.8793898 (2019)
Wu, R., Chen, Z., Wang, Z., Yang, J., Marschner, S.: Brush stroke synthesis with a generative adversarial network driven by physically based simulation. In: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering. Expressive ’18, pp 12–11210. https://doi.org/10.1145/3229147.3229150 (2018)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp 694–711 (2016)
The Fine Art Reproduction Studio. https://fineartreproductionstudio.com/ (2022)
Nina R. Aide. https://www.ninaraide.com/ (2022)
Guo, C., Lu, Y., Lin, Y., Zhuo, F., Wang, F.-Y.: Parallel art: Artistic creation under human-machine collaboration. Chin. J. Intell. Sci. Technol. 1(4), 335–341 (2019). https://doi.org/10.11959/j.issn.2096-6652.201938
Guo, C., Bai, T., Lu, Y., Lin, Y., Xiong, G., Wang, X., Wang, F.-Y.: Skywork-Davinci: A novel CPSS-based painting support system. In: 2020 IEEE 16Th International Conference on Automation Science and Engineering (CASE), pp 673–678. https://doi.org/10.1109/CASE48305.2020.9216814 (2020)
Funding
This work is supported in part by Skywork Intelligence Culture & Technology LTD.
Author information
Authors and Affiliations
Contributions
All authors contributed to the research.
Fei-Yue Wang formulated research goals;
Fei-Yue Wang and Xiao Wang provided supervision and ideas;
Chao Guo, Tianxiang Bai, and Yue Lu developed methodology and the system;
Chao Guo and Xiangyu Zhang performed the experiments, and prepared the manuscript;
Chao Guo, Tianxiang Bai, Xingyuan Dai, Xiao Wang, and Fei-Yue Wang reviewed and edited the manuscript.
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Chao Guo and Tianxiang Bai These authors contributed equally to this work.
Rights and permissions
About this article
Cite this article
Guo, C., Bai, T., Wang, X. et al. ShadowPainter: Active Learning Enabled Robotic Painting through Visual Measurement and Reproduction of the Artistic Creation Process. J Intell Robot Syst 105, 61 (2022). https://doi.org/10.1007/s10846-022-01616-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-022-01616-1