Skip to main content
SpringerLink
Log in

A Robot for Artistic Painting in Authentic Colors

Journal of Intelligent & Robotic Systems volume 107, Article number: 34 (2023) Cite this article

Abstract

Artistic robotic painting automates the process of creating an artwork. This complex and challenging task includes several aspects: creating algorithms for rendering brushstrokes, reproducing the exact shape of a brushstroke, and developing the principles of mixing paints. This work contributes to the previously unsolved problem of accurately reproducing colors of brushstrokes by means of artistic paints. The main contributions of this paper include: the development of a novel 4-component data-driven mathematical model for artistic paint mixing; the design and implementation of a novel robot capable of accurately dosing and mixing acrylic paints thanks to the improved syringe pumps and the innovative paint mixer; the development of a novel pneumatic system for paint release with a build-in clogging detection mechanism. The capabilities of the designed robotic system are demonstrated by painting four artworks: replicas of Claude Monet’s and Arkady Rylov’s landscapes, a synthetic image generated using the StyleGAN2 neural network trained on Vincent van Gogh’s artistic heritage, and a synthetic image generated using the Midjourney neural network. The obtained results can be useful in various applications of computer creativity, as well as in artistic image replication and restoration, and also in colored 3D printing.

This is a preview of subscription content, access via your institution.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Data Availability

Supplementary video in which the developed robot paints a synthetic image is available at https://youtu.be/NeZX1sPo7P4.

References

  1. Zhao, C: A survey on image style transfer approaches using deep learning. In: Journal of physics: Conference series. IOP Publishing, vol. 1453, p 012129 (2020)

  2. Rosin, P.L., Lai, Y.-K., Mould, D., Yi, R., Berger, I., Doyle, L., Lee, S., Li, C., Liu, Y.-J., Semmo, A., et al.: NPRportrait 1.0: A three-level benchmark for non-photorealistic rendering of portraits. Comput Vis. Media. 8(3), 445–465 (2022)

    Article  Google Scholar 

  3. Zhang, C, Peng, Y: Stacking VAE and GAN for context-aware text-to-image generation. In: 2018 IEEE Fourth international conference on multimedia big data (BigMM). IEEE, pp 1–5 (2018)

  4. Aggarwal, A., Mittal, M., Battineni, G.: Generative adversarial network: an overview of theory and applications. Inter. J Inform. Manag. Data Insights 1(1), 100004 (2021)

    Google Scholar 

  5. Chen, Y., Zhang, H., Liu, L., Chen, X., Zhang, Q., Yang, K., Xia, R., Xie, J.: Research on image inpainting algorithm of improved GAN based on two-discriminations networks. Appl. Intell. 51 (6), 3460–3474 (2021)

    Article  Google Scholar 

  6. Cornelis, B., Ružić, T., Gezels, E., Dooms, A., Pižurica, A., Platiša, L., Cornelis, J., Martens, M., De Mey, M., Daubechies, I.: Crack detection and inpainting for virtual restoration of paintings: the case of the ghent altarpiece. Signal Process. 93(3), 605–619 (2013)

    Article  Google Scholar 

  7. Vasan, K., Janosov, M., Barabási, A.-L.: Quantifying NFT-driven networks in crypto art. Sci Rep 12(1), 1–11 (2022)

    Article  Google Scholar 

  8. Lindemeier, T., Metzner, J., Pollak, L., Deussen, O.: Hardware-based non-photorealistic rendering using a painting robot. In: Computer Graphics Forum. Wiley Online Library, vol. 34, pp 311–323 (2015)

  9. Luo, R.C., Hong, M.-J., Chung, P.-C.: Robot artist for colorful picture painting with visual control system. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp 2998–3003 (2016)

  10. Lindemeier, T., Gülzow, J.M., Deussen, O.: Painterly rendering using limited paint color palettes. In: VMV, pp. 135–145 (2018)

  11. Beltramello, A., Scalera, L., Seriani, S., Gallina, P.: Artistic robotic painting using the palette knife technique. Robot 9(1), 15 (2020)

    Article  Google Scholar 

  12. Scalera, L., Seriani, S., Gasparetto, A., Gallina, P.: Busker robot: A robotic painting system for rendering images into watercolour artworks. Springer (2018)

  13. Scalera, L., Seriani, S., Gasparetto, A., Gallina, P.: Watercolour robotic painting: a novel automatic system for artistic rendering. J. Intell. Robot. Syst. 95(3), 871–886 (2019)

    Article  Google Scholar 

  14. Song, D., Lee, T., Kim, Y.J.: Artistic pen drawing on an arbitrary surface using an impedance-controlled robot. IEEE (2018)

  15. Shen, L., Tao, H., Ni, Y., Wang, Y., Vladimir, S.: Improved YOLOv3 model with feature map cropping for multi-scale road object detection. Measurement Science and Technology (2023)

  16. Zhou, C., Tao, H., Chen, Y., Stojanovic, V., Paszke, W.: Robust point-to-point iterative learning control for constrained systems: a minimum energy approach. Inter. J. Robust Nonlinear Control. 32 (18), 10139–10161 (2022)

    Article  MathSciNet  Google Scholar 

  17. Stojanovic, V., Nedic, N.: Robust identification of OE model with constrained output using optimal input design. J. Franklin Inst. 353(2), 576–593 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  18. Cheng, P., Chen, M., Stojanovic, V., He, S.: Asynchronous fault detection filtering for piecewise homogenous Markov jump linear systems via a dual hidden markov model. Mech. Syst. Signal Process. 107353, 151 (2021)

    Google Scholar 

  19. Tutsoy, O.: Cpg based rl algorithm learns to control of a humanoid robot leg. Int. J. Robot. Autom. 30(2), 1–7 (2015)

    Google Scholar 

  20. Tutsoy, O., Barkana, D.E.: Model free adaptive control of the under-actuated robot manipulator with the chaotic dynamics. ISA Trans 118, 106–115 (2021)

    Article  Google Scholar 

  21. Gülzow, J.M., Grayver, L., Deussen, O.: Self-improving robotic brushstroke replication. Arts 7(4), 84 (2018)

    Article  Google Scholar 

  22. MOONWALK: https://scan.ppgrefinish.com/en/moonwalk/. Accessed 15 Jan 2023

  23. Sochorová, Š, Jamriška, O.: Practical pigment mixing for digital painting. ACM Transactions on Graphics (TOG) 40(6), 1–11 (2021)

    Article  Google Scholar 

  24. Berns, R.S., Mohammadi, M.: Single-constant simplification of kubelka-munk turbid-media theory for paint systems—a review. Color Research & application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Franç,ais de la Couleur 32(3), 201–207 (2007)

    Article  Google Scholar 

  25. Lu, J., DiVerdi, S., Chen, W.A., Barnes, C., Finkelstein, A.: Realpigment: Paint compositing by example. In: Proceedings of the workshop on non-photorealistic animation and rendering, pp. 21–30 (2014)

  26. Lu, J., Barnes, C., DiVerdi, S., Finkelstein, A.: Realbrush: Painting with examples of physical media. ACM Transactions on Graphics (TOG) 32(4), 1–12 (2013)

    Article  Google Scholar 

  27. Gossett, N., Chen, B.: Paint inspired color mixing and compositing for visualization. In: IEEE Symposium on information visualization. IEEE, pp 113–118 (2004)

  28. Yuan, J., Chen, C., Yao, D.: Chen, G.: 3D printing of oil paintings based on material jetting and its reduction of staircase effect. Polymers 12(11), 2536 (2020)

    Article  Google Scholar 

  29. Elkhuizen, W., Essers, T., Song, Y., Geraedts, J., Weijkamp, C., Dik, J., Pont, S.: Gloss, color, and topography scanning for reproducing a painting’s appearance using 3D printing. Journal on Computing and Cultural Heritage (JOCCH) 12(4), 1–22 (2019)

    Google Scholar 

  30. Yuan, J., Chen, G., Li, H., Prautzsch, H., Xiao, K.: Accurate and computational: a review of color reproduction in full-color 3D printing. Mater. Des. 209, 109943 (2021)

    Article  Google Scholar 

  31. Punpongsanon, P., Wen, X., Kim, D.S., Mueller, S.: Colormod: recoloring 3D printed objects using photochromic inks. In: Proceedings of the 2018 CHI Conference on human factors in computing systems, pp. 1–12 (2018)

  32. Karimov, A.I., Kopets, E.E., Rybin, V.G., Leonov, S.V., Voroshilova, A.I., Butusov, D.N.: Advanced tone rendition technique for a painting robot. Robot. Auton. Syst. 115, 17–27 (2019)

    Article  Google Scholar 

  33. Lambos, N., Vosniakos, G.-C., Papazetis, G.: Low-cost automatic identification of nozzle clogging in material extrusion 3d printers. Procedia Manuf. 51, 274–279 (2020)

    Article  Google Scholar 

  34. Tlegenov, Y., Hong, G.S., Lu, W.F.: Nozzle condition monitoring in 3d printing. Robot. Comput. Integr. Manuf. 54, 45–55 (2018)

    Article  Google Scholar 

  35. Nuchitprasitchai, S., Roggemann, M., Pearce, J.M.: Factors effecting real-time optical monitoring of fused filament 3d printing. Prog. Addit. Manuf. 2(3), 133–149 (2017)

    Article  Google Scholar 

  36. Cohen, P.: Harold cohen and AARON. AI Mag. 37(4), 63–66 (2016)

    Google Scholar 

  37. Gurney, J.: Vangobot. http://gurneyjourney.blogspot.com/2012/12/vangobot.html. Accessed 23 Aug 2022 (2012)

  38. Lu, Y., Lam, J.H., Yam, Y.: Preliminary study on vision-based pen-and-ink drawing by a robotic manipulator. In: 2009 IEEE/ASME International conference on advanced intelligent mechatronics. IEEE, pp 578–583 (2009)

  39. Adamik, M., Goga, J., Pavlovicova, J., Babinec, A., Sekaj, I.: Fast robotic pencil drawing based on image evolution by means of genetic algorithm. Robot. Auton. Syst. 148, 103912 (2022)

    Article  Google Scholar 

  40. Gao, Q., Chen, H., Yu, R., Yang, J., Duan, X.: A robot portraits pencil sketching algorithm based on face component and texture segmentation. In: 2019 IEEE International Conference on Industrial Technology (ICIT). IEEE, pp 48–53 (2019)

  41. Wu, P.-L., Hung, Y.-C., Shaw, J.-S.: Artistic robotic pencil sketching using closed-loop force control. Proceedings of the Institution of Mechanical Engineers, Part c: Journal of Mechanical Engineering Science 09544062221096946 (2022)

  42. Montagner, C., Bacci, M., Bracci, S., Freeman, R., Picollo, M.: Library of UV–vis–NIR reflectance spectra of modern organic dyes from historic pattern-card coloured papers. Spectrochim. Acta A Mol. Biomol. Spectrosc. 79(5), 1669–1680 (2011)

    Article  Google Scholar 

  43. Robotart: winners. https://robotart.org/2017-winners/. Accessed 23 Aug 2022 (2017)

  44. Kaufman, S.L.: Artist Sougwen Chung wanted collaborators. So she designed and built her own AI robots. https://www.washingtonpost.com/business/2020/11/05/ai-artificial-intelligence-art-sougwen-chung/s. Accessed 23 Aug 2022 (2020)

  45. Scalera, L., Mazzon, E., Gallina, P., Gasparetto, A.: Airbrush robotic painting system: Experimental validation of a colour spray model. In: International conference on robotics in Alpe-Adria Danube Region. Springer, pp 549–556 (2017)

  46. Scalera, L., Canever, G., Seriani, S., Gasparetto, A., Gallina, P.: Robotic sponge and watercolor painting based on image-processing and contour-filling algorithms. In: Actuators, vol. 11, p. 62. MDPI (2022)

  47. Song, D., Lim, E., Park, J., Jung, M., Kim, Y.J.: Tsp-bot:, Robotic tsp pen art using high-dof manipulators. arXiv:http://arxiv.org/abs2210.07592 (2022)

  48. Ilinkin, I., Song, D., Kim, Y.J.: Stroke-based rendering and planning for robotic performance of artistic drawing. arXiv:2210.07590 (2022)

  49. Bidgoli, A., De Guevara, M.L., Hsiung, C., Oh, J., Kang, E.: Artistic style in robotic painting; a machine learning approach to learning brushstroke from human artists. In: 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, pp 412–418 (2020)

  50. Schaldenbrand, P., McCann, J., Oh, J.: Frida:, A collaborative robot painter with a differentiable, real2sim2real planning environment. arXiv:2210.00664 (2022)

  51. Rohani, N., Pouyet, E., Walton, M., Cossairt, O., Katsaggelos, A.K.: Pigment unmixing of hyperspectral images of paintings using deep neural networks. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 3217–3221 (2019)

  52. Kalloniatis, M., Luu, C.: The perception of color webvision: the organization of the retina and visual system [internet] (2007)

  53. Levkowitz, H., Herman, G.T.: GLHS: A generalized lightness, hue, and saturation color model. CVGIP:, Graphical Models and Image Processing 55(4), 271–285 (1993)

    Google Scholar 

  54. Hanbury, A.: Constructing cylindrical coordinate colour spaces. Pattern Recogn. Lett. 29(4), 494–500 (2008)

    Article  Google Scholar 

  55. Caivano, J.L.: Color order systems, color mixtures, and the role of cesia. Color. Res. Appl. 46 (6), 1169–1179 (2021)

    Article  Google Scholar 

  56. Gonzalez-Lopez, J., Ventura, S., Cano, A.: Distributed nearest neighbor classification for large-scale multi-label data on spark. Futur. Gener. Comput. Syst. 87, 66–82 (2018)

    Article  Google Scholar 

  57. Garcia, S., Derrac, J., Cano, J., Herrera, F.: Prototype selection for nearest neighbor classification: Taxonomy and empirical study. IEEE Trans. Pattern Anal. Mach. Intell. 34(3), 417–435 (2012)

    Article  Google Scholar 

  58. Song, Y., Liang, J., Lu, J., Zhao, X.: An efficient instance selection algorithm for k nearest neighbor regression. Neurocomputing 251, 26–34 (2017)

    Article  Google Scholar 

  59. Peterson, L.E., Coleman, M.A.: Machine learning-based receiver operating characteristic (ROC) curves for crisp and fuzzy classification of dna microarrays in cancer research. Int. J. Approx. Reason. 47(1), 17–36 (2008)

    Article  MATH  Google Scholar 

  60. Ostertagová, E.: Modelling using polynomial regression. Procedia Eng. 48, 500–506 (2012)

    Article  Google Scholar 

  61. Shin, Y., Xiu, D.: On a near optimal sampling strategy for least squares polynomial regression. J. Comput. Phys. 326, 931–946 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  62. Karimov, A., Nepomuceno, E.G., Tutueva, A., Butusov, D.: Algebraic method for the reconstruction of partially observed nonlinear systems using differential and integral embedding. Math. 8(2), 300 (2020)

    Article  Google Scholar 

  63. van Wieringen, W.N.: Lecture notes on ridge regression. arXiv:1509.09169 (2015)

  64. Tikhonov, A.N.: On the solution of ill-posed problems and the method of regularization. In: Doklady Akademii Nauk. Russian Academy of Sciences, vol. 151, pp 501–504 (1963)

  65. Tikhonov, A.N., Arsenin, V.J., Arsenin, V.I., Arsenin, V.Y., et al.: Solutions of Ill-posed Problems Vh Winston (1977)

  66. Kopets, E., Karimov, A., Scalera, L., Butusov, D.: Estimating natural frequencies of Cartesian 3D printer based on kinematic scheme. Appl. Sci. 12(9), 4514 (2022)

    Article  Google Scholar 

  67. Tischler, A.: How to MIX COLOR - Can you mix any colour with a limited palette??? on YouTube. https://www.youtube.com/watch?v=A1RdRqsRadI. Accessed 23 Aug 2022 (2020)

  68. Campus 5 x Acrylic Paint 100ml: https://www.amazon.de/-/en/N131800-00-Campus-Acrylic-Paint-100ml/dp/B0048DMN2S. Accessed 23 Aug 2022

  69. Kohavi, R., et al.: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. In: Ijcai. Montreal, Canada, vol. 14, pp 1137–1145 (1995)

  70. Cheng, J., Dekkers, J.C., Fernando, R.L.: Cross-validation of best linear unbiased predictions of breeding values using an efficient leave-one-out strategy. J. Anim. Breeding Genet. 138(5), 519–527 (2021)

    Article  Google Scholar 

  71. Wong, T.-T.: Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recogn. 48(9), 2839–2846 (2015)

    Article  MATH  Google Scholar 

  72. Monet, C.: Saintrépuscule.jpg. https://commons.wikimedia.org/wiki/File:Claude_Monet,_Saint-Georges_majeur_au_cr%C3%A9puscule.jpg. Accessed 23 Aug 2022

  73. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of styleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

  74. Van Gogh worldwide: https://vangoghworldwide.org. Accessed 23 Aug 2022

  75. Rylov, A.: Belaya Noch. Nochnaya Zarya. https://ar.culture.ru/ru/subject/belaya-noch-nochnaya-zarya. Accessed 23 Aug 2022

  76. Midjourney: https://www.midjourney.com/home/. Accessed 19 Sep 2022

  77. Hertzmann, A.: Can computers create art?. In: Arts, vol. 7, p. 18. MDPI (2018)

  78. Dall⋅E mini: https://github.com/borisdayma/dalle-mini. Accessed 19 Sep 2022 (2022)

  79. Karimov, A.I., Pesterev, D.O., Ostrovskii, V.Y., Butusov, D.N., Kopets, E.E.: Brushstroke rendering algorithm for a painting robot. In: 2017 International Conference “Quality Management, Transport and Information Security, Information Technologies” (IT&QM&IS), pp. 331–334. IEEE (2017)

Download references

Acknowledgements

The authors are grateful to T. Shpilevaya for the work dedicated to learning Style GAN 2 and generating the image in Fig. 14b.

Funding

This study is supported by Russian Science Foundation, project 22-79-00171.

Author information

Authors and Affiliations

  1. Youth Research Institute, St. Petersburg Electrotechnical University, 5 Professora Popova, St. Petersburg, 197022, Russia

    Artur Karimov, Ekaterina Kopets & Denis Butusov

  2. Public Relations Department, St. Petersburg Electrotechnical University, 5 Professora Popova, St. Petersburg, 197022, Russia

    Sergey Leonov

  3. Polytechnic Department of Engineering and Architecture, University of Udine, via delle Scienze, 206, Udine, 33100, Italy

    Lorenzo Scalera

Authors
  1. Artur Karimov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ekaterina Kopets
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergey Leonov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lorenzo Scalera
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Denis Butusov
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study’s conception and design. Artur Karimov and Denis Butusov conceptualized the proposed robotic system and were the supervisors of this project. Experimental robotic setup design and software development were performed by Artur Karimov and Ekaterina Kopets. The mathematical model of color mixing was developed by Artur Karimov and Sergey Leonov. The first draft of the manuscript was written by Artur Karimov, Denis Butusov, and Ekaterina Kopets. Ekaterina Kopets contributed to the 3D modeling and design of the planetary mixing device. Lorenzo Scalera is responsible for the experimentation part with image synthesis and drawing as well as for manuscript review and editing. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Artur Karimov or Denis Butusov.

Ethics declarations

Ethics approval

Not applicable.

Consent for Publication

Informed consent was obtained from all the co-authors of this publication.

Competing interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karimov, A., Kopets, E., Leonov, S. et al. A Robot for Artistic Painting in Authentic Colors. J Intell Robot Syst 107, 34 (2023). https://doi.org/10.1007/s10846-023-01831-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01831-4

Keywords

search

Navigation