Abstract
Artificial intelligence (AI) has played a significant role in imitating and producing large-scale designs such as e-commerce banners. However, it is less successful at creative and collaborative design outputs. Most humans express their ideas as rough sketches, and lack the professional skills to complete pleasing paintings. Existing AI approaches have failed to convert varied user sketches into artistically beautiful paintings while preserving their semantic concepts. To bridge this gap, we have developed SmartPaint, a co-creative drawing system based on generative adversarial networks (GANs), enabling a machine and a human being to collaborate in cartoon landscape painting. SmartPaint trains a GAN using triples of cartoon images, their corresponding semantic label maps, and edge detection maps. The machine can then simultaneously understand the cartoon style and semantics, along with the spatial relationships among the objects in the landscape images. The trained system receives a sketch as a semantic label map input, and automatically synthesizes its edge map for stable handling of varied sketches. It then outputs a creative and fine painting with the appropriate style corresponding to the human’s sketch. Experiments confirmed that the proposed SmartPaint system successfully generates high-quality cartoon paintings.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Atarsaikhan G, Iwana BK, Narusawa A, et al., 2017. Neural font style transfer. Proc 14th IAPR Int Conf on Document Analysis and Recognition, p.51–56. https://doi.org/10.1109/ICDAR.2017.328
Belongie S, Malik J, Puzicha J, 2001. Shape context: a new descriptor for shape matching and object recognition. Proc 13th Int Conf on Neural Information Processing Systems, p.798–804.
Benedetti L, Winnemöller H, Corsini M, et al., 2014. Painting with Bob: assisted creativity for novices. Proc 27th Annual ACM Symp on User Interface Software and Technology, p.419–428. https://doi.org/10.1145/2642918.2647415
Bowman SR, Vilnis L, Vinyals O, et al., 2016. Generating sentences from a continuous space. Proc 20th SIGNLL Conf on Computational Natural Language Learning, p.10–21. https://doi.org/10.18653/v1/K16-1002
Canny J, 1987. A computational approach to edge detection. In: Fischler MA, Firschein O (Eds.), Readings in Computer Vision: Issues, Problem, Principles, and Paradigms. Elsevier, Amsterdam, p.184–203. https://doi.org/10.1016/B978-0-08-051581-6.50024-6
Champandard AJ, 2016. Semantic style transfer and turning two-bit doodles into fine artworks. https://doi.org/1603.01768
Chen C, Lin JC, Liao MH, et al., 2016. Learning to detect salient curves of cartoon images based on composition rules. Proc 11th Int Conf on Computer Science & Education, p.808–813. https://doi.org/10.1109/ICCSE.2016.7581686
Chen T, Cheng MM, Tan P, et al., 2009. Sketch2Photo: Internet image montage. ACM Trans Graph, 28(5), Article 124. https://doi.org/10.1145/1618452.1618470
Chen Y, Lai YK, Liu YJ, 2018. CartoonGAN: generative adversarial networks for photo cartoonization. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.9465–9474. https://doi.org/10.1109/CVPR.2018.00986
Chu NSH, Tai CL, 2004. Real-time painting with an expressive virtual Chinese brush. IEEE Comput Graph Appl, 24(5):76–85. https://doi.org/10.1109/MCG.2004.37
Ci YZ, Ma XZ, Wang ZH, et al., 2018. User-guided deep anime line art colorization with conditional adversarial networks. Proc 26th ACM Int Conf on Multimedia, p.1536–1544. https://doi.org/10.1145/3240508.3240661
Cummmings D, Vides F, Hammond T, 2012. I don’t believe my eyes! Geometric sketch recognition for a computer art tutorial. Proc Int Symp on Sketch-Based Interfaces and Modeling, p.97–106. https://doi.org/10.2312/SBM/SBM12/097-106
Davis NM, 2013. Human-computer co-creativity: blending human and computational creativity. Proc 9th Artificial Intelligence and Interactive Digital Entertainment Conf, p.9–12.
Davis NM, Hsiao CP, Singh KY, et al., 2015. Drawing apprentice: an enactive co-creative agent for artistic collaboration. Proc ACM SIGCHI Conf on Creativity and Cognition, p.185–186. https://doi.org/10.1145/2757226.2764555
Davis NM, Hsiao CP, Singh KY, et al., 2016a. Co-creative drawing agent with object recognition. Proc 12th Artificial Intelligence and Interactive Digital Entertainment Conf, p.9–15.
Davis NM, Hsiao CP, Yashraj Singh K, et al., 2016b. Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent. Proc 21st Int Conf on Intelligent User Interfaces, p.196–207. https://doi.org/10.1145/2856767.2856795
Dekel T, Gan C, Krishnan D, et al., 2018. Sparse, smart contours to represent and edit images. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.3511–3520. https://doi.org/10.1109/CVPR.2018.00370
Dixon D, Prasad M, Hammond T, 2010. iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. Proc SIGCHI Conf on Human Factors in Computing Systems, p.897–906. https://doi.org/10.1145/1753326.1753459
Eitz M, Richter R, Hildebrand K, et al., 2011. Photosketcher: interactive sketch-based image synthesis. IEEE Comput Graph Appl, 31(6):56–66. https://doi.org/10.1109/MCG.2011.67
Gatys LA, Ecker AS, Bethge M, 2016. Image style transfer using convolutional neural networks. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2414–2423. https://doi.org/10.1109/CVPR.2016.265
Goodfellow I, Pouget-Abadie J, Mirza M, et al., 2014. Generative adversarial nets. Proc 27th Int Conf on Neural Information Processing Systems, p.2672–2680.
Güçlütürk Y, Güçlü U, van Lier R, et al., 2016. Convolutional sketch inversion. European Conf on Computer Vision, p.810–824. https://doi.org/10.1007/978-3-319-46604-0_56
Ha D, Eck D, 2017. A neural representation of sketch drawings. https://doi.org/1704.03477
Huang HZ, Wang H, Luo WH, et al., 2017. Real-time neural style transfer for videos. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.783–791. https://doi.org/10.1109/CVPR.2017.745
Isola P, Zhu JY, Zhou TH, et al., 2017. Image-to-image translation with conditional adversarial networks. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1125–1134. https://doi.org/10.1109/CVPR.2017.632
Karimi P, Davis N, Grace K, et al., 2018. Deep learning for identifying potential conceptual shifts for co-creative drawing. https://doi.org/1801.00723
Lee YJ, Zitnick CL, Cohen MF, 2011. ShadowDraw: realtime user guidance for freehand drawing. ACM Trans Graph, 30(4), Article 27. https://doi.org/10.1145/2010324.1964922
Li MJ, Huang HZ, Ma L, et al., 2018. Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. Proc 15th European Conf on Computer Vision, p.186–201. https://doi.org/10.1007/978-3-030-01240-3_12
Liu YF, Qin ZC, Wan T, et al., 2018. Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing, 311:78–87. https://doi.org/10.1016/j.neucom.2018.05.045
Luan FJ, Paris S, Shechtman E, et al., 2017. Deep photo style transfer. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4990–4998. https://doi.org/10.1109/CVPR.2017.740
Mirza M, Osindero S, 2014. Conditional generative adversarial nets. https://doi.org/1411.1784
Ning X, Laga H, Saito S, et al., 2011. Contour-driven Sumi-e rendering of real photos. Comput Graph, 35(1):122–134. https://doi.org/10.1016/j.cag.2010.11.017
Oh C, Song J, Choi J, et al., 2018. I lead, you help but only with enough details: understanding user experience of co-creation with artificial intelligence. Proc CHI Conf on Human Factors in Computing Systems, Article 649. https://doi.org/10.1145/3173574.3174223
Portenier T, Hu QY, Szabó A, et al., 2018. Faceshop: deep sketch-based face image editing. ACM Trans Graph, 37(4), Article 99. https://doi.org/10.1145/3197517.3201393
Roberts A, Engel J, Eck D, 2017. Hierarchical variational autoencoders for music. Workshop on Machine Learning for Creativity and Design, NIPS.
Sangkloy P, Lu JW, Fang C, et al., 2017. Scribbler: controlling deep image synthesis with sketch and color. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.5400–5409. https://doi.org/10.1109/cvpr.2017.723
Selim A, Elgharib M, Doyle L, 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Trans Graph, 35(4), Article 129. https://doi.org/10.1145/2897824.2925968
Simo-Serra E, Iizuka S, Sasaki K, et al., 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans Graph, 35(4), Article 121. https://doi.org/10.1145/2897824.2925972
Wang TC, Liu MY, Zhu JY, et al., 2018. High-resolution image synthesis and semantic manipulation with conditional GANs. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.8798–8807. https://doi.org/10.1109/CVPR.2018.00917
Xian WQ, Sangkloy P, Agrawal V, et al., 2018. TextureGAN: controlling deep image synthesis with texture patches. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.8456–8465. https://doi.org/10.1109/cvpr.2018.00882
Zhang YK, Hu KK, Ren PR, et al., 2017. Layout style modeling for automating banner design. Proc Thematic Workshops of ACM Multimedia, p.451–459. https://doi.org/10.1145/3126686.3126718
Zhao NX, Cao Y, Lau RWH, 2018. What characterizes personalities of graphic designs? ACM Trans Graph, 37(4), Article 116. https://doi.org/10.1145/3197517.3201355
Zhu JY, Krähenbühl P, Shechtman E, et al., 2016. Generative visual manipulation on the natural image manifold. Proc 14th European Conf on Computer Vision, p.597–613. https://doi.org/10.1007/978-3-319-46454-1_36
Zhu JY, Park T, Isola P, et al., 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proc IEEE Int Conf on Computer Vision, p.2223–2232. https://doi.org/10.1109/ICCV.2017.244
Author information
Authors and Affiliations
Corresponding author
Additional information
Compliance with ethics guidelines
Lingyun SUN, Pei CHEN, Wei XIANG, Peng CHEN, Wei-yue GAO, and Ke-jun ZHANG declare that they have no conflict of interest.
Project supported by the National Science and Technology Innovation 2030 Major Project of the Ministry of Science and Technology of China (No. 2018AAA0100703), the National Natural Science Foundation of China (No. 61672451), the Provincial Key Research and Development Plan of Zhejiang Province, China (No. 2019C03137), the China Postdoctoral Science Foundation (No. 2018M630658), and the Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant
Rights and permissions
About this article
Cite this article
Sun, L., Chen, P., Xiang, W. et al. SmartPaint: a co-creative drawing system based on generative adversarial networks. Front Inform Technol Electron Eng 20, 1644–1656 (2019). https://doi.org/10.1631/FITEE.1900386
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.1900386