Abstract
Image-guided drawing can compensate for a lack of skill but often requires a significant number of repetitive strokes to create textures. Existing automatic stroke synthesis methods are usually limited to predefined styles or require indirect manipulation that may break the spontaneous flow of drawing. We present an assisted drawing system to autocomplete repetitive short strokes during a user’s normal drawing process. Users draw over a reference image as usual; at the same time, our system silently analyzes the input strokes and the reference to infer strokes that follow the user’s input style when certain repetition is detected. Users can accept, modify, or ignore the system’s predictions and continue drawing, thus maintaining fluid control over drawing. Our key idea is to jointly analyze image regions and user input history to detect and predict repetition. The proposed system can effectively reduce the user’s workload when drawing repetitive short strokes, helping users to create results with rich patterns.
References
Dunn, A. Pen and Ink Drawing: A Simple Guide. New Jersey: Three Minds Press, 2015.
Adobe. Paint stylized strokes with the Art History Brush. 2017. Available at https://helpx.adobe.com/photoshop/using/painting-stylized-strokes-art-history.html.
Martín, D.; Arroyo, G.; Rodríguez, A.; Isenberg, T. A survey of digital stippling. Computers & Graphics Vol. 67, 24–44, 2017.
Hertzmann, A.; Jacobs, C. E.; Oliver, N.; Curless, B.; Salesin, D. H. Image analogies. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 327–340, 2001.
Fišer, J.; Jamriška, O.; Lukác, M.; Shechtman, E.; Asente, P.; Lu, J.; Sýkora, D. StyLit: Illumination-guided example-based stylization of 3D renderings. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 92, 2016.
Gerl, M.; Isenberg, T. Interactive example-based hatching. Computers & Graphics Vol. 37, Nos. 1–2, 65–80, 2013.
Hegde, S.; Gatzidis, C.; Tian, F. Painterly rendering techniques: A state-of-the-art review of current approaches. Computer Animation and Virtual Worlds Vol. 24, No. 1, 43–64, 2013.
Kazi, R. H.; Igarashi, T.; Zhao, S. D.; Davis, R. Vignette: Interactive texture design and manipulation with freeform gestures for pen-and-ink illustration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1727–1736, 2012.
Xing, J.; Chen, H. T.; Wei, L. Y. Autocomplete painting repetitions. ACM Transactions on Graphics Vol. 33, No. 6, Article No. 172, 2014.
Xie, J.; Hertzmann, A.; Li, W.; Winnemöller, H. PortraitSketch: Face sketching assistance for novices. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 407–417, 2014.
Kang, H. W.; He, W. J.; Chui, C. K.; Chakraborty, U. K. Interactive sketch generation. The Visual Computer Vol. 21, Nos. 8–10, 821–830, 2005.
Su, Q.; Li, W. H. A.; Wang, J.; Fu, H. EZ-sketching: Three-level optimization for error-tolerant image tracing. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 54, 2014.
Li, G. B.; Bi, S.; Wang, J.; Xu, Y. Q.; Yu, Y. Z. ColorSketch: A drawing assistant for generating color sketches from photos. IEEE Computer Graphics and Applications Vol. 37, No. 3, 70–81, 2017.
Iarussi, E.; Bousseau, A.; Tsandilas, T. The drawing assistant: Automated drawing guidance and feedback from photographs. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, 183–192, 2013.
Matsui, Y.; Shiratori, T.; Aizawa, K. DrawFromDrawings: 2D drawing assistance via stroke interpolation with a sketch database. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 7, 1852–1862, 2017.
Williford, B.; Doke, A.; Pahud, M.; Hinckley, K.; Hammond, T. DrawMyPhoto: Assisting novices in drawing from photographs. In: Proceedings of the 2019 on Creativity and Cognition, 198–209, 2019.
Haeberli, P. Paint by numbers: Abstract image representations. ACM SIGGRAPH Computer Graphics Vol. 24, No. 4, 207–214, 1990.
Benedetti, L.; Winnemöller, H.; Corsini, M.; Scopigno, R. Painting with Bob: Assisted creativity for novices. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 419–428, 2014.
Tsai, H. C.; Lee, Y. H.; Lee, R. R.; Chu, H. K. User-guided line abstraction using coherence and structure analysis. Computational Visual Media Vol. 3, No. 2, 177–188, 2017.
Kyprianidis, J. E.; Collomosse, J.; Wang, T. H.; Isenberg, T. State of the “art”: A taxonomy of artistic stylization techniques for images and video. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 5, 866–885, 2013.
Salisbury, M. P.; Wong, M. T.; Hughes, J. F.; Salesin, D. H. Orientable textures for image-based pen-and-ink illustration. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 401–406, 1997.
Hiller, S.; Hellwig, H.; Deussen, O. Beyond stippling—Methods for distributing objects on the plane. Computer Graphics Forum Vol. 22, No. 3, 515–522, 2003.
Kalogerakis, E.; Nowrouzezahrai, D.; Breslav, S.; Hertzmann, A. Learning hatching for pen-and-ink illustration of surfaces. ACM Transactions on Graphics Vol. 31, No. 1, Article No. 1, 2012.
Gatys, L. A.; Ecker, A. S.; Bethge, M.; Hertzmann, A.; Shechtman, E. Controlling perceptual factors in neural style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3730–3738, 2017.
Kaspar, A.; Neubert, B.; Lischinski, D.; Pauly, M.; Kopf, J. Self tuning texture optimization. Computer Graphics Forum Vol. 34, No. 2, 349–359, 2015.
Nancel, M.; Cockburn, A. Causality: A conceptual model of interaction history. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1777–1786, 2014.
Xing, J.; Wei, L. Y.; Shiratori, T.; Yatani, K. Autocomplete hand-drawn animations. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 169, 2015.
Peng, M.; Wei, L. Y.; Kazi, R. H.; Kim, V. G. Autocomplete animated sculpting. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, 760–777, 2020.
Peng, M. Q.; Xing, J.; Wei, L. Y. Autocomplete 3D sculpting. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 132, 2018.
Suzuki, R.; Yatani, K.; Gross, M. D.; Yeh, T. Tabby: Explorable design for 3D printing textures. In: Proceedings of the 26th Pacific Conference on Computer Graphics and Applications: Short Papers, 29–32, 2018.
Fišer, J.; Asente, P.; Sýkora, D. ShipShape: A drawing beautification assistant. In: Proceedings of the International Symposium on Sketch-based Interfaces and Modeling, 49–57, 2015.
Zitnick, C. L. Handwriting beautification using token means. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 53, 2013.
Barla, P.; Breslav, S.; Markosian, L.; Thollot, J. Interactive hatching and stippling by example. INRIA Research Report RR-6461, 2006.
Ijiri, T.; Mêch, R.; Igarashi, T.; Miller, G. An example-based procedural system for element arrangement. Computer Graphics Forum Vol. 27, No. 2, 429–436, 2008.
Alves dos Passos, V.; Walter, M.; Sousa, M. C. Sample-based synthesis of illustrative patterns. In: Proceedings of the 18th Pacific Conference on Computer Graphics and Applications, 109–116, 2010.
Hsu, C. Y.; Wei, L. Y.; You, L.; Zhang, J. J. Autocomplete element fields. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–13, 2020.
Mortensen, E. N.; Barrett, W. A. Intelligent scissors for image composition. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 191–198, 1995.
Ma, C.; Wei, L. Y.; Tong, X. Discrete element textures. ACM Transactions on Graphics Vol. 30, No. 4, Article No. 62, 2011.
Zhao, M.; Zhu, S. C. Customizing painterly rendering styles using stroke processes. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-Photorealistic Animation and Rendering, 137–146, 2011.
Ma, C.; Wei, L. Y.; Lefebvre, S.; Tong, X. Dynamic element textures. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 90, 2013.
Zhao, H. S.; Shi, J. P.; Qi, X. J.; Wang, X. G.; Jia, J. Y. Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6230–6239, 2017.
Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. In: Proceedings of the ACM SIGGRAPH 2004 Papers, 309–314, 2004.
Kyprianidis, J. E.; Kang, H. Image and video abstraction by coherence-enhancing filtering. Computer Graphics Forum Vol. 30, No. 2, 593–602, 2011.
Hart, S. G.; Staveland, L. E. Development of NASA-TLX (task load index): Results of empirical and theoretical research. In: Advances in Psychology. Amsterdam: Elsevier, 139–183, 1988.
Chen, X.; Zhao, Z. Y.; Yu, F. W.; Zhang, Y. L.; Duan, M. N. Conditional diffusion for interactive segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7325–7334, 2021.
Tu, P. H.; Wei, L. Y.; Yatani, K.; Igarashi, T.; Zwicker, M. Continuous curve textures. ACM Transactions on Graphics Vol. 39, No. 6, Article No. 168, 2020.
Acknowledgements
We are grateful to Li-Yi Wei for his insightful comments and suggestions. We also thank the anonymous reviewers for feedback, and funding from Adobe Research and the Deutsche Forschungsgemeinschaft, Project-ID 251654672-TRR 161.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Yilan Chen received her Ph.D. degree from the School of Creative Media, City University of Hong Kong, Hong Kong, China. Her research interests include computer graphics and human-computer interaction.
Kin Chung Kwan received his B.Sc. and Ph.D. degrees from the Chinese University of Hong Kong in 2009 and 2015, respectively. He is now a postdoctoral researcher at the University of Konstanz. His research interests include computer graphics and human-computer interaction.
Hongbo Fu is a professor in the School of Creative Media, City University of Hong Kong. Previously, he had postdoctoral research training at the Imager Lab, University of British Columbia, Canada, and the Department of Computer Graphics, Max-Planck-Institut Informatik, Germany. He received his Ph.D. degree in computer science from Hong Kong University of Science and Technology in 2007 and his B.S. degree in information sciences from Peking University, China, in 2002. His primary research interests fall in the fields of computer graphics and human-computer interaction. He has served as an associate editor of The Visual Computer, Computers & Graphics, and Computer Graphics Forum.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Chen, Y., Kwan, K.C. & Fu, H. Autocompletion of repetitive stroking with image guidance. Comp. Visual Media 9, 581–596 (2023). https://doi.org/10.1007/s41095-022-0288-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-022-0288-2
Keywords
- interaction
- autocompletion
- digital drawing
- prediction
- texture synthesis