Skip to main content
SpringerLink
Log in

Sketch-Based Creativity Support Tools Using Deep Learning

  • Chapter
  • First Online:
Artificial Intelligence for Human Computer Interaction: A Modern Approach

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines’ ability in understanding and generating visual content. An exciting area of development explores deep-learning approaches used to model human sketches, opening opportunities for creative applications. This chapter describes three fundamental steps in developing deep-learning-driven creativity support tools that consume and generate sketches: (1) a data collection effort that generated a new paired dataset between sketches and mobile user interfaces; (2) a sketch-based user interface retrieval system adapted from state-of-the-art computer vision techniques; and, (3) a conversational sketching system that supports the novel interaction of a natural-language-based sketch/critique authoring process. In this chapter, we survey relevant prior work in both the deep-learning and human-computer interaction communities, document the data collection process and the systems’ architectures in detail, present qualitative and quantitative results, and paint the landscape of several future research directions in this exciting area.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This total of 3702 sketches differs from original Swire publication [17]. We discovered that 100 trial sketches from a pilot study were accidentally included in the original stated total and we have corrected the numbers in this chapter.

  2. 2.

    An encoding of class information that is an array of bits where only the corresponding position for the class to be encoded is 1, and all other bits are 0s.

  3. 3.

    For some object categories, we found that increasing the KL weight to 1.0 improves the authors’ perceived quality of generated sketches.

References

  1. von Ahn L, Dabbish L (2008) Designing games with a purpose. Commun ACM 51(8):58–67. https://doi.org/10.1145/1378704.1378719

  2. Aksan E, Deselaers T, Tagliasacchi A, Hilliges O (2020) CoSE: compositional stroke embeddings. Adv Neural Inf Process Syst 33

    Google Scholar 

  3. Bonnardel N (1999) Creativity in design activities: the role of analogies in a constrained cognitive environment. In: Proceedings of the 3rd conference on creativity & cognition, C&C ’99. ACM, New York, NY, USA, pp 158–165. https://doi.org/10.1145/317561.317589

  4. Buxton B (2007) Sketching user experiences: getting the design right and the right design. Morgan Kaufmann Publishers Inc., San Francisco

    Google Scholar 

  5. Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 6:679–698

    Article  Google Scholar 

  6. Deka B, Huang Z, Kumar R (2016) ERICA: interaction mining mobile apps. In: Proceedings of the 29th annual symposium on user interface software and technology, UIST’16. ACM, New York, NY, USA, pp 767–776. https://doi.org/10.1145/2984511.2984581

  7. Deka B, Huang Z, Franzen C, Hibschman J, Afergan D, Li Y, Nichols J, Kumar R (2017) Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th annual ACM symposium on user interface software and technology, UIST’17. ACM, New York, NY, USA, pp 845–854. https://doi.org/10.1145/3126594.3126651

  8. Dow SP, Glassco A, Kass J, Schwarz M, Schwartz DL, Klemmer SR (2010) Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans Comput-Hum Interact 17(4):18:1–18:24. https://doi.org/10.1145/1879831.1879836

  9. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects? ACM Trans Graph (Proc SIGGRAPH) 31(4):44:1–44:10

    Google Scholar 

  10. Fernquist J, Grossman T, Fitzmaurice G (2011) Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning. In: Proceedings of the 24th annual ACM symposium on user interface software and technology, UIST’11. ACM, New York, NY, USA, pp 373–382. https://doi.org/10.1145/2047196.2047245

  11. Gao C, Liu Q, Xu Q, Wang L, Liu J, Zou C (2020) SketchyCOCO: image generation from freehand scene sketches. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5174–5183

    Google Scholar 

  12. Gervais P, Deselaers T, Aksan E, Hilliges O (2020) The DIDI dataset: digital ink diagram data. arXiv:200209303

  13. Ha D, Eck D (2018) A neural representation of sketch drawings. In: 6th international conference on learning representations, ICLR 2018, conference track proceedings, Vancouver, BC, Canada, April 30–May 3, 2018. https://openreview.net/forum?id=Hy6GHpkCW

  14. Hennessey JW, Liu H, Winnemöller H, Dontcheva M, Mitra NJ (2017) How2Sketch: generating easy-to-follow tutorials for sketching 3D objects. In: Symposium on interactive 3D graphics and games

    Google Scholar 

  15. Herring SR, Chang CC, Krantzler J, Bailey BP (2009) Getting inspired!: understanding how and why examples are used in creative design practice. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’09. ACM, New York, NY, USA, pp 87–96. https://doi.org/10.1145/1518701.1518717

  16. Huang F, Canny JF (2019) Sketchforme: composing sketched scenes from text descriptions for interactive applications. In: Proceedings of the 32nd annual ACM symposium on user interface software and technology, UIST’19. Association for Computing Machinery, New York, NY, USA, pp 209–220. https://doi.org/10.1145/3332165.3347878

  17. Huang F, Canny JF, Nichols J (2019) Swire: sketch-based user interface retrieval. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI’19. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3290605.3300334

  18. Jongejan J, Rowley H, Kawashima T, Kim J, Fox-Gieg N (2016) The quick, draw! - AI experiment. https://quickdraw.withgoogle.com/

  19. Kim JH, Kitaev N, Chen X, Rohrbach M, Zhang BT, Tian Y, Batra D, Parikh D (2019) CoDraw: collaborative drawing as a testbed for grounded goal-driven communication. In: Proceedings of the 57th annual meeting of the association for computational linguistics. Association for Computational Linguistics, Florence, Italy, pp 6495–6513. https://doi.org/10.18653/v1/P19-1651

  20. Kumar R, Talton JO, Ahmad S, Klemmer SR (2011) Bricolage: example-based retargeting for web design. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’11. ACM, New York, NY, USA, pp 2197–2206. https://doi.org/10.1145/1978942.1979262

  21. Landay JA (1996) SILK: sketching interfaces like krazy. In: Conference companion on human factors in computing systems, CHI’96. ACM, New York, NY, USA, pp 398–399. https://doi.org/10.1145/257089.257396

  22. Lasecki WS, Kim J, Rafter N, Sen O, Bigham JP, Bernstein MS (2015) Apparition: crowdsourced user interfaces that come to life as you sketch them. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI’15. ACM, New York, NY, USA, pp 1925–1934. https://doi.org/10.1145/2702123.2702565

  23. Lee YJ, Zitnick CL, Cohen MF (2011) ShadowDraw: real-time user guidance for freehand drawing. ACM Trans Graph 30(4):27:1–27:10. https://doi.org/10.1145/2010324.1964922

  24. Li M, Lin Z, Mech R, Yumer E, Ramanan D (2019) Photo-sketching: inferring contour drawings from images. In: 2019 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1403–1412

    Google Scholar 

  25. Limpaecher A, Feltman N, Treuille A, Cohen M (2013) Real-time drawing assistance through crowdsourcing. ACM Trans Graph 32(4):54:1–54:8. https://doi.org/10.1145/2461912.2462016

  26. Lin J, Newman MW, Hong JI, Landay JA (2000) DENIM: finding a tighter fit between tools and practice for web site design. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’00. ACM, New York, NY, USA, pp 510–517. https://doi.org/10.1145/332040.332486

  27. Munoz-Salinas R (2012) ArUco: a minimal library for augmented reality applications based on OpenCV. Universidad de Córdoba

    Google Scholar 

  28. Newman MW, Landay JA (2000) Sitemaps, storyboards, and specifications: a sketch of web site design practice. In: Proceedings of the 3rd conference on designing interactive systems: processes, practices, methods, and techniques, DIS’00. ACM, New York, NY, USA, pp 263–274. https://doi.org/10.1145/347642.347758

  29. Nguyen TA, Csallner C (2015) Reverse engineering mobile application user interfaces with REMAUI (T). In: 2015 30th IEEE/ACM international conference on automated software engineering (ASE), pp 248–259. https://doi.org/10.1109/ASE.2015.32

  30. Pennington J, Socher R, Manning CD (2014) GloVe: global vectors for word representation. In: Empirical methods in natural language processing (EMNLP), pp 1532–1543. http://www.aclweb.org/anthology/D14-1162

  31. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein MS, Berg AC, Li F (2014) ImageNet large scale visual recognition challenge. CoRR abs/1409.0575. arXiv:1409.0575

  32. Sain A, Bhunia AK, Yang Y, Xiang T, Song YZ (2020) Cross-modal hierarchical modelling for fine-grained sketch based image retrieval. In: Proceedings of the 31st British machine vision virtual conference (BMVC 2020). British Machine Vision Association, pp 1–14

    Google Scholar 

  33. Sangkloy P, Burnell N, Ham C, Hays J (2016) The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans Graph 35(4):119:1–119:12. https://doi.org/10.1145/2897824.2925954

  34. Schroff F, Kalenichenko D, Philbin J (2015) FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

    Google Scholar 

  35. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations

    Google Scholar 

  36. Su Q, Li WHA, Wang J, Fu H (2014) EZ-sketching: three-level optimization for error-tolerant image tracing. ACM Trans Graph 33(4):54:1–54:9. https://doi.org/10.1145/2601097.2601202

  37. Swearngin A, Dontcheva M, Li W, Brandt J, Dixon M, Ko AJ (2018) Rewire: interface design assistance from examples. In: Proceedings of the 2018 CHI conference on human factors in computing systems, CHI’18. ACM, New York, NY, USA, pp 504:1–504:12. https://doi.org/10.1145/3173574.3174078

  38. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30. Curran Associates, Inc., pp 5998–6008. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf

  39. Xie J, Hertzmann A, Li W, Winnemöller H (2014) PortraitSketch: face sketching assistance for novices. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, UIST’14. ACM, New York, NY, USA, pp 407–417. https://doi.org/10.1145/2642918.2647399

  40. Yu Q, Liu F, Song YZ, Xiang T, Hospedales T, Loy CC (2016) Sketch me that shoe. In: Computer vision and pattern recognition

    Google Scholar 

  41. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV)

    Google Scholar 

  42. Zou C, Yu Q, Du R, Mo H, Song YZ, Xiang T, Gao C, Chen B, Zhang H (2018) SketchyScene: richly-annotated scene sketches. In: ECCV. Springer International Publishing, pp 438–454. https://doi.org/10.1007/978-3-030-01267-0_26, https://github.com/SketchyScene/SketchyScene

Download references

Author information

Authors and Affiliations

  1. University of California, Berkeley, USA

    Forrest Huang, Eldon Schoop & John Canny

  2. Google Brain, Tokyo, Japan

    David Ha

  3. Apple, Inc., Cupertino, USA

    Jeffrey Nichols

Authors
  1. Forrest Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eldon Schoop
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Ha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeffrey Nichols
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. John Canny
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Forrest Huang .

Editor information

Editors and Affiliations

  1. Google Research (United States), Mountain View, CA, USA

    Yang Li

  2. Advanced Interactive Technologies Lab, ETH Zurich, Zurich, Switzerland

    Otmar Hilliges

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Huang, F., Schoop, E., Ha, D., Nichols, J., Canny, J. (2021). Sketch-Based Creativity Support Tools Using Deep Learning. In: Li, Y., Hilliges, O. (eds) Artificial Intelligence for Human Computer Interaction: A Modern Approach. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-82681-9_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82681-9_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82680-2

  • Online ISBN: 978-3-030-82681-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation