Skip to main content
SpringerLink
Log in

SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches

  • Conference paper
  • First Online:
HCI International 2021 - Late Breaking Papers: Cognition, Inclusion, Learning, and Culture (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13096))

Included in the following conference series:

Abstract

In this work, we propose an interactive general instruction framework SketchMeHow to guidance the common users to complete the daily tasks in real-time. In contrast to the conventional augmented reality-based instruction systems, the proposed framework utilizes the user sketches as system inputs to acquire the users’ production intentions from the drawing interfaces. Given the user sketches, the designated task instruction can be analyzed based on the subtask division and spatial localization for each task. The projector-camera system is adopted in the projection guidance to the end-users with the spatial augmented reality technology. To verify the proposed framework, we conducted two case studies of domino arrangement and bento production. From our user studies, the proposed systems can help novice users complete the tasks efficiently with user satisfaction. We believe the proposed SketchMeHow can broaden the research topics in sketch-based real-world applications in human-computer interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ajisaka, S., et al.: Learning Rubik’s cube through user operation history. In: 2020 Nicograph International (NicoInt), pp. 43–46. IEEE (2020). https://doi.org/10.1109/NicoInt50878.2020.00015

  2. Blattgerste, J., Renner, P., Strenge, B., Pfeiffer, T.: In-situ instructions exceed side-by-side instructions in augmented reality assisted assembly. In: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, PETRA 2018, pp. 133–140. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3197768.3197778

  3. Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., Alexa, M.: Sketch-based shape retrieval. ACM Trans. Graph. (TOG) 31(4), 31:1–31:10 (2012). https://doi.org/10.1145/2185520.2185527

  4. Flagg, M., Rehg, J.M.: Projector-guided painting. In: Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, pp. 235–244 (2006). https://doi.org/10.1145/1166253.1166290

  5. Funk, M., Kosch, T., Schmidt, A.: Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions, UbiComp 2016, pp. 934–939. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2971648.2971706

  6. Ghosh, A., et al.: Interactive sketch & fill: multiclass sketch-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1171–1180. IEEE, Long Beach (2019). https://doi.org/10.1109/ICCV.2019.00126

  7. He, Y., Xie, H., Zhang, C., Yang, X., Miyata, K.: Sketch-based normal map generation with geometric sampling. In: Nakajima, M., Kim, J.G., Lie, W.N., Kemao, Q. (eds.) International Workshop on Advanced Imaging Technology (IWAIT) 2021, vol. 11766, pp. 261–266. International Society for Optics and Photonics, SPIE (2021). https://doi.org/10.1117/12.2590760

  8. He, Z., Xie, H., Miyata, K.: Interactive projection system for calligraphy practice. In: 2020 Nicograph International (NicoInt), pp. 55–61 (2020).https://doi.org/10.1109/NicoInt50878.2020.00018

  9. Hu, Z., Xie, H., Fukusato, T., Sato, T., Igarashi, T.: Sketch2VF: sketch-based flow design with conditional generative adversarial network. Comput. Anim. Virtual Worlds 30(3–4), e1889 (2019). https://doi.org/10.1002/cav.1889

  10. Huang, Z., et al.: dualFace: two-stage drawing guidance for freehand portrait sketching. Comput. Vis. Media (2021). https://doi.org/10.1007/s41095-021-0227-7

  11. Igarashi, T., Matsuoka, S., Tanaka, H.: Teddy: a sketching interface for 3D freeform design. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 409–416. ACM, New York (1999). https://doi.org/10.1145/311535.311602

  12. Jasche, F., Hoffmann, S., Ludwig, T., Wulf, V.: Comparison of different types of augmented reality visualizations for instructions. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 2021. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3411764.3445724

  13. Jones, B., et al.: RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 637–644. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2642918.2647383

  14. Li, S., et al.: Skeleton-based interactive fabrication for large-scale newspaper sculpture. In: 2021 Nicograph International (NicoInt). IEEE (2021). https://doi.org/10.1109/NICOINT52941.2021.00021

  15. Luo, S., Xie, H., Miyata, K.: Sketch-based anime hairstyle editing with generative inpainting. In: 2021 Nicograph International (NicoInt). IEEE (2021). https://doi.org/10.1109/NICOINT52941.2021.00009

  16. Noh, S., Hashimoto, S., Yamanaka, D., Kamiyama, Y., Inami, M., Igarashi, T.: Design and enhancement of painting interface for room lights. Vis. Comput. 30(5), 467–478 (2013). https://doi.org/10.1007/s00371-013-0872-7

    Article  Google Scholar 

  17. Peng, Y., et al.: Sketch2Domino: interactive chain reaction design and guidance. In: 2020 Nicograph International (NicoInt), pp. 32–38. IEEE (2020). https://doi.org/10.1109/NicoInt50878.2020.00013

  18. Piper, B., Ratti, C., Ishii, H.: Illuminating clay: a 3-D tangible interface for landscape analysis. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2002, pp. 355–362. Association for Computing Machinery, New York (2002). https://doi.org/10.1145/503376.503439

  19. Portenier, T., Hu, Q., SzabĂł, A., Bigdeli, S.A., Favaro, P., Zwicker, M.: FaceShop: deep sketch-based face image editing. ACM Trans. Graph. (TOG) 37(4) (2018). https://doi.org/10.1145/3197517.3201393

  20. Rivers, A., Adams, A., Durand, F.: Sculpting by numbers. ACM Trans. Graph. 31(6) (2012). https://doi.org/10.1145/2366145.2366176

  21. Sakamoto, D., Honda, K., Inami, M., Igarashi, T.: Sketch and run: a stroke-based interface for home robots. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2009, pp. 197–200. Association for Computing Machinery, New York (2009). https://doi.org/10.1145/1518701.1518733

  22. Sano, Y., Sato, K., Shiraishi, R., Otsuki, M.: Sports support system: augmented ball game for filling gap between player skill levels. In: Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces, pp. 361–366 (2016). https://doi.org/10.1145/2992154.2996781

  23. Smith, E., Semple, G., Evans, D., McRae, K., Blackwell, P.: Augmented instructions: analysis of performance and efficiency of assembly tasks. In: Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2020. LNCS, vol. 12191, pp. 166–177. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49698-2_12

    Chapter  Google Scholar 

  24. Wang, H., Kanayama, H., Peng, Y., Xie, H., Okada, S., Miyata, K.: Sketch2bento: Sketch-based arrangement guidance for lunch boxes. In: 2021 Nicograph International (NicoInt). IEEE (2021). https://doi.org/10.1109/NICOINT52941.2021.00032

  25. Xie, H., Peng, Y., Chen, N., Xie, D., Chang, C.M., Miyata, K.: BalloonFAB: digital fabrication of large-scale balloon art. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–6 (2019). https://doi.org/10.1145/3290607.3312947

  26. Xie, J., Hertzmann, A., Li, W., Winnemöller, H.: PortraitSketch: face sketching assistance for novices. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST), pp. 407–417. ACM, New York (2014). https://doi.org/10.1145/2642918.2647399

  27. Xing, J., Kazi, R.H., Grossman, T., Wei, L.Y., Stam, J., Fitzmaurice, G.: Energy-brushes: interactive tools for illustrating stylized elemental dynamics. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST 2016, pp. 755–766. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2984511.2984585

Download references

Acknowledgement

The authors thank the anonymous reviewers for their valuable comments. We greatly thank Yuki Mishima, Yamato Igarashi, Ryoma Miyauchi, Masahiro Okawa, Haruka Kanayama, Shogo Yoshida, Shogo Okada for idea discussion and experiment design. We would thank Ze Wang for sharing the codes of drawer interface. This project has been partially funded by JAIST Research Grant, Hayao Nakayama Foundation for Science & Technology and Culture Grant A and JSPS KAKENHI grant JP20K19845, Japan.

Author information

Authors and Affiliations

  1. Japan Advanced Institute of Science and Technology, Ishikawa, 9231292, Japan

    Haoran Xie, Yichen Peng, Hange Wang & Kazunori Miyata

Authors
  1. Haoran Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yichen Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hange Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kazunori Miyata
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haoran Xie .

Editor information

Editors and Affiliations

  1. University of Crete and Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. Coventry University, Coventry, UK

    Don Harris

  3. Safety and Accident Investigation Centre, Cranfield University, Cranfield, UK

    Wen-Chin Li

  4. Soar Technology Inc., Orlando, FL, USA

    Dylan D. Schmorrow

  5. Design Interactive, Inc., Orlando, FL, USA

    Cali M. Fidopiastis

  6. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  7. Tsinghua University, Beijing, China

    Qin Gao

  8. Chongqing University, Chongqing, China

    Jia Zhou

  9. Department of Multimedia and Graphic Arts, Cyprus University of Technology, Limassol, Cyprus

    Panayiotis Zaphiris

  10. Department of Multimedia and Graphic Arts, Cyprus University of Technology, Limassol, Cyprus

    Andri Ioannou

  11. Soar Technology Inc., Orlando, FL, USA

    Robert A. Sottilare

  12. MMS, Fraunhofer FKIE, Wachtberg, Nordrhein-Westfalen, Germany

    Jessica Schwarz

  13. Industrial Design, Eindhoven University of Technology, Eindhoven, Noord-Brabant, The Netherlands

    Matthias Rauterberg

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xie, H., Peng, Y., Wang, H., Miyata, K. (2021). SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Cognition, Inclusion, Learning, and Culture. HCII 2021. Lecture Notes in Computer Science(), vol 13096. Springer, Cham. https://doi.org/10.1007/978-3-030-90328-2_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90328-2_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90327-5

  • Online ISBN: 978-3-030-90328-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation