Skip to main content

Gesture Coding: Easing the Introduction to Block-Based Programming Languages with Motion Controls

Part of the Lecture Notes in Networks and Systems book series (LNNS,volume 594)


Current approaches that introduce young learners to programming depend heavily on an exposure to block-based languages. Most, if not all of these, rely on traditional inputs, such as pointing and clicking with a mouse, and typing commands with a keyboard. While these input mechanics usually do not pose major usability concerns in their adoption, we see an opportunity in exploring novel interaction paradigms for sparking interest in programming within this population. In this paper we present Gesture Coding, an interactive block-based language, inspired by Scratch, that is controlled using Nintendo Switch Joy-Cons. We expect that interactive, playful, and non-traditional input controls would simplify user onboarding, hence easing the acceptance and appropriation of the language. We discuss the usability and perceived usefulness of the natural interaction metaphors built in Gesture Coding for developing short programs, covering sequential and conditional commands. Our results show that, despite requiring more effort to interact with motion-based controls, young learners show a more positive attitude for exploring and connecting blocks when compared to doing the same tasks with a mouse and keyboard. The obtained results are highly promising, highlighting the potential adoption of the tool.


  • Natural user interfaces
  • Gestural inputs
  • Playful interaction
  • Children
  • Design
  • Empirical study

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

  2. 2.


  1. Ab Rahman, M.S., Ali, N.M., Mohd, M.: Natural user interface for children: from requirement to design. In: International Visual Informatics Conference, vol. 10645, pp. 612–624. Springer, Cham (2017).

  2. Barron-Estrada, M.L., Zatarain-Cabada, R., Cardenas-Sainz, B.A.: A natural user interface implementation for an interactive learning environment. In: 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT), pp. 341–343. IEEE (2020)

    Google Scholar 

  3. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol.3(2), 77–101 (2006).

  4. Emerson, A., et al.: Cluster-based analysis of novice coding misconceptions in block-based programming. In: Zhang, J., Sherriff, M., Heckman, S., Cutter, P.A., Monge, A.E. (eds.) Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE 2020, Portland, OR, USA, 11-14 March 2020, pp. 825–831. ACM (2020).

  5. Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 74, 381–391 (1954)

    CrossRef  Google Scholar 

  6. Gomez, M.J., Moresi, M., Benotti, L.: Text-based programming in elementary school: a comparative study of programming abilities in children with and without block-based experience. In: Scharlau, B., McDermott, R., Pears, A., Sabin, M. (eds.) Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, Aberdeen, Scotland, UK, 15-17 July 2019, pp. 402–408. ACM (2019).

  7. Hart, S.G.: Nasa-task load index (NASA-TLX); 20 years later. Proc. Hum. Factors Ergon. Soc. Ann. Meet. 50(9), 904–908 (2006).

    CrossRef  Google Scholar 

  8. Jin, Q., Wang, D., Deng, X., Zheng, N., Chiu, S.: Ar-maze: a tangible programming tool for children based on AR technology. In: Proceedings of the 17th ACM Conference on Interaction Design and Children, pp. 611–616 (2018)

    Google Scholar 

  9. Kakavas, P., Ugolini, F.C.: Computational thinking in primary education: a systematic literature review. Res. Educ. Media 11(2), 64–94 (2019)

    CrossRef  Google Scholar 

  10. Kelleher, C., Pausch, R.: Lowering the barriers to programming: a taxonomy of programming environments and languages for novice programmers. ACM Comput. Surv. (CSUR) 37(2), 83–137 (2005)

    CrossRef  Google Scholar 

  11. Oram, A., Wilson, G.: Making Software: What Really Works, and Why we Believe it. O’Reilly Media, Inc. Sebastopol (2010)

    Google Scholar 

  12. Repenning, A., Basawapatna, A.R.: Smacking screws with hammers: Experiencing affordances of block-based programming through the hourglass challenge. In: Sherriff, M., Merkle, L.D., Cutter, P.A., Monge, A.E., Sheard, J. (eds.) SIGCSE 2021: The 52nd ACM Technical Symposium on Computer Science Education, Virtual Event, USA, 13-20 March 2021, pp. 267–273. ACM (2021).

  13. Resnick, M., Maloney, J.H., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J.S., Silverman, B., Kafai, Y.B.: Scratch: programming for all. Commun. ACM 52(11), 60–67 (2009).

    CrossRef  Google Scholar 

  14. Rich, K.M., Strickland, C., Binkowski, T.A., Moran, C., Franklin, D.: K-8 learning trajectories derived from research literature: sequence, repetition, conditionals. ACM Inroads 9(1), 46–55 (2018)

    CrossRef  Google Scholar 

  15. Robinson, W.: From scratch to patch: Easing the blocks-text transition. In: Vahrenhold, J., Barendsen, E. (eds.) Proceedings of the 11th Workshop in Primary and Secondary Computing Education, WiPSCE 2016, Münster, Germany, 13-15 October 2016, pp. 96–99. ACM (2016).

  16. Sullivan, A., Johnson, E.K.: Beaded adventures: crafting stem learning. In: Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 351–358 (2019)

    Google Scholar 

  17. Williams, K.D.: The effects of video game controls on hostility, identification, and presence. Mass Commun. Soc. 16(1), 26–48 (2013)

    CrossRef  Google Scholar 

  18. Yu, J., Zheng, C., Tamashiro, M.A., Gonzalez-millan, C., Roque, R.: Codeattach: engaging children in computational thinking through physical play activities. In: Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 453–459 (2020)

    Google Scholar 

Download references


This work has been partially funded by FONDECYT – Initiation to Research Program, grant: 11190248.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Sebastián Toro-Guajardo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Toro-Guajardo, S., Lizama, E., Gutierrez, F.J. (2023). Gesture Coding: Easing the Introduction to Block-Based Programming Languages with Motion Controls. In: Bravo, J., Ochoa, S., Favela, J. (eds) Proceedings of the International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2022). UCAmI 2022. Lecture Notes in Networks and Systems, vol 594. Springer, Cham.

Download citation