Considerations for the Design of Gesture-Augmented Learning Environments

Living reference work entry


A new genre of learning technologies is emerging that integrates computer simulations with physical or “embodied” interactions such as hand gestures. While this genre presents new opportunities for innovative digital environments that physically engage learners, there is very little guidance on how to design these environments to optimize learning. This chapter presents considerations specifically for the design of gesture-augmented learning environments. Design considerations are discussed in three main areas related to (1) what gestural interactions are used, (2) constraints of the learning environment, and (3) what social and contextual supports are offered. The term considerations is used rather than principles or guidelines to highlight the real tradeoffs and legitimate decisions to be made when designing gesture-based technologies for learning. These considerations are illustrated with detailed examples from a project that implements students’ gestures as the primary method of interaction with digital science simulations. Although the examples specifically pertain to learning in science, the considerations are framed such that they can be applied to a broad range of domains.


Computer simulations Embodied learning Explanations Gesture Science education 



This material is based upon work supported by the National Science Foundation under Grant No. DUE-1432424. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.


  1. Abrahamson, D., & Lindgren, R. (2014). Embodiment and embodied design. In Cambridge handbook of the Learning Sciences (2nd ed., pp. 358–376). Cambridge, UK: Cambridge University Press. 10.1017/CBO9781139519526.022.CrossRefGoogle Scholar
  2. Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: Evidence from learners’ and teachers’ gestures. The Journal of the Learning Sciences, 21, 247–286. 10.1080/10508406.2011.611446.CrossRefGoogle Scholar
  3. Black, J. B., Segal, A., Vitale, J., & Fadjo, C. (2012). Embodied cognition and learning environment design. In D. Jonassen & S. Lamb (Eds.), Theoretical foundations of student-centered learning environments (2nd ed., pp. 198–223). New York: Routledge.Google Scholar
  4. Chase, C. C., Chin, D. B., Oppezzo, M. A., & Schwartz, D. L. (2009). Teachable agents and the protégé effect: Increasing the effort towards learning. Journal of Science Education and Technology, 18, 334–352.CrossRefGoogle Scholar
  5. Clement, J. (2013). Roles for explanatory models and analogies in conceptual change. In S. Vosniadou (Ed.), International handbook of research on conceptual change (2nd ed., pp. 412–446). New York: Routledge.Google Scholar
  6. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning and instruction: Essays in honour of Robert Glaser (pp. 453–494). Hillsdale, NJ: Lawrence Erlbaum & Associates.Google Scholar
  7. Dourish, P. (2001). Where the action is: The foundations of embodied interaction. Cambridge, MA: MIT Press.Google Scholar
  8. Engelkamp, J., & Zimmer, H. D. (1994). The human memory: A multi-modal approach. Seattle, WA: Hogrefe & Huber.Google Scholar
  9. Flood, V. J., Amar, F. G., Nemirovsky, R., Harrer, B. W., Bruce, M. R. M., & Wittmann, M. C. (2014). Paying attention to gesture when students talk chemistry: Interactional resources for responsive teaching. Journal of Chemical Education, 92, 11–22. 10.1021/ed400477b.CrossRefGoogle Scholar
  10. Gallagher, S. (2005). How the body shapes the mind. Oxford: Oxford University Press.CrossRefGoogle Scholar
  11. Gibson, J. J. (1962). Observations on active touch. Psychological Review, 69, 477–491.CrossRefGoogle Scholar
  12. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.Google Scholar
  13. Goldin-Meadow, S. (2005). Hearing gesture: How our hands help us think. Cambridge, MA: Harvard University Press.Google Scholar
  14. Goldin-Meadow, S., Cook, S. W., & Mitchell, Z. a. (2009). Gesturing gives children new ideas about math. Psychological Science, 20, 267–272. 10.1111/j.1467-9280.2009.02297.x.CrossRefGoogle Scholar
  15. Han, I., & Black, J. B. (2011). Incorporating haptic feedback in simulation for learning physics. Computers & Education, 57, 2281–2290.CrossRefGoogle Scholar
  16. Hannafin, M., & Peck, K. (1988). The design, development, and evaluation of instructional software. New York: Macmillan.Google Scholar
  17. Hartman, B. A., Miller, B. K., & Nelson, D. L. (2000). The effects of hands-on occupation versus demonstration on children’s recall memory. American Journal of Occupational Therapy, 54, 477–483.CrossRefGoogle Scholar
  18. Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15, 495–514. 10.3758/PBR.15.3.495.CrossRefGoogle Scholar
  19. Isbister, K., & Mueller, F. F. (2015). Guidelines for the design of movement-based games and their relevance to HCI. Human Computer Interaction, 30, 366–399.CrossRefGoogle Scholar
  20. Johnson-Glenberg, M. C., Birchfield, D. A., Tolentino, L., & Koziupa, T. (2014a). Collaborative embodied learning in mixed reality motion-capture environments: Two science studies. Journal of Educational Psychology, 106, 86–104. 10.1037/a0034008.CrossRefGoogle Scholar
  21. Johnson-Glenberg, M. C., Savio-Ramos, C., & Henry, H. (2014b). “Alien Health”: A nutrition instruction exergame using the kinect sensor. Games for Health: Research, Development, and Clinical Applications, 3, 241–251.CrossRefGoogle Scholar
  22. Johnson-Glenberg, M. C., Megowan-Romanowicz, C., Birchfield, D. A., & Savio-Ramos, C. (2016). Effects of embodied learning and digital platform on the retention of physics content: Centripetal force. Frontiers in Psychology, 7, 1–22. 10.3389/fpsyg.2016.01819.CrossRefGoogle Scholar
  23. Kim, M., Roth, W. M., & Thom, J. (2011). Children’s gestures and the embodied knowledge of geometry. International Journal of Science and Mathematics Education, 9, 207–238. 10.1007/s10763-010-9240-5.CrossRefGoogle Scholar
  24. Lindgren, R. (2015). Getting into the cue: Embracing technology-facilitated body movements as a starting point for learning. In V. R. Lee (Ed.), Learning technologies and the body: Integration and implementation in formal and informal learning environments (pp. 39–54). New York: Routledge.Google Scholar
  25. Lindgren, R., & Johnson-Glenberg, M. C. (2013). Emboldened by embodiment: Six precepts for research on embodied learning and mixed reality. Educational Researcher, 42, 445–452. 10.3102/0013189X13511661.CrossRefGoogle Scholar
  26. Lindgren, R., Wallon, R. C., Brown, D. E., Mathayas, N., & Kimball, N. (2016). “Show me” what you mean: Learning and design implications of eliciting gesture in student explanations. In C. Looi, J. Polman, U. Cress, & P. Reimann (Eds.), Proceedings of the Twelfth International Conference of the Learning Sciences (pp. 1014–1017). Singapore: National Institute of Education.Google Scholar
  27. Mathayas, N., Brown, D. E., & Lindgren, R. (2016). Exploring middle school students’ sense making of a computer simulation about thermal conduction. In C. Looi, J. Polman, U. Cress, & P. Reimann (Eds.), Proceedings of the Twelfth International Conference of the Learning Sciences (pp. 1267–1268). Singapore: National Institute of Education.Google Scholar
  28. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar
  29. Nielsen, M., Störring, M., Moeslund, T. B., & Granum, E. (2003). A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In International gesture workshop (pp. 409–420). Springer: Heidelberg.Google Scholar
  30. Paek, S. (2012). The impact of multimodal virtual manipulatives on young children’s mathematics learning (doctoral dissertation). Retrieved from ProQuest dissertations & theses full text (3554708). Ann Arbor, MI.Google Scholar
  31. Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: Beyond human-computer interaction. New York: Wiley.Google Scholar
  32. Radford, L. (2009). Why do gestures matter? Sensuous cognition and the palpability of mathematical meanings. Educational Studies in Mathematics, 70, 111–126. 10.1007/s10649-008-9127-3.CrossRefGoogle Scholar
  33. Resnick, M. (2002). Rethinking learning in the digital age. In G. S. Kirkman, P. K. Cornelius, J. D. Sachs, & K. Schwab (Eds.), The global information technology report 2001–2002: Readiness for the networked world. New York: Oxford University Press.Google Scholar
  34. Roth, W.-M. (2001). Gestures: Their role in teaching and learning. Review of Educational Research, 71, 365–392. 10.3102/00346543071003365.CrossRefGoogle Scholar
  35. Schwartz, R. N. (2010). Considering the activity in interactivity: A multimodal perspective (doctoral dissertation). Retrieved from ProQuest dissertations & theses full text (3404551). Ann Arbor, MI.Google Scholar
  36. Segal, A. (2011). Do gestural interfaces promote thinking? Embodied interaction: Congruent gestures and direct touch promote performance in math (doctoral dissertation). Retrieved from ProQuest dissertations & theses full text (3453956). Ann Arbor, MI.Google Scholar
  37. Shapiro, L. (2010). Embodied cognition. New York: Routledge.Google Scholar
  38. Singer, M., Radinsky, J., & Goldman, S. R. (2008). The role of gesture in meaning construction. Discourse Processes, 45, 365–386. 10.1080/01638530802145601.CrossRefGoogle Scholar
  39. Wallon, R. C., Brown, D. E., & Lindgren, R. (2016). Student gestures during shifts from descriptions to explanations of gas pressure. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Baltimore, MD.Google Scholar
  40. Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9, 625–636. 10.3758/BF03196322.CrossRefGoogle Scholar
  41. Yoon, C., Thomas, M. O., & Dreyfus, T. (2011). Gestures and insight in advanced mathematical thinking. International Journal of Mathematical Education in Science and Technology, 42, 891–901.CrossRefGoogle Scholar

Authors and Affiliations

  1. 1.College of EducationUniversity of Illinois at Urbana-ChampaignChampaignUSA

Personalised recommendations