Affording embodied cognition through touchscreen and above-the-surface gestures during collaborative tabletop science learning

Soni, Nikita; Darrow, Alice; Luc, Annie; Gleaves, Schuyler; Schuman, Carrie; Neff, Hannah; Chang, Peter; Kirkland, Brittani; Alexandre, Jeremy; Morales, Amanda; Stofer, Kathryn A.; Anthony, Lisa

doi:10.1007/s11412-021-09341-x

Published: 13 May 2021

Affording embodied cognition through touchscreen and above-the-surface gestures during collaborative tabletop science learning

Nikita Soni ORCID: orcid.org/0000-0002-8231-0795¹,
Alice Darrow¹,
Annie Luc¹,
Schuyler Gleaves¹,
Carrie Schuman²,
Hannah Neff³,
Peter Chang⁴,
Brittani Kirkland⁵,
Jeremy Alexandre⁶,
Amanda Morales¹,
Kathryn A. Stofer⁵ &
Lisa Anthony¹

International Journal of Computer-Supported Collaborative Learning volume 16, pages 105–144 (2021)Cite this article

863 Accesses
1 Citations
1 Altmetric
Metrics details

Abstract

This paper draws upon the theory of embodied cognition to provide a robust account of how gestural interactions with and around multi-touch tabletops can play an important role in facilitating collaborative meaning-making, particularly in the context of science data visualizations. Embodied cognition is a theory of learning that implies that thinking and perception are shaped by interactions with the physical environment. Previous research has used embodied cognition as a theoretical framework to inform the design of large touchscreen learning applications such as for multi-touch tabletops. However, this prior work has primarily assumed that learning is occurring during any motion or interaction, without considering how specific interactions may be linked to particular instances of collaborative learning supported by embodiment. We investigate this question in the context of collaborative learning from data visualizations of global phenomena such as ocean temperatures. We followed a user-centered, iterative design approach to build a tabletop prototype that facilitated collaborative meaning-making and used this prototype as a testbed in a laboratory study with 11 family groups. We qualitatively analyzed learner groups’ co-occurring utterances and gestures to identify the nature of gestural interactions groups used when their utterances signaled the occurrence of embodiment during collaborative meaning-making. Our findings present an analysis of both touchscreen and above-the-surface gestural interactions that were associated with instances of embodied cognition. We identified four types of gestural interactions that promote scientific discussion and collaborative meaning-making through embodied cognition: (T1) gestures for orienting the group; (T2) cooperative gestures for facilitating group meaning-making; (T3) individual intentional gestures for facilitating group meaning-making; and (T4) gestures for articulating conceptual understanding to the group. Our work illustrates interaction design opportunities for affording embodied cognition and will inform the design of future interactive tabletop experiences in the domain of science learning.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Notes

References

Abrahamson, D., Andrade, A., Bakker, A., Nathan, M. J., Walkington, C., Lindgren, R., … Lindwall, O. (2018). Moving forward: In search of synergy across diverse views on the role of physical movement in design for STEM education. Proceedings of international conference of the learning sciences, 2, 1243–1250.
Abrahamson, D., & Lindgren, R. (2014). Embodiment and embodied design. In In R. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences. https://doi.org/10.1017/CBO9781139519526.022.
Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: Evidence from learners’ and teachers’ gestures. Journal of the Learning Sciences, 21(2), 247–286. https://doi.org/10.1080/10508406.2011.611446.
Article Google Scholar
Anthony, L., Stofer, K. A., Luc, A., & Wobbrock, J. O. (2016). Gestures by children and adults on touch tables and touch walls in a public science center. Proceedings of the Conference on Interaction Design and Children, 344–355. https://doi.org/10.1145/2930674.2930682.
Anthony, L., Yang, J., & Koedinger, K. R. (2012). A paradigm for handwriting-based intelligent tutors. International Journal of Human-Computer Studies, 70(11), 866–887. https://doi.org/10.1016/j.ijhcs.2012.04.003.
Article Google Scholar
Baccaglini-Frank, A., & Maracci, M. (2015). Multi-touch technology and preschoolers’ development of number sense. Digital Experiences in Mathematics Education, 1, 7–27. https://doi.org/10.1007/s40751-015-0002-4.
Article Google Scholar
Becvar, L. A., Hollan, J., & Hutchins, E. (2005). Hands as molecules: Representational gestures used for developing theory in a scientific laboratory. Semiotica, 2005(156), 89–112. https://doi.org/10.1515/semi.2005.2005.156.89.
Article Google Scholar
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa.
Article Google Scholar
Cheek, K. A. (2010). Commentary: A summary and analysis of twenty-seven years of geoscience conceptions research. Journal of Geoscience Education, 58(3), 122–134. https://doi.org/10.5408/1.3544294.
Article Google Scholar
Danish, J. A., Enyedy, N., Saleh, A., & Humburg, M. (2020). Learning in embodied activity framework: A sociocultural framework for embodied cognition. International Journal of Computer-Supported Collaborative Learning, 15, 49–87. https://doi.org/10.1007/s11412-020-09317-3.
Article Google Scholar
Dasu, K., Ma, K.-L., Ma, J., & Frazier, J. (2019). Sea of Genes: Combining Animation and Narrative Strategies to Visualize Metagenomic Data for Museums. Retrieved June 10, 2020, from https://arxiv.org/abs/1906.01071
Davidsen, J., & Ryberg, T. (2017). “This is the size of one meter”: Children’s bodily-material collaboration. International Journal of Computer-Supported Collaborative Learning, 12(1), 65–90. https://doi.org/10.22318/cscl2015.150.
Article Google Scholar
Devine-Wright, P. (2013). Think global, act local? The relevance of place attachments and place identities in a climate changed world. Global Environmental Change, 23(1), 61–69. https://doi.org/10.1016/j.gloenvcha.2012.08.003.
Article Google Scholar
Dillenbourg, P. (1999). What do you mean by collaborative learning? In P. Dillenbourg (Ed.), Collaborative-learning: Cognitive and computational approaches (pp. 1–19). Oxford: Elsevier.
Google Scholar
Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. In Technology-Enhanced Learning (pp. 3–19). https://doi.org/10.1007/978-1-4020-9827-7_1.
Dubé, A. K., & McEwen, R. N. (2015). Do gestures matter? The implications of using touchscreen devices in mathematics instruction. Learning and Instruction, 40, 89–98. https://doi.org/10.1016/j.learninstruc.2015.09.002.
Article Google Scholar
Epps, J., Oviatt, S., & Chen, F. (2004). Integration of speech and gesture inputs during multimodal interaction. Proceedings of the Australian Conference on Human-Computer Interaction.
Fleck, R., Rogers, Y., Yuill, N., Marshall, P., Carr, A., Rick, J., & Bonnett, V. (2009). Actions speak loudly with words: Unpacking collaboration around the table. Proceedings of the Conference on Interactive Tabletops and Surfaces, 189–196. https://doi.org/10.1145/1731903.1731939.
Geller, T. (2006). Interactive tabletop exhibits in museums and galleries. IEEE Computer Graphics and Applications, 26(5), 6–11. https://doi.org/10.1109/MCG.2006.111.
Article Google Scholar
Goldin-Meadow, S., & Beilock, S. L. (2010). Action’s influence on thought: The case of gesture. Perspectives on Psychological Science, 5(6), 664–674. https://doi.org/10.1177/1745691610388764.
Article Google Scholar
Goldinger, S. D., Papesh, M. H., Barnhart, A. S., Hansen, W. A., & Hout, M. C. (2016). The poverty of embodied cognition. Psychonomic Bulletin and Review, 23(4), 959–978. https://doi.org/10.3758/s13423-015-0860-1.
Article Google Scholar
Goldman, K. H., & Gonzalez, J. (2014). Open exhibits multitouch table use findings. Ideum Open Exhibits Papers. Retrieved from http://openexhibits.org/exhibit-design/multitouch-table-research-findings/8801/
Goldman, K. H., Kessler, C., & Danter, E. (2010). Science On a Sphere: Cross-site Summative Evaluation. In Institute for Learning Innovation. Retrieved from http://www.oesd.noaa.gov/network/SOS_evals/SOS_Final_Summative_Report.pdf
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. https://doi.org/10.1016/S0378-2166(99)00096-X.
Article Google Scholar
Greenberg, S., Carpendale, S., Marquardt, N., & Buxton, B. (2011). Sketching user experiences: The workbook. Elsevier.
Gutwill, J. P., & Allen, S. (2010). Facilitating family group inquiry at science museum exhibits. Science Education, 94(4), 710–742. https://doi.org/10.1002/sce.20387.
Article Google Scholar
Hoadley, C. P. (2002). Creating context: Design-based research in creating and understanding CSCL. Proceedings of the Conference on Computer Support for Collaborative Learning, 453–462. https://doi.org/10.3115/1658616.1658679.
Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition: Toward a new Foundation for human-computer interaction research. Proceedings of Transactions on Computer-Human Interaction (TOCHI’00), 7(2), 174–196. https://doi.org/10.1145/353485.353487.
Article Google Scholar
Horn, M., Leong, Z. A., Block, F., Diamond, J., Evans, E. M., Phillips, B., & Shen, C. (2012). Of BATs and APEs: An Interactive Tabletop Game for Natural History Museums. Proceedings of the Conference on Human Factors in Computing Systems, 2059–2068. https://doi.org/10.1145/2207676.2208355.
Hornecker, E. (2005). A design theme for tangible interaction: Embodied facilitation. Proceedings of the European Conference on Computer Supported Cooperative Work, 23–43. https://doi.org/10.5555/1242029.1242031.
Hornecker, E. (2008). “I Don’t understand it either, but it is cool” - visitor interactions with a multi-touch table in a museum. IEEE Workshop on Horizontal Interactive Human Computer Systems, 113–120. https://doi.org/10.1109/TABLETOP.2008.4660193.
Hullman, J., Adar, E., & Shah, P. (2011). Benefitting InfoVis with visual difficulties. IEEE Transactions on Visualization and Computer Graphics, 17(12), 2213–2222. https://doi.org/10.1109/TVCG.2011.175.
Article Google Scholar
Kharrufa, A. S., & Olivier, P. (2010). Exploring the requirements of tabletop interfaces for education. International Journal of Learning Technology, 5(1), 42–62.
Article Google Scholar
Kim, K. T., & Elmqvist, N. (2012). Embodied lenses for collaborative visual queries on tabletop displays. Information Visualization, 11(4), 319–338. https://doi.org/10.1177/1473871612441874.
Article Google Scholar
Kirschner, P. A., Sweller, J., Kirschner, F., Zambrano, R., & J. (2018). From cognitive load theory to collaborative cognitive load theory. International Journal of Computer-Supported Collaborative Learning, 13(2), 213–233. https://doi.org/10.1007/s11412-018-9277-y.
Article Google Scholar
Kirsh, D. (2013). Embodied cognition and the magical future of interaction design. ACM Transactions on Computer-Human Interaction, 20(1), 30 pages. https://doi.org/10.1145/2442106.2442109, 1, 30.
Article Google Scholar
Kiverstein, J. D., & Rietveld, E. (2018). Reconceiving representation-hungry cognition: An ecological-enactive proposal. Adaptive Behavior, 26(4), 147–163. https://doi.org/10.1177/1059712318772778.
Article Google Scholar
Köpsel, A., & Bubalo, N. (2015). Benefiting from legacy Bias. Interactions, 22(5), 44–47. https://doi.org/10.1145/2803169.
Article Google Scholar
Lakoff, G., & Johnson, M. (2003). Metaphors we live by. Chicago: University of Chicago Press.
Book Google Scholar
Lin, M., Preston, A., Kharrufa, A., & Kong, Z. (2016). Making L2 learners’ reasoning skills visible: The potential of computer supported collaborative learning environments. Thinking Skills and Creativity, 22, 303–322. https://doi.org/10.1016/j.tsc.2016.06.004.
Article Google Scholar
Lobben, A., & Lawrence, M. (2015). Synthesized model of geospatial thinking. The Professional Geographer, 67(3), 307–318. https://doi.org/10.1080/00330124.2014.935155.
Article Google Scholar
Ma, J. (2008). Visitors’ Interpretations of Images of the Nanoscale. In Nanoscale Informal Science Education Network. Retrieved from www.nisenet.org/sites/default/catalog/eval/uploads/2009/04/1353/visitor_interpretations.pdf.
Ma, J., Liao, I., Ma, K. L., & Frazier, J. (2012). Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors. IEEE Transactions on Visualization and Computer Graphics, 18(12), 2799–2808. https://doi.org/10.1109/TVCG.2012.244.
Article Google Scholar
McNeill, D. (1992). Hand and mind : What gestures reveal about thought. University of Chicago Press.
Morris, M. R., Huang, A., Paepcke, A., & Winograd, T. (2006). Cooperative gestures: Multi-user gestural interactions for co-located groupware. Proceedings of the Conference on Human Factors in Computing Systems, 1201–1210. https://doi.org/10.1145/1124772.1124952.
Oviatt, S. (1997). Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 12(1–2), 93–129. https://doi.org/10.1080/07370024.1997.9667241.
Article Google Scholar
Piper, A. M., Friedman, W., & Hollan, J. D. (2012). Setting the stage for embodied activity: Scientific discussion around a multi-touch tabletop display. International Journal of Learning Technology, 7(1), 58–78. https://doi.org/10.1504/IJLT.2012.046866.
Article Google Scholar
Piper, A. M., & Hollan, J. D. (2009). Tabletop displays for small group study: Affordances of paper and digital materials. Proceedings of the Conference on Human Factors in Computing Systems, 1227–1236. https://doi.org/10.1145/1518701.1518885.
Price, S., Falcão, T. P., Sheridan, J. G., & Roussos, G. (2009). The effect of representation location on interaction in a tangible learning environment. Proceedings of the Conference on Tangible and Embedded Interaction, 85–92. https://doi.org/10.1145/1517664.1517689.
Rick, J., Marshall, P., & Yuill, N. (2011). Beyond one-size-fits-all: How interactive tabletops support collaborative learning. Proceedings of the Conference on Interaction Design and Children, 109–117. https://doi.org/10.1145/1999030.1999043.
Rogers, Y., & Muller, H. (2006). A framework for designing sensor-based interactions to promote exploration and reflection in play. International Journal of Human Computer Studies, 64(1), 1–14. https://doi.org/10.1016/j.ijhcs.2005.05.004.
Article Google Scholar
Rust, K., Malu, M., Anthony, L., & Findlater, L. (2014). Understanding child-defined gestures and Children’s mental models for touchscreen tabletop interaction. Proceedings of the Conference on Interaction Design and Children, 201–204. https://doi.org/10.1145/2593968.2610452.
Schuman, C., Stofer, K. A., Anthony, L., Neff, H., Chang, P., Soni, N., et al. (2020). Ocean data visualization on a Touchtable demonstrates group content learning, science practices use, and potential embodied cognition. Research in Science Education, 1–13. https://doi.org/10.1007/s11165-020-09951-9.
Segal, A. (2011). Do gestural interfaces promote thinking? Embodied Interaction: Congruent Gestures and Direct Touch Promote Performance in Math. Columbia University.
Sinclair, N., & Pimm, D. (2015). Mathematics using multiple senses: Developing finger gnosis with three- and four-year-olds in an era of multi-touch technologies. Asia-Pacific Journal of Research in Early Childhood Education, 9, 99–110. https://doi.org/10.17206/apjrece.2015.9.3.99.
Article Google Scholar
Soni, N., & Anthony, L. (2019). HCI methodologies for designing natural user interactions that do not interfere with learning. Paper for the “making the learning sciences count: Impacting Association for Computing Machinery Communities in human-computer interaction” workshop, International Conference of Computer-Supported Collaborative Learning, 5 pages.
Soni, N., Darrow, A., Luc, A., Gleaves, S., Schuman, C., Neff, H., … Anthony, L. (2019b). Analysis of touchscreen interactive gestures during embodied cognition in collaborative tabletop science learning experiences. Proceedings of the Conference of Computer Supported Collaborative Learning, 9–16.
Soni, N., Gleaves, S., Neff, H., Morrison-Smith, S., Esmaeili, S., Mayne, I., et al. (2019c). Do user-defined gestures for Flatscreens generalize to interactive spherical displays for adults and children? Proceedings of the ACM International Symposium on Pervasive Displays, 7. https://doi.org/10.1145/3321335.3324941.
Soni, N., Gleaves, S., Neff, H., Morrison-Smith, S., Esmaeili, S., Mayne, I., et al. (2020). Adults’ and Children’s mental models for gestural interactions with interactive spherical displays. Proceedings of the Conference on Human Factors in Computing Systems, 1–12. https://doi.org/10.1145/3313831.3376468.
Stahl, G. (2018). A Theory of Group Cognition in CSCL. Retrieved from http://gerrystahl.net/pub/theory.pdf
Steier, R., Kersting, M., & Silseth, K. (2019). Imagining with improvised representations in CSCL environments. International Journal of Computer-Supported Collaborative Learning, 14, 109–136. https://doi.org/10.1007/s11412-019-09295-1.
Article Google Scholar
Stofer, K. A. (2016). When a picture Isn’t worth 1000 words: Learners struggle to find meaning in data visualizations. Journal of Geoscience Education, 64(3), 231–241. https://doi.org/10.5408/14-053.1.
Article Google Scholar
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.
Article Google Scholar
Tang, J. C. (1991). Findings from observational studies of collaborative work. International Journal of Man-Machine Studies, 34(2), 143–160. https://doi.org/10.1016/0020-7373(91)90039-A.
Article Google Scholar
Tse, E., Shen, C., Greenberg, S., & Forlines, C. (2006). Enabling interaction with single user applications through speech and gestures on a multi-user tabletop. Proceedings of the Conference on Advanced Visual Interfaces, 336–343. https://doi.org/10.1145/1133265.1133336.
Vygotsky, L. (1986). Thought and language. Cambridge Mass: MIT Press.
Google Scholar
Vygotsky, L. S. (1978). Mind in society: Development of higher psychological processes. Harvard University Press.
Wee, L. (2005). Class-inclusion and correspondence models as discourse types: A framework for approaching metaphorical discourse. Language in Society, 34(2), 219–238. https://doi.org/10.1017/S0047404505050098.
Article Google Scholar
Wilson, A. D., & Golonka, S. (2013). Embodied Cognition is Not What You Think It Is. Frontiers in Psychology, 4, 58 pages. https://doi.org/10.3389/fpsyg.2013.00058.
Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin and Review, 9, 625–636. https://doi.org/10.3758/BF03196322.
Article Google Scholar
Xambó, A., Jewitt, C., & Price, S. (2014). Towards an integrated methodological framework for understanding embodiment in HCI. Proceedings of the Conference on Human Factors in Computing Systems, 1411–1416. https://doi.org/10.1145/2559206.2581276.
Zheng, L., Cui, P., & Zhang, X. (2020). Does collaborative learning design align with enactment? An innovative method of evaluating the alignment in the CSCL context. International Journal of Computer-Supported Collaborative Learning, 15, 193–226. https://doi.org/10.1007/s11412-020-09320-8.
Article Google Scholar

Download references

Acknowledgments

This work is partially supported by National Science Foundation Grant Awards #DRL-1612485 and #IIS-1552598. Any opinions, findings, and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect these agencies’ views. The authors also thank the Florida Museum of Natural History for allowing us to recruit participants from their visitors; our advisory board members, Betty Dunckel and Julia Plummer, for contributing to our thinking related to embodied cognition; and all our pilot participants. A preliminary version of the analysis in this paper that included only T1 and T2 was presented at CSCL 2019 (Soni et al. 2019b). This work was conducted while author Jeremy Alexandre was a summer intern at the University of Florida.

Availability of data and material

The authors plan to make the anonymized study data publicly available on the open science framework data repository in the future: https://osf.io/rjzkh/

Code availability

The authors plan to make the codebase for the multi-touch tabletop application used in the study publicly available on the open science framework data repository in the future: https://osf.io/rjzkh/

Funding

This work is partially supported by National Science Foundation Grant Awards #DRL-1612485 and #IIS-1552598.

Author information

Authors and Affiliations

Department of Computer and Information Science and Engineering, University of Florida, Gainesville, FL, USA
Nikita Soni, Alice Darrow, Annie Luc, Schuyler Gleaves, Amanda Morales & Lisa Anthony
School of Natural Resources and Environment, University of Florida, Gainesville, FL, USA
Carrie Schuman
Department of Sociology and Criminology and Law, University of Florida, Gainesville, FL, USA
Hannah Neff
Department of Biology, University of Florida, Gainesville, FL, USA
Peter Chang
Department of Agricultural Education & Communication, University of Florida, Gainesville, FL, USA
Brittani Kirkland & Kathryn A. Stofer
City University of New York, Brooklyn College, Brooklyn, NY, USA
Jeremy Alexandre

Authors

Nikita Soni
View author publications
You can also search for this author in PubMed Google Scholar
Alice Darrow
View author publications
You can also search for this author in PubMed Google Scholar
Annie Luc
View author publications
You can also search for this author in PubMed Google Scholar
Schuyler Gleaves
View author publications
You can also search for this author in PubMed Google Scholar
Carrie Schuman
View author publications
You can also search for this author in PubMed Google Scholar
Hannah Neff
View author publications
You can also search for this author in PubMed Google Scholar
Peter Chang
View author publications
You can also search for this author in PubMed Google Scholar
Brittani Kirkland
View author publications
You can also search for this author in PubMed Google Scholar
Jeremy Alexandre
View author publications
You can also search for this author in PubMed Google Scholar
Amanda Morales
View author publications
You can also search for this author in PubMed Google Scholar
Kathryn A. Stofer
View author publications
You can also search for this author in PubMed Google Scholar
Lisa Anthony
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikita Soni.

Ethics declarations

Conflicts of interest/competing interests

Not Applicable.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Soni, N., Darrow, A., Luc, A. et al. Affording embodied cognition through touchscreen and above-the-surface gestures during collaborative tabletop science learning. Intern. J. Comput.-Support. Collab. Learn 16, 105–144 (2021). https://doi.org/10.1007/s11412-021-09341-x

Download citation

Received: 01 July 2020
Accepted: 22 February 2021
Published: 13 May 2021
Issue Date: March 2021
DOI: https://doi.org/10.1007/s11412-021-09341-x

Keywords

User-centered design
Touchscreen interaction
Collaborative learning
Embodied cognition
Science museum exhibits
Multi-touch tabletops
Science data visualization