Abstract
Multimodal Learning Analytics (MMLA) grant us insight into learners’ physiological, cognitive, and behavioral activity as it unfolds. In this chapter, we query the relations among modalities, intermodality, in the context of a design-based research program studying the relations between learning to move in new ways and learning to think in new ways. In the first part, we reflect on how different methods have afforded purchase on the investigation, development, and elaboration of theoretical claims about the multimodal enactment of cognitive events, culminating in the use of Recurrence Quantification Analysis (RQA) to quantify the microgenesis of stable new patterns in hand movement and gaze. In the second part, we analyze an RQA case study spanning across hand and gaze modalities to examine the emergence of intermodal coordination at a critical moment in the mathematical task. We conclude with implications and open questions around intermodality in embodied learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We use the term intermodality to signify the dynamic process of coordinating perceptual information across modalities. This usage is distinct from intermodality as multisensory integration (e.g., Ernst, 2008).
- 2.
cRQA quantifies dynamical aspects of the coordination of two time series (here, the left- and right-hand position time series). aRQA quantifies recurrent patterns within a single time series. The gaze data are categorically coded according to areas of interest relative to the hands such as the top of the left bar or the middle of the right bar (see Abdu et al., under review). Thus, cRQA analysis compares the continuous hand position time series, whereas the categorical aRQA compares the categorical gaze time series to itself.
- 3.
The brief drop in hand coupling at the end of the Discovery stage is likely a by-product of the participant attempting to generalize the ratio to an inverted position where the left hand was above the right.
- 4.
Although not visual or kinesthetic, it is worth noting that the auditory modality also played a role here as the tutor offered prompts and encouragements that solicited changes in dynamics, most notably by encouraging the participant to try moving-in-green.
References
Abdu, R., Tancredi, S., Abrahamson, D., & Balasubramaniam, R. (under review). A complex systems outlook on hand-eye coordination in mathematical learning. In M. Schindler, A. Shvarts, & A. Lilienthal (Eds.), Eye-tracking research in mathematics education [Special issue]. Educational Studies in Mathematics.
Abdullah, A., Adil, M., Rosenbaum, L., Clemmons, M., Shah, M., Abrahamson, D., & Neff, M. (2017). Pedagogical agents to support embodied, discovery-based learning. In J. Beskow, C. Peters, G. Castellano, C. O'Sullivan, I. Leite, & S. Kopp (Eds.), Proceedings of 17 th international conference on intelligent virtual agents (IVA 2017) (pp. 1–14). Springer International Publishing. https://doi.org/10.1007/978-3-319-67401-8_1
Abrahamson, D. (2009). Embodied design: Constructing means for constructing meaning. Educational Studies in Mathematics, 70(1), 27–47. https://doi.org/10.1007/s10649-008-9137-1
Abrahamson, D. (2014). Building educational activities for understanding: An elaboration on the embodied-design framework and its epistemic grounds. International Journal of Child-Computer Interaction, 2(1), 1–16. https://doi.org/10.1016/j.ijcci.2014.07.002
Abrahamson, D. (2015). The monster in the machine, or why educational technology needs embodied design. In V. R. Lee (Ed.), Learning technologies and the body: integration and implementation implementation in formal and informal learning environments (pp. 21–38). Routledge.
Abrahamson, D., & Abdu, R. (2020). Towards an ecological-dynamics design framework for embodied-interaction conceptual learning: The case of dynamic mathematics environments. In T. J. Kopcha, K. D. Valentine, & C. Ocak (Eds.), Embodied cognition and technology for learning [special issue]. Educational Technology Research and Development. https://doi.org/10.1007/s11423-020-09805-1
Abrahamson, D., & Howison, M. (2008). Kinemathics: Kinetically induced mathematical learning UC Berkeley Gesture Study Group (E. Sweetser, Organizer), December 5, 2008. Retrieved June 20, 2019 from https://edrl.berkeley.edu/wp-content/uploads/2019/06/Abrahamson-Howison-2008_kinemathics.pdf
Abrahamson, D., & Howison, M. (2010). Embodied artifacts: Coordinated action as an object-to-think-with. In D. L. Holton (Organizer & Chair) & J. P. Gee (Discussant), Embodied and enactive approaches to instruction: implications and innovations. Paper presented at the annual meeting of the American Educational Research Association, April 30–May 4, Denver, CO.
Abrahamson, D., & Sánchez-García, R. (2016). Learning is moving in new ways: The ecological dynamics of mathematics education. Journal of the Learning Sciences, 25(2), 203–239. https://doi.org/10.1080/10508406.2016.1143370
Abrahamson, D., & Trninic, D. (2011). Toward an embodied-interaction design framework for mathematical concepts. In P. Blikstein & P. Marshall (Eds.), Proceedings of the 10th annual interaction design and children conference (IDC 2011) (Vol. Full papers, pp. 1–10). IDC.
Abrahamson, D., Trninic, D., Gutiérrez, J. F., Huth, J., & Lee, R. G. (2011). Hooks and shifts: A dialectical study of mediated discovery. Technology, Knowledge, and Learning, 16(1), 55–85.
Abrahamson, D., Black, J. B., DeLiema, D., Enyedy, N., Hoyer, D., Fadjo, C. L., Gutiérrez, J. F., Martin, H. T., Petrick, C. J., Steen, F. F., & Trninic, D. (2012a). You’re it! Body, action, and object in STEM learning. In D. Abrahamson (Chair & Organizer) and M. Eisenberg (Discussant). In J. v. Aalst, K. Thompson, M. J. Jacobson, & P. Reimann (Eds.), Proceedings of the International Conference of the Learning Sciences: Future of Learning (ICLS 2012) (Vol. 1: Full papers, pp. 283–290). University of Sydney / ISLS.
Abrahamson, D., Gutiérrez, J. F., Charoenying, T., Negrete, A. G., & Bumbacher, E. (2012b). Fostering hooks and shifts: Tutorial tactics for guided mathematical discovery. Technology, Knowledge, and Learning, 17(1–2), 61–86. https://doi.org/10.1007/s10758-012-9192-7
Abrahamson, D., Lee, R. G., Negrete, A. G., & Gutiérrez, J. F. (2014). Coordinating visualizations of polysemous action: Values added for grounding proportion. In F. Rivera, H. Steinbring, & A. Arcavi (Eds.), visualization as an epistemological learning tool [special Isuue]. ZDM Mathematics Education, 46(1), 79–93. https://doi.org/10.1007/s11858-013-0521-7
Abrahamson, D., Shayan, S., Bakker, A., & Van der Schaaf, M. F. (2016). Eye-tracking Piaget: Capturing the emergence of attentional anchors in the coordination of proportional motor action. Human Development, 58(4–5), 218–244.
Abrahamson, D., Flood, V. J., Miele, J. A., & Siu, Y.-T. (2019). Enactivism and ethnomethodological conversation analysis as tools for expanding universal Design for Learning: The case of visually impaired mathematics students. ZDM Mathematics Education, 51(2), 291–303. https://doi.org/10.1007/s11858-018-0998-1
Adolph, K. E. (2019). An ecological approach to learning in (not and) development. Human Development, 63, 180–201. https://doi.org/10.1159/000503823
Alberto, R., Shvarts, A., Drijvers, P., & Bakker, A. (2021). Action-based embodied design for mathematics learning: A decade of variations on a theme. International Journal of Child-Computer Interaction, 100419. https://doi.org/10.1016/j.ijcci.2021.100419
Allen, L.K., Perret, C., Likens, A., McNamara, D.S. (2017). What’d you say again?: Recurrence quantification analysis as a method for analyzing the dynamics of discourse in a reading strategy tutor. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference (LAK ‘17). (pp. 373–382). ACM. https://doi.org/10.1145/3027385.3027445
Amon, M. J., Vrzakova, H., & D’Mello, S. K. (2019). Beyond dyadic coordination: Multimodal behavioral irregularity in triads predicts facets of collaborative problem solving. Cognitive Science, 43(10). https://doi.org/10.1111/cogs.12787
Ba, H., & Abrahamson, D. (2021). Taking design to task: A dialogue on task-initiation in STEM activities. Educational Designer, 4(14), 1–21. http://www.educationaldesigner.org/ed/volume4/issue14/article54/
Bakker, A., Shvarts, A., & Abrahamson, D. (2019). Generativity in design research: the case of developing a genre of action-based mathematics learning activities. In U. T. Jankvist, M. H. A. M. v. d. Heuvel-Panhuizen, & M. Veldhuis (Eds.), Proceedings of the 11th Congress of the European Society for Research in Mathematics Education (CERME 11) (Vol. TWG17: Theoretical perspectives and approaches in mathematics education research, pp. 3096–3103). Utrecht University.
Bernstein, N. A. (1996). In M. L. Latash & M. T. Turvey (Eds.), Dexterity and its development (pp. 3–235). Lawrence Erlbaum Associates.
Bongers, T. J. D. (2020). Transfer of embodied experiences in a tablet environment towards a pen and paper task (Unpublished Master’s thesis, Utrecht University).
Burton, H. (2003). Visual cortex activity in early and late blind people. Journal of Neuroscience, 23(10), 4005–4011.
Di Paolo, E. A., Chemero, A., Heras-Escribano, M., & McGann, M. (Eds.). (2021). Enaction and ecological psychology: Convergences and complementarities [Research topic]. Frontiers in Psychology. https://doi.org/10.3389/978-2-88966-431-3
Dourish, P. (2001). Where the action is: The foundations of embodied interaction. MIT Press.
Duijzer, A. C. G., Shayan, S., Bakker, A., van der Schaaf, M. F., & Abrahamson, D. (2017). Touchscreen tablets: Coordinating action and perception for mathematical cognition. In J. Tarasuik, G. Strouse, & J. Kaufman (Eds.), Touchscreen tablets touching children's lives [Special issue] [Original Research]. Frontiers in Psychology, 8(144). https://doi.org/10.3389/fpsyg.2017.00144
Ernst, M. O. (2008). Multisensory integration: A late bloomer. Current Biology, 18(12), R519–R521.
Fleuchaus, E., Kloos, H., Kiefer, A. W., & Silva, P. L. (2020). Complexity in science learning: Measuring the underlying dynamics of persistent mistakes. Journal of Experimental Education, 88(3). https://doi.org/10.1080/00220973.2019.1660603
Flood, V. J., Harrer, B. W., & Abrahamson, D. (2016). The interactional work of configuring a mathematical object in a technology-enabled embodied learning environment. In C.-K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), “Transforming learning, empowering learners,” Proceedings of the International Conference of the Learning Sciences (ICLS 2016) (Vol. 1, "Full Papers", pp. 122–129). International Society of the Learning Sciences.
Flood, V. J., Shvarts, A., & Abrahamson, D. (2020). Teaching with embodied learning technologies for mathematics: Responsive teaching for embodied learning. ZDM Mathematics Education, 52(7), 1307–1331. https://doi.org/10.1007/s11858-020-01165-7
Gibson, J. J. (1966). The senses considered as perceptual systems. Houghton Mifflin.
Gibson, E. J. (1969). Principles of perceptual learning and development. .
Hutto, D. D., & Sánchez-García, R. (2015). Choking RECtified: Embodied expertise beyond Dreyfus. Phenomenology and the Cognitive Sciences, 14(2), 309–331. https://doi.org/10.1007/s11097-014-9380-0
Kelton, M. L., & Ma, J. Y. (2018). Reconfiguring mathematical settings and activity through multi-party, whole-body collaboration [journal article]. Educational Studies in Mathematics, 98(2), 177–196. https://doi.org/10.1007/s10649-018-9805-8
Kostrubiec, V., Zanone, P.-G., Fuchs, A., & Kelso, J. A. S. (2012). Beyond the blank slate: Routes to learning new coordination patterns depend on the intrinsic dynamics of the learner—Experimental evidence and theoretical model. Frontiers in Human Neuroscience, 6. https://doi.org/10.3389/fnhum.2012.00222
Marwan, N., Romano, M. C., Thiel, M., & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438, 237–239.
Mechsner, F. (2003). Gestalt factors in human movement coordination. Gestalt Theory, 25(4), 225–245.
Mechsner, F. (2004). A psychological approach to human voluntary movements. Journal of Motor Behavior, 36(4), 355–370.
Negrete, A. G., Lee, R. G., & Abrahamson, D. (2013). Facilitating discovery learning in the tablet era: Rethinking activity sequences Vis-à-Vis digital practices. In M. Martinez & A. Castro Superfine (Eds.), “Broadening perspectives on mathematics thinking and learning”—Proceedings of the 35th Annual Meeting of the North-American Chapter of the International Group for the Psychology of Mathematics Education (PME-NA 35) (Vol. 10: “Technology”, p. 1205). University of Illinois at Chicago.
Newen A., Bruin L.D., & Gallagher S. (Eds.). (2018). The Oxford handbook of 4e cognition. Oxford University Press.
Ou, L., Andrade, A., Alberto, R. A., Bakker, A., & Bechger, T. (2020a). Identifying qualitative between-subject and within-subject variability: A method for clustering regime-switching dynamics. Frontiers in Psychology, 11, 1136. https://doi.org/10.3389/fpsyg.2020.01136
Ou, L., Andrade, A., Alberto, R., van Helden, G., & Bakker, A. (2020b). Using a cluster-based regime-switching dynamic model to understand embodied mathematical learning. In K. Verbert, M. Scheffel, N. Pinkwart, & V. Kovanonic (Eds.), Proceedings of the 10th international conference on Learning Analytics & Knowledge (pp. 496–501). ACM. https://doi.org/10.1145/3375462.3375513
Palatnik, A., & Abrahamson, D. (2018). Rhythmic movement as a tacit enactment goal mobilizing the emergence of mathematical structures. Educational Studies in Mathematics, 99(3), 293–309. https://doi.org/10.1007/s10649-018-9845-0
Pardos, Z. A., Hu, C., Meng, P., Neff, M., & Abrahamson, D. (2018). Classifying learner behavior from high frequency touchscreen data using recurrent neural networks. In UMAP’18 adjunct: 26th conference on user Modeling, adaptation and personalization adjunct (pp. 317–322). ACM. https://doi.org/10.1145/3213586.3225244
PhET Interactive Simulations. (2021, September). Ratio and proportion. phet.colorado.edu/en/simulations/ratio-and-proportion/.
Piaget, J. (1968). Quantification, conservation, and nativism. Science, 162(3857), 976–979. https://doi.org/10.1126/science.162.3857.97
Piaget, J. (1970). Genetic epistemology (E. Duckworth, Trans.). Columbia University Press.
Richardson, M. J., & Chemero, A. (2014). Complex dynamical systems and embodiment. In L. Shapiro (Ed.), The Routledge handbook of embodied cognition (pp. 39–50). Routledge.
Rosen, D. M., Palatnik, A., & Abrahamson, D. (2016). Tradeoffs of situatedness: Iconicity constrains the development of content-oriented sensorimotor schemes. In M. B. Wood, E. E. Turner, M. Civil, & J. A. Eli (Eds.), Sin fronteras: Questioning borders with(in) mathematics education - Proceedings of the 38th annual meeting of the North-American Chapter of the International Group for the Psychology of Mathematics Education (PME-NA) (Vol. 12, “Technology”, pp. 1509–1516). University of Arizona.
Shayan, S., Abrahamson, D., Bakker, A., Duijzer, A. C. G., & Van der Schaaf, M. F. (2015). The emergence of proportional reasoning from embodied interaction with a tablet application: An eye-tracking study. In L. G. Chova, A. L. Martínez, & I. C. Torres (Eds.), Proceedings of the 9 th international technology, education, and development conference (INTED 2015) (pp. 5732–5741) International Academy of Technology, Education, and Development.
Shvarts, A., & Abrahamson, D. (2019). Dual-eye-tracking Vygotsky: A microgenetic account of a teaching/learning collaboration in an embodied-interaction technological tutorial for mathematics. Learning, Culture and Social Interaction, 22, 100316. https://doi.org/10.1016/j.lcsi.2019.05.003
Shvarts, A., Alberto, R., Bakker, A., Doorman, M., & Drijvers, P. (2021). Embodied instrumentation in learning mathematics as the genesis of a body-artifact functional system. Educational Studies in Mathematics, 107(3), 447–469. https://doi.org/10.1007/s10649-021-10053-0
Smith, C., King, B., & Gonzalez, D. (2016). Using multimodal learning analytics to identify patterns of interactions in a body-based mathematics activity. Journal of Interactive Learning Research, 27(4), 355–379.
Steffe, L. P., & Kieren, T. (1994). Radical constructivism and mathematics education. Journal for Research in Mathematics Education, 25(6), 711–733.
Stephen, D. G., Dixon, J. A., & Isenhower, R. W. (2009). Dynamics of representational change: Entropy, action, and cognition. Journal of Experimental Psychology: Human Perception and Performance, 35(6), 1811–1832.
Stoffregen, T. A., & Bardy, B. G. (2001). On specification and the senses. The Behavioral and Brain Sciences, 24(2), 195–261. https://doi.org/10.1017/s0140525x01003946
Stoffregen, T. A., Mantel, B., & Bardy, B. G. (2017). The senses considered as one perceptual system. Ecological Psychology, 29(3), 165–197. https://doi.org/10.1080/10407413.2017.1331116
Tancredi, S., Abdu, R., Abrahamson, D., & Balasubramaniam, R. (2021). Modeling nonlinear dynamics of fluency development in an embodied-design mathematics learning environment with recurrence quantification analysis. International Journal of Child-Computer Interaction, 100297. https://doi.org/10.1016/j.ijcci.2021.100297
Tancredi, S., Chen, R. S. Y., Krause, C. M., & Siu, Y.-T. (2022). The need for SpEED: Reimagining accessibility through Special Education Embodied Design. In S. L. Macrine & J. M. B. Fugate (Eds.), Movement matters: How embodied cognition informs teaching and learning (pp. 197–216). M.I.T. Press.
Thelen, E., & Smith, L. B. (1994). A dynamic systems approach to the development of cognition and action. MIT Press.
Varela, F. J., Thompson, E., & Rosch, E. (1991). The embodied mind: Cognitive science and human experience. MIT Press.
von Glasersfeld, E. (1987). Learning as a constructive activity. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics (pp. 3–18). Lawrence Erlbaum.
Vygotsky, L. S. (1926/1997). Educational psychology (R. H. Silverman, Trans.). CRC Press LLC. (Original work published 1926).
Webber, C. L., & Zbilut, J. P. (1994). Dynamical assessment of physiological systems and states using recurrence plot strategies. Journal of Applied Physiology, 76(2), 965–973. 10(1), 3058. https://doi.org/10.1038/s41598-020-60066-7
Yamamoto, K., Shinya, M., & Kudo, K. (2020). The influence of attractor stability of intrinsic coordination patterns on the adaptation to new constraints. Scientific Reports, 3058. https://doi.org/10.1038/s41598-020-60066-7
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Tancredi, S., Abdu, R., Balasubramaniam, R., Abrahamson, D. (2022). Intermodality in Multimodal Learning Analytics for Cognitive Theory Development: A Case from Embodied Design for Mathematics Learning. In: Giannakos, M., Spikol, D., Di Mitri, D., Sharma, K., Ochoa, X., Hammad, R. (eds) The Multimodal Learning Analytics Handbook. Springer, Cham. https://doi.org/10.1007/978-3-031-08076-0_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-08076-0_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08075-3
Online ISBN: 978-3-031-08076-0
eBook Packages: Computer ScienceComputer Science (R0)