Skip to main content
SpringerLink
Log in

Everyone Matters: A Multimodal Learning Analysis Framework Based on Individual Time Series

  • Original research
  • Published:
Technology, Knowledge and Learning Aims and scope Submit manuscript

Abstract

Multimodal learning analysis emphasizes using diverse data from various sources and forms for precise examination of learning patterns. Despite recent rapid advancements in this field, conventional learning analysis remains predominantly cross-sectional and group-focused, which is insufficient for understanding continuous and personalized learning processes, especially in multimodal situations. Moreover, multimodal learning analysis poses a significant technological challenge, with research indicating growing difficulties for educators. Thus, we introduce an innovative framework utilizing individual learners’ temporal sequences for multimodal learning analysis, offering multifaceted evidence for understanding individual learning journeys. We elucidate the analysis process and extend the framework to pre-service teachers (Case 1) and in-service educators (Case 2), promoting their involvement in advancing multimodal learning analysis. Case findings underscore the framework’s feasibility and comprehensibility, suggesting its potential to drive personalized learning analysis in future practices and research endeavors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

References

  • Agus, R., & Mohamad Samuri, S. (2018). Learning analytics contribution in education and child development: A review on learning analytics. Asian Journal of Assessment in Teaching and Learning, 8, 36–47. https://doi.org/10.37134/AJATEL.VOL8.4.2018.

    Article  Google Scholar 

  • Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., Williams, D., Dawson, S., & Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, & C. Penstein Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ‘16 (pp. 329–338). ACM Press. https://doi.org/10.1145/2883851.2883944.

  • Cao, Z., Hidalgo, G., Simon, T., Wei, S. E., & Sheikh, Y. (2021). Openpose: Realtime Multi-person 2D pose estimation using Part Affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1), 172–186. https://doi.org/10.1109/TPAMI.2019.2929257.

    Article  Google Scholar 

  • Chejara, P., Prieto, L. P., Ruiz-Calleja, A., Rodríguez-Triana, M. J., & Shankar, S. K. (2019). Exploring the Triangulation of Dimensionality Reduction when interpreting Multimodal Learning Data from authentic settings. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Lecture notes in Computer Science. Transforming learning with Meaningful technologies (Vol. 11722, pp. 664–667). Springer International Publishing. https://doi.org/10.1007/978-3-030-29736-7_62.

  • Chen, L., Li, X., Xia, Z., Song, Z., Morency, L. P., & Dubrawski, A. (2016). Riding an Emotional Roller-Coaster: Riding an Emotional Roller-Coaster: A Multimodal Study of Young Child’s Math Problem Solving Activities. In 9th International Conference on Educational Data Mining, North Carolina, USA.

  • Chen, L. K., Ramsey, J., & Artur, D. (2021). Affect, support, and personal factors: Multimodal Causal models of one-on-one coaching. Journal of Educational Data Mining, 13(3), 36–68.

    Google Scholar 

  • Clow, D. (2012). The learning analytics cycle. In S. Dawson, C. Haythornthwaite, S. Buckingham Shum, D. Gasevic, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 134–138). ACM. https://doi.org/10.1145/2330601.2330636.

  • Crescenzi-Lanna, L. (2020). Multimodal learning analytics research with young children: A systematic review. British Journal of Educational Technology, 51(5), 1485–1504. https://doi.org/10.1111/bjet.12959.

    Article  Google Scholar 

  • Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches. Sage.

  • Deeva, G., de Smedt, J., & de Weerdt, J. (2022). Educational sequence mining for Dropout Prediction in MOOCs: Model Building, evaluation, and Benchmarking. IEEE Transactions on Learning Technologies, 15(6), 720–735. https://doi.org/10.1109/TLT.2022.3215598.

    Article  Google Scholar 

  • Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 338–349. https://doi.org/10.1111/jcal.12288.

    Article  Google Scholar 

  • Drachsler, H., & Schneider, J. (2018). JCAL special issue on multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 335–337. https://doi.org/10.1111/jcal.12291.

    Article  Google Scholar 

  • Ferguson, R. (2012). Learning analytics: Drivers, developments, and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304. https://doi.org/10.1504/IJTEL.2012.051816.

    Article  Google Scholar 

  • Fournier, H., Kop, R., & Sitlia, H. (2011). The value of learning analytics to networked learning on a personal learning environment. In P. Long, G. Siemens, G. Conole, & D. Gašević (Eds.), Proceedings of the 1st International Conference on Learning Analytics and Knowledge (pp. 104–109). ACM. https://doi.org/10.1145/2090116.2090131.

  • Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://doi.org/10.1080/23735082.2017.1286142.

    Article  Google Scholar 

  • Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology & Society, 15(3), 42–57.

    Google Scholar 

  • Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.

    Article  Google Scholar 

  • IIya Prigogine. (2007). From being to becoming. Ke Xue Su Yang Wen Ku Ke Xue yuan dian cong shu. Bei jing ta xue chu ban she.

  • Jirsa, V., & Sheheitli, H. (2022). Entropy, free energy, symmetry and dynamics in the brain. Journal of Physics: Complexity, 3(1), 15007. https://doi.org/10.1088/2632-072X/ac4bec.

    Article  Google Scholar 

  • Jovic, A., Brkic, K., & Bogunovic, N. (2014). An overview of free software tools for general data mining. In 2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) (pp. 1112–1117). IEEE. https://doi.org/10.1109/MIPRO.2014.6859735.

  • Koh, E., Shibani, A., Tan, J. P. L., & Hong, H. (2016). A pedagogical framework for learning analytics in collaborative inquiry tasks. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, & C. Penstein Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ‘16 (pp. 74–83). ACM Press. https://doi.org/10.1145/2883851.2883914.

  • Kulkarni, R. V., Revathy, S., & Patil, S. H. (2022). Smart pools of data with ensembles for adaptive learning in dynamic data streams with class imbalance. IAES International Journal of Artificial Intelligence (IJ-AI), 11(1), 310. https://doi.org/10.11591/ijai.v11.i1.pp310-318.

    Article  Google Scholar 

  • Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., de Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. https://doi.org/10.1371/journal.pdig.0000198.

    Article  Google Scholar 

  • Lim, B., Arik, S. O., Loeff, N., & Pfister, T. (2019). Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastinghttps://doi.org/10.48550/arXiv.1912.09363.

  • Liu Qin. (2021). Lectures on modern western thought (Di 1 ban). Xin xing chu ban she.

  • Mangaroska, K., & Giannakos, M. (2019). Learning analytics for Learning Design: A systematic literature review of Analytics-Driven Design to Enhance Learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://doi.org/10.1109/TLT.2018.2868673.

    Article  Google Scholar 

  • Marcelo Worsley (2018). Multimodal learning analytics’ past, present, and, poten- tial futures. In CEUR Workshop, Aachen, Germany.

  • Martinez-Maldonado, R., Schneider, B., Charleer, S., Shum, S. B., Klerkx, J., & Duval, E. (2016). Interactive surfaces and learning analytics. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, & C. Penstein Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ‘16 (pp. 124–133). ACM Press. https://doi.org/10.1145/2883851.2883873.

  • Martinez-Maldonado, R., Kay, J., Buckingham Shum, S., & Yacef, K. (2019). Collocated Collaboration Analytics: Principles and dilemmas for mining Multimodal Interaction Data. Human–Computer Interaction, 34(1), 1–50. https://doi.org/10.1080/07370024.2017.1338956.

    Article  Google Scholar 

  • Melero, J., Hernández-Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location-based learning design. British Journal of Educational Technology, 46(2), 317–329. https://doi.org/10.1111/bjet.12238.

    Article  Google Scholar 

  • Mu, S., Cui, M., & Huang, X. (2020). Multimodal Data Fusion in Learning analytics: A systematic review. Sensors (Basel Switzerland), 20(23). https://doi.org/10.3390/s20236856.

  • NIST (2001). CUMULATIVE AVERAGE. https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/cumumean.htm.

  • Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019). Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior, 100, 298–304. https://doi.org/10.1016/j.chb.2018.12.019.

    Article  Google Scholar 

  • Papamitsiou, Z., & Economides, A. A. (2016). Learning Analytics for Smart Learning Environments: A Meta-Analysis of Empirical Research Results from 2009 to 2015. In M. J. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, Design, and Technology (pp. 1–23). Springer International Publishing. https://doi.org/10.1007/978-3-319-17727-4_15-1.

  • Papavlasopoulou, S., Sharma, K., & Giannakos, M. N. (2018). How do you feel about learning to code? Investigating the effect of children’s attitudes towards coding using eye-tracking. International Journal of Child-Computer Interaction, 17, 50–60. https://doi.org/10.1016/j.ijcci.2018.01.004.

    Article  Google Scholar 

  • Ramakrishnan, A., Ottmar, E., LoCasale-Crouch, J., & Whitehill, J. (2019). Toward Automated Classroom Observation: Predicting Positive and Negative Climate. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1–8). IEEE. https://doi.org/10.1109/FG.2019.8756529.

  • Reimann, P. (2016). Connecting learning analytics with learning research: The role of design-based research. Learning: Research and Practice, 2(2), 130–142. https://doi.org/10.1080/23735082.2016.1210198.

    Article  Google Scholar 

  • Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture(S), (33), 134–154. https://doi.org/10.55612/s-5002-033-007.

  • Romero, C., Ventura, S., Zafra, A., & de Bra, P. (2009). Applying web usage mining for personalizing hyperlinks in web-based adaptive educational systems. Computers & Education, 53(3), 828–840. https://doi.org/10.1016/j.compedu.2009.05.003.

    Article  Google Scholar 

  • Sayes, E. (2014). Actor-network theory and methodology: Just what does it mean to say that nonhumans have agency? Social Studies of Science, 44(1), 134–149. https://doi.org/10.1177/0306312713511867.

    Article  Google Scholar 

  • Scherer, S., Worsley, M., & Morency, L. P. (2012). 1st international workshop on multimodal learning analytics. In L.-P. Morency, D. Bohus, H. Aghajan, J. Cassell, A. Nijholt, & J. Epps (Eds.), Proceedings of the 14th ACM international conference on Multimodal interaction - ICMI ‘12 (p. 609). ACM Press. https://doi.org/10.1145/2388676.2388803.

  • Schmidt, K. L., & Cohn, J. F. (2001). Human facial expressions as adaptations: Evolutionary questions in facial expression research. American Journal of Physical Anthropology, Suppl 33, 3–24. https://doi.org/10.1002/ajpa.20001.

    Article  Google Scholar 

  • Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450–1484. https://doi.org/10.1111/bjet.12993.

    Article  Google Scholar 

  • Siemens, G., & Baker, R. S. J. (2012). Learning analytics and educational data mining. In S. Dawson, C. Haythornthwaite, S. Buckingham Shum, D. Gasevic, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 252–254). ACM. https://doi.org/10.1145/2330601.2330661.

  • Singer, G., Golan, M., Shiff, R., & Kleper, D. (2022). Evaluating the effectiveness of accommodations given to students with learning impairments: Ordinal and interpretable machine-learning-based methodology. IEEE Transactions on Learning Technologies, 15(6), 736–746. https://doi.org/10.1109/TLT.2022.3214537.

    Article  Google Scholar 

  • Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366–377. https://doi.org/10.1111/jcal.12263.

    Article  Google Scholar 

  • Tsao, A., Yousefzadeh, S. A., Meck, W. H., Moser, M. B., & Moser, E. I. (2022). The neural bases for timing of durations. Nature Reviews Neuroscience, 23(11), 646–665. https://doi.org/10.1038/s41583-022-00623-3.

    Article  Google Scholar 

  • Worsley, M., & Blikstein, P. (2018). A Multimodal analysis of making. International Journal of Artificial Intelligence in Education, 28(3), 385–419. https://doi.org/10.1007/s40593-017-0160-1.

    Article  Google Scholar 

  • Worsley, M., Abrahamson, D., Blikstein, P., Grover, S., Schneider, B., & Tissenbaum, M. (2016). Situating Multimodal Learning Analytics. International Conference for the Learning Sciences 2016, 1346, 1349.

    Google Scholar 

  • Yusheng Huang (2005). On Augustine’s Transformation of Time View - saving phenomenon and defending God. Zhejiang Academic Journal, 4, 15–21. https://doi.org/10.16235/j.cnki.33-1005/c.2005.04.007.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Teachers Education, Zhaoqing University, 1st Zhaoqing Avenue, Duanzhou, Zhaoqing City, Guangdong, China

    Ran Bao

  2. Department of Computer Technology, Guangzhou Institute of Science and Technology China Location, 638 Xingtai Third Road, Taihe Town, Baiyun District, Guangzhou, China

    Jianyong Chen

Authors
  1. Ran Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianyong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ran Bao.

Ethics declarations

Competing Interests

The authors report there are no competing interests to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bao, R., Chen, J. Everyone Matters: A Multimodal Learning Analysis Framework Based on Individual Time Series. Tech Know Learn (2024). https://doi.org/10.1007/s10758-024-09742-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10758-024-09742-5

Keywords

Search

Navigation