Skip to main content

Advertisement

SpringerLink
Log in

Multimodal learning analytics for assessing teachers’ self-regulated learning in planning technology-integrated lessons in a computer-based environment

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Teachers’ self-regulated learning (SRL) plays a crucial role in developing technological pedagogical content knowledge (TPACK), a complex professional skill. It is crucial to identify teachers’ SRL activities that may lead to favorable TPACK. Previous studies have focused on the analysis of individual data sources from self-reported surveys or log files, which are insufficient to capture all SRL activities in the TPACK context. While multimodal learning analytics (MMLA) has the potential to improve SRL measurement, it remains unknown how multimodal data collected from different sources can be combined to identify salient features of SRL activities and examine how TPACK outcomes can be predicted by SRL activities identified from multimodal data. This study combined multimodal data from computer logs and think-aloud data to analyze teachers’ SRL activities in designing a technology-integrated lesson. We identified the salient features of SRL from the combined data and explored how identified SRL activities might predict TPACK outcomes reflected in teacher-generated lesson plans. The results of random forest regression analysis show that three SRL activities from the logs and two from the think-aloud data formed the best combination that explained a significant proportion of variances in TPACK performance. The impact of MMLA in SRL measurement and the implication of this study are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are not publicly available due to the Ethics requirements but are available from the corresponding author upon reasonable request.

References

  • Ahn, B. “Tony,” & Harley, J. M. (2020). Facial expressions when learning with a Queer History App: Application of the control value theory of achievement emotions. British Journal of Educational Technology, 51(5), 1563–1576. https://doi.org/10.1111/bjet.12989

  • Angeli, C., & Valanides, N. (2009). Epistemological and methodological issues for the conceptualization, development, and assessment of ICT–TPCK: Advances in technological pedagogical content knowledge (TPCK). Computers & Education, 52(1), 154–168. https://doi.org/10.1016/j.compedu.2008.07.006

    Article  Google Scholar 

  • Antonietti, A., Colombo, B., & Di Nuzzo, C. (2015). Metacognition in self-regulated multimedia learning: Integrating behavioural, psychophysiological and introspective measures. Learning, Media and Technology, 40(2), 187–209. https://doi.org/10.1080/17439884.2014.933112

    Article  Google Scholar 

  • Azevedo, R., & Gašević, D. (2019). Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges. Computers in Human Behavior, 96, 207–210. https://doi.org/10.1016/j.chb.2019.03.025

    Article  Google Scholar 

  • Biau, G., & Scornet, E. (2016). A random forest guided tour. TEST, 25(2), 197–227. https://doi.org/10.1007/s11749-016-0481-7

    Article  MathSciNet  MATH  Google Scholar 

  • Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 2. https://doi.org/10.18608/jla.2016.32.11

    Article  Google Scholar 

  • Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.

    Article  MATH  Google Scholar 

  • Cho, M.-H., & Yoo, J. S. (2017). Exploring online students’ self-regulated learning with self-reported surveys and log files: A data mining approach. Interactive Learning Environments, 25(8), 970–982. https://doi.org/10.1080/10494820.2016.1232278

    Article  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284

    Article  Google Scholar 

  • Ejubović, A., & Puška, A. (2019). Impact of self-regulated learning on academic performance and satisfaction of students in the online environment. Knowledge Management & E-Learning, 11(3), 345–363. https://doi.org/10.34105/j.kmel.2019.11.018

  • Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251. https://doi.org/10.1037/0033-295X.87.3.215

    Article  Google Scholar 

  • Fan, Y., Lim, L., van der Graaf, J., Kilgour, J., Raković, M., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2022a). Improving the measurement of self-regulated learning using multi-channel data. Metacognition and Learning. https://doi.org/10.1007/s11409-022-09304-z

    Article  Google Scholar 

  • Fan, Y., van der Graaf, J., Lim, L., Raković, M., Singh, S., Kilgour, J., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2022b). Towards investigating the validity of measurement of self-regulated learning based on trace data. Metacognition and Learning. https://doi.org/10.1007/s11409-022-09291-1

    Article  Google Scholar 

  • Farrell, T. S. C. (2002). Lesson planning. In J. C. Richards & W. A. Renandya (Eds.), Methodology of language teaching: An anthology of current practice (pp. 30–39). Cambridge University Press.

    Chapter  Google Scholar 

  • Gašević, D., Greiff, S., & Shaffer, D. W. (2022). Towards strengthening links between learning analytics and assessment: Challenges and potentials of a promising new bond. Computers in Human Behavior, 134, 107304. https://doi.org/10.1016/j.chb.2022.107304

    Article  Google Scholar 

  • Greene, J. A., Bolick, C. M., & Robertson, J. (2010). Fostering historical knowledge and thinking skills using hypermedia learning environments: The role of self-regulated learning. Computers & Education, 54(1), 230–243. https://doi.org/10.1016/j.compedu.2009.08.006

    Article  Google Scholar 

  • Greene, J. A., Plumley, R. D., Urban, C. J., Bernacki, M. L., Gates, K. M., Hogan, K. A., Demetriou, C., & Panter, A. T. (2021). Modeling temporal self-regulatory processing in a higher education biology course. Learning and Instruction, 72, 101201. https://doi.org/10.1016/j.learninstruc.2019.04.002

    Article  Google Scholar 

  • Haataja, E., Dindar, M., Malmberg, J., & Järvelä, S. (2022). Individuals in a group: Metacognitive and regulatory predictors of learning achievement in collaborative learning. Learning and Individual Differences, 96, 102146. https://doi.org/10.1016/j.lindif.2022.102146

    Article  Google Scholar 

  • Hu, C. C., Yeh, H. C., & Chen, N. S. (2021). Teacher development in robot and IoT knowledge, skills, and attitudes with the use of the TPACK-based Support-Stimulate-Seek approach. Interactive Learning Environments, 1–20. https://doi.org/10.1080/10494820.2021.2019058

  • Huang, L., Li, S., Poitras, E. G., & Lajoie, S. P. (2020). Latent profiles of self‐regulated learning and their impacts on teachers’ technology integration. British Journal of Educational Technology, 52(2), 695–713. https://doi.org/10.1111/bjet.13050

  • Huang, L., & Lajoie, S. P. (2021). Process analysis of teachers’ self-regulated learning patterns in technological pedagogical content knowledge development. Computers & Education, 166, 104–169. https://doi.org/10.1016/j.compedu.2021.104169

  • Kayri, M., Kayri, I., & Gencoglu, M. T. (2017). The performance comparison of Multiple Linear Regression, Random Forest and Artificial Neural Network by using photovoltaic and atmospheric data. 2017 14th International Conference on Engineering of Modern Electric Systems (EMES), 1–4. https://doi.org/10.1109/EMES.2017.7980368

  • Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., & Baker, R. (2015). Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics, 2(3), 3. https://doi.org/10.18608/jla.2015.23.6

    Article  Google Scholar 

  • Lee, Y., & Lee, J. (2014). Enhancing pre-service teachers’ self-efficacy beliefs for technology integration through lesson planning practice. Computers & Education, 73, 121–128. https://doi.org/10.1016/j.compedu.2014.01.001

    Article  Google Scholar 

  • Malmberg, J., Järvelä, S., Holappa, J., Haataja, E., Huang, X., & Siipo, A. (2019). Going beyond what is visible: What multichannel data can reveal about interaction in the context of collaborative learning? Computers in Human Behavior, 96, 235–245. https://doi.org/10.1016/j.chb.2018.06.030

    Article  Google Scholar 

  • Ordóñez, L. D., Schweitzer, M. E., Galinsky, A. D., & Bazerman, M. H. (2009). Goals gone wild: The systematic side effects of overprescribing soal setting. Academy of Management Perspectives, 23(1), 6–16. https://doi.org/10.5465/amp.2009.37007999

    Article  Google Scholar 

  • Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825–2830.

    MathSciNet  MATH  Google Scholar 

  • Poitras, E., Doleck, T., Huang, L., Li, S., & Lajoie, S. P. (2017). Advancing teacher technology education using open-ended learning environments as research and training platforms. Australasian Journal of Educational Technology, 33(3). https://doi.org/10.14742/ajet.3498

  • Poitras, E. G., Fazeli, N., & Mayne, Z. R. (2018). Modeling student teachers’ information-seeking behaviors while learning with network-based tutors. Journal of Educational Technology Systems, 47(2), 227–247. https://doi.org/10.1177/0047239518797086

    Article  Google Scholar 

  • Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450–1484. https://doi.org/10.1111/bjet.12993

    Article  Google Scholar 

  • Siadaty, M., Gašević, D., & Hatala, M. (2016). Associations between technological scaffolding and micro-level processes of self-regulated learning: A workplace study. Computers in Human Behavior, 55, 1007–1019. https://doi.org/10.1016/j.chb.2015.10.035

    Article  Google Scholar 

  • Smith, P. F., Ganesh, S., & Liu, P. (2013). A comparison of random forest regression and multiple linear regression for prediction in neuroscience. Journal of Neuroscience Methods, 220(1), 85–91. https://doi.org/10.1016/j.jneumeth.2013.08.024

    Article  Google Scholar 

  • Sonnenberg, C., & Bannert, M. (2019). Using process mining to examine the sustainability of instructional support: How stable are the effects of metacognitive prompting on self-regulatory behavior? Computers in Human Behavior. https://doi.org/10.1016/j.chb.2018.06.003

    Article  Google Scholar 

  • Srivastava, N., Fan, Y., Rakovic, M., Singh, S., Jovanovic, J., van der Graaf, J., Lim, L., Surendrannair, S., Kilgour, J., Molenaar, I., Bannert, M., Moore, J., & Gasevic, D. (2022). Effects of Internal and External Conditions on Strategies of Self-regulated Learning: A Learning Analytics Study. LAK22: 12th International Learning Analytics and Knowledge Conference, 392–403. https://doi.org/10.1145/3506860.3506972

  • Winne, P. H. (2018). Theorizing and researching levels of processing in self-regulated learning. The British Journal of Educational Psychology, 88(1), 9–20. https://doi.org/10.1111/bjep.12173

    Article  Google Scholar 

  • Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), The educational psychology series. Metacognition in educational theory and practice (pp. 277–304). Lawrence Erlbaum Associates Publishers.

  • Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–566). Academic. https://doi.org/10.1016/B978-012109890-2/50045-7

    Chapter  Google Scholar 

  • Zheng, J., Huang, L., Li, S., Lajoie, S. P., Chen, Y., & Hmelo-Silver, C. E. (2021). Self regulation and emotion matter: A case study of instructor interactions with a learning analytics dashboard. Computers & Education, 161, 104061. https://doi.org/10.1016/j.compedu.2020.104061

  • Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://doi.org/10.1016/j.learninstruc.2012.03.004

    Article  Google Scholar 

Download references

Funding

This research was supported in part by the doctoral scholarship from Fonds de Recherche du Québec—Société’s et Culture (FRQSC) and RGC Post-doctoral Fellowship (PDFS2122-7H03) awarded to Dr. Lingyun Huang, and Social Sciences and Humanities Research Council (SSHRC) Partnership Grant of Canada (895–2011-1006) awarded to Prof. Susanne Lajoie. This research is also partially supported by the National Natural Science Foundation of China (No. 61977023) and Eastern Scholar Chair Professorship Fund (No. JZ2017005) from the Shanghai Municipal Education Commission of China awarded to Prof. Minhong Wang.

Author information

Authors and Affiliations

  1. Faculty of Education, The University of Hong Kong, Hong Kong SAR, China

    Lingyun Huang, Boyin Chen & Minhong Wang

  2. Canada Research Chair, Simon Fraser University, Vancouver, Canada

    Tenzin Doleck

  3. Department of Educational and Counselling Psychology, McGill University, Montreal, Canada

    Xiaoshan Huang, Chengyi Tan & Susanne P. Lajoie

Authors
  1. Lingyun Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tenzin Doleck
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boyin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoshan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chengyi Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Susanne P. Lajoie
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Minhong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Lingyun Huang: Conceptualization, Data analysis, Formal analysis, Investigation, Methodology, Writing (original draft, review, and editing)

Boyin Chen: Writing (original draft, review, and editing)

Tenzin Doleck: Writing (original draft, review, and editing)

Xiaoshan Huang: Data curation, Writing (original draft)

Chengyi Tan: Data curation, Writing (original draft)

Susanne P. Lajoie: Supervision, Writing (review and editing)

Minhong Wang: Supervision, Writing (review and editing)

Corresponding author

Correspondence to Lingyun Huang.

Ethics declarations

Ethics statement

The research obtained Ethics approval from the Research Ethics Board (IRB) of McGill University. All procedures performed in this research were in accordance with the ethical guidelines of the IRB of McGill University.

Conflict of interest disclosure

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, L., Doleck, T., Chen, B. et al. Multimodal learning analytics for assessing teachers’ self-regulated learning in planning technology-integrated lessons in a computer-based environment. Educ Inf Technol 28, 15823–15843 (2023). https://doi.org/10.1007/s10639-023-11804-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-023-11804-7

Keywords

Search

Navigation