Skip to main content
SpringerLink
Log in

Measuring and Validating Assumptions About Self-Regulated Learning with Multimodal Data

  • Chapter
  • First Online:
Unobtrusive Observations of Learning in Digital Environments

Part of the book series: Advances in Analytics for Learning and Teaching ((AALT))

Abstract

Individuals who engage in self-regulated learning (SRL) tend to perform better in complex learning tasks. However, learners’ ability to self-regulate can vary. To understand and support learners’ SRL, collecting information about their engagement in specific learning processes in the context of learning tasks is necessary. However, SRL is sufficiently complex that it is not directly observable. Capturing the SRL processes that occur during learning, as students interact with elements of tasks hosted on virtual learning technologies (e.g., learning management systems; LMS), is possible because learners’ actions generate observable events that these technologies log. However, discerning how these events reflect SRL processes poses several major theoretical, methodical, and analytical challenges. To address these challenges, we present two projects to illustrate how researchers validated inferences about SRL processes. We demonstrate how observational indicators drawn from multiple channels of event data must be (a) collected from the technologies’ log files and the record of learners’ self-reports of their learning process, (b) instrumented to describe learner, event, and context, and (c) integrated and temporally aligned. Afterward, we show how researchers can hypothesize about the SRL processes digital events reflect and test inferences using secondary channels of explanatory data provided by learners during the tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 59.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Change history

  • 30 July 2023

    A correction has been published.

References

  • Azevedo, R. (2018). Using hypermedia as a metacognitive tool for enhancing student learning? The role of self-regulated learning. In Educational psychologist (pp. 199–209). Routledge.

    Google Scholar 

  • Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of Educational Psychology, 96(3), 523.

    Google Scholar 

  • Azevedo, R., Taub, M., & Mudrick, N. V. (2017). Understanding and reasoning about real-time cognitive, affective, and metacognitive processes to foster self-regulation with advanced learning technologies. In Handbook of self-regulation of learning and performance (pp. 254–270). Routledge.

    Google Scholar 

  • Azevedo, R., Taub, M., Mudrick, N. V., Martin, S. A., & Grafsgaard, J. (2018). Using multi-channel trace data to infer and foster self-regulated learning between humans and advanced learning technologies. In Handbook of self-regulation of learning and performance (2nd ed., pp. 254–270). Routledge.

    Google Scholar 

  • Ben-Eliyahu, A., & Bernacki, M. L. (2015). Addressing complexities in self-regulated learning: A focus on contextual factors, contingencies, and dynamic relations. Metacognition and Learning, 10(1), 1–13.

    Google Scholar 

  • Bernacki, M. L. (2018). Examining the cyclical, loosely sequenced, and contingent features of self-regulated learning: Trace data and their analysis. In Handbook of self-regulation of learning and performance (pp. 370–387). Routledge.

    Google Scholar 

  • Binbasaran Tuysuzoglu, B., & Greene, J. A. (2015). An investigation of the role of contingent metacognitive behavior in self-regulated learning. Metacognition and Learning, 10(1), 77–98.

    Google Scholar 

  • Biswas, G., Leelawong, K., Schwartz, D., Vye, N., & The Teachable Agents Group at Vanderbilt. (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19(3–4), 363–392.

    Google Scholar 

  • Callan, G. L., Rubenstein, L. D., Ridgley, L. M., & McCall, J. R. (2021). Measuring self-regulated learning during creative problem-solving with SRL microanalysis. Psychology of Aesthetics, Creativity, and the Arts, 15(1), 136.

    Google Scholar 

  • Cleary, T. J., Zimmerman, B. J., & Keating, T. (2006). Training physical education students to self-regulate during basketball free throw practice. Research Quarterly for Exercise and Sport, 77(2), 251–262.

    Google Scholar 

  • Cleary, T. J., Callan, G. L., & Zimmerman, B. J. (2012). Assessing self-regulation as a cyclical, context-specific phenomenon: Overview and analysis of SRL microanalytic protocols. Education Research International, 2012, 1.

    Google Scholar 

  • Cleary, T. J., & Callan, G. L. (2018). Assessing self-regulated learning using microanalytic methods. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (pp. 338–351). Routledge/Taylor & Francis Group

    Google Scholar 

  • Cohen, J. (1968). Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4), 213.

    Google Scholar 

  • Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46(1), 6–25.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. The MIT Press.

    Google Scholar 

  • Fiel, J., Lawless, K. A., & Brown, S. W. (2018). Timing matters: Approaches for measuring and visualizing behaviours of timing and spacing of work in self-paced online teacher professional development courses. Journal of Learning Analytics, 5(1), 25–40.

    Google Scholar 

  • Fox, E. (2009). The role of reader characteristics in processing and learning from informational text. Review of Educational Research, 79(1), 197–261.

    Google Scholar 

  • Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137(2), 316.

    Google Scholar 

  • Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71.

    Google Scholar 

  • Greene, J. A., & Azevedo, R. (2007). A theoretical review of Winne and Hadwin’s model of self-regulated learning: New perspectives and directions. Review of Educational Research, 77(3), 334–372.

    Google Scholar 

  • Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their relations to the acquisition of a sophisticated mental model of a complex system. Contemporary Educational Psychology, 34(1), 18–29.

    Google Scholar 

  • Greene, J. A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive and metacognitive processes while using computer-based learning environments. Educational Psychologist, 45(4), 203–209.

    Google Scholar 

  • Greene, J. A., Robertson, J., & Costa, L. J. C. (2011). Assessing self-regulated learning using think-aloud methods. In Handbook of self-regulation of learning and performance (pp. 313–328). Routledge.

    Google Scholar 

  • Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015). Domain-specificity of self-regulated learning processing in science and history. Contemporary Educational Psychology, 42, 111–128.

    Google Scholar 

  • Greene, J. A., Deekens, V. M., Copeland, D. Z., & Yu, S. (2017). Capturing and modeling self-regulated learning using think-aloud protocols. In Handbook of self-regulation of learning and performance (pp. 323–337). Routledge.

    Google Scholar 

  • Greene, J. A., Lobczowski, N. G., Freed, R., Cartiff, B. M., Demetriou, C., & Panter, A. T. (2020). Effects of a science of learning course on college students’ learning with a computer. American Educational Research Journal, 57(3), 947–978.

    Google Scholar 

  • Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2019). What multimodal data can tell us about the students’ regulation of their learning process. Learning and Instruction, 72(7), 4.

    Google Scholar 

  • Karabenick, S. A., & Zusho, A. (2015). Examining approaches to research on self-regulated learning: Conceptual and methodological considerations. Metacognition and Learning, 10(1), 151–163.

    Google Scholar 

  • Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1), 30–43.

    Google Scholar 

  • Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459.

    Google Scholar 

  • Lombardi, D., Shipley, T. F., & Astronomy Team, Biology Team, Chemistry Team, Engineering Team, Geography Team, Geoscience Team, and Physics Team. (2021). The curious construct of active learning. Psychological Science in the Public Interest, 22(1), 8–43.

    Google Scholar 

  • Malmberg, J., Järvelä, S., Holappa, J., Haataja, E., Huang, X., & Siipo, A. (2019). Going beyond what is visible: What multichannel data can reveal about interaction in the context of collaborative learning? Computers in Human Behavior, 96, 235–245.

    Google Scholar 

  • McCardle, L., & Hadwin, A. F. (2015). Using multiple, contextualized data sources to measure learners’ perceptions of their self-regulated learning. Metacognition and Learning, 10(1), 43–75.

    Google Scholar 

  • Mudrick, N. V., Azevedo, R., & Taub, M. (2019). Integrating metacognitive judgments and eye movements using sequential pattern mining to understand processes underlying multimedia learning. Computers in Human Behavior, 96, 223–234.

    Google Scholar 

  • Ochoa, X., Lang, A. C., & Siemens, G. (2017). Multimodal learning analytics. In The handbook of learning analytics (1st ed., pp. 129–141). Society for Learning Analytics Research.

    Google Scholar 

  • Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422.

    Google Scholar 

  • Pintrich, P. R. (1995). Understanding self-regulated learning. New Directions for Teaching and Learning, 1995(63), 3–12.

    Google Scholar 

  • Pintrich, P. R., Smith, D. A., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813.

    Google Scholar 

  • Roll, I., & Winne, P. H. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7–12.

    Google Scholar 

  • Rovers, S. F., Clarebout, G., Savelberg, H. H., de Bruin, A. B., & van Merriënboer, J. J. (2019). Granularity matters: Comparing different ways of measuring self-regulated learning. Metacognition and Learning, 14(1), 1–19.

    Google Scholar 

  • Salehian Kia, F., Hatala, M., Baker, R. S., & Teasley, S. D. (2021, April). Measuring students’ self-regulatory phases in LMS with behavior and real-time self report. In LAK21: 11th international learning analytics and knowledge conference (pp. 259–268).

    Google Scholar 

  • Salkind, N. J. (2006). Encyclopedia of measurement and statistics. Sage.

    Google Scholar 

  • Salmerón, L., Kintsch, W., & Caãs, J. J. (2006). Reading strategies and prior knowledge in learning from hypertext. Memory & Cognition, 34(5), 1157–1171.

    Google Scholar 

  • Schellings, G. (2011). Applying learning strategy questionnaires: Problems and possibilities. Metacognition and Learning, 6(2), 91–109.

    Google Scholar 

  • Schooler, J. W. (2011). Introspecting in the spirit of William James: Comment on Fox, Ericsson, and Best (2011). Psychological Bulletin, 137(2), 345–350.

    Google Scholar 

  • Schunk, D. H., & Greene, J. A. (Eds.). (2018). Handbook of Self-Regulation of Learning and Performance (2nd ed.). Routledge.

    Google Scholar 

  • Segedy, J. R., Kinnebrew, J. S., & Biswas, G. (2015, June). Coherence over time: Understanding day-to-day changes in students’ open-ended problem-solving behaviors. In International conference on artificial intelligence in education (pp. 449–458). Springer.

    Google Scholar 

  • Sonnenberg, C., & Bannert, M. (2019). Using Process Mining to examine the sustainability of instructional support: How stable are the effects of metacognitive prompting on self-regulatory behavior? Computers in Human Behavior, 96, 259–272.

    Google Scholar 

  • Veenman, M. V. (2011). Alternative assessment of strategy use with self-report instruments: A discussion. Metacognition and Learning, 6(2), 205–211.

    Google Scholar 

  • Winne, P. H. (2013). Self-regulated learning viewed from models of information processing. In Self-regulated learning and academic achievement (pp. 145–178). Routledge.

    Google Scholar 

  • Winne, P. H. (2018). Cognition and metacognition within self-regulation. In Handbook of self-regulation of learning and performance. Routledge.

    Google Scholar 

  • Winne, P. H. (2020). Commentary: A proposed remedy for grievances about self-report methodologies. Frontline Learning Research, 8(3), 164–173.

    Google Scholar 

  • Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Erlbaum.

    Google Scholar 

  • Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572.

    Google Scholar 

  • Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In Handbook of self-regulation (pp. 531–566). Academic.

    Google Scholar 

  • Wolters, C. A., & Won, S. (2017). Validity and the use of self-report questionnaires to assess self-regulated learning. In Handbook of self-regulation of learning and performance (pp. 307–322). Routledge.

    Google Scholar 

  • Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419.

    Google Scholar 

  • Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In Handbook of self-regulation (pp. 13–39). Academic.

    Google Scholar 

  • Zimmerman, B. J., & Schunk, D. H. (2011). Handbook of self-regulation of learning and performance. Routledge/Taylor & Francis Group.

    Google Scholar 

  • Zimmerman, B. J., & Schunk, D. H. (2013). Reflections on theories of self-regulated learning and academic achievement. In Self-regulated learning and academic achievement (pp. 282–301). Routledge.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of British Columbia, Vancouver, BC, Canada

    Fatemeh Salehian Kia

  2. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Mathew L. Bernacki & Jeffrey A. Greene

Authors
  1. Fatemeh Salehian Kia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mathew L. Bernacki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeffrey A. Greene
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatemeh Salehian Kia .

Editor information

Editors and Affiliations

  1. Centre for Change and Complexity, University of South Australia, Adelaide, SA, Australia

    Vitomir Kovanovic

  2. School of Modeling Simulation and Training, University of Central Florida, Orlando, FL, USA

    Roger Azevedo

  3. Data Science in Higher Education Learning and Teaching, Curtin University, Bentley, WA, Australia

    David C. Gibson

  4. Learning, Design and Technology, University of Mannheim, Mannheim, BW, Germany

    Dirk lfenthaler

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Salehian Kia, F., Bernacki, M.L., Greene, J.A. (2023). Measuring and Validating Assumptions About Self-Regulated Learning with Multimodal Data. In: Kovanovic, V., Azevedo, R., Gibson, D.C., lfenthaler, D. (eds) Unobtrusive Observations of Learning in Digital Environments. Advances in Analytics for Learning and Teaching. Springer, Cham. https://doi.org/10.1007/978-3-031-30992-2_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30992-2_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30991-5

  • Online ISBN: 978-3-031-30992-2

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation