Metacognition and Learning

, Volume 9, Issue 2, pp 187–215 | Cite as

Analyzing the temporal evolution of students’ behaviors in open-ended learning environments

  • John S. Kinnebrew
  • James R. Segedy
  • Gautam BiswasEmail author


Metacognition and self-regulation are important for developing effective learning in the classroom and beyond, but novice learners often lack effective metacognitive and self-regulatory skills. However, researchers have demonstrated that metacognitive processes can be developed through practice and appropriate scaffolding. Betty’s Brain, an open-ended computer-based learning environment, helps students practice their cognitive skills and develop related metacognitive strategies as they learn science topics. In this paper, we analyze students’ activity sequences in a study that compared different categories of adaptive scaffolding in Betty’s Brain. The analysis techniques for measuring students’ cognitive and metacognitive processes extend our previous work on using sequence mining methods to discover students’ frequently-used behavior patterns by (i) developing a systematic approach for interpreting derived behavior patterns using a cognitive/metacognitive task model and (ii) analyzing the evolution of students’ frequent behavior patterns over time. Our results show that it is possible to identify students’ learning behaviors and analyze their evolution as they work in the Betty’s Brain environment. Further, the results illustrate that changes in student behavior were generally consistent with the scaffolding provided, suggesting that these metacognitive strategies can be taught to middle school students in computer-based learning environments.


Cognitive/metacognitive models Open-ended learning environments Scaffolding Metacognitive strategies Sequence mining Temporal evolution 


  1. Agrawal, R., & Srikant, R. (1995). Mining sequential patterns. In Proceedings of the 11th IEEE international conference on data engineering (ICDE) (pp. 3–14).Google Scholar
  2. Aleven, V., McLaren, B., Roll, I., Koedinger, K. (2006). Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16(2), 101–128.Google Scholar
  3. Aleven, V., Roll, I., McLaren, B., Koedinger, K. (2010). Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system. Educational Psychologist, 45(4), 224–233.CrossRefGoogle Scholar
  4. Amershi, S., & Conati, C. (2009). Combining unsupervised and supervised classification to build user models for exploratory learning environments. Journal of Educational Data Mining, 1(1), 18–71.Google Scholar
  5. Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning: the role of self-regulated learning. Educational Psychologist, 40(4), 199–209.CrossRefGoogle Scholar
  6. Azevedo, R., & Aleven, V. (2013). Metacognition and learning technologies: an overview of current interdisciplinary research. In R. Azevedo & V. Aleven (Eds.), International handbook on metacognition and learning technologies (pp. 1–16). New York: Springer.Google Scholar
  7. Azevedo, R., & Witherspoon, A. (2009). Self-regulated use of hypermedia. In A. Graesser, J. Dunlosky, D. Hacker (Eds.), Handbook of metacognition in education. Erlbaum, Mahwah.Google Scholar
  8. Azevedo, R., Moos, D.C., Johnson, A.M., Chauncey, A.D. (2010). Measuring cognitive and metacognitive regulatory processes during hypermedia learning: issues and challenges. Educational Psychologist, 45(4), 210–223.CrossRefGoogle Scholar
  9. Azevedo, R., Cromley, J., Moos, D.C., JGreene, J., Winters, F. (2011). Adaptive content and process scaffolding: a key to facilitating students self-regulated learning with hypermedia. Psychological Testing and Assessment Modeling, 53, 106–140.Google Scholar
  10. Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-Behnagh, R., Bouchet, F., Landis, R. (2013). Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self regulatory processes during learning with multi-agent systems. In R. Azevedo & V. Aleven (Eds.), International handbook on metacognition and learning technologies (pp. 427–449). New York: Springer.Google Scholar
  11. Bandura, A. (1982). The assessment and predictive generality of self-percepts of efficacy. Journal of Behavior Therapy and Experimental Psychiatry, 13(3), 195–199.CrossRefGoogle Scholar
  12. Bandura, A. (1984). Recycling misconceptions of perceived self-efficacy. Cognitive Therapy and Research, 8(3), 231–255.CrossRefGoogle Scholar
  13. Bandura, A. (1997). Self-efficacy: the exercise of control. New York: Freeman.Google Scholar
  14. Biswas, G., Leelawong, K., Schwartz, D., Vye, N., Vanderbilt, T. (2005). Learning by teaching: a new agent paradigm for educational software. Applied Artificial Intelligence, 19(3), 363–392.CrossRefGoogle Scholar
  15. Biswas, G., Jeong, H., Kinnebrew, J., Sulcer, B., Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning, 5(02), 123–152.CrossRefGoogle Scholar
  16. Biswas, G., Kinnebrew, J.S., Segedy, J.R. (2013). Analyzing students’ metacognitive strategies in open-ended learning environments. In: Proceedings of the 35th annual meeting of the cognitive science society. Germany: Berlin.Google Scholar
  17. Bransford, J., Brown, A., Cocking, R. (Eds.) (2000). How people learn. Washington, DC: National Academy Press.Google Scholar
  18. Brown, A., Bransford, J.D., Ferrara, R.A., Campione, J.C. (1983). Learning, remembering, and understanding. In P. Mussen (Ed.), Handbook of child psychology. Hoboken: John Wiley.Google Scholar
  19. Butler, D., & Winne, P. (1995). Feedback and self-regulated learning: a theoretical synthesis. Review of Educational Research, 65(3), 245.CrossRefGoogle Scholar
  20. Chi, M., Glaser, R., Farr, M. (1988). The nature of expertise. Lawrence Erlbaum Associates, Inc.Google Scholar
  21. Chipman, S.F., Schraagen, J.M., Shalin, V.L. (2000). Introduction to cognitive task analysis. In J.M. Schraagen, S.F. Chipman, V.L. Shalin (Eds.), Cognitive task analysis (pp. 3–23). Psychology Press.Google Scholar
  22. Cross, D., & Paris, S. (1988). Developmental and instructional analyses of children’s metacognition and reading comprehension. Journal of Educational Psychology, 80(2), 131.CrossRefGoogle Scholar
  23. Devolder, A., van Braak, J., Tondeur, J. (2012). Supporting self-regulated learning in computer-based learning environments: systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning, 28, 557–573.CrossRefGoogle Scholar
  24. Flavell, J. (1979). Metacognition and cognitive monitoring: a new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906.CrossRefGoogle Scholar
  25. Ge, X., & Land, S.M. (2003). Scaffolding students problem-solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology Research and Development, 51(1), 21–38.CrossRefGoogle Scholar
  26. Grotzer, T., & Mittlefehldt, S. (2012). The role of metacognition in students understanding and transfer of explanatory structures in science. Metacognition in science education (pp. 79–99).Google Scholar
  27. Hadwin, A., Nesbit, J., Jamieson-Noel, D., Code, J., Winne, P. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2), 107–124.CrossRefGoogle Scholar
  28. Hennessey, M. (1999). Probing the dimensions of metacognition: implications for conceptual change teaching-learning. In NARST, ERIC.Google Scholar
  29. Jain, A., & Dubes, R. (1988). Algorithms for clustering data. Upper Saddle River: Prentice-Hall.Google Scholar
  30. Kinnebrew, J.S., & Biswas, G. (2012). Identifying learning behaviors by contextualizing differential sequence mining with action features and performance evolution. In Proceedings of the 5th international conference on educational data mining (EDM). Chania.Google Scholar
  31. Kinnebrew, J.S., Loretz, K.M., Biswas, G. (2013a). A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining, 5(1), 190–219.Google Scholar
  32. Kinnebrew, J.S., Mack, D.L., Biswas, G. (2013b). Mining temporally-interesting learning behavior patterns. In S.K. DMello, R.A. Calvo, A. Olney (Eds.), Proceedings of the 6th international conference on educational data mining (pp. 252–255). Memphis .Google Scholar
  33. Kramarski, B., & Mevarech, Z. (2003). Enhancing mathematical reasoning in the classroom: the effects of cooperative learning and metacognitive training. American Educational Research Journal, 40(1), 281–310.CrossRefGoogle Scholar
  34. Lajoie, S., & Derry, S. (1993). Computers as cognitive tools. Hillsdale: Lawrence Erlbaum Associates.Google Scholar
  35. Land, S. (2000). Cognitive requirements for learning with open-ended learning environments. Educational Technology Research and Development, 48(3), 61–78.CrossRefGoogle Scholar
  36. Larkin, J., McDermott, J., Simon, D., Simon, H. (1980). Expert and novice performance in solving physics problems. Science, 208(4450), 1335–1342.CrossRefGoogle Scholar
  37. Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: the Betty’s Brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.Google Scholar
  38. Lester, J., Mott, B., Robinson, J., Rowe, J., Shores, L. (2013). Supporting seld-regulated learning in narrative-centered environments. In R. Azevedo, & V. Aleven (Eds.), International handbook on metacognition and learning technologies (pp. 471–483). New York: Springer.Google Scholar
  39. Li, C., & Biswas, G. (2002). Unsupervised learning with mixed numeric and nominal data. IEEE Transactions on Knowledge and Data Engineering, 14(4), 673–690.CrossRefGoogle Scholar
  40. Martinez, M.E. (2006). What is metacognition? Phi Delta Kappan (pp. 696–699).Google Scholar
  41. Mashima, D., Kobourov, S., Hu, Y. (2012). Visualizing dynamic data with maps. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1424–1437.CrossRefGoogle Scholar
  42. Mayer, R.E. (1996). Learning strategies for making sense out of expository text: the SOI model for guiding three cognitive processes in knowledge construction. Educational Psychology Review, 8(4), 357–371.CrossRefGoogle Scholar
  43. Molenaar, I., Roda, C., van Boxtel, C., Sleegers, P. (2012). Dynamic scaffolding of socially regulated learning in a computer-based learning environment. Computers and Education, 59(2), 515–523.CrossRefGoogle Scholar
  44. Murtagh, F. (1983). A survey of recent advances in hierarchical clustering algorithms. The Computer Journal, 26(4), 354–359.CrossRefGoogle Scholar
  45. Nesbit, J., Zhou, M., Xu, Y., Winne, P. (2007). Advancing log analysis of student interactions with cognitive tools. In 12th biennial conference of the european association for research on learning and insruction (EARLI).Google Scholar
  46. Perera, D., Kay, J., Koprinska, I., Yacef, K., Zaïane, O. (2009). Clustering and sequential pattern mining of online collaborative learning data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759–772.CrossRefGoogle Scholar
  47. Pieschl, S., & Stahl, E. RB. (2013). Adaptation to context as core component of self-regulated learning: the example of complexity and epistemic beliefs. In R. Azevedo, & V. Aleven (Eds.), International handbook on metacognition and learning technologies (pp. 53–65). New York: Springer.Google Scholar
  48. Pintrich, P. (2000). An achievement goal theory perspective on issues in motivation terminology, theory, and research* 1. Contemporary Educational Psychology, 25(1), 92–104.CrossRefGoogle Scholar
  49. Pintrich, P., Smith, D., García, T., McKeachie, W. (1991). A manual for the use of the motivated strategies for learning questionnaire (mslq) (p. 48109:1259). Ann Arbor Michigan: National Center for Research to Improve Postsecondary Teaching and Learning.Google Scholar
  50. Plantevit, M., Laurent, A., Teisseire, M. (2006). Hype: mining hierarchical sequential patterns. In Proceedings of the 9th ACM international workshop on data warehousing and OLAP (pp. 19–26). ACM.Google Scholar
  51. Pleil, J., Stiegel, M., Madden, M., Sobus, J. (2011). Heat map visualization of complex environmental and biomarker measurements. Chemosphere, 84(5), 716–723.CrossRefGoogle Scholar
  52. Puntambekar, S., & Hubscher, R. (2005). Tools for scaffolding students in a complex learning environment: what have we gained and what have we missed?Educational Psychologist, 40(1), 1–12.CrossRefGoogle Scholar
  53. Sabourin, J., Mott, B., Lester, J. (2013). Utilizing dynamic bayes nets to improve early prediction models of self-regulated learning. User Modeling, Adaptation, and Personalization, 228–241.Google Scholar
  54. Schraw, G., Kauffman, D., Lehman, S. (2002). Self-regulated learning theory. In L. Nadel (Ed.), The encyclopedia of cognitive science (pp. 1063–1073). London: Nature Publishing Company.Google Scholar
  55. Schraw, G., Crippen, K., Hartley, K. (2006). Promoting self-regulation in science education: metacognition as part of a broader perspective on learning. Research in Science Education, 36(1), 111–139.CrossRefGoogle Scholar
  56. Schraw, G., Olafson, L., Weibel, M., Sewing, D. (2012). Metacognitive knowledge and field-based science learning in an outdoor environmental education program. Metacognition in Science Education, 57–77.Google Scholar
  57. Schunk, D., & Zimmerman, B. (1994). Self-regulation of learning and performance: issues and educational applications. Mahwah: Lawrence Erlbaum Associates, Inc.Google Scholar
  58. Schunk, D., & Zimmerman, B. (1997). Social origins of self-regulatory competence. Educational Psychologist, 32(4), 195–208.CrossRefGoogle Scholar
  59. Segedy, J.R., Kinnebrew, J.S., Biswas, G. (2012). Supporting student learning using conversational agents in a teachable agent environment. In Proceedings of the 10th international conference of the learning sciences.Google Scholar
  60. Segedy, J.R., Kinnebrew, J.S., Biswas, G. (2013). The effect of contextualized conversational feedback in a complex open-ended learning environment. Educational Technology Research and Development, 61(1), 71–89.CrossRefGoogle Scholar
  61. Segedy, J.R., Biswas, G., Sulcer, B. (2014). A model-based behavior analysis approach for open-ended environments. Educational Technology & Society, 17(1).Google Scholar
  62. Sotiriou, C., Wirapati, P., Loi, S., Harris, A., Fox, S., Smeds, J., Nordgren, H., Farmer, P., Praz, V., Haibe-Kains, B., et al. (2006). Gene expression profiling in breast cancer: understanding the molecular basis of histologic grade to improve prognosis. Journal of the National Cancer Institute, 98(4), 262–272.CrossRefGoogle Scholar
  63. Sperling, R., Howard, B., Miller, L., Murphy, C. (2002). Measures of children’s knowledge and regulation of cognition. Contemporary Educational Psychology, 27(1), 51–79.CrossRefGoogle Scholar
  64. Su, J.M., Tseng, S.S., Wang, W., Weng, J.F., Yang, J., Tsai, W.N. (2006). Learning portfolio analysis and mining for scorm compliant environment. Journal of Educational Technology and Society, 9(1), 262–275.Google Scholar
  65. Tang, T., & McCalla, G. (2002). Student modeling for a web-based learning environment: a data mining approach. In Proceedings of the 18th national conference on artificial intelligence (pp. 967–968). AAAI Press: Association for the Advancement of Artificial Intelligence.Google Scholar
  66. VanLehn, K. (1996). Cognitive skill acquisition. Annual Review of Psychology, 47(1), 513–539.CrossRefGoogle Scholar
  67. Veenman, M., & Spaans, M. (2005). Relation between intellectual and metacognitive skills: age and task differences. Learning and Individual Differences, 15(2), 159–176.CrossRefGoogle Scholar
  68. Wagster, J., Tan, J., Wu, Y., Biswas, G., Schwartz, D. (2007). Do learning by teaching environments with metacognitive support help students develop better learning behaviors? In Proceedings of the 29th annual meeting of the cognitive science society (pp. 695–700).Google Scholar
  69. Weinstein, C.E., & Mayer, R.E. (1986). The teaching of learning strategies. In M.C. Wittrock (Ed.), Handbook of research on teaching, 3rd edn. (pp. 315–327). New York: Macmillan.Google Scholar
  70. Whitebread, D., & Cárdenas, V. (2012). Self-regulated learning and conceptual development in young children: the development of biological understanding. Metacognition in Science Education, 101–132.Google Scholar
  71. Wilkinson, L., & Friendly, M. (2009). The history of the cluster heat map. The American Statistician, 63(2), 179–184.CrossRefGoogle Scholar
  72. Winne, P. (1996). A metacognitive view of individual differences in self-regulated learning. Learning and Individual Differences, 8(4), 327–353.CrossRefGoogle Scholar
  73. Winne, P. (2008). The weave of motivation and self-regulated learning. In D. Schunk, & B. Zimmerman (Eds.), Motivation and self-regulated learning: theory, research, and applications (pp. 297–314). NY: Taylor & Francis.Google Scholar
  74. Winne, P., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572.CrossRefGoogle Scholar
  75. Winne, P.H., & Hadwin, A.F. (1998). Studying as self-regulated learning. In D.J. Hacker, J. Dunlosky, A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Lawrence Erlbaum Associates Publishers.Google Scholar
  76. Zhang, M., & Quintana, C. (2012). Scaffolding strategies for supporting middle school students online inquiry processes. Computers and Education, 58(1), 181–196.CrossRefGoogle Scholar
  77. Zimmerman, B. (2001). Theories of self-regulated learning and academic achievement. An overview and analysis. In B. Zimmerman, & D. Schunk (Eds.), Self-regulated learning and academic achievement: theoretical perspectives (pp. 1–37). Mahwah: Erlbaum.Google Scholar
  78. Zimmerman, B., & Martinez-Pons, M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23(4), 614–628.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • John S. Kinnebrew
    • 1
  • James R. Segedy
    • 1
  • Gautam Biswas
    • 1
    Email author
  1. 1.Department of EECS/ISISVanderbilt UniversityNashvilleUSA

Personalised recommendations