The effect of contextualized conversational feedback in a complex open-ended learning environment
Betty’s Brain is an open-ended learning environment in which students learn about science topics by teaching a virtual agent named Betty through the construction of a visual causal map that represents the relevant science phenomena. The task is complex, and success requires the use of metacognitive strategies that support knowledge acquisition, causal map construction, and progress monitoring. Previous research has established that middle school students struggle at such tasks without proper scaffolding and feedback. In Betty’s Brain, this feedback is provided by Betty and Mr. Davis, another virtual agent designed to provide guidance and suggestions as students work. This paper discusses our implementation of contextualized conversational (CC) feedback, and then presents the results of an experimental study exploring the effects of this feedback in two 8th-grade science classrooms. The results illustrate some advantages of the CC feedback in comparison with a baseline dialogue mechanism that presents similar strategies in a non-conversational, non-contextualized form. While both groups showed significant pre-to-post test learning gains, the difference in learning gains between the groups was not statistically significant. However, students who received CC feedback more often performed actions in accordance with the advised strategies, and they created higher quality causal maps.
KeywordsConversational agents Open-ended learning environment Metacognition Student learning behaviors Mixed-initiative dialogue
This work has been supported by the National Science Foundation’s Information and Intelligent Systems Award #0904387.
- Adams, E. (2010). Fundamentals of game design (2nd ed.). Berkeley, CA: New Riders Pub.Google Scholar
- Biswas, G., Jeong, H., Kinnebrew, J. S., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning, 5(2), 123–152. doi: 10.1142/S1793206810000839.CrossRefGoogle Scholar
- Gouli, E., Gogoulou, A., Papanikolaou, K., & Grigoriadou, M. (2004). Compass: An adaptive web-based concept map assessment tool. In A. J. Cañas, J. D. Novak, F. M. González (Eds.), Proceedings of the first international conference on concept mapping, Pamplona, Spain.Google Scholar
- Jeong, H., & Biswas, G. (2008). Mining student behavior models in learning-by-teaching environments. In R. S. J. d. Baker, T. Barnes, & J. E. Beck (Eds.), Educational data mining 2008: 1st international conference on educational data mining, proceedings (pp. 127–136). Montreal, Quebec, Canada.Google Scholar
- Kinnebrew, J. S., Loretz, K. M., & Biswas, G. (in press). A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining.Google Scholar
- Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The Betty’s Brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.Google Scholar
- Mendicino, M., Razzaq, L., & Hefferman, N. T. (2009). A comparison of traditional homework to computer-supported homework. Journal of Research on Technology in Education, 41(3), 331–359.Google Scholar
- Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Metacognitive practice makes perfect: Improving students’ self-assessment skills with an intelligent tutoring system. In G. Biswas, S. Bull. J. Kay, & T. Mitrovic (Eds.), Proceedings of the 15th international conference on artificial intelligence in education (pp. 288–295), Berlin: Springer.Google Scholar
- Segedy, J. R., Kinnebrew, J. S., & Biswas, G. (2011). Modeling learner’s cognitive and metacognitive strategies in an open-ended learning environment. Advances in cognitive systems: Papers from the AAAI Fall Symposium (pp. 297–304). Menlo Park, CA: AAAI Press.Google Scholar
- Segedy, J. R., Kinnebrew, J. S., & Biswas, G. (2012). Supporting student learning using conversational agents in a teachable agent environment. In J. van Aalst, K. Thompson, M. J. Jacobson, & P. Reimann (Eds.), The future of learning: Proceedings of the 10th international conference of the learning sciences (ICLS 2012): Vol. 2. Short Papers, Symposia, and Abstracts (pp. 251–255). International Society of the Learning Sciences: Sydney, NSW, Australia.Google Scholar
- VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.Google Scholar