The effect of contextualized conversational feedback in a complex open-ended learning environment

  • James R. SegedyEmail author
  • John S. Kinnebrew
  • Gautam Biswas
Research Article


Betty’s Brain is an open-ended learning environment in which students learn about science topics by teaching a virtual agent named Betty through the construction of a visual causal map that represents the relevant science phenomena. The task is complex, and success requires the use of metacognitive strategies that support knowledge acquisition, causal map construction, and progress monitoring. Previous research has established that middle school students struggle at such tasks without proper scaffolding and feedback. In Betty’s Brain, this feedback is provided by Betty and Mr. Davis, another virtual agent designed to provide guidance and suggestions as students work. This paper discusses our implementation of contextualized conversational (CC) feedback, and then presents the results of an experimental study exploring the effects of this feedback in two 8th-grade science classrooms. The results illustrate some advantages of the CC feedback in comparison with a baseline dialogue mechanism that presents similar strategies in a non-conversational, non-contextualized form. While both groups showed significant pre-to-post test learning gains, the difference in learning gains between the groups was not statistically significant. However, students who received CC feedback more often performed actions in accordance with the advised strategies, and they created higher quality causal maps.


Conversational agents Open-ended learning environment Metacognition Student learning behaviors Mixed-initiative dialogue 



This work has been supported by the National Science Foundation’s Information and Intelligent Systems Award #0904387.

Supplementary material

11423_2012_9275_MOESM1_ESM.pdf (473 kb)
Supplementary material 1 (PDF 473 kb)


  1. Adams, E. (2010). Fundamentals of game design (2nd ed.). Berkeley, CA: New Riders Pub.Google Scholar
  2. Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The role of self-regulated learning. Educational Psychologist, 40(4), 199–209. doi: 10.1207/s15326985ep4004_2.CrossRefGoogle Scholar
  3. Biswas, G., Leelawong, K., Schwartz, D. L., & Vye, N. (2005). Learning by teaching: Anew agent paradigm for educational software. Applied Artificial Intelligence, 19(3–4), 363–392. doi: 10.1080/08839510590910200.CrossRefGoogle Scholar
  4. Biswas, G., Jeong, H., Kinnebrew, J. S., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning, 5(2), 123–152. doi: 10.1142/S1793206810000839.CrossRefGoogle Scholar
  5. Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. In A. Iran-Nejad & P. D. Pearson (Eds.). Review of Research in Education, 24, 61–101. doi: 10.3102/0091732X024001061.Google Scholar
  6. Forbus, K. D. (1984). Qualitative process theory. Artificial Intelligence, 24(1–3), 85–168. doi: 10.1016/0004-3702(84)90038-9.CrossRefGoogle Scholar
  7. Gouli, E., Gogoulou, A., Papanikolaou, K., & Grigoriadou, M. (2004). Compass: An adaptive web-based concept map assessment tool. In A. J. Cañas, J. D. Novak, F. M. González (Eds.), Proceedings of the first international conference on concept mapping, Pamplona, Spain.Google Scholar
  8. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. doi: 10.3102/003465430298487.CrossRefGoogle Scholar
  9. Jeong, H., & Biswas, G. (2008). Mining student behavior models in learning-by-teaching environments. In R. S. J. d. Baker, T. Barnes, & J. E. Beck (Eds.), Educational data mining 2008: 1st international conference on educational data mining, proceedings (pp. 127–136). Montreal, Quebec, Canada.Google Scholar
  10. Jonassen, D. H., & Ionas, I. G. (2008). Designing effective supports for causal reasoning. Educational Technology Research and Development, 56(3), 287–308. doi: 10.1007/s11423-006-9021-6.CrossRefGoogle Scholar
  11. Kinnebrew, J. S., Loretz, K. M., & Biswas, G. (in press). A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining.Google Scholar
  12. Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264. doi: 10.1007/s10648-007-9049-0.CrossRefGoogle Scholar
  13. Land, S. (2000). Cognitive requirements for learning with open-ended learning environments. Educational Technology Research and Development, 48(3), 61–78. doi: 10.1007/BF02319858.CrossRefGoogle Scholar
  14. Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The Betty’s Brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.Google Scholar
  15. Luckie, D., Harrison, S. H., & Ebert-May, D. (2011). Model-based reasoning: Using visual tools to reveal student learning. Advances in Physiology Education, 35, 59–67. doi: 10.1152/advan.00016.2010.CrossRefGoogle Scholar
  16. Lukasenko, R., Anohina-Naumeca, A., Vilkelis, M., & Grundspenkis, J. (2010). Feedback in the concept map based intelligent knowledge assessment system. Scientific Journal of Riga Technical University, 41, 17–26. doi: 10.2478/v10143-010-0020-z.CrossRefGoogle Scholar
  17. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14–19. doi: 10.1037/0003-066X.59.1.14.CrossRefGoogle Scholar
  18. Mendicino, M., Razzaq, L., & Hefferman, N. T. (2009). A comparison of traditional homework to computer-supported homework. Journal of Research on Technology in Education, 41(3), 331–359.Google Scholar
  19. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286. IEEE Computer Society.CrossRefGoogle Scholar
  20. Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Metacognitive practice makes perfect: Improving students’ self-assessment skills with an intelligent tutoring system. In G. Biswas, S. Bull. J. Kay, & T. Mitrovic (Eds.), Proceedings of the 15th international conference on artificial intelligence in education (pp. 288–295), Berlin: Springer.Google Scholar
  21. Segedy, J. R., Kinnebrew, J. S., & Biswas, G. (2011). Modeling learner’s cognitive and metacognitive strategies in an open-ended learning environment. Advances in cognitive systems: Papers from the AAAI Fall Symposium (pp. 297–304). Menlo Park, CA: AAAI Press.Google Scholar
  22. Segedy, J. R., Kinnebrew, J. S., & Biswas, G. (2012). Supporting student learning using conversational agents in a teachable agent environment. In J. van Aalst, K. Thompson, M. J. Jacobson, & P. Reimann (Eds.), The future of learning: Proceedings of the 10th international conference of the learning sciences (ICLS 2012): Vol. 2. Short Papers, Symposia, and Abstracts (pp. 251–255). International Society of the Learning Sciences: Sydney, NSW, Australia.Google Scholar
  23. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. doi: 10.3102/0034654307313795.CrossRefGoogle Scholar
  24. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.Google Scholar

Copyright information

© Association for Educational Communications and Technology 2012

Authors and Affiliations

  • James R. Segedy
    • 1
    Email author
  • John S. Kinnebrew
    • 1
  • Gautam Biswas
    • 1
  1. 1.Institute for Software Integrated SystemsVanderbilt UniversityNashvilleUSA

Personalised recommendations