Towards Improving (Meta)cognition by Adapting to Student Uncertainty in Tutorial Dialogue

Part of the Springer International Handbooks of Education book series (SIHE, volume 28)


We hypothesize that enhancing computer tutors to respond to student uncertainty over and above correctness is one method for increasing both student learning and self-monitoring abilities. We test this hypothesis using spoken data from both wizarded and fully-automated versions of a spoken tutorial dialogue system, where tutor responses to uncertain and/or incorrect student answers were manipulated. Although we find no significant improvement in metacognitive metrics (computed using speech and language information) when responding to uncertainty and incorrectness as compared to when responding only to incorrectness, we find that some metacognitive metrics significantly correlate with student learning. Our results suggest that monitoring and responding to student uncertainty has the potential to improve both cognitive and metacognitive student abilities.



This work is supported by NSF Award 0631930. A preliminary version of the wizarded results was presented at the AAAI Symposium on Cognitive and Metacognitive Educational Systems (Litman & Forbes-Riley, 2009a), while a subset of our automated results (the correlations) were previously published at ITS 2010 (Forbes-Riley & Litman, 2010).


  1. Aist, G., Kort, B., Reilly, R., Mostow, J., & Picard, R. (2002). Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: Adding human-provided emotional scaffolding to an automated reading tutor that listens. In Proceedings of Intelligent Tutoring Systems Workshop on Empirical Methods for Tutorial Dialogue Systems (pp. 483–490), San Sebastian, Spain.Google Scholar
  2. Aleven, V., & Roll, I. (Eds.) (2007). AIED workshop on metacognition and self-regulated learning in intelligent tutoring systems.Google Scholar
  3. Brennan, S.E., & Williams, M. (1995). The feeling of another’s knowing: Prosody and filled pauses as cues to listeners about the metacognitive states of speakers. Journal of Memory and Language,34, 383–398.Google Scholar
  4. Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media,29(3), 241–250.Google Scholar
  5. Dijkstra, C., Krahmer, E. J., & Swerts, M. (2006). Mani­pulating uncertainty: The contribution of different audio­visual prosodic cues to the perception of confidence. In R. Hoffmann & H. Mixdorff (Eds.), Proceed­ings of Speech Prosody 2006, Dresden: TUDpress.Google Scholar
  6. Forbes-Riley, K., & Litman, D. J. (2008). Analyzing dependencies between student certainness states and tutor responses in a spoken dialogue corpus. In L. Dybkjaer & W. Minker (Eds.), Recent Trends in Discourse and Dialogue. Berlin: Springer.Google Scholar
  7. Forbes-Riley, K., & Litman, D. (2009a). Adapting to student uncertainty improves tutoring dialogues. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, AIED 2009, July 6–10, 2009, Brighton, UK. Frontiers in Artificial Intelligence and Applications 200 IOS Press 2009, ISBN 978-1-60750-028-5.Google Scholar
  8. Forbes-Riley, K., & Litman, D. (2009b). A user modeling-based performance analysis of a wizarded uncertainty-adaptive dialogue system corpus. In Proceedings of the 10th Annual Conference of the International Speech Communication Association, Brighton, UK.Google Scholar
  9. Forbes-Riley, K., & Litman, D. (2010). Metacognition and learning in spoken dialogue computer tutoring. In Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS), Pittsburgh, PA.Google Scholar
  10. Forbes-Riley, K., & Litman, D. (2011a). Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Communication, 53(9–10), 1115–1136.Google Scholar
  11. Forbes-Riley, K., & Litman, D. (2011b). Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Computer Speech and Language, 25(1), 105–126.Google Scholar
  12. Forbes-Riley, K., Litman, D., & Rotaru, M. (2008a). Responding to student uncertainty during computer tutoring: A preliminary evaluation. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS), Montreal, Canada, June.Google Scholar
  13. Forbes-Riley, K., Rotaru, M., & Litman, D. (2008b). The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Modeling and User-Adapted Interaction,18(1–2), 11–43Google Scholar
  14. Huang, X. D., Alleva, F., Hon, H. W., Hwang, M. Y., Lee, K. F.,& Rosenfeld, R. (1993). The SPHINX-II speech recognition system: An overview. Computer Speech and Language,2, 137–148.Google Scholar
  15. Jordan, P., Hall, B., Ringenberg, M., Cui, Y., and Rosé, C. (2007). Tools for authoring a dialogue agent that participates in learning studies. In Artificial Intelligence in Education (AIED), pp. 43–50.Google Scholar
  16. Kelemen, W. L., Frost, P. J., & Weaver, C. A. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. Memory and Cognition,28, 92–107.Google Scholar
  17. Liscombe, J., Venditti, J., & Hirschberg, J. (2005, September). Detecting certainness in spoken tutorial dialogues. In Proceedings of Interspeech/Eurospeech Conference on Speech Communication and Techno­logy, Lisbon, Portugal.Google Scholar
  18. Litman, D., & Forbes-Riley, K. (2009a). Improving (meta)cognitive tutoring by detecting and responding to uncertainty. Technical Report FS-09-0:Cognitive and Metacognitive Educational Systems: Papers from the AAAI Symposium, AAAI Arlington, VA, November.Google Scholar
  19. Litman, D., & Forbes-Riley, K. (2009b). Spoken tutorial dialogue and the feeling of another’s knowing. In Proceedings of the 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), London.Google Scholar
  20. Litman, D., Moore, J., Dzikovska, M., & Farrow, E. (2009, July). Using natural language processing to analyze tutorial dialogue corpora across domains and modalities. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, AIED 2009, Brighton, UK. Frontiers in Artificial Intelligence and Applications 200 IOS Press 2009, ISBN 978-1-60750-028-5.Google Scholar
  21. Nicholas, G., Rotaru, M., & Litman, D. J. (2006). Exploiting word-level features for emotion prediction. In Proceedings of IEEE/ACL Workshop on Spoken Language Technology, Aruba.Google Scholar
  22. Nietfeld, J. L., Enders, C. K., & Schraw, G. (2006). A Monte Carlo comparison of measures of relative and absolute monitoring accuracy. Educational and Psychological Measurement,66, 258–271.Google Scholar
  23. Pon-Barry, H. (2008). Prosodic manifestations of confidence and uncertainty in spoken language. In Proceedings of the 9th Annual Conference of the International Speech Communication Association, September 2008 (pp. 74–77). Brisbane, Australia.Google Scholar
  24. Pon-Barry, H., Schultz, K., Bratt, E. O., Clark, B., & Peters, S. (2006). Responding to student uncertainty in spoken tutorial dialogue systems. International Journal of Artificial Intelligence in Education,16, 171–194.Google Scholar
  25. Roll, I., & Aleven, V., (Eds.) (2008). ITS Workshop on meta-cognition and self-regulated rearning in educational technologies.Google Scholar
  26. Rotaru, M., & Litman, D. J. (2009). Discourse structure and performance analysis: Beyond the correlation. In Proceedings of the 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), London.Google Scholar
  27. Saadawi, G. M. E., Azevedo, R., Castine, M., Payne, V., Medvedeva, O., Tseytlin, E., Legowski, E., azen Jukic, D., & Crowley, R. S. (2009). Factors affecting feeling-of-knowing in a medical intelligent tutoring system: The role of immediate feedback as a metacognitive scaffold. Advances in Health Sciences Education,15, 9–30.Google Scholar
  28. Smith, V. L. and Clark, H. H. (1993). On the course of answering questions. Journal of Memory and Language,32, 25–38.Google Scholar
  29. Tsukahara, W., & Ward, N. (2001). Responding to subtle, fleeting changes in the user’s internal state. In Proceedings of SIG-CHI on Human Factors in Computing Systems (pp.77–84).Google Scholar
  30. VanLehn, K., Jordan, P. W., Rosé, C., Bhembe, D., Böttner, M., Gaydos, A., Makatchev, M., Pappuswamy, U., Ringenberg, M., Roque, A., Siler, S., Srivastava, R., & Wilson, R. (2002). The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In Proceedings of the International Conference on Intelligent Tutoring Systems (pp.158–167).Google Scholar
  31. VanLehn, K., Siler, S., & Murray, C. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction,21(3), 209–249.Google Scholar
  32. Witten, I. H., & Frank, E. (1999). Data Mining: Practical Machine Learning Tools and Techniques, I. H. Witten & E. Frank and Mark Hall, January 2011, Morgan Kaufmann Publishers (ISBN: 978-0-12-374856-0).Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Learning Research and Development CenterUniversity of PittsburghPittsburghUSA

Personalised recommendations