Toward Non-intrusive Assessment in Dialogue-Based Intelligent Tutoring Systems

  • Vasile Rus
  • Dan Stefanescu
Conference paper
Part of the Lecture Notes in Educational Technology book series (LNET)


This paper describes a study whose goal was to assess students’ prior knowledge level with respect to a target domain based solely on characteristics of the natural language interaction between students and a state-of-the-art conversational ITS. We report results on data collected from two conversational ITSs: a micro-adaptive-only ITS and a fully adaptive (micro- and macro-adaptive) ITS. Our models rely on both dialogue and session interaction features including time-on-task, student-generated content features (e.g., vocabulary size or domain-specific concept use), and pedagogy-related features (e.g., level of scaffolding measured as number of hints). Linear regression models were explored based on these features in order to predict students’ knowledge level, as measured with a multiple-choice pre-test, and yielded in the best cases an r = 0.949 and adjusted r-square = 0.878.


Intelligent tutoring systems Knowledge assessment Tutorial dialogues 



This research was supported by the Institute for Education Sciences (IESs) under award R305A100875 to Dr. Vasile Rus. All opinions and findings presented here are solely the authors’.


  1. 1.
    Chi, M. T. H., Siler, S. A., & Jeong, H. (2004). Can tutors monitor students’ understanding accurately? Cognition and Instruction, 22(3), 363–387.Google Scholar
  2. 2.
    Chi, M. T. H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. G. (2001). Learning from human tutoring. Cognitive Science, 25, 471–533.CrossRefGoogle Scholar
  3. 3.
    Corcoran, T., Mosher, F. A., & Rogat, A. (2009). Learning progressions in science: An evidence-based approach to reform. Consortium for Policy Research in Education Report #RR-63. Philadelphia, PA: Consortium for Policy Research in Education.Google Scholar
  4. 4.
    Forbes-Riley, K., & Litman, D. J. (2006). Modelling user satisfaction and student learning in a spoken dialogue tutoring system with generic, tutoring, and user affect parameters. In Proceedings of Human Language Technology Conference.Google Scholar
  5. 5.
    Lintean, M., Rus, V., & Azevedo, R. (2011). Automatic detection of student mental models during prior knowledge activation in metatutor. International Journal of Artificial Intelligence in Education, 21(3), 169–190.Google Scholar
  6. 6.
    Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates.Google Scholar
  7. 7.
    Romero, C., López, M. I., Luna, J. M., & Ventura, S. (2013). Predicting students’ final performance from participation in on-line discussion forums. Computers and Education, 68, 458–472.CrossRefGoogle Scholar
  8. 8.
    Rus, V., D’Mello, S., Hu, X., & Graesser, A. C. (2013). Recent advances in conversational intelligent tutoring systems. AI Magazine, 34(3). Lang. Syst. 15, 5 (Nov. 1993), 795–825.Google Scholar
  9. 9.
    Rus, V., Stefanescu, D., Baggett, W., Niraula, N., Franceschetti, D., & Graesser, A. C. (2014). Macro-adaptation in Conversational Intelligent Tutoring Matters, The 12th International Conference on Intelligent Tutoring Systems, June 5–9, Honolulu, Hawaii.Google Scholar
  10. 10.
    Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.Google Scholar
  11. 11.
    Shute, V. J., & Ventura, M. (2013). Measuring and Supporting Learning in Games: Stealth Assessment. Cambridge, MA: The MIT Press.Google Scholar
  12. 12.
    Taraban, R., & Rynearson, K. (1998). Computer-based comprehension research in a content area. Journal of Developmental Education, 21, 10–18.Google Scholar
  13. 13.
    VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62.CrossRefGoogle Scholar
  14. 14.
    Williams, C., & D’Mello, S. (2010). Predicting student knowledge level from domain-independent function and content words. In Intelligent Tutoring Systems (pp. 62–71). Berlin: Springer.Google Scholar
  15. 15.
    Woolf, B. (2008). Building Intelligent Interactive Tutors, Student-Centered Strategies for Revolutionizing E-Learning. Elsevier & Morgan Kaufmann Publishers.Google Scholar
  16. 16.
    Yoo, J. -B., & Kim, J. (2012). Predicting learner’s project performance with dialogue features in online Q&A discussions. In Intelligent Tutoring Systems (pp. 570–575).Google Scholar

Copyright information

© Springer Science+Business Media Singapore 2016

Authors and Affiliations

  1. 1.Department of Computer Science, Institute for Intelligent SystemsThe University of MemphisMemphisUSA

Personalised recommendations