Skip to main content

Toward Non-intrusive Assessment in Dialogue-Based Intelligent Tutoring Systems

Part of the Lecture Notes in Educational Technology book series (LNET)

Abstract

This paper describes a study whose goal was to assess students’ prior knowledge level with respect to a target domain based solely on characteristics of the natural language interaction between students and a state-of-the-art conversational ITS. We report results on data collected from two conversational ITSs: a micro-adaptive-only ITS and a fully adaptive (micro- and macro-adaptive) ITS. Our models rely on both dialogue and session interaction features including time-on-task, student-generated content features (e.g., vocabulary size or domain-specific concept use), and pedagogy-related features (e.g., level of scaffolding measured as number of hints). Linear regression models were explored based on these features in order to predict students’ knowledge level, as measured with a multiple-choice pre-test, and yielded in the best cases an r = 0.949 and adjusted r-square = 0.878.

Keywords

  • Intelligent tutoring systems
  • Knowledge assessment
  • Tutorial dialogues

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-981-287-868-7_26
  • Chapter length: 11 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   169.00
Price excludes VAT (USA)
  • ISBN: 978-981-287-868-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   219.99
Price excludes VAT (USA)
Hardcover Book
USD   219.99
Price excludes VAT (USA)

References

  1. Chi, M. T. H., Siler, S. A., & Jeong, H. (2004). Can tutors monitor students’ understanding accurately? Cognition and Instruction, 22(3), 363–387.

    Google Scholar 

  2. Chi, M. T. H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. G. (2001). Learning from human tutoring. Cognitive Science, 25, 471–533.

    CrossRef  Google Scholar 

  3. Corcoran, T., Mosher, F. A., & Rogat, A. (2009). Learning progressions in science: An evidence-based approach to reform. Consortium for Policy Research in Education Report #RR-63. Philadelphia, PA: Consortium for Policy Research in Education.

    Google Scholar 

  4. Forbes-Riley, K., & Litman, D. J. (2006). Modelling user satisfaction and student learning in a spoken dialogue tutoring system with generic, tutoring, and user affect parameters. In Proceedings of Human Language Technology Conference.

    Google Scholar 

  5. Lintean, M., Rus, V., & Azevedo, R. (2011). Automatic detection of student mental models during prior knowledge activation in metatutor. International Journal of Artificial Intelligence in Education, 21(3), 169–190.

    Google Scholar 

  6. Pennebaker, J. W., Francis, M. E., & Booth, R. J. (2001). Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates.

    Google Scholar 

  7. Romero, C., López, M. I., Luna, J. M., & Ventura, S. (2013). Predicting students’ final performance from participation in on-line discussion forums. Computers and Education, 68, 458–472.

    CrossRef  Google Scholar 

  8. Rus, V., D’Mello, S., Hu, X., & Graesser, A. C. (2013). Recent advances in conversational intelligent tutoring systems. AI Magazine, 34(3). Lang. Syst. 15, 5 (Nov. 1993), 795–825.

    Google Scholar 

  9. Rus, V., Stefanescu, D., Baggett, W., Niraula, N., Franceschetti, D., & Graesser, A. C. (2014). Macro-adaptation in Conversational Intelligent Tutoring Matters, The 12th International Conference on Intelligent Tutoring Systems, June 5–9, Honolulu, Hawaii.

    Google Scholar 

  10. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.

    Google Scholar 

  11. Shute, V. J., & Ventura, M. (2013). Measuring and Supporting Learning in Games: Stealth Assessment. Cambridge, MA: The MIT Press.

    Google Scholar 

  12. Taraban, R., & Rynearson, K. (1998). Computer-based comprehension research in a content area. Journal of Developmental Education, 21, 10–18.

    Google Scholar 

  13. VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62.

    CrossRef  Google Scholar 

  14. Williams, C., & D’Mello, S. (2010). Predicting student knowledge level from domain-independent function and content words. In Intelligent Tutoring Systems (pp. 62–71). Berlin: Springer.

    Google Scholar 

  15. Woolf, B. (2008). Building Intelligent Interactive Tutors, Student-Centered Strategies for Revolutionizing E-Learning. Elsevier & Morgan Kaufmann Publishers.

    Google Scholar 

  16. Yoo, J. -B., & Kim, J. (2012). Predicting learner’s project performance with dialogue features in online Q&A discussions. In Intelligent Tutoring Systems (pp. 570–575).

    Google Scholar 

Download references

Acknowledgments

This research was supported by the Institute for Education Sciences (IESs) under award R305A100875 to Dr. Vasile Rus. All opinions and findings presented here are solely the authors’.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Vasile Rus .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer Science+Business Media Singapore

About this paper

Cite this paper

Rus, V., Stefanescu, D. (2016). Toward Non-intrusive Assessment in Dialogue-Based Intelligent Tutoring Systems. In: , et al. State-of-the-Art and Future Directions of Smart Learning. Lecture Notes in Educational Technology. Springer, Singapore. https://doi.org/10.1007/978-981-287-868-7_26

Download citation

  • DOI: https://doi.org/10.1007/978-981-287-868-7_26

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-287-866-3

  • Online ISBN: 978-981-287-868-7

  • eBook Packages: EducationEducation (R0)