When Does Disengagement Correlate with Learning in Spoken Dialog Computer Tutoring?

  • Kate Forbes-Riley
  • Diane Litman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6738)


We investigate whether an overall student disengagement label and six different labels of disengagement type are predictive of learning in a spoken dialog computer tutoring corpus. Our results show first that although students’ percentage of overall disengaged turns negatively correlates with the amount they learn, the individual types of disengagement correlate differently with learning: some negatively correlate with learning, while others don’t correlate with learning at all. Second, we show that these relationships change somewhat depending on student prerequisite knowledge level. Third, we show that using multiple disengagement types to predict learning improves predictive power. Overall, our results suggest that although adapting to disengagement should improve learning, maximizing learning requires different system interventions depending on disengagement type.


types of disengagement learning correlations spoken dialog computer tutors manual annotation natural language processing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aleven, V., McLaren, B., Roll, I., Koedinger, K.: Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 227–239. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Merheranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing disengagement with non-invasive interventions. In: Proc. Artificial Intelligence in Education (AIED), pp. 195–202 (2007)Google Scholar
  3. 3.
    Baker, R.S., Corbett, A., Roll, I., Koedinger, K.: Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction (UMUAI) 18(3), 287–314 (2008)CrossRefGoogle Scholar
  4. 4.
    Beck, J.: Engagement tracking: using response times to model student disengagement. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED), Amsterdam, pp. 88–95 (2005)Google Scholar
  5. 5.
    Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction 19(3), 267–303 (2009)CrossRefGoogle Scholar
  6. 6.
    Forbes-Riley, K., Litman, D.: Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Communication (2011) (in press)Google Scholar
  7. 7.
    Forbes-Riley, K., Litman, D.: Annotating disengagement for spoken dialogue computer tutoring. In: D’Mello, S., Calvo, R. (eds.) Affect and Learning Technologies (to appear, 2011)Google Scholar
  8. 8.
    Jordan, P., Hall, B., Ringenberg, M., Cui, Y., Rose, C.: Tools for authoring a dialogue agent that participates in learning studies. In: Proc. Artificial Intelligence in Education (2007)Google Scholar
  9. 9.
    Lehman, B., Matthews, M., D’Mello, S., Person, N.: What are you feeling? Investigating student affective states during expert human tutoring sessions. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 50–59. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  10. 10.
    Murray, R.C.: vanLehn, K.: Effects of dissuading unnecessary help requests while providing proactive help. In: Proc. of the International Conference on Artificial Intelligence in Education, pp. 887–889 (2005)Google Scholar
  11. 11.
    Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Modeling and User-Adapted Interaction: The Journal of Personalization Research 18, 125–173 (2008)CrossRefGoogle Scholar
  12. 12.
    de Vicente, A., Pain, H.: Informing the detection of the students’ motivational state: An empirical study. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 933–943. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Walonoski, J., Heffernan, N.: Prevention of off-task gaming behavior in intelligent tutoring systems. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 722–724. Springer, Heidelberg (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Kate Forbes-Riley
    • 1
  • Diane Litman
    • 1
  1. 1.Learning R&D CtrUniversity of PittsburghPittsburghUSA

Personalised recommendations