To Quit or Not to Quit: Predicting Future Behavioral Disengagement from Reading Patterns

  • Caitlin Mills
  • Nigel Bosch
  • Art Graesser
  • Sidney D’Mello
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8474)


This research predicted behavioral disengagement using quitting behaviors while learning from instructional texts. Supervised machine learning algorithms were used to predict if students would quit an upcoming text by analyzing reading behaviors observed in previous texts. Behavioral disengagement (quitting) at any point during the text was predicted with an accuracy of 76.5% (48% above chance), before students even began engaging with the text. We also predicted if a student would quit reading on the first page of a text or continue reading past the first page with an accuracy of 88.5% (29% above chance), as well as if students would quit sometime after the first page with an accuracy of 81.4% (51% greater than chance). Both actual quits and predicted quits were significantly related to learning, which provides some evidence for the predictive validity of our model. Implications and future work related to ITSs are also discussed.


engagement disengagement affect detection reading ITSs 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kelly, K.M., Heffernan, N., D’Mello, S., Namais, J., Strain, A.: Added Teacher-Created Motiational Video to an ITS. In: The Twenty-Sixth International FLAIRS Conference, pp. 503–508. AAAI Press, Menlo Park (2013)Google Scholar
  2. 2.
    Calvo, R.A., D’Mello, S.: Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans. on Affect. Comput. 1, 18–37 (2010)CrossRefGoogle Scholar
  3. 3.
    Pekrun, R., Linnenbrink-Garcia, L.: Academic emotions and student engagement. In: Handbook of Research on Student Engagement, pp. 259–282. Springer (2012)Google Scholar
  4. 4.
    Baker, R.S., Corbett, A.T., Koedinger, K.R.: Detecting student misuse of intelligent tutoring systems. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 531–540. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  5. 5.
    Beck, J.E.: Using response times to model student disengagement. In: Proceedings of the ITS 2004 Workshop on Social and Emotional Intelligence in Learning Environments, pp. 13–20 (2004)Google Scholar
  6. 6.
    D’Mello, S., Cobian, J., Hunter, M.: Automatic Gaze-Based Detection of Mind Wandering during Reading. In: Proceedings of the 6th International Conference on Educational Data Mining, pp. 364–365. International Educational Data Mining Society (2013)Google Scholar
  7. 7.
    Forbes-Riley, K., Litman, D.: When does disengagement correlate with learning in spoken dialog computer tutoring? In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 81–89. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  8. 8.
    Rowe, J.P., McQuiggan, S.W., Robison, J.L., Lester, J.C.: Off-Task Behavior in Narrative-Centered Learning Environments. In: AIED, pp. 99–106 (2009)Google Scholar
  9. 9.
    Jang, H.: Supporting students’ motivation, engagement, and learning during an uninteresting activity. UMAP 2012 100, 798 (2008)CrossRefGoogle Scholar
  10. 10.
    Baker, R.S.J.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1059–1068 (2007)Google Scholar
  11. 11.
    Baker, R.S.J.d., et al.: Adapting to when students game an intelligent tutoring system. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 392–401. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  12. 12.
    Cocea, M., Weibelzahl, S.: Eliciting motivation knowledge from log files towards motivation diagnosis for Adaptive Systems. In: Conati, C., McCoy, K., Paliouras, G. (eds.) UM 2007. LNCS (LNAI), vol. 4511, pp. 197–206. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  13. 13.
    Cocea, M., Weibelzahl, S.: Disengagement Detection in Online Learning: Validation Studies and Perspectives. IEEE Trans. Learn. Technol. 4, 114–124 (2011)CrossRefGoogle Scholar
  14. 14.
    Baker, R.S.J., Rossi, L.M.: Assessing the Disengaged Behaviors of Learners. Des. Recomm. Intell. Tutoring Syst. 155 (2013)Google Scholar
  15. 15.
    Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: Potential of the concept, state of the evidence. Rev. Educ. Res. 74, 59–109 (2004)CrossRefGoogle Scholar
  16. 16.
    D’Mello, S., Graesser, A.C.: The half-life of cognitive-affective states during complex learning. Cogn. Emot. 25, 1299–1308 (2011)CrossRefGoogle Scholar
  17. 17.
    Brusilovsky, P., Schwarz, E., Weber, G.: ELM-ART: An intelligent tutoring system on World Wide Web. In: Lesgold, A.M., Frasson, C., Gauthier, G. (eds.) ITS 1996. LNCS, vol. 1086, pp. 261–269. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  18. 18.
    McNamara, D.S., Levinstein, I.B., Boonthum, C.: iSTART: Interactive strategy training for active reading and thinking. Behav. Res. Methods Instrum. Comput. 36, 222–233 (2004)CrossRefGoogle Scholar
  19. 19.
    Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A.C., Halpern, D.: Operation ARIES!: A serious game for teaching scientific inquiry. Serious Games Edutainment Appl., 169–195 (2011)Google Scholar
  20. 20.
    Rosenthal, R., Rosnow, R.L.: Essentials of behavioral analysis: Methods and data analysis. McGraw-Hill, New York (1984)Google Scholar
  21. 21.
    Graesser, A.C., Person, N.K.: Question asking during tutoring. Am. Educ. Res. J. 31, 104–137 (1994)CrossRefGoogle Scholar
  22. 22.
    Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240 (2006)Google Scholar
  23. 23.
    Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)CrossRefGoogle Scholar
  24. 24.
    Baker, R.S.J., De Carvalho, A.: Labeling student behavior faster and more precisely with text replays. In: Proceedings of the 1st International Conference on Educational Data Mining, pp. 38–47 (2008)Google Scholar
  25. 25.
    Jang, H.: Supporting students’ motivation, engagement, and learning during an uninteresting activity. J. Educ. Psychol. 100, 798 (2008)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Caitlin Mills
    • 1
  • Nigel Bosch
    • 1
  • Art Graesser
    • 2
  • Sidney D’Mello
    • 1
  1. 1.Departments of Psychology and Computer ScienceUniversity of Notre DameNotre DameUSA
  2. 2.Department of Psychology and Institute for Intelligent SystemsUniversity of MemphisMemphisUSA

Personalised recommendations