To Quit or Not to Quit: Predicting Future Behavioral Disengagement from Reading Patterns
This research predicted behavioral disengagement using quitting behaviors while learning from instructional texts. Supervised machine learning algorithms were used to predict if students would quit an upcoming text by analyzing reading behaviors observed in previous texts. Behavioral disengagement (quitting) at any point during the text was predicted with an accuracy of 76.5% (48% above chance), before students even began engaging with the text. We also predicted if a student would quit reading on the first page of a text or continue reading past the first page with an accuracy of 88.5% (29% above chance), as well as if students would quit sometime after the first page with an accuracy of 81.4% (51% greater than chance). Both actual quits and predicted quits were significantly related to learning, which provides some evidence for the predictive validity of our model. Implications and future work related to ITSs are also discussed.
Keywordsengagement disengagement affect detection reading ITSs
Unable to display preview. Download preview PDF.
- 1.Kelly, K.M., Heffernan, N., D’Mello, S., Namais, J., Strain, A.: Added Teacher-Created Motiational Video to an ITS. In: The Twenty-Sixth International FLAIRS Conference, pp. 503–508. AAAI Press, Menlo Park (2013)Google Scholar
- 3.Pekrun, R., Linnenbrink-Garcia, L.: Academic emotions and student engagement. In: Handbook of Research on Student Engagement, pp. 259–282. Springer (2012)Google Scholar
- 5.Beck, J.E.: Using response times to model student disengagement. In: Proceedings of the ITS 2004 Workshop on Social and Emotional Intelligence in Learning Environments, pp. 13–20 (2004)Google Scholar
- 6.D’Mello, S., Cobian, J., Hunter, M.: Automatic Gaze-Based Detection of Mind Wandering during Reading. In: Proceedings of the 6th International Conference on Educational Data Mining, pp. 364–365. International Educational Data Mining Society (2013)Google Scholar
- 8.Rowe, J.P., McQuiggan, S.W., Robison, J.L., Lester, J.C.: Off-Task Behavior in Narrative-Centered Learning Environments. In: AIED, pp. 99–106 (2009)Google Scholar
- 10.Baker, R.S.J.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1059–1068 (2007)Google Scholar
- 14.Baker, R.S.J., Rossi, L.M.: Assessing the Disengaged Behaviors of Learners. Des. Recomm. Intell. Tutoring Syst. 155 (2013)Google Scholar
- 19.Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A.C., Halpern, D.: Operation ARIES!: A serious game for teaching scientific inquiry. Serious Games Edutainment Appl., 169–195 (2011)Google Scholar
- 20.Rosenthal, R., Rosnow, R.L.: Essentials of behavioral analysis: Methods and data analysis. McGraw-Hill, New York (1984)Google Scholar
- 22.Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240 (2006)Google Scholar
- 24.Baker, R.S.J., De Carvalho, A.: Labeling student behavior faster and more precisely with text replays. In: Proceedings of the 1st International Conference on Educational Data Mining, pp. 38–47 (2008)Google Scholar