Face Forward: Detecting Mind Wandering from Video During Narrative Film Comprehension

  • Angela StewartEmail author
  • Nigel Bosch
  • Huili Chen
  • Patrick Donnelly
  • Sidney D’MelloEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10331)


Attention is key to effective learning, but mind wandering, a phenomenon in which attention shifts from task-related processing to task-unrelated thoughts, is pervasive across learning tasks. Therefore, intelligent learning environments should benefit from mechanisms to detect and respond to attentional lapses, such as mind wandering. As a step in this direction, we report the development and validation of the first student-independent facial feature-based mind wandering detector. We collected training data in a lab study where participants self-reported when they caught themselves mind wandering over the course of completing a 32.5 min narrative film comprehension task. We used computer vision techniques to extract facial features and bodily movements from videos. Using supervised learning methods, we were able to detect a mind wandering with an F1 score of .390, which reflected a 31% improvement over a chance model. We discuss how our mind wandering detector can be used to adapt the learning experience, particularly for online learning contexts.


Mind wandering Attention aware interfaces 



This research was supported by the National Science Foundation (NSF) (DRL 1235958 and IIS 1523091). Any opinions, findings and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the NSF.


  1. 1.
    D’Mello, S.K.: Giving eyesight to the blind: towards attention-aware AIED. Int. J. Artif. Intell. Educ. 26(2), 645–659 (2016)CrossRefGoogle Scholar
  2. 2.
    Killingsworth, M.A., Gilbert, D.T.: A wandering mind is an unhappy mind. Science 330, 932 (2010)CrossRefGoogle Scholar
  3. 3.
    Bixler, R., D’Mello, S.: Toward fully automated person-independent detection of mind wandering. In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, G.-J. (eds.) UMAP 2014. LNCS, vol. 8538, pp. 37–48. Springer, Cham (2014). doi: 10.1007/978-3-319-08786-3_4 Google Scholar
  4. 4.
    Kopp, K., Mills, C., D’Mello, S.K.: Mind wandering during film comprehension: the role of prior knowledge and situational interest. Psychon. Bull. Rev. 23, 842–848 (2015)CrossRefGoogle Scholar
  5. 5.
    Franklin, M.S., Smallwood, J., Schooler, J.W.: Catching the mind in flight: using behavioral indices to detect mindless reading in real time. Psychon. Bull. Rev. 18, 992–997 (2011)CrossRefGoogle Scholar
  6. 6.
    Szpunar, K.K., Moulton, S.T., Schacter, D.L.: Mind wandering and education: from the classroom to online learning. Front. Psychol. 4, 495 (2013)CrossRefGoogle Scholar
  7. 7.
    Randall, J.G., Oswald, F.L., Beier, M.E.: Mind-wandering, cognition, and performance: a theory-driven meta-analysis of attention regulation. Psychol. Bull. 140, 1411 (2014)CrossRefGoogle Scholar
  8. 8.
    Bixler, R., D’Mello, S.: Automatic gaze-based detection of mind wandering with metacognitive awareness. In: Ricci, F., Bontcheva, K., Conlan, O., Lawless, S. (eds.) UMAP 2015. LNCS, vol. 9146, pp. 31–43. Springer, Cham (2015). doi: 10.1007/978-3-319-20267-9_3 CrossRefGoogle Scholar
  9. 9.
    Blanchard, N., Bixler, R., Joyce, T., D’Mello, S.: Automated physiological-based detection of mind wandering during learning. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 55–60. Springer, Cham (2014). doi: 10.1007/978-3-319-07221-0_7 CrossRefGoogle Scholar
  10. 10.
    Pham, P., Wang, J.: AttentiveLearner: improving mobile MOOC learning via implicit heart rate tracking. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS, vol. 9112, pp. 367–376. Springer, Cham (2015). doi: 10.1007/978-3-319-19773-9_37 CrossRefGoogle Scholar
  11. 11.
    Mills, C., Bixler, R., Wang, X., D’Mello, S.K.: Automatic gaze-based detection of mind wandering during film viewing. In: Proceedings of the 9th International Conference on Educational Data Mining. International Educational Data Mining Society, Raleigh (2016)Google Scholar
  12. 12.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)CrossRefGoogle Scholar
  13. 13.
    Faber, M., Bixler, R., D’Mello, S.K.: An automated behavioral measure of mind wandering during computerized reading. Behav. Res. Methods 1–17 (2017)Google Scholar
  14. 14.
    Reichle, E.D., Reineberg, A.E., Schooler, J.W.: Eye movements during mindless reading. Psychol. Sci. 21, 1300–1310 (2010)CrossRefGoogle Scholar
  15. 15.
    Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (CERT). In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 298–305. IEEE (2011)Google Scholar
  16. 16.
    Ekman, P., Friesen, W.V.: Facial action coding system (1977)Google Scholar
  17. 17.
    Westlund, J.K., D’Mello, S.K., Olney, A.M.: Motion tracker: camera-based monitoring of bodily movements using motion silhouettes. PLoS ONE 10, e0130293 (2015)CrossRefGoogle Scholar
  18. 18.
    Dixon, W.J., Yuen, K.K.: Trimming and winsorization: a review. Statistische Hefte 15, 157–170 (1974)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Allison, P.D.: Multiple Regression: A Primer. Pine Forge Press, Thousand Oaks (1999)Google Scholar
  20. 20.
    Kononenko, I.: Estimating attributes: analysis and extensions of RELIEF. In: Bergadano, F., Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 171–182. Springer, Heidelberg (1994). doi: 10.1007/3-540-57868-4_57 CrossRefGoogle Scholar
  21. 21.
    Holmes, G., Donkin, A., Witten, I.H.: Weka: a machine learning workbench. In: Proceedings of the 1994 Second Australian and New Zealand Conference on Intelligent Information Systems, pp. 357–361. IEEE (1994)Google Scholar
  22. 22.
    Platt, J., et al.: Sequential minimal optimization: a fast algorithm for training support vector machines (1998)Google Scholar
  23. 23.
    Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)zbMATHGoogle Scholar
  24. 24.
    Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Erlbaum, Hillsdale (1988)zbMATHGoogle Scholar
  25. 25.
    Szpunar, K.K., Khan, N.Y., Schacter, D.L.: Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proc. Natl. Acad. Sci. 110, 6313–6317 (2013)CrossRefGoogle Scholar
  26. 26.
    Moss, J., Schunn, C.D., Schneider, W., McNamara, D.S.: The nature of mind wandering during reading varies with the cognitive control demands of the reading strategy. Brain Res. 1539, 48–60 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of Notre DameNotre DameUSA
  2. 2.University of Illinois at Urbana-ChampaignUrbanaUSA
  3. 3.Massachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations