Advertisement

User Modeling and User-Adapted Interaction

, Volume 26, Issue 1, pp 33–68 | Cite as

Automatic gaze-based user-independent detection of mind wandering during computerized reading

  • Robert BixlerEmail author
  • Sidney D’Mello
Article

Abstract

Mind wandering is a ubiquitous phenomenon where attention involuntarily shifts from task-related thoughts to internal task-unrelated thoughts. Mind wandering can have negative effects on performance; hence, intelligent interfaces that detect mind wandering can improve performance by intervening and restoring attention to the current task. We investigated the use of eye gaze and contextual cues to automatically detect mind wandering during reading with a computer interface. Participants were pseudorandomly probed to report mind wandering while an eye tracker recorded their gaze during the reading task. Supervised machine learning techniques detected positive responses to mind wandering probes from eye gaze and context features in a user-independent fashion. Mind wandering was detected with an accuracy of 72 % (expected accuracy by chance was 60 %) when probed at the end of a page and an accuracy of 67 % (chance was 59 %) when probed in the midst of reading a page. Global gaze features (gaze patterns independent of content, such as fixation durations) were more effective than content-specific local gaze features. An analysis of the features revealed diagnostic patterns of eye gaze behavior during mind wandering: (1) certain types of fixations were longer; (2) reading times were longer than expected; (3) more words were skipped; and (4) there was a larger variability in pupil diameter. Finally, the automatically detected mind wandering rate correlated negatively with measures of learning and transfer even after controlling for prior knowledge, thereby providing evidence of predictive validity. Possible improvements to the detector and applications that utilize the detector are discussed.

Keywords

Gaze tracking Mind wandering User modeling 

Notes

Acknowledgments

We would first like to thank our collaborators at the University of Memphis for assistance with data collection. We also thank Kris Kopp, Caitlin Mills, Nigel Bosch, Jennifer Neale, Jacqueline Kory, Jonathan Cobian, and Matthew Hunter for help with data collection and analysis. The authors would also like to thank the individuals who reviewed the initial draft of this paper prior to publication. This research was supported by the National Science Foundation (NSF) (ITR 0325428, HCC 0834847, DRL 1235958).

References

  1. Baayen, R., Harald, P., Richard, G.: The CELEX Lexical Database (Release 2) (1995)Google Scholar
  2. Baer, R.A.: Mindfulness training as a clinical intervention: a conceptual and empirical review. Clin. Psychol. 10(2), 125–143 (2003)Google Scholar
  3. Barbuceanu, F., Antonya, C., Duguleana, M., et al.: Attentive User Interface for Interaction Within Virtual Reality Environments Based on Gaze Analysis. Human-Computer Interaction, pp. 204–213. Springer, Interaction Techniques and Environments, New York (2011)Google Scholar
  4. Ba, Sileye O., Odobez, J-M.: A Study on Visual Focus of Attention Recognition from Head Pose in a Meeting Room, pp. 75–87. Springer, In Machine Learning for Multimodal Interaction, New York (2006)Google Scholar
  5. Bergasa, L.M., Nuevo, J., Sotelo, M.A., et al.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transp. Sys. 7(1), 63–77 (2006)CrossRefGoogle Scholar
  6. Bixler, R., D’Mello, S.: Toward fully automated person-independent detection of mind wandering. In: User Modeling, Adaptation, and Personalization, pp. 37–48. Springer, New York (2014)Google Scholar
  7. Bixler, R., Kopp, K., D’Mello, S.: Evaluation of a Personalized Method for Proactive Mind Wandering Reduction. In: Proceedings of the 4th Workshop on Personalization Approaches for Learning Environments (PALE 2014), pp. 33, (2014)Google Scholar
  8. Blanchard, N., Bixler, R., Joyce, T., et al.: Automated physiological-based detection of mind wandering during learning. In: Intelligent Tutoring Systems, pp. 55–60. Springer, New York (2014)Google Scholar
  9. Börner, D., Kalz, M., Specht, M.: Lead me gently: facilitating knowledge gain through attention-aware ambient learning displays. Comput. & Educ. 78, 10–19 (2014)CrossRefGoogle Scholar
  10. Chawla, N.V., Bowyer, K. W., Hall, L.O., et al.: SMOTE: Synthetic Minority Over-Sampling Technique.’ J. Artif. Intell. Res. 16(1), 321–357 (2011)Google Scholar
  11. Christoff, K., Gordon, A.M., Smallwood, J., et al.: experience Sampling during fMRI reveals default network and executive system contributions to mind wandering. Proc. Natl. Acad. Sci. 106(21), 8719–8724 (2009)CrossRefGoogle Scholar
  12. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960)CrossRefGoogle Scholar
  13. Davidson, R.J.: Empirical explorations of mindfulness: conceptual and methodological conundrums. Emotion 10(1), 8–11 (2010)CrossRefGoogle Scholar
  14. D’Mello, S., Cobian, J., Hunter, M.: Automatic gaze-based detection of mind wandering during reading. Proceedings of the 6th International Conference on Educational Data Mining, International Educational Data Mining Society, pp. 364–365 (2013)Google Scholar
  15. Dong, L., Di, H., Tao, L., et al.: Visual Focus of Attention Recognition in the Ambient Kitchen. In: Computer Vision–ACCV 2009, pp. 548–559. Springer, New York (2010)Google Scholar
  16. Dong, Y., Hu, Z., Uchimura, K., et al.: Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans. Intell. Transp. Syst. 12(2), 596–614 (2011)CrossRefGoogle Scholar
  17. D’Orazio, T., Leo, M., Guaragnella, C., et al.: A visual approach for driver inattention detection. Pattern Recognit. 40(8), 2341–2355 (2007)CrossRefzbMATHGoogle Scholar
  18. Drummond, J., Litman, D.: In the Zone: towards Detecting Student Zoning Out Using Supervised Machine Learning, pp. 306–308. Springer, New York, In Intelligent Tutoring Systems (2010)Google Scholar
  19. Feng, S., D’Mello, S., Graesser, A.C.: Mind wandering while reading easy and difficult texts. Psychon. Bull. & Rev. 20(3), 586–592 (2013)CrossRefGoogle Scholar
  20. Fletcher, L., Zelinsky, A.: Driver inattention detection based on eye gaze-road event correlation. Int. J. Robot. Res. 28(6), 774–801 (2009)CrossRefGoogle Scholar
  21. Forster, S., Lavie, N.: Distracted by your mind? Individual differences in distractibility predict mind wandering. J. Exp. Psychol. 40(1), 251–260 (2014)Google Scholar
  22. Foulsham, T., Farley, J., Kingstone, A.: Mind wandering in sentence reading: decoupling the link between mind and eye. Can. J. Exp. Psychol./Rev. Can de Psychol Exp 67(1), 51–59 (2013)Google Scholar
  23. Frank, D.J., Nara, B., Zavagnin, M., et al.: Validating older adults’ reports of less mind-wandering: an examination of eye movements and dispositional influences. Psychol. Aging 30(2), 266–278 (2015)CrossRefGoogle Scholar
  24. Franklin, M.S., Broadway, J.M., Mrazek, M.D., et al.: Window to the wandering mind: pupillometry of spontaneous thought while reading. Q. J. Exp. Psychol. 66(12), 2289–2294 (2013)CrossRefGoogle Scholar
  25. Franklin, M.S., Smallwood, J., Schooler, J.W.: Catching the mind in flight: using behavioral indices to detect mindless reading in real time. Psychon. Bull. & Rev. 18(5), 992–997 (2011)CrossRefGoogle Scholar
  26. Furugori, S., Yoshizawa, N., Iname, C., et al.: Estimation of driver fatigue by pressure distribution on seat in long term driving. Rev. Automot. Eng. 26(1), 053–058 (2005)Google Scholar
  27. Grandchamp, R., Braboszcz, C., Delorme, A.: Oculometric variations during mind wandering. Front. Psychol. 5, 1000–1078 (2014)CrossRefGoogle Scholar
  28. Gruberger, M., Ben-Simon, E., Levkovitz, Y., et al.: Towards a neuroscience of mind-wandering. Front. Hum. Neurosci. 5, 56–60 (2011)CrossRefGoogle Scholar
  29. Hall, M.A.: Correlation-based feature selection for machine learning. PhD Thesis, department of Computer Science, University of Waikato, New Zealand. (1999)Google Scholar
  30. Hall, M., Frank, E., Holmes, G., et al.: The WEKA data mining software: an update. ACM SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  31. Halpern, D.F., Millis, K., Graesser, A.C., et al.: Operation ARA: a computerized learning game that teaches critical thinking and scientific reasoning. Think. Skills Creat. 7(2), 93–100 (2012)CrossRefGoogle Scholar
  32. Holmqvist, K., Nyström, M., Andersson, R., et al.: Eye Tracking: a Comprehensive Guide to Methods and Measures. Oxford University Press, Oxford (2011)Google Scholar
  33. Horvitz, E., Jacobs, A., Hovel D.: Attention-sensitive alerting’. In: Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, pp. 305–313. Morgan Kaufmann Publishers Inc. (1999)Google Scholar
  34. Horvitz, E., Kadie, C., Paek, T., et al.: Models of attention in computing and communication: from principles to applications. Commun. ACM 46(3), 52–59 (2003)CrossRefGoogle Scholar
  35. Jha, A.P., Krompinger, J., Baime, M.J.: Mindfulness training modifies subsystems of attention. Cogn. Affect. & Behav. Neurosci. 7(2), 109–119 (2007)CrossRefGoogle Scholar
  36. Just, M.A., Carpenter, P.A.: A theory of reading: from eye fixations to comprehension. Psychol. Rev. 87(4), 329 (1980)CrossRefGoogle Scholar
  37. Kane, M.J., Brown, L.H., McVay, J.C., et al.: For whom the mind wanders, and when an experience-sampling study of working memory and executive control in daily life. Psychol. Sci. 18(7), 614–621 (2007)CrossRefGoogle Scholar
  38. Killingsworth, M.A., Gilbert, D.T.: A wandering mind is an unhappy mind. Science 330(6006), 932–932 (2010)CrossRefGoogle Scholar
  39. Knipling, R.F., Wierwille, W.W.: America, IVHS: Vehicle-Based Drowsy Driver Detection: Current Status and Future Prospects. National Highway Traffic Safety Administration,Office of Crash Avoidance Research (1994)Google Scholar
  40. Kopp, K., Bixler, R., D’Mello, S.: Identifying Learning Conditions that Minimize Mind Wandering by Modeling Individual Attributes, pp. 94–103. Springer, New York, In Intelligent Tutoring Systems (2014)Google Scholar
  41. Lutz, A., Slagter, H.A., Rawlings, N.B., et al.: Mental training enhances attentional stability: neural and behavioral evidence. J. Neurosci. 29(42), 13418–13427 (2009)CrossRefGoogle Scholar
  42. Mason, M.F., Norton, M.I., Van Horn, J.D., et al.: Wandering minds: the default network and stimulus-independent thought. Science 315(5810), 393–395 (2007)CrossRefGoogle Scholar
  43. Matsumoto, Y., Ogasawara, T., Zelinsky, A.: Behavior recognition based on head pose and gaze direction measurement. In: Intelligent Robots and Systems, 2000. (IROS 2000). Proceedings. 2000 IEEE/RSJ International Conference on, IEEE pp. 2127–2132 (2000)Google Scholar
  44. Mills, C., D’Mello, S.: In Press. Toward a real-time (Day) dreamcatcher: detecting mind wandering episodes during online reading. In: Proceedings of the 8th International Conference on Educational Data Mining, International Educational Data Mining Society pp. XX–XXGoogle Scholar
  45. Mooneyham, B.W., Schooler, J.W.: The costs and benefits of mind-wandering: a review. Can. J. Exp. Psychol./Rev. Can. de Psychol. Exp. 67(1), 11–18 (2013)Google Scholar
  46. Mrazek, M.D., Franklin, M.S., Phillips, D.T., et al.: Mindfulness training improves working memory capacity and gre performance while reducing mind wandering. Psychol. Sci. 24(5), 776–781 (2013)CrossRefGoogle Scholar
  47. Mrazek, M.D., Smallwood, J., Schooler, J.W.: Mindfulness and mind-wandering: finding convergence through opposing constructs. Emotion 12(3), 442–448 (2012)CrossRefGoogle Scholar
  48. Muir, M., Conati, C.: An analysis of attention to student—adaptive hints in an educational game. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., et al. (eds.) Intelligent Tutoring Systems, Lecture Notes in Computer Science, pp. 112–122. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  49. Navalpakkam, V., Kumar, R., Li, L., et al.: Attention and selection in online choice tasks. In: Masthoff, J., Mobasher, B., Desmarais, M.C., et al. (eds.) User Modeling, Adaptation, and Personalization, Lecture Notes in Computer Science, pp. 200–211. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  50. Pham, P., Wang, J.: Attentive learner: improving mobile MOOC learning via implicit heart rate tracking. In: Conati, C., Heffernan, N., Mitrovic, A., et al. (eds.) Artificial Intelligence in Education, pp. 367–376. Springer International Publishing, New York (2015)CrossRefGoogle Scholar
  51. Randall, J.G., Oswald, F.L., Beier, M.E.: Mind-wandering, cognition, and performance: a theory-driven meta-analysis of attention regulation. Psychol. Bull. 140(6), 1411–1431 (2014)CrossRefGoogle Scholar
  52. Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124(3), 372 (1998)CrossRefGoogle Scholar
  53. Reichle, E.D., Pollatsek, A., Fisher, D.L., et al.: Toward a model of eye movement control in reading. Psychol. Rev. 105(1), 125 (1998)CrossRefGoogle Scholar
  54. Reichle, E.D., Reineberg, A.E., Schooler, J.W.: Eye movements during mindless reading. Psychol. Sci. 21(9), 1300–1310 (2010)CrossRefGoogle Scholar
  55. Robertson, L.H., Manly, T., Andrade, J., et al.: Oops!: performance correlates of everyday attentional failures in traumatic brain injured and normal subjects. Neuropsychologia 35(6), 747–758 (1997)CrossRefGoogle Scholar
  56. Roda, C., Nabeth, T.: Supporting attention in learning environments: attention support services, and information management. In: Creating New Learning Experiences on a Global Scale, pp. 277–291. Springer, New York (2007)Google Scholar
  57. Roda, C., Thomas, J.: Attention aware systems. Encycl. Hum. Comput. Interact. 58, 38 (2006)CrossRefGoogle Scholar
  58. Schad, D.J., Nuthmann, A., Engbert, R.: Your mind wanders weakly, your mind wanders deeply: objective measures reveal mindless reading at different levels. Cognition 125(2), 179–194 (2012)CrossRefGoogle Scholar
  59. Schooler, J.W., Reichle, E.D., Halpern D.V.: Zoning out while reading: evidence for dissociations between experience and metaconsciousness. In: Daniel T.L. (ed.) Thinking and Seeing: Visual Metacognition in Adults and Children, pp. 203–226. Cambridge, Mass.: MIT Press (2004)Google Scholar
  60. Schooler, J.W., Smallwood, J., Christoff, K., et al.: Meta-awareness, perceptual decoupling and the wandering mind. Trends Cognit. Sci. 15(7), 319–326 (2011)Google Scholar
  61. Seibert, P.S., Ellis, H.C.: Irrelevant thoughts, emotional mood states, and cognitive task performance. Mem. & Cognit. 19(5), 507–513 (1991)CrossRefGoogle Scholar
  62. Selker, T.: Visual attentive interfaces. BT Technol. J. 22(4), 146–150 (2004)CrossRefGoogle Scholar
  63. Sewell, W., Komogortsev, O.: Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network. In: CHI’10 Extended Abstracts on Human Factors in Computing Systems, ACM pp. 3739–3744, (2010)Google Scholar
  64. Smallwood, J.: Mind-wandering while reading: attentional decoupling, mindless reading and the cascade model of inattention. Lang. Linguist. Compass 5(2), 63–77 (2011)CrossRefGoogle Scholar
  65. Smallwood, J., Beach, E., Schooler, J.W., et al.: Going AWOL in the brain: mind wandering reduces cortical analysis of external events. J. Cognit. Neurosci. 20(3), 458–469 (2008)CrossRefGoogle Scholar
  66. Smallwood, J., Brown, K.S., Tipper, C., et al.: Pupillometric evidence for the decoupling of attention from perceptual input during offline thought ’ed. Sam Gilbert. PLoS ONE 6(3), e18298 (2011)CrossRefGoogle Scholar
  67. Smallwood, J., Davies, J.B., Heim, D., et al.: Subjective experience and the attentional lapse: task engagement and disengagement during sustained attention. Consci. Cognit. 13(4), 657–690 (2004)CrossRefGoogle Scholar
  68. Smallwood, J., Fishman, D.J., Schooler, J.W.: Counting the cost of an absent mind: mind wandering as an underrecognized influence on educational performance. Psychon. Bull. & Rev. 14(2), 230–236 (2007)CrossRefGoogle Scholar
  69. Smallwood, J., McSpadden, M., Schooler, J.W.: When attention matters: the curious incident of the wandering mind. Mem. & Cognit. 36(6), 1144–1150 (2008)CrossRefGoogle Scholar
  70. Smallwood, J., Schooler, J.W.: The restless mind. Psychol. Bull. 132(6), 946–958 (2006)CrossRefGoogle Scholar
  71. Smilek, D., Carriere, J.S.A., Cheyne, J.A.: Out of mind, out of sight: eye blinking as indicator and embodiment of mind wandering. Psychol. Sci. 21(6), 786–789 (2010)CrossRefGoogle Scholar
  72. Stiefelhagen, R.: Tracking focus of attention in meetings. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, IEEE Computer Society pp. 273, (2002)Google Scholar
  73. Stiefelhagen, R., Yang, J., Waibel, A.: Estimating focus of attention based on gaze and sound. In: Proceedings of the 2001 workshop on Perceptive user interfaces, ACM pp. 1–9 (2001)Google Scholar
  74. Stiefelhagen, R., Zhu, J.: Head orientation and gaze direction in meetings. In: CHI’02 Extended Abstracts on Human Factors in Computing Systems, ACM pp. 858–859 (2002)Google Scholar
  75. Su, Mu-Chun, Hsiung, Chao-Yueh, Huang De-Yuan (2006) ’A Simple Approach to Implementing a System for Monitoring Driver Inattention’. In Systems, Man and Cybernetics, 2006. SMC’06. IEEE International Conference on, IEEE pp. 429–433Google Scholar
  76. Sussman, E., Bishop, H., Madnick, B., et al.: Driver inattention and highway safety. Transp. Res. Rec. 1047, 40–48 (1985)Google Scholar
  77. Tawari, A., Sivaraman, S., Trivedi, M.M. et al.: Looking-in and looking-out vision for urban intelligent assistance: estimation of driver attentive state and dynamic surround for safe merging and braking. In: Intelligent Vehicles Symposium Proceedings, 2014 IEEE, IEEE pp. 115–120 (2014)Google Scholar
  78. Toet, A.: Gaze directed displays as an enabling technology for attention aware systems. Comput. Hum. Behav. 22(4), 615–647 (2006)CrossRefGoogle Scholar
  79. Torkkola, K., Massey, N., Wood. C.: Driver inattention detection through intelligent analysis of readily available sensors. In: Intelligent Transportation Systems, 2004. Proceedings. The 7th International IEEE Conference on, IEEE pp. 326–331 (2004)Google Scholar
  80. Ugurlu, Y.: User attention analysis for e-learning systems using gaze and speech information. In: Information Science, Electronics and Electrical Engineering (ISEEE), 2014 International Conference on, IEEE pp. 1–5 (2014)Google Scholar
  81. Uzzaman, S., Steve, J.: The eyes know what you are thinking: eye movements as an objective measure of mind wandering. Consci. Cognit. 20(4), 1882–1886 (2011)CrossRefGoogle Scholar
  82. Vertegaal, R., Shell, J.S., Chen, D., et al.: Designing for augmented attention: towards a framework for attentive user interfaces. Comput. Hum. Behav. 22(4), 771–789 (2006)CrossRefGoogle Scholar
  83. Voit, M., Stiefelhagen, R.: Deducing the visual focus of attention from head pose estimation in dynamic multi-view meeting scenarios. In: Proceedings of the 10th International Conference on Multimodal Interfaces, ACM pp. 173–180 (2008)Google Scholar
  84. Voßkühler, A., Nordmeier, V., Kuchinke, L., et al.: open gaze and mouse analyzer): open-source software designed to analyze eye and mouse movements in slideshow study design. Behav. Res. Methods 40(4), 1150–1162 (2008)CrossRefGoogle Scholar
  85. Warren, T., White, S.J., Reichle, E.D.: Investigating the causes of wrap-up effects: evidence from eye movements and e-z reader. Cognition 111(1), 132–137 (2009)CrossRefGoogle Scholar
  86. Yeo, M.V.M., Li, X., Shen, K., et al.: ’Can SVM be used for automatic EEG detection of drowsiness during car driving? Saf. Sci. 47(1), 115–124 (2009)CrossRefGoogle Scholar
  87. Yonetani, R., Kawashima, H., Matsuyama, T.: Multi-mode saliency dynamics model for analyzing gaze and attention. Proceedings of the Symposium on Eye Tracking Research and Applications, ETRA ’12, New York, NY, pp. 115–122. ACM, New York (2012)CrossRefGoogle Scholar
  88. Zeidan, F., Johnson, S.K., Diamond, B.J., et al.: Mindfulness meditation improves cognition: evidence of brief mental training. Consci. Cognit. 19(2), 597–605 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of Notre DameNotre DameUSA

Personalised recommendations