Skip to main content
SpringerLink
Log in

A case study of prevalence and causes of eye tracking data loss in a middle school classroom

  • Research Article
  • Published:
Educational technology research and development Aims and scope Submit manuscript

Abstract

Recent advances in eye-tracking technology afford the possibility to collect rich data on attentional focus in a wide variety of settings outside the lab. However, apart from anecdotal reports, it is not clear how to maximize the validity of these data and prevent data loss from tracking failures. Particularly helpful in developing these techniques would be to describe the prevalence and causes of tracking failures in authentic environments. To meet this goal, we analyzed video records aligned with eye-tracking data collected from screen-mounted eye trackers employed in a middle-school classroom. Our sample includes records from 35 students recorded during multiple eye-tracking sessions. We compared student head position, body posture, attentiveness, and social interactions for time periods associated with successful and unsuccessful eye tracking. Overall, we observed substantial data loss and found that student inattentiveness, movements toward the eye tracker, and head rotations were the most prevalent factors inducing data loss. In addition, we observed a substantial relationship between data loss and apparent low involvement in the learning task. These data suggest that eye-tracking data loss is an important problem and that it can present a threat to validity because it can bias datasets to overrepresent high involvement behaviors. Based on these findings, we present several recommendations for increasing the proportion of usable data and to counter possible biases that data loss may introduce.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Anderson, D. R. (1985). Online cognitive processing of television. In L. F. Alwitt & A. M. Mitchell (Eds.), Psychological processes and advertising effects: Theory, research, and application (pp. 177–199). Hillsdale.

    Google Scholar 

  • Bidwell, J., & Fuchs, H. (2011). Classroom analytics: Measuring student engagement with automated gaze Tracking: Technical report. University of North Carolina at Chapel Hill.

    Google Scholar 

  • Biswas, G., Segedy, J. R., & Bunchongchit, K. (2016). From design to implementation to practice – A learning by teaching system: Betty’s brain. International Journal of Artificial Intelligence in Education, 26(1), 350–364.

    Article  Google Scholar 

  • Catrysse, L., Gijbels, D., Donche, V., Maeyer, S. D., Lesterhuis, M., & Bossche, P. V. (2018). How are learning strategies reflected in the eyes? Combining results from self-reports and eye-tracking. British Journal of Educational Psychology, 88, 118–137.

    Article  Google Scholar 

  • Chen, N., & Guastella, A. (2014). Eye tracking during a psychosocial stress simulation: Insights into social anxiety disorder. In M. Horsley, M. Eliot, B. Knight, & R. Reilly (Eds.), Current trends in eye tracking research. Springer.

    Google Scholar 

  • Colliot, T., & Jamet, É. (2018). Understanding the effects of a teacher video on learning from a multimedia document: An eye-tracking study. Educational Technology Research and Development, 66(6), 1415–1433.

    Article  Google Scholar 

  • Dalrymple, K. A., Manner, M. D., Harmelink, K. A., Teska, E. P., & Elison, J. T. (2018). An examination of recording accuracy and precision from eye tracking data from toddlerhood to adulthood. Frontiers in Psychology, 9, 803.

    Article  Google Scholar 

  • Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press.

    Google Scholar 

  • Holmqvist, K., Nyström, M., & Mulvey, F. (2012). Eye tracker data quality: What it is and how to measure it. Eye Tracking Research and Applications Symposium (ETRA). https://doi.org/10.1145/2168556.2168563

    Article  Google Scholar 

  • Hutt, S., Mills, C., White, S., Donnelly, P. J., & D'Mello, S. K. (2016). The Eyes Have It: Gaze-Based Detection of Mind Wandering during Learning with an Intelligent Tutoring System. Proceedings of the 9th International Educational Data Mining Society.

  • Jankovski, C., & Schofield, D. (2017). The eyes have it: Using eye tracking technology to assess the usability of learning management systems in elementary schools. European Journal of Education Studies, 3(10), 425–458. https://doi.org/10.5281/zenodo.1034181

    Article  Google Scholar 

  • Levin, D. T., & Keliikuli, K. (2020). An empirical assessment of cinematic continuity. Psychology of Aesthetics, Creativity, and the Arts. https://doi.org/10.1037/aca0000344

    Article  Google Scholar 

  • Miller, B. W. (2015). Using reading times and eye-movements to measure cognitive engagement. Educational Psychologist, 50(1), 31–42. https://doi.org/10.1080/00461520.2015.1004068

    Article  Google Scholar 

  • Mudrick, N. V., Azevedo, R., & Taub, M. (2019). Integrating metacognitive judgments and eye movements using sequential pattern mining to understand processes underlying multimedia learning. Computers in Human Behavior, 96, 223–234.

    Article  Google Scholar 

  • Niehorster, D. C., Cornelissen, T. H. W., Holmqvist, K., Hooge, I. T. C., & Hessels, R. S. (2018). What to expect from your remote eye-tracker when participants are unrestrained. Behavior Research Methods, 50(1), 213–222. https://doi.org/10.3758/s13428-017-0863-0

    Article  Google Scholar 

  • Ooms, K., Dupont, L., Lapon, L., & Popelka, S. (2015). Accuracy and precision of fixation locations recorded with the low-cost Eye Tribe tracker in different experimental set-ups. Journal of Eye Movement Research, 8(1), 1–24. https://doi.org/10.16910/jemr.8.1.5

    Article  Google Scholar 

  • Rajendran, R., Kumar, A., Carter, K. E., Levin, D. T., & Biswas, G. (2018). Predicting learning by analyzing eye-gaze data of reading behavior [Paper Presentation]. 11th International Conference on Educational Data Mining, Buffalo, NY.

  • Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422. https://doi.org/10.1037/0033-2909.124.3.372

    Article  Google Scholar 

  • Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506. https://doi.org/10.1080/17470210902816461

    Article  Google Scholar 

  • San Diego, J. P., Aczel, J. C., Hodgson, B. K., & Scanlon, E. (2012). Digital approaches to researching learners’ computer interactions using gazes, actions, utterances and sketches. Educational Technology Research and Development, 60(5), 859–881.

    Article  Google Scholar 

  • Špakov, O., Istance, H., Hyrskykari, A., Siirtola, H., & Räihä, K. (2019). Improving the performance of eye trackers with limited spatial accuracy and low sampling rates for reading analysis by heuristic fixation-to-word mapping. Behavior Research Methods, 51, 2661–2687. https://doi.org/10.3758/s13428-018-1120-x

    Article  Google Scholar 

  • Spichtig, A. N., Pascoe, J. P., Ferrara, J. D., & Vorstius, C. (2017). A comparison of eye movement measures across reading efficiency groups in us elementary, middle, and high school students. Journal of Eye Movement Research, 10(4), 5. https://doi.org/10.16910/jemr.10.4.5

    Article  Google Scholar 

  • Takacs, Z. K., & Bus, A. G. (2016). Benefits of motion in animated storybooks for children’s visual attention and story comprehension an eye-tracking study. Frontiers in Psychology, 7, 1591.

    Article  Google Scholar 

  • Teicher, M. H., Ito, Y., Glod, C. A., & Barber, N. I. (1996). Objective measurement of hyperactivity and attentional problems in ADHD. Journal of the American Academy of Child & Adolescent Psychiatry, 35(3), 334–342.

    Article  Google Scholar 

  • van Gog, T., & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitive and metacognitive processes in computer-based learning environments. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies. Springer international handbooks of education. (Vol. 28). New York: Springer.

    Google Scholar 

  • Yang, F. Y., Chang, C. Y., Chien, W. R., Chien, Y. T., & Tseng, Y. H. (2013). Tracking learners’ visual attention during a multimedia presentation in a real classroom. Computers & Education, 62, 208–220. https://doi.org/10.1016/j.compedu.2012.10.009

    Article  Google Scholar 

Download references

Acknowledgements

The research in this report was supported by NSF grant 1623625 to DTL and GB.

Funding

This paper was supported by grant NSF Grant No. 1623625.

Author information

Authors and Affiliations

  1. Peabody College, Vanderbilt University, #552, 230 Appleton Place, Nashville, TN, 37203-5721, USA

    Xiaorui Xue, Shiwei Xie, Shitanshu Mishra, Anna M. Wright, Gautam Biswas & Daniel T. Levin

Authors
  1. Xiaorui Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiwei Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shitanshu Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anna M. Wright
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gautam Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel T. Levin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel T. Levin.

Ethics declarations

Conflict of interest

There are no financial conflicts of interest.

Ethical approval

This paper was supported by grant XXXX, from XXXX. The research involves human participants and it was approved by the Vanderbilt University Institutional Review Board. All participants assented to participate in the research and in addition, parental/guardian consent was obtained. There are no financial conflicts of interest.

Research involving human and animal rights

The research involves human participants and it was approved by the Vanderbilt University Institutional Review Board. All participants assented to participate in the research and in addition, parental/guardian consent was obtained

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A coding scheme

Social interaction (note: this section only describes participants’ the concrete verbal behaviors not whether they were on-task or off-task).

  1. 1.

    Verbal Talking: Is the student talking? 0 = no 1 = yes (note: if it is not specified, the coding for following other behaviors is aligned with this)

  2. 2.

    Non-verbal Sounds: This refers to the student makes noises. Is the student making non-verbal sounds?

  3. 3.

    Listening: Is the student listening to others?

  4. 4.

    Interacting with Teacher/Research Staff: Is the student talking or listening to Teacher/Research Staff?

  5. 5.

    Interacting with Classmate/Small Group of classmates: Is the student talking or listening to other students?

Task orientation and involvement

  1. 6.

    Off-task—Alone: Is the student, on his/her own, engaging in activities that do not relate to Betty's Brain? For example, the student is performing random movements that do not seem to have a goal, reading other materials, playing, etc.

  2. 7.

    Off-task—Social: Is the student engaging in activities with another student that do not relate to Betty's Brain? For example, the student is interacting with other students, doing things not associated with the Betty's Brain system.

  3. 8.

    Associated Activity—Alone: Is the student associative with others? Talking or listening to other students/teacher/research staff about Betty's Brain without asking or giving instruction. Is the student associative with others?

  4. 9.

    Associated Activity—Social: Talking or listening to other students/teacher/research staff about Betty's Brain without asking or giving instruction.

  5. 10.

    On-task—Alone. Is the student using the Betty's Brain system on his/her own?

  6. 11.

    On-task—Social. Is the student using the Betty's Brain system along with another student or with staff?

  7. 12.

    High: Mostly on-task both two seconds before and after the coding point.

  8. 13.

    Medium: On-task at least at the coding point but off task for some of the two seconds before and/or after the coding point.

  9. 14.

    Low: Mostly off-task both two seconds before and after the coding point.

Head position and camera occlusion

  1. 15.

    Off-screen Look: Is the student looking at anywhere else than the screen?

  2. 16.

    Forward Bending: Is the student forward bending? 0 = no 1 = yes, head 2 = yes, body 3 = yes, both head and body.

  3. 17.

    Backward Bending: Is the student backward bending? The student is leaning back toward the back of the chair.

  4. 18.

    Lateral Bending: Is the student lateral bending?

  5. 19.

    Rotation: Is the student's head rotating to the left or right?

  6. 20.

    Close to Screen: Is the student close to the screen? The full width of the student’s face takes approximately half or more of the width of the video window; the length of the student’s face takes approximately 2/3 or more of the height of the video window (The same computer was used to do the coding and reliability check).

  7. 21.

    Far from Screen: Is the student far from the screen? The student is leaning back, and the shoulder reaches the back of the chair.

  8. 22.

    Eyes-out-of-screen or Occluded: Are the student’s eyes out of the screen? 0 = no 1 = yes, one eye 2 = yes, both eyes

  9. 23.

    Not Present: Is the student not present in the video?

  10. 24.

    Eye Closed: Is the student closing eyes? 0 = no 1 = yes, half-closed/one eye closed 2 = yes, both eyes closed

Other general behaviors

  1. 25.

    Glasses: Is the student wearing glasses?

  2. 26.

    Touch Eye-tracker: Is the student touching the eye-tracker?

  3. 27.

    Move Computer: Is the student moving the eye-tracker?

  4. 28.

    Typing: Is the student typing?

  5. 29.

    Hat or hoodies: Is the student wearing a hat or hoodies?

  6. 30.

    Hands on Face: Are student’s hands on the face? 0 = no 1 = yes, on face 2 = yes, covering some of the eyes 3 = yes, covering the eyes

  7. 31.

    Stand: Is the student standing?

  8. 32.

    Multiple faces: Are there multiple faces in the video?

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, X., Xie, S., Mishra, S. et al. A case study of prevalence and causes of eye tracking data loss in a middle school classroom. Education Tech Research Dev 70, 2017–2032 (2022). https://doi.org/10.1007/s11423-022-10154-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-022-10154-4

Keywords

Search

Navigation