Technology, Knowledge and Learning

, Volume 23, Issue 1, pp 65–81 | Cite as

The Temporal Attentive Observation (TAO) Scale: Development of an Instrument to Assess Attentive Behavior Sequences During Serious Gameplay

  • James E. FolkestadEmail author
  • Brian McKernan
  • Stephanie Train
  • Rosa Mikeal Martey
  • Matthew G. Rhodes
  • Kate Kenski
  • Adrienne Shaw
  • Jennifer Stromer-Galley
  • Benjamin A. Clegg
  • Tomek Strzalkowski
Original research


The engaging nature of video games has intrigued learning professionals attempting to capture and retain learners’ attention. Designing learning interventions that not only capture the learner’s attention, but also are designed around the natural cycle of attention will be vital for learning. This paper introduces the temporal attentive observation (TAO) instrument, an instrument developed to assess attentive behavior sequences during serious gameplay. We use an established three-step process for developing observational systems that includes identifying the construct, determining validity, and demonstrating practicality criteria. We conclude that the TAO instrument reliably measures attention behaviors where participants’ faces can be recorded during an experiment. Furthermore, we suggest that TAO should be considered as a part of an attention measurement package.


Attention Behavior Instrument Serious games Temporal 



This work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Air Force Research Laboratory contract number FA8650-11-C-7176. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, AFRL, or the US Government.


  1. Attfield, S., Kazai, G., Lalmas, M., & Piwowarski, B. (February 2011). Towards a science of user engagement (position paper). In WSDM workshop on user modeling for web applications.Google Scholar
  2. Bailey, B. P., & Konstan, J. A. (2006). On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state. Computers in Human Behavior, 22(4), 685–708.CrossRefGoogle Scholar
  3. Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  4. Boyle, E. A., Connolly, T. M., Hainey, T., & Boyle, J. M. (2011). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28(3), 771–780.CrossRefGoogle Scholar
  5. Castel, A. D., Farb, N., & Craik, F. I. M. (2007). Memory for general and specific value information in younger and older adults: Measuring the limits of strategic control. Memory and Cognition, 35, 689–700.CrossRefGoogle Scholar
  6. Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. (2014). Digital games, design, and learning: A systematic review and meta-analysis. Menlo Park, CA: SRI International.Google Scholar
  7. Clark, R. C. (2008). Building expertise: Cognitive methods for training and performance improvement. Munich: Pfeiffer.Google Scholar
  8. Clegg, B., Martey, R., Folkestad, J. E., Stromer-Galley, J., Kenski, K., Saulnier, T., et al. (2014). Game-based training to mitigate three forms of cognitive bias. In Interservice/industry training, simulation and education conference.Google Scholar
  9. Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation and Gaming, 33(4), 441–467.CrossRefGoogle Scholar
  10. Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in Entertainment (CIE), 1(1), 20.CrossRefGoogle Scholar
  11. Godwin, K. E., Almeda, M. V., Petroccia, M., Baker, R. S., & Fisher, A. V. (2013). Classroom activities and off-task behavior in elementary school children. Cognitive Science Society.Google Scholar
  12. Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (no. NAWCTSD-TR-2005-004). Naval Air Warfare Center Training Systems Division.Google Scholar
  13. Herbert, J., & Attridge, C. (1975). A guide for developers and users of observation systems and manuals. American Educational Research Journal, 12(1), 1–20.CrossRefGoogle Scholar
  14. Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19(1), 59–71.Google Scholar
  15. Holmqvist, K., Nyström, M., & Mulvey, F. (March 2012). Eye tracker data quality: What it is and how to measure it. In Proceedings of the symposium on eye tracking research and applications (pp. 45–52). ACM.Google Scholar
  16. Hornof, A. J., & Halverson, T. (2002). Cleaning up systematic error in eye-tracking data by using required fixation locations. Behavior Research Methods, Instruments, and Computers, 34(4), 592–604.CrossRefGoogle Scholar
  17. Ismail, E., Sidek, S. N., Khan, M. R., Jalaludin, N. A. (July 2012). Analysis of engagement factor in trajectory tracking-based experiment. In 2012 international conference on computer and communication engineering (ICCCE). Kuala Lumpur.Google Scholar
  18. Kahneman, D. (2011). Thinking, fast and slow. New York: Macmillan.Google Scholar
  19. Karimi, A., & Lim, Y. P. (December 2010). Children, engagement, and enjoyment in digital narrative. In Proceedings of ASCILITE 2010. Sydney. Retrieved June 28, 2013 from
  20. Larrick, R. P. (2004). Debiasing. In D. J. Koehler & N. Harvey (Eds.), Blackwell handbook of judgment and decision making (pp. 316–338). Malden, MA: Blackwell.CrossRefGoogle Scholar
  21. Lim, C. P. (2008). Spirit of the game: Empowering students as designers in schools? British Journal of Educational Technology, 39(6), 996–1003.CrossRefGoogle Scholar
  22. López, F., Menez, M., & Hernández-Guzmán, L. (2005). Sustained attention during learning activities: An observational study with pre-school children. Early Child Development and Care, 175(2), 131–138.CrossRefGoogle Scholar
  23. Martey, R. M., Kenski, K., Folkestad, J., Feldman, L., Gordis, E., Shaw, A., et al. (2014). Measuring game engagement multiple methods and construct complexity. Simulation and Gaming, 45, 528–547.CrossRefGoogle Scholar
  24. McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. New York, NY: Penguin.Google Scholar
  25. Medina, J. (2008). Brain rules: 12 principles for surviving and thriving at work, home, and school. Edmonds: Pear Press.Google Scholar
  26. Medina, J. (2014). Brain rules for baby, updated and expanded: How to raise a smart and happy child from zero to five. Edmonds: Pear Press.Google Scholar
  27. Mezirow, J. (1990). Fostering critical reflection in adulthood. San Francisco: Jossey-Bass.Google Scholar
  28. Middlebrooks, C. D., McGillivray, S., Murayama, K., & Castel, A. D. (2015). Memory for allergies and health foods: How younger and older adults strategically remember critical health information. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 71(3), 389–399.Google Scholar
  29. Mullinix, G., Gray, O., Colado, J., Veinott, E., Leonard, J., Papautsky, E. L., & Todd, P. M. (September 2013). Heuristica: Designing a serious game for improving decision making. In 2013 IEEE international games innovation conference (IGIC) (pp. 250–255). IEEE.Google Scholar
  30. O’Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6), 938–955.CrossRefGoogle Scholar
  31. O’Brien, H. L., Toms, E. G., Kelloway, E. K., & Kelley, E. (2008). Developing and evaluating a reliable measure of user engagement. Proceedings of the American Society for Information Science and Technology, 45(1), 1–10.CrossRefGoogle Scholar
  32. O’Neil, H. F., Wainess, R., & Baker, E. L. (2005). Classification of learning outcomes: Evidence from the computer games literature. The Cirriculum Journal, 16(4), 455–474.CrossRefGoogle Scholar
  33. Packard, R. G. (1970). The control of “classroom attention”: A group contingency for complex behavior 1. Journal of Applied Behavior Analysis, 3(1), 13–28.CrossRefGoogle Scholar
  34. Prensky, M. (2003). Digital game-based learning. Computers in Entertainment (CIE), 1(1), 21.CrossRefGoogle Scholar
  35. Ravaja, N., Saari, T., Salminen, M., Laarni, J., & Kallinen, K. (2006). Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology, 8, 343–367.CrossRefGoogle Scholar
  36. Read, J. C., MacFarlane, S. J., & Casey, C. (2002). Endurability,  engagement and expectations: Measuring children’s fun. Interaction Design and Children, 2, 1–23.Google Scholar
  37. Reese, D. D. (2015). CyGaMEs selene player log dataset: Gameplay assessment, flow dimensions and non-gameplay assessments. British Journal of Educational Technology, 46(5), 1005–1014.CrossRefGoogle Scholar
  38. Renshaw, T., Stevens, R., & Denton, P. D. (2009). Towards understanding engagement in games: An eye-tracking study. On the Horizon, 17, 408–420.CrossRefGoogle Scholar
  39. Reynolds, R. E., & Anderson, R. C. (1982). Influence of questions on the allocation of attention during reading. Journal of Educational Psychology, 74(5), 623.CrossRefGoogle Scholar
  40. Ricci, K., Salas, E., & Cannon-Bowers, J. A. (1996). Do computer-based games facilitate knowledge acquisition and retention? Military Psychology, 8(4), 295–307.CrossRefGoogle Scholar
  41. Squire, K. (2011). Video games and learcning: Teaching and participatory culture in the digital age. Technology, Education--Connections (the TEC Series). New York: Teachers College Press.Google Scholar
  42. Stevens, C., & Bavelier, D. (2012). The role of selective attention on academic foundations: A cognitive neuroscience perspective. Developmental Cognitive Neuroscience, 2, S30–S48.CrossRefGoogle Scholar
  43. Symborski, C., Barton, M., Quinn, M., Morewedge, C., Kassam, K., Korris, J., & Hollywood, C. A. (2014). Missing: A serious game for the mitigation of cognitive biases. In Proceedings of the interservice/industry training, simulation, and education conference (I/ITSEC) (pp. 1–13).Google Scholar
  44. Teixeira, T., Wedel, M., & Pieters, R. (2012). Emotion-induced engagement in Internet video advertisements. Journal of Marketing Research, 49, 144–159.CrossRefGoogle Scholar
  45. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.CrossRefGoogle Scholar
  46. Vogel, J. F., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34, 229–243.CrossRefGoogle Scholar
  47. Wozniak, R. H. (1999). Introduction to memory: Hermann Ebbinghaus (1885/1913). In Classics in the history of psychology. Bristol: Thoemmes Press.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2017

Authors and Affiliations

  • James E. Folkestad
    • 1
    Email author
  • Brian McKernan
    • 2
  • Stephanie Train
    • 3
  • Rosa Mikeal Martey
    • 3
  • Matthew G. Rhodes
    • 4
  • Kate Kenski
    • 5
  • Adrienne Shaw
    • 6
  • Jennifer Stromer-Galley
    • 7
  • Benjamin A. Clegg
    • 4
  • Tomek Strzalkowski
    • 8
  1. 1.School of EducationColorado State UniversityFort CollinsUSA
  2. 2.Department of Interdisciplinary StudiesThe Sage CollegesAlbanyUSA
  3. 3.Journalism and Technical CommunicationColorado State UniversityFort CollinsUSA
  4. 4.Department of PsychologyColorado State UniversityFort CollinsUSA
  5. 5.Department of CommunicationUniversity of ArizonaTucsonUSA
  6. 6.Temple UniversityPhiladelphiaUSA
  7. 7.Syracuse UniversitySyracuseUSA
  8. 8.The Institute for Informatics, Logics, and Security StudiesUniversity of Albany, SUNYAlbanyUSA

Personalised recommendations