A Meta-Analysis of Data Collection in Serious Games Research

  • Shamus P. SmithEmail author
  • Karen Blackmore
  • Keith Nesbitt
Part of the Advances in Game-Based Learning book series (AGBL)


Serious game analytics share many of the challenges of data analytics for computer systems involving human activity. Key challenges include how to collect data without influencing its generation, and more fundamentally, how to collect and validate data from humans where a primary emphasis is on what people are thinking and doing. This chapter presents a meta-analysis of data collection activities in serious games research. A systematic review was conducted to consider metrics and measures across the human–computer interaction, gaming, simulation, and virtual reality literature. The review focus was on the temporal aspect of data collection to identify if data is collected before, during, or after gameplay and if so what fundamental processes are used to collect data. The review found that the majority of data collection occurred post-game, then pre-game, and finally during gameplay. This reflects traditional difficulties of capturing gameplay data and highlights opportunities for new data capture approaches oriented towards data analytics. Also we identify how researchers gather data to answer fundamental questions about the efficacy of serious games and the design elements that might underlie their efficacy. We suggest that more standardized and better-validated data collection techniques, that allow comparing and contrasting outcomes between studies, would be beneficial.


Data collection Serious games Meta-review Data analytics 


  1. Adams, P. C. (1998). Teaching and learning with SimCity 2000. Journal of Geography, 97(2), 47–55.CrossRefGoogle Scholar
  2. Anderson, E., McLoughlin, L., Liarokapis, F., Peters, C., Petridis, P., & Freitas, S. (2010). Developing serious games for cultural heritage: a state-of-the-art review. Virtual Reality, 14(4), 255–275. doi: 10.1007/s10055-010-0177-3.CrossRefGoogle Scholar
  3. Bearman, M., Smith, C. D., Carbone, A., Slade, S., Baik, C., Hughes-Warrington, M., et al. (2012). Systematic review methodology in higher education. Higher Education Research & Development, 31(5), 625–640.CrossRefGoogle Scholar
  4. Blakely, G., Skirton, H., Cooper, S., Allum, P., & Nelmes, P. (2009). Educational gaming in the health sciences: Systematic review. Journal of Advanced Nursing, 65(2), 259–269. doi: 10.1111/j.1365-2648.2008.04843.x.CrossRefGoogle Scholar
  5. Blunt, R. (2007). Does game-based learning work? Results from three recent studies. In Proceedings of the Interservice/Industry Training, Simulation, & Education Conference (pp. 945–955). Orlando, FL: National Defense Industrial Association.Google Scholar
  6. Bowers, C. A., & Jentsch, F. (2001). Use of commercial off-the-shelf, simulations for team research. In C. A. Bowers & E. Salas (Eds.), Advances in human performance and cognitive engineering research (pp. 293–317). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  7. Boyle, E. A., Connolly, T. M., & Hainey, T. (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2(2), 69–74.CrossRefGoogle Scholar
  8. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686. doi: 10.1016/j.compedu.2012.03.004.CrossRefGoogle Scholar
  9. Crookall, D. (2010). Serious games, debriefing, and simulation/gaming as a discipline. Simulation & Gaming, 41(6), 898–920.CrossRefGoogle Scholar
  10. Crown, S. W. (2001). Improving visualization skills of engineering graphics students using simple JavaScript web based games. Journal of Engineering Education, 90(3), 347–355.CrossRefGoogle Scholar
  11. Day, E. A., Arthur, W., Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86(5), 1022–1033.CrossRefGoogle Scholar
  12. Dennis, K. A., & Harris, D. (1998). Computer-based simulation as an adjunct to ab initio flight training. International Journal of Aviation Psychology, 8(3), 261–276.CrossRefGoogle Scholar
  13. Garris, R., & Ahlers, R. (2001, December). A game-based training model: Development, application, and evaluation. Paper presented at the Interservice/Industry Training, Simulation & Education Conference, Orlando, FL.Google Scholar
  14. Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467.CrossRefGoogle Scholar
  15. Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning, 29(3), 207–219. doi: 10.1111/j.1365-2729.2012.00489.x.CrossRefGoogle Scholar
  16. González, L. S., Rubio, F. G., González, F. R., & Velthuis, M. P. (2010). Measurement in business processes: A systematic review. Business Process Management Journal, 16(1), 114–134.CrossRefGoogle Scholar
  17. Graafland, M., Schraagen, J. M., & Schijven, M. P. (2012). Systematic review of serious games for medical education and surgical skills training. British Journal of Surgery, 99(10), 1322–1330.CrossRefGoogle Scholar
  18. Hwang, G.-J., & Wu, P.-H. (2012). Advancements and trends in digital game-based learning research: A review of publications in selected journals from 2001 to 2010. Journal of Educational Technology, 43(1), E6–E10. doi: 10.1111/j.1467-8535.2011.01242.x. Blackwell.CrossRefGoogle Scholar
  19. Jennett, C., Cox, A. L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., et al. (2008). Measuring and defining the experience of immersion in games. International Journal of Human Computer Studies, 66(9), 641–661.CrossRefGoogle Scholar
  20. Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering—A systematic literature review. Information and Software Technology, 51(1), 7–15.CrossRefGoogle Scholar
  21. Lazar, J., Feng, J. H., & Hochheiser, H. (2010). Research methods in human-computer interaction. Chichester, England: Wiley.Google Scholar
  22. Loh, C. S. (2009). Research and developing serious games as interactive learning instructions. International Journal of Gaming and Computer-Mediated Simulations, 1(4), 1–19. IGI Global.CrossRefGoogle Scholar
  23. Loh, C. S. (2011). Using in situ data collection to improve the impact and return of investment of game-based learning. In Proceedings of ICEM-SIIE 2011, the 61st International Council for Educational Media (ICEM) and the XIII International Symposium on Computers in Education (SIIE) Joint Conference (pp. 801–811). ICEM-SIIE.Google Scholar
  24. Loh, C. S. (2012). Information trails: In-process assessment for game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 123–144). New York: Springer.CrossRefGoogle Scholar
  25. Marks, M. A. (2000). A critical analysis of computer simulations for conducting team research. Small Group Research, 31(6), 653–675.CrossRefGoogle Scholar
  26. Marsh, T., Smith, S. P., Yang, K., & Shahabi, C. (2006) Continuous and unobtrusive capture of user-player behaviour and experience to assess and inform game design and development. In 1st World Conference for Fun’n Games (FNG 2006) (pp. 79–86). Preston, England: University of Central Lancaster.Google Scholar
  27. Mayer, I., Bekebrede, G., Harteveld, C., Warmelink, H., Zhou, Q., Ruijven, T., et al. (2014). The research and evaluation of serious games: Toward a comprehensive methodology. British Journal of Educational Technology, 45(3), 502–527.CrossRefGoogle Scholar
  28. McFarlane, A., Sparrowhawk, A., & Heald, Y. (2002). Report on the educational use of games: An exploration by TEEM of the contribution which games can make to the education process. Last Access on 11th April, from
  29. McMahan, R. P., Ragan, E. D., Leal, A., Beaton, R. J., & Bowman, D. A. (2011). Considerations for the use of commercial video games in controlled experiments. Entertainment Computing, 2(1), 3–9.CrossRefGoogle Scholar
  30. Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical education: A literature review. Computers & Education, 53(3), 603–622. doi: 10.1016/j.compedu.2009.04.001.CrossRefGoogle Scholar
  31. Parker, L. E., & Lepper, M. R. (1992). Effects of fantasy contexts on children’s learning and motivation: Making learning more fun. Journal of Personality and Social Psychology, 62(4), 625–633.CrossRefGoogle Scholar
  32. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879.CrossRefGoogle Scholar
  33. Rogers, Y., Sharp, H., & Preece, J. (2011). Interaction design: Beyond human-computer interaction (3rd ed.). Hoboken, NJ: Wiley.Google Scholar
  34. Slater, M. (2004). How colorful was your day? Why questionnaires cannot assess presence in virtual environments. Presence, 13(4), 484–493.CrossRefGoogle Scholar
  35. Šmite, D., Wohlin, C., Gorschek, T., & Feldt, R. (2010). Empirical evidence in global software engineering: A systematic review. Empirical Software Engineering, 15(1), 91–118.CrossRefGoogle Scholar
  36. Smith, S. P., Stibric, M., & Smithson, D. (2013). Exploring the effectiveness of commercial and custom-built games for cognitive training. Computers in Human Behavior, 29(6), 2388–2393.CrossRefGoogle Scholar
  37. Squire, K., Giovanetto, L., Devane, B., & Durga, S. (2005). From users to designers: Building a self organizing game-based learning environment. TechTrends, 49(5), 34–42.CrossRefGoogle Scholar
  38. Van Eck, R. (2006). Digital game based learning: It’s not just the digital natives who are restless. Educause Review, 41(2), 17–30.Google Scholar
  39. Vandercruysse, S., Vandewaetere, M., & Clarebout, G. (2012). Game based learning: A review on the effectiveness of educational games. In M. M. Cruz-Cunha (Ed.), Handbook of research on serious games as educational, business, and research tools (pp. 628–647). Hershey, PA: IGI Global.CrossRefGoogle Scholar
  40. Wattanasoontorn, V., Boada, I., García, R., & Sbert, M. (2013). Serious games for health. Entertainment Computing, 4(4), 231–247.CrossRefGoogle Scholar
  41. Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., et al. (2009). Relationships between game attributes and learning outcomes review and research proposals. Simulation & Gaming, 40(2), 217–266.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Shamus P. Smith
    • 1
    Email author
  • Karen Blackmore
    • 2
  • Keith Nesbitt
    • 2
  1. 1.School of Electrical Engineering and Computer ScienceThe University of NewcastleCallaghanAustralia
  2. 2.School of Design, Communication and Information TechnologyThe University of NewcastleCallaghanAustralia

Personalised recommendations