Abstract
Serious game analytics share many of the challenges of data analytics for computer systems involving human activity. Key challenges include how to collect data without influencing its generation, and more fundamentally, how to collect and validate data from humans where a primary emphasis is on what people are thinking and doing. This chapter presents a meta-analysis of data collection activities in serious games research. A systematic review was conducted to consider metrics and measures across the human–computer interaction, gaming, simulation, and virtual reality literature. The review focus was on the temporal aspect of data collection to identify if data is collected before, during, or after gameplay and if so what fundamental processes are used to collect data. The review found that the majority of data collection occurred post-game, then pre-game, and finally during gameplay. This reflects traditional difficulties of capturing gameplay data and highlights opportunities for new data capture approaches oriented towards data analytics. Also we identify how researchers gather data to answer fundamental questions about the efficacy of serious games and the design elements that might underlie their efficacy. We suggest that more standardized and better-validated data collection techniques, that allow comparing and contrasting outcomes between studies, would be beneficial.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Adams, P. C. (1998). Teaching and learning with SimCity 2000. Journal of Geography, 97(2), 47–55.
Anderson, E., McLoughlin, L., Liarokapis, F., Peters, C., Petridis, P., & Freitas, S. (2010). Developing serious games for cultural heritage: a state-of-the-art review. Virtual Reality, 14(4), 255–275. doi:10.1007/s10055-010-0177-3.
Bearman, M., Smith, C. D., Carbone, A., Slade, S., Baik, C., Hughes-Warrington, M., et al. (2012). Systematic review methodology in higher education. Higher Education Research & Development, 31(5), 625–640.
Blakely, G., Skirton, H., Cooper, S., Allum, P., & Nelmes, P. (2009). Educational gaming in the health sciences: Systematic review. Journal of Advanced Nursing, 65(2), 259–269. doi:10.1111/j.1365-2648.2008.04843.x.
Blunt, R. (2007). Does game-based learning work? Results from three recent studies. In Proceedings of the Interservice/Industry Training, Simulation, & Education Conference (pp. 945–955). Orlando, FL: National Defense Industrial Association.
Bowers, C. A., & Jentsch, F. (2001). Use of commercial off-the-shelf, simulations for team research. In C. A. Bowers & E. Salas (Eds.), Advances in human performance and cognitive engineering research (pp. 293–317). Mahwah, NJ: Lawrence Erlbaum.
Boyle, E. A., Connolly, T. M., & Hainey, T. (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2(2), 69–74.
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686. doi:10.1016/j.compedu.2012.03.004.
Crookall, D. (2010). Serious games, debriefing, and simulation/gaming as a discipline. Simulation & Gaming, 41(6), 898–920.
Crown, S. W. (2001). Improving visualization skills of engineering graphics students using simple JavaScript web based games. Journal of Engineering Education, 90(3), 347–355.
Day, E. A., Arthur, W., Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86(5), 1022–1033.
Dennis, K. A., & Harris, D. (1998). Computer-based simulation as an adjunct to ab initio flight training. International Journal of Aviation Psychology, 8(3), 261–276.
Garris, R., & Ahlers, R. (2001, December). A game-based training model: Development, application, and evaluation. Paper presented at the Interservice/Industry Training, Simulation & Education Conference, Orlando, FL.
Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467.
Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning, 29(3), 207–219. doi:10.1111/j.1365-2729.2012.00489.x.
González, L. S., Rubio, F. G., González, F. R., & Velthuis, M. P. (2010). Measurement in business processes: A systematic review. Business Process Management Journal, 16(1), 114–134.
Graafland, M., Schraagen, J. M., & Schijven, M. P. (2012). Systematic review of serious games for medical education and surgical skills training. British Journal of Surgery, 99(10), 1322–1330.
Hwang, G.-J., & Wu, P.-H. (2012). Advancements and trends in digital game-based learning research: A review of publications in selected journals from 2001 to 2010. Journal of Educational Technology, 43(1), E6–E10. doi:10.1111/j.1467-8535.2011.01242.x. Blackwell.
Jennett, C., Cox, A. L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., et al. (2008). Measuring and defining the experience of immersion in games. International Journal of Human Computer Studies, 66(9), 641–661.
Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering—A systematic literature review. Information and Software Technology, 51(1), 7–15.
Lazar, J., Feng, J. H., & Hochheiser, H. (2010). Research methods in human-computer interaction. Chichester, England: Wiley.
Loh, C. S. (2009). Research and developing serious games as interactive learning instructions. International Journal of Gaming and Computer-Mediated Simulations, 1(4), 1–19. IGI Global.
Loh, C. S. (2011). Using in situ data collection to improve the impact and return of investment of game-based learning. In Proceedings of ICEM-SIIE 2011, the 61st International Council for Educational Media (ICEM) and the XIII International Symposium on Computers in Education (SIIE) Joint Conference (pp. 801–811). ICEM-SIIE.
Loh, C. S. (2012). Information trails: In-process assessment for game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 123–144). New York: Springer.
Marks, M. A. (2000). A critical analysis of computer simulations for conducting team research. Small Group Research, 31(6), 653–675.
Marsh, T., Smith, S. P., Yang, K., & Shahabi, C. (2006) Continuous and unobtrusive capture of user-player behaviour and experience to assess and inform game design and development. In 1st World Conference for Fun’n Games (FNG 2006) (pp. 79–86). Preston, England: University of Central Lancaster.
Mayer, I., Bekebrede, G., Harteveld, C., Warmelink, H., Zhou, Q., Ruijven, T., et al. (2014). The research and evaluation of serious games: Toward a comprehensive methodology. British Journal of Educational Technology, 45(3), 502–527.
McFarlane, A., Sparrowhawk, A., & Heald, Y. (2002). Report on the educational use of games: An exploration by TEEM of the contribution which games can make to the education process. Last Access on 11th April, from https://pantherfile.uwm.edu/tjoosten/LTC/Gaming/teem_gamesined_full.pdf
McMahan, R. P., Ragan, E. D., Leal, A., Beaton, R. J., & Bowman, D. A. (2011). Considerations for the use of commercial video games in controlled experiments. Entertainment Computing, 2(1), 3–9.
Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical education: A literature review. Computers & Education, 53(3), 603–622. doi:10.1016/j.compedu.2009.04.001.
Parker, L. E., & Lepper, M. R. (1992). Effects of fantasy contexts on children’s learning and motivation: Making learning more fun. Journal of Personality and Social Psychology, 62(4), 625–633.
Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879.
Rogers, Y., Sharp, H., & Preece, J. (2011). Interaction design: Beyond human-computer interaction (3rd ed.). Hoboken, NJ: Wiley.
Slater, M. (2004). How colorful was your day? Why questionnaires cannot assess presence in virtual environments. Presence, 13(4), 484–493.
Šmite, D., Wohlin, C., Gorschek, T., & Feldt, R. (2010). Empirical evidence in global software engineering: A systematic review. Empirical Software Engineering, 15(1), 91–118.
Smith, S. P., Stibric, M., & Smithson, D. (2013). Exploring the effectiveness of commercial and custom-built games for cognitive training. Computers in Human Behavior, 29(6), 2388–2393.
Squire, K., Giovanetto, L., Devane, B., & Durga, S. (2005). From users to designers: Building a self organizing game-based learning environment. TechTrends, 49(5), 34–42.
Van Eck, R. (2006). Digital game based learning: It’s not just the digital natives who are restless. Educause Review, 41(2), 17–30.
Vandercruysse, S., Vandewaetere, M., & Clarebout, G. (2012). Game based learning: A review on the effectiveness of educational games. In M. M. Cruz-Cunha (Ed.), Handbook of research on serious games as educational, business, and research tools (pp. 628–647). Hershey, PA: IGI Global.
Wattanasoontorn, V., Boada, I., García, R., & Sbert, M. (2013). Serious games for health. Entertainment Computing, 4(4), 231–247.
Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., et al. (2009). Relationships between game attributes and learning outcomes review and research proposals. Simulation & Gaming, 40(2), 217–266.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Smith, S.P., Blackmore, K., Nesbitt, K. (2015). A Meta-Analysis of Data Collection in Serious Games Research. In: Loh, C., Sheng, Y., Ifenthaler, D. (eds) Serious Games Analytics. Advances in Game-Based Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-05834-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-05834-4_2
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-05833-7
Online ISBN: 978-3-319-05834-4
eBook Packages: Humanities, Social Sciences and LawEducation (R0)