Serious Games Evaluation: Processes, Models, and Concepts

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9970)


Serious games are developed with the goal of having a certain impact on players which goes beyond mere entertainment. This purpose-driven design is immanent to serious games and can be stated as the key characteristic that distinguishes serious games from other digital games. Hence, verifying that a serious game has the intended effect on the players needs to be an essential part in the development process. This and the following chapters are therefore dedicated to give a guidance how evaluation procedures can be planned and realized. The main focus is on aspects which are particularly distinctive to the evaluation of serious games, while methods and principles related to the evaluation of digital games in general will not be covered in detail. The structure of this chapter is as follows: After emphasizing the specific importance of evaluation for serious games, we describe a set of challenges which might occur in this context. In order to enable the reader to face these challenges, we present a framework of evaluation-driven design which offers guidance in the evaluation process. Other models which address different challenges are described before three examples of commendably evaluated serious games are discussed. These examples are intended to demonstrate how the presented abstract models can be applied in concrete evaluation procedures.


Evaluation Process Virtual Reality Game Development Game Design Preparation Phase 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Baranowski, T.: Measurement method bias in games for health research. Games Health J. 3(4), 193–194 (2014)CrossRefGoogle Scholar
  2. 2.
    Chen, S., Michael, D.: Proof of learning: assessment in serious games (2005).
  3. 3.
    Chin, J., Dukes, R., Gamson, W.: Assessment in simulation and gaming: a review of the last 40 years. Simul. Gaming 40(4), 553–568 (2009)CrossRefGoogle Scholar
  4. 4.
    Coleman, J.S., Livingston, S.A., Fennessey, G.M., Edwards, K.J., Kidder, S.J.: The hopkins games program: conclusions from seven years of research. Educ. Res. 2(8), 3–7 (1973)CrossRefGoogle Scholar
  5. 5.
    Connolly, T., Stansfield, M., Hainey, T.: Towards the development of a games-based learning evaluation framework. In: Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices. IGI Global, Hershey (2009)Google Scholar
  6. 6.
    Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M.: A systematic literature review of empirical evidence on computer games and serious games. Comput. Educ. 59(2), 661–686 (2012)CrossRefGoogle Scholar
  7. 7.
    DeSmet, A., van Ryckeghem, D., Compernolle, S., Baranowski, T., Thompson, D., Crombez, G., Poels, K., van Lippevelde, W., Bastiaensens, S., van Cleemput, K., Vandebosch, H., de Bourdeaudhuij, I.: A meta-analysis of serious digital games for healthy lifestyle promotion. Prev. Med. 69, 95–107 (2014)CrossRefGoogle Scholar
  8. 8.
    Faber, A.W., Patterson, D.R., Bremer, M.: Repeated use of immersive virtual reality therapy to control pain during wound dressing changes in pediatric and adult burn patients. J. Burn Care Res. 34(5), 563–568 (2013)CrossRefGoogle Scholar
  9. 9.
    Girard, C., Ecalle, J., Magnan, A.: Serious games as new educational tools: how effective are they? a meta-analysis of recent studies. J. Comput. Assist. Learn. 29(3), 207–219 (2013)CrossRefGoogle Scholar
  10. 10.
    Göbel, S., Gutjahr, M., Hardy, S.: Evaluation of serious games. In: Bredl, K., Bösche, W. (eds.) Serious Games and Virtual Worlds in Education, Professional Development, and Healthcare, pp. 105–115. IGI Global, Hershey (2013)CrossRefGoogle Scholar
  11. 11.
    Hainey, T., Connolly, T.: Evaluating games-based learning. Int. J. Virtual Per. Learn. Environ. (IJVPLE) 1(1), 57–71 (2010)CrossRefGoogle Scholar
  12. 12.
    Hoffman, H.G., Chambers, G.T., Meyer, W.J., Arceneaux, L.L., Russell, W.J., Seibel, E.J., Richards, T.L., Sharar, S.R., Patterson, D.R.: Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures. Ann. Behav. Med. 41(2), 183–191 (2011)CrossRefGoogle Scholar
  13. 13.
    Hoffman, H.G., Patterson, D.R., Seibel, E., Soltani, M., Jewett-Leahy, L., Sharar, S.R.: Virtual reality pain control during burn wound debridement in the hydrotank. Clin. J. Pain 24(4), 299–304 (2008)CrossRefGoogle Scholar
  14. 14.
    Hoffman, H.G., Seibel, E.J., Richards, T.L., Furness, T.A., Patterson, D.R., Sharar, S.R.: Virtual reality helmet display quality influences the magnitude of virtual reality analgesia. J. Pain 7(11), 843–850 (2006)CrossRefGoogle Scholar
  15. 15.
    Huizenga, J., Admiraal, W., Akkerman, S., Dam, G.T.: Mobile game-based learning in secondary education: engagement, motivation and learning in a mobile city game. J. Comput. Assist. Learn. 25(4), 332–344 (2009)CrossRefGoogle Scholar
  16. 16.
    Kato, P.M., Cole, S.W., Bradlyn, A.S., Pollock, B.H.: A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 122(2), e305–e317 (2008)CrossRefGoogle Scholar
  17. 17.
    Kirkley, S.E., Tomblin, S., Kirkley, J.: Instructional design authoring support for the development of serious games and mixed reality training. In: Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) (2005)Google Scholar
  18. 18.
    Kirkpatrick, D.L., Kirkpatrick, J.D.: Evaluating Training Programs: The Four Levels, 3rd edn. Berrett-Koehler Publishers, San Francisco (2006)MATHGoogle Scholar
  19. 19.
    Loh, C.S., Sheng, Y., Ifenthaler, D.: Serious games analytics: theoretical framework. In: Loh, C.S., Sheng, Y., Ifenthaler, D. (eds.) Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement. Advances in Game-Based Learning, pp. 3–29. Springer International Publishing, Cham (2015)CrossRefGoogle Scholar
  20. 20.
    Maani, C.V., Hoffman, H.G., Morrow, M., Maiers, A., Gaylord, K., McGhee, L.L., DeSocio, P.A.: Virtual reality pain control during burn wound debridement of combat-related burn injuries using robot-like arm mounted VR goggles. J. Trauma: Injury, Infection Crit. Care 71(supplement), 125–130 (2011)CrossRefGoogle Scholar
  21. 21.
    Mayer, I.: Towards a comprehensive methodology for the research and evaluation of serious games. Procedia Comput. Sci. 15, 233–247 (2012)CrossRefGoogle Scholar
  22. 22.
    Michael, D.R., Chen, S.: Serious Games: Games That Educate, Train and Inform. Course Technology PTR, Boston (2005)Google Scholar
  23. 23.
    Mitgutsch, K., Alvarado, N.: Purposeful by design?: a serious game design assessment framework. In: Proceedings of the International Conference on the Foundations of Digital Games, FDG 2012, New York, NY, USA, pp. 121–128. ACM (2012)Google Scholar
  24. 24.
    Molenda, M.: In search of the elusive ADDIE model. Perform. Improv. 42(5), 34–36 (2003)CrossRefGoogle Scholar
  25. 25.
    O’Neil, H.F., Wainess, R., Baker, E.L.: Classification of learning outcomes: evidence from the computer games literature. Curriculum J. 16(4), 455–474 (2005)CrossRefGoogle Scholar
  26. 26.
    Papastergiou, M.: Exploring the potential of computer and video games for health and physical education: a literature review. Comput. Educ. 53(3), 603–622 (2009)CrossRefGoogle Scholar
  27. 27.
    Sanchez, A., Cannon-Bowers, J.A., Bowers, C.: Establishing a science of game based learning. In: Sanchez, A., Cannon-Bowers, J.A., Bowers, C. (eds.) Serious Game Design and Development: Technologies for Training and Learning, pp. 290–304. IGI Global, Hershey (2010)CrossRefGoogle Scholar
  28. 28.
    Schmitt, Y.S., Hoffman, H.G., Blough, D.K., Patterson, D.R., Jensen, M.P., Soltani, M., Carrougher, G.J., Nakamura, D., Sharar, S.R.: A randomized, controlled trial of immersive virtual reality analgesia, during physical therapy for pediatric burn injuries. Burns: J. Int. Soc. Burn Injuries 37(1), 61–68 (2011)CrossRefGoogle Scholar
  29. 29.
    Shapiro, M.A., Peña, J.: Generalizability and validity in digital game research. In: Ritterfeld, U., Cody, M.J., Vorderer, P. (eds.) Serious Games: Mechanisms and Effects, pp. 389–403. Routledge, New York (2009)Google Scholar
  30. 30.
    Tate, R., Haritatos, J., Cole, S.: HopeLab’s approach to re-mission. Int. J. Learn. Media 1(1), 29–35 (2009)CrossRefGoogle Scholar
  31. 31.
    Wender, R., Hoffman, H.G., Hunner, H.H., Seibel, E.J., Patterson, D.R., Sharar, S.R.: Interactivity influences the magnitude of virtual reality analgesia. J. Cyber Ther. Rehabil. 2(1), 27–33 (2009)Google Scholar

Further Reading

  1. 32.
    Abt, C.C.: Serious Games. University Press of America, Lanham (1987)Google Scholar
  2. 33.
    Ainsworth, S.: Evaluation methods for learning environments. In: A Tutorial for the 11th International Conference on Artificial Intelligence Education, Amsterdam.
  3. 34.
    All, A., Castellar, E.P.N., Van Looy, J.: Towards a conceptual framework for assessing the effectiveness of digital game-based learning. Comput. Educ. 88, 29–37 (2015)CrossRefGoogle Scholar
  4. 35.
    Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum. Comput. Interact. 2013 (2013). Article no. 1Google Scholar
  5. 36.
    Bernhaupt, R. (ed.): Evaluating User Experience in Games: Concepts and Methods. Human-Computer Interaction Series. Springer, London and New York (2010)Google Scholar
  6. 37.
    Bredl, K., Bösche, W. (eds.): Serious Games and Virtual Worlds in Education. Professional Development, and Healthcare. Premier reference source, Information Science Reference, Hershey (2013)Google Scholar
  7. 38.
    Dondi, C., Moretti, M.: A methodological proposal for learning games selection and quality assessment. Br. J. Educ. Technol. 38(3), 502–512 (2007). CrossRefGoogle Scholar
  8. 39.
    El-Nasr, M.S., Drachen, A., Canossa, A. (eds.): Game Analytics: Maximizing the Value of Player Data. Springer, London and New York (2013)Google Scholar
  9. 40.
    Göbel, S., Gutjahr, M., Steinmetz, R.: What makes a good serious game - conceptual approach towards a metadata format for the description and evaluation of serious games. In: Gouscous, D., Meimaris, M. (eds.) 5th European Conference on Games Based Learning, pp. 202–210. Academic Conferences Limited, Reading (2011)Google Scholar
  10. 41.
    Law, E.L.-C., Kickmeier-Rust, M.D., Albert, D., Holzinger, A.: Challenges in the development and evaluation of immersive digital educational games. In: Holzinger, A. (ed.) USAB 2008. LNCS, pp. 19–30. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-89350-9_2 CrossRefGoogle Scholar
  11. 42.
    Loh, C.S., Sheng, Y., Ifenthaler, D.: Serious Games Analytics: Methodologiesfor Performance Measurement, Assessment, and Improvement. Advances in Game-Based Learning. Springer International Publishing, Cham (2015)CrossRefGoogle Scholar
  12. 43.
    Mayer, I., Bekebrede, G., Harteveld, C., Warmelink, H., Zhou, Q., Ruijven, T., Lo, J., Kortmann, R., Wenzler, I.: The research and evaluation of serious games: toward a comprehensive methodology. Br. J. Educ. Technol. 45(3), 502–527 (2014)CrossRefGoogle Scholar
  13. 44.
    Schultz, C.P., Bryant, R.: Game Testing All in One, 2nd edn. Mercury Learning & Information, Dulles (2011)Google Scholar
  14. 45.
    Serrano, Á., Marchiori, E.J., Blanco, Á.D., Torrente, J., Fernández-Manjón, B.: A framework to improve evaluation in educational games. In: Global Engineering Education Conference (EDUCON), pp. 1–8. IEEE (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Entertainment Computing GroupUniversity of Duisburg-EssenDuisburgGermany
  2. 2.Graph Theory and Complex Network Analysis GroupUniversity of KaiserslauternKaiserslauternGermany

Personalised recommendations