A Provisional Framework for Multimodal Evaluation: Establishing Serious Games Quality Label for Use in Training and Talent Development

  • Wee Hoe TanEmail author
  • Ivan Boo
Part of the Advances in Game-Based Learning book series (AGBL)


One common challenge faced by game-based learning practitioners is how to choose the right game at the right time and use the right game-based approach. This chapter depicts an attempt of the Serious Games Association in establishing a provisional framework of multimodal evaluation that can be used to generate quality labels for serious games, particularly in the training and talent development sector. This chapter aims to juxtapose components of the framework, aligning structural elements of serious games to specific knowledge, training evaluation, and reliability tests. An initial framework was created in 2017, intended to evaluate the quality of serious games submitted for a regional serious games marketplace. The framework was expected to assure constructive alignment between game goals and intended training outcomes in the healthcare and wellness industry. Also, it maintains originality and relevance in providing a feasible solution to specific problems encountered by practitioners. In practice, specific quality labels are defined, described, classified, and organized according to criteria adopted from the star rating system established by the Dutch Society for Simulation in Healthcare. This, in turn, would enable game-based training managers to make an optimum selection of appropriate serious games.


Provisional framework Multimodal evaluation Serious games Quality label Training and talent development Star rating 



We thank Ms. Ang Shin Yuh, the Senior Nurse Manager of Nursing Research and Quality, and Ms. Andrea Choh and Ms. Teo Kai Yun who are nurse clinicians, Nursing Quality Management in SGH, who developed the blood transfusion game.


  1. Abt, C. C. (1970). Serious games. Lanham, MD: University Press of America.Google Scholar
  2. Adams, E. (2004). The designer’s notebook: Postmodernism and the 3 types of immersion. Retrieved December 25, 2018, from
  3. Adams, E. (2013). Fundamentals of game design (3rd ed.). San Francisco, CA: New Riders.Google Scholar
  4. Aebersold, M. (2018). Simulation-based learning: No longer a novelty in undergraduate education. The Online Journal of Issues in Nursing, 23, 2. Scholar
  5. Barnum, C. M. (2011). Usability testing essentials. Burlington, MA: Morgan Kaufmann.Google Scholar
  6. Biggs, J., & Tang, C. (2007). Teaching for quality learning at university: What the student does (3rd ed.). Maidenhead, UK: Society for Research into Higher Education and Open University Press.Google Scholar
  7. Brady, M., Devitt, A., & Jameson, E. (2016). Towards a framework for the evaluation of digital game-based learning at the third level. In Proceedings of the 6th Irish Conference of Game-based Learning (pp. 110–115). Trinity College Dublin.Google Scholar
  8. Cant, R. P., & Cooper, S. J. (2017). The value of simulation-based learning in pre-licensure nurse education: A state-of-theart review and meta-analysis. Nurse Education in Practice, 27, 45–62.CrossRefGoogle Scholar
  9. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.CrossRefGoogle Scholar
  10. Csikzentmihalyi, M. (2008). Flow: The psychology of optimal experience. New York, NY: Harper Perennial.Google Scholar
  11. De Freitas, S. (2006). Learning in immersive worlds. A review of game-based learning. Bristol, UK: JISC.Google Scholar
  12. Doyen, B., Mert, A., Meijer, H. A. W., Nijboer, T. C. W., & Dankbaar, M. E. W. (2018). A quality label for serious games. Medical Training Magazine, 1, 10–13.Google Scholar
  13. Dunning, D., Johnson, K., Ehrlinger, J., & Kruger, J. (2003). Why people fail to recognize their own incompetence. Current Directions in Psychological Science, 12(3), 33–37. Scholar
  14. Dutch Game Garden. (2016). Growing Games. The evaluation of health-oriented serious games and apps: A differentiated approach. Retrieved from
  15. Falstein, N. (2007). Serious games production. In The eLearning Guild’s Online Forums: Strategies and Techniques for Building Immersive Learning Simulations (ILS).Google Scholar
  16. Feldt, L. S. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder Richardson coefficient twenty is the same for two tests. Psychometrika, 34(3), 363–373.CrossRefGoogle Scholar
  17. Fries, J. F., Spitz, P. R. N., Kraines, R. G., & Holman, H. R. (1980). Measurement of patient outcome in arthritis. Arthritis and Rheumatism, 23, 137–145. Scholar
  18. Glatter, R. (2016). Barton Health first to offer new digital medicine developed by Proteus Digital Health. Forbes. Retrieved May 21, 2019, from
  19. Gloria, A. D., Bellotti, F., Berta, R., & Lavagnino, E. (2014). Serious games for education and training. International Journal of Serious Games, 1(1).
  20. Hays, R. T. (2006). The science of learning: A systems theory perspective. Boca Raton, FL: BrownWalker Press.Google Scholar
  21. Horton, H. (2001). Evaluation e-learning. Alexandria, Egypt: American Society for Training and Development (ASTD).Google Scholar
  22. Hughes, G. (2017). Ipsative assessment and learning gain: International case studies. Basingstoke, UK: Palgrave Macmillan.CrossRefGoogle Scholar
  23. International Age Rating Coalition. (2016). How IARC works? Retrieved December 28, 2016, from
  24. Iuppa, N., & Borst, T. (2010). End-to-end game development: Creating independent serious games and simulations from start to finish. Burlington, MA: Focal Press.Google Scholar
  25. Janan, D., & Wray, D. (2014). Reassessing the accuracy and use of readability formulae. Malaysian Journal of Learning and Instruction, 11, 127–145.Google Scholar
  26. Jenkins, H. (2006). Game design as narrative architecture. In K. Salen & E. Zimmerman (Eds.), The game design reader: A rules of play anthology (pp. 670–689). Cambridge, MA: MIT Press.Google Scholar
  27. Jones, F., Passos-Neto, C. E., & Braghiroli, O. F. M. (2015). Simulation in medical education: Brief history and methodology. Principles and Practice of Clinical Research, 1(2), 56–63.Google Scholar
  28. Kirakowski, J., & Corbett, M. (1993). SUMI: The software usability measurement inventory. British Journal of Educational Technology, 24(3), 210–212. Scholar
  29. Lin, Y., Cheng, A., Hecker, K., Grant, V., & Currie, G. R. (2018). Implementing economic evaluation in simulation-based medical education: Challenges and opportunities. Medical Education, 52(2), 150–160.CrossRefGoogle Scholar
  30. MacArthur, R. H., & Wilson, E. O. (1967). The theory of island biogeography. Princeton, NJ: Princeton University Press.Google Scholar
  31. McDougall, E. M. (2015). Simulation in education for health care professionals. British Columbia Medical Journal, 57(10), 444–448.Google Scholar
  32. Nutter, F. W., Gleason, M. L., & Christians, N. C. (1993). Assessing the accuracy, intra-rater repeatability, and inter-rater reliability of disease assessment systems. Phytopathology, 83, 806–812.CrossRefGoogle Scholar
  33. Popescu, M. M., Romero, M., & Usart, M. (2013). Serious games for serious learning-using SG for business, management and defence education. International Journal of Computer Science Research and Applications, 3(1), 5–15.Google Scholar
  34. Prensky, M. (2007). Digital game-based learning. New York, NY: Paragon House.Google Scholar
  35. Sawyer, B., & Smith, P. (2008). Serious games taxonomy. Retrieved December 25, 2018, from
  36. Serious Games Society. (2008). Serious games initiative: About serious games initiative. Retrieved December 25, 2018, from
  37. Sonawane, K. (2017). Serious games market – Global opportunity analysis and industry forecast, 2016–2023. Singapore, Singapore: Allied Market Research.Google Scholar
  38. Tan, W. H. (2013). Developing aesthetics assessment skills in Bachelor of Design Programmes: An introspective account in UPSI. International Journal of Assessment and Evaluation in Education, 3, 92–104. Retrieved from Scholar
  39. Tan, W. H. (2015). Gamifikasi dalam pendidikan: Pembelajaran berasaskan permainan. Tanjung Malim, Malaysia: Penerbit UPSI.Google Scholar
  40. Tan, W. H., Noor, N. F. M., & Wang, Y. (2016). Inventive problem-solving puzzle game design and development. International Journal of Computational Vision and Robotics, 6(3), 244–252. Scholar
  41. U.S. Army. (2017). America’s Army: America’s Army Providing Grounds: The official game of the US Army. Retrieved January 25, 2017, from
  42. Wang, Y., & Yatim, M. H. M. (2019). A framework of childhood obesity prevention. In W. H. Tan (Ed.), Design, motivation, and frameworks of game-based learning (pp. 154–183). Hershey, PA: IGI Global.CrossRefGoogle Scholar
  43. Wattanasoontorn, V., Boada, I., Garcia, R., & Sbert, M. (2013). Serious games for health. Entertainment Computing, 4, 231–247. Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Faculty of Art, Computing and Creative IndustrySultan Idris Education UniversityTanjong MalimMalaysia
  2. 2.Serious Games AssociationSingaporeSingapore

Personalised recommendations