Skip to main content
Log in

Performance assessment in serious games: Compensating for the effects of randomness

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41 % of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players’ achievements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abt, C. (1970). Serious games. New York: Viking.

    Google Scholar 

  • Aldrich, C. (2005). Learning by doing: the essential guide to simulations, computer games, and pedagogy E-learning and other educational experiences. San Francisco: Wiley.

    Google Scholar 

  • Becker, K., & Parker, J. R. (2011). The guide to computer simulations and games. Indianapolis: Wiley.

    Google Scholar 

  • Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of Serious Games: An Overview. Advances in Human-Computer Interaction, 1–11. http://www.hindawi.com/journals/ahci/2013/136864/#B18.

  • Bente, G., & Breuer, J. (2009). Making the implicit explicit: Embedded measurement in serious games. In U. Ritterfield, M. J. Cody, & P. Vorderer (Eds.), Serious Games: Mechanisms and Effects (pp. 322–343). New York: Routledge.

    Google Scholar 

  • Boston, C. (2002). The concept of formative assessment. Practical Assessment, Research & Evaluation, 8 (9). http://pareonline.net/getvn.asp?v=8&n=9.

  • Brennan, R. L. (2006). Educational measurement (4th ed.). Lanham: Rowman & Littlefield Publishers.

    Google Scholar 

  • Chin, J., Dukes, R., & Gamson, W. (2009). Assessment in simulation and gaming: a review of the last 40 years. Simulation & Gaming, 40(4), 553–568.

    Article  Google Scholar 

  • Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of the empirical evidence on computer games and serious games. Computers and Education, 59, 661–686.

    Article  Google Scholar 

  • Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41, 1040–1048.

    Article  Google Scholar 

  • Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (Vol. 5). New Jersey: Prentice Hall.

    Google Scholar 

  • Farr, J. L., Hofmann, D. A., & Ringenbach, K. L. (1993). Goal orientation and action control theory: Implications for industrial and organizational psychology. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (pp. 193–232). New York: John Wiley.

    Google Scholar 

  • Guttormsen Schär, S., Schluep, S., Schierz, C., & Krueger, H. (2000). Interaction for Computer-Aided Learning. Interactive Multimedia Electronic Journal of Computer-enhanced learning 2(1). http://imej.wfu.edu/articles/2000/1/03/.

  • Kolb, D. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs: Prentice Hall.

    Google Scholar 

  • Nuffic (2006). Fact sheet cijfers ontcijferd. Nuffic: The Hague. http://www.nuffic.nl/bestanden/documenten/over-de-nuffic/publicaties/factsheet-cijfers-ontcijferd.pdf/view.

  • Redeker, C., Punie, Y., & Ferrari, A. (2012). eAssessment for 21st century learning and skills. In A. Ravenscroft, S. Lindsteadt, C. D. Kloos, & D. Hernandez-Leo (Eds.), 21st Century Learning for 21st Century Skills. Proceedings of the 7th European Conference on technology-enhanced learning EC-TEL, Saarbrücken, 2012 (pp. 292–305). Heidelberg: Springer.

    Google Scholar 

  • Reese, H.W. (2011). The Learning-by-Doing Principle. Behavioral Development Bulletin, 11, 1–19. http://www.baojournal.com/BDB%20WEBSITE/archive/BDB-2011-11-01-001-019.pdf.

  • Schank, R. C. (1995). Engines for education. New York: Lawrence Erlbaum.

    Google Scholar 

  • Schank, R. C., Berman, T. R., & Macpherson, K. A. (1999). Learning by doing. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory (Vol. II, pp. 161–181). Mahwah: Lawrence Erlbaum Associates.

    Google Scholar 

  • Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

    Google Scholar 

  • Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning: flow and grow. In U. Ritterfeld, M. Cody, & M. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). New York: Routledge.

    Google Scholar 

  • Vargas, J. S. (1986). Instructional Design Flaws in Computer-Assisted Instruction. The Phi Delta Kappan, 67(10), 738–744. Retrieved from http://www.jstor.org/stable/20403230.

  • Westera, W., Hommes, M. A., Houtmans, M., & Kurvers, H. J. (2003). Computer-supported training of psycho-diagnostic skills. Interactive Learning Environments, 11(3), 215–231.

    Article  Google Scholar 

  • Westera, W., Nadolski, R., Hummel, H., & Wopereis, I. (2008). Serious games for higher education: a framework for reducing design complexity. Journal of Computer-Assisted Learning, 24(5), 420–432.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Wim Westera.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Westera, W. Performance assessment in serious games: Compensating for the effects of randomness. Educ Inf Technol 21, 681–697 (2016). https://doi.org/10.1007/s10639-014-9347-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-014-9347-3

Keywords

Navigation