Operationalization and Measurement of Evaluation Constructs

  • Katharina Emmerich
  • Natalya Bogacheva
  • Mareike Bockholt
  • Viktor Wendel
Chapter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9970)

Abstract

This chapter deals with the operationalization and measurement of evaluation constructs, an important and challenging part of the serious games evaluation process. Hereby, advices will be given on what has to be measured and how to quantify an abstract concept. Thus, the chapter makes two main contributions. First, general data gathering methods are described and discussed in terms of advantages and disadvantages. Second, main psychological concepts and evaluation constructs relevant in the context of serious games, as well as their theoretical foundations, are introduced. In order to support the reader on planning future serious game evaluations, a list and description of concrete techniques and questionnaires addressing concepts like motivation, player experience, learning outcomes, health, well-being, and attitudes are compiled.

Keywords

Operationalization Measurement methods Player experience Psychological constructs Evaluation 

References

  1. 1.
    Aronson, E., Wilson, T.D., Akert, R.M.: Social Psychology, 7th edn. Pearson, Boston (2010). Global ednGoogle Scholar
  2. 2.
    Bandura, A.: Guide for constructing self-efficacy scales. In: Pajares, F., Urdan, T.C. (eds.) Self-Efficacy Beliefs of Adolescents, pp. 307–337. Information Age Publishing, Charlotte (2006)Google Scholar
  3. 3.
    Bartle, R.: Hearts, clubs, diamonds, spades: players who suit muds. J. Virtual Environ. 1(1) (1996). http://mud.co.uk/richard/hcds.htm
  4. 4.
    Bartle, R.A.: Designing Virtual Worlds. New Riders Pub., Indianapolis (2004)Google Scholar
  5. 5.
    Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum. Comput. Interact. 2013, 1–11 (2013)Google Scholar
  6. 6.
    Bernhaupt, R. (ed.): Evaluating User Experience in Games: Concepts and Methods. Human-Computer Interaction Series. Springer, London, New York (2010)Google Scholar
  7. 7.
    Bernhaupt, R.: User experience evaluation in entertainment. In: Bernhaupt, R. (ed.) Evaluating User Experience in Games. Human-ComputerInteraction Series, pp. 3–7. Springer, London, New York (2010)CrossRefGoogle Scholar
  8. 8.
    Biocca, F., Harms, C., Burgoon, J.: Towards a more robust theory and measure of social presence: Review and suggested criteria. Presence Teleoper. Virtual Environ. 12(5), 456–480 (2003)CrossRefGoogle Scholar
  9. 9.
    Bircher, J., Kuruvilla, S.: Defining health by addressing individual, social, and environmental determinants: new opportunities for health care and public health. J. Public Health Policy 35(3), 363–386 (2014). http://www.palgrave-journals.com/jphp/journal/v35/n3/pdf/jphp.201419a.pdf CrossRefGoogle Scholar
  10. 10.
    Campbell, D.T., Stanley, J.C., Gage, N.L.: Experimental and Quasi-Experimental Designs for Research. Ravenio books, New York (2015)Google Scholar
  11. 11.
    Chin, J., Dukes, R., Gamson, W.: Assessment in simulation and gaming a review of the last 40 years. Simul. Gaming 40(4), 553–568 (2009)CrossRefGoogle Scholar
  12. 12.
    Connolly, T., Stansfield, M., Hainey, T.: Towards the development of a games-based learning evaluation framework. In: Games-Based Learning Advancements for Multisensory Human Computer Interfaces: Techniques and Effective Practices (2009)Google Scholar
  13. 13.
    Csikszentmihalyi, M.: Flow: The Psychology of Optimal Experience. Harper Perennial, New York (1990)Google Scholar
  14. 14.
    De Kort, Y.A.W., Ijsselsteijn, W.A.: People, places, and play: player experience in a socio-spatial context. Comput. Entertain. 6(2), 18 (2008)Google Scholar
  15. 15.
    De Kort, Y.A.W., Ijsselsteijn, W.A., Poels, K.: Digital games as social presence technology: development of the social presence in gaming questionnaire (SPGQ). In: Proceedings of the 10th Annual International Workshop on Presence, pp. 195–203 (2007)Google Scholar
  16. 16.
    Dignan, A.: Game frame: Using Games as a Strategy for Success. Free Press, New York (2011). 1st free press hardcover ednGoogle Scholar
  17. 17.
    Dillon, R.: On The Way to Fun: An Emotion-Based Approach to Successful Game Design. A K Peters, Natick (2010)CrossRefGoogle Scholar
  18. 18.
    Dodge, R., Daly, A., Huyton, J., Sanders, L.: The challenge of defining wellbeing. Int. J. Wellbeing 2(3), 222–235 (2012)CrossRefGoogle Scholar
  19. 19.
    Dugard, P., Todman, J.: Analysis of pre-test-post-test control group designs in educational research. Educ. Psychol. 15(2), 181–198 (1995)CrossRefGoogle Scholar
  20. 20.
    Ermi, L., Mäyrä, F.: Fundamental components of the gameplay experience: Analysing immersion. In: DiGRA 2005, Proceedings of the 2005 DiGRA International Conference: Changing Views: Worlds in Play (2005). http://www.digra.org/wp-content/uploads/digital-library/06276.41516.pdf
  21. 21.
    Fernandez, A.: Fun experience with digital games: a model proposition. In: Leino, O., Wirman, H., Fernandez, A. (eds.) Extending Experiences, pp. 181–190. Lapland University Press, Rovaniemi (2008)Google Scholar
  22. 22.
    Ferrara, J.: Playful Design: Creating Game Experiences in Everyday Interfaces. Rosenfeld Media, Brooklyn (2012)Google Scholar
  23. 23.
    Garfield, S., Clifford, S., Eliasson, L., Barber, N., Willson, A.: Suitability of measures of self-reported medication adherence for routine clinical use: a systematic review. BMC Med. Res. Methodol. 11, 149 (2011)CrossRefGoogle Scholar
  24. 24.
    Garris, R., Ahlers, R., Driskell, J.E.: Games, motivation, and learning: a research and practice model. Simul. Gaming 33(4), 441–467 (2002)CrossRefGoogle Scholar
  25. 25.
    Gee, J.P.: What video games have to teach us about learning and literacy. Comput. Entertain. 1(1), 20 (2003)CrossRefGoogle Scholar
  26. 26.
    Gekker, A.: Health games. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 13–30. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33687-4_2 CrossRefGoogle Scholar
  27. 27.
    Gill, P., Stewart, K., Treasure, E., Chadwick, B.: Methods of data collection in qualitative research: interviews and focus groups. Br. Dent. J. 204(6), 291–295 (2008)CrossRefGoogle Scholar
  28. 28.
    Girard, C., Ecalle, J., Magnan, A.: Serious games as new educational tools: how effective are they? a meta-analysis of recent studies. J. Comput. Assist. Learn. 29(3), 207–219 (2013)CrossRefGoogle Scholar
  29. 29.
    Goldstein, G., Hersen, M.: Handbook of Psychological Assessment. Elsevier, Hoboken (2000)Google Scholar
  30. 30.
    Goodwin, C.J., Goodwin, K.A.: Research in Psychology: Methods and Design, 7th edn. Wiley, Hoboken (2012)Google Scholar
  31. 31.
    Greenwald, A.G., McGhee, D.E., Schwartz, J.L.: Measuring individual differences in implicit cognition: the implicit association test. J. Person. Soc. Psychol. 74(6), 1464–1480 (1998)CrossRefGoogle Scholar
  32. 32.
    Guilford, J.P.: Psychometric Methods, 2nd edn. McGraw-Hill, New York (1954)Google Scholar
  33. 33.
    Hudson, M., Cairns, P.: Measuring social presence in team-based digital games. In: Riva, G., Waterworth, J., Murray, D. (eds.) Interacting with Presence: HCI and the Sense of Presence in Computer-Mediated Environments. DE GRUYTER OPEN, Warsaw, Poland (2014)Google Scholar
  34. 34.
    Igroup Project Consortium: Igroup presence questionnaire (ipq)Google Scholar
  35. 35.
    IJsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., Bellotti, F: Characterising and measuring user experiences in digital games. In: ACE 2007 International Conference on Advances in Computer Entertainment Technology, Workshop ‘Methods for Evaluating Games - How to measure Usability and User Experience in Games’ (2007)Google Scholar
  36. 36.
    Ijsselsteijn, W.A., De Kort, Y.A.W., Poels, K.: The game experience questionnaire: Development of a self-report measure to assess the psychological impact of digital games. Manuscript in preparationGoogle Scholar
  37. 37.
    Kahneman, D., Krueger, A.B.: Developments in the measurement of subjective well-being. J. Econ. Perspect. 20(1), 3–24 (2006)CrossRefGoogle Scholar
  38. 38.
    Kato, P.M., Cole, S.W., Bradlyn, A.S., Pollock, B.H.: A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 122(2), e305–e317 (2008). 18676516CrossRefGoogle Scholar
  39. 39.
    Kirkpatrick, D.L.: Evaluating Training Programs. Tata McGraw-Hill Education, San Francisco (1975)Google Scholar
  40. 40.
    Kivikangas, J.M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., Ravaja, N.: A review of the use of psychophysiological methods in game research. J. Gaming Virtual World 3(3), 181–199 (2011)CrossRefGoogle Scholar
  41. 41.
    Kline, P.: Psychometrics and Psychology. Academy Press, Washington, D.C. (1979)Google Scholar
  42. 42.
    Koster, R.: A Theory of Fun for Game Design, 2nd edn. O’Reilly Media Inc., Sebastopol (2014)Google Scholar
  43. 43.
    Krosnick, J.A., Presser, S.: Question and questionnaire design. In: Marsden, P.V. (ed.) Handbook of Survey Research. Emerald Group Publishing, Bingley (2010)Google Scholar
  44. 44.
    Landers, R.N., Bauer, K.N.: Quantitative methods and analyses for the study of players and their behaviour. In: Lankoski, P., Björk, S. (eds.) Game Research Methods, pp. 151–174. ETC Press, Pittsburgh (2015)Google Scholar
  45. 45.
    Law, E.L.C., Roto, V., Hassenzahl, M., Vermeeren, A.P., Kort, J.: Understanding, scoping and defining user experience. In: Olsen, D.R., Arthur, R.B., Hinckley, K., Morris, M.R., Hudson, S., Greenberg, S. (eds.) The SIGCHI Conference, p. 719 (2009)Google Scholar
  46. 46.
    Lazzaro, N.: Why we play games: Four keys to more emotion without story (2004)Google Scholar
  47. 47.
    Loh, C.S., Anantachai, A., Byun, J., Lenox, J.: Assessing what players learned in serious games: in situ data collection, information trails, and quantitative analysis. In: 10th International Conference on Computer Games: AI, Animation, Mobile, Educational & Serious Games (CGAMES 2007), pp. 25–28 (2007)Google Scholar
  48. 48.
    Loh, C.S., Sheng, Y., Ifenthaler, D. (eds.): Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement. Springer, Heidelberg (2015)Google Scholar
  49. 49.
    Maki, P.L.: Assessing for Learning: Building a Sustainable Commitment Across the Institution. Stylus Publishing, LLC, Menlo Park (2012)Google Scholar
  50. 50.
    Mandryk, R.L., Inkpen, K.M.: Physiological indicators for the evaluation of co-located collaborative play. In: Herbsleb, J., Olson, G. (eds.) The 2004 ACM Conference, p. 102 (2004)Google Scholar
  51. 51.
    McAuley, E., Duncan, T., Tammen, V.V.: Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: a confirmatory factor analysis. Res. Q. Exerc. Sport 60(1), 48–58 (1989)CrossRefGoogle Scholar
  52. 52.
    Nacke, L.E., Bateman, C., Mandryk, R.L.: BrainHex: preliminary results from a neurobiological gamer typology survey. In: Anacleto, J.C., Fels, S., Graham, N., Kapralos, B., Saif El-Nasr, M., Stanley, K. (eds.) ICEC 2011. LNCS, vol. 6972, pp. 288–293. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24500-8_31 CrossRefGoogle Scholar
  53. 53.
    Nacke, L.E.: Affective ludology: scientific measurement of user experience in interactive entertainment, Blekinge Institute of Technology doctoral dissertation series, vol. 2009: 04. School of Computing, Blekinge Institute of Technology, Karlskrona (2010)Google Scholar
  54. 54.
    O’Neil, H.F., Wainess, R., Baker, E.L.: Classification of learning outcomes: evidence from the computer games literature. Cirriculum J. 16(4), 455–474 (2005)CrossRefGoogle Scholar
  55. 55.
    Plass, J.L., Homer, B.D., Kinzer, C.K., Chang, Y.K., Frye, J., Kaczetow, W., Isbister, K., Perlin, K.: Metrics in simulations and games for learning. In: El-Nasr, M.S., Drachen, A., Canossa, A. (eds.) Game Analytics, pp. 697–729. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  56. 56.
    Poels, K., de Kort, Y., Ijsselsteijn, W.: It is always a lot of fun! In: Kapralos, B., Katchabaw, M., Rajnovich, J. (eds.) The 2007 Conference, p. 83 (2007)Google Scholar
  57. 57.
    Rammstedt, B., Kemper, C.J., Klein, M.C., Beierlein, C., Kovaleva, A.: A short scale for assessing the big five dimensions of personalityGoogle Scholar
  58. 58.
    Ravaja, N., Saari, T., Turpeinen, M., Laarni, J., Salminen, M., Kivikangas, M.: Spatial presence and emotions during video game playing: does it matter with whom you play? Presence Teleoper. Virtual Environ. 15(4), 381–392 (2006)CrossRefGoogle Scholar
  59. 59.
    Rheinberg, F.: Motivation, Kohlhammer-Urban-Taschenbücher, vol. 555. Kohlhammer, Stuttgart, 7, aktualisierte aufl. edn. (2008)Google Scholar
  60. 60.
    Ryan, D.: Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp. Educ. Psychol. 25(1), 54–67 (2000)CrossRefGoogle Scholar
  61. 61.
    Ryan, R.M., Rigby, C.S., Przybylski, A.: The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30(4), 344–360 (2006)CrossRefGoogle Scholar
  62. 62.
    Sadler, D.: Formative assessment and the design of instructional systems. Instr. Sci. 18(2), 119–144 (1989). http://dx.doi.org/10.1007/BF00117714 CrossRefGoogle Scholar
  63. 63.
    Sano, A., Picard, R.W.: Stress recognition using wearable sensors and mobile phones. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 671–676 (2013)Google Scholar
  64. 64.
    Shute, V.J., Ventura, M., Bauer, M., Zapata-Rivera, D.: Melding the power of serious games and embedded assessment to monitor and foster learning. Serious Games Mech. Effects 2, 295–321 (2009)Google Scholar
  65. 65.
    Slater, M.: A note on presence terminology. Presence Connect 3(3), 1–5 (2003)MathSciNetGoogle Scholar
  66. 66.
    Spielberger, C.D.: State-trait anxiety inventory. In: Weiner, I.B., Craighead, W.E. (eds.) The Corsini Encyclopedia of Psychology. Wiley, Hoboken (2010)Google Scholar
  67. 67.
    Strube, M.J., Newman, L.C.: Psychometrics. In: Cacioppo, J., Tassinary, L.G., Berntson, G.G. (eds.) The Handbook of Psychophysiology, pp. 789–811. Cambridge University Press, Cambridge (2007)Google Scholar
  68. 68.
    Sweetser, P., Wyeth, P.: Gameflow: a model for evaluating player enjoyment in games. Comput. Entertain. 3(3), 3 (2005)CrossRefGoogle Scholar
  69. 69.
    Takatalo, J., Häkkinen, J., Kaistinen, J., Nyman, G.: Presence, involvement, and flow in digital games. In: Bernhaupt, R. (ed.) Evaluating User Experience in Games, pp. 23–46. Human-Computer Interaction Series, Springer, London, New York (2010)CrossRefGoogle Scholar
  70. 70.
    Takatalo, J., Häkkinen, J., Komulainen, J., Särkelä, H., Nyman, G.: Involvement and presence in digital gaming. In: Mørch, A., Morgan, K., Bratteteig, T., Ghosh, G., Svanaes, D. (eds.) The 4th Nordic Conference, pp. 393–396 (2006)Google Scholar
  71. 71.
    UQO Cyberpsychology Lab: Immersive tendencies questionnaire (itq) (2004)Google Scholar
  72. 72.
    Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence questionnaire. Presence Teleoper. Virtual Environ. 7(3), 225–240 (1998)CrossRefGoogle Scholar
  73. 73.
    Yee, N.: Motivations for play in online games. Cyberpsychol. Behav. 9(6), 772–775 (2006). The impact of the Internet, multimedia and virtual reality on behavior and societyCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Katharina Emmerich
    • 1
  • Natalya Bogacheva
    • 2
  • Mareike Bockholt
    • 3
  • Viktor Wendel
    • 4
  1. 1.Entertainment Computing GroupUniversity of Duisburg-EssenDuisburgGermany
  2. 2.Moscow State UniversityMoscowRussia
  3. 3.Graph Theory and Complex Network Analysis GroupTU KaiserslauternKaiserslauternGermany
  4. 4.Multimedia Communications LabTU DarmstadtDarmstadtGermany

Personalised recommendations