The Experimental Method as an Evaluation Tool in Serious Games Research and Development

  • Nataliya V. Bogacheva
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9970)


This chapter aims to provide the reader with basic knowledge about the experiment as a general method that can be applied towards serious games research and evaluation. It explains the main terms and rules of the experimental design, as well as points out the main risks and difficulties to avoid. The chapter also explains the differences in possible conclusions between true experiments, quasi-experiments, and correlational studies.


Experiment Experimental design Variables Validity Biases Samples Quasi-experiment Correlational study 


  1. 1.
    Adams, K.A., Lawrence, E.K.: Research Methods, Statistics and Applications. SAGE Publications, London (2014)Google Scholar
  2. 2.
    Adejumobi, B., Franck, N., Janzen, M.: Designing and testing a racing car serious game module. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 192–198. Springer, Heidelberg (2014)Google Scholar
  3. 3.
    Anderson, C.A., Dill, K.E.: Video games and aggressive thoughts, feelings, and behavior in the laboratory and in life. J. Person. Soc. Psychol. 78(4), 772–790 (2000)CrossRefGoogle Scholar
  4. 4.
    Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich, J., Kong, E., Larraburo, Y., Rolle, C., Johnston, E., Gazzaley, A.: Video game training enhances cognitive control in older adults. Nature 501, 97–101 (2013)CrossRefGoogle Scholar
  5. 5.
    Baek, Y., Ko, R., Marsh, T. (eds.): Trends and Applications of Serious Gaming and Social Media. Gaming Media and Social Effects. Springer, New York (2014)Google Scholar
  6. 6.
    Bidarra, R., Gambon, D., Kooij, R., Nagel, D., Schutjes, M., Tziouvara, I.: Gaming at the Dentist’s–serious game design for pain and discomfort distraction. In: Schouten, B., Fedtke, S., Bekker, T., Schijven, M., Gekker, A. (eds.) Games for Health 2013. Proceeding of the 3rd Conference on Gaming and Playful Interaction in Healthcare, pp. 207–215. Springer, Heidelberg (2013)Google Scholar
  7. 7.
    Boinodiris, P., Fingar, P.: Serious Games for Business: Using Gamification to Fully Engage Customers, Employees and Partners. Meghan-Kiffer Press, Tampa (2014)Google Scholar
  8. 8.
    Botella, C., Breton-Lópeza, J., Queroa, S., Bañosb, R.M., García-Palaciosa, A., Zaragozac, I., Alcanizc, M.: Treating cockroach phobia using a serious game on a mobile phone and augmented reality exposure: a single case study. Comput. Hum. Behav. 27(1), 217–227 (2011). Current Research Topics in Cognitive Load Theory. Third International Cognitive Load Theory ConferenceCrossRefGoogle Scholar
  9. 9.
    Brom, C., Buchtová, M., Šisler, V., Děchtěrenko, F., Palme, R., Glenk, L.M.: Flow, social interaction anxiety and salivary cortisol responses in serious games: a quasi-experimntal study. Comput. Educ. 79, 69–100 (2014)CrossRefGoogle Scholar
  10. 10.
    Bushman, B.J., Anderson, C.: Violent video games and hostile expectations: a test of the general aggression model. Pers. Soc. Psychol. Bull. 28(12), 1679–1686 (2002)CrossRefGoogle Scholar
  11. 11.
    Campbell, D.T., Cook, D.T.: Quasy-Experimental Design and Analysis Issues for Field Setting. Rand McNally, Chicago (1979)Google Scholar
  12. 12.
    Casti, J.L.: Reality Rules, Picturing the World in Mathematics–The Fundamentals, vol. I. Wiley, New York (1997)zbMATHGoogle Scholar
  13. 13.
    Choo, A., Tong, X., Gromala, D., Hollander, A.: Virtual reality and mobius floe: cognitive distraction as non-pharmacological analgesic for pain management. In: Schouten, B., Fedtke, S., Schijven, M., Vosmeer, M., Gekker, A. (eds.) Games for Health 2014 Proceeding of the 4th conference on gaming and playful interaction in healthcare, pp. 8–12. Springer Fachmedien Wiesbaden, Heidelberg (2014)Google Scholar
  14. 14.
    Adams, K.A.: Cram101 Textbooks Reviews, Just the facts101 Textbook Key Facts: Research Methods, Statistics, and Applications. Content Technologies (2014).
  15. 15.
    De Freitas, S., Liarokapis, F.: Serious games: a new paradigm for education? In: Ma, M., Oikonomou, A., Jain, L.C. (eds.) Serious Games and Edutainment Applications, pp. 9–23. Springer, London (2011)CrossRefGoogle Scholar
  16. 16.
    De Gloria, A., Bellotti, F., Berta, R.: Serious games for education and training. Int. J. Serious Games. 1(1).
  17. 17.
    Djaouti, D., Alvarez, J., Jessel, J., Rampnoux, O.: Origins of Serious Games. In: Ma, M., Oikonomou, A., Jain, L.C. (eds.) Serious Games and Edutainment Applications, pp. 25–43. Springer, London (2011)CrossRefGoogle Scholar
  18. 18.
    Essential facts about the computer and video game industry. 2016 Sales, Demographic and Usage Data. ESA (2016).
  19. 19.
    Ethical Principles of Psychologisys and Code of Conduct. APA (2010).
  20. 20.
    Experiment. In: Merriam-Webster.
  21. 21.
    Ferguson, C., Kilburn, J.: Much ado about nothing: the misestimation and overinterpretation of violent video game effects in eastern and western nations: a comment on Anderson, et al. Psychol. Bull. 136(2), 174–178 (2010)CrossRefGoogle Scholar
  22. 22.
    Foster, J., Barkus, E., Yavorsky, C.: Understanding and Using Advanced Statistics: A Practical Guide for Students. SAGE Puclications, London, Thousand Oaks (2006)CrossRefGoogle Scholar
  23. 23.
    Girard, C., Ecalle, J., Magnan, A.: Serious games as new educational tools: how effective are they? a meta-analysis of recent studies. J. Comput. Assist. Learn. 29(3), 207–219 (2013)CrossRefGoogle Scholar
  24. 24.
    Goodwin, C.J.: Research in Psychology Methods and Design. Wiley, New York (2010)Google Scholar
  25. 25.
    Hess, M., Wiemeyer, J., Hamacher, K., Goesele, M.: Serious games for solving protein sequence alignments - combining citizen science and gaming. In: Göbel, S., Wiemeyer, J. (eds.) GameDays 2014. LNCS, vol. 8395, pp. 175–185. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  26. 26.
    Hypothesis. In: Merriam-Webster.
  27. 27.
    Kantowitz, B.H., Roediger, H.L., Elmes, D.G.: Experimental Psychlogy, vol. III. Wadsworth, Belmont (2009)Google Scholar
  28. 28.
    Kaufman, S.B.: Sex differences in mental rotation and spatial visualization ability: can they be accounted for by differences in working memory capacity? Intelligence 35, 211–223 (2007)CrossRefGoogle Scholar
  29. 29.
    King, D.L., Delfabbro, P., Griffiths, M.D.: The psychological study of video game players: methodological challenges and practical advice. Int. J. Mental Health Addict. 7(4), 555–562 (2009)CrossRefGoogle Scholar
  30. 30.
    Kornilova, T.V.: Experimental Psychology. Jurajt, Moscow (2014). (in Russian)Google Scholar
  31. 31.
    Kuhn, T.S.: The Structure of Scientific Revolutions. International Encyclopedia of United Science, vol. 2. The University of Chicago Press, Chicago (1962)Google Scholar
  32. 32.
    Lim, C., Jung, H.: A study on the military serious game. Adv. Sci. Technol. Lett. 39, 73–77 (2013). Proceedings. SERSC 2013CrossRefGoogle Scholar
  33. 33.
    Marchand, A., Hennig-Thurau, T.: Value creation in the video game industry: industry economics, consumer benefits, and research opportunities. J. Interact. Market. 27, 141–157 (2013)CrossRefGoogle Scholar
  34. 34.
    Marsden, J.: The essential checklist for making an awesome video game, According to Futurlab (2013).
  35. 35.
    Martin, D.W.: Doing Psychology Experiments. Thompson Higher Education, Belmont (2008)Google Scholar
  36. 36.
    Martins, T., Araújo, M., Carvalho, V., Soares, F., Torrão, L.: PhysioVinci – a first approach on a physical rehabilitation game. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 1–9. Springer, Heidelberg (2014)Google Scholar
  37. 37.
    Martin-SanJosé, J.-F., Juan, M.-C., Segui, I., Garcia-Garcia, I.: The effects of computer-based games and collaboration in large groups vs collaboration in pair or traditional methods. Comput. Educ. 87, 42–54 (2015)CrossRefGoogle Scholar
  38. 38.
    Mayer, I., Zhou, Q., Keijser, X., Abspoel, L.: Gaming the future of the ocean: the marine spatial planning challenge 2050. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 150–162. Springer, Heidelberg (2014)Google Scholar
  39. 39.
    Omelina, L., Jansen, B., Bonnechère, B., Van Sint Jan, S., Cornelis, J.: Serious games for physical rehabilitation: designing highly configurable and adaptable games. In: Proceeding 9th International Conference on Disability, Virtual Reality and Associated Technologies Laval, France, 10–12 September 2012 (ICDVRAT 2012), pp. 195–201. ICDVRAT (2012)Google Scholar
  40. 40.
    Operationalization. Wikipedia.
  41. 41.
    Popper, K.: Normal Science and its Dangers. In: Lakatos, I., Musgrave, A. (eds.) Criticism and the Growth of Knowledge, pp. 51–58. Cambridge University Press, Cambridge (1970)CrossRefGoogle Scholar
  42. 42.
    Popper, K.: The Logic of Scientific Discovery. Routledge, London (2002)zbMATHGoogle Scholar
  43. 43.
    Raulin, M.L., Graziano, A.M.: Quasi-experiments and correlational studies. In: Colman, A.M. (ed.) Companion Encyclopedia of Psychology, vol. 2, pp. 1124–1141. Routledge, London (1994)Google Scholar
  44. 44.
    Reliability. In: Merriam-Webster.
  45. 45.
    Ricciardi, F., De Paolis, L.T.: A Comprehensive review of serious games in health professions. Int. J. Comput. Games Technol. (2014).
  46. 46.
    Senevirathne, S.G., Kodagoda, M., Kadle, V., Haake, S.J., Senior, T., Heller, B.W.: Application of serious games to sports, health and exercise. In: Proceedings of the 6th SLIIT Research Symposium, Sri Lanka, vol. 4, pp. 6–9 (2011)Google Scholar
  47. 47.
    Shubert, T., Strobach, T.: Video game experience optimized executive control skills—on false positives and false negatives: reply to boot and simons. Acta Psychol. 141, 278–280 (2012)CrossRefGoogle Scholar
  48. 48.
    Tomé, R.M., Pereira, J.M., Oliveira, M.: Using serious games for cognitive disabilities. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 34–47. Springer, Heidelberg (2014)Google Scholar
  49. 49.
    Tong, T., Chignell, M., Tierney, M.C., Masella, C.: A Serious game for clinical assessment of cognitive status: validation study. JMIR Serious Games 4(1), e7 (2016). doi: 10.2196/games.5006 CrossRefGoogle Scholar
  50. 50.
    Validity. In: Merriam-Webster.
  51. 51.
    Van der Spek, E.D.: Experiments in serious game design: a cognitive approach. Utrecht University Repository (Dissertation). Utrecht University, Utrecht (2011)Google Scholar
  52. 52.
    Van Muijden, J., Band, G.P.H., Hommel, B.: Online games training aging brains: limited transfer to cognitive control functions. Front. Hum. Neurosci. 6, 221 (2012). doi: 10.3389/fnhum.2012.00221 Google Scholar
  53. 53.
    Wouters, P., van Nimwegen, C., van Oostendorp, H., van der Spek, E.D.: A meta-analysis of the cognitive and motivational effects of serious games. J. Educ. Psychol. 105, 249 (2013). doi: 10.1037/a0031311 CrossRefGoogle Scholar
  54. 54.
    Wrzesien, M., Alcañiz, M., Botella, C., Burkhardt, J.-M., Lopez, J.B., Ortega, A.R.: A pilot evaluation of a therapeutic game applied to small animal phobia treatment. In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 10–20. Springer, Heidelberg (2014)Google Scholar
  55. 55.
    Yoo, K., Ahn, J., Lee, W.: A design of the stress relief game based on autonomic nervous system. In: Park, J.H., Jeong, Y.-S., Park, S.O., Chen, H.-C. (eds.) EMC 2012. LNEE, vol. 181, pp. 371–376. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  56. 56.
    Zakari, H.M., Ma, M., Simmons, D.: A review of serious games for children with autism spectrum disorders (ASD). In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 93–106. Springer, Heidelberg (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Pedagogics and Medical Psychology Department, Faculty of Higher Nursing Training, Psychology and Social WorkSechenov First Moscow State Medical UniversityMoscowRussia

Personalised recommendations