Advertisement

A Quantitative Approach for Developing Serious Games for Aptitude and Trait Assessment

  • Brenton M. WiernikEmail author
  • Michael D. Coovert
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11899)

Abstract

We describe a development process for serious games to create psychometrically rigorous measures of individual aptitudes (abilities, skills) and traits (habits, tendencies, behaviors). We begin with a discussion of serious games and how they can instantiate appropriate cognitive states for relevant aptitudes and traits to manifest. This can have numerous advantages over traditional assessment modalities. We then describe the iterative approach to aptitude and trait measurement that emphasizes (1) careful definition and specification of the traits and aptitudes to be measured, (2) rigorous assessment of reliability and validity, and (3) revision of gameplay elements and metrics to improve measurement properties.

Keywords

Assessment Validity Factor analysis Personality Cognitive ability 

References

  1. 1.
    Kozlowski, S.W.J., DeShon, R.P.: A psychological fidelity approach to simulation-based training: theory, research and principles. In: Schiflett, S.G., Elliott, L.R., Salas, E., Coovert, M.D. (eds.) Scaled Worlds: Development, Validation and Applications. Routledge, London (2017).  https://doi.org/10.4324/9781315243771
  2. 2.
    Society for Industrial and Organizational Psychology, American Psychological Association: Principles for the validation and use of personnel selection procedures (Fifth edition) (2018). https://www.apa.org/ed/accreditation/about/policies/personnel-selection-procedures.pdf
  3. 3.
    Ford, J.K., Meyer, T.: Advances in training technology: meeting the workplace challenges of talent development, deep specialization, and collaborative learning. In: Coovert, M.D., Thompson, L.F. (eds.) The Psychology of Workplace Technology. Routledge, New York (2013).  https://doi.org/10.4324/9780203735565
  4. 4.
    Long, D.T., Mulch, C.M.: Interactive wargaming cyberwar: 2025 (2017). https://apps.dtic.mil/docs/citations/AD1053350
  5. 5.
    Wiemeyer, J., Hardy, S.: Serious games and motor learning: concepts, evidence, technology. In: Bredl, B., Bösche, W. (eds.) Serious Games and Virtual Worlds in Education, Professional Development, and Healthcare, pp. 197–220. IGI Global, Hershey (2013).  https://doi.org/10.4018/978-1-4666-3673-6.ch013
  6. 6.
    Wiemeyer, J., Kliem, A.: Serious games in prevention and rehabilitation—a new panacea for elderly people? Eur. Rev. Aging Phys. Act. 9, 41–50 (2012).  https://doi.org/10.1007/s11556-011-0093-x
  7. 7.
    The O*NET® Content Model. https://www.onetcenter.org/content.html
  8. 8.
    Ludoscience: A collaborative classification of serious games. http://serious.gameclassification.com/
  9. 9.
    Dörner, R., Göbel, S., Effelsberg, W., Wiemeyer, J. (eds.): Serious Games: Foundations, Concepts and Practice. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-40612-1
  10. 10.
    Coovert, M.D., Winner, J., Bennett, W.: Construct development and validation in game-based research. Simul. Gaming. 48, 236–248 (2017).  https://doi.org/10.1177/1046878116682661
  11. 11.
    Lievens, F., Patterson, F.: The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J. Appl. Psychol. 96, 927–940 (2011).  https://doi.org/10.1037/a0023496
  12. 12.
    Schell, J.: The Art of Game Design: A Book of Lenses CRC Press, Boca Raton (2008).  https://doi.org/10.1201/9780080919171
  13. 13.
    Anderson, N., Salgado, J.F., Hülsheger, U.R.: Applicant reactions in selection: comprehensive meta-analysis into reaction generalization versus situational specificity. Int. J. Sel. Assess. 18, 291–304 (2010).  https://doi.org/10.1111/j.1468-2389.2010.00512.x
  14. 14.
    Anderson, N.: Applicant and recruiter reactions to new technology in selection: a critical review and agenda for future research. Int. J. Sel. Assess. 11, 121–136 (2003).  https://doi.org/10.1111/1468-2389.00235
  15. 15.
    Gilliland, S.W.: Fairness from the applicant’s perspective: reactions to employee selection procedures. Int. J. Sel. Assess. 3, 11–18 (1995).  https://doi.org/10.1111/j.1468-2389.1995.tb00002.x
  16. 16.
    Ones, D.S., Viswesvaran, C., Reiss, A.D.: Role of social desirability in personality testing for personnel selection: the red herring. J. Appl. Psychol. 81, 660–679 (1996).  https://doi.org/10.1037/0021-9010.81.6.660
  17. 17.
    Eklöf, H.: Skill and will: test taking motivation and assessment quality. Assess. Educ. Princ. Policy Pract. 17, 345–356 (2010).  https://doi.org/10.1080/0969594X.2010.516569
  18. 18.
    McFarland, L.A., Yun, G.J., Harold, C.M., Viera, L., Moore, L.G.: An examination of impression management use and effectiveness across assessment center exercises: the role of competency demands. Pers. Psychol. 58, 949–980 (2005).  https://doi.org/10.1111/j.1744-6570.2005.00374.x
  19. 19.
    Wilson, M.A. (ed.): The Handbook of Work Analysis: Methods, Systems, Applications and Science of Work Measurement in Organizations. Routledge, New York (2013).  https://doi.org/10.4324/9780203136324
  20. 20.
    Coovert, M.D., Winner, J., Bennett, Jr., W., Howard, D.J.: Serious games are a serious tool for team research. Int. J. Serious Games. 4 (2017).  https://doi.org/10.17083/ijsg.v4i1.141
  21. 21.
    Campbell, J.P., Wiernik, B.M.: The modeling and assessment of work performance. Annu. Rev. Organ. Psychol. Organ. Behav. 2 47–74 (2015).  https://doi.org/10.1146/annurev-orgpsych-032414-111427
  22. 22.
    Wiemeyer, J., Kickmeier-Rust, M., Steiner, Christina M.: Performance assessment in serious games. In: Dörner, R., Göbel, S., Effelsberg, W., Wiemeyer, J. (eds.) Serious Games, pp. 273–302. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-40612-1_10CrossRefGoogle Scholar
  23. 23.
    Nandakumar, R.: Assessing essential unidimensionality of real data. Appl. Psychol. Meas. 17, 29–38 (1993).  https://doi.org/10.1177/014662169301700108
  24. 24.
    Gignac, G.E., Watkins, M.W.: Bifactor modeling and the estimation of model-based reliability in the WAIS-IV. Multivar. Behav. Res. 48, 639–662 (2013).  https://doi.org/10.1080/00273171.2013.804398
  25. 25.
    Coovert, M.D., McNelis, K.: Determining the number of common factors in factor analysis: a review and program. Educ. Psychol. Meas. 48, 687–692 (1988).  https://doi.org/10.1177/0013164488483012
  26. 26.
    Comrey, A.L.: A First Course in Factor Analysis, 2 edn. Psychology Press, New York (2013).  https://doi.org/10.4324/9781315827506
  27. 27.
    Reise, S.P.: The rediscovery of bifactor measurement models. Multivar. Behav. Res. 47, 667–696 (2012).  https://doi.org/10.1080/00273171.2012.715555
  28. 28.
    Wiernik, B.M., Wilmot, M.P., Kostal, J.W.: How data analysis can dominate interpretations of dominant general factors. Ind. Organ. Psychol. 8, 438–445 (2015).  https://doi.org/10.1017/iop.2015.60
  29. 29.
    Giordano, C.A., Waller, N.G.: Recovering bifactor models: a comparison of seven methods. Psychol. Methods (2019).  https://doi.org/10.1037/met0000227CrossRefGoogle Scholar
  30. 30.
    McArdle, J.J.: Latent variable modeling of differences and changes with longitudinal data. Annu. Rev. Psychol. 60, 577–605 (2009).  https://doi.org/10.1146/annurev.psych.60.110707.163612
  31. 31.
    Coovert, M., Miller, E., Bennett, Jr., W.: Assessing trust and effectiveness in virtual teams: latent growth curve and latent change score models. Soc. Sci. 6, 87 (2017).  https://doi.org/10.3390/socsci6030087
  32. 32.
    Bollen, K.A., Curran, P.J.: Latent Curve Models: A Structural Equation Perspective. Wiley, Hoboken (2005).  https://doi.org/10.1002/0471746096
  33. 33.
    Kuhn, M., Johnson, K.: Applied Predictive Modeling. Springer, New York (2013).  https://doi.org/10.1007/978-1-4614-6849-3CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of South FloridaTampaUSA

Personalised recommendations