Evaluating the Validity of a Non-invasive Assessment Procedure

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7528)


Recent developments in serious games allow for in-game adaptations to enhance the learner´s current cognitive, motivational or emotional state. Providing suitable adaptations requires a valid assessment of the psycho-pedagogical constructs the game should adapt to. An explicit assessment, e.g. by questionnaires occurring repeatedly on the screen, would impair the learner´s game flow. Therefore, a non-invasive and implicit assessment procedure is required. In the course of the European research project TARGET, we established an assessment procedure which is based on the interpretation of the learner´s actions in the virtual environment, calledBehavioural Indicators (BIs). A set of 16 BIs has been formulated to assess the learner´s current emotional, motivational and clearness state. In this present work, we describe how these BIs can be validated and focus on the innovative elements of the methodological procedure, the material, experiential considerations and the statistical analysis to be applied in an empirical study.


Evaluation Validation Non-invasive Assessment Motivation Emotion Problem Solving 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Csikszentmihalyi, M.: Flow: The psychology of optimal experience. Harper & Collins, New York (1991)Google Scholar
  2. 2.
    Kopeinik, S., Bedek, M.A., Seitlinger, P.C., Albert, D.: The Artificial Mentor: An assessment based approach to adaptively enhance learning processes in virtual learning environments. In: Proceedings of the 19th International Conference on Computers in Education, pp. 106–110. Asia-Pacific Society for Computers in Education, Chiang Mai (2011)Google Scholar
  3. 3.
    Bedek, M.A., Seitlinger, P., Kopeinik, S., Albert, D.: Multivariate Assessment of Motivation and Emotion in Digital Educational Games. In: Proceedings of the 5th European Conference on Games-Based Learning, pp. 18–25. ACI, Athens (2011)Google Scholar
  4. 4.
    Atkinson, J.W.: Motivational determinants of risk-taking behavior. Psychological Review 64(6), 359–372 (1957)CrossRefGoogle Scholar
  5. 5.
    Covington, M.V., Omelich, C.L.: Need achievement revisited: verification of Atkinson’s original 2 x 2 model. In: Spielberger, C.D., Sarason, I.G., Kulcsar, Z., Van Heck, G.L. (eds.) Stress and Emotion: Anxiety, Anger and Curiosity, pp. 85–105. Hemisphere, New York (1991)Google Scholar
  6. 6.
    Larsen, R.J., Diener, E.: Promises and problems with the circumplex model of emotion. In: Clark, M.S. (ed.) Review of Personality and Social Psychology: Emotion, pp. 25–29. Sage, Newbury Park (1992)Google Scholar
  7. 7.
    Insko, B.E.: Measuring Presence: Subjective, behavioral and physiological methods. In: Riva, G., Davide, F., Ijsselsteijn, W.A. (eds.) Being There: Concepts, Effects and Measurement of User Presence in Synthetic Environments, pp. 110–118. IOS Press, Amsterdam (2003)Google Scholar
  8. 8.
    Van Reekum, C.M., Johnstone, T., Banse, R., Etter, A., Wehrle, T., Scherer, K.R.: Psychophysiological responses to appraisal dimensions in a computer game. Cognition and Emotion 18(5), 663–688 (2004)CrossRefGoogle Scholar
  9. 9.
    Cai, H., Lin, Y.: Modeling of operators’ emotion and task performance in a virtual driving environment. International Journal of Human-Computer Studies 69(9), 571–586 (2011)CrossRefGoogle Scholar
  10. 10.
    Lang, P.J.: Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski, J.B., Johnson, J.H., Williams, T.A. (eds.) Technology in Mental Health Care Delivery System, pp. 119–137. Ablex, Norwood (1980)Google Scholar
  11. 11.
    Rheinberg, F., Vollmeyer, R., Burns, B.D.: FAM: Ein Fragebogen zur Er-fassung aktueller Motivation in Lern- und Leistungssituationen. Diagnostica 2, 57–66 (2001)CrossRefGoogle Scholar
  12. 12.
    Peter, C., Herbon, A.: Emotion representation and physiology assignments in digital systems. Interacting with Computers 18, 139–170 (2006)CrossRefGoogle Scholar
  13. 13.
    Lane, R.D., Nadel, L.: Cognitive Neuroscience of Emotion. Oxford University Press, New York (2000)Google Scholar
  14. 14.
    Mehrabian, A., Russell, J.A.: An approach to environmental psychology. MIT Press, Cambridge (1974)Google Scholar
  15. 15.
    Bradley, M.M., Lang, P.J.: Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavioral Therapy & Experimental Psychiatry 25, 49–59 (1994)CrossRefGoogle Scholar
  16. 16.
    Sokolowski, K., Schmalt, H.D., Langens, T.A., Puca, R.M.: Assessing Achievement, Affiliation, and Power Motives All at Once: The Multi-Motive Grid (MMG). Journal of Personality Assessment 74, 126–145 (2000)CrossRefGoogle Scholar
  17. 17.
    Ziegler, M., Schmukle, S., Egloff, B., Bühner, M.: Investigating Measures of Achievement Motivation. Journal of Individual Differences 31, 15–21 (2007)CrossRefGoogle Scholar
  18. 18.
    McClelland, D.C.: Biological aspects of human motivation. In: Halisch, F., Kulh, J. (eds.) Motivation, Intention, and Volition, pp. 11–19. Springer, Heidelberg (1987)CrossRefGoogle Scholar
  19. 19.
    Spangler, W.D.: Validity of questionnaire and TAT measures of need for achievement: Two meta-analyses. Psychological Bulletin 112, 140–154 (1992)CrossRefGoogle Scholar
  20. 20.
    Schmalt, H.D.: The measurement of the Achievement Motivation. Hogrefe, Göttingen (1976)Google Scholar
  21. 21.
    Puca, R.M., Schmalt, H.D.: Task enjoyment: A mediator between achievement motives and performance. Motivation and Emotion 23, 15–29 (1999)CrossRefGoogle Scholar
  22. 22.
    Sokolowski, K., Kehr, H.: Zum differentiellen Einfluss von Motivation auf Führungstrainings (MbO). Zeitschrift für Differentielle und Diagnostische Psychologie 20, 192–202 (1999)CrossRefGoogle Scholar
  23. 23.
    Schmalt, H.D., Langens, T.A.: Projective, semiprojective and self-report measures of human motivation predict private cognitive events: Strivings, memories and daydreams., University of Wuppertal (2001) (unpubl. manuscript)Google Scholar
  24. 24.
    Pretz, J.E., Naples, A.J., Sternberg, R.J.: Recognizing, Defining, and Repre-senting Problems. In: Davidson, J.E., Steinberg, R.J. (eds.) The Psychology of Problem Solving, pp. 3–31. Cambridge University Press, Cambridge (2003)CrossRefGoogle Scholar
  25. 25.
    Pirolli, P., Card, S.K.: Information Foraging. Psychological Review 106, 643–675 (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Knowledge Management InstituteGraz University of TechnologyGrazAustria

Personalised recommendations