BARGAIN: behavioral affective rule-based games adaptation interface–towards emotionally intelligent games: application on a virtual reality environment for socio-moral development

Abstract

This paper presents a framework for adapting game elements to the player’s affective state and the integration of the framework in a virtual reality environment for moral development. These game elements include gestural and facial expressions of avatars during dialogues with the player, background music, the score, game mechanics, aesthetics and learning. The framework BARGAIN (Behavioral Affective Rule-based Games Adaptation Interface) is an authoring tool for affective game design providing a visual interface based on finite state machine (FSM) technique to represent the affective rules as state transitions graph dependent on the player emotional state assessed using facial expression recognition system based on electroencephalography (EEG) data. We conducted a user study (n = 29) examining the effects of the resulting affective virtual reality game on players’ experience using the Game experience Questionnaire (GEQ) (IJsselsteijn et al. in The game experience questionnaire, Technische Universiteit Eindhoven, Eindhoven, 2013). The results show significant correlation between the GEQ dimensions and the player's facial expressions during his interaction with the Non-Player Characters (NPCs) within the VR game. These findings highlight that adapting games to user's emotions enhance the players’ experience.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30

Notes

  1. 1.

    www.oculus.com.

  2. 2.

    www.emotiv.com.

  3. 3.

    We have not used Emotiv affective suite that detects facial states like: blink, frown, raised eyebrows, wink and smile.

  4. 4.

    https://imotions.com/emotient/.

  5. 5.

    Unity's Prefab allows to create, configure and store a gameobject complete with all its components, property values and child gameobjects.

  6. 6.

    Very joyful Music: https://www.playonloop.com/2017-music-loops/doggo-brothers/.

  7. 7.

    Moderately joyful Music: www.audiomicro.com/feelin-good-wav-royalty-free-stock-music-1438027.

  8. 8.

    Less joyful Music: https://freesound.org/people/axtoncrolley/sounds/172707/.

  9. 9.

    https://en.wikipedia.org/wiki/Bonferroni_correction.

  10. 10.

    https://jasp-stats.org/.

  11. 11.

    https://en.wikipedia.org/wiki/Beta_distribution#Jeffreys'_prior_probability_(Beta(1/2,1/2)_for_a_Bernoulli_or_for_a_binomial_distribution).

  12. 12.

    https://forum.cogsci.nl/index.php?p=/discussion/2161/setting-priors-in-jasp.

References

  1. Abdessalem, H.B., Frasson, C.: Real-Time Brain Assessment for Adaptive Virtual Reality Game: a Neurofeedback Approach. Springer, Cham (2017)

    Google Scholar 

  2. Ahmad, I., Hamid, E., Abdullasim, N., Jaafar, A.: Game Interface Design: Measuring the Player’s Gameplay Experience. Springer, Cham (2017)

    Google Scholar 

  3. Arellano, D.G., Tokarchuk, L., Gunes, H.: Measuring Affective, Physiological and Behavioural Differences in Solo, Competitive and Collaborative Games. Springer, Cham (2016)

    Google Scholar 

  4. Bekinschtein, T., Niklison, J., Sigman, L., Manes, F., Leiguarda, R., Armony, J., Owen, A., Carpintiero, S., Olmos, L.: Emotion processing in the minimally conscious state. J Neurol Neurosurg Psychiatry 75(5), 788–788 (2004)

    Article  Google Scholar 

  5. Benlamine, M., Chaouachi, M., Frasson, C., Dufresne, A.: Physiology-based recognition of facial micro-expressions using EEG and identification of the relevant sensors by emotion. In: Proceedings of the 3rd International Conference on Physiological Computing Systems—Volume 1: PhyCS (2016a)

  6. Benlamine, M.S., Chaouachi, M., Frasson, C., Dufresne, A.: Predicting Spontaneous Facial Expressions from EEG. Springer, Berlin (2016b)

    Google Scholar 

  7. Benlamine, M.S., Villata, S., Ghali, R., Frasson, C., Gandon, F., Cabrio, E.: Persuasive Argumentation and Emotions: An Empirical Evaluation with Users. Springer, Cham (2017)

    Google Scholar 

  8. Bernhaupt, R., Eckschlager, M., Tscheligi, M.: Methods for Evaluating Games: How to Measure Usability and User Experience in Games? ACM, New York (2007)

    Google Scholar 

  9. Böckle, M., Micheel, I., Bick, M., Novak, J.: A Design Framework for Adaptive Gamification Applications. Proceedings of the 51st Hawaii international conference on system sciences (HICSS 2018), 1227–1236 (2018)

  10. Bontchev, B.: Adaptation in affective video games: a literature review. Cybern Inf Technol 16(3), 3–34 (2016)

    MathSciNet  Google Scholar 

  11. Chaouachi, M., Frasson, C.: Mental Workload, Engagement and Emotions: An Exploratory Study for Intelligent Tutoring Systems. Springer, Berlin (2012)

    Google Scholar 

  12. Chaouachi, M., Chalfoun, P., Jraidi, I., Frasson, C.: Affect and Mental Engagement: Towards Adaptability for Intelligent Systems. AAAI Press, Palo Alto (2010)

    Google Scholar 

  13. Chaouachi, M., Jraidi, I., Frasson, C.: MENTOR: A Physiologically Controlled Tutoring System. Springer, Cham (2015)

    Google Scholar 

  14. Chiasson, V., Vera-Estay, E., Lalonde, G., Dooley, J., Beauchamp, M.: Assessing social cognition: age-related changes in moral reasoning in childhood and adolescence. Clin Neuropsychol 31(3), 515–530 (2017)

    Article  Google Scholar 

  15. Choi, J., Peters, M., Mueller, R.O.: Correlational analysis of ordinal data: from Pearson’s r to Bayesian polychoric correlation. Asia Pac Educ Rev 11(4), 459–466 (2010)

    Article  Google Scholar 

  16. Christy, T., Kuncheva, L.I.: Technological advancements in affective gaming: a historical survey. GSTF J Comput (JoC) 3(4), 32–41 (2018)

    Google Scholar 

  17. de Byl, P.: A conceptual affective design framework for the use of emotions in computer game design. Cyberpsychol J Psychosoc Res Cyberspace 9(3), 4 (2015). https://doi.org/10.5817/CP2015-3-4

    Article  Google Scholar 

  18. Dekker, A., Champion, E.: Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game using Biofeedback. In: Proceedings of DiGRA 2007 Conference: Situated Play, 550–558 (2007)

  19. Drachen, A., Nacke, L.E., Yannakakis, G., Pedersen, A.L.: Correlation Between Heart Rate, Electrodermal Activity and Player Experience in First-Person Shooter Games. ACM, New York (2010)

    Google Scholar 

  20. Fairclough, S.H.: Physiological Computing: Interfacing with the Human Nervous System. Springer, New York (2010)

    Google Scholar 

  21. Garner, T.A.: Game Sound from Behind the Sofa: An Exploration into the Fear Potential of Sound and Psychophysiological Approaches to Audio-Centric, Adaptive Gameplay. Ph D Thesis, Aalborg University (2013)

  22. Gibbs, J.C.: Moral Development and Reality: Beyond the Theories of Kohlberg, Hoffman, and Haidt. Oxford University Press, Oxford (2013)

    Google Scholar 

  23. Goldberg, L.R.: The development of markers for the Big-Five factor structure. Psychol Assess 4(1), 26 (1992)

    Article  Google Scholar 

  24. Hagemann, D., Naumann, E., Thayer, J.F., Bartussek, D.: Does resting electroencephalograph asymmetry reflect a trait? An application of latent state-trait theory. J Personal Soc Psychol 82(4), 619 (2002)

    Article  Google Scholar 

  25. Hamari, J., Shernoff, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T.: Challenging games help students learn: an empirical study on engagement, flow and immersion in game-based learning. Comput Hum Behav 54, 170–179 (2016)

    Article  Google Scholar 

  26. Harmon-Jones, E., Gable, P.A.: On the role of asymmetric frontal cortical activity in approach and withdrawal motivation: an updated review of the evidence. Psychophysiology 55(1), e12879 (2018)

    Article  Google Scholar 

  27. Harmon-Jones, E., Gable, P.A., Peterson, C.K.: The role of asymmetric frontal cortical activity in emotion-related phenomena: a review and update. Biol Psychol 84(3), 451–462 (2010)

    Article  Google Scholar 

  28. i Badia, S.B., Quintero, L.V., Cameirao, M.S., Chirico, A., Triberti, S., Cipresso, P., Gaggioli, A.: Towards emotionally-adaptive virtual reality for mental health applications. IEEE J Biomed Health Inform 23(5), 1877–1887 (2018)

    Article  Google Scholar 

  29. IJsselsteijn, W., De Kort, Y., Poels, K.: The Game Experience Questionnaire. Technische Universiteit Eindhoven, Eindhoven (2013)

  30. Kataria, M.: Confirmation: What’s in the evidence? J Behav Exp Econ 65, 9–15 (2016)

    Article  Google Scholar 

  31. Kosunen, I., Salminen, M., Järvelä, S., Ruonala, A., Ravaja, N., Jacucci, G.: RelaWorld: Neuroadaptive and Immersive Virtual Reality Meditation System. ACM, New York (2016)

    Google Scholar 

  32. Lehman, B., Matthews, M., D’Mello, S., Person, N.: What are You Feeling? Investigating Student Affective States During Expert Human Tutoring Sessions. Springer, Berlin (2008)

    Google Scholar 

  33. Molho, C., Tybur, J.M., Güler, E., Balliet, D., Hofmann, W.: Disgust and anger relate to different aggressive responses to moral violations. Psychol Sci 28(5), 609–619 (2017)

    Article  Google Scholar 

  34. Mulder, J.: Bayes factors for testing order-constrained hypotheses on correlations. J Math Psychol 72, 104–115 (2016)

    MathSciNet  Article  Google Scholar 

  35. Nacke, L., Lindley, C.A.: Flow and Immersion in First-Person Shooters: Measuring the Player’s Gameplay Experience. ACM, New York (2008)

    Google Scholar 

  36. Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L.: Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction. ACM, New York (2011)

    Google Scholar 

  37. Nogueira, P.A., Rodrigues, R.A., Oliveira, E.C., Nacke, L.E.: Guided Emotional State Regulation: Understanding and Shaping Players' Affective Experiences in Digital Games. In: The 9th Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE-13), AAAI Press, 51–58 (2013)

  38. Philip, L., Martin, J.-C., Clavel, C.: Rapid facial reactions in response to facial expressions of emotion displayed by real versus virtual faces. i-Perception 9(4), 2041669518786527 (2018)

    Article  Google Scholar 

  39. Picard, R.W.: Affective Computing. MIT press, Cambridge (1997)

    Google Scholar 

  40. Pope, A.T., Bogart, E.H., Bartolome, D.S.: Biocybernetic system evaluates indices of operator engagement in automated task. Biol Psychol 40(1–2), 187–195 (1995)

    Article  Google Scholar 

  41. Putman, P., van Peer, J., Maimari, I., van der Werff, S.: EEG theta/beta ratio in relation to fear-modulated response-inhibition, attentional control, and affective traits. Biol Psychol 83(2), 73–78 (2010)

    Article  Google Scholar 

  42. Quintana, D.S., Williams, D.R.: Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP. BMC Psychiatry 18(1), 178 (2018)

    Article  Google Scholar 

  43. Simpson, J., Carter, S., Anthony, S.H., Overton, P.G.: Is disgust a homogeneous emotion? Motiv Emot 30(1), 31–41 (2006)

    Article  Google Scholar 

  44. Sitaram, R., Lee, S., Ruiz, S., Rana, M., Veit, R., Birbaumer, N.: Real-time support vector classification and feedback of multiple emotional brain states. Neuroimage 56(2), 753–765 (2011)

    Article  Google Scholar 

  45. Tato, A.A.N., Nkambou, R., Dufresne, A., Beauchamp, M.H.: Convolutional neural network for automatic detection of sociomoral reasoning level. In: Proceedings of the 10th International Conference on Educational Data Mining (EDM2017), 284–289 (2017)

  46. Tato, A.A.N., Dufresne, A., Nkambou, R., Morasse, F., Beauchamp, M.H.: Preliminary Evaluation of a Serious Game for Socio-Moral Reasoning. Springer, Berlin (2018)

    Google Scholar 

  47. Villata, S., Cabrio, E., Jraidi, I., Benlamine, S., Chaouachi, M., Frasson, C., Gandon, F.: Emotions and personality traits in argumentation: an empirical evaluation 1. Argum Comput 8(1), 61–87 (2017)

    Article  Google Scholar 

  48. Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., Gronau, Q.F., Dropmann, D., Boutin, B.: Bayesian inference for psychology. Part II: Example applications with JASP. Psychon Bull Rev 25(1), 58–76 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

The authors acknowledge support of the NSERC (Natural Sciences and Engineering Research Council of Canada), the FRQSC (Fonds de Recherche du Québec Société et Culture) and BMU Labs for funding this research.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mohamed S. Benlamine.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Benlamine, M.S., Dufresne, A., Beauchamp, M.H. et al. BARGAIN: behavioral affective rule-based games adaptation interface–towards emotionally intelligent games: application on a virtual reality environment for socio-moral development. User Model User-Adap Inter (2021). https://doi.org/10.1007/s11257-020-09286-0

Download citation

Keywords

  • Authoring tool
  • Adaptive games
  • Affective computing
  • VR