Skip to main content

An empirical study of players’ emotions in VR racing games based on a dataset of physiological data

Multimedia Tools and Applications volume 79, pages 33657–33686 (2020)Cite this article

Abstract

A video game is an interactive software able to arouse intense emotions in players. Consequentially, different theories have been proposed to understand which game aspects are able to affect the players’ emotional state. However, only few works have tried to use empirical evidence to investigate the effects of different game aspects of the players’ emotions. In this paper, we present the results of a set of experiments aimed at predicting the players’ emotions during video games sessions using their physiological data. We have created a physiological dataset from the data acquired by 33 participants during video game fruition using a standard monitor and a Virtual Reality headset. The dataset contains information about electrocardiogram, 5 facials electromyographies, electrodermal activity, and respiration. Furthermore, we have asked the players to self-assess their emotional state on the Arousal and Valence space. We have then analyzed the contribution of each physiological signal to the overall definition of the players’ mental state. Finally, we have applied Machine Learning techniques to predict the emotional state of players during game sessions at a precision of one second. The obtained results can contribute to define game devices and engines able to detect physiological data, as well to improve the game design process.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Notes

  1. https://www.projectcarsgame.com

  2. https://34bigthings.com/portfolio/redout/

  3. https://github.com/grano00/GameVRRacingPhysioDB

References

  1. Abadi MK, Subramanian R, Kia SM, Avesani P, Patras I, Sebe N (2015) Decaf: meg-based multimodal database for decoding affective physiological responses. IEEE Trans Affect Comput 6(3):209–222

    Article  Google Scholar 

  2. Angie AD, Connelly S, Waples EP, Kligyte V (2011) The influence of discrete emotions on judgement and decision-making: a meta-analytic review. Cogn Emotion 25(8):1393–1422

    Article  Google Scholar 

  3. Bartle RA (2003) Designing virtual worlds. New Riders

  4. Benedek M, Kaernbach C (2010) A continuous measure of phasic electrodermal activity. J Neurosci Methods 190(1):80–91

    Article  Google Scholar 

  5. Betella A, Verschure P (2016) The affective slider: a digital self-assessment scale for the measurement of human emotions. PLoS ONE, 11(2)

  6. Boccignone G, Conte D, Cuculo V, Lanzarotti R (2017) AMHUSE: a multimodal dataset for humour sensing. In: Proceedings of the 19th ACM international conference on multimodal interaction. ACM, pp 438–445

  7. Boccignone G, Cuculo V, Grossi G, Lanzarotti R, Migliaccio R (2017) Virtual EMG via facial video analysis. In: International conference on image analysis and processing. Springer, pp 197–207

  8. Boccignone G, Conte D, Cuculo V, D’Amelio A, Grossi G, Lanzarotti R (2018) Deep construction of an affective latent space via multimodal enactment. IEEE Trans Cogn Develop Syst 10(4):865–880

    Article  Google Scholar 

  9. Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Therapy Exper Psych 25(1):49–59

    Article  Google Scholar 

  10. Braithwaite JJ, Watson DG, Jones R, Rowe M (2013) A guide for analysing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments. Psychophysiology 49(1):1017–1034

    Google Scholar 

  11. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    MATH  Article  Google Scholar 

  12. Castellar EN, Oksanen K, Van Looy J (2014) Assessing game experience: heart rate variability, in-game behavior and self-report measures. In: 2014 Sixth international workshop on quality of multimedia experience (QoMEX). IEEE, pp 292–296

  13. Caywood MS, Roberts DM, Colombe JB, Greenwald HS, Weiland MZ (2017) Gaussian process regression for predictive but interpretable machine learning models: an example of predicting mental workload across tasks. Frontiers Human Neurosci 10:647

    Article  Google Scholar 

  14. Chandrashekar G, Sahin F (2014) A survey on feature selection methods. Comput Electr Eng 40(1):16–28

    Article  Google Scholar 

  15. Clynes M (1978) Sentics: the touch of the emotions. Anchor Press/Doubleday

  16. Conover MB (2003) Understanding electrocardiography. Elsevier Health Sciences

  17. Critchley HD (2002) Electrodermal responses: what happens in the brain. Neuroscientist 8(2):132–142

    Article  Google Scholar 

  18. Csikszentmihalyi M, Nakamura J (1979) The concept of flow. Play Learn, 257–274

  19. Dalgleish T (20za04) The emotional brain. Nat Rev Neurosci 5(7):583

  20. Darwin C (1872) The expression of the emotions in man and animals. Chicago University Press, USA

    Book  Google Scholar 

  21. De Felice D, Granato M, Ripamonti L, Trubian M, Gadia D, Maggiorini D (2017) Effect of different looting systems on the behavior of players in a MMOG: simulation with real data, vol 181 LNICST

  22. Deterding S, Dixon D, Khaled R, Nacke L (2011) From game design elements to gamefulness: defining gamification. In: Proceedings of the 15th international academic MindTrek conference: envisioning future media environments. ACM, pp 9–15

  23. Donoho DL, et al. (2000) High-dimensional data analysis: the curses and blessings of dimensionality. AMS Math Challenges Lect 1:32

    Google Scholar 

  24. Draper JV, Kaber DB, Usher JM (1998) Telepresence. Human Factors 40 (3):354–375

    Article  Google Scholar 

  25. Drucker H, Burges CJ, Kaufman L, Smola AJ, Vapnik V (1997) Support vector regression machines. In: Advances in neural information processing systems, pp 155–161

  26. Dubin D (2000) Rapid interpretation of EKG’s, 6th edn. Cover Pub Co

  27. Ekman P (1971) Universals and cultural differences in facial expressions of emotion. In: Nebraska symposium on motivation. University of Nebraska Press, pp 207–282

  28. Englehart K, Hudgins B (2003) A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans Biomed Eng 50(7):848–854

    Article  Google Scholar 

  29. Fedotov A (2016) Selection of parameters of bandpass filtering of the ECG signal for heart rhythm monitoring systems. Biomed Eng 50(2):114–118

    Article  Google Scholar 

  30. Fernandes T, Chec A, Olczak D, Ferreira H (2015) Physiological computing gaming: use of electrocardiogram as an input for video gaming. In: Proceedings of the international conference on physiological computing systems (PhyCS), SCITEPRESS (Science and Technology Publications, Lda), Angers, pp 11–13

  31. Folgieri R, Lucchiari C, Granato M, Grechi D (2014) Brain, technology and creativity. Brainart: a BCI-based entertainment tool to enact creativity and create drawing from cerebral rhythms. In: Digital Da Vinci. Springer, pp 65–97

  32. Fourati N, Pelachaud C (2018) Perception of emotions and body movement in the emilya database. IEEE Trans Affect Comput 9(1):90–101

    Article  Google Scholar 

  33. Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat, 1189–1232

  34. Freeman DE (2003) Creating emotion in games: the craft and art of emotioneering. New Riders

  35. Fullerton T (2014) Game design workshop: a playcentric approach to creating innovative games. CRC Press

  36. Furnham A (1986) Response bias, social desirability and dissimulation. Person Individ Diff 7(3):385–400

    Article  Google Scholar 

  37. Gadia D, Garipoli G, Bonanomi C, Albani L, Rizzi A (2014) Assessing stereo blindness and stereo acuity on digital displays. Displays 35(4):206–212

    Article  Google Scholar 

  38. Gadia D, Granato M, Maggiorini D, Ripamonti L, Vismara C (2018) Consumer-oriented head mounted displays: analysis and evaluation of stereoscopic characteristics and user preferences. Mob Netw Appl 23(1):136–146

    Article  Google Scholar 

  39. Granato M, Gadia D, Maggiorini D, Ripamonti L (2017) Emotions detection through the analysis of physiological information during video games fruition. In: Springer lecture notes in computer science (proceedings of 6th international conference of games and learning alliance - GALA 2017), vol 10653, pp 197–207

  40. Granato M, Gadia D, Maggiorini D, Ripamonti L (2018) Feature extraction and selection for real-time emotion recognition in video games players. In: Proceedings - 14th international conference on signal-image technology and internet-based systems, SITIS 2018. IEEE

  41. Granato M, Gadia D, Maggiorini D, Ripamonti L (2018) Software and hardware setup for emotion recognition during video game fruition. In: Proceedings of the 4th EAI international conference on smart objects and technologies for social good, Goodtechs ’18. ACM, New York, pp 19–24

  42. Granic I, Lobel A, Engels RC (2014) The benefits of playing video games. Am Psychol 69(1):66

    Article  Google Scholar 

  43. Grodal T (2000) Video games and the pleasures of control. Media Entertainment: The Psychology of its Appeal, 197–213

  44. Grossi G, Lanzarotti R, Lin J (2016) Robust face recognition providing the identity and its reliability degree combining sparse representation and multiple features. Int J Pattern Recogn Artif Intell 30(10):1656007

    Article  Google Scholar 

  45. Hamilton P (2002) Open source ECG analysis. In: Computers in cardiology, 2002. IEEE, pp 101–104

  46. Han JS, Song WK, Kim JS, Bang WC, Lee H, Bien Z (2000) New emg pattern recognition based on soft computing techniques and its application to control of a rehabilitation robotic arm. In: Proc. of 6th international conference on soft computing (IIZUKA2000), pp 890–897

  47. Hazlett RL (2006) Measuring emotional valence during interactive experiences: boys at video game play. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1023–1026

  48. Herrmann O (1970) Design of nonrecursive digital filters with linear phase. Electron Lett 6(11):328–329

    Article  Google Scholar 

  49. Ho TK (1995) Random decision forests. In: Proceedings of the Third international conference on document analysis and recognition, 1995, vol 1. IEEE, pp 278–282

  50. Huang HP, Chen CY (1999) Development of a myoelectric discrimination system for a multi-degree prosthetic hand. In: 1999 IEEE International conference on robotics and automation, 1999. Proceedings, vol 3. IEEE, pp 2392–2397

  51. Huizinga J (2014) Homo Ludens: a study of the play-element in culture. Martino Fine Books

  52. Isbister K (2016) How games move us: emotion by design. Mit Press

  53. Knutas A, Van Roy R, Hynninen T, Granato M, Kasurinen J, Ikonen J (2019) A process for designing algorithm-based personalized gamification. Multimed Tools Appl 78(10):13593–13612

    Article  Google Scholar 

  54. Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  55. Kohavi R, et al. (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol 14, Stanford, pp 1137–1145

  56. Koster R (2013) Theory of fun for game design. O’Reilly Media Inc.

  57. Kreyszig E (2011) Advanced engineering mathematics. Wiley

  58. Lazzaro N (2004) Why we play games: four keys to more emotion without story. Games Developer Conference

  59. Lykken DT, Venables PH (1971) Direct measurement of skin conductance: a proposal for standardization. Psychophysiology 8(5):656–672

    Article  Google Scholar 

  60. Lyons RG (2011) Understanding digital signal processing. Pearson Education India

  61. Mauss IB, Robinson MD (2009) Measures of emotion: a review. Cogn Emot 23(2):209–237

    Article  Google Scholar 

  62. Menze BH, Kelm BM, Masuch R, Himmelreich U, Bachert P, Petrich W, Hamprecht FA (2009) A comparison of random forest and its gini importance with standard chemometric methods for the feature selection and classification of spectral data. BMC Bioinform 10(1):213

    Article  Google Scholar 

  63. Oshiro TM, Perez PS, Baranauskas JA (2012) How many trees in a random forest?. In: International workshop on machine learning and data mining in pattern recognition. Springer, pp 154–168

  64. Oskoei MA, Hu H (2006) GA-based feature subset selection for myoelectric classification. In: IEEE International conference on robotics and biomimetics, 2006. ROBIO’06. IEEE, pp 1465–1470

  65. Parnandi A, Gutierrez-Osuna R (2018) Visual biofeedback and game adaptation in relaxation skill transfer. IEEE Trans Affect Comput, 1–1

  66. Phinyomark A, Limsakul C, Phukpattaranont P (2009) A novel feature extraction for robust EMG pattern recognition. J Comput 1(1):71–80

    Google Scholar 

  67. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23 (10):1175–1191

    Article  Google Scholar 

  68. Ravaja N, Salminen M, Holopainen J, Saari T, Laarni J, Järvinen A (2004) Emotional response patterns and sense of presence during video games: potential criterion variables for game design. In: Proceedings of the third Nordic conference on human-computer interaction. ACM, pp 339–347

  69. Rechy-Ramirez EJ, Hu H (2011) Stages for developing control systems using EMG and EEG signals: a survey. School of Computer Science and Electronic Engineering. University of Essex, pp 1744–8050

  70. Ringeval F, Sonderegger A, Sauer J, Lalanne D (2013) Introducing the recola multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE International conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–8

  71. Ripamonti L, Granato M, Trubian M, Knutas A, Gadia D, Maggiorini D (2018) Multi-agent simulations for the evaluation of looting systems design in MMOG and MOBA games. Simul Model Pract Theory 83:124–148

    Article  Google Scholar 

  72. Rukavina S, Gruss S, Walter S, Hoffmann H, Traue HC (2015) Open_emorec_ii-a multimodal corpus of human-computer interaction. Int J Comput Electr Autom Control Inform Eng 9(5):977–983

    Google Scholar 

  73. Russell JA (1991) Culture and the categorization of emotions. Psychol Bull 110(3):426–450

    Article  Google Scholar 

  74. Russell JA, Mehrabian A (1977) Evidence for a three-factor theory of emotions. J Res Person 11(3):273–294

    Article  Google Scholar 

  75. Schmidt S, Walach H (1999) Electrodermal activity (eda): state-of-the-art measurement and techniques for parapsychological purposes. J Parapsychol 63 (3):221–221

    Google Scholar 

  76. Sheffield B (2013) The journey to create journey – the quest for emotion. https://www.gamasutra.com/view/news/186282/The_journey_to_create_Journey_the_quest_for_emotion.php

  77. Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems, pp 2951–2959

  78. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42–55

    Article  Google Scholar 

  79. Stojanović R, Čaplánová A, Kovačević Ž, Nemanja N, Bundalo Z (2015) Alternative approach to addressing infrastructure needs in biomedical engineering programs (case of emerging economies). Folia Medica Facultatis Medicinae Universitatis Saraeviensis 50(1):29–33

    Google Scholar 

  80. Tognetti S, Garbarino M, Bonarini A, Matteucci M (2010) Modeling enjoyment preference from physiological responses in a car racing game. In: 2010 IEEE Symposium on computational intelligence and games (CIG). IEEE, pp 321–328

  81. Van Boxtel A (2010) Facial EMG as a tool for inferring affective states. In: Proceedings of measuring behavior. Noldus Information Technology Wageningen, pp 104–108

  82. Venables PH, Christie MJ (1980) Electrodermal activity. Techniques Psychophysiol, 54(3)

  83. Vishay (2017) Datasheet. NTCLE203E3...SB0. http://www.vishay.com/docs/29118/ntcle203.pdf

  84. Vismara C, Granato M, Ripamonti L, Maggiorini D, Gadia D (2017) Analysis of stereoscopic visualization in a consumer-oriented head mounted display. In: Lecture notes of the institute for computer sciences, social-informatics and telecommunications engineering, LNICST, vol 195 LNICST, pp 274–283

  85. Williams CK, Rasmussen CE (2006) Gaussian processes for machine learning the. MIT Press

  86. Zafar M, Ahmed B, Al-Rihawi R, Gutierrez-Osuna R (2018) Gaming away stress: using biofeedback games to learn paced breathing. IEEE Transactions on Affective Computing - In Press

Download references

Acknowledgments

The authors want to thank 34BigThings for having provided a sample copy of RedOut. We also thank Prof. Nicolò Cesa-Bianchi, Dr. Vittorio Cuculo, and Prof. Giuseppe Boccignone for their comments and suggestions, that greatly improved the overall research.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Milan, Via Celoria 18, 20133, Milan, Italy

    Marco Granato, Davide Gadia, Dario Maggiorini & Laura A. Ripamonti

Authors
  1. Marco Granato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Davide Gadia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dario Maggiorini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laura A. Ripamonti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Granato.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Granato, M., Gadia, D., Maggiorini, D. et al. An empirical study of players’ emotions in VR racing games based on a dataset of physiological data. Multimed Tools Appl 79, 33657–33686 (2020). https://doi.org/10.1007/s11042-019-08585-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08585-y

Keywords

  • Affective computing
  • Video games
  • Emotions recognition
  • ECG
  • EMG
  • GSR
  • EDA
  • Respiration
  • Physiological dataset
  • Valence
  • Arousal
  • Machine learning
  • Virtual reality
  • Players’ emotions