Skip to main content

Non-intrusive Measurement of Player Engagement and Emotions - Real-Time Deep Neural Network Analysis of Facial Expressions During Game Play

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12211)

Abstract

Prior research suggests and reveals that there is a correlation between human emotional responses and the subjective qualities of digital interactive experiences. Using facial analysis done by deep neural networks presents a true non-intrusive way of measuring emotional responses and engagement assessed as the desire to continue playing. This paper proposes a tool to measure emotional responses across eight different emotions and in real time of any game. The emotional recognition system achieves an accuracy of 98% and the continuation desire system achieves 93.3% accuracy in a pilot test with a two player game and 78.5% accuracy in a single player game. This forms a strong tool that shows a correlation between emotions and the continuation desire of a player, which can be used to evaluate engagement in games and digital interactive experiences, e.g. in critical stages of development of said content.

Keywords

  • Continuation desire
  • Conation
  • Emotion
  • Machine learning
  • Facial expressions
  • Design tools/technologies
  • Game and flow
  • Game immersion
  • Player engagement assessment

Supported by Samsung Media Innovation Lab for Education (SMILE Lab).

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-50164-8_24
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-50164-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Fig. 1.

Source: [48]

Fig. 2.

Source: [46]

Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

Notes

  1. 1.

    https://www.playstation.com/en-us/games/littlebigplanet-3-ps4/.

  2. 2.

    John Cooney, 2009 (jmtb02) http://www.jmtb02.com/this-is-the-only-level.

References

  1. Nonverbal Communication: Science and Applications AU - Matsumoto, David AU - Hwang, Hyi Sung. SAGE Publications, Inc., Thousand Oaks (2013). https://doi.org/10.4135/9781452244037. http://sk.sagepub.com/books/nonverbal-communication

  2. Modl.ai (2019). http://modl.ai/

  3. Sensum (2019). https://sensum.co/

  4. Albanie, S., Nagrani, A., Vedaldi, A., Zisserman, A.: Emotion recognition in speech using cross-modal transfer in the wild. arXiv preprint arXiv:1808.05561 (2018)

  5. Amir Zaib Abbasi, D.H.T., Hlavacs, H.: Engagement in games: developing an instrument to measure consumer videogame engagement and its validation. Int. J. Comput. Games Technol. 2017, 1–10 (2017)

    CrossRef  Google Scholar 

  6. Arriaga, O., Valdenegro-Toro, M., Ploger, P.: Real-time convolutional neural networks for emotion and gender classification. CoRR abs/1710.07557 (2017)

    Google Scholar 

  7. Bitouk, D., Verma, R., Nenkova, A.: Class-level spectral features for emotion recognition. Speech Commun. 52(7), 613–625 (2010)

    CrossRef  Google Scholar 

  8. Brett, A., Smith, M., Price, E., Huitt, W.: Overview of the affective domain. Educ. Psychol. Interact, 1–21 (2003)

    Google Scholar 

  9. Brown, S., Vaughan, C.: Play: How it Shapes the Brain, Opens the Imagination, and Invigorates the Soul. Avery (2009). https://books.google.dk/books?id=ESQDsgqfgusC

  10. Buchanan, R., Csikszentmihalyi, M.: Flow: the psychology of optimal experience. Des. Issues 8(1), 80-1 (1991)

    Google Scholar 

  11. Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology. Cambridge University Press, Cambridge (2007)

    Google Scholar 

  12. Chen, J.: Flow in games (and everything else). Commun. ACM 50, 31–34 (2007)

    CrossRef  Google Scholar 

  13. Christy, T., Kuncheva, L.I.: Technological advancements in affective gaming: a historical survey. GSTF Int. J. Comput. (JoC) 3(4), 32–41 (2013)

    Google Scholar 

  14. Darwin, C.: The Expression of the Emotions in Man and Animals. D. Appleton and Co., New York (1872)

    CrossRef  Google Scholar 

  15. Debeauvais, T.: Challenge and Retention in Games. UC Irvine. ProQuest ID: Debeauvais\(\_\)uci\(\_\)0030D\(\_\)13948. Merritt ID: ark:/13030/m53n6r1p (2016). https://escholarship.org/uc/item/6k3357qx

  16. Drachen, A., Nacke, L.E., Yannakakis, G., Pedersen, A.L.: Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In: Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 49–54. ACM (2010)

    Google Scholar 

  17. Ekman, P.: Body position, facial expression, and verbal behavior during interviews. 68, 295 (1964). https://doi.org/10.1037/h0040225

  18. Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)

    CrossRef  Google Scholar 

  19. Ekman, P.: Emotions revealed. 12 (2004). https://doi.org/10.1136/sbmj.0405184

  20. Frith, C.: Role of facial expressions in social interactions. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364(1535), 3453–3458 (2009)

    CrossRef  Google Scholar 

  21. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    CrossRef  Google Scholar 

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

  23. Hilgard, E.R.: The trilogy of mind: cognition, affection, and conation. J. Hist. Behav. Sci. 16(2), 107–117 (1980)

    CrossRef  Google Scholar 

  24. Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. CoRR abs/1608.06993 (2016). http://arxiv.org/abs/1608.06993

  25. Hudlicka, E., Broekens, J.: Foundations for modelling emotions in game characters: modelling emotion effects on cognition, pp. 1–6 (2009). https://doi.org/10.1109/ACII.2009.5349473

  26. Huitt, W., Cain, S.: An overview of the conative domain. Educ. Psychol. Interact., 1–20 (2005)

    Google Scholar 

  27. Ijsselsteijn, W., et al.: Measuring the experience of digital game enjoyment. In: Proceedings of Measuring Behavior (2008)

    Google Scholar 

  28. Jang, E.H., Park, B.J., Kim, S.H., Eum, Y., Sohn, J.H.: Identification of the optimal emotion recognition algorithm using physiological signals, pp. 1–6, November 2011

    Google Scholar 

  29. Jennett, C., et al.: Measuring and defining the experience of immersion in games. Int. J. Hum.-Comput. Stud. 66(9), 641–661 (2008)

    CrossRef  Google Scholar 

  30. Lee, C., Yoo, S., Park, Y., Kim, N., Jeong, K., Lee, B.: Using neural network to recognize human emotions from heart rate variability and skin resistance. In: Engineering in Medicine and Biology 27th Annual Conference (2005)

    Google Scholar 

  31. Lombard, M., Ditton, T.: At the heart of it all: the concept of presence. J. Comput. Mediat. Commun. 3(2), 0 (1997)

    CrossRef  Google Scholar 

  32. Matsumoto, D., Willingham, B.: Spontaneous facial expressions of emotion of congenitally and noncongenitally blind individuals. J. Pers. Soc. Psychol. 96, 1–10 (2009). https://doi.org/10.1037/a0014037

    CrossRef  Google Scholar 

  33. Matsumotol, D., Hwang, H.S.: Reading facial expressions of emotion. Psychol. Sci. 115, 541–558 (2011)

    Google Scholar 

  34. Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Comput. Vis. Image Underst. 115(4), 541–558 (2011)

    CrossRef  Google Scholar 

  35. O’Brien, H.L., Toms, E.G.: What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inform. Sci. Technol. 59(6), 938–955 (2008). https://doi.org/10.1002/asi.20801

    CrossRef  Google Scholar 

  36. Ouellet, S.: Real-time emotion recognition for gaming using deep convolutional network features. CoRR abs/1408.3750 (2014)

    Google Scholar 

  37. Picard, R.W.: Emotion research by the people, for the people. Emot. Rev. 2(3), 250–254 (2010)

    CrossRef  Google Scholar 

  38. Pollreisz, D., TaheriNejad, N.: A simple algorithm for emotion recognition, using physiological signals of a smart watch. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2353–2356 (2017)

    Google Scholar 

  39. Posner, J., Russell, J.A., Peterson, B.S.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(3), 715–734 (2005)

    CrossRef  Google Scholar 

  40. Rashid, M., Abu-Bakar, S.A.R., Mokji, M.: Human emotion recognition from videos using spatio-temporal and audio features. Vis. Comput. 29(12), 1269–1275 (2012). https://doi.org/10.1007/s00371-012-0768-y

    CrossRef  Google Scholar 

  41. Russell, J.A.: A circumplex model of affect. Pers. Soc. Psychol. 39, 1161 (1980)

    CrossRef  Google Scholar 

  42. Russell, J.A., Barrett, L.F.: Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J. Pers. Soc. Psychol. 76, 805 (1999)

    CrossRef  Google Scholar 

  43. Saarni, C.: The Development of Emotional Competence. Guilford Series on Social and Emotional Development. Guilford Publications, New York (1999)

    Google Scholar 

  44. Sagi, B.R., Silvestrini, R.: Application of combinatorial tests in video game testing. 29, 745–759 (2017). https://doi.org/10.1080/08982112.2017.1300919

  45. Schoenau-Fog, H.: Hooked! – evaluating engagement as continuation desire in interactive narratives. In: Si, M., Thue, D., André, E., Lester, J.C., Tanenbaum, J., Zammitto, V. (eds.) ICIDS 2011. LNCS, vol. 7069, pp. 219–230. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25289-1_24

    CrossRef  Google Scholar 

  46. Schoenau-Fog, H.: The player engagement process - an exploration of continuation desire in digital games. In: DiGRA - Proceedings of the 2011 DiGRA International Conference: Think Design Play. DiGRA/Utrecht School of the Arts, January 2011

    Google Scholar 

  47. Schoenau-Fog, H.: At the Core of Player Experience: Continuation Desire in Digital Games, pp. 388–410. Wiley (2014). https://doi.org/10.1002/9781118796443.ch14

  48. Schoenau-Fog, H.: Designing and evaluating conative game-based learning scenarios. In: Busch, C. (ed.) Proceedings of The 8th European Conference on Games Based Learning – ECGBL 2014, pp. 512–518. Academic Conferences and Publishing International, October 2014

    Google Scholar 

  49. Schoenau-Fog, H., Birke, A., Reng, L.: Evaluation of continuation desire as an iterative game development method. In: Proceeding of the 16th International Academic MindTrek Conference, MindTrek 2012, pp. 241–243. ACM, New York (2012). https://doi.org/10.1145/2393132.2393185

  50. Schoenau-Fog, H., Louchart, S., Lim, T., Soto-Sanfiel, M.T.: Narrative engagement in games-a continuation desire perspective. In: FDG, pp. 384–387 (2013)

    Google Scholar 

  51. Selvig, D., Stovring, N., Sjoblom, A., Korsholm, J.: Continuation desire and its physiological underpinnings - unpublished (2018)

    Google Scholar 

  52. Seng, K.P., Ang, L., Ooi, C.S.: A combined rule-based amp; machine learning audio-visual emotion recognition approach. IEEE Trans. Affect. Comput. 9(1), 3–13 (2018)

    CrossRef  Google Scholar 

  53. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    CrossRef  Google Scholar 

  54. Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR abs/1602.07261 (2016)

    Google Scholar 

  55. Tawari, A., Trivedi, M.M.: Face expression recognition by cross modal data association. IEEE Trans. Multimedia 15(7), 1543–1552 (2013)

    CrossRef  Google Scholar 

  56. Tawari, A., Trivedi, M.M.: Audio visual cues in driver affect characterization: issues and challenges in developing robust approaches, pp. 2997–3002 (2011). https://doi.org/10.1109/IJCNN.2011.6033615

  57. Tokuno, S., et al.: Usage of emotion recognition in military health care, pp. 1–5 (2011). https://doi.org/10.1109/DSR.2011.6026823

  58. Vyvey, T., Castellar, E.N., Van Looy, J.: Loaded with fun? The impact of enjoyment and cognitive load on brand retention in digital games. 18, 72–82 (2018). https://doi.org/10.1080/15252019.2018.1446370

  59. Wang, C., Pun, T., Chanel, G.: A comparative survey of methods for remote heartrate detection from frontal face videos. Front. Bioeng. Biotechnol. 6, 33 (2018)

    CrossRef  Google Scholar 

  60. Xerfi: Global video game turnover between 2011 and 2019 (2019). https://www.statista.com/statistics/862278/global-video-game-revenues-worldwide/. Accessed 19 Mar 2019

  61. Yu, S.N., Chen, S.F.: Emotion state identification based on heart rate variability and genetic algorithm, pp. 538–541, August 2015. https://doi.org/10.1109/EMBC.2015.7318418

  62. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Li, Y.: Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans. Cybern. 49(3), 839–847 (2018)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Dines Rae Selvig or Henrik Schoenau-Fog .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Rae Selvig, D., Schoenau-Fog, H. (2020). Non-intrusive Measurement of Player Engagement and Emotions - Real-Time Deep Neural Network Analysis of Facial Expressions During Game Play. In: Fang, X. (eds) HCI in Games. HCII 2020. Lecture Notes in Computer Science(), vol 12211. Springer, Cham. https://doi.org/10.1007/978-3-030-50164-8_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50164-8_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50163-1

  • Online ISBN: 978-3-030-50164-8

  • eBook Packages: Computer ScienceComputer Science (R0)