Skip to main content
SpringerLink
Log in

Webcam-based detection of emotional states

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Game designers have to deal with the complex task of monitoring the emotional state of players in games. There are different elements with the game, which effect the player’s emotional status. Since the game play experience occurs almost unconsciously, traditional methods such as think aloud may disrupt the playing experience, thus skewing the results obtained. Other methods include fitting cables and electrodes to the player to monitor biological information. Although such devices can offer significant accurate results, they are not commonly found and may cause discomfort while playing games. Because of this, we propose a webcam-based heart rate monitoring method that can be used to predict the player’s emotional state. We first analyzed the change in heart rate with respect to the players emotional state. This allowed us to find a correlation between emotional states, such as frustration, fun, challenge, and boredom. The second objective was to create a webcam-based method to monitor the heart rate. This was performed by extracting the RGB channels from the face region and then retrieving the underlying components using a dimensionality-reduction method. The results obtained from the webcam-based method were far from perfect, but this was expected, since we were performing the tests under realistic conditions. The last objective was to predict the player’s emotional state using the heart rate obtained from the webcam-based method. The accuracy of the prediction was up to 76 %, which exceeded our initial aim. This system will be implemented in Unity 3D to make its integration and adoption easier.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://www.wilddivine.com/.

  2. A window size of 10 was chosen since the device sends an update of the blood pulse value 30 times per second and at most, there is one heart beat every 10 frames.

References

  1. Bahreini, K., Nadolski, R., Westera, W.: Towards multimodal emotion recognition in e-learning environments. Interact. Learn. Environ. 24(3), 590–605 (2016)

    Article  Google Scholar 

  2. Bousefsaf, F., Maaoui, C., Pruski, A.: Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomed. Signal Process. Control 8(6), 568–574 (2013)

    Article  Google Scholar 

  3. Burelli, P., Yannakakis, G.N.: Towards adaptive virtual camera control in computer games. In: Smart Graphics, Springer, New York, pp. 25–36 (2011)

  4. Chen, J.: Flow in games (and everything else). Commun. ACM 50(4), 31–34 (2007)

    Article  Google Scholar 

  5. Csikszentmihalyi, M., Csikzentmihaly, M.: Flow: the psychology of optimal experience, vol. 41. HarperPerennial, New York (1991)

  6. Drachen, A., Nacke, L.E., Yannakakis, G., Pedersen, A.L.: Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In: Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 49–54 (2010)

  7. Ganglbauer, E., Schrammel, J., Deutsch, S., Tscheligi, M.: Applying psychophysiological methods for measuring user experience: possibilities, challenges and feasibility. In: Workshop on User Experience Evaluation Methods in Product Development, Citeseer (2009)

  8. Hudlicka, E.: Affective game engines: motivation and requirements. In: Proceedings of the 4th International Conference on Foundations of Digital Games, ACM, pp. 299–306 (2009)

  9. Kwon, S., Kim, H., Park, K.S.: Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. In: Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, pp. 2174–2177 (2012)

  10. Nacke, L., et al. Playability and player experience research. In: Proceedings of DiGRA 2009: Breaking New Ground: Innovation in Games, Play, Practice and Theory. DiGRA (2009)

  11. Lee, D., Kim, J., Kwon, S., Park, K.: Heart rate estimation from facial photoplethysmography during dynamic illuminance changes. In: Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, pp. 2758–2761 (2015)

  12. Lewandowska, M., Rumiński, J., Kocejko, T., et al.: Measuring pulse rate with a webcama non-contact method for evaluating cardiac activity. In: Computer Science and Information Systems (FedCSIS), 2011 Federated Conference on, IEEE, pp. 405–410 (2011)

  13. Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271 (2014)

  14. Mandryk, R.L., Inkpen, K.M., Calvert, T.W.: Using psychophysiological techniques to measure user experience with entertainment technologies. Behav. Inf. Technol. 25(2), 141–158 (2006)

  15. Martinez, H.P., Jhala, A., Yannakakis, G.N.: Analyzing the impact of camera viewpoint on player psychophysiology. In: Affective Computing and Intelligent Interaction and Workshops, ACII 2009, 3rd International Conference on, IEEE, pp. 1–6 (2009)

  16. Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling player experience for content creation. Comput. Intell. AI Game IEEE Trans. 2(1), 54–67 (2010)

    Article  Google Scholar 

  17. Poh, M.-Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)

    Article  Google Scholar 

  18. Pursche, T., Krajewski, J., Moeller, R.: Video-based heart rate measurement from human faces. In: Consumer Electronics (ICCE), 2012 IEEE International Conference on, IEEE, pp. 544–545 (2012)

  19. Schell, J.: The Art of Game Design: A Book of Lenses. CRC Press, New York (2014)

  20. Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: A game-based corpus for analysing the interplay between game context and player experience. In: Affective Computing and Intelligent Interaction, pp. 547–556. Springer, New York (2011)

  21. Sweetser, P., Wyeth, P.: Gameflow: a model for evaluating player enjoyment in games. Comput. Entertain. (CIE) 3(3), 3–3 (2005)

    Article  Google Scholar 

  22. Taylor, R.: Interpretation of the correlation coefficient: a basic review. J. Diagnos. Med. Sonogr. 6(1), 35–39 (1990)

    Article  Google Scholar 

  23. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008)

    Article  Google Scholar 

  24. Wei, L., Tian, Y., Wang, Y., Ebrahimi, T., Huang, T.: Automatic webcam-based human heart rate measurements using laplacian eigenmap. In: Computer Vision–ACCV 2012, pp. 281–292. Springer, New York (2013)

  25. Wild Divine: Iom, user manual. http://support.wilddivine.com/wp-content/uploads/2015/01/IomPE-Users-Manual.pdf (2014)

  26. Wong, T.Y.: Contactless Heart Rate Monitor for Multiple Persons in a Video. PhD thesis, UTAR (2015)

  27. Yannakakis, G.N., Hallam, J.: Entertainment modeling through physiology in physical play. Int. J. Hum. Comput. Stud. 66(10), 741–755 (2008)

    Article  Google Scholar 

  28. Yannakakis, G.N., Hallam, J.: Ranking vs. preference: a comparative study of self-reporting. In: Affective Computing and Intelligent Interaction, pp. 437–446. Springer, New York (2011)

  29. Yu, S., You, X., Jiang, X., Zhao, K., Mou, Y., Ou, W., Tang, Y., Chen, C.: Human heart rate estimation using ordinary cameras under natural movement. In: Systems, Man, and Cybernetics (SMC), IEEE International Conference on, pp. 1041–1046. IEEE (2015)

  30. Zaunseder, S., Heinke, A., Trumpp, A., Malberg, H.: Heart beat detection and analysis from videos. In: Electronics and Nanotechnology (ELNANO), IEEE 34th International Conference on, pp. 286–290. IEEE (2014)

Download references

Author information

Authors and Affiliations

  1. Department of Intelligent Computer Systems, University of Malta, Msida, Malta

    Alexiei Dingli & Andreas Giordimaina

Authors
  1. Alexiei Dingli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Giordimaina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexiei Dingli.

Additional information

The authors would like to thank the Institute of Digital Games for providing the resources used during the experiments and for their feedback and advice throughout the research.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dingli, A., Giordimaina, A. Webcam-based detection of emotional states. Vis Comput 33, 459–469 (2017). https://doi.org/10.1007/s00371-016-1309-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-016-1309-x

Keywords

Search

Navigation