Advertisement

Accuracy Evaluation of Remote Photoplethysmography Estimations of Heart Rate in Gaming Sessions with Natural Behavior

  • Fernando Bevilacqua
  • Henrik Engström
  • Per Backlund
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10714)

Abstract

Remote photoplethysmography (rPPG) can be used to remotely estimate heart rate (HR) of users to infer their emotional state. However natural body movement and facial actions of users significantly impact such techniques, so their reliability within contexts involving natural behavior must be checked. We present an experiment focused on the accuracy evaluation of an established rPPG technique in a gaming context. The technique was applied to estimate the HR of subjects behaving naturally in gaming sessions whose games were carefully designed to be casual-themed, similar to off-the-shelf games and have a difficulty level that linearly progresses from a boring to a stressful state. Estimations presented mean error of 2.99 bpm and Pearson correlation \(r=0.43\), \(p < 0.001\), however with significant variations among subjects. Our experiment is the first to measure the accuracy of an rPPG technique using boredom/stress-inducing casual games with subjects behaving naturally.

Keywords

Games Emotion assessment Remote photoplethysmography Computer vision Affective computing 

Notes

Acknowledgment

The authors would like to thank the participants and all involved personnel for their valuable contributions. This work has been performed with support from: CNPq, Conselho Nacional de Desenvolvimento Científico e Tecnológico - Brasil; University of Skövde; EU Interreg ÖKS project Game Hub Scandinavia; UFFS, Federal University of Fronteira Sul.

References

  1. 1.
    Allen, J.: Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 28(3), R1–R39 (2007).  https://doi.org/10.1088/0967-3334/28/3/r01 CrossRefGoogle Scholar
  2. 2.
    Appelhans, B.M., Luecken, L.J.: Heart rate variability as an index of regulated emotional responding. Rev. Gen. Psychol. 10(3), 229 (2006)CrossRefGoogle Scholar
  3. 3.
    Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437. Institute of Electrical and Electronics Engineers (IEEE), June 2013.  https://doi.org/10.1109/cvpr.2013.440
  4. 4.
    Bevilacqua, F., Backlund, P., Engstrom, H.: Variations of facial actions while playing games with inducing boredom and stress. In: 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp. 1–8. Institute of Electrical and Electronics Engineers (IEEE), IEEE, September 2016.  https://doi.org/10.1109/vs-games.2016.7590374
  5. 5.
    Bousefsaf, F., Maaoui, C., Pruski, A.: Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomed. Signal Process. Control 8(6), 568–574 (2013).  https://doi.org/10.1016/j.bspc.2013.05.010 CrossRefGoogle Scholar
  6. 6.
    Bousefsaf, F., Maaoui, C., Pruski, A.: Remote assessment of the heart rate variability to detect mental stress. In: Proceedings of the ICTs for Improving Patients Rehabilitation Research Techniques, pp. 348–351. Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (ICST), IEEE (2013).  https://doi.org/10.4108/icst.pervasivehealth.2013.252181
  7. 7.
    Boyle, E.A., Connolly, T.M., Hainey, T., Boyle, J.M.: Engagement in digital entertainment games: A systematic review. Comput. Hum. Behav. 28(3), 771–780 (2012)CrossRefGoogle Scholar
  8. 8.
    Brogni, A., Vinayagamoorthy, V., Steed, A., Slater, M.: Variations in physiological responses of participants during different stages of an immersive virtual environment experiment. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology - VRST 2006, pp. 376–382. ACM, Association for Computing Machinery (ACM) (2006).  https://doi.org/10.1145/1180495.1180572
  9. 9.
    Choi, J., Gutierrez-Osuna, R.: Using heart rate monitors to detect mental stress. In: 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks, pp. 219–223. IEEE, Institute of Electrical and Electronics Engineers (IEEE), June 2009.  https://doi.org/10.1109/bsn.2009.13
  10. 10.
    Datcu, D., Cidota, M., Lukosch, S., Rothkrantz, L.: Noncontact automatic heart rate analysis in visible spectrum by specific face regions. In: Proceedings of the 14th International Conference on Computer Systems and Technologies, CompSysTech 2013, pp. 120–127 (2013). ACM, New York. ISBN 978-1-4503-2021-4.  https://doi.org/10.1145/2516775.2516805, http://doi.acm.org/10.1145/2516775.2516805
  11. 11.
    de Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013).  https://doi.org/10.1109/tbme.2013.2266196 CrossRefGoogle Scholar
  12. 12.
    Edwards, G.J., Taylor, C.J., Cootes, T.F.: Interpreting face images using active appearance models. In: Proceedings of the 3rd International Conference on Face & Gesture Recognition, FG 1998, p. 300, Washington, DC, USA (1998). IEEE Computer Society. ISBN 0-8186-8344-9. http://dl.acm.org/citation.cfm?id=520809.796067
  13. 13.
    Fenton-O’Creevy, M., Lins, J.T., Vohra, S., Richards, D.W., Davies, G., Schaaff, K.: Emotion regulation and trader expertise: Heart rate variability on the trading floor. J. Neurosci. Psychol. Econ. 5(4), 227 (2012)CrossRefGoogle Scholar
  14. 14.
    Garde, A., Laursen, B., Jørgensen, A., Jensen, B.: Effects of mental and physical demands on heart rate variability during computer work. Eur. J. Appl. Physiol. 87(4–5), 456–461 (2002)CrossRefGoogle Scholar
  15. 15.
    Giannakakis, G., Pediaditis, M., Manousos, D., Kazantzaki, E., Chiarugi, F., Simos, P.G., Marias, K., Tsiknakis, M.: Stress and anxiety detection using facial cues from videos. Biomed. Signal Process. Control 31, 89–101 (2017).  https://doi.org/10.1016/j.bspc.2016.06.020 CrossRefGoogle Scholar
  16. 16.
    Grundlehner, B., Brown, L., Penders, J., Gyselinckx, B.: The design and analysis of a real-time, continuous arousal monitor. In: 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks, pp. 156–161. Institute of Electrical and Electronics Engineers, IEEE, June 2009.  https://doi.org/10.1109/bsn.2009.21
  17. 17.
    Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6(2), 156–166 (2005)CrossRefGoogle Scholar
  18. 18.
    Hsu, Y.C., Lin, Y.-L., Hsu, W.: Learning-based heart rate detection from remote photoplethysmography features. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4433–4437. Institute of Electrical and Electronics Engineers (IEEE), IEEE, May 2014.  https://doi.org/10.1109/icassp.2014.6854440
  19. 19.
    Irani, R., Nasrollahi, K., Moeslund, T.B.: Improved pulse detection from head motions using DCT. Institute for Systems and Technologies of Information, Control and Communication (2014)Google Scholar
  20. 20.
    Jerritta, S., Murugappan, M., Nagarajan, R., Wan, K.: Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, pp. 410–415. IEEE, Institute of Electrical and Electronics Engineers (IEEE), March 2011.  https://doi.org/10.1109/cspa.2011.5759912
  21. 21.
    Kivikangas, J.M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., Ravaja, N.: A review of the use of psychophysiological methods in game research. J. Gaming Virtual Worlds 3(3), 181–199 (2011).  https://doi.org/10.1386/jgvw.3.3.181_1
  22. 22.
    Kukolja, D., Popović, S., Horvat, M., Kovač, B., Ćosić, K.: Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int. J. Hum. Comput. Stud. 72(10–11), 717–727 (2014).  https://doi.org/10.1016/j.ijhcs.2014.05.006 CrossRefGoogle Scholar
  23. 23.
    Landowska, A.: Emotion monitoring verification of physiological characteristics measurement procedures. Metrol. Meas. Syst. 21(4), 719–732 (2014). ISSN 2300–1941.  https://doi.org/10.2478/mms-2014-0049
  24. 24.
    Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situations. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271. Institute of Electrical & Electronics Engineers (IEEE), June 2014.  https://doi.org/10.1109/cvpr.2014.543
  25. 25.
    McDuff, D., Gontarek, S., Picard, R.: Remote measurement of cognitive stress via heart rate variability. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2957–2960. Institute of Electrical and Electronics Engineers (IEEE), August 2014.  https://doi.org/10.1109/embc.2014.6944243
  26. 26.
    McDuff, D., Gontarek, S., Picard, R.W.: Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Trans. Biomed. Eng. 61(10), 2593–2601 (2014).  https://doi.org/10.1109/tbme.2014.2323695 CrossRefGoogle Scholar
  27. 27.
    McDuff, D.J., Estepp, J.R., Piasecki, A.M., Blackford, E.B.: A survey of remote optical photoplethysmographic imaging methods. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6398–6404. IEEE, Institute of Electrical and Electronics Engineers (IEEE), August 2015.  https://doi.org/10.1109/embc.2015.7319857
  28. 28.
    McDuff, D.J., Hernandez, J., Gontarek, S., Picard, R.W.: COGCAM: Contact-free measurement of cognitive stress during computer tasks with a digital camera. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI 2016. Association for Computing Machinery (ACM) (2016).  https://doi.org/10.1145/2858036.2858247
  29. 29.
    Monkaresi, H., Calvo, R.A., Yan, H.: A machine learning approach to improve contactless heart rate monitoring using a webcam. IEEE J. Biomed. Health Inform. 18(4), 1153–1160 (2014).  https://doi.org/10.1109/jbhi.2013.2291900 CrossRefGoogle Scholar
  30. 30.
    Poh, M.-Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762 (2010).  https://doi.org/10.1364/oe.18.010762 CrossRefGoogle Scholar
  31. 31.
    Poh, M.-Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011)CrossRefGoogle Scholar
  32. 32.
    Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Anal. Appl. 9(1), 58–69 (2006)CrossRefGoogle Scholar
  33. 33.
    Ravaja, N., Saari, T., Laarni, J., Kallinen, K., Salminen, M., Holopainen, J., Järvinen, A.: The psychophysiology of video gaming: Phasic emotional responses to game events. In: International DiGRA Conference (2005)Google Scholar
  34. 34.
    Roald, N.G.: Estimation of vital signs from ambient-light non-contact photoplethysmography (2013)Google Scholar
  35. 35.
    Rodriguez, A., Rey, B., Vara, M.D., Wrzesien, M., Alcaniz, M., Banos, R.M., Perez-Lopez, D.: A VR-based serious game for studying emotional regulation in adolescents. IEEE Comput. Grap. Appl. 35(1), 65–73 (2015).  https://doi.org/10.1109/mcg.2015.8 CrossRefGoogle Scholar
  36. 36.
    Rouast, P.V., Adam, M.T.P., Chiong, R., Cornforth, D., Lux, E.: Remote heart rate measurement using low-cost RGB face video: A technical literature review. Front. Comput. Sci. 1–15 (2016).  https://doi.org/10.1007/s11704-016-6243-6
  37. 37.
    Schubert, C., Lambertz, M., Nelesen, R.A., Bardwell, W., Choi, J.-B., Dimsdale, J.E.: Effects of stress on heart rate complexitya comparison between short-term and chronic stress. Biol. Psychol. 80(3), 325–332 (2009)CrossRefGoogle Scholar
  38. 38.
    Sharma, R., Khera, S., Mohan, A., Gupta, N., Ray, R.B.: Assessment of computer game as a psychological stressor. Indian J. Physiol. Pharmacol. 50(4), 367 (2006)Google Scholar
  39. 39.
    Takano, C., Ohta, Y.: Heart rate measurement based on a time-lapse image. Med. Eng. Phys. 29(8), 853–857 (2007).  https://doi.org/10.1016/j.medengphy.2006.09.006 CrossRefGoogle Scholar
  40. 40.
    Tijs, T.J.W., Brokken, D., IJsselsteijn, W.A.: Dynamic game balancing by recognizing affect. In: Markopoulos, P., de Ruyter, B., IJsselsteijn, W., Rowland, D. (eds.) Fun and Games. LNCS, vol. 5294, pp. 88–93. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-88322-7_9 CrossRefGoogle Scholar
  41. 41.
    Tran, D.N., Lee, H., Kim, C.: A robust real time system for remote heart rate measurement via camera. In: 2015 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE, Institute of Electrical and Electronics Engineers (IEEE), June 2015.  https://doi.org/10.1109/icme.2015.7177484
  42. 42.
    Vandeput, S., Taelman, J., Spaepen, A., Van Huffel, S.: Heart rate variability as a tool to distinguish periods of physical and mental stress in a laboratory environment. In: Proceedings of the 6th International Workshop on Biosignal Interpretation (BSI), New Haven, CT, pp. 187–190 (2009)Google Scholar
  43. 43.
    Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434 (2008).  https://doi.org/10.1364/oe.16.021434 CrossRefGoogle Scholar
  44. 44.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  45. 45.
    Wang, W., den Brinker, A., Stuijk, S., de Haan, G.: Algorithmic principles of remote-PPG. IEEE Trans. Biomed. Eng. 1 (2016).  https://doi.org/10.1109/tbme.2016.2609282
  46. 46.
    Wang, W., Stuijk, S., de Haan, G.: A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016).  https://doi.org/10.1109/TBME.2015.2508602. ISSN 0018–9294CrossRefGoogle Scholar
  47. 47.
    Xiao, X., Wang, J.: Towards attentive, bi-directional MOOC learning on mobile devices. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction - ICMI 2015, pp. 163–170. ACM, Association for Computing Machinery (ACM) (2015).  https://doi.org/10.1145/2818346.2820754
  48. 48.
    Yamaguchi, M., Wakasugi, J., Sakakima, J.: Evaluation of driver stress using biomarker in motor-vehicle driving simulator. In: 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1834–1837. IEEE, Institute of Electrical and Electronics Engineers (IEEE), August 2006.  https://doi.org/10.1109/iembs.2006.260001
  49. 49.
    Yamakoshi, T., Yamakoshi, K., Tanaka, S., Nogawa, M., Shibata, M., Sawada, Y., Rolfe, P., Hirose, Y.: A preliminary study on driver’s stress index using a new method based on differential skin temperature measurement. In: 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 722–725. IEEE, Institute of Electrical and Electronics Engineers (IEEE), August 2007.  https://doi.org/10.1109/iembs.2007.4352392
  50. 50.
    Yun, C., Shastri, D., Pavlidis, I., Deng. Z.: O’ game, can you feel my frustration? In: Proceedings of the 27th International Conference on Human Factors in Computing Systems - CHI 2009, pp. 2195–2204. ACM, Association for Computing Machinery (ACM) (2009).  https://doi.org/10.1145/1518701.1519036
  51. 51.
    Zhao, F., Li, M., Qian, Y., Tsien, J.Z.: Remote measurements of heart and respiration rates for telemedicine. PLoS ONE 8(10), e71384 (2013).  https://doi.org/10.1371/journal.pone.0071384 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of SkövdeSkövdeSweden
  2. 2.Federal University of Fronteira SulChapecóBrazil

Personalised recommendations