Advertisement

Heart Rate Based Face Synthesis for Pulse Estimation

  • Umur Aybars CiftciEmail author
  • Lijun Yin
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11844)

Abstract

With the technological advancements in non-invasive heart rate (HR) detection, it becomes more feasible to estimate heart rate using commodity digital cameras. However, achieving high accuracy in HR estimation still remains a challenge. One of the bottlenecks is the lack of sufficient facial videos annotated with corresponding HR signals. In order to prevent this bottleneck, we propose to create videos enriched with different HR values from existing data sets with an attempt to increase the data size in a controllable manner. This paper presents a new method to generate facial videos with various heart rate values through a video synthesis procedure. Our method involves the synthesis of heart beat effects from skin colors of a face. New face video is generated with various heart rate values while taking identity information into account. The quality of the synthetic videos is evaluated by comparing to the original ground truth videos at the pixel level as well as by computing their differentiability across the synthetic face videos. Furthermore, the usability of the new data is assessed through the application of HR estimation from remote video approaches.

Keywords

Heart rate synthesis Remote heart rate estimation Face synthesis and analysis 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgement

The material is based upon the work supported in part by the National Science Foundation under grants CNS-1629898 and CNS-1205664.

References

  1. 1.
    Aarts, L.A., Jeanne, V., Cleary, J.P., Lieber, C., Nelson, J.S., Oetomo, S.B., Verkruysse, W.: Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit-a pilot study. Early Human Dev. 89(12), 943–948 (2013)CrossRefGoogle Scholar
  2. 2.
    Al-Ali, A., Diab, M.K., Coverston, R., Maurer, G., Schmidt, J., Schulz, C.: Pulse oximetry sensor, 9 October 2007, patent 7,280,858Google Scholar
  3. 3.
    Baltrusaitis, T., Robinson, P., Morency, L.P.: Openface: an open source facial behavior analysis toolkit. In: WACV, pp. 1–10. IEEE Computer Society (2016)Google Scholar
  4. 4.
    Bao, S.D., Zhang, Y.T., Shen, L.F.: Physiological signal based entity authentication for body area sensor networks and mobile healthcare systems. In: IEEE Engineering in Medicine and Biology (2006)Google Scholar
  5. 5.
  6. 6.
    Bobbia, S., Benezeth, Y., Dubois, J.: Remote photoplethysmography based on implicit living skin tissue segmentation. In: IEEE ICPR, pp. 361–365 (2016)Google Scholar
  7. 7.
    Chekmenev, S.Y., Rara, H., Farag, A.A.: Non-contact, wavelet-based measurement of vital signs using thermal imaging. In: The 1st International Conference on Graphics, Vision, and Image Processing (GVIP), pp. 107–112 (2005)Google Scholar
  8. 8.
    Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: IEEE CVPR (2017)Google Scholar
  9. 9.
    De Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)CrossRefGoogle Scholar
  10. 10.
    Feng, L., Po, L., Xu, X., Li, Y., Ma, R.: Motion-resistant remote imaging photoplethysmography based on the optical properties of skin. IEEE Trans. Circuits Syst. Video Technol. 25(5), 879–891 (2015)CrossRefGoogle Scholar
  11. 11.
    Gibert, G., D’Alessandro, D., Lance, F.: Face detection method based on photoplethysmography. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) (2013)Google Scholar
  12. 12.
    Gordon, J.W.: Certain molar movements of the human body produced by the circulation of the blood. J. Anat. Physiol. 11(Pt 3), 533–536 (1877)Google Scholar
  13. 13.
    Hassan, M., et al.: Heart rate estimation using facial video: a review. Biomed. Signal Process. Control 38, 346–360 (2017)CrossRefGoogle Scholar
  14. 14.
    Hsu, G., Ambikapathi, A., Chen, M.: Deep learning with time-frequency representation for pulse estimation from facial videos. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 383–389, October 2017Google Scholar
  15. 15.
    Hsu, Y., Lin, Y., Hsu, W.: Learning-based heart rate detection from remote photoplethysmography features. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4433–4437 (2014)Google Scholar
  16. 16.
    Inan, O.T., et al.: Ballistocardiography and seismocardiography: a review of recent advances. IEEE J. Biomed. Health Inf. 19(4), 1414–1427 (2015)CrossRefGoogle Scholar
  17. 17.
    Jose, A.D., Collison, D.: The normal range and determinants of the intrinsic heart rate in man. Cardiovasc. Res. 4(2), 160–167 (1970)CrossRefGoogle Scholar
  18. 18.
    Kakria, P., Tripathi, N.K., Kitipawang, P.: A real-time health monitoring system for remote cardiac patients using smartphone and wearable sensors. Int. J. Telemed. Appl. 2015, 373474 (2015)Google Scholar
  19. 19.
    Kranjec, J., Beguš, S., Geršak, G., Drnovšek, J.: Non-contact heart rate and heart rate variability measurements: a review. Biomed. Sig. Process. Control 13, 102–112 (2014)CrossRefGoogle Scholar
  20. 20.
    Kumar, M., Veeraraghavan, A., Sabharwal, A.: DistancePPG: robust non-contact vital signs monitoring using a camera. Biomed. Optics Express. 6(5), 1565–1588 (2015)CrossRefGoogle Scholar
  21. 21.
    Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3640–3648 (2015)Google Scholar
  22. 22.
    Lawson, G., Belcher, R., Dawes, G., Redman, C.: A comparison of ultrasound (with autocorrelation) and direct electrocardiogram fetal heart rate detector systems. Am. J. Obstet. Gynecol. 147(6), 721–722 (1983)CrossRefGoogle Scholar
  23. 23.
    Lee, A., Kim, Y.: Photoplethysmography as a form of biometric authentication. In: IEEE SENSORS, pp. 1–2 (2015)Google Scholar
  24. 24.
    Lee, T.W.: Independent component analysis. In: Lee, T.W. (ed.) Independent Component Analysis, pp. 27–66. Springer, Boston (1998).  https://doi.org/10.1007/978-1-4757-2851-4_2CrossRefGoogle Scholar
  25. 25.
    Lewandowska, M., Rumiński, J., Kocejko, T., Nowak, J.: Measuring pulse rate with a webcam a non-contact method for evaluating cardiac activity. In: IEEE Federated Conference on Computer Science and Information Systems (FedCSIS) (2011)Google Scholar
  26. 26.
    Li, X., Chen, J., Zhao, G., Pietikäinen, M.: Remote heart rate measurement from face videos under realistic situations. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271, June 2014 Google Scholar
  27. 27.
    Lu, G., Yang, F., Taylor, J.A., Stein, J.F.: A comparison of photoplethysmography and ECG recording to analyse heart rate variability in healthy subjects. J. Med. Eng. Technol. 33(8), 634–641 (2009)CrossRefGoogle Scholar
  28. 28.
    Mestha, L.K., Kyal, S., Xu, B., Lewis, L.E., Kumar, V.: Towards continuous monitoring of pulse rate in neonatal intensive care unit with a webcam. In: IEEE Engineering in Medicine and Biology Society (EMBC) (2014)Google Scholar
  29. 29.
    Niu, X., Han, H., Shan, S., Chen, X.: SynRhythm: learning a deep heart rate estimator from general to specific. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585 (2018)Google Scholar
  30. 30.
    Poh, M.Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)CrossRefGoogle Scholar
  31. 31.
    Poh, M.Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011)CrossRefGoogle Scholar
  32. 32.
    Rouast, P.V., Adam, M.T., Chiong, R., Cornforth, D., Lux, E.: Remote heart rate measurement using low-cost RGB face video: a technical literature review. Front. Comput. Sci. 12(5), 858–872 (2018)CrossRefGoogle Scholar
  33. 33.
    Rouast, P.V., Adam, M.T.P., Cornforth, D.J., Lux, E., Weinhardt, C.: Using contactless heart rate measurements for real-time assessment of affective states. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B. (eds.) Information Systems and Neuroscience. LNISO, vol. 16, pp. 157–163. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-41402-7_20CrossRefGoogle Scholar
  34. 34.
    Sameni, R., Clifford, G.D.: A review of fetal ECG signal processing; issues and promising directions. Open Pacing Electrophysiol. Ther. J. 3, 4 (2010)Google Scholar
  35. 35.
    Scalise, L., Bernacchia, N., Ercoli, I., Marchionni, P.: Heart rate measurement in neonatal patients using a webcamera. In: IEEE International Symposium on Medical Measurements and Applications (MeMeA) (2012)Google Scholar
  36. 36.
    Sodi-Pallares, D.: Electrophysiology of the heart. Am. J. Cardiol. (1961) Google Scholar
  37. 37.
    Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)CrossRefGoogle Scholar
  38. 38.
    Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  39. 39.
    Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008)CrossRefGoogle Scholar
  40. 40.
    Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Algorithmic principles of remote PPG. IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017)CrossRefGoogle Scholar
  41. 41.
    Wang, W., Stuijk, S., De Haan, G.: Unsupervised subject detection via remote PPG. IEEE Trans. Biomed. Eng. 62(11), 2629–2637 (2015)CrossRefGoogle Scholar
  42. 42.
    Wang, W., Stuijk, S., De Haan, G.: A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016)CrossRefGoogle Scholar
  43. 43.
    Wang, W., Stuijk, S., de Haan, G.: Living-skin classification via remote-PPG. IEEE Trans. Biomed. Eng. 64(12), 2781–2792 (2017)CrossRefGoogle Scholar
  44. 44.
    Yan, Y., Ma, X., Yao, L., Ouyang, J.: Noncontact measurement of heart rate using facial video illuminated under natural light and signal weighted analysis. Biomed. Mater. Eng. 26(s1), S903–S909 (2015)Google Scholar
  45. 45.
    Yu, Y.P., Kwan, B.H., Lim, C.L., Wong, S.L., Raveendran, P.: Video-based heart rate measurement using short-time Fourier transform. In: Intelligent Signal Processing and Communications Systems (ISPACS), pp. 704–707. IEEE (2013)Google Scholar
  46. 46.
    Zhang, Z., et al.: Multimodal spontaneous emotion corpus for human behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3438–3446 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.State University of New York at BinghamtonBinghamton NYUSA

Personalised recommendations

Cite paper

Buy options