On the Diffusion Process for Heart Rate Estimation from Face Videos Under Realistic Conditions

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10496)


This work addresses the problem of estimating heart rate from face videos under real conditions using a model based on the recursive inference problem that leverages the local invariance of the heart rate. The proposed solution is based on the canonical state space representation of an Itō process and a Wiener velocity model. Empirical results yield to excellent real-time and estimation performance of heart rate in presence of disturbing factors, like rigid head motion, talking and facial expressions under natural illumination conditions making the process of heart rate estimation from face videos applicable in a much broader sense. To facilitate comparisons and to support research we made the code and data for reproducing the results public available.


  1. 1.
    Arnold, I.: Ordinary Differential Equations. MIT Press, Cambridge (1973)zbMATHGoogle Scholar
  2. 2.
    Bland, J., Altman, D.: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 327(8476), 307–310 (1986)CrossRefGoogle Scholar
  3. 3.
    Blanik, N., Blazek, C., Pereira, C., Blazek, V., Leonhardt, S.: Wearable photoplethysmographic sensors: past and present. In: Proceedings of the SPIE 9034, Medical Imaging: Image Processing (2014)Google Scholar
  4. 4.
    Blazek, C., Hülsbusch, M.: Assessment of allergic skin reactions and their hemodynamical quantification using photoplethysmography imaging. In: Proceedings of 11th International Symposium CNVD, Computer-Aided Noninvasive Vascular Diagnostics, vol. 3, 85–90 (2005)Google Scholar
  5. 5.
    Blazek, V.: Optoelektronische Erfassung und rechnerunterstützte Analyse der Mikrozirkulations-Rhythmik. Biomed. Techn. 30(1), 121–122 (1985)CrossRefGoogle Scholar
  6. 6.
    Blazek, V., Blanik, N., Blazek, C., Paul, M., Pereira, C., Koeny, M., Venema, B., Leonhardt, S.: Active and passive optical imaging modality for unobtrusive cardiorespiratory monitoring and facial expressions assessment. Assessment. Anesth Analg. 124, 104–119 (2017)CrossRefGoogle Scholar
  7. 7.
    Bloom, H., Bar-Shalom, Y.: The interacting multiple model algorithm for systems with markovian switching coefficients. IEEE Trans. Autom. Control 33(8), 780–783 (1988)CrossRefzbMATHGoogle Scholar
  8. 8.
    Cardoso, J.: High-order contrasts for independent component analysis. Neural Comput. 11(1), 157–192 (1999)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Cox, D.: Some statistical methods connected with series of events. J. Roy. Stat. Soc. 17(2), 129–164 (1950)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Durbin, J., Koopman, S.: Time Series Analysis by State Space Methods. Oxford University Press, Oxford (2001)zbMATHGoogle Scholar
  11. 11.
    Feynman, R., Leighton, R., Sands, M.: The Feynman Lectures on Physics, vol. 1. Addison-Wesley, Boston (1963). Chap. 21zbMATHGoogle Scholar
  12. 12.
    Gray, R., Neuhoff, D.: Quantization. IEEE Trans. Inf. Theory 44(6), 2325–2383 (1998)CrossRefzbMATHGoogle Scholar
  13. 13.
    Grewal, M., Andrews, A.: Kalman Filtering Theory and Practice Using Matlab. Wiley Interscience, Hoboken (2001)zbMATHGoogle Scholar
  14. 14.
    de Haan, G., Jeanne, V.: Robust pulse-rate from chrominance-based rppg. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2014)CrossRefGoogle Scholar
  15. 15.
    de Haan, G., van Leest, A.: Improved motion robustness of remote-ppg by using the blood volume pulse signature. Physiol. Meas. 3(9), 1913–1926 (2014)CrossRefGoogle Scholar
  16. 16.
    Hertzman, A.: Photoelectric plethysmography of the fingers and toes in man. Exp. Biol. Med. 37(3), 529–534 (1937)CrossRefGoogle Scholar
  17. 17.
    Hülsbusch, M.: A functional imaging technique for opto-electronic assessment of skin perfusion. Ph.D. thesis, RWTH Aachen University (2008)Google Scholar
  18. 18.
    Itô, K.: On Stochastic Differential Equations, vol. 4. Memoris of The American Mathematical Society (1951)Google Scholar
  19. 19.
    Jazwinski, A.: Stochastic Processes and Filtering Theory. Academic Press, New York (1970)zbMATHGoogle Scholar
  20. 20.
    Jones, R.H.: Fitting multivariate models to unequally spaced data. In: Parzen, E. (ed.) Time Series Analysis of Irregularly Observed Data. LNS, vol. 25, pp. 158–188. Springer, New York (1984). doi: 10.1007/978-1-4684-9403-7_8 CrossRefGoogle Scholar
  21. 21.
    Kalman, R., Bucy, R.: New results in linear filtering and prediction theory. Trans. ASME-J. Basic Eng. 83, 95–108 (1961)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Khintchine, A.: Korrelationstheorie der stationären stochastischen Prozesse. Springer-Mathematische Annalen 109, 604–615 (1934)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: IEEE International Conference on Computer Vision, pp. 3640–3648 (2015)Google Scholar
  24. 24.
    Lewandowska, M., Ruminski, J., Kocejko, T., Nowak, J.: Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity. In: Proceedings of the FedCSIS, Szczecin, Poland, pp. 405–410 (2011)Google Scholar
  25. 25.
    Li, X., Chen, J., Zhao, G., Pietikinen, M.: Remote heart rate measurement from face videos under realistic situations. In: IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH (2014)Google Scholar
  26. 26.
    Lomb, N.: Least-squares frequency analysis of unequally spaced data. Astrophys. Space Sci. 39(2), 447–462 (1976)CrossRefGoogle Scholar
  27. 27.
    Makhnin, O.: Filtering for some stochastic processes with discrete observations. Ph.D. thesis, Department of Statistics and Probability, Michigan State University (2002)Google Scholar
  28. 28.
    McDuff, D., Gontarek, S., Picard, R.: Remote measurement of cognitive stress via heart rate variability. In: 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2957–2960 (2014)Google Scholar
  29. 29.
    Moço, A., Stuijk, S., de Haan, G.: Ballistocardiographic artifacts in PPG imaging. IEEE Trans. Biomed. Eng. 63(9), 1804–1811 (2015)CrossRefGoogle Scholar
  30. 30.
    Moço, A., Stuijk, S., de Haan, G.: Motion robust PPG-imaging through color channel mapping. Biomed. Opt. Express 7, 1737–1754 (2016)CrossRefGoogle Scholar
  31. 31.
    Molitor, H., Knaizuk, M.: A new bloodless method for continuous recording of peripheral change. J. Pharmacol. Exp. Theret. 27, 5–16 (1936)Google Scholar
  32. 32.
    Øksendal, B.: Stochastic Differential Equations. Springer, Heidelberg (2003)CrossRefzbMATHGoogle Scholar
  33. 33.
    Oliver, B., Pierce, J., Shannon, C.: The philosophy of PCM. Proc. IRE 36, 1324–1331 (1948)CrossRefGoogle Scholar
  34. 34.
    Osman, A., Turcot, J., Kaliouby, R.E.: Supervised learning approach to remote heart rate estimation from facial videos. In: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2015)Google Scholar
  35. 35.
    Poh, M., McDuff, J., Picard, R.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762–10774 (2010)CrossRefGoogle Scholar
  36. 36.
    Ramirez, G., Fuentes, O., Crites, S., Jimenez, M., Ordonez, J.: Color analysis of facial skin: detection of emotional state. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 474–479 (2014)Google Scholar
  37. 37.
    Särkkä, S.: Recursive Bayesian inference on stochastic differential equations. Ph.D. thesis, Helsinki University of Technology (2006)Google Scholar
  38. 38.
    Särkkä, S., Solin, A., Nummenmaa, A., Vehtari, T., Vanni, F.L.: Dynamic retrospective filtering of physiological noise in BOLD fMRI. NeuroImage 60(2), 1517–1527 (2012)CrossRefGoogle Scholar
  39. 39.
    Scargle, J.: Studies in astronomical time series analysis. II - statistical aspects of spectral analysis of unevenly spaced data. Astrophys. J. 263(1), 835–853 (1982)CrossRefGoogle Scholar
  40. 40.
    Teplov, V.: Blood pulsation imaging. Ph.D. thesis, Department of Applied Physics, University of Eastern Finland (2014)Google Scholar
  41. 41.
    Tulyakov, S., Pineda, X.A., Ricci, E., Yin, L., Cohn, J., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: Computer Vision and Pattern Recognition (2016)Google Scholar
  42. 42.
    Verkruysse, W., Svaasand, L., Nelson, J.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434–21445 (2008)CrossRefGoogle Scholar
  43. 43.
    Viola, P., Jones, M.: Robust real-time object detection. Int. J. Comput. Vis. 57, 137–154 (2001)CrossRefGoogle Scholar
  44. 44.
    Wang, W., Stuijk, S., de Haan, G.: A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2015)CrossRefGoogle Scholar
  45. 45.
    Wiener, N.: The average of an analytical functional and the brownian movement. Proc. Nat. Acad. Sci. USA 7(1), 294–298 (1921)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Wiener, N.: Generalized harmonic analysis. Acta Mathematica 55, 117–258 (1930)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Zakai, M.: On the optimal filtering of diffusion processes. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 11(3), 230–243 (1969)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.CanControls GmbHAachenGermany
  2. 2.Institute of Safety TechnologyUniversity of WuppertalWuppertalGermany
  3. 3.Philips Chair for Medical Information Technology, Helmholtz-Institute for Biomedical EngineeringRWTH Aachen UniversityAachenGermany

Personalised recommendations