Abstract
Remote photoplethysmography (rPPG) is a contactless method for estimating physiological signals from facial videos. Without large supervised datasets, learning a robust rPPG estimation model is extremely challenging. Instead of merely focusing on model learning, we believe data augmentation may be of greater importance for this task. In this paper, we propose a novel multi-task learning framework to simultaneously augment training data while learning the rPPG estimation model. We design three networks: rPPG estimation network, Image-to-Video network, and Video-to-Video network, to estimate rPPG signals from face videos, to generate synthetic videos from a source image and a specified rPPG signal, and to generate synthetic videos from a source video and a specified rPPG signal, respectively. Experimental results on three benchmark datasets, COHFACE, UBFC, and PURE, show that our method successfully generates photo-realistic videos and significantly outperforms existing methods with a large margin. (The code is publicly available at https://github.com/YiAnLee/Multi-Task-Learning-for-Simultaneous-VideoGeneration-and-Remote-Photoplethysmography-Estimation).
Y.-Y. Tsou and Y.-A. Lee—The first two authors contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Li, X., et al.: The OBF database: a large face video database for remote physiological signal measurement and atrial fibrillation detection. In: 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 242–249 (2018)
Yu, Z., Li, X., Zhao, G.: Recovering remote photoplethysmograph signal from facial videos using spatio-temporal convolutional networks. CoRR abs/1905.02419 (2019)
Chen, W., McDuff, D.: DeepPhys: video-based physiological measurement using convolutional attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 356–373. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_22
Chen, W., McDuff, D.J.: DeepMag: source specific motion magnification using gradient ascent. CoRR abs/1808.03338 (2018)
Wang, Z.K., Kao, Y., Hsu, C.T.: Vision-based heart rate estimation via a two-stream CNN. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 3327–3331 (2019)
de Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60, 2878–2886 (2013)
Hernandez-Ortega, J., Fierrez, J., Morales, A., Tome, P.: Time analysis of pulse-based face anti-spoofing in visible and NIR. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2018)
Liu, Y., Jourabloo, A., Liu, X.: Learning deep models for face anti-spoofing: binary or auxiliary supervision. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 389–398 (2018)
Liu, S., Lan, X., Yuen, P.C.: Remote photoplethysmography correspondence feature for 3D mask face presentation attack detection (2018)
Liu, S., Yuen, P.C., Zhang, S., Zhao, G.: 3D mask face anti-spoofing with remote photoplethysmography. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 85–100. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_6
S̆petlík, R., Franc, V., C̆ech, J., Matas, J.: Visual heart rate estimation with convolutional neural network. In: Proceedings of British Machine Vision Conference (2018)
Bobbia, S., Macwan, R., Benezeth, Y., Mansouri, A., Dubois, J.: Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recogn. Lett. 124, 82–90 (2017)
Stricker, R., Müller, S., Gross, H.M.: Non-contact video-based pulse rate measurement on a mobile service robot, vol. 2014, pp. 1056–1062 (2014)
Niu, X., Han, H., Shan, S., Chen, X.: SynRhythm: learning a deep heart rate estimator from general to specific. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585 (2018)
Heusch, G., Anjos, A., Marcel, S.: A reproducible study on remote heart rate measurement. CoRR abs/1709.00962 (2017)
Benezeth, Y., Bobbia, S., Nakamura, K., Gomez, R., Dubois, J.: Probabilistic signal quality metric for reduced complexity unsupervised remote photoplethysmography, pp. 1–5 (2019)
Li, P., Yannick Benezeth, K.N., Gomez, R., Yang, F.: Model-based region of interest segmentation for remote photoplethysmography. In: 14th International Conference on Computer Vision Theory and Applications, pp. 383–388 (2019)
Li, X., Chen, J., Zhao, G., Pietikäinen, M.: Remote heart rate measurement from face videos under realistic situations. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271 (2014)
Macwan, R., Benezeth, Y., Mansouri, A.: Heart rate estimation using remote photoplethysmography with multi-objective optimization. Biomed. Signal Process. Control 49, 24–33 (2019)
Macwan, R., Bobbia, S., Benezeth, Y., Dubois, J., Mansouri, A.: Periodic variance maximization using generalized eigenvalue decomposition applied to remote photoplethysmography estimation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1413–14138 (2018)
Wang, W., Stuijk, S., de Haan, G.: A novel algorithm for remote photoplethysmography: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63, 1974–1984 (2016)
Tsou, Y.Y., Lee, Y.A., Hsu, C.T., Chang, S.H.: Siamese-rPPG network: remote photoplethysmography signal estimation from face video. In: The 35th ACM/SIGAPP Symposium on Applied Computing (SAC 2020) (2020)
Yu, Z., Peng, W., Li, X., Hong, X., Zhao, G.: Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement. In: International Conference on Computer Vision (ICCV) (2019)
Dvornik, N., Mairal, J., Schmid, C.: Modeling visual context is key to augmenting object detection datasets. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 375–391. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01258-8_23
Frid-Adar, M., Klang, E., Amitai, M., Goldberger, J., Greenspan, H.: Synthetic data augmentation using GAN for improved liver lesion classification, pp. 289–293 (2018)
Qiu, Y., Liu, Y., Arteaga-Falconi, J., Dong, H., Saddik, A.E.: EVM-CNN: real-time contactless heart rate estimation from facial video. IEEE Trans. Multimed. 21, 1778–1787 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Tsou, YY., Lee, YA., Hsu, CT. (2021). Multi-task Learning for Simultaneous Video Generation and Remote Photoplethysmography Estimation. In: Ishikawa, H., Liu, CL., Pajdla, T., Shi, J. (eds) Computer Vision – ACCV 2020. ACCV 2020. Lecture Notes in Computer Science(), vol 12626. Springer, Cham. https://doi.org/10.1007/978-3-030-69541-5_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-69541-5_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69540-8
Online ISBN: 978-3-030-69541-5
eBook Packages: Computer ScienceComputer Science (R0)