Abstract
In this work, we propose the use of thermal vision sensors to estimate the frontal body landmarks of an inhabitant. The use of thermal sensors is being promoted to collect human patterns while protecting inhabitants’ privacy in smart environments. On the other hand, deep learning approaches have provided encouraging results in estimating body, hand and facial landmarks. Here, we present a residual neural network which produces body landmarks from images collected by a low cost thermal sensor. In order to solve the problems of capturing and labeling data, which hinder learning in deep learning models, we propose an auto-labeling approach with dual visible-spectrum and thermal cameras, including the recognition of keypoints by the OpenPose model. A case study developed with four inhabitants in different poses shows encouraging results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Al-Sarawi, S., Anbar, M., Alieyan, K., Alzubaidi, M.: Internet of Things (IoT) communication protocols. In: 2017 8th International Conference on Information Technology (ICIT), pp. 685–690. IEEE (2017)
Ali Hamad, R., Järpe, E., Lundström, J.: Stability analysis of the t-SNE algorithm for human activity pattern data. In: The 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC 2018) (2018)
Badave, H., Kuber, M.: Evaluation of person recognition accuracy based on openpose parameters. In: 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 635–640. IEEE (2021)
Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 172–186 (2019)
Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291–7299 (2017)
Csurka, G.: Domain adaptation for visual applications: a comprehensive survey. arXiv preprint arXiv:1702.05374 (2017)
Espinosa, Á.V., López, J.L.L., Mata, F.M., Estevez, M.E.E.: Application of IoT in healthcare: keys to implementation of the sustainable development goals. Sensors 21(7), 2330 (2021)
Griffiths, E., Assana, S., Whitehouse, K.: Privacy-preserving image processing with binocular thermal cameras. Proc. ACM Interact. Mob. Wearab. Ubiquit. Technol. 1(4), 1–25 (2018)
Han, J., Bhanu, B.: Human activity recognition in thermal infrared imagery. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005)-Workshops, pp. 17. IEEE (2005)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hidalgo, G., Raaj, Y., Idrees, H., Xiang, D., Joo, H., Simon, T., Sheikh, Y.: Single-network whole-body pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6982–6991 (2019)
Hiriyannaiah, S., Akanksh, B.S., Koushik, A.S., Siddesh, G.M., Srinivasa, K.G.: Deep learning for multimedia data in IoT. In: Tanwar, S., Tyagi, S., Kumar, N. (eds.) Multimedia Big Data Computing for IoT Applications. ISRL, vol. 163, pp. 101–129. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-8759-3_4
Jais, I.K.M., Ismail, A.R., Nisa, S.Q.: Adam optimization algorithm for wide and deep neural network. Knowl. Eng. Data Sci. 2(1), 41–46 (2019)
Jara-Quito, H.J., Guerrero-Vasquez, L.F., Parra-Luzuriaga, K.A., Ojeda-Sanchez, M.V., Bravo-Torres, J.F.: Avatar: human-computer interface for interaction with children using a live animation process based in facial and body landmarks recognition. In: 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 715–720. IEEE (2021)
Jin, S., et al.: Whole-body human pose estimation in the wild. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 196–214. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_12
Kong, X., Meng, Z., Meng, L., Tomiyama, H.: A privacy protected fall detection IoT system for elderly persons using depth camera. In: 2018 International Conference on Advanced Mechatronic Systems (ICAMechS), pp. 31–35. IEEE (2018)
Li, X., Liu, Y., Wang, Y., Yan, D.: Computing homography with RANSAC algorithm: a novel method of registration. In: Electronic Imaging and Multimedia Technology IV, vol. 5637, pp. 109–112. International Society for Optics and Photonics (2005)
López-Medina, M., Espinilla, M., Cleland, I., Nugent, C., Medina, J.: Fuzzy cloud-fog computing approach application for human activity recognition in smart homes. J. Intell. Fuzzy Syst. 38(1), 709–721 (2020)
Martínez-González, A., Villamizar, M., Canévet, O., Odobez, J.M.: Efficient convolutional neural networks for depth-based multi-person pose estimation. IEEE Trans. Circuits Syst. Video Technol. 30(11), 4207–4221 (2019)
Medina-Quero, J., Zhang, S., Nugent, C., Espinilla, M.: Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition. Expert Syst. Appl. 114, 441–453 (2018)
Mohammadmoradi, H., Munir, S., Gnawali, O., Shelton, C.: Measuring people-flow through doorways using easy-to-install IR array sensors. In: 2017 13th International Conference on Distributed Computing in Sensor Systems (DCOSS), pp. 35–43. IEEE (2017)
Ordóñez, F.J., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 115 (2016)
Osokin, D.: Real-time 2D multi-person pose estimation on CPU: lightweight openpose. arXiv preprint arXiv:1811.12004 (2018)
Patricia, N., Caputo, B.: Learning to learn, from transfer learning to domain adaptation: a unifying perspective. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1442–1449 (2014)
Polo-Rodriguez, A., Cruciani, F., Nugent, C.D., Medina, J.: Domain adaptation of binary sensors in smart environments through activity alignment. IEEE Access 8, 228804–228817 (2020)
Quero, J.M., Burns, M., Razzaq, M.A., Nugent, C., Espinilla, M.: Detection of falls from non-invasive thermal vision sensors using convolutional neural networks. In: Multidisciplinary Digital Publishing Institute Proceedings, vol. 2, p. 1236 (2018)
Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 1–48 (2019)
Sixsmith, A., Johnson, N.: A smart sensor to detect the falls of the elderly. IEEE Perv. Comput. 3(2), 42–47 (2004)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Voulodimos, A., Doulamis, N., Doulamis, A., Protopapadakis, E.: Deep learning for computer vision: a brief review. In: Computational Intelligence and Neuroscience 2018 (2018)
Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recogn. Lett. 119, 3–11 (2019)
Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Zhang, S., Wei, Z., Nie, J., Huang, L., Wang, S., Li, Z.: A review on human activity recognition using vision-based method. J. Healthcare Eng. 2017 (2017)
Acknowledgements
This contribution has been supported by the Spanish Institute of Health ISCIII by means of the project DTS21-00047 and by the Spanish Ministry of Science throughout Project RTI2018-095993-B-I00 and by J. Andalucía through Project P18-RT-1193 and by the European Regional Development Fund (ERDF). Funding for this research is provided by EU Horizon 2020 Pharaon Project ‘Pilots for Healthy and Active Ageing’, Grant agreement no. 857188. Moreover, this research received funding under the REMIND project Marie Sklodowska-Curie EU Framework for Research and Innovation Horizon 2020, under grant agreement no. 734355..
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Polo-Rodríguez, A., Lupión, M., Ortigosa, P.M., Medina-Quero, J. (2022). Estimating Frontal Body Landmarks from Thermal Sensors Using Residual Neural Networks. In: Rojas, I., Valenzuela, O., Rojas, F., Herrera, L.J., Ortuño, F. (eds) Bioinformatics and Biomedical Engineering. IWBBIO 2022. Lecture Notes in Computer Science(), vol 13346. Springer, Cham. https://doi.org/10.1007/978-3-031-07704-3_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-07704-3_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-07703-6
Online ISBN: 978-3-031-07704-3
eBook Packages: Computer ScienceComputer Science (R0)