Abstract
Action recognition is at the core of egocentric camera-based assistive technologies, as it enables automatic and continuous monitoring of Activities of Daily Living (ADLs) without any conscious effort on the part of the user. This study explores the feasibility of using 2D hand and object pose information for egocentric action recognition. While current literature focuses on 3D hand pose information, our work shows that using 2D skeleton data is a promising approach for hand-based action classification and potentially allows for reduced computational power. The study implements a state-of-the-art transformer-based method to recognise actions. Our approach achieves an accuracy of 95% in validation and 88% in test subsets on the publicly available benchmark, outperforming other existing solutions by 9% and proving that the presented technique offers a successful alternative to 3D-based approaches. Finally, the ablation study shows the significance of each network input and explores potential ways to improve the presented methodology in future research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
https://www.ray-ban.com/usa/ray-ban-stories (Accessed 01.06.2023).
- 2.
References
Bandini, A., Zariffa, J.: Analysis of the hands in egocentric vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020.2986648
Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexible image augmentations. Information 11(2), 125 (2020). https://doi.org/10.3390/info11020125
Carreira, J., Zisserman, A.: Quo Vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017). https://doi.org/10.1109/CVPR.2017.502
Cartas, A., Radeva, P., Dimiccoli, M.: Contextually driven first-person action recognition from videos. In: Presentation at EPIC@ ICCV2017 Workshop, p. 8 (2017)
Damen, D., et al.: Scaling egocentric vision: the dataset. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 753–771. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_44
Das, P., Ortega, A.: Symmetric sub-graph spatio-temporal graph convolution and its application in complex activity recognition. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3215–3219. IEEE (2021). https://doi.org/10.1109/ICASSP39728.2021.9413833
Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=YicbFdNTTy
Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6202–6211 (2019). https://doi.org/10.1109/ICCV.2019.00630
Grauman, K., et al.: Ego4D: around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995–19012 (2022). https://doi.org/10.1109/CVPR52688.2022.01842
Kwon, T., Tekin, B., Stühmer, J., Bogo, F., Pollefeys, M.: H2O: two hands manipulating objects for first person interaction recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10138–10148, October 2021. https://doi.org/10.1109/ICCV48922.2021.00998
Mucha, W., Kampel, M.: Addressing privacy concerns in depth sensors. In: Miesenberger, K., Kouroupetroglou, G., Mavrou, K., Manduchi, R., Covarrubias Rodriguez, M., Penaz, P. (eds.) Computers Helping People with Special Needs. ICCHP-AAATE 2022. LNCS, vol. 13342, pp. 526–533. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08645-8_62
Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12036–12045 (2019). https://doi.org/10.1109/CVPR.2019.01231
Núñez-Marcos, A., Azkune, G., Arganda-Carreras, I.: Egocentric vision-based action recognition: a survey. Neurocomputing 472, 175–197 (2022). https://doi.org/10.1016/j.neucom.2021.11.081
Tekin, B., Bogo, F., Pollefeys, M.: H+O: unified egocentric recognition of 3D hand-object poses and interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4511–4520 (2019). https://doi.org/10.1109/CVPR.2019.00464
Vaswani, A., et al..: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017). https://doi.org/10.5555/3295222.3295349
Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: Yolov7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7464–7475 (2023)
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018). https://doi.org/10.5555/3504035.3504947
Zhan, K., Faux, S., Ramos, F.: Multi-scale conditional random fields for first-person activity recognition. In: 2014 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 51–59. IEEE (2014). https://doi.org/10.1016/j.pmcj.2014.11.004
Zhang, F., et al.: Mediapipe hands: on-device real-time hand tracking. arXiv preprint arXiv:2006.10214 (2020)
Acknowledgements
This work was supported by VisuAAL ITN H2020 (grant agreement No. 861091) and by KIIS Austrian Research Promotion Agency (grant agreement No. 879744).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mucha, W., Kampel, M. (2023). Hands, Objects, Action! Egocentric 2D Hand-Based Action Recognition. In: Christensen, H.I., Corke, P., Detry, R., Weibel, JB., Vincze, M. (eds) Computer Vision Systems. ICVS 2023. Lecture Notes in Computer Science, vol 14253. Springer, Cham. https://doi.org/10.1007/978-3-031-44137-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-44137-0_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44136-3
Online ISBN: 978-3-031-44137-0
eBook Packages: Computer ScienceComputer Science (R0)