Abstract
Nowadays, the trend is to use hand gestures to interact with digital devices such as computers, robots, drones, VR interfaces, etc. While interacting with digital devices, selection, pick and place, and navigation are important tasks which can be performed using pointing gestures. Thus, detection of pointing gestures is an important step for pointing gesture based interaction. In computer vision-based analysis of gestures, depth images of the hand region have been predominantly used. Currently, the only existing method to detect pointing gesture from depth images of the hand region has sub optimal performance as shown in our experiments. This can be attributed to the lack of a large data-set that could be used to detect pointing gestures. To overcome this limitation, we create a new large data-set (1,00,395 samples) for pointing gesture detection using depth images of the hand region. The data-set has a large variation in the hand poses and in the depth of the hand with respect to the depth sensor. The data-set will be made publicly available. We also propose a 3D convolutional neural network based real-time technique for pointing gesture detection from depth images of the hand region. The proposed technique performs much better than the existing technique with respect to various evaluation measures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The link to the data-set is at https://github.com/shomedas/PGD.
References
Amir, A., et al.: A low power, fully event-based gesture recognition system. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7243–7252 (2017)
Benitez-Garcia, G., Olivares-Mercado, J., Sanchez-Perez, G., Yanai, K.: IPN hand: a video dataset and benchmark for real-time continuous hand gesture recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4340–4347. IEEE (2021)
Boboc, R.G., Dumitru, A.I., Antonya, C.: Point-and-command paradigm for interaction with assistive robots. Int. J. Adv. Rob. Syst. 12(6), 75 (2015)
Bojja, A.K., et al.: HandSeg: an automatically labeled dataset for hand segmentation from depth images. In: 2019 16th Conference on Computer and Robot Vision (CRV), pp. 151–158. IEEE (2019)
Boulabiar, M.-I., Burger, T., Poirier, F., Coppin, G.: A low-cost natural user interaction based on a camera hand-gestures recognizer. In: Jacko, J.A. (ed.) HCI 2011. LNCS, vol. 6762, pp. 214–221. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21605-3_24
Cordo, S., Bernardino, A., Gomes, R.: Fingerpointer-pointing gestures for collaborative robots (2015)
Droeschel, D., Stückler, J., Behnke, S.: Learning to interpret pointing gestures with a time-of-flight camera. In: Proceedings of the 6th International Conference on Human-Robot Interaction, pp. 481–488 (2011)
Gromov, B., Gambardella, L.M., Giusti, A.: Robot identification and localization with pointing gestures. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3921–3928. IEEE (2018)
Hoang, V.T.: Hgm-4: a new multi-cameras dataset for hand gesture recognition. Data Brief 30, 105676 (2020)
Huang, Y., Liu, X., Zhang, X., Jin, L.: A pointing gesture based egocentric interaction system: dataset, approach and application. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16–23 (2016)
Joo, S.I., Weon, S.H., Choi, H.I.: Real-time depth-based hand detection and tracking. Sci. World J. 2014 (2014)
Joo, S.I., Weon, S.H., Hong, J.M., Choi, H.I.: Hand detection in depth images using features of depth difference. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV), p. 1. The Steering Committee of The World Congress in Computer Science, Computer (2013)
Karbasi, M., Bhatti, Z., Nooralishahi, P., Shah, A., Mazloomnezhad, S.M.R.: Real-time hands detection in depth image by using distance with kinect camera. Int. J. Internet Things 4(1A), 1–6 (2015)
Lai, Y., Wang, C., Li, Y., Ge, S.S., Huang, D.: 3D pointing gesture recognition for human-robot interaction. In: Control and Decision Conference (CCDC), 2016 Chinese, pp. 4959–4964. IEEE (2016)
Marin, G., Fraccaro, M., Donadeo, M., Dominio, F., Zanuttigh, P.: Palm area detection for reliable hand gesture recognition. In: Proceedings of MMSP, vol. 2013, p. 120 (2013)
Matikainen, P., Pillai, P., Mummert, L., Sukthankar, R., Hebert, M.: Prop-free pointing detection in dynamic cluttered environments. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 374–381. IEEE (2011)
Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., Kautz, J.: Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4207–4215 (2016)
Nguyen, H.D., Na, I.S., Kim, S.H.: Hand segmentation and fingertip tracking from depth camera images using deep convolutional neural network and multi-task SegNet. arXiv preprint arXiv:1901.03465 (2019)
Nickel, K., Scemann, E., Stiefelhagen, R.: 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction scenario. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings. pp. 565–570. IEEE (2004)
Nuzzi, C., Pasinetti, S., Pagani, R., Coffetti, G., Sansoni, G.: Hands: an RGB-D dataset of static hand-gestures for human-robot interaction. Data Brief 35, 106791 (2021)
Obo, T., Kawabata, R., Kubota, N.: Pointing gesture detection for human-robot communication in informationally structured space. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–5. IEEE (2017)
Park, C.B., Lee, S.W.: Real-time 3D pointing gesture recognition for mobile robots with cascade HMM and particle filter. Image Vis. Comput. 29(1), 51–63 (2011)
Shukla, D., Erkent, Ö., Piater, J.: A multi-view hand gesture RGB-D dataset for human-robot interaction scenarios. In: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 1084–1091. IEEE (2016)
Wikipedia Contributors: Precision and recall – Wikipedia, the free encyclopedia (2022). https://en.wikipedia.org/w/index.php?title=Precision_and_recall &oldid=1086319787. Accessed 6 May 2022
Wolf, M.T., Assad, C., Vernacchia, M.T., Fromm, J., Jethani, H.L.: Gesture-based robot control with variable autonomy from the JPL BioSleeve. In: 2013 IEEE International Conference on Robotics and Automation, pp. 1160–1165. IEEE (2013)
Zhang, Y., Cao, C., Cheng, J., Lu, H.: EgoGesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans. Multimedia 20(5), 1038–1050 (2018)
Zhang, Z., Zhang, Z., Zhang, H., Zeng, D.: Accurate per-pixel hand detection from a single depth image. Opt. Eng. 56(3), 033107 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Das, S.S. (2022). A Data-Set and a Real-Time Method for Detection of Pointing Gesture from Depth Images. In: Raman, B., Murala, S., Chowdhury, A., Dhall, A., Goyal, P. (eds) Computer Vision and Image Processing. CVIP 2021. Communications in Computer and Information Science, vol 1567. Springer, Cham. https://doi.org/10.1007/978-3-031-11346-8_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-11346-8_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-11345-1
Online ISBN: 978-3-031-11346-8
eBook Packages: Computer ScienceComputer Science (R0)