Skip to main content
SpringerLink
Log in

Grassmann manifold based dynamic hand gesture recognition using depth data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel Grassmann manifold based framework for dynamic hand gesture recognition from depth data. Automated dynamic hand gesture recognition is important for improving man-machine communication and understanding human behavior. It finds various applications such as human computer interaction, ambient assisted living, automated driver assisted systems. We use depth data or skeleton information to detect the fingertip and store the fingertip points to create the trajectory. In fingertip detection using depth data first we detect the hand using the depth data and used hand shape properties such as finger thickness, finger length, finger width and finger orientation angle to find the shape of the hand. If skeleton data is available we use skeleton information to detect the fingertip in each frame. Then geometrical features are extracted and a unique gesture subspaces created using SVD for each feature vector matrix of each gesture set. These gesture subspaces lie on a Grassmann manifold and capture the intra-class variations and increase the inter-class discriminatory power. We apply Grassmann manifold based discriminant analysis for recognizing each test gesture. We perform experiments on standard datasets and the results show that we have achieved recognition accuracy comparable to the state-of-the-art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. http://lshao.staff.shef.ac.uk/data/SheffieldKinectGesture.htm

  2. http://www-rech.telecom-lille.fr/DHGdataset/,de2016skeleton

  3. https://github.com/davidespano/3cent-dataset,caputo20173

References

  1. Cabido R, Pantrigo J, Montemayor AS, Núnez JC, Vélez JF (2018) Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition. In: Journal of Pattern Recognition, vol 76, pp 80–94

  2. Caputo FM, Prebianca P, Carcangiu A, Spano LD, Giachetti A (2017) A 3 Cent Recognizer: Simple and Effective Retrieval and Classification of Mid-air Gestures from Single 3D Traces. In: Smart tools and apps for graphic eurographics association

  3. Chen F, Fu C, Huang C (2003) Hand gesture recognition using a real-time tracking method and hidden markov models. Image Video Comput 21(8):745–758

    Article  Google Scholar 

  4. Chunyong M, Wang A, Chen G, Xu C (2018) Hand joints-based gesture recognition for noisy dataset using nested interval unscented Kalman filter with LSTM network. In: The visual computer, vol 34, pp 1053–1063

  5. Darrell T, Pentland A (1993) Space-time gestures. In: IEEE Conference on computer vision and pattern recognition, pp 335–340

  6. De Smedt Q, Wannous H, Vandeborre J-P (2016) 3D Hand Gesture Recognition by Analyzing Set-of-Joints Trajectories. In: International workshop on understanding human activities through 3d sensors. Springer, Cham, 86–97

  7. De Smedt Q, Wannous H, Vandeborre J-P (2016) Skeleton-based dynamic hand gesture recognition. In: IEEE Conference on computer vision and pattern recognition workshops, pp 1–9

  8. Douglas D, Peucker T (1973) Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Can Cartogr 10(2):112–122

    Article  Google Scholar 

  9. Guo H, Wang G, Chen X, Zhang L (2018) Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. In: 2017 IEEE International Conference on Image Processing (ICIP), pp 2881–2885

  10. Hamm J, Lee DD (2008) GRassmann discriminant analysis A unifying view on subspace-based learning. In: 25Th international conference on machine learning, pp 376–383

  11. Harandi MT, Sanderson C, Shirazi S, Lovell BC (2011) Graph embedding discriminant analysis on Grassmannian manifolds for improved image set matching. In: IEEE Conference on computer vision and pattern recognition, pp 2705–2712

  12. Huang Z, Wang R, Shan S, Chen X (2015) Projection metric learning on Grassmann manifold with application to video based face recognition. In: IEEE Conference on computer vision and pattern recognition, pp 140–149

  13. Klaser A, Marszałek M, Schmid C (2008) A Spatio-temporal descriptor based on 3D-gradients. In: 9Th british machine vision conference, pp 275–285

  14. Kurakin A, Zhang Z, Liu Z (2012) A real time system for dynamic hand gesture recognition with a depth sensor. In: European signal processing conference, pp 1975–1979

  15. Liu L, Shao L (2013) Learning discriminative representations from RGB-D video data. In: Twenty-third international joint conference on artificial intelligence (IJCAI), vol 4, pp 8

  16. Mitra S, Acharya T (2007) Gesture recognition: A survey. IEEE Trans Syst Man CybernPart C (Appl Rev) 37(3):311–324

    Article  Google Scholar 

  17. Molchanov P, Gupta S, Kim K, Kautz J (2015) Hand gesture recognition with 3D convolutional neural networks. In: IEEE Conference on computer vision and pattern recognition workshops, pp 1–7

  18. Monnier C, German S, Ost A (2014) A multi-scale boosted detector for efficient and robust gesture recognition. In: European conference on computer vision workshops, pp 491–502

    Chapter  Google Scholar 

  19. Nagi J, Ducatelle F, Di Caro GA, Cireṡan D, Meier U, Giusti A, Nagi F, Schmidhuber J, Gambardella LM (2011) Max-pooling convolutional neural networks for vision-based hand gesture recognition. In: 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA). IEEE, pp 342–347

  20. Ohn-Bar E, Trivedi MM (2014) Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans Intell Transp Syst 15(6):2368–2377

    Article  Google Scholar 

  21. Pavlovic V, Sharma R, Huang T (1997) Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Trans Pattern Anal Mach Intell 19 (7):677–695

    Article  Google Scholar 

  22. Plouffe G, Cretu A-M (2016) Static and dynamic hand gesture recognition in depth data using Dynamic Time Warping. IEEE Trans Instrum Meas 65(2):305–316

    Article  Google Scholar 

  23. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  24. Sathyanarayana S, Littlewort G, Bartlett M (2013) Hand gestures for intelligent tutoring systems: dataset, techniques and evaluation. In: IEEE International conference on computer vision workshops, pp 769–776

  25. Shen X, Hua G, Williams L, Wu Y (2012) Dynamic hand gesture recognition: an exemplar-based approach from motion divergence fields. Image Vis Comput 30 (3):227–235

    Article  Google Scholar 

  26. Suarez J, Murphy RR (2012) Hand gesture recognition with depth images: a review. In: The 21st IEEE international symposium on robot and human interactive communication, pp 411–417

  27. Verma B, Choudhary A (2017) Unsupervised learning based static hand gesture recognition from rgb-d sensor. Springer Adv Intell Syst Comput (SoCPAR 2016) 614:1–8

    Google Scholar 

  28. Wang H, Kläser A, Schmid C, Liu C-L (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79

    Article  MathSciNet  Google Scholar 

  29. Wang C, Liu Z, Chan S-C (2015) Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed 17(1):29–39

    Article  Google Scholar 

  30. Xi F, Moutarde W, Devineau G, Yan J (2018) Deep Learning for Hand Gesture Recognition on Skeletal Data. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 106–113

  31. Yang MH, Ahuja N, Tabb M (2002) Extraction of 2D motion trajectories and its application to hand gesture recognition. IEEE Trans Pattern Anal Mach Intell 24 (8):1061–1074

    Article  Google Scholar 

  32. Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the kinect sensor. IEEE Trans Circ Syst Video Technol 24(11):1935–1944

    Article  Google Scholar 

  33. Yu M, Liu L, Shao L (2016) Structure-preserving binary representations for RGB-D action recognition. IEEE Trans Pattern Anal Mach Intell 38(8):1651–1664

    Article  Google Scholar 

  34. Zhang C, Yang X, Tian Y (2013) Histogram of 3D facets: a characteristic descriptor for hand gesture recognition. In: IEEE International conference and workshops on automatic face and gesture recognition, pp 1–8

  35. Zhu G, Zhang L, Shen P, Song J (2017) Multimodal gesture recognition using 3d convolution and convolutional lstm. IEEE Access 5:4517–4524

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer and System Sciences, Jawaharlal Nehru University, New Delhi, India

    Bindu Verma & Ayesha Choudhary

Authors
  1. Bindu Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayesha Choudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayesha Choudhary.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Verma, B., Choudhary, A. Grassmann manifold based dynamic hand gesture recognition using depth data. Multimed Tools Appl 79, 2213–2237 (2020). https://doi.org/10.1007/s11042-019-08266-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08266-w

Keywords

Search

Navigation