Grassmann manifold based dynamic hand gesture recognition using depth data

  • Bindu Verma
  • Ayesha ChoudharyEmail author


In this paper, we propose a novel Grassmann manifold based framework for dynamic hand gesture recognition from depth data. Automated dynamic hand gesture recognition is important for improving man-machine communication and understanding human behavior. It finds various applications such as human computer interaction, ambient assisted living, automated driver assisted systems. We use depth data or skeleton information to detect the fingertip and store the fingertip points to create the trajectory. In fingertip detection using depth data first we detect the hand using the depth data and used hand shape properties such as finger thickness, finger length, finger width and finger orientation angle to find the shape of the hand. If skeleton data is available we use skeleton information to detect the fingertip in each frame. Then geometrical features are extracted and a unique gesture subspaces created using SVD for each feature vector matrix of each gesture set. These gesture subspaces lie on a Grassmann manifold and capture the intra-class variations and increase the inter-class discriminatory power. We apply Grassmann manifold based discriminant analysis for recognizing each test gesture. We perform experiments on standard datasets and the results show that we have achieved recognition accuracy comparable to the state-of-the-art.


Dynamic hand gesture recognition Grassmann Manifold RGB-D sensors Hand gesture Subspace learning 



  1. 1.
    Cabido R, Pantrigo J, Montemayor AS, Núnez JC, Vélez JF (2018) Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition. In: Journal of Pattern Recognition, vol 76, pp 80–94Google Scholar
  2. 2.
    Caputo FM, Prebianca P, Carcangiu A, Spano LD, Giachetti A (2017) A 3 Cent Recognizer: Simple and Effective Retrieval and Classification of Mid-air Gestures from Single 3D Traces. In: Smart tools and apps for graphic eurographics associationGoogle Scholar
  3. 3.
    Chen F, Fu C, Huang C (2003) Hand gesture recognition using a real-time tracking method and hidden markov models. Image Video Comput 21(8):745–758CrossRefGoogle Scholar
  4. 4.
    Chunyong M, Wang A, Chen G, Xu C (2018) Hand joints-based gesture recognition for noisy dataset using nested interval unscented Kalman filter with LSTM network. In: The visual computer, vol 34, pp 1053–1063Google Scholar
  5. 5.
    Darrell T, Pentland A (1993) Space-time gestures. In: IEEE Conference on computer vision and pattern recognition, pp 335–340Google Scholar
  6. 6.
    De Smedt Q, Wannous H, Vandeborre J-P (2016) 3D Hand Gesture Recognition by Analyzing Set-of-Joints Trajectories. In: International workshop on understanding human activities through 3d sensors. Springer, Cham, 86–97Google Scholar
  7. 7.
    De Smedt Q, Wannous H, Vandeborre J-P (2016) Skeleton-based dynamic hand gesture recognition. In: IEEE Conference on computer vision and pattern recognition workshops, pp 1–9Google Scholar
  8. 8.
    Douglas D, Peucker T (1973) Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Can Cartogr 10(2):112–122CrossRefGoogle Scholar
  9. 9.
    Guo H, Wang G, Chen X, Zhang L (2018) Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. In: 2017 IEEE International Conference on Image Processing (ICIP), pp 2881–2885Google Scholar
  10. 10.
    Hamm J, Lee DD (2008) GRassmann discriminant analysis A unifying view on subspace-based learning. In: 25Th international conference on machine learning, pp 376–383Google Scholar
  11. 11.
    Harandi MT, Sanderson C, Shirazi S, Lovell BC (2011) Graph embedding discriminant analysis on Grassmannian manifolds for improved image set matching. In: IEEE Conference on computer vision and pattern recognition, pp 2705–2712Google Scholar
  12. 12.
    Huang Z, Wang R, Shan S, Chen X (2015) Projection metric learning on Grassmann manifold with application to video based face recognition. In: IEEE Conference on computer vision and pattern recognition, pp 140–149Google Scholar
  13. 13.
    Klaser A, Marszałek M, Schmid C (2008) A Spatio-temporal descriptor based on 3D-gradients. In: 9Th british machine vision conference, pp 275–285Google Scholar
  14. 14.
    Kurakin A, Zhang Z, Liu Z (2012) A real time system for dynamic hand gesture recognition with a depth sensor. In: European signal processing conference, pp 1975–1979Google Scholar
  15. 15.
    Liu L, Shao L (2013) Learning discriminative representations from RGB-D video data. In: Twenty-third international joint conference on artificial intelligence (IJCAI), vol 4, pp 8Google Scholar
  16. 16.
    Mitra S, Acharya T (2007) Gesture recognition: A survey. IEEE Trans Syst Man CybernPart C (Appl Rev) 37(3):311–324CrossRefGoogle Scholar
  17. 17.
    Molchanov P, Gupta S, Kim K, Kautz J (2015) Hand gesture recognition with 3D convolutional neural networks. In: IEEE Conference on computer vision and pattern recognition workshops, pp 1–7Google Scholar
  18. 18.
    Monnier C, German S, Ost A (2014) A multi-scale boosted detector for efficient and robust gesture recognition. In: European conference on computer vision workshops, pp 491–502CrossRefGoogle Scholar
  19. 19.
    Nagi J, Ducatelle F, Di Caro GA, Cireṡan D, Meier U, Giusti A, Nagi F, Schmidhuber J, Gambardella LM (2011) Max-pooling convolutional neural networks for vision-based hand gesture recognition. In: 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA). IEEE, pp 342–347Google Scholar
  20. 20.
    Ohn-Bar E, Trivedi MM (2014) Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans Intell Transp Syst 15(6):2368–2377CrossRefGoogle Scholar
  21. 21.
    Pavlovic V, Sharma R, Huang T (1997) Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Trans Pattern Anal Mach Intell 19 (7):677–695CrossRefGoogle Scholar
  22. 22.
    Plouffe G, Cretu A-M (2016) Static and dynamic hand gesture recognition in depth data using Dynamic Time Warping. IEEE Trans Instrum Meas 65(2):305–316CrossRefGoogle Scholar
  23. 23.
    Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120CrossRefGoogle Scholar
  24. 24.
    Sathyanarayana S, Littlewort G, Bartlett M (2013) Hand gestures for intelligent tutoring systems: dataset, techniques and evaluation. In: IEEE International conference on computer vision workshops, pp 769–776Google Scholar
  25. 25.
    Shen X, Hua G, Williams L, Wu Y (2012) Dynamic hand gesture recognition: an exemplar-based approach from motion divergence fields. Image Vis Comput 30 (3):227–235CrossRefGoogle Scholar
  26. 26.
    Suarez J, Murphy RR (2012) Hand gesture recognition with depth images: a review. In: The 21st IEEE international symposium on robot and human interactive communication, pp 411–417Google Scholar
  27. 27.
    Verma B, Choudhary A (2017) Unsupervised learning based static hand gesture recognition from rgb-d sensor. Springer Adv Intell Syst Comput (SoCPAR 2016) 614:1–8Google Scholar
  28. 28.
    Wang H, Kläser A, Schmid C, Liu C-L (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79MathSciNetCrossRefGoogle Scholar
  29. 29.
    Wang C, Liu Z, Chan S-C (2015) Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed 17(1):29–39CrossRefGoogle Scholar
  30. 30.
    Xi F, Moutarde W, Devineau G, Yan J (2018) Deep Learning for Hand Gesture Recognition on Skeletal Data. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 106–113Google Scholar
  31. 31.
    Yang MH, Ahuja N, Tabb M (2002) Extraction of 2D motion trajectories and its application to hand gesture recognition. IEEE Trans Pattern Anal Mach Intell 24 (8):1061–1074CrossRefGoogle Scholar
  32. 32.
    Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the kinect sensor. IEEE Trans Circ Syst Video Technol 24(11):1935–1944CrossRefGoogle Scholar
  33. 33.
    Yu M, Liu L, Shao L (2016) Structure-preserving binary representations for RGB-D action recognition. IEEE Trans Pattern Anal Mach Intell 38(8):1651–1664CrossRefGoogle Scholar
  34. 34.
    Zhang C, Yang X, Tian Y (2013) Histogram of 3D facets: a characteristic descriptor for hand gesture recognition. In: IEEE International conference and workshops on automatic face and gesture recognition, pp 1–8Google Scholar
  35. 35.
    Zhu G, Zhang L, Shen P, Song J (2017) Multimodal gesture recognition using 3d convolution and convolutional lstm. IEEE Access 5:4517–4524CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Computer and System SciencesJawaharlal Nehru UniversityNew DelhiIndia

Personalised recommendations