3D Hand Gesture Recognition by Analysing Set-of-Joints Trajectories

  • Quentin De Smedt
  • Hazem Wannous
  • Jean-Philippe Vandeborre
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10188)


Hand gesture recognition is recently becoming one of the most attractive field of research in Pattern Recognition. In this paper, a skeleton-based approach is proposed for 3D hand gesture recognition. Specifically, we consider the sequential data of hand geometric configuration to capture the hand shape variation, and explore the temporal character of hand motion. 3D Hand gesture are represented as a set of relevant spatiotemporal motion trajectories of hand-parts in an Euclidean space. Trajectories are then interpreted as elements lying on Riemannian manifold of shape space to capture their shape variations and achieve gesture recognition using a linear SVM classifier.

The proposed approach is evaluated on a challenging hand gesture dataset containing 14 gestures, performed by 20 participants performing the same gesture with two different numbers of fingers. Experimental results show that our skeleton-based approach consistently achieves superior performance over a depth-based approach.


Gesture recognition Riemaniann manifold Hand skeleton Depth image 


  1. 1.
    Kuznetsova, A., Leal-Taixé, L., Rosenhahn, B.: Real-time sign language recognition using a consumer depth camera. In: IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 83–90, December 2013Google Scholar
  2. 2.
    Wang, H., Wang, Q., Chen, X.: Hand posture recognition from disparity cost map. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7725, pp. 722–733. Springer, Heidelberg (2013). Scholar
  3. 3.
    Ren, Z., Yuan, J., Zhang, Z.: Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: ACM International Conference on Multimedia, MM 2011, pp. 1093–1096. ACM, New York (2011)Google Scholar
  4. 4.
    Cheng, H., Dai, Z., Liu, Z.: Image-to-class dynamic time warping for 3D hand gesture recognition. In: 2013 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, July 2013Google Scholar
  5. 5.
    Pugeault, N., Bowden, R.: Spelling it out: real-time ASL fingerspelling recognition. In: IEEE Computer Vision Workshops (ICCV Workshops), pp. 1114–1119, November 2011Google Scholar
  6. 6.
    Dong, C., Leu, M.C., Yin, Z.: American sign language alphabet recognition using microsoft kinect. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 44–52, June 2015Google Scholar
  7. 7.
    Marin, G., Dominio, F., Zanuttigh, P.: Hand gesture recognition with leap motion and kinect devices. In: IEEE International Conference on Image Processing (ICIP), pp. 1565–1569 (2014)Google Scholar
  8. 8.
    Kurakin, A., Zhang, Z., Liu, Z.: A real time system for dynamic hand gesture recognition with a depth sensor. In: 20th European Signal Processing Conference (EUSIPCO), pp. 1975–1979, August 2012Google Scholar
  9. 9.
    Zhang, C., Yang, X., Tian, Y.: Histogram of 3D facets: a characteristic descriptor for hand gesture recognition. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8, April 2013Google Scholar
  10. 10.
    Escalera, S., et al.: ChaLearn looking at people challenge 2014: dataset and results. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 459–473. Springer, Cham (2015). Scholar
  11. 11.
    Monnier, C., German, S., Ost, A.: A multi-scale boosted detector for efficient and robust gesture recognition. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 491–502. Springer, Cham (2015). Scholar
  12. 12.
    Neverova, N., Wolf, C., Taylor, G.W., Nebout, F.: ModDrop: adaptive multi-modal gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell., April 2016Google Scholar
  13. 13.
    Ohn-Bar, E., Trivedi, M.M.: Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans. Intell. Transp. Syst. 15(6), 2368–2377 (2014)CrossRefGoogle Scholar
  14. 14.
    De Smedt, Q., Wannous, H., Vandeborre, J.P.: Skeleton-based dynamic hand gesture recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2016Google Scholar
  15. 15.
    Lee, C.S., Elgammal, A.M.: Modeling view and posture manifolds for tracking. In: IEEE International Conference on Computer Vision, pp. 1–8 (2007)Google Scholar
  16. 16.
    Lui, Y.M.: Advances in matrix manifolds for computer vision. Image Vis. Comput. 30, 380–388 (2012)CrossRefGoogle Scholar
  17. 17.
    Harandi, M.T., Sanderson, C., Shirazi, S., Lovell, B.C.: Kernel analysis on grassmann manifolds for action recognition. Pattern Recogn. Lett. 34, 1906–1915 (2013)CrossRefGoogle Scholar
  18. 18.
    Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D skeletons as points in a lie group. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, pp. 588–595. IEEE Computer Society, Washington, DC (2014)Google Scholar
  19. 19.
    Slama, R., Wannous, H., Daoudi, M., Srivastava, A.: Accurate 3D action recognition using learning on the Grassmann manifold. Pattern Recogn. 48(2), 556–567 (2015)CrossRefGoogle Scholar
  20. 20.
    Slama, R., Wannous, H., Daoudi, M.: 3D human motion analysis framework for shape similarity and retrieval. Image Vis. Comput. 32(2), 131–154 (2014)CrossRefGoogle Scholar
  21. 21.
    Joshi, S.H., Klassen, E., Srivastava, A., Jermyn, I.: A novel representation for Riemannian analysis of elastic curves in \(R^n\). In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, pp. 1–7, June 2007Google Scholar
  22. 22.
    Karcher, H.: Riemannian center of mass and mollifier smoothing. Comm. Pure Appl. Math. 30, 509–541 (1977)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 27:1–27:27 (2011)CrossRefGoogle Scholar
  24. 24.
    Oreifej, O., Liu, Z.: HON4D: histogram of oriented 4D normals for activity recognition from depth sequences. In: IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 716–723 (2013)Google Scholar
  25. 25.
    Ohn-Bar, E., Trivedi, M.M.: Joint angles similarities and HOG2 for action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2013, Portland, OR, USA, 23–28 June 2013, pp. 465–470 (2013)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Quentin De Smedt
    • 1
  • Hazem Wannous
    • 2
  • Jean-Philippe Vandeborre
    • 1
  1. 1.Télécom Lille, CNRS, UMR 9189 - CRIStALLilleFrance
  2. 2.Télécom Lille, Univ. Lille, CNRS, UMR 9189 - CRIStALLilleFrance

Personalised recommendations