Skip to main content

Hand gesture recognition with jointly calibrated Leap Motion and depth sensor

Multimedia Tools and Applications volume 75, pages 14991–15015 (2016)Cite this article

Abstract

Novel 3D acquisition devices like depth cameras and the Leap Motion have recently reached the market. Depth cameras allow to obtain a complete 3D description of the framed scene while the Leap Motion sensor is a device explicitly targeted for hand gesture recognition and provides only a limited set of relevant points. This paper shows how to jointly exploit the two types of sensors for accurate gesture recognition. An ad-hoc solution for the joint calibration of the two devices is firstly presented. Then a set of novel feature descriptors is introduced both for the Leap Motion and for depth data. Various schemes based on the distances of the hand samples from the centroid, on the curvature of the hand contour and on the convex hull of the hand shape are employed and the use of Leap Motion data to aid feature extraction is also considered. The proposed feature sets are fed to two different classifiers, one based on multi-class SVMs and one exploiting Random Forests. Different feature selection algorithms have also been tested in order to reduce the complexity of the approach. Experimental results show that a very high accuracy can be obtained from the proposed method. The current implementation is also able to run in real-time.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Aha DW, Bankert RL (1996) A comparative evaluation of sequential feature selection algorithms. In: Fisher D, Lenz H-J (eds) Learning from data, volume 112 of lecture notes in statistics. Springer, New York, pp 199–206

    Google Scholar 

  2. Ballan L, Taneja A, Gall J, Van Gool L, Pollefeys M (2012) Motion capture of hands in action using discriminative salient points. In: European conference on computer vision (ECCV). Florence

  3. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MathSciNet  MATH  Google Scholar 

  4. Chang C-C, Lin C-J (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Tech 2:27:1–27:27

    Article  Google Scholar 

  5. Chen Y-W, Lin C-J (2006) Combining svms with various feature selection strategies. In: Feature extraction, pp 315–324. Springer

  6. Dal Mutto C, Zanuttigh P, Cortelazzo GM (2012) Time-of-flight cameras and microsoft kinect. SpringerBriefs in electrical and computer engineering. Springer

  7. Dominio F, Donadeo M, Marin G, Zanuttigh P, Cortelazzo GM (2013) Hand gesture recognition with depth data. In: Proceedings of the 4th ACM/IEEE international workshop on Analysis and retrieval of tracked events and motion in imagery stream, pp 9–16. ACM

  8. Dominio F, Donadeo M, Zanuttigh P (2014) Combining multiple depth-based descriptors for hand gesture recognition. Pattern Recogn Lett 50:101–111

    Article  Google Scholar 

  9. Guerrero-Rincon C, Uribe-Quevedo A, Leon-Rodriguez H, Park J-O (2013) Hand-based tracking animatronics interaction. In: 44th international symposium on robotics (ISR), 2013, pp 1–3

  10. Herrera D, Kannala J, Heikkilä J (2012) Joint depth and color camera calibration with distortion correction. IEEE Trans Pattern Anal Mach Intell 34(10):2058–2064

    Article  Google Scholar 

  11. Keskin C, Kirac F, Kara YE, Akarun L (2011) Real time hand pose estimation using depth sensors. In: ICCV workshops, pp 1228–1234

  12. Kosmopoulos DI, Doulamis A, Doulamis N (2005) Gesture-based video summarization. In: IEEE international conference on image processing, 2005. ICIP 2005, vol 3, pp III–1220–3

  13. Kumar N, Belhumeur PN, Biswas A, Jacobs DW, Kress WJ, Lopez I, Soares JVB (2012) Leafsnap: a computer vision system for automatic plant species identification. In: Fitzgibbon A, Lazebnik S, Perona P, Sato Y, Schmid C (eds) Computer vision ECCV 2012. Lecture notes in computer science. Springer Berlin Heidelberg, pp 502–516

  14. Kurakin A, Zhang Z, Liu Z (2012) A real time system for dynamic hand gesture recognition with a depth sensor. In: Proceedings of the 20th European signal processing conference, EUSIPCO 2012. Bucharest, Romania, pp 1975–1979

  15. Manay S, Cremers D, Hong B-W, Yezzi AJ, Soatto S (2006) Integral invariants for shape matching. IEEE Trans PAMI 28(10):1602–1618

    Article  MathSciNet  MATH  Google Scholar 

  16. Marin G, Dominio F, Zanuttigh P (2014) Hand gesture recognition with leap motion and kinect devices. In: Proceedings of IEEE international conference on image processing (ICIP). Paris, France

  17. Marin G, Fraccaro M, Donadeo M, Dominio F, Zanuttigh P (2013) Palm area detection for reliable hand gesture recognition. In: Proceedings of Multimedia Signal Processing, Pula, Italy

  18. Mohandes M, Aliyu S, Deriche M (2014) Arabic sign language recognition using the leap motion controller. In: IEEE 23rd international symposium on industrial electronics (ISIE), pp 960–965

  19. Nigam I, Vatsa M, Singh R (2014) Leap signature recognition using hoof and hot features. In: Proceedings of IEEE international conference on image processing (ICIP). Paris, France

  20. Pedersoli F, Adami N, Benini S, Leonardi R (2012) Xkin - extendable hand pose and gesture recognition library for kinect. In: Proceedings of ACM conference on multimedia 2012 - open source competition. Nara, Japan

  21. Potter LE, Araullo J, Carter L (2013) The leap motion controller: a view on sign language. In: Proceedings of the 25th australian computer-human interaction conference: augmentation, application, innovation, collaboration, OzCHI ’13, pp 175–178. ACM, New York

  22. Ren Z, Meng J, Yuan J (2011) Depth camera based hand gesture recognition and its applications in human-computer-interaction. In: Proceedings of ICICS, pp 1–5

  23. Ren Z, Yuan J, Zhang Z (2011) Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: Proceedings of ACM Conference on Multimedia, pp 1093–1096. ACM

  24. Shotton J, Fitzgibbon A, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. In: IEEE conference on computer vision and pattern recognition (CVPR), 2011, pp 1297–1304. IEEE

  25. Suryanarayan P, Subramanian A, Mandalapu D (2010) Dynamic hand pose recognition using depth data. In: Proceedings of ICPR, pp 3105–3108

  26. Vikram S, Li L, Russell S (2013) Writing and sketching in the air, recognizing and controlling on the fly. In: CHI ’13 extended abstracts on human factors in computing systems. CHI EA ’13, Paris, France. ACM, New York, pp 1179–1184. doi:10.1145/2468356.2468567

  27. Wachs JP, Kölsch M, Stern H, Edan Y (2011) Vision-based hand-gesture applications. Commun ACM 54(2):60–71

    Article  Google Scholar 

  28. Wang J, Liu Z, Chorowski J, Chen Z, Wu Y (2012) Robust 3D action recognition with random occupancy patterns. In: Fitzgibbon A, Lazebnik S, Perona P, Sato Y, Schmid C (eds) Computer vision ECCV 2012. Lecture notes in computer science. Springer Berlin Heidelberg, pp 872–885. doi:10.1007/978-3-642-33709-3_62

  29. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Engineering, University of Padova, Padova, Italy

    Giulio Marin, Fabio Dominio & Pietro Zanuttigh

Authors
  1. Giulio Marin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fabio Dominio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pietro Zanuttigh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pietro Zanuttigh.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Marin, G., Dominio, F. & Zanuttigh, P. Hand gesture recognition with jointly calibrated Leap Motion and depth sensor. Multimed Tools Appl 75, 14991–15015 (2016). https://doi.org/10.1007/s11042-015-2451-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2451-6

Keywords

  • Depth
  • Gesture recognition
  • Calibration
  • Kinect
  • Leap Motion
  • SVM