Skip to main content
SpringerLink
Log in

Kinect-based hand gesture recognition using trajectory information, hand motion dynamics and neural networks

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Hand gestures are spatio-temporal patterns which can be characterized by collections of spatio-temporal features. Recognition of hand gestures is to find the re-occurrences of such spatio-temporal patterns through pattern matching. However, dynamic hand gestures have many obstacles for accurate recognition, including poor lighting conditions, camera’s inability to capture dynamic gesture in focus, occlusion due to finger movement, color variations due to lighting conditions. The Microsoft Kinect device provides an effective way to solve the above issues and also provides the skeleton for more convenient hand localization and tracking. The aim of this study is to develop a new trajectory-based method for hand gesture recognition using Kinect. In the first step, trajectory-based hand gesture features including spatial position and direction of fingertips, are derived from Kinect. The properties associated with the hand motion dynamics are preserved in these features. In the second step, radial basis function (RBF) neural networks are employed to model and approximate the hand motion dynamics derived from different hand gestures which represent Arabic numbers (0–9) and English alphabets (A–Z). The trained patterns of the approximated hand motion dynamics is stored in constant RBF networks. In the last step, a bank of dynamical estimators is constructed for all the training patterns, in which the constant RBF networks are embedded in. By comparing the set of estimators with a test gesture pattern, a set of recognition errors are generated, in which the average \(L_1\) norms of the errors are taken as the recognition measure based on the smallest error principle. Finally, experiments are carried out to assess the performance of the proposed method compared with other state-of-the-art approaches. By using the twofold and tenfold cross-validation styles, the correct recognition rates for Arabic numbers (0–9) and English alphabets (A–Z) are reported to be \(95.83\%, 97.25\%\), and \(91.35\%, 92.63\%\), respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Beh J, Han D, Ko H (2014) Rule-based trajectory segmentation for modeling hand motion trajectory. Pattern Recognit 47(4):1586–1601

    Article  Google Scholar 

  • Cheng H, Dai Z, Liu Z, Zhao Y (2016) An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition. Pattern Recognit 55:137–147

    Article  Google Scholar 

  • Farrell J (1988) Stability and approximator convergence in nonparametric nonlinear adaptive control. IEEE Trans Neural Netw 9(5):1008–1020

    Article  Google Scholar 

  • Farzad A, Mashayekhi H, Hassanpour H (2017) A comparative performance analysis of different activation functions in LSTM networks for classification. Neural Comput Appl. https://doi.org/10.1007/s00521-017-3210-6

    Article  Google Scholar 

  • Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334

    Article  Google Scholar 

  • Han F, Reily B, Hoff W, Zhang H (2017) Space-time representation of people based on 3D skeletal data: a review. Comput Vis Image Underst 158:85–105

    Article  Google Scholar 

  • Herekar RR, Dhotre SR (2014) Handwritten character recognition based on zoning using Euler number for English alphabets and numerals. IOSR J Comput Eng 16(4):75–88

    Article  Google Scholar 

  • Ibañez R, Soria Á, Teyseyre A, Rodréguez G, Campo M (2017) Approximate string matching: a lightweight approach to recognize gestures with Kinect. Pattern Recognit 62:73–86

    Article  Google Scholar 

  • Jadooki S, Mohamad D, Saba T, Almazyad AS, Rehman A (2017) Fused features mining for depth-based hand gesture recognition to classify blind human communication. Neural Comput Appl 28(11):3285–3294

    Article  Google Scholar 

  • Jain S, Chauhan R (2018) Recognition of handwritten digits using DNN, CNN, and RNN. In: International conference on advances in computing and data sciences, pp 239–248

  • Kane L, Khanna P (2016) A framework to plot and recognize hand motion trajectories towards development of non-tactile interfaces. Proc Comput Sci 84:6–13

    Article  Google Scholar 

  • Kiliboz NC, Gudukbay U (2015) A hand gesture recognition technique for human-computer interaction. J Vis Commun Image Represent 28:97–104

    Article  Google Scholar 

  • Kim IC, Chien SI (2001) Analysis of 3d hand trajectory gestures using stroke-based composite hidden markov models. Appl Intell 15(2):131–143

    Article  MATH  Google Scholar 

  • Kundu S, Chhabra HS, Ara SS, Mishra RP (2017) Optical character recognition using 26-point feature extraction and ANN. Int J Adv Res Comput Sci Softw Eng 7(5):156–162

    Article  Google Scholar 

  • LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  • Leite DQ, Duarte JC, Neves LP, de Oliveira JC, Giraldi GA (2016) Hand gesture recognition from depth and infrared Kinect data for CAVE applications interaction. Multimed Tools Appl 76(20):20423–20455

    Article  Google Scholar 

  • Lu G, Zhou Y, Li X, Kudo M (2016) Efficient action recognition via local position offset of 3D skeletal body joints. Multimed Tools Appl 75(6):3479–3494

    Article  Google Scholar 

  • Lu W, Tong Z, Chu J (2016) Dynamic hand gesture recognition with Leap Motion controller. IEEE Signal Process Lett 23(9):1188–1192

    Article  Google Scholar 

  • Lun R, Zhao W (2015) A survey of applications and human motion recognition with microsoft kinect. Int J Pattern Recognit Artif Intell 29(5):1555008

    Article  Google Scholar 

  • Maqueda AI, del-Blanco CR, Jaureguizar F, GarcaN N (2015) Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns. Comput Vis Image Underst 141:126–137

    Article  Google Scholar 

  • Marin G, Dominio F, Zanuttigh P (2016) Hand gesture recognition with jointly calibrated Leap Motion and depth sensor. Multimed Tools Appl 75(22):14991–15015

    Article  Google Scholar 

  • Misra S, Singha J, Laskar RH (2018) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135

    Article  Google Scholar 

  • Oksuz O, Gudukbay U, Cetin AE (2008) A video-based text and equation editor for LaTeX. Eng Appl Artif Intell 21(6):952–960

    Article  Google Scholar 

  • Patwardhan KS, Roy SD (2007) Hand gesture modelling and recognition involving changing shapes and trajectories, using a Predictive EigenTracker. Pattern Recognit Lett 28(3):329–334

    Article  Google Scholar 

  • Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: a review. Comput Vis Image Understand 141:152–165

    Article  Google Scholar 

  • Qiao J, Wang G, Li W, Chen M (2018) An adaptive deep Q-learning strategy for handwritten digit recognition. Neural Netw 107:61–71

    Article  Google Scholar 

  • Raheja JL, Minhas M, Prashanth D, Shah T, Chaudhary A (2015) Robust gesture recognition using Kinect: a comparison between DTW and HMM. Optik-Int J Light Electron Opt 126(11):1098–1104

    Article  Google Scholar 

  • Raheja JL, Chandra M, Chaudhary A (2017) 3D gesture based real-time object selection and recognition. Pattern Recognit Lett. https://doi.org/10.1016/j.patrec.2017.09.034

    Article  Google Scholar 

  • Rautaray SS, Agrawal A (2017) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54

    Article  Google Scholar 

  • Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  • Roh MC, Fazli S, Lee SW (2016) Selective temporal filtering and its application to hand gesture recognition. Appl Intell 45(2):255–264

    Article  Google Scholar 

  • Sahoo MK, Nayak J, Mohapatra S, Nayak BK, Behera HS (2015) Character recognition using firefly based back propagation neural network. In: Computational intelligence in data mining, vol 2. Springer, New Delhi

  • Singha J, Misra S, Laskar RH (2016) Effect of variation in gesticulation pattern in dynamic hand gesture recognition system. Neurocomputing 208:269–280

    Article  Google Scholar 

  • Song Y, Demirdjian D, Davis R (2012) Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans Interact Intell Syst 2(1):5

    Article  Google Scholar 

  • Stern H, Shmueli M, Berman S (2013) Most discriminating segment Longest common subsequence (MDSLCS) algorithm for dynamic hand gesture classification. Pattern Recognit Lett 34(15):1980–1989

    Article  Google Scholar 

  • Suk HI, Sin BK, Lee SW (2010) Hand gesture recognition based on dynamic Bayesian network framework. Pattern Recognit 43(9):3059–3072

    Article  MATH  Google Scholar 

  • Wang C, Hill DJ (2006) Learning from neural control. IEEE Trans Neural Netw 17(1):130–146

    Article  Google Scholar 

  • Wang C, Hill DJ (2007) Deterministic learning and rapid dynamical pattern recognition. IEEE Trans Neural Netw 18(3):617–630

    Article  Google Scholar 

  • Wang C, Hill DJ (2009) Deterministic learning theory for identification, recognition and control. CRC Press, Boca Raton

    Google Scholar 

  • Wang C, Chen T, Chen G, Hill DJ (2009) Deterministic learning of nonlinear dynamical systems. Int J Bifurc Chaos 19(4):1307–1328

    Article  MathSciNet  MATH  Google Scholar 

  • Wang C, Liu Z, Chan SC (2015) Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed 17(1):29–39

    Article  Google Scholar 

  • Xu S, Xue Y (2017) A long term memory recognition framework on multi-complexity motion gestures. In: IEEE international conference on document analysis and recognition, pp 201–205

  • Yang X, Tian YL (2014) Effective 3d action recognition using eigenjoints. J Vis Commun Image Represent 25(1):2–11

    Article  MathSciNet  Google Scholar 

  • Yang C, Han DK, Ko H (2017) Continuous hand gesture recognition based on trajectory shape information. Pattern Recognit Lett 99:39–47

    Article  Google Scholar 

  • Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the Kinect sensor. IEEE Trans Circuits Syst Video Technol 24(11):1935–1944

    Article  Google Scholar 

  • Zhang C, Tian Y (2015) Histogram of 3D facets: a depth descriptor for human action and hand gesture recognition. Comput Vis Image Underst 139:29–39

    Article  Google Scholar 

  • Zhou Y, Jiang G, Lin Y (2016) A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recognit 49:102–114

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 61773194, 61304084), by the Program for New Century Excellent Talents in Fujian Province University, by the Educational and Scientific Research Project for Middle-aged and Young Teachers of Fujian Province of China (Grant No. JAT170558), by the Science and Technology Project of Longyan City (Grant No. 2017LY69), by the Natural Science Foundation of Fujian Province of China (Grant No. 2019J01794) and by the Science and Technology Project of Longyan University (Grant No. LQ2015027).

Author information

Authors and Affiliations

  1. School of Mechanical and Electrical Engineering, Longyan University, Longyan, 364012, People’s Republic of China

    Fenglin Liu, Wei Zeng, Qinghui Wang & Ying Wang

  2. Department of Mechanical, Industrial and Systems Engineering, University of Rhode Island, Kingston, RI, 02881, USA

    Chengzhi Yuan

Authors
  1. Fenglin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengzhi Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qinghui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ying Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Zeng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, F., Zeng, W., Yuan, C. et al. Kinect-based hand gesture recognition using trajectory information, hand motion dynamics and neural networks. Artif Intell Rev 52, 563–583 (2019). https://doi.org/10.1007/s10462-019-09703-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-019-09703-w

Keywords

Search

Navigation