Advertisement

Kinect-based hand gesture recognition using trajectory information, hand motion dynamics and neural networks

  • Fenglin Liu
  • Wei ZengEmail author
  • Chengzhi Yuan
  • Qinghui Wang
  • Ying Wang
Article

Abstract

Hand gestures are spatio-temporal patterns which can be characterized by collections of spatio-temporal features. Recognition of hand gestures is to find the re-occurrences of such spatio-temporal patterns through pattern matching. However, dynamic hand gestures have many obstacles for accurate recognition, including poor lighting conditions, camera’s inability to capture dynamic gesture in focus, occlusion due to finger movement, color variations due to lighting conditions. The Microsoft Kinect device provides an effective way to solve the above issues and also provides the skeleton for more convenient hand localization and tracking. The aim of this study is to develop a new trajectory-based method for hand gesture recognition using Kinect. In the first step, trajectory-based hand gesture features including spatial position and direction of fingertips, are derived from Kinect. The properties associated with the hand motion dynamics are preserved in these features. In the second step, radial basis function (RBF) neural networks are employed to model and approximate the hand motion dynamics derived from different hand gestures which represent Arabic numbers (0–9) and English alphabets (A–Z). The trained patterns of the approximated hand motion dynamics is stored in constant RBF networks. In the last step, a bank of dynamical estimators is constructed for all the training patterns, in which the constant RBF networks are embedded in. By comparing the set of estimators with a test gesture pattern, a set of recognition errors are generated, in which the average \(L_1\) norms of the errors are taken as the recognition measure based on the smallest error principle. Finally, experiments are carried out to assess the performance of the proposed method compared with other state-of-the-art approaches. By using the twofold and tenfold cross-validation styles, the correct recognition rates for Arabic numbers (0–9) and English alphabets (A–Z) are reported to be \(95.83\%, 97.25\%\), and \(91.35\%, 92.63\%\), respectively.

Keywords

Hand gesture recognition Kinect Hand motion dynamics RBF neural networks 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 61773194, 61304084), by the Program for New Century Excellent Talents in Fujian Province University, by the Educational and Scientific Research Project for Middle-aged and Young Teachers of Fujian Province of China (Grant No. JAT170558), by the Science and Technology Project of Longyan City (Grant No. 2017LY69), by the Natural Science Foundation of Fujian Province of China (Grant No. 2019J01794) and by the Science and Technology Project of Longyan University (Grant No. LQ2015027).

References

  1. Beh J, Han D, Ko H (2014) Rule-based trajectory segmentation for modeling hand motion trajectory. Pattern Recognit 47(4):1586–1601Google Scholar
  2. Cheng H, Dai Z, Liu Z, Zhao Y (2016) An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition. Pattern Recognit 55:137–147Google Scholar
  3. Farrell J (1988) Stability and approximator convergence in nonparametric nonlinear adaptive control. IEEE Trans Neural Netw 9(5):1008–1020Google Scholar
  4. Farzad A, Mashayekhi H, Hassanpour H (2017) A comparative performance analysis of different activation functions in LSTM networks for classification. Neural Comput Appl.  https://doi.org/10.1007/s00521-017-3210-6 Google Scholar
  5. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334Google Scholar
  6. Han F, Reily B, Hoff W, Zhang H (2017) Space-time representation of people based on 3D skeletal data: a review. Comput Vis Image Underst 158:85–105Google Scholar
  7. Herekar RR, Dhotre SR (2014) Handwritten character recognition based on zoning using Euler number for English alphabets and numerals. IOSR J Comput Eng 16(4):75–88Google Scholar
  8. Ibañez R, Soria Á, Teyseyre A, Rodréguez G, Campo M (2017) Approximate string matching: a lightweight approach to recognize gestures with Kinect. Pattern Recognit 62:73–86Google Scholar
  9. Jadooki S, Mohamad D, Saba T, Almazyad AS, Rehman A (2017) Fused features mining for depth-based hand gesture recognition to classify blind human communication. Neural Comput Appl 28(11):3285–3294Google Scholar
  10. Jain S, Chauhan R (2018) Recognition of handwritten digits using DNN, CNN, and RNN. In: International conference on advances in computing and data sciences, pp 239–248Google Scholar
  11. Kane L, Khanna P (2016) A framework to plot and recognize hand motion trajectories towards development of non-tactile interfaces. Proc Comput Sci 84:6–13Google Scholar
  12. Kiliboz NC, Gudukbay U (2015) A hand gesture recognition technique for human-computer interaction. J Vis Commun Image Represent 28:97–104Google Scholar
  13. Kim IC, Chien SI (2001) Analysis of 3d hand trajectory gestures using stroke-based composite hidden markov models. Appl Intell 15(2):131–143zbMATHGoogle Scholar
  14. Kundu S, Chhabra HS, Ara SS, Mishra RP (2017) Optical character recognition using 26-point feature extraction and ANN. Int J Adv Res Comput Sci Softw Eng 7(5):156–162Google Scholar
  15. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324Google Scholar
  16. Leite DQ, Duarte JC, Neves LP, de Oliveira JC, Giraldi GA (2016) Hand gesture recognition from depth and infrared Kinect data for CAVE applications interaction. Multimed Tools Appl 76(20):20423–20455Google Scholar
  17. Lu G, Zhou Y, Li X, Kudo M (2016) Efficient action recognition via local position offset of 3D skeletal body joints. Multimed Tools Appl 75(6):3479–3494Google Scholar
  18. Lu W, Tong Z, Chu J (2016) Dynamic hand gesture recognition with Leap Motion controller. IEEE Signal Process Lett 23(9):1188–1192Google Scholar
  19. Lun R, Zhao W (2015) A survey of applications and human motion recognition with microsoft kinect. Int J Pattern Recognit Artif Intell 29(5):1555008Google Scholar
  20. Maqueda AI, del-Blanco CR, Jaureguizar F, GarcaN N (2015) Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns. Comput Vis Image Underst 141:126–137Google Scholar
  21. Marin G, Dominio F, Zanuttigh P (2016) Hand gesture recognition with jointly calibrated Leap Motion and depth sensor. Multimed Tools Appl 75(22):14991–15015Google Scholar
  22. Misra S, Singha J, Laskar RH (2018) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135Google Scholar
  23. Oksuz O, Gudukbay U, Cetin AE (2008) A video-based text and equation editor for LaTeX. Eng Appl Artif Intell 21(6):952–960Google Scholar
  24. Patwardhan KS, Roy SD (2007) Hand gesture modelling and recognition involving changing shapes and trajectories, using a Predictive EigenTracker. Pattern Recognit Lett 28(3):329–334Google Scholar
  25. Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: a review. Comput Vis Image Understand 141:152–165Google Scholar
  26. Qiao J, Wang G, Li W, Chen M (2018) An adaptive deep Q-learning strategy for handwritten digit recognition. Neural Netw 107:61–71Google Scholar
  27. Raheja JL, Minhas M, Prashanth D, Shah T, Chaudhary A (2015) Robust gesture recognition using Kinect: a comparison between DTW and HMM. Optik-Int J Light Electron Opt 126(11):1098–1104Google Scholar
  28. Raheja JL, Chandra M, Chaudhary A (2017) 3D gesture based real-time object selection and recognition. Pattern Recognit Lett.  https://doi.org/10.1016/j.patrec.2017.09.034 Google Scholar
  29. Rautaray SS, Agrawal A (2017) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54Google Scholar
  30. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120Google Scholar
  31. Roh MC, Fazli S, Lee SW (2016) Selective temporal filtering and its application to hand gesture recognition. Appl Intell 45(2):255–264Google Scholar
  32. Sahoo MK, Nayak J, Mohapatra S, Nayak BK, Behera HS (2015) Character recognition using firefly based back propagation neural network. In: Computational intelligence in data mining, vol 2. Springer, New DelhiGoogle Scholar
  33. Singha J, Misra S, Laskar RH (2016) Effect of variation in gesticulation pattern in dynamic hand gesture recognition system. Neurocomputing 208:269–280Google Scholar
  34. Song Y, Demirdjian D, Davis R (2012) Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans Interact Intell Syst 2(1):5Google Scholar
  35. Stern H, Shmueli M, Berman S (2013) Most discriminating segment Longest common subsequence (MDSLCS) algorithm for dynamic hand gesture classification. Pattern Recognit Lett 34(15):1980–1989Google Scholar
  36. Suk HI, Sin BK, Lee SW (2010) Hand gesture recognition based on dynamic Bayesian network framework. Pattern Recognit 43(9):3059–3072zbMATHGoogle Scholar
  37. Wang C, Hill DJ (2006) Learning from neural control. IEEE Trans Neural Netw 17(1):130–146Google Scholar
  38. Wang C, Hill DJ (2007) Deterministic learning and rapid dynamical pattern recognition. IEEE Trans Neural Netw 18(3):617–630Google Scholar
  39. Wang C, Hill DJ (2009) Deterministic learning theory for identification, recognition and control. CRC Press, Boca RatonGoogle Scholar
  40. Wang C, Chen T, Chen G, Hill DJ (2009) Deterministic learning of nonlinear dynamical systems. Int J Bifurc Chaos 19(4):1307–1328MathSciNetzbMATHGoogle Scholar
  41. Wang C, Liu Z, Chan SC (2015) Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed 17(1):29–39Google Scholar
  42. Xu S, Xue Y (2017) A long term memory recognition framework on multi-complexity motion gestures. In: IEEE international conference on document analysis and recognition, pp 201–205Google Scholar
  43. Yang X, Tian YL (2014) Effective 3d action recognition using eigenjoints. J Vis Commun Image Represent 25(1):2–11MathSciNetGoogle Scholar
  44. Yang C, Han DK, Ko H (2017) Continuous hand gesture recognition based on trajectory shape information. Pattern Recognit Lett 99:39–47Google Scholar
  45. Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the Kinect sensor. IEEE Trans Circuits Syst Video Technol 24(11):1935–1944Google Scholar
  46. Zhang C, Tian Y (2015) Histogram of 3D facets: a depth descriptor for human action and hand gesture recognition. Comput Vis Image Underst 139:29–39Google Scholar
  47. Zhou Y, Jiang G, Lin Y (2016) A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recognit 49:102–114Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.School of Mechanical and Electrical EngineeringLongyan UniversityLongyanPeople’s Republic of China
  2. 2.Department of Mechanical, Industrial and Systems EngineeringUniversity of Rhode IslandKingstonUSA

Personalised recommendations