Skip to main content

Advertisement

SpringerLink
Log in

Non-trajectory-based gesture recognition in human-computer interaction based on hand skeleton data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Currently, no efficient, accurate and flexible gesture recognition algorithm has been developed to recognize non-trajectory-based gesture recognition. Therefore, we aim to construct a gesture recognition algorithm to not only complete gesture recognition accurately and quickly but also adapt to individual differences. In this paper, we present a novel non-trajectory-based gesture recognition method (NT-GRM) based on hand skeleton information and a hidden Markov model (HMM). To recognize a static gesture, the direction information of each bone section of the hands was taken as the observation data to construct the HMM. In addition, multiple static gestures were detected in turn to identify a dynamic gesture. As determined by experimental verification, the NT-GRM can complete recognition in a system containing ten interactive gestures with a recognition accuracy of over 95% and a recognition speed of 21.73 ms. The training time required for each static gesture model is 2.56 s. And the NT-GRM can identify static and dynamic gestures accurately and quickly with small training samples in different functional modes. In conclusion, the NT-GRM can be applied to the development of gesture interaction systems to help developers realize practical functions such as gesture library construction, user gesture customization, and user gesture adaptation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Al-Hammadi M, Muhammad G, Abdul W, et al. (2020) Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access 8:192527–192542. https://doi.org/10.1109/ACCESS.2020.3032140

    Article  Google Scholar 

  2. Ameur S, Khalifa AB, Bouhlel MS (2016) A comprehensive leap motion database for hand gesture recognition. In: 2016 7th international conference on sciences of electronics, technologies of information and telecommunications (SETIT), pp 514–519. https://doi.org/10.1109/SETIT.2016.7939924

  3. Arjun AM, Sreehari S, Nandakumar R (2020) The interplay of hand gestures and facial expressions in conveying emotions A CNN–BASED APPROACH. In: 2020 Fourth international conference on computing methodologies and communication (ICCMC), pp 833–837. https://doi.org/10.1109/ICCMC48092.2020.ICCMC-000154

  4. Bilmes JA (1998) A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models. International Computer Science Institute 4(510):126

    Google Scholar 

  5. Chen Q, Georganas ND, Petriu EM (2008) Hand gesture recognition using Haar-like features and a stochastic context-free grammar. IEEE Trans Instrum Meas 57(8):1562–1571. https://doi.org/10.1109/TIM.2008.922070

    Article  Google Scholar 

  6. Chen L, Fu J, Wu Y, et al. (2020) Hand gesture recognition using compact CNN via surface electromyography signals. Sensors 20(3):672. https://doi.org/10.3390/s20030672

    Article  Google Scholar 

  7. Chuan CH, Regina E, Guardino C (2014) American sign language recognition using leap motion sensor. In: 13th International conference on machine learning and applications, pp 541–544. https://doi.org/10.1109/ICMLA.2014.110

  8. Chung WK, Wu X, Xu Y (2009) A realtime hand gesture recognition based on Haar wavelet representation. In: 2008 IEEE international conference on robotics and biomimetics, pp 336–341. https://doi.org/10.1109/ROBIO.2009.4913026

  9. Dardas NH, Georganas ND (2011) Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Trans Instrum Meas 60(11):3592–3607. https://doi.org/10.1109/TIM.2011.2161140

    Article  Google Scholar 

  10. Ding J, Chang CW (2016) An adaptive hidden Markov model-based gesture recognition approach using Kinect to simplify large-scale video data processing for humanoid robot imitation. Multimed Tools Appl 75(23):15537–15551

    Article  Google Scholar 

  11. Elmezain M, Al-Hamadi A, Krell G, El-Etriby S, Michaelis B (2007) Gesture recognition for alphabets from hand motion trajectory using hidden markov models. In: 2007 IEEE international symposium on signal processing and information technology, pp 1192–1197. https://doi.org/10.1109/ISSPIT.2007.4458209

  12. Fok KY, Ganganath N, Cheng CT, Chi KT (2015) A real-time asl recognition system using leap motion sensors. In: 2015 International conference on cyber-enabled distributed computing and knowledge discovery, pp 411–414. https://doi.org/10.1109/CyberC.2015.81

  13. Gieser SN, Boisselle A, Makedon F (2015) Real-time static gesture recognition for upper extremity rehabilitation using the leap motion. In: International conference on digital human modeling and applications in health, safety, ergonomics and risk management. Springer, Cham, pp 144–154. https://doi.org/10.1007/978-3-319-21070-4_15

  14. Hakonen M, Piitulainen H, Visala A (2015) Current state of digital signal processing in myoelectric interfaces and related applications. Biomedical Signal Processing and Control 18:334–359. https://doi.org/10.1016/j.bspc.2015.02.009

    Article  Google Scholar 

  15. Jirak D, Tietz S, Ali H et al (2020) Echo state networks and long short-term memory for continuous gesture recognition: a comparative study. Cognitive Computation:1-13. https://doi.org/10.1007/s12559-020-09754-0

  16. Joshi A, Monnier C, Betke M, Sclaroff S (2017) Comparing random forest approaches to segmenting and classifying gestures. Image Vis Comput 58:86–95. https://doi.org/10.1016/j.imavis.2016.06.001

    Article  Google Scholar 

  17. Khan FR, Ong HF, Bahar N (2016) A sign language to text converter using leap motion. International Journal on Advanced Science, Engineering and Information Technology 6(6):1089–1095

    Article  Google Scholar 

  18. Kohlschein C (2013) An introduction to hidden markov models. Current protocols in bioinformatics, Appendix 3(Appendix 3), Appendix 3A

  19. Lee A, Cho Y, Jin S, et al. (2020) Enhancement of surgical hand gesture recognition using a capsule network for a contactless interface in the operating room. Comput Methods Prog Biomed 190:105385. https://doi.org/10.1016/j.cmpb.2020.105385

    Article  Google Scholar 

  20. Li D, Li JX, Zhang Y, Zeng Z (2014) Gesture recognition of data glove based on PSO-improved BP neural network. Electr Mach Control 18(8):87–93

    Google Scholar 

  21. Liao TL, Chen HC, Yan JJ (2020) Design of real-time face position tracking and gesture recognition system based on image segmentation algorithm. Journal of Network Intelligence 5(4):226–239. https://doi.org/10.1007/978-3-030-69178-3_2

    Article  Google Scholar 

  22. Lien J, Gillian N, Karagozler ME, et al. (2016) Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35 (4):1–19. https://doi.org/10.1145/2897824.2925953

    Article  Google Scholar 

  23. Liu X, Shi H, Hong X et al (2020) 3D skeletal gesture recognition via hidden states exploration. IEEE Trans Image Process 29:4583–4597. https://doi.org/10.1109/TIP.2020.2974061

    Article  Google Scholar 

  24. Louzir A, Haskou A, Pesin A (2021) A simple RF-based solution for gesture recognition. In: 2021 IEEE international conference on consumer electronics (ICCE), pp 1–5. https://doi.org/10.1109/ISCAS51556.2021.9401065

  25. Lu L, Zhang J, Zhu Y, Liu H (2015) A static gesture recognition method based on data glove. Journal of Computer-Aided Design & Computer Graphics 27(12):2411–2416

    Google Scholar 

  26. Mantecón T, del Blanco CR, Jaureguizar F, Garcíav N (2019) A real-time gesture recognition system using near-infrared imagery. PLOS ONE 14 (10):1–17. https://doi.org/10.1371/journal.pone.0223320. Article e0223320

    Article  Google Scholar 

  27. Mapari RB, Kharat G (2015) Real time human pose recognition using leap motion sensor. In: 2015 IEEE international conference on research in computational intelligence and communication networks (ICRCICN), pp 323–328. https://doi.org/10.1109/ICRCICN.2015.7434258

  28. Marin G, Dominio F, Zanuttigh P (2014) Hand gesture recognition with leap motion and kinect devices. In: 2014 IEEE international conference on image processing (ICIP), pp 1565–1569. https://doi.org/10.1109/ICIP.2014.7025313

  29. Marin G, Dominio F, Zanuttigh P (2016) Hand gesture recognition with jointly calibrated leap motion and depth sensor. Multimed Tools Appl 75 (22):14991–15015. https://doi.org/10.1007/s11042-015-2451-6

    Article  Google Scholar 

  30. McCartney R, Yuan J, Bischof HP (2015) Gesture recognition with the leap motion controller. In: The international conference on image processing, computer vision, and pattern recognition. https://scholarworks.rit.edu/other/857

  31. Mitra S, Acharya T (2007) Gesture recognition: a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37(3):311–324. https://doi.org/10.1109/TSMCC.2007.893280

    Article  Google Scholar 

  32. Nielsen M, Störring M, Moeslund TB, Granum E (2003) A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In: International gesture workshop, pp 409–420 https://doi.org/10.1007/978-3-540-24598-8_38

  33. Parcheta Z, Martínez-Hinarejos CD (2017) Sign language gesture recognition using HMM. In: Iberian conference on pattern recognition and image analysis, pp 419–426. https://doi.org/10.1007/978-3-319-58838-4_46

  34. Qin S, Zhu X, Yang Y, Jiang Y (2014) Real-time hand gesture recognition from depth images using convex shape decomposition method. J Signal Process Syst 74(1):47–58. https://doi.org/10.1007/s11265-013-0778-7

    Article  Google Scholar 

  35. Rabiner L, Juang B (1986) An introduction to hidden Markov models. IEEE ASSP Mag 3(1):4–16. https://doi.org/10.1109/MASSP.1986.1165342

    Article  Google Scholar 

  36. Rahim MA, Shin J, Islam MR (2020) Hand gesture recognition-based non-touch character writing system on a virtual keyboard. Multimed Tools Appl:1-24. https://doi.org/10.1007/s11042-019-08448-6

  37. Sagayam KM, Hemanth DJ (2017) Hand posture and gesture recognition techniques for virtual reality applications: a survey. Virtual Reality 21 (2):91–107. https://doi.org/10.1007/s10055-016-0301-0

    Article  Google Scholar 

  38. Saha S, Lahiri R, Konar A, Banerjee B, Nagar AK (2017) HMM-based gesture recognition system using Kinect sensor for improvised human-computer interaction. In: 2017 International joint conference on neural networks (IJCNN), pp 2776–2783. https://doi.org/10.1109/IJCNN.2017.7966198

  39. Schmidt T, Araujo FP, Pappa GL et al (2014) Real-time hand gesture recognition based on sparse positional data. Brazalian Workshop on Computer Vision

  40. Shanthakumar VA, Peng C, Hansberger J et al (2020) Design and evaluation of a hand gesture recognition approach for real-time interactions. Multimed Tools Appl:1–24. https://doi.org/10.1007/s11042-019-08520-1

  41. Singh JB, Jaison B (2021) Gyro-stabilized camera control in drones for military applications. IOP Conference Series: Materials Science and Engineering 1012(1):012017

    Article  Google Scholar 

  42. Tornay S, Aran O, Doss MM (2020) An HMM approach with inherent model selection for sign language and gesture recognition. In: Proceedings of the 12th language resources and evaluation conference, pp 6049–6056. https://aclanthology.org/2020.lrec-1.741

  43. Vamsikrishna KM, Dogra DP, Desarkar MS (2015) Computer-vision-assisted palm rehabilitation with supervised learning. IEEE Trans Biomed Eng 63(5):991–1001. https://doi.org/10.1109/TBME.2015.2480881

    Article  Google Scholar 

  44. Werner C, Kardaris N, Koutras P, Zlatintsi A, Maragos P, Bauer JM, Hauer K (2020) Improving gesture-based interaction between an assistive bathing robot and older adults via user training on the gestural commands. Arch Gerontol Geriatr 87:103996. https://doi.org/10.1016/j.archger.2019.103996

    Article  Google Scholar 

  45. Wu XY (2019) A hand gesture recognition algorithm based on DC-CNN. Multimed Tools Appl:1-13. https://doi.org/10.1007/s11042-019-7193-4

  46. Wu CF, Xie J, Yu L, Zhou KD (2016) Static gesture recognition arithmetic based on CHMM. Computer Systems & Applications (8):115–119. https://doi.org/10.15888/j.cnki.csa.005266 (in Chinese)

  47. Wu Y, Wu Z, Fu C (2018) Continuous arm gesture recognition based on natural features and logistic regression. IEEE Sensors J 18(19):8143–8153. https://doi.org/10.1109/JSEN.2018.2863044

    Article  Google Scholar 

  48. Wu H, Wang Y, Liu J, Qiu J, Zhang XL (2020) User-defined gesture interaction for in-vehicle information systems. Multimed Tools Appl 79 (1):263–288. https://doi.org/10.1007/s11042-019-08075-1

    Article  Google Scholar 

  49. Wysoski SG (2003) A rotation invariant static hand gesture recognition system using boundary information and neural networks. ME thesis, Nagoya Institute of Technology, Japan

  50. Zhang XH, Wang JJ, Wang X, Ma XL (2016) Improvement of dynamic hand gesture recognition based on HMM algorithm. In: 2016 International conference on information system and artificial intelligence (ISAI), pp 401–406. https://doi.org/10.1109/ISAI.2016.0091

  51. Zhao Z, Luo H, Song GH, Chen Z, Lu ZM, Wu X (2018) Web-based interactive drone control using hand gesture. Rev Sci Instrum 89(1):014707. https://doi.org/10.1063/1.5004004

    Article  Google Scholar 

Download references

Funding

This work is supported in part by the National Natural Science Foundation of China (No. 71901061, No. 71871056), Science and Technology on Avionics Integration Laboratory and Aeronautical Science Fund (No. 20185569008), Fundamental Research Funds for the Central Universities (No.2242019k1G016).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Southeast University, Nanjing, 211189, China

    Lesong Jia, Xiaozhou Zhou & Chengqi Xue

Authors
  1. Lesong Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaozhou Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengqi Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Lesong Jia, Xiaozhou Zhou, and Chengqi Xue. The first draft of the manuscript was written by Lesong Jia and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

More about the author’s contribution is as follows:

Conceptualization: Xiaozhou Zhou, Chengqi Xue, Lesong Jia; Methodology: Lesong Jia, Xiaozhou Zhou; Formal analysis and investigation: Lesong Jia; Writing - original draft preparation: Lesong Jia; Writing - review and editing: Xiaozhou Zhou, Chengqi Xue; Funding acquisition:Xiaozhou Zhou, Chengqi Xue; Resources: Xiaozhou Zhou, Chengqi Xue, Lesong Jia; Supervision: Chengqi Xue.

Corresponding author

Correspondence to Xiaozhou Zhou.

Ethics declarations

This study was approved by the IEC for Clinical Research of Zhongda Hospital, Affiliated to Southeast University. All subjects understood the contents of the experiment and signed an informed consent form before the experiment.

Conflict of Interests

No conflict of interest exits in the submission of this manuscript.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Availability of Data and Material

The supplementary data to this article can be found online at https://github.com/LELEJIA/The-Results-of-the-gesture-recognition-verification-experiments.

Appendix

Appendix

Fig. 17
figure 17

Diagram of gesture action

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jia, L., Zhou, X. & Xue, C. Non-trajectory-based gesture recognition in human-computer interaction based on hand skeleton data. Multimed Tools Appl 81, 20509–20539 (2022). https://doi.org/10.1007/s11042-022-12355-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12355-8

Keywords

Search

Navigation