Skip to main content
SpringerLink
Log in

Gesture objects detection and tracking for virtual text entry keyboard interface

  • 1178: Pattern Recognition for Adaptive User Interfaces
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Detection and tracking of the gesturing object is a vital stage in the field of dynamic gesture recognition. It becomes more challenging in the practical environment due to the variation in illumination, occlusion, pose, rotation, speed, and presence of impostors. To overcome these complexities and provides ease to the user, we designed algorithms to (i) detect and track the gesture objects (red-color marker, bare-hand); (ii) recognize the gesture. To detect the red-color marker, a region of interest-based detection model is proposed by utilizing movement and color information. This model achieves ~13% improvement over the existing baseline models. For bare hand detection, a three-degree-information algorithm is proposed by incorporating region of interest (color, movement) and AlexNet. An improvement of 14% is achieved over the baseline models. To track the bare hand, a detection and tracking algorithm is proposed by utilizing AlexNet and point-tracker. This model achieves ~10% improvement over the baseline models. Evaluation of the proposed models for detection and tracking is performed on NITS hand gesture (I-IV, VII), Oxford hand, OUHands, EgoHands databases. To recognize gesture trajectories, the deep convolutional neural network is utilized. This model is able to achieve a relative improvement of ~7% over the baseline recognition models. To evaluate the performance of the recognition model, the NITS hand gesture (I-IV, VII), MNIST, and SVHN databases are used. In addition, we introduce the NITS-Net database consisting of bare-hand, non-bare-hand images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Abavisani M, Joze HRV, Patel VM (2019) Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

  2. Chan YT, Hu AGC, Plant JB (1979) A Kalman filter based tracking scheme with input estimation. IEEE Trans Aerosp Electron Syst 2:237–244

    Article  Google Scholar 

  3. Farahi F (2020) Probabilistic Kalman filter for moving object tracking. Sig Process Image Commun 82:115751

    Article  Google Scholar 

  4. Gao Q, Liu J, Ju Z, Zhang X (2019)Dual-hand detection for human-robot interaction by a parallel network based on hand detection and body pose estimation. IEEE Trans Ind Electron 66(12):9663–9672

    Article  Google Scholar 

  5. Han Y et al (2020) Parametric analysis of KLT algorithm in autonomous driving. 2020 20th International Conference on Control, Automation and Systems (ICCAS). IEEE

  6. Henriques JF et al (2014)High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

  7. Le T, Ngan H et al (2017) Robust hand detection and classification in vehicles and in the wild. CVPR Workshops

  8. Li C, Kitani KM (2013)Pixel-level hand detection in ego-centric videos. Proceedings of the IEEE conference on computer vision and pattern recognition

  9. Li SE et al (2018) Kalman filter-based tracking of moving objects using linear ultrasonic sensor array for road vehicles. Mech Syst Signal Process 98:173–189

    Article  Google Scholar 

  10. McBride TJ, Vandayar N, Nixon KJ (2019) A comparison of skin detection algorithms for hand gesture recognition. In: 2019 Southern African Universities Power Engineering Conference/Robotics and Mechatronics/Pattern Recognition Association of South Africa (SAUPEC/RobMech/PRASA). IEEE, pp 211-216

  11. Misra S, Laskar RH (2019) Integrated features and GMM Based Hand Detector Applied to Character Recognition System under Practical Conditions. Multimed Tools Appl 78(24):34927–34961

    Article  Google Scholar 

  12. Misra S, Rabul Hussain Laskar (2019) Development of a hierarchical dynamic keyboard character recognition system using trajectory features and scale-invariant holistic modeling of characters. J Ambient Intell Humaniz Comput 10(12):4901–4923

    Article  Google Scholar 

  13. Misra S, Laskar RH (2019) Comparative framework for vision-based gesturing modes and implementation of robust colour-marker detector for practical environments. IET Image Process 13(9):1460–1469

  14. Misra S, Laskar RH (2021) Comprehensive analysis of a diverse group of features and development of vision-based two-level hand detector under practical environment conditions. Int J Comput Vis Robot 11(2):175–200

    Article  Google Scholar 

  15. Misra S, Singha J, Rabul Hussain L (2018)Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135

    Article  Google Scholar 

  16. Misra S, Paluri KV, Laskar RH (2020) 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). IEEE

  17. Monsley KA et al (2021) Segregation of meaningful strokes, a pre-requisite for self co-articulation removal in isolated dynamic gestures. IET Image Proc 15(5):1166–1178

    Article  Google Scholar 

  18. Mukherjee S et al (2019) Fingertip detection and tracking for recognition of air-writing in videos. Expert Syst Appl 136:217–229

    Article  Google Scholar 

  19. Nayak T, Bhoi N (2020) Object detection and tracking using watershed segmentation and KLT tracker. Glob J Comput Sci Technol

  20. Ong E-J, Bowden R (2004) A boosted classifier tree for hand shape detection. Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings. IEEE

  21. Ozcan T (2019) Transfer learning-based convolutional neural networks with heuristic optimization for hand gesture recognition. Neural Comput Appl 31(12):8955–8970

    Article  Google Scholar 

  22. Roy A, Singha J, Manam L, Laskar RH (2017) Combination of adaptive vector median filter and weighted mean filter for removal of high-density impulse noise from color images. IET Image Proc 11(6):352–361

    Article  Google Scholar 

  23. Russakovsky O et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115(3):211–252

    Article  Google Scholar 

  24. Saboo S, Singha J (2021) Vision based two-level hand tracking system for dynamic hand gestures in indoor environment. Multimed Tools Appl 80(13):20579–20598

    Article  Google Scholar 

  25. Saboo S, Singha J, Laskar RH (2021) Dynamic hand gesture recognition using combination of two-level trackertrajectory-guided features. Multimed Syst: 1–12

  26. Schlemper J et al (2017) A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans Med Imaging 37(2):491–503

    Article  Google Scholar 

  27. Sigal L, Sclaroff S, Athitsos V (2004) Skin color-based video segmentation under time-varying illumination. IEEE Trans Pattern Anal Mach Intell 26(7):862–877

    Article  Google Scholar 

  28. Singha J, Laskar RH (2017) Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimedia Syst 23(4):499–514

    Article  Google Scholar 

  29. Singha J, Misra S, Laskar RH (2016) Effect of variation in gesticulation pattern in dynamic hand gesture recognition system. Neurocomputing 208:269-280

  30. Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141

    Article  Google Scholar 

  31. Skaria S et al (2019)Hand-gesture recognition using two-antenna Doppler radar with deep convolutional neural networks. IEEE Sens J 19(8):3041–3048

    Article  Google Scholar 

  32. Yadav K, Singh, Singha J (2020) Facial expression recognition using modified Viola-John’s algorithm and KNN classifier. Multimed Tools Appl 79(19):13089–13107

    Article  Google Scholar 

  33. Yanay T (2020)Air-writing recognition using smart-bands. Pervasive Mob Comput 66:101183

    Article  Google Scholar 

  34. Yongyong D, Xinhua H, Wu Z (2020) Image stabilization algorithm based on KLT motion tracking. International Conference on Computer Vision, Image and Deep Learning (CVIDL). IEEE

  35. Zhang M et al (2020) Using computer vision to automate hand detection and tracking of surgeon movements in videos of open surgery. AMIA Annual Symposium Proceedings, vol American Medical Informatics Association

Download references

Acknowledgements

This research work is funded by the SERB IMPRINT - IMP/2018/000098 project [sanction order SERB/F/10220/2018-19 ] We thank the department of electronics and communication engineering of NITS for providing us with the necessary facilities to carry out this work. The authors thank the anonymous reviewers whose valuable comments greatly improved the article.

Author information

Authors and Affiliations

  1. Electronics and Communication Engineering, NIT Silchar, Assam, 788010, India

    Kuldeep Singh Yadav,  Anish Monsley K. & Rabul Hussain Laskar

Authors
  1. Kuldeep Singh Yadav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anish Monsley K.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rabul Hussain Laskar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kuldeep Singh Yadav.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yadav, K.S., Anish Monsley K. & Laskar, R.H. Gesture objects detection and tracking for virtual text entry keyboard interface. Multimed Tools Appl 82, 5317–5342 (2023). https://doi.org/10.1007/s11042-021-11874-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11874-0

Keywords

Search

Navigation