Development of a hierarchical dynamic keyboard character recognition system using trajectory features and scale-invariant holistic modeling of characters

  • Songhita MisraEmail author
  • R. H. Laskar
Original Research


A robust, intuitive, effortless, and novel dynamic hand gesture based virtual keyboard system is developed in this study. Firstly, a new hierarchical approach is applied, which, based on self-coarticulation and position features, effectively sub-groups a large gesture vocabulary. Additionally, new trajectory features are proposed which shall extract the local structural statistics of the gestures. All state-of-art models are based on temporal trajectory features which are based on the frame-wise 2D sequential path it followed. Due to this, the trajectory features are path dependent and vulnerable to trajectory noises or any other variations in pattern, speed, or scale. In contrast to this, an image-based approach of gesture recognition has been proposed in this study, which is independent of the sequential gesturing path of the gesture. Since the image-models (a holistic view) are not obtained frame-wise, unlike existing image-models, they are pattern, speed, and scale invariant in nature and also immune to trajectory distortions. To this end, image-based features and significant trajectory features are fused to develop a hybrid hierarchical classification model which exhibits an exceptional increase in accuracy by 3.9% as compared to baseline non-hierarchical trajectory based model using an Artificial neural network (ANN). Classification models such as Voronoi diagram based classifier (VDBC) and neuro-fuzzy (NF) classifier have also been explored and displayed motivating performance. Reduction in misclassification has been observed for gestures such as ‘(and)’, ‘{and}’, ‘0 and O’, ‘Z and 2’. The present system can also identify any static/dynamic imposters present in the gesture environment.


Holistic model Dynamic gestures Keyboard characters Hierarchical model Trajectory features 



Authors are thankful to the Ministry of Electronics and Information Technology, Government of India for the financial support under the Visvesvaraya Ph.D. scheme. Authors also acknowledge Speech and Image Processing Laboratory, ECE Dept., National Institute of Technology Silchar, India, for providing all necessary facilities to carry out the research work.


The study was funded by Ministry of Electronics and Information Technology, Government of India with Grant no. PhD-MLA/4(74)/2015-16.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Baig F, Beg S, Khan MF, Nawaz SJ (2015) A method to control home appliances based on writing commands over the air. J Control Autom Elect Syst 26(4):421–429. CrossRefGoogle Scholar
  2. Bhuyan MK, Ghosh D, Bora PK (2006) Hand motion tracking and trajectory matching for dynamic hand gesture recognition. J Exp Theor Artif Intell 18(4):435–447. CrossRefGoogle Scholar
  3. Bhuyan MK, Kumar DA, MacDorman KF, Iwahori Y (2014) A novel set of features for continuous hand gesture recognition. J Multimodal User Interfaces 8(4):333–343. CrossRefGoogle Scholar
  4. Bobick AF, Davis JW (2001) The recognition of human movement using temporal templates. IEEE Trans Pattern Anal Mach Intell 23(3):257–267CrossRefGoogle Scholar
  5. Bretzner L, Laptev I, Lindeberg T (2002) Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In: Automatic face and gesture recognition, 2002. Proceedings. Fifth IEEE international conference on, IEEE. pp 423–428.
  6. Carlsson S, Sullivan J (2001) Action recognition by shape matching to key frames. In: Workshop on models versus exemplars in computer vision, vol 1, p 18Google Scholar
  7. Carreira M, Ting KLH, Csobanka P, Gonçalves D (2017) Evaluation of in-air hand gestures interaction for older people. Univ Access Inf Soc 16(3):561–580. CrossRefGoogle Scholar
  8. Cetisli B (2010) Development of an adaptive neuro-fuzzy classifier using linguistic hedges: part 1. Expert Syst Appl 37(8):6093–6101. CrossRefGoogle Scholar
  9. Chang CL (1974) Finding prototypes for nearest neighbor classifiers. IEEE Trans Comput 100(11):1179–1184. CrossRefzbMATHGoogle Scholar
  10. Choi H, Park H (2014) A hierarchical structure for gesture recognition using RGB-D sensor. In: Proceedings of the second international conference on Human-agent interaction, ACM, pp 265–268.
  11. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Computer vision and pattern recognition, IEEE computer society conference, vol 1, pp 886–893.
  12. Deepthi DR, Eswaran K (2010) A new hierarchical pattern recognition method using mirroring neural networks. Int J Comput Appl 1(12):70–78Google Scholar
  13. Douglas DH, Peucker TK (2011) Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. In: Classics in cartography: reflections on influential articles from cartographica. Wiley Online Library, Hoboken, pp 15–28.
  14. Elmezain M, Al-Hamadi A, Krell G, El-Etriby S, Michaelis B (2007) Gesture recognition for alphabets from hand motion trajectory using hidden markov models. In: 2007 IEEE international symposium on signal processing and information technology, pp 1192–1197.
  15. Fang G, Gao W, Zhao D (2003) Large vocabulary sign language recognition based on hierarchical decision trees. In: Proceedings of the 5th international conference on multimodal interfaces, ACM. pp 125–131.
  16. Kobayashi T, Otsu N (2008) Image feature extraction using gradient local auto-correlations. In: European conference on computer vision. Springer, Berlin, Heidelberg, pp 346–358.
  17. Kundu AS, Mazumder O, Lenka PK, Bhaumik S (2017) Hand gesture recognition based omnidirectional wheelchair control using IMU and EMG sensors. J Intell Robot Syst. Google Scholar
  18. Li C, Xie C, Zhang B, Chen C, Han J (2018) Deep Fisher discriminant learning for mobile hand gesture recognition. Pattern Recognit 77:276–288. CrossRefGoogle Scholar
  19. Liu F, Zeng W, Yuan C, Wang Q, Wang Y, Lu B (2018) Trajectory-based hand gesture recognition using kinect via deterministic learning. In: Chinese control conference.
  20. Misra S, Laskar RH (2018) Approach toward extraction of sparse texture features and their application in robust two-level bare-hand detection. J Electron Imaging 27(5):051209. CrossRefGoogle Scholar
  21. Misra S, Singha J, Laskar RH (2018) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135. CrossRefGoogle Scholar
  22. Moschetti A, Fiorini L, Esposito D, Dario P, Cavallo F (2017) Toward an unsupervised approach for daily gesture recognition in assisted living applications. IEEE Sens J 17(24):8395–8403. CrossRefGoogle Scholar
  23. Ogale A, Karapurkar A, Guerra-Filho G, Aloimonos Y (2004) View-invariant identification of pose sequences for action recognition. In: Video analysis and content extraction workshop (VACE), p 14Google Scholar
  24. Pang Y, Yuan Y, Li X, Pan J (2011) Efficient R-HOG human detection. Sig Process 91(4):773–781. CrossRefzbMATHGoogle Scholar
  25. Quesada L, López G, Guerrero L (2017) Automatic recognition of the American sign language fingerspelling alphabet to assist people living with speech or hearing impairments. J Ambient Intell Humaniz Comput 8(4):625–635. CrossRefGoogle Scholar
  26. Rautaray SS, Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54. CrossRefGoogle Scholar
  27. Rittscher J, Blake A (1999) Classification of human body motion. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference, vol 1, pp 634–639.
  28. Sahbi H, Geman D (2006) A hierarchy of support vector machines for pattern detection. J Mach Learn Res 7:2087–2123MathSciNetzbMATHGoogle Scholar
  29. Shen X, Hua G, Williams L, Wu Y (2012) Dynamic hand gesture recognition: an exemplar-based approach from motion divergence fields. Image Vis Comput 30(3):227–235. CrossRefGoogle Scholar
  30. Silva EJ, Zanchettin CA (2016) Voronoi diagram based classifier for multiclass imbalanced data sets. In: Intelligent systems (BRACIS), 5th Brazilian conference, pp 109–114.
  31. Singha J, Laskar RH (2016a) Self co-articulation detection and trajectory guided recognition for dynamic hand gestures. IET Comput Vision 10(2):143–152. CrossRefGoogle Scholar
  32. Singha J, Laskar RH (2016b) Recognition of global hand gestures using self co-articulation information and classifier fusion. J Multimodal User Interfaces 10(1):77–93. CrossRefGoogle Scholar
  33. Singha J, Laskar RH (2017) Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimedia Syst 23(4):499–514. CrossRefGoogle Scholar
  34. Singha J, Misra S, Laskar RH (2016) Effect of variation in gesticulation pattern in dynamic hand gesture recognition system. Neurocomputing 208:269–280. CrossRefGoogle Scholar
  35. Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141. CrossRefGoogle Scholar
  36. Sun CT, Jang JS (1993) A neuro-fuzzy classifier and its applications. In: Fuzzy systems, second IEEE international conference, pp 94–98.
  37. Thabet E, Khalid F, Sulaiman PS, Yaakob R (2018) Fast marching method and modified features fusion in enhanced dynamic hand gesture segmentation and detection method under complicated background. J Ambient Intell Humaniz Comput 9(3):755–769. CrossRefGoogle Scholar
  38. Thiam P, Kessler V, Schwenker F (2017) Hierarchical combination of video features for personalised pain level recognition. In: 25th European symposium on artificial neural networks, computational intelligence and machine learning, pp 465–470Google Scholar
  39. Yamato J, Ohya J, Ishii K (1992) Recognizing human action in time-sequential images using hidden markov model. In: Computer vision and pattern recognition, IEEE computer society conference, pp 379–385.
  40. Yilmaz A, Shah M (2005) Actions sketch: a novel action representation. In: Computer vision and pattern recognition, IEEE computer society conference on, vol 1, pp 984–989.
  41. Yoon HS, Soh J, Bae YJ, Yang HS (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognit 34(7):1491–1501. CrossRefzbMATHGoogle Scholar
  42. Yun SS, Nguyen Q, Choi J (2017) Recognition of emergency situations using audio–visual perception sensor network for ambient assistive living. J Ambient Intell Humaniz Comput. Google Scholar
  43. Zeng W, Wang C, Wang Q (2018) Hand gesture recognition using leap motion via deterministic learning. Multimedia Tools Appl. Google Scholar
  44. Zhang Z, Tian Z, Zhou M (2018) HandSense: smart multimodal hand gesture recognition based on deep neural networks. J Ambient Intell Humaniz Comput. Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of ECENIT SilcharSilcharIndia

Personalised recommendations