Journal on Multimodal User Interfaces

, Volume 8, Issue 4, pp 333–343 | Cite as

A novel set of features for continuous hand gesture recognition

  • M. K. Bhuyan
  • D. Ajay Kumar
  • Karl F. MacDorman
  • Yuji Iwahori
Original Paper

Abstract

Applications requiring the natural use of the human hand as a human–computer interface motivate research on continuous hand gesture recognition. Gesture recognition depends on gesture segmentation to locate the starting and end points of meaningful gestures while ignoring unintentional movements. Unfortunately, gesture segmentation remains a formidable challenge because of unconstrained spatiotemporal variations in gestures and the coarticulation and movement epenthesis of successive gestures. Furthermore, errors in hand image segmentation cause the estimated hand motion trajectory to deviate from the actual one. This research moves toward addressing these problems. Our approach entails using gesture spotting to distinguish meaningful gestures from unintentional movements. To avoid the effects of variations in a gesture’s motion chain code (MCC), we propose instead to use a novel set of features: the (a) orientation and (b) length of an ellipse least-squares fitted to motion-trajectory points and (c) the position of the hand. The features are designed to support classification using conditional random fields. To evaluate the performance of the system, 10 participants signed 10 gestures several times each, providing a total of 75 instances per gesture. To train the system, 50 instances of each gesture served as training data and 25 as testing data. For isolated gestures, the recognition rate using the MCC as a feature vector was only 69.6 % but rose to 96.0 % using the proposed features, a 26.1 % improvement. For continuous gestures, the recognition rate for the proposed features was 88.9 %. These results show the efficacy of the proposed method.

Keywords

Human–computer interaction (HCI) Gesture recognition  Motion chain code (MCC) Conditional random fields (CRF) 

References

  1. 1.
    Segouat J, Braffort A (2010) Toward modeling sign language coarticulation. Lecture Notes in Computer Science (Gesture in Embodied Communication and Human–Computer Interaction), vol 5934, pp 325–336Google Scholar
  2. 2.
    Yang R, Sarkar S (2006) Detecting coarticulation in sign language using conditional random fields. In: 18th international conference on pattern recognition (ICPR), vol 2, pp 108–112Google Scholar
  3. 3.
    Eisenstein J, Davis R (2005) Gestural Cues for Sentence Segmentation. Technical report, MIT AI Memo, pp 1–14Google Scholar
  4. 4.
    Lee HK, Kim JH (1999) An HMM-based threshold model approach for gesture recognition. IEEE Trans Pattern Anal Mach Intell 21(2):961–972Google Scholar
  5. 5.
    Bhuyan MK, Bora PK, Ghosh D (2011) An integrated approach to the recognition of a wide class of continuous hand gestures. Int J Pattern Recognit Artif Intell 25(2):227–252CrossRefGoogle Scholar
  6. 6.
    Yang H-D, Sclaroff S, Lee SW (2009) Sign language spotting with a threshold model based on conditional random fields. IEEE Trans Pattern Anal Mach Intell 31(7):1264–1277CrossRefGoogle Scholar
  7. 7.
    Alon J, Athitsos V, Quan Y, Sclaroff S (2009) A unified framework for gesture recognition and spatiotemporal gesture segmentation. IEEE Trans Pattern Anal Mach Intell 31(9):1685–1699CrossRefGoogle Scholar
  8. 8.
    Zaki MM, Shaheen IS (2011) Sign language recognition using a combination of new vision-based features. Pattern Recognit Lett 32(4):572–577CrossRefGoogle Scholar
  9. 9.
    Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of 18th international conference on machine learning, pp 282–289Google Scholar
  10. 10.
    Wallach HM (2004) Conditional random fields: an introduction. University of PennsylvaniaGoogle Scholar
  11. 11.
    Rabiner LR (1989) A tutorial on hidden markov models and selected applications in speech recognition. In: Proceedings of the IEEE, vol 77, no 2, pp 257–286Google Scholar
  12. 12.
    Chai D, Ngan KN (1999) Face segmentation using skin color map in videophone applications. IEEE Trans Circuits Syst. Video Technol. 9(4):551–564CrossRefGoogle Scholar
  13. 13.
    Fitzgibbon A, Pilu M, Fisher RB (1999) Direct least square fitting of ellipses. IEEE Trans. Pattern Anal. Mach. Intell. 21(5):476–480CrossRefGoogle Scholar
  14. 14.
    Quattoni A, Wang S, Morency LP, Collins M, Darrell T (2007) Hidden conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 29(10):1848–1852CrossRefGoogle Scholar

Copyright information

© OpenInterface Association 2014

Authors and Affiliations

  • M. K. Bhuyan
    • 1
  • D. Ajay Kumar
    • 2
  • Karl F. MacDorman
    • 1
  • Yuji Iwahori
    • 3
  1. 1.School of Informatics and Computing, Indiana University Purdue University (IUPUI)IndianapolisUSA
  2. 2.Department of Electronics and Electrical EngineeringIITGuwahatiIndia
  3. 3.Department of Computer ScienceChubu UniversityKasugaiJapan

Personalised recommendations