A Visual System for Hand Gesture Recognition in Human-Computer Interaction

  • Matti-Antero Okkonen
  • Vili Kellokumpu
  • Matti Pietikäinen
  • Janne Heikkilä
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4522)


Visual hand gestures offer an interesting modality for Human-Computer-Interaction (HCI) applications. Gesture recognition and hand tracking, however, are not trivial tasks and real environments set a lot of challenges to algorithms performing such activities. In this paper, a novel combination of techniques is presented for tracking and recognition of hand gestures in real, cluttered environments. In addition to combining existing techniques, a method for locating a hand and segmenting it from an arm in binary silhouettes and a foreground model for color segmentation is proposed. A single hand is tracked with a single camera and the trajectory information is extracted along with recognition of five different gestures. This information is exploited for replacing the operations of a normal computer mouse. The silhouette of the hand is extracted as a combination of different segmentation methods: An adaptive colour model based segmentation is combined with intensity and chromaticity based background subtraction techniques to achieve robust performance in cluttered scenes. An affine-invariant Fourier-descriptor is derived from the silhouette, which is then classified to a hand shape class with support vector machines (SVM). Gestures are recognized as changes in the hand shape with a finite state machine (FSM).


Support Vector Machine Finite State Machine Gesture Recognition Foreground Object Hand Shape 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Kurata, T., Okuma, T., Kourogi, M., Sakaue, K.: The hand mouse: Gmm hand-color classification and mean shift tracking. In: Proc. of the IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems (RATFG-RTS’01), Vancouver, Canada, July 2001, pp. 119–124 (2001)Google Scholar
  2. 2.
    Wagner, S., Alefs, B., Picus, C.: Framework for a portable gesture interface. In: FGR ’06: 7th International Conference on Automatic Face and Gesture Recognition, pp. 275–280 (2006)Google Scholar
  3. 3.
    Zhai, H., Wu, X., Han, H.: Research of a real-time hand tracking algorithm. In: Proc. of the International Conference on Neural Networks and Brain (ICNN&B’05), Beijing, China, October 2005, pp. 1233–1235 (2005)Google Scholar
  4. 4.
    Pantrigo, J., Montemayor, A., Sanchez, A.: Local search particle filter applied to human-computer interaction. In: Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis, ISPA 2005, pp. 279–284 (2005)Google Scholar
  5. 5.
    Quek, F., Mysliwiec, T., Zhao, M.: Fingermouse: A freehand computer pointing interface. In: Proc. of Int’l Conf. on Automatic Face and Gesture Recognition, pp. 372–377 (1995),
  6. 6.
    Horimoto, S., Arita, D., Taniguchi, R.-i.: Real-time hand shape recognition for human interface. In: Proc. of the 12th International Conference on Image Analysis and Processing (ICIAP ’03), Mantova, Italy, September 2003, pp. 20–25 (2003)Google Scholar
  7. 7.
    Liu, Y., Jia, Y.: A robust hand tracking for gesture-based interaction of wearable computers. In: Eighth International Symposium on Wearable Computers, ISWC 2004, pp. 22–29 (2004)Google Scholar
  8. 8.
    Malik, S., Laszlo, J.: Visual touchpad: a two-handed gestural input device. In: Proc. of the 6th international conference on Multimodal interfaces (ICMI’04), State College, PA, USA, October 2004, pp. 289–296 (2004)Google Scholar
  9. 9.
    von Hardenberg, C., Bérard, F.: Bare-hand human-computer interaction. In: Proceedings of the 2001 workshop on Perceptive user interfaces (PUI’01), Orlando, FL, USA, November 2001, pp. 1–8 (2001)Google Scholar
  10. 10.
    Pavlovic, V.I., Sharma, R., Huang, T.S.: Visual interpretation of hand gestures for human-computer interaction: A review. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 677–695 (1997)CrossRefGoogle Scholar
  11. 11.
    Ong, S.C.W., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27(6), 873–891 (2005)CrossRefGoogle Scholar
  12. 12.
    Swain, M.J., Ballard, D.H.: Color indexing. Int. J. Comput. Vision 7(1), 11–32 (1991)CrossRefGoogle Scholar
  13. 13.
    Soriano, M., Martinkauppi, B., Huovinen, S., Laaksonen, M.: Adaptive skin color modeling using the skin locus for selecting training pixels. Pattern Recognition 36(3), 681–690 (2003)CrossRefGoogle Scholar
  14. 14.
    Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2(2), 121–167 (1998), CrossRefGoogle Scholar
  15. 15.
    Arbter, K., Snyder, W.E., Burhardt, H., Hirzinger, G.: Application of affine-invariant fourier descriptors to recognition of 3-d objects. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 640–647 (1990), doi:10.1109/34.56206CrossRefGoogle Scholar
  16. 16.
    Kellokumpu, V., Pietikäinen, M., Heikkilä, J.: Human activity recognition using sequences of postures. In: Proceedings of the IAPR Conference on Machine Vision Applications (MVA 2005), pp. 570–573 (2005)Google Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Matti-Antero Okkonen
    • 1
  • Vili Kellokumpu
    • 1
  • Matti Pietikäinen
    • 1
  • Janne Heikkilä
    • 1
  1. 1.Department of Electrical and Information Engineering, P.O. Box 4500, FI-90014, University of OuluFinland

Personalised recommendations