Real-Time Recognition of 3D-Pointing Gestures for Human-Machine-Interaction

  • Kai Nickel
  • Rainer Stiefelhagen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2781)


We present a system capable of visually detecting pointing gestures and estimating the 3D pointing direction in real-time. We use Hidden Markov Models (HMMs) trained on different phases of sample pointing gestures to detect the occurrence of a gesture. For estimating the pointing direction, we compare two approaches: 1) The line of sight between head and hand and 2) the forearm orientation. Input features for the HMMs are the 3D trajectories of the person’s head and hands. They are extracted from image sequences provided by a stereo camera. In a person-independent test scenario, our system achieved a gesture detection rate of 88%. For 90% of the detected gestures, the correct pointing target (one out of eight objects) was identified.


Hide Markov Model Gesture Recognition American Sign Stereo Camera Hold Phase 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: Real-Time Tracking of the Human Body. IEEE Transaction on Pattern Analysis and Machine Intelligence 19(7) (1997)Google Scholar
  2. 2.
    Azarbayejani, A., Pentland, A.: Real-time self-calibrating stereo person tracking using 3-D shape estimation from blob features. In: Proceedings of 13th ICPR (1996)Google Scholar
  3. 3.
    Darrell, T., Gordon, G., Harville, M., Woodfill, J.: Integrated person tracking using stereo, color, and pattern detection. In: IEEE Conference on Computer Vision and Pattern Recognition, Santa Barbara, CA (1998)Google Scholar
  4. 4.
    Starner, T., Pentland, A.: Visual Recognition of American Sign Language Using Hidden Markov Models. M.I.T. Media Laboratory, Perceptual Computing Section, Cambridge, USA (1994)Google Scholar
  5. 5.
    Becker, D.A.: Sensei: A Real-Time Recognition, Feedbak and Training System for T’ai Chi Gestures. M.I.T. Media Lab Perceptual Computing Group Technical Report No. 426 (1997)Google Scholar
  6. 6.
    Wilson, A.D., Bobick, A.F.: Recognition and Interpretation of Parametric Gesture. In: Intl. Conference on Computer Vision ICCV, pp. 329–336 (1998)Google Scholar
  7. 7.
    Jojic, N., Brumitt, B., Meyers, B., Harris, S., Huang, T.: Detection and Estimation of Pointing Gestures in Dense Disparity Maps. In: IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France (2000)Google Scholar
  8. 8.
    Konolige, K.: Small Vision Systems: Hardware and Implementation. In: 8th International Symposium on Robotics Research, Hayama, Japan (1997)Google Scholar
  9. 9.
    Yang, J., Lu, W., Waibel, A.: Skin-color modeling and adaption. Technical Report of School of Computer Science, CMU, CMU-CS-97-146 (1997)Google Scholar
  10. 10.
    Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proc. IEEE 77(2), 257–286 (1989)CrossRefGoogle Scholar
  11. 11.
    Campbell, L.W., Becker, D.A., Azarbayejani, A., Bobick, A.F., Pentland, A.: Invariant features for 3-D gesture recognition. In: Second International Workshop on Face and Gesture Recognition, Killington VT (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Kai Nickel
    • 1
  • Rainer Stiefelhagen
    • 1
  1. 1.Interactive Systems LaboratoriesUniversität Karlsruhe (TH)Germany

Personalised recommendations