Recognizing Human Behavior Using Hidden Markov Models

  • Junji Yamato
Part of the The Kluwer International Series in Video Computing book series (VICO, volume 3)


This chapter describes a new human behavior recognition method based on Hidden Markov Models (HMM). We use a feature-based bottom-up approach using HMM, which can provide a learning capability and time-scale invariability. To apply HMM to our aim, time sequential images are transformed to an image feature vector sequence, and the sequence is converted to a symbol sequence by Vector Quantization. In learning human behavior categories, the model parameters of HMM are optimized so as to best describe training sequences. For recognition, the model that best matches the observed sequence is chosen. A new VQ method and HMM configuration for behavior recognition is proposed and evaluated. Experimental results of real time-sequential images of sports scenes show a higher than 90% recognition rate. We also describe an example application of this behavior recognition method to content-based video database retrieval.


Behavior recognition Hidden Markov Models Vector quantization content-based image retrieval 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Aloimonos, J. (1990). Purposive and Qualitative Active Vision. In Proceeding of ICPR, pages 346–360.Google Scholar
  2. Bobick, A. and Wilson, A. (1997). A state-based approach to the representation and recognition of gesture. IEEE Transaction on Pattern Analysis and Machine Intelligence.Google Scholar
  3. Brand, M. (1997). Couples hidden markov models for modeling interacting processes. Technical report, “The Media Lab, MIT”.Google Scholar
  4. Brand, M., Oliver, N., and Pentlan, A. (1997). Coupled hidden markov models for complex action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
  5. Cedras, C. and Shah, M. (1995). Motion Based Recognition: A Survey. Image and Vision Computing, 13(2): 129–155.CrossRefGoogle Scholar
  6. Davis, J. and Shah, M. (1994). Visual Gesture Recognition. Vision, Image and Signal Processing, 141(2): 101–106.CrossRefGoogle Scholar
  7. He, Y. and Kundu, A. (1991). Planar Shape Classification using Hidden Markov Model. In Proc. CVPR’91, pages 10–15.Google Scholar
  8. Hogg, D. (1983). Model-based vision: a program to see a walking person. Image and vision computing, 1(1):5–20.CrossRefGoogle Scholar
  9. Horowitz, B. and Pentland, A. (1991). Recovery of Non-Rigid Motion and Structure. In Proc. CVPR′91, pages 325–330.Google Scholar
  10. Huang, X. D., Ariki, Y., and Jack, M. A. (1990). Hidden Markov Models for Speech Recognition. Edinburgh Univ. Press.Google Scholar
  11. Kunda, A., He, Y., and Bahl, P. (1989). Handwritten Word Recognition: a Hidden Markov Model Based Approach. Pattern Recognition, pages 283–297.Google Scholar
  12. Linde, Y., Buzo, A., and Gray, R. (1980). An Algorithm for Vector Quantization. IEEE Trans. Commin., 28(1):84–95.CrossRefGoogle Scholar
  13. Mao, W. and Kung, S. (1990). An Object Recognition System Using Stochastic Knowledge Source and VLSI Parallel Architecuture. In Proc. 10th ICPR, pages 832–836.Google Scholar
  14. O′Rouke, J. and Badler, N. (1980). Model-Based Image Analysis of Human Motion Using Constraint Propagation. IEEE Trans. PAMI, PAMI-2(6):522–536.Google Scholar
  15. Pentland, A. (2000). Looking at People: Sensing for Ubiquitous and Wearable Computing. IEEE Transctions on Pattern Analysis and Machine Intelligence, 22(1): 107–119CrossRefGoogle Scholar
  16. Rabiner, R. and Juang, B.-H. (1986). An Introduction to Hidden Markov Models. IEEE ASSP MAGAZINE, pages 4–16.Google Scholar
  17. Rimey, R. and Brown, C. (1990). Selective Attention as Sequential Behavior: Modeling Eye Movements with an Augmented Hidden Markov Model. In Proc. DARPA Image Understanding Workshop, pages 840–849.Google Scholar
  18. Sakaguchi, T., Ohya, J., and Kishino, F. (1995). Facial expression recognition using hmms. Television society, 49(8): 1060–1067.Google Scholar
  19. Samaria, F. (1993). Face Segmentation for Identification using Hidden Markov Models. In Proceedings of 4th British Machine Vision Conference.Google Scholar
  20. Samaria, F. and Fallside, F. (1993). Automated Face Identification using Hidden Markov Models. In Proceedings of the International Conference on Advanced Mechatronics.Google Scholar
  21. Terzopoulos, D. and Metaxas, D. (1991). Dynamic 3D Models with Local and Global Deformations: Deformable Superquadrics. IEEE Trans. on PAMI, 13(7):703–714.CrossRefGoogle Scholar
  22. Umeda, M. (1982). Recognition of Multi-Font Printed Chinese Characters. In Proc. 6th ICPR, pages 793–796.Google Scholar
  23. Yamamoto, M. (1991). Human Motion Analysis Based on A Robot Arm Model. In Proc. CVPR′91, pages 664–665.Google Scholar
  24. Yamato, J., Ohya, J., and Ishii, K. (1992). Recognizing Human Action in Time-Sequential Images using Hidden Markov Models. In Proc. CVPR′92, pages 379–387.Google Scholar

Copyright information

© Springer Science+Business Media New York 2002

Authors and Affiliations

  • Junji Yamato
    • 1
  1. 1.R&D strategy departmentNippon Telegraph and Telephone CorporationTokyoJapan

Personalised recommendations