A Bayesian Computer Vision System for Modeling Human Interactions

  • Nuria Oliver
  • Barbara Rosario
  • Alex Pentland
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1542)


We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system is particularly concerned with detecting when interactions between people occur, and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely HMMs and CHMMs, for modeling behaviors and interactions. The CHMM model is shown to work much more efficiently and accurately.

Finally, to deal with the problem of limited training data, a synthetic ‘Alife-style’ training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.


Feature Vector Hide Markov Model False Alarm Rate Synthetic Data Dynamic Time Warping 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    R.K. Bajcsy. Active perception vs. passive perception. In CVWS85, pages 55–62, 1985.Google Scholar
  2. 2.
    A. Bobick and R. Bolles. The representation space paradigm of concurrent evolving object descriptions. PAMI, pages 146–156, February 1992.Google Scholar
  3. 3.
    A.F. Bobick. Computers seeing action. In Proceedings of BMVC, volume 1, pages 13–22, 1996.Google Scholar
  4. 4.
    Matthew Brand. Coupled hidden markov models for modeling interacting processes. Submitted to Neural Computation, November 1996.Google Scholar
  5. 5.
    Matthew Brand, Nuria Oliver, and Alex Pentland. Coupled hidden markov models for complex action recognition. In In Proceedings of IEEE CVPR97, 1996.Google Scholar
  6. 6.
    W.L. Buntine. Operations for learning with graphical models. Journal of Artificial Intelligence Research, 1994.Google Scholar
  7. 7.
    W.L. Buntine. A guide to the literature on learning probabilistic networks from data. IEEE Transactions on Knowledge and Data Engineering, 1996.Google Scholar
  8. 8.
    Hilary Buxton and Shaogang Gong. Advanced visual surveillance using bayesian networks. In International Conference on Computer Vision, Cambridge, Massachusetts, June 1995.Google Scholar
  9. 9.
    C. Castel, L. Chaudron, and C. Tessier. What is going on? a high level interpretation of sequences of images. In Proceedings of the workshop on conceptual descriptions from images, ECCV, pages 13–27, 1996.Google Scholar
  10. 10.
    T. Darrell and A. Pentland. Active gesture recognition using partially observable markov decision processes. In ICPR96, page C9E.5, 1996.Google Scholar
  11. 11.
    J.H. Fernyhough, A.G. Cohn, and D.C. Hogg. Building qualitative event models automatically from visual input. In ICCV98, pages 350–355, 1998.Google Scholar
  12. 12.
    Zoubin Ghahramani and Michael I. Jordan. Factorial hidden Markov models. In David S. Touretzky, Michael C. Mozer, and M.E. Hasselmo, editors, NIPS, volume 8, Cambridge, MA, 1996. MITP.Google Scholar
  13. 13.
    David Heckerman. A tutorial on learning with bayesian networks. Technical Report MSR-TR-95-06, Microsoft Research, Redmond, Washington, 1995. Revised June 96.Google Scholar
  14. 14.
    T. Huang, D. Koller, J. Malik, G. Ogasawara, B. Rao, S. Russel, and J. Weber. Automatic symbolic traffic scene analysis using belief networks. pages 966–972. Proceedings 12th National Conference in AI, 1994.Google Scholar
  15. 15.
    R. Kauth, A. Pentland, and G. Thomas. Blob: An unsupervised clustering approach to spatial preprocessing of mss imagery. In 11th Int’l Symp. on Remote Sensing of the Environment, Ann Harbor MI, 1977.Google Scholar
  16. 16.
    B. Moghaddam and A. Pentland. Probabilistic visual learning for object detection. In ICCV95, pages 786–793, 1995.Google Scholar
  17. 17.
    H.H. Nagel. From image sequences towards conceptual descriptions. IVC, 6(2):59–74, May 1988.Google Scholar
  18. 18.
    N. Oliver, F. Berard, and A. Pentland. Lafter: Lips and face tracking. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR97), S.Juan, Puerto Rico, June 1997.Google Scholar
  19. 19.
    N. Oliver, B. Rosario, and A. Pentland. Graphical models for recognizing human interactions. In To appear in Proceedings of NIPS98, Denver, Colorado, USA, November 1998.Google Scholar
  20. 20.
    A. Pentland. Classification by clustering. In IEEE Symp. on Machine Processing and Remotely Sensed Data, Purdue, IN, 1976.Google Scholar
  21. 21.
    A. Pentland and A. Liu. Modeling and prediction of human behavior. In DARPA97, page 201 206, 1997.Google Scholar
  22. 22.
    Lawrence R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. PIEEE, 77(2):257–285, 1989.Google Scholar
  23. 23.
    Lawrence K. Saul and Michael I. Jordan. Boltzmann chains and hidden Markov models. In Gary Tesauro, David S. Touretzky, and T.K. Leen, editors, NIPS, volume 7, Cambridge, MA, 1995. MITP.Google Scholar
  24. 24.
    Padhraic Smyth, David Heckerman, and Michael Jordan. Probabilistic independence networks for hidden Markov probability models. AI memo 1565, MIT, Cambridge, MA, Feb 1996.Google Scholar
  25. 25.
    C. Williams and G. E. Hinton. Mean field networks that learn to discriminate temporally distorted strings. In Proceedings, connectionist models summer school, pages 18–22, San Mateo, CA, 1990. Morgan Kaufmann.Google Scholar
  26. 26.
    C. Wren, A. Azarbayejani, T. Darrell, and A. Pentland. Pfinder: Real-time tracking of the human body. InPhotonics East, SPIE, volume 2615, 1995. Bellingham, WA.Google Scholar
  27. 27.
    C.R. Wren, A. Azarbayejani, T. Darrell, and A. Pentland. Pfinder: Real-time tracking of the human body. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):780–785, July 1997.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1999

Authors and Affiliations

  • Nuria Oliver
    • 1
  • Barbara Rosario
    • 1
  • Alex Pentland
    • 1
  1. 1.Vision and Modeling. Media LaboratoryMITCambridgeUSA

Personalised recommendations