Learning to Recognize Complex Actions Using Conditional Random Fields

  • Christopher I. Connolly
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4842)


Surveillance systems that operate continuously generate large volumes of data. One such system is described here, continuously tracking and storing observations taken from multiple stereo systems. Automated event recognition is one way of annotating track databases for faster search and retrieval. Recognition of complex events in such data sets often requires context for successful disambiguation of apparently similar activities. Conditional random fields permit straightforward incorporation of temporal context into the event recognition task. This paper describes experiments in activity learning, using conditional random fields to learn and recognize composite events that are captured by the observation stream.


Video Tracking Conditional Random Fields Learning Event Recognition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Burns, J.B.: Detecting independently moving objects and their interactions in georeferenced airborne video. In: Proceedings of the IEEE Workshop on Detection and Recognition of Events in Video, pp. 12–19. IEEE, Los Alamitos (2001)CrossRefGoogle Scholar
  2. 2.
    Duong, T., Bui, H., Phung, D., Vekatesh, S.: Activity recognition and abnormality detection with the switching hidden semi-Markov model. In: IEEE International Conference on Computer Vision and Pattern Recognition (2005)Google Scholar
  3. 3.
    Harville, M., Li, D.: Fast, integrated person tracking and activity recognition with plan-view templates from a single stereo camera. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (June 2004)Google Scholar
  4. 4.
    Iocchi, L., Bolles, R.: Integrating plan-view tracking and color-based person models for multiple people tracking. In: International Conference on Image Processing, pp. 872–875 (2005)Google Scholar
  5. 5.
    Ivanov, Y., Stauffer, C., Bobick, A., Grimson, E.: Video surveillance of interactions. In: Proceedings of the CVPR ’99 Workshop on Visual Surveillance (1998)Google Scholar
  6. 6.
    Kudo, T.: CRF++, yet another CRF toolkit. Web Page,
  7. 7.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of ICML 2001, pp. 282–289 (2001)Google Scholar
  8. 8.
    Medioni, G.G., Cohen, I., Bremond, F., Hongeng, S., Nevatia, R.: Event detection and analysis from video streams. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(8), 873–889 (2001)CrossRefGoogle Scholar
  9. 9.
    Moore, D.J., Essa, I.A., Hayes, M.H.: Exploiting human actions and object context for recognition tasks. In: ICCV (1), pp. 80–86 (1999)Google Scholar
  10. 10.
    Nevatia, R., Hobbs, J., Bolles, B.: An ontology for video event representation. In: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2004), vol. 7, p. 119 (2004)Google Scholar
  11. 11.
    Sminchisescu, C., Kanaujia, A., Li, Z., Metaxas, D.: Conditional models for contextual human motion recognition. In: Proceedings of the International Conference on Computer Vision ICCV 2005 (2005)Google Scholar
  12. 12.
    Sutton, C., McCallum, A.: An Introduction to Conditional Random Fields for Relational Learning. ch. 4. MIT Press, Cambridge (2006)Google Scholar
  13. 13.
    Toshev, A., Bremond, F., Thonnat, M.: An APRIORI-based method for frequent composite event discovery in videos. In: Computer Vision Systems, p. 10 (2006)Google Scholar
  14. 14.
    Vail, D.L., Veloso, M.M., Lafferty, J.D.: Conditional random fields for activity recognition. In: Proceedings of the 2007 Conference on Autonomous Agents and Multiagent Systems (2007)Google Scholar
  15. 15.
    Vasilescu, M.A.O., Terzopoulos, D.: Multilinear subspace analysis of image ensembles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2003), pp. 93–99 (June 2003)Google Scholar
  16. 16.
    Wang, T., Li, J., Diao, Q., Hu, W., Zhang, Y., Dulong, C.: Semantic event detection using conditional random fields. In: Semantic Learning Applications in Multimedia, p. 109 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Christopher I. Connolly
    • 1
  1. 1.SRI International, 333 Ravenswood Avenue, Menlo Park, CA 

Personalised recommendations