Integrated Tracking and Recognition of Human Activities in Shape Space

  • Bi Song
  • Amit K. Roy-Chowdhury
  • N. Vaswani
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4338)


Activity recognition consists of two fundamental tasks: tracking the features/objects of interest, and recognizing the activities. In this paper, we show that these two tasks can be integrated within the framework of a dynamical feedback system. In our proposed method, the recognized activity is continuously adapted based on the output of the tracking algorithm, which in turn is driven by the identity of the recognized activity. A non-linear, non-stationary stochastic dynamical model on the “shape” of the objects participating in the activities is used to represent their motion, and forms the basis of the tracking algorithm. The tracked observations are used to recognize the activities by comparing against a prior database. Measures designed to evaluate the performance of the tracking algorithm serve as a feedback signal. The method is able to automatically detect changes and switch between activities happening one after another, which is akin to segmenting a long sequence into homogeneous parts. The entire process of tracking, recognition, change detection and model switching happens recursively as new video frames become available. We demonstrate the effectiveness of the method on real-life video and analyze its performance based on such metrics as detection delay and false alarm.


Video Sequence Tracking Error Particle Filter Activity Recognition Tracking Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bar-Shalom, Y., Fortmann, T.E.: Tracking and Data Association. Academic Press, London (1988)MATHGoogle Scholar
  2. 2.
    Cham, T.-J., Rehg, J.M.: A multiple hypothesis approach to figure tracking. In: Computer Vision and Pattern Recognition (1999)Google Scholar
  3. 3.
    Doucet, A., Freitas, N., Gordon, N.: Sequential Monte Carlo Methods in Practice. Springer, Heidelberg (2001)MATHGoogle Scholar
  4. 4.
    Dryden, I., Mardia, K.: Statistical Shape Analysis. John Wiley and Sons, Chichester (1998)MATHGoogle Scholar
  5. 5.
    Grimson, W., Lee, L., Romano, R., Stauffer, C.: Using Adaptive Tracking to Classify and Monitor Activities in a Site. In: Computer Vision and Pattern Recognition, pp. 22–31 (1998)Google Scholar
  6. 6.
    Harville, M., Li, D.: Fast, integrated person tracking and activity recognition with plan-view templates from a single stereo camera. In: Computer Vision and Pattern Recognition, pp. 398–405 (2004)Google Scholar
  7. 7.
    Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Trans. on Systems, Man, Cybernetics - Part C: Applications and Reviews 34(3) (2004)Google Scholar
  8. 8.
    Isard, M., Blake, A.: Condensation: Conditional Density Propagation for Visual Tracking. In: International Journal of Computer Vision, pp. 5–28 (1998)Google Scholar
  9. 9.
    Jackson, J., Yezzi, A., Soatto, S.: Tracking deformable moving objects under severe occlusions. In: IEEE Conference on Decision and control (December 2004)Google Scholar
  10. 10.
    Li, Y., Boult, T.: Understanding Images of Graphical User Interfaces: A new approach to activity recognition for visual surveillance. In: ACM UIST (2003)Google Scholar
  11. 11.
    Liao, L., Fox, D., Kautz, H.: Location-based activity recognition using relational markov networks. In: Proc. of the International Joint Conference on Artificial Intelligence (2005)Google Scholar
  12. 12.
    Niethammer, M., Tannenbaum, A.: Dynamic level sets for visual tracking. In: IEEE Conference on Decision and Control (2004)Google Scholar
  13. 13.
    North, B., Blake, A., Isard, M., Rittscher, J.: Learning and classification of complex dynamics. IEEE Trans. on Pattern Analysis and Machine Intelligence 22(9), 1016–1034 (2000)CrossRefGoogle Scholar
  14. 14.
    Rathi, Y., Vaswani, N., Tannenbaum, A., Yezzi, A.: Particle Filtering for Geometric Active Contours with Application to Tracking Moving and Deforming Objects. In: Computer Vision and Pattern Recognition (2005)Google Scholar
  15. 15.
    Rittscher, J., Blake, A.: Classification of human body motion. In: International Conf. on Computer Vision, vol. 2, pp. 634–639 (1999)Google Scholar
  16. 16.
    Vaswani, N.: Change Detection in Partially Observed Nonlinear Dynamic Systems with Unknown Change Parameters. In: American Control Conference (2004)Google Scholar
  17. 17.
    Vaswani, N., Chellappa, R.: NonStationary Shape Activities. In: Proc. of IEEE Conf. on Decision and Control (2005)Google Scholar
  18. 18.
    Vaswani, N., Roy-Chowdhury, A., Chellappa, R.: Shape Activities: A Continuous State HMM for Moving/Deforming Shapes with Application to Abnormal Activity Detection. IEEE Trans. on Image Processing (October 2005)Google Scholar
  19. 19.
    Zhai, Y., Shah, M.: A general framework for temporal video scene segmentation. In: International Conf. on Computer Vision (2005)Google Scholar
  20. 20.
    Zhou, S., Chellappa, R., Moghaddam, B.: Visual Tracking and Recognition Using Appearance-Adaptive Models in Particle Filters. IEEE Trans. on Image Processing 13(11), 1491–1506 (2004)CrossRefGoogle Scholar
  21. 21.
    Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of human faces from video. Computer Vision and Image Understanding 91, 214–245 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Bi Song
    • 1
    • 2
  • Amit K. Roy-Chowdhury
    • 1
    • 2
  • N. Vaswani
    • 1
    • 2
  1. 1.University of CaliforniaRiversideUSA
  2. 2.Iowa State UniversityUSA

Personalised recommendations