Learning Semantic Scene Models by Trajectory Analysis

  • Xiaogang Wang
  • Kinh Tieu
  • Eric Grimson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3953)


In this paper, we describe an unsupervised learning framework to segment a scene into semantic regions and to build semantic scene models from long-term observations of moving objects in the scene. First, we introduce two novel similarity measures for comparing trajectories in far-field visual surveillance. The measures simultaneously compare the spatial distribution of trajectories and other attributes, such as velocity and object size, along the trajectories. They also provide a comparison confidence measure which indicates how well the measured image-based similarity approximates true physical similarity. We also introduce novel clustering algorithms which use both similarity and comparison confidence. Based on the proposed similarity measures and clustering methods, a framework to learn semantic scene models by trajectory analysis is developed. Trajectories are first clustered into vehicles and pedestrians, and then further grouped based on spatial and velocity distributions. Different trajectory clusters represent different activities. The geometric and statistical models of structures in the scene, such as roads, walk paths, sources and sinks, are automatically learned from the trajectory clusters. Abnormal activities are detected using the semantic scene models. The system is robust to low-level tracking errors.


Spectral Cluster Trajectory Analysis Local Path Scene Model Trajectory Cluster 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kaucic, R., Perera, A., Brooksby, G., Kaufhold, J., Hoogs, A.: A Unified Framework for Tracking through Occlusions and across Sensor Gaps. In: Proceedings of CVPR (2005)Google Scholar
  2. 2.
    Stauffer, C., Grimson, E.: Learning Patterns of Activity Using Real-Time Tracking. IEEE Trans. on PAMI 22(8), 747–757 (2000)Google Scholar
  3. 3.
    Makris, D., Ellis, T.: Path Detection in Video Surveillance. Image and Vision Computing 20, 859–903 (2002)CrossRefGoogle Scholar
  4. 4.
    Fernyhough, J.H., Cohn, A.G., Hogg, D.C.: Generation of Semantic Regions from Image Sequences. In: Proc. of ECCV (1996)Google Scholar
  5. 5.
    Makris, D., Ellis, T.: Automatic Learning of an Activity-Based Semantic Scene Model. In: Proc. of IEEE Conference on Advanced Video and Signal Based Surveillance (2003)Google Scholar
  6. 6.
    Mckenna, S.J., Nait-Charif, H.: Learning Spatial Context from Tracking Using Penalized Likelihood. In: Proc. of ICPR (2004)Google Scholar
  7. 7.
    Bose, B., Grimson, E.: Improving Object Classification in Far-Field Video. In: Proc. CVPR (2004)Google Scholar
  8. 8.
    Stauffer, C.: Minimally-Supervised Classification using Multiple Observation Sets. In: ICCV (2003)Google Scholar
  9. 9.
    Dubuisson, M.P., Jain, A.K.: A Modified Hausdorff distance for Object Matching. In: Proc. of ICPR (1994)Google Scholar
  10. 10.
    Meila, M., Shi, J.: A Random Walk View of Spectral Segmentation. In: Proc. of AISTATS (2001)Google Scholar
  11. 11.
    Wang, X., Tieu, K., Grimson, E.: Learning Semantic Scene Models by Trajectory Analysis. Tech. Rep. MIT-CSAIL-TR-2006-08,
  12. 12.
    Gumbel, E.J., Greenwood, J.A.: The Circular Normal Distribution: Theory and Tables. J. Amer. Stat. Soc. 48(261), 131–152 (1953)zbMATHMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Xiaogang Wang
    • 1
  • Kinh Tieu
    • 1
  • Eric Grimson
    • 1
  1. 1.Computer Science and Artificial Intelligence LaboratoryMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations