Spatio-temporal Discovery: Appearance + Behavior = Agent

  • Prithwijit Guha
  • Amitabha Mukerjee
  • K. S. Venkatesh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4338)


Experiments in infant category formation indicate a strong role for temporal continuity and change in perceptual categorization. Computational approaches to model discovery in vision have traditionally focused on static images, with appearance features such as shape playing an important role. In this work, we consider integrating agent behaviors with shape for the purpose of agent discovery. Improved algorithms for video segmentation and tracking under occlusion enable us to construct models that characterize agents in terms of motion and interaction with other objects. We present a preliminary approach for discovering agents based on a combination of appearance and motion histories. Using uncalibrated camera images, we characterize objects discovered in the scene by their shape and motion attributes, and cluster these using agglomerative hierarchical clustering. Even with very simple feature sets, initial results suggest that the approach forms reasonable clusters for diverse categories such as people, and for very distinct clusters (animals), and performs above average on other classes.


Agglomerative Hierarchical Cluster Motion History Video Segmentation Agent Categorization Variable Length Sequence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Quinn, P., Eimas, P.: The emergence of category representations during infancy: Are separate perceptual and conceptual processes required? Journal of Cognition and Development 1, 55–61 (2000)CrossRefGoogle Scholar
  2. 2.
    Spelke, E.S.: Principles of object perception. Cognitive Science 14, 29–56 (1990)CrossRefGoogle Scholar
  3. 3.
    Gredebaeck, G., von Hofsten, C.: Infants evolving representations of object motion during occlusion: A longitudinal study of 6- to 12-month-old infants. Infancy 6, 165–184 (2004)CrossRefGoogle Scholar
  4. 4.
    Mandler, J.M.: Foundations of Mind. Oxford University Press, New York (2004)Google Scholar
  5. 5.
    Mareschal, D., French, R.M., Quinn, P.: A connectionist account of asymmetric category learning in early infancy. Developmental Psychology 36, 635–645 (2000)CrossRefGoogle Scholar
  6. 6.
    Blind1: Details withheld. In: for Blind Review (2006)Google Scholar
  7. 7.
    Zivkovic, Z.: Improved adaptive gaussian mixture model for background subtraction. In: Proceedings of the 17th International Conference on Pattern Recognition, vol. 2, pp. 28–31 (2004)Google Scholar
  8. 8.
    Proesmans, M., Gool, L.V., Pauwels, E., Osterlinck, A.: Determination of optical flow and its discontinuities using non-linear diffusion. In: The 3rd Eurpoean Conference on Computer Vision, vol. 2, pp. 295–304 (1994)Google Scholar
  9. 9.
    Collins, Lipton, Kanade, Fujiyoshi, Duggins, Tsin, Tolliver, Enomoto, Hasegawa: A system for video surveillance and monitoring: Vsam final report. Technical Report CMU-RI-TR-00-12, Robotics Institute, Carnegie Mellon University (2000)Google Scholar
  10. 10.
    Duda, R., Hart, P., Stork, D.: Pattern Recognition, 2nd edn. John Wiley and Sons, Chichester (2003)Google Scholar
  11. 11.
    Johnson, N., Hogg, D.: Learning the distribution of object trajectories for event recognition. In: Proceedings of the 6th British conference on Machine vision, vol. 2, pp. 583–592. BMVA Press (1995)Google Scholar
  12. 12.
    Galata, A., Johnson, N., Hogg, D.: Learning variable-length markov models of behavior. Computer Vision and Image Understanding 81, 398–413 (2001)MATHCrossRefGoogle Scholar
  13. 13.
    Buxton, H.: Learning and understanding dynamic scene activity: a review. Image and Vision Computing 21, 125–136 (2003)CrossRefGoogle Scholar
  14. 14.
    Johnson, N., Galata, A., Hogg, D.: The acquisition and use of interaction behavior models. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 866–871. IEEE Computer Society, Los Alamitos (1998)Google Scholar
  15. 15.
    Galata, A., Cohn, A.G., Magee, D., Hogg, D.: Modeling interaction using learnt qualitative spatio-temporal relations and variable length markov models. In: van Harmelen, F. (ed.) Proceedings of European Conference on Artificial Intelligence, pp. 741–745 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Prithwijit Guha
    • 1
  • Amitabha Mukerjee
    • 2
  • K. S. Venkatesh
    • 1
  1. 1.Department of Electrical EngineeringIndian Institute of Technology, KanpurKanpurUttar Pradesh
  2. 2.Department of Computer Science & EngineeringIndian Institute of Technology, KanpurKanpurUttar Pradesh

Personalised recommendations