Visual Tracking Using a Pixelwise Spatiotemporal Oriented Energy Representation

  • Kevin J. Cannons
  • Jacob M. Gryn
  • Richard P. Wildes
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6314)


This paper presents a novel pixelwise representation for visual tracking that models both the spatial structure and dynamics of a target in a unified fashion. The representation is derived from spatiotemporal energy measurements that capture underlying local spacetime orientation structure at multiple scales. For interframe motion estimation, the feature representation is instantiated within a pixelwise template warping framework; thus, the spatial arrangement of the pixelwise energy measurements remains intact. The proposed target representation is extremely rich, including appearance and motion information as well as information about how these descriptors are spatially arranged. Qualitative and quantitative empirical evaluation on challenging sequences demonstrates that the resulting tracker outperforms several alternative state-of-the-art systems.


Motion Estimation Feature Representation Visual Tracking Illumination Change Appearance Change 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Supplementary material

978-3-642-15561-1_37_MOESM1_ESM.avi (14.3 mb)
Electronic Supplementary Material (14,634 KB)


  1. 1.
  2. 2.
    Adelson, E., Bergen, J.: Spatiotemporal energy models for the perception of motion. JOSA A 2(2), 284–299 (1985)CrossRefGoogle Scholar
  3. 3.
    Babenko, B., Yang, M., Belongie, S.: Visual tracking with online multiple instance learning. In: CVPR, pp. 983–990 (2009)Google Scholar
  4. 4.
    Bergen, J., Anandan, P., Hanna, K., Hingorani, R.: Hierarchical model-based motion estimation. In: Sandini, G. (ed.) ECCV 1992. LNCS, vol. 588, pp. 237–252. Springer, Heidelberg (1992)Google Scholar
  5. 5.
    Birchfield, S., Rangarajan, S.: Spatiograms versus histograms for region-based tracking. In: CVPR, vol. 2, pp. 1158–1163 (2005)Google Scholar
  6. 6.
    Black, M., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. CVIU 63(1), 75–104 (1996)Google Scholar
  7. 7.
    Bogomolov, Y., Dror, G., Lapchev, S., Rivlin, E., Rudzsky, M.: Classification of moving targets based on motion and appearance. In: BMVC, pp. 142–149 (2003)Google Scholar
  8. 8.
    Burt, P., Bergen, J., Hingorani, R., Kolczynski, R., Lee, W., Leung, A., Lubin, J., Shvayster, H.: Object tracking with a moving camera. In: Motion Wkshp, pp. 2–12 (1989)Google Scholar
  9. 9.
    Cannons, K.: A review of visual tracking. Technical Report CSE-2008-07, York University, Department of Computer Science and Engineering (2008)Google Scholar
  10. 10.
    Cannons, K., Wildes, R.: Spatiotemporal oriented energy features for visual tracking. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) ACCV 2007, Part I. LNCS, vol. 4843, pp. 532–543. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  11. 11.
    Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. PAMI 25(5), 564–575 (2003)Google Scholar
  12. 12.
    Derpanis, K., Sizintsev, M., Cannons, K., Wildes, R.: Efficient action spotting based on a spacetime oriented structure representation. In: CVPR (2010)Google Scholar
  13. 13.
    Elgammal, A., Duraiswami, R., Davis, L.: Probabilistic tracking in joint feature-spatial spaces. In: CVPR, pp. 781–788 (2003)Google Scholar
  14. 14.
    Freeman, W., Adelson, E.: The design and use of steerable filters. PAMI 13(9), 891–906 (1991)Google Scholar
  15. 15.
    Granlund, G., Knutsson, H.: Signal Processing for Computer Vision. Kluwer, Dordrecht (1995)Google Scholar
  16. 16.
    Hager, G., Dewan, M., Stewart, C.: Multiple kernel tracking with SSD. In: CVPR, vol. 1, pp. 790–797 (2004)Google Scholar
  17. 17.
    Horn, B.: Robot Vision. MIT Press, Cambridge (1986)Google Scholar
  18. 18.
    Huber, P.: Robust Statistical Procedures. SIAM Press, Philadelphia (1977)zbMATHGoogle Scholar
  19. 19.
    Irani, M., Rousso, B., Peleg, S.: Computing occluding and transparent motions. IJCV 12(1), 5–16 (1994)CrossRefGoogle Scholar
  20. 20.
    Jepson, A., Fleet, D., El-Maraghi, T.: Robust on-line appearance models for visual tracking. PAMI 25(10), 1296–1311 (2003)Google Scholar
  21. 21.
    Lucas, B., Kanade, T.: An iterative image registration technique with application to stereo vision. In: DARPA IUW, pp. 121–130 (1981)Google Scholar
  22. 22.
    Meyer, F., Bouthemy, P.: Region-based tracking using affine motion models in long image sequences. CVGIP: Image Understanding 60(2), 119–140 (1994)CrossRefGoogle Scholar
  23. 23.
    Ross, D., Lim, J., Lin, R., Yang, M.: Incremental learning for robust visual tracking. IJCV 77, 125–141 (2008)CrossRefGoogle Scholar
  24. 24.
    Sato, K., Aggarwal, J.: Temporal spatio-velocity transformation and its applications to tracking and interaction. CVIU 96(2), 100–128 (2004)Google Scholar
  25. 25.
    Shi, J., Tomasi, C.: Good features to track. In: CVPR, pp. 593–600 (1994)Google Scholar
  26. 26.
    Sizintsev, M., Wildes, R.: Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching. In: CVPR, pp. 493–500 (2009)Google Scholar
  27. 27.
    Stauffer, C., Grimson, W.: Learning patterns of activity using real-time tracking. PAMI 22(8), 747–757 (2000)Google Scholar
  28. 28.
    Takala, V., Pietikainen, M.: Multi-object tracking using color, texture and motion. In: ICCV (2007)Google Scholar
  29. 29.
    Wildes, R., Bergen, J.: Qualitative spatiotemporal analysis using an oriented energy representation. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 768–784. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  30. 30.
    Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: Real-time tracking of the human body. PAMI 19(7), 780–785 (1997)Google Scholar
  31. 31.
    Zaharescu, A., Wildes, R.: Anomalous behaviour detection using spatiotemporal oriented energies, subset inclusion histogram comparison and event-driven processing. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 563–576. Springer, Heidelberg (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Kevin J. Cannons
    • 1
  • Jacob M. Gryn
    • 1
  • Richard P. Wildes
    • 1
  1. 1.Department of Computer Science and EngineeringYork UniversityTorontoCanada

Personalised recommendations