Aligning Spatio-Temporal Signals on a Special Manifold

  • Ruonan Li
  • Rama Chellappa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6315)


We investigate the spatio-temporal alignment of videos or features/signals extracted from them. Specifically, we formally define an alignment manifold and formulate the alignment problem as an optimization procedure on this non-linear space by exploiting its intrinsic geometry. We focus our attention on semantically meaningful videos or signals, e.g., those describing or capturing human motion or activities, and propose a new formalism for temporal alignment accounting for executing rate variations among realizations of the same video event. By construction, we address this static and deterministic alignment task in a dynamic and stochastic manner: we regard the search for optimal alignment parameters as a recursive state estimation problem for a particular dynamic system evolving on the alignment manifold. Consequently, a Sequential Importance Sampling iteration on the alignment manifold is designed for effective and efficient alignment. We demonstrate the performance on several types of input data that arise in vision problems.


Dynamic Time Warping Optimal Alignment Point Trajectory Alignment Problem Spatial Alignment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lee, L., Romano, R., Stein, G.: Monitoring activities from multiple video streams: Establishing a common coordinate frame. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 758–767 (2000)CrossRefGoogle Scholar
  2. 2.
    Wolf, L., Zomet, A.: Sequence to sequence self calibration. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 370–382. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  3. 3.
    Rao, C., Gritaiand, A., Shah, M., Syeda-Mahmood, T.: View-invariant alignment and matching of video sequences. In: ICCV (2003)Google Scholar
  4. 4.
    Laptev, I., Belongie, S., Perez, P., Wills, J.: Periodic motion detection and segmentation via approximate sequence alignment. In: ICCV (2005)Google Scholar
  5. 5.
    Caspi, Y., Simakov, D., Irani, M.: Feature-based sequence-to-sequence matching. International Journal of Computer Vision 68, 53–64 (2006)CrossRefGoogle Scholar
  6. 6.
    Wolf, L., Zomet, A.: Wide baseline matching between unsynchronized video sequences. International Journal of Computer Vision 68, 43–52 (2006)CrossRefGoogle Scholar
  7. 7.
    Padua, F., Carceroni, R., Santos, G., Kutulakos, K.: Linear sequence-to-sequence alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 304–320 (2010)CrossRefGoogle Scholar
  8. 8.
    Caspi, Y., Irani, M.: A step towards sequence-to-sequence alignment. In: CVPR (2000)Google Scholar
  9. 9.
    Caspi, Y., Irani, M.: Alignment of non-overlapping sequences. In: ICCV (2001)Google Scholar
  10. 10.
    Ukrainitz, Y., Irani, M.: Aligning sequences and actions by maximizing space-time correlations. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 538–550. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Veeraraghavan, A., Srivastava, A., Roy-Chowdhury, A., Chellappa, R.: Rate-invariant recognition of humans and their activities. IEEE Transactions on Image Processing 18(6), 1326–1339 (2009)CrossRefGoogle Scholar
  12. 12.
    Zhou, F., de la Torre, F.: Canonical time warping for alignment of human behavior. In: NIPS (2009)Google Scholar
  13. 13.
    Gordon, N., Salmond, D., Smith, A.: Novel approach to nonlinear/non-gaussian bayesian state estimation. IEE Proceedings F on RAdar and Signal Processing 140, 107–113 (1993)CrossRefGoogle Scholar
  14. 14.
    Arias, T., Edelman, A., Smith, S.: The geometry of algorithms with orthogonality constraints. SIAM Journal of Matrix Analysis and Applications 20, 303–353 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Maybank, S.: The Fisher-Rao metric for projective transformations of the line. International Journal of Computer Vision 63, 191–206 (2005)CrossRefGoogle Scholar
  16. 16.
    Srivastava, A., Jermyn, I., Joshi, S.: Riemannian analysis of probability density functions with applications in vision. In: CVPR (2007)Google Scholar
  17. 17.
    Srivastava, A., Klassen, E.: Bayesian and geometric subspace tracking. Advances in Applied Probability 36, 43–56 (2004)zbMATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    Wu, Y., Wu, B., Liu, J., Lu, H.: Probabilistic tracking on riemannian manifolds. In: ICPR (2008)Google Scholar
  19. 19.
    Kwon, J., Lee, K.M., Park, F.C.: Visual tracking via geometric particle filtering on the affine group with optimal importance functions. In: CVPR (2009)Google Scholar
  20. 20.
    Porikli, F., Pan, P.: Refressed importance sampling on manifolds for efficient object tracking. In: AVSS (2009)Google Scholar
  21. 21.
    Li, R., Chellappa, R., Zhou, S.K.: Learning multi-modal densities on discriminative temporal interaction manifold for group activity recognition. In: CVPR (2009)Google Scholar
  22. 22.
    Li, R., Chellappa, R.: Group motion segmentation using a spatio-temporal driving force model. In: CVPR (2010)Google Scholar
  23. 23.
    Sarkar, S., Phillips, P.J., Liu, Z., Robledo, I., Grother, P., Bowyer, K.W.: The human id gait challenge problem: Data sets, performance, and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 162–177 (2005)CrossRefGoogle Scholar
  24. 24.
    Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: A local svm approach. In: ICPR (2004)Google Scholar
  25. 25.
    Lui, Y.M., Beveridge, J.R.: Grassmann registration manifolds for face recognition. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 44–57. Springer, Heidelberg (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ruonan Li
    • 1
  • Rama Chellappa
    • 1
  1. 1.Center for Automation ResearchUniversity of MarylandCollege ParkUSA

Personalised recommendations