Aligning Sequences and Actions by Maximizing Space-Time Correlations
We introduced an algorithm for sequence alignment, based on maximizing local space-time correlations. Our algorithm aligns sequences of the same action performed at different times and places by different people, possibly at different speeds, and wearing different clothes. Moreover, the algorithm offers a unified approach to the problem of sequence alignment for a wide range of scenarios (e.g., sequence pairs taken with stationary or jointly moving cameras, with the same or different photometric properties, with or without moving objects). Our algorithm is applied directly to the dense space-time intensity information of the two sequences (or to filtered versions of them). This is done without prior segmentation of foreground moving objects, and without prior detection of corresponding features across the sequences. Examples of challenging sequences with complex actions are shown, including ballet dancing, actions in the presence of other complex scene dynamics (clutter), as well as multi-sensor sequence pairs.
KeywordsVideo Sequence Input Sequence Alignment Algorithm Temporal Derivative Pyramid Level
Unable to display preview. Download preview PDF.
- 1.Burt, P., Hingorani, R., Kolczynski, R.: Mechanisms for isolating component patterns in the sequential analysis of multiple motion. In: Workshop on Visual Motion (1991)Google Scholar
- 2.Carlsson, S.: Recognizing walking people. IJRR 22, 359–370 (2003)Google Scholar
- 3.Caspi, Y., Irani, M.: Aligning non-overlapping sequences. IJCV 48 (2002)Google Scholar
- 4.Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. T-PAMI (2002)Google Scholar
- 5.Caspi, Y., Simakov, D., Irani, M.: Feature-based sequence-to-sequence matching. In: VMODS (2002)Google Scholar
- 6.Giese, M.A., Poggio, T.: Synthesis and recognition of biological motion patterns based on linear superposition of prototypical motion sequences. In: IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes (1999)Google Scholar
- 7.Gonzalez, R.C., Woods, R.E.: Digital Image Processing. Addison-Wesley, Reading (1993)Google Scholar
- 8.Irani, M., Anandan, P.: Robust multi-sensor image alignment. In: ICCV (1998)Google Scholar
- 10.Rao, C., Gritai, A., Shah, M., Syeda-Mahmood, T.: View-invariant alignment and matching of video sequences. In: ICCV, pp. 939–945 (2004)Google Scholar
- 11.Shechtman, E., Irani, M.: Space-time behavior based correlation. In: CVPR (2005)Google Scholar
- 12.Thomas, G., Finney, R.: Calculus and Analytic Geometry, 9th edn. Addison-Wesley, Reading (1996)Google Scholar