International Journal of Computer Vision

, Volume 68, Issue 1, pp 53–64 | Cite as

Feature-Based Sequence-to-Sequence Matching

  • Yaron CaspiEmail author
  • Denis Simakov
  • Michal Irani


This paper studies the problem of matching two unsynchronized video sequences of the same dynamic scene, recorded by different stationary uncalibrated video cameras. The matching is done both in time and in space, where the spatial matching can be modeled by a homography (for 2D scenarios) or by a fundamental matrix (for 3D scenarios). Our approach is based on matching space-time trajectories of moving objects, in contrast to matching interest points (e.g., corners), as done in regular feature-based image-to-image matching techniques. The sequences are matched in space and time by enforcing consistent matching of all points along corresponding space-time trajectories.

By exploiting the dynamic properties of these space-time trajectories, we obtain sub-frame temporal correspondence (synchronization) between the two video sequences. Furthermore, using trajectories rather than feature-points significantly reduces the combinatorial complexity of the spatial point-matching problem when the search space is large. This benefit allows for matching information across sensors in situations which are extremely difficult when only image-to-image matching is used, including: (a) matching under large scale (zoom) differences, (b) very wide base-line matching, and (c) matching across different sensing modalities (e.g., IR and visible-light cameras). We show examples of recovering homographies and fundamental matrices under such conditions.


sequence-to-sequence matching alignment in space and time dynamic information multi-sensor alignment wide base-line matching trajectory matching 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bergen, J., Anandan, P., Hanna, K., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In European Conference on Computer Vision (ECCV). Santa Margarita Ligure, pp. 237–252.Google Scholar
  2. Burt, P. and Kolczynski, R. 1993. Enhanced image capture through fusion. In: International Conference on Computer Vision (ICCV). Berlin, pp. 173–182.Google Scholar
  3. Caspi, Y. and Irani, M. 2000. A step towards sequence-to-sequence alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hilton Head Island, South Carolina, pp. 682–689.Google Scholar
  4. Caspi, Y., Simakov, D., and Irani, M. 2002. Feature-based sequence-to-sequence matching. In: ECCV, VAMODS Workshop. Copenhagen.Google Scholar
  5. Faugeras, O., Luong, Q., and Papadopoulo, T. 2001. The Geometry of Multiple Images. MIT Press.Google Scholar
  6. Ferrari, V., Tuytelaars, T., and Gool, L.V. 2003. Wide-baseline multiple-view Correspondences. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1. Madison, Wisconsin, pp. 718–725.Google Scholar
  7. Fischler, M.A. and Bolles, R. 1981. RANSAC random sample concensus: A paradigm for model fitting with applications to image analysis and automated cartography. In Communications of the ACM, Vol. 24. pp. 381–395.CrossRefMathSciNetGoogle Scholar
  8. Giese, M.A. and Poggio, T. 2000. Morphable models for the analysis and synthesis of complex motion patterns. International Journal of Computer Vision 38(1):59–73.CrossRefGoogle Scholar
  9. Hampel, F., Rousseeuw, P., Ronchetti, E., and Stahel, W. 1986. Robust Statistics: The Approach Based on Influence Functions. John Wiley: New York.Google Scholar
  10. Harris, C. and Stephens, M. 1988. A combined corner and edge detector. In 4th Alvey Vision Conference, pp. 147–151.Google Scholar
  11. Hartley, R. and Zisserman, A. 2000. Multiple View Geometry in Computer Vision. Cambridge University Press: Cambridge.Google Scholar
  12. Kadir, T., Zisserman, A., and Brady, M. 2004. An affine invariant salient region detector. In European Conference on Computer Vision (ECCV). Prague, Czech Republic.Google Scholar
  13. Lee, L., Romano, R., and Stein, G. 2000. Monitoring activities from multiple video streams: establishing a common coordinate frame. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) (Special Issue on Video Surveillance and Monitoring), 22:758–767.Google Scholar
  14. Lowe, D.G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91–110.CrossRefGoogle Scholar
  15. Lucas, B. and Kanade, T. 1981, ‘An iterative image registration technique with an application to stereo vision. In: Image Understanding Workshop, pp. 121–130.Google Scholar
  16. Matas, J., Chum, O., Martin, U., and Pajdla, T. 2002. Robust wide baseline stereo from maximally stable extremal regions. In P.L. Rosin and D. Marshall (eds.), British Machine Vision Conference, Vol. 1. London, UK, pp. 384–393.Google Scholar
  17. Mikolajczyk, K. and Schmid, C., 2004. Scale and affine invariant interest point detectors. International Journal of Computer Vision 60(1):63–86.CrossRefGoogle Scholar
  18. Mindru, F., Tuytelaars, T., Gool, L.V., and Moons, T. 2004. Moment invariants for recognition under changing viewpoint and illumination. Comput. Vis. Imag Underst., 94(1–3):3–27.CrossRefGoogle Scholar
  19. Mundy, J. and Zisserman, A. 1992. Geometric invariance in computer vision. In MIT Press.Google Scholar
  20. Rousseeuw, P. 1987. Robust Regression and Outlier Detection. Wiley: New York.Google Scholar
  21. Shechtman, E., Caspi, Y., and Irani, M. 2002. Increasing video resolution in tim and space. In European Conference on Computer Vision (ECCV), Copenhagen.Google Scholar
  22. Stauffer, C., and Tieu, K., 2003. Automated multi-camera planar tracking correspondence modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Madison, Wisconsin, pp. 259–266.Google Scholar
  23. Stein, G.P. 1998. Tracking from multiple view points: self-calibration of space and time. In DARPA IU Workshop, Montery CA, pp. 1037–1042.Google Scholar
  24. Szeliski, R. and Shum., H.-Y. 1997. Creating full view panoramic image mosaics and environment maps. In Computer Graphics Proceedings, Annual Conference Series, pp. 251–258.Google Scholar
  25. Tomasi, C. and Kanade, T. 1991. Detection and tracking of point features. Technical Report CMU-CS-91-132, Carnegie Mellon University.Google Scholar
  26. Tresadern, P. and Reid, I. 2003. Synchronizing image sequences of non-rigid objects. In British Machine Vision Conference, Vol. 2, Norwich, pp. 629–638.Google Scholar
  27. Tuytelaars, T. and Gool, L.V. 2004. Matching widely separated views based on affine invariant regions. International Journal of Computer Vision, 59(1):61–85.CrossRefGoogle Scholar
  28. Wong, K.-Y. K. and Cipolla, R. 2001. Structure and motion from silhouettes. In International Conference on Computer Vision (ICCV), Vol. II. Vancouver, Canada, pp. 217–222.Google Scholar
  29. Xu, C. and Zhang, Z. 1996. Epipolar Geometry in Stereo, Motion and Object Recognition. Kluwer Academic Publishers: Dordecht, The Netherlands.Google Scholar
  30. Zhang, Z., Deriche R., Faugeras, O., and Luong, Q. 1995, ‘A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artificial Intelligence 78:87–119.Google Scholar
  31. Zoghlami, I., Faugeras, O., and Deriche, R. 1997. Using geometric corners to build a 2d mosaic from a set of images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 420–425.Google Scholar

Copyright information

© Springer Science + Business Media, LLC. 2006

Authors and Affiliations

  1. 1.Dept. of Computer Science and Applied MathThe Weizmann Institute of ScienceRehovotIsrael

Personalised recommendations