Advertisement

International Journal of Computer Vision

, Volume 22, Issue 3, pp 199–218 | Cite as

Spline-Based Image Registration

  • Richard Szeliski
  • James Coughlan
Article

Abstract

The problem of image registration subsumes a number of problems and techniques in multiframe image analysis, including the computation of optic flow (general pixel-based motion), stereo correspondence, structure from motion, and feature tracking. We present a new registration algorithm based on spline representations of the displacement field which can be specialized to solve all of the above mentioned problems. In particular, we show how to compute local flow, global (parametric) flow, rigid flow resulting from camera egomotion, and multiframe versions of the above problems. Using a spline-based description of the flow removes the need for overlapping correlation windows, and produces an explicit measure of the correlation between adjacent flow estimates. We demonstrate our algorithm on multiframe image registration and the recovery of 3D projective scene geometry. We also provide results on a number of standard motion sequences.

motion analysis multiframe image analysis hierarchical image registration optical flow splines global motion models structure from motion direct motion estimation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adelson, E. H. and Bergen, J. R. 1985. Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America, A2(2):284–299.Google Scholar
  2. Amit, Y. 1993. Anon-linear variational problem for image matching. unpublished manuscript (from Newton Institute).Google Scholar
  3. Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3):283–310.Google Scholar
  4. Bajcsy, R. and Broit, C. 1982. Matching of deformed images. In Sixth International Conference on Pattern Recognition (ICPRs'82), IEEE Computer Society Press: Munich, Germany, pp. 351–353.Google Scholar
  5. Bajcsy, R. and Kovacic, S. 1989. Multiresolution elastic matching. Computer Vision, Graphics, and Image Processing, 46(1):1–21.Google Scholar
  6. Barnard, S. T. and Fischler, M. A. 1982. Computational stereo. Computing Surveys, 14(4):553–572.Google Scholar
  7. Barron, J. L., Fleet, D. J., and Beauchemin, S. S. 1994. Performance of optical flow techniques. International Journal of Computer Vision, 12(1):43–77.Google Scholar
  8. Beier, T. and Neely, S. 1992. Feature-based image metamorphosis. Computer Graphics (SIGGRAPHs'92), 26(2):35–42.Google Scholar
  9. Bergen, J. R., Anandan, P., Hanna, K. J., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Second European Conference on Computer Vision (ECCVs'92), Santa Margherita Liguere, Springer-Verlag: Italy, pp. 237–252.Google Scholar
  10. Beymer, D., Shashua, A., and Poggio, T. 1993. Example based image analysis and synthesis. A. I. Memo 1431, Massachusetts Institute of Technology.Google Scholar
  11. Blake, A., Curwen, R., and Zisserman, A. 1993. A framework for spatio-temporal control in the tracking of visual contour. International Journal of Computer Vision, 11(2):127–145.Google Scholar
  12. Bolles, R. C., Baker, H. H., and Marimont, D. H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. International Journal of Computer Vision, 1:7–55.Google Scholar
  13. Brown, L. G. 1992. A survey of image registration techniques. Computing Surveys, 24(4):325–376.Google Scholar
  14. Burr, D. J. 1981. A dynamic model for image registration. Computer Graphics and Image Processing, 15(2):102–112.Google Scholar
  15. Burt, P. J. and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications, COM-31(4):532–540.Google Scholar
  16. Carlbom, I., Terzopoulos, D., and Harris, K. M. 1991. Reconstructing and visualizing models of neuronal dendrites. In Scientific Visualization of Physical Phenomena, N. M. Patrikalakis (Ed.), Springer-Verlag: New York, pp. 623–638.Google Scholar
  17. Dhond, U. R. and Aggarwal, J. K. 1989. Structure from stereo—A review. IEEE Transactions on Systems, Man, and Cybernetics, 19(6):1489–1510.Google Scholar
  18. Dreschler, L. and Nagel, H.-H. 1982. Volumetric model and 3D trajectory of a moving car derived from monocular TV frame sequences of a stree scene. Computer Graphics and Image Processing, 20:199–228.Google Scholar
  19. Enkelmann, W. 1988. Investigations of multigrid algorithms for estimation of optical flow fields in image sequences. Computer Vision, Graphics, and Image Processing, pp. 150–177.Google Scholar
  20. Farin, G. E. 1992. Curves and Surfaces for Computer Aided Geometric Design. Academic Press: Boston, Massachusetts, 3rd edition.Google Scholar
  21. Faugeras, O. D. 1992. What can be seen in three dimensions with an uncalibrated stereo rig? In Second European Conference on Computer Vision (ECCVs'92), Santa Margherita Liguere, Springer-Verlag: Italy, pp. 563–578.Google Scholar
  22. Fleet, D. and Jepson, A. 1990. Computation of component image velocity from local phase information. International Journal of Computer Vision, 5:77–104.Google Scholar
  23. Fuh, C.-S. and Maragos, P. 1991. Motion displacement estimation using an affine model for image matching. Optical Engineering, 30(7):881–887.Google Scholar
  24. Geiger, D., Ladendorf, B., and Yuille, A. 1992. Occlusions and binocular stereo. In Second European Conference on Computer Vision (ECCVs'92), Santa Margherita Liguere, Springer-Verlag, Italy, pp. 425–433.Google Scholar
  25. Gennert, M. A. 1988. Brightness-based stereo matching. In Second International Conference on Computer Vision (ICCVs'88), IEEE Computer Society Press: Tampa, Florida, pp. 139–143.Google Scholar
  26. Goshtasby, A. 1986. Piecewise linear mapping functions for image registration. Pattern Recognition, 19(6):459–466.Google Scholar
  27. Goshtasby, A. 1988. Image registration by local approximation methods. Image and Vision Computing, 6(4):255–261.Google Scholar
  28. Hanna, K. J. 1991. Direct multi-resolution estimation of ego-motion and structure from motion. In IEEE Workshop on Visual Motion, IEEE Computer Society Press: Princeton, New Jersey, pp. 156–162.Google Scholar
  29. Hartley, R. and Gupta, R. 1993. Computing matched-epipolar projections. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'93), IEEE Computer Society Press: New York, pp. 549–555.Google Scholar
  30. Hartley, R., Gupta, R., and Chang, T. 1992. Stereo from uncalibrated cameras. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'92), IEEE Computer Society Press: Champaign, Illinois, pp. 761–764,.Google Scholar
  31. Heeger, D. J. 1987. Optical flow from spatiotemporal filters. In First International Conference on Computer Vision (ICCVs'87), IEEE Computer Society Press: London, England, pp. 181–190.Google Scholar
  32. Hildreth, E. C. 1986. Computing the velocity field along contours. In Motion: Representation and Perception, N. I. Badler and J. K. Tsotsos (Eds.), North-Holland, New York, pp. 121–127.Google Scholar
  33. Horn, B. K. P. and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.Google Scholar
  34. Horn, B. K. P. and Weldon, E. J., Jr. 1988. Direct methods for recovering motion. International Journal of Computer Vision, 2(1):51–76.Google Scholar
  35. Kass, M., Witkin, A., and Terzopoulos, D. 1988. Snakes: Active contour models. International Journal of Computer Vision, 1(4):321–331.Google Scholar
  36. Koenderink, J. J. and van Doorn, A. J. 1991. Affine structure from motion. Journal of the Optical Society of America A, 8:377–385,538.Google Scholar
  37. Le Gall, D. 1991. MPEG: A video compression standard for multimedia applications. Communications of the ACM, 34(4):44–58.Google Scholar
  38. Lucas, B. D. 1984. Generalized Image Matching by the Method of Differences. Ph. D. Thesis, Carnegie Mellon University.Google Scholar
  39. Lucas, B. D. and Kanade, T. 1981. An iterative image registration technique with an application in stereo vision. In Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), Vancouver, pp. 674–679.Google Scholar
  40. Manmatha, R. and Oliensis, J. 1992. Measuring the affine transform —I: Scale and rotation. Technical Report 92-74, University of Massachussets, Amherst, Massachussets.Google Scholar
  41. Matthies, L. H., Szeliski, R., and Kanade, T. 1989. Kalman filter-based algorithms for estimating depth from image sequences. International Journal of Computer Vision, 3:209–236.Google Scholar
  42. Menet, S., Saint-Marc, P., and Medioni, G. 1990. B-snakes: implementation and applications to stereo. In Image Understanding Workshop, Morgan Kaufmann Publishers: Pittsburgh, Pennsylvania, pp. 720–726.Google Scholar
  43. Mohr, R., Veillon, L., and Quan, L. 1993. Relative 3D reconstruction using multiple uncalibrated images. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'93), New York, pp. 543–548.Google Scholar
  44. Nagel, H.-H. 1987. On the estimation of optical flow: Relations between different approaches and some new results. Artificial Intelligence, 33:299–324.Google Scholar
  45. Okutomi, M. and Kanade, T. 1992. A locally adaptive window for signal matching. International Journal of Computer Vision, 7(2):143–162.Google Scholar
  46. Okutomi, M. and Kanade, T. 1993. A multiple baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):353–363.Google Scholar
  47. Poggio, T., Torre, V., and Koch, C. 1985. Computational vision and regularization theory. Nature, 317(6035):314–319.Google Scholar
  48. Press, W. H., Flannery, B. P., Teukolsky, S. A., and Vetterling, W. T. 1992. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press: Cambridge, England, 2nd edition.Google Scholar
  49. Quam, L. H. 1984. Hierarchical warp stereo. In Image Understanding Workshop, Science Applications International Corporation: New Orleans, Louisiana, pp. 149–155.Google Scholar
  50. Rehg, J. and Witkin, A. 1991. Visual tracking with deformation models. In IEEE International Conference on Robotics and Automation, IEEE Computer Society Press: Sacramento, California, pp. 844–850.Google Scholar
  51. Sethi, I. K. and Jain, R. 1987. Finding trajectories of feature points in a monocular image sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(1):56–73.Google Scholar
  52. Shi, J. and Tomasi, C. 1994. Good features to track. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'94), IEEE Computer Society: Seattle, Washington, pp. 593–600.Google Scholar
  53. Simoncelli, E. P., Adelson, E. H., and Heeger, D. J. 1991. Probability distributions of optic flow. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'91), IEEE Computer Society Press: Maui, Hawaii, pp. 310–315.Google Scholar
  54. Singh, A. 1990. An estimation-theoretic framework for image-flow computation. In Third International Conference on Computer Vision (ICCVs'90), IEEE Computer Society Press: Osaka, Japan, pp. 168–177.Google Scholar
  55. Szeliski, R. 1989. Bayesian Modeling of Uncertainty in Low-Level Vision. Kluwer Academic Publishers: Boston, Massachusetts.Google Scholar
  56. Szeliski, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):513–528.Google Scholar
  57. Szeliski, R. 1996. Video mosaics for virtual environments. IEEE Computer Graphics and Applications, 16(2):22–30.Google Scholar
  58. Szeliski, R. and Kang, S. B. 1994. Recovering 3D shape and motion from image streams using nonlinear least squares. Journal of Visual Communication and Image Representation, 5(1):10–28.Google Scholar
  59. Szeliski, R. and Kang, S. B. 1995. Direct methods for visual scene reconstruction. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, pp. 26–33.Google Scholar
  60. Szeliski, R. and Shum, H.-Y. 1996. Motion estimation with quadtree splines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(12):1199–1210.Google Scholar
  61. Szeliski, R., Kang, S. B., and Shum, H.-Y. 1995. A parallel feature tracker for extended image sequences. In IEEE International Symposium on Computer Vision, Coral Gables, Florida, pp. 241–246.Google Scholar
  62. Terzopoulos, D. 1986. Regularization of inverse visual problems involving discontinuities. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(4):413–424.Google Scholar
  63. Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision, 9(2):137–154.Google Scholar
  64. Witkin, A., Terzopoulos, D., and Kass, M. 1987. Signal matching through scale space. International Journal of Computer Vision, 1:133–144.Google Scholar
  65. Wolberg, G. 1990. Digital Image Warping. IEEE Computer Society Press: Los Alamitos, California.Google Scholar
  66. Xu, G., Tsuji, S., and Asada, M. 1987. Amotion stereo method based on coarse-to-fine control strategy. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(2):332–336.Google Scholar
  67. Zheng, Q. and Chellappa, R. 1992. Automatic feature point extraction and tracking in image sequences for arbitrary camera motion. Technical Report CAR-TR-628, Computer Vision Laboratory, Center for Automation Research, University of Maryland.Google Scholar

Copyright information

© Kluwer Academic Publishers 1997

Authors and Affiliations

  • Richard Szeliski
    • 1
  • James Coughlan
    • 2
  1. 1.Microsoft ResearchRedmond
  2. 2.Department of PhysicsHarvard UniversityCambridge

Personalised recommendations