Epipolar-plane image analysis: An approach to determining structure from motion

  • Robert C. Bolles
  • H. Harlyn Baker
  • David H. Marimont
Article

Abstract

We present a technique for building a three-dimensional description of a static scene from a dense sequence of images. These images are taken in such rapid succession that they form a solid block of data in which the temporal continuity from image to image is approximately equal to the spatial continuity in an individual image. The technique utilizes knowledge of the camera motion to form and analyze slices of this solid. These slices directly encode not only the three-dimensional positions of objects, but also such spatiotemporal events as the occlusion of one object by another. For straight-line camera motions, these slices have a simple linear structure that makes them easier to analyze. The analysis computes the three-dimensional positions of object features, marks occlusion boundaries on the objects, and builds a three-dimensional map of “free space.” In our article, we first describe the application of this technique to a simple camera motion, and then show how projective duality is used to extend the analysis to a wider class of camera motions and object types that include curved and moving objects.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  1. 1.
    E.H. Adelson and J.R. Bergen, “Spatiotemporal energy models for the perception of motion,” Journal of the Optical Society of America A 2, pp. 284–299, 1985.Google Scholar
  2. 2.
    G. Adiv, “Determining three-dimensional motion and structure from optical flow generated by several moving objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-7, pp. 384–401, 1985.Google Scholar
  3. 3.
    J. Aloimonos and I. Rigoutsos, “Determining the 3-D motion of a rigid planar patch without correspondence, under perspective projection,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 167–174.Google Scholar
  4. 4.
    T. Atherton, Report of private discussion with B. Steer of the University of Warwick, 1986.Google Scholar
  5. 5.
    L. Auslander and R.E. MacKenzie, Introduction to Differentiable Manifolds, New York: Dover, 1977.Google Scholar
  6. 6.
    H.H. Baker, R.C. Bolles, and D.H. Marimont, “A new technique for obtaining depth information from a moving sensor,” in Proceedings of the ISPRS Commission II Symposium on Photogrammetric and Remote Sensing Systems for Data Processing and Analysis, Baltimore, 1986.Google Scholar
  7. 7.
    S.T. Barnard and W.B. Thompson, “Disparity analysis of images,” IEEE Transactions on Pattern Analysis and Machine Intelligence 2, 1980, pp. 333–340.Google Scholar
  8. 8.
    J. Barron, “A Survey of Approaches for Determining Optic Flow, Environmental Layout and Egomotion,” Department of Computer Science, University of Toronto, Report No. RBCV-TR-84-5, 1984.Google Scholar
  9. 9.
    J.V. Beck and K.J. Arnold, “Parameter Estimation in Engineerign and Science, New York: John Wiley and Sons, 1977.Google Scholar
  10. 10.
    W.H. Beyer (ed.), CRC Standard Mathematical Tables, 26th edn., Boca Raton, FL: CRC Press, 1981.Google Scholar
  11. 11.
    A.P. Blicher and S.M. Omohundro, “Unique recovery of motion and optic flow via lie algebras,” in Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI-85), Los Angeles, 1985, pp. 889–891.Google Scholar
  12. 12.
    R.C. Bolles, and H.H. Baker, “Epipolar-plane image analysis: A technique for analyzing motion sequences,” in Proceedings of the Third Workshop on Computer Vision: Representation and Control, Bellaire, MI, 1985, pp. 168–178.Google Scholar
  13. 13.
    N.J. Bridwell and T.S. Huang, “A discrete spatial representation for lateral motion stereo,” Computer Vision, Graphics, and Image Processing 21, pp. 33–57, 1983.Google Scholar
  14. 14.
    T.J. Broida and R. Chellappa, “Estimation of object motion parameters from noisy images,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8, pp. 90–99, 1986.Google Scholar
  15. 15.
    T.J., Broida and R. Chellappa, “Kinematics and structure of a rigid object from a sequence of noisy images,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 95–100.Google Scholar
  16. 16.
    A.R. Bruss and B.K.P. Horn, “Passive navigation,” Computer Vision, Graphics, and Image Processing, 21, pp. 3–20, 1983.Google Scholar
  17. 17.
    B.F. Buxton and H. Buxton, “Monocular depth perception from optical flow by space time signal processing,” Proceedings of the Royal Society of London B, 218, pp. 27–47, 1983.Google Scholar
  18. 18.
    B.F. Buxton and H. Buxton, “Computation of optic flow from the motion of edge features in image sequences,” Image and Vision Computing, 2, 1984, pp. 59–75.Google Scholar
  19. 19.
    R. Chatila and J.-P. Laumond, “Position referencing and consistent world modeling for mobile robots,” in Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, 1985, pp. 138–145.Google Scholar
  20. 20.
    H.S.M. Coxeter, Projective Geometry, New York: Blaisdell, 1964.Google Scholar
  21. 21.
    L.S. Dreschler and H.-H. Nagel, “Volumetric model and 3-D trajectory of a moving car derived from nomocular TV-frame sequences of a street scene,” in Proceedings of the Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), Vancouver, 1981, pp. 692–697.Google Scholar
  22. 22.
    L.S. Dreschler and H.-H. Nagel, “On the selection of critical points and local curvature extrema of region boundaries for interframe matching,” in Image sequence Processing and Dynamic Scene Analysis, T.S. Huang (ed.), Berlin: Spring-Verlag, 1983, pp. 457–470.Google Scholar
  23. 23.
    J.-Q. Fang and T.S. Huang, “Solving three-dimensional small-rotation motion equations: Uniqueness, algorithms, and numerical results, Computer Vision, Graphics, and Image Processing 26, pp. 183–206, 1984.Google Scholar
  24. 24.
    J.-Q. Fang and T.S. Huang, “Some experiments on estimating the 3-D motion parameters of a rigid body from two consecutive image frames,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6, pp. 545–554, 1984.Google Scholar
  25. 25.
    C.L. Fennema and W.B. Thompson, “Velocity determination in scenes containing several moving objects,” Computer Graphics and Image Processing 9, pp. 301–315, 1979.Google Scholar
  26. 26.
    D.B. Gennery, “Tracking known three-dimensional objects,” in Proceedings of the Second National Conference on Artificial Intelligence (AAAI-82), Pittsburgh, 1982, pp. 13–17.Google Scholar
  27. 27.
    M.J. Hannah, “Bootstrap stereo,” in Proceedings of the Image Understanding Workshop, College Park, MD, 1980, pp. 201–208.Google Scholar
  28. 28.
    S.M. Haynes and R. Jain, “Detection of moving edges,” Computer Vision, Graphics, and Image Processing, 21, 1983, pp. 345–367.Google Scholar
  29. 29.
    S.M. Haynes and R. Jain, “Low level motion events: trajectory discontinuities,”.in Proceedings of the First Conference on AI Applications, Denver, 1984, pp. 251–256.Google Scholar
  30. 30.
    D.J. Heeger, “Depth and flow from motion energy,” in Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI-86), Philadelphia, 1986, pp. 657–663.Google Scholar
  31. 31.
    E.C. Hildreth, “Computations underlying the measurement of visual motion,” Artificial Intelligence, 23, pp. 309–354, 1984.Google Scholar
  32. 32.
    E.C. Hildreth and N.M. Grzywacz, “The incremental recovery of structure from motion: Position vs. velocity based formulations,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 137–143.Google Scholar
  33. 33.
    D.D. Hoffman, “Inferring local surface orientation from motion fields,” Journal of the Optical Society of America 72, 888–892, 1982.Google Scholar
  34. 34.
    D.D. Hoffman and B.E. Flinchbaugh, “The interpretation of biological motion,” Biological Cybernetics 42, pp. 195–204, 1982.Google Scholar
  35. 35.
    B.K.P. Horn and B.G. Schunk, “Determining optical flow,” Artificial Intelligence 17, pp. 185–203, 1981.Google Scholar
  36. 36.
    B.K.P. Horn and E.J. Weldon, Jr., “Robust direct methods for recovering motion,” unpublished manuscript, February 1986.Google Scholar
  37. 37.
    T.S. Huang (ed.), Image sequence Analysis, Berlin: Springer-Verlag, 1981.Google Scholar
  38. 38.
    D.A. Huffman, “Impossible objects as nonsense sentences,” Machine Intelligence 6, pp. 295–324, 1971.Google Scholar
  39. 39.
    D.A. Huffman, “A duality concept for the analysis of polyhedral scenes,” Machine Intelligence 8, pp. 475–492, 1977.Google Scholar
  40. 40.
    R.Jain, “Direct computation of the focus of expansion,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-5, pp. 58–63, 1983.Google Scholar
  41. 41.
    R. Jain, “Detection on moving edges” in Proceedings of the First Conference on AI Applications, Denver, 1984, pp. 142–148.Google Scholar
  42. 42.
    K. Kanatani, “Tracing planar surface motion from projection without knowing correspondence,” Computer Vision, Graphics, and Image Processing 29, pp. 1–12, 1985.Google Scholar
  43. 43.
    K. Kanatani, “Detecting the motion of a planar surface by line and surface integrals,” Computer Vision, Graphics, and Image Processing 29, pp. 13–22, 1985.Google Scholar
  44. 44.
    K. Kanatani, “Structure from motion without correspondence: General principle,” in Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI-85), Los Angeles, 1985, pp. 886–888.Google Scholar
  45. 45.
    K. Kanatani, “Transformation of optical flow by camera rotation,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 113–118.Google Scholar
  46. 46.
    J.J. Koenderink and A.J.van Doorn, “Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer,” Optica Acta 22, pp. 773–791, 1975.Google Scholar
  47. 47.
    J.J. Koenderink and A.J.van Doorn, “Local structure of movement parallax of the plane,” Journal of the Optical Society of America 66, pp. 717–723, 1976.Google Scholar
  48. 48.
    J.J. Koenderink and A.J.van Doorn, “The singularities of the visual mapping,” Biological Cybernetics 24, pp. 51–59, 1976.Google Scholar
  49. 49.
    J.J. Koenderink and A.J. van Doorn, “How an ambulant observer can construct a model of the environment from the geometrical structure of the visual inflow,” in Kybernetik 77, Hauske and Butenandt (eds.), 1977, pp. 224–227.Google Scholar
  50. 50.
    J.J. Koenderink and A.J.van Doorn, “Exterospecific component of the motion parallax field,” Journal of the Optical Society of America 71, pp. 953–957, 1981.Google Scholar
  51. 51.
    D.T. Lawton, “Processing translational motion sequences,” Computer Vision, Graphics, and Image Processing 22, pp. 116–144, 1983.Google Scholar
  52. 52.
    Y. Liu and T.S. Huang, “Estimation of rigid body motion using straight line correspondences,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 47–52.Google Scholar
  53. 53.
    H.D. Longuet-Higgins, “A computer algorithm for reconstructing a scene from two projections,” Nature 293, pp. 133–135, 1981.Google Scholar
  54. 54.
    H.D. Longuet-Higgins, “The visual ambiguity of a moving plane,” Proceedings of the Royal Society of London B 223, pp. 165–175, 1984.Google Scholar
  55. 55.
    H.D. Longuet-Higgins and K. Prazdny, “The interpretation of a moving retinal image,” Proceedings of the Royal Society of London B 208, pp. 385–397, 1980.Google Scholar
  56. 56.
    A.K. Mackworth, “Interpreting pictures of polyhedral scenes,” Artificial Intelligence 4, pp. 121–137, 1977.Google Scholar
  57. 57.
    D.H. Marimont, “Inferring spatial structure from feature correspondences,” PhD dissertation, Stanford University, 1986.Google Scholar
  58. 58.
    D.H. Marimont, “Projective duality and the analysis of image sequences,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 7–14.Google Scholar
  59. 59.
    D. Marr and E. Hildreth, “Theory of edge detection,” Proceedings of the Royal Society of London B 207, pp. 187–217, 1980.Google Scholar
  60. 60.
    E.A. Maxwell, The Methods of Plane Projective Geometry Based on the Use of General Homogeneous Coordinates, Cambridge: Cambridge University Press, 1946.Google Scholar
  61. 61.
    E.A. Maxwell, General Homogeneous Coordinates in Space of Three Dimensions, Cambridge: Cambridge University Press, 1959.Google Scholar
  62. 62.
    J.L. Melsa and D.L. Cohn, Decision and Estimation Theory, New York: McGraw-Hill, 1978.Google Scholar
  63. 63.
    A. Mitiche, S. Seida, and J.K. Aggarwal, “Line-based computation of structure and motion using angular invariance,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 175–180.Google Scholar
  64. 64.
    H.P. Moravec, “Visual mapping by a robot rover,” in Proceedings of the International Joint Conference on Artificial Intelligence, Tokyo, 1979, pp. 598–600.Google Scholar
  65. 65.
    H.P. Moravec and A. Elfes, “High resolution maps from wide angle sonar,” in Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, 1985, pp. 116–120.Google Scholar
  66. 66.
    H.-H. Nagel, “Displacement vectors derived from second-order intensity variations in image sequences,” Computer Vision, Graphics, and Image Processing 21, pp. 85–117, 1983.Google Scholar
  67. 67.
    S. Negahdaripour and B.K.P. Horn, “Direct passive navigation,” MIT Artif. Intell. Lab., Massachusetts Inst. Technol., AI Memo 821, February 1985.Google Scholar
  68. 68.
    S. Negahdaripour, “Direct passive navigation: analytical solution for planes,” MIT Artif. Intell. Lab., Massachusetts Inst. Technol., AI Memo 863, August 1985.Google Scholar
  69. 69.
    R. Nevatia, “Depth measurement from motion stereo,” Computer Graphics and Image Processing, 5, 1976, pp. 203–214.Google Scholar
  70. 70.
    A.P. Pentland, “Perceptual organization and the representation of natural form,” Artificial Intelligence 28, pp. 293–331, 1986.Google Scholar
  71. 71.
    K. Prazdny, “Motion and structure from optical flow,” in Proceedings of the Sixth International Joint Conference on Artificial Intelligence (IJCAI-79), Tokyo, 1979, pp. 702–704.Google Scholar
  72. 72.
    K. Prazdny, “Egomotion and relative depth map from optical flow,” Biological Cybernetics, 36, pp. 87–102, 1980.Google Scholar
  73. 73.
    K. Prazdny, “Determining the instantaneous direction of motion from optical flow generated by a curvilinearly moving observer,” Computer and Image Processing 17, pp. 238–248, 1981.Google Scholar
  74. 74.
    K. Prazdny, “On the information in optical flows,” Computer Vision, Graphics, and Image Processing 22, pp. 239–259, 1983.Google Scholar
  75. 75.
    K. Ramer, “An iterative procedure for the polygonal approximation of plane curves,” Computer Graphics and Image Processing 1, 1972, pp. 224–256.Google Scholar
  76. 76.
    J.W. Roach and J.K. Aggarwal, “Determining the movement of objects from a sequence of images,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-2, pp. 554–562, 1980.Google Scholar
  77. 77.
    J.W. Roach and J.S. Wright, “Spherical dual images: A 3D representation method for solid objects that combines dual space and Gaussian spheres,” in Proceedings of the IEEE International Conference on Robotics and Automation, San Francisco, 1986, pp. 1087–1092.Google Scholar
  78. 78.
    M. Subbarao, “Interpretation of image motion fields: A spatio-temporal approach,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 157–165.Google Scholar
  79. 79.
    M. Subbarao and A.M. Waxman, “On the uniqueness of image flow solutions for planar surfaces in motion,” in Proceedings of the Third Workshop on Computer Vision: Representation and Control, Bellaire, MI, 1985, pp. 129–140.Google Scholar
  80. 80.
    E.H. Thompson, “A rational algebraic formulation of the problem of relative orientation,” Photogrammetric Record 3, pp. 152–159, 1959.Google Scholar
  81. 81.
    W.B. Thompson and S.T. Barnard, “Lower-level estimation and interpretation of visual motion,” Computer 14, 1981, pp. 20–28.Google Scholar
  82. 82.
    W.B. Thompson, K.M. Mutch, and V.A. Berzins, “Dynamic occlusion analysis in optical flow fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-7, pp. 374–383, 1985.Google Scholar
  83. 83.
    R.Y. Tsai, “Estimating 3-D motion parameters and object surface structures from the image motion of conic arcs. I: Theoretical basis,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Boston, 1983, pp. 122–125.Google Scholar
  84. 84.
    R.Y. Tsai, “Estimating 3-D motion parameters and object surface structures from the image motion of curved edges,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, 1983, pp. 259–266.Google Scholar
  85. 85.
    R.Y. Tsai and T.S. Huang, “Estimating three-dimensional motion parameters of a rigid planar patch,” IEEE Transactions on Acoustics, Speech, and Signal Processing 29, pp. 1147–1152, 1981.Google Scholar
  86. 86.
    R.Y. Tsai and T.S. Huang, “Estimating three-dimensional motion parameters of a rigid planar patch. II: Singular value decomposition,” IEEE Transactions on Acoustics, Speech, and Signal Processing 30, pp. 525–533, 1982.Google Scholar
  87. 87.
    R.Y. Tsai and T.S. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6, pp. 13–27, 1984.Google Scholar
  88. 88.
    S. Ullman, The Interpretation of Visual Motion, Cambridge, MA: MIT Press, 1979.Google Scholar
  89. 89.
    S. Ullman, “Recent computational studies in the interpretation of structure from motion,” in Human and Machine Vision, J. Beck, B. Hope, and A. Rosenfeld (eds.), Orlando, FL: Academic Press, 1983, pp. 459–480.Google Scholar
  90. 90.
    S. Ullman, “Maximizing rigidity: The incremental recovery of 3-D structure from rigid and rubbery motion,” Perception 13, pp. 255–274, 1984.Google Scholar
  91. 91.
    A.M. Waxman, “An image flow paradigm,” in Proceedings of the Second IEEE Workshop on Computer Vision: Representation and Control, Annapolis, 1984, pp. 49–57.Google Scholar
  92. 92.
    A.M. Waxman, and S. Ullman, “Surface structure and 3-D motion from image flow: A kinematic analysis,” Center for Automation Research, University of Maryland, CAR Tech. Report 24, October 1983.Google Scholar
  93. 93.
    A.M. Waxman, and S. Ullman, “Surface structure and three-dimensional motion from image flow kinematics,” International Journal of Robotics Research 4, pp. 72–94, 1985.Google Scholar
  94. 94.
    A.M. Waxman and K. Wohn, “Contour evolution, neighborhood deformation, and global image flow: Planar surfaces in motion,” Center for Automation Research, University of Maryland, CAR Tech. Report 58, April 1984.Google Scholar
  95. 95.
    A.M. Waxman and K. Wohn, “Contour evolution, neighborhood deformation, and global image flow: Planar surfaces in motion,” International Journal of Robotics Research 4, pp. 95–108, 1985.Google Scholar
  96. 96.
    J.A. Webb and J.K. Aggarwal, “Visually interpreting the motion of objects in space,” Computer 14, pp. 40–46, 1981.Google Scholar
  97. 97.
    K.W. Wong (author-editor), “Basic mathematics of photogrammetry,” in Manual of Photogrammetry, 4th edn., Falls Church, VA: American Society of Photogrammetry, 1980.Google Scholar
  98. 98.
    C.R. WylieJr., Introduction to Projective Geometry, New York: McGraw-Hill, 1970.Google Scholar
  99. 99.
    M. Yamamoto, “Motion analysis using the visualized locus method,” untranslated Japanese articles, 1981.Google Scholar
  100. 100.
    B.L. Yen and T.S. Huang, “Determining 3-D motion and structure of a rigid body using the spherical projection,” Computer Graphics and Image Processing, 21, pp. 21–32, 1983.Google Scholar
  101. 101.
    B.L. Yen and T.S. Huang, “Determining the 3-D motion and structure of a rigid body using straight line correspondences,” Image sequence Processing and Dynamic Scene Analysis, T.S. Huang (ed.), Berlin: Spring-Verlag, pp. 365–394, 1983.Google Scholar

Copyright information

© Kluwer Academic Publishers 1987

Authors and Affiliations

  • Robert C. Bolles
    • 1
  • H. Harlyn Baker
    • 1
  • David H. Marimont
    • 1
  1. 1.Artificial Intelligence CenterSRI InternationalMenlo Park

Personalised recommendations