# Epipolar-plane image analysis: An approach to determining structure from motion

- 2k Downloads
- 339 Citations

## Abstract

We present a technique for building a three-dimensional description of a static scene from a dense sequence of images. These images are taken in such rapid succession that they form a solid block of data in which the temporal continuity from image to image is approximately equal to the spatial continuity in an individual image. The technique utilizes knowledge of the camera motion to form and analyze slices of this solid. These slices directly encode not only the three-dimensional positions of objects, but also such spatiotemporal events as the occlusion of one object by another. For straight-line camera motions, these slices have a simple linear structure that makes them easier to analyze. The analysis computes the three-dimensional positions of object features, marks occlusion boundaries on the objects, and builds a three-dimensional map of “free space.” In our article, we first describe the application of this technique to a simple camera motion, and then show how projective duality is used to extend the analysis to a wider class of camera motions and object types that include curved and moving objects.

## Preview

Unable to display preview. Download preview PDF.

### Bibliography

- 1.E.H. Adelson and J.R. Bergen, “Spatiotemporal energy models for the perception of motion,”
*Journal of the Optical Society of America A***2**, pp. 284–299, 1985.Google Scholar - 2.G. Adiv, “Determining three-dimensional motion and structure from optical flow generated by several moving objects,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-7**, pp. 384–401, 1985.Google Scholar - 3.J. Aloimonos and I. Rigoutsos, “Determining the 3-D motion of a rigid planar patch without correspondence, under perspective projection,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 167–174.Google Scholar - 4.T. Atherton, Report of private discussion with B. Steer of the University of Warwick, 1986.Google Scholar
- 5.L. Auslander and R.E. MacKenzie,
*Introduction to Differentiable Manifolds*, New York: Dover, 1977.Google Scholar - 6.H.H. Baker, R.C. Bolles, and D.H. Marimont, “A new technique for obtaining depth information from a moving sensor,” in
*Proceedings of the ISPRS Commission II Symposium on Photogrammetric and Remote Sensing Systems for Data Processing and Analysis*, Baltimore, 1986.Google Scholar - 7.S.T. Barnard and W.B. Thompson, “Disparity analysis of images,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***2**, 1980, pp. 333–340.Google Scholar - 8.J. Barron, “A Survey of Approaches for Determining Optic Flow, Environmental Layout and Egomotion,” Department of Computer Science, University of Toronto, Report No. RBCV-TR-84-5, 1984.Google Scholar
- 9.J.V. Beck and K.J. Arnold, “
*Parameter Estimation in Engineerign and Science*, New York: John Wiley and Sons, 1977.Google Scholar - 10.W.H. Beyer (ed.),
*CRC Standard Mathematical Tables*, 26th edn., Boca Raton, FL: CRC Press, 1981.Google Scholar - 11.A.P. Blicher and S.M. Omohundro, “Unique recovery of motion and optic flow via lie algebras,” in
*Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI-85)*, Los Angeles, 1985, pp. 889–891.Google Scholar - 12.R.C. Bolles, and H.H. Baker, “Epipolar-plane image analysis: A technique for analyzing motion sequences,” in
*Proceedings of the Third Workshop on Computer Vision: Representation and Control*, Bellaire, MI, 1985, pp. 168–178.Google Scholar - 13.N.J. Bridwell and T.S. Huang, “A discrete spatial representation for lateral motion stereo,”
*Computer Vision, Graphics, and Image Processing***21**, pp. 33–57, 1983.Google Scholar - 14.T.J. Broida and R. Chellappa, “Estimation of object motion parameters from noisy images,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-8**, pp. 90–99, 1986.Google Scholar - 15.T.J., Broida and R. Chellappa, “Kinematics and structure of a rigid object from a sequence of noisy images,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 95–100.Google Scholar - 16.A.R. Bruss and B.K.P. Horn, “Passive navigation,”
*Computer Vision, Graphics, and Image Processing*,**21**, pp. 3–20, 1983.Google Scholar - 17.B.F. Buxton and H. Buxton, “Monocular depth perception from optical flow by space time signal processing,”
*Proceedings of the Royal Society of London B*,**218**, pp. 27–47, 1983.Google Scholar - 18.B.F. Buxton and H. Buxton, “Computation of optic flow from the motion of edge features in image sequences,”
*Image and Vision Computing*,**2**, 1984, pp. 59–75.Google Scholar - 19.R. Chatila and J.-P. Laumond, “Position referencing and consistent world modeling for mobile robots,” in
*Proceedings of the 1985 IEEE International Conference on Robotics and Automation*, St. Louis, 1985, pp. 138–145.Google Scholar - 20.H.S.M. Coxeter,
*Projective Geometry*, New York: Blaisdell, 1964.Google Scholar - 21.L.S. Dreschler and H.-H. Nagel, “Volumetric model and 3-D trajectory of a moving car derived from nomocular TV-frame sequences of a street scene,” in
*Proceedings of the Seventh International Joint Conference on Artificial Intelligence (IJCAI-81)*, Vancouver, 1981, pp. 692–697.Google Scholar - 22.L.S. Dreschler and H.-H. Nagel, “On the selection of critical points and local curvature extrema of region boundaries for interframe matching,” in
*Image sequence Processing and Dynamic Scene Analysis*, T.S. Huang (ed.), Berlin: Spring-Verlag, 1983, pp. 457–470.Google Scholar - 23.J.-Q. Fang and T.S. Huang, “Solving three-dimensional small-rotation motion equations: Uniqueness, algorithms, and numerical results,
*Computer Vision, Graphics, and Image Processing***26**, pp. 183–206, 1984.Google Scholar - 24.J.-Q. Fang and T.S. Huang, “Some experiments on estimating the 3-D motion parameters of a rigid body from two consecutive image frames,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-6**, pp. 545–554, 1984.Google Scholar - 25.C.L. Fennema and W.B. Thompson, “Velocity determination in scenes containing several moving objects,”
*Computer Graphics and Image Processing***9**, pp. 301–315, 1979.Google Scholar - 26.D.B. Gennery, “Tracking known three-dimensional objects,” in
*Proceedings of the Second National Conference on Artificial Intelligence (AAAI-82)*, Pittsburgh, 1982, pp. 13–17.Google Scholar - 27.M.J. Hannah, “Bootstrap stereo,” in
*Proceedings of the Image Understanding Workshop*, College Park, MD, 1980, pp. 201–208.Google Scholar - 28.S.M. Haynes and R. Jain, “Detection of moving edges,”
*Computer Vision, Graphics, and Image Processing*,**21**, 1983, pp. 345–367.Google Scholar - 29.S.M. Haynes and R. Jain, “Low level motion events: trajectory discontinuities,”.in
*Proceedings of the First Conference on AI Applications*, Denver, 1984, pp. 251–256.Google Scholar - 30.D.J. Heeger, “Depth and flow from motion energy,” in
*Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI-86)*, Philadelphia, 1986, pp. 657–663.Google Scholar - 31.E.C. Hildreth, “Computations underlying the measurement of visual motion,”
*Artificial Intelligence*,**23**, pp. 309–354, 1984.Google Scholar - 32.E.C. Hildreth and N.M. Grzywacz, “The incremental recovery of structure from motion: Position vs. velocity based formulations,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 137–143.Google Scholar - 33.D.D. Hoffman, “Inferring local surface orientation from motion fields,”
*Journal of the Optical Society of America***72**, 888–892, 1982.Google Scholar - 34.D.D. Hoffman and B.E. Flinchbaugh, “The interpretation of biological motion,”
*Biological Cybernetics***42**, pp. 195–204, 1982.Google Scholar - 35.B.K.P. Horn and B.G. Schunk, “Determining optical flow,”
*Artificial Intelligence***17**, pp. 185–203, 1981.Google Scholar - 36.B.K.P. Horn and E.J. Weldon, Jr., “Robust direct methods for recovering motion,” unpublished manuscript, February 1986.Google Scholar
- 37.T.S. Huang (ed.),
*Image sequence Analysis*, Berlin: Springer-Verlag, 1981.Google Scholar - 38.D.A. Huffman, “Impossible objects as nonsense sentences,”
*Machine Intelligence***6**, pp. 295–324, 1971.Google Scholar - 39.D.A. Huffman, “A duality concept for the analysis of polyhedral scenes,”
*Machine Intelligence***8**, pp. 475–492, 1977.Google Scholar - 40.R.Jain, “Direct computation of the focus of expansion,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-5**, pp. 58–63, 1983.Google Scholar - 41.R. Jain, “Detection on moving edges” in
*Proceedings of the First Conference on AI Applications*, Denver, 1984, pp. 142–148.Google Scholar - 42.K. Kanatani, “Tracing planar surface motion from projection without knowing correspondence,”
*Computer Vision, Graphics, and Image Processing***29**, pp. 1–12, 1985.Google Scholar - 43.K. Kanatani, “Detecting the motion of a planar surface by line and surface integrals,”
*Computer Vision, Graphics, and Image Processing***29**, pp. 13–22, 1985.Google Scholar - 44.K. Kanatani, “Structure from motion without correspondence: General principle,” in
*Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI-85)*, Los Angeles, 1985, pp. 886–888.Google Scholar - 45.K. Kanatani, “Transformation of optical flow by camera rotation,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 113–118.Google Scholar - 46.J.J. Koenderink and A.J.van Doorn, “Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer,”
*Optica Acta***22**, pp. 773–791, 1975.Google Scholar - 47.J.J. Koenderink and A.J.van Doorn, “Local structure of movement parallax of the plane,”
*Journal of the Optical Society of America***66**, pp. 717–723, 1976.Google Scholar - 48.J.J. Koenderink and A.J.van Doorn, “The singularities of the visual mapping,”
*Biological Cybernetics***24**, pp. 51–59, 1976.Google Scholar - 49.J.J. Koenderink and A.J. van Doorn, “How an ambulant observer can construct a model of the environment from the geometrical structure of the visual inflow,” in
*Kybernetik 77*, Hauske and Butenandt (eds.), 1977, pp. 224–227.Google Scholar - 50.J.J. Koenderink and A.J.van Doorn, “Exterospecific component of the motion parallax field,”
*Journal of the Optical Society of America***71**, pp. 953–957, 1981.Google Scholar - 51.D.T. Lawton, “Processing translational motion sequences,”
*Computer Vision, Graphics, and Image Processing***22**, pp. 116–144, 1983.Google Scholar - 52.Y. Liu and T.S. Huang, “Estimation of rigid body motion using straight line correspondences,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 47–52.Google Scholar - 53.H.D. Longuet-Higgins, “A computer algorithm for reconstructing a scene from two projections,”
*Nature***293**, pp. 133–135, 1981.Google Scholar - 54.H.D. Longuet-Higgins, “The visual ambiguity of a moving plane,”
*Proceedings of the Royal Society of London B***223**, pp. 165–175, 1984.Google Scholar - 55.H.D. Longuet-Higgins and K. Prazdny, “The interpretation of a moving retinal image,”
*Proceedings of the Royal Society of London B***208**, pp. 385–397, 1980.Google Scholar - 56.A.K. Mackworth, “Interpreting pictures of polyhedral scenes,”
*Artificial Intelligence***4**, pp. 121–137, 1977.Google Scholar - 57.D.H. Marimont, “Inferring spatial structure from feature correspondences,” PhD dissertation, Stanford University, 1986.Google Scholar
- 58.D.H. Marimont, “Projective duality and the analysis of image sequences,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 7–14.Google Scholar - 59.D. Marr and E. Hildreth, “Theory of edge detection,”
*Proceedings of the Royal Society of London B***207**, pp. 187–217, 1980.Google Scholar - 60.E.A. Maxwell,
*The Methods of Plane Projective Geometry Based on the Use of General Homogeneous Coordinates*, Cambridge: Cambridge University Press, 1946.Google Scholar - 61.E.A. Maxwell,
*General Homogeneous Coordinates in Space of Three Dimensions*, Cambridge: Cambridge University Press, 1959.Google Scholar - 62.J.L. Melsa and D.L. Cohn,
*Decision and Estimation Theory*, New York: McGraw-Hill, 1978.Google Scholar - 63.A. Mitiche, S. Seida, and J.K. Aggarwal, “Line-based computation of structure and motion using angular invariance,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 175–180.Google Scholar - 64.H.P. Moravec, “Visual mapping by a robot rover,” in
*Proceedings of the International Joint Conference on Artificial Intelligence*, Tokyo, 1979, pp. 598–600.Google Scholar - 65.H.P. Moravec and A. Elfes, “High resolution maps from wide angle sonar,” in
*Proceedings of the 1985 IEEE International Conference on Robotics and Automation*, St. Louis, 1985, pp. 116–120.Google Scholar - 66.H.-H. Nagel, “Displacement vectors derived from second-order intensity variations in image sequences,”
*Computer Vision, Graphics, and Image Processing***21**, pp. 85–117, 1983.Google Scholar - 67.S. Negahdaripour and B.K.P. Horn, “Direct passive navigation,” MIT Artif. Intell. Lab., Massachusetts Inst. Technol., AI Memo 821, February 1985.Google Scholar
- 68.S. Negahdaripour, “Direct passive navigation: analytical solution for planes,” MIT Artif. Intell. Lab., Massachusetts Inst. Technol., AI Memo 863, August 1985.Google Scholar
- 69.R. Nevatia, “Depth measurement from motion stereo,”
*Computer Graphics and Image Processing*,**5**, 1976, pp. 203–214.Google Scholar - 70.A.P. Pentland, “Perceptual organization and the representation of natural form,”
*Artificial Intelligence***28**, pp. 293–331, 1986.Google Scholar - 71.K. Prazdny, “Motion and structure from optical flow,” in
*Proceedings of the Sixth International Joint Conference on Artificial Intelligence (IJCAI-79)*, Tokyo, 1979, pp. 702–704.Google Scholar - 72.K. Prazdny, “Egomotion and relative depth map from optical flow,”
*Biological Cybernetics*,**36**, pp. 87–102, 1980.Google Scholar - 73.K. Prazdny, “Determining the instantaneous direction of motion from optical flow generated by a curvilinearly moving observer,”
*Computer and Image Processing***17**, pp. 238–248, 1981.Google Scholar - 74.K. Prazdny, “On the information in optical flows,”
*Computer Vision, Graphics, and Image Processing***22**, pp. 239–259, 1983.Google Scholar - 75.K. Ramer, “An iterative procedure for the polygonal approximation of plane curves,”
*Computer Graphics and Image Processing***1**, 1972, pp. 224–256.Google Scholar - 76.J.W. Roach and J.K. Aggarwal, “Determining the movement of objects from a sequence of images,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-2**, pp. 554–562, 1980.Google Scholar - 77.J.W. Roach and J.S. Wright, “Spherical dual images: A 3D representation method for solid objects that combines dual space and Gaussian spheres,” in
*Proceedings of the IEEE International Conference on Robotics and Automation*, San Francisco, 1986, pp. 1087–1092.Google Scholar - 78.M. Subbarao, “Interpretation of image motion fields: A spatio-temporal approach,” in
*Proceedings: Workshop on Motion: Representation and Analysis*, Kiawah Island, 1986, pp. 157–165.Google Scholar - 79.M. Subbarao and A.M. Waxman, “On the uniqueness of image flow solutions for planar surfaces in motion,” in
*Proceedings of the Third Workshop on Computer Vision: Representation and Control*, Bellaire, MI, 1985, pp. 129–140.Google Scholar - 80.E.H. Thompson, “A rational algebraic formulation of the problem of relative orientation,”
*Photogrammetric Record***3**, pp. 152–159, 1959.Google Scholar - 81.W.B. Thompson and S.T. Barnard, “Lower-level estimation and interpretation of visual motion,”
*Computer***14**, 1981, pp. 20–28.Google Scholar - 82.W.B. Thompson, K.M. Mutch, and V.A. Berzins, “Dynamic occlusion analysis in optical flow fields,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-7**, pp. 374–383, 1985.Google Scholar - 83.R.Y. Tsai, “Estimating 3-D motion parameters and object surface structures from the image motion of conic arcs. I: Theoretical basis,” in
*Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing*, Boston, 1983, pp. 122–125.Google Scholar - 84.R.Y. Tsai, “Estimating 3-D motion parameters and object surface structures from the image motion of curved edges,” in
*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, Washington, DC, 1983, pp. 259–266.Google Scholar - 85.R.Y. Tsai and T.S. Huang, “Estimating three-dimensional motion parameters of a rigid planar patch,”
*IEEE Transactions on Acoustics, Speech, and Signal Processing***29**, pp. 1147–1152, 1981.Google Scholar - 86.R.Y. Tsai and T.S. Huang, “Estimating three-dimensional motion parameters of a rigid planar patch. II: Singular value decomposition,”
*IEEE Transactions on Acoustics, Speech, and Signal Processing***30**, pp. 525–533, 1982.Google Scholar - 87.R.Y. Tsai and T.S. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,”
*IEEE Transactions on Pattern Analysis and Machine Intelligence***PAMI-6**, pp. 13–27, 1984.Google Scholar - 88.S. Ullman,
*The Interpretation of Visual Motion*, Cambridge, MA: MIT Press, 1979.Google Scholar - 89.S. Ullman, “Recent computational studies in the interpretation of structure from motion,” in
*Human and Machine Vision*, J. Beck, B. Hope, and A. Rosenfeld (eds.), Orlando, FL: Academic Press, 1983, pp. 459–480.Google Scholar - 90.S. Ullman, “Maximizing rigidity: The incremental recovery of 3-D structure from rigid and rubbery motion,”
*Perception***13**, pp. 255–274, 1984.Google Scholar - 91.A.M. Waxman, “An image flow paradigm,” in
*Proceedings of the Second IEEE Workshop on Computer Vision: Representation and Control*, Annapolis, 1984, pp. 49–57.Google Scholar - 92.A.M. Waxman, and S. Ullman, “Surface structure and 3-D motion from image flow: A kinematic analysis,” Center for Automation Research, University of Maryland, CAR Tech. Report 24, October 1983.Google Scholar
- 93.A.M. Waxman, and S. Ullman, “Surface structure and three-dimensional motion from image flow kinematics,”
*International Journal of Robotics Research***4**, pp. 72–94, 1985.Google Scholar - 94.A.M. Waxman and K. Wohn, “Contour evolution, neighborhood deformation, and global image flow: Planar surfaces in motion,” Center for Automation Research, University of Maryland, CAR Tech. Report 58, April 1984.Google Scholar
- 95.A.M. Waxman and K. Wohn, “Contour evolution, neighborhood deformation, and global image flow: Planar surfaces in motion,”
*International Journal of Robotics Research***4**, pp. 95–108, 1985.Google Scholar - 96.J.A. Webb and J.K. Aggarwal, “Visually interpreting the motion of objects in space,”
*Computer***14**, pp. 40–46, 1981.Google Scholar - 97.K.W. Wong (author-editor), “Basic mathematics of photogrammetry,” in
*Manual of Photogrammetry*, 4th edn., Falls Church, VA: American Society of Photogrammetry, 1980.Google Scholar - 98.C.R. WylieJr.,
*Introduction to Projective Geometry*, New York: McGraw-Hill, 1970.Google Scholar - 99.M. Yamamoto, “Motion analysis using the visualized locus method,” untranslated Japanese articles, 1981.Google Scholar
- 100.B.L. Yen and T.S. Huang, “Determining 3-D motion and structure of a rigid body using the spherical projection,”
*Computer Graphics and Image Processing*,**21**, pp. 21–32, 1983.Google Scholar - 101.B.L. Yen and T.S. Huang, “Determining the 3-D motion and structure of a rigid body using straight line correspondences,”
*Image sequence Processing and Dynamic Scene Analysis*, T.S. Huang (ed.), Berlin: Spring-Verlag, pp. 365–394, 1983.Google Scholar