Dense Structure-from-Motion: An Approach Based on Segment Matching

  • Fabian Ernst
  • Piotr Wilinski
  • Kees van Overveld
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2351)


For 3-D video applications, dense depth maps are required. We present a segment-based structure-from-motion technique. After image segmentation, we estimate the motion of each segment. With knowledge of the camera motion, this can be translated into depth. The optimal depth is found by minimizing a suitable error norm, which can handle occlusions as well. This method combines the advantages of motion estimation on the one hand, and structure-from-motion algorithms on the other hand. The resulting depth maps are pixel-accurate due to the segmentation, and have a high accuracy: depth differences corresponding to motion differences of 1/8th of a pixel can be recovered.


Motion Vector Camera Motion Camera Calibration Error Curve Relaxation Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    M. Accame, F.G.B. De Natale, and D. Giusto. Hierarchical block matching for disparity estimation in stereo sequences. In ICIP95, pages 374–377, 1995.Google Scholar
  2. 2.
    G. Adiv. Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE Trans. PAMI, 7:384–401, 1985.Google Scholar
  3. 3.
    S.T. Barnard and W.B. Thompson. Disparity analysis of images. IEEE Trans. PAMI, 2:333–340, 1980.Google Scholar
  4. 4.
    H.A. Beyer. Some aspects of the geometric calibration of CCD cameras. In ISPRS Intercomm. Conf. on Fast Processing of Photogrammetic Data, Interlaken, 1987.Google Scholar
  5. 5.
    G. de Haan and P. Biezen. Sub-pixel motion estimation with 3D recursive search block matching. Signal Processing: Image Communication, 6:229–239, 1994.CrossRefGoogle Scholar
  6. 6.
    R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge University Press, 2000.Google Scholar
  7. 7.
    MPEG-4 Video group ISO WG11. MPEG-4 overview (Maui Version). Technical Report ISO/IEC/JTC1/SC29/WG11 N3156, ISO, 1999.Google Scholar
  8. 8.
    J.R. Jain and A.K. Jain. Displacement measurement and its application in interframe image coding. IEEE Trans. Comm., 29:1799–1808, 1981.CrossRefGoogle Scholar
  9. 9.
    Tony Jebara, Ali Azarbayejani, and Alex Pentland. 3D structure from 2D motion. IEEE Signal Processing Magazine, pages 66–84, May 1999.Google Scholar
  10. 10.
    J.L. Mallet. Discrete smooth interpolation in geometric modelling. Computer Aided Design, 24:178–191, 1992.zbMATHCrossRefGoogle Scholar
  11. 11.
    M. Pollefeys, R. Koch, M. Vergauwen, and L. Van Gool. Metric 3D surface reconstruction from uncalibrated image sequences. In Proc. SMILE Workshop (post-ECCV’98), LNCS 1506, pages 138–153. Springer-Verlag, 1998.Google Scholar
  12. 12.
    P.A. Redert, E.A. Hendriks, and J. Biemond. Correspondence estimation in image pairs. IEEE Signal Processing Magazine, 16:29–46, 1999.CrossRefGoogle Scholar
  13. 13.
    R. Rodrigues, K. van Overveld, and P. Wilinski. Depth reconstruction based on irregular patches. In Proc. EPCG no 9, Marinha Grande, Portugal, 1999.Google Scholar
  14. 14.
    P. Salembier and F. Marques. Region-based representations of image and video: segmentation for multimedia services. IEEE Trans. CSVT, 9:1147–1169, 1999.Google Scholar
  15. 15.
    S. Soatto and P. Perona. Reducing “Structure from Motion”: A general framework for dynamic vision. part 1: Modeling. IEEE Trans. PAMI, 20:933–942, 1998.Google Scholar
  16. 16.
    H. Tao, H.S. Sawhney, and R. Kumar. A global matching framework for stereo computation. In Proc. ICCV, pages 532–539, Vancouver, Canada, 2001.Google Scholar
  17. 17.
    D. Terzopoulos. The computation of visible-surface representations. IEEE Trans. PAMI, 10:417–438, 1988.zbMATHGoogle Scholar
  18. 18.
    R.Y. Tsai. A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV camera lenses. IEEE Journal on Robotics and Automation, RA-3:323–344, 1987.CrossRefGoogle Scholar
  19. 19.
    C.W.A.M. van Overveld. The application of relaxation and optimisation methods in computer aided geometric design. In B. Özgüc and V. Akman, editors, Proc. of First Bilent Comp. Graphics Conf., pages 161–180, Ankara, Turkey, 1993.Google Scholar
  20. 20.
    L. Vincent and P. Soille. Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Trans. PAMI, 13:583–598, 1991.Google Scholar
  21. 21.
    D. Wang. Unsupervised video segmentation based on watersheds and temporal tracking. IEEE Trans. CSVT, 8:539–546, 1998.Google Scholar
  22. 22.
    A. Yezzi and S. Soatto. Stereoscopic segmentation. In Proc. ICCV, pages 59–66, Vancouver, Canada, 2001.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Fabian Ernst
    • 1
  • Piotr Wilinski
    • 1
  • Kees van Overveld
  1. 1.Philips ResearchEindhovenThe Netherlands

Personalised recommendations