International Journal of Computer Vision

, Volume 38, Issue 3, pp 199–218

A Theory of Shape by Space Carving

  • Kiriakos N. Kutulakos
  • Steven M. Seitz
Article

Abstract

In this paper we consider the problem of computing the 3D shape of an unknown, arbitrarily-shaped scene from multiple photographs taken at known but arbitrarily-distributed viewpoints. By studying the equivalence class of all 3D shapes that reproduce the input photographs, we prove the existence of a special member of this class, the photo hull, that (1) can be computed directly from photographs of the scene, and (2) subsumes all other members of this class. We then give a provably-correct algorithm, called Space Carving, for computing this shape and present experimental results on complex real-world scenes. The approach is designed to (1) capture photorealistic shapes that accurately model scene appearance from a wide range of viewpoints, and (2) account for the complex interactions between occlusion, parallax, shading, and their view-dependent effects on scene-appearance.

scene modeling photorealistic reconstruction multi-view stereo space carving voxel coloring shape-from-silhouettes visual hull volumetric shape representations metameric shapes 3D photography 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alfvin, R.L. and Fairchild, M.D. 1997. Observer variability in metameric color matches using color reproduction media. Color Research & Application, 22(3):174–178.Google Scholar
  2. Aloimonos, Y. 1988. Visual shape computation. Proc. IEEE, 76:899–916.Google Scholar
  3. Armstrong, M.A. 1983. Basic Topology. Springer-Verlag.Google Scholar
  4. Bascle, B. and Deriche, R. 1993. Stereo matching, reconstruction and refinement of 3D curves using deformable contours. In Proc. 4th Int. Conf. Computer Vision, pp. 421–430.Google Scholar
  5. Beardsley, P., Torr, P., and Zisserman, A. 1996. 3D model acquisition from extended image sequences. In Proc. 4th European Conf. on Computer Vision, pp. 683–695.Google Scholar
  6. Belhumeur, P.N. 1996. A bayesian approach to binocular stereopsis. Int. J. on Computer Vision, 19(3):237–260.Google Scholar
  7. Belhumeur, P.N. and Kriegman, D.J. 1996. What is the set of images of an object under all possible lighting conditions? In Proc. Computer Vision and Pattern Recognition, pp. 270–277.Google Scholar
  8. Bolles, R.C., Baker, H.H., and Marimont, D.H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. Int. J. Computer Vision, 1:7–55.Google Scholar
  9. Bolles, R.C. and Cain, R.A. 1982. Recognizing and locating partially-visible objects: The local-feature-focus method. Int. J. Robotics Research, 1(3):57–82.Google Scholar
  10. Cipolla, R. and Blake, A. 1992. Surface shape from the deformation of apparent contours. Int. J. Computer Vision, 9(2):83–112.Google Scholar
  11. Collins, R.T. 1996. A space-sweep approach to true multi-image matching. In Proc. Computer Vision and Pattern Recognition Conf., pp. 358–363.Google Scholar
  12. Cox, I., Hingorani, S., Rao, S., and Maggs, B. 1996. A maximum likelihood stereo algorithm. CVIU: Image Understanding, 63(3):542–567.Google Scholar
  13. Culbertson, W.B., Malzbender, T., and Slabaugh, G. 1999. Generalized voxel coloring. In Workshop on Vision Algorithms: Theory and Practice, Corfu, Greece.Google Scholar
  14. Curless, B. and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proc. SIGGRAPH'96, pp. 303–312.Google Scholar
  15. Debevec, P.E., Taylor, C.J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proc. SIGGRAPH'96, pp. 11–20.Google Scholar
  16. Epstein, R., Yuille, A.L., and Belhumeur, P.N. 1996. Learning object representations from lighting variations. In Object Representation in Computer Vision II, J. Ponce, A. Zisserman, and M. Hebert (Eds.). Springer-Verlag, pp. 179–199.Google Scholar
  17. Faugeras, O. 1995. Stratification of three-dimensional vision: Projective, affine, and metric representations. J. Opt. Soc. Am. A, 12(3):465–484.Google Scholar
  18. Faugeras, O.D. 1998. Personal communication.Google Scholar
  19. Faugeras, O. and Keriven, R. 1998. Complete dense stereovision using level set methods. In Proc. 5th European Conf. on Computer Vision, pp. 379–393.Google Scholar
  20. Faugeras, O.D. and Maybank, S. 1990. Motion from point matches: Multiplicity of solutions. Int. J. Computer Vision, 4:225–246.Google Scholar
  21. Foley, J.D., van Dam, A., Feiner, S.K., and Hughes, J.F. 1990. Computer Graphics Principles and Practice. Addison-Wesley Publishing Co.Google Scholar
  22. Forsyth, D. and Zisserman, A. 1991. Reflections on shading. IEEE Trans. Pattern Anal. Machine Intell., 13(7):671–679.Google Scholar
  23. Freund, J.E. 1992. Mathematical Statistics. Prentice Hall: Englewood Cliffs, NJ.Google Scholar
  24. Fua, P. and Leclerc, Y.G. 1995. Object-centered surface reconstruction: Combining multi-image stereo and shading. Int. J. Computer Vision, 16:35–56.Google Scholar
  25. Fuchs, H., Kedem, Z., and Naylor, B.F. 1980. On visible surface generation by a priori tree structures. In Proc. SIGGRAPH’ 80, pp. 39–48.Google Scholar
  26. Hoff, W. and Ahuja, N. 1989. Surfaces from stereo: Integrating feature matching, disparity estimation, and contour detection. IEEE Trans. Pattern Anal. Machine Intell., 11:121–136.Google Scholar
  27. Kakadiaris, I.A. and Metaxas, D. 1995. 3D human body model acquisition from multiple views. In Proc. Int. Conf. on Computer Vision, pp. 618–623.Google Scholar
  28. Kanade, T., Narayanan, P.J., and Rander, P.W. 1995. Virtualized reality: Concepts and early results. In Proc. Workshop on Representations of Visual Scenes, pp. 69–76.Google Scholar
  29. Kanade, T., Yoshida, A., Oda, K., Kano, H., and Tanaka, M. 1996. A stereo machine for video-rate dense depth mapping and its new applications. In Proc. Computer Vision and Pattern Recognition Conf. Google Scholar
  30. Kang, S.B. and Szeliski, R. 1996. 3-D scene data recovery using omnidirectional multibaseline stereo. In Proc. Computer Vision and Pattern Recognition Conf., pp. 364–370.Google Scholar
  31. Katayama, A., Tanaka, K., Oshino, T., and Tamura, H. 1995. A viewpoint dependent stereoscopic display using interpolation of multi-viewpoint images. In Proc. SPIE, Vol. 2409A, pp. 21–30.Google Scholar
  32. Koenderink, J.J. and van Doorn, A.J. 1991. Affine structure from motion. J. Opt. Soc. Am., A(2):377–385.Google Scholar
  33. Kutulakos, K.N. 1997. Shape from the light field boundary. In Proc. Computer Vision and Pattern Recognition, pp. 53–59.Google Scholar
  34. Kutulakos, K.N. 2000. Approximate N-view stereo. In Proc. European Conf. on Computer Vision.Google Scholar
  35. Kutulakos, K.N. and Dyer, C.R. 1994. Recovering shape by purposive viewpoint adjustment. Int. J. Computer Vision, 12(2):113–136.Google Scholar
  36. Kutulakos, K.N. and Dyer, C.R. 1995. Global surface reconstruction by purposive control of observer motion. Artificial Intelligence Journal, 78(1– 2):147–177.Google Scholar
  37. Langer, M.S. and Zucker, S.W. 1994. Shape-from-shading on a cloudy day. J. Opt. Soc. Am. A, 11(2):467–478.Google Scholar
  38. Laurentini, A. 1994. The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Machine Intell., 16(2):150–162.Google Scholar
  39. Marr, D. 1982. Vision. Freeman.Google Scholar
  40. Martin, W.N. and Aggarwal, J.K. 1983. Volumetric descriptions of objects from multiple views. IEEE Proc. Pattern Anal. Machine Intell., 5(2):150–158.Google Scholar
  41. Moezzi, S., Katkere, A., Kuramura, D.Y., and Jain, R. 1996. Reality modeling and visualization from multiple video sequences. IEEE Computer Graphics and Applications, 16(6):58–63.Google Scholar
  42. Mundy, J.L. and Zisserman, A. (Eds.). 1992. Geometric Invariance in Computer Vision. MIT Press.Google Scholar
  43. Narayanan, P.J., Rander, P.W., and Kanade, T. 1998. Constructing virtual worlds using dense stereo. In Proc. Int. Conf. on Computer Vision, pp. 3–10.Google Scholar
  44. Newell, M.E., Newell, R.G., and Sancha, T.L. 1972. A solution to the hidden surface problem. In Proc. ACM National Conference, pp. 443–450.Google Scholar
  45. Okutomi, M. and Kanade, T. 1993. A multiple-baseline stereo. IEEE Trans. Pattern Anal. Machine Intell., 15(4):353–363.Google Scholar
  46. Poggio, T., Torre, V., and Koch, C. 1985. Computational vision and regularization theory. Nature, 317(26):314–319.Google Scholar
  47. Pollefeys, M., Koch, R., and Gool, L.V. 1998. Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters. In Proc. 6th Int. Conf. on Computer Vision, pp. 90–95.Google Scholar
  48. Pritchett, P. and Zisserman, A. 1998. Wide baseline stereo matching. In Proc. 6th Int. Conf. on Computer Vision, pp. 754–760.Google Scholar
  49. Roy, S. and Cox, I.J. 1998. A maximum-flow formulation of the Ncamera stereo correspondence problem. In Proc. 6th Int. Conf. on Computer Vision, pp. 492–499.Google Scholar
  50. Sato, Y., Wheeler, M.D., and Ikeuchi, K. 1997. Object shape and reflectance modeling from observation. In Proc. SIGGRAPH'97, pp. 379–3870.Google Scholar
  51. Seales, W.B. and Faugeras, O. 1995. Building three-dimensional object models from image sequences. Computer Vision and Image Understanding, 61(3):308–324.Google Scholar
  52. Seitz, S.M. and Dyer, C.R. 1995. Complete scene structure from four point correspondences. In Proc. 5th Int. Conf. on Computer Vision, pp. 330–337.Google Scholar
  53. Seitz, S.M. and Dyer, C.R. 1999. Photorealistic scene reconstruction by voxel coloring. Int. J. Computer Vision, 35(2):151–173.Google Scholar
  54. Seitz, S.M. and Kutulakos, K.N. 1998. Plenoptic image editing. In Proc. 6th Int. Conf. Computer Vision, pp. 17–24.Google Scholar
  55. Smith, A.R. and Blinn, J.F. 1996. Blue screen matting. In Proc. SIGGRAPH'96, pp. 259–268.Google Scholar
  56. Stewart, C.V. 1995. MINPRAN: A new robust estimator for computer vision. IEEE Trans. Pattern Anal. Machine Intell., 17(10):925–938.Google Scholar
  57. Szeliski, R. 1993. Rapid octree construction from image sequences. CVGIP: Image Understanding, 58(1):23–32.Google Scholar
  58. Szeliski, R. and Golland, P. 1998. Stereo matching with transparency and matting. In Proc. 6th Int. Conf. on Computer Vision, pp. 517–524.Google Scholar
  59. Szeliski, R. and Weiss, R. 1994. Robust shape recovery from occluding contours using a linear smoother. In Real-time Computer Vision, C.M. Brown and D. Terzopoulos (Eds.). Cambridge University Press, pp. 141–165.Google Scholar
  60. Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: A factorization method. Int. J. Computer Vision, 9(2):137–154.Google Scholar
  61. Torrance, K.E. and Sparrow, E.M. 1967. Theory of off-specular reflection from roughened surface. Journal of the Optical Society of America, 57:1105–1114.Google Scholar
  62. Turk, G. and Levoy, M. 1994. Zippered polygon meshes from range images. In Proc. SIGGRAPH'94, pp. 311–318.Google Scholar
  63. Vaillant, R. and Faugeras, O.D. 1992. Using extremal boundaries for 3-D object modeling. IEEE Trans. Pattern Anal. Machine Intell., 14(2):157–173.Google Scholar
  64. van Veen, J.A.J.C. and Werkhoven, P. 1996. Metamerisms in structure-from-motion perception. Vision Research, 36(14):2197–2210.Google Scholar
  65. Woodham, R.J., Iwahori, Y., and Barman, R.A. 1991. Photometric stereo: Lambertian reflectance and light sources with unknown direction and strength. Technical Report 91-18, University of British Columbia, Laboratory for Computational Intelligence.Google Scholar
  66. Zhang, Z. 1998. Image-based geometrically-correct photorealistic scene/object modeling (IBPhM): A review. In Proc. 3rd Asian Conf. on Computer Vision, pp. 340–349.Google Scholar
  67. Zhao, C. and Mohr, R. 1996. Global three-dimensional surface reconstruction from occluding contours. ComputerVision and Image Understanding, 64(1):62–96.Google Scholar
  68. Zitnick, C.L. and Webb, J.A. 1996. Multi-baseline stereo using surface extraction. Technical Report CMU-CS-96-196, Carnegie Mellon University, Pittsburgh, PA.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Kiriakos N. Kutulakos
    • 1
  • Steven M. Seitz
    • 2
  1. 1.Department of Computer Science and Department of DermatologyUniversity of RochesterRochesterUSA
  2. 2.The Robotics InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations