Skip to main content
SpringerLink
Log in

Stereo for Image-Based Rendering using Image Over-Segmentation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper, we propose a stereo method specifically designed for image-based rendering. For effective image-based rendering, the interpolated views need only be visually plausible. The implication is that the extracted depths do not need to be correct, as long as the recovered views appear to be correct. Our stereo algorithm relies on over-segmenting the source images. Computing match values over entire segments rather than single pixels provides robustness to noise and intensity bias. Color-based segmentation also helps to more precisely delineate object boundaries, which is important for reducing boundary artifacts in synthesized views. The depths of the segments for each image are computed using loopy belief propagation within a Markov Random Field framework. Neighboring MRFs are used for occlusion reasoning and ensuring that neighboring depth maps are consistent. We tested our stereo algorithm on several stereo pairs from the Middlebury data set, and show rendering results based on two of these data sets. We also show results for video-based rendering.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Baker, S., Szeliski, R., and Anandan, P. 1998. A layered approach to stereo reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 434–441.

  • Bleyer, M. and Gelautz, M. 2004. A layered stereo algorithm using image segmentation and global visibility constraints. In ICIP, pp. 2997–3000.

  • Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11):1222–1239.

    Article  Google Scholar 

  • Broadhurst, A., Drummond, T.W., and Cipolla, R. 2001. A probabilistic framework for space carving. In International Conference on Computer Vision.

  • Chuang, Y.-Y., Curless, B., Salesin, D.H., and Szeliski, R. 2001. A Bayesian approach to digital matting. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 264–271.

  • Collins, R.T. 1996. A space-sweep approach to true multi-Image matching. In IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, pp. 358–363.

  • De Bonet, J.S. and Viola, P. 1999. Poxels: Probabilistic voxelized volume reconstruction. In International Conference on Computer Vision, pp. 418–425.

  • Debevec, P., Yu, Y., and Borshukov, G. 1998. Efficient view-dependent image-based rendering with projective Texture-mapping. In Eurographics Workshop on Rendering, pp. 105–116.

  • Debevec, P.E., Taylor, C.J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. Computer Graphics (SIGGRAPH), pp. 11–20.

  • Gortler, S.J., Grzeszczuk, R., Szeliski, R., and Cohen, M.F. 1996. The Lumigraph. In Computer Graphics (SIGGRAPH). New Orleans, pp. 43–54.

  • Han, J.H. and Park, J.S. 2000. Contour matching using epipolar geometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(4):358–370.

    Article  Google Scholar 

  • Hasinoff, S.W., Kang, S.B., and Szeliski, R. 2004. Boundary matting for view synthesis. In IEEE Workshop on Image and Video Registration. Washington, DC.

  • Heigl, B., Koch, R., Pollefeys, M., Denzler, J., and Van Gool, L. 1999. Plenoptic modeling and rendering from image sequences taken by hand-held camera. In DAGM, pp. 94–101.

  • Intille, S.S. and Bobick, A.F. 1994. Disparity-space images and large occlusion stereo. In European Conference on Computer Vision, pp. 179–186.

  • Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. Computer Graphics (SIGGRAPH), pp. 297–306.

  • Kanade, T. and Okutomi, M. 1994. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9):920–932.

    Article  Google Scholar 

  • Kanade, T., Rander, P.W., and Narayanan, P.J. 1997. Virtualized Reality: Constructing virtual worlds from real scenes. IEEE MultiMedia Magazine, 1(1):34–47.

    Article  Google Scholar 

  • Kang, S.B. 1999. A survey of image-based rendering techniques. In Videometrics VI (SPIE International Symposium on Electronic Imaging: Science and Technology), San Jose, CA, vol. 3641, pp. 2–16.

  • Kang, S.B., Szeliski, R., and Anandan, P. 2000. The geometry-image representation tradeoff for rendering. In International Conference on Image Processing.

  • Kang, S.B., Szeliski, R., and Chai, J. 2001. Handling occlusions in dense multi-view stereo. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 103–110.

  • Kim, J., Kolmogorov, V., and Zabih, R. 2003. Visual correspondence using energy minimization and mutual information. In International Conference on Computer Vision, pp. 1033–1040.

  • Kolmogorov, V. and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. In European Conference on Computer Vision, vol. 3, pp. 82–96.

  • Levoy, M. and Hanrahan, P. 1996. Light field rendering. Computer Graphics (SIGGRAPH), pp. 31–42.

  • Li, S.Z., Wang, H., Chan, K.L., and Petrou, M. 1997. Energy minimization and relaxation labeling. Journal of Mathematical Imaging and Vision, 7(2):149–161.

    Article  MathSciNet  Google Scholar 

  • Lin, S., Li, Y., Kang, S.B., Tong, X., and Shum, H.-Y. 2002. Simultaneous separation and depth recovery of specular reflections. In European Conference on Computer Vision, Copenhagen, Denmark, vol. 3, pp. 210–224.

  • Ohta, Y. and Kanade, T. 1985. Stereo by intra- and inter-scanline search using dynamic programming. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-7(2):139–154.

    Article  Google Scholar 

  • Okutomi, M. and Kanade, T. 1993. A multiple-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):353–363.

    Article  Google Scholar 

  • Okutomi, M., Katayama, Y., and Oka, S. 2002. A simple stereo algorithm to recover precise object boundaries and smooth surfaces. International Journal of Computer Vision, 47(1–3):261–273.

    Article  MATH  Google Scholar 

  • Perona, P. and Malik, J. 1990. Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7):629–639.

    Article  Google Scholar 

  • Pulli, K., Cohen, M., Duchamp, T., Hoppe, H., McDonald, J., Shapiro, L., and Stuetzle, W. 1997. View-based rendering: Visualizing real objects from scanned range and color data. In Eurographics Workshop on Rendering. St. Etienne, France.

  • Reche, A., Martin, I., and Drettakis, G. 2004. Volumetric reconstruction and interactive rendering of trees from photographs. In Proceedings of SIGGRAPH (ACM Transactions on Graphics), pp. 720–727.

  • Scharstein, D. and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1):7–42.

    Article  MATH  Google Scholar 

  • Schmid, C. and Zisserman, A. 1997. Automatic line matching across views. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 666–671.

  • Seitz, S.M. and Dyer, C.M. 1997. Photorealistic scene reconstruction by voxel coloring. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1067–1073.

  • Shade, J., Gortler, S., He, L.-W., and Szeliski, R. 1998. Layered depth images. In Computer Graphics (SIGGRAPH). Orlando, pp. 231–242.

  • Shum, H.-Y., Chan, S.-C., and Kang, S.B. 2006. Image-Based Rendering. Springer.

  • Sun, J., Li, Y., Kang, S.B., and Shum, H.-Y. 2005. Symmetric stereo matching for occlusion handling. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 399–406.

  • Sun, J., Shum, H.-Y., and Zheng, N.-N. 2003. Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7):787–800.

    Article  Google Scholar 

  • Swaminathan, R., Kang, S.B., Szeliski, R., Criminisi, A., and Nayar, S.K. 2002. On the motion and appearance of specularities in image sequences. In European Conference on Computer Vision, Copenhagen, Denmark, vol. 1, pp. 508–523.

  • Szeliski, R. 1999. Prediction error as a quality metric for motion and stereo. In International Conference on Computer Vision, pp. 781–788.

  • Szeliski, R. and Golland, P. 1999. Stereo matching with transparency and matting. International Journal of Computer Vision, 32(1):45–61.

    Article  Google Scholar 

  • Tao, H., Sawhney, H.S., and Kumar, R. 2001. A global matching framework for stereo computation. In International Conference on Computer Vision, vol. 1, pp. 532–539.

  • Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images. In International Conference on Computer Vision, pp. 839–846.

  • Torr, P.H.S. 2002. Quadratic programming and MAX CUT to solve matching, segmentation and other Markov Random Fields. Technical Report MSR-TR-2002-68, Microsoft Research.

  • Tsin, Y., Kang, S.B., and Szeliski, R. 2003. Stereo matching with reflections and translucency. In IEEE Conference on Computer Vision and Pattern Recognition, Madison, WI, vol. 1, pp. 702–709.

  • Vedula, S., Baker, S., Rander, P., Collins, R., and Kanade, T. 2005. Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3):475–480.

    Article  Google Scholar 

  • Wang, J. and Adelson, E. 1994. Representing moving images with layers. IEEE Transactions on Image Processing, 3(5):625–638.

    Article  Google Scholar 

  • Wexler, Y., Fitzgibbon, A.W., and Zisserman, A. 2002. Bayesian estimation of layers from multiple images. In European Conference on Computer Vision, vol. 3, pp. 487–501.

  • Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. Proceedings of SIGGRAPH (ACM Transactions on Graphics), 24(3):765–776.

    Google Scholar 

  • Zitnick, C.L. and Kanade, T. 2000. A cooperative algorithm for stereo matching and occlusion detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):675–684.

    Article  Google Scholar 

  • Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. Proceedings of SIGGRAPH (ACM Transactions on Graphics), 600–608.

Download references

Author information

Authors and Affiliations

  1. Microsoft Research, One Microsoft Way, Redmond, WA, USA, 98052

    C. Lawrence Zitnick & Sing Bing Kang

Authors
  1. C. Lawrence Zitnick
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sing Bing Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. Lawrence Zitnick.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zitnick, C.L., Kang, S.B. Stereo for Image-Based Rendering using Image Over-Segmentation. Int J Comput Vis 75, 49–65 (2007). https://doi.org/10.1007/s11263-006-0018-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-006-0018-8

Keywords

Search

Navigation