International Journal of Computer Vision

, Volume 75, Issue 1, pp 49–65 | Cite as

Stereo for Image-Based Rendering using Image Over-Segmentation

Article

Abstract

In this paper, we propose a stereo method specifically designed for image-based rendering. For effective image-based rendering, the interpolated views need only be visually plausible. The implication is that the extracted depths do not need to be correct, as long as the recovered views appear to be correct. Our stereo algorithm relies on over-segmenting the source images. Computing match values over entire segments rather than single pixels provides robustness to noise and intensity bias. Color-based segmentation also helps to more precisely delineate object boundaries, which is important for reducing boundary artifacts in synthesized views. The depths of the segments for each image are computed using loopy belief propagation within a Markov Random Field framework. Neighboring MRFs are used for occlusion reasoning and ensuring that neighboring depth maps are consistent. We tested our stereo algorithm on several stereo pairs from the Middlebury data set, and show rendering results based on two of these data sets. We also show results for video-based rendering.

Keywords

stereo correspondence multi-view stereo segmentation image-based rendering 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Baker, S., Szeliski, R., and Anandan, P. 1998. A layered approach to stereo reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 434–441.Google Scholar
  2. Bleyer, M. and Gelautz, M. 2004. A layered stereo algorithm using image segmentation and global visibility constraints. In ICIP, pp. 2997–3000.Google Scholar
  3. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11):1222–1239.CrossRefGoogle Scholar
  4. Broadhurst, A., Drummond, T.W., and Cipolla, R. 2001. A probabilistic framework for space carving. In International Conference on Computer Vision.Google Scholar
  5. Chuang, Y.-Y., Curless, B., Salesin, D.H., and Szeliski, R. 2001. A Bayesian approach to digital matting. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 264–271.Google Scholar
  6. Collins, R.T. 1996. A space-sweep approach to true multi-Image matching. In IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, pp. 358–363.Google Scholar
  7. De Bonet, J.S. and Viola, P. 1999. Poxels: Probabilistic voxelized volume reconstruction. In International Conference on Computer Vision, pp. 418–425.Google Scholar
  8. Debevec, P., Yu, Y., and Borshukov, G. 1998. Efficient view-dependent image-based rendering with projective Texture-mapping. In Eurographics Workshop on Rendering, pp. 105–116.Google Scholar
  9. Debevec, P.E., Taylor, C.J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. Computer Graphics (SIGGRAPH), pp. 11–20.Google Scholar
  10. Gortler, S.J., Grzeszczuk, R., Szeliski, R., and Cohen, M.F. 1996. The Lumigraph. In Computer Graphics (SIGGRAPH). New Orleans, pp. 43–54.Google Scholar
  11. Han, J.H. and Park, J.S. 2000. Contour matching using epipolar geometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(4):358–370.CrossRefGoogle Scholar
  12. Hasinoff, S.W., Kang, S.B., and Szeliski, R. 2004. Boundary matting for view synthesis. In IEEE Workshop on Image and Video Registration. Washington, DC.Google Scholar
  13. Heigl, B., Koch, R., Pollefeys, M., Denzler, J., and Van Gool, L. 1999. Plenoptic modeling and rendering from image sequences taken by hand-held camera. In DAGM, pp. 94–101.Google Scholar
  14. Intille, S.S. and Bobick, A.F. 1994. Disparity-space images and large occlusion stereo. In European Conference on Computer Vision, pp. 179–186.Google Scholar
  15. Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. Computer Graphics (SIGGRAPH), pp. 297–306.Google Scholar
  16. Kanade, T. and Okutomi, M. 1994. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9):920–932.CrossRefGoogle Scholar
  17. Kanade, T., Rander, P.W., and Narayanan, P.J. 1997. Virtualized Reality: Constructing virtual worlds from real scenes. IEEE MultiMedia Magazine, 1(1):34–47.CrossRefGoogle Scholar
  18. Kang, S.B. 1999. A survey of image-based rendering techniques. In Videometrics VI (SPIE International Symposium on Electronic Imaging: Science and Technology), San Jose, CA, vol. 3641, pp. 2–16.Google Scholar
  19. Kang, S.B., Szeliski, R., and Anandan, P. 2000. The geometry-image representation tradeoff for rendering. In International Conference on Image Processing.Google Scholar
  20. Kang, S.B., Szeliski, R., and Chai, J. 2001. Handling occlusions in dense multi-view stereo. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 103–110.Google Scholar
  21. Kim, J., Kolmogorov, V., and Zabih, R. 2003. Visual correspondence using energy minimization and mutual information. In International Conference on Computer Vision, pp. 1033–1040.Google Scholar
  22. Kolmogorov, V. and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. In European Conference on Computer Vision, vol. 3, pp. 82–96.Google Scholar
  23. Levoy, M. and Hanrahan, P. 1996. Light field rendering. Computer Graphics (SIGGRAPH), pp. 31–42.Google Scholar
  24. Li, S.Z., Wang, H., Chan, K.L., and Petrou, M. 1997. Energy minimization and relaxation labeling. Journal of Mathematical Imaging and Vision, 7(2):149–161.CrossRefMathSciNetGoogle Scholar
  25. Lin, S., Li, Y., Kang, S.B., Tong, X., and Shum, H.-Y. 2002. Simultaneous separation and depth recovery of specular reflections. In European Conference on Computer Vision, Copenhagen, Denmark, vol. 3, pp. 210–224.Google Scholar
  26. Ohta, Y. and Kanade, T. 1985. Stereo by intra- and inter-scanline search using dynamic programming. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-7(2):139–154.CrossRefGoogle Scholar
  27. Okutomi, M. and Kanade, T. 1993. A multiple-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):353–363.CrossRefGoogle Scholar
  28. Okutomi, M., Katayama, Y., and Oka, S. 2002. A simple stereo algorithm to recover precise object boundaries and smooth surfaces. International Journal of Computer Vision, 47(1–3):261–273.MATHCrossRefGoogle Scholar
  29. Perona, P. and Malik, J. 1990. Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7):629–639.CrossRefGoogle Scholar
  30. Pulli, K., Cohen, M., Duchamp, T., Hoppe, H., McDonald, J., Shapiro, L., and Stuetzle, W. 1997. View-based rendering: Visualizing real objects from scanned range and color data. In Eurographics Workshop on Rendering. St. Etienne, France.Google Scholar
  31. Reche, A., Martin, I., and Drettakis, G. 2004. Volumetric reconstruction and interactive rendering of trees from photographs. In Proceedings of SIGGRAPH (ACM Transactions on Graphics), pp. 720–727.Google Scholar
  32. Scharstein, D. and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1):7–42.MATHCrossRefGoogle Scholar
  33. Schmid, C. and Zisserman, A. 1997. Automatic line matching across views. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 666–671.Google Scholar
  34. Seitz, S.M. and Dyer, C.M. 1997. Photorealistic scene reconstruction by voxel coloring. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1067–1073.Google Scholar
  35. Shade, J., Gortler, S., He, L.-W., and Szeliski, R. 1998. Layered depth images. In Computer Graphics (SIGGRAPH). Orlando, pp. 231–242.Google Scholar
  36. Shum, H.-Y., Chan, S.-C., and Kang, S.B. 2006. Image-Based Rendering. Springer.Google Scholar
  37. Sun, J., Li, Y., Kang, S.B., and Shum, H.-Y. 2005. Symmetric stereo matching for occlusion handling. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 399–406.Google Scholar
  38. Sun, J., Shum, H.-Y., and Zheng, N.-N. 2003. Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7):787–800.CrossRefGoogle Scholar
  39. Swaminathan, R., Kang, S.B., Szeliski, R., Criminisi, A., and Nayar, S.K. 2002. On the motion and appearance of specularities in image sequences. In European Conference on Computer Vision, Copenhagen, Denmark, vol. 1, pp. 508–523.Google Scholar
  40. Szeliski, R. 1999. Prediction error as a quality metric for motion and stereo. In International Conference on Computer Vision, pp. 781–788.Google Scholar
  41. Szeliski, R. and Golland, P. 1999. Stereo matching with transparency and matting. International Journal of Computer Vision, 32(1):45–61.CrossRefGoogle Scholar
  42. Tao, H., Sawhney, H.S., and Kumar, R. 2001. A global matching framework for stereo computation. In International Conference on Computer Vision, vol. 1, pp. 532–539.Google Scholar
  43. Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images. In International Conference on Computer Vision, pp. 839–846.Google Scholar
  44. Torr, P.H.S. 2002. Quadratic programming and MAX CUT to solve matching, segmentation and other Markov Random Fields. Technical Report MSR-TR-2002-68, Microsoft Research.Google Scholar
  45. Tsin, Y., Kang, S.B., and Szeliski, R. 2003. Stereo matching with reflections and translucency. In IEEE Conference on Computer Vision and Pattern Recognition, Madison, WI, vol. 1, pp. 702–709.Google Scholar
  46. Vedula, S., Baker, S., Rander, P., Collins, R., and Kanade, T. 2005. Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3):475–480.CrossRefGoogle Scholar
  47. Wang, J. and Adelson, E. 1994. Representing moving images with layers. IEEE Transactions on Image Processing, 3(5):625–638.CrossRefGoogle Scholar
  48. Wexler, Y., Fitzgibbon, A.W., and Zisserman, A. 2002. Bayesian estimation of layers from multiple images. In European Conference on Computer Vision, vol. 3, pp. 487–501.Google Scholar
  49. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. Proceedings of SIGGRAPH (ACM Transactions on Graphics), 24(3):765–776.Google Scholar
  50. Zitnick, C.L. and Kanade, T. 2000. A cooperative algorithm for stereo matching and occlusion detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):675–684.CrossRefGoogle Scholar
  51. Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. Proceedings of SIGGRAPH (ACM Transactions on Graphics), 600–608.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Microsoft ResearchOne Microsoft WayRedmondUSA

Personalised recommendations