Advertisement

International Journal of Computer Vision

, Volume 72, Issue 2, pp 179–193 | Cite as

Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score

  • Jean-Philippe Pons
  • Renaud Keriven
  • Olivier Faugeras
Article

Abstract

We present a new variational method for multi-view stereovision and non-rigid three-dimensional motion estimation from multiple video sequences. Our method minimizes the prediction error of the shape and motion estimates. Both problems then translate into a generic image registration task. The latter is entrusted to a global measure of image similarity, chosen depending on imaging conditions and scene properties. Rather than integrating a matching measure computed independently at each surface point, our approach computes a global image-based matching score between the input images and the predicted images. The matching process fully handles projective distortion and partial occlusions. Neighborhood as well as global intensity information can be exploited to improve the robustness to appearance changes due to non-Lambertian materials and illumination changes, without any approximation of shape, motion or visibility. Moreover, our approach results in a simpler, more flexible, and more efficient implementation than in existing methods. The computation time on large datasets does not exceed thirty minutes on a standard workstation. Finally, our method is compliant with a hardware implementation with graphics processor units. Our stereovision algorithm yields very good results on a variety of datasets including specularities and translucency. We have successfully tested our motion estimation algorithm on a very challenging multi-view video sequence of a non-rigid scene.

Keywords

stereovision non-rigid 3D motion scene flow registration prediction error reprojection error variational method global image-based matching score cross correlation mutual information non-Lambertian surface level sets 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

Supplementary material (474 KB)

bust.mpg (544 kb)
Supplementary material (544 KB)
cactus.mpg (646 kb)
Supplementary material (646 KB)

Supplementary material (496 KB)

Supplementary material (324 KB)

Supplementary material (346 KB)

References

  1. Barron, J., Fleet, D., and Beauchemin, S. 1994. Performance of optical flow techniques. The International Journal of Computer Vision, 12(1):43–77.CrossRefGoogle Scholar
  2. Boykov, Y. and Kolmogorov, V. 2003. Computing geodesics and minimal surfaces via graph cuts. In International Conference on Computer Vision, 1:26–33.Google Scholar
  3. Broadhurst, A., Drummond, T., and Cipolla, R. 2001. A probabilistic framework for space carving. In International Conference on Computer Vision, 1:388–393.Google Scholar
  4. Carceroni, R. and Kutulakos, K. 2002. Multi-view scene capture by surfel sampling: From video streams to non-rigid 3D motion, shape and reflectance. The International Journal of Computer Vision, 49(2–3):175–214.CrossRefGoogle Scholar
  5. Caselles, V., Kimmel, R., and Sapiro, G. 1997. Geodesic active contours. The International Journal of Computer Vision, 22(1):61–79.CrossRefGoogle Scholar
  6. Duan, Y., Yang, L., Qin, H., and Samaras, D. 2004. Shape reconstruction from 3D and 2D data using PDE-based deformable surfaces. In European Conference on Computer Vision, 3:238–251.Google Scholar
  7. Faugeras, O. and Hermosillo, G. 2004. Well-posedness of two non-rigid multimodal image registration methods. Siam Journal of Applied Mathematics, 64(5):1550–1587.CrossRefMathSciNetGoogle Scholar
  8. Faugeras, O. and Keriven, R. 1998. Variational principles, surface evolution, PDE’s, level set methods and the stereo problem. IEEE Transactions on Image Processing, 7(3):336–344.CrossRefMathSciNetGoogle Scholar
  9. Goldlücke, B. and Magnor, M. 2004a. Space-time isosurface evolution for temporally coherent 3D reconstruction. In International Conference on Computer Vision and Pattern Recognition, 1:350–355.Google Scholar
  10. Goldlücke, B. and Magnor, M. 2004b. Weighted minimal hypersurfaces and their applications in computer vision. In European Conference on Computer Vision, 2:366–378.Google Scholar
  11. Hermosillo, G., Chefd’hotel, C., and Faugeras, O. 2002. Variational methods for multimodal image matching. The International Journal of Computer Vision, 50(3):329–343.CrossRefGoogle Scholar
  12. Hernández Esteban, C. and Schmitt, F. 2004. Silhouette and stereo fusion for 3D object modeling. Computer Vision and Image Understanding, 96(3):367–392.CrossRefGoogle Scholar
  13. Jin, H., Soatto, S., and Yezzi, A. 2005. Multi-view stereo reconstruction of dense shape and complex appearance. The International Journal of Computer Vision, 63(3):175–189.CrossRefGoogle Scholar
  14. Kanade, T. and Okutomi, M. 1994. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9):920–932.CrossRefGoogle Scholar
  15. Kass, M., Witkin, A., and Terzopoulos, D. 1987. Snakes: Active contour models. The International Journal of Computer Vision, 1(4):321–331.CrossRefGoogle Scholar
  16. Kolmogorov, V. and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. In European Conference on Computer Vision, 3:82–96.Google Scholar
  17. Kolmogorov, V. and Zabih, R. 2004. What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(2):147–159.CrossRefGoogle Scholar
  18. Kutulakos, K. 2000. Approximate n-View Stereo. In European Conference on Computer Vision, 1:67–83.Google Scholar
  19. Kutulakos, K. and Seitz, S. 2000. A theory of shape by space carving. The International Journal of Computer Vision, 38(3):199–218.CrossRefGoogle Scholar
  20. Lhuillier, M. and Quan, L. 2003. Surface reconstruction by integrating 3D and 2D data of multiple views. In International Conference on Computer Vision, 2:1313–1320.Google Scholar
  21. Neumann, J. and Aloimonos, Y. 2002. Spatio-temporal stereo using multi-resolution subdivision surfaces. The International Journal of Computer Vision, 47:181–193.CrossRefGoogle Scholar
  22. Osher, S. and Sethian, J. 1988. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton–Jacobi formulations. Journal of Computational Physics, 79(1):12–49.CrossRefMathSciNetGoogle Scholar
  23. Parzen, E. 1962. On estimation of a probability density function and mode. Annals Mathematical Statistics, 33:1065–1076.MathSciNetGoogle Scholar
  24. Roche, A., Malandain, G., Pennec, X., and Ayache, N. 1998. The correlation ratio as new similarity metric for multimodal image registration. In W. Wells, A. Colchester, and S. Delp (Eds.), Medical Image Computing and Computer-Assisted Intervention-MICCAI’98. Cambridge, MA, USA, pp. 1115–1124.Google Scholar
  25. Scharstein, D. and Szeliski, R. 1998. Stereo matching with nonlinear diffusion. The International Journal of Computer Vision, 28(2):155–174.CrossRefGoogle Scholar
  26. Scharstein, D. and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. The International Journal of Computer Vision, 47(1):7–42.CrossRefGoogle Scholar
  27. Segal, M., Korobkin, C., van Widenfelt, R., Foran, J., and Haeberli P. 1992. Fast shadows and lighting effects using texture mapping. Computer Graphics, 26(2):249–252.CrossRefGoogle Scholar
  28. Seitz, S. and Dyer, C. 1999. Photorealistic scene reconstruction by voxel coloring. The International Journal of Computer Vision, 35(2):151–173.CrossRefGoogle Scholar
  29. Soatto, S., Yezzi, A., and Jin, H. 2003. Tales of shape and radiance in multi-view stereo. In International Conference on Computer Vision, 2:974–981.Google Scholar
  30. Solem, J. and Overgaard, N. 2005. A geometric formulation of gradient descent for variational problems with moving surfaces. In International Conference on Scale Space and PDE Methods in Computer Vision, pp. 419–430.Google Scholar
  31. Strecha, C., Fransens, R., and Van Gool, L. 2004. Wide-baseline stereo from multiple views: A probabilistic account. In International Conference on Computer Vision and Pattern Recognition, 2:552–559.Google Scholar
  32. Strecha, C., Tuytelaars, T., and Van Gool, L. 2003. Dense matching of multiple wide-baseline views. In International Conference on Computer Vision, 2:1194–1201.Google Scholar
  33. Szeliski, R. 1999. Prediction error as a quality metric for motion and stereo. In International Conference on Computer Vision, 2:781–788.Google Scholar
  34. Treuille, A., Hertzmann, A., and Seitz, S. 2004. Example-based stereo with general brdfs. In European Conference on Computer Vision, 2:457–469.Google Scholar
  35. Vedula, S., Baker, S., and Kanade, T. 2002. Spatio-temporal view interpolation. In ACM Eurographics Workshop on Rendering, pp. 65–76.Google Scholar
  36. Vedula, S., Baker, S., Rander, P., Collins, R., and Kanade, T. 2005. Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3):475–480.CrossRefGoogle Scholar
  37. Viola, P. and Wells III, W. M. 1997. Alignement by maximization of mutual information. The International Journal of Computer Vision, 24(2):137–154.CrossRefGoogle Scholar
  38. Yang, R., Pollefeys, M., and Welch, G. 2003. Dealing with textureless regions and specular highlights: A progressive space carving scheme using a novel photo-consistency measure. In International Conference on Computer Vision, 1:576–584.Google Scholar
  39. Zhang, Y. and Kambhamettu, C. 2001. On 3D scene flow and structure estimation. In International Conference on Computer Vision and Pattern Recognition, 2:778–785.Google Scholar

Copyright information

© Springer Science + Business Media, LLC 2006

Authors and Affiliations

  • Jean-Philippe Pons
    • 1
  • Renaud Keriven
    • 1
  • Olivier Faugeras
    • 2
  1. 1.Odyssée Laboratory, ENPCMarne-la-ValléeFrance
  2. 2.Odyssée Laboratory, INRIASophia-AntipolisFrance

Personalised recommendations