Advertisement

Dense and Accurate Spatio-temporal Multi-view Stereovision

  • Jérôme Courchay
  • Jean-Philippe Pons
  • Pascal Monasse
  • Renaud Keriven
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5995)

Abstract

In this paper, we propose a novel method to simultaneously and accurately estimate the 3D shape and 3D motion of a dynamic scene from multiple-viewpoint calibrated videos. We follow a variational approach in the vein of previous work on stereo reconstruction and scene flow estimation. We adopt a representation of a dynamic scene by an animated mesh, i.e. a polygonal mesh with fixed connectivity whose time-varying vertex positions sample the trajectories of material points. Interestingly, this representation ensures a consistent coding of shape and motion by construction. Our method accurately recovers 3D shape and 3D motion by optimizing the positions of the vertices of the animated mesh. This optimization is driven by an energy function which incorporates multi-view and inter-frame photo-consistency, smoothness of the spatio-temporal surface and of the velocity field. Central to our work is an image-based photo-consistency score which can be efficiently computed and which fully handles projective distortion and partial occlusions. We demonstrate the effectiveness of our method on several challenging real-world dynamic scenes.

Keywords

Spatio-temporal stereovision Scene flow Motion capture 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aganj, E., Pons, J.P., Ségonne, F., Keriven, R.: Spatio-temporal shape from silhouette using four-dimensional Delaunay meshing. In: IEEE International Conference on Computer Vision (2007)Google Scholar
  2. 2.
    Ahmed, N., de Aguiar, E., Theobalt, C., Magnor, M., Seidel, H.P.: Automatic generation of personalized human avatars from multi-view video. In: Proc. VRST 2005, pp. 257–260 (2005)Google Scholar
  3. 3.
    Ahmed, N., Theobalt, C., Rössl, C., Thrun, S., Seidel, H.P.: Dense correspondence finding for parametrization-free animation reconstruction from video. In: IEEE Conference on Computer Vision and Pattern Recognition (2008)Google Scholar
  4. 4.
    de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.P., Thrun, S.: Performance capture from sparse multi-view video. In: ACM SIGGRAPH (2008)Google Scholar
  5. 5.
    de Aguiar, E., Theobalt, C., Stoll, C., Seidel, H.P.: Marker-less deformable mesh tracking for human shape and motion capture. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)Google Scholar
  6. 6.
    Furukawa, Y., Ponce, J.: Dense 3D motion capture from synchronized video streams. In: IEEE Conference on Computer Vision and Pattern Recognition (2008)Google Scholar
  7. 7.
    Goldlücke, B., Magnor, M.: Space-time isosurface evolution for temporally coherent 3D reconstruction. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 350–355 (2004)Google Scholar
  8. 8.
    Neumann, J., Aloimonos, Y.: Spatio-temporal stereo using multi-resolution subdivision surfaces. The International Journal of Computer Vision 47(1-3), 181–193 (2002)zbMATHCrossRefGoogle Scholar
  9. 9.
    Pons, J.P., Keriven, R., Faugeras, O.: Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. The International Journal of Computer Vision 72(2), 179–193 (2007)CrossRefGoogle Scholar
  10. 10.
    Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Computer Graphics and Applications 27(3), 21–31 (2007)CrossRefGoogle Scholar
  11. 11.
    Theobalt, C., Ahmed, N., Lensch, H., Magnor, M., Seidel, H.P.: Seeing people in different light-joint shape, motion, and reflectance capture. IEEE Transactions on Visualization and Computer Graphics 13(4), 663–674 (2007)CrossRefGoogle Scholar
  12. 12.
    Varanasi, K., Zaharescu, A., Boyer, E., Horaud, R.P.: Temporal surface tracking using mesh evolution. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 30–43. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  13. 13.
    Vedula, S., Baker, S., Kanade, T.: Image-based spatio-temporal modeling and view interpolation of dynamic events. ACM Transactions on Graphics 24(2), 240–261 (2005)CrossRefGoogle Scholar
  14. 14.
    Vedula, S., Baker, S., Seitz, S., Kanade, T.: Shape and motion carving in 6D. In: IEEE Conference on Computer Vision and Pattern Recognition (2000)Google Scholar
  15. 15.
    Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. ACM Transactions on Graphics 27(3) (2008)Google Scholar
  16. 16.
    Vedula, S., Baker, S.: Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(3), 475–480 (2005)CrossRefGoogle Scholar
  17. 17.
    Zhang, Y., Kambhamettu, C.: Integrated 3D scene flow and structure recovery from multiview image sequences. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 674–681 (2000)Google Scholar
  18. 18.
    Huguet, F., Devernay, F.: A variational method for scene flow estimation from stereo sequences. In: IEEE International Conference on Computer Vision (2007)Google Scholar
  19. 19.
    Pons, J.P., Keriven, R., Faugeras, O., Hermosillo, G.: Variational stereovision and 3D scene flow estimation with statistical similarity measures. In: IEEE International conference on Computer Vision, vol. 2, pp. 597–602Google Scholar
  20. 20.
    Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., Cremers, D.: Efficient dense scene flow from sparse or dense stereo data. In: European Conference on Computer Vision, pp. 739–751 (2008)Google Scholar
  21. 21.
    Zhang, Y., Kambhamettu, C.: On 3D scene flow and structure estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 778–785 (2001)Google Scholar
  22. 22.
    Süßmuth, J., Winter, M., Greiner, G.: Reconstructing animated meshes from time-varying point clouds. Computer Graphics Forum (Proc. Eurographics Symposium on Geometry Processing) 27(5), 1469–1476 (2008)Google Scholar
  23. 23.
    Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., Schilling, A.: Reconstruction of deforming geometry from time-varying point clouds. In: Eurographics Symposium on Geometry Processing, pp. 49–58 (2007)Google Scholar
  24. 24.
    Delaunoy, A., Prados, E., Gargallo, P., Pons, J.P., Sturm, P.: Minimizing the multi-view stereo reprojection error for triangular surface meshes. In: British Machine Vision Conference (2008)Google Scholar
  25. 25.
    Slabaugh, G.G., Unal, G.B.: Active polyhedron: Surface evolution theory applied to deformable meshes. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 84–91 (2005)Google Scholar
  26. 26.
    Vu, H.H., Keriven, R., Labatut, P., Pons, J.P.: Towards high-resolution large-scale multi-view stereo. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  27. 27.
    Pons, J.P., Boissonnat, J.D.: Delaunay deformable models: Topology-adaptive meshes based on the restricted Delaunay triangulation. In: IEEE Conference on Computer Vision and Pattern Recognition (2007)Google Scholar
  28. 28.
    Kobbelt, L., Campagna, S., Vorsatz, J., Seidel, H.P.: Interactive multi-resolution modeling on arbitrary meshes. In: International Conference on Computer Graphics and Interactive Techniques, pp. 105–114 (1998)Google Scholar
  29. 29.
    Courchay, J., Pons, J.P., Keriven, R., Monasse, P.: Dense and accurate spatio-temporal multi-view stereovision. Technical Report 09-43, IMAGINE (July 2009)Google Scholar
  30. 30.
    White, R., Crane, K., Forsyth, D.: Capturing and animating occluded cloth. In: SIGGRAPH (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Jérôme Courchay
    • 1
  • Jean-Philippe Pons
    • 2
  • Pascal Monasse
    • 1
  • Renaud Keriven
    • 1
  1. 1.IMAGINEMarne-la-ValléeFR
  2. 2.CSTBSophia AntipolisFR

Personalised recommendations