Advertisement

The Visual Computer

, Volume 21, Issue 8–10, pp 629–638 | Cite as

Scalable 3D video of dynamic scenes

  • Michael Waschbüsch
  • Stephan Würmlin
  • Daniel Cotting
  • Filip Sadlo
  • Markus Gross
original article

Abstract

In this paper we present a scalable 3D video framework for capturing and rendering dynamic scenes. The acquisition system is based on multiple sparsely placed 3D video bricks, each comprising a projector, two grayscale cameras, and a color camera. Relying on structured light with complementary patterns, texture images and pattern-augmented views of the scene are acquired simultaneously by time-multiplexed projections and synchronized camera exposures. Using space–time stereo on the acquired pattern images, high-quality depth maps are extracted, whose corresponding surface samples are merged into a view-independent, point-based 3D data structure. This representation allows for effective photo-consistency enforcement and outlier removal, leading to a significant decrease of visual artifacts and a high resulting rendering quality using EWA volume splatting. Our framework and its view-independent representation allow for simple and straightforward editing of 3D video. In order to demonstrate its flexibility, we show compositing techniques and spatiotemporal effects.

Keywords

3D video Free-viewpoint video Scene acquisition Point-based graphics 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Alexa, M., Gross, M., Pauly, M., Pfister, H., Stamminger, M., Zwicker, M.: Point-Based Computer Graphics. SIGGRAPH ’04 Course Notes (2004)Google Scholar
  2. 2.
    Bayakovski, Y., Levkovich-Maslyuk, L., Ignatenko, A., Konushin, A., Timasov, D., Zhirkov, A., Han, M., Park, I.K.: Depth image-based representations for static and animated 3D objects. In: ICIP ’02, 3, 25–28 (2002)Google Scholar
  3. 3.
    Bouguet, J.Y.: Camera calibration toolbox for matlab, http://www.vision.caltech.edu/ bouguetj/calib_docGoogle Scholar
  4. 4.
    Broadhurst, A., Drummond, T., Cipolla, R.: A probabilistic framework for the space carving algorithm. In: ICCV ’01, pp. 388–393 (2001)Google Scholar
  5. 5.
    Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.: Unstructured lumigraph rendering. In: SIGGRAPH ’01, pp. 425–432 (2001)Google Scholar
  6. 6.
    Carranza, J., Theobalt, C., Magnor, M., Seidel, H.P.: Free-viewpoint video of human actors. In: SIGGRAPH ’03, pp. 569–577 (2003)Google Scholar
  7. 7.
    Cockshott, W.P., Hoff, S., Nebel, J.C.: An experimental 3D digital TV studio. In: Vision, Image & Signal Processing ’03, pp. 28–33 (2003)Google Scholar
  8. 8.
    Cotting, D., Naef, M., Gross, M., Fuchs, H.: Embedding imperceptible patterns into projected images for simultaneous acquisition and display. In: ISMAR ’04, pp. 100–109 (2004)Google Scholar
  9. 9.
    Goldlücke, B., Magnor, M., Wilburn, B.: Hardware-accelerated dynamic light field rendering. In: VMV ’02, pp. 455–462 (2002)Google Scholar
  10. 10.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: SIGGRAPH ’96, pp. 43–54 (1996)Google Scholar
  11. 11.
    Gross, M., Würmlin, S., Näf, M., Lamboray, E., Spagno, C., Kunz, A., Moere, A.V., Strehlke, K., Lang, S., Svoboda, T., Koller-Meier, E., Gool, L.V., Staadt, O.: blue-c: A spatially immersive display and 3D video portal for telepresence. In: SIGGRAPH ’03, pp. 819–827 (2003)Google Scholar
  12. 12.
    Hofsetz, C., Ng, K., Max, N., Chen, G., Liu, Y., McGuinness, P.: Image-based rendering of range data with estimated depth uncertainty. IEEE CG&A 24(4), 34–42 (2005)Google Scholar
  13. 13.
    Iddan, G.J., Yahav, G.: 3D imaging in the studio (and elsewhere...). In: SPIE ’01, 4298, 48–55 (2001)Google Scholar
  14. 14.
    Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: construction of virtual worlds from real scenes. IEEE Multimedia 4(1), 34–47 (1997)Google Scholar
  15. 15.
    Kang, S., Szeliski, R.: Boundary matting for view synthesis. In: CVPRW ’04 (2004)Google Scholar
  16. 16.
    Kang, S., Webb, J., Zitnick, L., Kanade, T.: A multi-baseline stereo system with active illumination and real-time image acquisition. In: ICCV ’95, pp. 88–93 (1995)Google Scholar
  17. 17.
    Levin, D.: Mesh-independent surface interpolation. In: Geometric Modeling for Scientific Visualization, pp. 37–49, ed. by Brunnett, Hamann, Mueller, Springer 2003Google Scholar
  18. 18.
    Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH ’96, pp. 31–42 (1996)Google Scholar
  19. 19.
    Matusik, W., Buehler, C., McMillan, L.: Polyhedral visual hulls for real-time rendering. In: EGRW ’01, pp. 115–125 (2001)Google Scholar
  20. 20.
    Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: SIGGRAPH ’00, pp. 369–374 (2000)Google Scholar
  21. 21.
    Matusik, W., Pfister, H.: 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. In: SIGGRAPH ’04 (2004)Google Scholar
  22. 22.
    Mulligan, J., Daniilidis, K.: View-independent scene acquisition for tele-presence. In: International Symposium on Augmented Reality, pp. 105–110 (2000)Google Scholar
  23. 23.
    Pauly, M., Gross, M.: Spectral processing of point sampled geometry. In: SIGGRAPH ’01(2001)Google Scholar
  24. 24.
    Pauly, M., Gross, M., Kobbelt, L.: Efficient simplification of point-sampled geometry. In: VIS ’02, pp. 163–170 (2002)Google Scholar
  25. 25.
    Pfister, H., Zwicker, M., van Baar, J., Gross, M.: Surfels: Surface elements as rendering primitives. In: SIGGRAPH ’00, pp. 335–342 (2000)Google Scholar
  26. 26.
    Redert, A., de Beeck, M.O., Fehn, C., Ijsselsteijn, W., Pollefeys, M., Gool, L.V., Ofek, E., Sexton, I., Surman, P.: ATTEST: Advanced three-dimensional television system technologies. In: 3DPVT ’02, pp. 313–319 (2002)Google Scholar
  27. 27.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)Google Scholar
  28. 28.
    Shade, J., Gortler, S., He, L.W., Szeliski, R.: Layered depth images. In: SIGGRAPH ’98, pp. 231–242 (1998)Google Scholar
  29. 29.
    Smolic, A., Kimata, H.: Description of exploration experiments in 3DAV. In: JTC1/SC29/WG11 N6194. ISO/IEC (2003)Google Scholar
  30. 30.
    Vedula, S., Baker, S., Kanade, T.: Spatio-temporal view interpolation. In: EGRW ’02, pp. 65–76 (2002)Google Scholar
  31. 31.
    Weyrich, T., Pauly, M., Keiser, R., Heinzle, S., Scandella, S., Gross, M.: Post-processing of scanned 3D surface data. In: Eurographics Symposium on Point-Based Graphics ’04 (2004)Google Scholar
  32. 32.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. In: SIGGRAPH ’05 pp. 765–776 (2005)Google Scholar
  33. 33.
    Würmlin, S., Lamboray, E., Gross, M.: 3D video fragments: Dynamic point samples for real-time free-viewpoint video. In: Computers and Graphics ’04 28(1), 3–14 (2004)Google Scholar
  34. 34.
    Würmlin, S., Lamboray, E., Staadt, O.G., Gross, M.H.: 3D video recorder. In: Proceedings of Pacific Graphics 2002, pp. 325–334. IEEE Press, New York (2002)Google Scholar
  35. 35.
    Yang, J.C., Everett, M., Buehler, C., McMillan, L.: A real-time distributed light field camera. In: EGRW ’02, pp. 77–86 (2002)Google Scholar
  36. 36.
    Zhang, L., Curless, B., Seitz, S.M.: Spacetime stereo: shape recovery for dynamic scenes. In: CVPR ’03, pp. 367–374 (2003)Google Scholar
  37. 37.
    Zhang, L., Snavely, N., Curless, B., Seitz, S.M.: Spacetime faces: high resolution capture for modeling and animation. In: SIGGRAPH ’04, pp. 548–558 (2004)Google Scholar
  38. 38.
    Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. In: SIGGRAPH ’04, pp. 600–608 (2004)Google Scholar
  39. 39.
    Zwicker, M., Pfister, H., van Baar, J., Gross, M.: EWA splatting. IEEE Trans. Visual. Comput. Graph. 8(3), 223–238 (2002)Google Scholar

Copyright information

© Springer-Verlag 2005

Authors and Affiliations

  • Michael Waschbüsch
    • 1
  • Stephan Würmlin
    • 1
  • Daniel Cotting
    • 1
  • Filip Sadlo
    • 1
  • Markus Gross
    • 1
  1. 1.Computer Graphics Laboratory, Department of Computer ScienceSwiss Federal Institute of Technology (ETH)ZurichSwitzerland

Personalised recommendations