The Visual Computer

, Volume 21, Issue 8–10, pp 629–638 | Cite as

Scalable 3D video of dynamic scenes

  • Michael Waschbüsch
  • Stephan Würmlin
  • Daniel Cotting
  • Filip Sadlo
  • Markus Gross
original article


In this paper we present a scalable 3D video framework for capturing and rendering dynamic scenes. The acquisition system is based on multiple sparsely placed 3D video bricks, each comprising a projector, two grayscale cameras, and a color camera. Relying on structured light with complementary patterns, texture images and pattern-augmented views of the scene are acquired simultaneously by time-multiplexed projections and synchronized camera exposures. Using space–time stereo on the acquired pattern images, high-quality depth maps are extracted, whose corresponding surface samples are merged into a view-independent, point-based 3D data structure. This representation allows for effective photo-consistency enforcement and outlier removal, leading to a significant decrease of visual artifacts and a high resulting rendering quality using EWA volume splatting. Our framework and its view-independent representation allow for simple and straightforward editing of 3D video. In order to demonstrate its flexibility, we show compositing techniques and spatiotemporal effects.


3D video Free-viewpoint video Scene acquisition Point-based graphics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alexa, M., Gross, M., Pauly, M., Pfister, H., Stamminger, M., Zwicker, M.: Point-Based Computer Graphics. SIGGRAPH ’04 Course Notes (2004)Google Scholar
  2. 2.
    Bayakovski, Y., Levkovich-Maslyuk, L., Ignatenko, A., Konushin, A., Timasov, D., Zhirkov, A., Han, M., Park, I.K.: Depth image-based representations for static and animated 3D objects. In: ICIP ’02, 3, 25–28 (2002)Google Scholar
  3. 3.
    Bouguet, J.Y.: Camera calibration toolbox for matlab, bouguetj/calib_docGoogle Scholar
  4. 4.
    Broadhurst, A., Drummond, T., Cipolla, R.: A probabilistic framework for the space carving algorithm. In: ICCV ’01, pp. 388–393 (2001)Google Scholar
  5. 5.
    Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.: Unstructured lumigraph rendering. In: SIGGRAPH ’01, pp. 425–432 (2001)Google Scholar
  6. 6.
    Carranza, J., Theobalt, C., Magnor, M., Seidel, H.P.: Free-viewpoint video of human actors. In: SIGGRAPH ’03, pp. 569–577 (2003)Google Scholar
  7. 7.
    Cockshott, W.P., Hoff, S., Nebel, J.C.: An experimental 3D digital TV studio. In: Vision, Image & Signal Processing ’03, pp. 28–33 (2003)Google Scholar
  8. 8.
    Cotting, D., Naef, M., Gross, M., Fuchs, H.: Embedding imperceptible patterns into projected images for simultaneous acquisition and display. In: ISMAR ’04, pp. 100–109 (2004)Google Scholar
  9. 9.
    Goldlücke, B., Magnor, M., Wilburn, B.: Hardware-accelerated dynamic light field rendering. In: VMV ’02, pp. 455–462 (2002)Google Scholar
  10. 10.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: SIGGRAPH ’96, pp. 43–54 (1996)Google Scholar
  11. 11.
    Gross, M., Würmlin, S., Näf, M., Lamboray, E., Spagno, C., Kunz, A., Moere, A.V., Strehlke, K., Lang, S., Svoboda, T., Koller-Meier, E., Gool, L.V., Staadt, O.: blue-c: A spatially immersive display and 3D video portal for telepresence. In: SIGGRAPH ’03, pp. 819–827 (2003)Google Scholar
  12. 12.
    Hofsetz, C., Ng, K., Max, N., Chen, G., Liu, Y., McGuinness, P.: Image-based rendering of range data with estimated depth uncertainty. IEEE CG&A 24(4), 34–42 (2005)Google Scholar
  13. 13.
    Iddan, G.J., Yahav, G.: 3D imaging in the studio (and elsewhere...). In: SPIE ’01, 4298, 48–55 (2001)Google Scholar
  14. 14.
    Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: construction of virtual worlds from real scenes. IEEE Multimedia 4(1), 34–47 (1997)Google Scholar
  15. 15.
    Kang, S., Szeliski, R.: Boundary matting for view synthesis. In: CVPRW ’04 (2004)Google Scholar
  16. 16.
    Kang, S., Webb, J., Zitnick, L., Kanade, T.: A multi-baseline stereo system with active illumination and real-time image acquisition. In: ICCV ’95, pp. 88–93 (1995)Google Scholar
  17. 17.
    Levin, D.: Mesh-independent surface interpolation. In: Geometric Modeling for Scientific Visualization, pp. 37–49, ed. by Brunnett, Hamann, Mueller, Springer 2003Google Scholar
  18. 18.
    Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH ’96, pp. 31–42 (1996)Google Scholar
  19. 19.
    Matusik, W., Buehler, C., McMillan, L.: Polyhedral visual hulls for real-time rendering. In: EGRW ’01, pp. 115–125 (2001)Google Scholar
  20. 20.
    Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: SIGGRAPH ’00, pp. 369–374 (2000)Google Scholar
  21. 21.
    Matusik, W., Pfister, H.: 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. In: SIGGRAPH ’04 (2004)Google Scholar
  22. 22.
    Mulligan, J., Daniilidis, K.: View-independent scene acquisition for tele-presence. In: International Symposium on Augmented Reality, pp. 105–110 (2000)Google Scholar
  23. 23.
    Pauly, M., Gross, M.: Spectral processing of point sampled geometry. In: SIGGRAPH ’01(2001)Google Scholar
  24. 24.
    Pauly, M., Gross, M., Kobbelt, L.: Efficient simplification of point-sampled geometry. In: VIS ’02, pp. 163–170 (2002)Google Scholar
  25. 25.
    Pfister, H., Zwicker, M., van Baar, J., Gross, M.: Surfels: Surface elements as rendering primitives. In: SIGGRAPH ’00, pp. 335–342 (2000)Google Scholar
  26. 26.
    Redert, A., de Beeck, M.O., Fehn, C., Ijsselsteijn, W., Pollefeys, M., Gool, L.V., Ofek, E., Sexton, I., Surman, P.: ATTEST: Advanced three-dimensional television system technologies. In: 3DPVT ’02, pp. 313–319 (2002)Google Scholar
  27. 27.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)Google Scholar
  28. 28.
    Shade, J., Gortler, S., He, L.W., Szeliski, R.: Layered depth images. In: SIGGRAPH ’98, pp. 231–242 (1998)Google Scholar
  29. 29.
    Smolic, A., Kimata, H.: Description of exploration experiments in 3DAV. In: JTC1/SC29/WG11 N6194. ISO/IEC (2003)Google Scholar
  30. 30.
    Vedula, S., Baker, S., Kanade, T.: Spatio-temporal view interpolation. In: EGRW ’02, pp. 65–76 (2002)Google Scholar
  31. 31.
    Weyrich, T., Pauly, M., Keiser, R., Heinzle, S., Scandella, S., Gross, M.: Post-processing of scanned 3D surface data. In: Eurographics Symposium on Point-Based Graphics ’04 (2004)Google Scholar
  32. 32.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. In: SIGGRAPH ’05 pp. 765–776 (2005)Google Scholar
  33. 33.
    Würmlin, S., Lamboray, E., Gross, M.: 3D video fragments: Dynamic point samples for real-time free-viewpoint video. In: Computers and Graphics ’04 28(1), 3–14 (2004)Google Scholar
  34. 34.
    Würmlin, S., Lamboray, E., Staadt, O.G., Gross, M.H.: 3D video recorder. In: Proceedings of Pacific Graphics 2002, pp. 325–334. IEEE Press, New York (2002)Google Scholar
  35. 35.
    Yang, J.C., Everett, M., Buehler, C., McMillan, L.: A real-time distributed light field camera. In: EGRW ’02, pp. 77–86 (2002)Google Scholar
  36. 36.
    Zhang, L., Curless, B., Seitz, S.M.: Spacetime stereo: shape recovery for dynamic scenes. In: CVPR ’03, pp. 367–374 (2003)Google Scholar
  37. 37.
    Zhang, L., Snavely, N., Curless, B., Seitz, S.M.: Spacetime faces: high resolution capture for modeling and animation. In: SIGGRAPH ’04, pp. 548–558 (2004)Google Scholar
  38. 38.
    Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. In: SIGGRAPH ’04, pp. 600–608 (2004)Google Scholar
  39. 39.
    Zwicker, M., Pfister, H., van Baar, J., Gross, M.: EWA splatting. IEEE Trans. Visual. Comput. Graph. 8(3), 223–238 (2002)Google Scholar

Copyright information

© Springer-Verlag 2005

Authors and Affiliations

  • Michael Waschbüsch
    • 1
  • Stephan Würmlin
    • 1
  • Daniel Cotting
    • 1
  • Filip Sadlo
    • 1
  • Markus Gross
    • 1
  1. 1.Computer Graphics Laboratory, Department of Computer ScienceSwiss Federal Institute of Technology (ETH)ZurichSwitzerland

Personalised recommendations