Skip to main content
SpringerLink
Log in

Free-viewpoint video relighting from multi-view sequence under general illumination

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

We proposed an approach to create plausible free-viewpoint relighting video using multi-view cameras array under general illumination. Given the multi-view video dataset recorded using a set of industrial cameras under general uncontrolled and unknown illumination, we first reconstruct 3D model of the captured target using existing multi-view stereo approach. Using the coarse geometry reconstruction, we estimate the spatially varying surface reflectance in the spherical harmonics domain considering the spatial and temporal coherence. With the estimated geometry and reflectance, the 3D target is relit to the novel illumination with the environment map of the target environment. Relit performance is enhanced using a flow- and quotient-based transfer strategy to achieve detailed and plausible performance relighting. Finally, the free-viewpoint video is generated using a view-dependent rendering strategy. Experimental results on various dataset show that our approach enables plausible free-view relighting, and opens up a path towards relightable free-viewpoint video using less complex acquisition setups.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.P., Thrun, S.: Performance capture from sparse multi-view video. ACM TOG (Proc. SIGGRAPH 2008) 27, 1–10 (2008)

  2. Ballan, L., Cortelazzo, G.M.: Marker-less motion capture of skinned models in a four camera set-up using optical flow and silhouettes. In: 3DPVT (2008)

  3. Boivin, S., Gagalowicz, A.: Image-based rendering of diffuse, specular and glossy surfaces from a single image. In: Proceedings of SIGGRAPH, pp. 107–116. ACM, New York (2001)

  4. Cagniart, C., Boyer, E., Ilic, S.: Free-form mesh tracking: a patch-based approach. In: Proceedings of IEEE CVPR (2010)

  5. Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of SIGGRAPH 1998, pp. 189–198 (1998)

  6. Debevec, P.E., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of SIGGRAPH 2000, pp. 145–156 (2000)

  7. Einarsson, P., Chabert, C.F., Jones, A., Ma, W.C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., Debevec, P.: Relighting human locomotion with flowed reflectance fields. In: Proceedings of EGSR, pp. 183–194 (2006)

  8. Gall, J., Stoll, C., Aguiar, E., Theobalt, C., Rosenhahn, B., Seidel, H.P.: Motion capture using joint skeleton tracking and surface estimation. In: Proceedings of IEEE CVPR, pp. 1746–1753 (2009)

  9. Gao, Y., Tang, J., Hong, R., Yan, S., Dai, Q., Zhang, N., Chua, T.: Camera constraint-free view-based 3D object retrieval. IEEE Trans. Image Process. 21(4), 2269–2281 (2012)

    Article  MathSciNet  Google Scholar 

  10. Gao, Y., Wang, M., Tao, D., Ji, R., Dai, Q.: 3D object retrieval and recognition with hypergraph analysis. IEEE Trans. Image Process. 21(9), 4290–4303 (2012)

    Article  MathSciNet  Google Scholar 

  11. Gao, Y., Wang, M., Zha, Z., Tian, Q., Dai, Q., Zhang, N.: Less is more: efficient 3D object retrieval with query view selection. IEEE Trans. Multimedia 11(5), 1007–1018 (2011)

    Article  Google Scholar 

  12. Garces, E., Munoz, A., Lopez-Moreno, J., Gutierrez, D.: Intrinsic images by clustering. Comput. Graph. Forum (Proc. Eurograph. Symp. Render.) 31(4) (2012)

  13. Hong, R., Tang, J., Tan, H.K., Ngo, C.W., Yan, S., Chua, T.S.: Beyond search: event-driven summarization for web videos. ACM Trans. Multimedia Comput. Commun. Appl. 7(4), 35:1–35:18 (2011)

    Google Scholar 

  14. Hong, R., Wang, M., Li, G., Nie, L., Zha, Z.J., Chua, T.S.: Multimedia question answering. IEEE MultiMedia 19(4), 72–78 (2012)

    Article  Google Scholar 

  15. Lensch, H.P.A., Kautz, J., Goesele, M., Heidrich, W., Seidel, H.P.: Image-based reconstruction of spatial appearance and geometric detail. ACM TOG 22(2), 234–257 (2003)

    Article  Google Scholar 

  16. Li, G., Wu, C., Stoll, C., Liu, Y., Varanasi, K., Dai, Q., Theobalt, C.: Capturing relightable human performances under general uncontrolled illumination (2013)

  17. Liu, Y., Dai, Q., Xu, W.: A point-cloud-based multiview stereo algorithm for free-viewpoint video. IEEE Trans. Vis. Comput. Graph. 16, 407–418 (2010)

    Article  Google Scholar 

  18. Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Proceedings of SIGGRAPH 2000, pp. 369–374 (2000)

  19. Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. ACM TOG 22(3), 759–769 (2003)

    Article  Google Scholar 

  20. Matusik, W., Pfister, H., Ngan, A., Beardsley, P.A., Ziegler, R., McMillan, L.: Image-based 3d photography using opacity hulls. ACM TOG 21(3), 427–437 (2002)

    Article  Google Scholar 

  21. Peers, P., Tamura, N., Matusik, W., Debevec, P.E.: Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph. (Proc. SIGGRAPH 2007) 26(3), 52 (2007)

    Article  Google Scholar 

  22. Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of SIGGRAPH 2001, pp. 497–500 (2001)

  23. Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: Proceedings of SIGGRAPH 2001, pp. 117–128. ACM, New York (2001)

  24. Sato, Y., Wheeler, M.D., Ikeuchi, K.: Object shape and reflectance modeling from observation. In: Proceedings of SIGGRAPH 1997, pp. 379–387 (1997)

  25. Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)

    Article  Google Scholar 

  26. Theobalt, C., Ahmed, N., Lensch, H.P.A., Magnor, M.A., Seidel, H.P.: Seeing people in different light-joint shape, motion, and reflectance capture. IEEE TVCG 13(4), 663–674 (2007)

    Google Scholar 

  27. Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. ACM TOG (Proc. SIGGRAPH 2008) (2008)

  28. Wang, M., Hong, R., Li, G., Zha, Z.J., Yan, S., Chua, T.S.: Event driven web video summarization by tag localization and key-shot identification. IEEE Trans. Multimedia 14(4), 975–985 (2012)

    Article  Google Scholar 

  29. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., Debevec, P.:. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM TOG 24(3), 756–764 (2005)

    Google Scholar 

  30. Wu, C., Varanasi, K., Liu, Y., Seidel, H.P., Theobalt, C.: Shading-based dynamic shape refinement from multi-view video under general illumination. In: Proc. of IEEE ICCV (2011)

  31. Wu, C., Wilburn, B., Matsushita, Y., Theobalt, C.: High-quality shape from multi-view stereo and shading under general illumination. In: Proceedings of IEEE CVPR, pp. 969–976 (2011)

  32. Yu, Y., Debevec, P., Malik, J., Hawkins, T.: Inverse global illumination: recovering reflectance models of real scenes from photographs. In: SIGGRAPH, pp. 215–224 (1999)

  33. Yu, Y., Malik, J.: Recovering photometric properties of architectural scenes from photographs. In: Proc. SIGGRAPH, SIGGRAPH ’98, pp. 207–217. ACM, New York (1998)

  34. Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S.A.J., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM TOG 23(3), 600–608 (2004)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by The National High Technology Research and Development Program of China (863 Program) No.2013AA01A604 and the Project of NSFC (No.60932007 and No.61073072).

Author information

Authors and Affiliations

  1. Department of Automation, Tsinghua University, Beijing, China

    Guannan Li, Yebin Liu & Qionghai Dai

  2. Graduate School at Shenzhen, Tsinghua University, Shenzhen, China

    Guannan Li

Authors
  1. Guannan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yebin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qionghai Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qionghai Dai.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Li, G., Liu, Y. & Dai, Q. Free-viewpoint video relighting from multi-view sequence under general illumination. Machine Vision and Applications 25, 1737–1746 (2014). https://doi.org/10.1007/s00138-013-0559-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-013-0559-0

Keywords

Search

Navigation