Skip to main content
SpringerLink
Log in

An Approximate Shading Model with Detail Decomposition for Object Relighting

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present an object relighting system that allows an artist to select an object from an image and insert it into a target scene. Through simple interactions, the system can adjust illumination on the inserted object so that it appears naturally in the scene. To support image-based relighting, we build object model from the image, and propose a perceptually-inspired approximate shading model for the relighting. It decomposes the shading field into (a) a rough shape term that can be reshaded, (b) a parametric shading detail that encodes missing features from the first term, and (c) a geometric detail term that captures fine-scale material properties. With this decomposition, the shading model combines 3D rendering and image-based composition and allows more flexible compositing than image-based methods. Quantitative evaluation and a set of user studies suggest our method is a promising alternative to existing methods of object insertion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., et al. (2004). Interactive digital photomontage. ACM Transactions on Graphics, 23(3), 294–302.

    Article  Google Scholar 

  • Arsalan Soltani, A., Huang, H., Wu, J., Kulkarni, T. D., & Tenenbaum, J. B. (2017). Synthesizing 3D shapes via modeling multi-view depth maps and silhouettes with deep generative networks. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Barron, J. T., & Malik, J. (2012). Color constancy, intrinsic images, and shape estimation. In ECCV.

  • Basri, R., & Jacobs, D. (2003). Lambertian reflectance and linear subspaces. In PAMI.

  • Beck, J., & Prazdny, S. (1981). Highlights and the perception of glossiness. Attention, Perception and Psychophysics, 30, 407–410.

    Article  Google Scholar 

  • Berzhanskaya, J., Swaminathan, G., Beck, J., & Mingolla, E. (2005). Remote effects of highlights on gloss perception. Perception, 34, 565–575.

    Article  Google Scholar 

  • Burt, P. J., & Adelson, E. H. (1983). A multiresolution spline with application to image mosaics. ACM Transactions on Graphics, 2(4), 217–236.

    Article  Google Scholar 

  • Cavanagh, P. (2005). The artist as neuroscientist. Nature, 434, 301–307.

    Article  Google Scholar 

  • Chen, T., Cheng, M.-M., Tan, P., Shamir, A., & Hu, S.-M. (2009). Sketch2photo: internet image montage. ACM Transactions on Graphics, 28(5), 124:1–124:10.

    Google Scholar 

  • Debevec, P. (1998). Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH’98 (pp. 189–198). ACM.

  • Deshpande, A., Lu, J., Yeh, M.-C., Jin Chong, M., & Forsyth, D. (2017). Learning diverse image colorization. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Durou, J.-D., Falcone, M., & Sagona, M. (2008). Numerical methods for shape-from-shading: A new survey with benchmarks. Computer Vision and Image Understanding, 109(1), 22–43.

    Article  Google Scholar 

  • Esteban, C. H., Vogiatzis, G., & Cipolla, R. (2008). Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(3), 548–554.

    Article  Google Scholar 

  • Furukawa, Y., & Hernandez, C. (2015). Multi-vew Stereo: A tutorial. Foundations and trends?. In Computer graphics and vision.

  • Fyffe, G., Jones, A., Alexander, O., Ichikari, R., Graham, P., Nagano, K., Busch, J., & Debevec P. (2013). Driving high-resolution facial blendshapes with video performance capture. In ACM SIGGRAPH 2013 Talks, SIGGRAPH ’13 (p. 33:1), New York, NY, USA, 2013. ACM.

  • Fyffe, G., Nagano, K., Huynh, L., Saito, S., Busch, J., Jones, A., Li, H., & Debevec, P. (2017). Multi-view stereo on consistent face topology. In Computer Graphics Forum.

  • Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., & Debevec, P. (2011). Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics, 30(6), 129:1–129:10.

    Article  Google Scholar 

  • Grosse, R., Johnson, M. K., Adelson, E. H., & Freeman, W. T. (2009). Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV (pp. 2335–2342).

  • Hartley, R. I., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press. (ISBN: 0521540518).

    Book  MATH  Google Scholar 

  • Johnston, S. F. (2002). Lumo: Illumination for cel animation. In NPAR ’02.

  • Karsch, K., Hedau, V., Forsyth, D., & Hoiem, D. (2011). Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics (SIGGRAPH Asia), 30(6), 157:1–157:12.

    Google Scholar 

  • Karsch, K., Liu, C., Kang, S. B., & England, N. (2012). Depth extraction from video using non-parametric sampling. In ECCV.

  • Karsch, K., Sunkavalli, K., Hadap, S., Carr, N., Jin, H., Fonte, R., et al. (2014). Automatic scene inference for 3D object compositing. ACM Transactions on Graphics, 33(3), 32:1–32:15.

    Article  MATH  Google Scholar 

  • Kim, S., Park, K., Sohn, K., & Lin, S. (2016). Unified depth prediction and intrinsic image decomposition from a single image via joint convolutional neural fields. In European conference on computer vision.

  • Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3), 3.

    Article  Google Scholar 

  • Lettry, L., Vanhoey, K., & Van Gool L. (2016). Darn: a deep adversial residual network for intrinsic image decomposition. arXiv:1612.07899.

  • Liao, Z., Karsch, K., & Forsyth, D. (2015). An approximate shading model for object relighting. In CVPR.

  • Liao, Z., Rock, J., Wang, Y., & Forsyth, D. (2013). Non-parametric filtering for geometric detail extraction and material representation. In CVPR (pp. 963–970).

  • Narihira, T., Maire, M., & Yu, S. X. (2015). Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In Proceedings of the IEEE international conference on computer vision (pp. 2992–2992).

  • Niebner, M., Keinert, B., Fisher, M., Stamminger, M., Loop, C., & Schäfer, H. (2016). Real-time rendering techniques with hardware tessellation. Computer Graphics Forum, 35(1), 113–137.

    Article  Google Scholar 

  • Ostrovsky, Y., Cavanagh, P., & Sinha, P. (2005). Perceiving illumination inconsistencies in scenes. Perception, 34, 1301–1314.

    Article  Google Scholar 

  • Pérez, P., Gangnet, M., & Blake, A. (2003). Poisson image editing. ACM Transactions on Graphics, 22(3), 313–318.

    Article  Google Scholar 

  • Prasad, M., & Fitzgibbon, A. (2006). Single view reconstruction of curved surfaces. In CVPR (pp. 1345–1354).

  • Ramamoorthi, R., & Hanrahan, P. (2001). An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on computer graphics and interactive techniques, SIGGRAPH’01 (pp. 497–500).

  • Richardson, E., Sela, M., Or-El, R., & Kimmel R. (July 2017). Learning detailed face reconstruction from a single image. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., & Samaras, D. (2017). Neural face editing with intrinsic image disentangling. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Tarr, M. J., Kersten, D., & Bülthoff, H. H. (1998). Why the visual recognition system might encode the effects of illumination. Vision Research, 38, 2259–2275.

    Article  Google Scholar 

  • Trigeorgis, G., Snape, P., Kokkinos, I., & Zafeiriou, S. (2017). Face normals “in-the-wild” using fully convolutional networks. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Twarog, N., Tappen, M., & Adelson, E. (2012). Playing with puffball: simple scale-invariant inflation for use in vision and graphics. In Proceedings of the ACM symposium on applied perception, SAP’12 (pp. 45–54).

  • Wu, T.-P., Sun, J., Tang, C.-K., & Shum, H.-Y. (2008). Interactive normal reconstruction from a single image. ACM Transactions on Graphics, 27(5), 119:1–119:9.

    Article  Google Scholar 

  • Xia, T., Liao, B., & Yu, Y. (2009). Patch-based image vectorization with automatic curvilinear feature alignment. ACM Transactions on Graphics, 28(5), 115:1–115:10.

    Article  Google Scholar 

  • Zhang, R., Tsai, P.-S., Cryer, J., & Shah, M. (1999). Shape-from-shading: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8), 690–706.

    Article  MATH  Google Scholar 

Download references

Acknowledgements

DAF is supported in part by Division of Information and Intelligent Systems (US) (IIS 09-16014), Division of Information and Intelligent Systems (IIS-1421521) and Office of Naval Research (N00014-10-10934). ZL is supported in part by NSFC Grant No. 61602406, ZJNSF Grant No. Q15F020006 and a special fund from Alibaba – Zhejiang University Joint Institute of Frontier Technologies.

Author information

Authors and Affiliations

  1. College of Computer Science, Zhejiang University, Hangzhou, China

    Zicheng Liao & Hongyi Zhang

  2. Department of Computer Science, University of Illinois at Urbana-Champaign, Champaign, USA

    Zicheng Liao, Kevin Karsch & David Forsyth

Authors
  1. Zicheng Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kevin Karsch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongyi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Forsyth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zicheng Liao.

Additional information

Communicated by Zhouchen Lin.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3791 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, Z., Karsch, K., Zhang, H. et al. An Approximate Shading Model with Detail Decomposition for Object Relighting. Int J Comput Vis 127, 22–37 (2019). https://doi.org/10.1007/s11263-018-1090-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-018-1090-6

Keywords

Search

Navigation