Skip to main content

Controllable Shadow Generation Using Pixel Height Maps

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13683))

Included in the following conference series:

Abstract

Shadows are essential for realistic image compositing from 2D image cutouts. Physics-based shadow rendering methods require 3D geometries, which are not always available. Deep learning-based shadow synthesis methods learn a mapping from the light information to an object’s shadow without explicitly modeling the shadow geometry. Still, they lack control and are prone to visual artifacts. We introduce “Pixel Height", a novel geometry representation that encodes the correlations between objects, ground, and camera pose. The Pixel Height can be calculated from 3D geometries, manually annotated on 2D images, and can also be predicted from a single-view RGB image by a supervised approach. It can be used to calculate hard shadows in a 2D image based on the projective geometry, providing precise control of the shadows’ direction and shape. Furthermore, we propose a data-driven soft shadow generator to apply softness to a hard shadow based on a softness input parameter. Qualitative and quantitative evaluations demonstrate that the proposed Pixel Height significantly improves the quality of the shadow generation while allowing for controllability.

Y. Sheng and Y. Liu—Contributed equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Annen, T., Dong, Z., Mertens, T., Bekaert, P., Seidel, H.P., Kautz, J.: Real-time, all-frequency shadows in dynamic scenes. ACM TOG 27(3), 1–8 (2008)

    Article  Google Scholar 

  2. Assarsson, U., Akenine-Möller, T.: A geometry-based soft shadow volume algorithm using graphics hardware. ACM TOG 22(3), 511–520 (2003)

    Article  Google Scholar 

  3. Cao, Y., Wu, Z., Shen, C.: Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE TCSVT 28(11), 3174–3182 (2017)

    Google Scholar 

  4. Chan, E., Durand, F.: Rendering fake soft shadows with smoothies. In: Rendering Techniques, pp. 208–218. Citeseer (2003)

    Google Scholar 

  5. Cook, R.L., Porter, T., Carpenter, L.: Distributed ray tracing. In: ACM SIGGRAPH, pp. 137–145 (1984)

    Google Scholar 

  6. Crow, F.C.: Shadow algorithms for computer graphics. ACM SIGGRAPH 11(2), 242–248 (1977)

    Article  Google Scholar 

  7. Donnelly, W., Lauritzen, A.: Variance shadow maps. In: Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games, pp. 161–165 (2006)

    Google Scholar 

  8. Fernando, R.: Percentage-closer soft shadows. In: ACM SIGGRAPH, pp. 35-es (2005)

    Google Scholar 

  9. Franke, T.A.: Delta voxel cone tracing. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 39–44. IEEE (2014)

    Google Scholar 

  10. Fuchs, H., et al.: Fast spheres, shadows, textures, transparencies, and Images enhancements in pixel-planes. ACM SIGGRAPH 19(3), 111–120 (1985)

    Article  Google Scholar 

  11. Guennebaud, G., Barthe, L., Paulin, M.: Real-time soft shadow mapping by backprojection. In: Rendering Techniques, pp. 227–234 (2006)

    Google Scholar 

  12. Guennebaud, G., Barthe, L., Paulin, M.: High-quality adaptive soft shadow mapping. In: Computer Graphics Forum, vol. 26, pp. 525–533. Wiley Online Library (2007)

    Google Scholar 

  13. Heitz, E., Dupuy, J., Hill, S., Neubelt, D.: Real-time polygonal-light shading with linearly transformed cosines. ACM TOG 35(4), 1–8 (2016)

    Google Scholar 

  14. Hong, Y., Niu, L., Zhang, J., Zhang, L.: Shadow generation for composite image in real-world scenes. arXiv preprint arXiv:2104.10338 (2021)

  15. Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-ShadowGAN: learning to remove shadows from unpaired data. In: ICCV, pp. 2472–2481 (2019)

    Google Scholar 

  16. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 1125–1134 (2017)

    Google Scholar 

  17. Jiang, Y., et al.: SSH: a self-supervised framework for image harmonization. In: ICCV, pp. 4832–4841 (2021)

    Google Scholar 

  18. Kajiya, J.T.: The rendering equation. In: ACM SIGGRAPH, pp. 143–150 (1986)

    Google Scholar 

  19. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)

    Google Scholar 

  20. Li, Z., Snavely, N.: MegaDepth: learning single-view depth prediction from internet photos. In: CVPR, pp. 2041–2050 (2018)

    Google Scholar 

  21. Liu, D., Long, C., Zhang, H., Yu, H., Dong, X., Xiao, C.: ARShadowGAN: shadow generative adversarial network for augmented reality in single light scenes. In: CVPR, pp. 8139–8148 (2020)

    Google Scholar 

  22. Liu, Y., Qin, Z., Wan, T., Luo, Z.: Auto-painter: cartoon image generation from sketch by using conditional wasserstein generative adversarial networks. Neurocomputing 311, 78–87 (2018)

    Article  Google Scholar 

  23. Lu, H., Dai, Y., Shen, C., Xu, S.: Indices matter: learning to index for deep image matting. In: ICCV, pp. 3266–3275 (2019)

    Google Scholar 

  24. Mehta, S.U., Wang, B., Ramamoorthi, R.: Axis-aligned filtering for interactive sampled soft shadows. ACM TOG 31(6), 1–10 (2012)

    Article  Google Scholar 

  25. Ng, R., Ramamoorthi, R., Hanrahan, P.: All-frequency shadows using non-linear wavelet lighting approximation. In: ACM SIGGRAPH, pp. 376–381 (2003)

    Google Scholar 

  26. Öztireli, A.C.: Integration with stochastic point processes. ACM TOG 35(5), 1–16 (2016)

    Article  Google Scholar 

  27. Pellacini, F., Tole, P., Greenberg, D.P.: A user interface for interactive cinematic shadow design. ACM TOG 21(3), 563–566 (2002)

    Article  Google Scholar 

  28. Reeves, W.T., Salesin, D.H., Cook, R.L.: Rendering antialiased shadows with depth maps. In: ACM SIGGRAPH, pp. 283–291 (1987)

    Google Scholar 

  29. Remondino, F.: From point cloud to surface: the modeling and visualization problem. Int. Arch. Photogrammetry Remote Sens. Spat. Inf. Sci. 34 (2003)

    Google Scholar 

  30. Ren, Z., et al.: Real-time soft shadows in dynamic scenes using spherical harmonic exponentiation. In: ACM SIGGRAPH, pp. 977–986 (2006)

    Google Scholar 

  31. Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: PIFu: pixel-aligned implicit function for high-resolution clothed human digitization. In: ICCV (2019)

    Google Scholar 

  32. Saito, S., Simon, T., Saragih, J., Joo, H.: PIFuHD: multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In: CVPR, June 2020

    Google Scholar 

  33. Saxena, A., Chung, S.H., Ng, A.Y., et al.: Learning depth from single monocular images. In: NeurIPS, vol. 18, pp. 1–8 (2005)

    Google Scholar 

  34. Schwarz, M., Stamminger, M.: Bitmask soft shadows. In: Computer Graphics Forum, vol. 26, pp. 515–524. Wiley Online Library (2007)

    Google Scholar 

  35. Sen, P., Cammarano, M., Hanrahan, P.: Shadow silhouette maps. ACM TOG 22(3), 521–526 (2003)

    Article  Google Scholar 

  36. Sheng, Y., Zhang, J., Benes, B.: SSN: soft shadow network for image compositing. In: CVPR, pp. 4380–4390 (2021)

    Google Scholar 

  37. Sillion, F.X., Arvo, J.R., Westin, S.H., Greenberg, D.P.: A global illumination solution for general reflectance distributions. In: ACM SIGGRAPH, pp. 187–196 (1991)

    Google Scholar 

  38. Soler, C., Sillion, F.X.: Fast calculation of soft shadow textures using convolution. In: ACM SIGGRAPH, pp. 321–332 (1998)

    Google Scholar 

  39. Wang, Y., Curless, B.L., Seitz, S.M.: People as scene probes. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12355, pp. 438–454. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58607-2_26

    Chapter  Google Scholar 

  40. Westin, S.H., Arvo, J.R., Torrance, K.E.: Predicting reflectance functions from complex surfaces. In: ACM SIGGRAPH, pp. 255–264 (1992)

    Google Scholar 

  41. Williams, L.: Casting curved shadows on curved surfaces. In: ACM SIGGRAPH, pp. 270–274 (1978)

    Google Scholar 

  42. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: simple and efficient design for semantic segmentation with transformers. In: NeurIPS (2021)

    Google Scholar 

  43. Yin, W., Liu, Y., Shen, C.: Virtual normal: enforcing geometric constraints for accurate and robust depth prediction. IEEE TPAMI 44, 7282–7295 (2021)

    Google Scholar 

  44. Yin, W., Liu, Y., Shen, C., Yan, Y.: Enforcing geometric constraints of virtual normal for depth prediction. In: ICCV, pp. 5684–5693 (2019)

    Google Scholar 

  45. Zhang, S., Liang, R., Wang, M.: ShadowGAN: shadow synthesis for virtual objects with conditional adversarial networks. Comput. Vis. Media 5(1), 105–115 (2019). https://doi.org/10.1007/s41095-019-0136-1

    Article  Google Scholar 

Download references

Acknowledgment

Most of the work was done during Yifan and Yichen’s internship at Adobe. This work was also supported by a UKRI Future Leaders Fellowship [grant number G104084]. We thank Dr. Zhi Tian for the discussions.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yichen Sheng .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 11007 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sheng, Y. et al. (2022). Controllable Shadow Generation Using Pixel Height Maps. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20050-2_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20049-6

  • Online ISBN: 978-3-031-20050-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics