Advertisement

Learning to Factorize and Relight a City

Conference paper
  • 855 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12349)

Abstract

We propose a learning-based framework for disentangling outdoor scenes into temporally-varying illumination and permanent scene factors. Inspired by the classic intrinsic image decomposition, our learning signal builds upon two insights: 1) combining the disentangled factors should reconstruct the original image, and 2) the permanent factors should stay constant across multiple temporal samples of the same scene. To facilitate training, we assemble a city-scale dataset of outdoor timelapse imagery from Google Street View, where the same locations are captured repeatedly through time. This data represents an unprecedented scale of spatio-temporal outdoor imagery. We show that our learned disentangled factors can be used to manipulate novel images in realistic ways, such as changing lighting effects and scene geometry. Please visit http://factorize-a-city.github.io/ for animated results.

Notes

Acknowledgements

We would like to thank Richard Tucker, Richard Bowen, Ameesh Makadia, and Vincent Sitzmann for insightful discussions. We would also like to thank Angjoo Kanazawa and Tim Brooks for their help with preparing the manuscript. This work is supported, in part, by NSF grant IIS-1633310.

Supplementary material

504439_1_En_32_MOESM1_ESM.zip (79.9 mb)
Supplementary material 1 (zip 81817 KB)

References

  1. 1.
    Adelson, E.H., Pentland, A.P.: The perception of shading and reflectance. In: Knill, D.C., Richards, W. (eds.) Perception as Bayesian Inference, pp. 409–423. Cambridge University Press, New York (1996)CrossRefGoogle Scholar
  2. 2.
    Adelson, E.H., Bergen, J.R.: The plenoptic function and the elements of early vision. In: Landy, M., Movshon, J.A. (eds.) Computational Models of Visual Processing, pp. 3–20. MIT Press, Cambridge (1991)Google Scholar
  3. 3.
    Arietta, S.M., Efros, A.A., Ramamoorthi, R., Agrawala, M.: City forensics: using visual elements to predict non-visual city attributes. IEEE Trans. Visual Comput. Graphics 20(12), 2624–2633 (2014).  https://doi.org/10.1109/TVCG.2014.2346446CrossRefGoogle Scholar
  4. 4.
    Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. Trans. Pattern Anal. Mach. Intell. 37(8), 1670–1687 (2015)CrossRefGoogle Scholar
  5. 5.
    Barrow, H.G., Tenenbaum, J.M.: Recovering intrinsic scene characteristics from images. Comput. Vis. Syst. 2(3–26), 2 (1978)Google Scholar
  6. 6.
    Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graphics (SIGGRAPH) 33(4), 159:1–159:12 (2014).  https://doi.org/10.1145/2601097.2601206CrossRefGoogle Scholar
  7. 7.
    Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (2019)Google Scholar
  8. 8.
    Klingner, B., Martin, D., Roseborough, J.: Street view motion-from-structure-from-motion. In: Proceedings of the International Conference on Computer Vision (ICCV) (2013)Google Scholar
  9. 9.
    Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.A.: What makes Paris look like Paris? ACM Trans. Graphics (SIGGRAPH) 31(4), 101:1–101:9 (2012)CrossRefGoogle Scholar
  10. 10.
    Gebru, T., et al.: Using deep learning and google street view to estimate the demographic makeup of neighborhoods across the United States. Proc. Natl. Acad. Sci. 114(50), 13108–13113 (2017).  https://doi.org/10.1073/pnas.1700035114CrossRefGoogle Scholar
  11. 11.
    Gronat, P., Obozinski, G., Sivic, J., Pajdla, T.: Learning and calibrating per-location classifiers for visual place recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013Google Scholar
  12. 12.
    Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.F.: Deep outdoor illumination estimation. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), July 2017Google Scholar
  13. 13.
    Huang, G.B., Jain, V., Learned-Miller, E.: Unsupervised joint alignment of complex images. In: Proceedings of the International Conference on Computer Vision (ICCV) (2007)Google Scholar
  14. 14.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  15. 15.
    Jacobs, N., Roman, N., Pless, R.: Consistent temporal variations in many outdoor scenes. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), pp. 1–6, June 2007.  https://doi.org/10.1109/CVPR.2007.383258
  16. 16.
    Janner, M., Wu, J., Kulkarni, T.D., Yildirim, I., Tenenbaum, J.: Self-supervised intrinsic image decomposition. In: Neural Information Processing Systems, pp. 5936–5946. Curran Associates, Inc. (2017)Google Scholar
  17. 17.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  18. 18.
    Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 386–402. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01267-0_23CrossRefGoogle Scholar
  19. 19.
    Laffont, P.Y., Bazin, J.C.: Intrinsic decomposition of image sequences from local temporal variations. In: Proceedings of the International Conference on Computer Vision (ICCV), December 2015Google Scholar
  20. 20.
    Laffont, P.Y., Ren, Z., Tao, X., Qian, C., Hays, J.: Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graphics (SIGGRAPH) 33(4), 1–11 (2014)CrossRefGoogle Scholar
  21. 21.
    Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graphics (SIGGRAPH) 28(5), 1–10 (2009)CrossRefGoogle Scholar
  22. 22.
    Lee, S., Maisonneuve, N., Crandall, D., Efros, A.A., Sivic, J.: Linking past to present: discovering style in two centuries of architecture. In: IEEE International Conference on Computational Photography (ICCP) (2015)Google Scholar
  23. 23.
    Li, Z., Snavely, N.: CGIntrinsics: better intrinsic image decomposition through physically-based rendering. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 381–399. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01219-9_23CrossRefGoogle Scholar
  24. 24.
    Li, Z., Snavely, N.: Learning intrinsic image decomposition from watching the world. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  25. 25.
    Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: shape, spatially-varying lighting and SVBRDF from a single image. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2020)Google Scholar
  26. 26.
    Martin-Brualla, R., Gallup, D., Seitz, S.M.: Time-lapse mining from internet photos. ACM Trans. Graphics (SIGGRAPH) 34(4), 62:1–62:8 (2015).  https://doi.org/10.1145/2766903CrossRefGoogle Scholar
  27. 27.
    Meshry, M., et al.: Neural rerendering in the wild. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  28. 28.
    Naik, N., Philipoom, J., Raskar, R., Hidalgo, C.: Streetscore - predicting the perceived safety of one million streetscapes. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 793–799, June 2014.  https://doi.org/10.1109/CVPRW.2014.121
  29. 29.
    Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  30. 30.
    Philip, J., Gharbi, M., Zhou, T., Efros, A.A., Drettakis, G.: Multi-view relighting using a geometry-aware network. ACM Trans. Graphics (SIGGRAPH) 38(4) (2019). http://www-sop.inria.fr/reves/Basilic/2019/PGZED19
  31. 31.
    Rubinstein, M., Liu, C., Sand, P., Durand, F., Freeman, W.T.: Motion denoising with application to time-lapse photography. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), pp. 313–320, June 2011Google Scholar
  32. 32.
    Sengupta, S., Gu, J., Kim, K., Liu, G., Jacobs, D.W., Kautz, J.: Neural inverse rendering of an indoor scene from a single image. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)Google Scholar
  33. 33.
    Sengupta, S., Kanazawa, A., Castillo, C.D., Jacobs, D.W.: SfSNet: learning shape, reflectance and illuminance of faces in the wild. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  34. 34.
    Sunkavalli, K., Matusik, W., Pfister, H., Rusinkiewicz, S.: Factored time-lapse video. ACM Trans. Graphics (SIGGRAPH) (2007). SIGGRAPH 2007. ACM, New York.  https://doi.org/10.1145/1275808.1276504
  35. 35.
    Vo, N.N., Hays, J.: Localizing and orienting street views using overhead imagery. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 494–509. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46448-0_30CrossRefGoogle Scholar
  36. 36.
    Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  37. 37.
    Weiss, Y.: Deriving intrinsic images from image sequences. In: Proceedings of the International Conference on Computer Vision (ICCV) (2001)Google Scholar
  38. 38.
    Yu, Y., Smith, W.A.: InverseRenderNet: learning single image inverse rendering. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  39. 39.
    Zhou, T., Krähenbähl, P., Efros, A.A.: Learning data-driven reflectance priors for intrinsic image decomposition. In: Proceedings of the International Conference on Computer Vision (ICCV) (2015)Google Scholar
  40. 40.
    Zhou, Y., Berg, T.L.: Learning temporal transformations from time-lapse videos. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 262–277. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_16CrossRefGoogle Scholar
  41. 41.
    Zhu, J.Y., et al.: Visual object networks: image generation with disentangled 3D representations. In: Neural Information Processing Systems (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.GoogleBerkeleyUSA
  2. 2.UC BerkeleyBerkeleyUSA
  3. 3.Humen, Inc.San FranciscoUSA

Personalised recommendations