Skip to main content
SpringerLink
Log in

Virtual Home Staging: Inverse Rendering and Editing an Indoor Panorama under Natural Illumination

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14361))

Included in the following conference series:

  • 509 Accesses

Abstract

We propose a novel inverse rendering method that enables the transformation of existing indoor panoramas with new indoor furniture layouts under natural illumination. To achieve this, we captured indoor HDR panoramas along with real-time outdoor hemispherical HDR photographs. Indoor and outdoor HDR images were linearly calibrated with measured absolute luminance values for accurate scene relighting. Our method consists of three key components: (1) panoramic furniture detection and removal, (2) automatic floor layout design, and (3) global rendering incorporating scene geometry, new furniture objects, and real-time outdoor photograph. We demonstrate the effectiveness of our workflow in rendering indoor scenes under different outdoor illumination conditions. Additionally, we contribute a new calibrated HDR (Cali-HDR) dataset that consists of 137 calibrated indoor panoramas and their associated outdoor photographs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Araújo, A.B.: Drawing equirectangular VR panoramas with ruler, compass, and protractor. J. Sci. Technol. Arts 10(1), 15–27 (2018)

    Article  Google Scholar 

  2. Bolduc, C., Giroux, J., Hébert, M., Demers, C., Lalonde, J.F.: Beyond the pixel: a photometrically calibrated HDR dataset for luminance and color temperature prediction. arXiv preprint arXiv:2304.12372 (2023)

  3. Chen, B., Zhi, T., Hebert, M., Narasimhan, S.G.: Learning continuous implicit representation for near-periodic patterns. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13675, pp. 529–546. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19784-0_31

    Chapter  Google Scholar 

  4. Cheng, H.T., Chao, C.H., Dong, J.D., Wen, H.K., Liu, T.L., Sun, M.: Cube padding for weakly-supervised saliency prediction in 360 videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1420–1429 (2018)

    Google Scholar 

  5. Coughlan, J.M., Yuille, A.L.: Manhattan world: compass direction from a single image by Bayesian inference. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 941–947. IEEE (1999)

    Google Scholar 

  6. Cruz, S., Hutchcroft, W., Li, Y., Khosravan, N., Boyadzhiev, I., Kang, S.B.: Zillow indoor dataset: annotated floor plans with 360deg panoramas and 3D room layouts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2133–2143 (2021)

    Google Scholar 

  7. Debevec, P.: Image-based lighting. In: ACM SIGGRAPH 2006 Courses, pp. 4–es (2006)

    Google Scholar 

  8. Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: SIGGRAPH 2008 Classes, pp. 1–10. ACM (2008)

    Google Scholar 

  9. Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: SIGGRAPH 2008 classes, pp. 1–10. ACM (2008)

    Google Scholar 

  10. Fu, H., et al.: 3D-future: 3D furniture shape with texture. Int. J. Comput. Vis. 129, 3313–3337 (2021)

    Google Scholar 

  11. Gardner, M.A., Hold-Geoffroy, Y., Sunkavalli, K., Gagné, C., Lalonde, J.F.: Deep parametric indoor lighting estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7175–7183 (2019)

    Google Scholar 

  12. Gardner, M.A., et al.: Learning to predict indoor illumination from a single image. arXiv preprint arXiv:1704.00090 (2017)

  13. Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6908–6917 (2019)

    Google Scholar 

  14. Gkitsas, V., Sterzentsenko, V., Zioulis, N., Albanis, G., Zarpalas, D.: PanoDR: spherical panorama diminished reality for indoor scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3716–3726 (2021)

    Google Scholar 

  15. Gkitsas, V., Zioulis, N., Alvarez, F., Zarpalas, D., Daras, P.: Deep lighting environment map estimation from spherical panoramas. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 640–641 (2020)

    Google Scholar 

  16. Gkitsas, V., Zioulis, N., Sterzentsenko, V., Doumanoglou, A., Zarpalas, D.: Towards full-to-empty room generation with structure-aware feature encoding and soft semantic region-adaptive normalization. arXiv preprint arXiv:2112.05396 (2021)

  17. Guerrero-Viu, J., Fernandez-Labrador, C., Demonceaux, C., Guerrero, J.J.: What’s in my room? object recognition on indoor panoramic images. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 567–573. IEEE (2020)

    Google Scholar 

  18. Huang, J.B., Kang, S.B., Ahuja, N., Kopf, J.: Image completion using planar structure guidance. ACM Trans. Graph. (TOG) 33(4), 1–10 (2014)

    Google Scholar 

  19. Huang, S., Qi, S., Zhu, Y., Xiao, Y., Xu, Y., Zhu, S.C.: Holistic 3D scene parsing and reconstruction from a single RGB image. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 187–203 (2018)

    Google Scholar 

  20. Inanici, M.: Evalution of high dynamic range image-based sky models in lighting simulation. Leukos 7(2), 69–84 (2010)

    Article  Google Scholar 

  21. Inanici, M.N.: Evaluation of high dynamic range photography as a luminance data acquisition system. Lighting Res. Technol. 38(2), 123–134 (2006)

    Article  Google Scholar 

  22. Izadinia, H., Shan, Q., Seitz, S.M.: IM2CAD. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5134–5143 (2017)

    Google Scholar 

  23. Karsch, K., Hedau, V., Forsyth, D., Hoiem, D.: Rendering synthetic objects into legacy photographs. ACM Trans. Graph. (TOG) 30(6), 1–12 (2011)

    Article  Google Scholar 

  24. Kawai, N., Sato, T., Yokoya, N.: Diminished reality based on image inpainting considering background geometry. IEEE Trans. Visual Comput. Graphics 22(3), 1236–1247 (2015)

    Article  Google Scholar 

  25. Kulshreshtha, P., Lianos, N., Pugh, B., Jiddi, S.: Layout aware inpainting for automated furniture removal in indoor scenes. In: 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 839–844. IEEE (2022)

    Google Scholar 

  26. LeGendre, C., et al.: DeepLight: learning illumination for unconstrained mobile mixed reality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5918–5928 (2019)

    Google Scholar 

  27. Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: shape, spatially-varying lighting and SVBRDF from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2475–2484 (2020)

    Google Scholar 

  28. Li, Z., et al.: Physically-based editing of indoor scene lighting from a single image. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13666, pp. 555–572. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20068-7_32

    Chapter  Google Scholar 

  29. Liu, Y.L., et al.: Single-image HDR reconstruction by learning to reverse the camera pipeline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1651–1660 (2020)

    Google Scholar 

  30. Mitsunaga, T., Nayar, S.K.: Radiometric self calibration. In: Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), vol. 1, pp. 374–380. IEEE (1999)

    Google Scholar 

  31. Moeck, M.: Accuracy of luminance maps obtained from high dynamic range images. Leukos 4(2), 99–112 (2007)

    Article  Google Scholar 

  32. Nie, Y., Han, X., Guo, S., Zheng, Y., Chang, J., Zhang, J.J.: Total3DUnderstanding: joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 55–64 (2020)

    Google Scholar 

  33. Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting. Morgan Kaufmann, Burlington (2010)

    Google Scholar 

  34. Srinivasan, P.P., Mildenhall, B., Tancik, M., Barron, J.T., Tucker, R., Snavely, N.: Lighthouse: predicting lighting volumes for spatially-coherent illumination. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8080–8089 (2020)

    Google Scholar 

  35. Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., Debevec, P.: Direct HDR capture of the sun and sky. In: SIGGRAPH 2006 Courses, pp. 5-es. ACM (2006)

    Google Scholar 

  36. Suvorov, R., et al.: Resolution-robust large mask inpainting with Fourier convolutions. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2149–2159 (2022)

    Google Scholar 

  37. Wang, F.-E., et al.: Self-supervised learning of depth and camera motion from 360\(^\circ \) videos. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11365, pp. 53–68. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20873-8_4

    Chapter  Google Scholar 

  38. Wang, F.E., Yeh, Y.H., Sun, M., Chiu, W.C., Tsai, Y.H.: BiFuse: monocular 360 depth estimation via bi-projection fusion. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  39. Wang, F.E., Yeh, Y.H., Sun, M., Chiu, W.C., Tsai, Y.H.: LED2-Net: monocular 360\(^\circ \) layout estimation via differentiable depth rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12956–12965 (2021)

    Google Scholar 

  40. Xiao, J., Ehinger, K.A., Oliva, A., Torralba, A.: Recognizing scene viewpoint using panoramic place representation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2695–2702. IEEE (2012)

    Google Scholar 

  41. Yang, B., Jiang, T., Wu, W., Zhou, Y., Dai, L.: Automated semantics and topology representation of residential-building space using floor-plan raster maps. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 15, 7809–7825 (2022)

    Article  Google Scholar 

  42. Yang, S.T., Wang, F.E., Peng, C.H., Wonka, P., Sun, M., Chu, H.K.: DuLa-Net: a dual-projection network for estimating room layouts from a single RGB panorama. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3363–3372 (2019)

    Google Scholar 

  43. Yeh, Y.Y., et al.: PhotoScene: photorealistic material and lighting transfer for indoor scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18562–18571 (2022)

    Google Scholar 

  44. Zeng, Z., Li, X., Yu, Y.K., Fu, C.W.: Deep floor plan recognition using a multi-task network with room-boundary-guided attention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9096–9104 (2019)

    Google Scholar 

  45. Zhang, C., Liwicki, S., Smith, W., Cipolla, R.: Orientation-aware semantic segmentation on icosahedron spheres. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3533–3541 (2019)

    Google Scholar 

  46. Zhang, E., Cohen, M.F., Curless, B.: Emptying, refurnishing, and relighting indoor spaces. ACM Trans. Graph. (TOG) 35(6), 1–14 (2016)

    Google Scholar 

  47. Zhang, E., Martin-Brualla, R., Kontkanen, J., Curless, B.L.: No shadow left behind: removing objects and their shadows using approximate lighting and geometry. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16397–16406 (2021)

    Google Scholar 

  48. Zhang, Y., Song, S., Tan, P., Xiao, J.: PanoContext: a whole-room 3D context model for panoramic scene understanding. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 668–686. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_43

    Chapter  Google Scholar 

  49. Zhi, T., Chen, B., Boyadzhiev, I., Kang, S.B., Hebert, M., Narasimhan, S.G.: Semantically supervised appearance decomposition for virtual staging from a single panorama. ACM Transa. Graph. (TOG) 41(4), 1–15 (2022)

    Article  Google Scholar 

  50. Zhou, B., et al.: Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis. 127, 302–321 (2018)

    Google Scholar 

  51. Zou, C., Colburn, A., Shan, Q., Hoiem, D.: LayoutNet: reconstructing the 3D room layout from a single RGB image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2051–2059 (2018)

    Google Scholar 

Download references

Acknowledgement

This work was partially supported by a gift from Zillow Group, USA.

Author information

Authors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Guanzhou Ji, Azadeh O. Sawyer & Srinivasa G. Narasimhan

Authors
  1. Guanzhou Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azadeh O. Sawyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Srinivasa G. Narasimhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanzhou Ji .

Editor information

Editors and Affiliations

  1. University of Nevada Reno, Reno, NV, USA

    George Bebis

  2. Google Research, Mountain View, CA, USA

    Golnaz Ghiasi

  3. New York University, New York, USA

    Yi Fang

  4. Ben-Gurion University, Be'er Sheva, Israel

    Andrei Sharf

  5. Microsoft Research, Beijing, China

    Yue Dong

  6. The University of Oklahoma, Norman, OK, USA

    Chris Weaver

  7. University of Maryland, Collage Park, MD, USA

    Zhicheng Leo

  8. University of Central Florida, Orlando, FL, USA

    Joseph J. LaViola Jr.

  9. InnerOptic Technology, Hillsborough, NC, USA

    Luv Kohli

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ji, G., Sawyer, A.O., Narasimhan, S.G. (2023). Virtual Home Staging: Inverse Rendering and Editing an Indoor Panorama under Natural Illumination. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2023. Lecture Notes in Computer Science, vol 14361. Springer, Cham. https://doi.org/10.1007/978-3-031-47969-4_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47969-4_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47968-7

  • Online ISBN: 978-3-031-47969-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation