Abstract
We present a novel method to reconstruct the 3D layout of a room—walls, floors, ceilings—from a single perspective view in challenging conditions, by contrast with previous single-view methods restricted to cuboid-shaped layouts. This input view can consist of a color image only, but considering a depth map results in a more accurate reconstruction. Our approach is formalized as solving a constrained discrete optimization problem to find the set of 3D polygons that constitute the layout. In order to deal with occlusions between components of the layout, which is a problem ignored by previous works, we introduce an analysis-by-synthesis method to iteratively refine the 3D layout estimate. As no dataset was available to evaluate our method quantitatively, we created one together with several appropriate metrics. Our dataset consists of 293 images from ScanNet, which we annotated with precise 3D layouts. It offers three times more samples than the popular NYUv2 303 benchmark, and a much larger variety of layouts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Budroni, A., Boehm, J.: Automated 3D reconstruction of interiors from point clouds. Int. J. Archit. Comput. 8, 55–73 (2010)
Cabral, R., Furukawa, Y.: Piecewise planar and compact floorplan reconstruction from images. In: Conference on Computer Vision and Pattern Recognition (2014)
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European Conference on Computer Vision (2018)
Chen, Y., Huang, S., Yuan, T., Qi, S., Zhu, Y., Zhu, S.C.: Holistic++ scene understanding: single-view 3D holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In: International Conference on Computer Vision (2019)
Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Niessner, M.: ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: Conference on Computer Vision and Pattern Recognition (2017)
Godard, C., Aodha, O.M., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Conference on Computer Vision and Pattern Recognition (2017)
Guo, R., Hoiem, D.: Support surface prediction in indoor scenes. In: International Conference on Computer Vision (2013)
Hedau, V., Hoiem, D., Forsyth, D.: Recovering the spatial layout of cluttered rooms. In: International Conference on Computer Vision (2009)
Hirzer, M., Roth, P.M., Lepetit, V.: Smart hypothesis generation for efficient and robust room layout estimation. In: IEEE Winter Conference on Applications of Computer Vision (2020)
Howard-Jenkins, H., Li, S., Prisacariu, V.: Thinking outside the box: generation of unconstrained 3D room layouts. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11361, pp. 432–448. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20887-5_27
Huang, S., Qi, S., Xiao, Y., Zhu, Y., Wu, Y.N., Zhu, S.C.: Cooperative holistic scene understanding: unifying 3D object, layout, and camera pose estimation. In: Advances in Neural Information Processing Systems (2018)
Ikehata, S., Yang, H., Furukawa, Y.: Structured indoor modeling. In: International Conference on Computer Vision (2015)
Ku, J., Harakeh, A., Waslander, S.L.. In defense of classical image processing: fast depth completion on the CPU. In: CRV (2018)
Kuhn, H.W., Yaw, B.: The Hungarian method for the assignment problem. Naval Res. Logist, Quart (1955)
Kundu, A., Li, Y., Rehg, J.M.: 3D-RCNN: instance-level 3D object reconstruction via render-and-compare. In: Conference on Computer Vision and Pattern Recognition (2018)
Lee, C.Y., Badrinarayanan, V., Malisiewicz, T., Rabinovich, A.: Roomnet: end-to-end room layout estimation. In: International Conference on Computer Vision (2017)
Lee, J.H., Han, M.K., Ko, D.W., Suh, I.H.: From big to small: multi-scale local planar guidance for monocular depth estimation. arXiv Preprint (2019)
Liu, C., Kim, K., Gu, J., Furukawa, Y., Kautz, J.: PlanerCNN: 3D plane detection and reconstruction from a single image. In: Conference on Computer Vision and Pattern Recognition (2019)
Liu, C., Wu, J., Furukawa, Y.: FloorNet: A unified framework for floorplan reconstruction from 3D scans. In: European Conference on Computer Vision (2018)
Mallya, A., Lazebnik, S.: Learning informative edge maps for indoor scene layout prediction. In: International Conference on Computer Vision (2015)
Murali, S., Speciale, P., Oswald, M.R., Pollefeys, M.: Indoor Scan2BIM: building information models of house interiors. In: International Conference on Intelligent Robots and Systems (2017)
Nan, L., Wonka, P.: Polyfit: polygonal surface reconstruction from point clouds. In: International Conference on Computer Vision (2017)
Olson, C.F., Huttenlocher, D.P.: Automatic target recognition by matching oriented edge pixels. J. Mach. Learn. Res. 6, 103–113 (1997)
Ramalingam, S., Pillai, J.K., Jain, A., Taguchi, Y.: Manhattan junction catalogue for spatial reasoning of indoor scenes. In: Conference on Computer Vision and Pattern Recognition (2013)
Ramamonjisoa, M., Lepetit, V.: SharpNet: fast and accurate recovery of occluding contours in monocular depth estimation. In: International Conference on Computer Vision Workshops (2019)
Sanchez, V., Zakhor, A.: Planar 3D modeling of building interiors from point cloud data. In: International Conference on Computer Vision (2012)
Schwing, A.G., Hazan, T., Pollefeys, M., Urtasun, R.: Efficient structured prediction for 3D indoor scene understanding. In: Conference on Computer Vision and Pattern Recognition (2012)
Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4_54
Sun, C., Hsiao, C.W., Sun, M., Chen, H.T.: HorizonNet: learning room layout with 1D representation and pano stretch data augmentation. In: Conference on Computer Vision and Pattern Recognition (2019)
Xu, Y., Zhu, S.C., Tung, T.: DenseRaC: joint 3D pose and shape estimation by dense render-and-compare. In: International Conference on Computer Vision (2019)
Zhang, J., Kan, C., Schwing, A.G., Urtasun, R.: Estimating the 3D layout of indoor scenes and its clutter from depth sensors. In: International Conference on Computer Vision (2013)
Zhang, W., Zhang, W., Gu, J.: Edge-semantic learning strategy for layout estimation in indoor environment. IEEE Trans. Cybern. 50, 2730–2739 (2019)
Zhang, Y., Yu, F., Song, S., Xu, P., Seff, A., Xiao, J.: Large-scale scene understanding challenge: room layout estimation. In: Conference on Computer Vision and Pattern Recognition Workshops (2015)
Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S., Zhou, Z.: Structured3D: a large photo-realistic dataset for structured 3D modeling. In: European Conference on Computer Vision (2020)
Zou, C., Colburn, A., Shan, Q., Hoiem, D.: LayoutNet: reconstructing the 3D room layout from a single RGB image. In: Conference on Computer Vision and Pattern Recognition (2018)
Zou, C., et al.: 3D manhattan room layout reconstruction from a single 360 image. arXiv Preprint (2019)
Zou, C., Guo, R., Li, Z., Hoiem, D.: Complete 3D scene parsing from an RGBD image. Int. J. Comput. Vis. 127, 143–162 (2019)
Acknowledgment
This work was supported by the Christian Doppler Laboratory for Semantic 3D Computer Vision, funded in part by Qualcomm Inc.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Stekovic, S., Hampali, S., Rad, M., Sarkar, S.D., Fraundorfer, F., Lepetit, V. (2020). General 3D Room Layout from a Single View by Render-and-Compare. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12361. Springer, Cham. https://doi.org/10.1007/978-3-030-58517-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-58517-4_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58516-7
Online ISBN: 978-3-030-58517-4
eBook Packages: Computer ScienceComputer Science (R0)