Structured3D: A Large Photo-Realistic Dataset for Structured 3D Modeling

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12354)


Recently, there has been growing interest in developing learning-based methods to detect and utilize salient semi-global or global structures, such as junctions, lines, planes, cuboids, smooth surfaces, and all types of symmetries, for 3D scene modeling and understanding. However, the ground truth annotations are often obtained via human labor, which is particularly challenging and inefficient for such tasks due to the large number of 3D structure instances (e.g., line segments) and other factors such as viewpoints and occlusions. In this paper, we present a new synthetic dataset, Structured3D, with the aim of providing large-scale photo-realistic images with rich 3D structure annotations for a wide spectrum of structured 3D modeling tasks. We take advantage of the availability of professional interior designs and automatically extract 3D structures from them. We generate high-quality images with an industry-leading rendering engine. We use our synthetic dataset in combination with real images to train deep networks for room layout estimation and demonstrate improved performance on benchmark datasets.


Dataset 3D structure Photo-realistic rendering 



We would like to thank for providing the database of house designs and the rendering engine. We especially thank Qing Ye and Qi Wu from for the help on the data rendering. This work was partially supported by the National Key R&D Program of China (#2018AAA0100704) and the National Science Foundation of China (#61932020). Zihan Zhou was supported by NSF award #1815491.

Supplementary material

504446_1_En_30_MOESM1_ESM.pdf (47.7 mb)
Supplementary material 1 (pdf 48882 KB)


  1. 1.
  2. 2.
    Armeni, I., et al.: 3D scene graph: a structure for unified semantics, 3D space, and camera. In: ICCV, pp. 5664–5673 (2019)Google Scholar
  3. 3.
    Armeni, I., Sax, A., Zamir, A.R., Savarese, S.: Joint 2D–3D-semantic data for indoor scene understanding. CoRR abs/1702.01105 (2017)Google Scholar
  4. 4.
    Armeni, I., et al.: 3D semantic parsing of large-scale indoor spaces. In: CVPR, pp. 1534–1543 (2016)Google Scholar
  5. 5.
    Cadena, C., et al.: Past, present, and future of simultaneous localization and mapping: toward the robust-perception age. IEEE Trans. Robot. 32(6), 1309–1332 (2016)CrossRefGoogle Scholar
  6. 6.
    Chang, A.X., et al.: Matterport3D: learning from RGB-D data in indoor environments. In: 3DV, pp. 667–676 (2017)Google Scholar
  7. 7.
    Chen, J., Liu, C., Wu, J., Furukawa, Y.: Floor-SP: inverse CAD for floorplans by sequential room-wise shortest path. In: ICCV, pp. 2661–2670 (2019)Google Scholar
  8. 8.
    Chen, Y., Li, W., Chen, X., Van Gool, L.: Learning semantic segmentation from synthetic data: a geometrically guided input-output adaptation approach. In: CVPR, pp. 1841–1850 (2019)Google Scholar
  9. 9.
    Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: CVPR, pp. 5828–5839 (2017)Google Scholar
  10. 10.
    Dwibedi, D., Malisiewicz, T., Badrinarayanan, V., Rabinovich, A.: Deep cuboid detection: Beyond 2D bounding boxes. CoRR abs/1611.10010 (2016)Google Scholar
  11. 11.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B., Aubry, M.: A papier-mâché approach to learning 3D surface generation. In: CVPR, pp. 216–224 (2018)Google Scholar
  12. 12.
    Huang, K., Wang, Y., Zhou, Z., Ding, T., Gao, S., Ma, Y.: Learning to parse wireframes in images of man-made environments. In: CVPR, pp. 626–635 (2018)Google Scholar
  13. 13.
    Huang, S., Qi, S., Zhu, Y., Xiao, Y., Xu, Y., Zhu, S.-C.: Holistic 3D scene parsing and reconstruction from a single RGB image. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 194–211. Springer, Cham (2018). Scholar
  14. 14.
    Lee, C., Badrinarayanan, V., Malisiewicz, T., Rabinovich, A.: RoomNet: end-to-end room layout estimation. In: ICCV, pp. 4875–4884 (2017)Google Scholar
  15. 15.
    Li, W., et al.: InteriorNet: mega-scale multi-sensor photo-realistic indoor scenes dataset. In: BMVC, p. 77 (2018)Google Scholar
  16. 16.
    Liu, C., Kim, K., Gu, J., Furukawa, Y., Kautz, J.: Planercnn: 3D plane detection and reconstruction from a single image. In: CVPR. pp. 4450–4459 (2019)Google Scholar
  17. 17.
    Liu, C., Wu, J., Kohli, P., Furukawa, Y.: Raster-to-vector: revisiting floorplan transformation. In: ICCV. pp. 2214–2222 (2017)Google Scholar
  18. 18.
    Liu, C., Wu, J., Furukawa, Y.: Floornet: a unified framework for floorplan reconstruction from 3D scans. In: ECCV, pp. 203–219 (2018)Google Scholar
  19. 19.
    Liu, C., Yang, J., Ceylan, D., Yumer, E., Furukawa, Y.: Planenet: piece-wise planar reconstruction from a single RGB image. In: CVPR, pp. 2579–2588 (2018)Google Scholar
  20. 20.
    McCormac, J., Handa, A., Leutenegger, S., Davison, A.J.: Scenenet RGB-D: can 5m synthetic images beat generic imagenet pre-training on indoor segmentation? In: ICCV, pp. 2697–2706 (2017)Google Scholar
  21. 21.
    Purcell, T.J., Buck, I., Mark, W.R., Hanrahan, P.: Ray tracing on programmable graphics hardware. ACM Trans. Graph. 21(3), 703–712 (2002)CrossRefGoogle Scholar
  22. 22.
    Ros, G., Stent, S., Alcantarilla, P.F., Watanabe, T.: Training constrained deconvolutional networks for road scene semantic segmentation. CoRR abs/1604.01545 (2016)Google Scholar
  23. 23.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). Scholar
  24. 24.
    Song, S., Lichtenberg, S.P., Xiao, J.: SUN RGB-D: a RGB-D scene understanding benchmark suite. In: CVPR, pp. 567–576 (2015)Google Scholar
  25. 25.
    Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.A.: Semantic scene completion from a single depth image. In: CVPR, pp. 1746–1754 (2017)Google Scholar
  26. 26.
    Sun, C., Hsiao, C.W., Sun, M., Chen, H.T.: Horizonnet: Learning room layout with 1D representation and pano stretch data augmentation. In: CVPR, pp. 1047–1056 (2019)Google Scholar
  27. 27.
    Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: CVPR. pp. 7472–7481 (2018)Google Scholar
  28. 28.
    Tulsiani, S., Su, H., Guibas, L.J., Efros, A.A., Malik, J.: Learning shape abstractions by assembling volumetric primitives. In: CVPR. pp. 2635–2643 (2017)Google Scholar
  29. 29.
    Wald, I., Woop, S., Benthin, C., Johnson, G.S., Ernst, M.: Embree: a kernel framework for efficient CPU ray tracing. ACM Trans. Graph. 33(4), 143:1–143:8 (2014)CrossRefGoogle Scholar
  30. 30.
    Wang, K., Lin, Y.A., Weissmann, B., Savva, M., Chang, A.X., Ritchie, D.: Planit: planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Trans. Graph. 38(4), 1–15 (2019)Google Scholar
  31. 31.
    Witkin, A.P., Tenenbaum, J.M.: On the role of structure in vision. In: Beck, J., Hope, B., Rosenfeld, A. (eds.) Human and Machine Vision, pp. 481–543. Academic Press, Cambridge (1983)CrossRefGoogle Scholar
  32. 32.
    Wu, J., Xue, T., Lim, J.J., Tian, Y., Tenenbaum, J.B., Torralba, A., Freeman, W.T.: 3D interpreter networks for viewer-centered wireframe modeling. IJCV 126(9), 1009–1026 (2018). Scholar
  33. 33.
    Xiao, J., Ehinger, K.A., Oliva, A., Torralba, A.: Recognizing scene viewpoint using panoramic place representation. In: CVPR, pp. 2695–2702 (2012)Google Scholar
  34. 34.
    Xiao, J., Russell, B., Torralba, A.: Localizing 3D cuboids in single-view images. In: NeurIPS, pp. 746–754 (2012)Google Scholar
  35. 35.
    Yang, F., Zhou, Z.: Recovering 3D planes from a single image via convolutional neural networks. In: ECCV, pp. 87–103 (2018)Google Scholar
  36. 36.
    Yu, Z., Zheng, J., Lian, D., Zhou, Z., Gao, S.: Single-image piece-wise planar 3D reconstruction via associative embedding. In: CVPR. pp. 1029–1037 (2019)Google Scholar
  37. 37.
    Zhang, Y., Song, S., Tan, P., Xiao, J.: Panocontext: a whole-room 3D context model for panoramic scene understanding. In: ECCV, pp. 668–686 (2014)Google Scholar
  38. 38.
    Zhang, Y., et al.: Physically-based rendering for indoor scene understanding using convolutional neural networks. In: CVPR, pp. 5287–5295 (2017)Google Scholar
  39. 39.
    Zhang, Y., Yu, F., Song, S., Xu, P., Seff, A., Xiao, J.: Large-scale scene understanding challenge: room layout estimation (2016)Google Scholar
  40. 40.
    Zhou, Y., et al.: Learning to reconstruct 3D manhattan wireframes from a single image. In: ICCV, pp. 7698–7707 (2019)Google Scholar
  41. 41.
    Zou, C., Colburn, A., Shan, Q., Hoiem, D.: Layoutnet: reconstructing the 3D room layout from a single RGB image. In: CVPR, pp. 2051–2059 (2018)Google Scholar
  42. 42.
    Zou, C., et al.: 3D manhattan room layout reconstruction from a single 360 image. CoRR abs/1910.04099 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.KooLab, Kujiale.comHangzhouChina
  2. 2.ShanghaiTech UniversityShanghaiChina
  3. 3.Shanghai Engineering Research Center of Intelligent Vision and ImagingShanghaiChina
  4. 4.The Pennsylvania State UniversityState CollegeUSA

Personalised recommendations