Neural Procedural Reconstruction for Residential Buildings

  • Huayi ZengEmail author
  • Jiaye Wu
  • Yasutaka Furukawa
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11207)


This paper proposes a novel 3D reconstruction approach, dubbed Neural Procedural Reconstruction (NPR). NPR infers a sequence of shape grammar rule applications and reconstructs CAD-quality models with procedural structure from 3D points. While most existing methods rely on low-level geometry analysis to extract primitive structures, our approach conducts global analysis of entire building structures by deep neural networks (DNNs), enabling the reconstruction even from incomplete and sparse input data. We demonstrate the proposed system for residential buildings with aerial LiDAR as the input. Our 3D models boast compact geometry and semantically segmented architectural components. Qualitative and quantitative evaluations on hundreds of houses demonstrate that the proposed approach makes significant improvements over the existing state-of-the-art.


3D reconstruction CAD Deep learning Procedural modeling 



This research is partially supported by National Science Foundation under grant IIS 1540012 and IIS 1618685, and Google Faculty Research Award. We thank Nvidia for a generous GPU donation.

Supplementary material

474178_1_En_45_MOESM1_ESM.pdf (3.6 mb)
Supplementary material 1 (pdf 3736 KB)


  1. 1.
  2. 2.
    Bulat, A., Tzimiropoulos, G.: Human pose estimation via convolutional part heatmap regression. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 717–732. Springer, Cham (2016). Scholar
  3. 3.
    Chauve, A.L., Labatut, P., Pons, J.P.: Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1261–1268. IEEE (2010)Google Scholar
  4. 4.
  5. 5.
    Demir, I., Aliaga, D.G., Benes, B.: Procedural editing of 3D building point clouds. In: International Conference on Computer Vision, pp. 2147–2155 (2015)Google Scholar
  6. 6.
    Demir, I., Aliaga, D.G., Benes, B.: Proceduralization for editing 3D architectural models. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 194–202. IEEE (2016)Google Scholar
  7. 7.
    Dick, A.R., Torr, P.H.S., Cipolla, R.: A bayesian estimation of building shape using MCMC. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 852–866. Springer, Heidelberg (2002). Scholar
  8. 8.
    Dylla, K., Frischer, B., Müller, P., Ulmer, A., Haegler, S.: Rome reborn 2.0: a case study of virtual city reconstruction using procedural modeling techniques. Comput. Graph. World 16(6), 62–66 (2008)Google Scholar
  9. 9.
  10. 10.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Manhattan-world stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1422–1429. IEEE (2009)Google Scholar
  11. 11.
    Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 209–216. ACM Press/Addison-Wesley Publishing Co. (1997)Google Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
  13. 13.
    Henn, A., Grger, G., Stroh, V., Plmer, L.: Model driven reconstruction of roofs from sparse LiDAR point clouds 76, 17–29, February 2013Google Scholar
  14. 14.
    Huang, H., Brenner, C., Sester, M.: A generative statistical approach to automatic 3D building roof reconstruction from laser scanning data. ISPRS J. Photogramm. Remote Sens. 79, 29–43 (2013)CrossRefGoogle Scholar
  15. 15.
    Ikehata, S., Yang, H., Furukawa, Y.: Structured indoor modeling. In: IEEE International Conference on Computer Vision, pp. 1323–1331 (2015)Google Scholar
  16. 16.
    Jiang, H., Xiao, J.: A linear approach to matching cuboids in RGBD images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2171–2178 (2013)Google Scholar
  17. 17.
    Kazhdan, M., Hoppe, H.: Screened poisson surface reconstruction. ACM Trans. Graph. (TOG) 32(3), 29 (2013)CrossRefGoogle Scholar
  18. 18.
    Kim, V.G., Li, W., Mitra, N.J., Chaudhuri, S., DiVerdi, S., Funkhouser, T.: Learning part-based templates from large collections of 3D shapes. ACM Trans. Graph. (TOG) 32(4), 70 (2013)zbMATHGoogle Scholar
  19. 19.
    Li, Y., Wu, X., Chrysathou, Y., Sharf, A., Cohen-Or, D., Mitra, N.J.: Globfit: consistently fitting primitives by discovering global relations. In: ACM Transactions on Graphics (TOG), vol. 30, p. 52. ACM (2011)Google Scholar
  20. 20.
    Lin, H., Gao, J., Zhou, Y., Lu, G., Ye, M., Zhang, C., Liu, L., Yang, R.: Semantic decomposition and reconstruction of residential scenes from LiDAR data. ACM Trans. Graph. (TOG) 32(4), 66 (2013)Google Scholar
  21. 21.
    Martinovic, A., Van Gool, L.: Bayesian grammar learning for inverse procedural modeling. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 201–208 (2013)Google Scholar
  22. 22.
    Martinovic, A., Van Gool, L.: Hierarchical co-segmentation of building facades. In: 2014 2nd International Conference on 3D Vision (3DV), vol. 1, pp. 409–416. IEEE (2014)Google Scholar
  23. 23.
    Mathias, M., Martinovic, A., Weissenberg, J., Van Gool, L.: Procedural 3D building reconstruction using shape grammars and detectors. In: 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp. 304–311. IEEE (2011)Google Scholar
  24. 24.
    Merrell, P., Schkufza, E., Koltun, V.: Computer-generated residential building layouts. ACM Trans. Graph. (TOG) 29(6), 181 (2010)CrossRefGoogle Scholar
  25. 25.
    Mousavian, A., Anguelov, D., Flynn, J., Kosecka, J.: 3D bounding box estimation using deep learning and geometry. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  26. 26.
    Müller, P., Wonka, P., Haegler, S., Ulmer, A., Van Gool, L.: Procedural modeling of buildings. In: Acm Transactions On Graphics (Tog), vol. 25, pp. 614–623. ACM (2006)Google Scholar
  27. 27.
    Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of facades. ACM Trans. Graph. (TOG) 26(3), 85 (2007)CrossRefGoogle Scholar
  28. 28.
    Nguatem, W., Mayer, H.: Modeling urban scenes from pointclouds. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3857–3866, October 2017.
  29. 29.
    Nishida, G., Bousseau, A., Aliaga, D.G.: Procedural modeling of a building from a single image. Eurographics (2018)Google Scholar
  30. 30.
    Nishida, G., Garcia-Dorado, I., Aliaga, D.G., Benes, B., Bousseau, A.: Interactive sketching of urban procedural models. ACM Trans. Graph. (TOG) 35(4), 130 (2016)CrossRefGoogle Scholar
  31. 31.
    Quan, L., Tan, P., Zeng, G., Yuan, L., Wang, J., Kang, S.B.: Image-based plant modeling. In: ACM Transactions on Graphics (TOG), vol. 25, pp. 599–604. ACM (2006)Google Scholar
  32. 32.
    Ritchie, D., Thomas, A., Hanrahan, P., Goodman, N.: Neurally-guided procedural models: amortized inference for procedural graphics programs using neural networks. In: Advances In Neural Information Processing Systems, pp. 622–630 (2016)Google Scholar
  33. 33.
    Sinha, S., Steedly, D., Szeliski, R.: Piecewise planar stereo for image-based rendering. In: IEEE International Conference on Computer Vision (2009)Google Scholar
  34. 34.
    Talton, J.O., Lou, Y., Lesser, S., Duke, J., Měch, R., Koltun, V.: Metropolis procedural modeling. ACM Trans. Graph. (TOG) 30(2), 11 (2011)CrossRefGoogle Scholar
  35. 35.
    Tan, P., Zeng, G., Wang, J., Kang, S.B., Quan, L.: Image-based tree modeling. In: ACM Transactions on Graphics (TOG), vol. 26, p. 87. ACM (2007)CrossRefGoogle Scholar
  36. 36.
    Toshev, A., Mordohai, P., Taskar, B.: Detecting and parsing architecture at city scale from range data. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 398–405. IEEE (2010)Google Scholar
  37. 37.
    Wang, H., Zhang, W., Chen, Y., Chen, M., Yan, K.: Semantic decomposition and reconstruction of compound buildings with symmetric roofs from LiDAR data and aerial imagery. Remote Sens. 7(10), 13945–13974 (2015). Scholar
  38. 38.
    Xiao, J., Furukawa, Y.: Reconstructing the world’s museums. Int. J. Comput. Vis. 110(3), 243–258 (2014)CrossRefGoogle Scholar
  39. 39.
    Zebedin, L., Bauer, J., Karner, K., Bischof, H.: Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5305, pp. 873–886. Springer, Heidelberg (2008). Scholar
  40. 40.
    Zhou, Q.-Y., Neumann, U.: 2.5D dual contouring: a robust approach to creating building models from aerial LiDAR point clouds. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 115–128. Springer, Heidelberg (2010). Scholar
  41. 41.
    Zhou, Q.Y., Neumann, U.: 2.5D building modeling by discovering global regularities. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 326–333. IEEE (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Washington University in St. LouisSt. LouisUSA
  2. 2.Simon Fraser UniversityBurnabyCanada

Personalised recommendations