Semantically Aware Urban 3D Reconstruction with Plane-Based Regularization

  • Thomas HolzmannEmail author
  • Michael Maurer
  • Friedrich Fraundorfer
  • Horst Bischof
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11218)


We propose a method for urban 3D reconstruction, which incorporates semantic information and plane priors within the reconstruction process in order to generate visually appealing 3D models. We introduce a plane detection algorithm using 3D lines, which detects a more complete and less spurious plane set compared to point-based methods in urban environments. Further, the proposed normalized visibility-based energy formulation eases the combination of several energy terms within a tetrahedra occupancy labeling algorithm and, hence, is well suited for combining it with class specific smoothness terms. As a result, we produce visually appealing and detailed building models (i.e., straight edges and planar surfaces) and a smooth reconstruction of the surroundings.


Plane Detection Algorithm Plane Hypothesis Triple Line Poisson Surface Scene Parts 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was funded by the Austrian Science Fund (FWF) in the project V-MAV (I-1537). We thank Prof. Werner Lienhart and Slaven Kalenjuk from IGMS, TU Graz, Jesus Pestana and Christian Mostegel for providing datasets and Martin R. Oswald for discussion.

Supplementary material

474202_1_En_29_MOESM1_ESM.pdf (24.8 mb)
Supplementary material 1 (pdf 25356 KB)

Supplementary material 2 (mp4 32119 KB)


  1. 1.
    Blaha, M., Vogel, C., Richard, A., Wegner, J., Schindler, K., Pock, T.: Large-scale semantic 3D reconstruction: an adaptive multi-resolution model for multi-class volumetric labeling. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition (2016).
  2. 2.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Patt. Anal. Mach. Intell. 20(12), 1222–1239 (2001)CrossRefGoogle Scholar
  3. 3.
    CGAL. Computational Geometry Algorithms Library.
  4. 4.
    Dzitsiuk, M., Sturm, J., Maier, R., Ma, L., Cremers, D.: De-noising, stabilizing and completing 3D reconstructions on-the-go using plane priors. In: International Conference on Robotics and Automation, May 2017Google Scholar
  5. 5.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. IEEE Trans. Patt. Anal. Mach. Intell. 32(8), 1362–1376 (2010)CrossRefGoogle Scholar
  6. 6.
    Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: ACM Transactions on Graphics (SIGGRAPH), pp. 209–216. ACM Press/Addison-Wesley Publishing Co., New York (1997)Google Scholar
  7. 7.
    Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3D scene reconstruction and class segmentation. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition (2013)Google Scholar
  8. 8.
    Häne, C., Heng, L., Lee, G.H., Sizov, A., Pollefeys, M.: Real-time direct dense matching on fisheye images using plane-sweeping stereo. In: International Conference on 3D Vision (3DV) (2014)Google Scholar
  9. 9.
    Hofer, M., Maurer, M., Bischof, H.: Efficient 3D scene abstraction using line segments. Comput. Vis. Image Underst. 157, 167–178 (2016). Scholar
  10. 10.
    Holzmann, T., Fraundorfer, F., Bischof, H.: Regularized 3D modeling from noisy building reconstructions. In: Fourth International Conference on 3D Vision, 3DV 2016, Stanford, CA, USA, 25–28 October 2016, pp. 528–536 (2016)Google Scholar
  11. 11.
    Holzmann, T., Oswald, M.R., Pollefeys, M., Fraundorfer, F., Bischof, H.: Plane-based surface regularization for urban 3D reconstruction. In: 28th British Machine Vision Conference, vol. 28 (2017)Google Scholar
  12. 12.
    Hoppe, C., Klopschitz, M., Donoser, M., Bischof, H.: Incremental surface extraction from sparse structure-from-motion point clouds. In: Proceedings British Machine Vision Conference (2013)Google Scholar
  13. 13.
    Jia, Y., et al.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
  14. 14.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Eurographics Symposium on Geometry Processing (2006)Google Scholar
  15. 15.
    Korč, F., Förstner, W.: eTRIMS Image Database for interpreting images of man-made scenes. Technical report, TR-IGG-P-2009-01, April 2009.
  16. 16.
    Labatut, P., Pons, J.P., Keriven, R.: Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts. In: Proceedings International Conference on Computer Vision (2007)Google Scholar
  17. 17.
    Labatut, P., Pons, J.P., Keriven, R.: Hierarchical shape-based surface reconstruction for dense multi-view stereo. In: International Workshop on 3-D Digital Imaging and Modeling (3DIM), ICCV Workshops, Kyoto, Japan, pp. 1598–1605, October 2009Google Scholar
  18. 18.
    Lafarge, F., Alliez, P.: Surface reconstruction through point set structuring. Comput. Graph. Forum 32(2), 225–234 (2013)CrossRefGoogle Scholar
  19. 19.
    Lafarge, F., Keriven, R., Brédif, M., Vu, H.: A hybrid multiview stereo algorithm for modeling urban scenes. IEEE Trans. Patt. Anal. Mach. Intell. 35(1), 5–17 (2013)CrossRefGoogle Scholar
  20. 20.
    Li, M., Wonka, P., Nan, L.: Manhattan-world urban reconstruction from point clouds. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 54–69. Springer, Cham (2016). Scholar
  21. 21.
    Li, Y., Wu, X., Chrysanthou, Y., Sharf, A., Cohen-Or, D., Mitra, N.J.: GlobFit: consistently fitting primitives by discovering global relations. ACM Trans. Graph. 30(4), 52:1–52:12 (2011)Google Scholar
  22. 22.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  23. 23.
    Maurer, M., Hofer, M., Fraundorfer, F., Bischof, H.: Automated inspection of power line corridors to measure vegetation undercut using UAV-based images. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2017)CrossRefGoogle Scholar
  24. 24.
    Monszpart, A., Mellado, N., Brostow, G., Mitra, N.: RAPter: rebuilding man-made scenes with regular arrangements of planes. In: ACM SIGGRAPH 2015 (2015)Google Scholar
  25. 25.
    Nan, L., Wonka, P.: PolyFit: polygonal surface reconstruction from point clouds. In: Proceedings International Conference on Computer Vision (2017)Google Scholar
  26. 26.
    Oesau, S., Lafarge, F., Alliez, P.: Planar shape detection and regularization in tandem. Comput. Graph. Forum 35(1), 203–215 (2016)CrossRefGoogle Scholar
  27. 27.
  28. 28.
    Rothermel, M., Wenzel, K., Fritsch, D., Haala, N.: Sure: photogrammetric surface reconstruction from imagery. In: Proceedings LC3D Workshop, Berlin (2012)Google Scholar
  29. 29.
    Schnabel, R., Wahl, R., Klein, R.: Efficient ransac for point-cloud shape detection. Comput. Graph. Forum 26(2), 214–226 (2007)CrossRefGoogle Scholar
  30. 30.
    Strecha, C., von Hansen, W., Gool, L.V., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition (2008)Google Scholar
  31. 31.
  32. 32.
    Vogel, C., Richard, A., Pock, T., Schindler, K.: Semantic 3D reconstruction with finite element bases. In: 28th British Machine Vision Conference, vol. 28 (2017)Google Scholar
  33. 33.
    Waechter, M., Moehrle, N., Goesele, M.: Let there be color! Large-scale texturing of 3D reconstructions. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 836–850. Springer, Cham (2014). Scholar
  34. 34.
    Xiao, J., Furukawa, Y.: Reconstructing the world’s museums. Int. J. Comput. Vis. 110(3), 243–258 (2014)CrossRefGoogle Scholar
  35. 35.
    Zebedin, L., Bauer, J., Karner, K., Bischof, H.: Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5305, pp. 873–886. Springer, Heidelberg (2008). Scholar
  36. 36.
    Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: IEEE Computer Vision and Pattern Recognition (CVPR), pp. 1529–1537 (2015)., arXiv: 1502.03240

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Thomas Holzmann
    • 1
    Email author
  • Michael Maurer
    • 1
  • Friedrich Fraundorfer
    • 1
  • Horst Bischof
    • 1
  1. 1.Institute of Computer Graphics and VisionGraz University of TechnologyGrazAustria

Personalised recommendations