3D Reconstruction Using an n-Layer Heightmap

  • David Gallup
  • Marc Pollefeys
  • Jan-Michael Frahm
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6376)


We present a novel method for 3D reconstruction of urban scenes extending a recently introduced heightmap model. Our model has several advantages for 3D modeling of urban scenes: it naturally enforces vertical surfaces, has no holes, leads to an efficient algorithm, and is compact in size. We remove the major limitation of the heightmap by enabling modeling of overhanging structures. Our method is based on an an n-layer heightmap with each layer representing a surface between full and empty space. The configuration of layers can be computed optimally using a dynamic programming method. Our cost function is derived from probabilistic occupancy, and incorporates the Bayesian Information Criterion (BIC) for selecting the number of layers to use at each pixel. 3D surface models are extracted from the heightmap. We show results from a variety of datasets including Internet photo collections. Our method runs on the GPU and the complete system processes video at 13 Hz.


Bayesian Information Criterion Occupancy Grid Probabilistic Occupancy Photo Collection Urban Scene 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cornelis, N., Leibe, B., Cornelis, K., Gool, L.V.: 3d urban scene modeling integrating recognition and reconstruction. IJCV (2008)Google Scholar
  2. 2.
    Früh, C., Jain, S., Zakohr, A.: Data processing algorithms for generating textured 3d building facade meshes from laser scans and camera images. IJCV (2005)Google Scholar
  3. 3.
    Pollefeys, M., et al.: Detailed real-time urban 3d reconstruction from video. Int. Journal of Computer Vision (IJCV) (2008)Google Scholar
  4. 4.
    Zebedin, L., Bauer, J., Karner, K., Bischof, H.: Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 873–886. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Xiao, J., Quan, L.: Image-based street-side city modeling. In: Siggraph Asia (2009)Google Scholar
  6. 6.
    Gallup, D., Frahm, J.M., Pollefeys, M.: Piecewise planar and non-planar stereo for urban scene reconstruction. In: CVPR (2010)Google Scholar
  7. 7.
    Zach, C., Pock, T., Bischof, H.: A globally optimal algorithm for robust tv-l1 range image integration. In: ICCV (2007)Google Scholar
  8. 8.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. In: SIGGRAPH, pp. 835–846 (2006)Google Scholar
  9. 9.
    Li, X., Wu, C., Zach, C., Lazebnik, S., Frahm, J.M.: Modeling and recognition of landmark image collections using iconic scene graphs. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 427–440. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  10. 10.
    Agarwal, S., Snavely, N., Simon, I., Seitz, S.M., Szeliski, R.: Building Rome in a day. In: ICCV (2009)Google Scholar
  11. 11.
    Goesele, M., Snavely, N., Curless, B., Hoppe, H., Seitz, S.M.: Multi-view stereo for community photo collections. In: ICCV (2007)Google Scholar
  12. 12.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards internet-scale multi-view stereo. In: CVPR (2010)Google Scholar
  13. 13.
    Cornelis, N., Cornelis, K., Van Gool, L.: Fast compact city modeling for navigation pre-visualization. In: Computer Vision and Pattern Recognition (CVPR) (2006)Google Scholar
  14. 14.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Manhatten-world stereo. In: Proceedings IEEE CVPR (2009)Google Scholar
  15. 15.
    Sinha, S.N., Steedly, D., Szeliski, R.: Piecewise planar stereo for image-based rendering. In: Proceedings IEEE ICCV (2009)Google Scholar
  16. 16.
    Margaritis, D., Thrun, S.: Learning to locate an object in 3d space from a sequence of camera images. In: ICML (1998)Google Scholar
  17. 17.
    Pathak, K., Birk, A., Poppinga, J., Schwertfeger, S.: 3d forward sensor modeling and application to occupancy grid based sensor fusion. In: IROS (2007)Google Scholar
  18. 18.
    Guan, L., Franco, J.S., Pollefeys, M.: 3d object reconstruction with heterogeneous sensor data. In: 3DPVT (2008)Google Scholar
  19. 19.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards internet-scale multi-view stereo. In: CVPR (2010)Google Scholar
  20. 20.
    Merrell, P., Akbarzadeh, A., Wang, L., Mordohai, P., Frahm, J.M., Yang, R., Nister, D., Pollefeys, M.: Real-Time Visibility-Based Fusion of Depth Maps. In: Proceedings of International Conf. on Computer Vision (2007)Google Scholar
  21. 21.
    Szeliski, R.: Image alignment and stitching: A tutorial. Microsoft Research Technical Report (2005)Google Scholar
  22. 22.
    Cormen, T.H., Leisorson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. The MIT Press, Cambridge (2001)zbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • David Gallup
    • 1
  • Marc Pollefeys
    • 2
  • Jan-Michael Frahm
    • 1
  1. 1.Department of Computer ScienceUniversity of North Carolina 
  2. 2.Department of Computer ScienceETH Zurich 

Personalised recommendations