International Journal of Computer Vision

, Volume 61, Issue 2, pp 159–184 | Cite as

Data Processing Algorithms for Generating Textured 3D Building Facade Meshes from Laser Scans and Camera Images

  • Christian Frueh
  • Siddharth Jain
  • Avideh Zakhor


In this paper, we develop a set of data processing algorithms for generating textured facade meshes of cities from a series of vertical 2D surface scans and camera images, obtained by a laser scanner and digital camera while driving on public roads under normal traffic conditions. These processing steps are needed to cope with imperfections and non-idealities inherent in laser scanning systems such as occlusions and reflections from glass surfaces. The data is divided into easy-to-handle quasi-linear segments corresponding to approximately straight driving direction and sequential topological order of vertical laser scans; each segment is then transformed into a depth image. Dominant building structures are detected in the depth images, and points are classified into foreground and background layers. Large holes in the background layer, caused by occlusion from foreground layer objects, are filled in by planar or horizontal interpolation. The depth image is further processed by removing isolated points and filling remaining small holes. The foreground objects also leave holes in the texture of building facades, which are filled by horizontal and vertical interpolation in low frequency regions, or by a copy-paste method otherwise. We apply the above steps to a large set of data of downtown Berkeley with several million 3D points, in order to obtain texture-mapped 3D models.

3D city model occlusion hole filling image restoration texture synthesis urban simulation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., and Verdera, J. 2001. Filling in by joint interpolation of vector fields and gray levels. In IEEE Transactions on Image Processing, pp. 1200-1211.Google Scholar
  2. Ballester, C., Caselles, V., Verdera, J., Bertalmio, M., and Sapiro, G. A variational model for filling-in gray level and color images. In Proc. 8th IEEE Int’l Conference on Computer Vision, vol. 1, pp. 10-16.Google Scholar
  3. Bertalmio, M., Sapiro, G., Ballester, C., and Caselles, V. 2000. Image inpainting. In Proc. SIGGRAPH 2000, pp. 417-424.Google Scholar
  4. Chan, T. and Shen, J. 2001. Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics 62(3):1019-1043.CrossRefGoogle Scholar
  5. Chang, N.L. and Zakhor, A. 1999. A multivalued representation for view synthesis. In Proc. Int’l Conference on Image Processing, Kobe, Japan, vol. 2, pp. 505-509.Google Scholar
  6. Curless, B. and Levoy, M. A volumetric method for building com-plex models from range images. In SIGGRAPIL, New Orleans, pp. 303-312.Google Scholar
  7. Debevec, P.E., Taylor, C.J., and Malik, J. 1996. Modeling and ren-dering architecture from photographs. In Proc. ACM SIGGRAPH.Google Scholar
  8. Dick, A., Torr, P., Ruffle, S., and Cipolla, R. 2001. Combining single view recognition and multiple view stereo for architectural scenes. In International Conference on Computer, Vision, Vancouver, Canada, pp. 268-274.Google Scholar
  9. Elfros, A. and Freeman W. 2001. Image quilting for texture synthesis and transfer. In Proc. SIGGRAPI, pp. 341-346.Google Scholar
  10. Fox, D., Thrun, S, Dellaert, E. and Burgard, W. 2000. Particle filters for mobile robot localization. In Sequential Monte Carlo Methods in Practice, A. Doucet, N. de Freitas, and N. Gordon (Eds.), Springer Verlag, New York.Google Scholar
  11. Frere, D., Vandekerckhove, J., Moons, T., and Van Gool, L. 1998. Automatic modelling and 3D reconstruction of urban build-ings from aerial imagery. In IEEE International Geoscience and Remote Sensing Symposium Proceedings, Seattle, pp. 2593-2596.Google Scholar
  12. Frueh, C. 2002. Automated 3D Model Generation for Urban Environments. Ph.D. Thesis, University of Karlsruhe.Google Scholar
  13. Frueh, C., Flynn, J., Foroosh, HI., and Zakhor, A. 2001. Fast 3D model generation in urban environments. In Workshop on the Conver-gence of Graphics, Vision, and Video (CGW’OI), Berkeley, USA.Google Scholar
  14. Frueh, C. and Zakhor, A. 2001a. Fast 3D model generation in urban environments. In IEEE Conf on Multisensor Fusion and Integra-tionfor Intelligent Systems, Baden-Baden, Gemany, pp. 165-170.Google Scholar
  15. Frueh, C. and Zakhor, A. 2001b. 3D model generation of cities using aerial photographs and ground level laser scans. In Computer Vi-sion and Pattern Recognition, Hawaii, USA, vol. 2.2. pp. II-31-8.Google Scholar
  16. Frueh, C. and Zakhor, A. 2003. Constructing 3D city models by merging ground-based and airborne views. IEEE Computer Graphics and Applications, Special Issue Nov/Dlec. pp. 52-41.Google Scholar
  17. Garland, M. and EIeckbert, P. 1997. Surface Simplification Using Quadric Error Metrics. In SIGGRAPH ‘97, Los Angeles, pp. 209-216.Google Scholar
  18. Hlaala, N. and Brenner, C. 1997. Generation of 3D city models from airborne laser scanning data. In Proc. EARSEL Workshop on LIDAR Remnote Sensing on Land and Sea, Tallin, Esonia, pp. 105-112.Google Scholar
  19. Kim, Z., Huertas, A., and Nevatia, R. 2001. Automatic description of Buildings with complex rooftops from multiple images. In Computer Vision ard Pattern Recognition, Kauai, pp. 272-279.Google Scholar
  20. Koch, R., Pollefeys, M., and van Gool, L. 1999. Realistic 3D Scene modeling from uncalibrated image sequences. In ICIP’99, Kobe, Japan, Oct., pp. II: 500-504.Google Scholar
  21. Maas, H.-G. 2001. The suitability of airborne laser scanner data for automatic 3D object reconstruction. 3. In Int’l Workshop on Automatic Extraction of Man-Made Objects, Ascona, Switzerland.Google Scholar
  22. Masnou, S. and Morel, J. 1998. Level-lines based disocclusion. In 5th IEEE Int’l Conference on Image Processing, pp. 259-263.Google Scholar
  23. Nitzberg, M., Mumford, D., and Shiota, T. 1993. Filtering, Segmentation and Depth. Springer-Verlag: Berlin.Google Scholar
  24. Stamos, I. and Allen, PK. 2002. Geometry and texture recov-ery of scenes of large scale. In Computer Vision and Image Understanding (CVIU), 88(2):94-118.CrossRefGoogle Scholar
  25. Stulp, F., Dell’Acqua, F., and Fisher, R.B. 2001. Reconstruction of surfaces behind occlusions in range images. In Proc. 3rd Int. Conf. on 3-D Digital Imaging and Modeling, Montreal, Canada, pp. 232-239.Google Scholar
  26. Thrun, S., Burgard, W., and Fox, D. 2000. A real-time algorithm for mobile robot mapping with applications to multi-robot and 3D mapping. In Proc. of International Conference on Robotics and Automation, San Francisco, vol. 1. pp. 321-328.Google Scholar
  27. Wang, X., Totaro, S., Taillandier, IR, Hanson, A., and Teller, S. 2002. Recovering facade texture and microstructure from real-world images. In Proc. 2nd International Workshop on Texture Analysis and Synthesis in Conjunction with European Conference on Computer Vision, pp. 145-149.Google Scholar
  28. Zhao, H. and Shibasaki, R. 1999. A system for reconstructing urban 3D objects using ground-based range and CCD images. In Proc. of International Workshop on Urban Multi-Media/3D Mapping, Tokyo.Google Scholar
  29. Zhao, H. and Shibasaki, R. 2001. Reconstructing urban 3D model using vehicle-borne laser range scanners. In Proc. of the 3rd International Conference on 3D Digital Imaging and Modeling, Quebec, Canada.Google Scholar

Copyright information

© Kluwer Academic Publishers 2005

Authors and Affiliations

  • Christian Frueh
    • 1
  • Siddharth Jain
    • 1
  • Avideh Zakhor
    • 1
  1. 1.Video and Image Processing Laboratory, Department of Electrical Engineering and Computer SciencesUniversity of CaliforniaBerkeleyUSA

Personalised recommendations