An Automation System of Rooftop Detection and 3D Building Modeling from Aerial Images

  • Fanhuai Shi
  • Yongjian Xi
  • Xiaoling Li
  • Ye Duan


This paper presents a prototype system of rooftop detection and 3D building modeling from aerial images. In this system, without the knowledge of the position and orientation information of the aerial vehicle a priori, the parameters of the camera pose and ground plane are first estimated by simple human–computer interaction. Next, after an over-segmentation of the aerial image by the Mean-Shift algorithm, the rooftop regions are coarsely detected by integrating multi-scale SIFT-like feature vectors with SVM-based visual object recognition. 2D cues alone however might not always be sufficient to separate regions such as parking lots from building roofs. Thus in order to further refine the accuracy of the roof-detection result and remove the misclassified non-rooftop regions such as parking lots, we further resort to 3D depth information estimated based on multi-view geometry. More specifically, we determine whether a candidate region is a rooftop or not according to its height information relative to the ground plane, whereas the candidate region’s height information is obtained by a novel, hierarchical, asymmetry correlation-based corner matching scheme. The output of the system will be a water-tight triangle mesh based 3D building model texture mapped with the aerial images. We developed an interactive 3D viewer based on OpenGL and C+ + to allow the user to virtually navigate the reconstructed 3D scene with mouse and keyboard. Experimental results are shown on real aerial scenes.


Image-based modeling Asymmetry corner matching 3D reconstruction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baillard, C., Schmid, C., Zisserman, A., Fitzgibbon, A.: Automatic line matching and 3D reconstruction of buildings from multiple views. In: Proc. of ISPRS Conference on Automatic Extraction of GIS Objects from Digital Imagery, IAPRS, vol. 32, Part 3-2W5, pp. 69–80 (1999)Google Scholar
  2. 2.
    Bouguet, J.Y.: Camera Calibration Toolbox for Matlab (2008). Available at:
  3. 3.
    Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–714 (1986)CrossRefGoogle Scholar
  4. 4.
    Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)CrossRefGoogle Scholar
  5. 5.
    Elgammal, A., Harwood, D., Davis, L.: Non-parametric model for background subtraction. In: Proc. of ECCV’00, pp. 751–767 (2000)Google Scholar
  6. 6.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Franc, V., Hlavac, V.: Statistical Pattern Recognition Toolbox for Matlab (2008). Available at:
  8. 8.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, UK (2000)zbMATHGoogle Scholar
  9. 9.
    Hu, J., You, S., Neumann, U.: Integrating LiDAR, aerial image and ground images for complete urban building modeling. In: Proc. of 3DPVT, pp. 184–191 (2006)Google Scholar
  10. 10.
    Jaynes, C., Riseman, E., Hanson, A.: Recognition and reconstruction of buildings from multiple aerial images. Comput. Vis. Image Underst. 90(1), 68–98 (2003)CrossRefGoogle Scholar
  11. 11.
    Kim, Z.W., Nevatia, R.: Automatic description of complex buildings from multiple images. Comput. Vis. Image Underst. 96(1), 60–95 (2004)CrossRefGoogle Scholar
  12. 12.
    Kovesi, P.: Edge Linking and Line Segment Fitting (2007). Available at:
  13. 13.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2(60), 91–110 (2004)CrossRefGoogle Scholar
  14. 14.
    Maloof, M.A., Langley, P., Binford, T.O., Nevatia, R., Sage, S.: Improved rooftop detection in aerial images with machine learning. Mach. Learn. 53(1–2), 157–191 (2003)CrossRefGoogle Scholar
  15. 15.
    Mayer, H.: Automatic object extraction from aerial imagery—a survey focusing on buildings. Comput. Vis. Image Underst. 74(2), 138–149 (1999)CrossRefGoogle Scholar
  16. 16.
    Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 756–770 (2004)CrossRefGoogle Scholar
  17. 17.
    Porway, J., Wang, K., Yao, B., Zhu, S.C.: A hierarchical and contextual model for aerial image understanding. In: Proc. CVPR’08, pp. 1–8. Anchorage, Alaska (2008)Google Scholar
  18. 18.
    Verma, V., Kumar, R., Hsu, S.: 3D building detection and modeling from aerial LIDAR data. In: Proc. CVPR’06, vol. 2, pp. 2213–2220 (2006)Google Scholar
  19. 19.
    Vestri, C., Devernay, F.: Using robust methods for automatic extraction of buildings. In: Proc. CVPR’01, vol. 1, pp. 133–138 (2001)Google Scholar
  20. 20.
    Wei, L., Prinet, V.: Building detection from high-resolution satellite image using probability model, geoscience and remote sensing symposium, In: Proc. of IGARSS’05, pp. 25–29 (2005)Google Scholar
  21. 21.
    Zhang, Z., Deriche, R., Faugeras, O., Luong, Q.T.: A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artif. Intell. J. 78(1–2), 87–119 (1995)CrossRefGoogle Scholar
  22. 22.
    Zhao, F., Huang, Q., Gao, W.: Image matching by normalized cross-correlation. In: Proc. of ICASSP’06, vol. 2, pp. 729–732. Toulouse, France (2006)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Fanhuai Shi
    • 1
    • 2
  • Yongjian Xi
    • 1
  • Xiaoling Li
    • 1
  • Ye Duan
    • 1
  1. 1.University of Missouri-ColumbiaColumbiaUSA
  2. 2.Shanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations