Advertisement

A robust three-stage approach to large-scale urban scene recognition

  • Jinglu WangEmail author
  • Yonghua Lu
  • Jingbo Liu
  • Long Quan
Moop

Abstract

To obtain the ultimate high-level description of urban scenes, we propose a three-stage approach to recognizing the 3D reconstructed scene with efficient representations. First, we develop a joint semantic labeling method to obtain a semantic labeling of the triangular mesh-based representation by exploiting both image features and geometric features. The labeling is formulated over a conditional random field (CRF) that incorporates local spacial smoothness and multi-view consistency. Then, based on the labeled reconstructed meshes, we refine the man-made object segmentation in the recomposed global orthographic map with a graph partition algorithm, and propagate the coherent segmentation to the entire 3D meshes. Finally, we propose to generate a compact, abstracted geometric representation for each man-made object which is more visually appealing than the original cluttered models. This abstraction algorithm also leverages CRF formation to partition building footprints into minimal sets of structural linear features which are then used to construct profiles for large-scale scenes. The proposed recognition approach is able to robustly handle reconstructions with poor geometry and connectivity, thanks to the higher order CRF formulations which impose the ubiquitous regularity priors in urban scenes. Each stage performs an individual and uncoupling task. The intensive experiments have demonstrated the superior performance of our approach in robustness, accuracy and applicability.

Keywords

3D reconstruction urban scene recongition semantic segmentation CRF 

Supplementary material

11432_2017_9178_MOESM1_ESM.pdf (2.7 mb)
A Robust Three-stage Approach To Large-Scale Urban Scene Recognition

Supplementary material, approximately 25.8 MB.

Supplementary material, approximately 83.3 MB.

References

  1. 1.
    Ochmann S, Vock R, Wessel R, et al. Automatic generation of structural building descriptions from 3d point cloud scans. In: Proceedings of International Conference on Computer Graphics Theory and Applications (GRAPP), Lisbon, 2014. 1–8Google Scholar
  2. 2.
    Yu Z D, Xu C J, Liu J Z, et al. Automatic object segmentation from large scale 3d urban point clouds through manifold embedded mode seeking. In: Proceedings of the 19th ACM International Conference on Multimedia, Scottsdale, 2011. 1297–1300CrossRefGoogle Scholar
  3. 3.
    Matei B C, Sawhney H S, Samarasekera S, et al. Building segmentation for densely built urban regions using aerial LIDAR data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, 2008. 1–8Google Scholar
  4. 4.
    Drauschke M, Schuster H-F, Förstner W. Detectability of buildings in aerial images over scale space. In: Proceedings of Conference on Photogrammetric Computer Vision, Dresden, 2006. 7–12Google Scholar
  5. 5.
    Mayer H. Automatic object extraction from aerial imagery—a survey focusing on buildings. Comput Vis Image Underst, 1999, 74: 138–149CrossRefGoogle Scholar
  6. 6.
    Suveg I, Vosselman G. Reconstruction of 3d building models from aerial images and maps. ISPRS J Photogramm Remote Sens, 2004, 58: 202–224CrossRefGoogle Scholar
  7. 7.
    Kraus K, Pfeifer N. Determination of terrain models in wooded areas with airborne laser scanner data. ISPRS J Photogramm Remote Sens, 1998, 53: 193–203CrossRefGoogle Scholar
  8. 8.
    Verma V, Kumar R, Hsu S. 3d building detection and modeling from aerial LIDAR data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New York, 2006. 2213–2220Google Scholar
  9. 9.
    Zhang H H, Wang J L, Fang T, et al. Joint segmentation of images and scanned point cloud in large-scale street scenes with low-annotation cost. IEEE Trans Image Process, 2014, 23: 4763–4772MathSciNetCrossRefGoogle Scholar
  10. 10.
    Brédif M, Boldo D, Deseilligny M P, et al. 3d building reconstruction with parametric roof superstructures. In: Proceedings of 14th IEEE International Conference on Image Processing (ICIP), San Antonio, 2007. 537–540Google Scholar
  11. 11.
    Rottensteiner F, Trinder J, Clode S, et al. Automated delineation of roof planes from LIDAR data. In: Proceedings of ISPRS Workshop Laser Scanning 2005, Enschede, 2005. 221–226Google Scholar
  12. 12.
    Guo Y L, Sawhney H S, Kumar R, et al. Learning-based building outline detection from multiple aerial images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, 2001. II-545–II-552Google Scholar
  13. 13.
    Lafarge F, Descombes X, Zerubia J, et al. Structural approach for building reconstruction from a single DSM. IEEE Trans Pattern Anal Mach Intell, 2010, 32: 135–147CrossRefGoogle Scholar
  14. 14.
    Wang J L, Fang T, Su Q K, et al. Image-based building regularization using structural linear features. IEEE Trans Vis Comput Graph, 2016, 22: 1760–1772CrossRefGoogle Scholar
  15. 15.
    Liu J B, Wang J L, Fang T, et al. Higher-order CRF structural segmentation of 3d reconstructed surfaces. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Santiago, 2015. 2093–2101Google Scholar
  16. 16.
    Zhou Q Y, Neumann U. 2.5 d building modeling by discovering global regularities. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, 2012. 326–333Google Scholar
  17. 17.
    Malik J, Belongie S, Leung T, et al. Contour and texture analysis for image segmentation. Int J Comput Vision, 2001, 43: 7–27CrossRefzbMATHGoogle Scholar
  18. 18.
    Kohli P, Ladicky L, Torr P. Robust higher order potentials for enforcing label consistency. Int J Comput Vision, 2009, 82: 302–324CrossRefGoogle Scholar
  19. 19.
    Liu J B, Wang J L, Fang T, et al. Higher-order CRF structural segmentation of 3d reconstructed surfaces. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Santiago, 2015. 2093–2101Google Scholar
  20. 20.
    Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell, 2002, 24: 603–619CrossRefGoogle Scholar
  21. 21.
    Ladicky L, Russell C, Kohli P, et al. Associative hierarchical CRFs for object class image segmentation. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Kyoto, 2009. 739–746Google Scholar
  22. 22.
    Shotton J, Johnson M, Cipolla R. Semantic texton forests for image categorization and segmentation. In: Criminisi A, Shotton J, eds. Decision Forests for Computer Vision and Medical Image Analysis. London: Springer, 2008Google Scholar
  23. 23.
    Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell, 2001, 23: 1222–1239CrossRefGoogle Scholar
  24. 24.
    Shi J, Malik J. Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell, 2000, 22: 888–905CrossRefGoogle Scholar
  25. 25.
    Suzuki S. Topological structural analysis of digitized binary images by border following. Comput Vision Graph Image Process, 1985, 30: 32–46CrossRefzbMATHGoogle Scholar
  26. 26.
    Ester M, Kriegel H-P, Sander J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland. 1996. 226–231Google Scholar
  27. 27.
    Kohli P, Kumar M P, Torr P H S. P3 & beyond: solving energies with higher order cliques. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, 2007. 1–8Google Scholar
  28. 28.
    Lhuillier M, Quan L. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Trans Pattern Anal Mach Intell, 2005, 27: 418–433CrossRefGoogle Scholar
  29. 29.
    Kazhdan M, Bolitho M, Hoppe H. Poisson surface reconstruction. In: Proceedings of Eurographics Symposium on Geometry Processing, Cagliari, 2006. 61–70Google Scholar
  30. 30.
    Sinha S N, Steedly D, Szeliski R, et al. Interactive 3d architectural modeling from unordered photo collections. ACM Trans Graph, 2008, 27: 159CrossRefGoogle Scholar
  31. 31.
    Anguelov D, Taskarf B, Chatalbashev V, et al. Discriminative learning of Markov random fields for segmentation of 3d scan data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Diego, 2005. 169–176Google Scholar
  32. 32.
    Munoz D, Bagnell J A, Vandapel N, et al. Contextual classification with functional max-margin Markov networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 975–982Google Scholar
  33. 33.
    Zhang H H, Xiao J X, Quan L. Supervised label transfer for semantic segmentation of street scenes. In: Proceedings of the 11th European Conference on Computer Vision, Heraklion, 2009. 561–574Google Scholar

Copyright information

© Science China Press and Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Jinglu Wang
    • 1
    Email author
  • Yonghua Lu
    • 2
  • Jingbo Liu
    • 1
  • Long Quan
    • 1
  1. 1.Department of Computer Science and EngineeringThe Hong Kong University of Science and TechnologyHong KongChina
  2. 2.School of Resource and Environmental SciencesWuhan UniversityWuhanChina

Personalised recommendations