Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

A robust three-stage approach to large-scale urban scene recognition

  • 250 Accesses

  • 3 Citations

Abstract

To obtain the ultimate high-level description of urban scenes, we propose a three-stage approach to recognizing the 3D reconstructed scene with efficient representations. First, we develop a joint semantic labeling method to obtain a semantic labeling of the triangular mesh-based representation by exploiting both image features and geometric features. The labeling is formulated over a conditional random field (CRF) that incorporates local spacial smoothness and multi-view consistency. Then, based on the labeled reconstructed meshes, we refine the man-made object segmentation in the recomposed global orthographic map with a graph partition algorithm, and propagate the coherent segmentation to the entire 3D meshes. Finally, we propose to generate a compact, abstracted geometric representation for each man-made object which is more visually appealing than the original cluttered models. This abstraction algorithm also leverages CRF formation to partition building footprints into minimal sets of structural linear features which are then used to construct profiles for large-scale scenes. The proposed recognition approach is able to robustly handle reconstructions with poor geometry and connectivity, thanks to the higher order CRF formulations which impose the ubiquitous regularity priors in urban scenes. Each stage performs an individual and uncoupling task. The intensive experiments have demonstrated the superior performance of our approach in robustness, accuracy and applicability.

This is a preview of subscription content, log in to check access.

References

  1. 1

    Ochmann S, Vock R, Wessel R, et al. Automatic generation of structural building descriptions from 3d point cloud scans. In: Proceedings of International Conference on Computer Graphics Theory and Applications (GRAPP), Lisbon, 2014. 1–8

  2. 2

    Yu Z D, Xu C J, Liu J Z, et al. Automatic object segmentation from large scale 3d urban point clouds through manifold embedded mode seeking. In: Proceedings of the 19th ACM International Conference on Multimedia, Scottsdale, 2011. 1297–1300

  3. 3

    Matei B C, Sawhney H S, Samarasekera S, et al. Building segmentation for densely built urban regions using aerial LIDAR data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, 2008. 1–8

  4. 4

    Drauschke M, Schuster H-F, Förstner W. Detectability of buildings in aerial images over scale space. In: Proceedings of Conference on Photogrammetric Computer Vision, Dresden, 2006. 7–12

  5. 5

    Mayer H. Automatic object extraction from aerial imagery—a survey focusing on buildings. Comput Vis Image Underst, 1999, 74: 138–149

  6. 6

    Suveg I, Vosselman G. Reconstruction of 3d building models from aerial images and maps. ISPRS J Photogramm Remote Sens, 2004, 58: 202–224

  7. 7

    Kraus K, Pfeifer N. Determination of terrain models in wooded areas with airborne laser scanner data. ISPRS J Photogramm Remote Sens, 1998, 53: 193–203

  8. 8

    Verma V, Kumar R, Hsu S. 3d building detection and modeling from aerial LIDAR data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New York, 2006. 2213–2220

  9. 9

    Zhang H H, Wang J L, Fang T, et al. Joint segmentation of images and scanned point cloud in large-scale street scenes with low-annotation cost. IEEE Trans Image Process, 2014, 23: 4763–4772

  10. 10

    Brédif M, Boldo D, Deseilligny M P, et al. 3d building reconstruction with parametric roof superstructures. In: Proceedings of 14th IEEE International Conference on Image Processing (ICIP), San Antonio, 2007. 537–540

  11. 11

    Rottensteiner F, Trinder J, Clode S, et al. Automated delineation of roof planes from LIDAR data. In: Proceedings of ISPRS Workshop Laser Scanning 2005, Enschede, 2005. 221–226

  12. 12

    Guo Y L, Sawhney H S, Kumar R, et al. Learning-based building outline detection from multiple aerial images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, 2001. II-545–II-552

  13. 13

    Lafarge F, Descombes X, Zerubia J, et al. Structural approach for building reconstruction from a single DSM. IEEE Trans Pattern Anal Mach Intell, 2010, 32: 135–147

  14. 14

    Wang J L, Fang T, Su Q K, et al. Image-based building regularization using structural linear features. IEEE Trans Vis Comput Graph, 2016, 22: 1760–1772

  15. 15

    Liu J B, Wang J L, Fang T, et al. Higher-order CRF structural segmentation of 3d reconstructed surfaces. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Santiago, 2015. 2093–2101

  16. 16

    Zhou Q Y, Neumann U. 2.5 d building modeling by discovering global regularities. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, 2012. 326–333

  17. 17

    Malik J, Belongie S, Leung T, et al. Contour and texture analysis for image segmentation. Int J Comput Vision, 2001, 43: 7–27

  18. 18

    Kohli P, Ladicky L, Torr P. Robust higher order potentials for enforcing label consistency. Int J Comput Vision, 2009, 82: 302–324

  19. 19

    Liu J B, Wang J L, Fang T, et al. Higher-order CRF structural segmentation of 3d reconstructed surfaces. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Santiago, 2015. 2093–2101

  20. 20

    Comaniciu D, Meer P. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell, 2002, 24: 603–619

  21. 21

    Ladicky L, Russell C, Kohli P, et al. Associative hierarchical CRFs for object class image segmentation. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Kyoto, 2009. 739–746

  22. 22

    Shotton J, Johnson M, Cipolla R. Semantic texton forests for image categorization and segmentation. In: Criminisi A, Shotton J, eds. Decision Forests for Computer Vision and Medical Image Analysis. London: Springer, 2008

  23. 23

    Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Trans Pattern Anal Mach Intell, 2001, 23: 1222–1239

  24. 24

    Shi J, Malik J. Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell, 2000, 22: 888–905

  25. 25

    Suzuki S. Topological structural analysis of digitized binary images by border following. Comput Vision Graph Image Process, 1985, 30: 32–46

  26. 26

    Ester M, Kriegel H-P, Sander J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland. 1996. 226–231

  27. 27

    Kohli P, Kumar M P, Torr P H S. P3 & beyond: solving energies with higher order cliques. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, 2007. 1–8

  28. 28

    Lhuillier M, Quan L. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Trans Pattern Anal Mach Intell, 2005, 27: 418–433

  29. 29

    Kazhdan M, Bolitho M, Hoppe H. Poisson surface reconstruction. In: Proceedings of Eurographics Symposium on Geometry Processing, Cagliari, 2006. 61–70

  30. 30

    Sinha S N, Steedly D, Szeliski R, et al. Interactive 3d architectural modeling from unordered photo collections. ACM Trans Graph, 2008, 27: 159

  31. 31

    Anguelov D, Taskarf B, Chatalbashev V, et al. Discriminative learning of Markov random fields for segmentation of 3d scan data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Diego, 2005. 169–176

  32. 32

    Munoz D, Bagnell J A, Vandapel N, et al. Contextual classification with functional max-margin Markov networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 975–982

  33. 33

    Zhang H H, Xiao J X, Quan L. Supervised label transfer for semantic segmentation of street scenes. In: Proceedings of the 11th European Conference on Computer Vision, Heraklion, 2009. 561–574

Download references

Author information

Correspondence to Jinglu Wang.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Lu, Y., Liu, J. et al. A robust three-stage approach to large-scale urban scene recognition. Sci. China Inf. Sci. 60, 103101 (2017). https://doi.org/10.1007/s11432-017-9178-8

Download citation

Keywords

  • 3D reconstruction
  • urban
  • scene recongition
  • semantic segmentation
  • CRF