Advertisement

Synthesis and Completion of Facades from Satellite Imagery

Conference paper
  • 799 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12347)

Abstract

Automatic satellite-based reconstruction enables large and widespread creation of urban areas. However, satellite imagery is often noisy and incomplete, and is not suitable for reconstructing detailed building facades. We present a machine learning-based inverse procedural modeling method to automatically create synthetic facades from satellite imagery. Our key observation is that building facades exhibit regular, grid-like structures. Hence, we can overcome the low-resolution, noisy, and partial building data obtained from satellite imagery by synthesizing the underlying facade layout. Our method infers regular facade details from satellite-based image-fragments of a building, and applies them to occluded or under-sampled parts of the building, resulting in plausible, crisp facades. Using urban areas from six cities, we compare our approach to several state-of-the-art image completion/in-filling methods and our approach consistently creates better facade images.

Keywords

Image synthesis and completion Inverse procedural modeling Satellite imagery 

Notes

Acknowledgements

This research was supported in part by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number D17PC00280. Additional support came from National Science Foundation grants #10001387 and #1835739.

Supplementary material

504434_1_En_34_MOESM1_ESM.pdf (951 kb)
Supplementary material 1 (pdf 950 KB)

References

  1. 1.
    Bokeloh, M., Wand, M., Seidel, H.P.: A connection between partial symmetry and inverse procedural modeling. ACM Trans. Graph. 29 (2010) Google Scholar
  2. 2.
    Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_49CrossRefGoogle Scholar
  3. 3.
    Cohen, A., Schwing, A.G., Pollefeys, M.: Efficient structured parsing of facades using dynamic programming. In: IEEE Computer Vision and Pattern Recognition, pp. 3206–3213 (2014)Google Scholar
  4. 4.
    Demir, I., Aliaga, D.G., Benes, B.: Procedural editing of 3D building point clouds. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2147–2155, December 2015.  https://doi.org/10.1109/ICCV.2015.248
  5. 5.
    Demir, I., Aliaga, D.G., Benes, B.: Coupled segmentation and similarity detection for architectural models. ACM Trans. Graph. 34(4), 1–11 (2015)CrossRefGoogle Scholar
  6. 6.
    Fathalla, R., Vogiatzis, G.: A deep learning pipeline for semantic facade segmentation. In: Proceedings of the British Machine Vision Conference 2016, BMVC 2017, September 2017. c 2017. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms. http://publications.aston.ac.uk/id/eprint/31805/
  7. 7.
    Gadde, R., Marlet, R., Paragios, N.: Learning grammars for architecture-specific facade parsing. Int. J. Comput. Vis. 117(3), 290–316 (2016).  https://doi.org/10.1007/s11263-016-0887-4MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
  9. 9.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)Google Scholar
  10. 10.
    Kelly, T., Guerrero, P., Steed, A., Wonka, P., Mitra, N.J.: FrankenGAN: guided detail synthesis for building mass-models using style-synchonized GANs. ACM Trans. Graph. 37(6) (2018).  https://doi.org/10.1145/3272127.3275065
  11. 11.
    Kozinski, M., Gadde, R., Zagoruyko, S., Obozinski, G., Marlet, R.: A MRF shape prior for facade parsing with occlusions. In: IEEE Computer Vision and Pattern Recognition, pp. 2820–2828 (2015)Google Scholar
  12. 12.
    Koziński, M., Obozinski, G., Marlet, R.: Beyond procedural facade parsing: bidirectional alignment via linear programming. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 79–94. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-16817-3_6CrossRefGoogle Scholar
  13. 13.
    Leotta, M.J., et al.: Urban semantic 3D reconstruction from multiview satellite imagery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019Google Scholar
  14. 14.
    Liu, H., Zhang, J., Zhu, J., Hoi, S.C.H.: DeepFacade: a deep learning approach to facade parsing. In: International Joint Conference on Artificial Intelligence, pp. 2301–2307 (2017)Google Scholar
  15. 15.
    Martinovic, A., Van Gool, L.: Bayesian grammar learning for inverse procedural modeling. In: IEEE Computer Vision and Pattern Recognition, pp. 201–208 (2013)Google Scholar
  16. 16.
    Mathias, M., Martinović, A., Van Gool, L.: ATLAS: a three-layered approach to facade parsing. Int. J. Comput. Vis. 118(1), 22–48 (2016).  https://doi.org/10.1007/s11263-015-0868-zMathSciNetCrossRefGoogle Scholar
  17. 17.
    Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of facades. ACM Trans. Graph. 26(3), 85–es (2007).  https://doi.org/10.1145/1276377.1276484
  18. 18.
    Musialski, P., Wonka, P., Aliaga, D.G., Wimmer, M., Van Gool, L., Purgathofer, W.: A survey of urban reconstruction. Comput. Graph. Forum 32, 146–177 (2013)CrossRefGoogle Scholar
  19. 19.
    Nguatem, W., Mayer, H.: Modeling urban scenes from pointclouds. In: IEEE International Conference on Computer Vision, pp. 3837–3846 (2017)Google Scholar
  20. 20.
    Nishida, G., Bousseau, A., Aliaga, D.G.: Procedural modeling of a building from a single image. Comput. Graph. Forum 37, 415–429 (2018)CrossRefGoogle Scholar
  21. 21.
    Ozcanli, O.C., Dong, Y., Mundy, J.L., Webb, H., Hammoud, R., Tom, V.: A comparison of stereo and multiview 3-D reconstruction using cross-sensor satellite imagery. In: IEEE Computer Vision and Pattern Recognition Workshops, pp. 17–25 (2015)Google Scholar
  22. 22.
    Qin, R.: Automated 3D recovery from very high resolution multi-view satellite images. In: ASPRS (IGTF) Annual Conference, p. 10 (2017)Google Scholar
  23. 23.
    Riemenschneider, H., et al.: Irregular lattices for complex shape grammar facade parsing. In: IEEE Computer Vision and Pattern Recognition, pp. 1640–1647 (2012)Google Scholar
  24. 24.
    Ritchie, D., Mildenhall, B., Goodman, N.D., Hanrahan, P.: Controlling procedural modeling programs with stochastically-ordered sequential Monte Carlo. ACM Trans. Graph. 34(4), 1–11 (2015)CrossRefGoogle Scholar
  25. 25.
    Sasaki, Y.: The truth of the f-measure. Teach Tutor Mater, January 2007Google Scholar
  26. 26.
    Talton, J.O., Lou, Y., Lesser, S., Duke, J., Měch, R., Koltun, V.: Metropolis procedural modeling. ACM Trans. Graph. 30(2), 1–14 (2011)CrossRefGoogle Scholar
  27. 27.
    Teboul, O., Kokkinos, I., Simon, L., Koutsourakis, P., Paragios, N.: Shape grammar parsing via reinforcement learning. In: IEEE Computer Vision and Pattern Recognition, pp. 2273–2280 (2011)Google Scholar
  28. 28.
    Vanegas, C.A., Aliaga, D.G., Beneš, B.: Building reconstruction using manhattan-world grammars. In: IEEE Computer Vision and Pattern Recognition (2010)Google Scholar
  29. 29.
    Yang, C., Han, T., Quan, L., Tai, C.L.: Parsing façade with rank-one approximation. In: IEEE Computer Vision and Pattern Recognition, pp. 1720–1727 (2012)Google Scholar
  30. 30.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. CoRR abs/1801.07892 (2018). http://arxiv.org/abs/1801.07892
  31. 31.
    Zhang, H., et al.: Context encoding for semantic segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  32. 32.
    Zhang, X., May, C., Nishida, G., Aliaga, D.: Progressive regularization of satellite-based 3D buildings for interactive rendering. In: Symposium on Interactive 3D Graphics and Games, I3D 2020. Association for Computing Machinery, New York (2020)Google Scholar
  33. 33.
    Zheng, C., Cham, T.J., Cai, J.: Pluralistic image completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1438–1447 (2019)Google Scholar
  34. 34.
    Zheng, E., Wang, K., Dunn, E., Frahm, J.M.: Minimal solvers for 3D geometry from satellite imagery. In: IEEE International Conference on Computer Vision, pp. 738–746 (2015)Google Scholar
  35. 35.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Purdue UniversityWest LafayetteUSA

Personalised recommendations