Skip to main content

Advertisement

SpringerLink
Log in

Guided pluralistic building contour completion

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image/sketch completion is a core task that addresses the problem of completing the missing regions of an image/sketch with realistic and semantically consistent content. We address one type of completion which is producing a tentative completion of an aerial view of the remnants of a building structure. The inference process may start with as little as 10% of the structure and thus is fundamentally pluralistic (e.g., multiple completions are possible). We present a novel pluralistic building contour completion framework. A feature suggestion component uses an entropy-based model to request information from the user for the next most informative location in the image. Then, an image completion component trained using self-supervision and procedurally generated content produces a partial or full completion. In our synthetic and real-world experiments for archaeological sites in Turkey, with up to only 4 iterations, we complete building footprints having only 10–15% of the ancient structure initially visible. We also compare to various state-of-the-art methods and show our superior quantitative/qualitative performance. While we show results for archaeology, we anticipate our method can be used for restoring highly incomplete historical sketches and for modern day urban reconstruction despite occlusions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing. 10(8), 1200–1211 (2001). https://doi.org/10.1109/83.935036

    Article  MathSciNet  MATH  Google Scholar 

  2. Bao, F., Yan, D.M., Mitra, N.J., Wonka, P.: Generating and exploring good building layouts. ACM Trans. Graph. (2013). https://doi.org/10.1145/2461912.2461977

    Article  MATH  Google Scholar 

  3. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (2000). https://doi.org/10.1145/344779.344972

  4. Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI (2017)

  5. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.: Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR (2015)

  6. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV (2018)

  7. Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (TOG) 31(4), 1–10 (2012)

  8. Demir, I., Aliaga, D.G., Benes, B.: Proceduralization for editing 3d architectural models. In: 2016 Fourth International Conference on 3D Vision (3DV) (2016)

  9. Ghosh, A., Zhang, R., Dokania, P.K., Wang, O., Efros, A.A., Torr, P.H.S., Shechtman, E.: Interactive sketch & fill: Multiclass sketch-to-image translation. In: ICCV (2019)

  10. He, K., Sun, J.: Statistics of patch offsets for image completion. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C.: (eds.) ECCV Springer Berlin, Heidelberg, 16–29 (2012)

  11. He, X., Li, H., Ming, Y.: Connected contours: A new contour completion model that respects the closure effect. In: CVPR (2012)

  12. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and Locally Consistent Image Completion. ACM Trans. Graph. (Proc. of SIGGRAPH 2017) 36(4), 1–14 (2017)

  13. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. CVPR (2017)

  14. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)

  15. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)

  16. Köhler, R., Schuler, C., Schölkopf, B., Harmeling, S.: Mask-specific inpainting with deep neural networks. In: Pattern Recognition (2014)

  17. Köppel, M., Ben Makhlouf, M., Müller, K., Wiegand, T.: Fast image completion method using patch offset statistics. In: (ICIP) (2015)

  18. Lee, Y.J., Zitnick, C.L., Cohen, M.F.: Shadowdraw: Real-time user guidance for freehand drawing. ACM Trans. Graph. (2011)

  19. Levin, Zomet, Weiss: Learning how to inpaint from global image statistics. In: Proceedings Ninth IEEE International Conference on Computer Vision (2003)

  20. Lin, H., Fu, Y., Xue, X., Jiang, Y.: Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt. In: (CVPR) (2020)

  21. Liu, F., Deng, X., Lai, Y.K., Liu, Y.J., Ma, C., Wang, H.: Sketchgan: Joint sketch completion and recognition with generative adversarial network. In: (CVPR) (2019)

  22. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR) (2015)

  23. Müller, P., Wonka, P., Haegler, S., Ulmer, A., Van Gool, L.: Procedural modeling of buildings. ACM Trans. Graph. 25(3) 614–623 (2006)

  24. Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: Edgeconnect: Structure guided image inpainting using edge prediction. In: (ICCV) Workshops (2019)

  25. Nishida, G., Garcia-Dorado, I., Aliaga, D.G., Benes, B., Bousseau, A.: Interactive sketching of urban procedural models. ACM Trans. Graph. 35(4), 1–11 (2016)

  26. Parish, Y.I.H., Müller, P.: Procedural modeling of cities. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (2001)

  27. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)

  28. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR (2016)

  29. Poma, X.S., Riba, E., Sappa, A.: Dense extreme inception network: Towards a robust cnn model for edge detection. In: (WACV) (2020)

  30. Ren, J.S., Xu, L., Yan, Q., Sun, W.: Shepard convolutional neural networks. In: NIPS (2015)

  31. Ren, X., Fowlkes, C.C., Malik, J.: Scale-invariant contour completion using conditional random fields. In: ICCV (2005)

  32. Ritchie, D., Mildenhall, B., Goodman, N.D., Hanrahan, P.: Controlling procedural modeling programs with stochastically-ordered sequential monte carlo. ACM Trans. Graph. 34(4), 1–11 (2015)

    Article  Google Scholar 

  33. Ronneberger, O., P.Fischer, Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: (MICCAI) (2015)

  34. Sasaki, K., Iizuka, S., Simo-Serra, E., Ishikawa, H.: Joint Gap Detection and Inpainting of Line Drawings. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  35. Su, G., Qi, Y., Pang, K., Yang, J., Song, Y.Z.: Sketchhealer: A graph-to-sequence network for recreating partial human sketches. In: 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020 (2020)

  36. Vanegas, C.A., Aliaga, D.G., Benes, B.: Building reconstruction using manhattan-world grammars. In: CVPR (2010)

  37. Vanegas, C.A., Garcia-Dorado, I., Aliaga, D.G., Benes, B., Waddell, P.: Inverse design of urban procedural models. ACM Trans. Graph. 31(6) 1–168 (2012)

  38. Wan, Z., Zhang, J., Chen, D., Liao, J.: High-fidelity pluralistic image completion with transformers. In: (ICCV) (2021)

  39. Xu, J., Collins, M.D., Singh, V.: Incorporating User Interaction and Topological Constraints within Contour Completion via Discrete Calculus. In: CVPR (2013)

  40. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: (CVPR) (2017)

  41. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: (CVPR) (2018)

  42. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: (ICCV) (2019)

  43. Zhang, X., May, C., Aliaga, D.: Synthesis and completion of facades from satellite imagery. In: ECCV (2020)

  44. Zhao, Z., Liu, W., Xu, Y., Chen, X., Luo, W., Jin, L., Zhu, B., Liu, T., Zhao, B., Gao, S.: Prior based human completion. In: (CVPR) (2021)

  45. Zheng, C., Cham, T.J., Cai, J.: Pluralistic image completion. In: (CVPR) (2019)

Download references

Funding

This research was funded in part by National Science Foundation grants #1816514 CHS: Small: Functional Proceduralization of 3D Geometric Models, #1835739 U-Cube: A Cyberinfrastructure for Unified and Ubiquitous Urban Canopy Parameterization, and #2107096 Deep Generative Modeling for Urban and Archaeological Recovery

Author information

Authors and Affiliations

  1. Purdue University, West Lafayette, USA

    Xiaowei Zhang, Wufei Ma, Gunder Varinlioglu, Nick Rauh, Liu He & Daniel Aliaga

  2. Mimar Sinan Fine Arts University, Istanbul, Turkey

    Gunder Varinlioglu

Authors
  1. Xiaowei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wufei Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gunder Varinlioglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nick Rauh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Liu He
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel Aliaga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaowei Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no known potential conflicts of interest that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Ma, W., Varinlioglu, G. et al. Guided pluralistic building contour completion. Vis Comput 38, 3205–3216 (2022). https://doi.org/10.1007/s00371-022-02532-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02532-z

Keywords

Search

Navigation