Abstract
The use of repetition detection is an effective approach for increasing the efficiency of urban modeling. In practice, repetition detection can benefit from the apparent regularities and strong contextual relationships in façades. In view of this, we propose a novel algorithm for automatically detecting and inferring repetitive elements with accurate locations and shapes from façades. More specifically, firstly, starting from a rectification of the input façade, we employ the color clustering method to automatically derive candidate templates. Secondly, to detect the non- and partially occluded repetitive elements matching with the derived templates, we construct an adaptive region descriptor and a repetitive characteristic curve. Finally, the fully occluded elements are inferred by utilizing the Bayesian probability network, which can be learned from a database of the selected façades. The accuracy of our detection and inference is tested through a variety of experiments, and all of them justify the robustness of our algorithm to outliers such as appearance variations and occlusions.
Similar content being viewed by others
References
Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of façades. ACM Trans. Gr. 26(3), 85 (2007)
Wu, C., Frahm, J., Pollefeys, M.: Repetition-based dense single-view reconstruction. Rodng 42(7), 3113–3120 (2011)
Hou, F., Qin, H., Qi, Y.: Procedure-based component and architecture modeling from a single image. Vis. Comput. 32(2), 151–166 (2016)
Ji, L., Przemyslaw, M., Peter, W., Jieping, Y.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)
Musialski, P., Wonka, P., Recheis, M., Maierhofer, S., Purgathofer, W.: Symmetry-based façade repair. In: Vision Modeling and Visualization, Workshop, pp. 3–10 (2009)
Fuzhang, W., Yan, D.M., Dong, W., Zhang, X., Wonka, P.: Inverse procedural modeling of façade layouts. ACM Trans. Gr. 33(4), 70–79 (2014)
Zhang, H., Kai, X., Jiang, W., Lin, J., Cohen-Or, D., Chen, B.: Layered analysis of irregular façades via symmetry maximization. ACM Trans. Gr. 32(4), 96–96 (2013)
Yunliang, C., George, B.: Detecting, grouping, and structure inference for invariant repetitive patterns in images. IEEE Trans. Image Process. 22(6), 2343–2355 (2013)
Minwoo, P., Kyle, B., Collins, R.T., Yanxi, L.: Deformed lattice detection in real-world images using mean-shift belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 31(10), 1804–1816 (2009)
Shen, C., Huang, S., Hu, S.: Adaptive partitioning of urban façades. ACM Trans. Gr. 30(6), 61–64 (2012)
Teboul, O., Kokkinos, I., Simon, L., Koutsourakis, P., Paragios, N.: Shape grammar parsing via reinforcement learning. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2273–2280 (2011)
Changchang, W., Frahm, J.M., Pollefeys, M.: Detecting large repetitive structures with salient boundaries. Lect. Notes Comput. Sci. 6312, 142–155 (2010)
Zhang, H., Yang, L., Zhao, P., Quan, L.: Per-pixel translational symmetry detection, optimization, and segmentation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 526–533 (2012)
Zhao, P., Quan, L.: Translation symmetry detection in a fronto-parallel view. In: Proceedings/CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1009–1016 (2011)
Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Manhattan-world stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 1422–1429. IEEE (2009)
Lian, Y., Shen, X.: Probabilistic reasoning for repeatability detection from urban facade image. In: International Conference on Optical and Photonic Engineering (icOPEN2015), pp. 952429–952429. International Society for Optics and Photonics (2015)
Lian, Y., Shen, X.: Detecting repetitive elements with accurate locations and shapes from urban façade. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 1920–1924. IEEE (2015)
Martinovi, A., Mathias, M., Weissenberg, J., Van Gool, L.: A three-layered approach to facade parsing. Int. J. Comput. Vis. 7578(1), 416–429 (2012)
Zhao, P., Quan, L.: Translation symmetry detection in a fronto-parallel view. In: IEEE Conference on Computer Vision and Pattern Recognition 2011, pp. 1009–1016 (2011)
Peng, Z., Lei, Y., Honghui, Z., Long, Q.: Per-pixel translational symmetry detection, optimization, and segmentation. In: Computer Vision and Pattern Recognition, pp. 526–533 (2012)
Lee, S.C., Nevatia, R.: Extraction and integration of window in a 3d building model from ground view images. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II-113–II-120 (2004)
Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: European Conference on Computer Vision, pp. 391–405. Springer (2014)
Loy, G., Eklundh, J.O.: Detecting Symmetry and Symmetric Constellations of Features. Springer, Berlin (2006)
Jian, L., Ye, Z., Zou, Y.: Automatic generation of colorful patterns with wallpaper symmetries from dynamics. Vis. Comput. 23(6), 445–449 (2007)
Olivier, T., Iasonas, K., Loic, S., Panagiotis, K., Nikos, P.: Parsing façades with shape grammars and reinforcement learning. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1744–1756 (2013)
Fan, L., Musialski, P., Liu, L., Wonka, P.: Structure completion for façade layouts. ACM Trans. Gr. 33(6), 210:1–210:11 (2014)
Nan, L., Sharf, A., Zhang, H., Cohen-Or, D., Chen, B.: Smartboxes for interactive urban reconstruction. ACM Trans. Gr. 29(4), 157–166 (2010)
Nan, L., Jiang, C., Bernard, G., Peter, W.: Template assembly for detailed urban reconstruction. Comput. Gr. Forum 34(2), 217–228 (2015)
Rasmussen, C., Korah, T.: Spatiotemporal inpainting for recovering texture maps of partially occluded building facades. In: IEEE International Conference on Image Processing 2005, vol. 3, p. III-125. IEEE (2005)
Korah, T., Rasmussen, C.: Spatiotemporal inpainting for recovering texture maps of occluded building facades. IEEE Trans. Image Process. 16(9), 2262–2271 (2007)
Böhm, J.: Multi-image fusion for occlusion-free façade texturing. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 35(5), 867–872 (2004)
Ortin, D., Remondino, F.: Occlusion-free image generation for realistic texture mapping. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 36(5/W17), 7 (2005)
Wang, X., Totaro, S., Taillandier, F., Hanson, A.R, Teller, S.: Recovering façade texture and microstructure from real-world images. In: Proceedings of International Workshop on Texture Analysis and Synthesis at ECCV, pp. 381–386 (2002)
Mu, T.-J., Wang, J.-H., Du, S.-P., Hu, S.-M.: Stereoscopic image completion and depth recovery. Vis Comput 30(6), 833–843 (2014)
Yang, C., Han, T., Quan, L., Tai, C.-L.: Parsing façade with rank-one approximation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1720–1727. IEEE (2012)
Liang, X., Ren, X., Zhang, Z., Ma, Y.: Repairing sparse low-rank texture. In: European Conference on Computer Vision, pp. 482–495. Springer (2012)
Liang, X., Ren, X., Zhang, Z., Ma, Y.: Texture repairing by unified low rank optimization. J. Comput. Sci. Technol. 31(3), 525–546 (2016)
He, K., Sun, J.: Statistics of patch offsets for image completion. In: Computer Vision-ECCV 2012, pp. 16–29. Springer (2012)
Yeh, Y.-T., Breeden, K., Yang, L., Fisher, M., Hanrahan, P.: Synthesis of tiled patterns using factor graphs. ACM Trans. Gr. (TOG) 32(1), 3 (2013)
Komodakis, N.: Image completion using global optimization. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 1, pp. 442–452. IEEE (2006)
Pavić, D., Schönefeld, V., Kobbelt, L.: Interactive image completion with perspective correction. Vis. Comput. 22(9–11), 671–681 (2006)
Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)
Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)
Spath, H.: The cluster dissection and analysis theory FORTRAN programs examples. Prentice-Hall, Inc, Upper Saddle River, NJ, USA (1985)
Serra, J., Soille, P.: Mathematical Morphology and Its Applications to Image Processing. Kluwer, Dordrecht (1994)
Adams, R., Bischof, L.: Seeded region growing. IEEE Trans. Pattern Anal. Mach. Intell. 16(6), 641–647 (1994)
Kumar, S., Chauhan, A.: A survey on image feature selection techniques. Int. J. Comput. Sci. Inf. Technol. 5(5), 6449–6452 (2014)
Kazmi, I.K., You, L., Zhang, J.J.: A survey of 2d and 3d shape descriptors. In: Computer Graphics, Imaging and Visualization (CGIV), 2013 10th International Conference, pp. 1–10. IEEE (2013)
Heider, P., Pierre-Pierre, A., Li, R., Mueller, R., Grimm, C.: Comparing local shape descriptors. Vis. Comput. 28(9), 919–929 (2012)
Theodoridis, S., Koutroumbas, K.: Pattern Recognition, 4th edn. Academic Press, New York (2008)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2014)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Bengio, Y.: Learning deep architectures for AI. Found. Trends\({\textregistered }\) Mach. Learn. 2(1), 1–127 (2009)
Huang, R., Ye, M., Pei, X., Li, T., Dou, Y.: Learning to pool high-level features for face representation. Vis. Comput. 31(12), 1683–1695 (2015)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Tylecek, R., Šára, R.: Spatial pattern templates for recognition of objects with regular structure. In: German Conference on Pattern Recognition, pp. 364–374. Springer (2013)
Shao, H., Svoboda, T., Van Gool, L.: Zubud–zurich buildings database for image based recognition. Comput. Vis. Lab. Swiss Federal Inst. Technol. Switz. Tech. Rep. 260, 20 (2003)
Jensen, F.V.: An Introduction to Bayesian Networks, vol. 210. UCL Press, London (1996)
Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge (2009)
Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, Burlington (2014)
Rumí, R., Salmerón, A., Moral, S.: Estimating mixtures of truncated exponentials in hybrid Bayesian networks. Test 15(2), 397–421 (2006)
Cobb, B.R., Rumí, R., Salmerón, A.: Bayesian network models with discrete and continuous variables. In: Advances in Probabilistic Graphical Models, pp. 81–102. Springer Berlin Heidelberg (2007)
Romero, V., Rum, R., Salmern, A.: Learning hybrid Bayesian networks using mixtures of truncated exponentials. Int. J. Approx. Reason. 42(1), 54–68 (2006)
Cobb, B.R., Shenoy, P.P.: Inference in hybrid Bayesian networks with mixtures of truncated exponentials. Int. J. Approx. Reason. 41(3), 257–286 (2006)
Goutte, C., Gaussier, E.: A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: European Conference on Advances in Information Retrieval, pp. 345–359. Springer Berlin Heidelberg (2005)
Geng, Y., Hai-Miao, H., Zeng, G., Zheng, J.: A person re-identification algorithm by exploiting region-based feature salience. J. Vis. Commun. Image Rep. 29, 89–102 (2015)
Torralba, A., Murphy, K.P., Freeman, W.T.: Sharing features: efficient boosting procedures for multiclass object detection. In: Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 2, pp. II-762. IEEE (2004)
Krizhevsky, A., Sutskever, I., Hinton, G.E: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105. Curran Associates, Inc (2012)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Acknowledgements
This work was partially supported by the National High-Tech Research and Development Program of China (Grant No. 2013AA013803-1) and the National Natural Science Foundation of China (Grant No. 61202235).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lian, Y., Shen, X. & Hu, Y. Detecting and inferring repetitive elements with accurate locations and shapes from façades. Vis Comput 34, 491–506 (2018). https://doi.org/10.1007/s00371-017-1355-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-017-1355-z