Skip to main content
SpringerLink
Log in

Window Detection in Facades Using Heatmap Fusion

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Window detection is a key component in many graphics and vision applications related to 3D city modeling and scene visualization. We present a novel approach for learning to recognize windows in a colored facade image. Rather than predicting bounding boxes or performing facade segmentation, our system locates keypoints of windows, and learns keypoint relationships to group them together into windows. A further module provides extra recognizable information at the window center. Locations and relationships of keypoints are encoded in different types of heatmaps, which are learned in an end-to-end network. We have also constructed a facade dataset with 3 418 annotated images to facilitate research in this field. It has richly varying facade structures, occlusion, lighting conditions, and angle of view. On our dataset, our method achieves precision of 91.4% and recall of 91.0% under 50% IoU (intersection over union). We also make a quantitative comparison with state-of-the-art methods to verify the utility of our proposed method. Applications based on our window detector are also demonstrated, such as window blending.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cao Z, Simon T, Wei S, Sheikh Y. Realtime multi-person 2D pose estimation using part affinity fields. In Proc. the 30th IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.1302-1310.

  2. Newell A, Huang Z, Deng J. Associative embedding: Endto-end learning for joint detection and grouping. In Proc. the 31st Advances in Neural Information Processing Systems, December 2017, pp.2274-2284.

  3. Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. the 27th IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp.580–587.

  4. Girshick R. Fast R-CNN. In Proc. the 15th International Conference on Computer Vision, December 2015, pp.1440–1448.

  5. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proc. the 29th Advances in Neural Information Processing Systems, December 2015, pp.91–99.

  6. He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In Proc. the 16th International Conference on Computer Vision, October 2017, pp.2961-2969.

  7. Dai J, Li Y, He K, Sun J. R-FCN: Object detection via region-based fully convolutional networks. In Proc. the 30th Annual Conference in Neural Information Processing Systems, December 2016, pp.379–387.

  8. Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y. Deformable convolutional networks. In Proc. the 16th International Conference on Computer Vision, October 2017, pp.764–773.

  9. Zhu X, Hu H, Lin S, Dai J. Deformable ConvNets v2: More deformable, better results. In Proc. the 32nd IEEE Conference on Computer Vision and Pattern Recognition, June 2019, pp.9308–9316.

  10. Law H, Deng J. CornerNet: Detecting objects as paired keypoints. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.765-781.

  11. Zhou X, Zhuo J, Krahenbuhl P. Bottom-up object detection by grouping extreme and center points. In Proc. the 32nd IEEE Conference on Computer Vision and Pattern Recognition, June 2019, pp.850–859.

  12. Duan K, Bai S, Xie L, Qi H, Huang Q, Tian Q. Centernet: Keypoint triplets for object detection. In Proc. the 17th International Conference on Computer Vision, October 2019, pp.6568–6577.

  13. Ali H, Seifert C, Jindal N, Paletta L, Paar G. Window detection in facades. In Proc. the 14th International Conference on Image Analysis and Processing, September 2007, pp.837–842.

  14. Freund Y, Schapire R E. Experiments with a new boosting algorithm. In Proc. the 13th International Conference on Machine Learning, July 1996, pp.148–156.

  15. Shen C, Huang S, Fu H, Hu S. Adaptive partitioning of urban facades. ACM Trans. Graphics, 2011, 30(6): Article No. 184.

  16. Fan L, Musialski P, Liu L, Wonka P. Structure completion for facade layouts. ACM Trans. Graphics, 2014, 33(6): Article No. 210.

  17. Teboul O, Simon L, Koutsourakis P, Paragios N. Segmentation of building facades using procedural shape priors. In Proc. the 23rd IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2010, pp.3105–3112.

  18. Teboul O, Kokkinos I, Simon L, Koutsourakis P, Paragios N. Shape grammar parsing via reinforcement learning. In Proc. the 24th IEEE Conference on Computer Vision and Pattern Recognition, June 2011, pp.2273–2280.

  19. Koziński M, Obozinski G, Marlet R. Beyond procedural facade parsing: Bidirectional alignment via linear programming. In Proc. the 12th Asian Conference on Computer Vision, November 2014, pp.79–94.

  20. Kozinski M, Gadde R, Zagoruyko S, Obozinski G, Marlet R. A MRF shape prior for facade parsing with occlusions. In Proc. the 28th IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp.2820–2828.

  21. Gadde R, Marlet R, Paragios N. Learning grammars for architecture-specific facade parsing. International Journal of Computer Vision, 2016, 117(3): 290–316.

    Article  MathSciNet  Google Scholar 

  22. Weissenberg J, Riemenschneider H, Prasad M, van Gool L. Is there a procedural logic to architecture? In Proc. the 26th IEEE Conference on Computer Vision and Pattern Recognition, June 2013, pp.185–192.

  23. Liu H, Zhang J, Zhu J, Hoi S C. DeepFacade: A deep learning approach to facade parsing. In Proc. the 26th InternationalJoint Conference on Artificial Intelligence, Aug. 2017, pp.2301–2307.

  24. Recky M, Leberl F. Window detection in complex facades. In Proc. the 2nd European Workshop on Visual Information Processing, July 2010, pp.220–225.

  25. Wan G, Li S. Automatic facades segmentation using detected lines and vanishing points. In Proc. the 4th IEEE International Congress on Image and Signal Processing, October 2011, pp.1214–1217.

  26. Haugeard J E, Philipp-Foliguet S, Precioso F.Windows and facades retrieval using similarity on graph of contours. In Proc. the 16th International Conference on Image Processing, November 2009, pp.269–272.

  27. Cohen A, Schwing A G, Pollefeys M. Efficient structured parsing of facades using dynamic programming. In Proc. the 27th IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp.3206–3213.

  28. Li Y, Zheng Q, Sharf A, Cohen-Or D, Chen B, Mitra N J. 2D-3D fusion for layer decomposition of urban facades. In Proc. the 13th International Conference on Computer Vision, November 2011, pp.882–889.

  29. Alhalawani S, Yang Y, Liu H, Mitra N J. Interactive facades analysis and synthesis of semi-regular facades. Computer Graphics Forum, 2013, 32(2): 215–224.

    Article  Google Scholar 

  30. Bao F, Schwarz M, Wonka P. Procedural facade variations from a single layout. ACM Trans. Graphics, 2013, 32(1): Article No. 8.

  31. Ceylan D, Mitra N J, Li H, Weise T, Pauly M. Factored facade acquisition using symmetric line arrangements. Computer Graphics Forum, 2012, 31(2): 671–680.

    Article  Google Scholar 

  32. Haegler S, Wonka P, Arisona S M, van Gool L, Müller P. Grammar-based encoding of facades. Computer Graphics Forum, 2010, 29(4): 1479–1487.

    Article  Google Scholar 

  33. Zhuo H, Zhou S, Benes B, Whittinghill D. User-assisted inverse procedural facade modeling and compressed image rendering. In Proc. the 11th International Symposium on Visual Computing, December 2015, pp.126–136.

  34. Müller P, Zeng G, Wonka P, van Gool L. Image-based procedural modeling of facades. ACM Trans. Graphics, 2007, 26(3): Article No. 85.

  35. Zhang H, Xu K, Jiang W, Lin J, Cohen-Or D, Chen B. Layered analysis of irregular facades via symmetry maximization. ACM Trans. Graphics, 2013, 32(4): Article No. 121.

  36. Wang J, Liu C, Shen T, Quan L. Structure-driven facade parsing with irregular patterns. In Proc. the 3rd IAPR Asian Conference on Pattern Recognition, November 2015, pp.41–45.

  37. Cohen A, Oswald M R, Liu Y, Pollefeys M. Symmetryaware façade parsing with occlusions. In Proc. the 5th International Conference on 3D Vision, October 2017, pp.393–401.

  38. Xiao B, Wu H, Wei Y. Simple baselines for human pose estimation and tracking. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.472–487.

  39. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 29th IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770–778.

  40. Newell A, Deng J. Pixels to graphs by associative embedding. In Proc. the 31st Annual Conference on Neural Information Processing Systems, December 2017, pp.2171–2180.

  41. Tyleček R, Šára R. Spatial pattern templates for recognition of objects with regular structure. In Proc. the 35th German Conference on Pattern Recognition, September 2013, pp.364–374.

  42. Deng J, Dong W, Socher R, Li L, Li K, Li F. ImageNet: A large-scale hierarchical image database. In Proc. the 22nd IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp.248–255.

  43. Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C. L. Microsoft COCO: Common objects in context. In Proc. the 13th European Conference on Computer Vision, September 2014, pp.740–755.

  44. Korč F, Förstner W. eTRIMS image database for interpreting images of man-made scenes. Technical Report, University of Bonn, 2009. http://www.ipb.uni-bonn.de/projects/etrims_db/publ/korc-forstner-tr09-etrims.pdf, March 2020.

  45. Riemenschneider H, Krispel U, Thaller W, Donoser M, Havemann S, Fellner D, Bischof H. Irregular lattices for complex shape grammar facade parsing. In Proc. the 25th IEEE Conference on Computer Vision and Pattern Recognition, June 2012, pp.1640–1647.

  46. Pérez P, Gangnet M, Blake A. Poisson image editing. In Proc. the 30th ACM SIGGRAPH International Conference on Computer Graphics and Interactive Techniques, July 2003, pp.313–318.

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou, 310058, China

    Chuan-Kang Li, Hong-Xin Zhang, Jia-Xin Liu, Yuan-Qing Zhang, Shan-Chen Zou & Yu-Tong Fang

Authors
  1. Chuan-Kang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong-Xin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jia-Xin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuan-Qing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shan-Chen Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yu-Tong Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong-Xin Zhang.

Electronic supplementary material

ESM 1

(PDF 513 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, CK., Zhang, HX., Liu, JX. et al. Window Detection in Facades Using Heatmap Fusion. J. Comput. Sci. Technol. 35, 900–912 (2020). https://doi.org/10.1007/s11390-020-0253-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-020-0253-4

Keywords

Search

Navigation