Abstract
Although image completion has been used in many image filling and editing problems for a long time, it is rarely applied in the scenario regarding the perspective effect which is demonstrated largely in the design of urban architectures. Thus this paper proposes a method to capture the perspective information automatically based on a single image and employs it to image completion tasks effectively. Actually it is difficult to automatize the extraction from only a single image, but it has been observed that architectures usually contain plenty of repetitions and symmetries which could be described through periodicity. Therefore, our approach first determines the initial vanishing points through the cascaded hough transformation. Then, a refinement scheme based on SIFT feature matching is proposed to obtain the accurate vanishing points. At last, a good perspective mesh is generated from the geometry calculation. The experiments show that this method could be used to automatically extract perspective information from a single image and recover the image in a visually plausible way.
抽象
创新点
尽管图像填补技术已经广泛应用在各种图像编辑任务中, 但却很少作用在大量具有透视效果的场景中. 本文针对大量建筑物反映出的透视结构, 提出一种基于单幅图像的透视信息自动提取技术并有效应用于图像填补等问题上. 为了实现基于单幅图像的自动化提取, 首先应用级联霍夫变换算法取得初始灭点, 然后利用户外建筑结构具有大量重复性和对称性的特点提出了基于 SIFT 特征匹配的灭点修正方法, 最后通过几何计算得到图像填补所需要的透视网格. 准确的透视网格约束了图像填补工作, 实验结果证明, 该方法可以自动有效地从单幅图像中提取户外建筑结构的透视信息, 并合理地恢复具有透视约束的图像.
Similar content being viewed by others
References
Barrett W A, Cheney A S. Object-based image editing. ACM Trans Graph, 2002, 21: 777–784.
Criminisi A, Perez P, Toyama K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process, 2004, 13: 1200–1212.
Drori I, Cohen-Or D, Yeshurun H. Fragment-based image completion. ACM Trans Graph, 2003, 22: 303–312.
Sun J, Yuan L, Jia J, et al. Image completion with structure propagation. ACM Trans Graph, 2005, 24: 861–868.
Chen X W, Zhou B, Guo Y, et al. Structure guided texture inpainting through multi-scale patches and global optimization for image completion. Sci China Inf Sci, 2014, 57: 012102
Zhang G F, Jia J Y, Wong T T, et al. Consistent depth maps recovery from a video sequence. IEEE Trans Patt Anal Mach Intell, 2009, 31: 974–988.
Chen Y D, Hao C Y, Cai Z M, et al. Live accurate and dense reconstruction from a handheld camera. J Comput Animat Virtual Worlds, 2013, 24: 387–397.
Wu C C, Frahm J M, Pollefeys M. Detecting large repetitive structures with salient boundaries. In: Proceedings of the 11th European Conference on Computer Vision. Berlin: Springer, 2010. 142–155
Ballester C, Bertalmio M, Caselles V, et al. Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans Image Process, 2001, 10: 1200–1211.
Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 2000. 417–424
Oliveira M M, Bowen B, McKenna R, et al. Fast digital image inpainting. In: Proceedings of the International Conference on Visualization, Imaging and Image Processing. New York: ACM Press, 2001. 261–266
Efros A A, Leung T K. Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision. Washington DC: IEEE, 1999. 1033–1038
Wei L Y, Levoy M. Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 2000. 479–488
Ashikhmin M. Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics. New York: ACM Press, 2001. 217–226
Efros A A, Freeman W T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 2001. 341–346
Kwatra V, Schodl A, Essa I, et al. Graphcut textures: image and video synthesis using graph cuts. ACM Trans Graph, 2003, 22: 277–286.
Bertalmio M, Vese L A, Sapiro G, et al. Simultaneous structure and texture image inpainting. IEEE Trans Image Process, 2003, 12: 882–889.
Jia J Y, Tang C K. Image repairing: robust image synthesis by adaptive Nd tensor voting. In: Proceedings of the 2003 IEEE Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE, 2003. 643–650
Harrison P. A non-hierarchical procedure for re-synthesis of complex textures. In: Proceedings of the International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, Plzen, 2001. 190–197
Wilczkowiak M, Brostow G J, Tordoff B, et al. Hole filling through photomontage. In: Proceedings of the 16th British Machine Vision Conference, Oxford, 2005. 492–501
Kwatra V, Essa I, Bobick A, et al. Texture optimization for example-based synthesis. ACM Trans Graph, 2003, 24: 795–802.
Komodakis N, Tziritas G. Image completion using global optimization. In: Proceedings of the 2006 IEEE Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE, 2006. 442–452
Perez P, Gangnet M, Blake A. Patchworks: example-based region tiling for image editing. MSR-TR-2004-04. 2004
Darabi S, Shechtman E, Barnes C, et al. Image melding: combining inconsistent images using patch-based synthesis. ACM Trans Graph, 2012, 31: 1–10.
Pavic D, Schonefeld V, Kobbelt L. Interactive image completion with perspective correction. Vis Comput, 2006, 22: 671–681.
Barnard S T. Interpreting perspective images. Artif Intell, 1983, 21: 435–462.
Tuytelaars T, van Gool L, Proesmans M, et al. The cascaded hough transform as an aid in aerial image interpretation. In: Proceedings of the 6th International Conference on Computer Vision. Washington DC: IEEE, 1998. 67–72
Košsecka J, Zhang W. Video compass. In: Proceedings of the 7th European Conference on Computer Vision. Berlin: Springer, 2002. 476–490
Tardif J. Non-iterative approach for fast and accurate vanishing point detection. In: Proceedings of the 12th International Conference on Computer Vision. Washington DC: IEEE, 2009. 1250–1257
Wenzel S, Drauschke M, Forstner W. Detection of repeated structures in facade images. Patt Recog Image Anal, 2008, 18: 406–411.
Loy G, Eklundh J O. Detecting symmetry and symmetric constellations of features. In: Proceedings of the 9th European Conference on Computer Vision. Berlin: Springer, 2006. 508–521
Horry Y, Anjyo K I, Arai K. Tour into the picture: using a spidery mesh interface to make animation from a single image. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 1997. 225–232
Debevec P E, Taylor C J, Malik J. Modeling and rendering architecture from photographs: a hybrid geometryand image-based approach. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 1996. 11–20
Hao C Y, Wu E H. Interactive image completion with perspective constraint. In: Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-reality Continuum and Its Applications in Industry. New York: ACM Press, 2012. 153–160
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hao, C., Chen, Y., Wu, W. et al. Image completion with perspective constraint based on a single image. Sci. China Inf. Sci. 58, 1–12 (2015). https://doi.org/10.1007/s11432-015-5359-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11432-015-5359-x
Keywords
- perspective constraint
- image completion
- image inpainting
- texture synthesis
- mesh
- vanishing point
- hough transform
- SIFT features