Skip to main content
SpringerLink
Log in

Image completion with perspective constraint based on a single image

单幅图像上透视场景的图像填补

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Although image completion has been used in many image filling and editing problems for a long time, it is rarely applied in the scenario regarding the perspective effect which is demonstrated largely in the design of urban architectures. Thus this paper proposes a method to capture the perspective information automatically based on a single image and employs it to image completion tasks effectively. Actually it is difficult to automatize the extraction from only a single image, but it has been observed that architectures usually contain plenty of repetitions and symmetries which could be described through periodicity. Therefore, our approach first determines the initial vanishing points through the cascaded hough transformation. Then, a refinement scheme based on SIFT feature matching is proposed to obtain the accurate vanishing points. At last, a good perspective mesh is generated from the geometry calculation. The experiments show that this method could be used to automatically extract perspective information from a single image and recover the image in a visually plausible way.

抽象

创新点

尽管图像填补技术已经广泛应用在各种图像编辑任务中, 但却很少作用在大量具有透视效果的场景中. 本文针对大量建筑物反映出的透视结构, 提出一种基于单幅图像的透视信息自动提取技术并有效应用于图像填补等问题上. 为了实现基于单幅图像的自动化提取, 首先应用级联霍夫变换算法取得初始灭点, 然后利用户外建筑结构具有大量重复性和对称性的特点提出了基于 SIFT 特征匹配的灭点修正方法, 最后通过几何计算得到图像填补所需要的透视网格. 准确的透视网格约束了图像填补工作, 实验结果证明, 该方法可以自动有效地从单幅图像中提取户外建筑结构的透视信息, 并合理地恢复具有透视约束的图像.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Barrett W A, Cheney A S. Object-based image editing. ACM Trans Graph, 2002, 21: 777–784.

    Article  Google Scholar 

  2. Criminisi A, Perez P, Toyama K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process, 2004, 13: 1200–1212.

    Article  Google Scholar 

  3. Drori I, Cohen-Or D, Yeshurun H. Fragment-based image completion. ACM Trans Graph, 2003, 22: 303–312.

    Article  Google Scholar 

  4. Sun J, Yuan L, Jia J, et al. Image completion with structure propagation. ACM Trans Graph, 2005, 24: 861–868.

    Article  Google Scholar 

  5. Chen X W, Zhou B, Guo Y, et al. Structure guided texture inpainting through multi-scale patches and global optimization for image completion. Sci China Inf Sci, 2014, 57: 012102

    Google Scholar 

  6. Zhang G F, Jia J Y, Wong T T, et al. Consistent depth maps recovery from a video sequence. IEEE Trans Patt Anal Mach Intell, 2009, 31: 974–988.

    Article  Google Scholar 

  7. Chen Y D, Hao C Y, Cai Z M, et al. Live accurate and dense reconstruction from a handheld camera. J Comput Animat Virtual Worlds, 2013, 24: 387–397.

    Article  MATH  Google Scholar 

  8. Wu C C, Frahm J M, Pollefeys M. Detecting large repetitive structures with salient boundaries. In: Proceedings of the 11th European Conference on Computer Vision. Berlin: Springer, 2010. 142–155

    Google Scholar 

  9. Ballester C, Bertalmio M, Caselles V, et al. Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans Image Process, 2001, 10: 1200–1211.

    Article  MathSciNet  Google Scholar 

  10. Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 2000. 417–424

    Google Scholar 

  11. Oliveira M M, Bowen B, McKenna R, et al. Fast digital image inpainting. In: Proceedings of the International Conference on Visualization, Imaging and Image Processing. New York: ACM Press, 2001. 261–266

    Google Scholar 

  12. Efros A A, Leung T K. Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision. Washington DC: IEEE, 1999. 1033–1038

    Chapter  Google Scholar 

  13. Wei L Y, Levoy M. Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 2000. 479–488

    Google Scholar 

  14. Ashikhmin M. Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics. New York: ACM Press, 2001. 217–226

    Chapter  Google Scholar 

  15. Efros A A, Freeman W T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 2001. 341–346

    Google Scholar 

  16. Kwatra V, Schodl A, Essa I, et al. Graphcut textures: image and video synthesis using graph cuts. ACM Trans Graph, 2003, 22: 277–286.

    Article  Google Scholar 

  17. Bertalmio M, Vese L A, Sapiro G, et al. Simultaneous structure and texture image inpainting. IEEE Trans Image Process, 2003, 12: 882–889.

    Article  MATH  Google Scholar 

  18. Jia J Y, Tang C K. Image repairing: robust image synthesis by adaptive Nd tensor voting. In: Proceedings of the 2003 IEEE Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE, 2003. 643–650

    Chapter  Google Scholar 

  19. Harrison P. A non-hierarchical procedure for re-synthesis of complex textures. In: Proceedings of the International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, Plzen, 2001. 190–197

    Google Scholar 

  20. Wilczkowiak M, Brostow G J, Tordoff B, et al. Hole filling through photomontage. In: Proceedings of the 16th British Machine Vision Conference, Oxford, 2005. 492–501

    Google Scholar 

  21. Kwatra V, Essa I, Bobick A, et al. Texture optimization for example-based synthesis. ACM Trans Graph, 2003, 24: 795–802.

    Article  Google Scholar 

  22. Komodakis N, Tziritas G. Image completion using global optimization. In: Proceedings of the 2006 IEEE Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE, 2006. 442–452

    Chapter  Google Scholar 

  23. Perez P, Gangnet M, Blake A. Patchworks: example-based region tiling for image editing. MSR-TR-2004-04. 2004

  24. Darabi S, Shechtman E, Barnes C, et al. Image melding: combining inconsistent images using patch-based synthesis. ACM Trans Graph, 2012, 31: 1–10.

    Article  Google Scholar 

  25. Pavic D, Schonefeld V, Kobbelt L. Interactive image completion with perspective correction. Vis Comput, 2006, 22: 671–681.

    Article  Google Scholar 

  26. Barnard S T. Interpreting perspective images. Artif Intell, 1983, 21: 435–462.

    Article  Google Scholar 

  27. Tuytelaars T, van Gool L, Proesmans M, et al. The cascaded hough transform as an aid in aerial image interpretation. In: Proceedings of the 6th International Conference on Computer Vision. Washington DC: IEEE, 1998. 67–72

    Google Scholar 

  28. Košsecka J, Zhang W. Video compass. In: Proceedings of the 7th European Conference on Computer Vision. Berlin: Springer, 2002. 476–490

    Google Scholar 

  29. Tardif J. Non-iterative approach for fast and accurate vanishing point detection. In: Proceedings of the 12th International Conference on Computer Vision. Washington DC: IEEE, 2009. 1250–1257

    Google Scholar 

  30. Wenzel S, Drauschke M, Forstner W. Detection of repeated structures in facade images. Patt Recog Image Anal, 2008, 18: 406–411.

    Article  Google Scholar 

  31. Loy G, Eklundh J O. Detecting symmetry and symmetric constellations of features. In: Proceedings of the 9th European Conference on Computer Vision. Berlin: Springer, 2006. 508–521

    Google Scholar 

  32. Horry Y, Anjyo K I, Arai K. Tour into the picture: using a spidery mesh interface to make animation from a single image. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 1997. 225–232

    Google Scholar 

  33. Debevec P E, Taylor C J, Malik J. Modeling and rendering architecture from photographs: a hybrid geometryand image-based approach. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, 1996. 11–20

    Google Scholar 

  34. Hao C Y, Wu E H. Interactive image completion with perspective constraint. In: Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-reality Continuum and Its Applications in Industry. New York: ACM Press, 2012. 153–160

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Science and Technology, University of Macau, Macau, 999078, China

    ChuanYan Hao, YaDang Chen, Wen Wu & EnHua Wu

  2. State Key Laboratory of Computer Science, Institute of Software, Chinese, Academy of Sciences, Beijing, 100864, China

    EnHua Wu

Authors
  1. ChuanYan Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. YaDang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. EnHua Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to ChuanYan Hao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hao, C., Chen, Y., Wu, W. et al. Image completion with perspective constraint based on a single image. Sci. China Inf. Sci. 58, 1–12 (2015). https://doi.org/10.1007/s11432-015-5359-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11432-015-5359-x

Keywords

关键词

Search

Navigation