Skip to main content
SpringerLink
Log in

Image completion by structure reconstruction and texture synthesis

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

We present a new image completion method that can deal with large holes surrounded by different types of structure and texture. Our approach is based upon creating image structure in the hole while preserving global image structure, and then creating texture in the hole constrained by this structure. The images are segmented into homogeneous regions. Similar regions touching the hole are linked, resulting in new areas in the hole that are flood-filled and made to match the geometry of the surrounding structure to provide a globally spatially coherent and plausible topology. This reconstructed structure is then used as a constraint for texture synthesis. The contribution of the paper is two-fold. Firstly, we propose an algorithm to link regions around the hole to create topologically consistent structure in the hole, the structure being then made to match that of the rest of the image, using a texture synthesis method. Secondly, we propose a synthesis method akin to simulated annealing that allows global randomness and fine detail that match given examples. This method was developed particularly to create structure (texture in label images) but can also be used for continuous valued images (texture).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. We understand that some authors have used the term ‘inpainting’ to mean general image completion. We limit the meaning to completion of small areas, as described in the text.

References

  1. Alotaibi N (2009) Image region completion by structure reconstruction and texture synthesis. PhD thesis, Department of Computer Science, Aberystyth University, UK

  2. Ashikhmin M (2001) Synthesizing natural textures. In Proceedings of the symposium on Interactive 3D graphics, pp 217–226

  3. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) PatchMatch: a randomized correspondence algorithm for structural image editing. In: Proceedings of SIGGRAPH

  4. Bertalmío M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of SIGGRAPH, pp 417–424

  5. Bertalmío M, Vese LA, Sapiro G, Osher S (2003) Simultaneous structure and texture image inpainting. IEEE Trans Image Process 12(8):882–889

    Article  Google Scholar 

  6. Bonneel N, van de Panne M, Lefebvre S, Drettakis G (2010) Proxy-guided texture synthesis for rendering natural scenes. In: Proceedings of vision modeling and visualization, pp 87–95

  7. Bornard R, Lecan E, Laborelli L, Chenot J-H (2002) Missing data correction in still images and image sequences. In: Proceedings of the ACM international conference on multimedia, pp 355–361

  8. Chan T, Shen J (2000) Non-texture inpainting by curvature-driven diffusions (CDD). Technical report, Department of Mathematics, University of California, Los Angeles

  9. Chan T, Shen J (2002) Mathematical models for local nontexture inpaintings. J Appl Math (SIAM 02) 62(3):1019–1043

    Article  MATH  MathSciNet  Google Scholar 

  10. Criminisi A, Pérez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13(9):1200–1212

    Article  Google Scholar 

  11. De Bonet JS (1997) Multiresolution sampling procedure for analysis and synthesis of texture images. In Proceedings of SIGGRAPH, pp 361–368

  12. Deng Y, Manjunath BS (2001) Unsupervised segmentation of color-texture regions in images and video. IEEE Trans Pattern Anal Mach Intell 23(8):800–810

    Article  Google Scholar 

  13. Drori I, Cohen-Or D, Yeshurun H (2003) Fragment-based image completion. ACM Trans Graphics 22(3):303–312

    Article  Google Scholar 

  14. Efros A, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of SIGGRAPH, pp 341–346

  15. Efros A, Leung T (1999) Texture synthesis by non-parametric sampling. In: Proceedings of the international conference on computer vision, pp 1033–1038

  16. Eisenacher C, Tappan C, Burley B, Teece D, Shek A (2010) Example-based texture synthesis on Disney’s tangled. In: Proceedings of ACM SIGGRAPH, production talks

  17. Hertzmann S, Jacobs CE, Oliver N, Curless B, Salesin DH (2001) Image analogies. In: Proceedings of SIGGRAPH, pp 327–340

  18. Hoiem D, Efros AA, Hebert M (2011) Recovering occlusion boundaries from an image. Int J Comput Vision 91(3):328–346

    Article  MATH  MathSciNet  Google Scholar 

  19. Holtzman-Gazit M, Yavneh I (2008) A scale consistent approach to image completion. Int J Multiscale Comput Eng 6(6):617–628

    Article  Google Scholar 

  20. Hongying Z, Qicong P, Yadong W (2007) Image completion algorithm based on texture synthesis. J Syst Eng Electron 18(2):385–391

    Article  Google Scholar 

  21. Igehy H, Pereira L (1997) Image replacement through texture synthesis. In: Proceedings of the international conference on image processing, pp 186–189

  22. Jia J, Tang C-K (2004) Inference of segmented color and texture description by tensor voting. IEEE Trans Pattern Anal Mach Intell 26(6):771–786

    Article  Google Scholar 

  23. Komodakis N, Tziritas G (2007) Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans Image Process 16(11):2649–2661

    Google Scholar 

  24. Kwatra V, Schödl A, Essa I, Turk G, Bobick A (2003) Graphcut textures: image and video synthesis using graph cuts. ACM Trans Graphics 22(3):277–286

    Article  Google Scholar 

  25. Labrosse F (2003) On the editing of images: selecting, cutting and filling-in. In: Proceedings of the international conference on vision, video, and graphics, pp 71–78

  26. Lefebvre S, Hoppe H (2005) Parallel controllable texture synthesis. ACM Trans Graphics 24(3):777–786

    Article  Google Scholar 

  27. Liang L, Liu C, Xu Y-Q, Guo B, Shum H-Y (2001) Real-time texture synthesis by patch-based sampling. ACM Trans Graphics 20(3):127–150

    Article  Google Scholar 

  28. Long J, Mould D (2007) Improved image quilting. In: Proceedings of graphics interface, pp 257–264

  29. Mansfield A, Prasad M, Rother C, Sharp T, Kohli P, Van Gool L (2011) Transforming image completion. In: Proceedings of the british machine vision conference

  30. Masnou S (2002) Disocclusion: a variational approach using level lines. IEEE Trans Image Process 11(2):68–76

    Article  MathSciNet  Google Scholar 

  31. Mumford D (1994) Elastica and computer vision. In: Bajaj CL (ed) Algebraic geometry and its applications. Springer, New York, pp 491–506

  32. Nealen A, Alexa M (2004) Fast and high quality overlap repair for patch-based texture synthesis. In: Proceedings of computer graphics international, pp 582–585

  33. Rareş A, Reinders MJT, Biemond J (2005) Constrained texture restoration. EURASIP J Appl Signal Process 2005(17):2758–2771

    Article  MATH  Google Scholar 

  34. Rosenberger A, Cohen-Or D, Lischinski D (2009) Layered shape synthesis: automatic generation of control maps for non-stationary textures. ACM Trans Graphics 28(5)

  35. Shekhovtsov A, Kohli P, Rother C (2012) Curvature prior for MRF-based segmentation and shape inpainting. Pattern recognition, pp 41–51

  36. Shen J, Jin X, Zhou C, Wang CCL (2007) Gradient based image completion by solving the poisson equation. Comput Graphics 31(1):119–126

    Article  Google Scholar 

  37. Shin S, Nishita T, Shin SY (2006) On pixel-based texture synthesis by non-parametric sampling. Comput Graphics 30(5):767–778

    Article  Google Scholar 

  38. Sun J, Yuan L, Jia J, Shum H-Y (2005) Image completion with structure propagation. ACM Trans Graphics 24(3):861–868

    Article  Google Scholar 

  39. Wei L-Y, Levoy M (2000) Fast texture synthesis using tree-structured vector quantization. In: Proceedings of SIGGRAPH

  40. Wei L-Y, Levoy M (2002) Order-independent texture synthesis. Technical Report TR-2002-01. Computer Science Department, Stanford University

  41. Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans Pattern Anal Mach Intell 29(3):463–476

    Article  Google Scholar 

  42. Zhang Y, Xiao X, Shah M (2004) Region completion in a single image. In: Proceedings of the eurographics conference, vol 23

Download references

Acknowledgments

The authors wish to thank A. Criminisi, P. Pérez, K. Toyama, I. Drori, D. Cohen-Or and H. Yeshurun for allowing us to use their images, and the anonymous reviewers for their constructive and useful comments.

Author information

Authors and Affiliations

  1. Department of Computer Science, Aberystwyth University, Llandinam Building, Aberystwyth, SY23 3DB, UK

    Najm Alotaibi & Frédéric Labrosse

Authors
  1. Najm Alotaibi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frédéric Labrosse
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Frédéric Labrosse.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Alotaibi, N., Labrosse, F. Image completion by structure reconstruction and texture synthesis. Pattern Anal Applic 18, 333–350 (2015). https://doi.org/10.1007/s10044-013-0348-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-013-0348-4

Keyword

Search

Navigation