The Visual Computer

, Volume 31, Issue 6–8, pp 1113–1122 | Cite as

Effective structure restoration for image completion using internet resources

  • Miao Hua
  • Wencheng Wang
Original Article


Structure restoration plays an important role in image completion. Though much progress has been made, existing techniques are ineffective to restore plausible complex structures, due to the high complexity of structure registration between neighboring regions. We address this challenge by taking two measures. The first is to get the images whose structures are potential to be well merged with the structures around the hole. This can be achieved by effectively retrieving the mass images on the internet, which provide enough candidates. The second is to coherently blend the retrieved structures with the structures around the hole, which is by suppressing the connection differences between these structures. Experimental results show that we can easily complete images, especially with complex structures or for filling large holes.


Image completion Structure retrieval Iterative closest point 



This work is partially supported by National Natural Science Foundation of China (61379087), the Knowledge Innovation Program of the Chinese Academy of Sciences and the European Unions Seventh Framework Programme (FP7/2007-2013) under grant agreement (612627).


  1. 1.
    Bar, M.: Visual objects in context. Nat. Rev. Neurosci. 5(8), 617–629 (2004)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24:1–24:11 (2009)CrossRefGoogle Scholar
  3. 3.
    Bertalmío, M., Vese, L.A., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. IEEE Trans. Image Process. 12(8), 882–889 (2003)CrossRefGoogle Scholar
  4. 4.
    Besl, P.J., McKay, N.D.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)CrossRefGoogle Scholar
  5. 5.
    Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992)CrossRefGoogle Scholar
  6. 6.
    Cheng, M.M., Zhang, F.L., Mitra, N.J., Huang, X., Hu, S.M.: Repfinder: finding approximately repeated scene elements for image editing. ACM Trans. Graph. 29, 83:1–83:8 (2010)CrossRefGoogle Scholar
  7. 7.
    Chia, A.Y.S., Zhuo, S., Gupta, R.K., Tai, Y.W., Cho, S.Y., Tan, P., Lin, S.: Semantic colorization with internet images. ACM Trans. Graph. 30(6), 156:1–156:8 (2011)CrossRefGoogle Scholar
  8. 8.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Img. Proc. 13(9), 1200–1212 (2004)Google Scholar
  9. 9.
    Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31(4), 82:1–82:10 (2012)CrossRefGoogle Scholar
  10. 10.
    Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. 31(4), 44:1–44:10 (2012)Google Scholar
  11. 11.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Vis. Comput. Graph. 17(11), 1624–1636 (2011)CrossRefGoogle Scholar
  12. 12.
    Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27(3), 67:1–67:10 (2008)CrossRefGoogle Scholar
  13. 13.
    Goldberg, C., Chen, T., Zhang, F.L., Shamir, A., Hu, S.M.: Data-driven object manipulation in images. Comput. Graph. Forum 31(2), 265–274 (2012)CrossRefGoogle Scholar
  14. 14.
    Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Trans. Graph. 26(3) (2007)Google Scholar
  15. 15.
    He, K., Sun, J.: Statistics of patch offsets for image completion. In: ECCV’12, pp. 16–29 (2012)Google Scholar
  16. 16.
    Ho, J., Peter, A.M., Rangarajan, A., Yang, M.H.: An algebraic approach to affine registration of point sets. In: ICCV, pp. 1335–1340 (2009)Google Scholar
  17. 17.
    Hu, S.M., Chen, T., Xu, K., Cheng, M.M., Martin, R.: Internet visual media processing: a survey with graphics and vision applications. Vis.Comput. 29(5), 393–405 (2013)CrossRefGoogle Scholar
  18. 18.
    Huang, H., Yin, K., Gong, M., Lischinski, D., Cohen-Or, D., Ascher, U., Chen, B.: “Mind the gap”: tele-registration for structure-driven image completion. ACM Trans. Graph. 32(6), 174:1–174:10 (2013)Google Scholar
  19. 19.
    Jia, J., Tang, C.K.: Image repairing: robust image synthesis by adaptive and tensor voting. In: CVPR, pp. 643–650 (2003)Google Scholar
  20. 20.
    Komodakis, N.: Image completion using global optimization. In: CVPR, pp. 442–452 (2006)Google Scholar
  21. 21.
    Liu, M.Y., Tuzel, O., Veeraraghavan, A., Chellappa, R.: Fast directional chamfer matching. In: CVPR, pp. 1696–1703 (2010)Google Scholar
  22. 22.
    Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., Gool, L.J.V.: Transforming image completion. In: BMVC, pp. 1–11 (2011)Google Scholar
  23. 23.
    Mu, T.J., Wang, J.H., Du, S.P., Hu, S.M.: Stereoscopic image completion and depth recovery. Vis. Comput. 30(6–8), 833–843 (2014)CrossRefGoogle Scholar
  24. 24.
    Nill, N.B., Bouzas, B.H.: Objective image quality measure derived from digital image power spectra. Opt. Eng. 31 (1992)Google Scholar
  25. 25.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: SIGGRAPH, pp. 313–318 (2003)Google Scholar
  26. 26.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)Google Scholar
  27. 27.
    Pritch, Y., Kav-Venaki, E., Peleg, S.: Shift-map image editing. In: ICCV, pp. 151–158 (2009)Google Scholar
  28. 28.
    Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: CVPR, pp. 1–8 (2008)Google Scholar
  29. 29.
    Sivic, J., Zisserman, A.: Video google: a text retrieval approach to object matching in videos. In: ICCV, pp. 1470–1477 (2003)Google Scholar
  30. 30.
    Sun, J., Yuan, L., Jia, J., Shum, H.Y.: Image completion with structure propagation. In: SIGGRAPH, pp. 861–868 (2005)Google Scholar
  31. 31.
    Wexler, Y., Shechtman, E., Irani, M.: Space–time video completion. In: CVPR, pp. 120–127 (2004)Google Scholar
  32. 32.
    Wexler, Y., Shechtman, E., Irani, M.: Space–time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.State Key Laboratory of Computer Science, Institute of SoftwareChinese Academy of SciencesBeijingChina
  2. 2.University of Chinese Academy of SciencesBeijingChina

Personalised recommendations