Skip to main content
SpringerLink
Log in

Image Inpainting Using Wasserstein Generative Adversarial Imputation Network

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2021 (ICANN 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12892))

Included in the following conference series:

Abstract

Image inpainting is one of the important tasks in computer vision which focuses on the reconstruction of missing regions in an image. The aim of this paper is to introduce an image inpainting model based on Wasserstein Generative Adversarial Imputation Network. The generator network of the model uses building blocks of convolutional layers with different dilation rates, together with skip connections that help the model reproduce fine details of the output. This combination yields a universal imputation model that is able to handle various scenarios of missingness with sufficient quality. To show this experimentally, the model is simultaneously trained to deal with three scenarios given by missing pixels at random, missing various smaller square regions, and one missing square placed in the center of the image. It turns out that our model achieves high-quality inpainting results on all scenarios. Performance is evaluated using peak signal-to-noise ratio and structural similarity index on two real-world benchmark datasets, CelebA faces and Paris StreetView. The results of our model are compared to biharmonic imputation and to some of the other state-of-the-art image inpainting methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/vasatdan/wgain-inpaint.

  2. 2.

    https://www.tensorflow.org.

  3. 3.

    https://scikit-image.org/.

References

  1. Amrani, N., Serra-Sagristà, J., Peter, P., Weickert, J.: Diffusion-based inpainting for coding remote-sensing data. IEEE Geosci. Remote Sens. Lett. 14(8), 1203–1207 (2017)

    Article  Google Scholar 

  2. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001)

    Article  MathSciNet  Google Scholar 

  3. Barnum, A., Jiao, J.: Adaptive biharmonic in-painting for sparse acquisition using variance frames. Microsc. Microanal. 23(S1), 148–149 (2017). https://doi.org/10.1017/S1431927617001428

    Article  Google Scholar 

  4. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417–424 (2000)

    Google Scholar 

  5. Cai, W., Wei, Z.: PiiGAN: generative adversarial networks for pluralistic image inpainting. IEEE Access 8, 48451–48463 (2020)

    Article  Google Scholar 

  6. Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression. arXiv preprint arXiv:1401.4112 (2014)

  7. Damelin, S.B., Hoang, N.S.: On surface completion and image inpainting by biharmonic functions: numerical aspects. Int. J. Math. Math. Sci. 2018, 1–8 (2018)

    Article  MathSciNet  Google Scholar 

  8. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346 (2001)

    Google Scholar 

  9. Friedjungová, M., Vašata, D., Balatsko, M., Jiřina, M.: Missing features reconstruction using a Wasserstein generative adversarial imputation network. In: Krzhizhanovskaya, V.V., et al. (eds.) ICCS 2020. LNCS, vol. 12140, pp. 225–239. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50423-6_17

    Chapter  Google Scholar 

  10. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates, Inc. (2014)

    Google Scholar 

  11. Hua, P., Liu, X., Liu, M., Dong, L., Hui, M., Zhao, Y.: Image inpainting using Wasserstein generative adversarial network. In: Optics and Photonics for Information Processing XII, vol. 10751, pp. 183–194. SPIE (2018). https://doi.org/10.1117/12.2320212

  12. Hui, Z., Li, J., Wang, X., Gao, X.: Image fine-grained inpainting. arXiv preprint arXiv:2002.02609 (2020)

  13. Jam, J., Kendrick, C., Drouard, V., Walker, K., Hsu, G.S., Yap, M.H.: Symmetric skip connection Wasserstein GAN for high-resolution facial image inpainting. arXiv preprint arXiv:2001.03725 (2020)

  14. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017)

  15. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

  16. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2016). https://doi.org/10.1109/cvpr.2016.278

  17. Rubner, Y., Guibas, L.J., Tomasi, C.: The earth mover’s distance, multi-dimensional scaling, and color-based image retrieval. In: Proceedings of the ARPA Image Understanding Workshop, vol. 661, p. 668 (1997)

    Google Scholar 

  18. Shin, Y.G., Sagong, M.C., Yeo, Y.J., Kim, S.W., Ko, S.J.: Pepsi++: fast and lightweight network for image inpainting. IEEE Trans. Neural Netw. Learn. Syst. (2020)

    Google Scholar 

  19. Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

    Google Scholar 

  20. Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. Tools 9(1), 23–34 (2004). https://doi.org/10.1080/10867651.2004.10487596

    Article  Google Scholar 

  21. Villani, C.: Optimal Transport: Old and New, vol. 338. Springer, Heidelberg (2008)

    Google Scholar 

  22. Wang, Z., Bovik, A.C.: Mean squared error: love it or leave it? a new look at signal fidelity measures. IEEE Signal Process. Mag. 26(1), 98–117 (2009)

    Article  Google Scholar 

  23. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  24. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. CoRR abs/1511.07122 (2016)

    Google Scholar 

  25. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)

    Google Scholar 

  26. Zheng, C., Cham, T.J., Cai, J.: Pluralistic image completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1438–1447 (2019)

    Google Scholar 

Download references

Acknowledgements

This research has been supported by SGS grant No. SGS20/213/OHK3/3T/18, by GACR grant No. GA18-18080S, and by the Student Summer Research Program 2020 of FIT CTU in Prague, Czech Republic.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Czech Technical University in Prague, Prague, Czech Republic

    Daniel Vašata, Tomáš Halama & Magda Friedjungová

Authors
  1. Daniel Vašata
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tomáš Halama
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Magda Friedjungová
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Magda Friedjungová .

Editor information

Editors and Affiliations

  1. Comenius University in Bratislava, Bratislava, Slovakia

    Igor Farkaš

  2. iMotions A/S, Copenhagen, Denmark

    Paolo Masulli

  3. University of Tübingen, Tübingen, Baden-Württemberg, Germany

    Sebastian Otte

  4. Universität Hamburg, Hamburg, Germany

    Stefan Wermter

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vašata, D., Halama, T., Friedjungová, M. (2021). Image Inpainting Using Wasserstein Generative Adversarial Imputation Network. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12892. Springer, Cham. https://doi.org/10.1007/978-3-030-86340-1_46

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86340-1_46

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86339-5

  • Online ISBN: 978-3-030-86340-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation