Skip to main content
SpringerLink
Log in

A Survey of Face Image Inpainting Based on Deep Learning

  • Conference paper
  • First Online:
Cloud Computing (CloudComp 2021)

Included in the following conference series:

  • 976 Accesses

Abstract

In recent years, deep learning has become the mainstream method of image inpainting. It can not only repair the texture of the image, obtain high-level abstract features of the image, but also recover semantic images such as human faces. Among these methods, attention mechanisms, semantic methods, and progressive networks have become very promising image inpainting models. These models implement end-to-end image inpainting and generate visually reasonable and clear image structure and texture. This paper briefly describes the face inpainting technology and summarizes the existing face image inpainting methods. We try to collect most of the face inpainting methods based on deep learning, divide them into attentional, semantic-based, and progressive inpainting networks, and prorate the methods proposed by researchers in each category in recent years. Then we summarize the dataset proposed by the predecessors and the evaluation index of the algorithm performance. Finally, we summarize the current situation and future development trends of face inpainting.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27 (2014)

    Google Scholar 

  2. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  3. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)

    Google Scholar 

  4. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)

    Article  Google Scholar 

  5. Huang, J.B., Kang, S.B., Ahuja, N., Kopf, J.: Image completion using planar structure guidance. ACM Trans. Graph. 33(4), 1–10 (2014)

    Google Scholar 

  6. Yan, Z., Li, X., Li, M., Zuo, W., Shan, S.: Shift-Net: image inpainting via deep feature rearrangement. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_1

    Chapter  Google Scholar 

  7. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  8. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)

    Google Scholar 

  9. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 1–14 (2017)

    Article  Google Scholar 

  10. Song, Y., et al.: Contextual-based image inpainting: infer, match, and translate. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 3–18. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_1

    Chapter  Google Scholar 

  11. Song, Y., Yang, C., Shen, Y., Wang, P., Huang, Q., Kuo, C.C.J.: SPG-Net: segmentation prediction and guidance network for image inpainting. arXiv preprint arXiv:03356 (2018)

  12. Jo, Y., Park, J.: SC-FEGAN: face editing generative adversarial network with user’s sketch and color. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1745–1753 (2019)

    Google Scholar 

  13. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4471–4480 (2019)

    Google Scholar 

  14. Xiao, Q., Li, G., Chen, Q.: Deep inception generative network for cognitive image inpainting. arXiv preprint arXiv:01458 (2018)

  15. Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: EdgeConnect: generative image inpainting with adversarial edge learning. arXiv preprint arXiv:00212 (2019)

  16. Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6

    Chapter  Google Scholar 

  17. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  18. Xiao, Z., Li, D.: Generative image inpainting by hybrid contextual attention network. In: Lokoč, J., Patras, I. (eds.) MMM 2021. LNCS, vol. 12572, pp. 162–173. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67832-6_14

    Chapter  Google Scholar 

  19. Xie, C., et al.: Image inpainting with learnable bidirectional attention maps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8858–8867 (2019)

    Google Scholar 

  20. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)

    Google Scholar 

  21. Mohite, T.A., Phadke, G.S.: Image inpainting with contextual attention and partial convolution. In: 2020 International Conference on Artificial Intelligence and Signal Processing (AISP), pp. 1–6. IEEE (2020)

    Google Scholar 

  22. He, X., Cui, X., Li, Q.J.I.A.: Image inpainting based on inside-outside attention and wavelet decomposition. IEEE Access 8, 62343–62355 (2020)

    Article  Google Scholar 

  23. Qiu, J., Gao, Y.: Position and channel attention for image inpainting by semantic structure. In: 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 1290–1295. IEEE (2020)

    Google Scholar 

  24. Wu, H., Zhou, J.: IID-Net: image inpainting detection network via neural architecture search and attention. IEEE Trans. Circ. Technol. Syst. Video (2021)

    Google Scholar 

  25. Wang, C., Wang, J., Zhu, Q., Yin, B.: Generative image inpainting based on wavelet transform attention model. In: 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5. IEEE (2020)

    Google Scholar 

  26. Li, J., Wang, N., Zhang, L., Du, B., Tao, D.: Recurrent feature reasoning for image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7760–7768 (2020)

    Google Scholar 

  27. Wang, N., Ma, S., Li, J., Zhang, Y., Zhang, L.J.P.R.: Multistage attention network for image inpainting. Pattern Recognit. 106, 107448 (2020)

    Article  Google Scholar 

  28. Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4634–4643 (2019)

    Google Scholar 

  29. Song, L., et al.: Unsupervised domain adaptive re-identification: theory and practice. Pattern Recognit. 102, 107173 (2020)

    Article  Google Scholar 

  30. Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4170–4179 (2019)

    Google Scholar 

  31. Chen, B., Li, P., Sun, C., Wang, D., Yang, G., Lu, H.: Multi attention module for visual tracking. Pattern Recogn. 87, 80–93 (2019)

    Article  Google Scholar 

  32. Uddin, S., Jung, Y.J.: Global and local attention-based free-form image inpainting. Sensors 20(11), 3204 (2020)

    Article  Google Scholar 

  33. Jiao, L., Wu, H., Wang, H., Bie, R.: Multi-scale semantic image inpainting with residual learning and GAN. Neurocomputing 331, 199–212 (2019)

    Article  Google Scholar 

  34. Zheng, C., Cham, T.J., Cai, J.: Pluralistic image completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1438–1447 (2019)

    Google Scholar 

  35. Zeng, Y., Fu, J., Chao, H., Guo, B.: Learning pyramid-context encoder network for high-quality image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1486–1494 (2019)

    Google Scholar 

  36. Vitoria, P., Sintes, J., Ballester, C.: Semantic image inpainting through improved Wasserstein generative adversarial networks. arXiv preprint arXiv:01071 (2018)

  37. Yeh, R.A., Chen, C., Yian Lim, T., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5485–5493

    Google Scholar 

  38. Qiu, J., Gao, Y., Shen, M.: Semantic-SCA: semantic structure image inpainting with the spatial-channel attention. IEEE Access 9, 12997–13008 (2021)

    Article  Google Scholar 

  39. Zhang, F., Wang, X., Sun, T., Xu, X.: SE-DCGAN: a new method of semantic image restoration. Cogn. Comput. 13, 1–11 (2021)

    Article  Google Scholar 

  40. Zhang, H., Hu, Z., Luo, C., Zuo, W., Wang, M.: Semantic image inpainting with progressive generative networks. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1939–1947 (2018)

    Google Scholar 

  41. Wang, W., Gu, E., Fang, W.: An improvement of coherent semantic attention for image inpainting. In: Sun, X., Wang, J., Bertino, E. (eds.) ICAIS 2020. CCIS, vol. 1252, pp. 267–275. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-8083-3_24

    Chapter  Google Scholar 

  42. Yang, W., Li, X., Zhang, L.: Toward semantic image inpainting: where global context meets local geometry. J. Electron. Imaging 30(2), 023028 (2021)

    Google Scholar 

  43. Ciobanu, S., Ciortuz, L.: Semantic image inpainting via maximum likelihood. In: 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 153–160. IEEE (2020)

    Google Scholar 

  44. Shen, Z., Lai, W.S., Xu, T., Kautz, J., Yang, M.H.: Deep semantic face deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8260–8269 (2018)

    Google Scholar 

  45. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6721–6729 (2017)

    Google Scholar 

  46. Ma, B., An, X., Sun, N.: Face image inpainting algorithm via progressive generation network. In: 2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP), pp. 175–179. IEEE (2020)

    Google Scholar 

  47. Xiong, W., et al.: Foreground-aware image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5840–5848 (2019)

    Google Scholar 

  48. Zeng, Yu., Lin, Z., Yang, J., Zhang, J., Shechtman, E., Lu, H.: High-resolution image inpainting with iterative confidence feedback and guided upsampling. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 1–17. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_1

    Chapter  Google Scholar 

  49. Yen, S.H., Yeh, H.Y., Chang, H.W.: Progressive completion of a panoramic image. Multimedia Tools Appl. 76(9), 11603–11620 (2017)

    Article  Google Scholar 

  50. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:10196 (2017)

  51. Guo, Z., Chen, Z., Yu, T., Chen, J., Liu, S.: Progressive image inpainting with full-resolution residual network. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2496–2504 (2019)

    Google Scholar 

  52. Huang, Z., Qin, C., Liu, R., Weng, Z., Zhu, Y.: Semantic-aware context aggregation for image inpainting. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2465–2469. IEEE (2021)

    Google Scholar 

  53. Li, J., He, F., Zhang, L., Du, B., Tao, D.: Progressive reconstruction of visual structure for image inpainting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5962–5971 (2019)

    Google Scholar 

  54. Zamir, S.W., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)

    Google Scholar 

  55. Yang, Y., Guo, X., Ma, J., Ma, L., Ling, H.: LAFIN: generative landmark guided face inpainting. arXiv preprint arXiv:11394 (2019)

  56. Gao, W., et al.: The CAS-PEAL large-scale Chinese face database and baseline evaluations. IEEE Trans. Syst. Man Syst. Cybernet. Part A Hum. 38(1), 149–161 (2007)

    Google Scholar 

  57. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  58. Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_49

    Chapter  Google Scholar 

  59. Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. arXiv preprint arXiv (2014)

    Google Scholar 

  60. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Intell. Mach. 40(6), 1452–1464 (2017)

    Article  Google Scholar 

  61. Liu, Z., Luo, P., Wang, X., Tang, X.: Large-scale CelebFaces attributes (CelebA) dataset. Retrieved August 15, 11 (2018)

    Google Scholar 

  62. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded by the National Natural Science Foundation of China (12163004), the basic applied research program of Yunnan Province (202001AT070135, 202101AS070007, 202002AD080002, 2018FB105).

Author information

Authors and Affiliations

  1. College of Big Data and Intelligent Engineering, Southwest Forestry University, Kunming, 650224, China

    Shiqi Su, Xiaofeng Shao, Yuxuan Zuo & Zhenping Qiang

  2. Yunnan Institute of Product Quality Supervision and Inspection, Kunming, 650214, China

    Miao Yang

  3. Information Security College, Yunnan Police College, Kunming, 650223, China

    Libo He

Authors
  1. Shiqi Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Libo He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaofeng Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuxuan Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhenping Qiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenping Qiang .

Editor information

Editors and Affiliations

  1. Persian Gulf University, Bushehr, Iran

    Mohammad R. Khosravi

  2. Swinburne University of Technology, Hawthorn, VIC, Australia

    Qiang He

  3. Nanjing University, Nanjing, China

    Haipeng Dai

Rights and permissions

Reprints and permissions

Copyright information

© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Su, S., Yang, M., He, L., Shao, X., Zuo, Y., Qiang, Z. (2022). A Survey of Face Image Inpainting Based on Deep Learning. In: Khosravi, M.R., He, Q., Dai, H. (eds) Cloud Computing. CloudComp 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 430. Springer, Cham. https://doi.org/10.1007/978-3-030-99191-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99191-3_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99190-6

  • Online ISBN: 978-3-030-99191-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation