Abstract
Image inpainting is a challenging task due to the loss of the image information. Recently, GAN-based approaches have shown promising performance in the field of image inpainting. For this task, a superior similarity measurement of extracted patches from known and missing regions is important. Existing approaches usually adopt cosine distance to measure this similarity for missing region reconstruction. However, from the semantic-level perspective, these methods often generate content with inconsistent color and disorder structure due to the ignorance of the magnitude distance of the attended patches. To resolve this problem, we propose a Hybrid Contextual Attention Network (HCA-Net) with a novel attention module called hybrid contextual attention module (HCAM). HCAM takes account of both cosine distance and Euclidean distance as the measurement of the extracted patches and gives a better prediction of missing features. Besides, a Spectral-Normalization patch discriminator and the cosine loss are added into the model for patch-level and pixel-level consistency enhancement. Extensive results on three public datasets (Paris Street View, Celeba-HQ, and Places2), have both validated that our approach significantly outperforms the state-of-the-art approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. (Proc. SIGGRAPH) 28(3), 1–11 (2009)
Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv. Neural. Inf. Process. Syst. 30, 6626–6637 (2017)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image com-pletion. ACM Trans. Graph. (ToG) 36(4), 1–14 (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Lim, J.H., Ye, J.C.: Geometric gan. arXiv preprint arXiv:1705.02894 (2017)
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (2015)
Marnerides, D., Bashford-Rogers, T., Hatchett, J., Debattista, K.: Expandnet: a deep convolutional neural network for high dynamic range expansion from low dynamic range content. In: Computer Graphics Forum, vol. 37, pp. 37–49. Wiley Online Library (2018)
Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context en- coders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Ting, H., Chen, S., Liu, J., Tang, X.: Image inpainting by global structure and texture propagation. In: Proceedings of the 15th ACM international conference on Multimedia, pp. 517–520 (2007)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. IEEE Trans. Image Process. 19(5), 1153–1165 (2010)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4471–4480 (2019)
Zeng, Y., Fu, J., Chao, H., Guo, B.: Learning pyramid-context encoder network for high-quality image inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1486–1494 (2019)
Zhong, S.h., Liu, Y., Zhang, Y., Chung, F.l.: Attention modeling for face recognition via deep learning. In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 34 (2012)
Zhou, B., Khosla, A., Lapedriza, A., Torralba, A., Oliva, A.: Places2: a large-scale database for scene understanding. Arxiv preprint (2015)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (Grant 62002230).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Xiao, Z., Li, D. (2021). Generative Image Inpainting by Hybrid Contextual Attention Network. In: Lokoč, J., et al. MultiMedia Modeling. MMM 2021. Lecture Notes in Computer Science(), vol 12572. Springer, Cham. https://doi.org/10.1007/978-3-030-67832-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-67832-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67831-9
Online ISBN: 978-3-030-67832-6
eBook Packages: Computer ScienceComputer Science (R0)