Skip to main content
SpringerLink
Log in

Facial mask attention network for identity-aware face super-resolution

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Face Super-Resolution (FSR) is a crucial research topic in image restoration field, which is a fundamental task for subsequent face applications, such as cross- and low-resolution face recognition. Recently, supported by deep convolutional neural networks, the previous FSR methods have achieved great success in generating high quality face images. However, they mainly focus on improving the visual effects of the images while retaining a challenge of restoring identity information from low-resolution faces. Specifically, some face structure information is discarded, such as the position and the shape of the face components, containing useful identity-related details. To solve this issue, we propose the Facial Mask Attention Network utilizing this information to generate faces of both high resolution and identity fidelity. Furthermore, we present an efficient pixel loss function, MaskPix loss, which selectively emphasizes those significant pixels to focus the model on the face regions with dense identity features. Extensive experiments on popular datasets demonstrate that our restored face images not only have more natural textures and facial details, but also preserve higher identity fidelity compared to the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2d and 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In: Proceedings of the IEEE international conference on computer vision, pp 1021–1030

  2. Bulat A, Tzimiropoulos G (2018) Super-fan: integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 109–117

  3. Chen C, Gong D, Wang H et al (2020) Learning spatial attention for face super-resolution. IEEE Trans Image Process 30:1219–1231

    Article  Google Scholar 

  4. Chen C, Li X, Yang L, et al (2021) Progressive semantic-aware style transformation for blind face restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11,896–11,905

  5. Chen Y, Tai Y, Liu X, et al (2018) Fsrnet: End-to-end learning face super-resolution with facial priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2492–2501

  6. Deng J, Dong W, Socher R, et al (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255

  7. Deng J, Guo J, Xue N, et al (2019) Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4690–4699

  8. Dong C, Loy CC, He K et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  9. Farooq M, Dailey MN, Mahmood A et al (2021) Human face super-resolution on poor quality surveillance video footage. Neural Comput Appl 33(20):13,505-13,523

    Article  Google Scholar 

  10. Gao G, Zhu D, Yang M et al (2020) Face image super-resolution with pose via nuclear norm regularized structural orthogonal procrustes regression. Neural Comput Appl 32(9):4361–4371

    Article  Google Scholar 

  11. Guo Y, Zhang L, Hu Y, et al (2016) Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In: European conference on computer vision. Springer, pp 87–102

  12. He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  13. Heusel M, Ramsauer H, Unterthiner T, et al (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv Neural Inf Process Syst 30

  14. Huang GB, Mattar M, Berg T, et al (2008) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: Workshop on faces in ’Real-Life’ Images: detection, alignment, and recognition

  15. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, pp 694–711

  16. Kim D, Kim M, Kwon G, et al (2019) Progressive face super-resolution via attention to facial landmark. arXiv:1908.08239

  17. Kim J, Lee JK, Lee KM (2016a) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  18. Kim J, Lee JK, Lee KM (2016b) Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1645

  19. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  20. Ledig C, Theis L, Huszár F, et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  21. Li Y, Liu S, Yang J, et al (2017) Generative face completion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3911–3919

  22. Liu Z, Luo P, Wang X, et al (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision, pp 3730–3738

  23. Lu T, Zeng K, Qu S et al (2020) Face super-resolution via nonlinear adaptive representation. Neural Comput Appl 32(15):11,637-11,649

    Article  Google Scholar 

  24. Lu T, Wang Y, Zhang Y, et al (2021) Face hallucination via split-attention in split-attention network. In: Proceedings of the 29th ACM international conference on multimedia, pp 5501–5509

  25. Ma C, Jiang Z, Rao Y, et al (2020) Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5569–5578

  26. Meishvili G, Jenni S, Favaro P (2020) Learning to have an ear for face super-resolution. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1364–1374

  27. Moschoglou S, Papaioannou A, Sagonas C, et al (2017) Agedb: the first manually collected, in-the-wild age database. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 51–59

  28. Paszke A, Gross S, Massa F et al (2019) Pytorch: an imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 32:8026–8037

    Google Scholar 

  29. Qin D, Gu X (2020) Single-image super-resolution with multilevel residual attention network. Neural Comput Appl 32(19):15615–15628

    Article  Google Scholar 

  30. Sagonas C, Tzimiropoulos G, Zafeiriou S, et al (2013) A semi-automatic methodology for facial landmark annotation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 896–903

  31. Shi W, Caballero J, Huszár F, et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  32. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  33. Tai Y, Yang J, Liu X (2017a) Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3147–3155

  34. Tai Y, Yang J, Liu X, et al (2017b) Memnet: A persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vision, pp 4539–4547

  35. Wang TC, Liu MY, Zhu JY, et al (2018) High-resolution image synthesis and semantic manipulation with conditional gans. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8798–8807

  36. Wang X, Li Y, Zhang H, et al (2021) Towards real-world blind face restoration with generative facial prior. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9168–9178

  37. Yi D, Lei Z, Liao S, et al (2014) Learning face representation from scratch. arXiv:1411.7923

  38. Yu X, Fernando B, Ghanem B, et al (2018a) Face super-resolution guided by facial component heatmaps. In: Proceedings of the European conference on computer vision (ECCV), pp 217–233

  39. Yu X, Fernando B, Hartley R, et al (2018b) Super-resolving very low-resolution face images with supplementary attributes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 908–917

  40. Zhang M, Ling Q (2020) Supervised pixel-wise GAN for face super-resolution. IEEE Trans Multimed 23:1938–1950

    Article  Google Scholar 

  41. Zhang R, Isola P, Efros AA, et al (2018a) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 586–595

  42. Zhang Y, Tian Y, Kong Y, et al (2018b) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  43. Zhang Y, Tsang IW, Luo Y, et al (2020) Copy and paste GAN: face hallucination from shaded thumbnails. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7355–7364

Download references

Acknowledgements

This work was supported by Key-Area Research and Development Program of Guangdong Province under Grant (2020B1111010002, 2018B010109001, 2019B020214001) and Guangdong Marine Economic Development Project under Grant GDNRC[2020]018.

Author information

Authors and Affiliations

  1. School of Automation Science and Engineering, South China University of Technology, Guangzhou, 510641, China

    Zhengzheng Sun, Lianfang Tian, Qiliang Du, Jameel A. Bhutto & Zhaolin Wang

  2. Southern Marine Science and Engineering Guangdong Laboratory, Zhuhai, 519000, China

    Lianfang Tian

  3. Key Laboratory of Marine Environmental Survey Technology and Application, Ministry of Natural Resources, Guangzhou, 510641, China

    Lianfang Tian

  4. Key Laboratory of Autonomous Systems and Network Control of Ministry of Education, Guangzhou, 510641, China

    Qiliang Du

Authors
  1. Zhengzheng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lianfang Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiliang Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jameel A. Bhutto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhaolin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhengzheng Sun or Qiliang Du.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest in connection with the work entitled “Facial Mask Attention Network for Identity-Aware Face Super-Resolution.”

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, Z., Tian, L., Du, Q. et al. Facial mask attention network for identity-aware face super-resolution. Neural Comput & Applic 35, 8243–8257 (2023). https://doi.org/10.1007/s00521-022-08098-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-08098-0

Keywords

Search

Navigation