Skip to main content
SpringerLink
Log in

Capsule GAN for robust face super resolution

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Face hallucination is an emerging sub-field of Super-Resolution (SR) which aims to reconstruct the High-Resolution (HR) facial image given its Low-Resolution (LR) counterpart. The task becomes more challenging when the LR image is extremely small due to the image distortion in the super-resolved results. A variety of deep learning-based approaches has been introduced to address this issue by using attribute domain information. However, a more complex dataset or even further networks is required for training these models. In order to avoid these complexities and yet preserve the precision in reconstructed output, a robust Multi-Scale Gradient capsule GAN for face SR is proposed in this paper. A novel similarity metric called Feature SIMilarity (FSIM) is introduced as well. The proposed network surpassed state-of-the-art face SR systems in all metrics and demonstrates more robust performance while facing image transformations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Brock A, Donahue J, Simonyan K (2018) Large scale gan training for high fidelity natural image synthesis. arXiv:180911096

  2. Chen Y, Tai Y, Liu X, Shen C, Yang J (2018) Fsrnet: End-to-end learning face super-resolution with facial priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2492–2501

  3. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275

    Article  Google Scholar 

  4. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  5. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  6. Karnewar A, Wang O, Iyengar RS (2019) MSG-GAN: multi-scale gradient GAN for stable image synthesis. arXiv:1903.06048

  7. Kim D, Kim M, Kwon G, Kim DS (2019) Progressive face super-resolution via attention to facial landmark. arXiv:190808239

  8. Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  9. Kolouri S, Rohde GK (2015) Transport-based single frame super resolution of very low resolution face images. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 4876–4884

  10. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, et al. (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  11. Lee CH, Zhang K, Lee HC, Cheng CW, Hsu W (2018) Attribute augmented convolutional neural network for face hallucination. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 721–729

  12. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision, pp 3730–3738

  13. Liu C, Shum H Y, Freeman W T (2007) Face hallucination: Theory and practice. Int. J. Comput. Vis. 75(1):115–134

    Article  Google Scholar 

  14. Majdabadi M M, Ko SB (2020) Msg-capsgan: Multi-scale gradient capsule gan for face super resolution. In: 2020 International Conference on Electronics, Information, and Communication (ICEIC). IEEE, pp 1–3

  15. Rajput S S, Arya K (2019) A robust facial image super-resolution model via mirror-patch based neighbor representation. Multimed Tool Appl 78 (18):25407–25426

    Article  Google Scholar 

  16. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp 3856–3866. http://papers.nips.cc/paper/6975-dynamic-routing-between-capsules.pdf

  17. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  18. Wang Z, Bovik A C, Sheikh H R, Simoncelli E P (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  19. Wang Z, Simoncelli E P, Bovik A C (2003) Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol 2. IEEE, pp 1398–1402

  20. Wang X, Tang X (2005) Hallucinating face by eigentransformation. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 35(3):425–434

    Article  Google Scholar 

  21. Yu X, Fernando B, Hartley R, Porikli F (2018) Super-resolving very low-resolution face images with supplementary attributes. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 908–917

  22. Yu X, Porikli F (2016) Ultra-resolving face images by discriminative generative networks. In: European conference on computer vision. Springer, pp 318–333

  23. Yu X, Porikli F (2017) Face hallucination with tiny unaligned images by transformative discriminative neural networks. In: Thirty-First AAAI Conference on Artificial Intelligence

  24. Yu X, Porikli F (2017) Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3760–3768

  25. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  26. Zhu S, Liu S, Loy CC, Tang X (2016) Deep cascaded bi-network for face hallucination. In: European conference on computer vision. Springer, pp 614–630

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Saskatchewan, Saskatoon, Canada

    Mahdiyar Molahasani Majdabadi & Seok-Bum Ko

  2. Division of Biomedical Engineering, University of Saskatchewan, Saskatoon, Canada

    Seok-Bum Ko

Authors
  1. Mahdiyar Molahasani Majdabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seok-Bum Ko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seok-Bum Ko.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is the expansion of “MSG-CapsGAN: Multi-Scale gradient capsule GAN for face super-resolution,” in 2020 International Conference on Electronics, Information, and Communication (ICEIC), Barcelona, Spain, Jan. 2020

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Molahasani Majdabadi, M., Ko, SB. Capsule GAN for robust face super resolution. Multimed Tools Appl 79, 31205–31218 (2020). https://doi.org/10.1007/s11042-020-09489-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09489-y

Keywords

Search

Navigation