Skip to main content
SpringerLink
Log in

SRPSGAN: Super-resolution with pose and expression robust spatial-aware generative adversarial network for makeup transfer

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we address special scenario makeup-transfer tasks designed to transfer makeup features from a low-resolution (LR) reference image to a LR source image and generate high-resolution (HR) results. Existing methods are not suitable when limited by storage space, especially when HR images are unavailable. To overcome this problem, we propose the super-resolution (SR) pose-and-expression robust spatial-aware (PS) generative adversarial network (GAN), which makes full use of the prior information while preserving the original network properties (i.e., robustness of pose and expression during makeup transfer). Specifically, it feeds the results of the PSGAN (Jiang et al. 20) into the prior estimation network (PEnet) to estimate facial landmarks and parsing maps. Then, a feature map containing makeup information extracted from the PSGAN is used as additional prior information. Both the feature map and the PEnet results are sent to the SR network to generate HR makeup-transfer images. Moreover, we propose a novel makeup recommender that integrates three evaluation metrics to provide superior decision support. Our extensive experiments show that the SRPSGAN achieves excellent results in SR makeup-transfer tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available in the google drive, https://drive.google.com/drive/folders/1ubqJ49ev16NbgJjjTt-Q75mNzvZ7sEEn

References

  1. Alashkar T, Jiang S, Fu Y (2017) Rule-based facial makeup recommendation system. In: 2017 12th IEEE International conference on automatic face & gesture recognition (FG 2017), pp 325–330. IEEE

  2. Alashkar T, Jiang S, Wang S, Fu Y (2017) Examples-rules guided deep neural network for makeup recommendation. In: Proceedings of the AAAI Conference on artificial intelligence, vol 31

  3. Blau Y, Mechrez R, Timofte R, Michaeli T, Zelnik-Manor L (2018) The 2018 pirm challenge on perceptual image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp 0–0

  4. Center XCR (2014) Chinese image law tutorial: Women’s Personal dress style sub-book ChinaTextileApparelPress, Beijing

  5. Chan KC, Wang X, Xu X, Gu J, Loy CC (2021) Glean: Generative latent bank for large-factor image super-resolution. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 14245–14254

  6. Chang H, Lu J, Yu F, Finkelstein A (2018) Pairedcyclegan: Asymmetric style transfer for applying and removing makeup. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 40–48

  7. Chen H-J, Hui K-M, Wang S-Y, Tsao L-W, Shuai H-H, Cheng W-H (2019) Beautyglow: On-demand makeup transfer framework with reversible generative network. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 10042–10050

  8. Chen C, Li X, Yang L, Lin X, Zhang L, Wong K-YK (2021) Progressive semantic-aware style transformation for blind face restoration. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 11896–11905

  9. Chen Y, Shen C, Wei X-S, Liu L, Yang J (2017) Adversarial posenet: a structure-aware convolutional network for human pose estimation. In: Proceedings of the IEEE International conference on computer vision, pp 1212–1221

  10. Chen Y, Tai Y, Liu X, Shen C, Yang J (2018) Fsrnet: End-to-end learning face super-resolution with facial priors. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 2492–2501

  11. Deng H, Han C, Cai H, Han G, He S (2021) Spatially-invariant style-codes controlled makeup transfer. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 6549–6557

  12. Dong C, Deng Y, Loy CC, Tang X (2015) Compression artifacts reduction by a deep convolutional network. In: Proceedings of the IEEE International conference on computer vision, pp 576–584

  13. Gan J, Li L, Zhai Y, Liu Y (2014) Deep self-taught learning for facial beauty prediction. Neurocomputing 144:295–303

    Article  Google Scholar 

  14. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Advances in neural information processing systems 27

  15. Gowda SN (2019) Using colorization as a tool for automatic makeup suggestion. arXiv:1906.07421

  16. Gu Q, Wang G, Chiu MT, Tai Y-W, Tang C-K (2019) Ladn:Local adversarial disentangling network for facial makeup and de-makeup. In: Proceedings of the IEEE/CVF International conference on computer vision, pp 10481–10490

  17. Gulati K, Verma G, Mohania M, Kundu A (2021) Beautifai–a personalised occasion-oriented makeup recommendation system. arXiv:2109.06072

  18. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30

  19. Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 1125–1134

  20. Jiang W, Liu S, Gao C, Cao J, He R, Feng J, Yan S (2020) Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 5194–5202

  21. Jiang J, Wang C, Liu X, Ma J (2021) Deep learning-based face super-resolution:, A survey. arXiv:2101.03749

  22. Jiang J, Yu Y, Hu J, Tang S, Ma J (2018) Deep cnn denoiser and multi-layer neighbor component embedding for face hallucination. arXiv:1806.10726

  23. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp 694–711. Springer

  24. Kalarot R, Li T, Porikli F (2020) Component attention guided face super-resolution network: Cagface. In: Proceedings of the IEEE/CVF Winter conference on applications of computer vision, pp 370–380

  25. Kips R, Gori P, Perrot M, Bloch I (2020) Ca-gan: Weakly supervised color aware gan for controllable makeup transfer. In: European Conference on Computer Vision, pp 280–296. Springer

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  27. Kupyn O, Martyniuk T, Wu J, Wang Z (2019) Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In: Proceedings of the IEEE/CVF International conference on computer vision, pp 8878–8887

  28. Ledig C, Theis L, Huszár F., Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 4681–4690

  29. Li T, Qian R, Dong C, Liu S, Yan Q, Zhu W, Lin L (2018) Beautygan: Instance-level facial makeup transfer with deep generative adversarial network. In: Proceedings of the 26th ACM International conference on multimedia, pp 645–653

  30. Li Z, Yang J, Liu Z, Yang X, Jeon G, Wu W (2019) Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 3867–3876

  31. Liao J, Yao Y, Yuan L, Hua G, Kang SB (2017) Visual attribute transfer through deep image analogy. arXiv:1705.01088

  32. Liu S, Ou X, Qian R, Wang W, Cao X (2016) Makeup like a superstar:, Deep localized makeup transfer network. arXiv:1604.07102

  33. Liu L, Xing J, Liu S, Xu H, Zhou X, Yan S (2014) Wow! you are so beautiful today!. ACM Transactions on Multimedia Computing Communications, and Applications (TOMM) 11(1s):1–22

    Article  Google Scholar 

  34. Lyu Y, Dong J, Peng B, Wang W, Tan T (2021) Sogan: 3d-aware shadow and occlusion robust gan for makeup transfer. In: Proceedings of the 29th ACM International conference on multimedia, pp 3601–3609

  35. Majdabadi MM, Ko S-B (2020) Capsule gan for robust face super resolution. Multimed Tools Appl 79(41):31205–31218

    Article  Google Scholar 

  36. Mittal A, Soundararajan R, Bovik AC (2012) Making a “completely blind” image quality analyzer. IEEE Signal Process Lett 20(3):209–212

    Article  Google Scholar 

  37. Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. In: European conference on computer vision, pp 483–499. Springer

  38. Nguyen TV, Liu L (2017) Smart mirror: Intelligent makeup recommendation and synthesis. In: Proceedings of the 25th ACM International conference on multimedia, pp 1253–1254

  39. Nguyen T, Tran AT, Hoai M (2021) Lipstick ain’t enough: beyond color matching for in-the-wild makeup transfer. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 13305–13314

  40. Organisciak D, Ho ES, Shum HP (2021) Makeup style transfer on low-quality images with weighted multi-scale attention. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp 6011–6018. IEEE

  41. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: British machine vision conference

  42. Ren Y, Sun Y, Wu D, Cui Z, Qin AK (2019) A new makeup transfer with super-resolution. Aust J Intell Inf Process Syst 16(2):59–68

    Google Scholar 

  43. Scherbaum K, Ritschel T, Hullin M, Thormählen T, Blanz V, Seidel H-P (2011) Computer-suggested facial makeup. In: Computer graphics forum, vol 30, pp 485–492. Wiley Online Library

  44. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  45. Singh A, Singh J (2020) Survey on single image based super-resolution—implementation challenges and solutions. Multimed Tools Appl 79(3):1641–1672

    Article  Google Scholar 

  46. Song Y, Zhang J, He S, Bao L, Yang Q (2017) Learning to hallucinate face images via component generation and enhancement. arXiv:1708.00223

  47. Sun Z, Chen Y, Xiong S (2022) Ssat: a symmetric semantic-aware transformer network for makeup transfer and removal. In: Proceedings of the AAAI Conference on artificial intelligence, vol 36, pp 2325–2334

  48. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 2818–2826

  49. Wan Z, Chen H, An J, Jiang W, Yao C, Luo J (2022) Facial attribute transformers for precise and robust makeup transfer. In: Proceedings of the IEEE/CVF Winter conference on applications of computer vision, pp 1717–1726

  50. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84

    Article  Google Scholar 

  51. Wang X, Li Y, Zhang H, Shan Y (2021) Towards real-world blind face restoration with generative facial prior. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 9168–9178

  52. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The 37th Asilomar Conference on signals, systems & computers, 2003, vol 2, pp 1398–1402. IEEE

  53. Wang C, Zhong Z, Jiang J, Zhai D, Liu X (2020) Parsing map guided multi-scale attention network for face hallucination. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 2518–2522. IEEE

  54. Yang C, He W, Xu Y, Gao Y (2022) Elegant:, Exquisite and locally editable gan for makeup transfer. arXiv:2207.09840

  55. Yang T, Ren P, Xie X, Zhang L (2021) Gan prior embedded network for blind face restoration in the wild. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 672–681

  56. Yang L, Wang S, Ma S, Gao W, Liu C, Wang P, Ren P (2020) Hifacegan: Face renovation via collaborative suppression and replenishment. In: Proceedings of the 28th ACM International conference on multimedia, pp 1551–1560

  57. Yu X, Fernando B, Ghanem B, Porikli F, Hartley R (2018) Face super-resolution guided by facial component heatmaps. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 217–233

  58. Yu C, Wang J, Peng C, Gao C, Yu G, Sang N (2018) Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 325–341

  59. Yu X, Zhang L, Xie W (2021) Semantic-driven face hallucination based on residual network. IEEE Trans Biom Behav Identity Sci 3(2):214–228

    Article  Google Scholar 

  60. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 586–595

  61. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International conference on computer vision, pp 2223–2232

Download references

Funding

This work was partially supported by the Guangdong Basic and Applied Basic Research Fund Regional Joint Fund Project (2020B1515120089), the Guangdong Colleges and Universities Special Project Foundation in Key Areas of Artificial Intelligence (2019KZDZX1033), and the Science and Technology Innovation Project of Foshan City, Guangdong (2016AG100472).

Author information

Authors and Affiliations

  1. School of Software, South China Normal University, Taoyuan West Road, Foshan, 528000, Guangdong, China

    Kai Lin, Yiming Ru, Jingzeng Huang, Songsen Yu & Yihua Chen

Authors
  1. Kai Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiming Ru
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingzeng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Songsen Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yihua Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yihua Chen.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflicts of interest related to this study and the commercial capabilities provided.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, K., Ru, Y., Huang, J. et al. SRPSGAN: Super-resolution with pose and expression robust spatial-aware generative adversarial network for makeup transfer. Multimed Tools Appl 83, 10147–10165 (2024). https://doi.org/10.1007/s11042-023-15440-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15440-8

Keywords

Search

Navigation