Advertisement

CooGAN: A Memory-Efficient Framework for High-Resolution Facial Attribute Editing

Conference paper
  • 562 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12356)

Abstract

In contrast to great success of memory-consuming face editing methods at a low resolution, to manipulate high-resolution (HR) facial images, i.e., typically larger than \(768^2\) pixels, with very limited memory is still challenging. This is due to the reasons of 1) intractable huge demand of memory; 2) inefficient multi-scale features fusion. To address these issues, we propose a NOVEL pixel translation framework called Cooperative GAN(CooGAN) for HR facial image editing. This framework features a local path for fine-grained local facial patch generation (i.e., patch-level HR, LOW memory) and a global path for global low-resolution (LR) facial structure monitoring (i.e., image-level LR, LOW memory), which largely reduce memory requirements. Both paths work in a cooperative manner under a local-to-global consistency objective (i.e., for smooth stitching). In addition, we propose a lighter selective transfer unit for more efficient multi-scale features fusion, yielding higher fidelity facial attributes manipulation. Extensive experiments on CelebA-HQ well demonstrate the memory efficiency as well as the high image generation quality of the proposed framework.

Keywords

Generative adversarial networks Conditional GANs Face attributes editing 

Notes

Acknowledgements

This work was supported by National Science Foundation of China (61976137, U1611461, U19B2035) and STCSM(18DZ1112300).

Supplementary material

504452_1_En_39_MOESM1_ESM.pdf (4.1 mb)
Supplementary material 1 (pdf 4193 KB)

References

  1. 1.
  2. 2.
    IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018. IEEE Computer Society (2018)Google Scholar
  3. 3.
    IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019. Computer Vision Foundation/IEEE (2018)Google Scholar
  4. 4.
    Awiszus, M., Ackermann, H., Rosenhahn, B.: Learning disentangled representations via independent subspaces. CoRR abs/1908.08989 (2019)Google Scholar
  5. 5.
    Cao, J., Huang, H., Li, Y., Liu, J., He, R., Sun, Z.: Biphasic learning of GANs for high-resolution image-to-image translation. CoRR abs/1904.06624 (2019)Google Scholar
  6. 6.
    Chen, Y., et al.: Facelet-bank for fast portrait manipulation. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018 [2], pp. 3541–3549 (2018).  https://doi.org/10.1109/CVPR.2018.00373
  7. 7.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A Meeting of SIGDAT, A Special Interest Group of the ACL, pp. 1724–1734. ACL (2014). https://www.aclweb.org/anthology/volumes/D14-1/
  8. 8.
    Choi, Y., Choi, M., Kim, M., Ha, J., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018 [2], pp. 8789–8797 (2018).  https://doi.org/10.1109/CVPR.2018.00916
  9. 9.
    Chung, J., Gülçehre, Ç., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR abs/1412.3555 (2014)Google Scholar
  10. 10.
    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. CoRR abs/1704.00028 (2017)Google Scholar
  11. 11.
    Guyon, I., et al. (eds.): Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017 (2017)Google Scholar
  12. 12.
    He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: AttGAN: facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28(11), 5464–5478 (2019).  https://doi.org/10.1109/TIP.2019.2916751MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017)Google Scholar
  14. 14.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April–3 May 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/group?id=ICLR.cc/2018/Conference
  15. 15.
    Kemelmacher-Shlizerman, I., Suwajanakorn, S., Seitz, S.M.: Illumination-aware age progression. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, 23–28 June 2014, pp. 3334–3341. IEEE Computer Society (2014).  https://doi.org/10.1109/CVPR.2014.426. https://ieeexplore.ieee.org/xpl/conhome/6909096/proceeding
  16. 16.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html
  17. 17.
    Lample, G., Zeghidour, N., Usunier, N., Bordes, A., Denoyer, L., Ranzato, M.: Fader networks: manipulating images by sliding attributes. In: Guyon et al. [11], pp. 5967–5976Google Scholar
  18. 18.
    Lee, C., Liu, Z., Wu, L., Luo, P.: MaskGAN: towards diverse and interactive facial image manipulation. CoRR abs/1907.11922 (2019)Google Scholar
  19. 19.
    Lei, T., Zhang, Y., Wang, S.I., Dai, H., Artzi, Y.: Simple recurrent units for highly parallelizable recurrence. In: Riloff, E., Chiang, D., Hockenmaier, J., Tsujii, J. (eds.) Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October–4 November 2018, pp. 4470–4481. Association for Computational Linguistics (2018).  https://doi.org/10.18653/v1/d18-1477
  20. 20.
    Liu, M., et al.: STGAN: a unified selective transfer network for arbitrary image attribute editing. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019 [3], pp. 3673–3682Google Scholar
  21. 21.
    Liu, M., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Guyon et al. [11], pp. 700–708Google Scholar
  22. 22.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV), December 2015Google Scholar
  23. 23.
    Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 122–138. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01264-9_8CrossRefGoogle Scholar
  24. 24.
    Odena, A., Olah, C., Shlens, J.: Conditional image synthesis with auxiliary classifier GANs. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017. Proceedings of Machine Learning Research, vol. 70, pp. 2642–2651. PMLR (2017). http://proceedings.mlr.press/v70/
  25. 25.
    Perarnau, G., van de Weijer, J., Raducanu, B., Álvarez, J.M.: Invertible conditional GANs for image editing. CoRR abs/1611.06355 (2016)Google Scholar
  26. 26.
    Qian, S., et al.: Make a face: towards arbitrary high fidelity face manipulation. CoRR abs/1908.07191 (2019)Google Scholar
  27. 27.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  28. 28.
    Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation. CoRR abs/1801.04381 (2018)Google Scholar
  29. 29.
    Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stamminger, M., Theobalt, C.: Real-time expression transfer for facial reenactment. ACM Trans. Graph. 34(6), 183:1–183:14 (2015).  https://doi.org/10.1145/2816795.2818056
  30. 30.
    Xu, S., Huang, H., Hu, S., Liu, W.: FaceShapeGene: a disentangled shape representation for flexible face image editing. CoRR abs/1905.01920 (2019)Google Scholar
  31. 31.
    Xu, T., et al.: AttnGAN: fine-grained text to image generation with attentional generative adversarial networks. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018 [2], pp. 1316–1324.  https://doi.org/10.1109/CVPR.2018.00143
  32. 32.
    Yi, R., Liu, Y., Lai, Y., Rosin, P.L.: APDrawingGAN: generating artistic portrait drawings from face photos with hierarchical GANs. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 10743–10752. Computer Vision Foundation/IEEE (2019).  https://doi.org/10.1109/CVPR.2019.01100
  33. 33.
    Yin, W., Liu, Z., Loy, C.C.: Instance-level facial attributes transfer with geometry-aware flow. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, 27 January–1 February 2019, pp. 9111–9118. AAAI Press (2019).  https://doi.org/10.1609/aaai.v33i01.33019111. https://www.aaai.org/Library/AAAI/aaai19contents.php
  34. 34.
    Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018 [2], pp. 6848–6856Google Scholar
  35. 35.
    Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., Tian, Q.: Variational convolutional neural network pruning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019 [3], pp. 2780–2789Google Scholar
  36. 36.
    Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 2242–2251. IEEE Computer Society (2017).  https://doi.org/10.1109/ICCV.2017.244. https://ieeexplore.ieee.org/xpl/conhome/8234942/proceeding

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Shanghai Jiao Tong UniversityShanghaiChina
  2. 2.Huawei HiSiliconShenzhenChina
  3. 3.HuaweiChina

Personalised recommendations