Photo-Realistic Facial Emotion Synthesis Using Multi-level Critic Networks with Multi-level Generative Model

  • Minho Park
  • Hak Gu Kim
  • Yong Man Ro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11296)


In this paper, we propose photo-realistic facial emotion synthesis by using a novel multi-level critic network with multi-level generative model. We devise a new facial emotion generator containing the proposed multi-level decoder to synthesize facial image with a desired variation. A proposed multi-level decoder and multi-level critic network help the generator to produce a photo-realistic and variation-realistic facial image in generative adversarial learning. The multi-level critic network consists of two discriminators, photo-realistic discriminator and variation-realistic discriminator. The photo-realistic discriminator in the multi-level critic network determines whether the multi-resolution facial image generated from the latent feature of the multi-level decoding module is photo-realistic or not. The variation-realistic discriminator determines whether the multi-resolution facial image has natural variation or not. Experimental results show that the proposed facial emotion synthesis method outperforms existing methods in terms of both qualitative performance and quantitative performance of expression recognition.


Facial variation image synthesis Adversarial learning Feature refinement 


  1. 1.
    Xu, Y., Li, X., Yang, J., Zhang, D.: Integrate the original face image and its mirror image for face recognition. Neurocomputing 131, 191–199 (2014)CrossRefGoogle Scholar
  2. 2.
    Kim, Y., Yoo, B., Kwak, Y., Choi, C., Kim, J.: Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv:1703.07140 (2017)
  3. 3.
    Kim, D.H., Baddar, W., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. In: IEEE Transactions on Affective Computing (2017)Google Scholar
  4. 4.
    Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. arXiv preprint arXiv:1702.08423 (2017)
  5. 5.
    Gu, G.M., Kim, S.T., Kim, K.H., Baddar, W., Ro, Y.M.: Differential generative adversarial networks: synthesizing non-linear facial variations with limited number of training data. arXiv preprint arXiv:1711.10267 (2017)
  6. 6.
    Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: unified generative adversarial networks for multi-domain image-to-image translation. arXiv preprint arXiv:1711.09020 (2017)
  7. 7.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  9. 9.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, AlF (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  10. 10.
    Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), E1454–E1462 (2014)CrossRefGoogle Scholar
  11. 11.
    Valstar, M., Pantic, M.: Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proceedings of 3rd International Workshop on EMOTION (Satellite of LREC): Corpora for Research on Emotion and Affect, p. 65 (2010)Google Scholar
  12. 12.
    Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H., Hawk, S.T., Van Knippenberg, A.: Presentation and validation of the radboud faces database. Cogn. Emot. 24(8), 1377–1388 (2010)CrossRefGoogle Scholar
  13. 13.
    Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004 (2016)
  14. 14.
    Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  15. 15.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Image and Video Systems Laboratory, School of Electrical EngineeringKAISTDaejeonSouth Korea

Personalised recommendations