Advertisement

Ultra-Resolving Face Images by Discriminative Generative Networks

  • Xin YuEmail author
  • Fatih Porikli
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9909)

Abstract

Conventional face super-resolution methods, also known as face hallucination, are limited up to \(2 \! \sim \! 4\times \) scaling factors where \(4 \sim 16\) additional pixels are estimated for each given pixel. Besides, they become very fragile when the input low-resolution image size is too small that only little information is available in the input image. To address these shortcomings, we present a discriminative generative network that can ultra-resolve a very low resolution face image of size \(16 \times 16\) pixels to its \(8\times \) larger version by reconstructing 64 pixels from a single pixel. We introduce a pixel-wise \(\ell _2\) regularization term to the generative model and exploit the feedback of the discriminative network to make the upsampled face images more similar to real ones. In our framework, the discriminative network learns the essential constituent parts of the faces and the generative network blends these parts in the most accurate fashion to the input image. Since only frontal and ordinary aligned images are used in training, our method can ultra-resolve a wide range of very low-resolution images directly regardless of pose and facial expression variations. Our extensive experimental evaluations demonstrate that the presented ultra-resolution by discriminative generative networks (UR-DGN) achieves more appealing results than the state-of-the-art.

Keywords

Super-resolution Discriminative Generative Networks Face 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Baker, S., Kanade, T.: Hallucinating faces. In: Proceedings - 4th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2000, pp. 83–88 (2000)Google Scholar
  2. 2.
    Liu, C., Shum, H., Zhang, C.: A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 1, 192–198 (2001)Google Scholar
  3. 3.
    Baker, S., Kanade, T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1167–1183 (2002)CrossRefGoogle Scholar
  4. 4.
    Wang, X., Tang, X.: Hallucinating face by eigen transformation. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 35(3), 425–434 (2005)CrossRefGoogle Scholar
  5. 5.
    Liu, C., Shum, H.Y., Freeman, W.T.: Face hallucination: theory and practice. Int. J. Comput. Vis. 75(1), 115–134 (2007)CrossRefGoogle Scholar
  6. 6.
    Jia, K., Gong, S.: Generalized face super-resolution. IEEE Trans. Image Process. 17(6), 873–886 (2008)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Ma, X., Zhang, J., Qi, C.: Hallucinating face by position-patch. Pattern Recogn. 43(6), 2224–2236 (2010)CrossRefGoogle Scholar
  9. 9.
    Tappen, M.F., Liu, C.: A Bayesian approach to alignment-based image hallucination. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 236–249. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  10. 10.
    Yang, C.Y., Liu, S., Yang, M.H.: Structured face hallucination. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1099–1106 (2013)Google Scholar
  11. 11.
    Zhou, E., Fan, H.: Learning face hallucination in the wild. In: Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 3871–3877 (2015)Google Scholar
  12. 12.
    Wang, N., Tao, D., Gao, X., Li, X., Li, J.: A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106(1), 9–30 (2014)CrossRefGoogle Scholar
  13. 13.
    Yang, C.-Y., Ma, C., Yang, M.-H.: Single-image super-resolution: a benchmark. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 372–386. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10593-2_25 Google Scholar
  14. 14.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07–49, University of Massachusetts, Amherst, October 2007Google Scholar
  15. 15.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV), December 2015Google Scholar
  16. 16.
    Dong, C., Loy, C.C., He, K.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  17. 17.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. arXiv:1511.04587 (2015)
  18. 18.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M.: Generative adversarial networks. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  19. 19.
    Denton, E., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Advances In Neural Information Processing Systems, pp. 1486–1494 (2015)Google Scholar
  20. 20.
    Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans. Image Process. 23(6), 2569–2582 (2014)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Yang, C.Y., Yang, M.H.: Fast direct super-resolution by simple functions. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 561–568 (2013)Google Scholar
  22. 22.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  23. 23.
    Chang, H., Yeung, D-Y., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR, vol. 1, pp. 275–282 (2004)Google Scholar
  24. 24.
    Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: ICCV, pp. 349–356 (2009)Google Scholar
  25. 25.
    Schulter, S., Leistner, C.: Fast and accurate image upscaling with super-resolution forests. In: CVPR, pp. 3791–3799 (2015)Google Scholar
  26. 26.
    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)Google Scholar
  27. 27.
    Lin, Z., Shum, H.Y.: Response to the comments on fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE Trans. Pattern Anal. Mach. Intell. 28(5), 847 (2006)CrossRefGoogle Scholar
  28. 28.
    Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM Trans. Graph. 28(3), 1–10 (2010)Google Scholar
  29. 29.
    Singh, A., Porikli, F., Ahuja, N.: Super-resolving noisy images. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2846–2853 (2014)Google Scholar
  30. 30.
    Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., Zhang, L.: Convolutional sparse coding for image super-resolution. In: ICCV (2015)Google Scholar
  31. 31.
    Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. In: ICLR (2016)Google Scholar
  32. 32.
    Li, Y., Cai, C., Qiu, G., Lam, K.M.: Face hallucination based on sparse local-pixel structure. Pattern Recogn. 47(3), 1261–1270 (2014)CrossRefGoogle Scholar
  33. 33.
    Kolouri, S., Rohde, G.K.: Transport-based single frame super resolution of very low resolution face images. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  34. 34.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv:1312.6114 (Ml), pp. 1–14 (2013)
  35. 35.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks, pp. 1–15 (2015). arXiv:1511.06434
  36. 36.
    Zeiler, M.D., Taylor, G.W., Fergus, R.: Adaptive deconvolutional networks for mid and high level feature learning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2018–2025 (2011)Google Scholar
  37. 37.
    Hinton, G.: Neural Networks for Machine Learning Lecture 6a: Overview of mini-batch gradient descent Reminder: The error surface for a linear neuronGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Australian National UniversityCanberraAustralia

Personalised recommendations

Cite paper

Buy options