Advertisement

Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement

  • Yibing Song
  • Jiawei Zhang
  • Lijun Gong
  • Shengfeng He
  • Linchao Bao
  • Jinshan PanEmail author
  • Qingxiong Yang
  • Ming-Hsuan Yang
Article
  • 143 Downloads

Abstract

We address the problem of restoring a high-resolution face image from a blurry low-resolution input. This problem is difficult as super-resolution and deblurring need to be tackled simultaneously. Moreover, existing algorithms cannot handle face images well as low-resolution face images do not have much texture which is especially critical for deblurring. In this paper, we propose an effective algorithm by utilizing the domain-specific knowledge of human faces to recover high-quality faces. We first propose a facial component guided deep Convolutional Neural Network (CNN) to restore a coarse face image, which is denoted as the base image where the facial component is automatically generated from the input face image. However, the CNN based method cannot handle image details well. We further develop a novel exemplar-based detail enhancement algorithm via facial component matching. Extensive experiments show that the proposed method outperforms the state-of-the-art algorithms both quantitatively and qualitatively.

Keywords

Face hallucination Face deblurring Convolutional Neural Network 

Notes

Acknowledgements

This work has been supported in part by the NSF CAREER (No. 1149783), NSF of China (No. 61872421 and 61572099), NSF of Jiangsu Province (No. BK20180471), and National Science and Technology Major Project (2018ZX04041001-007).

References

  1. Agustsson, E., Timofte, R., & Van Gool, L. (2017). Anchored regression networks applied to age estimation and super resolution. In IEEE international conference on computer vision.Google Scholar
  2. Cao, Q., Lin, L., Shi, Y., Liang, X., & Li, G. (2017). Attention-aware face hallucination via deep reinforcement learning. In IEEE conference on computer vision and pattern recognition.Google Scholar
  3. Chen, Y., Tai, Y., Liu, X., Shen, C., & Yang, J. (2017). Fsrnet: End-to-end learning face super-resolution with facial priors. arXiv preprint arXiv:1711.10703
  4. Delac, K., Grgic, M., & Grgic, S. (2005). Independent comparative study of PCA, ICA, and LDA on the FERET data set. International Journal of Imaging Systems and Technology, 15(5), 252–260.CrossRefGoogle Scholar
  5. Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In European conference on computer vision.Google Scholar
  6. Dong, C., Loy, C. C., He, K., & Tang, X. (2016a). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.Google Scholar
  7. Dong, C., Loy, C. C., & Tang, X. (2016b). Accelerating the super-resolution convolutional neural network. In European conference on computer vision.Google Scholar
  8. Eisemann, E., & Durand, F. (2004). Flash photography enhancement via intrinsic relighting. In ACM transactions on graphics (SIGGRAPH).Google Scholar
  9. Gross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2010). Multi-pie. Image and Vision Computing, 28(5), 807–813.CrossRefGoogle Scholar
  10. Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., & Zhang, L. (2015). Convolutional sparse coding for image super-resolution. In IEEE international conference on computer vision.Google Scholar
  11. Gunturk, B. K., Batur, A. U., Altunbasak, Y., Hayes, M. H., & Mersereau, R. M. (2003). Eigenface-domain super-resolution for face recognition. IEEE Transactions on Image Processing, 12(5), 597–606.CrossRefGoogle Scholar
  12. Hacohen, Y., Shechtman, E., & Lischinski, D. (2013). Deblurring by example using dense correspondence. In IEEE international conference on computer vision.Google Scholar
  13. He, K., Sun, J., & Tang, X. (2010). Guided image filtering. In European conference on computer vision.Google Scholar
  14. He, K., Sun, J., & Tang, X. (2013). Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 1397–1409.CrossRefGoogle Scholar
  15. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition.Google Scholar
  16. Jia, K., & Gong, S. (2005). Multi-modal tensor face for simultaneous super-resolution and recognition. In IEEE international conference on computer vision.Google Scholar
  17. Jia, K., & Gong, S. (2006). Multi-resolution patch tensor for facial expression hallucination. In IEEE conference on computer vision and pattern recognition.Google Scholar
  18. Jia, K., & Gong, S. (2008). Generalized face super-resolution. IEEE Transactions on Image Processing, 17(6), 873–886.MathSciNetCrossRefGoogle Scholar
  19. Jin, Y., & Bouganis, C. S. (2015). Robust multi-image based blind face hallucination. In IEEE conference on computer vision and pattern recognition.Google Scholar
  20. Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision.Google Scholar
  21. Jourabloo, A., Ye, M., Liu, X., & Ren, L. (2017). Pose-invariant face alignment with a single CNN. In IEEE international conference on computer vision.Google Scholar
  22. Karras, T., Aila, T., Laine, S., & Jaakko, L. (2018). Progressive growing of GANs for improved quality, stability, and variation. In International conference on learning representation.Google Scholar
  23. Kim, J., Kwon Lee, J., & Mu Lee, K. (2016a). Accurate image super-resolution using very deep convolutional networks. In IEEE conference on computer vision and pattern recognition.Google Scholar
  24. Kim, J., Kwon Lee, J., & Mu Lee, K. (2016b). Deeply-recursive convolutional network for image super-resolution. In IEEE conference on computer vision and pattern recognition.Google Scholar
  25. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. In International conference on learning representation.Google Scholar
  26. Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and similar classifiers for face verification. In IEEE international conference on computer vision.Google Scholar
  27. Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2017). Deep Laplacian pyramid networks for fast and accurate super-resolution. In IEEE conference on computer vision and pattern recognition.Google Scholar
  28. Lai, W. S., Huang, J. B., Hu, Z., Ahuja, N., & Yang, M. H. (2016). A comparative study for single image blind deblurring. In IEEE conference on computer vision and pattern recognition.Google Scholar
  29. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  30. Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). Enhanced deep residual networks for single image super-resolution. In The IEEE conference on computer vision and pattern recognition workshops.Google Scholar
  31. Liu, C., Shum, H. Y., & Freeman, W. T. (2007). Face hallucination: Theory and practice. International Journal of Computer Vision, 75(1), 115–134.CrossRefGoogle Scholar
  32. Liu, C., Yuen, J., & Torralba, A. (2011). Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), 978–994.CrossRefGoogle Scholar
  33. Liu, S., & Yang, M. H. (2014). Compressed face hallucination. In IEEE international conference on image processing.Google Scholar
  34. Liu, W., Lin, D., & Tang, X. (2005). Hallucinating faces: Tensorpatch super-resolution and coupled residue compensation. In IEEE conference on computer vision and pattern recognition.Google Scholar
  35. Ma, X., Zhang, J., & Qi, C. (2010). Hallucinating face by position-patch. Pattern Recognition, 43(6), 2224–2236.CrossRefGoogle Scholar
  36. Pan, J., Dong, J., Tai, Y. W., Su, Z., & Yang, M. H. (2017). Learning discriminative data fitting functions for blind image deblurring. In IEEE international conference on computer vision.Google Scholar
  37. Pan, J., Hu, Z., Su, Z., & Yang, M. H. (2014). Deblurring face images with exemplars. In European conference on computer vision.Google Scholar
  38. Pan, J., Sun, D., Pfister, H., & Yang, M. H. (2017). Deblurring images via dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence.  https://doi.org/10.1109/TPAMI.2017.2753804.
  39. Park, H., & Mu Lee, K. (2017). Joint estimation of camera pose, depth, deblurring, and super-resolution from a blurred image sequence. In IEEE international conference on computer vision.Google Scholar
  40. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004). Digital photography with flash and no-flash image pairs. In ACM transactions on graphics (SIGGRAPH).Google Scholar
  41. Sajjadi, M. S., Scholkopf, B., & Hirsch, M. (2017). Enhancenet: Single image super-resolution through automated texture synthesis. In IEEE international conference on computer vision.Google Scholar
  42. Shen, Z., Lai, W. S., Xu, T., Kautz, J., & Yang, M. H. (2018). Deep semantic face deblurring. In The IEEE conference on computer vision and pattern recognition.Google Scholar
  43. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., et al. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  44. Song, Y., Bao, L., He, S., Yang, Q., & Yang, M. H. (2017a). Stylizing face images via multiple exemplars. Computer Vision and Image Understanding, 162, 135–145.CrossRefGoogle Scholar
  45. Song, Y., Bao, L., Yang, Q., & Yang, M. H. (2014). Real-time exemplar-based face sketch synthesis. In European conference on computer vision.Google Scholar
  46. Song, Y., Zhang, J., Bao, L., & Yang, Q. (2017b). Fast preprocessing for robust face sketch synthesis. In International joint conference on artificial intelligence.Google Scholar
  47. Song, Y., Zhang, J., He, S., Bao, L., & Yang, Q. (2017c). Learning to hallucinate face images via component generation and enhancement. In International joint conference on artificial intelligence.Google Scholar
  48. Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  49. Tappen, M. F., & Liu, C. (2012). A bayesian approach to alignment-based image hallucination. In European conference on computer vision.Google Scholar
  50. Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., et al. (2017). Ntire 2017 challenge on single image super-resolution: Methods and results. In IEEE conference on computer vision and pattern recognition workshops.Google Scholar
  51. Wang, N., Tao, D., Gao, X., Li, X., & Li, J. (2014). A comprehensive survey to face hallucination. International Journal of Computer Vision, 106(1), 9–30.CrossRefGoogle Scholar
  52. Wang, X., & Tang, X. (2005). Hallucinating face by eigentransformation. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 35(3), 425–434.CrossRefGoogle Scholar
  53. Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.CrossRefGoogle Scholar
  54. Wang, Z., Liu, D., Yang, J., Han, W., & Huang, T. (2015). Deep networks for image super-resolution with sparse prior. In IEEE international conference on computer vision.Google Scholar
  55. Xu, X., Sun, D., Pan, J., Zhang, Y., Pfister, H., & Yang, M. H. (2017). Learning to super-resolve blurry face and text images. In IEEE international conference on computer vision.Google Scholar
  56. Yang, C. Y., Liu, S., & Yang, M. H. (2013). Structured face hallucination. In IEEE conference on computer vision and pattern recognition.Google Scholar
  57. Yang, C. Y., Liu, S., & Yang, M. H. (2018a). Hallucinating compressed face images. International Journal of Computer Vision, 126(6), 597–614.MathSciNetCrossRefGoogle Scholar
  58. Yang, X., Xu, K., Song, Y., Zhang, Q., Wei, X., & Lau, R. W. (2018b). Image correction via deep reciprocating HDR transformation. In IEEE conference on computer vision and pattern recognition.Google Scholar
  59. Yu, F., Koltun, V., & Funkhouser, T. (2017). Dilated residual networks. In IEEE conference on computer vision and pattern recognition.Google Scholar
  60. Yu, X., & Porikli, F. (2016). Ultra-resolving face images by discriminative generative networks. In European conference on computer vision.Google Scholar
  61. Yu, X., & Porikli, F. (2017a). Face hallucination with tiny unaligned images by transformative discriminative neural networks. In AAAI conference on artificial intelligence.Google Scholar
  62. Yu, X., & Porikli, F. (2017b). Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In IEEE conference on computer vision and pattern recognition.Google Scholar
  63. Zhang, J., Pan, J., Ren, J., Song, Y., Bao, L., Lau, R. W., & Yang, M. H. (2018). Dynamic scene deblurring using spatially variant recurrent neural networks. In IEEE conference on computer vision and pattern recognition.Google Scholar
  64. Zhou, E., Fan, H., Cao, Z., Jiang, Y., & Yin, Q. (2015). Learning face hallucination in the wild. In AAAI conference on artificial intelligence.Google Scholar
  65. Zhu, S., Liu, S., Loy, C. C., & Tang, X. (2016). Deep cascaded bi-network for face hallucination. In European conference on computer vision.Google Scholar
  66. Zhu, X., & Ramanan, D. (2012). Face detection, pose estimation, and landmark localization in the wild. In IEEE conference on computer vision and pattern recognition.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Tencent AI LabShenzhenChina
  2. 2.Sensetime ResearchShenzhenChina
  3. 3.TencentShenzhenChina
  4. 4.South China University of TechnologyGuangzhouChina
  5. 5.Nanjing University of Science and TechnologyNanjingChina
  6. 6.University of Science and Technology of ChinaHefeiChina
  7. 7.University of California at MercedMercedUSA

Personalised recommendations