Skip to main content

Advertisement

SpringerLink
Log in

Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We address the problem of restoring a high-resolution face image from a blurry low-resolution input. This problem is difficult as super-resolution and deblurring need to be tackled simultaneously. Moreover, existing algorithms cannot handle face images well as low-resolution face images do not have much texture which is especially critical for deblurring. In this paper, we propose an effective algorithm by utilizing the domain-specific knowledge of human faces to recover high-quality faces. We first propose a facial component guided deep Convolutional Neural Network (CNN) to restore a coarse face image, which is denoted as the base image where the facial component is automatically generated from the input face image. However, the CNN based method cannot handle image details well. We further develop a novel exemplar-based detail enhancement algorithm via facial component matching. Extensive experiments show that the proposed method outperforms the state-of-the-art algorithms both quantitatively and qualitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Agustsson, E., Timofte, R., & Van Gool, L. (2017). Anchored regression networks applied to age estimation and super resolution. In IEEE international conference on computer vision.

  • Cao, Q., Lin, L., Shi, Y., Liang, X., & Li, G. (2017). Attention-aware face hallucination via deep reinforcement learning. In IEEE conference on computer vision and pattern recognition.

  • Chen, Y., Tai, Y., Liu, X., Shen, C., & Yang, J. (2017). Fsrnet: End-to-end learning face super-resolution with facial priors. arXiv preprint arXiv:1711.10703

  • Delac, K., Grgic, M., & Grgic, S. (2005). Independent comparative study of PCA, ICA, and LDA on the FERET data set. International Journal of Imaging Systems and Technology, 15(5), 252–260.

    Article  Google Scholar 

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In European conference on computer vision.

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2016a). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.

  • Dong, C., Loy, C. C., & Tang, X. (2016b). Accelerating the super-resolution convolutional neural network. In European conference on computer vision.

  • Eisemann, E., & Durand, F. (2004). Flash photography enhancement via intrinsic relighting. In ACM transactions on graphics (SIGGRAPH).

  • Gross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2010). Multi-pie. Image and Vision Computing, 28(5), 807–813.

    Article  Google Scholar 

  • Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., & Zhang, L. (2015). Convolutional sparse coding for image super-resolution. In IEEE international conference on computer vision.

  • Gunturk, B. K., Batur, A. U., Altunbasak, Y., Hayes, M. H., & Mersereau, R. M. (2003). Eigenface-domain super-resolution for face recognition. IEEE Transactions on Image Processing, 12(5), 597–606.

    Article  Google Scholar 

  • Hacohen, Y., Shechtman, E., & Lischinski, D. (2013). Deblurring by example using dense correspondence. In IEEE international conference on computer vision.

  • He, K., Sun, J., & Tang, X. (2010). Guided image filtering. In European conference on computer vision.

  • He, K., Sun, J., & Tang, X. (2013). Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 1397–1409.

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition.

  • Jia, K., & Gong, S. (2005). Multi-modal tensor face for simultaneous super-resolution and recognition. In IEEE international conference on computer vision.

  • Jia, K., & Gong, S. (2006). Multi-resolution patch tensor for facial expression hallucination. In IEEE conference on computer vision and pattern recognition.

  • Jia, K., & Gong, S. (2008). Generalized face super-resolution. IEEE Transactions on Image Processing, 17(6), 873–886.

    Article  MathSciNet  Google Scholar 

  • Jin, Y., & Bouganis, C. S. (2015). Robust multi-image based blind face hallucination. In IEEE conference on computer vision and pattern recognition.

  • Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision.

  • Jourabloo, A., Ye, M., Liu, X., & Ren, L. (2017). Pose-invariant face alignment with a single CNN. In IEEE international conference on computer vision.

  • Karras, T., Aila, T., Laine, S., & Jaakko, L. (2018). Progressive growing of GANs for improved quality, stability, and variation. In International conference on learning representation.

  • Kim, J., Kwon Lee, J., & Mu Lee, K. (2016a). Accurate image super-resolution using very deep convolutional networks. In IEEE conference on computer vision and pattern recognition.

  • Kim, J., Kwon Lee, J., & Mu Lee, K. (2016b). Deeply-recursive convolutional network for image super-resolution. In IEEE conference on computer vision and pattern recognition.

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. In International conference on learning representation.

  • Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and similar classifiers for face verification. In IEEE international conference on computer vision.

  • Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2017). Deep Laplacian pyramid networks for fast and accurate super-resolution. In IEEE conference on computer vision and pattern recognition.

  • Lai, W. S., Huang, J. B., Hu, Z., Ahuja, N., & Yang, M. H. (2016). A comparative study for single image blind deblurring. In IEEE conference on computer vision and pattern recognition.

  • Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In IEEE conference on computer vision and pattern recognition.

  • Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). Enhanced deep residual networks for single image super-resolution. In The IEEE conference on computer vision and pattern recognition workshops.

  • Liu, C., Shum, H. Y., & Freeman, W. T. (2007). Face hallucination: Theory and practice. International Journal of Computer Vision, 75(1), 115–134.

    Article  Google Scholar 

  • Liu, C., Yuen, J., & Torralba, A. (2011). Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), 978–994.

    Article  Google Scholar 

  • Liu, S., & Yang, M. H. (2014). Compressed face hallucination. In IEEE international conference on image processing.

  • Liu, W., Lin, D., & Tang, X. (2005). Hallucinating faces: Tensorpatch super-resolution and coupled residue compensation. In IEEE conference on computer vision and pattern recognition.

  • Ma, X., Zhang, J., & Qi, C. (2010). Hallucinating face by position-patch. Pattern Recognition, 43(6), 2224–2236.

    Article  Google Scholar 

  • Pan, J., Dong, J., Tai, Y. W., Su, Z., & Yang, M. H. (2017). Learning discriminative data fitting functions for blind image deblurring. In IEEE international conference on computer vision.

  • Pan, J., Hu, Z., Su, Z., & Yang, M. H. (2014). Deblurring face images with exemplars. In European conference on computer vision.

  • Pan, J., Sun, D., Pfister, H., & Yang, M. H. (2017). Deblurring images via dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2017.2753804.

  • Park, H., & Mu Lee, K. (2017). Joint estimation of camera pose, depth, deblurring, and super-resolution from a blurred image sequence. In IEEE international conference on computer vision.

  • Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004). Digital photography with flash and no-flash image pairs. In ACM transactions on graphics (SIGGRAPH).

  • Sajjadi, M. S., Scholkopf, B., & Hirsch, M. (2017). Enhancenet: Single image super-resolution through automated texture synthesis. In IEEE international conference on computer vision.

  • Shen, Z., Lai, W. S., Xu, T., Kautz, J., & Yang, M. H. (2018). Deep semantic face deblurring. In The IEEE conference on computer vision and pattern recognition.

  • Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., et al. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE conference on computer vision and pattern recognition.

  • Song, Y., Bao, L., He, S., Yang, Q., & Yang, M. H. (2017a). Stylizing face images via multiple exemplars. Computer Vision and Image Understanding, 162, 135–145.

    Article  Google Scholar 

  • Song, Y., Bao, L., Yang, Q., & Yang, M. H. (2014). Real-time exemplar-based face sketch synthesis. In European conference on computer vision.

  • Song, Y., Zhang, J., Bao, L., & Yang, Q. (2017b). Fast preprocessing for robust face sketch synthesis. In International joint conference on artificial intelligence.

  • Song, Y., Zhang, J., He, S., Bao, L., & Yang, Q. (2017c). Learning to hallucinate face images via component generation and enhancement. In International joint conference on artificial intelligence.

  • Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In IEEE conference on computer vision and pattern recognition.

  • Tappen, M. F., & Liu, C. (2012). A bayesian approach to alignment-based image hallucination. In European conference on computer vision.

  • Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., et al. (2017). Ntire 2017 challenge on single image super-resolution: Methods and results. In IEEE conference on computer vision and pattern recognition workshops.

  • Wang, N., Tao, D., Gao, X., Li, X., & Li, J. (2014). A comprehensive survey to face hallucination. International Journal of Computer Vision, 106(1), 9–30.

    Article  Google Scholar 

  • Wang, X., & Tang, X. (2005). Hallucinating face by eigentransformation. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 35(3), 425–434.

    Article  Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wang, Z., Liu, D., Yang, J., Han, W., & Huang, T. (2015). Deep networks for image super-resolution with sparse prior. In IEEE international conference on computer vision.

  • Xu, X., Sun, D., Pan, J., Zhang, Y., Pfister, H., & Yang, M. H. (2017). Learning to super-resolve blurry face and text images. In IEEE international conference on computer vision.

  • Yang, C. Y., Liu, S., & Yang, M. H. (2013). Structured face hallucination. In IEEE conference on computer vision and pattern recognition.

  • Yang, C. Y., Liu, S., & Yang, M. H. (2018a). Hallucinating compressed face images. International Journal of Computer Vision, 126(6), 597–614.

    Article  MathSciNet  Google Scholar 

  • Yang, X., Xu, K., Song, Y., Zhang, Q., Wei, X., & Lau, R. W. (2018b). Image correction via deep reciprocating HDR transformation. In IEEE conference on computer vision and pattern recognition.

  • Yu, F., Koltun, V., & Funkhouser, T. (2017). Dilated residual networks. In IEEE conference on computer vision and pattern recognition.

  • Yu, X., & Porikli, F. (2016). Ultra-resolving face images by discriminative generative networks. In European conference on computer vision.

  • Yu, X., & Porikli, F. (2017a). Face hallucination with tiny unaligned images by transformative discriminative neural networks. In AAAI conference on artificial intelligence.

  • Yu, X., & Porikli, F. (2017b). Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In IEEE conference on computer vision and pattern recognition.

  • Zhang, J., Pan, J., Ren, J., Song, Y., Bao, L., Lau, R. W., & Yang, M. H. (2018). Dynamic scene deblurring using spatially variant recurrent neural networks. In IEEE conference on computer vision and pattern recognition.

  • Zhou, E., Fan, H., Cao, Z., Jiang, Y., & Yin, Q. (2015). Learning face hallucination in the wild. In AAAI conference on artificial intelligence.

  • Zhu, S., Liu, S., Loy, C. C., & Tang, X. (2016). Deep cascaded bi-network for face hallucination. In European conference on computer vision.

  • Zhu, X., & Ramanan, D. (2012). Face detection, pose estimation, and landmark localization in the wild. In IEEE conference on computer vision and pattern recognition.

Download references

Acknowledgements

This work has been supported in part by the NSF CAREER (No. 1149783), NSF of China (No. 61872421 and 61572099), NSF of Jiangsu Province (No. BK20180471), and National Science and Technology Major Project (2018ZX04041001-007).

Author information

Authors and Affiliations

  1. Tencent AI Lab, Shenzhen, China

    Yibing Song & Linchao Bao

  2. Sensetime Research, Shenzhen, China

    Jiawei Zhang

  3. Tencent, Shenzhen, China

    Lijun Gong

  4. South China University of Technology, Guangzhou, China

    Shengfeng He

  5. Nanjing University of Science and Technology, Nanjing, China

    Jinshan Pan

  6. University of Science and Technology of China, Hefei, China

    Qingxiong Yang

  7. University of California at Merced, Merced, USA

    Ming-Hsuan Yang

Authors
  1. Yibing Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiawei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lijun Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shengfeng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Linchao Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jinshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Qingxiong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ming-Hsuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinshan Pan.

Additional information

Communicated by Chellappa, Liu, Kim, Torre and Loy.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, Y., Zhang, J., Gong, L. et al. Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement. Int J Comput Vis 127, 785–800 (2019). https://doi.org/10.1007/s11263-019-01148-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-019-01148-6

Keywords

Search

Navigation