Skip to main content
SpringerLink
Log in

Mixed noise face hallucination via adaptive weighted residual and nuclear-norm regularization

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Face hallucination has been extensively studied in recent years. The majority of existing approaches operate admirably in noise-free or single camera/atmospheric noise (i.e., Gaussian or impulse) situations. However, when both Gaussian and impulse noises (mixed Gaussian-impulse noise (MIXGIN)) jointly contaminate an image, face hallucination becomes a challenging task. To deal with mixture image corruption, we offer an Adaptive Weighted Residual and Nuclear-Norm Regularization approach in this paper. The method reveals some of the most crucial components of the face hallucination problem. By assigning the weights adaptively to the pixels according to their coding residuals, the proposed method could self-identify the mixed noise and alleviate its effects on the coding process. Moreover, considering the similarity of the training samples and the sparsity of representation coefficients, a nuclear-norm regularization is employed to represent and hallucinate a high resolution face image using correlated and discriminative samples. The method is adaptive to produce suitable coefficients in the presence of mixed noise. To optimize the model, the alternating direction method of multipliers (ADMM) is introduced. The experimental results show that the proposed algorithm gives better performance than the existing face hallucination methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Chen L, Pan JS, Jiang JJ, Zhang JW, Han Z, Bao LC (2021) Multi-stage degradation homogenization for super-resolution of face images with extreme degradations. IEEE Trans Image Process 30:5600–5612

    Article  Google Scholar 

  2. Chen YT, Liu LW, Phonevilay V, Gu K, Xia R, Xie J, Zhang Q, Yang K (2021) Image super-resolution reconstruction based on feature map attention mechanism. Appl Intell 51:4367–4380. https://doi.org/10.1007/s10489-020-02116-1

    Article  Google Scholar 

  3. Shi JG, Zhao GY (2019) Face hallucination via coarse-to-fine recursive kernel regression structure. IEEE Trans Multimedia 21(9):2223–2236. https://doi.org/10.1109/TMM.2019.2898752

    Article  Google Scholar 

  4. Shi J, Liu X, Zong Y, Qi C, Zhao G (2018) Hallucinating face image by regularization models in high-resolution feature space. IEEE Trans Image Process 27(6):2980–2995. https://doi.org/10.1109/TIP.2018.2813163

    Article  MathSciNet  MATH  Google Scholar 

  5. Rajput SS, Bohat VK, Arya KV (2019) Grey wolf optimization algorithm for facial image super-resolution. Appl Intell 49:1324–1338. https://doi.org/10.1007/s10489-018-1340-x

    Article  Google Scholar 

  6. Baker S, Kanade T (2000) Hallucinating faces. In: Proceding of IEEE International Conference Autom. Face Gesture Recognition, pp. 83–88

  7. Belhumeur PN, Hespanha JP, Kriegman D (1997) Eigenfaces vs. Fisherfaces: recognition using class speci_c linear projection. IEEE Trans Pattern Anal Mach Intell 19:711–720

    Article  Google Scholar 

  8. Wang XG, Tang XO (2005) Hallucinating face by eigentransformation. IEEE Trans Syst Man Cybern C Appl Rev 35:425–434. https://doi.org/10.1109/TSMCC.2005.848171

    Article  Google Scholar 

  9. Chakrabarti A, Rajagopalan AN, Chellappa R (2007) Super-resolution of face images using kernel pca-based prior. IEEE Trans Multimedia 9(4):888–892. https://doi.org/10.1109/TMM.2007.893346

    Article  Google Scholar 

  10. He X, Yan S, Hu Y, Niyogi P, Zhang HJ (2005) Face recognition using Laplacian faces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340. https://doi.org/10.1109/TPAMI.2005.55

    Article  Google Scholar 

  11. Huang H, He H, Fan X, Zhang J (2010) Super-resolution of human face image using canonical correlation analysis. Pattern Recogn 43(7):2532–2543. https://doi.org/10.1016/j.patcog.2010.02.007

    Article  MATH  Google Scholar 

  12. Zhang X, Peng S, Jiang J (2008) An adaptive learning method for face hallucination using locality preserving projections, In: Proceding of IEEE International Conference Autom. Face Gesture Recognition, pp. 1–8

  13. An L, Bhanu B (2014) Face image super-resolution using 2D CCA. Signal Process 103:184–194. https://doi.org/10.1016/j.sigpro.2013.10.004

    Article  Google Scholar 

  14. Tang SZ, Xiao L, Liu PF, Wu HC (2014) Partial least-squares regression on common feature space for single image superresolution. Journal of Electronic Imaging 23:053006. https://doi.org/10.1117/1.JEI.23.5.053006

    Article  Google Scholar 

  15. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  16. Song YB, Zhang J, He S, Bao L, Yang Q (2017) Learning to hallucinate face images via component generation and enhancement. In: Proceedings of the 27th international joint conference on Arti_cial intelligence, pp 4537-4543

  17. Zhang YL, Li KP, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep Residual Channel attention networks. In: Proceeding of IEEE European Conference on Computer Vision, pp. 294–310

  18. Chen Y, Tai Y, Liu X, Shen CH, Yang J (2018) Fsrnet: end-to-end learning face superresolution with facial priors. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, 2492–2501

  19. Jiang K, Wang ZY, Yi P, Lu T, Jiang JJ, Xiong ZX (2020) Dual-path deep fusion network for face image hallucination. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2020.3027849

  20. Ledig C, Theis L, Huszr F, Caballero J et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 105–114

  21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778

  22. He K, Zhang X, Ren S, Sun J (2016) Sun identity mappings in deep residual networks. In: Proceeding of IEEE European Conference on Computer Vision, pp. 630–645

  23. Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 275–282

  24. Yang JC, Wright J, Huang T, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873. https://doi.org/10.1109/TIP.2010.2050625

    Article  MathSciNet  MATH  Google Scholar 

  25. Ma X, Zhang J, Qi C (2010) Hallucinating face by position-patch. Pattern Recogn 43(6):2224–2236. https://doi.org/10.1016/j.patcog.2009.12.019

    Article  Google Scholar 

  26. Jiang JJ, Hu RM, Wang ZY, Zhen H, Ma JY (2016) Facial image hallucination through coupled-layer neighbor embedding. IEEE Trans Circuits Syst Video Technol 26(9):1674–1684. https://doi.org/10.1109/TCSVT.2015.2433538

    Article  Google Scholar 

  27. Jung C, Jiao LC, Liu B, Gong M (2011) Position-patch based face hallucination using convex optimization. IEEE Signal Process Lett 18(6):367–370. https://doi.org/10.1109/LSP.2011.2140370

    Article  Google Scholar 

  28. Zeng K, Lu T, Liang X, Liu K, Chen H, Zhang Y (2019) Face superresolution via bilayer contextual representation. Signal Process Image Commun 75:147–157. https://doi.org/10.1016/j.image.2019.03.019

    Article  Google Scholar 

  29. Jiang JJ, Ma JY, Chen C, Jiang X, Wang Z (2016) Noise robust face image super-resolution through smooth sparse representation. IEEE Trans Cybern 47(11):3991–4002. https://doi.org/10.1109/TCYB.2016.2594184

    Article  Google Scholar 

  30. Jiang JJ, Chen C, Huang K, Cai Z, Hu RM (2016) Noise robust position patch based face super-resolution via Tikhonov regularized neighbor representation. Inf Sci 367:354–372. https://doi.org/10.1016/j.ins.2016.05.032

    Article  Google Scholar 

  31. Rajput SS, Singh A, Arya KV, Jiang JJ (2018) Noise robust face hallucination algorithm using local content prior based error shrunk nearest neighbors representation. Signal Process 147:233–246. https://doi.org/10.1016/j.sigpro.2018.01.030

    Article  Google Scholar 

  32. Jiang JJ, Wang ZY, Chen C, Lu T (2016) L1-L1 norms for face super-resolution with mixed Gaussian-impulse noise. IEEE international conference on acoustics, speech and signal processing, 2089-2093

  33. Nagar S, Jain A, Singh P, Kumar A (2020) Pixel-wise dictionary learning based locality-constrained representation for noise robust face hallucination. Digital Signal Processing 99:102667. https://doi.org/10.1016/j.dsp.2020.102667

    Article  Google Scholar 

  34. Liu LC, Li ST, Chen CLP (2019) Iterative relaxed collaborative representation with adaptive weights learning for noise robust face hallucination. IEEE Trans Circuits Syst Video Technol 29(5):1284–1295. https://doi.org/10.1109/TCSVT.2018.2829758

    Article  Google Scholar 

  35. Li N, Liu LC, Li ST, Lin H (2020) Robust face hallucination via locality-constrained multiscale coding. Inf Sci 512:369–380. https://doi.org/10.1016/j.ins.2019.06.041

    Article  Google Scholar 

  36. Liu LC, Han L, Li ST, Chen CL (2020) Face hallucination via multiple feature learning with hierarchical structure. Inf Sci 512:416–430

    Article  Google Scholar 

  37. Jiang JJ, Yu Y, Tang SH, Ma JY, Aizawa A, Aizawa K (2020) Context-patch face hallucination based on thresholding locality-constrained representation and reproducing learning. IEEE Trans Cybern 50(1):324–337. https://doi.org/10.1109/TCYB.2018.2868891

    Article  Google Scholar 

  38. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326. https://doi.org/10.1126/science.290.5500.2323

    Article  Google Scholar 

  39. Huber PJ (1981) Robust statistics. Wiley, New York

    Book  MATH  Google Scholar 

  40. Liu GC, Lin ZC, Yan SC, Sun J, Yu Y, Ma Y (2012) Robust recovery of subspace structures by low-rank representation. IEEE Trans Pattern Anal Mach Intell 35(1):171–184. https://doi.org/10.1109/TPAMI.2012.88

    Article  Google Scholar 

  41. Chen YY, Xiao XL, Zhou YC (2019) Low-rank quaternion approximation for color image processing. IEEE Trans Image Process 29:1426–1439. https://doi.org/10.1109/TIP.2019.2941319

    Article  MathSciNet  MATH  Google Scholar 

  42. Cai C, Li G, Poor HV, Chen Y (2019) Nonconvex low-rank tensor completion from noisy data. In: Proceeding of Advances in Neural Inf. Process. Syst., pp 1863-1874

  43. Eckstein J, Bertsekas DP (1992) On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math Program 55(1):293–318. https://doi.org/10.1007/BF01581204

    Article  MathSciNet  MATH  Google Scholar 

  44. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122. https://doi.org/10.1561/2200000016

    Article  MATH  Google Scholar 

  45. Cai JF, Cands EJ, Shen Z (2010) A singular value thresholding algorithm for matrix completion. SIAM J Optim 20(4):1956–1982. https://doi.org/10.1137/080738970

    Article  MathSciNet  MATH  Google Scholar 

  46. Thomaz CE, Giraldi GA (2010) A new ranking method for principal components analysis and its application to face image analysis. Image Vis Comput 28(6):902–913. https://doi.org/10.1016/j.imavis.2009.11.005

    Article  Google Scholar 

  47. Tang XO, Wang XG (2002) Face photo recognition using sketch. In: Proceeding of IEEE International Conference on Image Processing, pp. 257–260

  48. Hwang H, Haddad RA (1995) Adaptive median filters: new algorithm and results. IEEE Trans Image Process 4(4):499–502. https://doi.org/10.1109/83.370679

    Article  Google Scholar 

  49. Sun L, Ma C et al (2020) Low rank component induced spatial-spectral kernel method for hyperspectral image classification. IEEE Trans Cir Syst Video Technol 30(10):3829–3842. https://doi.org/10.1109/TCSVT.2019.2946723

    Article  Google Scholar 

  50. Sun L, Wu F et al (2020) Weighted nonlocal low-rank tensor decomposition method for sparse Unmixing of hyperspectral images. IEEE J Sel Top Appl Earth Obs Remote Sens 13(1):1174–1188. https://doi.org/10.1109/JSTARS.2020.2980576

    Article  Google Scholar 

  51. Yeh CH, Huang CH, Kang LW (2019) Multi-scale deep residual learning-based single image haze removal via image decomposition. IEEE Trans Image Process 29:3153–3167. https://doi.org/10.1109/TIP.2019.2957929

    Article  MATH  Google Scholar 

  52. Duan S, Chen Z, Wu Q, Cai L, Lu D (2021) Multi-scale gradients self-attention residual learning for face photo-sketch transformation. IEEE Trans Inf Forensics Secur 16:1218–1230. https://doi.org/10.1109/TIFS.2020.3031386

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China [Grant No. 61603159, 62006097, 61902160], Natural Science Foundation of Jiangsu Province [Grant No. BK20160293, BK20200593], Excellent Key Teachers of QingLan Project in Jiangsu Province, Excellent Scientific and Technological Innovation Team of Higher Education in Jiangsu Province(2019-29).

Author information

Authors and Affiliations

  1. Department of Criminal Science and Technology, Nanjing Forest Police College, Nanjing, 210023, China

    Songze Tang

  2. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, 650500, China

    Zhenqiu Shu

Authors
  1. Songze Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenqiu Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Songze Tang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, S., Shu, Z. Mixed noise face hallucination via adaptive weighted residual and nuclear-norm regularization. Appl Intell 53, 11979–11996 (2023). https://doi.org/10.1007/s10489-022-04018-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04018-w

Keywords

Search

Navigation