Noisy practical facial super-resolution method via deformable constrained model with small dataset

  • Liang ChenEmail author
  • Qing Li
  • Junjun Jiang


Face Super-Resolution (FSR) is to infer high resolution facial image(s) from given low resolution one(s). But when large-scale training samples are absent, FSR may fail in inferring high resolution image for practical low resolution facial image with complex degradation. To solve this problem, we present a novel position patch-based FSR method via latent Deformable Constrained Model (FSR-DCM). Different from conventional FSR methods that view an image patch as a fixed-length vector, we train the target image patch as a matrix in a flexible deformation flow form. This enables the dictionary to cover patterns that do not appear in training examples, resulting in our FSR method to be more expressive and able to solve the outlier (heterogeneous) problem. Besides, instead of explicit Euclidean distance, we use latent deformable similarity as distance criteria to measure the patch similarity, which facilitates our FSR method to work in low-quality scene by emphasizing neighbor relationship and enlarging the distance difference. Through enforcing such a constraint, the expressive capability of the FSR method can be improved, and the restoration failure caused by the coefficients of wrongly emphasized candidates can be overcome. Experiments on the public face datasets CAS-PEAL-R1 [9] and FEI [25] demonstrate the superiority of the proposed algorithm against the existing solutions to the problem of enhancing facial images of very low resolution both quantitatively (Peak-Singal-to-Noise Ratio, i.e., PSNR and Structural Similarity, i.e., the SSIM [29]) and qualitatively (subjective performance).


Face super resolution (FSR) Deformable constrained model (DCM) Latent distance constraint Facial image pattern 



This work was supported in part by the National Nature Science Foundation of China under Grant No. 61901117, U1805262, in part by the Natural Science Foundation of Fujian Province under the Grant 2019J05060 and Grant 2019J01271, in part by the Fujian Provincial Education Department Project under Grant JT180094 and Grant JT180095, in part by the Hubei Key Laboratory of Intelligent Robot, Wuhan Institute of Technology, under Grant HBIR201906, in part by the Special Funds of the Central Government Guiding Local Science and Technology Development under Grant 2017L3009, and in part by the National Key Research and Development Program of China under Grant 2016YFB1001001.


  1. 1.
    Baker S, Kanade T (2002) Limits on super-resolution and how to break them. IEEE Trans Pattern Anal Mach Intell 24(9):1167–1183CrossRefGoogle Scholar
  2. 2.
    Chakrabarti A, Rajagopalan AN, Chellappa R (2007) Super-resolution of face images using kernel PCA-based prior. IEEE Trans Multimed 9(4):888–892CrossRefGoogle Scholar
  3. 3.
    Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. Comput Vis Pattern Recogn 1:275–282Google Scholar
  4. 4.
    Chen L, Hu R, Jiang J, Han Z (2014) Efficient learning based face hallucination approach via facial standard deviation prior. In: IEEE International Symposium on Circuits and Systems, pp 2057–2060Google Scholar
  5. 5.
    Chen L, Hu R, Liang C, Li Q, Han Z (2017) A novel face super resolution approach for noisy images using contour feature and standard deviation prior. Multimed Tool Appl 76(2):2467–2493CrossRefGoogle Scholar
  6. 6.
    Dong X, Hu R, Jiang J, Han Z, Chen L, Gao G (2014) Noise face image hallucination via data-driven local eigentransformation. In: Pacific Rim Conference on Multimedia, pp 183–192Google Scholar
  7. 7.
    Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307CrossRefGoogle Scholar
  8. 8.
    Everingham M, Sivic J, Zisserman A (2006) Hello! My name is... Buffy–automatic naming of characters in TV video. In: British Machine Vision Conference, pp 899–908Google Scholar
  9. 9.
    Gao W, Cao B, Shan S (2008) The CAS-PEAL large-scale Chinese face database and baseline evaluations[J]Google Scholar
  10. 10.
    Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. International Conference on Computer Vision 349–356Google Scholar
  11. 11.
    Huang H, Wu N (2011) Fast facial image super-resolution via local linear transformations for resource-limited applications. IEEE Trans Circ Syst Video Technol 21(10):1363–1377MathSciNetCrossRefGoogle Scholar
  12. 12.
    Jiang J, Hu R, Han Z, Huang K, Lu T (2013) Graph discriminant analysis on multi-manifold (GDAMM): a novel super-resolution method for face recognition. IEEE International Conference on Image Processing 1465–1468Google Scholar
  13. 13.
    Jiang J, Hu R, Liang C, Han Z, Zhang C (2014) Face image super-resolution through locality-induced support regression. Signal Process 103:168–183CrossRefGoogle Scholar
  14. 14.
    Jiang J, Hu R, Wang Z, Han Z (2014) Face super-resolution via multilayer locality-constrained iterative neighbor embedding and intermediate dictionary learning. IEEE Trans Image Process 23(10):4220–4231MathSciNetCrossRefGoogle Scholar
  15. 15.
    Jiang J, Hu R, Wang Z, Han Z (2014) Noise robust face hallucination via locality-constrained representation. IEEE Trans Multimed 16(5):1268–1281CrossRefGoogle Scholar
  16. 16.
    Lan C, Hu R, Huang K, Han Z (2010) Face hallucination with shape parameters projection constraint. In: Proceedings of the ACM international conference on Multimedia, pp 883–886Google Scholar
  17. 17.
    Lee H, Battle A, Raina R, Ng AY (2007) Efficient sparse coding algorithms. In: Advances in neural information processing systems, pp 801–808Google Scholar
  18. 18.
    Liu S, Yang MH (2014) Compressed face hallucination. In: IEEE International Conference on Image Processing, pp 4032–4036Google Scholar
  19. 19.
    Liu C, Yuen J, Torralba A (2011) Sift flow: dense correspondence across scenes and its applications. IEEE Trans Pattern Anal Mach Intell 33(5):978–994CrossRefGoogle Scholar
  20. 20.
    Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence, pp 674–679Google Scholar
  21. 21.
    Ma X, Zhang J, Qi C (2009) Position-based face hallucination method. In: ICME 2009 IEEE International Conference on Multimedia and ExpoGoogle Scholar
  22. 22.
    Qu S, Hu R, Chen S, Jiang J, Wang Z, Chen J (2014) Face hallucination via re-identified K-nearest neighbors embedding. In: IEEE International Conference on Multimedia and Expo, pp 1–6Google Scholar
  23. 23.
    Rowley H, Baluja S, Kanade T (1998) Neural network-based face detection. IEEE Trans Pattern Anal Mach Intell 20(1):23–38CrossRefGoogle Scholar
  24. 24.
    Sun J, Zheng NN, Tao H, Shum HY (2003) Image hallucination with primal sketch priors. Comput Vis Pattern Recogn 2, II:729–36Google Scholar
  25. 25.
    Thomaz CE, Giraldi GA (2010) A new ranking method for principal components analysis and its application to face image analysis. Image Vis Comput 28(6):902–913CrossRefGoogle Scholar
  26. 26.
    Tibshirani R (1996) Regression shrinkage and selection via the lasso. J Royal Stat Soc Series B (Methodological) 267–288MathSciNetzbMATHGoogle Scholar
  27. 27.
    Timofte R, De Smet V, Van Gool L (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International Conference on Computer Vision, pp 1920–1927Google Scholar
  28. 28.
    Wang X, Tang X (2005) Hallucinating face by eigentransformation . IEEE Trans Syst, Man, Cybern Part C (Applications and Reviews) 35(3):425–434CrossRefGoogle Scholar
  29. 29.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13 (4):600–612CrossRefGoogle Scholar
  30. 30.
    Yang J, Tang H, Ma Y, Huang T (2008) Face hallucination via sparse coding. In: IEEE International Conference on Image Processing, pp 1264–1267Google Scholar
  31. 31.
    Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873MathSciNetCrossRefGoogle Scholar
  32. 32.
    Yang CY, Liu S, Yang MH (2013) Structured face hallucination. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1099–1106Google Scholar
  33. 33.
    Zhou E, Fan H, Cao Z, Jiang Y, Yin Q (2015) Learning face hallucination in the wild. Assoc Advan Artif Intell 3871–3877Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Fujian Provincial Engineering Technology Research Center of Photoelectric Sensing Application, Key Laboratory of OptoElectronic Science and Technology for Medicine of Ministry of Education, Fujian Provincial Key Laboratory of Photonics TechnologyFujian Normal UniversityFuzhouChina
  2. 2.The Hong Kong Polytechnic UniversityHong KongPeople’s Republic of China
  3. 3.Harbin Institute of TechnologyHarbinPeople’s Republic of China

Personalised recommendations