Hallucinating Compressed Face Images

Abstract

A face hallucination algorithm is proposed to generate high-resolution images from JPEG compressed low-resolution inputs by decomposing a deblocked face image into structural regions such as facial components and non-structural regions like the background. For structural regions, landmarks are used to retrieve adequate high-resolution component exemplars in a large dataset based on the estimated head pose and illumination condition. For non-structural regions, an efficient generic super resolution algorithm is applied to generate high-resolution counterparts. Two sets of gradient maps extracted from these two regions are combined to guide an optimization process of generating the hallucination image. Numerous experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art hallucination methods on JPEG compressed face images with different poses, expressions, and illumination conditions.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Baker, S., & Kanade, T. (2002). Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9), 1167–1183.

    Article  Google Scholar 

  2. Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010). The generalized PatchMatch correspondence algorithm. In Proceedings of European conference on computer vision.

  3. Buades, A., Coll, B., & Morel, J. M. (2005). A non-local algorithm for image denoising. In Proceedings of IEEE conference on computer vision and pattern recognition.

  4. Choi, I., Kim, S., Brown, M., & Tai, Y. W. (2013). A learning-based approach to reduce JPEG artifacts in image matting. In Proceedings of IEEE international conference on computer vision.

  5. Figueiredo, M. A. T., Dias, J. B., Oliveira, J. P., & Nowak, R. (2006). On total variation denoising: A new majorization-minimization algorithm and an experimental comparison with wavelet denoising. In Proceedings of IEEE international conference on image processing.

  6. Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE Transactions on Image Processing, 16(5), 1395–1411.

    MathSciNet  Article  Google Scholar 

  7. Gross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2008). Multi-PIE. In Proceedings of IEEE conference on automatic face and gesture recognition.

  8. Jia, K., & Gong, S. (2005). Multi-modal tensor face for simultaneous super-resolution and recognition. In Proceedings of IEEE international conference on computer vision.

  9. Jiang, J., Hu, R., Wang, Z., & Han, Z. (2014). Noise robust face hallucination via locality-constrained representation. IEEE Transactions on Multimedia, 16(5), 1268–1281.

    Article  Google Scholar 

  10. Kim, K. I., & Kwon, Y. (2008). Example-based learning for single-image super-resolution and JPEG artifact removal. Max-Planck-Institut Technical Report.

  11. Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and simile classifiers for face verification. In Proceedings of IEEE international conference on computer vision.

  12. Li, Y., Guo, F., Tan, R. T., & Brown, M. S. (2014). A contrast enhancement framework with JPEG artifacts suppression. In Proceedings of European conference on computer vision.

  13. Liang, Y., Lai, J. H., Yuen, P. C., Zou, W. W., & Cai, Z. (2014). Face hallucination with imprecise-alignment using iterative sparse representation. Pattern Recognition, 47(10), 3327–3342.

    Article  Google Scholar 

  14. Liu, C., Shum, H. Y., & Freeman, W. T. (2007). Face hallucination: Theory and practice. International Journal of Computer Vision, 75(1), 115–134.

    Article  Google Scholar 

  15. Liu, S., & Bovik, A. C. (2002). Efficient DCT-domain blind measurement and reduction of blocking artifacts. IEEE Transactions on Circuits and Systems for Video Technology, 12(12), 1139–1149.

    Article  Google Scholar 

  16. Liu, S., & Yang, M. H. (2014). Compressed face hallucination. In Proceedings of IEEE international conference on image processing.

  17. Ma, X., Zhang, J., & Qi, C. (2010). Hallucinating face by position-patch. Pattern Recognition, 43(6), 2224–2236.

    Article  Google Scholar 

  18. Mairal, J., Bach, F., Ponce, J., Sapiro, G., & Zisserman, A. (2009). Non-local sparse models for image restoration. In Proceedings of IEEE international conference on computer vision.

  19. Park, J. S., & Lee, S. W. (2008). An example-based face hallucination method for single-frame, low-resolution facial images. IEEE Transactions on Image Processing, 17(10), 1806–1816.

    MathSciNet  Article  MATH  Google Scholar 

  20. Singh, S., Kumar, V., & Verma, H. K. (2007). Reduction of blocking artifacts in JPEG compressed images. Digital Signal Processing, 17(1), 225–243.

    Article  Google Scholar 

  21. Tappen, M. F., & Liu, C. (2012). A Bayesian approach to alignment-based image hallucination. In Proceedings of European conference on computer vision.

  22. Timofte, R., Smet, V. D., & Gool, L. V. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of Asian conference on computer vision.

  23. Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137–154.

    Article  Google Scholar 

  24. Voska, R., Mediachancecom. (2001). JpgQ—jpeg quality estimator. www.mediachance.com.

  25. Wang, N., Tao, D., Gao, X., Li, X., & Li, J. (2014). A comprehensive survey to face hallucination. International Journal of Computer Vision, 106(1), 9–30.

    Article  Google Scholar 

  26. Wang, X., & Tang, X. (2005). Hallucinating face by eigentransformation. IEEE Transactions on Systems, Man, and Cybernetics, 35(3), 425–434.

    Article  Google Scholar 

  27. Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  Google Scholar 

  28. Xiong, X., & la Torre, F. D. (2013). Supervised descent method and its application to face alignment. In Proceedings of IEEE conference on computer vision and pattern recognition.

  29. Xiong, Z., Sun, X., & Wu, F. (2010). Robust web image/video super-resolution. IEEE Transactions on Image Processing, 19(8), 2017–2028.

    MathSciNet  Article  MATH  Google Scholar 

  30. Yang, CY., Liu, S., & Yang, M. H. (2013). Structured face hallucination. In Proceedings of IEEE conference on computer vision and pattern recognition.

  31. Yang, J., Wright, J., Huang, T., & Ma, Y. (2008). Image super-resolution via sparse representation of raw image patches. In Proceedings of IEEE conference on computer vision and pattern recognition.

  32. Yang, J., Wright, J., Huang, T., & Ma, Y. (2010). Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11), 2861–2873.

    MathSciNet  Article  MATH  Google Scholar 

  33. Zhai, G., Zhang, W., Yang, X., Lin, W., & Xu, Y. (2008). Efficient deblocking with coefficient regularization, shape-adaptive filtering, and quantization constraint. IEEE Transactions on Multimedia, 10(5), 735–745.

    Article  Google Scholar 

  34. Zhu, X., & Ramanan, D. (2012). Face detection, pose estimation, and landmark localization in the wild. In Proceedings of IEEE conference on computer vision and pattern recognition.

Download references

Acknowledgements

This work is supported by NSF CAREER Grant 1149783, and gifts from Adobe and Nvidia.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ming-Hsuan Yang.

Additional information

Communicated by T.E. Boult.

Appendix

Appendix

Solving (2) Given a LR image D(L), and gradient maps U, we generate a HR image \(H^*\) through

$$\begin{aligned} H^* = \underset{H}{{\text {argmin}}} \Vert \nabla H - U\Vert ^2\;\; \text{ s.t. } \;\;(H \otimes G_\sigma ) \downarrow _s = D(L). \end{aligned}$$

To handle the nonlinear constraint, we relax the problem by

$$\begin{aligned} H^* = \underset{H}{{\text {argmin}}} \; \Vert \nabla H - U\Vert ^2 + \beta \Vert (H\otimes G_\sigma )\downarrow _s - D(L)\Vert ^2, \nonumber \\ \end{aligned}$$
(9)

where \(\beta \) is a weight parameter. We use the gradient descent method to solve the optimization problem.

Algorithm 1 shows the details how (9) is solved. The original energy value e is computed on Line 9, and a descent direction for generating a new image is computed on Line 10, where the \(\text{ Div }(\cdot )\) is a divergence operator and \(U^k\) means the k-th map in U for one of the eight derivative directions. We carry out a line search on Lines 11 to 15 and record the energy values of all step lengths in an array r. We find the best step index \(j^*\) and check the energy value \(r[j^*]\) on Line 17. If the new energy value \(r[j^*]\) is smaller than the original energy value e, the image is updated on Lines 18 to 19.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, C., Liu, S. & Yang, M. Hallucinating Compressed Face Images. Int J Comput Vis 126, 597–614 (2018). https://doi.org/10.1007/s11263-017-1044-4

Download citation

Keywords

  • Face hallucination
  • Super resolution
  • JPEG compression
  • Image denoising
  • Landmark points