Early Exit or Not: Resource-Efficient Blind Quality Enhancement for Compressed Images

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12361)


Lossy image compression is pervasively conducted to save communication bandwidth, resulting in undesirable compression artifacts. Recently, extensive approaches have been proposed to reduce image compression artifacts at the decoder side; however, they require a series of architecture-identical models to process images with different quality, which are inefficient and resource-consuming. Besides, it is common in practice that compressed images are with unknown quality and it is intractable for existing approaches to select a suitable model for blind quality enhancement. In this paper, we propose a resource-efficient blind quality enhancement (RBQE) approach for compressed images. Specifically, our approach blindly and progressively enhances the quality of compressed images through a dynamic deep neural network (DNN), in which an early-exit strategy is embedded. Then, our approach can automatically decide to terminate or continue enhancement according to the assessed quality of enhanced images. Consequently, slight artifacts can be removed in a simpler and faster process, while the severe artifacts can be further removed in a more elaborate process. Extensive experiments demonstrate that our RBQE approach achieves state-of-the-art performance in terms of both blind quality enhancement and resource efficiency.


Blind quality enhancement Compressed images Resource-efficient Early-exit 



This work was supported by the NSFC under Project 61876013, Project 61922009, and Project 61573037.

Supplementary material

504471_1_En_17_MOESM1_ESM.pdf (561 kb)
Supplementary material 1 (pdf 561 KB)


  1. 1.
    Arbeláez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011). Scholar
  2. 2.
    Cai, Q., Song, L., Li, G., Ling, N.: Lossy and lossless intra coding performance evaluation: HEVC, H. 264/AVC, JPEG 2000 and JPEG LS. In: Asia Pacific Signal and Information Processing Association Annual Summit and Conference, pp. 1–9. IEEE (2012)Google Scholar
  3. 3.
    Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1251–1258 (2017)Google Scholar
  4. 4.
    Cisco Systems Inc.: Cisco visual networking index: global mobile data traffic forecast update, 2017–2022 white paper.
  5. 5.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. (TIP) 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Dang-Nguyen, D.T., Pasquini, C., Conotter, V., Boato, G.: Raise: a raw images dataset for digital image forensics. In: The 6th ACM Multimedia Systems Conference, pp. 219–224. ACM (2015)Google Scholar
  7. 7.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255. IEEE (2009)Google Scholar
  8. 8.
    Dong, C., Deng, Y., Change Loy, C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: IEEE International Conference on Computer Vision (ICCV), pp. 576–584 (2015)Google Scholar
  9. 9.
    Fan, Z., Wu, H., Fu, X., Huang, Y., Ding, X.: Residual-guide network for single image deraining. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1751–1759 (2018)Google Scholar
  10. 10.
    Fu, C.M., et al.: Sample adaptive offset in the HEVC standard. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 22(12), 1755–1764 (2012)CrossRefGoogle Scholar
  11. 11.
    Gluck, M.A., Myers, C.E.: Hippocampal mediation of stimulus representation: a computational theory. Hippocampus 3(4), 491–516 (1993)CrossRefGoogle Scholar
  12. 12.
    Guan, Z., Xing, Q., Xu, M., Yang, R., Liu, T., Wang, Z.: MFQE 2.0: a new approach for multi-frame quality enhancement on compressed video. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), 1 (2019).
  13. 13.
    Guo, J., Chao, H.: Building dual-domain representations for compression artifacts reduction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 628–644. Springer, Cham (2016). Scholar
  14. 14.
    Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1712–1722 (2019)Google Scholar
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  16. 16.
    He, X., Hu, Q., Zhang, X., Zhang, C., Lin, W., Han, X.: Enhancing HEVC compressed videos with a partition-masked convolutional neural network. In: IEEE International Conference on Image Processing (ICIP), pp. 216–220. IEEE (2018)Google Scholar
  17. 17.
    Hennings-Yeomans, P.H., Baker, S., Kumar, B.V.: Simultaneous super-resolution and feature extraction for recognition of low-resolution faces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8. IEEE (2008)Google Scholar
  18. 18.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700–4708 (2017)Google Scholar
  19. 19.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  20. 20.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  21. 21.
    Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Artificial Intelligence and Statistics, pp. 562–570 (2015)Google Scholar
  22. 22.
    Li, K., Bare, B., Yan, B.: An efficient deep convolutional neural networks model for compressed image deblocking. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 1320–1325. IEEE (2017)Google Scholar
  23. 23.
    Li, L., Zhou, Y., Lin, W., Wu, J., Zhang, X., Chen, B.: No-reference quality assessment of deblocked images. Neurocomputing 177, 572–584 (2016)CrossRefGoogle Scholar
  24. 24.
    Li, L., Zhu, H., Yang, G., Qian, J.: Referenceless measure of blocking artifacts by Tchebichef kernel analysis. IEEE Signal Process. Lett. 21(1), 122–125 (2013)CrossRefGoogle Scholar
  25. 25.
    Li, S., Xu, M., Ren, Y., Wang, Z.: Closed-form optimization on saliency-guided image compression for HEVC-MSP. IEEE Trans. Multimed. (TMM) 20(1), 155–170 (2017)CrossRefGoogle Scholar
  26. 26.
    Liu, Y., Hamidouche, W., Déforges, O., Lui, Y., Dforges, O.: Intra Coding Performance Comparison of HEVC, H.264/AVC, Motion-JPEG2000 and JPEGXR Encoders. Research report, IETR/INSA Rennes, September 2018.
  27. 27.
    Lundh, F.: Python imaging library (PIL).
  28. 28.
    Marcellin, M.W., Gormish, M.J., Bilgin, A., Boliek, M.P.: An overview of JPEG-2000. In: Data Compression Conference (DCC), pp. 523–541. IEEE (2000)Google Scholar
  29. 29.
    Mukundan, R., Ong, S., Lee, P.A.: Image analysis by Tchebichef moments. IEEE Trans. Image Process. (TIP) 10(9), 1357–1364 (2001)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: The 27th International Conference on Machine Learning (ICML), pp. 807–814 (2010)Google Scholar
  31. 31.
    Nguyen, T., Marpe, D.: Performance analysis of HEVC-based intra coding for still image compression. In: Picture Coding Symposium (PCS), pp. 233–236. IEEE (2012)Google Scholar
  32. 32.
    Nguyen, T., Marpe, D.: Objective performance evaluation of the HEVC main still picture profile. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 25(5), 790–797 (2014)CrossRefGoogle Scholar
  33. 33.
    Norkin, A., et al.: HEVC deblocking filter. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 22(12), 1746–1754 (2012)CrossRefGoogle Scholar
  34. 34.
    Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3937–3946 (2019)Google Scholar
  35. 35.
    Seshadrinathan, K., Soundararajan, R., Bovik, A.C., Cormack, L.K.: Study of subjective and objective quality assessment of video. IEEE Trans. Image Process. (TIP) 19(6), 1427–1441 (2010)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  37. 37.
    Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 22(12), 1649–1668 (2012)CrossRefGoogle Scholar
  38. 38.
    Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: a persistent memory network for image restoration. In: IEEE International Conference on Computer Vision (ICCV), pp. 4539–4547 (2017)Google Scholar
  39. 39.
    Tan, T.K., Weerakkody, R., Mrak, M., Ramzan, N., Baroncini, V., Ohm, J.R., Sullivan, G.J.: Video quality evaluation methodology and verification testing of HEVC compression performance. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 26(1), 76–90 (2015)CrossRefGoogle Scholar
  40. 40.
    Wallace, G.K.: The JPEG still picture compression standard. IEEE Trans. Consum. Electron. (TCE) 38(1), xviii–xxxiv (1992).
  41. 41.
    Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: ECA-Net: efficient channel attention for deep convolutional neural networks. arXiv preprint arXiv:1910.03151 (2019)
  42. 42.
    Wang, T., Chen, M., Chao, H.: A novel deep learning-based method of improving coding efficiency from the decoder-end for HEVC. In: Data Compression Conference (DCC), pp. 410–419. IEEE (2017)Google Scholar
  43. 43.
    Wang, Z., Liu, D., Chang, S., Ling, Q., Yang, Y., Huang, T.S.: D3: deep dual-domain based fast restoration of JPEG-compressed images. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2764–2772 (2016)Google Scholar
  44. 44.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. (TIP) 13(4), 600–612 (2004)CrossRefGoogle Scholar
  45. 45.
    Xu, M., Li, T., Wang, Z., Deng, X., Yang, R., Guan, Z.: Reducing complexity of HEVC: a deep learning approach. IEEE Trans. Image Process. (TIP) 27(10), 5044–5059 (2018)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Yang, R., Xu, M., Liu, T., Wang, Z., Guan, Z.: Enhancing quality for HEVC compressed videos. IEEE Trans. Circuits Syst. Video Technol. (TCSVT) 2039–2054 (2018) Google Scholar
  47. 47.
    Yang, R., Xu, M., Wang, Z., Li, T.: Multi-frame quality enhancement for compressed video. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6664–6673 (2018)Google Scholar
  48. 48.
    Zhang, H., Yang, J., Zhang, Y., Nasrabadi, N.M., Huang, T.S.: Close the loop: joint blind image restoration and recognition with sparse representation prior. In: IEEE International Conference on Computer Vision (ICCV), pp. 770–777. IEEE (2011)Google Scholar
  49. 49.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. (TIP) 26(7), 3142–3155 (2017)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. (TIP) 27(9), 4608–4622 (2018)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: UNet++: a nested U-Net architecture for medical image segmentation. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 3–11. Springer, Cham (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Electronic and Information EngineeringBeihang UniversityBeijingChina
  2. 2.Hangzhou Innovation Institute of Beihang UniversityHangzhouChina

Personalised recommendations