The Visual Computer

, Volume 34, Issue 6–8, pp 1065–1076 | Cite as

Efficient image super-resolution integration

  • Ke Xu
  • Xin Wang
  • Xin Yang
  • Shengfeng He
  • Qiang Zhang
  • Baocai Yin
  • Xiaopeng Wei
  • Rynson W. H. Lau
Original Article


The super-resolution (SR) problem is challenging due to the diversity of image types with little shared properties as well as the speed required by online applications, e.g., target identification. In this paper, we explore the merits and demerits of recent deep learning-based and conventional patch-based SR methods and show that they can be integrated in a complementary manner, while balancing the reconstruction quality and time cost. Motivated by this, we further propose an integration framework to take the results from FSRCNN and A+ methods as inputs and directly learn a pixel-wise mapping between the inputs and the reconstructed results using the Gaussian conditional random fields. The learned pixel-wise integration mapping is flexible to accommodate different upscaling factors. Experimental results show that the proposed framework can achieve superior SR performance compared with the state of the arts while being efficient.


Image super-resolution Image processing Gaussian conditional random fields 



We thank the anonymous reviewers for the insightful and constructive comments. This work is in part supported by an SRG grant from City University of Hong Kong (Ref. 7004889), and by NSFC grant from National Natural Science Foundation of China (Ref. 91748104, 61632006, 61425002, 61702194).


This study was funded by an SRG grant from City University of Hong Kong (Ref. 7004889), and by NSFC grant from National Natural Science Foundation of China (Ref. 91748104, 61632006, 61425002, 61702194).

Compliance with Ethical Standards

Conflict of interest

Ke Xu, XinWang, Xin Yang, Shengfeng He, Qiang Zhang, Baocai Yin, Xiaopeng Wei and Rynson W.H. Lau declare that they have no conflict of interest.


  1. 1.
    Chang, H., Yeung, D., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR (2004)Google Scholar
  2. 2.
    Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. In: NIPS (2015)Google Scholar
  3. 3.
    Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. In: EG (2015)Google Scholar
  4. 4.
    Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y.: Soft edge smoothness prior for alpha channel super resolution. In: CVPR (2007)Google Scholar
  5. 5.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: ECCV (2014)Google Scholar
  6. 6.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE TPAMI 38, 295–307 (2016)CrossRefGoogle Scholar
  7. 7.
    Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: ECCV (2016)Google Scholar
  8. 8.
    Duchon, C.: Lanczos filtering in one and two dimensions. J. Appl. Meteorol. 18, 1016–1022 (1979)CrossRefGoogle Scholar
  9. 9.
    Efrat, N., Glasner, D., Apartsin, A., Nadler, B., Levin, A.: Accurate blur models vs. image priors in single image super-resolution. In: ICCV (2013)Google Scholar
  10. 10.
    Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14, 47–57 (2004)CrossRefGoogle Scholar
  11. 11.
    Fattal, R.: Image upsampling via imposed edge statistics. In: ACM TOG (2007)Google Scholar
  12. 12.
    Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM TOG 30, 12 (2011)CrossRefGoogle Scholar
  13. 13.
    Freeman, W., Jones, T., Pasztor, E.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 56–65 (2002)CrossRefGoogle Scholar
  14. 14.
    Freeman, W., Liu, C.: Markov random fields for super-resolution and texture synthesis. Adv. Markov Random Fields Vis. Image Process. 1, 3 (2011)Google Scholar
  15. 15.
    He, X., Zemel, R.S., Carreira-Perpiñán, M.Á.: Multiscale conditional random fields for image labeling. In: CVPR (2004)Google Scholar
  16. 16.
    Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR (2015)Google Scholar
  17. 17.
    Jancsary, J., Nowozin, S., Rother, C.: Loss-specific training of non-parametric image restoration models: a new state of the art. In: ECCV (2012)Google Scholar
  18. 18.
    Jancsary, J., Nowozin, S., Sharp, T., Rother, C.: Regression tree fields an efficient, non-parametric approach to image labeling problems. In: CVPR (2012)Google Scholar
  19. 19.
    Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2, 109–122 (2016)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Khler, T., Huang, X., Schebesch, F., Aichert, A., Maier, A., Hornegger, J.: Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans. Comput. Imaging 2, 42–58 (2016)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: CVPR (2016)Google Scholar
  22. 22.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)Google Scholar
  23. 23.
    Kim, K., Franz, M., Schölkopf, B.: Kernel hebbian algorithm for single-frame super-resolution. In: ECCV Workshop on Statistical Learning in Computer Vision (2004)Google Scholar
  24. 24.
    Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE TPAMI 32, 1127–1133 (2010)CrossRefGoogle Scholar
  25. 25.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)Google Scholar
  26. 26.
    Michaeli, T., Irani, M.: Nonparametric blind super-resolution. In: ICCV (2013)Google Scholar
  27. 27.
    Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20, 21–36 (2003)CrossRefGoogle Scholar
  28. 28.
    Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  29. 29.
    Salvador, J., Perez-Pellitero, E.: Naive bayes super-resolution forest. In: ICCV (2015)Google Scholar
  30. 30.
    Schmidt, U., Rother, C., Nowozin, S., Jancsary, J., Roth, S.: Discriminative non-blind deblurring. In: CVPR (2013)Google Scholar
  31. 31.
    Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: CVPR (2015)Google Scholar
  32. 32.
    Shen, X., Tao, X., Gao, H., Zhou, C., Jia, J.: Deep automatic portrait matting. In: ECCV (2016)Google Scholar
  33. 33.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556
  34. 34.
    Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR (2017)Google Scholar
  35. 35.
    Tappen, M., Liu, C., Adelson, E., Freeman, W.: Learning gaussian conditional random fields for low-level vision. In: CVPR (2007)Google Scholar
  36. 36.
    Thévenaz, P., Blu, T., Unser, M.: Image interpolation and resampling. Handb. Med. Imaging Process. Anal. 1, 393–420 (2000)CrossRefGoogle Scholar
  37. 37.
    Timofte, R., De, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: ICCV (2013)Google Scholar
  38. 38.
    Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: ACCV (2014)Google Scholar
  39. 39.
    Wang, Q., Tang, X., Shum, H.: Patch based blind image super resolution. In: ICCV (2005)Google Scholar
  40. 40.
    Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: ICCV (2015)Google Scholar
  41. 41.
    Xie, Z., Xu, K., Liu, L., Xiong, Y.: 3d shape segmentation and labeling via extreme learning machine. In: CGF (2014)Google Scholar
  42. 42.
    Xu, K., Shi, Y., Zheng, L., Zhang, J., Liu, M., Huang, H., Su, H., Cohen-Or, D., Chen, B.: 3d attention-driven depth acquisition for object identification. ACM TOG 35, 238 (2016)Google Scholar
  43. 43.
    Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE TIP 19, 2861–2873 (2010)MathSciNetzbMATHGoogle Scholar
  44. 44.
    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces (2010)Google Scholar
  45. 45.
    Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM TOMS 23, 550–560 (1997)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Ke Xu
    • 1
    • 3
  • Xin Wang
    • 1
  • Xin Yang
    • 1
  • Shengfeng He
    • 2
  • Qiang Zhang
    • 1
  • Baocai Yin
    • 1
  • Xiaopeng Wei
    • 1
  • Rynson W. H. Lau
    • 3
  1. 1.Dalian University of TechnologyDalianChina
  2. 2.South China University of TechnologyGuangzhouChina
  3. 3.City University of Hong KongKowloonHong Kong

Personalised recommendations