Advertisement

Deep Learning Based Single Image Super-resolution: A Survey

  • Viet Khanh Ha
  • Jin-Chang RenEmail author
  • Xin-Ying Xu
  • Sophia Zhao
  • Gang Xie
  • Valentin Masero
  • Amir Hussain
Review

Abstract

Single image super-resolution has attracted increasing attention and has a wide range of applications in satellite imaging, medical imaging, computer vision, security surveillance imaging, remote sensing, objection detection, and recognition. Recently, deep learning techniques have emerged and blossomed, producing “the state-of-the-art” in many domains. Due to their capability in feature extraction and mapping, it is very helpful to predict high-frequency details lost in low-resolution images. In this paper, we give an overview of recent advances in deep learning-based models and methods that have been applied to single image super-resolution tasks. We also summarize, compare and discuss various models from the past and present for comprehensive understanding and finally provide open problems and possible directions for future research.

Keywords

Image super-resolution convolutional neural network high-resolution image low-resolution image deep learning 

Notes

Acknowledgements

The authors would like acknowledge the support from the Shanxi Hundred People Plan of China and colleagues from the Image Processing Group in Strathclyde University (UK), Anhui University (China) and Taibah Valley (Taibah University, Saudi Arabia) respectively, for their valuable suggestions.

References

  1. [1]
    D. Glasner, S. Bagon, M. Irani. Super-resolution from a single image. In Proceedings of the 12th International Conference on Computer Vision, IEEE, Kyoto, Japan, pp. 349–356, 2009. DOI:  https://doi.org/10.1109/ICCV.2009.5459271.Google Scholar
  2. [2]
    J. B. Huang, A. Singh, N. Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 5197–5206, 2015. DOI:  https://doi.org/10.1109/CVPR.2015.7299156.Google Scholar
  3. [3]
    W. T. Freeman, E. C. Pasztor, O. T. Carmichael. Learning low-level vision. International Journal of Computer Vision, vol. 40, no. 1, pp. 25–47, 2000. DOI:  https://doi.org/10.1023/A:1026501619075.CrossRefzbMATHGoogle Scholar
  4. [4]
    W. T. Freeman, T. R. Jones, E. C. Pasztor. Example-based super-resolution. IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56–65, 2002. DOI:  https://doi.org/10.1109/38.988747.CrossRefGoogle Scholar
  5. [5]
    H. Chang, D. Y. Yeung, Y. M. Xiong. Super-resolution through neighbor embedding. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, USA, 2004. DOI:  https://doi.org/10.1109/CVPR.2004.1315043.Google Scholar
  6. [6]
    C. Y. Yang, M. H. Yang. Fast direct super-resolution by simple functions. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 561–568, 2013. DOI:  https://doi.org/10.1109/ICCV.2013.75.Google Scholar
  7. [7]
    R. Timofte, V. De Smet, L. Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 1920–1927, 2013. DOI:  https://doi.org/10.1109/ICCV.2013.241.Google Scholar
  8. [8]
    R. Timofte, V. De Smet, L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the 12th Asian Conference on Computer Vision, Springer, Singapore, pp. 111–126, 2015. DOI:  https://doi.org/10.1007/978-3-319-16817-3_8.Google Scholar
  9. [9]
    S. Schulter, C. Leistner, H. Bischof. Fast and accurate image upscaling with super-resolution forests. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 3791–3799, 2015. DOI:  https://doi.org/10.1109/CVPR.2015.7299003.Google Scholar
  10. [10]
    E. Pérez-Pellitero, J. Salvador, J. Ruiz-Hidalgo, B. Rosenhahn. PSyCo: Manifold span reduction for super resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1837–1845, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.203.Google Scholar
  11. [11]
    J. C. Yang, J. Wright, T. S. Huang, Y. Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010. DOI:  https://doi.org/10.1109/TIP.2010.2050625.MathSciNetCrossRefzbMATHGoogle Scholar
  12. [12]
    J. C. Yang, Z. W. Wang, Z. Lin, S. Cohen, T. Huang. Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3467–3478, 2012. DOI:  https://doi.org/10.1109/TIP.2012.2192127.MathSciNetCrossRefzbMATHGoogle Scholar
  13. [13]
    T. Peleg, M. Elad. A statistical prediction model based on sparse representations for single image super-resolution. IEEE Transactions on Image Processing, vol. 23, no. 6, pp. 2569–2582, 2014. DOI:  https://doi.org/10.1109/TIP.2014.2305844.MathSciNetCrossRefzbMATHGoogle Scholar
  14. [14]
    S. L. Wang, L. Zhang, Y. Liang, Q. Pan. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, pp. 2216–2223, 2012. DOI:  https://doi.org/10.1109/CV-PR.2012.6247930.Google Scholar
  15. [15]
    L. He, H. R. Qi, R. Zaretzki. Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA, pp. 345–352, 2013. DOI:  https://doi.org/10.1109/CVPR.2013.51.Google Scholar
  16. [16]
    C. Dong, C. C. Loy, K. M. He, X. O. Tang. Learning a deep convolutional network for image super-resolution. In Proceedings of the 13th European Conference on Computer Vision, Springer, Zurich, Switzerland, pp. 184–199, 2014. DOI:  https://doi.org/10.1007/978-3-319-10593-2_13.Google Scholar
  17. [17]
    C. Dong, C. C. Loy, K. M. He, X. O. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. DOI:  https://doi.org/10.1109/TPAMI.2015.2439281.CrossRefGoogle Scholar
  18. [18]
    J. Kim, J. Kwon Lee, K. Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1646–1654, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.182.Google Scholar
  19. [19]
    J. Kim, J. Kwon Lee, K. Mu Lee. Deeply-recursive convolutional network for image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1637–1645, 2016. DOI:  https://doi.org/10.1109/CVPR.2016.181.Google Scholar
  20. [20]
    Y. Tai, J. Yang, X. M. Liu. Image super-resolution via deep recursive residual network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, vol. 1, pp. 2790–2798, 2017. DOI:  https://doi.org/10.1109/CVPR.2017.298.Google Scholar
  21. [21]
    X. J. Mao, C. H. Shen, Y. B. Yang. Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections, [Online], Available: https://arxiv.org/abs/1606.08921, May, 2018.Google Scholar
  22. [22]
    J. Yamanaka, S. Kuwashima, T. Kurita. Fast and accurate image super resolution by deep CNN with skip connection and network in network. In Proceedings of the 24th International Conference on Neural Information Processing, Springer, Guangzhou, China, 2017. DOI:  https://doi.org/10.1007/978-3-319-70096-0_23.Google Scholar
  23. [23]
    T. Tong, G. Li, X. J. Liu, Q. Q. Gao. Image super-resolution using dense skip connections. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4809–4817, 2017. DOI:  https://doi.org/10.1109/ICCV.2017.514.Google Scholar
  24. [24]
    Y. L. Zhang, K. P. Li, K. Li, L. C. Wang, B. N. Zhong, Y. Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 286–301, 2018. DOI:  https://doi.org/10.1007/978-3-030-01234-2_18.Google Scholar
  25. [25]
    Z. S. Zhong, T. C. Shen, Y. B. Yang, Z. C. Lin, C. Zhang. Joint sub-bands learning with clique structures for wavelet domain super-resolution. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Curran Associates, Inc., Montréal, Canada, pp. 165–175, 2018.Google Scholar
  26. [26]
    Y. L. Zhang, Y. P. Tian, Y. Kong, B. N. Zhong, Y. Fu. Residual dense network for image super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 2472–2481, 2018. DOI:  https://doi.org/10.1109/CVPR.2018.00262.Google Scholar
  27. [27]
    J. H. Yu, Y. C. Fan, J. C. Yang, N. Xu, Z. W. Wang, X. C. Wang, T. Huang. Wide Activation for Efficient and Accurate Image Super-resolution, [Online], Available: https://arxiv.org/abs/1808.08718v1, April 8, 2019.Google Scholar
  28. [28]
    N. Ahn, B. Kang, K. A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 252–268, 2018. DOI:  https://doi.org/10.1007/978-3-030-01249-6_16.Google Scholar
  29. [29]
    Z. Hui, X. M. Wang, X. B. Gao. Fast and accurate single image super-resolution via information distillation network. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 723–731, 2018. DOI:  https://doi.org/10.1109/CVPR.2018.00082.Google Scholar
  30. [30]
    W. S. Lai, J. B. Huang, N. Ahuja, M. H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, vol. 2, pp. 5835–5843, 2017. DOI:  https://doi.org/10.1109/CVPR.2017.618.Google Scholar
  31. [31]
    R. S. Asamwar, K. M. Bhurchandi, A. S. Gandhi. Interpolation of images using discrete wavelet transform to simulate image resizing as in human vision. International Journal of Automation and Computing, vol. 7, no. 1, pp. 9–16, 2010. DOI:  https://doi.org/10.1007/s11633-010-0009-7.CrossRefGoogle Scholar
  32. [32]
    B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Honolulu, USA, vol. 1, pp. 1132–1140, 2017. DOI:  https://doi.org/10.1109/CVPRW.2017.151.Google Scholar
  33. [33]
    Y. F. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine-Hornung, C. Schroers. A fully progressive approach to single-image super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Salt Lake City, USA, 2018. DOI:  https://doi.org/10.1109/CVPRW.2018.00131.Google Scholar
  34. [34]
    M. Haris, G. Shakhnarovich, N. Ukita. Deep back-projection networks for super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1664–1673, 2018. DOI:  https://doi.org/10.1109/CVPR.2018.00179.Google Scholar
  35. [35]
    K. Zhang, W. M. Zuo, L. Zhang. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3262–3271, 2018. DOI:  https://doi.org/10.1109/CVPR.2018.00344.Google Scholar
  36. [36]
    A. Shocher, N. Cohen, M. Irani. Zero-shot super-resolution using deep internal learning. In Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3118–3126, 2018. DOI:  https://doi.org/10.1109/CVPR.2018.00329.CrossRefGoogle Scholar
  37. [37]
    Q. L. Liao, T. Poggio. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex, [Online], Available: https://arxiv.org/abs/1604.03640, July 10, 2018.Google Scholar
  38. [38]
    W. Han, S. Y. Chang, D. Liu, M. Yu, M. Witbrock, T. S. Huang. Image super-resolution via dual-state recurrent networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1654–1663, 2018. DOI:  https://doi.org/10.1109/CV-PR.2018.00178.Google Scholar
  39. [39]
    Y. Tai, J. Yang, X. M. Liu, C. Y. Xu. MemNet: A persistent memory network for image restoration. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4539–4547, 2017. DOI:  https://doi.org/10.1109/ICCV.2017.486.Google Scholar
  40. [40]
    X. L. Wang, R. Girshick, A. Gupta, K. M. He. Non-local neural networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 7794–7803, 2018. DOI:  https://doi.org/10.1109/CVPR.2018.00813.Google Scholar
  41. [41]
    D. Liu, B. H. Wen, Y. C. Fan, C. C. Loy, T. S. Huang. Non-local recurrent network for image restoration. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Curran Associates, Inc., Montréal, Canada, pp. 1680–1689, 2018.Google Scholar
  42. [42]
    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 2672–2680, 2014.Google Scholar
  43. [43]
    C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. H. Wang, W. Z. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, vol. 2, pp. 105–114, 2017. DOI:  https://doi.org/10.1109/CVPR.2017.19.Google Scholar
  44. [44]
    M. S. Sajjadi, B. Schölkopf, M. Hirsch. EnhanceNet: Single image super-resolution through automated texture synthesis. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4501–4510, 2017. DOI:  https://doi.org/10.1109/ICCV.2017.481.Google Scholar
  45. [45]
    J. Johnson, A. Alahi, F. F. Li. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 694–711, 2016. DOI:  https://doi.org/10.1007/978-3-319-46475-6_43.Google Scholar
  46. [46]
    S. J. Park, H. Son, S. Cho, K. S. Hong, S. Lee. SRFeat: Single image super-resolution with feature discrimination. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 439–455, 2018. DOI:  https://doi.org/10.1007/978-3-030-01270-0_27.Google Scholar
  47. [47]
    M. Bevilacqua, A. Roumy, C. Guillemot, M. L. Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of British Machine Vision Conference, BMVA Press, Surrey, UK, 2012.Google Scholar
  48. [48]
    R. Zeyde, M. Elad, M. Protter. On single image scale-up using sparse-representations. In Proceedings of the 7th International Conference on Curves and Surfaces, Springer, Avignon, France, pp. 711–730, 2010. DOI:  https://doi.org/10.1007/978-3-642-27413-8_47.Google Scholar
  49. [49]
    D. Martin, C. Fowlkes, D. Tal, J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings the 8th IEEE International Conference on Computer Vision, IEEE, Vancouver, Canada, 2001. DOI:  https://doi.org/10.1109/ICCV.2001.937655.Google Scholar
  50. [50]
    V. K. Ha, J. C. Ren, X. Y. Xu, S. Zhao, G. Xie, V. M. Vargas. Deep learning based single image super-resolution: A survey. In Proceedings of the 9th International Conference on Brain Inspired Cognitive Systems, Springer, Xi’an, China, pp. 106–119, 2018. DOI:  https://doi.org/10.1007/978-3-030-00563-4_11.CrossRefGoogle Scholar
  51. [50]
    T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen. Improved techniques for training GANs. In Proceedings of the 30th Conference on Neural Information Processing Systems, Curran Associates, Inc., Barcelona, Spain, pp. 2234–2242, 2016.Google Scholar
  52. [51]
    M. Arjovsky, L. Bottou. Towards Principled Methods for Training Generative Adversarial Networks, [Online], Available: https://arxiv.org/abs/1701.04862, April 8, 2018.Google Scholar
  53. [52]
    L. Metz, B. Poole, D. Pfau, J. Sohl-Dickstein. Unrolled Generative Adversarial Networks, [Online], Available: https://arxiv.org/abs/1611.02163, June 10–20, 2018.Google Scholar
  54. [53]
    X. T. Wang, K. Yu, C. Dong, C. C. Loy. Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform, [Online], Available: https://arxiv.org/abs/1804.02815, October, 2018.CrossRefGoogle Scholar

Copyright information

© Institute of Automation, Chinese Academy of Sciences and Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electronic and Electrical EngineeringUniversity of StrathclydeGlasgowUK
  2. 2.College of Information EngineeringTaiyuan University of TechnologyTaiyuanChina
  3. 3.School of Electronic Information EngineeringTaiyuan University of Science and TechnologyTaiyuanChina
  4. 4.Department of Computer Systems and Telematics EngineeringUniversity of ExtremaduraBadajozSpain
  5. 5.School of ComputingEdinburgh Napier UniversityEdinburghUK
  6. 6.School of Computer Science and TechnologyAnhui UniversityAnhuiChina

Personalised recommendations