Skip to main content
Log in

Deep Learning Based Single Image Super-resolution: A Survey

  • Review
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

Single image super-resolution has attracted increasing attention and has a wide range of applications in satellite imaging, medical imaging, computer vision, security surveillance imaging, remote sensing, objection detection, and recognition. Recently, deep learning techniques have emerged and blossomed, producing “the state-of-the-art” in many domains. Due to their capability in feature extraction and mapping, it is very helpful to predict high-frequency details lost in low-resolution images. In this paper, we give an overview of recent advances in deep learning-based models and methods that have been applied to single image super-resolution tasks. We also summarize, compare and discuss various models from the past and present for comprehensive understanding and finally provide open problems and possible directions for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. D. Glasner, S. Bagon, M. Irani. Super-resolution from a single image. In Proceedings of the 12th International Conference on Computer Vision, IEEE, Kyoto, Japan, pp. 349–356, 2009. DOI: https://doi.org/10.1109/ICCV.2009.5459271.

    Google Scholar 

  2. J. B. Huang, A. Singh, N. Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 5197–5206, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7299156.

    Google Scholar 

  3. W. T. Freeman, E. C. Pasztor, O. T. Carmichael. Learning low-level vision. International Journal of Computer Vision, vol. 40, no. 1, pp. 25–47, 2000. DOI: https://doi.org/10.1023/A:1026501619075.

    MATH  Google Scholar 

  4. W. T. Freeman, T. R. Jones, E. C. Pasztor. Example-based super-resolution. IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56–65, 2002. DOI: https://doi.org/10.1109/38.988747.

    Google Scholar 

  5. H. Chang, D. Y. Yeung, Y. M. Xiong. Super-resolution through neighbor embedding. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, USA, 2004. DOI: https://doi.org/10.1109/CVPR.2004.1315043.

    Google Scholar 

  6. C. Y. Yang, M. H. Yang. Fast direct super-resolution by simple functions. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 561–568, 2013. DOI: https://doi.org/10.1109/ICCV.2013.75.

    Google Scholar 

  7. R. Timofte, V. De Smet, L. Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 1920–1927, 2013. DOI: https://doi.org/10.1109/ICCV.2013.241.

    Google Scholar 

  8. R. Timofte, V. De Smet, L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the 12th Asian Conference on Computer Vision, Springer, Singapore, pp. 111–126, 2015. DOI: https://doi.org/10.1007/978-3-319-16817-3_8.

    Google Scholar 

  9. S. Schulter, C. Leistner, H. Bischof. Fast and accurate image upscaling with super-resolution forests. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 3791–3799, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7299003.

    Google Scholar 

  10. E. Pérez-Pellitero, J. Salvador, J. Ruiz-Hidalgo, B. Rosenhahn. PSyCo: Manifold span reduction for super resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1837–1845, 2016. DOI: https://doi.org/10.1109/CVPR.2016.203.

    Google Scholar 

  11. J. C. Yang, J. Wright, T. S. Huang, Y. Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010. DOI: https://doi.org/10.1109/TIP.2010.2050625.

    MathSciNet  MATH  Google Scholar 

  12. J. C. Yang, Z. W. Wang, Z. Lin, S. Cohen, T. Huang. Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3467–3478, 2012. DOI: https://doi.org/10.1109/TIP.2012.2192127.

    MathSciNet  MATH  Google Scholar 

  13. T. Peleg, M. Elad. A statistical prediction model based on sparse representations for single image super-resolution. IEEE Transactions on Image Processing, vol. 23, no. 6, pp. 2569–2582, 2014. DOI: https://doi.org/10.1109/TIP.2014.2305844.

    MathSciNet  MATH  Google Scholar 

  14. S. L. Wang, L. Zhang, Y. Liang, Q. Pan. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, pp. 2216–2223, 2012. DOI: https://doi.org/10.1109/CV-PR.2012.6247930.

    Google Scholar 

  15. L. He, H. R. Qi, R. Zaretzki. Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA, pp. 345–352, 2013. DOI: https://doi.org/10.1109/CVPR.2013.51.

    Google Scholar 

  16. C. Dong, C. C. Loy, K. M. He, X. O. Tang. Learning a deep convolutional network for image super-resolution. In Proceedings of the 13th European Conference on Computer Vision, Springer, Zurich, Switzerland, pp. 184–199, 2014. DOI: https://doi.org/10.1007/978-3-319-10593-2_13.

    Google Scholar 

  17. C. Dong, C. C. Loy, K. M. He, X. O. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. DOI: https://doi.org/10.1109/TPAMI.2015.2439281.

    Google Scholar 

  18. J. Kim, J. Kwon Lee, K. Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1646–1654, 2016. DOI: https://doi.org/10.1109/CVPR.2016.182.

    Google Scholar 

  19. J. Kim, J. Kwon Lee, K. Mu Lee. Deeply-recursive convolutional network for image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1637–1645, 2016. DOI: https://doi.org/10.1109/CVPR.2016.181.

    Google Scholar 

  20. Y. Tai, J. Yang, X. M. Liu. Image super-resolution via deep recursive residual network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, vol. 1, pp. 2790–2798, 2017. DOI: https://doi.org/10.1109/CVPR.2017.298.

    Google Scholar 

  21. X. J. Mao, C. H. Shen, Y. B. Yang. Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections, [Online], Available: https://arxiv.org/abs/1606.08921, May, 2018.

    Google Scholar 

  22. J. Yamanaka, S. Kuwashima, T. Kurita. Fast and accurate image super resolution by deep CNN with skip connection and network in network. In Proceedings of the 24th International Conference on Neural Information Processing, Springer, Guangzhou, China, 2017. DOI: https://doi.org/10.1007/978-3-319-70096-0_23.

    Google Scholar 

  23. T. Tong, G. Li, X. J. Liu, Q. Q. Gao. Image super-resolution using dense skip connections. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4809–4817, 2017. DOI: https://doi.org/10.1109/ICCV.2017.514.

    Google Scholar 

  24. Y. L. Zhang, K. P. Li, K. Li, L. C. Wang, B. N. Zhong, Y. Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 286–301, 2018. DOI: https://doi.org/10.1007/978-3-030-01234-2_18.

    Google Scholar 

  25. Z. S. Zhong, T. C. Shen, Y. B. Yang, Z. C. Lin, C. Zhang. Joint sub-bands learning with clique structures for wavelet domain super-resolution. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Curran Associates, Inc., Montréal, Canada, pp. 165–175, 2018.

    Google Scholar 

  26. Y. L. Zhang, Y. P. Tian, Y. Kong, B. N. Zhong, Y. Fu. Residual dense network for image super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 2472–2481, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00262.

    Google Scholar 

  27. J. H. Yu, Y. C. Fan, J. C. Yang, N. Xu, Z. W. Wang, X. C. Wang, T. Huang. Wide Activation for Efficient and Accurate Image Super-resolution, [Online], Available: https://arxiv.org/abs/1808.08718v1, April 8, 2019.

    Google Scholar 

  28. N. Ahn, B. Kang, K. A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 252–268, 2018. DOI: https://doi.org/10.1007/978-3-030-01249-6_16.

    Google Scholar 

  29. Z. Hui, X. M. Wang, X. B. Gao. Fast and accurate single image super-resolution via information distillation network. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 723–731, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00082.

    Google Scholar 

  30. W. S. Lai, J. B. Huang, N. Ahuja, M. H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, vol. 2, pp. 5835–5843, 2017. DOI: https://doi.org/10.1109/CVPR.2017.618.

    Google Scholar 

  31. R. S. Asamwar, K. M. Bhurchandi, A. S. Gandhi. Interpolation of images using discrete wavelet transform to simulate image resizing as in human vision. International Journal of Automation and Computing, vol. 7, no. 1, pp. 9–16, 2010. DOI: https://doi.org/10.1007/s11633-010-0009-7.

    Google Scholar 

  32. B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Honolulu, USA, vol. 1, pp. 1132–1140, 2017. DOI: https://doi.org/10.1109/CVPRW.2017.151.

    Google Scholar 

  33. Y. F. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine-Hornung, C. Schroers. A fully progressive approach to single-image super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Salt Lake City, USA, 2018. DOI: https://doi.org/10.1109/CVPRW.2018.00131.

    Google Scholar 

  34. M. Haris, G. Shakhnarovich, N. Ukita. Deep back-projection networks for super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1664–1673, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00179.

    Google Scholar 

  35. K. Zhang, W. M. Zuo, L. Zhang. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3262–3271, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00344.

    Google Scholar 

  36. A. Shocher, N. Cohen, M. Irani. Zero-shot super-resolution using deep internal learning. In Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3118–3126, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00329.

    Google Scholar 

  37. Q. L. Liao, T. Poggio. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex, [Online], Available: https://arxiv.org/abs/1604.03640, July 10, 2018.

    Google Scholar 

  38. W. Han, S. Y. Chang, D. Liu, M. Yu, M. Witbrock, T. S. Huang. Image super-resolution via dual-state recurrent networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1654–1663, 2018. DOI: https://doi.org/10.1109/CV-PR.2018.00178.

    Google Scholar 

  39. Y. Tai, J. Yang, X. M. Liu, C. Y. Xu. MemNet: A persistent memory network for image restoration. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4539–4547, 2017. DOI: https://doi.org/10.1109/ICCV.2017.486.

    Google Scholar 

  40. X. L. Wang, R. Girshick, A. Gupta, K. M. He. Non-local neural networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 7794–7803, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00813.

    Google Scholar 

  41. D. Liu, B. H. Wen, Y. C. Fan, C. C. Loy, T. S. Huang. Non-local recurrent network for image restoration. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Curran Associates, Inc., Montréal, Canada, pp. 1680–1689, 2018.

    Google Scholar 

  42. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 2672–2680, 2014.

    Google Scholar 

  43. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. H. Wang, W. Z. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, vol. 2, pp. 105–114, 2017. DOI: https://doi.org/10.1109/CVPR.2017.19.

    Google Scholar 

  44. M. S. Sajjadi, B. Schölkopf, M. Hirsch. EnhanceNet: Single image super-resolution through automated texture synthesis. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4501–4510, 2017. DOI: https://doi.org/10.1109/ICCV.2017.481.

    Google Scholar 

  45. J. Johnson, A. Alahi, F. F. Li. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 694–711, 2016. DOI: https://doi.org/10.1007/978-3-319-46475-6_43.

    Google Scholar 

  46. S. J. Park, H. Son, S. Cho, K. S. Hong, S. Lee. SRFeat: Single image super-resolution with feature discrimination. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 439–455, 2018. DOI: https://doi.org/10.1007/978-3-030-01270-0_27.

    Google Scholar 

  47. M. Bevilacqua, A. Roumy, C. Guillemot, M. L. Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of British Machine Vision Conference, BMVA Press, Surrey, UK, 2012.

    MATH  Google Scholar 

  48. R. Zeyde, M. Elad, M. Protter. On single image scale-up using sparse-representations. In Proceedings of the 7th International Conference on Curves and Surfaces, Springer, Avignon, France, pp. 711–730, 2010. DOI: https://doi.org/10.1007/978-3-642-27413-8_47.

    MATH  Google Scholar 

  49. D. Martin, C. Fowlkes, D. Tal, J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings the 8th IEEE International Conference on Computer Vision, IEEE, Vancouver, Canada, 2001. DOI: https://doi.org/10.1109/ICCV.2001.937655.

    Google Scholar 

  50. V. K. Ha, J. C. Ren, X. Y. Xu, S. Zhao, G. Xie, V. M. Vargas. Deep learning based single image super-resolution: A survey. In Proceedings of the 9th International Conference on Brain Inspired Cognitive Systems, Springer, Xi’an, China, pp. 106–119, 2018. DOI: https://doi.org/10.1007/978-3-030-00563-4_11.

    Google Scholar 

  51. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen. Improved techniques for training GANs. In Proceedings of the 30th Conference on Neural Information Processing Systems, Curran Associates, Inc., Barcelona, Spain, pp. 2234–2242, 2016.

    Google Scholar 

  52. M. Arjovsky, L. Bottou. Towards Principled Methods for Training Generative Adversarial Networks, [Online], Available: https://arxiv.org/abs/1701.04862, April 8, 2018.

    Google Scholar 

  53. L. Metz, B. Poole, D. Pfau, J. Sohl-Dickstein. Unrolled Generative Adversarial Networks, [Online], Available: https://arxiv.org/abs/1611.02163, June 10–20, 2018.

    Google Scholar 

  54. X. T. Wang, K. Yu, C. Dong, C. C. Loy. Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform, [Online], Available: https://arxiv.org/abs/1804.02815, October, 2018.

    Google Scholar 

Download references

Acknowledgements

The authors would like acknowledge the support from the Shanxi Hundred People Plan of China and colleagues from the Image Processing Group in Strathclyde University (UK), Anhui University (China) and Taibah Valley (Taibah University, Saudi Arabia) respectively, for their valuable suggestions.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jin-Chang Ren.

Additional information

Recommended by Associate Editor Bin Luo

Viet Khanh Ha received the B. Eng. degrees in electrical and electronics from Le Quy Don University, Viet Nam in 2008, the M. Eng. degree in electrical and electronics from Wollongong University, Australia in 2012. He is currently a Ph. D. degree candidate at the University of Strathclyde, UK.

His research interests include image super resolution using deep learning.

Jin-Chang Ren received the B. Eng. degree in computer software in 1992, the M. Eng. degree in image processing in 1997, the Ph. D. degree in computer vision in 2000, all from the North-western Polytechnical University (NWPU), China. He was also awarded a Ph. D. in electronic imaging and media communication from Bradford University, UK in 2009. Currently, he is with Centre for Signal and Image Processing (CeSIP), University of Strathclyde, UK. He has published over 150 peer reviewed journals and conferences papers. He acts as an associate editor for two international journals including Multidimensional Systems and Signal Processing and International Journal of Pattern Recognition and Artificial Intelligence.

His research interests focus mainly on visual computing and multi-media signal processing, especially on semantic content extraction for video analysis and understanding more recently hyperspectral imaging.

Xin-Ying Xu received the B. Sc. and Ph. D. degrees from the Taiyuan University of Technology, China, in 2002 and 2009, respectively. He is currently a professor with the College of Information Engineering, Taiyuan University of Technology, China. He has published more than 30 academic papers. He is a member of the Chinese Computer Society, and has been a visiting scholar in Department of Computer Science, San Jose State University, USA.

His research interests include computational intelligence, data mining, wireless networking, image processing, and fault diagnosis.

Sophia Zhao received the B. Sc. degree in education from Henan University, China in 1999, and several qualifications from Shipley College, UK during 2003–2005. Currently, she is a research assistant with the Department of Electronic and Electrical Engineering, University of Strathclyde, UK.

Her research interests include image/signal analysis, machine learning and optimisation.

Gang Xie received the B. S. degree in control theory and the Ph. D. degree in circuits and systems from the Taiyuan University of Technology, China, in 1994 and 2006, respectively. He has been a professor and vice principle of Taiyuan University of Science and Technology, China. He has published over 80 research papers.

His research interests include rough sets, intelligent computing, image processing, automation and big data analysis.

Valentin Masero received the B. Eng. degree in computer science and business administration from University of Extremadura (UEX), Spain, and another B. Eng. degree in computer engineering specialized in software development and artificial intelligence from University of Granada, Spain. He received the Ph. D. degree in computer engineering from UEX, Spain. Now he is an associate professor at UEX.

His research interests include image processing, machine learning, artificial intelligence, computer graphics, computer programming, software development, computer applications in industrial engineering, computer applications in agricultural engineering and computer applications in healthcare.

Amir Hussain received the B. Eng. and Ph. D. degrees from the University of Strathclyde in Glasgow, UK, in 1992 and 1997, respectively. Following postdoctoral and academic positions at the Universities of West of Scotland (1996–1998), Dundee (1998–2000) and Stirling (2000-2018), respectively, he joined Edinburgh Napier University (UK) in 2018, as founding director of the Cognitive Big Data and Cybersecurity (CogBiD) Research Laboratory, managing over 25 academic and research staffs. He has been appointed to invited visiting professorships at several Universities and Research and Innovation Centres, including at Anhui University (China) and Taibah Valley (Taibah University, Saudi Arabia). He has (co)authored three international patents, around 400 publications, including over a dozen books and 150 journal papers. He has led major multi-disciplinary research projects, funded by national and European research councils, local and international charities and industry, and supervised more than 35 Ph. D. students. He is founding Editor-in-Chief of (Springer Nature’s) Cognitive Computation journal and BMC Big Data Analytics journal. He has been appointed Associate Editor of several other world-leading journals including, IEEE Transactions on Neural Networks and Learning Systems, (Elsevier’s) Information Fusion journal, IEEE Transactions on Emerging Topics in Computational Intelligence, and IEEE Computational Intelligence Magazine. Amongst other distinguished roles, he is General Chair for IEEE WCCI 2020 (the world’s largest and top IEEE technical event in computational intelligence, comprising IJCNN, FUZZ-IEEE and IEEE CEC), Vice-Chair of Emergent Technologies Technical Committee of the IEEE Computational Intelligence Society, and chapter Chair of the IEEE UK & Ireland, Industry Applications Society Chapter.

His research interests include developing cognitive data science and AI technologies, to engineer the smart and secure systems of tomorrow.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ha, V.K., Ren, JC., Xu, XY. et al. Deep Learning Based Single Image Super-resolution: A Survey. Int. J. Autom. Comput. 16, 413–426 (2019). https://doi.org/10.1007/s11633-019-1183-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-019-1183-x

Keywords

Navigation