Abstract
Single Image Super- Resolution (SISR) is a complex restoration method to recover high-resolution (HR) image from degraded low-resolution (LR) form. SISR is used in many applications, such as microscopic image analysis, medical imaging, security and surveillance, astronomical observation, hyperspectral imaging, and text image super-resolution. Convolutional Neural Networks (CNNs) are most widely used technique to solve Super-Resolution (SR) problems. This paper presents review of SISR methods based on CNN. The SISR CNN models are analyzed based on the design and their performance on benchmark datasets: Set 5, Set 14, BSD 100, and Urban 100. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are used for quantitative analysis. ESRGAN model shows the best results on all benchmark datasets and reconstructs images with good visual quality at large upscaling factors. The model performs excellently with PSNR 27.03 dB and SSIM 0.8153 on the Urban 100 dataset for ×4 upscaling factor. The models are further analyzed on the basis of the loss function, scalability, processing time, and number of parameters. The framework and implementation setup of SISR CNN models are also discussed. Perceptual loss function can help to boost the network performance by increasing the visual quality of the reconstructed images. Hence, it has emerged as a new research trend in recent years. It is also observed that there is tremendous growth in the field of blind or unsupervised SISR. The research has shifted to developing reference less performance evaluation parameters for unsupervised SISR.
Similar content being viewed by others
Data Availability
NA
Code Availability
NA
References
Yuan Y, Zheng X, Lu X (2017) Hyperspectral image super resolution by transfer learning. IEEE J Select Top Appl Earth Observ Remote Sens 10(5):1963–1974
Li Y, Hu J, Zhao X, Tao R (2017) Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing. 266:29–41
Ribeiro E, Uhl FF (2019) Iris super-resolution using CNNs: photo-realism important to iris recognition. IET Biometrics 8(1):69–78
Bisen D, Shukla R, Rajpoot N (2022) Responsive human-computer interaction model based on recognition of facial landmarks using machine learning algorithms. Multimed Tools Appl 81:18011–18031. https://doi.org/10.1007/s11042-022-12775-6
Sharma A, Shrivastava BP (2023) Facial image super-resolution using progressive network interleaved correlation filter. Multimed Tools Appl. https://doi.org/10.1007/s11042-023-14765-8
Pham C, Ducournau A, Fablet R, Rousseau F (2017) Brain MRI super-resolution using deep 3D convolutional networks. Proc. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp.187-200
Khan A, Khan M, Obaid F, Jadoon S, Khan MA, Sikandar M (2015) A novel multi-frame super resolution algorithm for surveillance camera image reconstruction. Proc. 2015 First International Conf. on Anti-Cybercrime (ICACC), pp. 1-6
Tang J, Huang C, Zhu H, Liu J (2020) Image Super-Resolution Based on CNN Using Multilabel Gene Expression Programming. Appl Sci 10(3):854–868
Zhang H, Liu D, Xiong Z (2017) CNN-based text image super-resolution tailored for OCR, Proc. 2017 IEEE Visual Communications and Image Process. (VCIP), pp. 1-4
Duchon C (1979) Lanczos filtering in one and two dimensions. J Appl Meteorol 18(8):1016–1022
Sun J, Sun J, Xu Z, Shum H (2011) Gradient Profile Prior and Its Applications in Image Super-Resolution and Enhancement. IEEE Trans Image Process 20(6):1529–1542
Freeman W, Jones T, Pasztor E (2002) Example based super resolution. IEEE Comput Graph Appl 22(2):56–65
Dai S, Han M, Xu W, Wu Y (2009) Soft cuts: a soft edge smoothness prior for color image super-resolution. IEEE Trans Image Process 18(5):969–981
Sun J, Xu Z, Shum H (2008) Image super-resolution using gradient profile prior. Proc. Computer Vision and Pattern Recognition , Anchorage, AK, pp. 1-8
Yan Q, Xu Y, Yang X, Nguyen T (2015) Single image super resolution based on gradient profile sharpness. IEEE Trans Image Process 24(10):3187–3202
Shukla AK, Pandey RK (2021) Yadav S (2021) Adaptive fractional masks and super resolution based approach for image enhancement. Multimed Tools Appl 80:30213–30236. https://doi.org/10.1007/s11042-020-08968-6
Hu J, Luo Y (2015) Noise-robust video super-resolution using an adaptive spatial-temporal filter. Multimed Tools Appl 74:9259–9278. https://doi.org/10.1007/s11042-014-2079-y
Gupta S, Sharma DK, Ranta S (2022) A new hybrid image enlargement method using singular value decomposition and cubic spline interpolation. Multimed Tools Appl 81:4241–4254
Freedman G, Fattal R (2011) Image and video upscaling from local self-examples. ACM Trans. on. Graphics (TOG) 12(8):651–664
Wang Z, Yang Y, Wang Z, Chang S, Yang J, Huang T (2015) Learning super-resolution jointly from external and internal examples. IEEE Trans Image Process 24(11):4359–4371
Guo F, Zhang C, Zhang M (2021) Hyperspectral image super-resolution through clustering-based sparse representation. Multimed Tools Appl 80:7351–7366. https://doi.org/10.1007/s11042-020-09952-w
Hardiansyah B, Lu Y (2021) Single image super-resolution via multiple linear mapping anchored neighborhood regression. Multimed Tools Appl 80:28713–28730. https://doi.org/10.1007/s11042-021-11062-0
Hou M, Feng Z, Wang H (2022) An adaptive regression based single-image super-resolution. Multimed Tools Appl 81:28231–28248. https://doi.org/10.1007/s11042-022-12911-2
Yang X, Wu W, Lu L (2020) Multiple Regressions based Image Super-resolution. Multimed Tools Appl 79:8911–8927. https://doi.org/10.1007/s11042-019-7716-z
Zhang K, Zuo W, Zhang L (2018) FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising. IEEE Trans Image Process 27(9):4608–4621
Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks, Neural Information Processing Systems, 1097-1105
Bhuyan HK Ravi V (2021) Analysis of Subfeature for Classification in Data Mining IEEE Trans Eng Manage. 1-16. https://doi.org/10.1109/TEM.2021.3098463
Bhuyan HK, Chakraborty C, Shelke Y, Pani SK (2021) COVID-19 diagnosis system by deep learning approaches. Expert Systems 39(3):1-18 https://doi.org/10.1111/exsy.12776
Bhuyan HK, Chakraborty C (2022) Explainable Machine Learning for Data Extraction Across Computational Social System. IEEE Trans Comput Social Syst 99:1–15. https://doi.org/10.1109/TCSS.2022.3164993
Bhuyan HK, Ravi V, Yadav MS (2022) Multi-objective optimization-based privacy in data mining. Cluster Comput 25:4275–4287. https://doi.org/10.1007/s10586-022-03667-3
Mishra J, Goyal S (2022) An effective automatic traffic sign classification and recognition deep convolutional networks. Multimed Tools Appl 81:18915–18934. https://doi.org/10.1007/s11042-022-12531-w
Salau AO, Jain S (2019) Feature Extraction: A Survey of the Types, Techniques, Applications. 2019 International Conference on Signal Processing and Communication (ICSC), pp. 158-164. doi: https://doi.org/10.1109/ICSC45622.2019.8938371
Mutlag WK, Ali SK, Aydam ZM, Taher BH (2020) Feature Extraction Methods: A Review. The Fifth International Scientific Conference of Al-Khwarizmi. 1591, 012028. https://doi.org/10.1088/1742-6596/1591/1/012028
Dong C, Loy C, He K, Tang X (2016) Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans Patt Anal Mach Intell 38(2):295–307
Kim J, Lee J, Lee K (2016) Accurate Image Super-Resolution Using Very Deep Convolutional Networks. Proc. 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1646-1654
Zhang K, Zuo W, Gu S, Zhang L (2017) Learning Deep CNN Denoiser Prior for Image Restoration. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2808-2817
Zhang K, Zuo W, Chen Y (2017) Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Trans Image Process 26(7):3142–3155
Mao X, Shen C, Yang Y (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Proc. Neural Information Process. System, pp. 2810–2818
Kim J, Lee J, Lee K (2016) Deeply-Recursive Convolutional Network for Image Super-Resolution. Proc. 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1637-1645
Tai Y, Yang J, Liu X (2017) Image Super-Resolution via Deep Recursive Residual Network. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2790-2798
Jiao J, Tu W, He S, Lau RWH (2017) FormResNet: Formatted Residual Learning for Image Restoration. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1034-1042
Tai Y, Yang J, Liu X, Xu C (2017) MemNet: A persistent memory network for image restoration. Proc.Computer Vision Pattern Recognition. Venice, pp. 4539-4547
Han W, Chang S, Liu D, Yu M, Witbrock M, Huang S (2018) Image super-resolution via dual-state recurrent networks. Proc. Computer Vision PatternRecognition, pp.1654-1663
Hu Y, Gao X., Li J, Huang Y, Wang H (2018) Single image super resolution via cascaded multi-scale cross network. Proc. Computer Vision Pattern Recognition, pp.512-531
Ren H, Khamy M, Lee J (2017) Image Super Resolution Based on Fusing Multiple Convolution Neural Networks. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.1050-1057
Dong C, Loy C, Tang X (2016) Accelerating the super resolution convolutional neural network. Proc. European Conf. on Computer Vision, pp.1187-1202
Fan Y, Shi H, Yu J, Liu D (2017) Balanced two-stage residual networks for image super-resolution. Proc. Computer Vision Pattern Recognition, pp. 1157-1164
Tong T, Li G, Liu, X, Gao Q (2017) Image Super-Resolution using Dense Skip Connections. Proc. 2017 IEEE International Conf. on Computer Vision (ICCV), pp. 4809-4817
Hui Z, Wang X, Gao X (2018) Fast and accurate single image super-resolution via information distillation network. Proc. Computer Vision Pattern Recognition, pp.723-731
Shi W, Caballero J, Huszár F, Totz J, Aitken A, Bishop R, Rueckert D, Wang Z (2016) Real-Time Single Image and Video Super-Resolution using an Efficient Sub-Pixel Convolutional Neural Network. Proc. 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1874-1883
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta, A, Aitken A (2017) Photo-Realistic Single Image Super-Resolution using a Generative Adversarial Network. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp.105-114
Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual Dense Network for Image Super-Resolution. Proc. 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pp. 2472-2481
Lim B, Son S, Kim H, Nah S, Lee K (2017) Enhanced deep residual networks for single image super-resolution. Proc. Computer Vision Pattern Recognition, International Conf. on Computer Vision, pp. 136-144
Park S, Son H, Cho S, Hong K, Lee S (2018) SRFeat: Single image super-resolution with feature discrimination. Proc. International Conf. on Computer Vision, pp. 439-455
Guo D, Niu Y, Xie P (2019) Speedy and accurate image super-resolution via deeply recursive CNN with skip connection and network in network. IET Image Process 13(7):1201–1209
Yamanaka J (2020) Fast and Accurate Image Super Resolution by Deep CNN with Skip Connection and Network in Network. Computer vision and pattern recognition, pp.456-471
Zhang K, Zuo W, Zhang L (2018) Learning a Single Convolutional Super-Resolution Network for Multiple Degradations. Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City. UT, pp. 3262-3271
Mei S, Jiang R, Li X, Du Q (2020) Spatial and Spectral Joint Super-Resolution Using Convolutional Neural Network. IEEE Trans Geosci Remote Sens 58(7):4590–4603
Lai W, Huang J, Ahuja N, Yang M (2017) Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 5835-5843
Wang Y, Perazzi F, McWilliams B (2018) A fully progressive approach to single-image super-resolution. Proc. Computer Vision Pattern Recognition, pp. 977-986
Haris M, Shakhnarovich G, Ukita N (2018) Deep Back-Projection Networks for Super-Resolution. Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1664-1673
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proc. IEEE Computer Vision and Pattern Recognition, pp. 770–778
Ahn N, Kang B, Sohn K (2018) Fast, accurate, and, lightweight super-resolution with cascading residual network. Proc. Computer Vision Pattern Recognition, pp. 256-272
Li J, Fang F, Mei K (2018) Multi-scale residual network for image super-resolution. Proc. International Conference on Computer Vision, pp. 517-532
Lan R, Sun L, Liu Z, Su Z, Lu H (2020) Cascading and Enhanced Residual Networks for Accurate Single-Image Super-Resolution, IEEE Trans. on Cybernetics, pp. 1-11. doi: https://doi.org/10.1109/TCYB.2019.2952710
Liu Z, Yuan L, Sun L (2022) Frequency separation-based multi-scale cascading residual block network for image super resolution. Multimed Tools Appl 81:6827–6848
Ahmadian K, Reza Alikhani H (2022) Single image super-resolution with self-organization neural networks and image laplace gradient operator. Multimed Tools Appl 81:10607–10630
Song Z, Zhao X, Jiang H (2021) Gradual deep residual network for super-resolution. Multimed Tools Appl 80:9765–9778
Barzegar S, Sharifi A, Manthouri M (2020) Super-resolution using lightweight detailnet network. Multimed Tools Appl 79:1119–1136. https://doi.org/10.1007/s11042-019-08218-4
Peng Y (2020) Super-resolution Reconstruction Using Multiconnection Deep Residual Network Combined an Improved Loss Function for Single-frame Image. Multimed Tools Appl 79:9351–9362
Wen R, Fu K, Sun H, Sun X (2018) Image Super resolution Using Densely Connected Residual Networks. IEEE Signal Process. Letters. 25(10):1565–1569
Hu S, Wang G, Wang Y (2020) Accurate image super-resolution using dense connections and dimension reduction network. Multimed Tools Appl 79(5):1427–1443
Lan R, Sun L, Liu Z, Lu H, Pang C, Luo X (2020) MADNet: A Fast and Lightweight Network for Single-Image Super Resolution. IEEE Trans Cybern 63(7):2256–2284
Sajjadi M, Scholkopf B, Hirsch M (2017) Enhancenet: Single image super-resolution through automated texture synthesis. Proc. International Conf. on Computer Vision. pp. 4491-4500 doi: arXiv:1612.07919
Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Loy C (2018) ESRGAN: Enhanced super-resolution generative adversarial networks. Proc. Computer Vision Pattern Recognition, pp.63-79
Shocher A, Cohen N, Irani M. (2017) Zero-shot super-resolution using deep internal learning. Proc. Computer Vision Pattern Recognition, pp. 3118-3126. doi: arXiv:1712.06087
Yi Z, Zhang H, Tan P, Gong M (2017) DualGAN: Unsupervised dual learning for image-to-image translation, Proc. Computer Vision Pattern Recognition, pp.2849-2857
Zhu J, Park T, Isola P, Efros A (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Proc. 2017 IEEE International Conf. on Computer Vision, pp. 2242-2251
Yuan Y, Liu S, Zhang J, Zhang Y, Dong C, Lin L (2018) Unsupervised Image Super-Resolution Using Cycle-in-Cycle Generative Adversarial Networks. Proc. 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 814-824
Zhang Y, Li K, Wang L, Zhong B, Fu Y (2018) Image super resolution using very deep residual channel attention networks. Proc. Computer Vision Pattern Recognition, pp.294-310.
Choi J, Kim M (2017) A Deep Convolutional Neural Network with Selection Units for Super-Resolution. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW) Honolulu HI, pp. 1150-1156
Kim J, Choi J, Cheon M (2018) RAM: Residual attention module for single image super-resolution, Proc. Computer Vision Pattern Recognition, 886-895
Wu H, Zou Z, Gui J (2019) Multi-grained Attention Networks for Single Image Super Resolution. IEEE Trans Circuits Syst Video Technol 5(11):567–587
Ying W, Dong T, Shentu C (2023) Accurate stereo image super-resolution using spatial-attention-enhance residual network. Multimed Tools Appl 82:12117–12133. https://doi.org/10.1007/s11042-022-13815-x
Wang Y, Li X, Nan F (2022) Image super-resolution reconstruction based on generative adversarial network model with feedback and attention mechanisms. Multimed Tools Appl 81:6633–6652. https://doi.org/10.1007/s11042-021-11679-1
Oord A, Kalchbrenner N (2016) Conditional image generation with pixelcnn decoders. Proc. Neural Information Process. System, pp.4797–4805
Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks, Proc. 33rd International Conf. on Machine Learning, pp. 1747–1756
Salimans T, Karpathy A, Chen X., Kingma D (2017) PixelCNN++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. International Conf. on Learning Representations, pp.236-248
Huang H, He R, Sun Z, Tan T (2017) Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution. Proc. 2017 IEEE International Conf. on Computer Vision (ICCV), pp.1698-1706
Liu P, Zhang H, Zhang K, Lin L, Zuo W (2018) Multi-level wavelet-CNN for image restoration. Proc. Computer Vision Pattern Recognition, pp. 886-895
Guo T, Mousavi H, Vu T, Monga V (2017) Deep Wavelet Prediction for Image Super-Resolution. Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops, pp. 1100-1109
Russakovsky O, Deng J, Berg A, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252
Bevilacqua M, Roumy A, Guillemot C, Morel M (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding, Proc. British Machine Vision Conf., pp. 1-10
Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse representations, Proc. International conf. on curves and surfaces, Berlin, Heidelberg, pp. 711-730
Huang J, Singh A, Ahuja N (2015) Single image super resolution from transformed self-exemplars, Proc. Computer Vision Pattern Recognition, pp. 5197-5206
Timofte R, Smet V, Gool L (2017) NTIRE 2017 challenge on single image super-resolution: methods and results, Proc. Computer Vision Pattern Recognition, pp. 1110-1121
Martin D, Fowlkes C, Tal D (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, Proc. International Conf. on Computer Vision, pp. 416-423
Fujimoto A, Ogawa T, Yamamoto K. (2016) Manga109 dataset and creation of metadata, Proc. International Workshop on Analysis, Process. and Understanding, pp. 1-5
Yang C, Ma C, Yang M (2014) Single-image super-resolution: A benchmark. Proc. European Conf. Computer Vision. Zurich. Switzerland. September, pp.372–386
Wang Z, Bovik A, Sheikh H (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(11):600–612
Wang Z, Simoncelli E (2003) A.:Multiscale structural similarity for image quality assessment. The Thrity-Seventh Asilomar Conf. on Signals, Systems & Computers, pp. 1398-1402.
Venkata N, Kite T, Geisler W (2000) Image quality assessment based on a degradation model. IEEE Trans Image Process 9(11):636–650
Sheikh H, Bovik A, Veciana G (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14(11):2117–2128
Mittal A, Soundararajan R, Bovik AC (2013) Making a “completely blind” image quality analyzer. IEEE Signal Process Lett 20(3):209–212
Zhang L, Mou X, Zhang D (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378–2386
Mittal A, Moorthy AK, Bovik AC (2012) No-reference image quality assessment in the spatial domain. IEEE Trans Image Process 21(12):4695–4708. https://doi.org/10.1109/TIP.2012.2214050
Saad MA, Bovik AC, Charrier C (2012) Blind image quality assessment: a natural scene statistics approach in the DCT domain. IEEE Trans Image Process 21(8):3339–3352. https://doi.org/10.1109/TIP.2012.2191563
Moorthy AK, Bovik A (2011) Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Trans Image Process 20(12):3350–3364. https://doi.org/10.1109/TIP.2011.2147325
Greeshma MS, Bindu VR (2020) Super-resolution Quality Criterion (SRQC): a super-resolution image quality assessment metric. Multimed Tools Appl 79:35125–35146
Bruna J, Sprechmann P, LeCun Y (2016) Super-resolution with deep convolutional sufficient statistics. International conference on learning representations (ICLR)
Wu Q, Fan C, Li Y (2020) A novel perceptual loss function for single image super-resolution. Multimed Tools Appl 79:21265–21278. https://doi.org/10.1007/s11042-020-08878-7
Acknowledgments
Authors would like to express their gratitude to the anonymous reviewers for their insightful and productive feedback.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Dixit, M., Yadav, R.N. A Review of Single Image Super Resolution Techniques using Convolutional Neural Networks. Multimed Tools Appl 83, 29741–29775 (2024). https://doi.org/10.1007/s11042-023-16786-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-16786-9