Skip to main content
SpringerLink
Log in

Effects of the deep learning-based super-resolution method on thermal image classification applications

  • 1212: Deep Learning Techniques for Infrared Image/Video Understanding
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Thermal imaging can be used in many sectors such as public security, health, and defense in image processing. However, thermal imaging systems are very costly, limiting their use, especially in the medical field. Also, thermal camera systems obtain blurry images with low levels of detail. Therefore, the need to improve their resolution has arisen. Here, super-resolution techniques can be a solution. Developments in deep learning in recent years have increased the success of super-resolution (SR) applications. This study proposes a new deep learning-based approach TSRGAN model for SR applications performed on a new dataset consisting of thermal images of premature babies. This dataset was created by downscaling the thermal images (ground truth) of premature babies as traditional SR studies. Thus, a dataset consisting of high-resolution (HR) and low-resolution (LR) thermal images were obtained. SR images created due to the applications were compared with LR, bicubic interpolation images, and obtained SR images using state-of-the-art models. The success of the results was evaluated using image quality metrics of peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM). The results show that the proposed model achieved the second-best PSNR value and the best SSIM value. Additionally, a CNN-based classifier model was developed to perform task-based evaluation, and classification applications were carried out separately on LR, HR, and reconstructed SR image sets. Here, the success of classifying unhealthy and healthy babies was compared. This study showed that the classification accuracy of SR images increased by approximately 5% compared to the classification accuracy of LR images. In addition, the classification accuracy of SR thermal images approached the classification accuracy of HR thermal images by about 2%. Therefore, with the approach proposed in this study, it has been proven that LR thermal images can be used in classification applications by increasing their resolution. Thus, widespread use of thermal imaging systems with lower costs in the medical field will be achieved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Achanta SDM, Karthikeyan T, Vinothkanna R (2019) A novel hidden Markov model-based adaptive dynamic time warping (HMDTW) gait analysis for identifying physically challenged persons. Soft Comput 23:8359–8366. https://doi.org/10.1007/s00500-019-04108-x

    Article  Google Scholar 

  2. Achanta SDM, Karthikeyan T, Vinothkanna R (2020) A wireless IOT system towards gait detection technique using FSR sensor and wearable IOT devices. Int J Intell Unmanned Syst 8(1):43–54. https://doi.org/10.1108/IJIUS-01-2019-0005

    Article  Google Scholar 

  3. Anwar S, Khan S, Barnes N (2020) A deep journey into super-resolution: a survey. ACM Comput Surv 53:1–34. https://doi.org/10.1145/3390462

    Article  Google Scholar 

  4. Choi Y, Kim N, Hwang S, Kweon IS (2016) Thermal image enhancement using convolutional neural network. In: IEEE/RSJ ınternational conference on ıntelligent robots and systems (IROS), Oct 2016. pp 223–230. https://doi.org/10.1109/IROS.2016.7759059

  5. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. 1800–1807. https://doi.org/10.1109/CVPR.2017.195

  6. Chudasama V et al (2020) TherISuRNet—a computationally efficient thermal image super-resolution network. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), Seattle, WA. pp 388–397. https://doi.org/10.1109/CVPRW50498.2020.00051

  7. Dong C, Loy CC, He K, Tan X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38:295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  8. Dong W, Fu F, Shi G, Cao X, Wu J, Li G, Li X (2016) Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans Image Process 25(5):2337–2352. https://doi.org/10.1109/TIP.2016.2542360

    Article  MathSciNet  MATH  Google Scholar 

  9. Dosovitskiy A, Brox T (2016) Generating images with perceptual similarity metrics based on deep networks. In: Advances in neural information processing systems (NIPS). pp 658–666

  10. Fan et al (2018) Dim infrared image enhancement based on convolutional neural network. Neurocomputing 272:396–404. https://doi.org/10.1016/j.neucom.2017.07.017

    Article  Google Scholar 

  11. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. In: IEEE 12th ınternational conference on computer vision. pp 349–356. https://doi.org/10.1109/ICCV.2009.5459271

  12. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. In: Advances in neural ınformation processing systems (NIPS). pp 2672–2680. https://doi.org/10.1145/3422622

  13. Gu Y et al (2020) MedSRGAN: medical images super-resolution using generative adversarial networks. Multimed Tools Appl 79:21815–21840. https://doi.org/10.1007/s11042-020-08980-w

    Article  Google Scholar 

  14. Guei A, Akhloufi M (2018) Deep learning enhancement of infrared face images using generative adversarial networks. Appl Opt 57(18):98. https://doi.org/10.1364/AO.57.000D98

    Article  Google Scholar 

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  16. He Z, Tang S, Yang J, Cao Y, Yang MY, Cao Y (2019) Cascaded deep networks with multiple receptive fields for infrared image super-resolution. IEEE Trans Circuits Syst Video Technol 29(8):2310–2322. https://doi.org/10.1109/TCSVT.2018.2864777

    Article  Google Scholar 

  17. Hwang S, Park J, Kim N, Choi Y, Kweon IS (2015) Multispectral pedestrian detection: benchmark dataset and baselines. In: Proceedings of IEEE conference on Computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2015.7298706

  18. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of The 32nd ınternational conference on machine learning (ICML). pp 448–456

  19. Javaid H, Babar TK, Rasool A, Saghir RU (2013) Video colour variation detection and motion magnification to observe subtle changes. MSc Thesis, Blekinge Institute of Technology, Faisalabad

  20. Johnson J, Alahi A, Li F (2016) Perceptual losses for real-time style transfer and super resolution. In: European conference on computer vision (ECCV). Springer, pp 694–711. https://doi.org/10.1007/978-3-319-46475-6_43

  21. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR). pp 1637–1645. https://doi.org/10.1109/CVPR.2016.181

  22. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: IEEE CVPR. pp 1646–1654. https://doi.org/10.1109/CVPR.2016.182

  23. Lai Z, Deng H (2018) Medical image classification based on deep features extracted by deep model and statistic feature fusion with multilayer perceptron. Comput Intell Neurosci. https://doi.org/10.1155/2018/2061516

    Article  Google Scholar 

  24. Ledig C et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu. pp 105–114. https://doi.org/10.1109/CVPR.2017.19

  25. Lee K, Lee J, Lee J, Hwang S, Lee S (2017) Brightness-based convolutional neural network for thermal image enhancement. IEEE Access 5:26867–26879. https://doi.org/10.1109/ACCESS.2017.2769687

    Article  Google Scholar 

  26. Li J, Fang F, Mei K, Zhang G (2018) Multi-scale residual network for image super-resolution. In: Proceedings of the European conference on computer vision (ECCV). pp 517–532. https://doi.org/10.1145/3390462

  27. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolutaion. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW). pp 1132–1140. https://doi.org/10.1109/CVPRW.2017.151

  28. Liu S et al (2019) Infrared image super resolution using gan with infrared image prior. In: IEEE 4th international conference on signal and ımage processing (ICSIP). pp 1004–1009. https://doi.org/10.1109/SIPROCESS.2019.8868566

  29. Loussaief S, Abdelkrim A (2018) Machine learning framework for image classification. Adv Sci Technol Eng Syst J. https://doi.org/10.25046/aj030101

    Article  Google Scholar 

  30. Mandanici E, Tavasci L, Corsini FF, Gandolfi S (2019) A multi-image super-resolution algorithm applied to thermal imagery. Appl Geomat 11(3):215–228. https://doi.org/10.1007/s12518-019-00253-y

    Article  Google Scholar 

  31. Miranda E, Aryuni M, Irwansyah E (2016) A survey of medical image classification techniques. In: International conference on ınformation management and technology. pp 56–61. https://doi.org/10.1109/ICIMTech.2016.7930302

  32. Nguyen K, Fookes C, Sridharan S, Denman S (2013) Feature-domain super-resolution for iris recognition. Comput Vis Image Underst 117(10):1526–1535. https://doi.org/10.1016/j.cviu.2013.06.010

    Article  Google Scholar 

  33. Ornek AH, Ceylan M, Ervural S (2019) Health status detection of neonates using infrared thermography and deep convolutional neural networks. Infrared Phys Technol 103:103044. https://doi.org/10.1016/j.infrared.2019.103044

    Article  Google Scholar 

  34. Park SJ, Son H, Cho S, Hong KS, Lee S (2018) Srfeat: Single image super-resolution with feature discrimination. In: Proceedings of the European conference on computer vision (ECCV). pp 439–455. https://doi.org/10.1007/978-3-030-01270-0_27

  35. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint. https://arxiv.org/abs/1511.06434.

  36. Rivadeneira RE, Suarez PL, Sappa A, Vintimilla BX (2019) Thermal image super resolution through deep convolutional neural network. In: International conference on image analysis and recognition. pp 417–426

  37. Rivadeneira R, Sappa A, Vintimilla B (2020) Thermal image super-resolution: a novel architecture and dataset. In: 15th international conference on computer vision theory and applications. pp 111–119. https://doi.org/10.5220/0009173601110119

  38. Savasci D, Ceylan M, Ornek AH, Konak M, Soylu H (2020) Heart disease detection from neonatal infrared thermograms using multiresolution features and data augmentation. Int J Intell Syst Appl Eng 8(1):28–36. https://doi.org/10.18201/ijisae.2020158886

    Article  Google Scholar 

  39. Senalp FM, Ceylan M (2020) Enhancement of low resolution thermal face image resolution using deep learning. Eur J Sci Technol. https://doi.org/10.31590/ejosat.802174

    Article  Google Scholar 

  40. Shi W et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE conference on computer vision and pattern recognition (CVPR). pp 1874–1883. https://doi.org/10.1109/CVPR.2016.207

  41. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale ımage recognition. arXiv. https://arxiv.org/abs/1409.1556

  42. Singh K, Gupta A, Kapoor R (2015) Fingerprint image super-resolution via ridge orientation-based clustered coupled sparse dictionaries. J Electron Imaging 24(4):043015. https://doi.org/10.1117/1.JEI.24.4.043015

    Article  Google Scholar 

  43. Szegedy C et al (2015) Going deeper with convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  44. Toyran M (2008) Reconstructing super resolution images from low resolution images. MSc Thesis, Institute of Science, Istanbul

  45. Wang M, Chen Z, Wu QMJ, Jian M (2020) Improved face super-resolution generative adversarial networks. Mach Vis Appl 31:22. https://doi.org/10.1007/s00138-020-01073-6

    Article  Google Scholar 

  46. Weixiang Du, Addepalli P, Zhao Y (2019) The spatial resolution enhancement for a thermogram enabled by controlled sub-pixel movements. IEEE Trans Instrum Meas 69(6):3566–3575. https://doi.org/10.1109/TIM.2019.2932175

    Article  Google Scholar 

  47. Xin M, Wang Y (2019) Research on image classification model based on deep convolution neural network. EURASIP J Image Video Process. https://doi.org/10.1186/s13640-019-0417-8

    Article  Google Scholar 

  48. Yue L, Shen H, Li J, Yuan Q, Zhang H, Zhang L (2018) Image super-resolution: the techniques, applications, and future. Signal Process 128:389–408. https://doi.org/10.1016/j.sigpro.2016.05.002

    Article  Google Scholar 

  49. Zhang X, Li C, Meng Q, Liu S, Zhang Y, Wang J (2018) Infrared image super resolution by combining compressive sensing and deep learning. Sensors (Basel) 18(8):2587. https://doi.org/10.3390/s18082587

    Article  Google Scholar 

  50. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV). pp 286–301. https://doi.org/10.1007/978-3-030-01234-2_18

  51. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Computer vision (ICCV), IEEE ınternational conference. https://doi.org/10.1109/ICCV.2017.244

Download references

Acknowledgements

This project is financially supported by the Scientific Research Projects Coordinatorship of Konya Technical University (Project Number: 201102001).

The thermal images used in this study were obtained in project studies supported by the Scientific and Technological Research Council of Turkey (TUBITAK, Project Number: 215E019).

Author information

Authors and Affiliations

  1. Department of Electrical-Electronics Engineering, Konya Technical University, Konya, Turkey

    Fatih Mehmet Senalp & Murat Ceylan

Authors
  1. Fatih Mehmet Senalp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Murat Ceylan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatih Mehmet Senalp.

Ethics declarations

Ethical approval

Approval was obtained from the Ethics Committee of Non-Interventional Clinical Research in Selcuk University, Faculty of Medicine (Number: 2015/16 – Date: 06.01.2015).

Conflict of interest

Dr. Ceylan declares that he has no conflict of interest. Mr. Senalp declares that he has no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Senalp, F.M., Ceylan, M. Effects of the deep learning-based super-resolution method on thermal image classification applications. Multimed Tools Appl 81, 9313–9330 (2022). https://doi.org/10.1007/s11042-021-11436-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11436-4

Keywords

Search

Navigation