Skip to main content
SpringerLink
Log in

Cloud environment-based super resolution application for thermal images using the new approach TSRGAN+ model

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Thermal cameras can be used safely on living things because thermal camera systems provide harmless and contactless imaging. However, thermal cameras are high-cost systems, which limits their widespread usage. Therefore, low-cost thermal cameras facilitate the use of thermal imaging in different areas. Also, these thermal cameras can create blurry thermal images with low detail information. The purpose of this work is to provide an alternative to the high-cost problem and to create a system that can be easily accessed via the internet. Here, it is important to use super resolution techniques on these low quality images. In this study, a new dataset was created using two different thermal cameras. This dataset was obtained in different way from the datasets used in traditional image enhancement implementations. Here, the low-resolution thermal face images are obtained by means of the Flir One Pro© thermal camera, which can be easily integrated into the smartphone. The high-resolution (ground truth) thermal face images are obtained with the Variocam HD© thermal camera, which is an HD (high definition) thermal imaging system. In addition, the TSRGAN+ deep network model is proposed as a new approach for super resolution application on the new dataset. The obtained results were compared with state-of-the-art models using peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) image quality metrics. When the results were evaluated, it was observed that the success performance of the proposed model was superior to other models. Here, considering the PSNR and SSIM values, it was seen that the proposed model achieved approximately 0.5 dB and 6% more successful results than other models, respectively. After these studies, the proposed super resolution model was run in the cloud environment and the system can be easily accessed from anywhere through the developed Android interface software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

The data presented in this study are available on request from the corresponding author.

References

  1. Achanta SDM, Karthikeyan T, Vinothkanna R (2020) A wireless IOT system towards gait detection technique using FSR sensor and wearable IOT devices. Int J Intell Unmanned Syst 8(1):43–54. https://doi.org/10.1108/IJIUS-01-2019-0005

    Article  Google Scholar 

  2. Anwar S, Khan S, Barnes N (2020) A deep journey into super-resolution: a survey. ACM Comput Surv 53:1–34. https://doi.org/10.1145/3390462

    Article  Google Scholar 

  3. Choi Y, Kim N, Hwang S, Kweon IS (2016) Thermal image enhancement using convolutional neural network. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 223–230. https://doi.org/10.1109/IROS.2016.7759059

  4. Chudasama V, Patel H, Prajapati K, Upla K, Ramachandra R, Raja K, Busch C (2020) TherISuRNet- A computationally efficient thermal image super-resolution network. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 388–397. https://doi.org/10.1109/CVPRW50498.2020.00051

  5. Dong C, Loy CC, He K, Tan X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38:295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  6. Dong W, Fu F, Shi G, Cao X, Wu J, Li G, Li X (2016) Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans Image Process 25(5):2337–2352. https://doi.org/10.1109/TIP.2016.2542360

    Article  MathSciNet  MATH  Google Scholar 

  7. Dong C, Loy CC, Tang X (2016) Accelerating the Super-Resolution Convolutional Neural Network. Computer Vision ECCV 2016 Lecture Notes in Comput Sci Springer 9906. https://doi.org/10.1007/978-3-319-46475-6_25

  8. Dosovitskiy A, Brox T (2016) Generating images with perceptual similarity metrics based on deep networks. In advances in neural information processing systems (NIPS). Pp 658–666. https://doi.org/10.48550/arXiv.1602.02644

  9. Fan Z, Bi D, Xiong L, Ma S, He L, Ding W (2018) Dim infrared image enhancement based on convolutional neural network. Neurocomputing 272:396–404. https://doi.org/10.1016/j.neucom.2017.07.017

    Article  Google Scholar 

  10. Fraiwan L, Alkhodari M, Ninan J, Mustafa B, Saleh A, Ghazal M (2017) Diabetic foot ulcer mobile detection system using smartphone thermal camera: a feasibility study. Biomed Eng Online 16(1):117. https://doi.org/10.1186/s12938-017-0408-x

    Article  Google Scholar 

  11. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. IEEE 12th international conference on computer vision. Pp 349-356. https://doi.org/10.1109/ICCV.2009.5459271

  12. Goodfellow I et al (2014) Generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS). pp 2672–2680. https://doi.org/10.1145/3422622

  13. Gu Y, Zeng Z, Chen H, Wei J, Zhang Y, Chen B, Li Y, Qin Y, Xie Q, Jiang Z, Lu Y (2020) MedSRGAN: medical images super-resolution using generative adversarial networks. Multimed Tools Appl 79:21815–21840. https://doi.org/10.1007/s11042-020-08980-w

    Article  Google Scholar 

  14. Guei A, Akhloufi M (2018) Deep learning enhancement of infrared face images using generative adversarial networks. Appl Opt 57(18):98–D107. https://doi.org/10.1364/AO.57.000D98

    Article  Google Scholar 

  15. Javaid H, Babar TK, Rasool A, Saghir RU (2013) Video colour variation detection and motion magnification to observe subtle changes. M.Sc. Thesis, Blekinge Institute of Technology, Faisalabad, Pakistan

    Google Scholar 

  16. Johnson J, Alahi A, Li F (2016) Perceptual losses for real-time style transfer and super resolution. In European Conference on Computer Vision (ECCV) Springer pp 694–711. https://doi.org/10.1007/978-3-319-46475-6_43

  17. Kim J, Lee JK, Lee KM (2016) accurate image super-resolution using very deep convolutional networks. IEEE CVPR pp 1646–1654. https://doi.org/10.1109/CVPR.2016.182

  18. Ledig C et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE conference on computer vision and pattern recognition (CVPR) Honolulu. Pp 105-114. https://doi.org/10.1109/CVPR.2017.19

  19. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolutaion. IEEE conference on computer vision and pattern recognition workshops (CVPRW). Pp 1132–1140. https://doi.org/10.1109/CVPRW.2017.151

  20. Liu S et al (2019) Infrared image super resolution using Gan with infrared image prior. In: IEEE 4th International Conference on Signal and Image Processing (ICSIP), pp 1004–1009. https://doi.org/10.1109/SIPROCESS.2019.8868566

  21. Mandanici E, Tavasci L, Corsini F, Gandolfi S (2019) A multi-image super-resolution algorithm applied to thermal imagery. Appl Geomatics 11(3):215–228. https://doi.org/10.1007/s12518-019-00253-y

    Article  Google Scholar 

  22. Nguyen K, Fookes C, Sridharan S, Sv D (2013) Feature-domain super-resolution for iris recognition. Comput Vis Image Underst 117(10):1526–1535. https://doi.org/10.1016/j.cviu.2013.06.010

    Article  Google Scholar 

  23. Ornek AH, Ceylan M, Ervural S (2019) Health status detection of neonates using infrared thermography and deep convolutional neural networks. Infrared Phys Technol 103:103044. https://doi.org/10.1016/j.infrared.2019.103044

  24. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

  25. Rivadeneira RE, Suarez P, Sappa AD, Vintimilla BX (2019) Thermal image super resolution through deep convolutional neural network. International conference on image analysis and recognition. Pp 417–426

  26. Senalp FM, Ceylan M (2020) Enhancement of low resolution thermal face image resolution using deep learning. European J Sci Tech. Pp 131-135. https://doi.org/10.31590/ejosat.802174

  27. Senalp FM, Ceylan M (2021) Deep learning based super resolution and classification applications for neonatal thermal images. Traitement du signal 38(5):1361–1368. https://doi.org/10.18280/ts.380511

    Article  Google Scholar 

  28. Senalp FM, Ceylan M (2022) Effects of the deep learning-based super-resolution method on thermal image classification applications. Multimed Tools Appl 81:9313–9330. https://doi.org/10.1007/s11042-021-11436-4

    Article  Google Scholar 

  29. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Comput Vis Pattern Recognit arXiv 1409:1556

    Google Scholar 

  30. Singh K, Gupta A, Kapoor R (2015) Fingerprint image super-resolution via ridge orientation-based clustered coupled sparse dictionaries. J Electron Imaging 24(4):043015. https://doi.org/10.1117/1.JEI.24.4.043015

    Article  Google Scholar 

  31. Sun L, Sakaridis C, Liang J, Jiang Q, Yang K, Sun P, Ye Y, Wang K, Van Gool L (2022) MEFNet: multi-scale event fusion network for motion deblurring with cross-modal attention. Computer Vision – ECCV 2022, pp 412–428

  32. Toyran M (2008) Reconstructing super resolution images from low resolution images. M.Sc. Thesis, institute of science, Istanbul

  33. Voronin V, Semenishchev E, Frants V, Agaian S (2018) Smart cloud system for forensic thermal image enhancement using local and global logarithmic transform histogram matching. 2018 IEEE international conference on smart cloud (SmartCloud) New York. Pp 153-157. https://doi.org/10.1109/SmartCloud.2018.00033

  34. Wang M, Chen Z, Wu QMJ, Jian M (2020) Improved face super-resolution generative adversarial networks. Mach Vis Appl 31:22. https://doi.org/10.1007/s00138-020-01073-6

    Article  Google Scholar 

  35. Weixiang D, Addepalli P, Zhao Y (2019) The spatial resolution enhancement for a Thermogram enabled by controlled sub-pixel movements. IEEE Trans Instrum Meas 69(6):3566–3575. https://doi.org/10.1109/TIM.2019.2932175

    Article  Google Scholar 

  36. Yan R, Yang K, Wang K (2021) NLFNet: non-local fusion towards generalized multimodal semantic segmentation across RGB-depth, polarization, and thermal images. 2021 IEEE international conference on robotics and biomimetics (ROBIO), pp 1129-1135. https://doi.org/10.1109/ROBIO54168.2021.9739390

  37. Yue L, Shen H, Li J, Yuan Q, Zhang H, Zhang L (2018) Image super-resolution:the techniques, applications, and future. Signal Process 128:389–408. https://doi.org/10.1016/j.sigpro.2016.05.002

    Article  Google Scholar 

  38. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. Proceedings of the European conference on computer vision. Pp 286–301. https://doi.org/10.1007/978-3-030-01234-2_18

  39. Zhang X, Li C, Meng Q, Liu S, Zhang Y, Wang J (2018) Infrared image super resolution by combining compressive sensing and deep learning. Sensors (Basel) 18(8):2587. https://doi.org/10.3390/s18082587

    Article  Google Scholar 

  40. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. Comput Vis (ICCV) IEEE Int Conf, pp 2242–2251.https://doi.org/10.1109/ICCV.2017.244

Download references

Acknowledgments

This work is supported by the Scientific Research Project Fund of Konya Technical University under the project number 201102001.

This work is supported by the Scientific and Technological Research Council of Turkey (TUBITAK) under the project number 215E019.

Author information

Authors and Affiliations

  1. Department of Electrical Electronics Engineering, Konya Technical University, Konya, Turkey

    Fatih Mehmet Senalp, Batuhan Orhan & Murat Ceylan

Authors
  1. Fatih Mehmet Senalp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Batuhan Orhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Murat Ceylan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatih Mehmet Senalp.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Senalp, F.M., Orhan, B. & Ceylan, M. Cloud environment-based super resolution application for thermal images using the new approach TSRGAN+ model. Multimed Tools Appl 82, 18483–18500 (2023). https://doi.org/10.1007/s11042-022-14169-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14169-0

Keywords

Search

Navigation