Skip to main content

Thermal Image Super-Resolution: A Novel Unsupervised Approach

  • 383 Accesses

Part of the Communications in Computer and Information Science book series (CCIS,volume 1474)

Abstract

This paper proposes the use of a CycleGAN architecture for thermal image super-resolution under a transfer domain strategy, where middle-resolution images from one camera are transferred to a higher resolution domain of another camera. The proposed approach is trained with a large dataset acquired using three thermal cameras at different resolutions. An unsupervised learning process is followed to train the architecture. Additional loss function is proposed trying to improve results from the state of the art approaches. Following the first thermal image super-resolution challenge (PBVS-CVPR2020) evaluations are performed. A comparison with previous works is presented showing the proposed approach reaches the best results.

Keywords

  • Thermal image super-resolution
  • Thermal images
  • Datasets
  • Challenge
  • Unpair thermal images

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-94893-1_23
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-94893-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Notes

  1. 1.

    FREE FLIR Thermal Dataset for Algorithm Training https://www.flir.in/oem/adas/adas-dataset-form/.

References

  1. Arbel, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)

    CrossRef  Google Scholar 

  2. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

    Google Scholar 

  3. Chang, H., Lu, J., Yu, F., Finkelstein, A.: PairedCycleGAN: asymmetric style transfer for applying and removing makeup. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 40–48 (2018)

    Google Scholar 

  4. Chen, Y.S., Wang, Y.C., Kao, M.H., Chuang, Y.Y.: Deep photo enhancer: unpaired learning for image enhancement from photographs with GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6306–6314 (2018)

    Google Scholar 

  5. Choi, Y., Kim, N., Hwang, S., Kweon, I.S.: Thermal image enhancement using convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 223–230. IEEE (2016)

    Google Scholar 

  6. Chudasama, V., et al.: TheriSuRNet-a computationally efficient thermal image super-resolution network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 86–87 (2020)

    Google Scholar 

  7. Davis, J.W., Keck, M.A.: A two-stage template approach to person detection in thermal imagery. In: 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION 2005), vol. 1, pp. 364–369. IEEE (2005)

    Google Scholar 

  8. Ding, M., Zhang, X., Chen, W.H., Wei, L., Cao, Y.F.: Thermal infrared pedestrian tracking via fusion of features in driving assistance system of intelligent vehicles. Proc. Inst. Mech. Eng. Part G: J. Aerosp. Eng. 233(16), 6089–6103 (2019)

    CrossRef  Google Scholar 

  9. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    CrossRef  Google Scholar 

  10. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    CrossRef  Google Scholar 

  11. Gade, R., Moeslund, T.B.: Thermal cameras and applications: a survey. Mach. Vis. Appl. 25(1), 245–262 (2013). https://doi.org/10.1007/s00138-013-0570-5

    CrossRef  Google Scholar 

  12. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 142–158 (2015)

    CrossRef  Google Scholar 

  13. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  14. Herrmann, C., Ruf, M., Beyerer, J.: CNN-based thermal infrared person detection by domain adaptation. In: Autonomous Systems: Sensors, Vehicles, Security, and the Internet of Everything, vol. 10643, p. 1064308. International Society for Optics and Photonics (2018)

    Google Scholar 

  15. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

    Google Scholar 

  16. Hwang, S., Park, J., Kim, N., Choi, Y., So Kweon, I.: Multispectral pedestrian detection: benchmark dataset and baseline. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1037–1045 (2015)

    Google Scholar 

  17. Kansal, P., Nathan, S.: A multi-level supervision model: a novel approach for thermal image super resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 94–95 (2020)

    Google Scholar 

  18. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

    Google Scholar 

  19. Kittler, J.: On the accuracy of the Sobel edge detector. Image Vis. Comput. 1(1), 37–42 (1983)

    CrossRef  Google Scholar 

  20. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  21. Mandanici, E., Tavasci, L., Corsini, F., Gandolfi, S.: A multi-image super-resolution algorithm applied to thermal imagery. Appl. Geomat. 11(3), 215–228 (2019). https://doi.org/10.1007/s12518-019-00253-y

    CrossRef  Google Scholar 

  22. Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, Vancouver (2001)

    Google Scholar 

  23. Matsui, Y., et al.: Sketch-based manga retrieval using manga109 dataset. Multimed. Tools Appl. 76(20), 21811–21838 (2016). https://doi.org/10.1007/s11042-016-4020-z

    CrossRef  Google Scholar 

  24. Mehri, A., Sappa, A.D.: Colorizing near infrared images through a cyclic adversarial approach of unpaired samples. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  25. Mudunuri, S.P., Biswas, S.: Low resolution face recognition across variations in pose and illumination. IEEE Trans. Pattern Anal. Mach. Intell. 38(5), 1034–1040 (2015)

    CrossRef  Google Scholar 

  26. Olmeda, D., Premebida, C., Nunes, U., Armingol, J.M., de la Escalera, A.: Pedestrian detection in far infrared images. Integr. Comput.-Aided Eng. 20(4), 347–360 (2013)

    CrossRef  Google Scholar 

  27. Park, S.J., Son, H., Cho, S., Hong, K.S., Lee, S.: SRFeat: single image super-resolution with feature discrimination. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 439–455 (2018)

    Google Scholar 

  28. Rivadeneira, R.E., et al.: Thermal image super-resolution challenge-PBVS 2020. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 96–97 (2020)

    Google Scholar 

  29. Rivadeneira, R.E., Sappa, A.D., Vintimilla, B.X.: Thermal image super-resolution: a novel architecture and dataset. In: VISIGRAPP (4: VISAPP), pp. 111–119 (2020)

    Google Scholar 

  30. Rivadeneira, R.E., Suárez, P.L., Sappa, A.D., Vintimilla, B.X.: Thermal image superresolution through deep convolutional neural network. In: Karray, F., Campilho, A., Yu, A. (eds.) ICIAR 2019. LNCS, vol. 11663, pp. 417–426. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27272-2_37

    CrossRef  Google Scholar 

  31. Shamsolmoali, P., Zareapoor, M., Jain, D.K., Jain, V.K., Yang, J.: Deep convolution network for surveillance records super-resolution. Multimed. Tools Appl. 78(17), 23815–23829 (2018). https://doi.org/10.1007/s11042-018-5915-7

    CrossRef  Google Scholar 

  32. Shi, W., Ledig, C., Wang, Z., Theis, L., Huszar, F.: Super resolution using a generative adversarial network, 15 March 2018. US Patent App. 15/706,428

    Google Scholar 

  33. Suarez, P.L., Sappa, A.D., Vintimilla, B.X., Hammoud, R.I.: Image vegetation index through a cycle generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  34. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: NTIRE 2017 challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

    Google Scholar 

  35. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    CrossRef  Google Scholar 

  36. Wu, Z., Fuller, N., Theriault, D., Betke, M.: A thermal infrared video benchmark for visual analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 201–208 (2014)

    Google Scholar 

  37. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., Chenin, P., Cohen, A., Gout, C., Lyche, T., Mazure, M.-L., Schumaker, L. (eds.) Curves and Surfaces 2010. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27413-8_47

    CrossRef  Google Scholar 

  38. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017)

    Google Scholar 

  39. Zhang, L., Zhang, H., Shen, H., Li, P.: A super-resolution reconstruction algorithm for surveillance images. Signal Process. 90(3), 848–859 (2010)

    CrossRef  Google Scholar 

  40. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)

    Google Scholar 

  41. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgements

This work has been partially supported by the Spanish Government under Project TIN2017-89723-P; and the “CERCA Programme/Generalitat de Catalunya”. The first author has been supported by Ecuador government under a SENESCYT scholarship contract.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Rafael E. Rivadeneira .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Rivadeneira, R.E., Sappa, A.D., Vintimilla, B.X. (2022). Thermal Image Super-Resolution: A Novel Unsupervised Approach. In: , et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2020. Communications in Computer and Information Science, vol 1474. Springer, Cham. https://doi.org/10.1007/978-3-030-94893-1_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-94893-1_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-94892-4

  • Online ISBN: 978-3-030-94893-1

  • eBook Packages: Computer ScienceComputer Science (R0)