Skip to main content
SpringerLink
Log in

Thermal image super-resolution via multi-path residual attention network

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Convolutional Neural Networks (CNN)-based Single-Image Super-Resolution (SISR) methods for RGB images have flourished rapidly. However, thermal images SR methods based on CNN are rarely studied. The performance of existing deep SR methods is limited by the narrow receptive field of single small convolution kernel (e.g., \(3\times 3\)). In this paper, we propose a thermal image SISR deep network MPRANet, combining multi-path residual and attention blocks. Specifically, an innovative design multi-path residual block, constructed by parallel depth-wise separable convolution paths composed of convolution kernels of different sizes, is used to extract local minute and global large features, effectively enhancing the capacity of MPRANet. Meanwhile, the attention block is formed by cascading channel attention and spatial attention modules to re-scale features in the channel and spatial dimensions sequentially. A Mixture of Data Augmentation (MoDA) strategy for meliorating MPRANet performance without increasing computational burden is proposed. MoDA makes full use of multiple pixel-domain data augmentation methods to raise the generalization of MPRANet. Qualitative and quantitative experiments on three test datasets show that the proposed MPRANet has obvious advantages over state-of-the-art thermal and RGB image SR methods for the preservation of details such as edges and textures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Availability of data and materials

The data that support the findings of this study are available on request from the corresponding author.

Notes

  1. https://www.iray-dataset.com/apply/E_Super_resolution.html/.

  2. https://www.flir.in/oem/adas/adas-dataset-form/.

References

  1. Chudasama, V., Patel, H., Prajapati, K., Upla, K.P., Ramachandra, R., Raja, K., Busch, C.: Therisurnet-a computationally efficient thermal image super-resolution network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 86–87 (2020)

  2. Rostami, M., Oussalah, M., Farrahi, V.: A novel time-aware food recommender-system based on deep learning and graph clustering. IEEE Access 10, 52508–52524 (2022)

    Article  Google Scholar 

  3. Azadifar, S., Rostami, M., Berahmand, K., Moradi, P., Oussalah, M.: Graph-based relevancy-redundancy gene selection method for cancer diagnosis. Comput. Biol. Med. 147, 105766 (2022)

    Article  Google Scholar 

  4. Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. 43(10), 3365–3387 (2020)

    Article  Google Scholar 

  5. Choi, Y., Kim, N., Hwang, S., Kweon, I.S.: Thermal image enhancement using convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 223–230. IEEE (2016)

  6. Rivadeneira, R.E., Suárez, P.L., Sappa, A.D., Vintimilla, B.X.: Thermal image superresolution through deep convolutional neural network. In: International Conference on Image Analysis and Recognition, pp. 417–426. Springer (2019)

  7. Zhou, L., Cai, H., Gu, J., Li, Z., Liu, Y., Chen, X., Qiao, Y., Dong, C.: Efficient image super-resolution using vast-receptive-field attention. arXiv preprint arXiv:2210.05960 (2022)

  8. Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

  9. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  10. Timofte, R., Rothe, R., Van Gool, L.: Seven ways to improve example-based single image super resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1865–1873 (2016)

  11. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv:1710.09412 (2017)

  12. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032 (2019)

  13. Yoo, J., Ahn, N., Sohn, K.-A.: Rethinking data augmentation for image super-resolution: a comprehensive analysis and a new strategy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8375–8384 (2020)

  14. DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv:1708.04552 (2017)

  15. Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., Bengio, Y.: Manifold mixup: better representations by interpolating hidden states. In: International Conference on Machine Learning, pp. 6438–6447. PMLR (2019)

  16. Gastaldi, X.: Shake-shake regularization. arXiv:1705.07485 (2017)

  17. Yamada, Y., Iwamura, M., Akiba, T., Kise, K.: Shakedrop regularization for deep residual learning. IEEE Access 7, 186126–186136 (2019)

    Article  Google Scholar 

  18. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  19. Ghiasi, G., Lin, T.-Y., Le, Q.V.: Dropblock: a regularization method for convolutional networks. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

  20. Choe, J., Lee, S., Shim, H.: Attention-based dropout layer for weakly supervised single object localization and semantic segmentation. IEEE Trans. Pattern Anal. 43(12), 4256–4271 (2020)

    Article  Google Scholar 

  21. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE. Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  22. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE. Trans. Pattern Anal. 38(2), 295–307 (2015)

    Article  Google Scholar 

  23. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016)

  24. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017)

  25. Li, J., Fang, F., Mei, K., Zhang, G.: Multi-scale residual network for image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 517–532 (2018)

  26. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)

  27. Rivadeneira, R.E., Sappa, A.D., Vintimilla, B.X., Guo, L., Hou, J., Mehri, A., Ardakani, P.B., Patel, H., Chudasama, V., Prajapati, K., Upla, K.P., Ramachandra, R., Raja, K., Busch, C., Almasri, F., Debeir, O., Nathan, S., Kansal, P., Gutierrez, N., Mojra, B., Beksi, W.J.: Thermal image super-resolution challenge—PBVS 2020. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2020)

  28. Rivadeneira, R.E., Sappa, A.D., Vintimilla, B.X., Nathan, S., Kansal, P., Mehri, A., Ardakani, P.B., Dalal, A., Akula, A., Sharma, D., et al.: Thermal image super-resolution challenge-pbvs 2021. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4359–4367 (2021)

  29. Prajapati, K., Chudasama, V., Patel, H., Sarvaiya, A., Upla, K.P., Raja, K., Ramachandra, R., Busch, C.: Channel split convolutional neural network (CHASNET) for thermal image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4368–4377 (2021)

  30. Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv:1511.07289 (2015)

  31. Fu, B., Dong, Y., Fu, S., Wu, Y., Ren, Y., Thanh, D.: Multistage supervised contrastive learning for hybrid-degraded image restoration. Signal Image Video, pp. 1–9 (2022)

  32. Chen, X., Yang, R., Guo, C.: A lightweight multi-scale residual network for single image super-resolution. Signal Image Video, pp. 1–9 (2022)

  33. Rivadeneira, R., Sappa, A., Vintimilla, B.: Thermal image super-resolution: a novel architecture and dataset. In: 15th International Conference on Computer Vision Theory and Applications (2020)

  34. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer (2016)

  35. Jo, Y., Kim, S.J.: Practical single-image super-resolution using look-up table. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 691–700 (2021)

  36. Martini, M.G., Hewage, C.T., Villarini, B.: Image quality assessment based on edge preservation. Signal Process. Image Commun. 27(8), 875–882 (2012)

Download references

Funding

This work was supported by the project from the National Natural Science Foundation of China under Grant 62073210.

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, 200444, China

    Haikun Zhang, Yueli Hu, Ming Yan & Bin Ma

Authors
  1. Haikun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yueli Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bin Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HZ completed most of the experiments and evaluations, YH contributed to the conception of the study, MY and BM prepared most of the charts.

Corresponding author

Correspondence to Yueli Hu.

Ethics declarations

Competing interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Hu, Y., Yan, M. et al. Thermal image super-resolution via multi-path residual attention network. SIViP 17, 2073–2081 (2023). https://doi.org/10.1007/s11760-022-02421-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02421-x

Keywords

Search

Navigation