Skip to main content
SpringerLink
Log in

Lightweight dynamic attention network for single thermal image super-resolution

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The embedding of attention mechanism in convolutional neural networks (CNN) effectively improves the performance of single image super-resolution (SISR). However, consistent employ of attention modules at distinct depths of the CNN failed conduct the congruous gain, or even degrades performance. In this paper, we propose LDANet, a lightweight SISR network based on dynamic attention mechanism for thermal image. The dynamic attention blocks in LDANet dynamically rescale the attention and non-attention branches according to input features. Specifically, the attention branch composed of pixel- and channel-wise attention blocks to extract the most informative features in pixel domain and channel dimension, respectively. While the no-attention branch consisting of single convolutional layer for extracting features that are ignored by the attention branch. Innovatively, we adaptively and averagely weight the average pooled and standard deviation pooled features within the channel attention block to fully take advantage of the pooled features. Quantitative and qualitative experiments on three thermal image testing datasets with \(\times \)2, \(\times \)3 and \(\times \)4 scale factors and plentiful scenes show that, compared with SISR models of similar size scope, the proposed LDANet accomplishes superior high-resolution thermal image reconstruction performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available on request from the corresponding author.

References

  1. Chudasama, V., Patel, H., Prajapati, K., Upla, K.P., Ramachandra, R., Raja, K., Busch, C.: Therisurnet-a computationally efficient thermal image super-resolution network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 86–87 (2020)

  2. Zhang, H., Hu, Y., Yan, M., Ma, B.: Thermal image super-resolution via multi-path residual attention network. Signal Image Video, 1–9 (2022)

  3. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  4. Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. 43(10), 3365–3387 (2020)

    Article  Google Scholar 

  5. Prajapati, K., Chudasama, V., Patel, H., Sarvaiya, A., Upla, K.P., Raja, K., Ramachandra, R., Busch, C.: Channel split convolutional neural network (chasnet) for thermal image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4368–4377 (2021)

  6. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  7. Hu, Y., Li, J., Huang, Y., Gao, X.: Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3911–3927 (2019)

    Article  Google Scholar 

  8. Kim, J.-H., Choi, J.-H., Cheon, M., Lee, J.-S.: Ram: residual attention module for single image super-resolution. 2(1), 2. arXiv:1811.12043 (2018)

  9. Zhao, H., Kong, X., He, J., Qiao, Y., Dong, C.: Efficient image super-resolution using pixel attention. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pp. 56–72. Springer (2020)

  10. Chen, H., Gu, J., Zhang, Z.: Attention in attention network for image super-resolution. arXiv:2104.09497 (2021)

  11. Yoo, J., Ahn, N., Sohn, K.-A.: Rethinking data augmentation for image super-resolution: a comprehensive analysis and a new strategy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8375–8384 (2020)

  12. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. 38(2), 295–307 (2015)

    Article  Google Scholar 

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  14. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016)

  15. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017)

  16. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  17. Lan, R., Sun, L., Liu, Z., Lu, H., Pang, C., Luo, X.: Madnet: a fast and lightweight network for single-image super resolution. IEEE Trans. Cybern. 51(3), 1443–1453 (2020)

    Article  Google Scholar 

  18. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)

  19. Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2359–2368 (2020)

  20. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep Laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 624–632 (2017)

  21. Wang, T., Zhang, X., Jiang, R., Zhao, L., Chen, H., Luo, W.: Video deblurring via spatiotemporal pyramid network and adversarial gradient prior. Comput. Vis. Image Underst. 203, 103135 (2021)

    Article  Google Scholar 

  22. Choi, Y., Kim, N., Hwang, S., Kweon, I.S.: Thermal image enhancement using convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 223–230. IEEE (2016)

  23. Rivadeneira, R.E., Suárez, P.L., Sappa, A.D., Vintimilla, B.X.: Thermal image superresolution through deep convolutional neural network. In: International Conference on Image Analysis and Recognition, pp. 417–426. Springer (2019)

  24. Rivadeneira, R., Sappa, A., Vintimilla, B.: Thermal image super-resolution: a novel architecture and dataset. In: 15th International Conference on Computer Vision Theory and Applications (2020)

  25. Wang, T., Zhang, K., Chen, X., Luo, W., Deng, J., Lu, T., Cao, X., Liu, W., Li, H., Zafeiriou, S.: A survey of deep face restoration: denoise, super-resolution, deblur, artifact removal. arXiv:2211.02831 (2022)

  26. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

  27. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  28. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018)

  29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

  30. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

  31. Cai, Y., Lin, J., Hu, X., Wang, H., Yuan, X., Zhang, Y., Timofte, R., Van Gool, L.: Coarse-to-fine sparse transformer for hyperspectral image reconstruction. In: European Conference on Computer Vision, pp. 686–704. Springer (2022)

  32. Wang, T., Zhang, K., Shen, T., Luo, W., Stenger, B., Lu, T.: Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 2654–2662 (2023)

  33. Liang, J., Cao, J., Fan, Y., Zhang, K., Ranjan, R., Li, Y., Timofte, R., Van Gool, L.: VRT: a video restoration transformer. arXiv:2201.12288 (2022)

  34. Liang, J., Fan, Y., Xiang, X., Ranjan, R., Ilg, E., Green, S., Cao, J., Zhang, K., Timofte, R., Van Gool, L.: Recurrent video restoration transformer with guided deformable attention. arXiv:2206.02146 (2022)

  35. Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., Liu, Z.: Dynamic convolution: attention over convolution kernels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030–11039 (2020)

  36. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer (2016)

  37. Jo, Y., Kim, S.J.: Practical single-image super-resolution using look-up table. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 691–700 (2021)

  38. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3147–3155 (2017)

  39. Wang, X., Wang, Q., Zhao, Y., Yan, J., Fan, L., Chen, L.: Lightweight single-image super-resolution network with attentive auxiliary feature learning. In: Proceedings of the Asian Conference on Computer Vision (2020)

  40. Wang, C., Li, Z., Shi, J.: Lightweight image super-resolution with adaptive weighted learning network. arXiv:1904.02358 (2019)

  41. Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2024–2032 (2019)

  42. Xie, C., Zhang, X., Li, L., Meng, H., Zhang, T., Li, T., Zhao, X.: Large kernel distillation network for efficient single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1283–1292 (2023)

  43. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3271 (2018)

  44. Ahn, N., Kang, B., Sohn, K.-A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 252–268 (2018)

Download references

Funding

This work was supported by the project from the National Natural Science Foundation of China under Grant 62073210.

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, 200444, China

    Haikun Zhang & Yueli Hu

Authors
  1. Haikun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yueli Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HZ completed the experiments and evaluations, YH contributed to the conception of the study. All authors reviewed the manuscript.

Corresponding author

Correspondence to Yueli Hu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Hu, Y. Lightweight dynamic attention network for single thermal image super-resolution. SIViP 18, 2195–2206 (2024). https://doi.org/10.1007/s11760-023-02886-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02886-4

Keywords

Search

Navigation