Skip to main content
SpringerLink
Log in

Down-scale simplified non-local attention networks with application to image denoising

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Non-local (NL) attention modules or transformer-based methods have been widely applied in various image processing tasks. However, the computation of the long-range similarity is very expensive, which greatly limits the further application of the NL attention modules. Motivated by the recurrence law of image patches across different scales, we propose an efficient down-scale simplified NL (DSNL) attention module. In our method, the deep feature maps are divided into several feature maps in the coarse scales, which contain the cleaner version of feature patches in the original feature maps. Then the NL attention can be implemented on smaller and clearer feature maps. Numerical experiments ons image denoising demonstrate that the proposed attention module consistently outperforms the original patch-based NL attention modules on both visual quality and GPU time. The classical ResNet which integrates the proposed attention module can product favorable results compared to many state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2

Similar content being viewed by others

Availability of data and materials

The datasets used for the current study are open online. We gratefully acknowledge Dr. Yiqun Mei, Jingzhao Xu, Nam Ik Cho and Yulun Zhang for supplying the codes and visual results of the PANet, the DUMRN, the VDIR and the RDN, respectively. The code and visual results for our DSNL implementation are available at https://pan.baidu.com/s/1k6VKWyeWlU5p9K9apTsCEw?pwd=9r51.

References

  1. Chen, D.Q.: Regularized generalized inverse accelerating linearized alternating minimization algorithm for frame-based poissonian image deblurring. SIAM J. Imag. Sci. 7(2), 716–739 (2014)

    Article  MathSciNet  Google Scholar 

  2. Bahat, Y., Irani, M.: Blind dehazing using internal patch recurrence[C]. In: IEEE international conference on computational photography. (2016)

  3. Kai, Z., Zuo, W., Gu, S., et al.: Learning deep CNN denoiser prior for image restoration[C]. IEEE conference on computer vision and pattern recognition (CVPR). (2017)

  4. Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. on Image Process. 15(12), 3736–3745 (2006)

    Article  MathSciNet  Google Scholar 

  5. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising[C]. In: IEEE computer vision and pattern recognition. (2005)

  6. Dabov, K., Foi, A., Katkovnik, V., et al.: Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  7. Zhang, J., Zhao, D., Gao, W.: Group-based sparse representation for image restoration[J]. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 23(8), 3336–3351 (2014)

    Article  MathSciNet  Google Scholar 

  8. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition[C]. In ICLR, (2015)

  9. He, K., Zhang, X., Ren, S., et al.: Identity mappings in deep residual networks[C], pp. 630–645. Cham, European conference on computer vision. Springer (2016)

  10. Luo, W., Li, Y., Urtasun, R., et al.: Understanding the effective receptive field in deep convolutional neural networks[J]. Adv. Neural Inf. Process. Syst., 29 (2016)

  11. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need[J]. Adv. Neural Inf. Process. Syst., 30 (2017)

  12. Wang, X., Girshick, R., Gupta, A., et al.: Non-local neural networks. In: CVPR (2018)

  13. Liang, J., Cao, J., Sun, G., et al.: Swinir: image restoration using swin transformer[c]. In: Proceedings of the IEEE/CVF international conference on computer vision: 1833–1844 (2021)

  14. Xia, Z., Pan, X., Song, S., et al.: Vision transformer with deformable attention[C]. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition: 4794–4803 (2022)

  15. Hassani, A., Walton, S., Li, J., et al.: Neighborhood attention transformer[J]. arXiv preprint arXiv:2204.07143, (2022)

  16. Zontak, M., Irani, M.: Internal statistics of a single natural image[C]. In: IEEE conference on computer vision and pattern recognition (CVPR). (2011)

  17. Zontak, M., Mosseri, I., Irani, M.: Separating signal from noise using patch recurrence across scales[C]. In: IEEE conference on computer vision and pattern recognition (CVPR). (2013)

  18. Dong, C., Loy, C.C., He, K., et al.: Image super-resolution using deep convolutional networks[J]. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  19. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network[C]. In: European conference on computer vision. Springer, Cham: 391-407 (2016)

  20. Kai, Z., Zuo, W., Chen, Y., et al.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising[J]. IEEE Trans. Image Process. 26(7), 3142–3155 (2016)

    MathSciNet  Google Scholar 

  21. Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN based image denoising[J]. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)

    Article  MathSciNet  Google Scholar 

  22. Lim, B., Son, S., Kim, H., et al.: Enhanced deep residual networks for single image super-resolution[C]. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW). (2017)

  23. Zhang, Y., Tian, Y., Kong, Y., et al.: Residual dense network for image restoration[J]. IEEE Trans. Pattern Anal. Mach. Intell. 43(7), 2480–2495 (2021)

    Article  Google Scholar 

  24. Liu, Z., Mao, X., Huang, J., et al.: Stratified attention dense network for image super-resolution[J]. SIViP 16, 715–722 (2022)

    Article  Google Scholar 

  25. Chen, Y., Xia, R., Yang, K., et al.: MFFN: image super-resolution via multi-level features fusion network. Vis. Comput. (2023). https://doi.org/10.1007/s00371-023-02795-0

    Article  Google Scholar 

  26. Chen, Y., Xia, R., Zou, K., Yang, K.: FFTI: image inpainting algorithm via features fusion and two-steps inpainting. J. Vis. Commun. Image Represent. 91, 103776 (2023)

    Article  Google Scholar 

  27. Chen, Y., Xia, R., Yang, K., Zou, K.: DARGS: Image inpainting algorithm via deep attention residuals group and semantics. J. King Saud Univ. Comput. Inf. Sci. 35(6), 101567 (2023)

    Google Scholar 

  28. Chen, Y., Xia, R., Yang, K., et al.: DGCA: high resolution image inpainting via DR-GAN and contextual attention. Multimed Tools Appl (2023). https://doi.org/10.1007/s11042-023-15313-0

    Article  Google Scholar 

  29. Chen, Y., Xia, R., Zou, K., et al.: RNON: image inpainting via repair network and optimization network. Int. J. Mach. Learn. Cyber. (2023). https://doi.org/10.1007/s13042-023-01811-y

    Article  Google Scholar 

  30. Buades, A., Coll, B., Morel, J.M.: Non-local means denoising[J]. Image Process. Line 1, 208–212 (2011)

    Article  Google Scholar 

  31. Zhu, Z., Xu, M., Bai, S., et al.: Asymmetric non-local neural networks for semantic segmentation[C]. Proceedings of the IEEE/CVF international conference on computer vision: 593–602 (2019)

  32. Fu, J., Liu, J., Tian, H., et al.: Dual attention network for scene segmentation[C]. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR). (2020)

  33. Ding, L., Wen, B., Fan, Y., et al.: Non-local recurrent network for image restoration[C]. In: NeurIPS (2018)

  34. Dai, T., Cai, J., Zhang, Y., et al.: Second-order attention network for single image super-resolution[C]. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR). (2019)

  35. Zhang, Y., Li, K., Li, K., et al.: Residual non-local attention networks for image restoration[C]. In: ICLR. (2019)

  36. Mei Y, Fan Y, Zhou Y, et al. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining[C]. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR). (2020)

  37. Mei, Y., Fan, Y., Zhang, Y., et al.: Pyramid attention networks for image restoration. In: Computer vision and pattern recognition (CVPR). in press. (2022)

  38. Hu, J., Shen, L., Albanie, S., et al.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020)

    Article  Google Scholar 

  39. Sajjadi, M., Scholkopf, B., Hirsch, M.: EnhanceNet: single image super-resolution through automated texture synthesis[C]. In: IEEE international conference on computer vision: 4491-4500. (2017)

  40. Timofte, R., Gu, S., Wu, J., Ntire, et al.: Challenge on single image super-resolution: methods and results[C]. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018, 852–863 (2018)

  41. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars[C]. In: Proceedings of the IEEE conference on computer vision and pattern recognition: 5197-5206. (2015)

  42. Martin, D., Fowlkes, C., Tal, D., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]. In: Proceedings eighth IEEE international conference on computer vision (ICCV). 2, 416–423 (2001)

  43. Dabov, K., Foi, A., Katkovnik, V., et al.: Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space[C]. In: 2007 IEEE international conference on image processing. (2007)

  44. Tian, C., Xu, Y., Zuo, W., et al.: Designing and training of a dual CNN for image denoising[J]. Knowl.-Based Syst. 226, 106949 (2021)

    Article  Google Scholar 

  45. Soh, J.W., Cho, N.I.: Variational deep image restoration[J]. IEEE Trans. Image Process. 31, 4363–4376 (2022)

    Article  Google Scholar 

  46. Xu, J., Yuan, M., Yan, D.M., et al.: Deep unfolding multi-scale regularizer network for image denoising[J]. Comput. Vis. Media 9(2), 335–350 (2023)

    Article  Google Scholar 

Download references

Funding

The work was supported in part by the National Natural Science Foundation of China under Grant 62076247 and the Supporting Program for Excellent Talents in Army Medical University.

Author information

Authors and Affiliations

  1. Department of Mathematics, Basic Medicine College, Army Medical University of PLA, Chongqing, Chongqing, 400038, People’s Republic of China

    Dai-Qiang Chen

Authors
  1. Dai-Qiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dai-Qiang Chen.

Ethics declarations

Conflict of interest

We declared no potential competing interests with respect to the research, author-ship, and publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, DQ. Down-scale simplified non-local attention networks with application to image denoising. SIViP 18, 47–54 (2024). https://doi.org/10.1007/s11760-023-02708-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02708-7

Keywords

Search

Navigation