Skip to main content
SpringerLink
Log in

A multi-focus image fusion method based on attention mechanism and supervised learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Multi-focus image fusion is always a difficult problem in digital image processing. To achieve efficient integration, we propose a new end-to-end network. This network uses the residual atrous spatial pyramid pooling module to extract multi-level features from the space of different scales and share parameters to ensure the consistency and correspondence of features. We also introduced a disparities attention module for the network which allows for information retention. These two parts can make our method overcome the difficulties of target edge artifacts, small range blur, poor detail capture, and so on. In addition, in order to improve the semantic ambiguity easily caused by unsupervised learning, we also proposed a new multi-focus image fusion dataset with groundtruth for supervised learning. We performed sufficient experiments, and the results show that the network can quickly capture the corresponding features of multi-focus images, and improve the fusion performance with less computation and lower storage cost. Compared with the existing nine fusion methods, our network is superior to other methods in subjective visual evaluation and objective evaluation, reaching a higher level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Li S, Kang X, Fang L, Hu J, Yin H (2017) Pixel-level image fusion:, A survey of the state of the art. Info Fus 33:100–112

    Article  Google Scholar 

  2. Liu Y, Chen X, Peng H, Wang Z (2017) Multi-focus image fusion with a deep convolutional neural network. Info Fus 36:191–207

    Article  Google Scholar 

  3. Yang B, Zhong J, Li Y, Chen Z (2017) Multi-focus image fusion and super-resolution with convolutional neural network. International Journal of Wavelets, Multiresolution and Information Processing 15 (04):136–146

    MathSciNet  MATH  Google Scholar 

  4. Wang L, Wang Y, Liang Z, Lin Z, Yang J, An W, Guo Y (2019) Learning parallax attention for stereo image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 12250–12259

  5. Nencini F, Garzelli A, Baronti S, Alparone L (2007) Remote sensing image fusion using the curvelet transform. Info Fus 8(2):143–156

    Article  Google Scholar 

  6. Chai Y, Li H, Zhang X (2012) Multifocus image fusion based on features contrast of multiscale products in nonsubsampled contourlet transform domain. Optik 123(7):569–581

    Article  Google Scholar 

  7. Sahu A, Bhateja V, Krishn A (2014) Medical image fusion with Laplacian pyramids. In: 2014 International conference on medical imaging, m-health and emerging communication systems (MedCom), pp 448–453

  8. Joshi K, Kirola M, Chaudhary S, Diwakar M, Joshi NK (2019) Multi-Focus Image fusion using discrete wavelet transform method international conference on advances in engineering science management and technology (ICAESMT)

  9. Li S, Kwok JT, Wang Y (2001) Combination of images with diverse focuses using the spatial frequency. Info Fus 2(3):169–176

    Article  Google Scholar 

  10. Aslantas V, Kurban R (2010) Fusion of multi-focus images using differential evolution algorithm. Expert Syst Appl 37(12):8861–8870

    Article  Google Scholar 

  11. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense SIFT. Info Fus 23:139–155

    Article  Google Scholar 

  12. Nejati M, Samavi S, Shirani S (2015) Multi-focus image fusion using dictionary-based sparse representation. Info Fus 25:72–84

    Article  Google Scholar 

  13. Li H, Wu XJ, Durrani T (2018) Multi-focus noisy image fusion using low-rank representation. arXiv:1804.09325

  14. Qiu X, Li M, Zhang L, Yuan X (2019) Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication 72:35–46

    Google Scholar 

  15. Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. Proceedings of the IEEE international conference on computer vision, pp 2794–2802

  16. Ma B, Ban X, Huang H, Zhu Y (2019) SESF-fuse: An unsupervised deep model for multi-focus image fusion. arXiv preprint arXiv:1908.01703

  17. Chen Q, Wang Z, Chai Y (2020) A multi focus image fusion method based on improved VGG network. Appl Opt 041(003):500–507

    Article  Google Scholar 

  18. Tang H, Xiao B, Li W, Wang G (2018) Pixel convolutional neural network for multi-focus image fusion. Inf Sci 433:125–141

    Article  MathSciNet  Google Scholar 

  19. Guo X, Nie R, Cao J, Zhou D, Qian W (2018) Fully convolutional network-based multifocus image fusion. Neural Comp 30(7):1775–1800

    Article  MathSciNet  Google Scholar 

  20. Lai R, Li Y, Guan J, Xiong A (2019) Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access 7:114385–114399

    Article  Google Scholar 

  21. Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020) IFCNN: A general image fusion framework based on convolutional neural network. Info Fus 54:99–118

    Article  Google Scholar 

  22. Ma H, Liao Q, Zhang J, Liu S, Xie JH (2019) An α-Matte Boundary Defocus Model Based Cascaded Network for Multi-focus Image Fusion. arXiv preprint arXiv:1910.13136

  23. Gregor K, Danihelka I, Graves A, Rezende DJ, Wierstra D (2015) Draw:, A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623

  24. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification, proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

  25. Zhang H, Goodfellow I, Metaxas D, Odena A (2019) Self-attention generative adversarial networks, international conference on machine learning, pp 7354–7363

  26. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  27. Everingham M, Winn J (2011) The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8

  28. Nejati M, Samavi S, Shirani S (2015) Multi-focus image fusion using dictionary-based sparse representation. Info Fus 25:72–84

    Article  Google Scholar 

  29. Savić S, Babić Z (2012) Multifocus image fusion based on empirical mode decomposition, 19th IEEE International conference on systems signals and image processing (IWSSIP)

  30. Xu H, Fan F, Zhang H, Le Z, Huang J (2020) A deep model for Multi-Focus image fusion based on gradients and connected regions. IEEE Access 8:26316–26327

  31. Ma B, Yin X, Wu D, Ban X, Huang H (2020) Gradient aware cascade network for multi-focus image fusion. arXiv preprint arXiv:2010.08751

  32. Tan W, Zhou H, Rong S, Qian K, Yu Y (2018) Fusion of multi-focus images via a Gaussian curvature filter and synthetic focusing degree criterion. App Opt 57(35):10092–10101

    Article  Google Scholar 

  33. Ma J, Zhou Z, Wang B, Miao L, Zong H (2019) Multi-focus image fusion using boosted random walks-based algorithm with two-scale focus maps. Neurocomputing 335:9–20

    Article  Google Scholar 

  34. Bai X, Zhang Y, Zhou F, Xue B (2015) Quadtree-based multi-focus image fusion using a weighted focus-measure. Info Fus 22:105–118

    Article  Google Scholar 

  35. Liu Z, Blasch E, Xue Z, Zhao J, Laganiere R, Wu W (2011) Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE transactions on pattern analysis and machine intelligence 34(1):94–109

    Article  Google Scholar 

  36. Zheng Y, Essock EA, Hansen BC, Haun AM (2007) A new metric based on extended spatial frequency and its application to DWT based fusion algorithms. Info Fus 8(2):177–192

    Article  Google Scholar 

  37. Yang C, Zhang JQ, Wang XR, Liu X (2008) A novel similarity based quality metric for image fusion. Info Fus 9(2):156–160

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Natural Science Foundation of China (61772319, 61773244, 61976125, 61976124), Shandong Natural Science Foundation of China (ZR2017MF049) and Yantai Key Research and Development Plan (2019XDHZ081).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong Technology and Business University, Yantai, 264005, China

    Limai Jiang & Jinjiang Li

  2. Co-innovation Center of Shandong Colleges and Universities Future Intelligent Computing, Yantai, 264005, China

    Hui Fan

Authors
  1. Limai Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinjiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinjiang Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, L., Fan, H. & Li, J. A multi-focus image fusion method based on attention mechanism and supervised learning. Appl Intell 52, 339–357 (2022). https://doi.org/10.1007/s10489-021-02358-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02358-7

Keywords

Search

Navigation