Skip to main content
SpringerLink
Log in

MSIMCNN: Multi-scale inception module convolutional neural network for multi-focus image fusion

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The aim of image fusion is to obtain a clear image by combining useful information coming from multiple images. However, the fused image usually has the problem of artifacts and unclear boundary. To address these problems, a deep convolutional neural network based framework for multi-focus image fusion is proposed in this paper, called multi-scale inception module convolutional neural network (MSIMCNN). MSIMCNN converts the entire image into a binary mask to estimate the focus characteristics, and obtains the clear boundary between focus and defocus. First of all, a pair of focus images and the corresponding feature images detected by the Laplace operator are inputted into the network. The Laplace operator can detect the edge and gradient of focus in the image, which can help us accurately reconstruct the focused area in the focus map and distinguish the focus and defocus boundaries. Then, in the feature extraction stage, different scales of convolution kernels are designed to extract the rich and complementary features at different scales of the source images. At the same time, the inception module is added to increase the width of the network and reduce the parameters, which can extract more focus features required for image reconstruction and reduce the complexity. Finally, the focus map of the source image pair can be obtained in the feature reconstruction stage. In this stage, an efficient method is proposed to make the focus mask, which is used for the calculation of the loss function and the generation of the training set. The experimental results on different data sets confirm the superiority and effectiveness of MSIMCNN compared with other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Zhan K, Li Q, Teng J, Wang M, Shi J (2015) Multifocus image fusion using phase congruency. Journal of Electronic Imaging 24(3):033014

    Article  Google Scholar 

  2. Liu Y, Wang Z (2015) Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Processing 9(5):347–357

    Article  Google Scholar 

  3. Goshtasby AA, Nikolov S (2007) Image fusion: advances in the state of the art. Information Fusion 2(8):114–118

    Article  Google Scholar 

  4. Guo X, Nie R, Cao J, Zhou D, Qian W (2018) Fully convolutional network-based multifocus image fusion. Neural Computation 30(7):1775–1800

    Article  MathSciNet  Google Scholar 

  5. Li S, Kwok JT, Wang Y (2001) Combination of images with diverse focuses using the spatial frequency. Information Fusion 2(3):169–176

    Article  Google Scholar 

  6. Huang W, Jing Z (2007) Multi-focus image fusion using pulse coupled neural network. Pattern Recognition Letters 28(9):1123–1132

    Article  Google Scholar 

  7. Luo X, Zhang Z, Zhang B, Wu X (2016) Image fusion with contextual statistical similarity and nonsubsampled shearlet transform. IEEE Sensors Journal 17(6):1760–1771

    Article  Google Scholar 

  8. Chen C-I (2017) Fusion of pet and mr brain images based on ihs and log-gabor transforms. IEEE Sensors Journal 17(21):6995–7010

    Article  Google Scholar 

  9. Aslantas V, Kurban R (2010) Fusion of multi-focus images using differential evolution algorithm. Expert Systems with Applications 37(12):8861–8870

    Article  Google Scholar 

  10. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense sift. Information Fusion 23:139–155

    Article  Google Scholar 

  11. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Transactions on Image Processing 22(7):2864–2875

    Article  Google Scholar 

  12. Li S, Kang X, Hu J, Yang B (2013) Image matting for fusion of multi-focus images in dynamic scenes. Information Fusion 14(2):147–162

    Article  Google Scholar 

  13. Nencini F, Garzelli A, Baronti S, Alparone L (2007) Remote sensing image fusion using the curvelet transform. Information Fusion 8(2):143–156

    Article  Google Scholar 

  14. Zhang Q, Guo B (2009) Multifocus image fusion using the nonsubsampled contourlet transform. Signal Processing 89(7):1334–1346

    Article  Google Scholar 

  15. Miao Q, Shi C, Xu P, Yang M, Shi Y (2011) A novel algorithm of image fusion using shearlets. Optics Communications 284(6):1540–1547

    Article  Google Scholar 

  16. Burt PJ, Adelson EH (1987) The laplacian pyramid as a compact image code. In: Readings in computer vision, pp 671–679. Elsevier

  17. Li S, Kwok JT, Wang Y (2002) Using the discrete wavelet frame transform to merge landsat tm and spot panchromatic images. Information Fusion 3(1):17–23

    Article  Google Scholar 

  18. Lewis JJ, O’Callaghan RJ, Nikolov SG, Bull DR, Canagarajah N (2007) Pixel-and region-based image fusion with complex wavelets. Information Fusion 8(2):119–130

    Article  Google Scholar 

  19. Mitianoudis N, Stathaki T (2008) Image fusion schemes using ica bases. Image Fusion Algorithms Appl:85

  20. Yang B, Li S (2009) Multifocus image fusion and restoration with sparse representation. IEEE Transactions on Instrumentation and Measurement 59(4):884–892

    Article  Google Scholar 

  21. Li S, Yang B, Hu J (2011) Performance comparison of different multi-resolution transforms for image fusion. Information Fusion 12(2):74–84

    Article  Google Scholar 

  22. Miao Q, Shi C, Xu P, Yang M, Shi Y (2011) A novel algorithm of image fusion using shearlets. Optics Communications 284(6):1540–1547

    Article  Google Scholar 

  23. Li S, Yang B (2008) Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recognition Letters 29(9):1295–1301

    Article  Google Scholar 

  24. Ma Jiayi, Chen Chen, Li Chang, Huang Jun (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Information Fusion 31:100–109

    Article  Google Scholar 

  25. Liu Y, Chen X, Peng H, Wang Z (2017) Multi-focus image fusion with a deep convolutional neural network. Information Fusion 36:191–207

    Article  Google Scholar 

  26. Amin-Naji M, Aghagolzadeh A, Ezoji M (2019) Cnns hard voting for multi-focus image fusion. J Ambient Intell Human Comput:1–21

  27. Tang H, Xiao B, Li W, Wang G (2018) Pixel convolutional neural network for multi-focus image fusion. Information Sciences 433:125–141

    Article  MathSciNet  Google Scholar 

  28. Yang Y, Nie Z, Huang S, Lin P, Wu J (2019) Multilevel features convolutional neural network for multifocus image fusion. IEEE Transactions on Computational Imaging 5(2):262–273

    Article  Google Scholar 

  29. Mustafa HT, Yang J, Zareapoor M (2019) Multi-scale convolutional neural network for multi-focus image fusion. Image and Vision Computing 85:26–35

    Article  Google Scholar 

  30. Yan X, Gilani SZ, Qin H, Mian A (2018) Unsupervised deep multi-focus image fusion. arXiv:1806.07272

  31. Gai D, Shen X, Chen H, Su P (2020) Multi-focus image fusion method based on two stage of convolutional neural network. Signal Processing 176:107681

    Article  Google Scholar 

  32. Nejati M, Samavi S, Shirani S (2015) Multi-focus image fusion using dictionary-based sparse representation. Information Fusion 25:72–84

    Article  Google Scholar 

  33. Kumar BKS (2015) Image fusion based on pixel significance using cross bilateral filter. Signal, Image and Video Processing 9(5):1193–1204

    Article  Google Scholar 

  34. Amin-Naji M, Aghagolzadeh A, Ezoji M (2019) Ensemble of cnn for multi-focus image fusion. Information Fusion 51:201–214

    Article  Google Scholar 

  35. Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020) Ifcnn: A general image fusion framework based on convolutional neural network. Information Fusion 54:99–118

    Article  Google Scholar 

  36. Xu H, Ma J, Le Z, Jiang J, Guo X (2020) Fusiondn: A unified densely connected network for image fusion. 34(07):12484–12491

  37. Lai R, Li Y, Guan J (2019) Xiong A (2019) Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access 7:114385–114399

    Article  Google Scholar 

  38. Ma B, Zhu Y, Yin X, Ban X, Huang H, Mukeshimana M (2021) Sesf-fuse: An unsupervised deep model for multi-focus image fusion. Neural Computing and Applications 33(11):5793–5804

    Article  Google Scholar 

  39. Liu Z, Blasch E, Xue Z, Zhao J, Laganiere R, Wu W (2011) Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(1):94–109

    Article  Google Scholar 

  40. Hossny M, Nahavandi S (2008) Creighton D (2008) Comments on’information measure for performance of image fusion’. Electronics Letters 44(18):1066–1067

    Article  Google Scholar 

  41. Xydeas CS, Petrovic V (2000) Objective image fusion performance measure. Electronics Letters 36(4):308–309

    Article  Google Scholar 

  42. Yang C, Zhang J-Q, Wang X-R, Liu X (2008) A novel similarity based quality metric for image fusion. Information Fusion 9(2):156–160

    Article  Google Scholar 

  43. Chen Y, Blum RS (2009) A new automated quality assessment algorithm for image fusion. Image and Vision Computing 27(10):1421–1432

    Article  Google Scholar 

  44. Li J, Guo X, Lu G, Zhang B, Xu Y, Wu F, Zhang D (2020) Drpl: Deep regression pair learning for multi-focus image fusion. IEEE Transactions on Image Processing 29:4816–4831

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their critical and constructive comments and suggestions. This research is partially supported by grant from the National Natural Science Foundation of China (No. 72071019), grants from the Natural Science Foundation of Chongqing (No. cstc2020jcyj-msxmX0068, No. cstc2021jcyj-msxmX0185), and grant from the Science and Technology Project of Chongqing Municipal Education Committee (No. KJQN201900520).

Author information

Authors and Affiliations

  1. College of Computer and Information Science, Chongqing Normal University, 401331, Chongqing, China

    Wenchang Gao, Lei Yu, Yao Tan & Pengna Yang

Authors
  1. Wenchang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yao Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pengna Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Yu.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest to report regarding the present study.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, W., Yu, L., Tan, Y. et al. MSIMCNN: Multi-scale inception module convolutional neural network for multi-focus image fusion. Appl Intell 52, 14085–14100 (2022). https://doi.org/10.1007/s10489-022-03160-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03160-9

Keywords

Search

Navigation