Skip to main content
SpringerLink
Log in

DDFN: a depth-differential fusion network for multi-focus image

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Dueto the limitations of digital image capturing equipment, it is usually difficult for the photographer to obtain a complete and clear image of a certain scene in the case of dual targets and multiple targets. This is because most digital imaging systems have a limited depth of field control range, so they can only focus on one or a few objects in the far or near distance, resulting in clear and blurred areas with clear boundaries, that is multi-focus image. This kind of image limits further image processing, such as target recognition, image segmentation, target tracking and so on. Often, two multi-focus images can basically integrate all scene information completely, and multiple multi-focus images can also be fused by cascading all images. Inspired by this, we propose a new image fusion method based on binocular depth estimation and binocular image difference, called depth-differential mapping fusion network (DDFN). In detail, DDFN is based on the idea of residual U-Net and the network structure. It takes two multi-focus images as input, extracts rich hierarchical features through the convolutional pooling pyramid, and learns the residuals between them and the corresponding groundtruth. In this process, DDFN will use their differential information to encode, merge the depth information, and finally perform the decoding process, so the features in the multi-focus image pair will be fully extracted. Finally a clear image without defocusing blur area is formed. We have conducted multiple ablation experiments and comparative experiments, furthermore, a large number of results fully demonstrate the effectiveness of our network structure.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Alantas V, Kurban R (2010) Fusion of multi-focus images using differential evolution algorithm. Expert Syst Appl 37(12):8861–8870

    Article  Google Scholar 

  2. Chen C, Mu S, Xiao W, Ye Z, Wu L, Ju Q (2019) Improving image captioning with conditional generative adversarial nets. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 8142–8150

  3. Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network. In: Advances in neural information processing systems, pp 2366–2374

  4. Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network. In: Advances in neural information processing systems, pp 2366–2374

  5. Everingham M, Winn J (2011) The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8

  6. Feng S, Zhao H, Shi F, Cheng X, Chen X (2020) CPFNet: context pyramid fusion network for medical image segmentation. IEEE Trans Med Imaging 1–1:99

    Google Scholar 

  7. Guo C, Li C, Guo J, Cong R, Fu H, Han P (2018) Hierarchical features driven residual learning for depth map super-resolution. IEEE Trans Image Process, 1–1

  8. Han J, Pauwels EJ, De Zeeuw P (2013) Fast saliency-aware multi-modality image fusion. Neurocomputing 111:70–80

    Article  Google Scholar 

  9. Hirschmuller H (2007) Stereo processing by semiglobal matching and mutual information. IEEE Trans Pattern Anal Mach Intell 30(2):328–341

    Article  Google Scholar 

  10. Jaritz M, Vu TH, Charette RD, Wirbel E, Pérez P (2020) xmuda: cross-modal unsupervised domain adaptation for 3d semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12605–12614

  11. Joshi K, Kirola M, Chaudhary S, Diwakar M, Joshi NK (2019) Multi-focus image fusion using discrete wavelet transform method. In: International conference on advances in engineering science management & technology (ICAESMT)-2019, Uttaranchal University, Dehradun, India

  12. Jung H, Kim Y, Jang H, Ha N, Sohn K (2020) Unsupervised deep image fusion with structure tensor representations. IEEE Trans Image Process 29:3845–3858

    Article  MATH  Google Scholar 

  13. Lai R, Li Y, Guan J, Xiong A (2019) Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access 7:114385–114399

    Article  Google Scholar 

  14. Li M, Cai W, Tan Z (2006) A region-based multi-sensor image fusion scheme using pulse-coupled neural network. Pattern Recogn Lett 27(16):1948–1956

    Article  Google Scholar 

  15. Li J, Guo X, Lu G, Zhang B, Xu Y, Wu F, Zhang D (2020) DRPL: deep regression pair learning for multi-focus image fusion. IEEE Trans Image Process 29:4816–4831

    Article  MATH  Google Scholar 

  16. Li S, Kang X, Fang L, Hu J, Yin H (2017) Pixel-level image fusion: a survey of the state of the art. Inform Fus 33:100–112

    Article  Google Scholar 

  17. Li S, Kwok JT, Wang Y (2001) Combination of images with diverse focuses using the spatial frequency. Inform Fus 2(3):169–176

    Article  Google Scholar 

  18. Li H, Li L, Zhang J (2015) Multi-focus image fusion based on sparse feature matrix decomposition and morphological filtering. Opt Commun 342:1–11

    Article  Google Scholar 

  19. Liu Z, Blasch E, Xue Z, Zhao J, Laganiere R, Wu W (2011) Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans Pattern Anal Mach Intell 34 (1):94–109

    Article  Google Scholar 

  20. Liu Y, Chen X, Peng H (2017) Multi-focus image fusion with a deep convolutional neural network. Inform Fus 36:191–207

    Article  Google Scholar 

  21. Liu Y, Jiang J, Sun J, Bai L, Wang Q (2020) A survey of depth estimation based on computer vision. In: 2020 IEEE Fifth international conference on data science in cyberspace (DSC), pp 135–141

  22. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense sift. Inform Fus 23:139–155

    Article  Google Scholar 

  23. Ma B, Ban X, Huang H, Zhu Y (2019) SESF-fuse: an unsupervised deep model for multi-focus image fusion. arXiv:1908.01703

  24. Ma T, Kuang P, Tian W (2020) An improved recurrent neural networks for 3D object reconstruction. Appl Intell 50:905–923

    Article  Google Scholar 

  25. Ma H, Liao Q, Zhang J, Liu S, Xue JH (2020) An α-matte boundary defocus model-based cascaded network for multi-focus image fusion. IEEE Trans Image Process 29:8668–8679

    Article  MATH  Google Scholar 

  26. Ma J, Ma Y, Li C (2019) Infrared and visible image fusion methods and applications: a survey. Information Fusion, 153–178

  27. Ma J, Yu W, Chen C, Liang P, Jiang J (2020) Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Inform Fus 62:110–120

    Article  Google Scholar 

  28. Ma J, Zhou Z, Wang B, Dong M (2017) Multi-focus image fusion based on multi-scale focus measures and generalized random walk. In: 2017 36th Chinese control conference (CCC), pp 5464–5468

  29. Ma J, Zhou Z, Wang B, Miao L, Zong H (2019) Multi-focus image fusion using boosted random walks-based algorithm with two-scale focus maps. Neurocomputing 335:9–20

    Article  Google Scholar 

  30. Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2794– 2802

  31. Mingrui C, Junyi Y, Guanghui C (2015) Multi-focus image fusion algorithm using LP transformation and PCNN. In: 2015 6th IEEE International conference on software engineering and service science (ICSESS), pp 237–241

  32. Naji MA, Aghagolzadeh A (2015) A new multi-focus image fusion technique based on variance in DCT domain. In: 2015 2nd International conference on knowledge-based engineering and innovation (KBEI), pp 478–484

  33. Nejati M, Samavi S, Shirani S (2015) Multi-focus image fusion using dictionary-based sparse representation. Inform Fus 25:72–84

    Article  Google Scholar 

  34. Nencini F, Garzelli A, Baronti S, Alparone L (2007) Remote sensing image fusion using the curvelet transform. Inform Fus 8(2):143–156

    Article  Google Scholar 

  35. Qiu X, Li M, Zhang L, Yuan X (2019) Guided filter-based multi-focus image fusion through focus region detection. Signal Process Image Commun 72:35–46

    Article  Google Scholar 

  36. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  37. Sahu A, Bhateja V, Krishn A (2014) Medical image fusion with Laplacian pyramids. In: 2014 International conference on medical imaging, m-health and emerging communication systems (MedCom), pp 448–453

  38. Savić S, Babić Z (2012) Multifocus image fusion based on empirical mode decomposition. In: 19th IEEE International conference on systems, signals and image processing (IWSSIP)

  39. Saxena A, Sun M, Ng AY (2008) Make3d: learning 3d scene structure from a single still image. IEEE Trans Pattern Anal Mach Intell 31(5):824–840

    Article  Google Scholar 

  40. Shin K (2020) Binocular depth estimation, https://githubcom/kkenshin1/Binocular-Depth-Estimation

  41. Shita D, Chanda B (2013) Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure. Inform Fus 14(2):136–146

    Article  Google Scholar 

  42. Tang H, Xiao B, Li W, Wang G (2018) Pixel convolutional neural network for multi-focus image fusion. Inform Sci 433:125–141

    Article  MathSciNet  Google Scholar 

  43. Xu H, Fan F, Zhang H, Le Z, Huang J (2020) A deep model for multi-focus image fusion based on gradients and connected regions. IEEE Access 8:26316–26327

  44. Yan X, Gilani SZ, Qin H, Mian A (2018) Unsupervised deep multi-focus image fusion. ArXiv:1806.07272

  45. Yang B, Li S (2010) Multifocus image fusion and restoration with sparse representation. IEEE Trans Instrum Measur 59(4):884–892

    Article  Google Scholar 

  46. Zafar R, Farid MS, Khan MH (2020) Multi-focus image fusion: algorithms, evaluation, and a library. J Imag 6(7):60

    Article  Google Scholar 

  47. Zhang X (2020) Multi-focus image fusion: a benchmark. arXiv:2005.01116

  48. Zhang Q, Li G, Cao Y, Han J (2020) Multi-focus image fusion based on non-negative sparse representation and patch-level consistency rectification. Pattern Recogn 104:107325

    Article  Google Scholar 

  49. Zhang Q, Liu Y, Blum RS, Han J, Tao D (2018) Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: a review. Inform Fus 40:57–75

    Article  Google Scholar 

  50. Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020) IFCNN: a general image fusion framework based on convolutional neural network. Inform Fus 54:99–118

    Article  Google Scholar 

  51. Zhang Q, Shi T, Wang F, Blum RS, Han J (2018) Robust sparse representation based multi-focus image fusion with dictionary construction and local spatial consistency. Pattern Recogn 83:299–313

    Article  Google Scholar 

  52. Zhao W, Wang D, Lu H (2018) Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Trans Circ Syst Video Technol 29(4):1102–1115

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Natural Science Foundation of China (61772319, 61773244, 61976125, 61976124), Shandong Natural Science Foundation of China (ZR2017MF049) and Yantai Key Research and Development Plan (2019XDHZ081).

Author information

Authors and Affiliations

  1. Co-innovation Center of Shandong Colleges and Universities: Future Intelligent, Shandong Technology and Business University, Yantai, Shandong, China

    Limai Jiang & Jinjiang Li

  2. School of Computer Science and Technology, Shandong Technology and Business University, Yantai, China

    Hui Fan & Jinjiang Li

Authors
  1. Limai Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinjiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinjiang Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, L., Fan, H. & Li, J. DDFN: a depth-differential fusion network for multi-focus image. Multimed Tools Appl 81, 43013–43036 (2022). https://doi.org/10.1007/s11042-022-12075-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12075-z

Keywords

Search

Navigation