Skip to main content

Advertisement

SpringerLink
Log in

MA-MFIF: When misaligned multi-focus Image fusion meets deep homography estimation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Multi-focus image fusion is a technique that combines multiple out-of-focus images to enhance the overall image quality. It has gained significant attention in recent years, thanks to the advancements in deep learning. However, one of the persistent challenges in this field is the processing of misaligned data, which can negatively impact the fusion results. To overcome this problem, a novel fusion framework with pre-registration is proposed for the fusion of misaligned multi-focus images. For pre-registration, content-aware deep homography estimation is used, which performs transfer learning on a real multi-focus image dataset to adapt to registration under defocused conditions. For fusion, a fusion module with dual-branch feature interaction is utilized to avoid invalid feature fusion and trained on real light field dataset to achieve better fusion performance. Qualitative and quantitative experimental results show that the proposed method has a 2-3 percentage point improvement in multiple evaluation metrics compared to existing advanced registration and fusion methods, and a maximum improvement of 4.83 percentage points in fusion performance when tested independently on the Lytro dataset. Additionally, We find that the value of the \(Q_{cv}\) metric is greatly influenced by the alignment status of the input images, leading to its inability to reflect the fusion quality of aligned images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availability

The Lytro dataset can be downloaded from https://mansournejati.ece.iut.ac.ir/content/lytro-multi-focus-dataset. The Real-MFF dataset can be found from https://github.com/Zancelot/Real-MFF. The V-1, V-2, and V-3 datasets mentioned in the paper can be found from https://github.com/Romatic-zbj/Multi-focus-image-fusion-registration./tree/main/Testdataset

References

  1. Hao F, Li X, Li M et al (2022) An accurate urine red blood cell detection method based on multi-focus video fusion and deep learning with application to diabetic nephropathy diagnosis. Electronics 11(24):4176. https://doi.org/10.3390/electronics11244176

    Article  Google Scholar 

  2. Dai Y, Song Y, Liu W et al (2021) Multi-focus image fusion based on convolution neural network for parkinson’s disease image classification. Diagnostics 11(12):2379. https://doi.org/10.3390/diagnostics11122379

    Article  Google Scholar 

  3. Li J, Guo X, Lu G et al (2020) Drpl: Deep regression pair learning for multi-focus image fusion. IEEE Transactions on Image Processing 29:4816–4831. https://doi.org/10.1109/TIP.2020.2976190

    Article  Google Scholar 

  4. Cheng C, Wu XJ, Xu T et al (2021) Unifusion: A lightweight unified image fusion network. IEEE Transactions on Instrumentation and Measurement 70:1–14. https://doi.org/10.1109/TIM.2021.3109379

    Article  Google Scholar 

  5. Xiao Y, Guo Z, Veelaert P et al (2022) Dmdn: Degradation model-based deep network for multi-focus image fusion. Signal Processing: Image Communication 101:116554. https://doi.org/10.1016/j.image.2021.116554

    Article  Google Scholar 

  6. Ji J, Pan F, Wang X et al (2023) An end-to-end anti-shaking multi-focus image fusion approach. Image Vis. Comput. 137. https://doi.org/10.1016/j.imavis.2023.104788

  7. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int. J. of Comput. Vis. 60:91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  8. Rublee E, Rabaud V, Konolige K, et al (2011) Orb: An efficient alternative to sift or surf. In: 2011 Int. Conf. on Comput. Vis. IEEE, pp 2564–2571. https://doi.org/10.1109/ICCV.2011.6126544

  9. Bay H, Ess A, Tuytelaars T et al (2008) Speeded-up robust features (surf). Comput. Vis. Image Underst. 110(3):346–359. https://doi.org/10.1016/j.cviu.2007.09.014

    Article  Google Scholar 

  10. DeTone D, Malisiewicz T, Rabinovich A (2016) Deep image homography estimation. arXiv:1606.03798https://doi.org/10.48550/arXiv.1606.03798

  11. Zhang J, Wang C, Liu S, et al (2020) Content-aware unsupervised deep homography estimation. In: Eu. Conf. on Comput. Vis., Springer, pp 653–669, https://doi.org/10.1007/978-3-030-58452-8$_$38

  12. DeTone D, Malisiewicz T, Rabinovich A (2016) Deep image homography estimation. arXiv:1606.03798https://doi.org/10.48550/arXiv.1606.03798

  13. Zhang J, Liao Q, Liu S et al (2020) Real-mff: A large realistic multi-focus image dataset with ground truth. Pattern Recognit. Lett. 138:370–377. https://doi.org/10.1016/j.patrec.2020.08.002

  14. Priya BL, Jayanthi K, Pottakkat B et al (2020) A modified framework for multislice image fusion for high contrast liver cancer detection. IETE J. Res. 66(2):139–149. https://doi.org/10.1080/03772063.2018.1479664

  15. Lakshmipriya B, Pavithra N, Saraswathi D (2020) Optimized convolutional neural network based colour image fusion. In: 2020 International Conference on System, Computation, Automation and Networking (ICSCAN), pp 1–4, 10.1109/ICSCAN49426.2020.9262439

  16. Zhou Y, Yu L, Zhi C et al (2022) A survey of multi-focus image fusion methods. Appl. Sci. 12(12):6281. https://doi.org/10.3390/app12126281

  17. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Trans. Image Process. 22(7):2864–2875. https://doi.org/10.1109/TIP.2013.2244222

  18. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense sift. Inf. Fusion. 23:139–155. DOIurl10.1016/j.inffus.2014.05.004

  19. Li S, Kwok JT, Wang Y (2001) Combination of images with diverse focuses using the spatial frequency. Inf. Fusion. 2(3):169–176. https://doi.org/10.1016/S1566-2535(01)00038-0

  20. Cao L, Jin L, Tao H et al (2014) Multi-focus image fusion based on spatial frequency in discrete cosine transform domain. IEEE Signal Process. Lett. 22(2):220–224. https://doi.org/10.1109/LSP.2014.2354534

  21. Toet A (1989) Image fusion by a ratio of low-pass pyramid. Pattern Recognit. Lett. 9(4):245–253. https://doi.org/10.1016/0167-8655(89)90003-2

  22. Petrovic VS, Xydeas CS (2004) Gradient-based multiresolution image fusion. IEEE Trans. Image Process. 13(2):228–237. https://doi.org/10.1109/TIP.2004.823821

  23. Liu Y, Chen X, Peng H et al (2017) Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion. 36:191–207. https://doi.org/10.1016/j.inffus.2016.12.001

  24. Luo H, U K, Zhao W, (2023) Multi-focus image fusion through pixel-wise voting and morphology. Multimed. Tools Appl. 82(1):899–925. https://doi.org/10.1007/s11042-022-13218-y

  25. Wu P, Hua Z, Li J (2023) Multi-scale siamese networks for multi-focus image fusion. Multimed. Tools Appl. 82(10):15651–15672. https://doi.org/10.1007/s11042-022-13949-y

  26. Avci D, Sert E, Özyurt F, et al (2023) Mfif-dwt-cnn: Multi-focus image fusion based on discrete wavelet transform with deep convolutional neural network. Multimed. Tools Appl. pp 1–18. https://doi.org/10.1007/s11042-023-16074-6

  27. Tang L, Deng Y, Ma Y et al (2022) Superfusion: A versatile image registration and fusion network with semantic awareness. IEEE/CAA J. Autom. Sin. 9(12):2121–2137. https://doi.org/10.1109/JAS.2022.106082

  28. Xu H, Yuan J, Ma J (2023) Murf: Mutually reinforcing multi-modal image registration and fusion. IEEE Trans. Pattern Anal. Mach. Intell. https://doi.org/10.1109/TPAMI.2023.3283682

  29. Nejati M, Samavi S, Shirani S (2015) Multi-focus image fusion using dictionary-based sparse representation. Inf. Fusion. 25:72–84. https://doi.org/10.1016/j.inffus.2014.10.004

  30. Chen P, Jiang J, Li L et al (2024) A defocus and similarity attention-based cascaded network for multi-focus and misaligned image fusion. Inf. Fusion. 103:102125. https://doi.org/10.1016/j.inffus.2023.102125

  31. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM. 24(6):381–395. https://doi.org/10.1145/358669.358692

  32. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: IJCAI’81: 7th international joint conference on Artificial intelligence, pp 674–679

  33. Nguyen T, Chen SW, Shivakumar SS et al (2018) Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robot. Autom. Lett. 3(3):2346–2353. https://doi.org/10.1109/LRA.2018.2809549

  34. Jaderberg M, Simonyan K, Zisserman A, et al (2015) Spatial transformer networks. Advances in neural information processing systems 28

  35. Nencini F, Garzelli A, Baronti S et al (2007) Remote sensing image fusion using the curvelet transform. Inf. Fusion. 8(2):143–156. https://doi.org/10.1016/j.inffus.2006.02.001

  36. Lewis JJ, O’Callaghan RJ, Nikolov SG, et al (2007) Pixel-and region-based image fusion with complex wavelets. Inf. Fusion. 8(2):119–130. https://doi.org/10.1016/j.inffus.2005.09.006

  37. Li H, Manjunath B, Mitra SK (1995) Multisensor image fusion using the wavelet transform. Graphical models and image processing 57(3):235–245. https://doi.org/10.1006/gmip.1995.1022

  38. Burt PJ, Adelson EH (1987) The laplacian pyramid as a compact image code. In: Read.Comp. Vis. Elsevier, p 671–679, https://doi.org/10.1016/B978-0-08-051581-6.50065-9

  39. Xu H, Fan F, Zhang H et al (2020) A deep model for multi-focus image fusion based on gradients and connected regions. IEEE Access 8:26316–26327. https://doi.org/10.1109/ACCESS.2020.2971137

  40. Lai R, Li Y, Guan J et al (2019) Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access 7:114385–114399. https://doi.org/10.1109/ACCESS.2019.2935006

  41. Ma H, Liao Q, Zhang J et al (2020) An \(\alpha \)-matte boundary defocus model-based cascaded network for multi-focus image fusion. IEEE Trans. Image Process. 29:8668–8679. https://doi.org/10.1109/TIP.2020.3018261

  42. Ma B, Zhu Y, Yin X et al (2021) Sesf-fuse: An unsupervised deep model for multi-focus image fusion. Neural Comput. Appl. 33:5793–5804. https://doi.org/10.1007/s00521-020-05358-9

  43. Xu S, Wei X, Zhang C, et al (2020) Mffw: A new dataset for multi-focus image fusion. arXiv preprint arXiv:2002.04780https://doi.org/10.48550/arXiv.2002.04780

  44. Cvejic N, Loza A, Bull D et al (2005) A similarity metric for assessment of image fusion algorithms. Int. J. Signal Process.2(3):178–182

  45. Chen H, Varshney PK (2007) A human perception inspired quality metric for image fusion based on regional information. Inf. fusion. 8(2):193–207. https://doi.org/10.1016/j.inffus.2005.10.001

  46. Li H, Wu XJ (2018) Densefuse: A fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5):2614–2623. https://doi.org/10.1109/TIP.2018.2887342

  47. Aslantas V, Bendes E (2015) A new image quality metric for image fusion: The sum of the correlations of differences. AEU - Int. J. Electron. Commun. 69(12):1890–1896. https://doi.org/10.1016/j.aeue.2015.09.004

  48. Ma J, Yu W, Liang P et al (2019) Fusiongan: A generative adversarial network for infrared and visible image fusion. Inf. fusion. 48:11–26. https://doi.org/10.1016/j.inffus.2018.09.004

  49. Li Y, Yi Y, Ren D, et al (2022) Learning dual-pixel alignment for defocus deblurring. arXiv preprint arXiv:2204.12105bhttps://doi.org/10.48550/arXiv.2204.12105

Download references

Acknowledgements

This work was sponsored by National Natural Science Foundation of China (No. 62276097), Natural Science Foundation of Shanghai (No. 22ZR1416500), and the project on Science and Technology Innovation plan Of Shanghai Science and Technology Commission (20dz1201400).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, East China University of Science and Technology, 200237, Shanghai, China

    Baojun Zhao, Fei Luo, Weichao Ding & Chunhua Gu

  2. Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China

    Fei Luo

  3. Department of Computer Science and Information Technologies, Universidad del Bio-Bio, 3780000, Chillán, Chile

    Joel Fuentes

Authors
  1. Baojun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joel Fuentes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weichao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chunhua Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Fei Luo or Weichao Ding.

Ethics declarations

Conflict of Interests

There is no confict of interest

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, B., Luo, F., Fuentes, J. et al. MA-MFIF: When misaligned multi-focus Image fusion meets deep homography estimation. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19385-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19385-4

Keywords

Search

Navigation