Abstract
Computed tomography (CT) and magnetic resonance imaging (MRI) image fusion is a popular technique for integrating information from two different modalities of medical images. This technique can improve image quality and diagnostic efficacy. To effectively extract and balance complementary information in the source images, we propose an end-to-end multimodal feature interaction network (MFINet) to fuse CT and MRI images. The MIFNet consists of a shallow feature extractor, a feature interaction (FI), and an image reconstruction. In the FI, we design a deep feature extraction module, which consists of a series of gated feature enhancement units (GFEUs) and convolutional layers. To extract key features from images, we introduce a gated normalization block in the GFEU, which can achieve feature selection. Comprehensive experiments demonstrate that the proposed end-to-end fusion network outperforms existing state-of-the-art methods in both qualitative and quantitative assessments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The data used in this work is available at http://www.med.harvard.edu/aanlib/home.html.
Notes
[Online]. Available online: http://www.med.harvard.edu/aanlib/home.html.
References
Ali FE, El-Dokany I, Saad A, Abd El-Samie F (2010) A curvelet transform approach for the fusion of mr and ct images. J Mod Opt 57(4):273–286
Anu PS, Khanaa V (2023) Multimodality brain tumor image fusion using wavelet and contourlet transformation. In: Joseph, F.J.J., Balas, V.E., Rajest, S.S., Regin, R. (eds) Computational intelligence for clinical diagnosis. Springer, pp 201–214
Azam MA, Khan KB, Salahuddin S, Rehman E, Khan SA, Khan MA, Kadry S, Gandomi AH (2022) A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics. Comput Biol Med 144:105253
Bhavana V, Krishnappa H (2015) Multi-modality medical image fusion using discrete wavelet transform. Procedia Comput Sci 70:625–631
Di W, Jinyuan L, Xin F, Liu Risheng (2022) Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration. In: International joint conference on artificial intelligence (IJCAI) 3508-3515
Du J, Li W, Xiao B, Nawaz Q (2016) Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing 194:326–339
Faragallah OS, El-Hoseny H, El-Shafai W, El-Rahman WA, El-sayed HS, El-Rabaie ES, El-Samie FA, Mahmoud KR, Geweid GG (2022) Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis. Multimed Tools Appl 81(10):14379–14401
Ganasala P, Prasad A (2018) Medical image fusion based on frei-chen masks in nsst domain. In: 2018 5th international conference on signal processing and integrated networks (SPIN). IEEE, pp 619–623
Gao M, Zhou Y, Zhai W, Zeng S, Li Q (2023) Saregan: a salient regional generative adversarial network for visible and infrared image fusion. Multimed Tools Appl 1–13
Haribabu M, Guruviah V, Yogarajah P (2023) Recent advancements in multimodal medical image fusion techniques for better diagnosis: an overview. Curr Med Imaging 19(7):673–694
Huang Y, Li W, Gao M, Liu Z (2018) Algebraic multi-grid based multi-focus image fusion using watershed algorithm. IEEE Access 6:47082–47091. https://doi.org/10.1109/ACCESS.2018.2866867
Jian L, Yang X, Liu Z, Jeon G, Gao M, Chisholm D (2020) Sedrfuse: a symmetric encoder-decoder with residual block network for infrared and visible image fusion. IEEE Trans Instrum Meas 70:1–15
Li H, Wu XJ (2019) Densefuse: a fusion approach to infrared and visible images. IEEE Trans Image Process 28(5):2614–2623
Li H, Wu XJ, Kittler J (2021) Rfn-nest: an end-to-end residual fusion network for infrared and visible images. Inf Fusion 73:72–86
Li W, Zhang Y, Wang G, Huang Y, Li R (2023) Dfenet: a dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion. Biomed Signal Process Control 80:104402
Ma J, Yu W, Liang P, Li C, Jiang J (2019) Fusiongan: a generative adversarial network for infrared and visible image fusion. Inf Fusion 48:11–26
Ma J, Xu H, Jiang J, Mei X, Zhang XP (2020) Ddcgan: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process 29:4980–4995
Ma J, Tang L, Xu M, Zhang H, Xiao G (2021) Stdfusionnet: an infrared and visible image fusion network based on salient target detection. IEEE Trans Instrum Meas 70:1–13
Maqsood S, Javed U (2020) Multi-modal medical image fusion based on two-scale image decomposition and sparse representation. Biomed Signal Process Control 57:101810
Singh R, Khare A (2014) Fusion of multimodal medical images using daubechies complex wavelet transform-a multiresolution approach. Inf Fusion 19:49–60
Song W, Zhai W, Gao M, Li Q, Chehri A, Jeon G (2023) Multiscale aggregation and illumination-aware attention network for infrared and visible image fusion. Concurr Comput Pract Exp e7712
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Woo S, Park J, Lee JY, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), Springer, Cham, pp. 3–19
Xu H, Ma J, Jiang J, Guo X, Ling H (2020a) U2fusion: a unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell 44:502–518
Xu H, Ma J, Le Z, Jiang J, Guo X (2020b) Fusiondn: a unified densely connected network for image fusion. Proceedings of the AAAI conference on artificial intelligence, New York, USA. vol 34, pp 12484–12491
Xu H, Zhang H, Ma J (2021) Classification saliency-based rule for visible and infrared image fusion. IEEE Trans Comput Imaging 7:824–836
Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang MH (2022) Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, Louisiana, USA. pp 5728–5739
Zhai W, Song W, Chen J, Zhang G, Li Q, Gao M (2023) Ct and mri image fusion via dual-branch gan. Int J Biomed Eng Technol 42(1):52–63
Zhang H, Ma J (2021) Sdnet: a versatile squeeze-and-decomposition network for real-time image fusion. Int J Comput Vis 129:1–25
Zhang H, Xu H, Xiao Y, Guo X, Ma J (2020) Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. In: Proceedings of the AAAI conference on artificial intelligence, New York, USA. pp 12797–12804
Zhang G, Nie R, Cao J, Chen L, Zhu Y (2023) Fdgnet: a pair feature difference guided network for multimodal medical image fusion. Biomed Signal Process Control 81:104545
Zhu Z, Zheng M, Qi G, Wang D, Xiang Y (2019) A phase congruency and local Laplacian energy based multi-modality medical image fusion method in nsct domain. IEEE Access 7:20811–20824
Acknowledgements
This work is supported by the National Natural Science Foundation of China (no. 62101310).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no Conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Song, W., Zeng, X., Li, Q. et al. CT and MRI image fusion via multimodal feature interaction network. Netw Model Anal Health Inform Bioinforma 13, 13 (2024). https://doi.org/10.1007/s13721-024-00449-2
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s13721-024-00449-2