Skip to main content
SpringerLink
Log in

Multimodal image sensor fusion in a cascaded framework using optimized dual channel pulse coupled neural network

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Medical image fusion plays an important role in the clinical diagnosis of several critical diseases by merging the complementary information present in multimodal images and also assists the radiologists. Therefore, in this paper, a cascaded multimodal medical image fusion scheme is proposed using an optimized dual-channel biologically-inspired spiking neural model in two-scale hybrid \(\ell _{1}\) − \(\ell _{0}\) layer decomposition (HLD) and nonsubsampled shearlet transform (NSST) domain. The different features of HLD and NSST in a cascade framework are utilized to preserve more structural and textural information available in the source images with the suppression of significant noise and artifacts. At stage 1, the source images are subjected to HLD, the base layer is fused by choose max rule to preserve the local luminescence information, and detail layers at scale-1 and 2 are further decomposed by NSST. At stage 2, fusion rule based on laws of texture energy features is utilized to fuse low-frequency coefficients to highlight the local energy, contrast, and textures of the source images, while to fuse high-frequency coefficients, an optimized dual-channel biologically-inspired spiking neural model is utilized to maximize the retention of sharp edges and enhance the visual quality of fused images. Differential evolution is used for optimization which uses a fitness function based on the edge index of the resultant fused image. To analyze the fusion performance, extensive experiments are conducted on CT-MR-T2, MR-T2-SPECT neurological image dataset. Experimental results show that the proposed method provides better-fused images and outperforms the other existing fusion method in both visual and quantitative assessments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Chai Y, Li H, Qu J (2010) Image fusion scheme using a novel dual-channel pcnn in lifting stationary wavelet domain. Opt Commun 283(19):3591–3602

    Article  Google Scholar 

  • Das S, Kundu MK (2012) Nsct-based multimodal medical image fusion using pulse-coupled neural network and modified spatial frequency. Med Biol Eng Comput 50(10):1105–1114

    Article  Google Scholar 

  • Das M, Gupta D, Radeva P, Bakde AM (2020) NSST domain CT-MR neurological image fusion using optimised biologically inspired neural network. IET Image Proc 14(16):4291–4305

    Article  Google Scholar 

  • Das M, Gupta D, Radeva P, Bakde A (2021a) Optimized bio-inspired spiking neural models based anatomical and functional neurological image fusion in NSST domain. In: 2021 National Conference on Communications (NCC), pp 1–6

  • Das M, Gupta D, Radeva P, Bakde AM (2021b) Multi-scale decomposition-based ct-mr neurological image fusion using optimized bio-inspired spiking neural model with meta-heuristic optimization. Int J Imaging Syst Technol 31(4):2170–2188

    Article  Google Scholar 

  • Das M, Gupta D, Radeva P, Bakde AM (2021c) Optimized CT-MR neurological image fusion framework using biologically inspired spiking neural model in hybrid l1–l0 layer decomposition domain. Biomed Signal Process Control 68:102535

    Article  Google Scholar 

  • Dogra A, Goyal B, Agrawal S (2017) From multi-scale decomposition to non-multi-scale decomposition methods: a comprehensive survey of image fusion techniques and its applications. IEEE Access 5:16040–16067

    Article  Google Scholar 

  • Du J, Li W, Lu K, Xiao B (2016) An overview of multi-modal medical image fusion. Neurocomputing 215:3–20

    Article  Google Scholar 

  • Ganasala P, Kumar V (2016) Feature-motivated simplified adaptive pcnn-based medical image fusion algorithm in nsst domain. J Digit Imaging 29(1):73–85

    Article  Google Scholar 

  • Ganasala P, Prasad AD (2020) Medical image fusion based on laws of texture energy measures in stationary wavelet transform domain. Int J Imaging Syst Technol 30(3):544–557

    Article  Google Scholar 

  • Gómez-Flores W, Hernández-López J (2020) Automatic adjustment of the pulse-coupled neural network hyperparameters based on differential evolution and cluster validity index for image segmentation. Appl Soft Comput 97:105547

    Article  Google Scholar 

  • Guo Z, Li X, Huang H, Guo N, Li Q (2019) Deep learning-based image segmentation on multimodal medical imaging. IEEE Trans Radiat Plasma Med Sci 3(2):162–169

    Article  Google Scholar 

  • Gupta D (2018) Nonsubsampled shearlet domain fusion techniques for ct-mr neurological images using improved biological inspired neural model. Biocybernet Biomed Eng 38(2):262–274

    Article  MathSciNet  Google Scholar 

  • Hu T, Khishe M, Mohammadi M, Parvizi GR, Karim SHT, Rashid TA (2021) Real-time covid-19 diagnosis from x-ray images using deep cnn and extreme learning machines stabilized by chimp optimization algorithm. Biomed Signal Process Control 68:102764

    Article  Google Scholar 

  • James AP, Dasarathy BV (2014) Medical image fusion: a survey of the state of the art. Inf Fusion 19:4–19

    Article  Google Scholar 

  • Jiang Q, Jin X, Lee SJ, Yao S (2017) A novel multi-focus image fusion method based on stationary wavelet transform and local features of fuzzy sets. IEEE Access 5:20286–20302

    Article  Google Scholar 

  • Khishe M, Mosavi MR (2020) Chimp optimization algorithm. Expert Syst Appl 149:113338

    Article  Google Scholar 

  • Khishe M, Caraffini F, Kuhn S (2021) Evolving deep learning convolutional neural networks for early covid-19 detection in chest x-ray images. Mathematics 9(9):1002

    Article  Google Scholar 

  • Labate D, Lim WQ, Kutyniok G, Weiss G (2005) Sparse multidimensional representation using shearlets. Wavelets XI Int Soc Opt Photon 5914:254–262

    Google Scholar 

  • Lahoud F, Süsstrunk S (2019) Zero-learning fast medical image fusion. In: 2019 22th International conference on information fusion (FUSION), IEEE, pp 1–8

  • Li X, Zhao J (2021) A novel multi-modal medical image fusion algorithm. J Ambient Intell Hum Comput 12(2):1995–2002

    Article  MathSciNet  Google Scholar 

  • Li S, Yang B, Hu J (2011) Performance comparison of different multi-resolution transforms for image fusion. Inf Fusion 12(2):74–84

    Article  Google Scholar 

  • Li S, Kang X, Fang L, Hu J, Yin H (2017) Pixel-level image fusion: a survey of the state of the art. Inf Fusion 33:100–112

    Article  Google Scholar 

  • Li X, Guo X, Han P, Wang X, Li H, Luo T (2020) Laplacian redecomposition for multimodal medical image fusion. IEEE Trans Instrum Meas 69(9):6880–6890

    Article  Google Scholar 

  • Li X, Zhou F, Tan H, Zhang W, Zhao C (2021) Multimodal medical image fusion based on joint bilateral filter and local gradient energy. Inf Sci 569:302–325

    Article  MathSciNet  Google Scholar 

  • Liang Z, Xu J, Zhang D, Cao Z, Zhang L (2018) A hybrid l1–l0 layer decomposition model for tone mapping, pp 4758–4766

  • Panigrahy C, Seal A, Mahato NK (2020) Mri and spect image fusion using a weighted parameter adaptive dual channel pcnn. IEEE Signal Process Lett 27:690–694

    Article  Google Scholar 

  • Qu G, Zhang D, Yan P (2002) Information measure for performance of image fusion. Electron Lett 38(7):313–315

    Article  Google Scholar 

  • Singh S, Anand R (2018) Ripplet domain fusion approach for ct and mr medical image information. Biomed Signal Process Control 46:281–292

    Article  Google Scholar 

  • Singh S, Anand R (2019) Multimodal neurological image fusion based on adaptive biological inspired neural model in nonsubsampled shearlet domain. Int J Imaging Syst Technol 29(1):50–64

    Article  Google Scholar 

  • Singh S, Anand R (2020) Multimodal medical image sensor fusion model using sparse k-svd dictionary learning in nonsubsampled shearlet domain. IEEE Trans Instrum Meas 69(2):593–607

    Article  Google Scholar 

  • Singh S, Gupta D (2021) Detail enhanced feature-level medical image fusion in decorrelating decomposition domain. IEEE Trans Instrum Meas 70:1–9

    Google Scholar 

  • Singh S, Gupta D, Anand R, Kumar V (2015) Nonsubsampled shearlet based ct and mr medical image fusion using biologically inspired spiking neural network. Biomed Signal Process Control 18:91–101

    Article  Google Scholar 

  • Singh S, Anand RS, Gupta D (2018) Ct and mr image information fusion scheme using a cascaded framework in ripplet and nsst domain. IET Image Proc 12(5):696–707

    Article  Google Scholar 

  • Sufyan A, Imran M, Shah SA, Shahwani H, Wadood AA (2021) A novel multimodality anatomical image fusion method based on contrast and structure extraction. Int J Imaging Syst Technol. https://doi.org/10.1002/ima.22649

    Article  Google Scholar 

  • Tan W, Tiwari P, Pandey HM, Moreira C, Jaiswal AK (2020) Multimodal medical image fusion algorithm in the era of big data. Neural Comput Appl. https://doi.org/10.1007/s00521-020-05173-2

    Article  Google Scholar 

  • Tawhid MA, Ibrahim AM (2020) A hybridization of grey wolf optimizer and differential evolution for solving nonlinear systems. Evol Syst 11(1):65–87

    Article  Google Scholar 

  • Toktas A, Erkan U, Toktas F, Yetgın Z (2021) Chaotic map optimization for image encryption using triple objective differential evolution algorithm. IEEE Access 9:127814–127832

    Article  Google Scholar 

  • Wang Z, Ma Y (2007) Dual-channel PCNN and its application in the field of image fusion. In: Third international conference on natural computation (ICNC 2007), vol 1, pp 755–759

  • Wang Z, Wang S, Zhu Y, Ma Y (2016) Review of image fusion based on pulse-coupled neural network. Arch Comput Methods Eng 23(4):659–671

    Article  MathSciNet  MATH  Google Scholar 

  • Wu C, Khishe M, Mohammadi M, Karim SHT, Rashid TA (2021) Evolving deep convolutional neutral network by hybrid sine-cosine and extreme learning machine for real-time covid19 diagnosis from x-ray images. Soft Comput. https://doi.org/10.1007/s00500-021-05839-6

    Article  Google Scholar 

  • Xu X, Wang G, Ding S, Cheng Y, Wang X (2017) Pulse-coupled neural networks and parameter optimization methods. Neural Comput Appl 28(1):671–681

    Article  Google Scholar 

  • Xu H, Ma J, Jiang J, Guo X, Ling H (2022) U2fusion: a unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell 44(1):502–518

    Article  Google Scholar 

  • Xydeas C, Petrovic V (2000) Objective image fusion performance measure. Electron Lett 36(4):308–309

    Article  Google Scholar 

  • Yang Y, Que Y, Huang S, Lin P (2016) Multimodal sensor medical image fusion based on type-2 fuzzy logic in nsct domain. IEEE Sens J 16(10):3735–3745

    Article  Google Scholar 

  • Yin M, Liu X, Liu Y, Chen X (2018) Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans Instrum Meas 68(1):49–64

    Article  Google Scholar 

  • Zhu Z, Zheng M, Qi G, Wang D, Xiang Y (2019) A phase congruency and local laplacian energy based multi-modality medical image fusion method in nsct domain. IEEE Access 7:20811–20824

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, Visvesvaraya National Institute of Technology, Nagpur, India

    Manisha Das & Deep Gupta

  2. Department of Mathematics and Computer Science, University of Barcelona, Barcelona, Spain

    Petia Radeva

  3. Department of Radio-Diagnosis, All India Institute of Medical Sciences, Nagpur, India

    Ashwini M. Bakde

Authors
  1. Manisha Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deep Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Petia Radeva
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ashwini M. Bakde
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deep Gupta.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Das, M., Gupta, D., Radeva, P. et al. Multimodal image sensor fusion in a cascaded framework using optimized dual channel pulse coupled neural network. J Ambient Intell Human Comput 14, 11985–12004 (2023). https://doi.org/10.1007/s12652-022-03749-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-022-03749-5

Keywords

Search

Navigation