Feature-Motivated Simplified Adaptive PCNN-Based Medical Image Fusion Algorithm in NSST Domain

Abstract

Multimodality medical image fusion plays a vital role in diagnosis, treatment planning, and follow-up studies of various diseases. It provides a composite image containing critical information of source images required for better localization and definition of different organs and lesions. In the state-of-the-art image fusion methods based on nonsubsampled shearlet transform (NSST) and pulse-coupled neural network (PCNN), authors have used normalized coefficient value to motivate the PCNN-processing both low-frequency (LF) and high-frequency (HF) sub-bands. This makes the fused image blurred and decreases its contrast. The main objective of this work is to design an image fusion method that gives the fused image with better contrast, more detail information, and suitable for clinical use. We propose a novel image fusion method utilizing feature-motivated adaptive PCNN in NSST domain for fusion of anatomical images. The basic PCNN model is simplified, and adaptive-linking strength is used. Different features are used to motivate the PCNN-processing LF and HF sub-bands. The proposed method is extended for fusion of functional image with an anatomical image in improved nonlinear intensity hue and saturation (INIHS) color model. Extensive fusion experiments have been performed on CT-MRI and SPECT-MRI datasets. Visual and quantitative analysis of experimental results proved that the proposed method provides satisfactory fusion outcome compared to other image fusion methods.

Introduction

Multimodality medical image fusion provides a composite image containing all the crucial information present in the source images. It is widely used in diagnosis, treatment planning, and follow-up studies of various diseases. It helps in precise localization and delineation of lesions. For the diagnosis and treatment planning of certain diseases, information provided by more than one imaging modality would be required as each medical imaging modality is having its own strengths and limitations. Anatomical imaging modalities like X-ray computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound imaging provide morphologic details of the human body. Whereas, functional imaging modalities like single photon emission computed tomography (SPECT) and positron emission computed tomography (PET) provide metabolic information without anatomical context. By the fusion of functional image with an anatomical image, it is possible to provide anatomical correlation to the functional information. This is used in oncology for tumor segmentation and localization for radiation therapy treatment planning [1]. Fusion of CT with MRI images is used in brachytherapy treatment planning [2].

Medical image fusion has been widely explored, and many schemes have been proposed by researchers. Depending on the technique used, these can be classified into the following categories: methods based on dimensionality reduction technique such as principle component analysis (PCA), morphological operators-based methods, fuzzy logic-based methods, neural network (NN)-based methods, and multi-scale decomposition transform-based methods or transform domain methods [3]. Multi-scale top hat transform-based image fusion algorithm is proposed in [4]. These morphological operator-based algorithms are sensitive to structuring element used to extract image features, and there is possibility of introduction of artifacts into the fused image. The performance of fuzzy logic-based algorithms [5] is dependent on the selection of membership functions, and the membership functions have to be varied depending on the modalities to be fused. In case of NN-based algorithm [6], images are segmented or divided into blocks of fixed sizes, and then they are fused. These methods are sensitive to segmentation algorithm or the block size [7, 8]. Functional and anatomical image fusion based on PCA and intensity hue saturation (IHS) color model has been proposed in [9]. This method suffered from spectral distortion because of the nonlinear IHS color model. PET-MRI fusion based on color tables has been tested in [10], and this method presents the functional information in different color palette. PET-CT fusion based on features in spatial domain is reported in [11]. This method is easy to implement. However, color palettes of source functional image and fused image are different in this case. Among the aforementioned techniques, transform domain image fusion methods are more feasible for medical image fusion. Because it is possible to extract image features of different resolutions and directions into different sub-bands using transform and combine these sub-bands in accordance with their characteristics. In transform domain image fusion methods, the major influencing factors are selection of multi-scale decomposition (MSD) transform and the coefficient fusion schemes. The MSD transforms like discrete wavelet transform (DWT), curvelet transform (CVT), and contourlet transform (CNT) do not provide better image fusion performance as they are shift variant transforms. Due to subsampling operation in the construction of these transforms, some information would be lost, and blocking artifacts would be introduced in the fused image. Again, due to up-sampling in their reconstruction, the error incurred would be enlarged. Hence, shift-invariance property of MSD resulting from elimination of down-samplers and up-samplers are very much essential for image fusion. One of the shift invariant MSD transforms is nonsubsampled contourlet transform (NSCT). It is able to represent the smoothness along the edges or contours properly. However, its high computational complexity impedes its use for medical image fusion. Nonsubsampled shearlet transform (NSST) [12, 13] is a shift invariant MSD transform that has high directional sensitivity and having less computational complexity compared to NSCT. These features of NSST are suitable for medical image fusion application. After fixing the MSD transform for directional decomposition of source images, the next step is to design the fusion rules for combining the coefficients of different sub-bands.

Fusion rules based on different strategies have been tested by researchers. These could be classified into fusion rules based on saliency measures, neural networks, and fuzzy logic. Entropy of square of the coefficients and sum of modified Laplacian has been used in NSCT domain [14]. Energy and regional feature-based fusion rules have been adapted in NSCT domain [15]. Regional energy and contrast feature-based fusion rules in NSST domain have been utilized in [16]. Directional vector norm and band-limited contrast-based fusion rules in NSST domain have been proposed in [17]. Non-classical RF model-based fusion rules in NSST domain have been tested in [18]. Fusion outcome of all these methods suffered from poor contrast and loss of information related to one of the source images. A special type of neural network inspired by the cat’s visual cortex called pulse-coupled neural network (PCNN) [19] has been applied for image fusion. The temporal synchronous pulse output of PCNN contains valuable information for many image-processing applications. The PCNN used in multi-resolution transform domain is promising for medical image fusion application. In the past years, researchers have used PCNN-based fused rules in transform domain. The basic PCNN model is utilized in NSCT domain [20]. Modified spatial frequency-motivated PCNN has been adapted in NSCT domain [21]. PCNN with adaptive-linking strength based on spatial frequency has been used in NSST domain [22]. Unit-linking PCNN with contrast-based linking strength has been used in [23]. The modified models of PCNN such as dual-channel PCNN [24, 25], intersecting cortical model (ICM) [26], and spiking cortical model (SCM) [27, 28] have been used for image fusion. However, in these state-of-the-art methods, external stimulus to PCNN for approximate and detail sub-bands is normalized coefficient value. PCNN-based fusion rules for both types of sub-bands have been modeled in the same way. In fact, the information present in approximate and detail sub-bands is different. The normalized coefficient value used to motivate PCNN in LF sub-band fusion leads to blur and loss of details in the fused image. In the present work, this point is taken into consideration, and a suitable feature is used to motivate the PCNN-processing LF sub-band. Values of other parameters of the PCNN are chosen based on the task to be performed by them in the network to get optimum performance.

Rest of the paper is organized as follows. PCNN neuron model used in the proposed method is discussed in section “Simplified Adaptive PCNN Model”. The proposed anatomical image fusion method is described in section “Proposed Anatomical Image Fusion Method”. Extension of this method to functional and anatomical image fusion is described in section “Functional and Anatomical Image Fusion”. The “Results and Discussions” section is divided into three subsections for clarity: “Methods Used for Comparison”, “CT-MRI Image Fusion Results”, and “SPECT-MRI Image Fusion Results”. Finally, conclusions are drawn in the “Conclusions” section.

Simplified Adaptive PCNN Model

PCNN is a laterally connected two-dimensional network of pulse-coupled neurons that have one to one correspondence with pixels of the image. The basic model of PCNN has more number of parameters and its detailed description is available in [27]. It is practically difficult to set the optimum value for each parameter. Hence, in the present work, PCNN is simplified in such a way to reduce the number of parameters to be adjusted. On other hand, all the unique capabilities of the PCNN are retained in this simplified model. Structure of the simplified adaptive PCNN neuron model adapted in the present work is depicted in Fig. 1. This consists of three fields: receptive field, modulation filed, and pulse generator. The receptive field comprises of two input parts: feeding input and linking input. Feeding input F _ij [n] given by Eq. (1) is the channel through which external stimulus S _ij and the output from neighboring neurons Y _ijkl [n] fed to the center neuron. Linking input L _ij [n] given by Eq. (2) is another channel through which neighboring neurons coupled to the center neuron.

Fig. 1

Neuron structure of simplified adaptive PCNN model

$$ {F}_{ij}\left[n\right]={S}_{ij}+{\displaystyle {\sum}_{k=-1}^1{\displaystyle {\sum}_{l=-1}^1{W}_{kl}{Y}_{ijkl}\left[n-1\right]}} $$

(1)

$$ {L}_{ij}\left[n\right]={\displaystyle {\sum}_{k=-1}^1{\displaystyle {\sum}_{l=-1}^1{M}_{kl}{Y}_{ijkl}\left[n-1\right]}} $$

(2)

Where, W and M are the synaptic weight matrices.

In the modulation field, the feeding and linking inputs are modulated in the second order fashion to obtain the total internal activity U _ij [n] of the neuron as described in Eq. (3).

$$ {U}_{ij}\left[n\right]={F}_{ij}\left[n\right]\left(1+{\beta}_{ij}{L}_{ij}\left[n\right]\right) $$

(3)

β _ij is the linking strength of the ijth neuron of the network.

In pulse generator part, internal activity U _ij [n] of the neuron is compared with its internal threshold θ _ij [n], and the pulse will be generated at the instances when the internal activity crosses the threshold value.

$$ {Y}_{ij}\left[n\right]=\left\{\begin{array}{c}\hfill \begin{array}{cc}\hfill 1\hfill & \hfill \mathrm{if}\ {U}_{ij}\left[n\right]\ge {\theta}_{ij}\left[n\right]\hfill \end{array}\hfill \\ {}\hfill \begin{array}{cc}\hfill 0\hfill & \hfill \kern3.25em \mathrm{otherwise}\hfill \end{array}\hfill \end{array}\right. $$

(4)

Threshold value is updated each iteration according to Eq. (5).

$$ {\theta}_{ij}\left[n\right]=\alpha {\theta}_{ij}\left[n-1\right]+v{Y}_{ij}\left[n-1\right] $$

(5)

Where α is threshold decay constant, and v is threshold magnifying constant.

The time matrix which holds the first firing instance of each neuron in the PCNN is described in Eq. (6). This holds important information, and it is used as output of PCNN in the present work.

$$ {T}_{ij}\left[n\right]=\left\{\begin{array}{cc}\hfill n\hfill & \hfill\ \mathrm{if}\ {Y}_{ij}\left[n\right]=1\ \mathrm{f}\mathrm{o}\mathrm{r}\ \mathrm{the}\ \mathrm{f}\mathrm{irst}\ \mathrm{time}\ \hfill \\ {}\hfill {T}_{ij}\left[n-1\right]\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right. $$

(6)

PCNN parameters are set to the following values:

• I. Synaptic weight matrices (W and M): The synaptic weight matrix couples the neighboring neurons to the center neuron. Generally, neighboring neurons close to the center neuron must be coupled with more weight than neighboring neurons far away. Hence, it is reasonable to make the synaptic weight matrices to be dependent on the Euclidian distance from the center neuron. As the synaptic weight matrices W and M in the feeding input F _ij [n] given in Eq. (1), and linking input L _ij [n] given in Eq. (2), respectively, serve the same purpose, they are set to the same value given in Eq. (7).

  $$ W=M=\left[\begin{array}{c}\hfill \begin{array}{ccc}\hfill 0.1035\hfill & \hfill 0.1465\hfill & \hfill 0.1035\hfill \end{array}\hfill \\ {}\hfill \begin{array}{ccc}\hfill 0.1465\hfill & \hfill \kern2em 0\kern0.75em \hfill & \hfill 0.1465\hfill \end{array}\hfill \\ {}\hfill \begin{array}{ccc}\hfill 0.1035\hfill & \hfill 0.1465\hfill & \hfill 0.1035\hfill \end{array}\hfill \end{array}\right] $$

  (7)

• II. Linking strength (β): Linking strength β modulates the linking input and feeding input to form total internal activity as given in Eq. (3). In order to get high internal activity from regions having remarkable features than the low activity regions, it is intuitive to use variable linking strength instead of using constant linking strength for all neurons in PCNN. Sum of directional gradients (SDG) feature at each location is used as the linking strength of the neuron present in the corresponding location.

  The sum of directional gradients of a function I(x, y) is defined as:

  $$ \mathrm{S}\mathrm{D}\mathrm{G}\left[I\left(x,y\right)\right]=\mathrm{dg}1+\mathrm{dg}2+\mathrm{dg}3+\mathrm{dg}4 $$

  (8)
  $$ \mathrm{dg}1=\sqrt{{\left(I\left(x,y\right)-I\left(x-1,y\right)\right)}^2+{\left(I\left(x,y\right)-I\left(x+1,y\right)\right)}^2} $$

  (9)
  $$ \mathrm{dg}2=\sqrt{{\left(I\left(x,y\right)-I\left(x,y-1\right)\right)}^2+{\left(I\left(x,y\right)-I\left(x,y+1\right)\right)}^2} $$

  (10)
  $$ \mathrm{dg}3=\sqrt{\left({\left(I\left(x,y\right)-I\left(x-1,y-1\right)\right)}^2+{\left(I\left(x,y\right)-I\left(x+1,y+1\right)\right)}^2\right)/2} $$

  (11)
  $$ \mathrm{dg}4=\sqrt{\left({\left(I\left(x,y\right)-I\left(x-1,y+1\right)\right)}^2+{\left(I\left(x,y\right)-I\left(x+1,y-1\right)\right)}^2\right)/2} $$

  (12)

• III. Threshold decay and magnifying constants: Threshold θij[n] of each neuron is updated by Eq. (5) every iteration of PCNN. The two parameters, threshold decay (α) and magnifying (v) constants, control the pulsing frequency of neurons. If the neuron has not pulsed, then threshold is reduced by a factor of α each iteration to reach to the internal activity so that neuron may get fired. To get moderate pulsing frequency, α is made equal to 0.75. Once the neuron fired, then threshold would be increased by an amount v. Hence, v is set to 20 a large value in order to avoid firing of neurons more number of times during PCNN processing.

• IV. Number of iterations: Number of iterations of PCNN is chosen adaptively. PCNN is iterated until all the neurons fired once. The ordinal value of the iteration in which the lost neuron fired gives the total number of iterations.

Proposed Anatomical Image Fusion Method

Although PCNN can be used to fuse the image pixels directly in spatial domain, usage of PCNN in multi-scale decomposition transform (MSDT) domain gives better results, because MSDT visualizes image feature of different orientations and different resolutions. Among existing MSDTs, nonsubsampled shearlet transform (NSST) has important characteristics of shift invariance and high directional sensitivity at lower computational complexity. In the present work, image fusion is performed in NSST domain. More details of NSST are available in [12, 29]. The proposed image fusion frame work is depicted in Fig. 2. The two source images A and B are represented in approximate/low-frequency (LF) sub-bands and directional/high-frequency (HF) sub-bands of various scales using NSST. The LF sub-bands A ^L and B ^L of source images A and B are combined to obtain fused LF sub-band F ^L using PCNN-based fusion algorithm in which sum of variation in squares (SVS) feature described in Eq. (14) is fed as external stimulus (S ^L _X , _{X=A or B} ) to the PCNN. The HF sub-bands A ^HF and B ^HF of the source images A and B, respectively, are combined using PCNN-based fusion algorithm in which coefficient absolute value (CAV) is fed as external stimulus (S ^HF _X , _{X=A or B} ) to the PCNN. PCNN-based fused algorithm is described in the following subsection. Inverse NSST of fused sub-bands gives the fused image (F).

Fig. 2

Schematic of proposed anatomical image fusion method

PCNN-Based Fusion Algorithm

• I. Initialize the PCNN: Feeding input F, linking input L, output Y, and time matrix T of PCNN shown in Fig. 1 initialized to zero. Threshold θ is initialised to one to avoid void iterations.

• II. Compute the linking strength β: At each coefficient of the sub-band, calculate the sum of directional gradients (SDG) feature using Eq. (8) and give it to PCNN as its linking strength β.

• III. Feed the PCNN: External stimulus S to the PCNN is made different for LF and HF sub-band fusion in view of type of information present in these sub-bands. LF sub-band is a smooth version of the image and it contains most of the signal energy. Hence, the sum of the variation in squares of the coefficients (SVS) is fed as external stimulus to PCNN-processing LF sub-band. SVS feature is calculated by using Eqs. (13) and (14).

  Let C(u, v) be LF sub-band coefficient at location (u, v). The variation in the square of the coefficients in 3 × 3 neighborhood is defined as (13).

  $$ \mathrm{v}\mathrm{s}\left[C\left(u,v\right)\right]={\displaystyle {\sum}_{m=-1}^1{\displaystyle {\sum}_{n=-1}^1\left|\left(C{\left(u,v\right)}^2-C{\left(u+m,v+n\right)}^2\right)\right|}} $$

  (13)

  Sum of variation in squares of coefficients at location (u, v) is defined in Eq. (14).

  $$ \mathrm{s}\mathrm{v}\mathrm{s}\left[fC\left(u,v\right)\right]={\displaystyle {\sum}_{i=-1}^1{\displaystyle {\sum}_{j=-1}^1w\left(i+1,j+i\right)\mathrm{v}\mathrm{s}\left[C\left(u+i,v+j\right)\right]}} $$

  (14)
  $$ w=\frac{1}{16}\left[\begin{array}{ccc}\hfill 1\hfill & \hfill 2\hfill & \hfill 1\hfill \\ {}\hfill 2\hfill & \hfill 4\hfill & \hfill 2\hfill \\ {}\hfill 1\hfill & \hfill 2\hfill & \hfill 1\hfill \end{array}\right] $$

  (15)

  HF sub-bands hold the detail information present in the image. Hence, coefficient absolute value (CAV) is fed as external stimulus to PCNN-processing HF sub-bands.

• IV. Obtain the time matrix T: Iterate each PCNN using Eqs. (1–5) until all neurons fire. Save the time matrices of PCNN-processing LF sub-bands A ^L and B ^L of source images A and B as T ^L _A and T ^L _B , respectively. Similarly, time matrices of PCNN-processing HF sub-band of rth scale and dth direction A ^r,d and B ^r,d of source images A and B as T ^r,d _A and T ^r,d _B , respectively.

• V. Derive the fused sub-bands: At each location of the sub-band, the source image coefficient which has fired earlier is selected. Fused LF sub-band F ^L is obtained by the Eq. (16).

  $$ {F}^L\left(i,j\right)=\left\{\begin{array}{cc}\hfill {A}^L\left(i,j\right)\hfill & \hfill \mathrm{if}\ {T}_A^L\left(i,j\right)\le {T}_B^L\left(i,j\right)\hfill \\ {}\hfill {B}^L\left(i,j\right)\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right. $$

  (16)

  Fused HF sub-band of rth scale and dth direction is obtained by the Eq. (17).

  $$ {F}^{r,d}\left(i,j\right)=\left\{\begin{array}{cc}\hfill {A}^{r,d}\left(i,j\right)\hfill & \hfill \mathrm{if}\ {T}_A^{r,d}\left(i,j\right)\le {T}_B^{r,d}\left(i,j\right)\hfill \\ {}\hfill {B}^{r,d}\left(i,j\right)\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right. $$

  (17)

Functional and Anatomical Image Fusion

Functional image is a low-resolution pseudocolor image in which color of the image holds the most vital information such as metabolic activity or blood flow etc., depending on the organ being imaged. However, anatomical image is a high-resolution grayscale image that gives the structural information. Fused functional and anatomical should have the same color as that of the functional image and all the details of anatomical image. One of the promising approaches to achieve this is fusion in de-correlated color model. So far, nonlinear IHS (NIHS) color model [30] and lαβ [31] color models have been used for medical image fusion application. NIHS color model introduces new colors in the fused image, but it is able to preserve the anatomical details with good contrast. lαβ color model has better de-correlation of chromatic and achromatic information, thereby, it does not incur any spectral distortion or change in color information. However, it does not provide good contrast for spatial details. To get better spectral and spatial characteristics, improved nonlinear IHS color model (INIHS) [32] is adapted in the present work for the first time for medical image fusion application. INIHS color model does not incur color gamut problem after fusion with an anatomical image. In INIHS color model, all the desired intensity changes can be achieved with the contrast being well maintained by adjusting the saturation to be within the maximum attainable range.

The proposed functional and anatomical image fusion technique is depicted in Fig. 3. The functional image which is in RGB color model is converted to INIHS color model. Thereby, we get intensity component (i), hue component (h), and saturation component (s). Before fusion of the intensity component with anatomical image, it is very important to normalize the intensity range of anatomical image (A) with that of the intensity component (i) of functional image. If A is an anatomical image, then normalized anatomical image A _new is given by Eq. (19).

Fig. 3

Schematic of proposed functional and anatomical image fusion frame work

$$ {A}_1=\left(A-\mathrm{minimum}\ \mathrm{of}\ A\right)/\left(\mathrm{maximum}\ \mathrm{of}\ A-\mathrm{minimum}\ \mathrm{of}\ A\right) $$

(18)

$$ {A}_{\mathrm{new}}={A}_1\left(\mathrm{maximum}\ \mathrm{of}\ i-\mathrm{minimum}\ \mathrm{of}\ i\right)+\mathrm{minimum}\ \mathrm{of}\ i $$

(19)

This new anatomical image A _new is fused with intensity component by the proposed anatomical image fusion method to obtain i _new containing all the details of anatomical image. The i _new, the original hue (h), and saturation (s) components converted to RGB color model give the fused image (F). Detail procedure of converting RGB color model to INIHS color model and INIHS color model to RGB color model is available in [32].

Results and Discussions

All simulations are done in Matlab7.1 on PC with Intel core2 Duo 3 GHz. In the proposed method, a number of NSST decomposition levels are set to 3 with 4, 8, and 8 directions from coarse to fine scale. The shearing filter width is set to 8, 16, and 16. The “pyr” filter is used as pyramidal filter. All the PCNN parameters are set as discussed in section 2. Extensive experiments have been conducted on 9 sets of CT-MRI images and 5 sets of SPECT-MRI neurological images of different pathologies to test the performance of the proposed method and its extension to functional-anatomical image fusion. All these pre-processed and registered datasets have been collected from http://www.med.harvard.edu/aanlib/home.html. Both quantitative and qualitative analysis has been done on experimental results. This section discusses the methods used for comparison, performance indexes used for quantitative evaluation, CT-MRI image fusion results, and SPECT-MRI image fusion results.

Methods Used for Comparison

The proposed method denoted as M6 is compared with the following five latest methods. NSST-based method with non-classic receptive field-based fusion rule [18] denoted as M1. Improved intersecting cortical model-based fusion rule adapted in NSST domain [26] denoted as M2. Energy and contrast-based fusion rules utilized in NSST domain [16] denoted as M3. An improved dual-channel PCNN-based fusion rule adapted in NSCT domain [25] denoted as M4. The spatial frequency-linked PCNN in NSST domain [22] denoted as M5. The parameter setting of these methods is done as per the details given by the corresponding references.

In order to prove the effectiveness of the improved nonlinear IHS (INIHS) color model, the proposed image fusion method for functional and anatomical images, is compared with the following NIHS color model-based methods. A simple NIHS color model-based method in which intensity component of functional image is replaced with anatomical image [30] denoted as m1. Image fusion method of references [16, 22, 25, 26] in NIHS color model domain denoted as m2, m3, m4, and m5, respectively. The proposed functional-anatomical image fusion method is denoted as m6.

Performance Indexes

• I. Mutual information based index (MI): Mutual information between fused image and the two source images is calculated and then added [14].

• II. Spatial frequency (SF): Spatial frequency [14] of fused image asses its activity level.

• III. Standard deviation (STD): Standard deviation of fused image measures its clarity.

• IV. Edge information-based quality index (Qab/f) [33]: Xydias and Petrovic proposed an image fusion quality index that quantifies the amount edge information transferred from source images to the fused image [33]. In case of anatomical image fusion (CT-MRI image fusion), Qab/f is calculated between fused image and the two anatomical images. In case of functional-anatomical image fusion, it is calculated for each spectral band of the functional image separately and then averaged.

• V. Q, Qw, and Qe: Gemma Piella et al. [34] proposed three image fusion quality indexes based on universal image quality index [35]. These quality indexes quantify the similarity without structural distortion between the fused and the original images. These have a dynamic range of 0 to 1.

• VI. Bias: Bias quantifies the spectral distortion in case of functional and anatomical image fusion. Bias is the absolute difference in the means of functional image and fused image in all spectral bands relative to the mean of the functional image. The less value of bias implies less spectral distortion and better fusion quality.

  $$ \mathrm{Bias}=\frac{1}{3}{\displaystyle \sum_{k=1}^3\frac{abs\left(\overline{f_k}-\overline{F_k}\right)}{\overline{f_k}}} $$

  $\overline{f_k}$, and $\overline{F_k}$ are the mean values of kth spectral band of functional image and fused image, respectively.

• VII. Filtered correlation coefficient (FCC) [36]: This quality index quantifies the anatomic detail information in functional and anatomical fused image. Fused and anatomical images are high pass filtered using laplacian filter mask. Correlation coefficient between each filtered fused image spectral band and filtered anatomical image is calculated, and then averaged to get filtered correlation coefficient (FCC) or high correlation coefficient [36]. Range of FCC is 0 to 1.

CT-MRI Image Fusion Experimental Results

Image fusion methods are tested on nine CT-MRI image pairs of different cases. Case 1 is an acute stroke case is shown in Fig. 4. Patient of this case had difficulty in reading. Cases 2 to 9 are shown in Fig. 5. Case 2 is an acute stroke case presenting as speech arrest. The MRI image of this case reveals infract having ribbon like geometry. Case 3 is hypertensive encephalopathy. Case 4 is a multiple embolic infractions case. Case 5 is a fatal stroke case in which enlarged right pupil can be seen. Case 6 is a metastatic bronchogenic carcinoma, and the MRI image of this case reveals a large mass with surrounding edema in left temporal region. Cases 7, 8, and 9 are meningioma, sarcoma, and cerebral toxoplasmosis cases, respectively. All these nine image pairs are of size 256 × 256 and are in 256 grayscale format.

Fig. 4

CT and MRI image fusion results of case 1: a CT image, b MRI image. Fused image by c M1 [18], d M2 [26], e M3 [16], f M4 [25], g M5 [22], and h the proposed method M6

Fig. 5

CT and MRI image fusion results of cases 2 to 9: a CT images, b MRI images. Fused images by c M1 [18], d M2 [26], e M3 [16], f M4 [25], g M5 [22], and h the proposed method M6

Visual Evaluation of CT-MRI Fusion Results

Visual evaluation of CT-MRI fusion results is performed with respect to the following aspects. Ability of fused image to retain hard tissue information present in the CT image and soft tissue information present in the MRI image with the same contrast as that of the source images. Structural distortion introduced into the fused image is tested. Ability of fused image to present the pathology of the source images clearly without any distortion is observed. The usefulness of fused image for clinical applications is evaluated. Fused images of acute stroke case that is case 1 by all the six methods including proposed method is presented in Fig. 4. The visual inspection of fused images reveals the following facts. Fused image by M1 shown in Fig. 4c retained the details of CT and MRI images. However, contrast is poor and few structures are darkened. Fused image by M2 shown in Fig. 4d is able to hold all the details of MRI image, but hard tissues present in CT image are lost. Fused image by M3 shown in Fig. 4e retained the CT image details clearly, but it has not maintained the desired contrast for soft tissues of MRI image. Fused image by M4 shown in Fig. 4f has the details of CT and MRI image. However, both details are blurred. Even though, fused image by M5 shown in Fig. 4g maintains good contrast for soft and hard tissues of MRI and CT; there are discrepancies in few tissues or structures compared to the original images. Fused image by the proposed method M6 shown in Fig. 4h presented all the details of MRI and CT images clearly without any type distortions. The pathology which is infarct in this case is presented clearly in the fused image by M6. Similar performance is perceived for the remaining 8 cases (cases 2 to 9) shown in Fig. 5.

Quantitative Evaluation of CT-MRI Image Fusion Results

Quantitative comparison of image fusion methods for all 9 cases of CT-MRI images is reported in Fig. 6. The mutual information based quality index (MI) [14] indicates the amount of information contained in the fused image about the source images. The larger the value of MI indicates better fusion quality. Comparison with respect to MI is depicted in Fig. 6a. For most of the cases, proposed method M6 exhibited better performance compared to all other methods with respect to MI. The method M1 showed poor performance with respect to MI. Spatial frequency (SF) index specifies the overall activity of fused image. The proposed method M6 showed superior performance with respect to SF by giving the highest value of SF for all cases compared to other methods as evident from Fig. 6b. M2 showed poor performance in terms of SF.

Fig. 6

Quantitative evaluation of CT-MRI image fusion results with respect to a mutual information-based index (MI), b spatial frequency (SF), c standard deviation (STD), and d edge information-based index (Qab/f) [33]. UIQI-based indexes [34] e Q, f Qw, and g Qe

The standard deviation (STD) index reveals the clarity of the fused image and the comparison with respect to STD is reported in Fig. 6c. The maximum value of STD is achieved by M6 for all the cases. This implies that the over clarity of fused images by the proposed method is better compared to that of other methods. The edge information-based quality index Qab/f represents the amount of edge details carried from source images to the fused image. It has a dynamic range of 0 to 1. The value 0 indicates the complete loss of edge information. The comparison with respect to Qab/f [33] is shown in Fig. 6d. For most of the cases, proposed method achieved the maximum value compared to other method. This implies that the fused images by proposed have maximum edge information compared to that of other methods. Comparison with respect to UIQI-based indexes Q, Qw, and Qe [34] is shown in Fig. 6e–g, respectively. All these three indexes have dynamic range of 0 to 1. As quality indexes reach 1, fusion quality would be the best. The index Q reveals the amount of salient information transferred from source images without structural distortion in the fused image. Method M4 and M6 showed better performance with respect to this index. The quality index Qw measures the amount salient information by giving more weightage to the perceptually important regions in the image. The quality index Qe gives importance to edge information. The proposed method M6 showed superior performance by achieving the maximum value of Qw and Qe for all cases compared to other methods. The average value of each quality index calculated by considering 9 CT-MRI cases is shown in Table 1. The best value of each quality index is highlighted. From this analysis, it is evident that the proposed method M6 gives superior performance compared to other methods.

Table 1 Quantitative comparison of CT-MRI image fusion results by different image fusion methods

SPECT-MRI Image Fusion Results

The proposed functional-anatomical image fusion method is tested on 5 SPECT-MRI cases. The first case which is a sub-acute stroke case is reported in Fig. 7. The SPECT image of this case presented in Fig. 7b shows hyper-perfusion in the right posterior parietal. Remaining cases 2 to 5 are presented in Fig. 8a, b. Case 2 is a stroke (aphasia) case. In the MRI image of this case, there are several areas of increased signal abnormality. SPECT image of this case showed the increased uptake of Thallium radioactive tracer representing the acute infraction in the left middle cerebral artery. Case 3 is cavernous hemangioma case in which SPECT images were made with Technetium-labeled red blood cells. Mixed signal in the MRI image of this case represents angiomas. Case 4 is anaplastic astrocytoma case. High tumor Thallium uptake in SPECT image indicates left parietal anaplastic astrocytoma. Case 5 is astrocytoma case in which MRI image shows cystic elements in a left occipital region, and SPECT image obtained with Thallium radioactive tracer shows an anterior border of high uptake. The visual and quantitative analysis of SPECT-MRI fusion results is discussed in the following subsections.

Fig. 7

SPECT-MRI fusion results of case 1: a MRI image, b SPECT image. Fused image by c m1 [30], d m2 [26], e m3 [16], f m4 [25], g m5 [22], and h the proposed method m6

Fig. 8

SPECT-MRI fusion results of cases 2 to 5: a MRI images, b SPECT images. Fused images by c m1 [30], d m2 [26], e m3 [16], f m4 [25], g m5 [22], and h proposed method m6

Visual Analysis of SPECT-MRI Fusion Results

SPECT image is a low-resolution pseudocolor image. Different levels of uptake of radioactive tracer are represented with different colors in SPECT image. MRI image is a high-resolution grayscale image presenting the anatomical details. Fused image of SPECT and MRI image should retain all the anatomical details of MRI image without altering the functional content that is without changing the color of SPECT image. Any change in the color of fused image compared to functional image is known as spectral distortion and loss of anatomical details is known as spatial distortion. Fused images of case 1 by different methods are illustrated in Fig. 7. Spectral distortion is evident in the fused images by methods m1 to m5 presented in Fig. 7c–g. The hyper-perfused region is changed to black color in fused images by m1 and m2, which is originally in white color in SPECT image. Non-functional region is presented as green color in fused images by m1 to m5. Because, methods m1 to m5 are based on NIHS color model, in which the changes made in intensity, the component during fusion process affects the spectral content. However, in the fused image by proposed method m6 presented in Fig. 7h retained the functional information. Anatomical details in fused images by m4 and m5 are smoothed. Proposed method m6 presented the anatomical details with good contrast. Fused images of cases 2 to 5 are presented in Fig. 8. Fused images by methods m1 to m5, spectral distortion is evident in non-functional area of SPECT images, whereas, in fused image by proposed method m6 shown in Fig. 8h, pathological tissues and other tissues are presented clearly.

Quantitative Analysis of SPECT-MRI Fusion Results

Quantitative analysis of SPECT-MRI image fusion for the 5 cases is presented in the Table 2. The best value of each quality index in each case is highlighted. To assess the spectral distortion incurred by various image fusion methods, bias index is adapted. Bias measures the absolute difference in the means of fused image and functional image relative to the mean of functional image. Hence, low value of bias indicates that the spectral distortion is low and fusion quality is better. Minimum value of bias is achieved by proposed method m6 for cases 1 to 4. This implies that spectral distortion is less in proposed method compared to other methods. Spatial distortion is quantified by filtered correlation coefficient (FCC) and edge information-based performance index (Qab/f). Proposed method achieved highest values for these quality indexes compared to other methods. This indicates that anatomical details of MRI images are better preserved in proposed compared to others.

Table 2 Quantitative analysis of SPECT-MRI fusion results by different methods

Conclusions

Multimodality medical image fusion method utilizing simplified adaptive PCNN model in NSST domain is proposed. The high directional sensitivity and shift-invariance property of NSST helps in extracting more directional information without introducing artifacts. Directional gradient feature at each location is used as the linking strength of the PCNN at corresponding locations. The adaptive PCNN-processing low-frequency and high-frequency sub-bands are motivated by different features consistent with the characteristics of corresponding sub-bands. The proposed method is compared with five state-of-the-art image fusion methods. A set of nine pairs of CT and MRI neurological images of different pathologies is used for testing the proposed method. Visual analysis of CT and MRI image fusion results proved that the proposed method is able to retain the salient information of CT and MRI images with good contrast compared to other methods. Quantitative comparison with respect to mutual information-based quality index, spatial frequency, standard deviation, edge information-based quality index, and UIQI-based quality indexes proved the superiority of proposed method over other methods.

The proposed method is extended for functional and anatomical image fusion in improved nonlinear IHS (INIHS) color model. INIHS color model does not incur color gamut problem and this helps in retaining the spectral information of functional image in the fusion process. The proposed functional and anatomical image fusion method is compared with five other methods. Set of five SPECT and MRI neurological images is used for testing. Visual analysis of SPECT and MRI fusion results reveal that the fused images by proposed method retain the color information of original images both in pathological regions and normal regions. In case of other methods, hyper-perfused areas and non-functional regions of SPECT images are changed to different colors, which hampers their practical utility for the diagnosis and treatment planning. The quantitative spectral and spatial distortion measures evidenced the superior performance of proposed functional and anatomical image fusion method over other methods. However, the proposed method is tested only on different neurological disorders. Because of the usage of the PCNN in the proposed method, computational complexity is slightly higher.