Multimodal non-rigid image registration based on elastodynamics

Abstract

In this paper, we present a new multimodal image registration technique established on elastodynamics notion. The main idea behind this concept is the progression of waves on an elastic body as soon as it is disturbed from its initial rest state. We propose to solve the multimodal registration problem by modeling the non-linear deformations as elastic waves and iteratively solving the elastodynamics wave equation to estimate the transformation. The inertial force in elastodynamics model is computed as the gradient of mutual information which considers the statistical relationship between the intensities of the images acquired using different imaging modalities. We tested our method on T1–T2 weighted MR brain image pairs and MR-CT brain image pairs. The proposed registration technique was compared against a variant of demons method proposed for multimodal images. The registration results were analyzed by examining the overlay images and by computing the normalized mutual information. The qualitative and quantitative analysis proved that our proposed method registers the images better than the compared method.

1 Introduction

Multimodal non-rigid image registration has attracted numerous research scientists since past few decades. It has widespread applications in medical image analysis and diagnostic procedures [1]. It helps in integrating complementary information obtained from different imaging modalities, thus facilitating therapy planning [2, 3]. Multimodal image registration is a difficult task as compared to monomodal image registration. The crucial problem of multimodal image registration lies in the non-linear intensity relationship of the images to be registered. However, for monomodal image registration, this problem is alleviated as the images have linear intensity mapping. Due to the complex relationship between the intensities of the two images, the similarity measures such as mean squared difference or normalized cross correlation do not give favorable registration results. For this reason, information-theoretic based similarity measures are used [4, 5] which could tackle the multimodal image registration very well. Mutual Information (MI) was incorporated into the viscous fluid model by defining the force field as the gradient of MI [6]. MI has also been used within diffeomorphic demons framework to register multimodal brain images [7, 8]. Another approach has been proposed by Orchard [9], where the joint intensity scatter plots (JISP) clusters are fitted with regressors to iteratively compute the deformation. Vegh et al. [10] proposed an optimization method based on stochastic differential equation for multimodal images. Optical flow technique along with Shannon entropy was proposed by Reducindo et al. [11] to solve multimodal image registration problem. Recently, spatial information was added into MI as similarity measure. The spatial information was obtained by segmenting the liver [12]. Similarly, point-wise MI was proposed by Lu et al. [13]. It included tissue class probability as prior information with diffeomorphic demons as transformation model. A Tsallis entropy based MI method was proposed by Martin et al. [14]. They combined this technique with simultaneous perturbation stochastic approximation (SPSA) algorithm. Khader et al. [5] proposed a similar approach where they optimized Jensen–Tsallis divergence via modified SPSA algorithm. Sabuncu and Ramadge [15] proposed to solve multimodal image registration problem using a minimal spanning graph with Jensen–Renyi divergence.

In this paper, we propose to model the non-linear deformations as elastic waves. We introduce the elastodynamics model (proposed previously by us for monomodal image registration [16]) into the multimodal registration scenario. For monomodal registration, the inertial force was modeled as the gradient of sum of squared difference of intensities [16]. Here, we tend to estimate the inertial force as the gradient of MI [17]. This information-theoretic based force field accounts for the non-linear mapping of image intensities, thus enabling the recovery of large variations among subjects.

The paper is organized as follows: Sect. 2 presents the material used for non-rigid registration experiments. The proposed registration scheme is described in Sect. 3. The results achieved by the proposed method and the compared registration scheme are discussed in Sect. 4. The paper concludes in Sect. 5.

2 Materials

The performance of the proposed multimodal non-rigid image registration was investigated on 2D MR brain images available from BrainWeb Simulated Brain Database¹ [18]. Two 2D axial slices of T2 weighted MR images were selected as reference images. The dimension of each slice was 217$\times$181 pixels with in-plane resolution of 1 $\mathrm {mm^{2}}$, $3~\%$ noise level and an intensity non-uniformity (INU) of $20~\%$. 30 T1 weighted 2D axial slices were selected as subject images, each having a dimension of 217$\times$181 pixels with in-plane resolution of 1 $\mathrm {mm^{2}}$, different noise levels (i.e. $0, 1, 3, 5, 7, 9~\%$) and an INU of $20~\%$. The subject images also underwent simulated deformations to ensure non-rigid differences between the reference and subject images. Figure 1a shows the two reference images and Fig. 1b shows the two synthetically deformed subject images.

Fig. 1

2D axial slices of two a T2 weighted reference images and b T1 weighted subject images with synthetic non-rigid deformations obtained from BrainWeb database

The second dataset comprised of 20 2D MR T1-T2 weighted brain image pairs obtained from Retrospective Image Registration Evaluation Project (RIRE) database.² The T2 and T1 weighted images served as reference and subject images, respectively. Each image had 256$\times$256 pixels. All the subject images had real non-rigid differences. Figure 2a shows the reference image and Fig. 2b shows the subject image.

Fig. 2

2D axial slices of a T2 weighted reference image and b T1 weighted subject image with real non-rigid deformations obtained from RIRE database

Image registration experiment was also performed on 20 2D MR-CT brain image pairs obtained from RIRE database.$^{2}$ CT images were used as reference images with 147$\times$188 pixels as shown in Fig. 3a. MR images were used as subject images with 256$\times$256 pixels as shown in Fig. 3b.

Fig. 3

2D axial slices of a CT reference image and b MR subject image with real non-rigid deformations obtained from RIRE database

3 Methods

3.1 The elastodynamics model

When an elastic body, initially at rest, is disturbed from its equilibrium state, then waves begin to travel across the body [19]. Let us assume that the Cauchy stress tensor is represented by $\mathbf {\sigma }$ and strain tensor is represented by $\mathbf {\epsilon }$. The relationship between strain and displacements $\mathbf {u}=\left[ u_{1}=u_{x},u_{2}=u_{y}\right]$ is given by [20];

$$\begin{aligned} \mathbf {\epsilon }_{ij}=\frac{1}{2}\left[ \frac{\partial }{\partial x_{i}}u_{j}+\frac{\partial }{\partial x_{j}}u_{i}\right] , \end{aligned}$$

(1)

where $i=j=1,2$ for the two dimensions and $[x_{1}=x, x_{2}=y]$ denotes the two dimensional Cartesian coordinates. According to Hooke’s Law, stress is proportional to strain and material properties as follows [21]:

$$\begin{aligned} \mathbf {\sigma }=C\mathbf {\epsilon }, \end{aligned}$$

(2)

where $C=C_{ijkl}$ is a stiffness tensor of fourth rank. The inertial force $\mathbf {f}=[f_{1}=f_{x},f_{2}=f_{y}]$ per unit volume is given by the divergence of stress tensor;

$$\begin{aligned} f_{x}= & {} \frac{\partial }{\partial x}\sigma _{11}+\frac{\partial }{\partial y}\sigma _{12} \end{aligned}$$

(3)

$$\begin{aligned} f_{y}= & {} \frac{\partial }{\partial x}\sigma _{21}+\frac{\partial }{\partial y}\sigma _{22}, \end{aligned}$$

(4)

The equation of motion is thus written as;

$$\begin{aligned} {\rho \frac{{\partial ^{2} u_{i} }}{{\partial t^{2} }}}&= f_{i} + \sum \limits _{j} \frac{\partial }{{\partial x_{j} }}\left\{ {C_{{ijkl}} \frac{1}{2}\left( {\frac{\partial }{{\partial x_{i} }}u_{j} + \frac{\partial }{{\partial x_{j} }}u_{i} } \right) } \right\} , \end{aligned}$$

(5)

where $\rho$ stands for material density. The matrix C for isotropic elastic solids is:

$$\begin{aligned} C=\begin{bmatrix} \lambda + 2\mu&\quad \lambda&\quad \lambda&\quad 0&\quad 0&\quad 0\\ \lambda&\quad \lambda + 2\mu&\quad \lambda&\quad 0&\quad 0&\quad 0\\ \lambda&\quad \lambda&\quad \lambda + 2\mu&\quad 0&\quad 0&\quad 0\\ 0&\quad 0&\quad 0&\quad 2\mu&\quad 0&\quad 0 \\ 0&\quad 0&\quad 0&\quad 0&\quad 2\mu&\quad 0 \\ 0&\quad 0&\quad 0&\quad 0&\quad 0&\quad 2\mu \end{bmatrix}, \end{aligned}$$

(6)

where $\mu$ and $\lambda$ are Lamé’s material parameters. Now the equation of motion can be written as a hyperbolic partial differential equation referred to as elastodynamics wave equation:

$$\begin{aligned} \rho \frac{\partial ^{2}\mathbf {u}}{\partial {t^{2}}}=\mu \nabla ^{2}\mathbf {u}+(\mu + \lambda )\nabla \left( \nabla .\mathbf {u}\right) +\mathbf {f}. \end{aligned}$$

(7)

3.2 Non-rigid registration formulation

Let $I_{1}$ be the reference image and let $I_{2}$ be the subject image with $\mathbf {p}=\left[ x,y\right]$ being the image pixel coordinates. The subject image is deformed towards the reference image with the transformation $\mathbf {\tau }=\left[ \tau _{x},\tau _{y}\right]$ where $\tau _{x}=x+u_{x}$ and $\tau _{y}=y+u_{y}$. The deformation field $\mathbf {u}$ is obtained using the elastodynamics model, where the image is modeled as an elastic body and the non-rigid deformations are modeled as elastic waves. The non-linear deformations are recovered as the waves progress over the body under consideration. The deformation field is obtained by solving Eq. (7) using finite difference approximation method given as follows:

$$\begin{aligned} \Bigg [\begin{array}{c} u_{x}^{(t+1)}\\ u_{y}^{(t+1)} \end{array}\Bigg ]&= \frac{1}{\rho }\Bigg \{\nabla ^{2}\mathbf {u}^{(t)}\Bigg [\begin{array}{c} 1\\ 1 \end{array}\Bigg ]+(\mu +\lambda )\Bigg [\begin{array}{c} \nabla (\nabla \cdot \mathbf {u}^{(t)})\big |_{x}\\ \nabla (\nabla \cdot \mathbf {u}^{(t)})\big |_{y} \end{array}\Bigg ]\nonumber \\&\quad +\Bigg [\begin{array}{c} f_{x}^{(t)}\\ f_{y}^{(t)} \end{array}\Bigg ]+\Bigg [\begin{array}{c} u_{x}^{(t)}-u_{x}^{(t-1)}\\ u_{y}^{(t)}-u_{y}^{(t-1)} \end{array}\Bigg ]\Bigg \}, \end{aligned}$$

(8)

The inertial force ($\mathbf {f}$) is the main driving component of the whole registration procedure. This inertial force depends on the similarity criterion used for the purpose of registration. Since, we are dealing with multimodal images, so mutual information (MI) would serve the purpose. Our inertial force is defined as the gradient of MI as derived by Hermosillo et al. [17] given below:

$$\begin{aligned} \mathbf {f}=\nabla \mathrm {MI}\big |_{\mathbf {u}}, \end{aligned}$$

(9)

where,

(10)

where r corresponds to the intensity value in $I_{1}$ and s corresponds to the intensity value in deformed subject image $I_{2}(\mathbf {\tau })$. $p_{\mathbf {u}}(r,s)$ is the joint intensity probability and p(r) and $p_{\mathbf {u}}(s)$ are the marginal intensity probabilities. Eq. (9) expands as follows:

(11)

(12)

The joint intensity probability $p_{\mathbf {u}}(r,s)$ is determined using the Parzen window kernel [6] $W_{h}(r,s)$ of width h. The inertial force is thus given as follows:

$$\begin{aligned} \mathbf {f}=\frac{1}{V}\Bigg [\frac{\partial W_{h}}{\partial r}L_{\mathbf {u}}\Bigg ](I_{2}(\mathbf {\tau }),I_{1}(\mathbf {p}))\nabla I_{2}(\mathbf {\tau })\Big |_{\mathbf {\tau }}, \end{aligned}$$

(13)

where V is the number of pixels in the overlap region between the reference and deformed subject image. This inertial force would be initially small due to substantial variations amongst the reference and subject images as can be seen in Fig. 4a. Its magnitude increases gradually (Fig. 4b, c) until the subject image matches well with the reference image. Thereafter, the inertial force remains constant with no further improvement in alignment of the images. Thus, the registration process comes to an end.

Fig. 4

(Color images) Inertial force field at a start, b middle and c end of registration procedure. Magnitude of inertial force is given by the corresponding color coding scheme

3.3 Implementation Details

We start by initializing the deformation field ($\mathbf {u}$), velocity field ($\dot{\mathbf {u}}$) and acceleration field ($\ddot{\mathbf {u}}$) for the first two time instants to be equal to zero. Initialize the material parameters as follows:

$$\begin{aligned} \mu =0.20,\;\;\rho =20,\;\;\lambda =0 \end{aligned}$$

The initial values of the parameters $\mu ,\rho ,\lambda$ were determined using two-fold cross validation method. This procedure involved breakdown of the dataset into training and test sets. Images in training dataset were used to determine the values of these parameters. The values which gave minimum error were selected and then test set was used to validate these values. For joint intensity probability estimation we use the Gaussian function as Parzen window kernel of width $h=1$ defined below:

$$\begin{aligned} W_{h}(r,s)=\frac{1}{2\pi h^{2}}\mathrm {exp}\Bigg (\frac{-r^{2}}{2h^{2}}\Bigg )\mathrm {exp}\Bigg (\frac{-s^{2}}{2h^{2}}\Bigg ) \end{aligned}$$

(14)

The Parzen window kernel defined in Eq. (14) is then convolved with joint histogram ($H_{\mathbf {u}}(r,s)$) of the two images with the bin size $N=128$.

$$\begin{aligned} p_{\mathbf {u}}(r,s)=W_{h}(r,s)*H_{\mathbf {u}}(r,s) \end{aligned}$$

(15)

The deformation field is obtained by iteratively solving Eq. (8) until the average value of the magnitude of inertial force becomes greater than the pre-defined threshold or the algorithm reaches the maximum number of iterations. This leads us to stage 2 of registration procedure where the value of $\mu$, $\rho$ and h are now reduced to continue estimating the deformation field. As soon as the change in inertial force starts diminishing, the registration process stops and the subject image is deformed with the final deformation field obtained.

4 Results

We performed the registration experiments with three different datasets. Images obtained from BrainWeb database had simulated non-rigid deformations and images obtained from RIRE database had real non-rigid deformations. We compared our method with the technique proposed by Kroon et al. [22]. In this method the classical Demons forces were modified to work for multimodal images. We used the MATLAB implementation of this method. For validation purpose, we used overlay image display as a qualitative tool and normalized mutual information (NMI) as a quantitative measure.

Figure 5 (top row) shows the overlay images of reference and subject images obtained from BrainWeb database while the two columns depict two different subject images. Figure 5 (middle row) shows the overlay images of reference and corresponding deformed subject image after Kroon’s method and Fig. 5 (bottom row) shows the registration result of our proposed elastodynamics based method. By comparing the registration results of the two methods, it could be concluded that our proposed method gives better registration results than the Kroon’s method. Our method completely aligned the subject image as it is notable around the lateral ventricles whereas Kroon’s method did not align the images well. Registration result for second dataset is shown in Fig. 6. Overlay image of reference (T2 MR) and subject (T1 MR) image is shown in Fig. 6a. Overlay images of reference and deformed subject image obtained after Kroon’s and elastodynamics based registration could be seen in Fig. 6b, c, respectively. As evident from results, the elastodynamics based method recovered real non-rigid deformations as well whereas Kroon’s method did not perform well. Figure 7a shows the overlay image of the reference (CT) and subject (MR) image obtained from RIRE database. The overlay image of reference and deformed subject image obtained with Kroon’s method is shown in Fig. 7b, c shows elastodynamics based registration result. The result obtained with our proposed method is much more aligned as compared to Kroon’s method.

Fig. 5

Overlay image of the reference and subject image (top row) before registration, (middle row) after Kroon’s registration [22] and (bottom row) after elastodynamics registration. The columns depict the subject images with two different deformations

Fig. 6

Overlay image result a before registration, b after Kroon’s registration [22] and c after elastodynamics registration for the reference and subject image obtained from RIRE database. Note that these images had real non-rigid deformations

Fig. 7

Overlay image result a before registration, b after Kroon’s registration [22] and c after elastodynamics registration for the reference (CT) and subject (MR) image obtained from RIRE database

Figure 8 (left column) shows the joint histogram of reference and subject image (BrainWeb dataset) before registration and Fig. 8 (right column) shows the same after registration with our proposed method. It could be noted that the histogram is dispersed initially and after registration it accumulates around the identity line. Table 1 gives the statistical summary of the values of NMI before registration and after Kroon’s and elastodynamics based registration method for all three datasets. It could be observed that our method achieved higher NMI values as compared to Kroon’s method for all the registration experiments. The overall mean ± SD of NMI achieved with Kroon’s method being 1.166 ± 0.101 and with elastodynamics method being 1.219 ± 0.062. Thus, Kroon’s method resulted in percentage improvement of 0.49 % and elastodynamics method showed 5.06 % improvement.

Fig. 8

Joint histogram of reference and subject image (BrainWeb dataset) (left column) before registration and (right column) after registration using elastodynamics model

Table 1 Statistical summary ($\mathrm{mean}\,\pm \,\mathrm{SD}$) of normalized mutual information (NMI) obtained with Kroon’s [22] and elastodynamics registration methods

5 Conclusion

This paper presented a new non-rigid registration approach for multimodal images. We proposed to use the elastodynamics model as the transformation function to obtain the underlying deformation field. The inertial force was modeled as the gradient of mutual information to deal with the statistical relationship between the intensities of underlying images. The proposed method was tested on three different datasets which included MR (T1, T2 weighted) and CT images with real and synthetic non-linear deformations. The presented registration technique was compared against a variant of Demon’s registration method proposed for multimodal images [22]. The qualitative and quantitative results indicated that the proposed method recovers non-rigid deformations to quite a good extent and performs better than the compared method.