Next-generation prognosis framework for pediatric spinal deformities using bio-informed deep learning networks

Abstract

Predicting pediatric spinal deformity (PSD) from X-ray images collected on the patient’s initial visit is a challenging task. This work builds on our previous method and provides a novel bio-informed framework based on a mechanistic machine learning technique with dynamic patient-specific parameters to predict PSD. We provide a geometry-based bone growth model that can be utilized in a range of applications to enhance the bio-informed mechanistic machine learning framework. The proposed technique is utilized to examine and predict spine curvature in PSD cases such as adolescent idiopathic scoliosis. The best fit of a segmented 3D volumetric geometry of the human spine acquired from 2D X-ray images is employed. Using an active contour model based on gradient vector flow snakes, the anteroposterior and lateral views of the X-ray images are segmented to derive the 2D contours surrounding each vertebra. Using minimal user input, the snake parameters are calibrated and automatically computed over the dataset, resulting in fast image segmentation and data collection. The 2D segmented outlines of each vertebra are transformed into a 3D image segmentation result. The Iterative Closest Point mesh registration technique is then used to establish a mesh morphing approach and creates a 3D atlas spine model. Using the comprehensive 3D volumetric model, one can automatically extract spinal geometry data as inputs to the mechanistic machine learning network. Moreover, the proposed bio-informed deep learning network with the modified bone growth model achieves competitive or even superior performance against other state-of-the-art learning-based methods.Please check and confirm if the author names and initials are correct for “Yongjie Jessica Zhang” and “Wing Kam Liu”.We confirm they are correct.

Appendices

Appendix A: Image segmentation using the Snakes method

Image segmentation using the Snakes method [42] is a fast and efficient technique to detect important object contours from images. The limitation of this method is that it requires initialization of the contour to be done manually. The initial contour determines the accuracy of the segmentation method. The framework involves a variational formulation in which the total energy functional consists of three main terms, namely the image energy term which attracts the contour to the salient features in the image, internal spline energy term which introduces smoothness and regularity in the evolving contour and external constrain term which aligns the contour near local minima. $v(s)=(x(s),y(s))$ is the parametric representation of a snake where parameter $s \in [0,1]$. As s changes smoothly, a closed contour on a plane is traced. The total energy functional proposed in [42] considers both image and constraint forces is given as

$$\begin{aligned} E(\mathbf{v} (s))= & {} \int _0^1 \! E_{\text {int}}(\mathbf{v} (s)) + E_{\text {con}}(\mathbf{v} (s)) \nonumber \\&\quad + E_{\text {img}}(\mathbf{v} (s)) \, \mathrm {d}s, \end{aligned}$$

(A1)

where $E_{\text {int}}(\mathbf{v} (s))$, $E_{\text {img}}(\mathbf{v} (s))$ and $E{\text {con}}(\mathbf{v} (s))$ are the energy functionals associated with internal spline energy, image force and external constraint energy, respectively. The $E_{\text {int}}(\mathbf{v} (s))$ term is given as

$$\begin{aligned} \begin{aligned} E_{\text {int}}(\mathbf{v} (s)) = \alpha \left|\mathbf{v} '(s) \right|^2 + \beta \left|\mathbf{v} ''(s) \right|^2, \end{aligned} \end{aligned}$$

(A2)

where $\alpha$ and $\beta$ are weights associated with the first- and second-order regularization terms which are elastic length and stiffness of the contour. $E_{\text {img}}(\mathbf{v} (s))$ is defined as

$$\begin{aligned} \begin{aligned}&E_{\text {img}}(\mathbf{v} (s)) = {}\, w_{\text {line}} E_{\text {line}}(\mathbf{v} (s)) + w_{\text {edge}}E_{\text {edge}}(\mathbf{v} (s)) \\&\quad + w_{\text {term}}E_{\text {term}}(\mathbf{v} (s)), \end{aligned} \end{aligned}$$

(A3)

where $w_{\text {line}}$, $w_{\text {edge}}$ and $w_{\text {term}}$ are the weighting coefficients associated with the energy functionals $E_{\text {line}} = I(x,y)$, $E_{\text {edge}} = -|\bigtriangledown I(x,y)|^2$ and $E_{\text {term}} = \frac{C_{yy}C_{x}^2 - 2C_{xy}C_xC_y + C_{xx}C_{y}^2}{(C_x^2 + C_y^2)^{\frac{2}{3}}}$, I(x, y) is the image intensity, $C(x,y)=G_\sigma (x,y)*I(x,y)$ and $G_\sigma$ is a Gaussian of standard deviation $\sigma$ [42].

Appendix B: Point cloud registration

This section introduces the employed non-rigid ICP approach, where a concise description of the approach is given based on the conventional ICP algorithm [50].

1.1 B.1 The approach

In the registration process of the non-rigid ICP, the source surface $\mathcal {S} = (\mathcal {V}, \mathcal {E})$, consisting of n vertices in $\mathcal {V}$ and m edges in $\mathcal {E}$, is registered to the target surface $\mathcal {T}$ step by step. Fig. 17 illustrates a step of the registration process. In the figure, the meshes are assumed to be triangular meshes, and the vertices are labeled by numbers. In this step, first, the correspondences between vertices $v_i$ in the source surface $\mathcal {S}$ (green) and vertices $u_i$ in the target surface $\mathcal {T}$ (red) are established.

In the use of a conventional ICP method, given a point on $\mathcal {S}$, the closest point on $\mathcal {T}$ is considered as its corresponding point [59]. Then $v_i$ is transformed by locally affine transformations ($X_i$) towards the target surface $\mathcal {T}$ (red). The transformed source surface is $\mathcal {S}(X)$ (blue). This procedure iterates till an optimal stable state [60,61,62,63] is obtained.

Fig. 17

Match the source surface to the target surface [64]

1.2 B.2 3D mesh registration

Here, based on the established correspondences ($v_i, u_i$), a cost function consisting of different terms is defined and then minimized with guaranteed stability, convergence, and robustness [50]. In the following, we introduce each term in the cost function first, and then we describe the optimization process based on the cost function.

For a non-rigid registration, the distance of the deformed source and the target should be minimized. Thus, a distance term is selected as the first component of the cost function to be minimised as

$$\begin{aligned} E_d&= \sum _{v_i \in \mathcal {V}}^{} w_i \left|X_iv_i-u_i \right|^2, \end{aligned}$$

(B4)

where $w_i$ is the weight of the distance term and X describes a set of transformations of displaced source vertices $\mathcal {V}(X)$. The transformation matrix $X_i$ for each vertex in the source is a $3 \times 4$ transformation matrix as:

$$\begin{aligned} X_i =\left[ \begin{matrix} r_{xx} &{} r_{xy} &{} r_{xz} &{} d_{x}\\ r_{yx} &{} r_{yy} &{} r_{yx} &{} d_{y}\\ r_{zx} &{} r_{zy} &{} r_{zz} &{} d_{z} \end{matrix} \right] , \end{aligned}$$

(B5)

where r, and d define all afine transformations. The transformation matrix X of all vertices is described in a $4n \times 3$ matrix as $X =\left[ X_1 \ldots X_n\right] ^{\text {T}}$.

A canonical form of Eq. (B4) is addressed in Eq. (B6), introduced by swapping the position of transformation matrix, and correspondences $(v_i, u_i)$. The sparse matrix D is formed to facilitate the transformation of the source vertices with the individual transformations contained in X via matrix multiplication, and denoted as $D = diag(v_1^{\text {T}}, v_2^{\text {T}}, \ldots , v_n^{\text {T}})$. The corresponding points are also arranged as $U~=~\left[ u_1 \ldots u_n\right] ^{\text {T}}$ and the distance term can be derived as:

$$\begin{aligned} E_d = \left|W\left( DX-U\right) \right|_F^2 \end{aligned}$$

(B6)

where W is a diagonal matrix consisting of weights $w_i$. To regularise the deformation, an additional stiffness term is employed. Using the Frobenius norm $\left|. \right|_F$, the stiffness term penalizes difference of the transformations of neighboring vertices, through a weighting matrix $G = diag(1, 1, 1, \gamma )$. We have

$$\begin{aligned} E_s = \sum _{i,j \in \mathcal {E}}\left|\left( X_i-X_j\right) G \right|_F^2. \end{aligned}$$

(B7)

During the deformation, $\gamma$ is a parameter to stress differences in the skew and rotational part against the translational part of the deformation. The value of $\gamma$ can be specified based on data units and the types of deformation [50].

Addressing the function of the stiffness term to penalise differences of transformation matrices of the neighboring vertices, the node-arc incidence matrix M (e.g. Dekker [65]) of the template mesh topology is employed to convert the stiffness term functional into a matrix form. As the matrix is fixed for directed graphs, the construction is one row for each edge of the mesh and one column per vertex. To establish the node-arc incidence matrix of the source topology, the indices (i.e. the subscripts) of edges and vertices are addressed, for any edge of r which is connected to vertices (i, j) , in $r^{th}$ row of M, and the nonzero entries are $M_{ri} = -1$ and $M_{rj} = 1$. Therefore, we formulate the stiffness term as

$$\begin{aligned} E_s = \left|\left( M \otimes G\right) X \right|^{2}_{F}. \end{aligned}$$

(B8)

Briefly, the Amberg’s method accounts for an optimal step with non-rigid ICP approach being capable to employ different regularisations, while they are using a range of lowering stiffness parameter. Thus, the cost function of Eq. (B6) and Eq. (B8) are changed to:

$$\begin{aligned} E(X) = \left|\left[ \begin{matrix} \gamma M \otimes G\\ WD\end{matrix}\right] X-\left[ \begin{matrix} 0\\ WU\end{matrix}\right] \right|^{2}_{F}. \end{aligned}$$

(B9)