Workflow
The workflow in which our software is to be used is illustrated in Fig. 2. Our proof-of-concept software performs steps 3–4 and 9. Steps 2 and 8 are currently performed with human intervention, using external software.
Firstly, our software allows a surgeon to plan a cement injection procedure using a pre-operatively acquired CT as template. Secondly, our software can compute digitally reconstructed radiographs (DRRs) from a pre-operative CT [11]. DRRs are simulated 2D X-ray radiographs, computed from CT. DRRs can be used in guidance, or in estimating cement quantities. Thirdly, our software allows for a rough quantitative estimate of the resulting 3D cement distribution to be made. This is done by comparing the observed intraoperative fluoroscopic images to DRRs that are generated from the pre-operative CT volume.
When using HipRFX, the first step is to load a pre-operative CT volume of the affected hip. The user then specifies a 3D target “mask” that delineates the sub-volume(s) that should be filled with bone cement, i.e. osteolytic lesion(s). In our experiments we used the stand-alone MITK software [20] to perform these segmentations, but any medical volume segmentation tool may be used for this purpose. The segmented cement mask is then read by HipRFX as a binary image volume input file.
Once cement injection targets have been defined, the user has the possibility of adding and positioning virtual cement injection needles that access the designated target(s). If desired, additional segmented anatomical structures may be concurrently loaded and displayed.
From the moment that a CT volume is loaded, our software is able to interactively simulate fluoroscopic images based on a virtual C-arm that may be freely rotated. The goal of this simulation is to subsequently compare it with the actual intraoperative fluoroscopic images. Images may be simulated with or without designated needles and injected cement. By comparing actual intraoperative fluoroscopic images to these simulated images—both in the complete absence and in the complete presence of the intended cement filling—we generate 2D difference images between observed and simulated outcomes. These difference images are transformed to a 2D “cement filling” images that may be back-projected into the 3D volume to create a volumetric cement filling map. The algorithm’s sensitivity to image noise may be determined for each position in the cement filling image. Noise has the highest impact in regions where the image intensity difference between a low and high cement filling is small. This is further described in “Sensitivity of the output to image noise” section.
Software user interface
HipRFX is implemented as a module in the DeVIDE Runtime Environment [5] and makes extensive use of the Visualization ToolKit (VTK), NumPy and SciPy [27].
The functionality contained in HipRFX is split across four separate panels, as explained in Fig. 3. Each panel has its own distinct purpose. We now describe the functionality and underlying methods of each panel separately. In its current proof-of-concept form, the image registration and intensity matching is performed by an external tool we built with scikit-image [28].
Planning panel
The first view the user is presented with is the planning panel, shown in Fig. 4. As soon as a CT volume is loaded, it is rendered in a slice view and as a 3D volume. The panel contains two viewports showing 3D renderings of the hip’s bony structures from anterior-posterior (A) and medio-lateral (B) perspectives. To facilitate the positioning of the needles, the isovalues used for 3D rendering may be adjusted to provide the right amount of context. The central viewport contains a slice-based viewer that can be manipulated interactively (C). The user may load 3D segmented structures of interest. Examples include osteolytic lesions, or arteries and nerves that need to be avoided during surgery. Any number of cement injection needles can be virtually inserted into the CT volume and manipulated. The right-hand panel (D) contains controls to adjust the viewports’ display parameters, to perform distance and angle measurements, and to add, manipulate and remove needles.
DRR panel
The digitally reconstructed radiograph (DRR) panel shown in Fig. 5 allows a simulated fluoroscope to be interactively viewed and manipulated. The majority of the panel is taken up by the interactively rendered fluoroscopic image (A). Each image frame is created by a ray casting algorithm that directly simulates the propagation of X-rays through the patient [4]. The user may manually adjust the brightness and contrast to match the operating room fluoroscope’s settings (B), either to predetermined values obtained from calibration or visually.
An interactive 3D representation of the patient and the fluoroscopic C-arm is provided (C). The orientation of the C-arm and accompanying fluoroscopic view may be interactively manipulated to match the operating room’s set-up.
The image may be inverted to either emulate fluoroscopy or X-ray radiographs—the simulated X-ray attenuation algorithm is identical between these modes. This is an aesthetic choice to be made by the modality that the surgeon is most familiar with.
Needles or segmented cement targets that have been loaded in the planning view (Fig. 4) are realistically overlayed in the DRR image. In this way, the operator sees a simulation that corresponds to the fluoroscopic view he/she would see intraoperatively.
Snapshots can be saved to the step-by-step guidance panel described in the next subsection. With each snapshot, the C-arm orientation is stored as well. Snapshots may be used as a road map to guide the surgeon along planned steps during the execution of a minimally invasive procedure.
Step-by-step guidance panel
In the DRR panel shown in Fig. 5, the operator is allowed to store a number of DRR snapshots for later reference. Snapshots are displayed in a grid view, and clicking on any snapshot recalls the corresponding C-arm orientation that was used. These snapshots can act as an intraoperative road map since at any time, the operator may visually compare them to the live fluoroscopy image.
Changes in fluoroscopic images may be subtle between steps, e.g. between partial and complete cement filling. The software can highlight the difference between any pair of snapshots—this creates a copy of the image with differences overlayed in blinking red (see Fig. 6).
Cement filling feedback panel
Differences between real intraoperative fluoroscopic images and DRRs may be analysed to yield estimates of cement filling. Along with filling estimates, sensitivity to image noise, i.e. “certainty”, may be computed. We discuss the way in which this “filling certainty ” is implemented in “Sensitivity of the output to image noise” section. The purpose of the cement filling feedback panel is to visualize these computed values. Using a bivariate colour map similar to those of Moreland [21], we represent certainty with luminance and filling with hue—see Fig. 7.
Algorithm
We are able to estimate the volume of injected cement by comparing an observed fluoroscopic image to simulated fluoroscopic images. First, we generate a set of two DRRs. The first DRR simulates the fluoroscopic view in the absence of percutaneously injected cement. The second DRR simulates the fluoroscopic view when the target is completely filled with cement. During the cement injection procedure, real observed fluoroscopic images represent scenarios that lie in between these two extremes. An example of these inputs to our algorithm is shown in Fig. 8.
By numerically computing the image intensity differences between the observed fluoroscopic image and the two DRRs, our software numerically estimates the amount of cement that was injected.
Computing the amount of cement
The DRR image formation process is illustrated in Fig. 9. Every element in the CT volume has at least one X-ray path that passes through it to the DRR image plane. The DRR formation is a discrete approximation of the physical X-ray imaging process.
We can estimate the total volume of cement in the 3D CT by estimating the distance that each ray travelled through cement on its path from X-ray source to the image. The distance that each ray travelled through cement can be expressed as the distance that it travelled through the target, multiplied by the fraction of this distance that was filled with cement. We call this fraction the “fill fraction”, \(F=s_\mathrm{c}/s_\mathrm{t}\) as shown in Fig. 10.
An X-ray beam’s flux exponentially decays as it passes through a uniform solid [14]. There exists a linear relationship between the length of an X-ray path through a uniform substance and the cumulative exponential absorption coefficient \(\mu _\mathrm{ray}\) by which the ray is attenuated. Using similar notation as in Fig. 10, we rewrite the fill fraction as \(F=\mu _\mathrm{c}/\mu _\mathrm{t}\). Here \(\mu _\mathrm{c}\) is the cumulative X-ray attenuation coefficient contributed by the cement-filled portion of the target and \(\mu _\mathrm{t}\) that which the target would have contributed if it was completely filled with cement. The superposition principle [13] allows us to express \(\mu _\mathrm{c}\) and \(\mu _\mathrm{t}\) each as the cumulative attenuation coefficient of the entire ray from source to image plane minus the portion that passes through no cement. We therefore rewrite the fill fraction as
$$\begin{aligned} F=\frac{\mu _\mathrm{partial}-\mu _\mathrm{none}}{\mu _\mathrm{full}-\mu _\mathrm{none}}. \end{aligned}$$
(1)
The subscript “none” refers to the entire ray where cement is completely absent—this represents the pre-operative scenario where no cement has yet been injected. “Partial” refers to the entire ray passing through the partially filled volume—this represents a typical intraoperative scenario. “Full” refers to the hypothetical ideal case where complete cement filling of the target is achieved.
By taking the response curve of the X-ray detector into account [6], one can show that the logarithm of the image brightness at any position in the fluoroscope’s image plane is linearly proportional to the accumulated X-ray attenuation along the path of the X-ray terminating at that point. Thus \(\log (I)\approx -k\cdot \mu _\mathrm{ray},\) where k is some constant and I is the image intensity expressed as a value between 0 and 1. This allows us to rewrite Eq. 1 in terms of image intensities:
$$\begin{aligned} F=\frac{-\log \left( I_\mathrm{partial}\right) +\log \left( I_\mathrm{none}\right) }{-\log \left( I_\mathrm{full}\right) +\log \left( I_\mathrm{none}\right) } \end{aligned}$$
which becomes
$$\begin{aligned} F=\frac{\log \left( I_\mathrm{partial}/I_\mathrm{none}\right) }{\log \left( I_\mathrm{full}/I_\mathrm{none}\right) }. \end{aligned}$$
(2)
\(I_\mathrm{none}\) and \(I_\mathrm{full}\) refer to the fluoroscopy images seen when either no cement is injected and when the target volume is completely cement-filled. In an intraoperative setting, one never has access to \(I_\mathrm{full}\) while \(I_\mathrm{none}\) could be recorded at the start of the procedure. We chose to simulate both \(I_\mathrm{none}\) and \(I_\mathrm{full}\) by using our DRR algorithm on the pre-operative CT volume. Note that \(I_\mathrm{partial}\) corresponds to the observed intraoperative fluoroscopy image that is available continuously during the cement injection procedure. An example of \(I_\mathrm{none}\), \(I_\mathrm{full}\) and \(I_\mathrm{partial}\) is shown in Fig. 8.
This triad of images allows us to compute F for all pixels in the 2D fluoroscopy image where these values are defined, as shown in Fig. 11. The value of F for an arbitrary pixel in the image is denoted by \(F_{x,y}\).
To obtain a quantitative cement volume estimate, we need to weigh each pixel’s \(F_{x,y}\) with the thickness \(s_{t}\) of the cement target at the point through which the associated ray passed. We call these the “projection weights”—illustrated in Fig. 12.
The projection weights correspond directly to \(s_{t}\) in Fig. 10, albeit appropriately scaled to yield a millilitre-valued output. The projection weights W are directly proportional to the attenuation coefficients that a completely cement-filled target would contribute to the rays terminating on each pixel. This is illustrated in Fig. 12. \(I_\mathrm{cement}\) is the image that would result from X-rays passing only through the completely filled cement target and nothing else, as shown in Fig. 13.
Mathematically:
$$\begin{aligned} W\propto s_\mathrm{t}\propto \mu _\mathrm{t}=-\frac{1}{k}\log I_\mathrm{cement}. \end{aligned}$$
As \(F=s_\mathrm{c}/s_\mathrm{t}\), it follows that \(W\cdot F\propto s_\mathrm{c}\). Looking back at Fig. 10, we note that summing \(s_\mathrm{c}\) for every ray through the CT volume provides an approximation to the amount of cement contained in the partially filled target.
We can now compute an absolute quantitative estimate of the volume of the filled portion of the cement target. This is done by calculating the weighted sum of the fill fractions over all image pixels and multiplying the result with the volume of the complete cement target, in millilitres:
$$\begin{aligned} V_\mathrm{filled}=\frac{\sum _{x,y}\left( W_{x,y}\cdot F_{x,y}\right) }{\sum _{x,y}W_{x,y}}\cdot V_\mathrm{target}. \end{aligned}$$
(3)
A necessary prerequisite for computing Eqs. 2 and 3 is that all relevant DRR and fluoroscopy images are registered, i.e. that their field of view are the same, that they have the same brightness and contrast and that their subject is in the same position. In our proof-of-concept system and in our experimental set-up, calibration was approximated visually. In a practical clinical system, such pre-operative registration should instead be automatically and robustly implemented, and orientation recorded via angle encoders attached to the imaging hardware. To correct for mismatches in our experiments, we applied suitable rigid similarity transformation that corrects for scale, rotation and translation in each fluoroscopic image. These parameters were estimated from four manually selected point correspondences in each image pair. Brightness and contrast were corrected by linearly adjusting the intensity of the fluoroscopic image to match that of the DRR in a least squares sense.
Additional caveats apply. The cement fill fraction F in Eq. 2 is only defined in image regions onto which the cement target project, and only where the image brightness is not fully saturated. This is illustrated in Fig. 13.
All other areas in the image are marked as non-computable. Non-computable areas include those where the thick metal prosthesis completely absorbs the X-ray beam, as this would result in a zero-valued denominator \(I_\mathrm{none}\) in Eq. 2. In non-computable areas falling inside the domain of \(I_\mathrm{cement}\), F is interpolated using thin plate spline radial basis functions, as in Figs. 11 and 14. In our experiments, we manually delineated suitable regions within the domain of \(I_\mathrm{cement}\) to serve as input to the interpolation algorithm, thereby excluding areas where needles were present or where cement leaked into the incision created during preparation of the test femurs. This is shown in Fig. 14.
Once F has been computed over the whole 2D projected cement filling domain, it may be back-projected along the ray path to 3D and visualized in the cement filling feedback panel as shown in Fig. 7.
Sensitivity of the output to image noise
Fluoroscopy images contain some degree of image noise that may distort the derived cement filling values. In areas occluded by radio-dense materials such as the prosthesis, little or no information about cement filling can be deduced. The uncertainty inherent in the resulting cement filling computation can be explicitly analysed. This uncertainty is essentially equal the degree that the result is affected by noise in the input image. Where dense metal objects like the prosthesis occlude the image, the useful signal is attenuated, while image noise remains constant. This results in a very low signal-to-noise ratio, resulting in unreliable cement filling estimates. For each pixel, the sensitivity of the filling fraction F to noise can be approximated as its derivative with respect to the observed fluoroscopic image intensity:
$$\begin{aligned} S= & {} \frac{\partial F}{\partial I_\mathrm{observed}}\\= & {} \frac{\partial }{\partial I_\mathrm{observed}}\frac{\log \left( I_\mathrm{observed}/I_\mathrm{none}\right) }{\log \left( I_\mathrm{full}/I_\mathrm{none}\right) }\\= & {} \frac{1}{I_\mathrm{observed}\log \left( I_\mathrm{full}/I_\mathrm{none}\right) }. \end{aligned}$$
S is only defined for the regions where F is directly computable. The reciprocal of the sensitivity function defines what we call the “certainty image” C :
$$\begin{aligned} C=I_\mathrm{observed}\log \left( I_\mathrm{full}/I_\mathrm{none}\right) . \end{aligned}$$
(4)
All non-computable pixels in C are set to zero. An example of the certainty image is shown in Fig. 11.
As was the case for F, we may re-project C along the projection rays and visualize it in three dimensions on the cement filling feedback panel. F and C are then combined in a single bi-variate colour mapping shown in Fig. 7.
The sensitivity of the overall cement volume estimate \(V_\mathrm{filled}\) may be computed by applying the weights W to each individual pixel’s filling value sensitivity S in the same way as it was done in Eq. 3:
$$\begin{aligned} S_\mathrm{filled}=\frac{\sum _{x,y}\left( W_{x,y}\cdot S_{x,y}\right) }{\sum _{x,y}W_{x,y}}\cdot V_\mathrm{target}. \end{aligned}$$
(5)