1 Introduction

X-ray fluoroscopy is a common imaging modality in clinical fields. Although fluoroscopic imaging does not provide three-dimensional (3D) information, in contrast to computed tomography (CT) and magnetic resonance imaging, the images can be acquired at high spatial resolution and in real time. Catheter-based therapy is performed under fluoroscopic imaging, although the vessel structures cannot be visualized. To overcome this limitation, angiographic images are captured with a contrast agent, and digital subtraction angiography (DSA) images (difference between the fluoroscopic and angiographic images) [1] are collected during the intervention. Patients typically have to hold their breath to match the respiratory phase during DSA acquisition, particularly in thoracoabdominal DSA. However, breath holding is difficult for some patients, such as the elderly. If the organs move during DSA acquisition, artifacts can occur on DSA images, and the DSA acquisition must be performed again. Unsuccessful DSA acquisition should be avoided as repeating the acquisition increases the quantity of injected contrast agent, the exposed X-ray dose and stress on the patient.

Fig. 1
figure 1

Flow of the proposed method

Full size image
Fig. 2
figure 2

Detailed way of respiratory phase matching including region of interest (ROI) setting

Full size image
Fig. 3
figure 3

Overview of the intra-operative execution system for intra-operative assessment experiment, including the conventional fluoroscopy system and the add-on execution system

Full size image

Prior studies have proposed motion correction methods between fluoroscopic and angiographic images using non-rigid image registration strategies  [2,3,4,5]. These methods can markedly reduce motion artifacts caused by movement, including heartbeat for example. Unfortunately, the computational times were unrealistic in all those reported strategies. As an alternative, numerous studies have proposed extraction of the vessel structures only from angiographic images  [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22]. In those studies, the centerlines were tracked or vessel regions were segmented from angiographic images. However, although these methods provide images for thick vessels, it remains difficult to accurately extract thin vessels compared with conventional DSA.

Fig. 4
figure 4

Example of changes in imaging geometry

Full size image

A further concern with DSA is that the acquired images are static, and the vessel behavior with respiratory motion cannot be evaluated. To observe the vessel structures, Atasoy et al.  [23] proposed a real-time roadmap system using fluoroscopic images and 3D vessel information extracted from 3D CT images. However, this system was only useful for thick vessels, as the vessel information was extracted from the 3D CT image. Thus, the amount of vessel information to be extracted largely depends on the spatial resolution of the CT image.

Fig. 5
figure 5

Coefficient of variations (CVs) of respiration-synchronized digital subtraction angiography (DSA) images for retrospective assessment. Red crosses represent CVs of the breath-hold DSA images. Blue line represents the average CV of the failed DSA images. Vertical bars, central rectangles, and horizontal lines inside the rectangles represent the ranges from minimum to maximum values, interquartile ranges, and median values, respectively (color figure online)

Full size image

In the present study, to resolve these problems with DSA imaging, we propose a respiratory-synchronized DSA (RSDSA) generation method based on respiratory phase matching method. The proposed method uses preoperative angiographic images and intra-operative fluoroscopic images collected under natural respiration during the operation. For each fluoroscopic image, a pattern matching is used to select an angiographic image in the most similar respiratory phase. We conducted two kinds of experiments to confirm that the proposed method can be used as the free breathing DSA and can generate a respiratory-synchronized roadmap.

Fig. 6
figure 6

Respiratory-synchronized DSA (RSDSA) and breath-hold DSA (BHDSA) image results

Full size image

2 Methods

2.1 Generation of respiratory-synchronized DSA images

We assumed that all image acquisition was performed after the patient was positioned on the couch, and when the tip of the catheter was positioned at the start point of the roadmap to be displayed. The workflow of the proposed method is shown in Fig. 1 and the detailed way of respiratory phase matching is shown in Fig. 2. In advance of forwarding the catheter, we obtained angiographic images under natural respiration in multiple phases for use as templates, and obtained real-time fluoroscopic images. In this step, a region of interest (ROI) was set around the diaphragm, ensuring that the diaphragm motion was contained within the ROI. The operator subjectively determined the position and size of the ROI by observing the diaphragm motion on the fluoroscopic images. After setting the ROI, the catheter operation can be undergone. For each fluoroscopic image (termed a frame), we then used a pattern matching technique to select an angiographic image at the most similar respiratory phase. Similarities of the diaphragm between the angiographic images and one fluoroscopic image in the ROI were calculated by normalized cross-correlation (NCC)  [24]. NCC between images A and B was defined as follows:

Fig. 7
figure 7

Comparison of RSDSA and BHDSA images (patient #9). Enlarged images are shown on the right side, and correspond to the red square areas of the left side images. Yellow arrows mark the edge position of the vertebra. The edge is more clearly observed on RSDSA images (color figure online)

Full size image
$$\begin{aligned} \hbox {NCC}=\frac{\sum _{i, j\in R} {\left( {A_{i, j} -\bar{{A}}} \right) \left( {B_{i, j} -\bar{{B}}} \right) } }{\sqrt{\sum _{i, j\in R} {\left( {A_{i,j} -\bar{{A}}} \right) ^{2}} }\sqrt{\sum _{i, j\in R} {\left( {B_{i,j} -\bar{{B}}} \right) ^{2}} }}. \end{aligned}$$
(1)

Here, \(A_{i,j}\) and \(B_{i,j}\) are the pixel values of A and B at coordinates (i, j), respectively, and \(\bar{{A}}\) and \(\bar{{B}}\) are the average values of A and B, respectively. The higher NCC is, the similar the pair of images is. By evaluating the NCC, the most similar angiographic image was selected. Because the number of acquired angiographic images was no more than a few dozen, the respiratory phase matching was performed using full searching with every entered fluoroscopic image. In the last step, DSA images were generated by subtracting each fluoroscopic image and its corresponding angiographic image. For display on the monitor, the pixel values were normalized between 0 and 255, as the simple subtracted image had pixel values ranging from − 255 to 255. We normalized the pixel values as follows:

$$\begin{aligned} V_{\mathrm{out}} =\left\lfloor {{V_{\mathrm{in}} }/2} \right\rfloor +128, \quad \hbox {where}\quad \left\lfloor x \right\rfloor =\hbox {max}\{ v \in {\mathbb {Z}} {\vert } v \le x \}.\nonumber \\ \end{aligned}$$
(2)

Here, \(V_{\mathrm{in}}\) and \(V_{\mathrm{out}}\) are the pixel values before and after normalization, respectively. Using this conversion, the − 255, 0, and 255 pixel values in the subtracted image became 0, 128, and 255, respectively, in the normalized image.

Fig. 8
figure 8

DSA images of patient #3. a Best RSDSA image. b Worst RSDSA image. c BHDSA image

Full size image
Fig. 9
figure 9

DSA images of patient #12. a Best RSDSA image. b Worst RSDSA image. Yellow arrows mark implemented coils (color figure online)

Full size image
Fig. 10
figure 10

A set of the worst RSDSA images of patient #10. a Generated RSDSA image. b Target fluoroscopic image. c Selected angiographic image

Full size image
Fig. 11
figure 11

Motion of the diaphragm with respiratory phase. a Patient #7 and b patient #10. Black squares represent the positions of the diaphragm on angiographic images. Red circles represent the positions of the diaphragm on fluoroscopic images. The frames of fluoroscopic images for which there were no corresponding angiographic images are enhanced in the regions shaded with cyan (color figure online)

Full size image

2.2 Intra-operative execution system

We constructed an add-on execution system for image acquisition and display of DSA for operative assessment, which also allowed normal use of the conventional fluoroscopy system. The intra-operative execution system is shown in Fig. 3. The add-on execution system consists of a computer and two monitors that display a doctor form and an operator form. The angiographic and fluoroscopic images are fed into a computer from the VGA output of the display located in the operation room through a frame-grabber board in real time. A DSA image is immediately generated with each fluoroscopic image. The generated DSA image is displayed on the doctor form (e.g., see Fig.  3b). Setting of the ROI and correction of parameters for image acquisition geometry are performed on the operator form (e.g., see Fig. 3c).

Image acquisition geometry is often changed by the physician during the operation, to center and/or enlarge the important region as shown in Fig. 4. In the proposed method, angiographic image acquisition was performed only once. Thus, scaling and shifting of the fluoroscopic images was performed optionally, to correct the image acquisition conditions between the angiographic and fluoroscopic images. The scaling process was performed based on the ratio between the digital zooming magnification factors. A shift correction was then performed by manual selection of corresponding feature points from both angiographic and fluoroscopic images. All pairs of corresponding feature points were manually selected from anatomical structures such as the transverse process or posterior tubercle of the vertebra, which do not move with respiration in the angiographic and fluoroscopic images.

3 Experimental results

3.1 Retrospective assessment experiment

Image acquisition tests were performed in 15 patients. All subjects gave informed consent to participate in this study, which was approved by the Ethical Review Board of Chiba University. Angiographic and fluoroscopic images were acquired under both natural respiration and breath holding. A contrast agent was injected into the common hepatic artery, and imaging was performed using the X-ray imaging device (DFP-8000D; Toshiba Medical Systems Co., Tochigi, Japan). The image size was \(1024\times 1024\) pixels, with an 8-bit pixel depth. The frame rates of both the angiographic and fluoroscopic images were 6.0 frames per second (fps). Three of the 15 patients failed to hold their breath and the obtained DSA images had unacceptable motion artifacts. In this paper, we call such DSA image failed DSA image and use as a reference. The numbers of angiographic and fluoroscopic images for the remaining 12 patients were approximately 29 and 39, respectively.

The quality of the RSDSA image depending on the ROI setting variability was evaluated and compared with image quality of the breath-hold DSA (BHDSA) image. ROIs for the respiratory phase matching were selected by six operators (medical engineering graduate students). We instructed each operator to select a region with respiratory motion, such as the diaphragm. Each operator viewed the fluoroscopic images and selected an ROI.

To evaluate the quality of the generated DSA image, a coefficient of variation (CV) of the background pixel values was calculated as follows:

$$\begin{aligned} CV\left( \mathbf{S} \right) ={\sigma \left( \mathbf{S} \right) }/\mu \left( \mathbf{S} \right) \end{aligned}$$
(3)

Here, \(\sigma (\cdot )\) and \(\mu (\cdot )\) represent the functions of the standard deviation and the average, respectively, and S represents a DSA image after removal of the contrast agent regions. When the similarity between a fluoroscopic image and an angiographic image is high, all background pixel values on the DSA image after application of Eq. (2) become 128, with a standard deviation close to zero. Thus, a low CV indicates a high quality DSA image. As described, the average CVs of the failed DSA images were also calculated and used as reference values to compare quality with that of the RSDSA images. We manually removed the contrast agent regions from all DSA images for the CV calculation.

The CVs of the DSA images for all patients are shown in Fig. 5. Although successful BHDSA images provided better CVs than for the RSDSA images, the CVs of the RSDSA images were generally better than the average CV of the failed DSA images. For patients #7 and #11, the CVs of the RSDSA images were similar to those of their BHDSA images. Data from patient #5 were also satisfactory. The RSDSA images of patients #5, #7, and #11 are shown in Fig. 6. We confirmed that all DSA images clearly visualized vessel structures, with few motion artifacts. Although the CVs of the RSDSA images patients #5 and #7 were higher than those for their corresponding BHDSA images, the overall image quality was acceptable. RSDSA and BHDSA images of patient #9 are shown in Fig. 7. Visual assessment indicated a similar image quality between the RSDSA and BHDSA images. The edges of the vertebra (yellow arrows) were partly located in the red squares (Fig. 7), suggesting that trunk motion affected the subtraction.

Patient #3 had CVs for the RSDSA images that were markedly higher than those for BHDSA images. Two RSDSA images (best and worst images) and one BHDSA image for patient #3 are shown in Fig. 8. BHDSA was performed successfully, with few motion artifacts. As most operators set the ROI around the right side of the diaphragm in RSDSA, there was a good consistency around this region. However, as shown in Fig. 8b, the intestines were slightly deformed and the left side of the diaphragm did not match at expiration phases. Nevertheless, most images could be used as roadmap images for catheter insertion, as the mismatched regions did not interfere with the contrast area, and as vessel structures were visualized equally well as those for BHDSA images.

Fig. 12
figure 12

CVs of respiratory-synchronized DSA images for intra-operative assessment. Vertical bars, central rectangles, and horizontal lines inside the rectangles represent the ranges from minimum to maximum values, interquartile ranges, and median values, respectively

Full size image
Fig. 13
figure 13

RSDSA images of patient R2. a–c Generated RSDSA images. d–f Enlarged images of a–c. Yellow arrows mark the positions of the tip of the catheter (color figure online)

Full size image

Patient #12 had CVs for the RSDSA images that were similar to the average CV of the failed DSA images. Two RSDSA images (CVs = 0.046 and 0.084) for patient #12 are shown in Fig. 9. This patient had implanted artificial coils (yellow arrows), and the areas around the coils showed markedly different pixel values. When the implanted coils did not match well, there was a large difference in the white and black pixel values. If patients have a metallic implant in their bodies, the CV easily becomes high even if the difference in pixel values is slight. However, we found that both RSDSA images of patient #12 were useful during surgery, as we were able to observe blood vessel structures.

For patient #10, the CVs of some of the RSDSA images were higher than those of the average CV of the failed DSA images. The worst RSDSA and corresponding fluoroscopic and angiographic images are shown in Fig. 10. The highest positions of the diaphragm on the angiographic and fluoroscopic images are shown in Fig. 11. For patient #7, most fluoroscopic images had a corresponding angiographic image. By contrast, for patient #10, half of the fluoroscopic images did not have corresponding angiographic images, as the respiration pattern was considerably different between angiographic and fluoroscopic image acquisitions. Unfortunately, it is sometimes difficult for some patients to control their respiration, often because of their advanced age or the effects of contrast agent injection. In such cases, acceptable RSDSA images cannot be obtained and angiographic image acquisition needs to re-performed.

3.2 Intra-operative assessment experiment

Image acquisition tests and real-time collection of RSDSA images were performed during catheter insertion in three patients using the implemented intra-operative execution system (see Sect. 2.2). The patients provided informed consent, as for retrospective assessment patients, and the same imaging device was used. The frame rates for the angiographic and fluoroscopic images were 15.0 and 7.5 fps, respectively. Approximately 55 angiographic images and 185 fluoroscopic images were acquired for each patient. The ROI sizes for these images were approximately \(200\times 200\) pixels. Angiographic images were acquired over one breathing cycle. Before catheter insertion, the operator visually confirmed that the ROI involved the diaphragm on all fluoroscopic images, from maximum inspiration to maximum exhalation. The X-ray tube voltage and current for angiographic imaging were 75 kV and 400 mA, respectively, and those of fluoroscopic imaging were 80 kV and 75 mA, respectively. Therefore, acquired fluoroscopic images had a higher noise than for the angiographic images, with a slight difference in image contrast. CV was used as an assessment index, as for the retrospective assessment experiment. To confirm the possibility of real-time presentment, the average processing time from image acquisition to display of the RSDSA image was also evaluated.

The CVs of RSDSA images are shown in Fig. 12, while the generated RSDSA images of patient R2 are shown in Fig. 13. All CVs were quite high, and the RSDSA images were very noisy. The edges of the vertebra and rib were clearly observed on images in Fig. 13. The artifacts and noise were likely because of the lower X-ray dose compared with the retrospective assessment experiment, and as the imaging position was shifted. In this case, because of the nature of the perspective projection, the image cannot be completely corrected by a simple pixel-shift process. However, the positions of the catheter tip (Fig. 13, yellow arrows) were superimposed onto the vessels in the RSDSA image. Thus, the positions and shapes of both the catheter and vessels were observed immediately, even if the patient was breathing. This suggests that add-on execution system can provide real-time guidance for catheter movement using fluoroscope images.

The average processing time of our method was \(88.2\pm 4.7\, \hbox {ms}\). As the frame rate of fluoroscopic images was 7.5 fps, the update time was approximately 133 ms. Thus, we confirmed that updating the RSDSA images using the proposed intra-operative execution system can be performed in real time. We also found that NCC was useful for measuring the similarity between fluoroscopic image and angiographic images, despite being a simple measurement function. However, there is no guarantee that NCC will work robustly when the image acquisition conditions (e.g., X-ray tube voltage or current) are very different between angiographic and fluoroscopic images. Nevertheless, many intelligent image recognition and visual tracking methods have been recently developed to provide automatic detection of effective regions for measurement of image similarities  [25,26,27,28]. Future studies using these novel techniques are required to reduce the manual operation and achieve more stable results with our system.

4 Conclusion

We proposed a respiratory-synchronized DSA generation method using pattern matching between angiographic and fluoroscopic images acquired under natural respiration. The effectiveness of our method was validated through retrospective assessment and intra-operative assessment experiments. The CVs of most RSDSA images were better than average CV of the failed DSA images. Furthermore, in three of 12 patients, those values were still better than for the BHDSA images. Moreover, as the processing was performed within \(88.2\pm 4.7\, \hbox {ms}\), the method can be easily applied during operation.

Despite confirming the effectiveness of our method, further validation and modification are needed as future works. For example, the correlation of the quantitative CV metric for measuring resulting image quality with diagnostic efficiency should be determined to find a suitable threshold. On the other hand, for more advanced respiration tracking, we are currently developing a method for automatic ROI selection not limited to the diaphragm, more intelligent similarity measure and more advanced geometry correction methods.