Evaluation of novice reader diagnostic performance in coronary CT angiography using an advanced cardiac software package

Purpose The purpose of this research was to evaluate whether a commercially available advanced cardiac software package for coronary CT angiography (CTA) interpretation may reliably assist inexperienced readers to screen for significant coronary artery stenoses. Methods Coronary CTA data sets of 61 consecutive patients with suspected coronary artery disease were evaluated by three novice readers with no experience in cardiac CT interpretation. In the first 15 patients, the novice readers were trained to use the advanced cardiac software package (includes automatic detection of coronary vessels, curved MPR and VRT reconstructions and a measurement too) knowing the results of an expert read. In the next 46 patients, the novice readers had to state whether there is a significant coronary artery stenosis (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$>$$\end{document}>50 %) and if they are confident with their diagnosis. The results of the novice readers were compared to the expert read. Results The 46 coronary CTA data sets contained 184 vessels with 15 stenoses in 9 patients. On a per-vessel analysis, novice reader 1/2/3 demonstrated 60 %/100 %/ 93% sensitivity, and 98 %/90 %/86 % specificity. Per patient, the readers diagnosed 36/28/29 cases correctly as free of stenoses, 6/9/8 correctly as having at least one stenosis, missed 3/0/1 cases with a stenosis and overdiagnosed 1/9/8 patients. Cohen’s kappa values for the three readers versus the expert were 0.60, 0.61 and 0.54. The three novice readers felt confident in the diagnosis of 36/33/30 patients. In these patients, they missed one significant stenosis, showed a sensitivity of 100 %/100 %/75 % and a specificity of 100 %/92 %/88 %. Conclusions The evaluated advanced cardiac software package successfully assists novice readers in interpreting coronary CTA data sets especially in ruling out significant coronary artery stenosis.


Introduction
Coronary computed tomography angiography (coronary CTA) can be used to identify and rule out coronary artery stenoses in selected patients [1][2][3]. In recent years, the number of coronary CTA studies has steadily increased for various reasons. More and more hospitals and radio-logical practices have access to multi-detector computed tomography systems which permit coronary CT imaging [4]. Furthermore, radiation exposure for routine coronary CTA can be reduced to less than 1 mSv per examination [5,6]. The majority of patients prefer CT to invasive coronary angiography [7], and an increasing amount of clinical data, for example regarding the use of coronary CTA in acute chest pain patients, demonstrates the method's efficacy [8].
However, the accurate identification of significant coronary stenoses is a challenging task that requires extensive training and experience, and according to Saur [9] is often constrained by the evaluator's knowledge and ability. Therefore, the American College of Cardiology and American Heart Association have developed clinical competence criteria to standardize training for interpretation of coronary CTA [10]. Expertise and experience are measured in competence levels (1)(2)(3), whereas level 1 can be achieved during a one month training and interpretation of 50 coronary CTA examinations-while level 3 requires 6 month of training, 300 interpreted examinations as well as ongoing teaching and research in the area of coronary CTA. However, some authors [11] claim that the learning curve is substantially longer than suggested by the competence criteria. As indicated by the 15fold increase in research on cardiac CT between 1996 and 2006 [12], most likely the number of coronary CTA studies dramatically increased, outweighing the number of expert readers.
The purpose of this research was to evaluate whether commercially available advanced cardiac software package for coronary CTA interpretation may reliably assist inexperienced readers to screen for significant coronary artery stenoses.

Materials and methods
The institutional review board approved this study and waived the need for informed consent.

Patient acquisition and scan technique
Data sets from 61 consecutive patients undergoing clinically indicated routine coronary CTA (24 male, 37 female; mean age 54.3 ± 9.4 years was retrospectively included for evaluation. Multidetector CT had been performed with a 128section dual source CT system (Somatom Flash , Siemens, Forchheim, Germany). Clinical indications for coronary CTA included 3 patients prior and 5 after aortic valve replacement, 3 before coronary bypass operation and 50 patients with chest pain and intermediate pretest likelihood of coronary artery disease. The patients' mean body weight was 84.7 ± 12.8 kg. Patients with a heart rate greater than 60 beats/min before the examination received oral administration of a β-blocker (atenolol, 100 mg) 1 h prior to the scan. During coronary CTA, the patients' mean heart rate was 65.9 ± 13.6. Adopted doses of 50-80 ml i.v. contrast agent (350 mg iodine/ml, Imeron , Bracco, Friedrichshafen, Germany), followed by 50 ml of saline solution were administered to all patients at a flow rate of 4-7 ml/s (54 patients received 60 ml, 5 patients 50 ml and 2 patients 80 ml contrast media). Images were acquired in prospectively ECG-triggered high-pitch spiral mode, and acquisition was timed to start at 60 % of the patients' R-peak to R-peak interval [13]. Tube voltage was 100 kV, tube current 320 mAs per rotation. Reconstructed slice thickness was 0.6 mm, slice increment 0.3 mm, and a 3D adaptive noise reduction algorithm termed B26 kernel by the manufacturer [14] was applied.
Advanced cardiac software package We investigated a commercially available advanced cardiac software package (syngo.via , Siemens, Forchheim, Germany). It semi-automatically reformats the original images, segments the coronary arteries and provides a measurement tool for stenosis quantification (Figs. 1, 2).

Standard of reference
A consensus read of two experienced readers (5 and 6 years experience) and one expert reader (15 years of experience in cardiac CT and Electron-Beam Tomography) supplied the reference standard. Therefore, they evaluated all 61 data sets for the presence of a significant coronary artery stenosis (>50 % reduction in diameter) in the left main (LM), the left anterior descending (LAD), the left circumflex (CX) and the right coronary artery (RCA), utilizing a standard 3D MPR visualization software (Leonardo, Siemens, Forchheim, Germany).

Readers
Three novice readers (medical students in their last semester of medical school), inexperienced in cardiac imaging and CT interpretation, were trained in the use of the advanced cardiac software package. Under the supervision of a radiologist experienced in using the advanced cardiac software package, the students interpreted the examinations of the first 15 patients applying the software and therefore were aware of the expert's interpretation.

Workflow of image evaluation applying the advanced cardiac software package
The CT data sets were transferred to a 3D workstation and automatically post-processed by the advanced cardiac soft-  One feature allows stenosis quantification by three adjustable bars, which are aligned orthogonally to the vessel direction (true axial) and are projected onto the curved MPR and the vertical-section profile. At the location of each bar, the diameter for the cross section of contrasted vessel is measured. By positioning the adjustable bars in accordance with the vertical-section profile of a vessel, for instance proximal, in and distal the narrowest part of a stenotic vessel segment, it calculates the degree of stenosis both with respect to the diameter and area of the remaining lumen. Furthermore, the axial view (in the orthogonal direction to the vessel) at the position of each bar (including the measured values) is displayed ware package. For the experimental read, the three novice readers independently interpreted the same 46 examinations without further assistance and they were blinded to the patients' clinical data. The workflow demanded the read-ers to apply the advanced cardiac software package detect stenoses and use the stenosis quantification tool. If a stenosis seemed to be overestimated by the advanced cardiac software package, for example due to artifacts, they were encouraged to use the assisted navigation to re-examine the stenosis in the corresponding location in multiplanar views. If in doubt, readers were instructed to base their final diagnosis on their perception of vessel stenosis rather than on the system's calculated degree of severity, using the software as assistance, but leaving the final diagnosis up to the human readers. Furthermore, the readers were asked to document their confidence about their diagnosis for each vessel (confident/not confident).

Statistical analysis
The statistical analysis on the 46 test patients was calculated on a per-vessel view and on a per-patient view for all three novice readers. Additionally, the readers' per-patient diagnostic performance (sensitivity, specificity, PPV, NPV) depending on their diagnostic confidence were evaluated.
The 46 experimental coronary CTA data sets contained 184 evaluable coronary arteries, 169 without, and 15 with significant stenoses (0/LM, 6/LAD, 4/CX, 5/RCA). On a perpatient basis, 37 patients had no stenoses, while 9 patients showed at least one significant coronary artery stenosis. No coronary CTA data set was excluded due to artifacts or insufficient image quality.

Discussion
As demonstrated by this work, novice readers revealed moderate sensitivity and specificity in detecting significant coro- nary artery stenosis when using a commercially available advanced cardiac software package. However, when taking readers' confidence level into consideration, novices displayed excellent sensitivity and NPV for detection of significant coronary artery stenoses. The presented workflow considers the advanced cardiac software as a tool inexperienced readers (e.g., interns) can apply to screen for significant stenoses in coronary CTA. If the expert would have reevaluated only cases which were marked as diseased or uncertain by the novice readers which corresponded to 40 % of all cases, he would have missed only one significant stenosis. Our results stand in accordance with the papers of Meyer et al. and Arnoldi et al. [17,18], who demonstrated computeraided systems to be helpful for detection of coronary stenosis by less-experienced readers. In contrast to our work, the software was applied as second reader, whereas the setup in our experiment simulates utilization of the software in the initial read. Additionally, in our experiment, the software never suggested a diagnosis. Instead, in the presented study, inexperienced readers apply the advanced cardiac software package as semi-automatic coronary artery visualization and segmentation tool, with the final decision up to the human interpreter. Our work stands in consistency with Goldenberg et al. [19] who propose a CADx as an initial analyzer (rather than a second reader), but leaves the final decision to the physician. As displayed by the excellent sensitivity and NPV, as well as the acceptable specificity and PPV, the presented method fulfills demands of a screening test [20]. According to Anders et al. and Ferencik et al. [21,22], the diagnostic accuracy of prerendered images, for example curved MPR and VRT, is inferior to conventional interactive reading of coronary CTA data. However, the investigated software offers interactive 3D multiplanar view and prerendered images, as well as real-time navigation between corresponding locations from the prerendered images and the interactive 3D multiplanar viewer. This provides the speed and convenience of post-processed images and the precision of interactive 3D multiplanar reading.
Pugliese et al. [11] studied the learning curve of members of a cCTA fellowship. They also investigated novices in reading coronary CTA. However, since all readers were radiology/cardiology fellows, in contrast to our study, they all had at least some level of experience in reading CT images or diagnosing coronary artery disease. After one year of fellowship, investigated readers reached competence level 3 for the number of interpreted examinations [10]. Although the results from [11] (naturally using different data sets), and our study (due to different experimental protocols, varying study hypotheses, distinct analyzing approaches, etc.) have to be compared very carefully and do not display a perfect concordance, we might compare the diagnostic odds ratio (DOR), a single and powerful indicator of test performance [23]. Even without considering level of confidence for our readers, they demonstrated a higher DOR (62, 145, 89) compared to [11] (between 15 and 27). Reported sensitivity (68-75 %) and specificity (88-92 %), both after one year of training were similar or worse to the overall performance of our readers.
Further, novices' diagnostic performance is validated by Cohen's kappa, attesting moderate to good interreader agreement to the reference standard. McNemar's value suggests high correlation between reader 1 and the reference standard. However, the need for dedicated and structured training in interpreting coronary CTA cannot be overemphasized.
One limitation of our study is that the novice readers did not initially analyze the coronary CTA data sets without the assistance of the advanced cardiac software. So we do not know if and how much the advanced software pack-age has increased the diagnostic performance of the novice readers. Another limitation is that the expert reading was not verified by invasive coronary angiography and significant CTA stenoses could not be verified to be hemodynamically relevant or insignificant. It is a well-known fact that coronary CTA has a limited positive predictive value, i.e., too many patients are incorrectly suspected of having a significant coronary artery stenosis [24]. However, the aim of our research was mainly set on comparing the interpretation results of the novices with an expert read. The clinical effect of coronary CTA stenosis possibly displaying hemodynamic significance could be the next step in future work. Moreover, our study focused on the evaluation of consecutive, unselected routine coronary CTA data sets. Most investigated patients presented with chest pain and intermediate pretest likelihood of coronary artery disease and after negative CTA did not receive further invasive coronary angiography. Finally, the patient number was relatively small and the number of diseased arteries relatively low. This low incidence, however, represents a clinical cohort of patients with an intermediate pretest likelihood of coronary artery disease better than a preselected group of diseased patients. Although coronary artery disease with 20 % accounts for the leading cause of death in the western world, our incidence indeed is way higher than the average 30 % lifetime prevalence for men and 15 % for women in Western countries [25].

Conclusions
The applied advanced cardiac software successfully assists novice readers in interpreting coronary CTA data sets. Evaluated software and displayed manner of use demonstrates moderate sensitivity and specificity for novice readers. When further taking readers' confidence level into account, sensitivity of detecting coronary stenosis was excellent. Therefore, the advanced cardiac software may be useful for novice readers as an initial analysis tool for screening and ruling out significant stenoses in coronary CT angiography.