Convolutional neural networks on risk stratification of patients with suspected coronary artery disease undergoing coronary computed tomography angiography

To assess the prognostic value of convolutional neural networks (CNN) on coronary computed tomography angiography (CCTA) in comparison to conventional computed tomography (CT) reporting and clinical risk scores. 5468 patients who underwent CCTA with suspected coronary artery disease (CAD) were included. Primary endpoint was defined as a composite of all-cause death, myocardial infarction, unstable angina or late revascularization (> 90 days after CCTA). Early revascularization was additionally included as a training endpoint for the CNN algorithm. Cardiovascular risk stratification was based on Morise score and the extent of CAD (eoCAD) as assessed on CCTA. Semiautomatic post-processing was performed for vessel delineation and annotation of calcified and non-calcified plaque areas. Using a two-step training of a DenseNet-121 CNN the entire network was trained with the training endpoint, followed by training the feature layer with the primary endpoint. During a median follow-up of 7.2 years, the primary endpoint occurred in 334 patients. CNN showed an AUC of 0.631 ± 0.015 for prediction of the combined primary endpoint, while combining it with conventional CT and clinical risk scores showed an improvement of AUC from 0.646 ± 0.014 (based on eoCAD only) to 0.680 ± 0.015 (p < 0.0001) and from 0.619 ± 0.0149 (based on Morise Score only) to 0.6812 ± 0.0145 (p < 0.0001), respectively. In a stepwise model including all prediction methods, it was found an AUC of 0.680 ± 0.0148. CNN analysis showed to improve conventional CCTA-derived and clinical risk stratification when evaluating CCTA of patients with suspected CAD.


Introduction
Coronary computed tomography angiography (CCTA) has been widely incorporated into the clinical setting as a first line strategy in ruling out obstructive coronary artery disease (CAD) in patients with low to intermediate risk [1].
Beyond the identification and grading of coronary artery stenosis, it allows the characterization of atherosclerotic plaque features that have prognostic implications such as low-attenuation plaque, spotty calcification, napkin-ring sign and remodeling [2,3].
Semi-automatic and semi-quantitative evaluations of high-risk plaques have been extensively developed for several years. However, CCTA-based risk assessments are not yet taken regularly into account in the clinical decisionmaking process, mainly because it demands quite a time expenditure of highly trained professionals for a still limited additional benefit compared to other risk prediction models.
With the rise of automatic machine learning (ML) algorithms including deep learning (DL) there is an expectation of improvement in diagnosis and prognostication for patients with cardiovascular diseases [6]. The identification of plaque features using ML tools has been shown already to outperform conventional quantitative and qualitative CCTA analysis [4][5][6].
A type of deep-learning algorithm, the so called convolutional neural networks (CNN), has been developed to process imaging data exhibiting natural spatial invariances [7]. Using a training sample of images, it is able to learn features from images and execute tasks such as labeling an image to a group or class, detecting an object or generating a new image, so that CNN is considered nowadays the state of the art in image analysis [8,9].
With the increasing importance of diagnostic imaging and the rapid expansion of medical recorded data, CNN may be helpful in evaluating computed tomography datasets more effectively and has the potential to even recognize imaging patterns that the human eye can not see in traditional grayscale computed tomography (CT) scans.
Therefore, the aim of our study was to evaluate the longterm prediction of major cardiovascular events using CNN on CCTA-images of patients with suspected CAD in comparison with clinical and conventional CCTA-based risk scores.

Study population
In this study, we enrolled 5468 consecutive patients who underwent CCTA for suspected coronary artery disease (CAD) at the German Heart Center in Munich, Germany from October 2004 to January 2018.
Patients with acute coronary syndrome, presence of a lifethreatening situation, a lack of stable sinus rhythm during the examination, prior stent implantation or coronary bypass surgery were excluded from analyses. Before examination, a structured interview was performed, including patient age, height and weight, as well as history of cardiac disease, present concerns and current medication.
Laboratory results and cardiac risk factors were assessed. The pretest probability of CAD was calculated using the Morise score [10], which includes age, gender, risk factors and symptoms to predict the probability of obstructive CAD. According to the number of coronary arteries with obstructive CAD (defined as ≥ 50% stenosis) the extent of coronary artery disease was classified as 0-, 1-, 2-or 3-vessel disease.
Follow-up information was gathered either through clinical visits, questionnaires sent by mail or phone contact. Of the 7770 patients initially enrolled in the study, 5605 could be reached for clinical follow-up. 25 patients had to be excluded due to absent individual cardiovascular risk factor values and further 137 individuals had missing or non-diagnostic images. Primary combined endpoint of the study consisted of major adverse cardiac events (MACE) defined as composite of all-cause death, myocardial infarction, unstable angina, or late revascularization (> 90 days after CCTA).
Training endpoint included additionally patients undergoing coronary revascularization within 90 days after CCTA and was used together with the primary endpoint in the twostep training of the full network.

Image acquisition
Throughout the study period 4 different CT scan generations were used for image acquisition (Fig. 1).
A 64-slice single source According to the patient's heart rate and absence of contraindications intravenous beta-blocker medication was administered targeting a heart rate less than 60 beats/min. Sublingual nitrates were applied if systolic blood pressure was higher than 100 mmHg.
The coronary prospective ECG-synchronized CTA was triggered into the diastolic phase (70% of RR-interval). Tube voltage was selected by the technician and/or physician between 70 and 120 kVp, tube current was adapted automatically based on body size (CARE Dose). Contrast circulation time was determined using a testbolus with 10 ml contrast media (Imeron 350, Bracco Imaging GmbH, Konstanz, Germany), followed by a 50 ml 0.9% saline chaser. The coronary CT angiogram was performed with a 50 ml contrast bolus at 5.0 ml/s, followed by 30 ml 0.9% saline chaser.
Axial thin slice images were reconstructed with 0.6 mm slice width and increment of 0.4.

Image annotation and preprocessing
The 3D dataset was analyzed using a commercially available software (Syngo.via, Siemens Healthineers, Erlangen, Germany) and the coronary artery tree was segmented automatically with manual correction of inconsistencies. This yielded centerlines and a mask denoting vessel lumen and vessel wall including plaques for all detectable vessel branches. The vessel regions containing non-calcified and partially calcified plaques were marked manually, calcified plaques were annotated automatically using a threshold algorithm: along the centerline of each segment mean and maximum contrast intensity was calculated. Calcification was marked, if pixel intensity was more than 150HU above maximum vessel contrast. To correct for outliers, maximum contrast was limited to 120% of mean contrast.
Coronary arteries were reformatted into 2D multi angle images as stretched curved planar reconstructions (SCPR). Up to five reformations (1 for RCA, 2 for LAD, and 2 for LCx territory) were then integrated in one image with a 224 × 224 matrix holding each pixel data, annotation mask, and distance from vessel ostium in one color channel ( Fig. 2), for each patient 36 reconstructions of different angles around the centerline were calculated.

Model architecture and model training
An ImageNet DenseNet-121 a binary classification layer was used.
The whole dataset was split randomly into five groups stratified by scanner generation, both endpoints, gender and age (dichotomized by median).
Hyperparameter optimization was done on a 4:1 trainingvalidation-test split. Hyperparameters are listed in Table 1. The parameters selected for the main training are marked The final results were acquired using five time cross validation with one group serving as validation group and the other as training group.
Densenet models are pretrained on nonmedical image. These models failed to converge on the clinical endpoint. We therefore chose a two-step approach. First the network was trained using the training endpoint, which included early revascularization; then it was further optimized using the primary endpoint.

Statistical analysis
The prediction of the fully trained model, normalized by the softmax function was used as a variable for further statistical tests. Outcome prediction and incremental value compared to the extent of CAD was done by receiver operating statistics. All statistical tests were performed two-sided and a significance level of 5% was used. The statistical package R version 2.10.1 including the package rms was used for statistical analysis.

Results
A total of 5468 patients were included, with a mean age of 61.1 ± 11.2 years, and 66.5% were male. In total, 334 primary endpoint events (168 deaths, 27 non-fatal myocardial infarction, 1 unstable angina and 154 late revascularization) occurred during a median follow-up duration of 7.2 years. Additionally early revascularizations occurred in 405 (7.4%) patients. Table 2 shows the characteristics of the study participants.
Of the 5,468 patients, 419 (7.6%) showed diabetes, 1757 (32.1%) were currently or had a history of smoking and 1885 (34.5%) had a positive family history of cardiovascular disease (CAD). Hypercholesterolemia was Fig. 2 Model architecture overview. Initially (step 1), coronary artery segments were reformatted in multi angle stretched curved planar reconstructions (SCPR). Next (step2), images were integrated with the annotation mask of non-calcified, partially calcified and calcified, as well as distance from vessel ostium. 224 × 224 matrix was used as input for an ImageNet pretrained DenseNet-121 with a two-step training: first, the full network was trained using the training endpoint (step 3), which included early revascularizations; then, the feature layer was further trained using only the primary endpoint (step 4) showed obstructive CAD. Baseline differences in terms of cardiovascular risk factors between groups with or without the occurrence of primary and training endpoint are shown in Table 3. The primary and secondary endpoints are shown in Table 4.
CNN based risk prediction for primary endpoints had an area under the curve (AUC) of 0.631 ± 0.015. Regarding the training endpoint it was observed an AUC of 0.720 ± 0.010 with the CNN algorithm. When combining CNN analysis with CT-based parameters, we found an improvement of AUC to predict primary endpoints from 0.646 ± 0.014 (based on eoCAD only) to 0.680 ± 0.015 using CNN in addition to eoCAD (p < 0.0001). Clinical risk assessment using the Morise score demonstrated an AUC of 0.619 ± 0.0149 for predicting the combined primary endpoint, while combining  it with CNN showed an increased AUC of 0.6812 ± 0.0145 (p < 0.0001). In a stepwise model combining all prediction methods, it was found an AUC from 0.619 ± 0.0149 for Morise score alone, increasing to 0.676 ± 0.015 after adding eoCAD (p < 0.0001) and, eventually, to 0.680 ± 0.0148 by means of Morise score, eoCAD and CNN combined together (p = 0.0001) (Fig. 3).

Discussion
Our study shows an improved risk prediction for MACE in patients undergoing CCTA combining CNN with conventional CT parameters and clinical risk factors. These results highlight the potential of integrating machine learning (ML)-based image analysis into the evaluation of coronary plaque features in order to improve prognostication of patients with suspected CAD.
To our knowledge, this is the first approach to use a CNN algorithm directly on risk assessment of patients with suspected CAD. Up to now, ML-based models were used to optimize prediction based on known plaque features and CNNs were used to automate the detection of these features. Our CNN model was fed with only a scarce amount of information about the coronary plaque characteristics and was able to enhance the prognostication of MACE evaluating not 3D CT data, but merely 2D integrated images of the coronary arteries.
Previously, ML-based models in cardiac CT imaging were either used to optimize prediction of cardiovascular outcome or to simply automate and enhance morphologic plaque characterization. CCTA-based qualitative and quantitative plaque features were used by Al'Aref et al. to create a ML model to predict culprit lesions among acute coronary syndrome (ACS) patients and showed a significantly higher AUC when compared to models based on high-risk plaque features, diameter stenosis and lesion-level plaque analysis [11]. This model also demonstrated a specificity of 89% for predicting non-culprit lesions in patients who underwent CCTA without presenting acute coronary syndrome. Motwani et al. [12] analyzed clinical and CCTA-based risk scores for the prediction of 5-years all-cause mortality and found an improved AUC using ML when compared to Framingham risk score (FRS) or CCTA data alone.
Our group performed a ML-based time-to-event analysis in a similar cohort of patients with suspected CAD [13], which showed a superior performance for the long-term prediction of MACE than the use of clinical and CCTA derived variables or scores, independently.
In a multicentric study, Lin et al. [14] developed and externally validated a deep learning based algorithm to measure total plaque volume and minimal luminal area that correlated closely with expert reader measurements and intravascular ultrasound. However, an association between an increased risk of myocardial infarction and deep learningbased total plaque volume could only be shown after adjustment for clinical risk scores and the presence of obstructive stenosis.

Fig. 3 ROC-Curves for Prediction of Major Cardiovascular Events
Using a multi-task recurrent convolutional neural network (RCNN) Zreik et al. [15] demonstrated the feasibility of an algorithm for an automatic detection and characterization of coronary plaques and stenosis. This method showed a high accuracy in detecting and determining the significance of coronary stenosis but only a moderate reliability in classifying coronary plaques, as the differentiation of the mixed plaque from the calcified and noncalcified plaques remains a major challenge.
The main focus in applying neuronal networks to X-ray coronary angiography (CAG) is automated stenosis detection and characterization. In CT angiography, there are several commercially available systems, but their algorithms are not known in detail. In invasive coronary angiography Stralen et al. compared three CNNs for stenosis detection in the right coronary artery in 9278 invasive angiography and identified EfficientDet D3 as the best performing model [16]. Cong et al. compared different CNN architectures for classifying stenosis as < 25% or > 25% based on QCA-data from 230 invasive angiographies and identified Inception-v3 as best performing model [17].
The aim of this study was not to automatically detect single image parameters but to use the clinical outcome as ground truth and the main challenge was to adapt one of the many available CNNs to this new endpoint and the additional variance. DenseNet family was chosen because of the relatively large size of the input matrix, its usage in other studies in the field and the good performance in classification of non-medical images [18,19].
Reliable risk assessment based on coronary plaque features is challenging as quantitative and qualitative analysis softwares are often time-consuming with more than 40 different plaque characteristics to be considered. Even after years of development, semi-automatic plaque evaluations still show restricted inter-and intraobserver agreement, especially in patients with higher coronary disease burden when evaluating calcified and low-attenuation plaques [4,[20][21][22]. Additionally, plaque analysis is not performed in a strictly standardized fashion among different research centers, since acquisition protocols, CT scans, software algorithms and levels of experience of CT readers may differ between medical care centers.
The good performance of the new ML algorithms emphasizes the complex nature of plaque analysis where different parameters carry only a fraction of the prognostic information. Assuming that relevant prognostic information lies in the coexistence of different parameters and part of this is still unknown, we tried to use the unbiased learning approach of CCNs to optimize prognostication and in addition to the image data only provided basic additional information of coronary segmentation and lesion localization.
The results demonstrate the feasibility of the approach. Prognostic value of the CNN algorithm alone was comparable with eoCAD and Morise score, but improved prediction significantly in combination with the others. It seems the algorithm can detect relevant prognostic information not used by standard CCTA assessment, but obviously it cannot use all information available.
Without providing coronary segmentation and lesion localization the algorithm did not improve at all, thus still requiring preprocessing of the data. The integration of fully automated lesion detection would be the logical next step of improvement. To account for the length of follow-up, it would also be relevant to use a time-to-event model in further studies.

Limitations
The results of the present study were not externally validated on a separate cohort. The majority of our patients were males from an urban area of mainly caucasian people. Throughout the image acquisition period of 160 months four different CT scan generations were used and improvement of image quality may have affected the results. Due to the limited number of primary endpoints, we could not set aside a testing sample to train the algorithm.

Conclusion
We developed a novel CNN model based on CCTA images to assess risk prediction for MACE and found an improved AUC when combining it with conventional CT and clinical parameters. Our results highlight the value of CNN tools in assessing CCTA images and hold great potential for further improvement in prognostication of patients with suspected CAD. In the future, we would like to identify which specific variables the CNN model had used to predict MACE. It would be also interesting to abdicate of plaque annotations or develop an automatic detection and characterization of coronary plaques and stenosis for further risk analysis tools.
Author contributions All authors have contributed to data collection. CES and MH drafted the analysis plan and performed the statistical analysis. The first draft of the manuscript was written by RA and all authors were involved in the review and final approval of the manuscript.
Funding Open Access funding enabled and organized by Projekt DEAL. The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.