Introduction

Coronavirus disease (COVID-19) caused by the Severe Acute Respiratory Syndrome Virus 2 (SARS-CoV-2) can progress to acute respiratory distress syndrome, multiorgan failure, and death in some individuals1. The COVID-19 infection-fatality risk varies widely with age, with Canadian estimates ranging from < 0.1% for children and younger adults to 25.7% for adults 80 and older2. The clinical presentation and progression of COVID-19 in patients is highly variable3, which makes it difficult for clinicians to triage patients and determine their risk of poor outcomes. While some patients may clearly present with severe disease, even patients presenting with mild symptoms may experience rapid decompensation4. A simple, validated prognostic tool utilizing data that is available at presentation can help clinicians better prognosticate and make clinical decisions. Numerous tools to predict mortality in COVID-19 patients have been developed, but many are limited due to small derivation cohort sizes and/or inadequate validation5.

The 4C mortality score is an accessible risk stratification score developed by the International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC)6. It was derived and internally validated on a large, diverse cohort within the United Kingdom but requires external validity to confirm its generalizability. The objective of this study was to validate the ability of the 4C score to prognosticate mortality in COVID-19 patients admitted to hospital in Ontario, Canada. We will also examine the performance of the score over time as there has been significant variation in in available treatments, vaccine uptake, and dominant viral lineages over the course of the pandemic.

Methods

Study design and setting

We conducted a retrospective validation study using records from the McMaster Multi-Regional Hospital Coronavirus Registry (COREG). COREG is a multi-center data registry collecting information on positive COVID-19 cases in the Kitchener-Waterloo and Hamilton regions of southern Ontario, Canada. The registry includes COVID-19-related emergency department (ED) visits and hospital admissions from six tertiary hospitals across the regions, including three academic and three community centres. The total number of inpatient beds across all sites is approximately 2400.

Participants

We selected all patients admitted to one of the participating hospitals with a positive SARS-CoV-2 PCR nasal swab test between March 4, 2020 and June 13, 2021. We set the end date on the inclusion window to four weeks before date of final data extraction to allow for at least a 28-day length of stay for all participants. All inpatients were tested for COVID-19 therefore asymptomatic patients admitted for other reasons were included. Elective procedures however required a negative COVID test and therefore were not included. If a patient was transferred between study sites or had two separate admissions only the latter admission was retained. For temporal analysis, patients were separated into waves based on their admission date. Wave 1 spanned March 4, 2020 to August 31, 2020, Wave 2 included dates from September 1, 2020 to February 28, 2021 and Wave 3 stretched from March 1, 2021 to June 13, 2021.

Measures

Data in COREG were manually abstracted from electronic medical records by trained abstractors using a modified case report form published by ISARIC and the World Health Organization. Data was regularly audited to ensure accuracy. We utilized demographic and clinical data at presentation, typically from the emergency department. For patients who were directly admitted, we utilized the first day of inpatient records. Supplementary Table S1 includes additional information on data collection procedures.

4C mortality score

The 4C mortality score was derived and validated within the ISARIC World Health Organization Clinical Characterisation Protocol UK study7. The score was derived from a population of over 35,000 hospital inpatients validation on over 22,000 inpatient records indicated good discriminability (area under the receiver operating characteristic curve (AUC) = 0.77)6.

The 4C score incorporates age, sex, comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow Coma Scale, blood urea nitrogen, and C-reactive protein. We adapted the score to match our available data as the Glasgow Coma Scale was not collected at presentation. We replaced this risk factor with the documented presence of altered consciousness or confusion. The 4C score ranges from 0 to 21 with risk groups defined as Low (0–3), Intermediate (4–8), High (9–14), and Very high (≥ 15).

Outcome

The primary outcome was all-cause in-hospital mortality. Patients who received a palliative discharge were included. As was done in the initial derivation paper, we did not place a limit on time until death.

Statistical analysis

We presented a demographic and clinical profile of study participants, stratified by outcome, and reported the prevalence and missingness of each of element of the 4C score. Missing data was treated by multiple imputation with chained questions with predictive mean matching, using 20 imputations and 50 iterations per imputation. The imputation model included all variables in the 4C score, site id, and outcome. We plotted the proportion of patients who died in hospital by 4C score and risk group and compared them to the initial derivation work in the UK.

We validated the 4C score using the AUC, with 95% confidence intervals calculated via bootstrapping with 2000 resamples. For comparative purposes, we also calculated the AUC using only the age components of the 4C score as age has consistently been shown to be among the strongest predictors of COVID-19 related mortality8 and can be easily collected and used to triage patients without electronic aid. We calculated all AUCs overall and by wave. Additionally, we calculated diagnostic accuracy measures for cut-offs at scores of 3, 8, 12, and 14. These cut-offs align with the predefined risk groups with an additional group within the “High” category as half of our study participants were classified within this group. Finally, we constructed a calibration plot using bootstrapped predicted probabilities. All analysis was done in R 4.0.39.

Ethics approval

Our study received ethics approval from the Tri-hospital Research Ethics Board and the Hamilton Integrated Research Ethics Board, who waived the requirement for informed consent as the data for this study was retrospectively collected from hospital medical records. All methods were performed in accordance with relevant guidelines and regulations.

Results

Our study included 959 patients of which 224 (23%) died (Table 1, Supplementary Table S2). Median age was 72 with slightly more males (55%) than females. Wave 2 was the largest wave with 46% of patients, followed by wave 1 at 32% and wave 3 at 22%. The missingness and post-imputation prevalence of each element of the 4C score can be found in Table 2. Missingness ranged from 0% for age to 57% for c-reactive protein.

Table 1 Characteristics of COVID-positive hospital inpatients and emergency department patients in the Kitchener-Waterloo region of Canada, March 2020 – January 2021.
Full size table
Table 2 Components of the 4C mortality score, missingness, and post-imputation prevalence.
Full size table

Validation

The plot of the proportion of patients who died in each risk score demonstrated a clear upwards trend (Fig. 1). Mortality rates within the risk groups were 0% (Low), 8.0% (Intermediate), 27.2% (High), and 54.2% (Very High). These mortality rates are similar to the rates reported in the original research (1.2%, 9.9%, 31.4%, and 61.5%)6.

Figure 1
figure 1

Proportion of COVID-19 inpatients who died by 4C mortality score and risk group.

Full size image

The AUC of the 4C score across all patients was 0.77, 95% CI (0.74, 0.80) (Table 3). The performance of the model was higher in wave 1 (0.81 (0.76, 0.86) than wave 2 (0.74 (0.69, 0.80)) or wave 3 (0.76 (0.69, 0.83), although confidence intervals overlapped. The age-only model was somewhat less discriminative than the full model in each wave: wave 1 (0.78 (0.72, 0.83)), wave 2 (0.68 (0.63, 0.73)), wave 3 (0.74 (0.67, 0.80)).

Table 3 Area under the receiver operating characteristic curve (AUC) for 4C score and age-only model overall and by wave.
Full size table

Diagnostic accuracy measures for cut-offs at 3, 8, 11, and 14 can be found in Table 4. The sensitivity of each cut-off ranged from 100% for > 3 to 28.3% for > 14 while the specificities ranged from 10.2% for > 3 to 92.7% for > 14. The calibration plot (Supplementary Figure S3) indicated good calibration with significant deviations only occurring at very high predicted probabilities where data was scarce.

Table 4 Diagnostic accuracy measures for mortality prediction at several cut-offs of the 4C score.
Full size table

Interpretation

We found that the 4C mortality score is a valid tool to prognosticate mortality among COVID-19 patients admission to hospitals in a Canadian population. We observed an overall AUC of 0.77, which is identical to the initial derivation research6. The AUC values across waves 1, 2, and 3 were 0.81, 0.74 and 0.76, respectively. The AUC of the 4C score was higher than an age-only model across each of the three waves.

The 4C score has been validated in jurisdictions outside of the United Kingdom, including Canada10,11. Our study is confirmatory of these findings and additionally contributes to the literature by conducting an analysis by wave to investigate changes in the performance of the score over time. Changes over time in treatment practices (e.g. use of steroids), vaccine distribution, and the dominant strain of SARS-CoV-2 could all potentially impact the predictive ability of a mortality risk score. The 4C score was derived during the initial phases of the pandemic, before variants of concern emerged, before steroids were demonstrated effective against severe disease12, and while vaccines were still in early trials. While these conditions are similar to wave 1 in Ontario, by the peak of wave 2 vaccine distribution was underway to individuals most at risk13 and the alpha variant (B.1.1.7) was spreading14.

Overall, the predictive ability of the 4C was maintained across the three waves, which is a promising indication given that COVID variants are likely to continue to emerge15. We observed a drop in the point estimate of the discriminative ability of 4C score in the later waves, although the differences were not statistically significant. Both the 4C score and age-only model had higher AUC values for wave 3 than wave 2, which may be due to the targeting of early vaccine distribution to older individuals, which would have muted the effect of age on mortality. Future research should examine if the score continues to be predictive as vaccine uptake reaches herd immunity levels.

The proliferation of COVID-19 risk models is evidence of the demand for an accurate, accessible, and generalizable tool16, and our research adds to the body of evidence supporting use of the 4C score. Our examination of cut-offs within the 4C score suggests that in practice the score may ultimately be most useful in identifying individuals at particularly low risk of death. The lower two cut-offs (3 and 8) demonstrated negative likelihood ratios of 0 and 0.2, while positive likelihood ratios for any cut-off never exceeded 4. Automated calculation of the 4C score in electronic medical records could be used guide resource management and support clinical decision making such as treatment initiation and admission to ICU.

Limitations

A key limitation of our study is that we were only able to include data from two regions within southern Ontario. While similarities between health systems across Canada suggest our findings will have excellent generalizability to other Canadian provinces and territories, our results may not generalize to geographically remote settings or to jurisdictions with substantially different health systems. Also, certain variables had high levels of missingness. Although we used multiple imputation per best practice17, model performance may have been nevertheless adversely affected by data missing not at random. Additionally, we were not able to collect and report data on race and ethnicity. Finally, the scores were calculated retrospectively and not in real-time.

Conclusion

The 4C mortality score is an valid prognostic tool for use in Canadian hospitals. It can be used to identify and prioritize care for COVID-19 patients at high risk of death.