Background

The COVID-19 pandemic, a disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has had a huge impact on healthcare systems worldwide and resulted in more than 523 million known infections and well over 6.2 million deaths globally as of May 24, 2022 [1]. The disease’s spectrum ranges from a minor illness that can be treated on an outpatient basis to severe acute respiratory failure that may require admission to the intensive care unit (ICU) or death [2,3,4].

The early detection of patients with COVID-19 who may have worse outcomes is a priority [5,6,7]. A highly accurate tool for predicting the clinical course of this disease could be very useful for risk stratification, clinical decision-making, and ultimately for reducing mortality. A number of prognostic models for COVID-19 have been proposed since the beginning of the pandemic. Several studies have established a link between the most severe cases of COVID-19 and blood cell count-derived ratios, such as the neutrophil-to-lymphocyte ratio (NLR), platelet-to-lymphocyte ratio (PLR), or neutrophil-to-platelet ratio (NPR) [8,9,10,11,12].

Our group created a prognostic nomogram (Risk of In-hospital Mortality Score in COVID-19 (RIM Score-COVID)) that is highly accurate for predicting in-hospital mortality [13]. The RIM Score-COVID includes five variables commonly measured upon a patient’s arrival at the emergency department (ED): age, sex, baseline oxygen saturation (SpO2), C-reactive protein (CRP) level, and NPR. Applying the nomogram to the validation cohort, the area under the receiver operating characteristics curve (AUROC) was 0.861 (95% CI 0.823–0.900).

The Spanish Society of Internal Medicine (SEMI, for its initials in Spanish) sponsored a nationwide COVID-19 patient registry in which 150 Spanish hospitals participated. Over 24,000 patients over 18 years of age were included. Several studies derived from the SEMI-COVID-19 Registry have reported clinical characteristics with prognostic value [4, 14,15,16] as well as prognostic scores [17, 18]. However, since the start of the pandemic, multiple factors have changed the course of disease because the clinical management [20], available resources, and treatments indicated have evolved since the beginning of the pandemic [19, 21]. Therefore, the objective of this study was to validate the prognostic utility of the RIM Score-COVID in the SEMI-COVID-19 Registry in three time periods: patients admitted in the first wave, patients admitted after first wave but before the start of vaccination, and patients admitted after the start of vaccination [22].

Methods

The RIM Score-COVID was validated using the SEMI-COVID-19 Registry, which retrospectively compiles data on the index admission of patients ≥ 18 years of age with COVID-19 confirmed microbiologically through a reverse transcription polymerase chain reaction (RT-PCR) or antigen test and with hospital discharge or in-hospital death from January 2020 to December 2021. More in-depth information about the justification, objectives, methodology, and preliminary results of the SEMI-COVID-19 Registry have recently been published [4, 14, 15].

Study population and registry information

Patients from the SEMI-COVID-19 Registry were treated at their attending physician’s discretion, according to local protocols and clinical judgement. Patients included in open-label clinical trials could be included in the registry, provided that all information about treatment was available.

An online electronic data capture system (DCS) was developed, which included a database manager along with procedures for the verification of data and contrasting of information against the original medical record to ensure the best possible quality of data collection. Patient identifiable data were dissociated and pseudonymized.

Data collection

Data were collected retrospectively and included approximately 300 variables grouped under various headings: (1) inclusion criteria, (2) epidemiological data, (3) RT-PCR and serology data, (4) personal medical and medication history, (5) symptoms and physical examination findings at admission, (6) laboratory (blood gasses, metabolic panel, complete blood count, coagulation) and diagnostic imaging tests, (7) additional data at 7 days after admission or at admission to the intensive care unit (ICU), (8) pharmacological treatment during the hospitalization (antiviral drugs, immunomodulators, antibiotics) and ventilatory support, (9) complications during the hospitalization, and (10) progress after discharge and/or 30 days from diagnosis. Variables were obtained from the patient’s electronic medical record [4].

To validate the results obtained with the initial RIM Score-COVID model, the cohort of patients was divided into three groups based on admission date.

The first time period (T1) was February 1 to June 10, 2020 (first wave). The cut-off point of June 10, 2020 was selected for two reasons: first, because it was the date that the last patient was included in the original validation model for the RIM Score-COVID and second, because it is the approximate date of transition between the first and second waves of the COVID-19 pandemic in Spain.

The second time period (T2) was from June 11 to December 31, 2020 (second wave and pre-vaccination period). The second cut-off point of December 31, 2020 was selected for two reasons: first, because vaccination in Spain began soon before on December 26, 2020 and second, because it is the approximate date of transition between the third and fourth waves in Spain.

The third time period (T3) was from January 1 to December 5, 2021. The third cut-off point of December 5, 2021 was selected because it was the end of data inclusion in the SEMI-COVID-19 Registry for this work. T3 is the vaccination period.

RIM Score-COVID

The RIM Score-COVID is a logistic regression model that includes five variables commonly measured upon the patient’s arrival at the ED: age, sex, baseline SpO2, CRP level, and NPR value. The NPR is the ratio of neutrophil count (× 109 cells/L) to platelet count (× 1011 cells/L). The initial nomogram was developed in a cohort of 2,543 patients with COVID-19 requiring hospital admission, of which 1,955 (76.8%) patients were included in the final analysis. The patients were randomly divided into training and validation cohorts; applying the nomogram to the latter cohort, the area under the receiver operating characteristics curve (AUROC) was 0.861 (95% CI 0.823–0.900). The nomogram developed is shown in Fig. 1 [13]. It should be noted that the RIM Score-COVID was developed in a cohort of COVID-19 patients admitted during the first wave of the pandemic, when there was a multitude of hospitalized patients, a shortage of RT-PCR diagnostic tests for SARS-CoV-2, and frequent changes in the diagnostic and treatment protocol proposed by the Spanish Ministry of Health. For several weeks in the first wave, COVID-19 was diagnosed based solely on clinical and radiological criteria.

Fig. 1
figure 1

Nomograms of the NPR model. SpO2 (%): oxygen saturation; NPR (high: > 3.0) neutrophil-to-platelet ratio; CRP (high: > 112 mg/L): C-reactive protein

Full size image

Statistical analysis

Data were expressed as medians (interquartile range) for continuous variables or as numbers and percentages for categorical variables. The normality of the distribution was assessed using the Kolmogorov–Smirnov test. The Kruskal–Wallis test was used for comparison of continuous independent variables (Kolmogorov–Smirnov test < 0.05 in all). Pearson's Chi-square test was performed to compare categorical variables between groups. ROC curves and their corresponding 95% AUC confidence intervals (CI) were calculated [23]. The Statistical Package for Social Sciences SPSS version 23.0 (Armonk, NY, USA) was used to perform all statistical analyses. Significance was established as p < 0.05. The Python package "scikit-learn" (1.0.1) was used to generate ROC curves and their corresponding AUC. Confidence intervals were estimated through bootstrapping 1000 tenfold cross-validation resampling, also calculated with scikit-learn 1.0.1.

Ethical aspects

The study was conducted in accordance with ethical principles and was conducted pursuant to good clinical practice standards and the principles of the latest versions of the Declaration of Helsinki (2013) and the Oviedo Convention (1997). This work was approved by the Institutional Research Ethics Committee of Málaga on March 27, 2020 (Ethics Committee code: SEMI-COVID-19 27–03-20), as per the guidelines of the Spanish Agency of Medicines and Medical Products. All patients gave informed consent. All data collected, processed, and analyzed in this work were anonymized and used only for the purposes of this project. TRIPOD statement guidelines were adhered to in the execution and reporting of the study [24].

Results

Data from 24,514 patients included in SEMI-COVID-19 Registry were analyzed, of which 22,566 (92.1%) were included. All patients were over 18 years old and none died in the ED. Patients admitted prior to February 1, 2020 (n = 15); those with platelet counts higher than × 106 cells/mm3 (n = 6); those with total leukocyte counts lower than the sum of total lymphocytes and total neutrophils (n = 481); and patients missing CRP (n = 767), SpO2 (n = 510), total neutrophils (n = 107), total platelets (n = 46), or sex (n = 16) were excluded, as shown in the flowchart in Fig. 2.

Fig. 2
figure 2

Patient inclusion flowchart. CRP C-reactive protein; SpO2 oxygen saturation

Full size image

The cohort of patients was divided into time periods according to the aforementioned cut-off points. A total of 15,976 (70.7%) were included in T1, 4,233 (18.7%) in T2, and 2,357 in T3 (10.4%). In total, 4,238 (18.7%) patients died: 3,113 (73.4%) in T1, 702 (16.5%) in T2, and 423 (9.9%) in T3. As to the mortality rate, the proportion of patients who died was highest in T1 (19.4%), followed by T3 (17.9%), and was the lowest in T2 (16.5%) (Table 1).

Table 1 Patients included in each time period, mortality, and mortality rate
Full size table

Table 2 shows the demographic data, comorbidities, evaluation of patients in the ED, as well as the outcomes of all patients included in the study. The groups of patients diagnosed in the different time periods were compared and significant differences were found in terms of age, sex, diabetes mellitus, hypertension, and cancer. Regarding the first evaluation in the ED, all parameters studied were significantly different between the time periods, as were the percentage of patients admitted to the ICU and the percentage of deaths (Table 2).

Table 2 Comparison of demographic, clinical, and biochemical characteristics upon admission and outcomes in three successive periods of the pandemic
Full size table

To predict the probability of death based on variables measured upon hospital admission, ROC curves and their corresponding AUC for the NPR model of the RIM Score were calculated for the total number of patients and for each time period. The AUC of the RIM Score-COVID in the Spanish SEMI-COVID-19 Registry was 0.823 (95% CI 0.819–0.827). The AUC for each time period was 0.834 (95% CI 0.830–0.839) for T1, 0.792 (95% CI 0.781–0.803) for T2, and 0.799 (95% CI 0.785–0.813) for T3 (Fig. 3 A–D).

Fig. 3
figure 3

AUC charts of the NPR RIM Score-COVID in the Spanish SEMI-COVID-19 Registry calculated for the total number of patients included (A) and for each time period (B, C, D). T1 first time period from February 1 to June 10, 2020 (B), T2 second time period from June 11 to 31 December 31, 2020 (C) T3 third time period from January 1 to December 5, 2021 (D); NPR, neutrophil-to-platelet ratio

Full size image

Discussion

According to our results, the RIM score-COVID (NPR model) is an effective and simple tool for predicting the risk of in-hospital mortality in patients with COVID-19. This nomogram incorporates the NPR, a novel ratio proposed by our group that is able to be calculated using values found in a complete blood count [13]. To our knowledge, this is the first study in which a nomogram designed in the first wave has been validated using data from later periods of the COVID-19 pandemic.

The RIM Score-COVID nomogram uses five accessible, routine, and simple-to-measure parameters commonly obtained in the patient’s first assessment in the ED. An AUC value of 0.861 was obtained in the validation cohort during the nomogram’s development [13]. Values close to 0.8 were reached in the three different time periods, during which the disease was managed differently due to greater knowledge of COVID-19 as the pandemic progressed, the emergence of different strains, and the start of a mass vaccination campaign. This reflects the RIM Score-COVID’s prognostic capacity for in-hospital mortality due to COVID-19 and, therefore, its clinical usefulness.

Several previous studies have published clinical scores that attempt to predict the risk of worse outcomes in patients with COVID-19 [16, 17, 24,25,26,27,28]. However, some of these studies have methodological limitations, including small patient samples, an unrepresentative selection of the control group, or short or incomplete follow-up periods [28].

On the other hand, many of the nomograms and scores published to date are complex, as they include many parameters (some up to 23) and show a lower predictive ability than ours [25,26,27,28,29,30]. Other scores have demonstrated a predictive ability similar to ours, but were developed with a smaller number of patients, do not include training and validation cohorts [31, 32], use parameters that are not accessible in many hospital EDs [33,34,35], or use imaging methods that cannot always be reported objectively in real time [35].

Two scores have been developed from the SEMI-COVID-19 Registry. The first was the PRIORITY model, which included age, dependency, cardiovascular disease, chronic kidney disease, dyspnea, tachypnea, confusion, systolic blood pressure, SpO2 ≤ 93%, and a requirement for oxygen. The AUC value of this model was 0.794 (95% CI: 0.775–0.813) in the validation cohort [17]. The other model developed and evaluated was a machine learning-based in-hospital COvid-19 Disease Outcome Predictor (CODOP), developed in a multicontinental retrospective study that included patients in the SEMI-COVID-19 Registry. CODOP uses 12 clinical parameters that are commonly measured at hospital admission and associated with the pathophysiology of COVID-19 (platelets, eosinophils, neutrophils, monocytes, CRP, creatinine, lactate dehydrogenase, sodium, potassium, glucose, and D-dimer) as well as age. CODOP has a high discriminative ability up to 9 days before clinical resolution (AUROC: 0.90–0.96, 95% CI: 0.879–0.970) [18].

A mortality prediction score has recently been published that is based on a cohort of 6,161 patients with COVID-19 on arrival at the ED of 127 Spanish hospitals. This score is similar to the RIM Score-COVID in that it uses the parameters of age, sex, SpO2, and NLR. It also includes two other parameters: dyspnea, which, although dichotomous (yes/no), is a subjective variable, and estimated glomerular filtration rate, which may not be so easy to obtain. However, its predictive capacity is lower than that of the RIM Score-COVID, with an AUC in the validation cohort of 0.845 (0.819–0.870) [36].

The first wave of COVID-19 posed an unprecedented challenge to healthcare systems around the world [37,38,39]. During those first months, limited resources, the scarcity of diagnostic tests, patients' own fears of going to hospitals, and the lack of knowledge of the disease led to a scenario that—despite the current overload of healthcare systems has fortunately not been repeated with such intensity for the time being [40, 41].

These aspects are reflected when comparing the cohorts of the three time periods: patients admitted in the first wave were older, had a more severe inflammatory state on arrival at the ED, and there were significantly higher percentages of patients admitted and deceased than in the successive waves [40, 42]. However, age, presence of comorbidities, and inflammatory status on arrival at the ED did not differ in deceased patients from the three time periods, with the exception of CRP, which, although elevated in both, was significantly different.

Nevertheless, despite the temporal differences that have led to different hospital care scenarios, the emergence of new SARS-CoV-2 strains, the varying availability of resources, the use of treatments with a higher degree of clinical evidence, and the start of the vaccination campaign, the RIM Score-COVID nomogram has continued to demonstrate prognostic capacity in the three time cohorts, with an AUC in successive waves of 0.79 and 0.80.

Some factors, such as age, hypoxemia, abnormal blood count ratios, and increased acute phase reactants, have been identified as risk factors for mortality and worse prognosis [18, 19, 43]. Early identification of patients at risk for moderate or severe forms of COVID-19 could lead to more forceful clinical behavior in the ED and to reducing admission thresholds using blood count-derived ratios in patients with COVID-19, especially the novel NPR.

Our findings may reflect an underlying inflammatory state that would become evident on the nomogram when weighing and combining complete blood count (CBC) parameters with the other factors identified as relevant. CBC-derived ratios, including the NPR, have been shown to be independent markers of mortality and worse prognosis in COVID-19 patients [11,12,13].

Our study has some limitations. The research has been performed at different time points during the pandemic with all the implications described previously. However, this could also be considered a strength, as it has allowed for the validation of a simple nomogram in patients with confirmed SARS-CoV-2 infection. The study has been performed on patients in Spain, but given the demographic variations among patients from different regions of Spain, it is reasonable to expect the model to perform similarly in other populations. On the other hand, although patients with COVID-19 who were discharged home from the ED were not included in the study, all patients who required hospital admission were included; therefore, our model should be applied to patients who require hospital admission for COVID-19. Lastly, the data were collected from electronic medical records by attending physicians, so important information may have been missing.

During the different study periods, the treatment protocols for patients with COVID-19 changed based on the scientific evidence available at that time. This has led to changes in the therapeutic approach to patients in the different periods/waves of the disease. We believe this aspect lends greater value to our findings since, without assessing the treatments administered, the information obtained from the patient on arrival at the ED is independent of the therapeutic protocols indicated when predicting the risk of in-hospital death due to COVID-19. This logic can also be extended to the effects of different SARS-CoV-2 strains, since although the RIM Score-COVID was developed when the Alpha variant was predominant, its discriminatory capacity has also been proven with later strains, such as the Delta variant. Further studies are needed to demonstrate its discriminatory capacity in new variants such as Omicron. Finally, the SEMI-COVID-19 Registry was not designed to collect information on vaccination in T3, so these data could not be analyzed. However, Spain is one of the countries where vaccination against SARS-CoV-2 has been very successful, with a high percentage of vaccination coverage in population. Vaccination has been shown to reduce the severity of SARS-CoV-2 infection. However, the progression of COVID-19 infection in patients vaccinated against SARS-CoV-2 is well described and related to the time that has elapsed since full vaccination and host characteristics (high comorbidity and immunosuppression). This suggests that decreased immunity and impaired immune responses after vaccination could help explain the high mortality rate [44].

Conclusion

The RIM Score-COVID is an easy and practical quantitative prediction tool. The nomogram uses objective parameters that are easy to obtain and reproducible in most healthcare facilities that treat COVID-19 patients worldwide. It can be calculated at no added cost and without the need for additional laboratory tests. This tool has been validated in a large cohort of patients hospitalized for COVID-19 in different periods of the pandemic in a large number of Spanish hospitals.

These assessments provide further evidence on this highly accurate tool’s predictive ability of the risk of in-hospital COVID-19 mortality, which showed good prognostic ability in successive waves of the pandemic. Further studies are needed to determine the use of this nomogram to support clinical judgment in other populations and healthcare settings around the world.