Introduction

Various scores have been developed for emergency department and hospitalised patients to determine the degree of illness or to predict an endpoint such as mortality. An example of a widely used score for all patients is National Early Warning Score 2 [1]. The vital signs or other parameters needed for the score are inserted in an electronic patient record and the record automatically shows you, for example, the trend of the score. Another example is Sequential Organ Failure Assessment (SOFA) score [2], commonly used for all intensive care unit (ICU) patients to predict mortality. The newcomers, BLOOMY 14-day and 6-month prediction scores [3], are developed for bloodstream infection (BSI) patients, they need to be calculated electronically, and they have not been yet studied in other cohorts than in the original BLOOMY study. Before roll-out, more studies are needed to test their reproducibility and validity [4,5,6,7].

The BLOOMY study is a multicentre cohort study at six German tertiary care university hospitals. It was the first study to develop a prediction score for patients with infection by use of machine-learning approach. The 14-day and 6-month mortality model predictors included, for example, age, body-mass index, platelet counts and C-reactive protein concentrations (see Tables 1 and 2). The total possible scores for the BLOOMY 14-day and 6-month score were 20 and 26, respectively. In prediction of mortality, sensitivity increased with higher scores but specificity decreased; the study group used mostly 6 or 7 points as cutoff. The researchers had a separate validation cohort to calculate predictive accuracy. The aim of our study was to assess how the score estimates mortality in our cohort of 497 Finnish BSI patients, i.e., to externally assess the sensitivity, specificity, and other predictive values. We also studied how quick SOFA (qSOFA) is comparable to BLOOMY 14-day score and tested how BLOOMY 6-month score predicts 1-year and 5-year mortality.

Table 1 Points allocations for BLOOMY 14-day and quick BLOOMY 14-day scores by variable assessed, as presented by Tacconelli et al. [3]
Full size table
Table 2 Points allocations for BLOOMY 6-month and quick BLOOMY 6-month scores by variable assessed, as presented by Tacconelli et al. [3]
Full size table

Materials and methods

Bacteraemia patient cohort

Tampere University Hospital is a tertiary hospital with a catchment population of ca. 525 000 inhabitants in the Pirkanmaa County, Finland. The study cohort included consecutive blood culture-positive patients in the Tampere University Hospital ED between March 1, 2012 and February 28, 2014. The cases in which positive blood culture were determined as contamination were excluded. The clinical data of the patients were collected retrospectively from patient records. A criterion for the qSOFA score was calculated based on Sepsis‐3 definitions [8]. The Pitt Bacteraemia Score was calculated as presented by Korvick et al. [9]. The site of infection was determined retrospectively. A more detailed description of the patient cohort is available in our previous publications [10,11,12]. The study was approved by the Ethics Committee of Tampere University Hospital, Finland (permit# R11099). The need for informed consent was waived as routine patient care was not modified.

Variables

The BLOOMY 14-day and 6-month scores were calculated as described in the original article apart from two modifications due to practical reasons. On BLOOMY 14-day scores, day 0 variables were used instead of day 3 variables. This has an effect on points allocated to hypotension, mental status, and laboratory values (see Table 1), but not on the permanent patient data such as malignancies and medication. BLOOMY 6-month score uses laboratory variables minimum of six days after the last day of antibiotic treatment (Table 2). Instead, day 0 laboratory parameters were used. Again, permanent patient date was collected as in original article. In dichotomous variables, only cases with missing data that would affect the result were excluded. In continuous variables all cases with missing data were excluded. Patients with multiple admissions (n = 12) were handled as they would be separate cases.

Statistical analyses

SPSS version 28.0 software (IBM Corp., NY, USA) was used for statistical analyses. AUC analysis was used to assess the scores’ discriminative value and specificity, sensitivity, negative predictive value (NPV), and positive predictive value (PPV) was used to assess the predictive accuracy. A P‐value of < 0.05 was considered significant. To assess the calibration of the models, the calibration belt was used [7, 13].

Results

A total of 497 cases (484 patients) with positive blood culture were included. Median age was 68 (range 16–95 years), 53% were male, and Gram-positive and Gram-negative bacteraemia were equally common. Other characteristics, the severity of the disease, and causative organism are shown in Table 3. Ninety-nine cases were missing weight and 3 cases day of admission C-reactive protein, no other missing data for the BLOOMY score calculations. We found no need to recalibrate the scores (see supplement report).

Table 3 Patient characteristics, severity of the disease and microbiological data of the study population
Full size table

We calculated the predictive value of BLOOMY 14-day score to estimate death by day 14. AUC was 0.87 (95% Confidence Interval 0.80–0.94). Using ≥ 6 points as a cutoff, sensitivity was 68.8% and specificity 88.1%. With ≥ 7 points, sensitivity was 42.9%, specificity 94.9%, PPV 48.8%, and NPV 93.5%. In our cohort, Pitt Bacteraemia Score had an AUC of 0.79 (0.72–0.86) to estimate 14-day mortality. For the same estimation, ≥ 2 qSOFA points had an AUC of 0.76 (0.69–0.83).

Using ≥ 6 points as a cutoff, BLOOMY 6-month score estimated 6-month mortality with a sensitivity of 98.3% and a specificity of 10.7%. Increasing cutoff to ≥ 7 points decreased sensitivity to 93.2% and increased specificity to 21.0% (data on 483 cases). BLOOMY 6-month score had an AUC of 0.79 (0.73–0.85). AUCs of 6-month score to estimate 1-year and 5-year mortality were 0.80 (0.74–0.85) and 0.77 (0.73–0.82), respectively; see Table 4 for other predictive values for both BLOOMY 14-day and 6-month score.

Table 4 The predictive values of BLOOMY 14-day/6-month score in the original validation cohort and BLOOMY 14-day/6-month and qSOFA score ≥ 2 in our cohort
Full size table

The quick BLOOMY 14-day score included three parameters: age, mental status, and hypotension. The predictive value (AUC) of quick BLOOMY 14-day score to estimate 14-day mortality was 0.81 (0.75–0.87). Quick BLOOMY score of ≥ 6 points predicted 19 of 55 deaths at 14 days with a sensitivity of 34.6%, a specificity of 95.5%, a PPV of 48.7%, and an NPV of 92.1%.

Discussion

We calculated the predictive value of BLOOMY 14-day score to estimate death by day 14. In our study AUC was 0.87, almost the same as in the original BLOOMY cohort (0.86). Using 6 points or more as a cutoff, our sensitivity to predict day 14 mortality was 68.8% (original 61.3%) and specificity 88.1% (original 86.4%). Quick BLOOMY 14-day score AUC was 0.81 in our cohort vs. 0·83 in the BLOOMY cohort. The AUC of Pitt Bacteraemia Score to estimate 14-day mortality was 0.79, almost identical than in the original cohort (0.78). The AUC of qSOFA was 0.76 in our study. In the BLOOMY study qSOFA was not calculated.

The predictive values of BLOOMY 6-month score to estimate 6-month mortality in our cohort were different from the original BLOOMY cohort. We calculated the score by using day 0 laboratory values instead of, as in the BLOOMY cohort, the last day of antibiotic treatment plus minimum of six days. This affected the specificity of the 6-point cutoff; it was much lower in our cohort (10.7%) than in the BLOOMY cohort (52.8%). However, AUC was still higher than in the original cohort, 0.79 vs. 0.74, respectively.

The BLOOMY study compared the new scores to Pitt Bacteraemia Score, Gram-Negative Bloodstream Infection Risk Mortality Score [14], and, indirectly, to results from a qSOFA meta-analysis [15]. They found the predictive values of BLOOMY 14-day and 6-month scores to be better than in these older scores. Combined with our results, it appears that the machine learning approach has developed a score that yields better 14-day and 6-month mortality estimations than these established scores.

BLOOMY 14-day and 6-month prediction scores are developed for BSI patients, and they need to be calculated electronically. One prediction score developed for patients with infection, e.g. qSOFA, is, in contrast to all BLOOMY scores, easily calculated bedside and therefore gained popularity. According to our experience, other prediction scores developed only for patients with infection have not been widely used in clinical settings. Pitt Bacteraemia Score would need either electronic record-based, web-based, or app-based calculation to obtain the results, but these calculations are rarely performed. The same might happen with these newcomers.

Our study had some limitations. We were able to calculate all BLOOMY scores retrospectively, but some values were missing. Ninety-nine cases were missing weight which is used in BLOOMY 14-day and 6-month score (BMI ≤ 20 kg/m2 equals 1 point). We decided not to reduce the total number to those with weight as the missing values affected less on the dichotomous variables (see Table 4). We had > 99% of other values. Other limitation is the difference in the days of evaluation in the original cohort and in our cohort. The original cohort used day 3 as an evaluation point in BLOOMY 14-day score and the last day of antibiotic treatment plus minimum of 6 days in BLOOMY 6-month score. In the present study, day 0 was used as an evaluation point in both BLOOMY 14-day and in the laboratory results of BLOOMY 6-month score. The difference in BLOOMY 14-day evaluation day can also been seen as an advantage of the present study, but the difference in BLOOMY 6-month might have been the reason for the differences in the sensitivity and specificity in the original and this study as the laboratory results add up to maximum of 10 points out of 26 total points. On the other hand, in clinical work, laboratory tests are often not routinely taken 6 days after antibiotic treatment.

In conclusion, the BLOOMY 14-day and BLOOMY 6-month scores performed well in our cohort. However, it is more likely that any new score is adopted in clinical routine if it is for general patient population and not a score for cases with a certain laboratory result (i.e. positive blood culture). It is possible that these new scores will remain mostly in research purposes.