Prognostic accuracy of age-adapted SOFA, SIRS, PELOD-2, and qSOFA for in-hospital mortality among children with suspected infection admitted to the intensive care unit

Abstract

Purpose

The Sepsis-3 consensus task force defined sepsis as life-threatening organ dysfunction caused by dysregulated host response to infection. However, the clinical criteria for this definition were neither designed for nor validated in children. We validated the performance of SIRS, age-adapted SOFA, quick SOFA and PELOD-2 scores as predictors of outcome in children.

Methods

We performed a multicentre binational cohort study of patients < 18 years admitted with infection to ICUs in Australia and New Zealand. The primary outcome was ICU mortality. SIRS, age-adapted SOFA, quick SOFA and PELOD-2 scores were compared using crude and adjusted area under the receiver operating characteristic curve (AUROC) analysis.

Results

Of 2594 paediatric ICU admissions due to infection, 151 (5.8%) children died, and 949/2594 (36.6%) patients died or experienced an ICU length of stay ≥ 3 days. A ≥ 2-point increase in the individual score was associated with a crude mortality increase from 3.1 to 6.8% for SIRS, from 1.9 to 7.6% for age-adapted SOFA, from 1.7 to 7.3% for PELOD-2, and from 3.9 to 8.1% for qSOFA (p < 0.001). The discrimination of outcomes was significantly higher for SOFA (adjusted AUROC 0.829; 0.791–0.868) and PELOD-2 (0.816; 0.777–0.854) than for qSOFA (0.739; 0.695–0.784) and SIRS (0.710; 0.664–0.756).

Conclusions

SIRS criteria lack specificity to identify children with infection at substantially higher risk of mortality. We demonstrate that adapting Sepsis-3 to age-specific criteria performs better than Sepsis-2-based criteria. Our findings support the translation of Sepsis-3 into paediatric-specific sepsis definitions and highlight the importance of robust paediatric organ dysfunction characterization.

Introduction

While the prevalence and mortality of paediatric sepsis has become comparable to figures reported in adult ICUs in high-income countries [1,2,3], defining sepsis in the absence of a gold standard remains a challenge [4]. Following the 2001 consensus statement of the Society of Critical Care Medicine, paediatric sepsis was defined as infection in the presence of at least two out of four criteria of the systemic inflammatory response syndrome (SIRS) [5, 6]. The 2005 Consensus definition for paediatric sepsis maintained the requirement for SIRS, providing further specification on organ failure definitions [6]. The validity of SIRS criteria to identify and risk-stratify patients with sepsis has been challenged in adults, where insufficient sensitivity and specificity were demonstrated [7, 8]. While tachycardia and tachypnoea represent adaptive mechanisms commonly seen in febrile childhood infections, including diseases with near-zero mortality (e.g. bronchiolitis [9]), the face, and construct validity and sensitivity of SIRS criteria have not been studied in large cohorts of critically ill children.

The recent Sepsis-3 consensus definition emphasized that sepsis is differentiated from uncomplicated infection by the presence of life-threatening organ dysfunction as a result of a dysregulated host response to infection [10]. The Delphi process, systematic reviews, and development and validation cohorts leading to Sepsis-3 were based on adult populations and the task force recognized “the need to develop similar updated definitions for pediatric populations” [11]. However, current paediatric sepsis definitions remain essentially based on Sepsis-2, representing a major obstacle towards research, benchmarking, coding, and quality monitoring [11, 12]. The operationalization of clinical criteria to identify individuals meeting outcomes consistent with sepsis in Sepsis-3 is based on the Sequential Organ Failure Assessment (SOFA) score, however neither SOFA nor quick SOFA (qSOFA) were developed for children.

We hypothesized that in children admitted to ICU with infection, the presence of organ dysfunction would better identify patients at substantially higher mortality in comparison to the presence of ≥ 2 SIRS criteria. We compared the performance of SIRS criteria with measures of organ failure to characterize outcomes of children with sepsis.

Methods

Study population

Multicentre binational cohort study of patients < 18 years admitted to ICUs in Australia and New Zealand. The study was approved by the Human Research and Ethics Committee. Patients were eligible if they had presence of suspected or proven infection [7, 8] at admission to an adult or combined adult/paediatric ICU which contributed data to the ANZICS Adult Patient Database. The ANZICS Adult Patient Database captures prospective information on more than 90% of all adult and mixed adult/paediatric ICU admissions in Australia and New Zealand, but does not include the specialized paediatric ICUs which contribute to the separate ANZPIC registry (Supplementary Material). For patients that were transferred to a PICU, cases were followed through the Australian and New Zealand Paediatric Intensive Care Registry [2, 13].

Outcomes and definitions

In-hospital mortality was defined as the primary outcome. The composite secondary outcome was defined as in-hospital mortality or ICU length of stay of 3 days or longer [8, 14].

Physiological parameters on cardiorespiratory, neurologic, hepatic, renal and haematological organ dysfunction were prospectively recorded, capturing the highest and lowest value recorded during the first 24 h of ICU admission. SIRS criteria, pediatric logistic organ dysfunction score-2 (PELOD-2), SOFA, and qSOFA were calculated (Supplementary Table 1). Age-specific cut-offs to define SIRS criteria, and definitions for severe sepsis were applied as per the 2005 Pediatric Sepsis Consensus statement and the correction provided in a subsequent author’s reply [6, 15]; paediatric SIRS was defined as presence of ≥ 2 SIRS criteria, one of which must be abnormal temperature or WCC. All PELOD-2 items, except for pupillary dilatation and serum lactate levels, were available in the database to allow calculation of a PELOD-2 ranging from zero (best) to 22 (worst) [16]. Given the absence of age-specific SOFA definitions, we developed an age-adapted SOFA by defining increasing severity of cardiovascular and renal dysfunction using the PELOD-2 cut-offs for mean arterial blood pressure and serum creatinine increase. qSOFA was defined as a score composed of three binary variables (tachypnoea, altered mentation, hypotension) [10]. Age-specific qSOFA scores were defined by applying age-specific cut-offs for respiratory rate and systolic blood pressure, respectively, as per the corrected 2005 Pediatric Sepsis definitions [6, 15].

Statistics

Data are presented as percentages and numbers or means with standard deviation. We measured the discrimination of each score using the area under the receiver operating characteristic curve (AUROC). The sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV) was calculated for each score. A baseline risk model was developed to reflect the underlying risk of a patient developing the primary and secondary outcomes using available information at the time of ICU admission not contained in any of the scores. Univariate mixed effects logistic regression models, with a random effect for each site, were used to assess associations between patient factors and the primary outcome. Variables with associations p < 0.2 where considered for inclusion in a multivariable model. The same model was applied to the secondary outcome.

Sensitivity analyses were performed by using age- and sex-specific systolic blood pressure cut-offs based on the 5th percentile previously validated in children with sepsis [13, 17], and by using systolic blood pressure cut-offs used to define arterial hypotension in the corrected 2005 consensus definition [6, 15]. Analyses were conducted using Stata (version 14.0, Stata Corp, College Station, TX, USA). p values < 0.05 were considered significant.

Results

Study population

Between 2000 and 2016, 2,715 patients aged < 18 years were admitted to an adult or mixed ICU because of infection and recorded in the ANZICS Adult Patient Database. 121 episodes were excluded: one (0.04%) duplicated record, 49 (1.8%) patients > 16 years were transferred alive to another adult ICU with unknown outcome and 71 (2.6%) had missing outcome data. A final cohort of 2594 paediatric ICU admission encounters due to infection with known outcomes were identified, with a median age of 13 years (IQR 1–16). 151 (5.8%) children died in hospital and 949 (36.6%) died in hospital or experienced an ICU length of stay of 3 days or more (Supplementary Table 2). 1510 (58.3%) were classified as severe sepsis. The mortality in this group was 7.4% and the secondary outcome was met in 43.9% of patients with severe sepsis.

SIRS criteria

Of all 2594 episodes, SIRS data was incomplete in 323 (12.4%). 57/2271(2.2%) children did not present with any SIRS criteria during the first 24 h of ICU admission, 356 (15.6%) met one SIRS criterion, 1858/2271 (81.8%) fulfilled at least two SIRS criteria, and 1675 (73.8%) met paediatric SIRS (Table 1, Fig. 1). Mortality increased from 3.1% in presence of < 2 SIRS criteria to 6.8% if ≥ 2 SIRS criteria were present (between-group difference, 3.6%, 95% CI 1.6–5.7, p = 0.005, Fig. 2, Supplementary Figs. 1 and 2). Using patients with < 2 SIRS criteria as a reference, the relative increase in the primary and secondary outcomes was not significant for 2 SIRS criteria, but was significant for 3 SIRS criteria (primary outcome OR 1.94, 95% CI 1.02–3.70); secondary outcome OR 1.46, 95% CI 1.12–1.92), and for 4 SIRS criteria (primary outcome OR 3.31, 95% CI 1.72–6.37; secondary outcome OR 2.97, 95% CI 2.19–4.03).

Table 1 Distribution of signs meeting SIRS criteria in children admitted to ICU with infection, according to primary outcome (mortality) and secondary outcome (mortality or ICU stay ≥ 3 days)

Fig. 1

Distribution of patients < 18 years with infection by SIRS criteria, PELOD-2 score, SOFA and qSOFA score measured during the first 24 h of ICU admission

Fig. 2

Mortality by SIRS criteria, PELOD-2 score, SOFA and qSOFA score measured during the first 24 h of ICU admission in patients < 18 years admitted with infection

Using models adjusted for baseline risk, the relative odds of death increased in the presence of paediatric SIRS (OR 1.83, 95% CI 1.13–2.99, p = 0.015), or the presence of any ≥ 2 SIRS criteria (OR 2.00, 95% CI 1.10–3.64, p = 0.023). The relative odds of the secondary outcome increased in the presence of paediatric SIRS (OR 1.54, 95% CI 1.25–1.90, p < 0.001), and in the presence of ≥ 2 SIRS criteria (OR 1.50, 95% CI 1.21–1.85, p < 0.001). Overall, each additional SIRS criterion was associated with a 50% increase in the relative odds for the primary outcome (OR 1.50, 95% CI 1.25–1.81, p < 0.001) and a 38% increase for the secondary outcome (OR 1.38, 95% CI 1.27–1.52, p < 0.001).

SOFA, PELOD-2 and qSOFA

1690 (74.2%) of infected patients had an age-adapted SOFA score of ≥ 2 (Fig. 1). The mortality increased from 1.9 to 7.6% if the SOFA score was ≥ 2 (between-group difference, 5.7%, 95% CI 4.0–7.4, p < 0.001, Fig. 2). The risk of the secondary outcome increased from 17.6 to 46.1% in those with a SOFA score of ≥ 2 (between-group difference, 28.5%, 95% CI 24.7–32.4, p < 0.001, Supplementary Fig. 2).

When assessing organ dysfunction using PELOD-2, a score of ≥ 8 performed best in identifying patients at higher mortality: There were 374 (14.4%) children with a PELOD-2 score ≥ 8, with a mortality of 22.2% (versus 3.0% in those with scores < 8, p < 0.001) and a secondary outcome incidence of 79.4% (versus 20.6%, p < 0.001). Among children with a PELOD-2 score of ≥ 2, mortality was 7.3% (versus 1.7% in those with scores < 2, p < 0.001) and the secondary outcome occured in 43.8% (versus 17.2%, p < 0.001).

For those who had a qSOFA (altered mentation, arterial hypotension, and tachypnea) score of ≥ 2, mortality was 8.1% (97/1200) in comparison to 3.9% (41/1059) with a qSOFA < 2 (between-group difference, 4.2%; 95% CI 2.3–6.1, p < 0.001).

Comparison of SIRS, severe sepsis, SOFA, PELOD-2 and qSOFA

There were significant differences in discrimination of both primary and secondary outcomes in crude and adjusted analyses (p < 0.001). For the primary outcome, discrimination was highest for SOFA (AUROC = 0.829) which was significantly higher than SIRS (AUROC = 0.727, p < 0.001), severe sepsis (AUROC = 0.711, p < 0.001), and qSOFA (AUROC = 0.739, p < 0.001), though not significantly higher than PELOD-2 (AUROC = 0.816, p = 0.970). For the secondary outcome, discrimination was highest for PELOD-2 (AUROC = 0.771), which was significantly higher than SIRS (AUROC = 0.676, p < 0.001), severe sepsis (AUROC = 0.677, p < 0.001), qSOFA (AUROC = 0.682, p < 0.001), and SOFA (AUROC = 0.751, p < 0.001) (Table 2, Fig. 3, Supplementary Table 3). The best binary performance for PELOD-2 was using a cutpoint score of ≥ 8, resulting in an adjusted AUROC for in-hospital mortality of 0.812 (95% CI 0.774–0.851), and a sensitivity of 88.1% and a specificity of 55.7%.

Table 2 Crude and adjusted AUROCs for discrimination characteristics of SIRS, SOFA, PELOD-2 and qSOFA on ICU admission in patients < 18 years with suspected or confirmed infection

Fig. 3

Comparison of area under the receiver operating characteristic curves (AUROCS) to discriminate in-hospital mortality (primary outcome) and in-hospital mortality or ICU length of stay of 3 days or more (secondary outcome) for SIRS criteria, SOFA, qSOFA, and PELOD scores at ICU admission. AUROCs are shown for primary (a, c) and secondary (b, d) outcomes using crude (a, b) and adjusted (c, d) models

Sensitivity analyses were performed using different adjusted models, and based on different thresholds to define qSOFA, which resulted similar (Online Supplementary Table 3).

Discussion

In this multicentre cohort of 2594 children aged < 18 years admitted to ICU with infection, we externally validated and assessed the discriminatory capacities of SIRS, severe sepsis, SOFA, PELOD-2, and qSOFA. We observed superior prognostic accuracy of SOFA and PELOD-2, both for in-hospital mortality and for the composite outcome of in-hospital mortality or ICU length of stay of ≥ 3 days, in comparison with SIRS, severe sepsis, or qSOFA. SIRS lacked specificity to identify children with infection at substantially higher mortality risk.

Key features underlying the Sepsis-3 consensus definition relate to the differentiation of sepsis from non-life-threatening infection, operationalization of the definition, and establishment through a data-driven process using large cohorts [18]. In contrast, paediatric Sepsis-2 definitions focus on systemic inflammation, applying non-validated criteria commonly seen outside sepsis, and have specific requirements for individual organ dysfunctions, attributing more weight to cardiovascular or respiratory organ dysfunction [19, 20]. The paradigm of SIRS as a feature of paediatric sepsis has been maintained for two decades, but neither SIRS nor the particular organ dysfunction criteria—which overlap with multiorgan dysfunction—to define severe sepsis in children have been externally validated in large ICU cohorts. Previous studies reported that SIRS criteria are met in > 90% of febrile children presenting to ED, of which < 5% require ICU admission [21]. We demonstrate that ≥ 2 SIRS criteria are present in 81.8% of paediatric patients admitted to ICU with infection, resulting in poor specificity and poor positive predictive value to capture patients at risk for adverse outcomes. While each additional SIRS criterion was associated with an increase in the relative odds of the primary, and secondary outcomes, significance was only reached when 3 or 4 SIRS criteria were present. Furthermore, the sensitivity of ≥ 2 SIRS criteria to discriminate in-hospital mortality was inferior to an incremental increase by ≥ 2 points in SOFA or PELOD-2, challenging the common notion that SIRS has excellent sensitivity for sepsis. Our findings are supported by a study in critically ill adults, demonstrating shortcomings in sensitivity, specificity, and validity of ≥ 2 SIRS criteria to define sepsis [7]. The limited utility if SIRS is further illustrated by substantial differences when applying Sepsis-3 versus Sepsis-2 criteria to infected children [11]. In a recent study on children with bloodstream infection, 30-day mortality was 1% in the presence of bacteraemia and SIRS without organ dysfunction, but increased to 17% when organ dysfunction was present [22]. Hence, our findings support abandoning SIRS since operationalising inflammation performs inferior to operationalising organ dysfunction when predicting death or prolonged ICU stay as outcomes.

Both age-adapted SOFA and PELOD-2 were superior to SIRS in identifying patients with infection at greater risk of mortality. Given the limited evidence on optimal blood pressure thresholds [23, 24], and to avoid overfitting of models, we applied the validated PELOD-2 age-specific cut-offs for cardiovascular and renal dysfunction, which may partially account for the similar performance observed between SOFA and PELOD-2 [25]. When simplifying the discrete scores (SIRS, SOFA, PELOD-2, qSOFA) to binary categorizations, PELOD ≥ 8 performed best. Notably, Sepsis-3 defined an increase in SOFA by ≥ 2 points based on the a priori requirement to identify presence of ≥ 1 (new) organ dysfunction to characterize sepsis, and not by post hoc derivation of optimal cut-offs. While a PELOD-2 ≥ 2 will capture patients with ≥ 1 organ dysfunction, PELOD ≥ 8 performed best in our study but such scores will predominantly reflect multiorgan dysfunction. Our findings support the operationalization of clinical criteria to paediatric patients with sepsis [10] and are highly comparable to a large external validation cohort in critically ill adults captured by the same database [8]. Leclerc and colleagues assessed PELOD-2 in 862 children with infection recruited in the original PELOD cohort and reported a high in-sample performance [23]. Our findings are further supported by a recent single centre PICU study including patients up to 21 years which tested a paediatric SOFA adaptation [26] based on the same PELOD-2 cut-offs for arterial hypotension and renal dysfunction but applying a more granular score increase, resulting in excellent performance. In contrast to the paper by Matics et al. [26] which did not report on SIRS or severe sepsis, we analyzed multicentre data of patients < 18 years, using mortality as the primary and mortality and/or PICU length of stay ≥ 3 days as the composite secondary outcome, and applied crude and adjusted analyses using similar methodology to adult Sepsis-3 validation cohorts [8].

qSOFA has been proposed as a screening tool in adults with infection, prompting assessment for evidence of organ dysfunction on hospital floors or in the ED [10]. In our study, the performance of our adapted qSOFA score to identify children who subsequently died or had prolonged length of stay was only moderate, which potentially could reflect the use of ventilation, sedation, and inotropes in ICU, altering respiratory rate, blood pressure, and GCS. In comparison, other PICU sepsis scores and paediatric Early Warning Tools have reported AUROCs of > 0.80 [12, 27]. Specific features of paediatric sepsis and septic shock [28], such as late development of arterial hypotension and a higher proportion of fulminant presentations [12, 22, 29, 30], warrant improved rapid identification of infected paediatric patients with organ dysfunction [30] across emergency department, hospital ward, and ICU settings. Given the high proportion of patients with organ dysfunction (74% with SOFA ≥ 2) in our cohort, future studies are needed to test the discriminatory performance in cohorts of lower average acuity.

The key strengths of this study relate to the application of stringent data-driven validation procedures to allow comparison of the prognostic accuracy of SIRS, severe sepsis, SOFA, PELOD-2 and qSOFA. Moreover, given the limitations of using mortality as an outcome, analyses included the composite outcome of mortality or ICU length of stay of 3 days or longer, aligned with adult Sepsis-3 studies [8, 13]. We deliberately did not restrict the study to admissions coded as sepsis but instead included all children admitted to ICUs with suspected and confirmed infection. Diagnoses were based on assessment by trained ICU specialists and not administrative coding.

Our study carries several limitations. First, it is based on a binational prospective dataset of adult and mixed adult-paediatric ICUs, whereas the main PICUs contribute to a different registry, which may have led to selection bias. In contrast to the paediatric ANZPIC registry, the ANZICS APD has been prospectively collecting data on SIRS, APACHE and SOFA score in patients admitted to the contributing units which ensured consistent practices of organ dysfunction assessment and data monitoring. Due to the high centralization of PICU services in Australia and New Zealand, it is common for critically ill children outside large metropolitan areas to be admitted to mixed ICUs. In the present study, 149 (5.8%) children required secondary interhospital transfer to a PICU, all of which were tracked to capture the primary and secondary outcome. Second, capturing the worst parameters within the first 24 h of ICU admission may not capture peak disease severity, resulting in lower performance of the scores [26]. However, prediction requires early assessment by definition and the passage of time is associated with competing risk bias due to death or discharge from ICU. In a recent study on a different cohort of septic children admitted to specialized PICUs we have demonstrated that a small set of clinical variables available within the first hour of PICU admission allows to establish robust severity stratification for paediatric sepsis mortality [12]. Third, although data collection for this binational ICU registry had been monitored using regular quality controls and mandatory audits, data had not been primarily captured for sepsis studies. Fourth, the SOFA score was modified as detailed information on vasopressor type and dose was not consistently available. Finally, two items of the PELOD score, including pupil size and serum lactate levels, were not available in the database, which may have reduced the performance of PELOD. Several studies have identified lactate as one of the best predictors of paediatric sepsis severity [12, 31, 32].

In conclusion, the two SIRS variables based sepsis criteria had poor specificity and diagnostic performance to discriminate children with infection at substantially higher mortality risk. In contrast, SOFA and PELOD-2 had significantly greater prognostic accuracy for in-hospital mortality. Our findings indicate that age-specific translation of Sepsis-3 definitions to critically ill children using validated measures of organ dysfunction should be considered in the next revision of paediatric sepsis definitions. In contrast, the performance of qSOFA to identify patients with organ dysfunction at risk for worse outcomes was poor, and may not be of sufficient clinical value to be recommended as a screening tool for paediatric age groups within the ICU.