Introduction

Sepsis is a life-threatening disease caused by a dysregulated host response to infection. Unfortunately, both early recognition and definitive confirmation of the diagnosis have proven to be difficult as sepsis is a very heterogeneous syndrome [1]. Since 1991, conceptual thinking about sepsis has focused on the presence of a systemic inflammatory response syndrome (SIRS). However, SIRS criteria are neither sensitive nor specific for infection and do not necessarily indicate a dysregulated or life-threatening host response [2, 3]. Furthermore, sepsis definitions that relied on SIRS criteria were highly sensitive to minor variations in frequency and timing, thereby affecting the reliability of the sepsis diagnosis [2].

Sepsis-3 definitions were developed to improve risk stratification among patients with a suspected infection, and their predictive validity regarding unfavorable clinical outcomes have been confirmed several times by now [4,5,6,7,8,9,10,11,12]. Rather than a systemic inflammatory response syndrome, these sepsis definitions require the development of organ failure during an infectious episode, which is operationalized by an increase in the Sequential Organ Failure Assessment (SOFA) score [13, 14]. Similarly, the septic shock-3 definition requires the presence of elevated serum lactate levels in addition to fluid-resistant hypotension [15].

Sepsis-3 definitions were also established to increase uniformity among reported incidence and mortality rates [13,14,15]. A consistent diagnosis of sepsis and septic shock between centers is particularly important for research and benchmarking purposes. Clinical data can be sensitive to different coding approaches, complicating comparisons of sepsis epidemiology among different cohorts [16, 17]. However, as only a little attention has been focused on the robustness of sepsis-3 criteria, we studied the effects of minor variations in the interpretation of the criteria on the incidence and related mortality of sepsis-3.

Materials and methods

Study design and population

This study was embedded within the Molecular Diagnosis and Risk Stratification of Sepsis (MARS) cohort [18]. Consecutive adult patients with newly suspected infection either upon presentation or during ICU stay were enrolled in two Dutch tertiary ICUs between June 2011 and April 2015 (University Medical Center Utrecht) or between June 2011 and January 2014 (Academic Medical Center Amsterdam).

Patients who had been admitted to another ICU for more than 1 day before transfer to one of the study centers were excluded, because information about possible previous infections and organ failures was not available. Patients who had been treated for an infection in the week prior to ICU admission and subsequently were admitted with a new infection were also excluded to avoid possible overlap between pre-existent and newly acquired organ failures. The institutional review board approved an opt-out consent procedure (protocol number 10-056C).

Data and definitions

Trained researchers attended daily multidisciplinary rounds in the participating ICUs and prospectively recorded the presence of infection, SIRS, and organ failure [18, 19]. In this study, we use the terms “MARS-sepsis” and “MARS-shock” to indicate severe sepsis and septic shock according to prospective assessment of the presence of SIRS and organ failure, based on the 1991 and 2001 definitions of sepsis [20, 21] (see Table 1). The incidence and related mortality of MARS-sepsis are shown for illustrative purposes only and are not intended to provide a head to head comparison with sepsis-3 (which would have no clinical significance) nor to appraise the robustness of sepsis-3.

Table 1 Sepsis definitions
Full size table

The terms “sepsis-3” and “septic shock-3” were used to indicate events meeting the updated definitions. Organ failure for sepsis-3 was defined as life-threatening organ dysfunction caused by a dysregulated host response to infection [14]. We operationalized organ failure as an acute SOFA score increase of ≥ 2 points compared to pre-existing (acute or chronic) organ dysfunction before the onset of infection (Table 1). The increase in SOFA score had to occur between 2 days before the onset of infection and 1 day after the onset of infection (i.e., a 4-day window, see Fig. 1). This window was used because organ dysfunction may occur prior to, near the moment, or after the infection is recognized [5]. An infection was registered when empirical antimicrobial therapy was started by attending clinicians irrespective of the presence of SIRS or organ failure, and this day was regarded as its onset. Subsequently, the likelihood of each infection was subsequently adjudicated as none, possible, probable or definite, using detailed definitions derived from Center of Disease Control and International Sepsis Forum Consensus Conference criteria [18, 23, 24]. Only first ICU infections occurring during a hospital admission were included for analysis. Infections present at admission (having onset between 1 day before and 2 days after ICU admittance) and ICU-acquired infections (having onset more than 2 days after ICU admittance) were analyzed separately since we hypothesized that the extent of new organ failure might vary between these types of infection.

Fig. 1
figure 1

Hypothetical cases showing the influence of variations in organ failure definitions. SOFA = Sequential Organ Failure Assessment. The onset of infection (i.e., start of empirical antibiotic therapy) is day 0. Case 1 does not fulfill the sepsis-3 definition as there is no SOFA score increase of ≥ 2 points within the 4-day (or 2-day) time-window. However, case 1 fulfills the criteria if sepsis is defined by the presence of an absolute SOFA score of ≥ 2 (both in the 4-day and 2-day time-window). Case 2 fulfills the sepsis-3 criteria since there is an increase of ≥ 2 points between day 0 and day 1. In a reduced time-window, there is no increase observed between the day before infection and day of the onset of infection, and sepsis-3 criteria are not met

Full size image

To reconstruct baseline SOFA scores, raw pre-ICU clinical data were extracted from the hospital electronic health care record. All ICU data were collected prospectively [19]. In cases on dialysis dependency or having chronic renal insufficiency, the renal SOFA was assumed to be 3.

To evaluate the robustness, we assessed the influence of minor variations in the implementation of the sepsis-3 definitions (see Table 1). We based our variations on the methodology that was used in previous studies [4, 6, 13, 15]. First, we shortened the time window of observation by only including the day of clinical diagnosis and 1 day before (2-day window). Second, we explored the effects of an absolute SOFA score at the time of recognition of infection. Third, to mimic settings in which lactate is not always available, only vasopressor-dependent hypotension was required to fulfill the septic shock definition in cases where lactate levels were missing (see Table 1 and Fig. 1 for further explanations).

Statistical analyses

We calculated apparent incidences and related in-hospital mortality of sepsis-3 and MARS-sepsis. We calculated the percent agreement as the percentage of cases in which two sepsis definitions corresponded with each other. Sensitivity analysis was performed by excluding subjects with rejected infection (i.e., a post hoc likelihood of none). All analyses were performed and reported separately for infections at admission and ICU-acquired infections. Missing data were handled as described in Additional file 1: Table S1. Differences at baseline and clinical characteristics between the subgroups were analyzed using a Mann-Whitney U test, chi-square test, or McNemar test, as appropriate. Differences in mortality were calculated accounting for partially overlapping samples [25]. A p value < 0.05 was considered statistically significant. All analyses were performed using SAS 9.2 (SAS Institute Inc.).

Results

Among 1743 patients treated for an infection in the ICU, 1081 with an infection at ICU admission and 501 with an ICU-acquired infection remained for analysis (Figs. 2 and 3). Patient and infection characteristics are presented in Table 2.

Fig. 2
figure 2

Flowchart. ICU = intensive care unit

Full size image
Fig. 3
figure 3

Venn diagram comparing MARS-sepsis and sepsis-3 definitions. ICU = intensive care unit. Presented as frequencies of patients (%)

Full size image
Table 2 Characteristics of patients with infection on admission and with ICU-acquired infection and stratified by presence of sepsis-3 criteria
Full size table

Incidence and associated mortality

Table 3 shows the apparent incidences and related percent agreement of sepsis and septic shock according to the various definitions.

Table 3 Incidences of sepsis and related mortality according to core definitions
Full size table

Compared to prospectively recorded MARS-sepsis events, more patients fulfilled sepsis-3 and septic shock-3 criteria at ICU admission (60% vs 90%, and 27% vs 38%, respectively). Furthermore, agreement between the definitions was only 64% and 80%, respectively. For patients with ICU-acquired infections, the overall incidences of sepsis (54% vs 52%) and septic shock (19% vs 18%) were similar, yet the MARS and sepsis-3 criteria selected different individuals (58% and 81% agreement for sepsis and septic shock, respectively) (Table 3).

Hospital mortality was lower for patients with sepsis-3 and septic shock-3 than for patients with MARS-sepsis and MARS-shock (Table 3). Indeed, those patients who were exclusively identified by sepsis-3 at admission (33% of all patients) had a lower mortality rate than patients with organ failure according to both MARS-sepsis and sepsis-3 (37% vs 14%, respectively) (Additional file 1: Table S2). Nevertheless, mortality was > 10% for all definitions (Table 3, Additional file 1: Table S2 and Table S3). There were 110 (10%) and 167 (33%) patients with a rejected infection (i.e., those with a post hoc likelihood rated as none) at ICU admittance and during admission respectively. The exclusion of patients with rejected infection had a negligible effect on apparent sepsis incidences, mortality, and agreement (Table 3).

Robustness of the sepsis-3 definitions

Table 4 shows the results of the analyses to assess the robustness of sepsis-3 criteria. Minor variations in the timing of observations and criteria for organ failure considerably affected the apparent incidence of sepsis-3 at admission, ranging from 68 to 96% for the most restrictive and the most liberal definition, respectively. Using the same criteria, the incidence of septic shock-3 varied from 30 to 42%. For ICU-acquired infections, the incidence of sepsis-3 and septic shock-3 ranged from 22 to 99% and from 7 to 28%, respectively. Whereas these minor variations did not affect hospital mortality rates for infections at admission, and only marginally for ICU-acquired sepsis (Table 4).

Table 4 The influence of minor variations in diagnostic criteria on the apparent incidence and related mortality of sepsis
Full size table

Discussion

We assessed the incidence, mortality, and robustness of the sepsis-3 definitions in a large prospectively monitored cohort of ICU patients. We found that virtually all patients with a suspected infection met clinical criteria for organ failure and, as such, the sepsis-3 criteria did not have discriminative power in our setting. Furthermore, minor variations in the precise interpretation of the criteria required to meet the sepsis-3 definitions considerably impacted the apparent incidences of both sepsis and septic shock, while mortality remained comparable among the variations.

An anticipated advantage of the sepsis-3 definitions is that they may increase the comparability of sepsis incidence and related mortality among studies. Organ failure is explicitly defined by means of the SOFA score, possibly reducing subjective interpretation. Still, studies published to date have used many subtle variations on the original definition. For example, the original publication suggested to define organ failure as an acute change in the SOFA score of ≥ 2 points as a consequence of infection [14]. Subsequent validation studies, however, have largely disregarded this requirement of an acute SOFA increase. Instead, they used an absolute SOFA score of ≥ 2 points, applied different time-windows, and used different ways of taking chronic comorbidities into account [4,5,6,7,8,9,10, 13]. By applying similar (minor) variations to our data, we explored the robustness of the criteria and observed considerable variations in the apparent incidences of sepsis-3. Similar variations in the incidence of sepsis-3 and septic shock-3 are likely to occur in other studies, hence affecting the comparability of study results. Standardization of the operationalization of sepsis-3 criteria is therefore paramount to improve the generalizability of studies.

One of the most used and straightforward methods of defining organ failure for sepsis diagnosis is the use of an absolute SOFA score, thereby disregarding any pre-existent organ failure. And yet, several problems might arise using this approach. First, almost all ICU patients fulfill these criteria, indicating that the criteria have no discriminatory power in ICU settings. Second, an absolute SOFA score disregards the etiology of organ failure. Organ failure might have been present already before infection (e.g., due to non-infectious diseases or pre-existent comorbidities) and is therefore not caused by the infection itself. To illustrate, in the current study, up to 33% of the patients who developed sepsis-3 actually did not have an infection in a post hoc adjudication. It therefore remains essential to differentiate between infectious and non-infectious causes for organ failure. We find that future efforts should also be directed to improve (risk) stratification of septic patients and enrich classification by inclusion of additional variables, such as type of organ failure, number of different organ dysfunctions, site of infection, and possibly biomarkers [26].

Our study has some limitations. First, organ failure data were often missing before ICU admission, which was also noticed in the original assessment of sepsis-3 [13]. Second, we based our severe sepsis and septic shock definitions on consensus literature. Nevertheless, the exact application of the definitions in our study might be different from others. Of note, some of the described restraints of the sepsis-3 criteria also apply to previous sepsis definitions.

Conclusions

Virtually all patients who have suspected infection upon presentation to the ICU meet sepsis-3 criteria, making this definition less suitable for risk stratification in this setting. Furthermore, caution should be taken when using the sepsis-3 definitions to report incidences and related outcomes of sepsis, as they are very sensitive to minor variations in timing and interpretation of organ failure criteria. These criteria should therefore be specified in great detail, and applied very consistently, in all future publications on the topic.