Background

Severe sepsis and septic shock are a substantial burden to health care, affecting millions of patients around the world each year [1]. The average cost of care for a patient with severe sepsis is about 22.000 USD [2]. It is often thought that severe sepsis is primarily seen on the intensive care unit (ICU). However, studies point out that the majority (50 – 68%) of patients with severe sepsis are admitted to a general medical ward [3], with a mortality rate around 26 – 29.5% [4, 5]. Since the diagnosis ‘severe sepsis’ is often poorly documented in the medical records by the treating clinicians, specific sepsis-targeted measures may not have been performed and antimicrobial use may have been inappropriate [3].

As the necessary first step in the improvement of appropriate use in patients with sepsis, guidelines have been developed that describe appropriate antimicrobial use in patients with sepsis admitted to a general medical ward or an Intensive Care Unit (ICU). Despite the availability of these guidelines, antimicrobials are used inappropriately: several studies show that inappropriate initial antimicrobial use in patients with severe sepsis or septic shock is associated with a reduction in survival [6–9]. As a second important step towards change and improvement of daily clinical care, the guideline-based development of quality indicators has been suggested [10, 11]. Quality indicators (QIs) are measurable elements that can be used to gain insight into the appropriateness of the given antimicrobial treatment, which is important to set priorities and to focus improvement. The aim of our study was to develop a concise and therefore feasible set of QIs to measure and monitor the appropriateness of antimicrobial use in adults with sepsis admitted to a general medical ward and/or ICU.

Methods

Delphi survey

The Dutch Working Party on Antibiotic Policy (SWAB) publishes evidence-based guidelines for antimicrobial use. We used the guideline for antimicrobial use in hospitalized patients with sepsis (published online in 2010) as a starting point for the development of a set of QIs [12]. This guideline covers antimicrobial use in all hospitalized adult patients with sepsis, except antimicrobial use in sepsis associated with indwelling intravascular devices that are not removed (tunnelled catheter or totally implantable vascular access devices) and therefore requires different therapy.

We used a systematic approach -the RAND-modified Delphi method [13, 14]- to develop a set of QIs in order to measure the appropriateness of antimicrobial treatment in adult patients with sepsis admitted to a general ward and/or ICU (Figure 1). For developing QIs by means of the Delphi method medical ethical approval was not required.

Figure 1
figure 1

The step-wise RAND-modified Delphi method. a. Accepted: the potential QI was selected for the next round because of an overall median score of 8 or 9, without disagreement. Disagreement was defined as the case in which less than 70% of the scores were in the top tertile (scores 7, 8, or 9). b. Discussion: the QI had a median score of 7 without disagreement or a median score of 8 or 9 with disagreement, and so it was discussed during the consensus meeting. c. Rejected: disagreement between panel members and the median was also lower than 8; the potential indicator was deselected and not discussed during the consensus meeting. d. Merged: multiple indicators were ‘rejected’ and merged into a composite, more generic indicator. e. Added: the indicator was proposed by one of the experts and was added to the initial set of indicators.

Full size image

Extraction of guideline recommendations

One infectious diseases physician and one quality-of-care specialist independently extracted key recommendations from the national guideline for antimicrobial use in hospitalized patients with sepsis. Disagreements were resolved by consensus. The selected recommendations were translated into potential QIs and each indicator was graded to determine its scientific soundness (level of evidence), using Tables 1 and 2[15].

Table 1 Grading system for methodological quality of individual studies[15]
Full size table
Table 2 Level of evidence of conclusions
Full size table

First questionnaire round

The list of the potentially relevant QIs was converted into a written questionnaire and used for the RAND-modified Delphi method to achieve expert consensus on these QIs. The consensus procedure was performed between February and April 2011. All authors of the above mentioned national guideline, and an additional intensive care specialist and a hospital pharmacist were approached, and they all agreed to participate in the multidisciplinary expert panel. Our final expert panel consisted of four infectious diseases physicians (all working primarily outside the ICU), two medical microbiologists, two hospital pharmacists, three intensive care specialists, two haematologists and one general surgeon (14 experts).

The questionnaire was sent by email to the experts, asking them to rate the QIs using the following criteria:

  • The potential QI leads to health gain for the patient, less bacterial resistance or promotes efficiency of care;

  • The potential QI is generalizable to all adult patients treated for sepsis with antimicrobial use;

  • There is enough scientific evidence or expert consensus to justify the recommended care.

The expert panel was asked to rate the potential QIs using a 9-point Likert scale (with 1 denoting “definitely not appropriate care” and 9 denoting “definitely appropriate care”). The answer category ‘cannot assess’ was also available. The panel members (experts) were asked to add suggestions and comments regarding the potential QIs, and also to add additional potential QIs or topics for consideration.

The results from the first questionnaire were analysed using a standardized Microsoft Office Access-based consensus tool. Potential QIs rated with an overall median score of 8 or 9 without disagreement were considered to be face valid and reliable [16], and were accepted as preliminary indicators. Disagreement was defined as the case in which less than 70% of the scores were in the top tertile (scores 7, 8, or 9) [16]. If there was disagreement and the median score was below 8, the potential indicator was rejected and not discussed during the consensus meeting. QIs with a median score of 7 without disagreement or a median score of 8 or 9 with disagreement were discussed during the consensus meeting.

Expert panel meeting

All panel members were invited for a consensus meeting during which an overview of the first round ratings was provided. Goals of the consensus meeting were to achieve consensus on the QIs with a median score of 7 without disagreement or a median score of 8 or 9 with disagreement, and to rephrase accepted indicators using the comments from the panel in the first round of questionnaires. These QIs and the suggested new QIs by the panel members were discussed and reformulated when necessary.

Second questionnaire round, ranking procedure

After the consensus meeting, all discussed, reformulated and added potential indicators were included in a second questionnaire. First, the panel was asked whether they agreed (yes or no) with the proposed indicators and their definitions. Redefined indicators were accepted if at least 70% of the experts agreed with the new formulation. Second, the panel was asked to prioritize the potential indicators by selecting the ‘top 5’ of most important indicators. For each number-one ranking by a panel member, we granted a potential QI five points, for each number-two ranking, we granted four points and so on. QIs receiving more than 15% of the maximum possible ranking points were considered to be the most relevant indicators.

Results

Extraction of guideline recommendations

Key recommendations were extracted from the national guideline independently by one infectious diseases physician (S.E.G.) and one quality-of-care specialist (M.E.L.J.H.). In consensus, 40 key recommendations were extracted from the national guideline and translated into potential QIs.Figure 1 shows the entire Delphi method as performed in the next steps.

First questionnaire round

The 40 potential QIs were scored by 12 of the 14 panel members during the first questionnaire round (86% response rate). Twenty-two potential QIs had a high score (8 or 9) without disagreement and were accepted, and nine potential indicators were rejected, see Figure 1 and Table 3. The panel did not agree on nine potential QIs, they either had a median score of 7 without disagreement or a median score of 8 or 9 with disagreement. One new potential indicator regarding adapting the antimicrobial dose to renal function was proposed by one panel member. Results are shown in Table 3.

Table 3 Results Delphi procedure: first questionnaire, consensus meeting and second questionnaire
Full size table

Expert panel meeting

Six panel members (four infectious diseases specialists, one medical microbiologist and one intensive care specialist) were present during the consensus meeting (43%). The 22 accepted QIs with comments from the panel, the nine potential QIs with disagreement and the new indicator were discussed during the meeting.

From these 32 QIs, this smaller panel accepted 13 potential QIs; two QIs were rephrased and one new potential QI was added. The other 17 QIs were rejected and merged into four new composite QIs. These potential QIs had a more generic formulation and contained: starting empirical antimicrobial therapy; duration of therapy; changing empirical therapy to pathogen-directed therapy; and harmonizing local guidelines with the national guideline (indicator 42 – 45, Table 3). This resulted in a total set of 20 potential QIs.

Second questionnaire round, ranking procedure

After the consensus meeting, the resulting 20 potential indicators were sent to the entire panel for comments and approval. This questionnaire was scored by 13 of the 14 panel members (93% response rate). All 20 indicators were accepted, because in all cases 70% or more of the panel members agreed with the new content and rephrasing. Indicator 42 and 43 (local guidelines should correspond to the national guideline and prescribe according to the national guideline) were merged into one indicator because four out of the 13 panel members (31%) found them to be overlapping indicators. In the same questionnaire the panel members were also asked to rank the QIs. Out of this set of 19 indicators, five indicators received more than 15% of the maximum possible ranking points and were prioritized. They were found to be the most important QIs for antimicrobial care in adult patients with sepsis. The results of the second questionnaire are shown in Table 3 and Table 4 shows the final set of QIs. For the attendance list of the participation of the Delphi procedure and the development of the sepsis guideline, see Additional file 1: Table S1.

Table 4 Final set of quality indicators to monitor antimicrobial use in hospitalized adult patients with sepsis
Full size table

Discussion

This systematic, stepwise method combining evidence and expert opinion, generated a valid, concise and therefore feasible set of five QIs to measure appropriate antimicrobial use in adult patients with sepsis admitted to general medical wards and/or ICUs. One QI specifically applies to patients with severe sepsis or septic shock (Table 4, indicator 2).

The issue of local resistance patterns and national versus local guideline recommendations for empirical treatment choices was extensively discussed during the Delphi consensus meeting. The national sepsis guideline underlines that hospitals can and should deviate from the recommendations based on local resistance patterns. We therefore favored to follow the national guidelines, but to guarantee the generalizability of the QIs another QI was added during the consensus meeting; local guidelines should correspond to the national guideline, but should deviate based on local resistance patterns (QI number 42).

In a systematic review performed by McGregor et al. empiric or definitive antibiotic therapy was considered to be appropriate if the regimen exhibited in vitro activity against the isolated pathogen(s) [17]. We derived our key recommendations for appropriate antibiotic therapy from the national, evidence-based guideline for antimicrobial use in hospitalized patients with sepsis. This, according to the national and international experts, implies more than correct (empirical or definitive) antibiotic therapy alone: it defines correct antimicrobial use at patient level along the entire antibiotic pathway, from start (including appropriate diagnostics) to streamlining and discontinuing of antimicrobial therapy.

The final set consists of independent QIs, that can be used to provide insight into the appropriateness of current antimicrobial use, to identify where there is room for improvement [18]. In a recent paper by our group [19] we found that patients with urinary tract infections who in particular adherence to the total set of QIs, showed a significant dose–response relationship with a shorter length of hospital stay. This argues for application of the QIs in a bundle approach.

This is the first study that specifically describes the development of QIs for the entire antimicrobial treatment of sepsis patients, also outside the ICU, via the modified Delphi technique. Several studies have described the use of quality measures for antimicrobial sepsis treatment [20–25]. However, some indicators were not systematically developed using a Delphi method [21], and some only focused on optimal sepsis care on the ICU [20, 24] or focused on the start of treatment (first 24 hours) and not on the entire clinical course [22, 23, 25].

The results of our study show resemblance with the concise Surviving Sepsis Campaign Care Bundle originated from the guideline [1, 26]. They also defined the optimal start of antimicrobial treatment and taking cultures as important parameters, only our panel members also defined streamlining as an important QI.

Our study has several strengths. We used the systematic modified Delphi method, a common and validated technique in which scientific evidence is combined with expert opinion [13, 14, 27, 28]. Boulkedid and colleagues recently reviewed its use and reporting, and formulated a practical guideline for using this RAND modified Delphi technique. Our procedure is consistent with their guideline [29]. Our panel was multidisciplinary, with 14 experts from 6 different specialties. Furthermore, the response rate of the first and second questionnaire (86% and 93%) was high, which increases the validity of the results.

A limitation of this study is the national setting in which the QIs were developed, with a Dutch national expert panel and a Dutch guideline. This leads to the question of whether the results can be generalized to a wider international population. However, the guideline reviewed and graded the recent international literature and the QI development was performed by a multidisciplinary panel, in which several members have international experience and expertise on the topic.

Another potential limitation was the attendance at the expert panel meeting, which was 43%. However an extensive summary concerning the results from the consensus meeting was sent to all panel members, as they were asked to give their final remarks and approval for the added and rephrased potential QIs. Since 93% returned the second questionnaire, we believe that an incomplete attendance did not undermine the validity of the results.

Conclusion

We describe the complete and precise development of a concise set of quality indicators for optimal antimicrobial use in hospitalized adult sepsis patients, by means of the Delphi method. This paper can be used as manual for others, since transparency of healthcare becomes more important worldwide, and QIs give insight into the appropriateness of daily clinical care. At this moment, we are testing the applicability of this set of QIs in practice in 22 hospitals. After establishing their clinimetric properties we will analyze the association between adherence to the QIs and outcomes like duration of hospital stay. In the future, our guideline-based indicators can be used for national monitoring of antimicrobial use in hospitalized adults with sepsis or for quality improvement projects. This information is crucial for antimicrobial stewardship teams to help set priorities and to focus improvement.