Background

Quality improvement collaboratives (QICs) bring together multidisciplinary teams across different organizations to work over a period of time towards defined improvement goals [1]. Though QICs have enjoyed enduring popularity as a means of securing improvement, evidence to support their effectiveness is equivocal [2]. This may be because studies of collaboratives have often used weak designs that are biased in favor of positive findings, tend towards measurement error, and perform poorly in ruling out alternative explanations for any improvement detected, including secular trends [3, 4]. Further, collaboratives may tackle complex, multi-faceted interventions, some of which may have weak underlying evidence for their effectiveness [5]. In this article, we report a cluster-randomized trial of a QIC based on the Breakthrough Series (BTS) model that sought to improve compliance with processes associated with high-quality stroke care in hospitals in the Northwest of England.

Stroke affects up to 15 million people per year worldwide [6]. Of these, 17% die within one month. Survivors often suffer long-term disability, resulting in increased health and social care costs. The impact of stroke can be substantially mitigated by optimal care, but data from the 1990s showed that UK patients had poorer outcomes than many European countries [7]. For the last decade, the English National Sentinel Audit of Stroke (hereafter ‘National Audit’) has collected data biennially from hospitals in England and Wales on over two hundred measures of stroke care [8]. This has revealed dramatic variation in the care provided by different organizations, and has identified nine evidence-based processes associated with improved outcomes [9]. In 2008, prior to the commencement of this research, National Audit data indicated that stroke care in the Northwest of England was among the worst in the UK; for example, only four of every ten patients received a brain scan within 24 hours.

Stroke 90:10 was a BTS collaborative that aimed to improve compliance to an average of 90% on each of the nine processes by the next National Audit (2010), up from the baseline average of 72% in 2008. The nine processes targeted by the collaborative were in effect pre-defined, since they had already been selected as the basis of performance assessment for the National Audit. However, the aims of Stroke 90:10 went beyond simply enhancing reported performance on the national measures. The QIC sought to improve not just average compliance, but also reliability. Reliability was defined to mean reducing variation and delivering higher levels of consistent performance. To motivate and assess reliability, the nine key processes were organized into two care bundles: one for early hours care (Bundle 1) and one for rehabilitation (Bundle 2) [10].

We conducted a cluster randomized controlled trial with an interrupted time series design to determine whether hospitals participating in the Stroke 90:10 collaborative improved more than non-participating controls, as assessed by compliance with the two bundles of care. We compared across collaborative and control hospitals the amount of improvement on bundle compliance.

Quality improvement collaboratives

A QIC is an organized, multifaceted approach to quality improvement that involves five essential features: an agreed topic and aim; clinical and quality improvement experts who provide support for improvement by acting as a faculty; multi-professional teams from multiple sites who participate in the QIC; use of an agreed model for improvement (for example, setting an aim, collecting data and testing changes); and the collaborative process involves a series of structured activities including face-to face-meetings [5].

As well as mastering quality improvement techniques and applying them in their own organizations, teams participating in QICs are encouraged to create a sense of common purpose and commitment, to generate enthusiasm, mutual support, and shared learning [11]. Several variants of QIC models exist, with the Institute of Healthcare Improvement’s BTS design among the most widely used [1]. A BTS collaborative typically involves collaborative faculty comprising quality improvement experts, three learning sessions (collaborative meetings), and action periods where participants implement activities, including Plan-Do-Study Act (PDSA) cycles. QICs are often used to implement evidence-based processes grouped into care ‘bundles’ or composites of processes. Behind the concept of bundles is the idea that groups of interrelated processes, rather than individual processes, are the appropriate target both for standards of practice and for analysis [12]. Bundles require that all patients receive all elements of the bundle (or have documented evidence of valid exception). There is also a theory that the bundle as a whole will achieve better results than the sum of its parts [13].

The stroke 90:10 QIC

The Stroke 90:10 collaborative ran from July 2008 to December 2010, with participating organizations recruited between July and December 2008. Participating site teams received an extensive package of support from the program office, which was based at Salford Royal Hospitals NHS Trust. Support included executive mentoring visits; direct access to the Stroke 90:10 project director; an improvement advisor and one another via a web-based portal (extranet); and weekly online sharing and learning sessions. Teams were asked to produce monthly reports to reflect on their performance. The project director (author MP) also met with each hospital’s chief executive and the team to review progress twice.

All hospitals participating in the collaborative were asked to commit to:

  1. 1.

    appointing an executive lead, a physician leader, a site lead, and a project team comprising relevant leaders from clinical and ward areas;

  2. 2.

    taking part in one two-day and two one-day learning sessions, 90 days apart, that provided instruction in the theory and practice of improvement, offered teams advice and guidance, and shared cumulative results;

  3. 3.

    participating in ongoing collaborative activities over the course of the program;

  4. 4.

    using ‘The Model for Improvement’ to implement changes at the point of care and test them for local feasibility, reliability, and evidence of improvement in relation to the two bundles of processes relating to stroke care (Table 1) [14];

  5. 5.

    collecting data on 20 randomly selected patients per month;

  6. 6.

    submitting the data to a bespoke web-based system linked to the National Audit.

Table 1 Care bundles for stroke 90:10
Full size table

Methods

Study design

A cluster randomized controlled trial was used with an interrupted time series design to compare the hospitals participating in the Stroke 90:10 QIC with the non-participating controls. The study sought to compare the amount of improvement on the two evidence-based bundles of care (early hours and rehabilitation) used in the QIC.

This study was approved by Tameside and Glossop Research Ethics Committee (Ref: 08/H1013/55) and was registered as a clinical trial with the International Standard Randomized Controlled Trial Number Register (Ref: ISRCTN13893902).

Setting and participants

All NHS hospital Trusts in the Northwest of England were invited to participate based on the pre-defined inclusion criteria of: a minimum of ten inpatient dedicated stroke beds (a ‘stroke unit’); agreement to participate signed by the chief executive; agreement to participate from a consultant in stroke medicine (or equivalent); a dedicated multidisciplinary stroke team; and availability of case notes for review.

Randomization

We used a stratified-randomization approach. Hospitals were stratified by stroke performance (Sentinel Audit score above or below 60) in the 12 months preceding baseline data collection (2007 and 2008). Within each group, a computer-generated list was used to randomly allocate 12 hospitals to the intervention group and 12 to the control group. The intervention group took part in the QIC from January to October 2009, but the control group did not join the QIC until January 2010. The control group of hospitals, who waited to join the QIC, provided an ‘untreated’ control group. The nature of the trial meant that participants could not be blinded to group allocation.

Sample size

A conservative approach to the power calculation was undertaken, based on a simple baseline average versus end of study average comparison. This was preferable to basing the power calculation on an interrupted time series using a fixed number of charts per month, because we did not know in advance how the sites would choose to implement their activities or the likely impact of these activities (for example, in a simple linear fashion, stepped or incremental, or any other way).

Various permutations of the power calculation were performed. Using data from the 2008 National Stroke Audit, the intra-class correlation coefficient was estimated at 0.149 for Bundle 1 and 0.217 for Bundle 2. Based on preliminary data, where at one hospital, compliance with Bundle 2 had improved to 80%, we assumed a 0.05 significance level, 90% power and a minimum detectable difference of 25% for Bundle 1 and 35% for Bundle 2 in relative increased uptake between the control and intervention. We estimated that for improvement on Bundle 1 to reach significance, up to 24 hospitals, with 12 on either arm, would be required (based on the control group improving to 40% compliance and the intervention group improving to 65% compliance). For Bundle 2 to reach significance, up to 19 hospitals (10 on either arm) would be required (based on the control group improving to 45% compliance and the intervention group improving to 70% compliance).

Outcomes and follow-up

Participating hospitals provided data to the QIC separately and independently from the National Audit (Table 2), although both were measuring the same processes. Random samples of 20 patients per month per hospital were used to generate data for both the intervention period (July 2008 to December 2009) and baseline pre-intervention period (July 2008 to December 2008).

Table 2 Baseline information for individual data (patient level)
Full size table

The primary measures for the trial were compliance with Bundle 1 and Bundle 2. The elements measured within those bundles were identical to the indicators used for the National Audit [8]. Bundle compliance was calculated using an ‘all or none’ measurement, where every process element of the bundle had to be undertaken for overall bundle compliance to be achieved [15]. Thus, omission of any one or more elements would result in a patient’s care being deemed ‘bundle non-compliant’ as illustrated in Figure 1.

Figure 1
figure 1

Bundle compliance. In this example only patient 3 would be classed as receiving care that was bundle compliant.

Full size image

Data collection

Once the QIC began in January 2009, intervention teams were asked to submit, every month, a complete registry of discharged patients coded for stroke from the previous month (based on ICD 10 codes 61, 63, and 64). Each month, the Stroke 90:10 faculty used this registry to produce the random sample of 20 patients for each hospital to audit. For each of the 20 patients included in the sample, data were collected on 45 questions (excluding basic demographic information) on compliance with elements of Bundle 1 and Bundle 2, the three stages of acute stroke care: stroke onset and hospital stay; comorbidities and risk factors; and standards by discharge. Data were entered into a web-based database developed in partnership with the Royal College of Physicians.

The QIC faculty collected the data for the baseline period for hospital teams in the intervention group, and throughout the entire study period for those in the control group. After the baseline period the intervention group were expected to collect and submit data themselves, though they sometimes required extensive faculty support to do so.

In both the control and intervention groups, hospitals were asked to audit all patients in months where there were fewer than 20 admissions. Where patient records were incomplete, could not be retrieved, or were miscoded, patients were excluded and replaced by the next available patient. In total, 1,328 (17%) records (control and intervention) were excluded (see Figure 2).

Figure 2
figure 2

Flow diagram of participation.

Full size image

Statistical analysis

We used a difference-in-difference approach to compare the differences between the intervention and control groups on bundle compliance. This approach measures the difference in bundle compliance over time (before and after the intervention) for the intervention group compared with the difference over the same period for the control group. The analysis was carried out using STATA (version 11, StataCorp LP, College Station, TX, USA).

Using a pre-post-test methodology, we aimed to determine whether there was a difference in relative average bundle compliance in the last three months of the baseline period (October 2008 to December 2008) compared with the last three months of the collaborative (October 2009 to December 2009). Specifically, we sought to determine a difference in difference of means. We fitted a logistic regression model with a random effects term to take account of clustering at the hospital level. Bundle compliance was the primary measure. Explanatory variables included group (control or intervention), time period (October 2008 to December 2008 and October 2009 to December 2009) and an interaction term between them. The interaction term provides the difference in differences that estimates the impact of the collaborative.

Results

Of the 25 eligible trusts in the Northwest of England, 24 (covering 30 hospitals) agreed to participate and were randomized (Additional file 1). Of these, data were available from 18 trusts for analysis (Figure 2), yielding 3533 patients in the intervention arm and 3,059 patients in the control arm. Comparisons of patient demographic and clinical data between the control and intervention groups (Table 2) suggested no significant differences at baseline, except that the intervention group patients showed higher co-morbidity from myocardial infarction or angina (16.4% versus 13.3%; p = 0.019). The collaborative program was run as designed. However, hospital sites did not consistently audit 20, or all, patients each month. Figure 2 shows that a small number of hospitals were excluded for having a reporting rate under 50%; this was pre-specified in the protocol.

Primary measures

Bundle 1 (early hours)

Average Bundle 1 compliance in the control group at baseline (October 2008 to December 2008) was 24.3%, rising to 37.5% by study end (Figure 3). In the intervention group, compliance was 19.6% at baseline, rising to 42.3% by study end. Using a difference-in-difference approach, this represents a relative increase in the odds of compliance of 1.56 (95% CI 1.06, 2.31 p = 0.025) in the intervention group compared with the control group (Table 3). These odds are equivalent to a relative improvement by the end of the study of 10.9% (95% CI 1.3%, 20.6%) in the intervention group relative to the control group.

Figure 3
figure 3

Bundle compliance over time in the control and intervention group. In bundle 1, for the control group, compliance went from 24.3% to 37.5% (13.2% change) and in the intervention group, from 19.6% to 42.3% (22.7% change). On crude comparison, this is a difference of 9.5%, but with all the various adjustments for clustering, this results in a relative benefit of 10.9% in the intervention relative to the control group.

Full size image
Table 3 The proportion of patients who received the bundle in the three months prior to the start of the collaborative intervention (October 2008 to December 2008) were compared with the proportion who received the bundle in the three months at end of the intervention (October 2009 to December 2009)
Full size table

Secondary analysis found, within the bundle processes, that the largest relative difference was seen in administering aspirin (or an alternative antiplatelet) within 24 hours of admission. The odds of aspirin (or suitable alternative) administration to patients where it was clinically indicated in the intervention relative to the control group increased by 1.52 (95% CI 1.05, 2.2 p = 0.027).

Bundle 2 (rehabilitation bundle)

Baseline bundle compliance with Bundle 2 in the control group was 21.9%, rising to 33.2% by study end. In the intervention group, baseline compliance was 27.3% and rose to 46.2% by study end (Figure 3). Using a difference-in-difference approach, this represents an increase in the odds of compliance of 1.61 (95% CI 1.07, 2.42 p = 0.023) in the intervention group compared with the control group. These odds are equivalent to a relative improvement by the end of the study of 11.2% (95% CI 1.4%, 21.5%) in the intervention group relative to the control group (Table 3).

Secondary analysis found that for the individual processes included in the bundle, both Mood Assessed during Hospital Admission (odds 1.74, 95% CI 1.17, 2.58; p = 0.006) and Rehabilitation Goals set by multidisciplinary teams MDT (odds 2.43, 95% CI 1.7, 3.47; p < 0.001) increased in the intervention group more than in the control group.

Discussion

Our study is among a small number of studies evaluating the effectiveness of a QIC that have used a controlled design. Like others [9, 16], we have found evidence of improvement in both the intervention and control groups pointing to an underlying secular trend of incremental improvement over time. Some of the reasons for this are likely to lie with the unprecedented national and regional attention on stroke coinciding with the period of study. During this time, managed clinical networks for stroke were being developed, a National Audit Office report was published and the Department of Health Stroke Improvement program was launched [17]. Delivery of thrombolysis was a national target, and the basic processes required for success in achieving this were closely linked to the processes of care packaged into our Early Hours bundle. Given the intensity of focus, it is unsurprising that both intervention and control groups demonstrated improved performance. The trends we observed are significant and signal the collective impact of direction setting (through national policy), implementation of national standards, regulation, clinical leadership, research implementation and other activities in creating a ‘rising tide’ of more reliable care for stroke patients.

Participation in a QIC does appear to offer a modest overall effect in the amplitude of change over and above secular trend. At first glance it may seem disappointing that the overall benefits are not more significant. However, the bundled nature of our primary outcome measures may have obscured some important and significant effects. These include the disproportionate impact of Stroke 90:10 on some process indicators of quality of care and the resistant nature of others. Looking at the bundles of care as individual items, it is possible to see that four care processes (administration of aspirin/antiplatelet, weighing patients on time, mood assessment and setting MDT rehabilitation goals) improved significantly in the group of hospitals that participated in the collaborative. Weighing patients and setting MDT goals conferred a 17% and 21% absolute improvement respectively. One possibility, therefore, is that the effects of QICs may be specific rather than generalized: some processes of care may be more susceptible to the improvement methods used in a collaborative program than others. This is an important new hypothesis that requires further testing. Furthermore, it appears that these processes could have some distinctive characteristics: they are (relatively) simple changes under the direct control of the hospital stroke team on the stroke unit (defined as a physical unit and not a collection of staff and services) and they are geographically bounded. This possible explanation is supported somewhat by an Australian study in acute stroke units, which showed that focused improvements in direct patient care, delivered on a ward, can be made using a collaborative method [18].

Processes of care that are less tractable may require more attention to engineering systems and processes. For example, delivering a CT scan within hours of admission requires coordination of multiple services and provider groups. Achieving the necessary changes in patterns of working, culture, and attitudes requires an approach different from that used for less socially complex problems. Our study may thus provide some evidence to suggest that the effects of QICs, rather than being uniform, may be mediated by the nature of the processes being targeted and the amount of control participants have over them. Collaboratives may be more suitable for some kinds of improvement interventions than others, and future research should investigate this possibility further. The kinds of implementation interventions needed to support more complex organizational change also require further study, now that promising examples are beginning to emerge [19–24].

Limitations

Though our study design represents improvement on the weaker designs typically used in evaluations of collaboratives, it does have some limitations. First, because the nine processes used in the study were those used by the National Audit, we may not have captured other processes of care that are clinically more important. Variation in data collection approaches across hospitals, although small, may have introduced bias to the results. Second, six hospitals did not provide data, and there was some variation in completeness rates between hospitals. This may introduce bias because it is unlikely the hospitals that failed to provide data, or did not produce complete data, were similar to those that did. Third, generalizability beyond an English context is limited. The socio-technical nature of this intervention, and the context within which it was set, suggests that it is not clear whether the same effects would be found in other populations or settings.

Fourth, our study was not designed to explain whether the increased process adherence resulted in improved outcomes or had any unintended consequences, for example longer lengths of stay or increased re-admissions to hospital. Future studies should investigate the possibility that there may be some causal mechanism that links improved performance on the nine processes to beneficial or unwanted outcomes. Finally, beyond evaluations of collaboratives, experimental evaluations of many complex socio-technical interventions demonstrate modest effect sizes, which might suggest the limitations of positivist evaluations in this context and signal a requirement for more sophisticated evaluation approaches. Some of these issues are the subject of an accompanying paper [25].

Conclusions

Our study suggests that the answer to whether a Breakthrough Series QIC can deliver the extra boost needed to induce improvement beyond secular trend is not straightforward. It does appear to support improvement in more consistent delivery of some processes of care grouped into bundles, but additional, or other kinds of, support may be needed for more complex organizational challenges. Our study reinforces the need, when researching health service improvements, for controlled studies using difference-in-difference analyses to avoid mistaking secular trends for treatment effects. Delivering consistently high quality of stroke care remains a key challenge.

Ethical approval

This study was approved by Tameside & Glossop Local Research Ethics Committee (Ref: 08/H1013/55).