Background

A growing body of research demonstrates the effectiveness of strategies to improve quality and enhance patient safety (QI strategies) [1]. Yet, at the same time the contextual factors affecting the implementation and effectiveness of these strategies are not well understood [2]. Grimshaw et al. provided the first comprehensive review on the effect of interventions to change provider behaviour [3]. This overview of systematic reviews provided invaluable insight into the effectiveness of quality improvement strategies. It raised concerns about the strength of the evidence base and noted that the majority of interventions are effective under some circumstances, although the authors did not systematically explore what such circumstances might be. Scott conducted a similar, albeit less detailed review and concluded that few studies so far have investigated contextual and implementation factors in detail [4]. More recently, Conry et al. conducted yet another review of interventions to improve the quality of care in hospitals and concluded that the lack of theoretically sound research methods that elucidated why interventions work (or do not work) might be a key reason for the slow uptake of the research evidence in health care settings [5].

It is now accepted that the mixed effect and success rates of QI strategies are in part due to the different contexts in which the interventions are planned and implemented [6–8]. An intervention that works in one setting does not necessarily work in another. ‘Context’ for quality improvement has been defined to include those factors that potentially mediate the effect of the intervention, such as leadership, personal skills, organizational resources or data availability [8]. More recently, specific definitions and categorizations of context have been proposed [9, 10]. Often it is neither feasible nor appropriate to adjust for these factors in an analysis of effectiveness. This is, firstly, because it is unlikely that data on all such factors are available and, secondly, because adjustment for context might mask rather than highlight the importance of such factors. As the potential generalisability of findings on the effectiveness of QI strategies (which often include organizational interventions) is much more limited than the generalizability of clinical trials (for example on the pharmacokinetic response to a drug in a defined group of patients), the question ‘does the QI strategy work’ is only of initial interest. The broader question ‘why, when, where, and for whom it works most effectively’ is of much greater concern and practical importance [11]. Having a thorough understanding of the underlying mechanisms that make an intervention work, will allow for successful application of the intervention in other settings and help improving its effectiveness.

The objectives of this paper are therefore to (i) describe the reporting of contextual factors in the literature on the effectiveness of QI strategies, (ii) assess the relationship between these contextual factors and the effectiveness of QI strategies, and (iii) analyze the importance of contextual factors.

Methods

Search strategy

We conducted a review of systematic reviews of the literature on the effectiveness of QI strategies [12]. The following electronic databases were searched: PubMed, Cochrane Database of Systematic Reviews, Embase and CINAHL. The search was limited to literature reviews published in English language between January 2000 and November 2012. The search focused on QI strategies included in the Cochrane Effective Practice and Organisation of Care (EPOC) Group taxonomy, which include various forms of continuing medical education (CME), quality assurance projects, financial, organizational, or regulatory interventions that can affect the ability of health care professionals to deliver services more effectively and efficiently [13]. A Boolean search strategy for PubMed was developed (see Additional file 1) covering all quality management topics, including a combination of text words and Medical Subject Headings (MeSH) terms, searched in titles and abstracts of studies. The Boolean search strategy was adapted for the other databases.

Methods of screening and selection criteria

The review strategy was guided by a manual for performing systematic literature reviews on a health services research topic [14]. The results from the databases were checked for duplicates using Reference Manager. This was followed by a three-step screening procedure. Studies were excluded if they did not focus on the effectiveness, performance or impact of quality management strategies in hospital settings. An initial screening of studies was based on titles and abstract, performed by two reviewers (OG, DK) independently. In the second screening, the full texts of the reviews were assessed for inclusion by OG and DK independently. In the third step, the final list of included studies was evaluated for their completeness by a panel of quality management experts from European countries, comprised of mostly senior researchers and medical professionals who participated in a European Commission (EC) funded research project (Deeping our understanding of quality improvement in Europe (DUQuE), see www.duque.eu and acknowledgment section). This evaluation led to five additions to the publication list. Only systematic literature reviews with a focus on the effectiveness, performance or impact of quality management strategies in hospital settings were included. This covered both qualitative and qualitative reviews and meta-analyses. Figure 1 shows the complete study selection process, including the number of disagreements among reviewers which were resolved by a third independent reviewer (RS).

Fig. 1
figure 1

Study selection process. Legend: Figure 1 indicates the study selection process of the systematic literature review

Full size image

Data extraction

Data on the effectiveness of quality management strategies and the influence of both internal and external contextual factors were extracted using a structured data entry form. Panel experts from the DUQuE team were each independently assigned to extract the data from a limited set of studies which fit with their expertise. The following information was extracted from the studies that met our inclusion criteria: type of QI intervention, number of included studies and participants, objective, description of intervention, description of primary and secondary outcome, effect of primary and secondary outcome, and contextual factors.

Data extraction on contextual factors was categorized according to an assessment tool based on the Model for Understanding Success in Quality (MUSIQ) (Textbox 1) [9]. We coded context-based factors against the MUSIQ tool and added up the number of studies which displayed each one and noted if the study had produced an effective outcome.

The MUSIQ tool was developed by Kaplan et al. to facilitate research on the contextual factors affecting QI strategies, and has been shown to be reliable and valid [9, 15]. It identifies 25 contextual factors for quality improvement, covering six overarching themes that they labelled as: external environment, organization, quality improvement capacity, the clinical microsystem, the quality improvement team, and a number of miscellaneous issues. Detailed descriptions of the six themes and 25 contextual factors are included in Additional file 2: Table S1.

Textbox 1: Domains of the MUSIQ Tool [9]

●

External environment (external motivators, project sponsorship),

●

Organization (QI leadership, senior leader project sponsor, culture supportive of QI, maturity of organizational QI, physician payment structure),

●

Quality improvement support and capacity (data infrastructure, resource availability, workforce focus on QI),

●

Microsystem (QI leadership, culture supportive of QI, capability for improvement, motivation to change),

●

Quality improvement team (team diversity, physician involvement, subject matter expert, team tenure, prior experience with QI, team leadership, team decision-making process, team norms and team QI skill) and

●

Miscellaneous (trigger events, task strategic importance to the organization).

All data entry forms were double-checked by two independent reviewers (OG, DK) and complemented or altered when both reviewers agreed. In case of disagreement, a third independent reviewer (RS) made the final decision. The quality of the included studies was assessed using a valid and reliable [16, 17] measurement tool to assess the methodological quality of systematic reviews (AMSTAR) [18]. The overall quality of each study was assessed by the proportion of items of the AMSTAR Checklist each study complied with (number of “Yes” answers to 11 questions). Studies are rated as low quality when AMSTAR score is 0–4, moderate quality when AMSTAR score is 5–8, and high quality when AMSTAR score 9–11 [19]. The quality assessment was undertaken by two independent reviewers (OG, DK).

Ethics

This research did not involve human subjects. Ethics approval is covered by the DUQUE project, financed by the EU Commission under grant agreement number 241822 and approved by the Health department of the Government of Catalonia, Spain.

Results

We included 56 systematic reviews in this study (Fig. 1).

The reviews address QI interventions such as health care accreditation (n = 3), local leadership (n = 1), continuing medical education (n = 3), nurturing patient safety cultures (n = 2), promoting organizational culture (n = 3), computerised clinical decision support systems (n = 16), guideline dissemination and implementation (n = 2), interventions to improve patient handovers (n = 5), patient-centred care interventions (n = 3), six sigma and lean continuous quality improvement interventions (n = 3), the use of performance information (n = 6), audit and feedback (n = 2), hospital incident reporting (n = 2), safety checklists (n = 1), educational outreach visits (n = 1), and multi-faceted quality improvement interventions (n = 3). Table 1 gives an overview of all studies included, by type of QI intervention, author, year, AMSTAR score, number of studies included (Table 1). In addition, it specifies the countries included and the number of participants. Reviews included between 2 and 235 individual studies, covering mostly North America, Europe and South-East Asia, with very few studies from South America and the African continent. The median AMSTAR score is 7 (moderate quality), but for studies published from 2010 onwards it increased to 8.5 (moderately high quality) (see also Table 1, footnote for comparison of average AMSTAR code by year).

Table 1 Study characteristics
Full size table

In Additional file 3: Table S2 we summarize the information on contextual factors extracted from the systematic reviews. For each study, we identified the type of contextual factor according to the MUSIQ model and describe how they relate to QI effectiveness (Additional file 3: Table S2). Only 35 of the 56 studies described contextual factors related with the effectiveness of QI interventions. Other studies were exclusively focussed on describing the effectiveness of QI interventions, keeping the contextual factors outside the scope of their study. The most frequently reported contextual factors that were found to be associated with the effectiveness of interventions were the following: external environment (external motivator, n = 4), organization (maturity of organizational QI, n = 5; QI leadership, n = 4), QI support and capacity (resource availability, n = 6; data infrastructure, n = 5), micro-system (capability for improvement, n = 5; culture supportive of QI, n = 3), and QI team (physician involvement, n = 5; team diversity, n = 4; including a subject matter expert, n = 3). Contextual factors identified in the MUSIQ tool but not reported in the reviews were QI team norms, task strategic importance to the organization and trigger events. Contextual factors that are not currently included in the MUSIQ tool, but were reported in various studies (n = 8) included organizational level of programme implementation, patient turnover and bed occupancy, staffing levels, quality of evidence and guidelines, maturity of systems on which CDSS are based, trust in and quality of information sources and educational outreach visitors [20–27]. In the following section we summarize these contextual factors using the six domains of the MUSIQ model: external environment, organization, quality improvement capacity, the clinical micro-system, the quality improvement team, and miscellaneous issues.

External environment

In terms of direct effects, financial incentives or administrative support were related with the effectiveness of QI strategies [5, 28]. Resource requirements, too, were a key factor, in particular for large up-front organizational support and capital investment required to introduce a Computerised Physician Order Entry [29]. QI strategies were not equally effective across health care settings: Clinical decision support systems (CDSS) were more effective in institutional settings than community settings due to types of conditions, stricter controls on professional behaviour, and different attitudes of professionals towards externally imposed rules [30]. The external environment also affected the interpretation of QI effectiveness more broadly. Priority areas of external inspections differed between high and low income countries, making the results of studies on the effectiveness of accreditation programmes difficult to compare and transfer [31].

Organization

There was a substantial amount of literature referring to the effect of supportive organizational cultures [21, 32, 33]. The creation of a patient safety culture (including having clear policies and actively support training) positively impacted on the effectiveness of infection control policies [21], on the implementation of Six Sigma and Lean approaches to QI) [32]. The visible support of managerial staff (both at ward/unit and above ward/unit level) also impacted positively on the effectiveness of accreditation of health care services [34], on clinical decision support systems [35], and on hospital incident reporting [26].

Another example for organizational context factors was the embedding of feedback systems in organizational QI: Feedback on a physician’s clinical performance was more likely to be effective when provided by an authoritative, credible source, systematically over multiple years [36], and a high frequency of performing audit and feedback increased its effectiveness in modifying health professionals’ behaviour [37]. Having a closed safety-feedback cycle (e.g. effective dissemination channels and the capacity for rapid action) at all levels of the organization positively affected the effectiveness of hospital incident reporting [26].

The absence of an effective multidisciplinary infection control team perceived as exercising positive leadership at ward or unit level was a risk for the effectiveness of infection control strategies on health care associated infections [21].

QI support and capacity

The availability and functionality of information technology (IT) systems facilitated data collection and improved the effectiveness of QI interventions [38], notably, interventions targeting handovers [24], accreditation programs in rural health care services [34], and Six Sigma [39]. Sufficient resources, too, were paramount for the implementation of QI strategies [38, 40, 41]. Insufficient administrative support impacted on the effectiveness of interventions promoting safety cultures [41] or on strategies aimed at implementing quality indicators [40]. An excessive workload (not matched to available staffing) and insufficient staff training were a risk for the effectiveness of Six Sigma [39] and infection control strategies on health care associated infections [21]. High staff turnover and an excessive use of external staff members limited the effectiveness of infection control strategies [21].

Microsystem

Clinical micro-systems have previously been described as the key settings in which QI interventions are implemented [42–46]. Low staff morale and scepticism of health care professionals towards the positive impact of QI interventions (e.g. accreditation programs or CDSS) on the quality of health care services were serious barriers to the successful implementation of QI interventions [21, 23, 33]. Alignment of physicians’ views on the content and implementation of interventions aimed at improving handovers was beneficial to its effectiveness [24, 47]. The training or education in the proper use of QI strategies (e.g. safety checklists, accreditation standards) was beneficial to the effectiveness of the QI strategies [34, 48]. Integrating QI strategies in the working practices of health professionals promoted the effectiveness of the QI strategies [23] and the ‘motivation to change’ [22, 40].

QI team

Team composition is a major determinant for QI effectiveness. Involving physicians in the development and implementation of QI interventions (such as CDSS, accreditation programs, audit and feedback) has been shown to be an important success factor for their effectiveness [22, 23, 34, 37, 49]. Involving multidisciplinary QI teams (including nurses and physicians, and sometimes also pharmacists) in the development and implementation of QI strategies increased the effectiveness of the intervention [22, 24, 50, 51]. ‘Subject matter experts’, where more than one team member has detailed knowledge about the outcome, process, or system being changed were shown to be beneficial for the a range of QI strategies (CDSS, CME, Six Sigma and Lean). Awareness, attitude, knowledge of and understanding performance data (generic ‘QI skills’) [52] were all essential facilitators for the implementation of QI interventions.

Miscellaneous

A number of additional contextual factors were identified from the literature, not all of which are addressed in the MUSIQ tool. These include, in particular, structural factors of service organization, including turnover of staff or bed occupancy [21], workload and time constraints [24], but also the flexibility to update the QI intervention (guidelines or computerised decision support systems) [22, 23]. Detailed information on the contextual factors identified in the literature is presented in Additional file 3.

Discussion

A number of previous studies have reviewed the effectiveness of quality improvement strategies, but this is to our best knowledge the first study to systematically assess a broad range of associated context factors and their relationship with the effectiveness of multiple quality improvement strategies in health care. This study has shown that, overall, context factors were poorly reported in the current literature. This is a very important finding since those studies that did report it demonstrated substantial differences in QI effectiveness, depending on the presence or absence of contextual factors. Given the heterogeneity of the literature few systematic reviews included in the analysis were able to pursue a quantitative synthesis and stratify this analysis on the context factors identified.

Context factors most frequently reported related to the three MUSIQ domains microsystem, QI support and capacity or QI team. This aligns with recent analysis of the concepts underlying the MUSIQ tool, which identified that these domains exhibit significant effects on QI performance outcomes [15]. Of significance for improvement efforts these domains are also most amenable to local adjustment. We also identified context factors that are not currently included in the MUSIQ tool, such as those relating to structural characteristics or the heterogeneity observed in the relationship between context and multiple outcome measures [20–27]. Since the publication of the MUSIQ tool several other key publications on the role of context have emerged [2, 10, 15, 53]. Our findings might help to inform these conceptual developments. Moreover, our data may lead to reflections on the broad conceptual nature of existing context models (which emphasize multi-level structures, the role of the external environment, the organization at large and clinical practice), while it seems that all three key domains emerging from this review broadly relate to the clinical microsystem.

Implications of the study

This study has important implications for future research on the relationship between QI context and effectiveness. For some QI interventions the evidence is substantial, and is supported by clear recommendation on how context factors mediate effectiveness (e.g. the Cochrane review on audit and feedback [37] gives clear indication on the factors and the magnitude to which they increase or attenuate the effect of QI strategies). For other QI interventions, the evidence base is still weak and adaptation and implementation should be pursued with caution, since generalisability is limited when context factors are unknown.

Future studies on the effectiveness of QI interventions need to place more emphasis on studying and reporting outcomes in relation to contextual factors [53]. Pooled averages are misleading and do not reflect the varying contexts in which QI are implemented. Ideally, alongside major evaluations of effectiveness, studies should be conducted looking at context factors and recording such factors using existing tools (e.g. [9, 53]). This would permit future reviews on the effectiveness of QI to stratify on the type of context, sample size permitting. In parallel, qualitative studies (including ethnographic studies) could be conducted to gain a deeper understanding of professional, organizational, cultural and structural context factors [54]. This would be a complex and difficult undertaking. For example, a true investigation of context factors cannot be achieved retrospectively at the stage of writing up a research project, but rather requires engaging multiple perspectives and stakeholders from the start to ensure all relevant aspects of QI implementation (both formal and informal) are considered. Once identified, such factors need to be observed and monitored over time and statistical models could aim to assess the association of context factors with the effectiveness of QI interventions, where appropriate. We have recently illustrated some of the methodological challenges related to these tasks [55, 56] and at current, the multiple relationships and pathways between exposure, outcome, and context variables in research on QI strategies are not yet sufficiently understood. Alternatively, context might be considered as an integral component of the subject area that evolves, changes and interacts with the intervention during the time period of QI project implementation. In this case, in-depth qualitative assessment is needed. Finally, the research output should report on the role of context factors in order to facilitate generalisability and replicability of the QI intervention. This issue, too, has been subject to debate and the Standards for Quality Improvement Reporting Excellence (SQUIRE) guidelines are one approach to address a better understanding of QI context factors [57]. The SQUIRE guidelines demonstrate the difficulties and practical issues when reporting on the factors that potentially impact on a QI intervention. Nevertheless, improving understanding, conceptualization, analysis and reporting of context factors in QI is important to advance the field of research. It will help in understanding the mixed results of some QI interventions and help replicate successful projects or, equally important, inform implementers were replication is unlikely to be successful due to different contexts. Our findings suggest that some of the most relevant context factors are those close to the clinical microsystem in which the QI intervention is delivered. This provides cues for action for improvement practitioners who may include an assessment of such factors, and dedicated change processes, in their local plans for quality improvement.

Limitations

This study has a number of limitations. The main limitation is that the field of research addressing the role of contextual factors in QI is still developing and currently there is no clear consensus on how to define or assess context factors, which has implications for the reviews included in our umbrella review. While we were able to apply the MUSIQ tool to categorize context factors and facilitate data extraction from the literature, the lack of clear search terms may mean that we might have missed reviews for our study. Moreover, it is unclear how contextual factors were assessed in the original studies included in our list of systematic reviews. This may potentially induce publication bias as positive associations are more likely to have been reported. A meta-regression analysis of the effect of context factors, adjusting for publication bias, would be desirable, but given the heterogeneous reporting of the findings in the literature this was not possible at present. We intentionally searched only for the literature that primarily addressed the effectiveness of QI strategies (and not for literature that primarily addressed context factors), as our key interest was the effect of QI and how it is affected by context. In comparison, much of the context literature does not report specifically on QI effectiveness and includes also largely qualitative and mixed methods research. An assessment of this literature would have been beyond the scope of this paper. While acknowledging these limitations, the findings of this review are nevertheless important for an advancement of the understanding of how context factors shape the effectiveness of quality improvement interventions.

Conclusions

Contextual factors may influence the effectiveness of quality improvement interventions, in particular at the level of the clinical microsystem. Future research on the implementation and effectiveness of QI interventions should emphasize formative evaluation to elicit information on context factors and report on them in a more systematic way in order to better appreciate their relative importance.