Background

Researching complex interventions in healthcare and nursing has gained increased awareness following the publication of the UK Medical Research Council’s (MRC) framework for development and evaluation of complex interventions in 2000 [1] and its update in 2008 [2, 3]. The MRC framework defined complex interventions as interventions comprising several components which may act either independently or inter-dependently [2]. However, there is no distinct boundary in the classification of whether an intervention is complex or not. Characteristics of complex interventions are, for example, the number of different professions or organisational levels targeted by the intervention and/or the degree of flexibility permitted for the intervention [4].

While the first framework [1] implied a linear approach comprising one preclinical and four clinical phases, the updated framework [2, 3] builds on a circular model which gives a better reflection of the flexibility or even non-linearity of the research process. However, both versions cover the same methodological steps of development, piloting and evaluation of a complex intervention, and the long-term implementation after the complex intervention has demonstrated its effectiveness. Also, both frameworks emphasise that the development of a complex intervention should include a theoretical basis and a clear description of the intended change processes. Prior to the evaluation, the intervention should be piloted in the target setting and its feasibility has to be tested. The evaluation study should not only focus on effectiveness and cost-effectiveness but also include the evaluation of the intended change process by investigating the dose delivered, fidelity and reach of the intervention (process evaluation) [5–7]. Thus, the development and evaluation of complex interventions requires several studies using different methods and study designs [1–4].

Comprehensive reporting of all steps of the development and evaluation of a complex intervention is crucial. Sufficient information must be available for the judgement of the intervention’s clinical benefits, for replication, or for adaption of an intervention to different settings or countries [8–10]. However, several analyses revealed shortcomings of the reporting of core aspects of complex intervention research, e.g. regarding the theoretical basis and assumptions guiding the development and piloting of the intervention, and the description of intervention delivery [10–13]. These studies evaluated interventions on defined topics, e.g. care of people after stroke [13] and reduction of physical restraints in geriatric care [12], or specific methodological aspects of research on complex interventions [10, 11]. An evaluation of the reporting quality of a sample of systematically identified complex interventions explicitly adhering to the MRC framework has not been performed so far.

We have recently published a list of specific criteria for reporting complex interventions in order to offer a structured guidance for researchers and authors covering the first three phases of the MRC framework [14]. In contrast to other reporting guidelines (e.g. CONSORT), CReDECI does not comprise design-specific items. The development and evaluation of complex interventions requires the use of different methodological approaches and the criteria list includes only items covering these specific methodological aspects [1–4, 14]. Therefore, CReDECI should be used alongside established study design-specific reporting statements.

Employing the CReDECI criteria, the aim of the present study was to evaluate the reporting quality of the development and evaluation of complex interventions adhering to the MRC framework. The second aim was to test the applicability of the criteria list.

Methods

Literature search and study selection

All trials which adhered to the MRC framework were eligible for inclusion. We use the term ‘trial’ for the entire research process covering development, piloting and evaluation of a complex intervention and all related publications or reports. Adherence to the MRC framework was defined as citing either the first version [1] or the updated framework [3] as methodological guide for the development and evaluation of a complex intervention. According to the aim of our study to test the applicability of all CReDECI criteria, the included trials should have undergone a controlled evaluation study.

The following inclusion criteria were defined: The intervention was (I) labelled as ‘complex intervention’, (II) developed and evaluated adhering to the MRC framework, and (III) a controlled evaluation study was published. All citations retrieved by database searching and ‘snowballing’ techniques were screened independently by two reviewers (RM and GB) and checked for inclusion. We conducted a database search and used snowballing techniques to identify relevant publications since we expected several publications reporting on different phases of the same trial [15].

A systematic search in Medline (via PubMed) was performed in April 2012. Medline was searched because this database is the primary resource for clinicians and researchers and most journals publishing evaluation studies of health care interventions are covered. The search was limited to German and English publications of the last ten years. We explored several search strategies to select the adequate search terms. Finally, we used a combination of terms related to complex interventions and to the MRC framework (the complete search strategy is described in Additional file 1). Since we used additional snowballing technique to identify further relevant publications, we believe the search strategy employed to be sufficiently specific. In a second step, we conducted forward citation tracking (via Scopus and Google Scholar). For all citations identified by the database search which reported on the development and/or piloting of a complex intervention we checked whether a publication of an evaluation study was available. In a third step, we identified the associated publications of the included trials, e.g. publications on the interventions’ development.

CReDECI

The CReDECI list [14] was developed based on a systematic literature review of methodological publications on complex intervention research and the updated MRC framework and reviewed by experts in the field. The criteria list is divided into three sections: development (n = 6 items), feasibility and piloting (n = 2 items), and introduction of the intervention and evaluation (n = 8 items). In contrast to most of the available reporting guidelines, the CReDECI list does not comprise items targeting a specific study-design, since the development and evaluation of complex interventions requires different study designs conducted subsequently or in parallel [2]. Instead, CReDECI focuses on criteria covering the core elements that are specific for the research of complex interventions. An overview of the criteria and their explanation is presented in Table 1.

Table 1 Overview of CReDECI
Full size table

Data extraction

Two independent reviewers (RM, GB) applied CReDECI to all publications related to the respective included trials. For each item it was recorded whether sufficient information was provided in any publication. In order to distinguish between sufficient and insufficient information, the raters assessed whether the information provided was judged as comprehensive. The information to the individual items may have been included in several publications. However, if a minimum of comprehensive information was offered, the items were rated as fulfilled. We did not assess the degree of comprehensibility.

The results of the independent ratings were compared and agreement between both raters was checked. Thereafter, differences in the ratings were discussed in order to reach consensus. If results on planned parts of the trial were missing (e.g. economic evaluation or process evaluation), the corresponding author was contacted. The number of trials fulfilling the criteria was calculated. The absolute number of different ratings per trial was determined with regard to inter-rater reliability as well as to the time spent on completing the list (i.e. mean time of both raters).

Results

A total of 274 publications were identified via database search and forward citation tracking. Eight trials met the inclusion criteria (Figure 1). Main reasons for exclusion were that the interventions were not labelled as complex or the studies did not adhere to the MRC framework. A total of 26 publications were identified as reporting on the eight included trials. In two cases, authors were contacted for missing information. The mean number of publications per trial was 3.25 (range 1 to 6). For one trial, only one publication was available [16]. Four trials were conducted in UK [16–19], two in Ireland [20, 21] and one in Germany [22] and the Netherlands [23], respectively. Most of the complex interventions offered structured approaches for improving the quality of care for patients with chronic or complex conditions. Target groups were people suffering from a stroke [16, 19], cancer [18, 22], coronary heart disease [20], or diabetes mellitus Type 2 [21], patients at risk from prescription and medication management errors in general practice [17], and frail elderly people [23]. The interventions offered improved care concepts or pathways [17, 19, 20, 22] or alternative or modified forms of information or support [16, 18, 21] e.g. peer support groups for patients and/or their relatives [21]. Table 2 presents an overview of the included trials and the underlying interventions. The quality of reporting on development and piloting of the intervention was judged as good (Table 3). Seven out of eight trials reported on all six items referring to the intervention’s development [17–23] and one trial reported on three items [16]. Most trials presented sufficient information on the feasibility and pilot phase. Concerning the introduction and evaluation phase, the quality of reporting varied. Only one trial reported on all eight items [21] and one trial on seven items [17]. Five trials reported on one to five items [18–20, 22, 23] and one trial offered no information on any item [16].

Figure 1
figure 1

Study selection flow diagram.

Full size image
Table 2 – Overview of included complex interventions
Full size table
Table 3 Results of the rating of the CReDECI items
Full size table

The number of items with different ratings per trial ranged from 0–5 with a mean of 2. The two reviewers disagreed on 16 out of the total of 128 ratings (12.5%). The time needed to rate a trial ranged from 30 to 90 minutes, depending on the number of publications per trial.

Discussion

We identified a small number of trials whose authors explicitly stated having adhered to the MRC framework for developing and evaluating complex interventions [2]. The quality of reporting in these trials was judged as good for the intervention development and piloting phases. Thus, our findings are contrary to previous analyses of the quality of reporting on complex interventions [12, 13, 41]. However, the trials included in our sample are more likely to fulfil the CReDECI criteria, since all trials confirmed adherence to the MRC framework. For the evaluation phase, quality of reporting varied. While two trials reported on nearly all criteria, six trials offered information only on half of the items or even less. These results confirm the findings from previous studies [9, 11, 12, 42]. Only half of the trials included an evaluation of the implementation process (CReDECI criterion 12). However, this information is required in order to get a deeper understanding of the effects of the evaluation and to describe barriers and facilitators influencing the intervention’s implementation. A process evaluation should be an integral part of the evaluation of complex interventions [2, 5, 6]. Only half of the trials offered information on the characteristics of the care delivered in the control group, which was often only described as usual or standard care. The lack of information hampers the replication of studies and the adaption of the intervention to different settings or countries [9, 10].

The integration of the local context and detailed information on the process evaluation has recently been described as frequently underreported [43, 44]. In our sample only a small number of trials included a process evaluation but most trials offered information on the integration of the context in the intervention’s development. However, we did not assess the completeness of the information provided.

The CReDECI list has proven its applicability and practicability. A high rate of agreement was reached by both reviewers. The time needed for assessment varied, depending on the number of publications per trial. Since both reviewers were familiar with complex interventions research and the MRC framework, application of the CReDECI list by untrained users might be more time-consuming.

The currently available reporting guidelines specifically address a defined study design, offering recommendations to improve the reporting of design-specific methodological aspects [45]. The CReDECI list employs a different approach since it comprises items covering three phases of complex interventions’ research: development, feasibility/piloting, and introduction of the intervention and evaluation. However, we recommend the additional use of design-specific reporting guidelines alongside the CReDECI list. Currently, a CONSORT extension for social and psychological interventions is being developed [46]. This extension might be a valuable supplementation to the CReDECI list.

Limitations of study

Although Medline includes a great number of journals on healthcare research, the search in only one database might be judged as limitation. Further publications not indexed in Medline were identified through forward and backward citation tracking. Thus, it is most likely that we have identified the majority of publications on the development and evaluation of complex intervention explicitly referring to the MRC framework.

In this study, we assessed whether the criteria were fulfilled. Therefore, our analysis offers an overview about relevant aspects of complex interventions’ development and evaluation which were considered in the trials included. However, this analysis offers no judgement on the quality of the information provided.

We included only a small number of trials, since our inclusion criteria were rather strict. Therefore, readers must be aware that the results of our analysis do not represent the reporting quality of publications on complex interventions in general.

Conclusions

In this study, adhering to the MRC framework for the development and evaluation of complex interventions seems to have a positive impact on the quality of reporting of the complex interventions’ development and feasibility/piloting. These results are in contrast to former analysis [10–13]. Reporting on the evaluation phase could still be improved.

The CReDECI list seems to be a practical instrument for checking the quality of reporting in publications on complex interventions. It could be used alongside established design-specific reporting guidelines such as the CONSORT statement. To further validate the CReDECI list, a formal consensus process with researchers and stakeholders is scheduled.