Diabetologia

, Volume 50, Issue 7, pp 1375–1383

Meta-analysis does not allow appraisal of complex interventions in diabetes and hypertension self-management: a methodological review

  • M. Lenz
  • A. Steckelberg
  • B. Richter
  • I. Mühlhauser
Review

DOI: 10.1007/s00125-007-0679-z

Cite this article as:
Lenz, M., Steckelberg, A., Richter, B. et al. Diabetologia (2007) 50: 1375. doi:10.1007/s00125-007-0679-z

Abstract

Common methodologies used in systematic reviews do not allow adequate appraisal of complex interventions. The aim of the present study was to describe and critically appraise current methods of systematic reviews on complex interventions, using diabetes and hypertension patient education as examples. PubMed, the Cumulative Index to Nursing and Allied Health (CINAHL), the Cochrane Library and Health Technology Assessment databases were searched. Systematic reviews focusing on diabetes or hypertension patient education were included. Authors were contacted. Two investigators independently evaluated the reviews. The available evidence of three patient education programmes of diabetes and hypertension self-management implemented in Germany was used as a reference. We included 14 reviews; 12 reviews mentioned that the included education programmes were multidimensional. Reviews on comparable topics identified different publications of the same programme. Identical programmes were classified differently within and between reviews. Education programmes were dissected to analyse effects of single components. Different components of identical programmes were used. Interdependencies between components were not explored. Six reviews performed meta-analysis across programmes with heterogeneous educational or organisational approaches. The complexity of efficacy measures was disregarded: e.g. HbA1c was used as an isolated outcome variable without considering treatment goals, effects on hypoglycaemia, body weight or quality of life. Our results indicate that methods of current systematic reviews are not fully equipped to appraise patient education and self-management programmes. Since these are complex and heterogeneous interventions, consideration of aggregated evidence is necessary.

Keywords

Diabetes Education Hypertension Methods Self-management Systematic reviews 

Abbreviations

RCT

randomised controlled trial

UKMRC

UK Medical Research Council

Introduction

In diabetes care there are single interventions (e.g. a single drug) and complex interventions (e.g. treatment regimens or diabetes self-management). The latter comprise a number of separate elements (active components), all of which seem essential to their proper functioning [1, 2]. A self-management programme for type 1 diabetes may illustrate the complexity of such interventions [3]. Decisive components are the insulin regimen used and the quality of the teaching process to empower patients to carry it out effectively and safely. Empowerment of patients to set individual treatment goals and to balance favourable blood glucose targets and an acceptable risk of hypoglycaemia by self-adaptation of insulin dosages to adjust to lifestyle may be more effective than defining normoglycaemia as the primary treatment goal and asking patients to adapt their lifestyle to match prescribed doses of insulin [4]. Liberalisation of the diet may be important to motivate patients to carry out an intensified insulin therapy regimen in the long term [5]. Though indispensable, knowledge by itself may not improve outcomes. The information and how it is transmitted are decisive [3]. Blood glucose self-monitoring may be at best useless unless patients have learned to interpret results and to react by adjusting insulin dosages [3].

A high-quality randomised controlled trial (RCT) is considered the most valid method of evaluating a medical intervention, and a systematic review of high-quality RCTs the most powerful evidence available [6]. A systematic review is a summary of the medical literature that uses explicit methods to systematically search, critically appraise and synthesise the literature on a specific issue [6]. They may, but need not, include a meta-analysis as a statistical method for combining the results of individual studies [6].

The UK Medical Research Council (UKMRC) has proposed a framework that allows a systematic and transparent evaluation of complex interventions. Five sequential phases of a ‘continuum of increasing evidence of complex interventions’ (hereafter referred to as ‘increasing evidence’) have been defined, which require both qualitative and quantitative evidence [1, 2]

It has been suggested that, for many reasons, current methods used in systematic reviews do not allow adequate appraisal of complex interventions such as diabetes or hypertension self-management programmes [7]. For example, systematic reviews do not consider the theory behind the complex intervention (e.g. behavioural models) and do not differentiate between trials designed to determine efficacy and those focusing on implementation.

If used appropriately, meta-analysis is a powerful tool for investigating overall effects. However, if studies are clinically or methodologically heterogeneous, data-pooling may be meaningless and genuine differences in effects may be obscured [8]. Complex interventions are heterogeneous in their goals, methods and target populations [9, 10, 11, 12, 13, 14]. Thus, using meta-analysis to evaluate complex interventions may disregard the complexity of the efficacy measures used in the original studies. For example, HbA1c is used as a single outcome variable without considering individual treatment goals or effects on hypoglycaemia, body weight or quality of life [9, 15, 16]. An increase in HbA1c from 6.5 to 7% might be considered a deterioration, but would clearly be a desirable outcome if it halved the rate of hypoglycaemia.

The aim of the present study was to describe and critically appraise available methodologies of systematic reviews on complex interventions. Patient self-management education programmes of diabetes and hypertension implemented in Germany were used as examples.

Materials and methods

The Cochrane Library, PubMed, Cumulative Index to Nursing and Allied Health (CINAHL) and Health Technology Assessment databases were systematically searched for systematic reviews published between 1997 and March 2006 (see search strategy in Electronic supplementary material [ESM]). Reference lists of retrieved reviews were screened for additional publications. Reviews were included if patient education programmes of diabetes and hypertension self-management were evaluated (see Selection criteria in ESM). Included reviews were analysed independently by two investigators (M. Lenz and A. Steckelberg) using standardised data extraction forms created following Cochrane criteria [17], according to predefined hypotheses (see Fig. 1, and Hypotheses and Data synthesis in ESM). All authors of the included reviews were contacted after data synthesis and a preliminary version of our review was provided. They were asked whether they felt that their review had been considered appropriately and their findings interpreted correctly.
Fig. 1

Flow diagram of study selection. The term ‘systematic review’ includes meta-analyses and health technology assessment reports. Fourteen reviews in 16 publications (superscript a). Some reviews address more than one topic (superscript b)

The ‘increasing evidence’ of three patient self-management education programmes—type 1 diabetes, type 2 diabetes and hypertension—was used as the reference. It was assessed: (1) whether the selection criteria and search strategies used in the included systematic reviews were appropriate for detection of the ‘increasing evidence’ of the three reference programmes (ESM Tables 1, 2, 3); (2) which publications related to our three reference programmes had been identified; (3) whether theoretical background literature (theoretical/preclinical phase) had been considered; (4) which active components had been identified and included (ESM Tables 4, 5, 6); (5) whether classification of interventions matched the type of intervention; and (6) whether all relevant patient outcomes had been included (ESM Tables 7, 8, 9).

Results

A total of 15 reviews (16 publications) were included (Table 1, excluded reviews are listed in ESM Table 10). Various types of interventions were investigated (e.g. individual and group counselling, and structured training courses for patients). A meta-analysis was performed in eight reviews [9, 10, 11, 13, 15, 16, 18, 19, 20]. A total of six reviews exclusively used qualitative data synthesis [21, 22, 23, 24, 25, 26, 27] (ESM Table 11).
Table 1

Characteristics of included reviews

First author

Topic

Number of included studies

Type of data synthesis

Boulware [11]

Patient-centred counselling, patient self-monitoring of BP and structured training courses on hypertension

n = 16

1. Meta-analysis: overall summary effects across pre-specified categoriesb of interventions according to study characteristics:

 Single-intervention analysis (n = 12): the intervention was part of a study comparing more than one intervention, but the remaining interventions of the study not meeting the inclusion criteria were excluded from analysis

 Between-intervention analysis (n = 7): the intervention group was compared with a control group also containing an intervention meeting the eligibility criteria or usual care

2. Meta-analysis: subgroup-analysis according to active components: articles (n = 5) focusing on counselling by categorising articles by publication year; leader of intervention; duration of intervention; age of participants; percentage of white participants; and quality scores of articles

Corabian [24]

Outpatient education for self-management in adults with type 2 diabetes

n = 24a

1. Qualitative assessment of quantitative studies within categories according to study characteristics: meta-analyses; primary studies on effectiveness; comprehensive and systematic reviews; Canadian studies

2. Qualitative assessment of qualitative research to identify factors that potentially influence the impact of the diabetes care and education regimen

Deakin [19]

Group-based, patient-centred training on type 2 diabetes

n = 11 (in 14 publications)

1. Meta-analysis: overall summary effects across pre-specified categories of interventions according to study characteristicsc: short-term follow-up (4–6 months); medium-term follow-up (6–12 months); long-term follow-up (≥12 months)

2. Descriptive summary according to outcome measuresd

3. Meta-analysis: overall summary effects across pre-specified category: follow-up term (12–14 months)

4. Meta-analysis: overall summary effects across pre-specified category: follow-up term 2 years

5. Meta-analysis: subgroup analysis according to ethnicity across specified categories of interventions: short-term follow-up (4–6 months), long-term follow-up (12–14 months)

6. Meta-analysis: subgroup analysis according to theoretical model; summary effect size calculated only across two studies, excluding those studies causing heterogeneity (only short-term follow-up [4–6 months])

7. Qualitative subgroup analysis according to the theoretical modeld

8. Meta-analysis: subgroup analysis according to the educator (nurse, dietician or a combination of both); summary effect size calculated only across five studies, excluding those studies causing heterogeneity (only long-term follow-up [12–14 months])

9. Meta-analysis: subgroup analysis according to primary care interventions across specified categoriesc: short-term follow-up (4–6 months), long-term follow-up (12–14 months)

10. Qualitative subgroup analysis according to primary and secondary cared

11. Qualitative subgroup analysis according to the number of participants in the group education programme for larger groups (16–18 participants)

Ellis [9]

Patient education on type 1 and type 2 diabetes

n = 21 (in total); n = 4 type 1 diabetes, n = 14 type 2 diabetes, n = 2 both, n = 1 data not available

1. Meta-analysis: overall summary effects

2. Meta-analysis: subgroup analysis in categories of interventions according to active components: type of delivery; content; teaching method; intensity of intervention

Fahey [10, 18]

Different strategies of care (e.g. self-management and patient education) used to improve the control and follow-up of patients with hypertension

n = 59 [10]; n = 56 [18]

Meta-analysis in pre-specified categories according to the types of intervention: self-monitoring education; education intervention directed at patients; education intervention directed at health professionals; health professional lead care; organisational interventions; appointment reminder systems

Gary [13]

Educational and behavioural interventions for individuals with type 2 diabetes

n = 18

1. Studies are descriptively summarised according to study characteristics and outcome measures

2. Meta-analysis: overall summary effects

3. Meta-analysis: subgroup analysis in categories of interventions according to active components: interventionist (physicians, nurse, dietician); mode of instruction (individual, group); topic of instruction (diet, medication, exercise, blood glucose self-monitoring)

Jack [23]

Educational methods in type 2 diabetes

n = 8

Studies are descriptively summarised according to outcome measures

Loveman [25]

Structured educational interventions for diabetes self-management

n = 24 (in total); n = 4 (type 1 diabetes); n = 16 (type 2 diabetes); n = 4 (type 1 or type 2 diabetes)

Studies are descriptively summarised within categories according to types of intervention: patient education for self-management in type 1 diabetes; patient education for self-management in type 2 diabetes; patient education for self-management in type 1 or type 2 diabetes

Newman [26]

Self-management interventions for chronic illness (type 2 diabetes, arthritis and asthma)

n = 63 (in total); n = 21 (type 2 diabetes); n = 24 (arthritis); n = 18 (asthma)

1. Studies are descriptively summarised within categories of interventions according to types of intervention: group-based; problem solving; stress management; cognitive behavioural; expressive writing; programme delivered by mail; combination of education with an action plan; exercise

2. Qualitative analysis to examine whether particular outcomes were focused upon, those in each study were classified into seven broad categories

Norris [27]

Self-management training in type 2 diabetes

n = 72

Studies are descriptively summarised within categories of primary educational focus: knowledge or information (didactic education or collaborative education); lifestyle behaviours; skill development; coping skills

Norris [15]

Self-management training in type 2 diabetes

n = 31

1. Meta-analysis: subgroup analysis according to study characteristics by the length of follow-up (follow-up ≥4 months after end of the intervention; follow-up 1–3 months after end of the intervention; follow-up during or immediately after the intervention)

2. Meta-analysis: regression analysis to investigate potential treatment interactions: patient age; baseline GHb; treatment (insulin, diet only, oral hypoglycaemic agents); number of contacts; total contact time; time frame over which the intervention was delivered; group vs individual presentation of the intervention; who delivered the intervention; educational focus (lifestyle, skills, knowledge, coping skills or mixed); follow-up in months; setting in the USA vs other countries

Norris [20]

Self-management training in type 1 and type 2 diabetes in community settings

n = 30 (in total); n = 8 (community gathering places); n = 10 (home); n = 10 (summer camps); n = 1 (schools); n = 1 (worksite)

1. Meta-analysis: overall summary effect sizes for evidence on effectiveness, summarised for each setting (community gathering places, home, camps, schools and worksite)e

2. Qualitative analysis: assessment within categories, summarised for each setting (community gathering places, home, camps, schools, and worksite): applicability; other positive and negative effects, economic efficiency; barriers for implementation

3. Additional descriptive information on positive and negative effects, applicability, economic efficiency, and barriers for implementation are described for each study

Renders [22]

Interventions targeted at health professionals and/or the structure of care to improve the management of patients with type 1 and type 2 diabetes

n = 41

1. Qualitative analysis: assessment whether the effectiveness of the interventions are influenced by pre-specified characteristics: types of intervention (professional, organisational, or financial); source of information (e.g. whether the intervention was carried or supported by professional organisation); type of diabetes

2. Additional descriptive information on effects are described for each study categorised according to types of intervention:

 Professional

 Organisational

 Professional and organisational in combination

3. Qualitative analysis: assessment whether the effectiveness of the interventions are influenced by active components of the programmes:

 Professional (professional interventions combined with patient education; postgraduate education of healthcare professionals combined with local consensus procedures and/or reminders and/or audit and feedback; education for both healthcare professionals and patients)

 Organisational (patient education and learner-centred counselling; nurse or pharmacist assumed part of the physician’s role and provided diabetes care in combination with a patient-orientated intervention; multidisciplinary teams in combination with arrangements for follow-up and patient education)

 Professional and organisational in combination (healthcare professionals received education through educational materials, educational meeting or both; arranging follow-up, audit and feedback, or generating reminders, or a combination; including patient education; involvement of nurses in diabetes management; including a telecommunication system)

Renders [21]

Interventions targeted at health professionals and/or the structure of care to improve the management of patients with type 1 and type 2 diabetes

n = 41

1. Qualitative analysis: effects are described for each study, categorised according to types of intervention:

 Professional

 Organisational

 Professional and organisational in combination

2. Qualitative analysis: assessment whether the effectiveness of the interventions are influenced by active components of the programmes

 Professional (professional interventions combined with patient education; postgraduate education of healthcare professionals combined with local consensus procedures and/or reminders and/or audit and feedback; education for both healthcare professionals and patients)

 Organisational (patient education and learner-centred counselling (nurse or pharmacist assumed part of the physician’s role and provided diabetes care in combination with a patient-orientated intervention; multidisciplinary teams in combination with arrangements for follow-up and patient education)

 Professional and organisational in combination (healthcare professionals received education through educational materials, educational meeting or both; arranging follow-up, audit and feedback, or generating reminders, or a combination; including patient education; involvement of nurses in diabetes management; including a telecommunication system)

Warsi [16]

Patient self-management educational programmes for chronic diseases (e.g. diabetes and hypertension)

n = 71

1. Meta-analysis: overall summary effect sizes only for evidence on efficacy for each outcome

2. Meta-analysis: regression across chronic diseases to identify which components were associated with greater clinical benefits. The dependent variable was the summary effect size across all diseases. Indicator variables were each disease and its endpoint. Independent variables were percentage of dropouts, number of educational sessions, programme duration, programme format, education mode (face-to-face education), and reference to a behavioural science model

aThe authors summarised seven studies to point out the research that had taken place over the last 10 years in Canada, although they did not meet their inclusion criteria; bsome interventions (n = 3) are allocated into both categories; cstudies were excluded from analysis if heterogeneity was substantial; dif there was substantial heterogeneity between studies meta-analysis was not performed; ethe effectiveness of interventions for type 1 and type 2 diabetes was examined separately

GHb glycated haemoglobin (HbA1, HbA1c)

The majority of search strategies were comprehensive and transparently reported (ESM Table 12). In eight reviews, experts in the field were contacted to identify additional publications [10, 11, 13, 15, 18, 19, 20, 25]. Reference tracking was performed in 11 reviews [10, 11, 13, 15, 16, 18, 19, 20, 21, 22, 24]. Predefined contact of all authors was not reported in any review.

In six reviews only RCTs were considered [9, 10, 13, 15, 18, 26, 27] (see ESM Table 12). Following the selection criteria of the reviews, 16 publications [28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43] referring to our three reference programmes should have been identified. In practice, a total of 11 were identified [5, 28, 30, 32, 34, 38, 39, 43, 44, 45, 46], and seven [28, 32, 34, 35, 38, 39, 46] were included in at least one of the reviews. In six reviews the main controlled trials of our reference programmes [28, 30, 31, 32, 38, 44] were included in data synthesis [10, 11, 16, 18, 19, 25]. Out of 25 [5, 33, 34, 35, 36, 37, 39, 41, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] replication trials (phase 4, see text box: Phases of ‘increasing evidence of complex interventions’) four [34, 35, 39, 46] were included in six reviews [16, 19, 21, 22, 24, 25]. Different reviews on identical topics included different publications referring to the same reference programme.

No review explicitly evaluated publications on the theoretical basis of the reference programmes (preclinical/theoretical phase of ‘increasing evidence’). The UKMRC approach of the evaluation of complex interventions [1, 2] was cited in three reviews [18, 19, 25]. We did not find that the approach influenced their methodology.

All included reviews reported at least some features of the assessed interventions, which were either active components (e.g. setting, duration, interventionist, formal syllabus) or study characteristics (e.g. follow-up, age of participants, study quality). The investigated features were heterogeneous. Six reviews used regression or subgroup analysis to analyse the effect of single active components on outcomes [9, 11, 13, 15, 16, 19]. Descriptive analysis of the influence of active components was performed in five reviews [19, 21, 22, 24, 26]. Most of the components of our three reference programmes had been identified, but only a few had been evaluated. The identified and included components were different in reviews on identical topics (ESM Table 13).

In 12 reviews the included programmes were categorised by intervention [9, 10, 11, 13, 15, 16, 18, 19, 21, 22, 25, 26, 27]. Categories were defined according to the active components of the included interventions (e.g. setting, duration, interventionist, formal syllabus) [9, 11, 13, 15, 16, 19, 21, 22], the type of intervention (e.g. type of disease, type of activity, organisational interventions) [10, 18, 21, 22, 25, 26, 27] aimed at patients or health professionals, or the study characteristics (e.g. follow-up, age of participants, study quality) [11, 13, 15, 19, 24]. The applied categories varied.

The calculation of overall effects within categories of interventions was performed in eight reviews [9, 10, 11, 13, 15, 16, 18, 19, 20]. Nine reviews included qualitative data synthesis within categories [13, 19, 20, 21, 22, 24, 25, 26, 27] (Table 1). In some reviews [10, 11, 18] the allocation into categories did not meet the type of intervention [32, 34, 35].

No review included all relevant patient outcome measures of our reference studies (ESM Table 14). If quantitative reviews were aimed at exploring multiple outcome measures, multiple meta-analyses were performed [10, 11, 13, 16, 18, 19, 20]. If quantitative analysis was used according to the effect of active components, regression analysis or subgroup analysis was conducted [9, 11, 13, 15, 16, 19]. Interdependencies between different outcome measures of the same educational intervention were considered in only three reviews [25, 26, 27], all of which performed qualitative data synthesis.

Four out of 12 contacted authors responded (P. Corabian, Alberta Heritage Foundation for Medical Research, Edmonton, AB, Canada; T. Deakin, Pendle Rossendale Primary Care Trust, Burnley General Hospital, Burnley, UK; T. Fahey, University of Dundee, Dundee, UK; C. M. Renders, Institute of Research in Extramural Medicine, Amsterdam, The Netherlands). All of them replied that their reviews were, in general, considered appropriately and their findings interpreted correctly. Specific comments on methodological problems such as study selection and categorisation of components of education programmes were acknowledged. We corrected five misinterpretations in reporting data and added unpublished information according to the authors’ comments.

Discussion

Most of the included systematic reviews discussed methodological challenges to appraise complex interventions. However, these considerations did not have adequate impact on the methods used in data synthesis.

The selection criteria used in most of the analysed reviews excluded study types other than RCTs; other important types of publications concerning the ‘increasing evidence’ were rarely included. The investigated reviews did not differentiate between the main controlled trial(s) and controlled replication trials referring to the same programme.

The importance of considering the theoretical basis of an education programme was widely discussed. However, we could not identify any approach to systematically assess the theoretical basis and its influence on judging the quality of the programmes.

The majority of reviews reported that the included programmes had been ‘multifaceted’ or ‘multidimensional’ or ‘consist of multiple active components.’ However, neither the UKMRC approach of ‘increasing evidence’ (published in 2000) [1, 2], nor a similar approach was integrated in any review. Methodological changes may take a long time to become accepted.

The categorisation of interventions used in the included reviews often seemed to be arbitrary: each review used different categories; none of them explained the rationale of their categorisation. Allocation of complex interventions into categories can be problematic, even if categories are derived from core components of programmes (e.g. education directed to the patient). If the categories refer to single but interdependent components, the compartmentalisation of efficacy is not possible.

Regression and subgroup analyses are the best tools for exploring heterogeneity [8]. However, these techniques should not be misused to identify the contributions of the various active components (e.g. intensity or duration of the programme) on the overall effect (e.g. knowledge of the target group or the importance perceived by the provider).

The analysed reviews did not consider all relevant patient outcome parameters. Components of complex outcome measures were singled out, especially if they used meta-analysis. The complex interdependency of individual treatment goals and outcomes (e.g. changes in medication and metabolic or blood pressure control) remained unexplored. Clinicians with experience in diabetes care might be consulted to focus the relevant clinical issues.

Recapitulating the research findings, we propose to take the following criteria into account when undertaking systematic reviews of complex interventions:
  1. 1.

    All studies referring to development, evaluation and implementation (‘increasing evidence’) should be considered, differentiating between phases of ‘increasing evidence’.

     
  2. 2.

    Literature searches should therefore not be limited by criteria such as certain types of studies, specific target groups and publication dates; reference tracking should be performed and authors should be contacted systematically.

     
  3. 3.

    Active components should be described and assessed, but should only be examined separately if they are independent and should not be disassembled if they work interdependently.

     
  4. 4.

    Education programmes should not be allocated into categories referring to interdependent components.

     
  5. 5.

    All relevant patient-orientated outcome parameters should be included.

     
  6. 6.

    Pooling of outcome measures across different programmes is usually inappropriate. Instead, the relative importance of outcomes [60] and the complex interdependency between treatment goals and outcomes should be described in detail.

     

Information necessary for the evaluation of education programmes is difficult or impossible to identify. Therefore, specific search strategies need to be developed and validated that aim to identify publications on all phases of ‘increasing evidence’. To facilitate the implementation of such an approach, what is needed is an electronic database that provides available programmes together with all relevant background information, or the implementation of such a system into existing databases.

Acknowledgement

We thank G. Meyer (University of Bremen, Bremen, Germany) for her insightful comments on the draft version of this review.

Duality of interest

None of the authors have a duality of interest with regard to this work.

Supplementary material

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  • M. Lenz
    • 1
  • A. Steckelberg
    • 1
  • B. Richter
    • 2
  • I. Mühlhauser
    • 3
  1. 1.Unit of Health Sciences and EducationUniversity of HamburgHamburgGermany
  2. 2.Department of General Practice, University HospitalHeinrich-Heine UniversityDusseldorfGermany
  3. 3.Unit of Health Sciences and EducationUniversity of HamburgHamburgGermany

Personalised recommendations