Background

Although there are earlier examples of statistical combination of data from multiple studies, the practice began in earnest in the 1970s in the social sciences, and the term meta-analysis was coined by Glass as recently as 1976 [1]. Systematic reviews, including statistical synthesis of the results of multiple studies, are now commonplace in medical research, notably through the work of The Cochrane Collaboration. Yet it is only about 30 years since these studies began to appear in medical journals.

Methodological rigour was a key principle of The Cochrane Collaboration from its inception two decades ago. Accordingly, statistical methods have always been seen as critically important. In this article, we provide a historical account of the beginnings of the Statistical Methods Group (SMG) of The Cochrane Collaboration and its subsequent work, and reflect on the challenges that the group faces in the future. Many individuals and organisations outside the Collaboration have made significant contributions to the development of statistical aspects of systematic review methods in health care; however, in this paper, we restrict our focus to contributions of the SMG and its members.

History of the Statistical Methods Group

A key publication pre-dating the founding of The Cochrane Collaboration was the two volume Effective Care in Pregnancy and Childbirth, which included a long chapter on methods for synthesising evidence from multiple studies (now called systematic reviews), including methods of meta-analysis [2]. It was recognised, however, that the field was in its infancy and there were many unresolved issues. Accordingly, the first Cochrane meeting on statistics was held at the UK Cochrane Centre in Oxford in July 1993, masterminded by Iain Chalmers and co-chaired by Ken Schulz and Doug Altman. This meeting took place just one week after a one-day meeting in London on ‘Systematic Reviews’, organised jointly by the British Medical Journal and the UK Cochrane Centre, which led to a book of the same name [3].

Although the statistics meeting predated (by a few months) the creation of The Cochrane Collaboration, the goal of reviewing the literature on healthcare interventions across specialties was already clear. There were 16 participants, most of whom were statisticians. The workshop was convened to develop guidelines on statistical methods for data synthesis. The hope was that the meeting would identify areas where there was reasonable consensus, and those which caused debate and could be the subject of further research. Opening the meeting, Andy Oxman described the need to develop guidelines for people preparing systematic reviews. His suggestion that it was desirable for the statistical methods to be consistent across reviews as far as possible was endorsed by most participants. It was agreed that the planned Cochrane Collaboration software should have explicit default methods of analysis and presentation but that reviewers be able to override these. Areas where there was much discussion but a lack of consensus included the use of odds ratio or relative risk (risk ratio) for meta-analysis of binary outcomes and the use of fixed or random effects analyses in the presence of heterogeneity. The minutes and list participants from this meeting are available in Additional file 1 - Oxford 1993 Workshop Report.

Following on from the meeting, Doug Altman and Ken Schulz became the founding co-convenors of the Statistical Methods Working Group (SMWG), as it was initially called. (The term ‘working’ was later dropped from the names of all Cochrane Methods Groups.) The SMWG produced an initial list of ‘possible research topics’ , shown in Table 1. They noted that “This list indicates some areas where either there is no consensus as yet about the best approach or where inadequate information is available. It should not be taken as an inclusive list of all intended topics of investigation.” From the 1993 recommendations onwards, the SMG has had a major influence on the specification of The Cochrane Collaboration software, especially in relation to meta-analysis, as discussed below.

Table 1 Research topics proposed in 1995 for the Cochrane Statistical Methods Working Group
Full size table

The participants at the 1993 meeting also noted the importance of studying the benefits of collecting individual patient data versus using published summary statistics. That topic was deemed to fit more within the remit of a specialist group, and a meeting of interested parties in Oxford in April 1994 led to the founding of the Meta-analyses using Individual Patient Data Methods Working Group [4].

In March 1995, an application was made for the “Cochrane Collaboration Working Group on Statistical Methods for Data Synthesis” to be formally registered. At that time the group put forward the following objectives:

  • To develop and update guidelines on statistical methods for data synthesis.

  • To develop and update guidelines for the integration of the methodological quality of randomised controlled trials into statistical methods for data synthesis.

  • To serve as a “clearinghouse” for names of individuals who are willing and able to provide technical consultations to those in The Cochrane Collaboration on specific methodological issues.

It was recognised early that the SMG would benefit from funding to support methodological explorations and development of enhanced guidance, and the SMG made the following argument in an unsuccessful attempt to get funding in 1997:

“Many of the statistical queries and issues in the Cochrane Collaboration are not straightforward. While the basic statistical methods for meta-analysis are well known, in any particular systematic review it is unusual for these to be applicable without problems. For example there are often difficulties in extracting the required information from publications, in addressing both clinical and statistical heterogeneity between trials, in considering the potential effects of publication bias, in undertaking appropriate sensitivity analyses to investigate the robustness the conclusions made, and in investigating the effects of methodological quality of the primary studies on the overall results and interpretation. These difficult issues require both empirical and methodological work.”

In the absence of dedicated funding for the SMG, however, methodological advances by members of the group have been motivated by SMG discussions (and other Cochrane activities) but not determined by the SMG. Thus, progress addressing these research topics has been piecemeal. Yet members of the group have made many key contributions to advancing the statistical aspects of systematic review methods, as well as authoring several books on methods for systematic reviews and meta-analyses (for example, [5–9]). Thus evolution of statistical methodology used in Cochrane reviews was informed mainly through discussion of new developments initiated outside the Collaboration, although many SMG members were involved in those advances. In 1997, the SMG merged with the Quality of Reporting Trials Methods Group, led by David Moher, who became a co-convenor of the SMG.

Communication within SMG was always primarily via an email discussion list. Membership of the SMG grew (from 33 members in 1995, to 133 by 2006), with the group acting primarily as a discussion forum. To try to advance understanding of specific methodological areas in the late 1990s, the SMG established subgroups for specific research topics and members were invited to join these. The most overt success came from a group of SMG members who were interested in the incorporation of crossover trials in meta-analyses. Their work led to a journal publication [10].

Statistical issues relating to systematic review of diagnostic test accuracy were never part of the SMG’s remit, as the Screening and Diagnostic Tests Methods Group had also existed from the early years of the Collaboration. However, other parts of the SMG’s scope have split off into specialist Methods Groups over time. In 2000, the activities relating to problems of reporting research were taken up by a newly formed Reporting Bias Methods group (renamed the Bias Methods Group in 2005). Statistical methods for other specific contexts have also been addressed by the formation of other new methods groups, notably the Non-randomised Studies Methods Group in 1999 and Comparing Multiple Interventions Methods Group in 2010.

The Statistical Methods Group and methodological developments in meta-analysis

The SMG has maintained and promoted a research agenda of issues important to the statistical synthesis of study findings. Important methodological developments have often been motivated through SMG members’ involvement in Cochrane reviews and through discussion on the SMG email list, at training events and during scientific meetings (see next section for more detail). Most of the research topics that were identified by the Group in 1995 (Table 1) have now been addressed, and the guidance based on this research is offered in the Cochrane Handbook for Systematic Reviews of Interventions (henceforth referred to as the Cochrane Handbook) [8]. The Cochrane Handbook has had substantial impact, with more than 7,000 citations, approximately 74% of which are from sources other than Cochrane Reviews [11]. While not all these citations can be attributed to the statistical chapters 9, 10 and 16 [12–14], these chapters are integral to the Cochrane Handbook, and are a key resource for statistical synthesis in systematic reviews.

The contribution of SMG members to the meta-analysis literature and the Cochrane Handbook has been extensive and has helped to shape the wider meta-analysis landscape. These contributions have included not only methodological developments, but also empirical studies that have evaluated existing methods. Below we highlight some landmark methodological developments that have influenced the field of meta-analysis. We include highly cited papers, and those that have received awards at Cochrane Colloquia (the annual conferences of The Cochrane Collaboration), made an important novel contribution, or generated important debate. This list is not exhaustive or fully representative of the research activity of SMG members. A full list of publications and current SMG members can be found online at smg.cochrane.org/publications and smg.cochrane.org/our-contributors.

The importance of the choice of the effect measure for dichotomous data in the presence of heterogeneity was examined by Deeks [15]. Results from this research have largely framed current guidance that suggests relative measures (odds ratio or risk ratio) to be preferable to absolute (risk difference) (for example, [9, 12, 16]). Sweeting et al.[17], Bradburn et al.[18] and others have evaluated the methodology for sparse dichotomous data and made suggestions for their optimal modelling [19, 20]. The potential of the ratio of means for continuous outcomes as an alternative effect measure to the mean difference and standardised mean difference was presented at the 15th and 16th Cochrane Colloquia (in 2007 and 2008). A series of papers examining the performance of this measure have been published [21–23]. Different methods for meta-analysis of skewed data have been evaluated [24]. Missing outcome data in trials with dichotomous outcomes have been addressed in meta-analyses primarily through sensitivity analysis methods [25, 26]. However, there are still many unanswered questions of how best to handle missing outcome data in meta-analysis of trials, particularly where the outcome is continuous, and this led The Cochrane Collaboration to fund a project on missing data in 2011. Methods for handling missing standard deviations in meta-analysis have been described and appraised in Wiebe et al.[27] and empirically evaluated [28]. Clinical fields where survival data are common have largely benefited from the development of methods to combine time-to-event data [29–31].

SMG members have made methodological contributions to the synthesis of results from trials with non-standard designs. Extensive research has been undertaken evaluating strategies for combining results from parallel group and cross-over trials [10, 32–34], and simple methods to deal with paired data (such as eyes or arms) have been proposed [35]. These methods are recommended in the Cochrane Handbook.

The I 2 statistic, which quantifies the amount of heterogeneity [36], is now probably the most popular method to evaluate heterogeneity (for example, [16, 37, 38]). The general medical journal article published in 2003 that describes I 2 has been cited more than 6,000 times [39], and the statistic has been included in RevMan (The Cochrane Collaboration’s software to prepare and maintain reviews [40]). Problems with interpretation of the I 2 statistic have been discussed [41]. Current guidance encourages reviewers to explore sources of heterogeneity by employing techniques such as subgroup analysis and meta-regression, although the pitfalls of such post-hoc analyses have been highlighted [42]. While meta-regression has not been implemented in RevMan, since this method is rarely appropriate in Cochrane reviews which typically include few studies [43], results and graphs from meta-regression fitted in statistical packages can be easily imported into RevMan. A significance test to investigate if there are differences between two or more subgroups has recently been added [9]. Cochrane Handbook guidance recommends the use of random effects in the presence of unexplained heterogeneity and the results are summarised in terms of an average combined effect size and its standard error. However, more recently, it has been proposed that in the presence of unexplained heterogeneity, an average effect is not informative, and that instead, the distribution of effects should be considered [44]. This has led to the development of different methods to calculate prediction intervals, which better convey heterogeneity of the results [44].

One particular source of heterogeneity that has been observed in meta-analyses is the difference in the magnitude of effects between small and large studies, often termed ‘small-study effects’. The Cochrane Handbook explicitly warns against misinterpreting funnel plot asymmetry as necessarily indicating publication bias, and the connections between heterogeneity, funnel plot asymmetry and small study effects have long been a focus of members of the SMG [45]. Consequently, SMG members have published several methodological developments evaluating the possibility of small study effects [46–52], visualizing them [53] and accounting for them [54–58].

SMG members have undertaken methodological work on network meta-analysis (also called multiple-treatments meta-analysis and mixed-treatment comparison), a multidimensional extension of meta-analysis aiming to make inferences about the relative effectiveness of many treatments for the same health condition. The methodology, presented in Higgins and Whitehead [59], takes advantage of indirect evidence in a network of interventions and has attracted much interest in recent years within the SMG and, more broadly, The Cochrane Collaboration. Workshops and contributing talks on the topic have been presented at Cochrane Colloquia since 2005. SMG members have contributed many methodological and applied publications on this topic in the scientific literature (see, for example, [60–69]).

The SMG recognised early on the potential of the method to answer policy-relevant questions and, because of the specialist nature of the methods, initiated the creation of the Comparing Multiple Interventions Methods Group (CMIMG). The role of the CMIMG is to develop and maintain guidance for undertaking and publishing Cochrane Reviews that compare multiple interventions via network meta-analysis. The promise of network meta-analysis has led to rapid development and adoption of the methods, stimulating collaboration between members of the CMIMG and researchers external to The Cochrane Collaboration.

Many of the SMG developments outlined above were initially presented at Cochrane Colloquia, and several have received the Thomas C Chalmers award, an award presented for best oral or poster presentation at a Cochrane Colloquium addressing methodological issues related to systematic reviews. SMG members have received in total 11 prizes on topics strictly within the scope of the SMG (Table 2), and several more for presentations that relate to bias, individual participant data and diagnostic tests.

Table 2 Thomas C Chalmers awards for statistical issues related to systematic reviews
Full size table

SMG members have also co-authored publications that fall under the remit of other Cochrane methods groups including the Individual Participant Data Meta-analysis Methods Group (ipdmamg.cochrane.org), Prospective Meta-Analysis Methods Group (pma.cochrane.org), Bias Methods Group (bmg.cochrane.org), Prognosis Methods Group (prognosismethods.cochrane.org) and the Non-Randomised Studies Methods Group.

Implementation and dissemination of statistical methods within The Cochrane Collaboration

The SMG is involved in a range of activities within the Collaboration that aim to provide support to systematic review authors, the staff of Cochrane Review Groups and statisticians. A particular role of the SMG is to provide guidance when there are alternative approaches, such as handling multi-arm trials, handling missing outcomes within trials, imputing standard deviations when not presented, and mixing means with medians.

All CRGs are expected to involve statisticians [70]. They contribute to the CRGs in various ways including being statistical editors of CRGs; peer review, collaboration and authorship on systematic reviews; development of methods resources for CRGs; involvement in methodological quality improvement; and responding to statistical queries. In addition, statisticians in some CRGs have undertaken research to address methods issues prompted by their CRG (for example, [71–74]). The SMG maintains an email list that provides a forum for discussion of statistical issues relevant to systematic reviews. Membership of the list constitutes membership of the SMG, and includes CRG statisticians and other statisticians/methodologists with an interest in systematic review methods. The list currently has 237 members (April 2013), from 27 countries, primarily based in the UK (37%), USA (12%), Australia (10%), Germany (9%) and Canada (8%).

Members of the SMG contribute to the dissemination of statistical methods through training at regional Cochrane events and Colloquia, reviewing of standard review author training materials, and contributions to the Cochrane Handbook[8]. A set of core workshops, targeting review authors and CRG staff, are presented at Colloquia covering introductory topics on meta-analysis including: basic ideas, meta-analysis of binary and continuous outcomes, dealing with heterogeneity, and inclusion of non-standard studies and non-standard data. A full list of workshops presented at Colloquia is available at smg.cochrane.org/teachingtraining-workshops. The SMG has run two major training events for Cochrane statisticians. The first ‘Summer school for Cochrane Statisticians’ was held in Oxford, UK, in July 2001, while the second, ‘Cochrane Statistical Methods Group Training Course: Addressing advanced issues in meta-analytical technique’ was held in Cardiff, UK, in April 2010. The latter course led to a series of freely available online materials, including slidecasts of the presentations (smg.cochrane.org/teachingtraining-workshops/smg_training_course_2010).

In addition to training workshops, SMG members have organised and presented in SMG scientific meetings, special methods sessions, and plenaries at Cochrane Colloquia. The purpose of these sessions has often been to discuss or present new or controversial issues in meta-analysis. Ad hoc meetings outside Cochrane Colloquia have also been convened to discuss specific methodological issues. Examples of key events are presented in Table 3.

Table 3 Description of the Statistical Methods Group scientific meetings, special methods sessions, and plenaries
Full size table

Discussion

What has statistics done for The Cochrane Collaboration?

The need for statistical synthesis methods to enable interpretation of results from multiple studies was recognised more than 50 years ago [1, 75–77]. Meta-analyses now form a core component of many Cochrane Reviews. Without such a method, interpreting the effectiveness of an intervention is difficult, if not impossible. Further, meta-analysis allows the combining of results from a series of small studies, to answer a question regarding the effectiveness of an intervention, which could otherwise not be answered from the individual studies. Extensions of meta-analysis methods provide systematic review authors with an extensive tool kit to answer questions beyond whether an intervention is effective. For example, which factors may modify the magnitude of the intervention effect [42], what is the likely effect of an intervention in an individual setting [44], or what is the relative effectiveness of many interventions for the same health condition [64]. These extensions increase the utility of systematic reviews in answering key questions of relevance to policy makers, health care decision makers and patients.

What has The Cochrane Collaboration done for statistics?

The Cochrane Collaboration has provided the impetus, structure and coordination for an international group of methodologists to contribute to the development of statistical aspects of systematic review methods. This has expedited development of systematic review methods that would not likely have been achieved as rapidly without such an organisation. Coupled with the benefits of an international collaboration of methodologists, the main output of The Cochrane Collaboration, the Cochrane Database of Systematic Reviews, has facilitated methodological research, by providing a large and unique repository of data on systematic reviews and meta-analyses (for example, [23, 43, 78–83]). A recent paper highlighted those articles published in statistical journals that appear to be having the most direct impact on general and internal medicine research [84]. The authors identified the 18 most-cited biostatistical articles; 11 were about methodological aspects of meta-analysis, 7 of which were co-authored by SMG members [17, 18, 32, 36, 42, 48, 64].

What are the research gaps in meta-analysis methods?

Further research to establish the optimal strategy for combining intervention effects estimated from analyses of final values and change scores is required [85]; SMG members are currently investigating this area [86]. Further research is also needed to develop and evaluate methods for combining results from trials where continuous outcome data have been categorised using varying cut-points [87]. A related issue is the best approach to deal with the scenario where trials present results for the same outcome based on a mixture of continuous and dichotomous measures. Statistical methods that adjust for potential bias in included studies (as, for example, in Turner et al.[57]) might be desirable, particularly when non-randomised studies are included. Evaluation of methods to adjust for ‘hidden’ clustering in individually randomised trials arising through the organisation of the intervention (for example, therapists treating groups of patients, or group therapy sessions, surgical procedures) would be useful [88, 89]. Development of methodology for meta-analysis of time-to-event data that allow for more complex multistate modeling, such as competing risks, is needed. Several developments have taken place in recent years in the field of multivariate meta-analysis [90–92]. The potential of different methods (for example, maximum likelihood, method of moments and Bayesian approaches) to estimate parameters of multivariate meta-analysis models and the estimation of within-study correlation in the absence of detailed study-level data needs further attention [93, 94]. Methods to describe and quantify heterogeneity (for example, extending the I 2 statistic) and visually display results from multivariate meta-analysis have been recently suggested [95] and their usefulness and performance remains to be tested in practice. Finally, the appropriateness of the statistical methods used in Cochrane Reviews requires regular assessment, particularly as new methods appear in the literature. Cochrane reviews have specific characteristics (for example, they typically have few studies [43]) and knowledge of how synthesis methods perform, in which circumstances their performance is most compromised, and how large the impact is when this occurs, is required to develop guidance for the Cochrane Handbook, and include the most appropriate methods in RevMan. Simulation studies and extensive meta-epidemiological studies provide the necessary research to underpin this guidance.

Thoughts on our future direction

While 10 years ago, say, some may have felt that all the main statistical issues in systematic reviews had been addressed, the reality is that new challenges have continued to arise, and the pace of methodological work has not slowed. While we cannot say in which areas future innovations will appear we are confident that such innovations will indeed be made. And we expect the SMG members to continue to be among those at the forefront of this work. The SMG will continue to have a key role in advising The Cochrane Collaboration on which methods are, or are not, recommended, or whether new approaches need more evaluation before a decision can be made. They also have an important role in further refinement and extension of the handbook and other training materials.

Despite the contribution SMG members have made to advance the statistical aspects of systematic review methods, funding the work of the SMG and, more broadly, methodological research in evidence synthesis remains an ongoing challenge. Methodological research funding programmes, such as those provided in the United Kingdom (for example, the Medical Research Council/National Institute for Health Research Methodology Research Programme), United States (for example, Agency for Healthcare Research and Quality research grants for comparative effectiveness), and Europe (for example, the European Commission’s Seventh Framework Programme and Horizon 2020), are critical for the ongoing development and evaluation of methods, and production of guidance. Specific funding for the SMG would allow for a more priority-driven and structured approach to addressing important gaps in statistical synthesis methods that are of particular relevance to Cochrane Reviews, thus expediting the availability of research on which to base well-informed decisions. Incorporation into the Cochrane Handbook and RevMan of new statistical synthesis methods that make the best use of evidence will contribute to The Cochrane Collaboration’s goal of producing high quality research evidence. Given the impact of the Cochrane Handbook and the widespread use of RevMan, the benefits would likely extend well beyond reviews completed under the auspices of The Cochrane Collaboration.

Conclusions

Over the past 20 years, the SMG has played a pivotal role within The Cochrane Collaboration in determining the direction of statistical methods used within Cochrane reviews. Members of the SMG have made key contributions to advancing statistical aspects of systematic reviews. Many research gaps in statistical methods of systematic reviews remain, and will continue to arise as the field of meta-analysis develops. Statisticians and methodologists will lead the development and evaluation of these methods, thus being critical players in the production of evidence.