Background

Overviews of systematic reviews are an increasingly popular method of evidence synthesis [1, 2], and there are a growing number of resources, including guidelines, recommendations, descriptions and systematic reviews, relating to overview methods [2,3,4,5,6]. While there are some areas of agreement in relation to optimal overview methods, particularly in the early stages of overview completion [5], there remains considerable uncertainty around some key areas of methodology [3, 5, 7, 8] and a need for clearer standards and reporting guidance, supported by research evidence, to enhance methodological quality of overviews [1,2,3, 5, 6].

At the UK Cochrane Symposium in 2016, a workshop focusing on the methods and challenges associated with overviews included presentations relating to five selected ongoing or recently completed overviews [9]. These were selected in order to highlight practical variations in the methods adopted within different overviews and to provide tangible examples of the decisions and challenges associated with the completion of an overview. During and following this workshop, we explored practical issues associated with planning and preparing these exemplar overviews, discussed the impact of methodological decisions and reached consensus on key methodological challenges and potential implications for future overview authors.

Subsequent to our author group reaching consensus, Ballard [3] published the results of a scoping review of methodological guidance for overviews. This synthesis identified areas where there was consensus relating to overview methods and highlighted five key areas where there was debate or uncertainty including “(i) overlapping systematic reviews, (ii) the scope of systematic reviews, (iii) evaluating the quality and reporting of included research, (iv) updating included systematic reviews, and (v) synthesizing and reporting the results of included systematic reviews” [3]. Since the search period of this comprehensive synthesis, a number of further papers relevant to overview methods have been published adding to the discussion and debate around key areas of uncertainty. These include summaries of guidance relating to overview methods [5], descriptions and debate relating to the use of GRADE [10,11,12] and AMSTAR [13], the development of a new tool to assess risk of bias (ROBIS [14]), and protocols for ongoing work in this field [15, 16]. The specific debate around methodological challenges presented by Ballard [3], and these subsequent related publications, afforded us a timely opportunity to compare the results of our independent consensus arising from our five exemplar overviews and to explore the practical implications of the current uncertainties around overview methods for authors planning protocols for new overviews.

Our aim is therefore to use five exemplar overviews:

  1. (i)

    To provide practical examples of methodological approaches to the planning and preparing of overviews and discuss the impact of methodological decisions

  2. (ii)

    To explore methodological challenges reported by overview authors and compare these with areas of debate identified within current literature

  3. (iii)

    To discuss practical implications which may support methodological decision making during development of protocols for future overviews

Methods

Methodological features of exemplar overviews

Methods of the five exemplar overviews were tabulated, relating to all key stages in the process of completing an overview. Justification for the selection and use of the methods at each stage were provided by the overview authors, based on the presentations provided at the Cochrane Workshop and supplemented by discussion (authors representing all five exemplar overviews are included as authors on this paper). Points of agreement and dissonance were highlighted and discussed. The key methodological challenges identified by each overview author were discussed at the workshop and synthesised into a list, with similar challenges merged. A description of each challenge, and the solutions (if any) implemented within individual overviews, was developed through discussion. A list of potential implications for future overview authors, arising from the practical experiences within these exemplar overviews, was developed iteratively. This initially comprised a list of key methodological decisions made by the overview authors arising from the tabulated descriptions of methods and methodological challenges; this was circulated amongst the overview authors who added to, and refined until consensus was reached on the final list.

Key challenges within exemplar overviews and complementarity with published literature on overview methods

The key methodological challenges identified within each of the exemplar overviews were systematically compared and contrasted with the recognised areas of debate [3], which were published after consensus had been reached on the methodological challenges identified within our exemplar overviews. Each of the methodological challenges from our exemplar overviews were tabulated, and overview authors considered levels of complementarity [17, 18] between the recognised areas of debate and the methodological challenges identified from our exemplar overviews, applying categories of “agreement” (the methodological challenges identified within the exemplar overviews is in agreement with the findings of Ballard [3]), “dissonance” (the methodological challenges identified within the exemplar overviews differs from, or conflicts with, the findings of Ballard [3]) or “silence” (the methodological challenges identified within the exemplar overviews were not addressed within the findings of Ballard [3, 17, 18]. One author applied the initial categorisation, which were then appraised by the other authors and any areas of disagreement highlighted. Disagreements were discussed between all authors until consensus was reached, the findings of other recent publications were considered, and any changes from the original categorisation noted.

Implications for development of protocols for future overviews

Finally, based on the presentations on the methods of the exemplar overviews from the Cochrane Symposium which all provided a chronological description of the overview process and the synthesis of methodological features, common features and differences between our exemplar overviews were identified and agreed. Considering these features and differences alongside the perceived methodological challenges and complementarity with published literature, overview authors then debated and agreed key implications for the development of protocols for future overviews.

Results

Methodological features of exemplar overviews

Table 1 summarises the key methodological features of the five exemplar overviews. Three of the overviews are Cochrane overviews [19,20,21], one is an overview of reviews of diagnostic test accuracy (DTA) [22], and one a mixed method overview carried out by the EPPI centre [23]. Two of the Cochrane overviews are produced by the same research team (AP is an author on [19] and [21]), while the remaining three are each produced by distinct author groups. Detailed protocols describing the planned methods were agreed and made available a priori for all five of the overviews; two of the overviews are now complete and published [19, 23], while three are in the final stages of completion and write-up [20,21,22]. Justification for the selection and use of the methods as described in the protocol and used at each stage of the overview process are provided in Table 1. Characteristics of the included reviews and any data relating to the results of meta-analyses which were extracted and reported within the systematic reviews are summarised in Table 2.

Table 1 Methodological features of exemplar overviews
Full size table
Table 2 Characteristics of included reviews which are reported in tables in overviews
Full size table

Key challenges within exemplar overviews and complementarity with published literature on overview methods

Discussion between overview authors led to agreement that there were a total of eight key methodological challenges encountered across all the exemplar overviews. These were the following:

Table 3 Summary of the perceived key methodological challenges associated with each of the exemplar overviews, a description of what the challenge was, and examples of how this challenge was dealt with within individual overviews
Full size table

  • Overlap between reviews (studies appearing in more than one review)

  • Reviews are out of date

  • Definition of “systematic review”

  • Assessment of methodological quality of reviews

  • Quality of reporting within reviews

  • Applying GRADE

  • Potential for publication bias

  • Summarising key findings in brief accessible format, suitable for informing decision making

The judgement of complementarity between these identified methodological challenges and the areas of debate identified within current literature are summarised in Table 3 and below.

Complementarity with published literature

  • Agreement: seven of our identified key methodological challenges were in agreement with the issues raised by Ballard [3]; although, some additional specific points (categorised as areas of dissonance) were raised by our exemplar overviews in relation to three of the issues (see Table 3 for details)

  • Silence:

    • o Methodological challenge highlighted by our overviews, but not raised by Ballard [3]): there was one area of silence, with our exemplar overviews identifying challenges relating to gaining agreement with the ROBIS tool. (Note: the ROBIS tool was published after the search period of Ballard [3])

    • o Methodological issue identified by Ballard [3] but not from our exemplar overviews: there were two areas of silence. These included challenges relating to the “scope of systematic reviews”, where there is a “mismatch” between the scope of the systematic review and the remit or focus of the overview, and challenges associated with the assessment of risk of bias of primary trials, where appropriate quality assessment was not used within the systematic review

  • Dissonance: there were no areas of dissonance between the methodological issues raised by our overviews and Ballard [3]. However, there was disagreement noted in relation to conclusions drawn by Ballard [3] relating to the function of overviews and that overviews cannot identify evidence gaps: three of our exemplar overviews [19, 21, 23] clearly concluded that the overview had successfully identified gaps in the evidence.

Challenges common to systematic reviews

Also common to all exemplar overviews were a number of challenges which were considered to be routine amongst most systematic reviews. These included challenges such as the management of large volumes of review information and data extraction, the available time and resources, and reaching consensual decisions amongst overview authors. Strategies implemented to address these challenges included methods of automating data extraction (for example downloading data files for Cochrane reviews) and audio-recording discussions between overview authors.

Implications for development of protocols for future overviews

The identified common methodological features, implications for future overviews and implications for development of protocols for future overviews are summarised in Table 4.

Table 4 Summary of common features and differences between the exemplar overviews, and implications for development of protocols for future overviews
Full size table

Discussion

Using five recently completed or ongoing overviews of reviews, we have explored the methodological features of overviews and the key methodological challenges which are reported by the overview authors and have systematically compared these findings with synthesised evidence relating to current guidance for overview methods (Ballard [3]). Our five overviews provide examples of a range of different types of overviews, including Cochrane overviews, mixed method overviews and overviews of reviews of diagnostic test accuracy. There are some methodological differences between our exemplar overviews which cannot be attributed to the type or aim of the overviews, and these arguably occur due to lack of information and guidance on the optimal methods for overviews and reinforce recent calls for methodological research and improved guidance for overviews [2, 3, 5, 24, 25].

Eight key methodological challenges were encountered during our five exemplar overviews; these challenges were clearly aligned with the areas of debate identified within current literature [3], with no areas of dissonance. Areas of silence where a methodological issue was identified by our exemplar overviews and not by the synthesis by Ballard, and vice versa, can arguably be explained by differing levels of reporting within these syntheses. Where there were methods debated within our exemplar overviews which were not discussed in the synthesis by Ballard [3], these were always related to a larger topic or issue for which there was no consensus on an optimal approach, for example dealing with overlap between reviews.

Our exemplar overviews supported the conclusion that overviews can successfully identify gaps in the evidence; this conflicts with Ballard [3] who concludes that overviews cannot fulfil the function of identifying evidence gaps. However, Ballard [3] qualifies this, stating that “overviews that fail to find a systematic review for every relevant comparison will not, by default, detect evidence gaps”. Our experiences suggest that there are situations where evidence gaps will be identified, specifically when there is documented knowledge of current clinical practice or existing interventions. For example, during protocol development McClurg [21] consulted with a stakeholder group comprising expert clinicians and patients, creating a comprehensive list and taxonomy of interventions delivered within clinical practice relevant to the scope of the overview. While Ballard implies that overviews cannot fulfil the function of identifying evidence gaps, we argue that this function can be fulfilled, where there is knowledge of existing interventions or current clinical practice. This supports the viewpoint that the involvement of key stakeholders within the overview process should be an essential component of overview methodology, serving to increase relevance, quality and rigor and reduce research waste, as has been proposed for systematic reviews [26,27,28,29,30].

Our experiences as authors completing overviews, despite an absence of guidance for many methodological features, and the evidence of the increasing number of new overviews published each year [1, 2] highlight that there is a need for recommendations and support for authors wishing to complete overviews during this time of methodological uncertainty. Clearly, the ultimate goal must be to address the uncertainties and establish evidence-based guidance for overview methods, and there is ongoing work which aims to inform and help prioritise future research in this field [15]. However, while there remains a lack of guidance and recommendations for optimal methods, there is currently decision-making required by the authors of future overviews. Until sufficient guidance is available, we recommend that overview authors make and transparently report decisions relating to the inevitable methodological choices. Based on our experiences as overview authors, we have proposed a list of implications to be considered during the development of protocols for future overviews (Table 4). Whilst we anticipate that these implications will be superseded by more robust and evidence-based recommendations as methodological research in this field is completed, we believe that these offer practical advice to those embarking on overviews whilst there remains methodological uncertainty. As this is an active area of research and the methods for overviews continue to develop, overview authors should ensure that they remain up-to-date with any new guidance or information on best practice and should take opportunities to build on the methods of completed overviews. Furthermore, as the methodological features of overviews are broadly derived from, and build upon, methods for systematic reviews of primary research [1, 4, 5], overview authors should utilise guidance and recommendations relating to the conduct of systematic reviews [1]. These include the QUality Of Reporting Of Meta-analyses (QUOROM) statement [31], Methodological Expectations of Cochrane Intervention Reviews guidance (MECIR) [32] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, as well as the growing body of guidance specific to overviews [1, 4, 33, 34]. As well as ensuring the overview is carried out to the highest methodological standards, it is essential to ensure that the overview is relevant and useful, and meaningful involvement of key stakeholders should be central to all overviews (Hunt H, Pollock A, Campbell P, Estcourt L, Brunton G: An introduction to overviews of reviews: planning a relevant research question and objective for an overview Systematic Reviews, submitted); resources currently being developed to support systematic review authors in achieving meaningful stakeholder involvement [35, 36] ought to be relevant to authors of overviews.

Strengths and limitations

The aim of this paper was to provide illustrated examples of the methods and challenges associated with a range of overviews. The objective was not to provide specific recommendations about how to do an overview, or to propose optimal methods for overviews. The five exemplar overviews were initially selected using contacts for overview authors whom attended a meeting on overviews at the 2015 Cochrane Colloquium in Vienna. As the exemplar overviews were initially selected for presentation at the 2016 UK Cochrane meeting, all the selected overviews were led by UK-based authors. Two of the exemplar overviews include some of the same authors [19, 21]. Thus, the overviews provided as examples in this paper are selected from a limited population of authors and cannot therefore be assumed to be representative or comprehensive of the range of different overviews which have been published internationally. None of the five exemplar overviews addresses a research question for which there was high quality trial and review evidence and none of these exemplar overviews completed network meta-analyses; these are clear gaps within these example overviews. However, whilst not purporting to be comprehensive, the examples reflect the shared challenges experienced by the authors of these five selected overviews, and those which were perceived to be the greatest or most difficult to deal with. The comparison of the methodological challenges independently identified from these exemplar overviews with the issues emerging from a comprehensive synthesis of current evidence adds significant strength to this paper, confirming that the experiences within these exemplar overviews are aligned with current evidence.

Conclusions

Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. Furthermore, there are considerable methodological challenges for which optimal solutions are not necessarily yet known. This paper has explored the variations in methodological approaches used within five selected overviews, and the challenges reported by the overview authors. Lessons learnt from these overviews have highlighted a number of methodological decisions which may need to be considered during the development of an overview protocol and led to the development of a list of implications to support the development of protocols for future overviews (Table 4).

While there remains a lack of empirical evidence to support selection of specific methodological approaches [1,2,3, 5, 6], authors planning protocols for Cochrane overviews are encouraged to consider and transparently report their decisions in response to a number of questions, including the following:

  • Is the overview to be limited to Cochrane reviews or will other systematic reviews also be included?

  • Is the search strategy to be limited to databases of reviews or will wider electronic databases be searched?

  • What action will be taken if there are overlapping reviews (reviews containing the same trials)?

  • What action will be taken if included reviews are out of date? (How will ‘out of date’ be defined?)

  • What tool will be used to assess the quality of the included reviews? (AMSTAR/ROBIS)

  • How will the quality of evidence within reviews be assessed? Will this be done using GRADE (and if so, how will GRADE be applied to the available evidence)?

  • Is any new statistical analysis to be carried out, using the data extracted from the reviews?

  • How will the evidence be brought together into an accessible summary which is useful to the potential audience/readership of the overview? What information should be included within this (in order to address the overview objective)?

As this is a new and developing methodological field, it is important that overview authors keep up-to-date with new developments and methods research, in order that their decisions relating to the methodological approach for a new overview are informed by current evidence.