Background

Overviews of systematic reviews (or umbrella reviews) attempt to systematically retrieve and summarise the results of multiple systematic reviews into a single document [1]. The number of published overviews of systematic reviews (henceforth termed overviews) has increased steadily in recent years, in part due to the proliferation of systematic reviews [2]. Methods for conducting, interpreting and reporting overviews are in their infancy [2]. To date, there has been no systematic review or evidence map examining the range of methods for overviews (particularly those which are unique to overviews) nor of the evidence for using these methods.

In general, the steps for undertaking an overview mirror those of a systematic review, with many of the methods used in systematic reviews being directly transferrable to overviews (e.g. independent study selection and data extraction) [3]. However, there are unique features of overviews that require the use of different or additional methods, for example, methods for assessing the quality or the risk of bias in systematic reviews; dealing with the inclusion of the same trial in multiple systematic reviews; dealing with out-of-date systematic reviews; and dealing with discordant results across systematic reviews [2].

Evidence maps provide a systematic method for mapping the evidence on a particular topic, with the resulting map facilitating identification of gaps in the literature [4, 5]. Bragge [6] defines evidence mapping as describing the yield, design and characteristics of research in broad topic areas, in contrast to systematic reviews, which usually address narrowly focused research questions. Evidence mapping has been primarily used to map the evidence for healthcare interventions [7, 8]; however, the approach may also be usefully applied for mapping the evidence on other topics. For example, evidence mapping may be useful for collating evidence on the range and performance of research methods. To our knowledge, evidence mapping has yet to be applied in this way [5].

Where possible, methodological guidance for conducting overviews should be based on empirical research; that is, based on methods that have been evaluated and shown to have better performance [9]. It is therefore timely to systematically review and map the available methods literature to determine where there are gaps or areas of uncertainty and hence what methods research should be undertaken as a priority. We will apply the evidence mapping approach to studies evaluating the methods for conducting, interpreting and reporting overviews of systematic reviews of interventions.

The objectives of this study are to develop and populate a framework of methods that have been used, or may be used, in conducting, interpreting and reporting overviews of systematic reviews of interventions (stage I); (2) create an evidence map of studies that have evaluated these methods (stage II); and (3) identify and describe unique methodological challenges of overviews.

Methods/design

The methods used in this study are based on published methods for mapping the evidence in broad content areas [4, 6, 10–13]. Methods for developing an evidence map involve several steps, namely, defining a topic and specific questions to be answered by the evidence map; searching for and selecting relevant studies; and reporting on the yield and study characteristics.

Our mapping study will involve two stages. In stage I, we will use the following sources to identify and describe methods that have been used or recommended for use in overviews: articles describing methods for overviews, methods studies that have described the characteristics of a cross section or cohort of overviews; guidance documents for conducting overviews; and commentaries or editorials. The identified methods will populate a framework of methods relating to the steps involved in conducting an overview (e.g. defining objectives, searching for systematic reviews, selecting studies). The second stage will involve the development of an evidence map of studies that have evaluated any of the identified methods (stage II). The steps involved in these stages are depicted in Fig. 1. The evidence map will facilitate identification of methods that have had little or no evaluation.

Fig. 1
figure 1

Stages in the development of an evidence map of overview methods

Full size image

We have used the Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocols (PRISMA-P) [14] to develop this protocol. The PRISMA-P checklist was developed for the preparation of protocols of systematic reviews that summarise aggregate data from studies, particularly the evaluations of the effects of interventions, and therefore, not all PRISMA-P items were applicable (Additional file 1). We did not register our evidence map on PROSPERO, the international prospective register of systematic reviews (http://www.crd.york.ac.uk/PROSPERO/), since reviews of ‘methodological issues’ do not met the inclusion criteria for this registry (as specified 29 June 2015).

Search methods

Several sources will be searched to identify available methods (stage I) and studies that have evaluated the performance of those methods (stage II). We will undertake searches of the following methods collections: Cochrane Methodology Register, Meth4ReSyn library, the Scientific Resource Center Methods library of the AHRQ Effective Health Care Program, and Cochrane Colloquium abstracts. In addition, we will undertake a search of Ovid MEDLINE from 2000 onwards. We will use variations of the following search terms: overview, umbrella reviews, overviews of reviews, overviews of systematic reviews, meta-review or review of reviews (Additional file 2).

We will conduct purposive searches to locate evaluations of the methods we have identified, where the above searches are unlikely to have located these evaluations. For example, although searching for systematic reviews is a fundamental step of overviews, there are other purposes aside from overviews (e.g. developing guidelines) for which retrieving systematic reviews is necessary. Therefore, papers describing the development and evaluation of search strategies for systematic reviews may reasonably not have mentioned ‘overviews’ (or its synonyms) and thus would not be identified in the searches we describe above. Details of these additional search strategies will be documented and reported.

Given the potential difficulties in locating methods literature on overviews, we will supplement our searches by examining the reference lists of all included articles and undertaking citation searches of seminal papers using Google Scholar, Scopus and Web of Science. All three of these databases will be searched for citations because searches run in each database have been shown to return unique material [15]. In addition, as part of a related research project to develop a search strategy to identify overviews in MEDLINE, we set aside any methods papers that we identified through screening citations [16]. Finally, we will contact relevant experts in the field to help identify additional methods papers.

Eligibility criteria

Separate eligibility criteria will be applied for stages I and II of the development of the evidence map (Fig. 1).

Stage I: populating the framework—identification of methods used in overviews of systematic reviews of interventions

Inclusion criteria:

  1. i)

    Articles describing methods for overviews

  2. ii)

    Studies examining methods used in a cross section or cohort of overviews

  3. iii)

    Guidance (e.g. handbooks and guidelines) for undertaking overviews

  4. iv)

    Commentaries or editorials that discuss methods for overviews

Exclusion criteria:

  1. i)

    Articles published in languages other than English

  2. ii)

    Studies describing methods for network meta-analysis

  3. iii)

    Articles exclusively about methods for overviews of other review types (i.e. not of interventions)

If necessary, the framework (stage I) will be further refined to include methods identified from studies in stage II, if these have not already been identified in stage I.

Stage II: populating the evidence map—identification of evaluations of methods for overviews of systematic reviews of interventions

Inclusion criteria:

  1. i)

    Systematic reviews of methods studies that have evaluated methods for overviews

  2. ii)

    Methods studies that have evaluated methods for overviews

Exclusion criteria:

  1. i)

    Articles published in languages other than English

  2. ii)

    Methods studies that have evaluated methods for network meta-analysis

In cases where we identify systematic reviews in which the search for studies was conducted before 2013, we will search for more recent methods studies and include the results of these in the evidence map, in addition to including the results of the systematic review.

We plan to populate the evidence map with evaluations of methods that are different or additional to those required to undertake a systematic review of primary research. For example, methods for screening articles for inclusion in an overview (e.g. whether one or more reviewers are required) do not differ to those used in undertaking a systematic review of primary research, so would not be included. However, evaluations of methods used to assess the quality or risk of bias of systematic reviews are of relevance to overviews, and not systematic reviews of primary research, and so would be included. It may be the case that methods have been evaluated in the context of other ‘overview’ products, such as guidelines, and if these methods are of relevance to overviews, they will be included.

The eligibility criteria will be pilot tested by three independent reviewers on a sample of articles retrieved from the search to ensure consistent application.

Study selection

Citations retrieved from the searches will be imported into EndNote. Duplicates will be identified and removed. Two reviewers will independently review titles and abstracts for their potential inclusion against the selection criteria. Full-text articles will be retrieved when both reviewers agree that inclusion criteria have been met or when there is uncertainty. Two reviewers will then independently assess the articles for inclusion; any disagreement will be resolved by discussion or by arbitration of a third reviewer. In instances where there is limited or incomplete information regarding a study’s eligibility (e.g. when only an abstract is available), the authors will be contacted to request the full text or further details.

Assessment of the risk of bias of evaluation studies (stage II)

There is no tool available to assess the risk of bias of studies that have evaluated methods (i.e. studies meeting the criteria for stage II). We will therefore report characteristics of the evaluation studies that may plausibly be associated with bias and have been used in other methodology reviews (e.g. [17–19]). These characteristics will include study design, method to select the cohort of studies included in the evaluation, process used to extract data and existence of a protocol. For systematic reviews of methods studies, we will use a tool to assess the risk of bias in systematic reviews (ROBIS) [20].

Data extraction, data coding and analysis

Developing the coding framework

The coding framework will be developed using an iterative process. Two reviewers (CL, SB) will independently code three articles using a simple framework comprised of codes representing the main steps in conducting an overview as outlined in the Cochrane Handbook (e.g. specifying eligibility criteria, data extraction) [21]. Additional codes will be generated inductively by the two reviewers from the text of the three articles. The independently developed frameworks will then be compared and combined into a single integrated framework. The resulting framework will be reviewed by all authors, then tested on three further articles purposively selected to reflect the diversity of article types (i.e. study examining methods in a cohort of overviews, guidance, editorial) included at stage I. Further refinements will be discussed and agreed by all authors. The preliminary framework of parent and illustrative child codes, with descriptions for interpreting each code, is presented in Table 1.

Table 1 Coding framework for methods used in overviews of systematic reviews of interventions (stage I)
Full size table

Stage I: populating the framework

Data will be coded from articles meeting the inclusion criteria of stage I using NVivo software. Coded data will be analysed thematically to further refine and populate the framework with descriptions of each method. This thematic analysis will be used to (1) determine the most appropriate categorisation of methods (i.e. the final structure and terminology used in the framework); (2) determine whether or not methods are distinct (i.e. identifying whether methods described using different terminology are indeed different and vice versa); and (3) write a description of each method by synthesising key concepts across included studies. In addition, data will be coded to identify the type of paper (article describing method for overviews, cohort/cross-sectional; commentary or editorial; guidance document); noted advantages/disadvantages of the method; examples of use; at which step in the overview process the method is used; and unique methodological considerations in conducting overviews.

Stage II: populating the evidence map

We plan to extract the following data from methods evaluation studies using a data extraction form: characteristics of the article (e.g. publication year, journal); characteristics of the study (study design, type of sample selection (e.g. random, consecutive, others), process to extract data, existence of a protocol); primary objective(s); secondary objective(s); methods evaluated for primary objective; evaluation analysis methods; and the quantitative results relating to the primary objective.

Coding and data extraction for both stages of the study will be undertaken independently by two reviewers. We anticipate that it will be difficult to pre-specify response categories for each data extraction item due to the potential variability in the type of evaluations. Therefore, much of the extracted data will be verbatim free text, which will be categorised following discussion between the reviewers. Discrepancies in extracted data will be discussed between the reviewers until consensus is reached or by arbitration of a third reviewer if required.

Data analysis

We will describe the evaluations that have been undertaken and map these evaluations to the framework of methods identified in stage I. The description of evaluations will include the yield and characteristics of the available evidence, the use, advantages and disadvantages of each method and a summary of the findings of the evaluations.

Discussion

Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no systematic review or evidence map examining either the range of methods for overviews or the evidence behind those methods. We plan to use evidence mapping methods to develop and populate a framework of methods used in conducting, interpreting and reporting overviews of systematic reviews of interventions (stage I) and create an evidence map of studies that have evaluated these methods (stage II). This will be a novel application of the evidence mapping methodology in a context where, to our knowledge, this method has not been used.

The results of the evidence map may influence the use of particular methods in overviews, either directly or by influencing guidance for overview methods. However, perhaps the most notable use of the evidence map will be in directing methods research to areas where there is limited evidence for the methods that are being used. Such a map may also be useful for national funding agencies in considering what methods research should be funded.

Strengths and limitations

We have developed a protocol to guide our research and reduce post hoc decision making. Two reviewers will undertake the screening, data coding and data extraction of all methods studies. Searching for methodological papers can prove very difficult in databases other than specialist methodology collections, such as the Cochrane Methodology Register and the Meth4ReSyn library [22]. In addition, while the coverage of two of the specialist methodology collections extends beyond the health and medical literature (Meth4ReSyn and Scientific Resource Center Methods library), the focus of our search is within health. These factors might mean that some relevant methodology papers are missed. However, we have included reference checking, citation searching and contacting relevant experts in the field to minimise the impact of these limitations.

Research status

At the time of submitting this protocol, we have undertaken the searching and screening for stage I. Two reviewers have independently coded three methods papers for stage I and developed an initial framework (Table 1). During the peer-review process, we amended the inclusion criteria to clarify that our focus was on methods for overviews of systematic reviews of interventions. We also revised our search strategies in response to suggestions from the reviewer.

Summary

Our results will lead to an inventory of evaluation studies of methods for overviews of systematic reviews of interventions. The evidence map will aid in the development and implementation of methods for overviews which will be relevant to a wide range of knowledge users, including researchers, funders and journal editors. The evidence map will help to prioritise the future research agenda in this field.