Background

Systematic reviews are a useful tool for decision-makers because they can be used to interpret the results of individual studies within the context of the totality of evidence and provide the evidence-base for knowledge translation products, such as patient decision aids, clinical practice guidelines or policy briefs [1]. However, due to the high level of methodological rigour, systematic reviews take from 0.5 to 2 years to conduct [2] and require considerable skill to execute. According to the Cochrane Collaboration, all procedures including screening citations (titles and abstracts), screening full-text articles, data abstraction, and risk of bias appraisal, should be conducted by two individuals, independently [3]. In addition, technical expertise from librarians, research coordinators, content experts, and statisticians is required.

Health decision-makers (including clinicians, patients, managers, and policy-makers) often need timely access to health information. Although this information can be obtained through a systematic review, these research endeavours require enormous resources to complete and the timeframe required to conduct a systematic review may not suit the needs of some decision-makers. For example, it has been estimated that systematic reviews take, on average, 1,139 hours (range 216–2,518 hours) to complete and usually require a budget of at least $100,000 [4]. Consequently, decision-makers may be forced to rely on less robust evidence, such as expert opinion or the results of a single small study [5], leading to suboptimal decision-making.

Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner [2]. Yet rapid reviews might be susceptible to biased results as a consequence of streamlining the systematic review process [6]. Although numerous rapid review programs exist internationally [7], few studies have examined their methodology. We aimed to examine rapid review approaches, guidance, impact, and comparisons through a scoping review.

Methods

Definition of a rapid review

A formal definition for a rapid review does not exist. As such, we used the following working definition, ‘a rapid review is a type of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a short period of time’ [2].

Protocol

A scoping review protocol was compiled using guidance from Arksey and O’Malley [8], and revised upon feedback received from the Canadian Institutes of Health Research peer review panel. It is available from the corresponding author upon request.

Information sources and literature search

To identify potentially relevant studies for inclusion, the following electronic databases were searched: MEDLINE; EMBASE; and the Cochrane Library. Since two systematic reviews have already been published on rapid reviews [6, 7], we limited our search from 2008 until May 2013. An experienced librarian (LP) drafted the literature searches based on the previous reviews, which was refined through team discussion. The MEDLINE search strategy is presented in Additional file 1: Appendix 1 and the other searches are available from the corresponding author upon request.

Our literature search was supplemented by targeted internet searches for unpublished rapid review reports posted on the websites of producers of rapid reviews. For this search, we took a random 10 % sample of the unpublished rapid reviews available on the producers’ websites. Often only the title was available for the rapid reviews, so, we focused inclusion to the full rapid review, if available. The reference lists of relevant reviews were scanned [6, 7], as were the reference lists of all included rapid reviews.

Inclusion criteria

Articles, papers, books, and reports were included if they evaluated, compared, used or described a rapid review according to the authors.

Screening process

The screening criteria were established a priori (as outlined in our protocol) and calibrated amongst the team through a series of pilot tests. After >90 % agreement was observed, pairs of reviewers screened the literature search results independently, and discrepancies were resolved through discussion. All screening was performed using our online tool, synthesi.sr [9].

Data items and data abstraction process

A data abstraction form was developed a priori and the draft form was calibrated amongst the team using a random sample of ten included studies. After this exercise, the data abstraction form was revised and all included studies were abstracted by two reviewers working independently. Discrepancies were resolved through discussion.

Data items included study characteristics (for example, first author, year of publication), terminology used to describe the rapid review, full citation of previous methods papers that were used to guide the rapid review design, timeframe (in months) for completing the rapid review, and operationalized steps of the rapid review, if reported. The rapid review type was categorized as an application (for example, a rapid review report), development (paper attempts to further refine the rapid review method), impact (examines the impact of rapid reviews) or comparison (compares the results of a rapid review to a systematic review). We abstracted the assessment of the rapid review approach, including accuracy of results, comprehensiveness, potential for risk of bias, timeliness, cost-effectiveness, and feasibility as reported by the publication authors. We also abstracted the skills or knowledge required to conduct the rapid review as reported by the authors.

Synthesis

To synthesize the descriptive results, we conducted qualitative analysis using NVivo 10 [10]. Content analysis was conducted by one team member (WZ) and verified by another team member (ACT) to synthesize common methodologies used across the included rapid reviews using a framework. The framework was developed by the review team and presented in Additional file 1: Appendix 2. The framework focused on the following steps for a rapid review: literature search (number of databases and grey literature); inclusion criteria (limited by date, language, and study design); screening (title/abstract and full-text); data abstraction; risk of bias/quality appraisal; and data synthesis. In order to depict the frequency of the terms used to describe the rapid reviews, a word cloud was created using Wordle, which is software that generates ‘word clouds’ from text that the user provides and places more emphasis on words that occur with greater frequency [11].

Results

Literature search

A total of 3,397 citations and 262 potentially relevant full-text papers were screened. Subsequently, 100 articles [2, 12–110] plus one companion report [111] fulfilled the eligibility criteria and were included [31] (Fig. 1). Forty-seven of the included papers were unpublished rapid reviews posted on websites [13, 24, 29, 31–36, 39, 45, 47, 50, 52–57, 62, 63, 66, 68, 70, 73–75, 77, 81–83, 86–94, 99, 100, 103, 104, 107, 109, 112].

Fig. 1
figure 1

Study flow diagram

Full size image

Rapid review characteristics and assessment

The rapid reviews were published between 1997 and 2013, and 58 were conducted in Europe, while 20 were conducted in North America (Table 1, Additional file 1: Appendix 3). The type of articles included 84 application papers (two did not report any methods), seven development papers, six impact papers, and four comparison papers; one article [20] was categorized in two categories. Ten of the rapid reviews were reported in 5 pages or less, suggesting that they were brief reports or research letters. Most of the articles (73 %) did not report the duration of conduct for the rapid review. For the minority that reported this, the duration ranged from less than 1 month to 12 months, and 18 were between 1 and 6 months. For the application articles, 74 % examined interventions, 12 % charted the frequency of literature (for example, regarding outcomes or frameworks), 5 % examined associations between exposure and disease, 5 % assessed diagnosis or screening techniques, and 2 % examined the patient experience or barriers/facilitators.

Table 1 Summary of study characteristics
Full size table

Sixty-five articles assessed rapid review characteristics (Table 2) [2, 12, 14–22, 24, 26–30, 32, 37–39, 41–43, 45–49, 51–59, 61, 63, 64, 66, 69, 72–76, 78–80, 84, 86, 88–94, 100, 103–105, 110]. Sixty percent of the authors reported that the report was timely, 29 % believed that the method had potential risk of bias, 23 % deemed that the approach was accurate compared to a full systematic review, 8 % believed the approach was comprehensive, 5 % reported that the approach was cost-effective, and 6 % believed it was a feasible approach.

Table 2 Assessing the characteristics of rapid reviews compared to systematic reviews
Full size table

Terminology used to describe the rapid review method

The most frequent term used to describe the rapid review approaches was ‘rapid review’, used in 34 of the included articles (Fig. 2). This was followed by ‘rapid evidence assessment’, which was used in 11 papers, ‘rapid systematic review’ in ten papers, and ‘health technology assessment’ or ‘rapid health technology assessment’ in six papers. All of the other terms occurred two times or less.

Fig. 2
figure 2

Word cloud for the frequency of terms

Full size image

Citation analysis

Twenty-six [2, 12, 13, 17, 20–22, 27, 28, 30, 40, 42–44, 48, 49, 61, 76, 78–80, 84, 88, 103, 105, 110] articles provided citations of previous methods papers that were used to guide the rapid review method (Fig. 3, Additional file 1: Appendix 4). The citations were Ganann and colleagues [6] (cited in eight papers), Watt and colleagues [7, 111] (cited in seven papers), a Civil Service paper [113] (cited in four papers), Ehlers and colleagues [114] (cited in one paper), Armitage and colleagues [14] (cited in one paper), and Grant and colleagues [115] (cited in one paper).

Fig. 3
figure 3

Citation analysis. *Twenty-six papers referenced another seminal paper to establish their rapid review framework

Full size image

Skills and knowledge required to conduct the rapid reviews

Thirteen [16, 32, 39, 42, 46, 48, 49, 52, 79, 84, 88, 90, 94] of the included papers reported the skills and knowledge required to conduct the rapid reviews (Table 3). These were content experts in seven articles [16, 32, 42, 48, 49, 79, 90], information specialists in five articles [39, 49, 52, 84, 88], systematic review methodologists in four papers [16, 42, 48, 79], staff experienced in conducting reviews in four papers [46, 48, 49, 84], and knowledge users in three papers [32, 79, 94].

Table 3 Skills required to conduct a rapid review
Full size table

Operationalized steps to conduct the rapid review applications

The 84 rapid review applications were categorized using our framework (Additional file 1: Appendix 2) and 50 unique methods were observed. Of these, only 16 occurred more than once; three approaches occurred five times [21, 36, 40, 44, 45, 47, 53, 54, 56, 57, 65, 75, 83, 91, 92], another two occurred four times [18, 37, 39, 64, 86, 93, 99, 107], three approaches were used three times [49, 51, 58, 61, 62, 69, 73, 76, 81], and eight approaches occurred two times [14, 16, 20, 25, 27, 30, 31, 66–68, 70, 79, 82, 96, 100, 104]. The characteristics of the rapid review approaches that occurred more than four times were analyzed (Table 4). Rapid Approach 1 had the most details reported, with 5/5 papers mentioning that it was accurate and timely (but did not report the amount of time it took to conduct their rapid review), and had limited comprehensiveness.

Table 4 Evaluation of rapid review approaches occurring more than four times
Full size table

Many of the steps used in the rapid reviews were not fully reported (Table 5, Additional file 1: Appendix 5). For example, 40 % (33/82) did not report whether reference lists were scanned and 67 % (55/82) did not report whether authors were contacted to obtain further material or information.

Table 5 Summary of rapid review streamlined approaches (n = 82 application studies)
Full size table

Streamlined methods that were used in the 82 rapid reviews included limiting the literature search to published literature (24 %) or one database (2 %), limiting inclusion criteria by date (68 %) or language (49 %), having one person screen and another verify or screen excluded studies (6 %), having one person abstract data and another verify (23 %), not conducting risk of bias/quality appraisal (7 %) or having only one reviewer conduct the quality appraisal (7 %), and presenting results as a narrative summary (78 %) (Fig. 4).

Fig. 4
figure 4

Streamlined steps used across the rapid reviews (n = 82 studies reporting this information). SR, systematic review

Full size image

Comparing results from rapid reviews to systematic reviews

Four studies were comparisons, providing details on differences in results between rapid reviews and systematic reviews [20, 31, 34, 106]. Cameron and colleagues identified rapid reviews from health technology assessment (HTA) organization websites and then conducted a literature search to identify systematic reviews on the same topic [31]. Eight rapid review products were identified on four different topics. However, the authors did not appraise the methodological quality of the systematic reviews, so it is unclear whether shortcuts were also taken in the included systematic reviews. The authors noted that the conclusions did not differ substantially between the rapid and systematic reviews. Corabian and colleagues compared six rapid review products (called ‘technotes’) with their final peer-reviewed publications [34]. The authors found that the conclusions differed only in 1/6 cases. Van de Velde and colleagues compared the results from their rapid review to a systematic review that was conducted by another group and published on the same topic [106]. Despite having literature searches that were conducted for the same dates, conflicting results were observed; the rapid review concluded that potato peel was effective for burns, while the systematic review concluded that potato peel was not effective for treating burns. Finally, Best and colleagues noted that two of the rapid reviews they conducted were in agreement with systematic reviews published at a later point in time on the same topic [20].

Development papers on rapid reviews

Seven papers proposed methods to refine the rapid review approach [2, 12, 16, 20, 46, 79, 80]. Best and colleagues (1997) described their experience conducting 63 rapid reviews for decision-making beginning in 1991, through the Development and Evaluation Committee in the UK [20]. Abrami and colleagues (2010) described ways to produce brief reviews efficiently, and presented a checklist for the conduct and reporting of brief reviews [12]. Bambra and colleagues (2010) described their experience conducting nine rapid reviews for the Secretary of State for Health [16]. Jahangirian and colleagues (2011) described their experience conducting five rapid reviews for the Research into Global Healthcare Tools consortium and proposed a framework for the conduct of rapid reviews [46]. Khangura and colleagues (2012) described their approach to the conduct of 11 rapid reviews through the collaboration between the Ottawa Hospital Research Institute and the Champlain Local Health Integrated Network [2]. Thigpen and colleagues (2012) described their experience conducting rapid reviews using the 6-step Prevention Synthesis and Translation System process for the Division of Violence Prevention, National Center for Injury Prevention and Control at the Centers for Disease Control and Prevention [79]. Thomas and colleagues (2013) described their experience of conducting two rapid reviews for the UK Treasury to inform the 2006/07 Comprehensive Spending Review [80].

Guidance to streamline the rapid review process varied, yet some consistencies were observed (Table 6). For example, four papers suggested using integrated knowledge translation, in which researchers work closely with the knowledge users to complete the rapid review [2, 16, 19, 79]. Four papers suggested the use of a research question with a limited scope [12, 16, 80, 110]. Seven publications recommended streamlining the literature search [2, 12, 16, 46, 79, 80, 110] and three suggested restricting the eligibility criteria [2, 12, 80]. Two papers provided suggestions for efficiently appraising risk of bias [2, 80] and none suggested conducting a meta-analysis as part of the rapid review.

Table 6 Guidance provided in development papers on rapid reviews
Full size table

Articles assessing the impact and use of rapid reviews

Six papers examined the impact of rapid reviews on decision-making [41–43, 60, 85, 110]. Hailey and colleagues (2000) examined the impact of 20 rapid review products [43] and found that 14 had an influence on policy decision-making, four provided guidance, and two had no perceived impact. McGregor and Brophy (2005) evaluated the success of the conduct of 16 rapid reviews for a hospital rapid review service [60]. The results of all 16 products were directly implemented in the hospital, saving approximately $3 million per year. Hailey (2006) wrote a paper summarizing the impact of HTA in general, as well as related to rapid HTA. Overall, it was concluded that these reports can influence decision-making. Hailey (2009) conducted a survey of HTA organizations to examine the use of rapid reviews for decision-making [42]. Fifteen rapid review products were included; all influenced a decision, including using the rapid review for reference material (67 %) and directly using the rapid review’s conclusions for the decision (53 %). Zechmeister (2012) examined the impact of 58 rapid assessments and observed that 56 of these products were directly used for reimbursement decisions and two were used for disinvestment decisions [85]. Finally, Batten (2012) wrote an editorial discussing how rapid reviews can be used by school nurses [110].

Discussion

Our results suggest that the conduct of rapid reviews is recondite across the literature. Through our study, 50 different rapid review approaches were identified and only 16 occurred more than once. Furthermore, many different terms were used to describe a rapid review, making the identification of these types of knowledge synthesis products difficult.

Using a framework of rapid review methods, we observed numerous strategies employed to conduct reviews in a streamlined manner. These included not using a protocol, limiting the literature search, limiting inclusion criteria, only having one person screen the literature search results, not conducting quality appraisal, and not conducting a meta-analysis. In general, combining multiple shortcuts led to a timelier conduct of the review.

Only four of the included studies compared the results of rapid reviews to systematic reviews. Three of these found that the results for both knowledge synthesis products were in agreement. However, the results of these studies should be interpreted with caution because a very small sample of reviews were included (ranging from 1 to 8) and none of these were prospectively conducted. The latter is of particular importance, since it is unclear whether the authors of the full systematic reviews used the rapid review as a starting point to identify articles for inclusion (or vice versa). Interestingly, none of the included studies compared the results across rapid reviews on the same topic. Such a study may provide further clarity into the impact of streamlining different steps on the risk of bias and comprehensiveness of the review.

Seven papers provided recommendations on making rapid reviews more efficient. Consistent guidance included using an integrated knowledge translation approach, limiting the scope of the question and literature search, and not conducting a meta-analysis. Furthermore, six papers examined the impact of rapid reviews on decision-making and all found that they were valuable products. These results suggest that decision-makers are currently using rapid reviews to inform their decision-making processes. Further supporting this observation was the recent Canadian Agency for Drugs and Technologies in Health Rapid review summit [116], for which a large number of international decision-making organizations were in attendance.

Across the application papers, many of the methods were poorly reported suggesting that improvement in the reporting of rapid reviews is warranted. Thorough reporting of the methods is important because it is difficult to judge the bias of these reports without fully understanding what shortcuts were taken. As well, transparent reporting allows the reproducibility of research. It is important to note that 10 % of the included papers were reported in 5 pages or less, suggesting that perhaps there was insufficient room to report the methods fully.

Prior to establishing a quality of reporting guidelines for rapid reviews, a common terminology and definition is required [117]. Some of the team members are currently involved with research that is attempting to tackle this issue. At the bare minimum, one of the included papers provided a checklist to examine the reporting of rapid reviews [12], which can be used by producers of rapid reviews to ensure their reports are reported in a consistent manner.

We have also conducted other research on rapid reviews that builds on this scoping review [118]. Specifically, we conducted an international survey of 40 rapid review producers who identified several rapid review approaches, such as updating the literature search of previous reviews and limiting the search strategy by date of publication. Most of the rapid review products were conducted within 12 weeks. A modified Delphi approach was used to include input from 113 stakeholders (for example, researchers, policy-makers, industry, journal editors, and healthcare providers) to agree upon an attractive rapid review method that would be used in a future comparative study. The stakeholders ranked the following method as being the most feasible, timely, and having a low perceived risk of bias: literature search limited by date and language; study selection by one reviewer only; and data abstraction and quality appraisal conducted by one reviewer and verified by a second reviewer. We are currently in the process of seeking funding of a comparative study to test the accuracy of this rapid review approach versus the gold standard, systematic review.

A recent project on rapid reviews was commissioned by the Agency for Healthcare Research and Quality in the United States [119, 120]. The authors summarized evidence from 12 review articles of rapid reviews [120], as well as 35 different rapid reviews produced by 20 different organizations [119]. This information was obtained through literature searches and key informant interviews with 18 individuals who had experience of conducting rapid reviews. The authors are currently conducting interviews with policy-makers to obtain their perceptions on rapid reviews, including their utility and importance.

Our scoping review has some limitations. To make our review more feasible, we were only able to include a random sample of rapid reviews from websites of rapid review producers. Further adding to this issue is that many rapid reviews contain proprietary information and are not publicly available. As such, our results are only likely generalizable to rapid reviews that are publicly available. Furthermore, this scoping review was an enormous undertaking and our results are only up to date as of May 2013. However, we believe that our results provide important information on rapid reviews and ours is the most comprehensive scoping review that we are currently aware of.

Conclusions

In conclusion, numerous rapid review approaches were identified and few were used consistently in the literature. Poor quality of reporting was observed. Further research on rapid reviews is warranted. In particular, the consequences of various methodological shortcuts should be investigated. This could be examined through a prospective study comparing the results of rapid reviews to those obtained through systematic reviews on the same topic. Team members are currently seeking funding to conduct such a study and it is hoped that our results will provide pertinent information on the utility and risk of bias of rapid reviews.