Background

Systematic reviews of interventions are used to inform clinical guidelines [1], to set research priorities [2], and to help patients and clinicians make healthcare decisions. Well-conducted systematic reviews focus on clear research questions and use reproducible methods to identify, select, describe, and synthesize information from relevant studies [3]. Such reviews can reduce uncertainty about the effectiveness of interventions, lead to faster adoption of safe and effective interventions, and identify research needs. On the other hand, poorly-conducted systematic reviews may be harmful if they contribute to suboptimal patient care or promote unnecessary research.

Each year, the number of published systematic reviews continues to increase throughout medicine, yet the quality of systematic reviews is highly variable [2, 4]. To be considered reliable, systematic reviews must be conducted using methods that minimize bias and error in the review process, and must be reported completely and transparently [3].

Because refractive error is the leading cause of visual impairment globally [5], effective interventions to correct refractive error are important to patients and to optometrists and ophthalmologists. The American Academy of Ophthalmology (AAO) updated their Preferred Practice Pattern (PPP) for Refractive Errors & Refractive Surgery in 2017. To assist the AAO in this task, Cochrane Eyes and Vision @ United States (CEV@US) investigators sought to identify reliable systematic reviews about interventions for refractive error using a database of systematic reviews in eyes and vision [6].

Methods

Systematic reviews were eligible for this project if they evaluated the effectiveness or safety of interventions for refractive error that were included in the 2012 AAO’s PPP Refractive Errors & Refractive Surgery [7]. We excluded systematic reviews related to cataract removal with implantation of an intraocular lens for correcting refractive error because this topic is covered in another PPP. We included all reports that claimed to be systematic reviews. Otherwise, we defined a systematic review as “a scientific investigation that focuses on a specific question and uses explicit, pre-specified scientific methods to identify, select, assess, and summarize similar but separate studies [8].” Consistent with this definition, eligible studies were not required to include meta-analyses. Whenever a systematic review had been updated since the initial publication, we reviewed the most recent update.

CEV@US maintains a database of systematic reviews related to vision research and eye care (see Online Additional file 1 for search strategies used to identify reviews). We conducted an initial search of PubMed and Embase in 2007, and we updated the search in 2009, 2012, 2014, and on May 15, 2016 [9]. Two individuals identified systematic reviews related to vision research and eye care. For the 2007, 2009, and 2012 searches, two people identified relevant refractive error systematic reviews independently. For the 2014 and 2016 searches, one person identified potentially eligible refractive error reviews, then a second person verified that the reports were eligible. Differences were resolved through discussion that sometimes included another team member.

We developed a data abstraction form that included components of the Critical Appraisal Skills Programme (CASP) [10], the Assessment of Multiple Systematic Reviews (AMSTAR) [11], and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [12]. We adapted a version of the form used in previous studies [4, 13] (see Online Additional file 2 for a copy of the electronic form). In addition to the results presented in this paper, we recorded other descriptive information about the reviews. We entered data electronically using the Systematic Review Data Repository (SRDR) [14].

Two authors (EMW and SN) abstracted data independently from eligible systematic reviews and resolved discrepancies through discussion. We classified systematic reviews as “reliable” when the systematic reviewers had (1) defined criteria for the selection of studies, (2) conducted comprehensive literature searches for eligible studies, (3) assessed the methodological quality (risk of bias) of the included studies, (4) used appropriate methods for meta-analyses (which we assessed only when meta-analyses were reported), and (5) presented conclusions that were supported by the evidence provided in the review. We considered a systematic review “unreliable” when one or more of these criteria were not met.

Results

Of 124 systematic reviews in our database and classified as related to refractive error, 39 met our eligibility criteria (Fig. 1). Systematic reviews that met our inclusion criteria were published between 1996 and 2014 (median = 2011). They included between 0 and 309 studies (median = 9.5). In 13/39 (33%) systematic reviews, we could not ascertain how many participants contributed data; the 26 remaining reviews included data from 0 to 9336 participants.

Fig. 1
figure 1

Flow chart showing the identification of systematic reviews of interventions for refractive error

Full size image

Of the 39 systematic reviews, 22 (56%) assessed surgical interventions, including reviews of LASEK or LASIK (18; 45%) and intraocular lenses (4; 10%). Other reviews assessed orthokeratology (10; 26%), atropine (2; 5%), monovision (1; 3%), contact lenses (1; 3%), spectacles (1; 3%), and multiple interventions (2; 5%). Participants with myopia were included in 30 (77%) reviews. Reviews also included participants with astigmatism (15; 38%), hyperopia (10; 26%), and presbyopia (3; 8%).

Of the 39 eligible systematic reviews, 11 (28%) were classified as reliable [15,16,17,18,19,20,21,22,23,24,25], and 28 (72%) were classified as unreliable (Fig. 2) [26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53]. One of the reliable reviews did not include any studies. All 11 reliable systematic reviews assessed interventions for myopia, either to slow progression of myopia in children (5 reviews), to correct myopia surgically (5 reviews), or to treat choroidal neovascularization secondary to pathologic myopia (1 review). The 11 reliable systematic reviews were published between 2002 and 2016, included data from 0 to 23 studies (median = 9), and analyzed data for 0 to 4696 participants (median = 666). Of the 11 systematic reliable reviews, 7 (64%) assessed surgical interventions, and included some of the same studies. Different groups of reviewers reached contradictory conclusions about the safety and effectiveness of LASEK compared with PRK (Table 1).

Fig. 2
figure 2

Assessment of reliability criteria for 40 systematic reviews on interventions for refractive error

Full size image
Table 1 Objectives, Participants, Interventions, Outcomes and Conclusions of the Reliable Systematic Reviews on Interventions for Refractive Error and Refractive Surgery (N = 11)
Full size table

The 28 systematic reviews that we classified as unreliable were published between 1996 and 2016 (see Online Additional file 3 for the data extracted for all included systematic reviews) [26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53]. Unreliable systematic reviews did not define the criteria for selecting studies (5; 13%), did not assess methodological quality of the individual studies (10; 26%), did not conduct comprehensive literature searches for eligible studies (17; 44%), or used inappropriate quantitative methods, such as combining randomized and non-randomized studies for meta-analysis (3; 8%). All systematic reviews that presented conclusions not supported by the data were also classified as unreliable for another reason (20; 51%).

Discussion

Our investigation revealed that most systematic reviews about interventions for refractive error published to date have been of low methodological quality; consequently, these reviews may result in inappropriate decisions about clinical care and future research. Some of the shortcomings we identified were related to reporting deficiencies. For example, authors of most reviews did not report criteria for selecting studies. Also, many authors did not report essential characteristics of the included studies, such as sample sizes. Other shortcomings were related to both poor reporting and poor conduct. Our findings are thus consistent with evidence that most systematic reviews published to date have been neither conducted nor reported following best practices [54, 55].

Although visual impairment due to uncorrected refractive error is particularly prevalent in low-income countries [5], most of the systematic reviews included in our study addressed interventions not widely available in those countries. Systematic reviewers may have focused on interventions of interest to decision-makers in high-income countries, and there may be few randomized trials of interventions suitable for use in low-income countries and thus available for inclusion in systematic reviews. Although we classified the systematic review reported by Pearce [29] as unreliable, it is the exception among the 40 systematic reviews for having dealt with an intervention with wide applicability.

A comprehensive and reproducible literature search lays the foundation for a high-quality systematic review. To be transparent and reproducible, systematic reviews should list all information sources searched, the exact search terms used, the operators used to combine terms, and the dates of the searches [56]; however, many systematic reviews of interventions for refractive error did not describe the search methods for identifying eligible studies. Moreover, the number of citations retrieved in many systematic reviews was so small that a knowledgeable reader might suspect the search methods were not sensitive. As a rule of thumb, systematic reviewers can expect to retrieve and review about 200 citations (titles and abstracts) and 5 full-text reports to identify one study eligible for inclusion in a systematic review [57].

Reviewers should describe the methodological quality of included studies using a method such as the Cochrane Risk of Bias Tool to identify potential sources of bias that could influence the credibility of results of individual studies and hence systematic reviews and meta-analysis [58, 59]. Many reviews did not mention having assessed the methodological quality of individual studies, and some reviews specified methods that are known to be unreliable (e.g., the Jadad Scale) [60]. Quality scores are problematic because they are inconsistent and lack validity; the same study may be assessed as excellent using one scoring method but poorly using another scoring method [61].

Finally, authors of seven systematic reviews used methods for data synthesis that were inappropriate. Of particular concern are reviews in which outcomes from randomized controlled trials were combined with outcomes from observational studies in the same meta-analysis. For a research question on the effectiveness of an intervention, randomized controlled trials, by design, have more protection against bias than observational studies. When both types of studies are available, systematic reviewers should justify the reasons for relying on observational data as a substitute for or a complement to data from randomized trials. It is important to understand and to appraise the strengths and weaknesses of observational data when they are used to assess the effectiveness of interventions. When dissimilar studies are combined in meta-analysis, the resulting estimates of intervention effects might be meaningless.

Systematic reviews about interventions for refractive error assessed only a few distinct interventions, a finding consistent with other evidence that systematic reviews are duplicative throughout medicine [62]. After completing our review and sharing the results with the AAO PPP, the panel informed us that one reliable systematic review was not relevant because it examined choroidal neovascularization and laser treatment in high myopia, which is considered a retina topic [24]. Furthermore, some of the reliable reviews included only a few small studies and thus provided inconclusive evidence about the effectiveness and safety of the targeted interventions. Although it is possible for reviews to use pre-specified methods and reach different conclusions (e.g., because of differences in outcomes or inclusion criteria), we find it concerning that some of the reliable reviews reached inconsistent conclusions about the same interventions, even though they included some of the same studies.

We searched for systematic reviews using PubMed and Embase, and we might have missed systematic reviews that are not indexed in these databases. It seems unlikely that systematic reviews in non-indexed journals would be of better quality than the reviews we identified, and their inclusion in our study would have been unlikely to lead us a different conclusion. More high-quality evidence is needed to inform clinical guidelines and to improve patient care for this common problem.

Journal editors could help improve the quality of systematic reviews by adopting four requirements. First, editors should require that authors of systematic reviews publish or provide protocols for their reviews that adhere to current best practices (e.g., the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols (PRISMA-P) statement [63]). Second, editors should require that all reports of systematic reviews include the information in the PRISMA statement [64], which should be documented in a completed PRISMA checklist that accompanies every systematic review manuscript and is confirmed by a peer reviewer or a member of the editorial team. By considering only those manuscripts that adhere to these requirements, editors could ensure that readers have the information needed to assess the methodological rigor of systematic reviews and to decide whether to apply their findings to practice. Third, to reduce the publication of reviews that are essentially redundant (i.e., reviews that include the same studies), editors should require that authors explain how their systematic reviews differ from other reviews of the same interventions or, if the systematic reviews are not different, editors should require that authors reference existing systematic reviews about the same topic. Finally, manuscripts that report systematic reviews should be reviewed by people knowledgeable about both systematic reviews and the clinical specialties that the reviews concern. Eight ophthalmology and optometry journals now have editors for systematic reviews, which may improve the quality of reviews in those journals (http://eyes.cochrane.org/associate-editors-eyes-and-vision-journals).

Conclusions

In conclusion, systematic reviews about interventions for refractive error could benefit patient care and avoid redundant research if the authors of such reviews followed widely accepted guidance, such as Cochrane or Institute of Medicine standards for conducting systematic reviews [3, 65]. The reliable systematic reviews we identified may be useful in updating the AAO PPP and in future updates of the American Optometric Association’s Evidence-Based Clinical Practice Guidelines.