Abstract
Systematic reviews are the most reliable and comprehensive statement about what works. They focus on a specific question and use clearly stated, prespecified scientific methods to identify, select, assess, and summarise the findings of similar but separate studies. A systematic review may or may not contain a meta-analysis for various reasons. Given the hierarchy of evidence-based medicine, a systematic review and meta-analysis are expected to provide robust evidence to guide clinical practice and research. However, the methodological rigour (design, conduct, analysis, interpretation, and reporting) of both, the systematic review and meta-analysis and the included studies deserve equal attention for judging the validity of the findings of a systematic review. Reproducibility is a critical aspect of science. Without transparency about what was done, and how it was done, it is difficult to reproduce the results, questioning the validity of any study. This chapter focuses on the critical appraisal of a systematic review and meta-analysis based on their principles and practice.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baker R, Jackson D. A new approach to outliers in meta-analysis. Health Care Manag Sci. 2008;11:121–31.
Balasubramanian H, Ananthan A, Rao S, Patole S. Odds ratio vs risk ratio in randomised controlled trials. Postgrad Med. 2015;127:359–67.
Bashir R, Surian D, Gunn AG. Time-to-update of systematic reviews relative to the availability of new evidence. Syst Rev. 2018;7:195. https://doi.org/10.1186/s13643-018-0856-9.
Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Med. 2010;7: https://doi.org/10.1371/journal.pmed.1000326.
Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, Thorlund K. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol. 2008;61:857–65.
Borenstein M, Hedges LV, Higggins JPT, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1:97–111.
Bowden J, Tierney JF, Copas AJ, et al. Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics. BMC Med Res Methodol. 2011;11:41. https://doi.org/10.1186/1471-2288-11-41.
Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017,6: Article 245. https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-017-0644-y.
Brown PA, Harniss MK, Schomer KG, Feinberg M, Cullen NK, Johnson KL. Conducting systematic evidence reviews: core concepts and lessons learned. Arch Phys Med Rehabil. 2012;93:S177–84.
Chess LE, Gagnier JJ. Applicable or non-applicable: investigations of clinical heterogeneity in systematic reviews. BMC Med Res Methodol. 2016;17(16):19. https://doi.org/10.1186/s12874-016-0121-7.
Clarke M, Hopewell S, Chalmers I. Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet. 2010;376:20–1.
Cooper NJ, Jones DR, Sutton AJ. The use of systematic reviews when designing studies. Clin Trials. 2005;2:260–4.
Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol. 2018;18:85. https://doi.org/10.1186/s12874-018-0545-3.
Coulson M, Healey M, Fidler F, Cumming G. Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Front Psychol. 2010;1:26. https://doi.org/10.3389/fpsyg.2010.00026. eCollection 2010.
da Costa BR, Juni P. Systematic reviews and meta-analyses of randomised trials: principles and pitfalls. Eur Heart J. 2014;14(35):3336–45.
Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, 2019. www.training.cochrane.org/handbook.
Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385–9.
EBM notebook: Weighted event rates. Werre SR, Walter-Dilks C. BMJ Evid Based Med. 2005;10:70. http://dx.doi.org/10.1136/ebm.10.3.70.
Ebrahim S. The use of numbers needed to treat derived from systematic reviews and meta-analysis: caveats and pitfalls. Eval Health Prof. 2001;24:152–64.
Egger M, Smith GD, Phillips AN. Meta-analysis: principles and procedures. BMJ. 1997;315(7121):1533–7.
Esteves SC, Majzoub A, Agarwal A. The problem of mixing ‘apples and oranges’ in meta-analytic studies. Transl Androl Urol. 2017;6:S412–3. https://doi.org/10.21037/tau.2017.03.23.
Evangelou E, Ioanidis JPA, Patsopoulos NA. Uncertainty in Heterogeneity Estimates in Meta-Analyses. BMJ. 2007;335:914–6.
Fuhr U, Hellmich M. Channelling the flood of meta-analyses. Eur J Clin Pharmacol. 2015;71:645–7.
Gagnier JJ, Moher D, Boon H, Beyene J, Bombardier C. Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature. BMC Med Res Methodol. 2012;30(12):111. https://doi.org/10.1186/1471-2288-12-111.
Garner P, Hopewell S, Chandler J, et al. When and how to update systematic reviews: consensus and checklist. BMJ. 2016;354: https://doi.org/10.1136/bmj.i3507.
Gibbs NM, Gibbs SV. Misuse of ‘trend’ to describe ‘almost significant’ differences in anaesthesia research. Br J Anaesth. 2015;115:337–9.
Glasziou PP, Shepperd S, Brassey J. Can we rely on the best trial? A comparison of individual trials and systematic reviews. BMC Med Res Methodol. 2010;18(10):23. https://doi.org/10.1186/1471-2288-10-23.
Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence and imprecision. J Clin Epidemiol 2011; 64: 1283e–1293.
Haddaway NR, Rytwinski T. Meta-analysis is not an exact science: Call for guidance on quantitative synthesis decisions. Environ Int. 2018;114:357–9.
Higgins J, Thompson S, Deeks JJ, Altman D. Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice. J Health Service Res Policy. 2002; 7:51–61.
Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;15(21):1539–58.
Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5:13.
Huang HY, Andrews E, Jones J, Skovron ML, Tilson H. Pitfalls in meta-analyses on adverse events reported from clinical trials. Pharmacoepidemiol Drug Saf. 2011;20:1014–20.
Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol Methods. 2006;11:193–206.
Hunter JE, Schmidt FL. Fixed effects vs. random effects meta-analysis models: Implications for cumulative research knowledge. Int J Sel Assess. 2000; 8: 275–292.
IntHout J, Ioannidis JP, Borm GF, Goeman JJ. Small studies are more heterogeneous than large ones: a meta-meta-analysis. J Clin Epidemiol. 2015;68:860–9.
Ioannidis JP. Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract. 2008;14:951–7.
Ioannidis JA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94:485–514. https://doi.org/10.1111/1468-0009.12210.
Ioannidis JP, Lau J. Completeness of safety reporting in randomised trials: an evaluation of 7 medical areas. JAMA. 2001;285:437–43.
Ioannidis JP, Cappelleri JC, Lau J. Issues in comparisons between meta-analyses and large trials. JAMA. 1998;8(279):1089–93.
Jackson D, Turner R. Power analysis for random-effects meta-analysis. Res Synth Methods. 2017;8:290–302.
Jeong W, Keighley C, Wolfe R, et al. The epidemiology and clinical manifestations of mucormycosis: a systematic review and meta-analysis of case reports. Syst Rev. 2019;25(1):26–34. https://doi.org/10.1016/j.cmi.2018.07.011.
Jin ZC, Zhou XH, He J. Statistical methods for dealing with publication bias in meta-analysis. Stat Med. 2015;34:343–60.
Jones JB, Blecker S, Shah NR. Meta-analysis 101: what you want to know in the era of comparative effectiveness. Am Health Drug Benefits. 2008;1:38–43.
Jørgensen AW, Maric KL, Tendal B, Faurschou A, Gøtzsche PC. Industry-supported meta-analyses compared with meta-analyses with non-profit or no support: differences in methodological quality and conclusions. BMC Med Res Methodol. 2008;9(8):60. https://doi.org/10.1186/1471-2288-8-60.
Kriston L. Dealing with clinical heterogeneity in meta-analysis. Assumptions, methods, interpretation. Int J Methods Psychiatr Res. 2013;22:1–15.
Lakens D, Hilgard J, Staaks J. On the reproducibility of meta-analyses: six practical recommendations. BMC Psychol. 2016;31(4):24. https://doi.org/10.1186/s40359-016-0126-3.
Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597–600.
Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ. 2001;322:1479–80.
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009; 62: e1–e34.
Liberati A. Meta-analysis: statistical alchemy for the 21st century: discussion. A plea for a more balanced view of meta-analysis and systematic overviews of the effect of health care interventions. J Clin Epidemiol. 1995;48:81–6.
Mahtani KR. All health researchers should begin their training by preparing at least one systematic review. J R Soc Med. 2016;109:264–8.
Malone DC, Hines LE, Graff JS. The good, the bad, and the different: a primer on aspects of heterogeneity of treatment effects. J Manage Care Special Pharm. 2014;20:555–63.
Melson WG, Bootsma MCJ, Rovers MM, Bonten MJM. The effects of clinical and statistical heterogeneity on the predictive values of results from meta-analyses. Clin Microbiol Infec. 2014;20:123–9.
Mohan BP, Adler DG. Heterogeneity in systematic review and meta-analysis: how to read between the numbers. Gastrointest Endosc. 2019;89:902–3.
Nikolakopoulou A, Mavridis D, Salanti G. How to interpret meta-analysis models: fixed effect and random effects meta-analyses. Evid Based Mental Health. 2014. https://doi.org/10.1136/eb-2014-101794.
Phan K, Tian DH, Cao C, Black D, Yan TD. Systematic review and meta-analysis: techniques and a guide for the academic surgeon. Ann Cardiothorac Surg. 2015;4:112–22.
Purgato M, Adams CE. Heterogeneity: the issue of apples, oranges and fruit pie. Epidemiol Psychiatr Sci. 2012;21:27–9.
Ranganathan P, Pramesh CS, Buyse M. Common pitfalls in statistical analysis: clinical versus statistical significance. Perspect Clin Res. 2015;6:169–70. https://doi.org/10.4103/2229-3485.159943.
Richardson M, Garner P, Donegan S. Interpretation of subgroup analyses in systematic reviews: a tutorial. Clin Epidemiol Global Health. 2018 (Published: May 28, 2018) https://doi.org/10.1016/j.cegh.2018.05.005.
Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995; 123:A12–3.
Roever L, Zoccai GB. Critical appraisal of systematic reviews and meta-analyses. Evid Based Med Pract. 2015;1:1. https://doi.org/10.4172/EBMP.1000e106.
Rücker G, Schwarzer G, Carpenter JR, et al. Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008;8:79.
Sanchez-Meca J, Marin-Martinez F. Confidence intervals for the overall effect size in random-effects meta-analysis. Psychol Methods. 2008;13:31–48.
Schmidt FL, Oh IS, Hayes TL. Fixed-versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. Br J Math Stat Psychol. 2009;62:97–128.
Scifres CM, Iams JD, Klebanoff M, Macones GA. Meta-analysis vs large clinical trials: which should guide our management? Am J Obstet Gynecol. 2009;200:484.e1–4845.
Sedgwick P. How to read a funnel plot in a meta-analysis. BMJ. 2015;351: https://doi.org/10.1136/bmj.h4718 (Published 16 September 2015).
Shokraneh F. Reproducibility and replicability of systematic reviews. World J Meta-Anal. 2019; 7(3): 66–71. https://dx.doi.org/10.13105/wjma.v7.i3.66.
Shrier I, Boivin JF, Platt RW, et al. The interpretation of systematic reviews with meta-analyses: an objective or subjective process? BMC Med Inform Decis Making. 2008;19. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472–6947-8-19.
Shuster JJ. Empirical vs natural weighting in random effects meta-analysis. Stat Med. 2010;30(29):1259–65.
Simonsohn U, Nelson LD, Simmons JP. P-curve: A key to the file-drawer. J Exp Psychol Gen. 2014;143:534–47.
Smith TO, Hing CB. “Garbage in, garbage out”- the importance of detailing methodological reasoning in orthopaedic meta-analysis. Int Orthop. 2011;35:301–2.
Stanley TD, Doucouliagos H. Neither fixed nor random: weighted least squares meta-analysis. Stat Med. 2015;15(34):2116–27.
Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;22(343): https://doi.org/10.1136/bmj.d4002.
Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;12(355): https://doi.org/10.1136/bmj.i4919.
Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting: Meta-analysis of observational studies in epidemiology (MOOSE) group. JAMA. 2000; 283: 2008–2012.
Subgroups in meta-analysis–Section 3.3.7: JBI Reviewer’s Manual–JBI GLOBAL WIKI https://wiki.joannabriggs.org/display/MANUAL/3.3.7+Subgroups+in+meta-analysis.
Thompson SG, Pocock SJ. Can meta-analyses be trusted? Lancet. 1991;2(338):1127–30.
Tricco AC, Straus SE, Moher D. How can we improve the interpretation of systematic reviews? BMC Med. 2011: 31. https://bmcmedicine.biomedcentral.com/articles/10.1186/1741-7015-9-31.
van der Knaap LM, Leeuw FL, Bogaerts S, Nijssen LTJ. Combining Campbell standard and the realist evaluation approach: the best of two worlds? Am J Eval. 2008;29:48–57.
van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(838–44): https://doi.org/10.1016/j.jclinepi.2008.09.010.
Viechtbauer W. Confidence intervals for the amount of heterogeneity in meta-analysis. Stat Med. 2007;26:37–52.
von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344–9.
von Hippel PT. The heterogeneity statistic I(2) can be biased in small meta-analyses. BMC Med Res Methodol. 2015;14(15):35.
Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of non-randomised studies in meta-analyses. The Ottawa Hospital Research Institute; 2000. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.
Xu C, Li L, Lin L, et al. Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions. J Clin Epidemiol. 2020; 123: 91–99.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Patole, S. (2021). Critical Appraisal of Systematic Reviews and Meta-Analyses. In: Patole, S. (eds) Principles and Practice of Systematic Reviews and Meta-Analysis. Springer, Cham. https://doi.org/10.1007/978-3-030-71921-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-71921-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71920-3
Online ISBN: 978-3-030-71921-0
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)