Skip to main content
SpringerLink
Log in

Certainty of Evidence Assessment in Systematic Reviews Published by High-Impact Sports Science Journals: A Meta-epidemiological Study

  • Systematic Review
  • Published:
Sports Medicine Aims and scope Submit manuscript

Abstract

Background

Assessing certainty of evidence is a key element of any systematic review. The aim of this meta-epidemiology study was to understand the frequency and ways with which certainty of evidence is assessed in contemporary systematic reviews published in high-impact sports science journals.

Methods

We searched PubMed and relevant journal web sites from 1 August 2016 to 11 October 2022 for systematic reviews published in the top-ten highest-impact journals within the 2020 Journal Citation Report for the Sports Sciences category. Pairs of independent reviewers screened items using a priori established criteria.

Results

Of 1250 eligible documents, 258 (20.6%) assessed the certainty of evidence, defined as using two or more distinct domains to provide an overall rating of the trustworthiness of findings across studies. Nine methods were cited for assessing certainty, with the most common being the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach (61.6%). The proportion of systematic reviews assessing certainty of evidence appeared to increase over the 6-year timeframe analyzed. Across all reviews analyzed, a large majority addressed the domains of risk of bias, imprecision, and inconsistency of the results. Other certainty domains including indirectness/applicability were less commonly assessed.

Discussion

Only one in five recent contemporary systematic reviews in the field of exercise and sports science assessed certainty of evidence. Organizational and institutional education on methods for assessing evidence may help further increase uptake of these methods and improve both the quality and clinical impact of systematic reviews in the field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol. 2019;70(1):747–70.

    Article  PubMed  Google Scholar 

  3. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med. 2016;21(4):125–7.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Paul M, Leibovici L. Systematic review or meta-analysis? Their place in the evidence hierarchy. Clin Microbiol Infect. 2014;20(2):97–100.

    Article  CAS  PubMed  Google Scholar 

  5. Berlin JA, Golub RM. Meta-analysis as evidence: building a better pyramid. JAMA. 2014;312(6):603–5.

    Article  CAS  PubMed  Google Scholar 

  6. Ebell MH, Siwek J, Weiss BD, Woolf SH, Susman J, Ewigman B, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract. 2004;17(1):59–67.

    Article  PubMed  Google Scholar 

  7. Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014;312(2):171–9.

    Article  CAS  PubMed  Google Scholar 

  8. West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al. Systems to rate the strength of scientific evidence. Evid Rep Technol Assess (Summ). 2002;47:1–11.

    Google Scholar 

  9. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, et al. What is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336(7651):995–8.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;29(372): n71.

    Article  Google Scholar 

  11. Ardern CL, Buttner F, Andrade R, Weir A, Ashe MC, Holden S, et al. Implementing the 27 PRISMA 2020 Statement items for systematic reviews in the sport and exercise medicine, musculoskeletal rehabilitation and sports science fields: the PERSiST (implementing Prisma in Exercise, Rehabilitation, Sport medicine and SporTs science) guidance. Br J Sports Med. 2022;56(4):175–95.

    Article  PubMed  Google Scholar 

  12. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–42.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust: National Academies Press; 2011.

  14. Siedler MR, Lamadrid P, Humphries MN, Mustafa RA, Falck-Ytter Y, Dahm P, et al. The quality of physical activity guidelines, but not the specificity of their recommendations, has improved over time: a systematic review and critical appraisal. Appl Physiol Nutr Metab. 2021;46(1):34–45.

    Article  PubMed  Google Scholar 

  15. Vancampfort D, Sweers K, Probst M, Mitchell AJ, Knapen J, De Hert M. Quality assessment of physical activity recommendations within clinical practice guidelines for the prevention and treatment of cardio-metabolic risk factors in people with schizophrenia. Community Ment Health J. 2011;47(6):703–10.

    Article  PubMed  Google Scholar 

  16. Armstrong JJ, Rodrigues IB, Wasiuta T, MacDermid JC. Quality assessment of osteoporosis clinical practice guidelines for physical activity and safe movement: an AGREE II appraisal. Arch Osteoporos. 2016;11:6.

    Article  PubMed  Google Scholar 

  17. Halperin I, Vigotsky AD, Foster C, Pyne DB. Strengthening the practice of exercise and sport-science research. Int J Sports Physiol Perform. 2018;13(2):127–34.

    Article  PubMed  Google Scholar 

  18. Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Oxford Levels of Evidence Working Group. The Oxford Levels of Evidence 2 [cited 2021 December 14]. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence.

  20. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;18(343): d5928.

    Article  Google Scholar 

  22. Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;28(366): l4898.

    Article  Google Scholar 

  23. Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;12(355): i4919.

    Article  Google Scholar 

  24. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377–84.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. 2011. https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

  26. Physiotherapy Evidence Database. PEDro Scale. 1999. https://pedro.org.au/english/resources/pedro-scale/.

  27. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344–9.

    Article  Google Scholar 

  28. Moher D. CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. JAMA. 1998;279(18):1489–91.

    Article  CAS  PubMed  Google Scholar 

  29. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348(mar07 3):g1687-g.

  30. Schünemann H BJ, Guyatt G, Oxman A. GRADE handbook; 2013.

  31. Higgins JPT TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). 2022 [cited 2023 June 27]. https://training.cochrane.org/handbook.

  32. van Tulder M, Furlan A, Bombardier C, Bouter L, Editorial Board of the Cochrane Collaboration Back Review G. Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976). 2003;28(12):1290–9.

  33. Conway A, Conway Z, Soalheira K, Sutherland J. High quality of evidence is uncommon in Cochrane systematic reviews in Anaesthesia, Critical Care and Emergency Medicine. Eur J Anaesthesiol. 2017;34(12):808–13.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Nasser SM, Cooke G, Kranzer K, Norris SL, Olliaro P, Ford N. Strength of recommendations in WHO guidelines using GRADE was associated with uptake in national policy. J Clin Epidemiol. 2015;68(6):703–7.

    Article  PubMed  Google Scholar 

  35. Schunemann HJ, Neumann I, Hultcrantz M, Brignardello-Petersen R, Zeng L, Murad MH, et al. GRADE guidance 35: update on rating imprecision for assessing contextualized certainty of evidence and making decisions. J Clin Epidemiol. 2022;150:225–42.

    Article  PubMed  Google Scholar 

  36. Werner SS, Binder N, Toews I, Schunemann HJ, Meerpohl JJ, Schwingshackl L. Use of GRADE in evidence syntheses published in high-impact-factor nutrition journals: A methodological survey. J Clin Epidemiol. 2021;135:54–69.

    Article  PubMed  Google Scholar 

  37. Gianola S, Bargeri S, Nembrini G, Varvello A, Lunny C, Castellini G. One-third of systematic reviews in rehabilitation applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system to evaluate certainty of evidence: a meta-research study. Arch Phys Med Rehabil. 2023;104(3):410–7.

    Article  PubMed  Google Scholar 

  38. Naude CE, Durao S, Harper A, Volmink J. Scope and quality of Cochrane reviews of nutrition interventions: a cross-sectional study. Nutr J. 2017;16(1):22.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Kane RL, Butler M, Ng W. Examining the quality of evidence to support the effectiveness of interventions: an analysis of systematic reviews. BMJ Open. 2016;6(5): e011051.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Betini M, Volpato ES, Anastacio GD, de Faria RT, El Dib R. Choosing the right journal for your systematic review. J Eval Clin Pract. 2014;20(6):834–6.

    Article  PubMed  Google Scholar 

  41. Suelzer EM, Jackson JL. Measures of impact for journals, articles, and authors. J Gen Intern Med. 2022;37(7):1593–7.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Ari MD, Iskander J, Araujo J, Casey C, Kools J, Chen B, et al. A science impact framework to measure impact beyond journal metrics. PLoS ONE. 2020;15(12): e0244407.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence—publication bias. J Clin Epidemiol. 2011;64(12):1277–82.

    Article  PubMed  Google Scholar 

  44. Sallis JF, Prochaska JJ, Taylor WC. A review of correlates of physical activity of children and adolescents. Med Sci Sports Exerc. 2000;32(5):963–75.

    Article  CAS  PubMed  Google Scholar 

  45. Lubans DR, Morgan PJ, Cliff DP, Barnett LM, Okely AD. Fundamental movement skills in children and adolescents: review of associated health benefits. Sports Med. 2010;40(12):1019–35.

    Article  PubMed  Google Scholar 

  46. Brignardello-Petersen R, Mustafa RA, Siemieniuk RAC, Murad MH, Agoritsas T, Izcovich A, Schünemann HJ, Guyatt GH, GRADE Working Group. GRADE approach to rate the certainty from a network meta-analysis: addressing incoherence. J Clin Epidemiol. 2019;108:77–85.

  47. Pescatello LS, Hennessy EA, Katzmarzyk PT, Kraus WE, Fish AF, Craft LL, et al. Best practices for meta-reviews in physical activity and health research: insights From the Physical Activity Guidelines for Americans Advisory Committee Scientific Report. J Phys Act Health. 2021;18(11):1437–45.

    Article  PubMed  Google Scholar 

  48. Wright JG, Einhorn TA, Heckman JD. Grades of recommendation. J Bone Jt Surg Am. 2005;87(9):1909–10.

    Article  Google Scholar 

  49. Wright JG. Revised grades of recommendation for summaries or reviews of orthopaedic surgical studies. J Bone Jt Surg Am. 2006;88(5):1161–2.

    Article  Google Scholar 

  50. Oxford Centre for Evidence-Based Medicine. Levels of Evidence. 2009. https://www.cebm.ox.ac.uk/resources/levels-of-evidence/oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009.

  51. Meeus M., Gebruers N. Health literacy: from reference to review. Leuven: Acco; 2016.

  52. World Health Organization. General Guidelines for Methodologies on Research and Evaluation of Traditional Medicine. Geneva; 2000.

Download references

Acknowledgements

The authors acknowledge Patrick S. Harty for assistance in initial title and abstract screening and Brett T. Norling for input on data extraction and interpretation.

Author information

Authors and Affiliations

  1. Evidence Foundation, Cleveland, OH, USA

    Madelin R. Siedler, Yngve Falck-Ytter, Reem A. Mustafa, Shahnaz Sultan, Philipp Dahm, Rebecca L. Morgan & M. Hassan Murad

  2. Department of Kinesiology and Sport Management, Lubbock, TX, USA

    Madelin R. Siedler, Katie N. Harris, Christian Rodriguez & Matthew T. Stratton

  3. Department of Health and Kinesiology, Texas A&M University, College Station, TX, USA

    Megan H. Lewis

  4. Exercise Science Program, Tampa, FL, USA

    Priscila Semidey-Lamadrid

  5. Department of Psychology, University of Notre Dame, Notre Dame, IN, USA

    Miguel Blacutt

  6. College of Kinesiology, University of Saskatchewan, Saskatoon, SK, Canada

    Zeinab Hosseini

  7. Division of Gastroenterology and Hepatology, Case Western Reserve University School of Medicine, Cleveland, OH, USA

    Yngve Falck-Ytter

  8. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada

    Reem A. Mustafa & Rebecca L. Morgan

  9. Department of Internal Medicine, University of Kansas Medical Center, Kansas City, KS, USA

    Reem A. Mustafa

  10. Division of Gastroenterology, University of Minnesota, Minneapolis, MN, USA

    Shahnaz Sultan

  11. Urology, Minneapolis VA Health Care System, Minneapolis, MN, USA

    Philipp Dahm

  12. Department of Urology, University of Minnesota, Minneapolis, MN, USA

    Philipp Dahm

  13. Department of Population and Quantitative Health Sciences, School of Medicine, Case Western Reserve University, Cleveland, OH, USA

    Rebecca L. Morgan

  14. Evidence-Based Practice Center, Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA

    M. Hassan Murad

Authors
  1. Madelin R. Siedler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Katie N. Harris
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Megan H. Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Priscila Semidey-Lamadrid
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthew T. Stratton
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Miguel Blacutt
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zeinab Hosseini
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yngve Falck-Ytter
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Reem A. Mustafa
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Shahnaz Sultan
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Philipp Dahm
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Rebecca L. Morgan
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. M. Hassan Murad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Hassan Murad.

Ethics declarations

Author contributions

Concept and design: M.R.S., R.L.M., and M.H.M. Screening and data extraction: M.R.S., K.N.H., C.R., M.H.L., P.S.-L., M.T.S., M.B., and Z.H. Data curation and analysis: M.R.S. Drafting of the manuscript: M.R.S. Critical review and approval of the manuscript: all authors.

Funding

No external funds were used in the development of this review.

Conflict of interest

The authors declare no financial conflicts of interest. M.R.S., Y.F.Y., R.A.M., S.S., P.D., R.L.M., and M.H.M. are members of the US GRADE Network. M.R.S. is a fellow of the Evidence Foundation and receives a direct stipend. All other authors declare no conflicts of interest.

Data availability

Data and data collection template forms are available from the corresponding author upon reasonable request.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Code availability

The analytic code is available as supplementary material.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 34 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Siedler, M.R., Harris, K.N., Rodriguez, C. et al. Certainty of Evidence Assessment in Systematic Reviews Published by High-Impact Sports Science Journals: A Meta-epidemiological Study. Sports Med 54, 473–484 (2024). https://doi.org/10.1007/s40279-023-01941-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40279-023-01941-x

Search

Navigation