Skip to main content

Advertisement

SpringerLink
Log in

Methodological and reporting quality assessment of network meta-analyses in anesthesiology: a systematic review and meta-epidemiological study

Évaluation de la qualité méthodologique et de communication des méta-analyses en réseau en anesthésiologie : revue systématique et étude méta-épidémiologique

  • Reports of Original Investigations
  • Published:
Canadian Journal of Anesthesia/Journal canadien d'anesthésie Aims and scope Submit manuscript

Abstract

Purpose

The scientific rigour of the conduct and reporting of anesthesiology network meta-analyses (NMAs) is unknown. This systematic review and meta-epidemiological study assessed the methodological and reporting quality of NMAs in anesthesiology.

Methods

We searched four databases, including MEDLINE, PubMed, Embase, and the Cochrane Systematic Reviews Database, for anesthesiology NMAs published from inception to October 2020. We assessed the compliance of NMAs against A Measurement Tool to Assess Systematic Reviews (AMSTAR-2), Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement for Network Meta-Analyses (PRISMA-NMA), and PRISMA checklists. We measured the compliance across various items in AMSTAR-2 and PRISMA checklists and provided recommendations to improve quality.

Results

Using the AMSTAR-2 rating method, 84% (52/62) of NMAs were rated “critically low.” Quantitatively, the median [interquartile range] AMSTAR-2 score was 55 [44–69]%, while the PRISMA score was 70 [61–81]%. Methodological and reporting scores showed a strong correlation (R = 0.78). Anesthesiology NMAs had a higher AMSTAR-2 score and PRISMA score if they were published in higher impact factor journals (P = 0.006 and P = 0.01, respectively) or followed PRISMA-NMA reporting guidelines (P = 0.001 and P = 0.002, respectively). Network meta-analyses from China had lower scores (P < 0.001 and P < 0.001, respectively). Neither score improved over time (P = 0.69 and P = 0.67, respectively).

Conclusion

The current study highlights numerous methodological and reporting deficiencies in anesthesiology NMAs. Although the AMSTAR tool has been used to assess the methodological quality of NMAs, dedicated tools for conducting and assessing the methodological quality of NMAs are urgently required.

Study registration

PROSPERO (CRD42021227997); first submitted 23 January 2021.

Résumé

Objectif

La rigueur scientifique de la conduite et de la communication des méta-analyses en réseau (MAR) en anesthésiologie est inconnue. Cette revue systématique et étude méta-épidémiologique a évalué la qualité méthodologique et de communication des MAR en anesthésiologie.

Méthode

Nous avons mené des recherches dans quatre bases de données, soit MEDLINE, PubMed, Embase et la base de données des revues systématiques Cochrane, pour trouver des MAR en anesthésiologie publiées depuis la création de ces bases de données jusqu’en octobre 2020. Nous avons évalué la conformité des MAR par rapport à trois outils, soit : AMSTAR-2 (outil de mesure pour évaluer les revues systématiques), PRISMA-NMA et les listes de contrôle PRISMA. Nous avons mesuré la conformité de divers éléments des listes de contrôle AMSTAR-2 et PRISMA et formulé des recommandations pour améliorer la qualité.

Résultats

En utilisant la méthode de notation AMSTAR-2, 84 % (52/62) des MAR ont reçu la cote « extrêmement faible ». Quantitativement, le score médian [écart interquartile] sur l’AMSTAR-2 était de 55 [44-69] %, tandis que le score PRISMA était de 70 [61-81] %. Les scores méthodologiques et de communication ont montré une forte corrélation (R = 0,78). Les MAR en anesthésiologie avaient un score AMSTAR-2 et un score PRISMA plus élevés si elles étaient publiées dans des revues à facteur d’impact plus élevé (P = 0,006 et P = 0,01, respectivement) ou avaient suivi les lignes directrices de PRISMA-NMA en matière de communication des résultats (P = 0,001 et P = 0,002, respectivement). Les méta-analyses en réseau provenant de Chine avaient des scores plus faibles (P < 0,001 et P < 0,001, respectivement). Aucun des deux scores ne s’est amélioré au fil du temps (P = 0,69 et P = 0,67, respectivement).

Conclusion

La présente étude met en évidence de nombreuses lacunes méthodologiques et de communication dans les MAR en anesthésiologie. Bien que l’outil AMSTAR ait été utilisé pour évaluer la qualité méthodologique des MAR, il est urgent de disposer d’outils spécialisés pour mener des MAR et en évaluer la qualité méthodologique.

Enregistrement de l’étude

PROSPERO (CRD42021227997); soumis pour la première fois le 23 janvier 2021.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Berlin JA, Golub RM. Meta-analysis as evidence: building a better pyramid. JAMA 2014; 312: 603–5. https://doi.org/10.1001/jama.2014.8167

    Article  CAS  PubMed  Google Scholar 

  2. Lee YH. Strengths and limitations of meta-analysis. Korean J Med 2019; 94: 391–5.

    Article  Google Scholar 

  3. Rouse B, Chaimani A, Li T. Network meta-analysis: an introduction for clinicians. Intern Emerg Med 2017; 12: 103–11. https://doi.org/10.1007/s11739-016-1583-7

    Article  PubMed  Google Scholar 

  4. Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health 2011; 14: 429–37. https://doi.org/10.1016/j.jval.2011.01.011

    Article  PubMed  Google Scholar 

  5. The Cochrane Collaboration. A network meta-analysis toolkit. Available from URL: https://methods.cochrane.org/cmi/network-meta-analysis-toolkit (accessed January 2023).

  6. Chaimani A, Caldwell DM, Li T, Higgins JP, Salanti G. Undertaking network meta-analyses. In: Higgins PT, Tomas J, Chandler J, et al. (Eds.). Cochrane Handbook for Systematic Reviews of Interventions, 2nd ed. Hoboken: Wiley-Blackwell; 2019: 285–320. https://doi.org/10.1002/9781119536604.ch11

  7. Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015; 4: 1. https://doi.org/10.1186/2046-4053-4-1

    Article  PubMed  PubMed Central  Google Scholar 

  8. Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. J Clin Epidemiol 2011; 64: 163–71. https://doi.org/10.1016/j.jclinepi.2010.03.016

    Article  PubMed  Google Scholar 

  9. Puhan MA, Schünemann HJ, Murad MH, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ 2014; 349: g5630. https://doi.org/10.1136/bmj.g5630

    Article  PubMed  Google Scholar 

  10. Hutton B, Catalá-López F, Moher D. The PRISMA statement extension for systematic reviews incorporating network meta-analysis: PRISMA-NMA [Spanish]. Med Clin (Barc) 2016; 147: 262–6. https://doi.org/10.1016/j.medcli.2016.02.025

    Article  PubMed  Google Scholar 

  11. Liu H, Zhou X, Yu G, Sun X. The effects of the PRISMA statement to improve the conduct and reporting of systematic reviews and meta-analyses of nursing interventions for patients with heart failure. Int J Nurs Pract 2019; 25: e12729. https://doi.org/10.1111/ijn.12729

    Article  PubMed  Google Scholar 

  12. Nawijn F, Ham WH, Houwert RM, Groenwold RH, Hietbrink F, Smeeing DP. Quality of reporting of systematic reviews and meta-analyses in emergency medicine based on the PRISMA statement. BMC Emerg Med 2019; 19: 19. https://doi.org/10.1186/s12873-019-0233-6

    Article  PubMed  PubMed Central  Google Scholar 

  13. Page MJ, Moher D. Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review. Syst Rev 2017; 6: 263. https://doi.org/10.1186/s13643-017-0663-8

    Article  PubMed  PubMed Central  Google Scholar 

  14. Veroniki AA, Tsokani S, Zevgiti S, et al. Do reporting guidelines have an impact? Empirical assessment of changes in reporting before and after the PRISMA extension statement for network meta-analysis. Syst Rev 2021; 10: 246. https://doi.org/10.1186/s13643-021-01780-9

    Article  PubMed  PubMed Central  Google Scholar 

  15. Salanti G, Higgins JP, Ades AE, Ioannidis JP. Evaluation of networks of randomized trials. Stat Methods Med Res 2008; 17: 279–301. https://doi.org/10.1177/0962280207080643

    Article  PubMed  Google Scholar 

  16. Ge L, Tian JH, Li XX, et al. Epidemiology characteristics, methodological assessment and reporting of statistical analysis of network meta-analyses in the field of cancer. Sci Rep 2016; 6: 37208. https://doi.org/10.1038/srep37208

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Tonin FS, Borba HH, Leonart LP, et al. Methodological quality assessment of network meta-analysis of drug interventions: implications from a systematic review. Int J Epidemiol 2019; 48: 620–32. https://doi.org/10.1093/ije/dyy197

    Article  PubMed  Google Scholar 

  18. Gao Y, Ge L, Ma X, Shen X, Liu M, Tian J. Improvement needed in the network geometry and inconsistency of Cochrane network meta-analyses: a cross-sectional survey. J Clin Epidemiol 2019; 113: 214–27. https://doi.org/10.1016/j.jclinepi.2019.05.022

    Article  PubMed  Google Scholar 

  19. Ho RS, Wu X, Yuan J, et al. Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respir Med 2015; 25: 14102. https://doi.org/10.1038/npjpcrm.2014.102

    Article  PubMed  PubMed Central  Google Scholar 

  20. Matthias K, Rissling O, Pieper D, et al. The methodological quality of systematic reviews on the treatment of adult major depression needs improvement according to AMSTAR 2: a cross-sectional study. Heliyon 2020; 6: e04776. https://doi.org/10.1016/j.heliyon.2020.e04776

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Kanters S, Ford N, Druyts E, Thorlund K, Mills EJ, Bansback N. Use of network meta-analysis in clinical guidelines. Bull World Health Organ 2016; 94: 782–4. https://doi.org/10.2471/blt.16.174326

    Article  PubMed  PubMed Central  Google Scholar 

  22. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017; 358: j4008. https://doi.org/10.1136/bmj.j4008

    Article  PubMed  PubMed Central  Google Scholar 

  23. Sehmbi H, Retter S, Shah UJ, Nguyen D, Martin J, Uppal V. Epidemiological, methodological, and statistical characteristics of network meta-analysis in anaesthesia: a systematic review. Br J Anaesth 2022; https://doi.org/10.1016/j.bja.2022.08.042

    Article  PubMed  Google Scholar 

  24. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009; 151: 264–9. https://doi.org/10.7326/0003-4819-151-4-200908180-00135

  25. Schiavo JH. PROSPERO: an international register of systematic review protocols. Med Ref Serv Q 2019; 38: 171–80. https://doi.org/10.1080/02763869.2019.1588072

    Article  PubMed  Google Scholar 

  26. Covidence. Covidence systematic review software. Available from URL: www.covidence.org (accessed January 2023).

  27. Patil I. Visualizations with statistical details: the 'ggstatsplot' approach. J Open Source Softw 2021; 6: 3167. https://doi.org/10.21105/joss.03167

  28. Schober P, Boer C, Schwarte LA. Correlation coefficients: appropriate use and interpretation. Anesth Analg 2018; 126: 1763–8. https://doi.org/10.1213/ane.0000000000002864

    Article  PubMed  Google Scholar 

  29. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282: 1054–60. https://doi.org/10.1001/jama.282.11.1054

    Article  PubMed  Google Scholar 

  30. Tricco AC, Cogo E, Page MJ, et al. A third of systematic reviews changed or did not specify the primary outcome: a PROSPERO register study. J Clin Epidemiol 2016; 79: 46–54. https://doi.org/10.1016/j.jclinepi.2016.03.025

    Article  PubMed  Google Scholar 

  31. Dos Santos MBF, Agostini BA, Bassani R, Rocha Pereira GK, Sarkis-Onofre R. Protocol registration improves reporting quality of systematic reviews in dentistry. BMC Med Res Methodol 2020; 20: 57. https://doi.org/10.1186/s12874-020-00939-7

    Article  PubMed  PubMed Central  Google Scholar 

  32. Sterne JA, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019; 366: l4898. https://doi.org/10.1136/bmj.l4898

    Article  Google Scholar 

  33. Lunny C, Tricco AC, Veroniki AA, et al. Methodological review to develop a list of bias items used to assess reviews incorporating network meta-analysis: protocol and rationale. BMJ Open 2021; 11: e045987. https://doi.org/10.1136/bmjopen-2020-045987

    Article  PubMed  PubMed Central  Google Scholar 

  34. Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in STATA. PLoS One 2013; 8: e76654. https://doi.org/10.1371/journal.pone.0076654

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Nikolakopoulou A, Higgins JPT, Papakonstantinou T, et al. CINeMA: an approach for assessing confidence in the results of a network meta-analysis. PLoS Med 2020; 17: e1003082. https://doi.org/10.1371/journal.pmed.1003082

    Article  PubMed  PubMed Central  Google Scholar 

  36. Chiocchia V, Nikolakopoulou A, Higgins JP, et al. ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis. BMC Med 2021; 19: 304. https://doi.org/10.1186/s12916-021-02166-3

    Article  PubMed  PubMed Central  Google Scholar 

  37. Sullivan SM, Coyle D, Wells G. What guidance are researchers given on how to present network meta-analyses to end-users such as policymakers and clinicians? A systematic review. PLoS One 2014; 9: e113277. https://doi.org/10.1371/journal.pone.0113277

    Article  CAS  PubMed  Google Scholar 

  38. Higgins J, Lasserson T, Chandler J, et al. Methodological expectations of cochrane intervention reviews (MECIR), 2022. Available from URL: https://community.cochrane.org/mecir-manual (accessed January 2023).

  39. Li T, Puhan MA, Vedula SS, Singh S, Dickersin K. Network meta-analysis-highly attractive but more methodological research is needed. BMC Med 2011; 9: 79. https://doi.org/10.1186/1741-7015-9-79

    Article  PubMed  PubMed Central  Google Scholar 

  40. Lee AW. Use of network meta-analysis in systematic reviews: a survey of authors. Syst Rev 2016; 5: 8. https://doi.org/10.1186/s13643-015-0174-4

    Article  PubMed  PubMed Central  Google Scholar 

  41. Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One 2011; 6: e27611. https://doi.org/10.1371/journal.pone.0027611

    Article  PubMed  PubMed Central  Google Scholar 

  42. Jansen JP. Heterogeneity and subgroup analysis in network meta-analysis. In: Ting N, Cappelleri JC, Ho S, Chen DG (Eds.). Design and Analysis of Subgroups with Biopharmaceutical Applications. Cham: Springer International Publishing; 2020: 369–85.

    Chapter  Google Scholar 

  43. Dias S, Ades AE, Welton NJ, Jansen JP, Sutton AJ. Meta-regression for relative treatment effects. In: Dias S, Ades AE, Welton NJ, Jansen JP, Sutton J (Ed.). Network Meta‐Analysis for Decision Making. Hoboken: Wiley; 2018: 227–71.

    Chapter  Google Scholar 

  44. Gwon Y, Mo M, Chen MH, et al. Network meta-regression for ordinal outcomes: applications in comparing Crohn's disease treatments. Stat Med 2020; 39: 1846–70. https://doi.org/10.1002/sim.8518

    Article  Google Scholar 

  45. Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA Statement for reporting literature searches in systematic reviews. Syst Rev 2021; 10: 39. https://doi.org/10.1186/s13643-020-01542-z

    Article  PubMed  PubMed Central  Google Scholar 

  46. Oh JH, Shin WJ, Park S, Chung JS. Reporting and methodologic evaluation of meta-analyses published in the anesthesia literature according to AMSTAR and PRISMA checklists: a preliminary study. Korean J Anesthesiol 2017; 70: 446–55. https://doi.org/10.4097/kjae.2017.70.4.446

    Article  PubMed  PubMed Central  Google Scholar 

  47. Efthimiou O, Mavridis D, Debray TP, et al. Combining randomized and non-randomized evidence in network meta-analysis. Stat Med 2017; 36: 1210–26. https://doi.org/10.1002/sim.7223

    Article  PubMed  Google Scholar 

  48. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003; 326: 1167–70. https://doi.org/10.1136/bmj.326.7400.1167

    Article  PubMed  PubMed Central  Google Scholar 

  49. Brignardello-Petersen R, Bonner A, Alexander PE, et al. Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. J Clin Epidemiol 2018; 93: 36–44. https://doi.org/10.1016/j.jclinepi.2017.10.005

    Article  PubMed  Google Scholar 

Download references

Author contributions

Herman Sehmbi, Janet Martin, and Vishal Uppal contributed to study design and protocol. Herman Sehmbi and Ushma J. Shah contributed to initial study screening. Herman Sehmbi, Susanne Retter, Derek Nguyen, and Ushma J. Shah contributed to full-text review and data extraction. Herman Sehmbi and Vishal Uppal contributed to statistical analysis. Herman Sehmbi, Susanne Retter, Ushma J. Shah, Derek Nguyen, Janet Martin, and Vishal Uppal reviewed the manuscript’s final version.

Acknowledgements

The authors would like to thank Darlene Chapman (MLIS), Dalhousie University, for her invaluable help with peer reviewing the literature search for this review.

Disclosures

Dr. Vishal Uppal is an Associate Editor of the Canadian Journal of Anesthesia/Journal canadien d’anesthésie. He had no involvement in the handling of this manuscript.

Funding statement

None.

Editorial responsibility

This submission was handled by Dr. Stephan K. W. Schwarz, Editor-in-Chief, Canadian Journal of Anesthesia/Journal canadien d’anesthésie.

Author information

Authors and Affiliations

  1. Department of Anesthesia & Perioperative Medicine, Schulich School of Medicine & Dentistry, Western University, London, ON, Canada

    Herman Sehmbi MD, MSc, Ushma J. Shah MD, FRCA & Janet Martin PharmD

  2. London Health Sciences Centre, London, ON, Canada

    Herman Sehmbi MD, MSc

  3. Department of Anesthesia, Pain Management & Perioperative Medicine, Dalhousie University, 10W Victoria Building, 1276 South Park St, Halifax, NS, B3H 2Y9, Canada

    Susanne Retter MD & Vishal Uppal MBBS, MSc, FRCA

  4. Nova Scotia Health Authority, Halifax, NS, Canada

    Susanne Retter MD & Vishal Uppal MBBS, MSc, FRCA

  5. IWK Health Centre, Halifax, NS, Canada

    Vishal Uppal MBBS, MSc, FRCA

  6. Schulich School of Medicine & Dentistry, London, ON, Canada

    Derek Nguyen BMSc

  7. Department of Epidemiology & Biostatistics, London Health Sciences Centre, Western University, London, ON, Canada

    Janet Martin PharmD

Authors
  1. Herman Sehmbi MD, MSc
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susanne Retter MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ushma J. Shah MD, FRCA
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Derek Nguyen BMSc
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Janet Martin PharmD
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Vishal Uppal MBBS, MSc, FRCA
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vishal Uppal MBBS, MSc, FRCA.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 1639 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sehmbi, H., Retter, S., Shah, U.J. et al. Methodological and reporting quality assessment of network meta-analyses in anesthesiology: a systematic review and meta-epidemiological study. Can J Anesth/J Can Anesth 70, 1461–1473 (2023). https://doi.org/10.1007/s12630-023-02510-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12630-023-02510-6

Keywords

Search

Navigation