Skip to main content
SpringerLink
Log in

The Fragility Index of randomized controlled trials in pediatric anesthesiology

L’indice de fragilité des études randomisées contrôlées en anesthésiologie pédiatrique

  • Reports of Original Investigations
  • Published:
Canadian Journal of Anesthesia/Journal canadien d'anesthésie Aims and scope Submit manuscript

Abstract

Purpose

The P value is a widely used measure of statistical importance but has many drawbacks and limitations, one being that it does not reflect the robustness of the results of a clinical trial. The Fragility Index (FI) was developed as a measure of how many outcome events would need to change to nonevents to render a significant P value nonsignificant (P ≥ 0.05). The FI of trials from other medical specialties is typically < 5. We aimed to determine the FI of pediatric anesthesiology randomized controlled trials (RCT) and to test for association with various characteristics of the included trials.

Methods

We conducted a comprehensive systematic search of high-impact anesthesia, surgical, and medical journals from the last 25 years for trials comparing an intervention between two groups with a statistically significant P value (< 0.05) for a dichotomous outcome. We also compared FI values for variables that reflect the quality and importance of a trial.

Results

The median [interquartile range] FI was 3 [1–7] and correlated positively with the number of participants (rS = 0.41; P < 0.001) and events (rS = 0.42; P < 0.001), and negatively with the P value (rPB = -0.36; P < 0.001). Other measures of trial quality and impact or importance were not strongly associated with the FI.

Conclusions

The FI of published trials in pediatric anesthesiology is similarly low as in other medical specialties. Larger trials with more events and P values ≤ 0.01 were associated with a higher FI.

Résumé

Objectif

La valeur P est une mesure d’importance statistique largement utilisée, mais elle présente de nombreux inconvénients et limites, notamment parce qu’elle ne reflète pas la robustesse des résultats d’une étude clinique. L’indice de fragilité (IF) a été mis au point pour mesurer le nombre d’événements du critère d’évaluation qui devraient se transformer en non-événements pour obtenir une valeur P non significative (P ≥ 0,05). L’IF des études d’autres spécialités médicales est généralement < 5. Notre objectif était de déterminer l’IF des études randomisées contrôlées (ERC) en anesthésiologie pédiatrique et de tester l’association avec diverses caractéristiques des études incluses.

Méthode

Nous avons réalisé une recherche systématique exhaustive dans les revues d’anesthésie, de chirurgie et médicales à fort impact des 25 dernières années pour trouver des études comparant une intervention entre deux groupes avec une valeur P significative d’un point de vue statistique (< 0,05) pour un résultat dichotomique. Nous avons également comparé les valeurs d’IF pour les variables qui reflètent la qualité et l’importance d’une étude.

Résultats

L’IF médian [écart interquartile] était de 3 [1 à 7] et était positivement corrélé avec le nombre de participant·es (rS = 0,41; P < 0,001) et d’événements (rS = 0,42; P < 0,001), et négativement avec la valeur P (rPB = -0,36; P < 0,001). D’autres mesures de la qualité et de l’impact ou de l’importance des études n’étaient pas fortement associées à l’IF.

Conclusion

L’IF des études publiées en anesthésiologie pédiatrique est tout aussi faible que dans d’autres spécialités médicales. Des études plus importantes avec plus d’événements et des valeurs P ≤ 0,01 étaient associées à un IF plus élevé.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Shafer SL, Dexter F. Publication bias, retrospective bias, and reproducibility of significant results in observational studies. Anesth Analg 2012; 114: 931–2. https://doi.org/10.1213/ane.0b013e31824a0b5b

    Article  PubMed  Google Scholar 

  2. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat 2016; 70: 129–33. https://doi.org/10.1080/00031305.2016.1154108

    Article  Google Scholar 

  3. Walsh M, Srinathan SK, McAuley DF, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. J Clin Epidemiol 2014; 67: 622–8. https://doi.org/10.1016/j.jclinepi.2013.10.019

    Article  PubMed  Google Scholar 

  4. Ahmed W, Fowler RA, McCredie VA. Does sample size matter when interpreting the fragility index? Crit Care Med 2016; 44: e1142. https://doi.org/10.1097/ccm.0000000000001976

  5. Mazzinari G, Ball L, Serpa Neto A, et al. The fragility of statistically significant findings in randomised controlled anaesthesiology trials: systematic review of the medical literature. Br J Anaesth 2018; 120: 935–41. https://doi.org/10.1016/j.bja.2018.01.012

    Article  CAS  PubMed  Google Scholar 

  6. Rickard M, Lorenzo AJ, Hannick JH, Blais AS, Koyle MA, Bägli DJ. Over-reliance on P values in urology: fragility of findings in the hydronephrosis literature calls for systematic reporting of robustness indicators. Urology 2019; 133: 204–10. https://doi.org/10.1016/j.urology.2019.03.045

    Article  PubMed  Google Scholar 

  7. Evaniew N, Files C, Smith C, et al. The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey. Spine J 2015; 15: 2188–97. https://doi.org/10.1016/j.spinee.2015.06.004

    Article  PubMed  Google Scholar 

  8. Ridgeon EE, Young PJ, Bellomo R, Mucchetti M, Lembo R, Landoni G. The fragility index in multicenter randomized controlled critical care trials. Crit Care Med 2016; 44: 1278–84. https://doi.org/10.1097/ccm.0000000000001670

    Article  PubMed  Google Scholar 

  9. Khan M, Evaniew N, Gichuru M, et al. The fragility of statistically significant findings from randomized trials in sports surgery: a systematic survey. Am J Sports Med 2017; 45: 2164–70. https://doi.org/10.1177/0363546516674469

    Article  PubMed  Google Scholar 

  10. Docherty KF, Campbell RT, Jhund PS, Petrie MC, McMurray JJ. How robust are clinical trials in heart failure? Eur Heart J 2017; 38: 338–45. https://doi.org/10.1093/eurheartj/ehw427

    Article  CAS  PubMed  Google Scholar 

  11. Chow JTY, Turkstra TP, Yim E, Jones PM. Sample size calculations for randomized clinical trials published in anesthesiology journals: a comparison of 2010 versus 2016. Can J Anesth 2018; 65: 611–8. https://doi.org/10.1007/s12630-018-1109-z

    Article  PubMed  Google Scholar 

  12. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996; 276: 637–9. https://doi.org/10.1001/jama.276.8.637

    Article  CAS  PubMed  Google Scholar 

  13. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019; 366: I4898. https://doi.org/10.1136/bmj.l4898

    Article  Google Scholar 

  14. Higgins JP, Savović J, Page MJ, Elberrs RG, Sterne JA. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JP, Thomas J, Chandler J, et al. (Eds.). Cochrane Handbook for Systematic Reviews of Interventions, 2nd edition, 2022. Available from URL: www.training.cochrane.org/handbook (accessed February 2023).

  15. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale: L. Erlbaum Associates; 1988.

  16. Ortega JL. Reliability and accuracy of altmetric providers: a comparison among Altmetric.com, PlumX and Crossref Event Data. Scientometrics 2018; 116: 2123–38. https://doi.org/10.1007/s11192-018-2838-z

  17. GitHub. rAltmetric: Retrieves altmetric data for any published paper from altmetrics.com. R package version 0.7. Available from URL: https://github.com/ropensci/rAltmetric (accessed February 2023).

  18. Holek M, Bdair F, Khan M, et al. Fragility of clinical trials across research fields: a synthesis of methodological reviews. Contemp Clin Trials 2020; 97: 106151. https://doi.org/10.1016/j.cct.2020.106151

  19. Goerke K, Parke M, Horn J, et al. Are results from randomized trials in anesthesiology robust or fragile? An analysis using the fragility index. Int J Evid Based Healthc 2020; 18: 116–24. https://doi.org/10.1097/xeb.0000000000000200

    Article  PubMed  Google Scholar 

  20. Turner RM, Bird SM, Higgins JP. The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One 2013; 8: e59202. https://doi.org/10.1371/journal.pone.0059202

  21. Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P. Reporting of sample size calculation in randomised controlled trials: review. BMJ 2009; 338: b1732. https://doi.org/10.1136/bmj.b1732

  22. Daniels JR, Dexter F, Espy JL, Brull SJ. Quantitative assessment of statistical reviews of patient safety research articles. J Patient Saf 2019; 15: 184–90. https://doi.org/10.1097/pts.0000000000000391

    Article  PubMed  Google Scholar 

  23. Dexter F, Shafer SL. Narrative review of statistical reporting checklists, mandatory statistical editing, and rectifying common problems in the reporting of scientific articles. Anesth Analg 2017; 124: 943–7. https://doi.org/10.1213/ane.0000000000001593

    Article  PubMed  Google Scholar 

  24. Smith SM, Dworkin RH. Prospective clinical trial registration: not sufficient, but always necessary. Anaesthesia 2018; 73: 538–41. https://doi.org/10.1111/anae.14189

    Article  CAS  PubMed  Google Scholar 

  25. Tahamtan I, Safipour Afshar A, Ahamdzadeh K. Factors affecting number of citations: a comprehensive review of the literature. Scientometrics 2016; 107: 1195–225. https://doi.org/10.1007/s11192-016-1889-2

    Article  Google Scholar 

  26. Fassoulaki A, Vassi A, Kardasis A, Chantziara V. Altmetrics should not be used for ranking of anaesthesia journals. Br J Anaesth 2018; 121: 514–6. https://doi.org/10.1016/j.bja.2018.05.048

    Article  CAS  PubMed  Google Scholar 

  27. Shochet LR, Kerr PG, Polkinghorne KR. The fragility of significant results underscores the need of larger randomized controlled trials in nephrology. Kidney Int 2017; 92: 1469–75. https://doi.org/10.1016/j.kint.2017.05.011

    Article  PubMed  Google Scholar 

  28. Gnech M, Lovatt CA, McGrath M, et al. Quality of reporting and fragility index for randomized controlled trials in the vesicoureteral reflux literature: where do we stand? J Pediatr Urol 2019; 15: 204–12. https://doi.org/10.1016/j.jpurol.2019.02.014

    Article  CAS  PubMed  Google Scholar 

  29. Costi D, Ellwood J, Wallace A, Ahmed S, Waring L, Cyna A. Transition to propofol after sevoflurane anesthesia to prevent emergence agitation: a randomized controlled trial. Paediatr Anaesth 2015; 25: 517–23. https://doi.org/10.1111/pan.12617

    Article  PubMed  Google Scholar 

  30. Caldwell JM, Youssefzadeh K, Limpisvasti O. A method for calculating the fragility index of continuous outcomes. J Clin Epidemiol 2021; 136: 20–5. https://doi.org/10.1016/j.jclinepi.2021.02.023

    Article  PubMed  Google Scholar 

  31. Potter GE. Dismantling the Fragility Index: a demonstration of statistical reasoning. Stat Med 2020; 39: 3720–31. https://doi.org/10.1002/sim.8689

    Article  PubMed  Google Scholar 

Download references

Author contributions

Jason Hayes and Mael Zuercher contributed to the study conception and design; acquisition, analysis, and interpretation of data; and drafting and revision of the manuscript. Nan Gai and Apala R. Chowdhury contributed to the acquisition, analysis, and interpretation of data and revised the manuscript. Kazuyoshi Aoyama contributed to the study conception and design; analysis, and interpretation of data; and drafting and revision of the manuscript.

Acknowledgements

The authors wish to thank Ms. Quenby Mahood, MI, The Hospital for Sick Children (SickKids; Toronto, ON, Canada) for conducting the literature searches.

Disclosures

The authors have no conflicts of interest to declare.

Funding statement

Support was provided from departmental sources.

Editorial responsibility

This submission was handled by Dr. Stephan K. W. Schwarz, Editor-in-Chief, Canadian Journal of Anesthesia/Journal canadien d’anesthésie.

Author information

Authors and Affiliations

  1. Department of Anesthesia and Pain Medicine, The Hospital for Sick Children (SickKids), 555 University Avenue, Toronto, ON, M5G 1X8, Canada

    Jason Hayes MD, Mael Zuercher MD, Nan Gai MD, Apala Roy Chowdhury MD, DNB & Kazuyoshi Aoyama MD, PhD

  2. Department of Anesthesiology and Pain Medicine, University of Toronto, Toronto, ON, Canada

    Jason Hayes MD, Nan Gai MD & Kazuyoshi Aoyama MD, PhD

  3. Program in Child Health Evaluative Sciences, SickKids Research Institute, Toronto, ON, Canada

    Kazuyoshi Aoyama MD, PhD

Authors
  1. Jason Hayes MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mael Zuercher MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan Gai MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Apala Roy Chowdhury MD, DNB
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kazuyoshi Aoyama MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason Hayes MD.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

eTable 1a

Anesthesiology top 25 highest Impact Factor journals according to 2019 Journal Citation Reports™ eTable 1bPediatrics top 10 highest Impact Factor journals according to 2019 Journal Citation Reports™eTable 1cGeneral medicine top 10 highest impact factor journals according to 2019 Journal Citation Reports™eTable 2Topics of study and journals of the included trialseTable 3Details of trials included for extraction and analysis of data (PDF 484 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hayes, J., Zuercher, M., Gai, N. et al. The Fragility Index of randomized controlled trials in pediatric anesthesiology. Can J Anesth/J Can Anesth 70, 1449–1460 (2023). https://doi.org/10.1007/s12630-023-02513-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12630-023-02513-3

Keywords

Search

Navigation