Skip to main content

Advertisement

SpringerLink
Log in

Predictors of Increased Fragility Index Scores in Surgical Randomized Controlled Trials: An Umbrella Review

  • Scientific Review
  • Published:
World Journal of Surgery Aims and scope Submit manuscript

Abstract

Background

The fragility index (FI) is defined as the minimum number of patients or subjects needed to switch experimental groups for statistical significance to be lost in a randomized control trial (RCT). This index is used to determine the robustness of a study’s findings and recently as a measure of evaluating RCT quality. The objective of this review was to identify and describe published systematic reviews utilizing FI to evaluate surgical RCTs and to determine if there were common factors associated with higher FI values.

Methods

Three databases (PubMed, MEDLINE [Ovid], Embase) were searched, followed by a subsequent abstract/title and full-text screening to yield 50 reviews of surgical RCTs. Authors, year of publication, name of journal, study design, number of RCTs, subspecialty, sample size, median FI, patients lost to follow-up, and associations between variables and FI scores were collected.

Results

Among 1007 of 2214 RCTs in 50 reviews reporting FI (median sample size 100), the pooled median FI was 3 (IQR: 1–7). Most reviews investigated orthopaedic surgery RCTs (n = 32). There was a moderate correlation between FI and p value (r = 0.-413), a mild correlation between FI and sample size (r = 0.188), and a mild correlation between FI and event number (r = 0.129).

Conclusion

Based on a limited sample of systematic reviews, surgical RCT FI values are still low (2–5). Future RCTs in surgery require improvement to study design in order to increase the robustness of statistically significant findings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Maldonado DR, Go CC, Huang BH et al (2021) The fragility index of hip arthroscopy randomized controlled trials: a systematic survey. Arthrosc Tech 37(6):1983–1989

    Article  Google Scholar 

  2. Khormaee S, Choe J, Ruzbarsky JJ et al (2018) The fragility of statistically significant results in pediatric orthopaedic randomized controlled trials as quantified by the fragility index: a systematic review. J Pediatr Orthop 38(8):e418–e423

    Article  PubMed  Google Scholar 

  3. Herndon CL, McCormick KL, Gazgalis A et al (2021) Fragility Index as a measure of randomized clinical trial quality in adult reconstruction: a systematic review. Arthroplast Today 11:239–251

    Article  PubMed  PubMed Central  Google Scholar 

  4. Walsh M, Srinathan SK, McAuley DF et al (2014) The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility Index. J Clin Epidemiol 67(6):622–628

    Article  PubMed  Google Scholar 

  5. Hegazy MA, El Nahas W, Roshdy S (2011) Surgical outcome of modified versus conventional parotidectomy in treatment of benign parotid tumors. J Surg Oncol 103(2):163–168

    Article  PubMed  Google Scholar 

  6. Marasco D, Russo J, Izzo A et al (2021) Static versus dynamic fixation of distal tibiofibular syndesmosis: a systematic review of overlapping meta-analyses. Knee Surg Sports Traumatol Arthrosc 29(11):3534–3542

    Article  PubMed  Google Scholar 

  7. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174

    Article  CAS  PubMed  Google Scholar 

  8. Kung J, Chiappelli F, Cajulis OO et al (2010) From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J 4:84–91

    PubMed  PubMed Central  Google Scholar 

  9. Anand S, Kainth D (2021) Fragility index of recently published meta-analyses in pediatric urology: a striking observation. Cureus 13(7):e16225

    PubMed  PubMed Central  Google Scholar 

  10. Bowers A, Meyer C, Tritz D et al (2018) Assessing quality of randomized trials supporting guidelines for laparoscopic and endoscopic surgery. J Surg Res 224:233–239

    Article  PubMed  Google Scholar 

  11. Budhiraja P, Kaplan B, Kalot M et al (2022) Current state of evidence on kidney transplantation: how fragile are the results. Transplantation 106(2):248–256

    Article  PubMed  Google Scholar 

  12. Carroll AH, Rigor P, Wright MA et al (2022) Fragility of randomized controlled trials on treatment of proximal humeral fracture. J Shoulder Elbow Surg 31(8):1610–1616

    Article  PubMed  Google Scholar 

  13. Checketts JX, Scott JT, Meyer C et al (2018) The robustness of trials that guide evidence-based orthopaedic surgery. J Bone Joint Surg Am 100(12):e85

    Article  PubMed  Google Scholar 

  14. Chertin L, Burman I, Haifler M (2021) Urologic oncology randomized controlled trials are frequently fragile—A review of the urology literature. Urol Oncol 39(10):735.e1-735.e8

    Article  PubMed  Google Scholar 

  15. Chin B, Copeland A, Gallo L et al (2019) The fragility of statistically significant randomized controlled trials in plastic surgery. Plast Reconstr Surg 144(5):1238–1245

    Article  CAS  PubMed  Google Scholar 

  16. Constant M, Trofa DP, Saltzman BM et al (2021) The fragility of statistical significance in patellofemoral instability research: a systematic review. Am J Sports Med. https://doi.org/10.1177/03635465211039202

    Article  PubMed  Google Scholar 

  17. Davey MS, Hurley ET, Doyle TR et al (2022) The fragility index of statistically significant findings from randomized controlled trials comparing the management strategies of anterior shoulder instability. Am J Sports Med. https://doi.org/10.1177/03635465221077268

    Article  PubMed  Google Scholar 

  18. Ehlers CB, Curley AJ, Fackler NP et al (2021) The statistical fragility of hamstring versus patellar tendon autografts for anterior cruciate ligament reconstruction: a systematic review of comparative studies. Am J Sports Med 49(10):2827–2833

    Article  PubMed  Google Scholar 

  19. Ehlers CB, Curley AJ, Fackler NP et al (2021) The statistical fragility of single-bundle vs double-bundle autografts for ACL reconstruction: a systematic review of comparative studies. Orthop J Sports Med 9(12):23259671211064624

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ekhtiari S, Gazendam AM, Nucci NW et al (2021) The fragility of statistically significant findings from randomized controlled trials in hip and knee arthroplasty. J Arthroplast 36(6):2211-2218.e1

    Article  Google Scholar 

  21. Evaniew N, Files C, Smith C et al (2015) The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey. Spine J 15(10):2188–2197

    Article  PubMed  Google Scholar 

  22. Forrester LA, Jang E, Lawson MM et al (2020) Statistical fragility of surgical and procedural clinical trials in orthopaedic oncology. J Am Acad Orthop Surg Glob Res Rev 4(6):e19.00152

    PubMed  PubMed Central  Google Scholar 

  23. Forrester LA, McCormick KL, Bonsignore-Opp L et al (2021) Statistical fragility of surgical clinical trials in orthopaedic trauma. J Am Acad Orthop Surg Glob Res Rev 5(11):e20.00197

    PubMed  PubMed Central  Google Scholar 

  24. Gnech M, Lovatt CA, McGrath M et al (2019) Quality of reporting and fragility index for randomized controlled trials in the vesicoureteral reflux literature: Where do we stand? J Pediatr Urol 15(3):204–212

    Article  CAS  PubMed  Google Scholar 

  25. Go CC, Maldonado DR, Go BC et al (2022) The fragility index of total hip arthroplasty randomized control trials: a systematic review. J Am Acad Orthop Surg 30(9):e741–e750

    Article  PubMed  Google Scholar 

  26. Khan M, Evaniew N, Gichuru M et al (2017) The fragility of statistically significant findings from randomized trials in sports surgery: a systematic survey. Am J Sports Med 45(9):2164–2170

    Article  PubMed  Google Scholar 

  27. Kyriakides PW, Schultz BJ, Egol K et al (2021) The fragility and reverse fragility indices of proximal humerus fracture randomized controlled trials: a systematic review. Eur J Trauma Emerg Surg. https://doi.org/10.1007/s00068-021-01684-2

    Article  PubMed  Google Scholar 

  28. Li B, Kong I, McGrath M et al (2021) Evaluating the literature on preoperative androgen stimulation for hypospadias repair using the fragility index - can we trust observational studies. J Pediatr Urol 17(5):661–669

    Article  PubMed  Google Scholar 

  29. McCormick KL, Tedesco LJ, Swindell HW et al (2021) Statistical fragility of randomized clinical trials in shoulder arthroplasty. J Shoulder Elbow Surg 30(8):1787–1793

    Article  PubMed  Google Scholar 

  30. Megafu M, Megafu E (2022) The fragility of statistical findings in distal radius fractures: a systematic review of randomized controlled trials. Injury 53(10):3352–3356

    Article  PubMed  Google Scholar 

  31. Morris SC, Gowd AK, Agarwalla A et al (2022) Fragility of statistically significant findings from randomized clinical trials of surgical treatment of humeral shaft fractures: a systematic review. World J Orthop 13(9):825–836

    Article  PubMed  PubMed Central  Google Scholar 

  32. Muthu S, Ramakrishnan E (2021) Fragility analysis of statistically significant outcomes of randomized control trials in spine surgery: a systematic review. Spine (Phila Pa 1976) 46(3):198–208

    Article  PubMed  Google Scholar 

  33. Narayan VM, Gandhi S, Chrouser K et al (2018) The fragility of statistically significant findings from randomized controlled trials in the urological literature. BJU Int 122(1):160–166

    Article  PubMed  Google Scholar 

  34. Nelms DW, Vargas HD, Bedi RS et al (2022) When the p value doesn’t cut it: the fragility index applied to randomized controlled trials in colorectal surgery. Dis Colon Rectum 65(2):276–283

    Article  PubMed  Google Scholar 

  35. Noel CW, McMullen C, Yao C et al (2018) The fragility of statistically significant findings from randomized trials in head and neck surgery. Laryngoscope 128(9):2094–2100

    Article  PubMed  Google Scholar 

  36. Parisien RL, Dashe J, Cronin PK et al (2019) Statistical significance in trauma research: too unstable to trust. J Orthop Trauma 33(12):e466–e470

    Article  PubMed  Google Scholar 

  37. Parisien RL, Trofa DP, Dashe J et al (2019) Statistical fragility and the role of p values in the sports medicine literature. J Am Acad Orthop Surg 27(7):e324–e329

    Article  PubMed  Google Scholar 

  38. Parisien RL, Ehlers C, Cusano A et al (2021) The statistical fragility of platelet-rich plasma in rotator cuff surgery: a systematic review and meta-analysis. Am J Sports Med 49(12):3437–3442

    Article  PubMed  Google Scholar 

  39. Parisien RL, Danford NC, Jarin IJ et al (2021) The fragility of statistical findings in achilles tendon injury research: a systematic review. J Am Acad Orthop Surg Glob Res Rev 5(9):e21.00018

    PubMed  PubMed Central  Google Scholar 

  40. Parisien RL, Constant M, Saltzman BM, et al (2021) The fragility of statistical significance in cartilage restoration of the knee: a systematic review of randomized controlled trials. Cartilage 13(1_suppl):147S-155S.

  41. Parisien RL, Trofa DP, O’Connor M et al (2021) The fragility of significance in the hip arthroscopy literature: a systematic review. JBJS Open Access 6(4):e21.00035

    Article  PubMed  PubMed Central  Google Scholar 

  42. Parisien RL, Trofa DP, Cronin PK et al (2021) Comparative studies in the shoulder literature lack statistical robustness: a fragility analysis. Arthrosc Sports Med Rehabil 3(6):e1899–e1904

    Article  PubMed  PubMed Central  Google Scholar 

  43. Pascoal E, Liu M, Lin L et al (2022) The fragility of statistically significant results in gynaecologic surgery: a systematic review. J Obstet Gynaecol Can 44(5):508–514

    Article  PubMed  Google Scholar 

  44. Rickard M, Lorenzo AJ, Hannick JH et al (2019) Over-reliance on p values in urology: fragility of findings in the hydronephrosis literature calls for systematic reporting of robustness indicators. Urology 133:204–210

    Article  PubMed  Google Scholar 

  45. Robinson T, Al-Shahwani N, Easterbrook B et al (2020) The fragility of statistically significant findings from randomized controlled trials in pediatric appendicitis: a systematic review. J Pediatr Surg 55(5):800–804

    Article  PubMed  Google Scholar 

  46. Robinson NB, Fremes S, Hameed I et al (2021) Characteristics of Randomized clinical trials in surgery from 2008 to 2020: a systematic review. JAMA Netw Open 4(6):e2114494

    Article  PubMed  PubMed Central  Google Scholar 

  47. Ruzbarsky JJ, Khormaee S, Rauck RC et al (2019) Fragility of randomized clinical trials of treatment of clavicular fractures. J Shoulder Elbow Surg 28(3):415–422

    Article  PubMed  Google Scholar 

  48. Ruzbarsky JJ, Rauck RC, Manzi J et al (2019) The fragility of findings of randomized controlled trials in shoulder and elbow surgery. J Shoulder Elbow Surg 28(12):2409–2417

    Article  PubMed  Google Scholar 

  49. Ruzbarsky JJ, Khormaee S, Daluiski A (2019) The fragility index in hand surgery randomized controlled trials. J Hand Surg Am 44(8):698.e1-698.e7

    Article  PubMed  Google Scholar 

  50. Schröder A, Muensterer OJ, Oetzmann VC (2021) Meta-analyses in paediatric surgery are often fragile: implications and consequences. Pediatr Surg Int 37(3):363–367

    Article  PubMed  PubMed Central  Google Scholar 

  51. Shen C, Shamsudeen I, Farrokhyar F et al (2018) Fragility of results in ophthalmology randomized controlled trials: a systematic review. Ophthalmology 125(5):642–648

    Article  PubMed  Google Scholar 

  52. Skinner M, Tritz D, Farahani C et al (2019) The fragility of statistically significant results in otolaryngology randomized trials. Am J Otolaryngol 40(1):61–66

    Article  PubMed  Google Scholar 

  53. Svantesson E, Hamrin SE, Danielsson A et al (2020) Strength in numbers? The fragility index of studies from the Scandinavian knee ligament registries. Knee Surg Sports Traumatol Arthrosc 28(2):339–352

    Article  PubMed  Google Scholar 

  54. Tignanelli CJ, Napolitano LM (2019) The fragility index in randomized clinical trials as a means of optimizing patient Care. JAMA Surg 154(1):74–79

    Article  PubMed  Google Scholar 

  55. Shochet LR, Kerr PG, Polkinghorne KR (2017) The fragility of significant results underscores the need of larger randomized controlled trials in nephrology. Kidney Int 92(6):1469–1475

    Article  PubMed  Google Scholar 

  56. Docherty KF, Campbell RT, Jhund PS et al (2017) How robust are clinical trials in heart failure. Eur Heart J 38(5):338–345

    CAS  PubMed  Google Scholar 

  57. Chase KB, Matt VB (2017) Unbreakable? An analysis of the fragility of randomized trials that support diabetes treatment guidelines. Diabetes Res Clin Pract 134:91–105

    Article  Google Scholar 

  58. Adie S, Harris IA, Naylor JM et al (2017) The quality of surgical versus non-surgical randomized controlled trials. Contemp Clin Trials Commun 5:63–66

    Article  PubMed  Google Scholar 

  59. Maggard MA, O’Connell JB, Liu JH et al (2003) Sample size calculations in surgery: are they done correctly? Surgery 134(2):275–279

    Article  PubMed  Google Scholar 

  60. Ioannidis JPA (2018) The proposal to lower p value thresholds to 005. JAMA 319(14):1429–1430

    Article  PubMed  Google Scholar 

  61. Lydersen S, Pradhan V, Senchaudhuri P et al (2007) Choice of test for association in small sample unordered r x c tables. Stat Med 26(23):4328–4343

    Article  CAS  PubMed  Google Scholar 

  62. Yusuf S, Peto R, Lewis J et al (1985) Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis 27(5):335–371

    Article  CAS  PubMed  Google Scholar 

  63. Bomze D, Asher N, Hasan AO et al (2020) Survival-inferred fragility index of phase 3 clinical trials evaluating immune checkpoint inhibitors. JAMA Netw Open 3(10):e2017675

    Article  PubMed  PubMed Central  Google Scholar 

  64. ESCP EAGLE Safe Anastomosis Collaborative. (2021) ESCP Safe Anastomosis ProGramme in CoLorectal SurgEry (EAGLE): study protocol for an international cluster randomised trial of a quality improvement intervention to reduce anastomotic leak following right colectomy. Colorectal Dis 23(10):2761-2771. doi:https://doi.org/10.1111/codi.15806

  65. Andrade C (2020) The use and limitations of the fragility index in the interpretation of clinical trial findings. J Clin Psychiatr 81(2):20f13334

    Article  Google Scholar 

Download references

Acknowledgements

This work was not supported by a federal or commercial grant, and there were no other authors who were of direct help in the reported work.

Author information

Authors and Affiliations

  1. Michael DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada

    Prushoth Vivekanantha

  2. Division of Orthopaedic Surgery, Postgraduate Medical Education, University of Toronto, Toronto, ON, Canada

    Ajay Shah & Graeme Hoit

  3. Division of Orthopaedic Surgery, Department of Surgery, McMaster University, Hamilton, ON, Canada

    Olufemi Ayeni

  4. Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, 149 College St, Toronto, ON, M5T 1P5, Canada

    Ajay Shah & Daniel Whelan

Authors
  1. Prushoth Vivekanantha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ajay Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Graeme Hoit
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Olufemi Ayeni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel Whelan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ajay Shah.

Ethics declarations

Conflict of interest

No financial support was received for this manuscript preparation and submission, and there are no relevant conflicts of interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vivekanantha, P., Shah, A., Hoit, G. et al. Predictors of Increased Fragility Index Scores in Surgical Randomized Controlled Trials: An Umbrella Review. World J Surg 47, 1163–1173 (2023). https://doi.org/10.1007/s00268-023-06928-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00268-023-06928-3

Search

Navigation