Advertisement

Sample size calculations for randomized clinical trials published in anesthesiology journals: a comparison of 2010 versus 2016

  • Jeffrey T. Y. Chow
  • Timothy P. Turkstra
  • Edmund Yim
  • Philip M. Jones
Reports of Original Investigations

Abstract

Purpose

Although every randomized clinical trial (RCT) needs participants, determining the ideal number of participants that balances limited resources and the ability to detect a real effect is difficult. Focussing on two-arm, parallel group, superiority RCTs published in six general anesthesiology journals, the objective of this study was to compare the quality of sample size calculations for RCTs published in 2010 vs 2016.

Methods

Each RCT’s full text was searched for the presence of a sample size calculation, and the assumptions made by the investigators were compared with the actual values observed in the results. Analyses were only performed for sample size calculations that were amenable to replication, defined as using a clearly identified outcome that was continuous or binary in a standard sample size calculation procedure.

Results

The percentage of RCTs reporting all sample size calculation assumptions increased from 51% in 2010 to 84% in 2016. The difference between the values observed in the study and the expected values used for the sample size calculation for most RCTs was usually > 10% of the expected value, with negligible improvement from 2010 to 2016.

Conclusion

While the reporting of sample size calculations improved from 2010 to 2016, the expected values in these sample size calculations often assumed effect sizes larger than those actually observed in the study. Since overly optimistic assumptions may systematically lead to underpowered RCTs, improvements in how to calculate and report sample sizes in anesthesiology research are needed.

Calculs de la taille des échantillons pour les essais cliniques randomisés publiés dans les journaux d’anesthésiologie : comparaison entre 2010 et 2016

Résumé

Objectif

Même si chaque essai clinique randomisé (ECR) nécessite des participants, la détermination de leur nombre idéal prenant en compte d’une part des ressources limitées et d’autre part la capacité à détecter un effet réel s’avère difficile. Se concentrant sur les ECR à deux groupes, à groupes parallèles, et essais de supériorité publiés dans six journaux d’anesthésiologie, l’objectif de cette étude était de comparer la qualité des calculs de taille d’échantillon pour les ECR publiés en 2010 et en 2016.

Méthodes

Le texte complet de chaque ECR a été analysé en fonction du calcul de la taille de l’échantillon et les hypothèses faites par les investigateurs ont été comparées aux valeurs réelles observées dans les résultats. Des analyses n’ont été pratiquées que pour les calculs de taille d’échantillon qu’il était possible de répliquer, en utilisant une mesure clairement identifiée, continue ou binaire, avec une procédure usuelle de calcul de taille d’échantillon.

Résultats

Le pourcentage des ECR indiquant toutes les hypothèses du calcul de taille de l’échantillon est passé de 51 % en 2010 à 84 % en 2016. La différence entre les valeurs observées dans les études et les valeurs attendues utilisées pour les calculs de taille d’échantillon de la majorité des ECR était habituellement plus de 10 % plus élevée que la valeur attendue, sans véritable amélioration entre 2010 et 2016.

Conclusion

Alors que la présentation des calculs de la taille des échantillons s’est améliorée entre 2010 et 2016, les valeurs attendues dans ces calculs ont souvent supposé des ampleurs d’effet supérieures à celles véritablement observées dans les études. Considérant que des hypothèses excessivement optimistes entraînent un manque de puissance des ECR, des améliorations sur la façon de calculer et présenter la taille des échantillons pour la recherche en anesthésiologie sont nécessaires.

Notes

Acknowledgements

We appreciate the support and advice offered by Neil Klar (PhD, Department of Epidemiology and Biostatistics, The University of Western Ontario) and Janet Martin (PharmD, Department of Anesthesia & Perioperative Medicine and Department of Epidemiology and Biostatistics, The University of Western Ontario).

Conflicts of interest

Philip M. Jones is an Associate Editor at the Canadian Journal of Anesthesia. No other authors have any conflicts of interest.

Editorial responsibility

This submission was handled by Dr. Gregory L. Bryson, Deputy Editor-in-Chief, Canadian Journal of Anesthesia.

Author contributions

Jeffrey T. Y. Chow and Philip M. Jones designed the study, analyzed the data, and wrote the manuscript. Jeffrey T. Y. Chow, Timothy P. Turkstra, Edmund Yim, and Philip M. Jones conducted the study, collected data, and revised the manuscript. Philip M. Jones supervised the study.

Funding

Jeffrey T. Y. Chow is supported by an Ontario Graduate Scholarship: Queen Elizabeth II Graduate Scholarship in Science and Technology (OGS: QEIIGSST) and the funder had no role in any part of the study. Philip M. Jones is supported by internal departmental funding. No other authors received funding for this study.

References

  1. 1.
    Guyatt G, Rennie D, Meade MO, Cook DJ. Users’ Guides to the Medical Literature - A Manual for Evidence-based Clinical Practice. 3rd ed. USA: McGraw-Hill Education; 2015 .Google Scholar
  2. 2.
    Bothwell LE, Greene JA, Podolsky SH, Jones DS. Assessing the gold standard — lessons from the history of RCTs. N Engl J Med 2016; 374: 2175-81.CrossRefPubMedGoogle Scholar
  3. 3.
    Noordzij M, Tripepi G, Dekker FW, Zoccali C, Tank MW, Jager KJ. Sample size calculations: basic principles and common pitfalls. Nephrol Dial Transplant 2010; 25: 1388-93.CrossRefPubMedGoogle Scholar
  4. 4.
    Gupta KK, Attri JP, Singh A, Kaur H, Kaur G. Basic concepts for sample size calculation: critical step for any clinical trials! Saudi J Anaesth 2016; 10: 328-31.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005; 365: 1348-53.CrossRefPubMedGoogle Scholar
  6. 6.
    Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c869.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Campbell MJ, Julious SA, Altman DG. Estimating sample sizes for binary, ordered categorical, and continuous outcomes in paired comparisons. BMJ 1995; 311: 1145-8.CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P. Reporting of sample size calculation in randomised controlled trials: review. BMJ 2009; 338: b1732.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Abdulatif M, Mukhtar A, Obayah G. Pitfalls in reporting sample size calculation in randomized controlled trials published in leading anaesthesia journals: a systematic review. Br J Anaesth 2015; 115: 699-707.CrossRefPubMedGoogle Scholar
  10. 10.
    Jones PM, Chow JT, Arango MF, et al. Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals. Anesth Analg 2017; 125: 1292-300.CrossRefPubMedGoogle Scholar
  11. 11.
    Thomson Reuters. Journal Citation Reports — Anesthesiology — 2016. ISI Web Knowl. Available from URL: http://admin-apps.webofknowledge.com/JCR/JCR (accessed February 2018).
  12. 12.
    Chow J, Jones P. Identifying areas to improve the quality of reporting in randomized clinical trials published in anesthesiology journals: a study protocol for a series of literature surveys assessing quality of trial registration, adherence to abstract reporting guidelines adequacy of sample size calculations, and impact of funding source. Figshare 2016. DOI:  https://doi.org/10.6084/m9.figshare.4490582.v1.Google Scholar
  13. 13.
    Jones PM. SSI: Stata module to estimate sample size for randomized controlled trials. IDEAS — 2010. Available from URL: https://ideas.repec.org/c/boc/bocode/s457150.html (accessed February 2018).
  14. 14.
    Vickers AJ. Underpowering in randomized trials reporting a sample size calculation. J Clin Epidemiol 2003; 56: 717-20.CrossRefPubMedGoogle Scholar
  15. 15.
    Chen H, Zhang N, Lu X, Chen S. Caution regarding the choice of standard deviations to guide sample size calculations in clinical trials. Clin Trials 2013; 10: 522-9.CrossRefPubMedGoogle Scholar
  16. 16.
    Wittes J. Sample size calculations for randomized controlled trials. Epidemiol Rev 2002; 24: 39-53.CrossRefPubMedGoogle Scholar
  17. 17.
    Aberegg SK, Richards DR, O’Brien JM. Delta inflation: a bias in the design of randomized controlled trials in critical care medicine. Crit Care 2010; 14: R77.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Wilson Van Voorhis CR, Morgan BL. Understanding power and rules of thumb for determining sample sizes. Tutor Quant Methods Psychol 2007; 3: 43-50.Google Scholar
  19. 19.
    Bacchetti P. Current sample size conventions: flaws, harms, and alternatives. BMC Med 2010; 8: 17.CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Ciarleglio MM, Arendt CD. Sample size determination for a binary response in a superiority clinical trial using a hybrid classical and Bayesian procedure. Trials 2017; 18: 83.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Bacchetti P, McCulloch CE, Segal MR. Simple, defensible sample sizes based on cost efficiency. Biometrics 2008; 64: 577-85.CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Chan AW, Hrobjartsson A, Jorgensen KJ, Gotzsche PC, Altman DG. Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ 2008; 337: a2299.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Canadian Anesthesiologists' Society 2018

Authors and Affiliations

  1. 1.Department of Epidemiology & BiostatisticsThe University of Western OntarioLondonCanada
  2. 2.Department of Anesthesia & Perioperative MedicineThe University of Western OntarioLondonCanada
  3. 3.Schulich School of Medicine & DentistryThe University of Western OntarioLondonCanada
  4. 4.Rm C3-110 - University HospitalLondon Health Sciences CentreLondonCanada

Personalised recommendations