An evaluation of the quality of statistical design and analysis of published medical research: results from a systematic survey of general orthopaedic journals
The application of statistics in reported research in trauma and orthopaedic surgery has become ever more important and complex. Despite the extensive use of statistical analysis, it is still a subject which is often not conceptually well understood, resulting in clear methodological flaws and inadequate reporting in many papers.
A detailed statistical survey sampled 100 representative orthopaedic papers using a validated questionnaire that assessed the quality of the trial design and statistical analysis methods.
The survey found evidence of failings in study design, statistical methodology and presentation of the results. Overall, in 17% (95% confidence interval; 10–26%) of the studies investigated the conclusions were not clearly justified by the results, in 39% (30–49%) of studies a different analysis should have been undertaken and in 17% (10–26%) a different analysis could have made a difference to the overall conclusions.
It is only by an improved dialogue between statistician, clinician, reviewer and journal editor that the failings in design methodology and analysis highlighted by this survey can be addressed.
- Petrie A: Statistics in orthopaedic papers. J Bone Joint Surg Br 2006, 88:1121–1136. CrossRef
- Parsons N, Hiskens R, Price CL, Costa ML: A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br 2011, 93:1154–1159. CrossRef
- Moher D: CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. JAMA 1998, 279:1489–1491.
- von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC: The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. Lancet 2007, 370:1453–1457. CrossRef
- Siebelt M, Siebelt T, Pilot P, Bloem RM, Bhandari M, Poolman RW: Citation analysis of orthopaedic literature: 18 major orthopaedic journals compared for Impact Factor and SCImago. BMC Musculoskelet Disord 2010, 11:4. CrossRef
- Web of Knowledge. [http://wok.mimas.ac.uk/]
- Kilkenny C, Parsons N, Kadyszewski E, Festing MFW, Cuthill IC, Fry D, Hutton J, Altman D: Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 2009, 4:e7824. CrossRef
- Altman DG, Bland JM: Units of analysis. BMJ 1874, 1997:314.
- Chow S-C, Shao J, Wang H: Sample size calculations in clinical research. New York: Chapman and Hall; 2008.
- Schlesselman JJ: Sample size requirements in cohort and case–control studies of disease. American J Epidemiol 1974, 99:381–384.
- Missing data analysis. [http://missingdata.lshtm.ac.uk/]
- Oliver D, Hall JC: Usage of statistics in the surgical literature and the 'orphan P' phenomenon. Aust N Z J Surg 1989, 59:449–451. CrossRef
- Altman DG, Bland JM: Parametric v non-parametric methods for data analysis. BMJ 2009, 338:a3167. CrossRef
- Bland M: An introduction to medical statistics. Oxford: OUP; 2003.
- Bland JM, Altman DG: Multiple significance tests: the Bonferroni method. BMJ 1995, 310:170. CrossRef
- Perneger TV: What's wrong with Bonferroni adjustments. BMJ 1998, 316:1236. CrossRef
- Bland M: How to upset the Statistical Referee. [http://www-users.york.ac.uk/~mb55/talks/upset.htm]
- Petrie A: Statistical power in testing a hypothesis. J Bone Joint Surg Br 2010, 92:1192–1194. CrossRef
- Simunovic N, Devereaux PJ, Bhandari M: Design considerations for randomised trials in orthopaedic fracture surgery. Injury 2008, 39:696–704. CrossRef
- Soucacos PN, Johnson EO, Babis G: Randomised controlled trials in orthopaedic surgery and traumatology: overview of parameters and pitfalls. Injury 2008, 39:636–642. CrossRef
- BMJ Statistics Notes Series. [http://openwetware.org/wiki/BMJ_Statistics_Notes_series]
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/12/60/prepub
- An evaluation of the quality of statistical design and analysis of published medical research: results from a systematic survey of general orthopaedic journals
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
BMC Medical Research Methodology
- Online Date
- April 2012
- Online ISSN
- BioMed Central
- Additional Links
- Author Affiliations
- 1. Warwick Medical School, University of Warwick, Coventry, CV2 2DX, UK
- 2. Public Health, Epidemiology and Biostatistics Group, University of Birmingham, Birmingham, B15 2TT, Uk
- 3. University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK