An evaluation of the quality of statistical design and analysis of published medical research: results from a systematic survey of general orthopaedic journals
The application of statistics in reported research in trauma and orthopaedic surgery has become ever more important and complex. Despite the extensive use of statistical analysis, it is still a subject which is often not conceptually well understood, resulting in clear methodological flaws and inadequate reporting in many papers.
A detailed statistical survey sampled 100 representative orthopaedic papers using a validated questionnaire that assessed the quality of the trial design and statistical analysis methods.
The survey found evidence of failings in study design, statistical methodology and presentation of the results. Overall, in 17% (95% confidence interval; 10–26%) of the studies investigated the conclusions were not clearly justified by the results, in 39% (30–49%) of studies a different analysis should have been undertaken and in 17% (10–26%) a different analysis could have made a difference to the overall conclusions.
It is only by an improved dialogue between statistician, clinician, reviewer and journal editor that the failings in design methodology and analysis highlighted by this survey can be addressed.
- Petrie, A (2006) Statistics in orthopaedic papers. J Bone Joint Surg Br 88: pp. 1121-1136 CrossRef
- Parsons, N, Hiskens, R, Price, CL, Costa, ML (2011) A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br 93: pp. 1154-1159 CrossRef
- Moher, D (1998) CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. JAMA 279: pp. 1489-1491
- von Elm, E, Altman, DG, Egger, M, Pocock, SJ, Gøtzsche, PC (2007) The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. Lancet 370: pp. 1453-1457 CrossRef
- Siebelt, M, Siebelt, T, Pilot, P, Bloem, RM, Bhandari, M, Poolman, RW (2010) Citation analysis of orthopaedic literature: 18 major orthopaedic journals compared for Impact Factor and SCImago. BMC Musculoskelet Disord 11: pp. 4 CrossRef
- Web of Knowledge. [http://wok.mimas.ac.uk/]
- Kilkenny, C, Parsons, N, Kadyszewski, E, Festing, MFW, Cuthill, IC, Fry, D, Hutton, J, Altman, D (2009) Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 4: pp. e7824 CrossRef
- Altman, DG, Bland, JM (1874) Units of analysis. BMJ 1997: pp. 314
- Chow, S-C, Shao, J, Wang, H (2008) Sample size calculations in clinical research. Chapman and Hall, New York
- Schlesselman, JJ (1974) Sample size requirements in cohort and case–control studies of disease. American J Epidemiol 99: pp. 381-384
- Missing data analysis. [http://missingdata.lshtm.ac.uk/]
- Oliver, D, Hall, JC (1989) Usage of statistics in the surgical literature and the 'orphan P' phenomenon. Aust N Z J Surg 59: pp. 449-451 CrossRef
- Altman, DG, Bland, JM (2009) Parametric v non-parametric methods for data analysis. BMJ 338: pp. a3167 CrossRef
- Bland, M (2003) An introduction to medical statistics. OUP, Oxford
- Bland, JM, Altman, DG (1995) Multiple significance tests: the Bonferroni method. BMJ 310: pp. 170 CrossRef
- Perneger, TV (1998) What's wrong with Bonferroni adjustments. BMJ 316: pp. 1236 CrossRef
- Bland M: How to upset the Statistical Referee. [http://www-users.york.ac.uk/~mb55/talks/upset.htm]
- Petrie, A (2010) Statistical power in testing a hypothesis. J Bone Joint Surg Br 92: pp. 1192-1194 CrossRef
- Simunovic, N, Devereaux, PJ, Bhandari, M (2008) Design considerations for randomised trials in orthopaedic fracture surgery. Injury 39: pp. 696-704 CrossRef
- Soucacos, PN, Johnson, EO, Babis, G (2008) Randomised controlled trials in orthopaedic surgery and traumatology: overview of parameters and pitfalls. Injury 39: pp. 636-642 CrossRef
- BMJ Statistics Notes Series. [http://openwetware.org/wiki/BMJ_Statistics_Notes_series]
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/12/60/prepub
- An evaluation of the quality of statistical design and analysis of published medical research: results from a systematic survey of general orthopaedic journals
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
BMC Medical Research Methodology
- Online Date
- April 2012
- Online ISSN
- BioMed Central
- Additional Links
- Author Affiliations
- 1. Warwick Medical School, University of Warwick, Coventry, CV2 2DX, UK
- 2. Public Health, Epidemiology and Biostatistics Group, University of Birmingham, Birmingham, B15 2TT, Uk
- 3. University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK