Skip to main content


Log in

Have Levels of Evidence Improved the Quality of Orthopaedic Research?

  • Clinical Research
  • Published:
Clinical Orthopaedics and Related Research®

A CORR Insights® to this article was published on 01 August 2013



Since 2003 many orthopaedic journals have adopted grading systems for levels of evidence (LOE). It is unclear if the quality of orthopaedic literature has changed since LOE was introduced.


We asked three questions: (1) Have the overall number and proportion of Level I and II studies increased in the orthopaedic literature since the introduction of LOE? (2) Is a similar pattern seen in individual orthopaedic subspecialty journals? (3) What is the interobserver reliability of grading LOE?


We assigned LOE to therapeutic studies published in 2000, 2005, and 2010 in eight major orthopaedic subspecialty journals. Number and proportion of Level I and II publications were determined. Data were evaluated using log-linear models. Twenty-six reviewers (13 residents and 13 attendings) graded LOE of 20 blinded therapeutic articles from the Journal of Bone and Joint Surgery for 2009. Interobserver agreement relative to the Journal of Bone and Joint Surgery was assessed using a weighted kappa.


The total number of Level I and II publications in subspecialty journals increased from 150 in 2000 to 239 in 2010. The proportion of high-quality publications increased with time (p < 0.001). All subspecialty journals other than the Journal of Pediatric Orthopaedics and the Journal of Orthopaedic Trauma showed a similar behavior. Average weighted kappa was 0.791 for residents and 0.842 for faculty (p = 0.209).


The number and proportion of Level I and II publications have increased. LOE can be graded reliably with high interobserver agreement. The number and proportion of high-level studies should continue to increase.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others


  1. Altman DG. Practical Statistics for Medical Research. 2nd ed. London, England: Chapman & Hall/CRC; 2006.

    Google Scholar 

  2. Bhandari M, Swiontkowski MF, Einhorn TA, Tornetta P 3rd, Schemitsch EH, Leece P, Sprague S, Wright JG. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of the Journal of Bone and Joint Surgery. J Bone Joint Surg Am. 2004;86:1717–1720.

    PubMed  Google Scholar 

  3. Brophy RH, Gardner MJ, Saleem O, Marx RG. An assessment of the methodological quality of research published in The American Journal of Sports Medicine. Am J Sports Med. 2005;33:1812–1815.

    Article  PubMed  Google Scholar 

  4. Cashin MS, Kelley SP, Douziech JR, Varghese RA, Hamilton QP, Mulpuri K. The levels of evidence in pediatric orthopaedic journals: where are we now? J Pediatr Orthop. 2011;31:721–725.

    Article  PubMed  Google Scholar 

  5. Chang EY, Pannucci CJ, Wilkins EG. Quality of clinical studies in aesthetic surgery journals: a 10-year review. Aesthet Surg J. 2009;29:144–147; discussion 147–149.

    Google Scholar 

  6. Cochrane AL. Archie Cochrane in his own words. Selections arranged from his 1972 introduction to “Effectiveness and Efficiency: Random Reflections on the Health Services” 1972. Control Clin Trials. 1989;10:428–433.

  7. Donegan DJ, Kim TW, Lee GC. Publication rates of presentations at an annual meeting of the American Academy of Orthopaedic Surgeons. Clin Orthop Relat Res. 2010;468:1428–1435.

    Article  PubMed  Google Scholar 

  8. Ellis J, Mulligan I, Rowe J, Sackett DL. Inpatient general medicine is evidence based. A-Team, Nuffield Department of Clinical Medicine. Lancet. 1995;346:407–410.

    Article  PubMed  CAS  Google Scholar 

  9. Esses SI, McGuire R, Jenkins J, Finkelstein J, Woodard E, Watters WC 3rd, Goldberg MJ, Keith M, Turkelson CM, Wies JL, Sluka P, Boyer KM, Hitchcock K. The treatment of symptomatic osteoporotic spinal compression fractures. J Am Acad Orthop Surg. 2011;19:176–182.

    PubMed  Google Scholar 

  10. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA. 1992;268:2420–2425.

    Article  Google Scholar 

  11. Hanzlik S, Mahabir RC, Baynosa RC, Khiabani KT. Levels of evidence in research published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91:425–428.

    Article  PubMed  Google Scholar 

  12. Keith MW, Masear V, Amadio PC, Andary M, Barth RW, Graham B, Chung K, Maupin K, Watters WC 3rd, Haralson RH 3rd, Turkelson CM, Wies JL, McGowan R. Treatment of carpal tunnel syndrome. J Am Acad Orthop Surg. 2009;17:397–405.

    PubMed  Google Scholar 

  13. Kuhn JE. Exercise in the treatment of rotator cuff impingement: a systematic review and a synthesized evidence-based rehabilitation protocol. J Shoulder Elbow Surg. 2009;18:138–160.

    Article  PubMed  Google Scholar 

  14. Lichtman DM, Bindra RR, Boyer MI, Putnam MD, Ring D, Slutsky DJ, Taras JS, Watters WC 3rd, Goldberg MJ, Keith M, Turkelson CM, Wies JL, Haralson RH 3rd, Boyer KM, Hitchcock K, Raymond L. Treatment of distal radius fractures. J Am Acad Orthop Surg. 2010;18:180–189.

    PubMed  Google Scholar 

  15. Loiselle F, Mahabir RC, Harrop AR. Levels of evidence in plastic surgery research over 20 years. Plast Reconstr Surg. 2008;121:207e–211e.

    Article  PubMed  CAS  Google Scholar 

  16. NHS National Institute for Health and Clinical Excellence. Methods for development of NICE public health guidance. Available at: Accessed June 20, 2013.

  17. Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P 3rd, Bhandari M. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87:2632–2638.

    Article  PubMed  Google Scholar 

  18. Richmond J, Hunter D, Irrgang J, Jones MH, Levy B, Marx R, Snyder-Mackler L, Watters WC 3rd, Haralson RH 3rd, Turkelson CM, Wies JL, Boyer KM, Anderson S, St Andre J, Sluka P, McGowan R; American Academy of Orthopaedic Surgeons. Treatment of osteoarthritis of the knee (nonarthroplasty). J Am Acad Orthop Surg. 2009;17:591–600.

    Google Scholar 

  19. Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest. 1986;89(2 suppl):2S–3S.

    PubMed  CAS  Google Scholar 

  20. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–72.

    Article  PubMed  CAS  Google Scholar 

  21. Schmidt AH, Zhao G, Turkelson C. Levels of evidence at the AAOS meeting: can authors rate their own submissions, and do other raters agree? J Bone Joint Surg Am. 2009;91:867–873.

    Article  PubMed  Google Scholar 

  22. Schwartz WB, Gorry GA, Kassirer JP, Essig A. Decision analysis and clinical judgment. Am J Med. 1973;55:459–472.

    Article  PubMed  CAS  Google Scholar 

  23. Thornton J, Alderson P, Tan T, Turner C, Latchem S, Shaw E, Ruiz F, Reken S, Mugglestone MA, Hill J, Neilson J, Westby M, Francis K, Whittington C, Siddiqui F, Sharma T, Kelly V, Ayiku L, Chamberlain K. Introducing GRADE across the NICE clinical guideline program. J Clin Epidemiol. 2013;66:124–131.

    Article  PubMed  Google Scholar 

  24. Turpen RM, Fesperman SF, Sultan S, Borawski KM, Norris RD, Klink J, Sur RL, Breau RH, Krupski TL, Dahm P. Levels of evidence ratings in the urological literature: an assessment of interobserver agreement. BJU Int. 2010;105:602–606.

    Article  PubMed  Google Scholar 

  25. Ubbink DT, Legemate DA. Evidence-based surgery. Br J Surg. 2004;91:1091–1092.

    Article  PubMed  CAS  Google Scholar 

  26. US Preventive Services Task Force. Guide to Clinical Preventive Services: Report of the US Preventive Services Task Force. Collingdale, PA: DIANE Publishing; 1989.

    Google Scholar 

  27. Voleti PB, Donegan DJ, Baldwin KD, Lee GC. Level of evidence of presentations at American Academy of Orthopaedic Surgeons annual meetings. J Bone Joint Surg Am. 2012;94:e50.

    Article  PubMed  Google Scholar 

  28. Voleti PB, Donegan DJ, Kim TWB, Lee GC. Level of evidence: does it change the rate of publication and time to publication of American Academy of Orthopaedic Surgeons presentations? J Bone Joint Surg Am. 2013;95:e2.

    PubMed  Google Scholar 

  29. Wasserman JM, Wynn R, Bash TS, Rosenfeld RM. Levels of evidence in otolaryngology journals. Otolaryngol Head Neck Surg. 2006;134:717–723.

    Article  PubMed  Google Scholar 

  30. Wingfield K, Matheson GO, Meeuwisse WH. Preparticipation evaluation: an evidence-based review. Clin J Sport Med. 2004;14:109–122.

    Article  PubMed  Google Scholar 

  31. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85:1–3.

    Article  PubMed  Google Scholar 

  32. Wupperman R, Davis R, Obremskey WT. Level of evidence in Spine compared to other orthopedic journals. Spine (Phila Pa 1976). 2007;32:388–393.

  33. Xu CC, Côté DW, Chowdhury RH, Morrissey AT, Ansari K. Trends in level of evidence in facial plastic surgery research. Plast Reconstr Surg. 2011;127:1499–1504.

    Article  PubMed  CAS  Google Scholar 

Download references


We thank Richard Gerkin MD, for assistance with the statistical analysis of the data and discussion of model results.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Ryan McLemore PhD.

Additional information

Work on this study was supported by the Herbert Louis Fund (ACM) at the Orthopaedic Research and Education Foundation and by Banner Good Samaritan Medical Center, Phoenix, AZ, USA.

All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research editors and board members are on file with the publication and can be viewed on request.

About this article

Cite this article

Cunningham, B.P., Harmsen, S., Kweon, C. et al. Have Levels of Evidence Improved the Quality of Orthopaedic Research?. Clin Orthop Relat Res 471, 3679–3686 (2013).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: