Clinical Orthopaedics and Related Research®

, Volume 473, Issue 7, pp 2163–2165 | Cite as

Editorial: “Pencil and Paper” Research? Network Meta-analysis and Other Study Designs That Do Not Enroll Patients

  • Seth S. LeopoldEmail author

Surgeons like to get their hands on things. Laparoscopy and arthroscopy—two tools that distanced the surgeon from the subject—were met with derision or worse by those who did not see their potential [8, 11]. So it goes for new tools in surgical science. When I speak with groups of orthopaedic surgeons, I am surprised by the frequency with which they deride meta-analyses—which are not so new anymore—as “pencil-and-paper research.” Surgeons seem to prefer studies that they can get their hands around: Give us a good randomized trial comparing treatment A to treatment B any day, they say, over some sort of mathemagical approach to data pooling.

But our approaches to understanding, processing, and using new evidence in practice continue to evolve. Neglecting these approaches is no better than turning one’s back on a proven surgical technique simply because it was not in wide use when we were residents. It is much better to learn how to use it. Meta-analysis allows us to pool the results of trials performed in different practice settings and patient populations, and so adds an element of external validity (generalizability) missing from randomized controlled trials. Since a well-performed meta-analysis starts with a thorough search and evaluation process, it also will bring to light some studies that individual readers may have missed, given that most of us cannot keep up with the pace with which research is being published. For these reasons and others, leaders in the evidence-based medicine movement, including the Centre for Evidence-Based Medicine [1] and The Cochrane Collaboration [13], put meta-analysis at the top of the pyramid, making it the focus of what they do.

Traditional meta-analysis, though, has a major shortcoming: It can only compare interventions that have been directly evaluated against one another in the studies whose data are being pooled. If individual trials have compared treatment A to treatment B, and treatment B has been compared to treatment C, but no head-to-head trials have compared A to C, a traditional meta-analysis must remain silent on whether treatment A or treatment C is more effective.

Enter network meta-analysis, a new tool that provides a robust means to compare interventions even when they have not been tried directly against one another in individual studies. This is an important advantage, and one worthy of our attention—even for those of us who are not methodologists or clinical researchers. Results from network meta-analyses will affect the decisions we make for our patients, and the way payors compensate surgeons for their services.

But as may be evident, network meta-analyses involve even more methodological heavy lifting compared to conventional meta-analyses. There is perhaps even more opportunity for studies of this design to confuse readers (or worse, to mislead them).

To minimize this likelihood, and to maximize the utility—and readers’ enjoyment—of this important new approach to analyzing clinical evidence, we are pleased to offer a pair of “User’s Guides” [2, 6] to help readers understand not just this article but any future network meta-analyses that come along. Alongside these User’s Guides, we also present a well-performed network meta-analysis on the care of a common fracture [7]. We hope that readers will take advantage of this opportunity to try something new, and to learn. Put the tibia fracture network meta-analysis on the left side of your desk (even if you do not treat tibial fractures), and the User’s Guides on the right, and dive in.

We are especially proud that this important work comes from a group led by CORR’s Deputy Editor for Evidence-based Orthopaedics, Mohit Bhandari MD, PhD. This is the same partnership that brings us the Cochrane in CORR ® column [4, 5], which we publish in alternate months in our Journal. That regular column synthesizes the latest and most-important musculoskeletal reviews from the Cochrane group, and is a must-read for the orthopaedic surgeon who wishes to stay current.

We are not the only ones who believe that network meta-analyses should influence how we read and practice. The American Academy of Orthopaedic Surgeons has used this tool to help support a number of its clinical practice guidelines in the last several years [9, 10], and has invested heavily in this approach. This is a good choice, as network meta-analysis is powerful, and often is the only suitable tool for the job. But since clinical practice guidelines are expected to play a larger role in medical decision-making, medical policy, and reimbursement [3, 12], surgeons must be facile with the tools used in the creation of these guidelines—first among them, network meta-analysis. The ability to evaluate a guideline’s validity and applicability is essential to knowing whether, when, and how to apply it in practice.

If you are familiar with network meta-analyses, I am confident you will enjoy the paper by Foote et al. [7] on tibial fractures in this issue of CORR ®. But if this study design is new to you, I hope you will take advantage of this opportunity to learn how to read and enjoy network meta-analyses by taking advantage of the User’s Guides [2, 6]. This will be time well spent, both because they are good reading, and also because we expect to see many more studies using this design in the months and years to come.


  1. 1.
    Centre for Evidence-based Medicine. Oxford Centre for Evidence-based Medicine 2011 levels of evidence. Available at Accessed on March 31, 2015.
  2. 2.
    Chaudhry H, Foote CJ, Guyatt G, Thabane L, Furukawa TA, Petrisor B, Bhandari M. Network Meta-analysis: Users’ guide for surgeons part II – certainty. [Published online ahead of print April 14, 2015]. Clin Orthop Relat Res. DOI:  10.1007/s11999-015-4287-9.
  3. 3.
    Clancy CM, Cronin K. Evidence-based Decision Making: Global Evidence, Local Decisions. Health Affairs. 2005;24:151–162.PubMedCrossRefGoogle Scholar
  4. 4.
    Evaniew N, Madden K, Bhandari M. Cochrane in CORR ®: Arthroplasties (with and without bone cement) for proximal femoral fractures in adults. Clin Orthop Relat Res. 2014;472:1367–1372.PubMedCentralPubMedCrossRefGoogle Scholar
  5. 5.
    Evaniew N, Madden K, Bhandari M. Cochrane in CORR ®: Arthroplasty versus fusion in single-level cervical degenerative disc disease. Clin Orthop Relat Res. 2014;472:802–808.PubMedCentralPubMedCrossRefGoogle Scholar
  6. 6.
    Foote CJ, Chaudhry H, Bhandari M, Thabane L, Furukawa TA, Petrisor B, Guyatt G. Network Meta-analysis: Users’ guide for surgeons part I – credibility. [Published online ahead of print April 14, 2015]. Clin Orthop Relat Res. DOI:  10.1007/s11999-015-4286-x.
  7. 7.
    Foote CJ, Guyatt GH, Vignesh N, Mundi R, Chaudhry H, Heels-Ansdell D, Thabane L, Tornetta III, P, Bhandari M. Which surgical treatment for open tibial shaft fractures results in the fewest reoperations? A network meta-analysis. [Published online ahead of print February 28, 2015]. Clin Orthop Relat Res. DOI:  10.1007/s11999-015-4224-y.
  8. 8.
  9. 9.
    Jacobs JJ, Mont MA, Bozic KJ, Della Valle CJ, Goodman SB, Lewis CG, Yates AC Jr, Boggio LN, Watters WC 3rd, Turkelson CM, Wies JL, Sluka P, Hitchcock K. Preventing Venous thromboembolic disease in patients undergoing elective hip and knee arthroplasty. evidence-based guideline and evidence report. Available at Accessed on March 20, 2015.
  10. 10.
    Jevsevar DS, Brown GA, Jones DL, Matzkin EG, Manner PA, Mooar P, Schousboe JT, Stovitz S, Sanders JO, Bozic KJ, Goldberg MJ, Martin WR 3rd, Cummins DS, Donnelly P, Woznica A, Gross L; American Academy of Orthopaedic Surgeons. Treatment of osteoarthritis of the knee. Evidence-based guideline. Available at: Accessed on March 20, 2015.
  11. 11.
    Page B. Chapter 22: Nezhat & the rise of advanced operative video-laparoscopy. Available at Accessed March 19, 2015.
  12. 12.
    Shea KG. EBM, CPGs, and OKO. Available at Accessed on March 20, 2015.
  13. 13.
    The Cochrane Collaboration. Strategy to 2020 (public access version). Available at Accessed March 31, 2015.

Copyright information

© The Association of Bone and Joint Surgeons® 2015

Authors and Affiliations

  1. 1.Clinical Orthopaedics and Related ResearchPhiladelphiaUSA

Personalised recommendations