Orthopaedic surgeons base clinical decisions in large measure upon what we read, and the quality of what we read depends critically upon the effectiveness of peer review. But the peer-review process has a number of important shortcomings, chief among them being its dependence on the generosity of busy volunteers and large disparities among reviewers in terms of experience, training, and skill.
Because we care so deeply about the quality of the reviews we receive, and for those who perform those reviews on behalf of our authors, the editors of Clinical Orthopaedics and Related Research ® believe it is critical to mitigate these shortcomings. This month, we are proud to release a web-based tool that can make reviewing an article faster yet more thorough. This tool also helps focus reviewers on the elements of each study’s design that are most likely to influence the robustness of its conclusions.
Our new peer-reviewer tool is free and publicly available to all at http://www.clinorthop.org/reviewertool.
Our new reviewer tool draws upon an algorithm we created that points the reviewer to the most-important methodological issues for each study type, and asks questions about importance of the research questions and generalizability of the results, which apply to studies of all designs. In addition, the reviewer tool provides some coaching about how to interpret no-difference (“negative”) studies thoughtfully and fairly, since there is evidence that reviewers have a preference for studies drawing “positive” conclusions [2]. This preference can result in meta-analyses and systematic reviews overestimating the apparent benefits of the treatments surgeons use [7], since those study designs depend on the searchable body of knowledge having no particular preference for positive studies.
The inspiration for this tool was the pioneering and transformative work of David Sackett OC, FRSC, Gordon Guyatt OC, FRSC at McMaster University in Canada, and Drummond Rennie MD, a longtime editor at The Journal of the American Medical Association (JAMA). I first encountered their concepts in the early 1990s in JAMA [5], though some of their ideas had been published even before that [1]. Indeed it was their approach to critical appraisal (a term coined by Dr. Sackett [8]) and evidence-based medicine (a term coined by Dr. Guyatt [3]) that first kindled my own interest in how clinicians know what we think we know, and sometimes—to quote the title of another interesting read—“how we know what isn’t so” [4].
For a deeper dive into the thoughtful analysis of clinical research, I heartily recommend the book-length treatment of this topic presented in Users’ Guides to the Medical Literature – A Manual for Evidence-based Clinical Practice [6], which can help all of us to read more thoughtfully. In the meantime, for the busy reviewer and reader, we at CORR ® offer our new, freely available online tool.
Going forward, we will invite our peer reviewers to use this tool if they find it helpful to do so. Our initial experiences in piloting this project have been quite favorable, both in terms of the feedback we have received from reviewers who have tried it, and the quality of reviews they generated using this tool.
While we believe this reviewer tool can make the work of being a reviewer easier, and while it can help less-experienced reviewers be more effective, we think this tool can help others, including authors, readers, and orthopaedic trainees. Although some of the questions the tool asks necessarily focus on housekeeping elements associated with journal-oriented peer review, most focus on key issues that should be of interest to all. By using this tool, authors can see what kinds of questions our reviewers apply to all articles (Supplemental Table 1; Supplemental material is available with the online version of CORR ®), as well as the specific methodological elements a good reviewer will look for when evaluating each article type (Supplemental Table 2; Supplemental material is available with the online version of CORR ®). Many readers—including trainees, but also those already in practice—confide that they do not know whether an article they read is methodologically robust. It certainly can be challenging. The discerning reader therefore needs to develop his or her own “toolkit” for critical reading. We hope that our online tool will provide a little hand-holding for those looking to exercise the critical-reading muscle. The tool also generates a document summarizing the article that the student or resident can use when presenting at Journal Club.
But above all else, we designed this tool for those individuals who selflessly donate their time in a cause that helps all of us and our patients—the external review of the original research of others as part of the publication process. We are deeply indebted to these individuals, who despite laboratory, clinical, and personal commitments continue to share their expertise with CORR’s authors, editors, and readers. We hope that our tool will help make their important job easier and more effective.
We will continue to modify and improve upon it based on the feedback we receive. Please send your suggestions to EIC@clinorthop.org.
References
Department of Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre. How to read clinical journals: I. Why to read them and how to start reading them critically. Can Med Assoc J. 1981;124:555–558.
Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, Leopold SS. Testing for the presence of positive outcome bias in peer review: A randomized controlled trial. Arch Intern Med. 2010;170:1934–1939.
Evidence-based Medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268: 2420–2425.
Gilovich T. How We Know What Isn’t So. The Fallibility of Human Reason in Everyday Life. New York, NY: The Free Press; 1991.
Guyatt, GH, Rennie D. Users’ guides to the medical literature. JAMA. 1993;270:2096–2097.
Guyatt G, Rennie D, Meade MO, Cook DJ. Users’ Guides to the Medical Literature—A Manual for Evidence-based Clinical Practice, Third Edition. New York, NY: McGraw Hill Education; 2015.
Leopold SS. Editorial: No difference studies make a big difference. Clin Orthop Relat Res. 2015;473:3329–3331.
Sackett DL, Parkes J. Teaching critical appraisal: No quick fixes. Can Med Assoc J. 1998;158:203–204.
Acknowledgments
I would like to thank Lee Beadling BA, Matthew B. Dobbs MD, Mark C. Gebhardt MD, Terence J. Gioe MD, Paul A. Manner MD, Raphaël Porcher PhD, Clare M. Rimnac PhD, and Montri D. Wongworawat MD for their suggestions about how to improve and deploy CORR ® ’s new online reviewer tool. I also would like to thank the CORR ® Board of Trustees for their support of this and other projects under development at our Journal, which we hope will improve how science is performed and reported on.
Author information
Authors and Affiliations
Corresponding author
Additional information
The author certifies that he, or any members of his immediate family, has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
The opinions expressed are those of the writers, and do not reflect the opinion or policy of CORR ® or the Association of Bone and Joint Surgeons®.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Leopold, S.S. Editorial: CORR’s New Peer-Reviewer Tool—Useful for More Than Peer Reviews. Clin Orthop Relat Res 474, 2321–2322 (2016). https://doi.org/10.1007/s11999-016-5050-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11999-016-5050-6