Peer review is fundamental to the editorial process, but the vital work of our reviewers is only one component of many in that process. As we begin the new year, this seems a suitable time to give both our readers and our authors a look behind the curtain: What do we know about peer review’s efficacy, can it be improved, and what happens at Clinical Orthopaedics and Related Research ® after the reviews have been completed?
We can trace the origins of peer review as far back as the 18th century [6], but consistent, contemporary-looking peer review of scientific and medical manuscripts became common only in the late-20th century. Perhaps the most influential general interest science journal of our time, Nature, only instituted peer review in 1967 [9]. Today, peer review is ubiquitous among the better journals of our specialty, and at CORR ®, we work hard to get it right. Thoughtful observers have raised questions about its efficacy [4, 11], but good evidence substantiates that peer review improves both article quality and clarity [3, 10]. Insofar as peer review is a decentralized system that depends in large measure on the work of volunteers, no doubt it can be improved, and this remains an area of active research. Simple reviewer training does not seem to make an impact [5], while certain labor-intensive add-ons might [1]. At CORR ®, we are developing a systematic refinement to guide reviewers through the editorial process. The idea is to create a tool that can walk a reviewer through the key elements of each article in such a way that the reviewer does not need to remember lessons learned in a lecture or training session. We hope that this tool, which will be used during the “routine” review process, may result in improved quality and consistency of reviews compared to simple mentoring or the implementation of still-more-rigid guidelines.
Since we depend on peer review to make good decisions, and because the process requires a considerable investment of time and effort, it is important to ask whether peer review can be improved upon. Alternatives to traditional review include open peer review (allowing authors to know the identities of reviewers), postpublication peer review, and portable peer review, among many others. We need to remain mindful of the shortcomings of traditional peer review, open minded about other options, and willing to invest in thoughtful experiments on the topic when appropriate. CORR ® is currently involved with a large-scale effort to investigate modifiable nonscientific elements of manuscript presentation that can influence reviewers’ decisions, and we hope to have results to share on the topic in the next year or so.
Although high-quality peer review is essential to a robust editorial process, it is neither a direct democracy nor an end in itself. On the former, thoughtful editors must do more than count votes, if for no other reason than reviewers often disagree about the applicability or validity of a paper’s methods, research questions, and conclusions. Rather than tallying opinions, editors need to read the reviews in detail, and use their content to determine whether articles are interesting, important, and methodologically sound. Quality reviews provide essential subject matter expertise to guide good editorial decisions. But reviewers’ manuscript-disposition recommendations are advisory to the editors; they do not constitute a binding vote of any kind. One substantive review identifying an article’s irremediable methodological flaws can and should sway a decision, even if two other more-cursory reviews missed those flaws and favored publication. While we commonly request additional reviews even after several have come in, we generally do so not to “break ties,” but to ensure we have a balanced perspective from the best possible experts on whether a paper is interesting enough to deserve the attention of our readers, or to verify the validity of subspecialized methodological elements.
An effective editorial process begins with peer review, but it does not end there. Reviewers with special subject-matter knowledge help us decide when papers on unusual topics are important or interesting enough to warrant attention from our readers. They also offer invaluable technical guidance on complex or uncommon methodological elements. But most papers study familiar topics using straightforward study designs. Specifically, the large majority of orthopaedic papers evaluate treatment approaches, and the large majority of those are case series [2], the “bones” of which are relatively similar [8] regardless of whether the topic is hip replacement or carpal tunnel release. Selection bias and study inclusion criteria, loss to followup, and outcomes assessment are essential to the validity of studies about surgical treatments, and good editors must ask about these elements whether or not the reviewers have. Editors should ensure that studies do not overstate their conclusions, or fail to disclose and discuss key limitations. Finally, editors work with authors to ensure that the presentation is as clear as it can be. Corporate standards for methodological rigor, completeness, and clarity are central to the integrity of a journal’s brand, and so are squarely within the purview of its editors. Editors who ask questions beyond those posed by reviewers are not violating the spirit of peer review, they are completing an important process.
Scientists, reviewers, and editors, will inevitably err. The last step in the process – no less important than any other – is the evaluation of new work by a thoughtful community of readers. Ideas (including those in papers that have passed through peer review), in time, get replaced – either incrementally or in revolutionary fashion – through further experiments that may fill in the blanks, correct the errors, or even shift the paradigm [7]. Unless something changes substantially about the way scientists evaluate new ideas, these new ideas themselves probably will pass through peer review. We think that is a good thing.
Often, the first inkling we get that an idea in an article is not quite right comes in the form of a letter to the editor. Send yours to eic@clinorthop.org.
References
Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O’Callaghan A, Kostov B, García L, Cirugeda L, Altman DG, González JA, Sànchez JA, Miras F, Urrutia A, Fonollosa V, Rey-Joly C, Vilardell M. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: Masked randomised trial. BMJ. 2011;343:d6783.
Cunningham BP, Harmsen S, Kweon C, Patterson J, Waldrop R, McLaren A, McLemore R. Have levels of evidence improved the quality of Orthopaedic Research? Clin Orthop Relat Res. 2013;471:3679–3686.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121:11–21.
Horton R. Genetically modified food: Consternation, confusion, and crack-Up. Med J Aust. 2000;172:148–148.
Houry G, Green S, Callaham M. Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Med Educ. 2012;12:83.
Kronick DA. Peer review in 18th-century scientific journalism. JAMA. 1990;263:1321–1322.
Kuhn TS. The Structure of Scientific Revolutions, 2nd ed. Chicago, IL: The University of Chicago Press; 1970:166–173.
Leopold SS. Let’s Talk About Level IV: The Bones of a Good Retrospective Case Series. Clin Orthop Relat Res. 2013. 471;353–354.
Nature Publishing Group. History of the journal Nature. Available at: http://www.nature.com/nature/history/timeline_1960s.html. Accessed September 22, 2014.
Roberts JC, Fletcher RH, Fletcher SW. Effects of peer review and editing on the readability of articles published in Annals of Internal Medicine. JAMA. 1994;272:119–121.
Smith R. Peer review: A flawed process at the heart of science and journals. J R Soc Med. 2006;99:178–182.
Acknowledgment
I am grateful to Lee Beadling BS, whose thoughtful suggestions improved this essay.
Author information
Authors and Affiliations
Corresponding author
Additional information
The author certifies that he, or any members of his immediate family, has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
The opinions expressed are those of the writers, and do not reflect the opinion or policy of CORR ® or the Association of Bone and Joint Surgeons®.
About this article
Cite this article
Leopold, S.S. Editorial: Peer Review and the Editorial Process – A Look Behind the Curtain. Clin Orthop Relat Res 473, 1–3 (2015). https://doi.org/10.1007/s11999-014-4031-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11999-014-4031-x