Abstract
Study Design
Retrospective analysis of peer review comments.
Objectives
To assess the likelihood that comments provided by peer reviewers of one orthopaedic journal would be similar to comments of reviewers from the same journal and a second journal.
Summary of Background Data
The consistency of the peer review process in orthopedic research has not been objectively examined.
Methods
Nine separate clinical papers related to spinal deformity were submitted for publication in major peer-reviewed journals and initially rejected. The exact same manuscripts were then submitted to different journals. All papers were returned with comments from two to three reviewers from each journal. Reviews were divided into distinct conceptual criticisms that were regarded as separate comments. Comments were compared between reviewers of the same journal and to comments from reviewers of the second journal.
Results
When comparing comments from reviewers of the same journal, an average of 11% of comments were repeated (range 0% [0/12] to 23% [3/13]). On average, 20% of comments from the first journal were repeated by a reviewer at the second journal (range 10% [1/10] to 33% [6/18]). If a comment was made by two or more reviewers from the first journal, it had a higher likelihood (43% [6/14]) of being repeated by a reviewer from the second journal.
Conclusion
When an identical manuscript is submitted to a second journal after being rejected, 80% of peer review comments from the first journal are not repeated by reviewers from the second journal. One may question if addressing every peer review comment in a rejected manuscript prior to resubmission is an efficient use of resources. Comments that appear twice or more in the first journal review are more likely to reappear and may warrant special attention from the researcher.
Level of Evidence
Level IV.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Rathwell P, Martyn C. Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain 2000;213:1964–9.
Scharschmidt BF, DeAmicis A, Bacchetti P, et al. Chance, concurrence and clustering: analysis of reviewers’ recommendations on 1000 submissions to the. J Clin Invest 1994;93:1877–80.
Peters DP, Ceci SJ. Peer review practices of psychological journals: the fate of published manuscripts submitted again. Behav Brain Sci 1982;5:187–255.
Sprowson AP, Rankin KS, McNamara I, et al. Improving the peer review process in orthopaedic journals. Bone Joint Res 2013;2:245–7.
Leopold SS. Editorial: peer review and the editorial process—a look behind the curtain. Clin Orthop Relat Res 2015;473:1–3.
Hing CB, Higgs D, Hooper L, et al. A survey of orthopaedic journal editors determining the criteria of manuscript selection for publication. J Orthop Surg Res 2011;6:19.
Donegan DJ, Kim TW, Lee GC. Publication rates of presentations at an annual meeting of the American Academy of Orthopaedic Surgeons. Clin Orthop Relat Res 2010;468:1428–35.
Yalçınkaya M, Bagatur E. Fate of abstracts presented at a National Turkish Orthopedics and Traumatology Congress: publication rates and consistency of abstracts compared with their subsequent full-text publications. Acta Orthop Traumatol Turc 2013;47:223–30.
Ohtori S, Kubota G, Inage K, et al. English publication rate of 3,205 abstracts presented at the Annual Meeting of the Japanese Orthopaedic Association and the Annual Research Meeting of the Japanese Orthopaedic Association. J Orthop Sci 2013;18:1031–6.
Eck JC. Publication rates of abstracts presented at Biennial Meetings of the International Society of Arthroscopy, Knee Surgery and Orthopaedic Sports Medicine. Knee Surg Sports Traumatol Arthrosc 2005;13:426–9.
Okike K, Kocher MS, Nwachukwu BU, et al. The fate of manuscripts rejected by. J Bone Joint Surg Am 2012;94:e130.
Stahel PF, Moore EE. Peer review for biomedical publications: we can improve the system. BMC Med 2014;12:179.
Houry G, Green S, Callaham M. Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Med Educ 2012;12:83.
Author information
Authors and Affiliations
Corresponding author
Additional information
Author disclosures
SEI (none), LMA (other from Eli Lilly, personal fees from Orthobullets, non-financial support from Pediatric Orthopaedic Society of North America, non-financial support from Scoliosis Research Society, outside the submitted work), DLS (grants from Pediatric Orthopaedic Society of North America & Scoliosis Research Society, personal fees from Biomet; Medtronic; Zipline Medical, Inc.; Orthobullets; Grand Rounds (a healthcare navigation company), other from Zipline Medical, Inc., non-financial support from Growing Spine Study Group, Scoliosis Research Society, Growing Spine Foundation, personal fees from Biomet; Medtronic; Johnson & Johnson, other from Medtronic & Biomet (patent holder), personal fees from Wolters Kluwer Health — Lippincott Williams & Wilkins; Biomet Spine, outside the submitted work).
All tables used in this manuscript are used with permission of the Children’s Orthopaedic Center, Los Angeles.
Approval for this study has been granted from the Children’s Hospital Los Angeles Institutional Review Board.
Rights and permissions
About this article
Cite this article
Iantorno, S.E., Andras, L.M. & Skaggs, D.L. Variability of Reviewers’ Comments in the Peer Review Process for Orthopaedic Research. Spine Deform 4, 268–271 (2016). https://doi.org/10.1016/j.jspd.2016.01.004
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1016/j.jspd.2016.01.004