Reviewing for Clinical Orthopaedics and Related Research

Abstract

Peer review in science was established in the 17^th Century and while not without detractors and some controversy, has been a mainstay of high-quality scientific publications ever since. Most believe peer review adds substantially to the value of papers that achieve publication. However, in practice, peer review can be practiced with varying degrees of rigor and the value of the review depends on rigor. The two primary tasks of a reviewer are to determine whether the manuscript makes a substantial contribution (in an age of information overload) and to determine whether there are any “fatal” flaws. If the reviewer recommends rejection, then he or she need only note the major flaws. If, however, the material is sufficiently novel and would substantially add to the literature, the reviewer’s secondary task is to ensure completeness and clarity by noting information that should be added and identifying unclear points; in these cases more detailed reviews are merited. To achieve this task, the reviewer must ask numerous questions related to the background and rationale, questions or purposes, study design and methods, findings, and synthesis with the literature. In this brief review I outline such key questions. An invitation to review is an honor and reflects the confidence of the editor in the reviewer’s expertise and accomplishments. Given proper reviews and recommendations, the majority of authors believe peer review adds great value to their papers and the reviewer makes contributions to the community and their own knowledge.

“It is what we think we know that keeps us from learning.”

Claude Bernard

Introduction

Peer review is an old and time-tested process to ensure a published paper contributes to scientific truth, and in a way to ensure the work reflects a substantial contribution. First used in the modern sense in the Philosophical Transactions of The Royal Society in 1665, it was routinely used beginning in 1731 in Medical Essays and Observations published by the Royal Society of Edinburgh [1]. Although many philosophical and practical arguments against its use have been proposed [1, 7], each argument has counterarguments and none has prevented its widespread use throughout scientific disciplines. Most would argue peer review helps ensure the best papers are published, reduces clutter in the literature, and adds to value to manuscripts that ultimately are published (there are few papers that cannot be improved by review and revision).

Bronowski emphasized that science seeks truth [4]. Publishing scientific work is a consequential responsibility of all investigators to materially extend the body of knowledge and lead us closer to truth: no work should be published or given serious credence which does not meet these two criteria. Although confirmatory repetition of initial observations is crucial to establishing truth, unnecessary repetition is unethical and wasteful. Kuhn [5] distinguished “normal science” from “revolutionary science.” The former builds on what we already know whereas the latter reflects a “paradigm shift” in which old concepts are abandoned for new explanations. The majority of science lies in the former domain, and so the question always arises whether a given study adds sufficiently and substantively new or confirmatory information to warrant publication. The reviewer assumes some responsibility for ensuring substantial contribution, particularly given the immensity of material being published, and that is his or her first task.

The reviewer’s second task is to ensure the author(s) has rigorously adhered to the traditional steps of the scientific method and the study has no “fatal” flaws. Fatal flaws (ie, those which preclude answering the question or hypothesis) must be distinguished from nonfatal flaws. The former include errors in logic (eg, in the formulation of the question or in experimental design) and invalid or inappropriate approaches (eg, incorrect method or application of method); no alterations in presentation can overcome these flaws. Nonfatal flaws are those that can be corrected either through additional data or modification of presentation (eg, unclear rationale, inadequate review of and synthesis with the literature).

If there are fatal flaws the reviewer should recommend rejection and need make only general comments noting major deficiencies (such as conclusions not supportable by the study design or data, inadequately rigorous methods to address the questions); detailed specific comments (particularly related to individual sentences) usually are not required. Reasons for rejection must be posed in objective, substantive, nonjudgmental terms. If a reviewer recommends conditional acceptance (ie, no fatal flaws), they may supplement general comments with specific points by line number (or page and paragraph when line numbers are absent), and raise questions when a statement is unclear or a description insufficient. Further questions may be posed when reviewers believe a reader who might be interested in the work, but who may not be an expert, will be confused. All reviews should be constructive and diplomatic.

While determining whether a given work leads us closer to truth, the reviewer must blind his- or herself from some inevitable biases: biases of scientific perspective and beliefs, biases arising from choices of approaches, biases toward particular authors. Minimizing bias is one of the most difficult aspects of reviewing, for it is not only a natural human tendency but arises from our own investment of time and energy in particular problems and approaches. Yet, we must overcome such biases to the extent possible.

Most scientific studies follow a specific pattern: Introduction and questions, Materials and Methods, Results, Discussion. The background and questions or purposes (in an Abstract and Introduction) should persuade the reviewer (and reader) the work adds sufficiently to the literature to warrant publication (the first task). The Materials and Methods should persuade the reviewer there are no fatal flaws in logic or approach (the second task) and because reproducibility is a pillar of science the methods should contain sufficient detail that another investigator could replicate the study. Each section should be evaluated with specific questions.

Introduction (formulation of questions, hypotheses, aims)

• (1) Is the rationale (based on cited critical observations, rigorous data, or accepted opinion) for the questions or purposes provided? Is the logic of that rationale clear? (The logic of a scientific paper might take the form of an Aristotelian modal syllogism: If A, and B, and C, then D, where A, B, and C are known observations and D is the question, hypothesis, or purpose [2].)

• (2) Do the authors pose clear questions, hypotheses, or purposes (goals, objectives, aims)? Clear questions and unambiguous hypotheses most efficiently advance science.

• (3) Once posed, is the question novel? Do the authors document novelty by proper citation? If the question is not novel, have others reported similar work, whether or not cited? If the work essentially repeats that of others, does it confirm previously unconfirmed or inadequately confirmed observations or does it address explicit controversies? Studies addressing issues adequately addressed in the literature do not generally bear repeating and do not pass the reviewer’s first task.

• (4) Are the questions important? Do the authors explore and explicitly state the implications of the answers?

• (5) Are the questions or hypotheses described in terms of independent and dependent variables? (Questions, hypotheses, or aims not posed in terms of explicit design variables cannot be addressed by the study design.)

Materials and Methods

• (1) Is the design of the study clear? A well-crafted study design will necessarily complement the questions or hypotheses or aims. If the questions are appropriately posed, the design and/or descriptions of statistical analyses can be inferred; similarly, if the study design and/or descriptions of the statistical analyses are properly written, the reader can infer the questions or purposes.

• (2) Is the design (including methods) appropriate and adequately rigorous to answer the question or test the hypothesis? Is the question answerable or the hypothesis addressable with the study design? These are the fatal flaw questions related to the reviewer’s second task. The best questions are those that can be unequivocally answered “yes” or “no” within the limitations of the study design and data and the best hypotheses are those that can be falsified [6].

• (3) In clinical studies, are all inclusion and exclusion criteria described and appropriate to the question(s)? In therapeutic studies (including surgery), are indications and contraindications for the treatments clearly described?

• (4) Do the authors have adequate numbers of patients, animals, or specimens to address the questions or purposes? In the case of rare conditions or rare events where a single study cannot likely address the questions, can the study materially contribute to some future systematic review or meta-analysis? Have they performed an a priori power analysis where there are multiple cohorts or groups?

• (5) In clinical studies with more than one group, are the groups demonstrably comparable (with use of proper statistics where appropriate)?

• (6) Are the controls adequately described? Are they appropriate controls?

• (7) Are all critical methods of assessment of dependent variables valid and reproducible? Are they described in adequate detail? Could another observer reproduce the data based on the description? Do the authors fully disclose sources and methods?

• (8) Do the authors justify the choice of statistical tests (considering assumptions of each)? (Again, a reader should be able to ascertain the questions and study design from the description of the statistical analyses.)

Results

• (1) Are the observations clearly presented and in order of the questions, purposes, or hypotheses? (In good writing, key findings will be apparent by reading only each sentence in the first paragraph of the Results section.)

• (2) Does the presentation of the findings persuade the reader the data have been rigorously obtained and reported? Do the authors report variability of the data in an appropriate fashion?

• (3) Are all essential observations arising from the experimental design reported (ie, do the authors omit or inadequately present potentially critical data)? On the other hand, do the authors present only essential data for answering the questions? (In the case of vaguely worded purposes, many findings can be - and often are - presented which do not directly lead the reader to some coherent view or explanation or conclusion [3].)

• (4) Do the findings unambiguously answer the question(s) or address the purpose(s) or hypothesis(es)?

Discussion (interpretation and synthesis of the results)

• (1) Are the assumptions, limitations, and source of bias adequately described? When describing the limitations, do the authors persuade the reader they do not seriously jeopardize the conclusions?

• (2) Do the authors adequately synthesize the observations with those in the literature? Are representative relevant past observations included in the Discussion, or do the authors selectively cite only a few and perhaps biased range of papers? Have the authors overlooked critical references?

• (3) Do the data support past observations or compare appropriately with published data? If not, do the authors adequately explore and explain any contradictions?

• (4) Does the synthesis of the authors’ data and that in the literature support all conclusions made by the authors?

Acknowledgements

• (1) Do the authors fully disclose sources of financial support and/or other sources of potential conflict of interest or bias?

• (2) Do the authors acknowledge individuals who might have contributed to the manuscript but might not meet criteria for authorship; and do the authors include every individual who would meet criteria for authorship?

Tables

• (1) Do the titles briefly describe the data or the question or purpose?

• (2) Are the data self-explanatory without reference to the text?

• (3) Are data in the table consistent with those in the text? Do the data supplement and complement, or merely repeat data in the text?

• (4) Are the tables clearly numbered?

Figures

• (1) Are all figures and illustrations necessary to address the key questions or purposes?

• (2) Are the figures clear and of publishable quality?

• (3) Do the legends adequately describe the figures and in the case of data figures or tables, do they include the key point?

• (4) Are the figures clearly numbered?

Discussion

By following these relatively straightforward principles, the reviewer will provide a valuable service to the scientific community by ensuring novel and valid material is made available while ensuring inappropriate or trivial material does not clutter the literature. Reviewers can be confident well-prepared reviews contribute to the literature: greater than 95% of our surveyed authors believe the process substantially enhanced the value of their manuscripts.

Siegelman, an editor of Radiology, suggested reviewers could be broadly considered as zealots or assassins [8]. Zealots were those who support their field and tend to believe that any manuscript in an area deserves publication whereas assassins are those who tend to believe only their work is important and that of others in a field is inferior. Both may do disservice to peer review: the former because they are inadequately critical and the latter because they fail to acknowledge the contributions of others.

Assuming no fatal flaws, these are the most common problems I see with manuscripts: (1) failure to review the literature before embarking on a study to determine whether and how the work substantively adds to the literature; (2) failure to ensure the questions, methods, statistics, and answers are clear and coherent; (3) failure to clearly and adequately describe the study design (which must be consistent with the questions); (4) in the case of clinical studies failure to describe the patient population in sufficient detail so a reader could determine all relevant aspects of bias; and (5) failure to appropriately synthesize relevant literature.

These are the most common problems I see with reviews: (1) failure to identify fatal flaws; (2) failure to identify and communicate to authors additional key information required to create a high-quality manuscript; and (3) failure to identify and denote statements requiring clarification. Depending on the complexity of the manuscript, most good and comprehensive reviews for conditionally accepted manuscripts will range from one-half to two pages.

I suspect many, if not most reviewers enjoy the process: it keeps them current with potentially new ideas or data, it contributes to their own thinking, and it gives them a sense of contributing to the field. Do clinicians and scientists have a responsibility to review? I would argue they do, realizing they are busy and have many other commitments. Further, surveys of our reviewers suggest greater than 70% believe their comments contribute to scientific development in the field and a majority believe it is a responsibility. Most individuals invited to review have published many articles in the field for which they have received an invitation. Past scientific experience contributes to expertise, and most editors will extend review invitations to experts. If reviewers have received value from peer review of their own works, then they have a responsibility to contribute value to the papers of others. Most peer-review journals receive more manuscripts than they can publish, and most that are published will be reviewed by multiple experts. Thus, for each paper published, many reviews will be required. That being the case, any expert who publishes can anticipate they will be extended many invitations for each paper they publish, and they have a duty to review in a responsible and unbiased (as possible) manner.

An invitation to peer review is, in my view, an honor: it implies the editor has sufficient trust and confidence in the reviewer’s expertise to competently identify the key strengths and weaknesses of a paper. Accepting an invitation to review implies a trust that the authors’ material and the reviewer’s opinions will remain confidential and that the reviewer will provide an unbiased and competent review of the authors’ work. Proper review of manuscripts contributes to authors, the community, and the reviewer’s own knowledge.