Getting a manuscript published in a journal is a process involving many steps, some of which may be more transparent than others. In Mesa and Wagner (2019), we sought to “open the door” to processes related to publishing in Educational Studies in Mathematics (ESM). We discussed in detail what happens to a manuscript from the moment it is received until a decision is made, and the involvement of the editors-in-chief and the associate editors, including changes since the journal was first published. We devoted two paragraphs to expectations for reviews drawn from responses to the question “what makes a good review?” Two criteria were mentioned: a good review “helps the editor argue their decision and (...) it helps authors produce a better manuscript” (p. 315). At the time we were writing that editorial, David and Vilma were thinking that an important role for us as editors was to support reviewers in getting better at writing “good reviews.” We also get frequently asked, and especially by junior researchers, what do we consider to be good reviews. Thus, in this editorial, we want to expand on what constitutes a “good review.”

Journals have an important role in growing, improving, and diversifying our field. Journals do that by publishing papers that have been reviewed by a group of knowledgeable peers who can provide an assessment of the work and propose ways to improve it, so as a whole, the result will advance our thinking in the discipline. In this context, reviewing is a key professional practice playing a significant role in such growth. Reviewing not only benefits the field, of course; reviewers benefit from reviewing other colleagues’ work. An invitation to review opens the opportunity to see how the field is changing: reviewers are exposed to current thinking, to connections that authors are making that might be new, to new contexts where known ideas are tested… and they can support new and seasoned researchers in the process of producing a better version of their work. Reviewers broaden their personal horizons in an area of interest to them. Reviewers can also get better in writing manuscripts and in mentoring others in that process, as reviewing creates an opportunity to practice evaluating manuscripts critically and constructively. When reviewers accept an invitation to review a manuscript submitted to the journal, they agree “to support others, especially emerging authors,” who might be new to the field.Footnote 1 Additionally, ESM reviewers benefit from reading the full set of reviews and thus can see how their own comments align with the comments of others with expertise specifically related to the paper they are reviewing.

Editors and reviewers are sometimes considered gatekeepers; and rejection rates of journals seem to stress that aspect of their work. Although some of the work of the editors results in rejecting manuscripts for publication in ESM, we strive to focus on the potential of manuscripts to become published articles; we seek to assist authors in improving their manuscripts. While a deficit perspective focuses on what a manuscript is lacking, instead we, ESM’s editors, promote an asset perspective that highlights the potential of the work and gives guidance on how to address the expectations of readers (cf. Wagner, 2021). We believe that good reviews help create publishable papers, that is, papers that will advance the field.

For the most part, and in our judgment, reviews in ESM tend to be of high quality, respectful, and fair. This editorial addresses our interest in making our processes more transparent and makes explicit what reviewers can do to produce good reviews: reviews that support the editors in making a decision about the overall contribution of a manuscript and that support the author in creating a better paper, a paper that can be eventually published. Such a review acknowledges the time and energy that other colleagues have put into their research and strives to understand the contribution that the authors see the study as making and to consider further options for the study’s possible contribution to the broader international field.

In this editorial, we highlight primarily the peer review process of manuscripts that are handled by the associate editors.Footnote 2 Throughout the piece, we refer to the editor in charge of handling the manuscript (an associate editor or an editor-in-chief) as the handling editor. This editorial has three sections. We start by describing how ESM handles the peer review process of manuscripts sent by authors for consideration. The next section includes the features that the team of current editors has agreed are desirable in a good review. We conclude with some final reflections about writing good reviews.

1 Current practices

In this section, we describe how ESM handles the review process using the Responsible Journals framework proposed by Horbach and Halffman (2019).Footnote 3 The framework—synthesized from a survey that reviewed processes from 361 journals indexed in Web of Sciences and covering a wide range of research fields—proposes 12 attributes of the peer review process. The attributes are stated in a question form and can be organized into four categories: timing and selectiveness, openness of the review, specialization of the review, and technological support in the review. These attributes help describe fully the current peer review process that ESM follows. We note that in doing this exercise, we were curious about the origin of some of these practices and other possibilities that are intriguing for us. These might be a matter of future discussion in our community and might be expanded upon in a future editorial.

1.1 Timing and selectiveness

  1. 1.

    At what stage of the publication process does the review take place? Like most journals, ESM uses a review process prior to publication. Three timing possibilities used by other journals include reviews that are conducted prior to submission (also known as data-free reviews; used to avoid bias against publishing only positive results) or conducted after the publication (a more recent form that seeks to accelerate knowledge exchange). The third possibility is to not require a review.

  2. 2.

    What quality criteria does the journal use for peer review? In ESM, we consider the following criteria: coherence that ties the literature, theory, methodology, and analysis together and lead to warranted claims; insight; anticipated contribution to already-available knowledge within and beyond the discipline; and fit with the journal’s scope.Footnote 4 For empirical studies, coherence refers to the alignment of the several components of the paper: the problem under investigation, the research questions, the theorization and conceptualization, the methods used, and the conclusions. For studies with theoretical or methodological foci, coherence refers to the alignment of the argument throughout their sections. ESM reviews strive to address these criteria; the final decisions made based on recommendations for these aspects are, however, tied to each individual contribution.

Openness of the review

  1. 3.

    What type of reviewers is included in the journal’s peer review process? Horbach and Halffman’s (2019) framework offers six options, including commercial review platforms, editor-in-chief, wider community, editorial committee, external reviewers suggested and selected by editors, and external reviewers suggested by authors. After an initial review, that indicates that the manuscript can be sent to reviewers, ESM invites at least two external reviewers, one of whom is customarily a member of the editorial board. These reviewers are selected and invited by the handling editor who will shepherd the whole review process. For some submissions, such as special issue proposals, we use an editorial committee composed of the editors-in-chief and current or past editors. Solicited manuscripts, such as editorials and commentaries, are usually reviewed by the editors-in-chief with assistance of an advisory editor and at least one associate editor. Book reviews are solicited by the book review editor.

  2. 4.

    To what extent are authors anonymized in the journal’s peer review process? In ESM the author identities are known to the editor and the reviewers; that is, we use a single-blind process of review (see Mesa & Wagner, 2019). We are discussing the possibility of making it optional for authors to have their papers go through a double-blind review if they prefer so.

  3. 5.

    To what extent are reviewers anonymized in the journal’s peer review process? In ESM the three or more chosen reviewers are anonymous both to authors and other reviewers, as well as to readers of the published manuscript. On rare occasions, the reviewers disclose their identity in their reviews.

  4. 6.

    To what extent are the review reports accessible? In addition to being available to authors and editors, the review reports are accessible to the other reviewers of the manuscript, as they are included in the decision letter that the editors send to the author. We do not share reviews with readers nor make them publicly accessible.

  5. 7.

    To what extent does the journal’s review process allow for interaction between reviewers and authors? ESM does not facilitate direct interaction between authors and reviewers. The interaction can be described as indirect. When manuscripts are sent out for review and when authors are invited to send a revised version of the manuscript, authors respond to the reviewers’ comments in the letter they send along with their revised manuscripts. If the revised manuscript is sent out for another round of reviews, the reviewers of the revised manuscript can see the authors’ responses to the comments made to the previous version of the manuscript.

Specialization of the review

  1. 8.

    To what extent is the journal’s review process structured? Until very recently, ESM asked reviewers to consider and answer a list of 16 questions about some general criteria for judgment (see the Appendix in Mesa & Wagner, 2019) and use those to write a review. In practice, few reviewers answered these questions and wrote instead unstructured reviews. Now, reviewers are not asked these questions. They choose how to organize their review and do not have questions or criteria for judgment to follow.

  2. 9.

    To what extent does the journal’s review process use specialist statistical review? We invite at least two experts in the topic of the manuscript, with at least one who can attend to the methodology, and a reviewer who has a broader and general view of the field. We do ask specific reviewers with specific statistical expertise, when there is a technique that is not well known (e.g., Bayesian statistics, network analyses, machine learning).

  3. 10.

    To what extent does the journal accept or use reviews from external sources? We receive manuscripts that have been transferred from other journals, but Springer, as a policy, does not transfer prior decisions or reviews.

Technological support in review

  1. 11.

    What forms of digital tools are used in the journal’s review process? ESM does not use the possible options for digital tools that can assist in the review process, which include automatic reference check, plagiarism detection, assessment of validity and consistency of statistics used, and detection of image manipulation, although Springer has announced that a program that detects plagiarism automatically will be set up in the near future. Occasionally editors receive warnings when there is high overlap in specific papers.

  2. 12.

    To what extent does the journal’s review process allow for commentary after publication of a manuscript? ESM does not proactively provide out-of-channel reader commentary, although occasionally an editor may ask a reviewer of the manuscript for a response. Authors or editors follow up mentions in social media, and readers can write letters to the editors about papers published, which can turn into commentaries that are published.

2 Features of a “good review”

In this section, we present a set of features that we think make a “good review,” which we define in this editorial as useful to the editor for making a decision about publication and useful for the authors to improve their work. We arrived at this set of features through a process that involved all the current editors, through several rounds of discussion. We separate these into several elements: structure, content, and tone.

2.1 Structure of a good review

A review is a piece of scholarly writing. It presents a position, taken by the reviewer, about the qualities of the manuscript reviewed backed by evidence from the manuscript and the expertise that the reviewer brings. In terms of structure, a review has a brief description of what the reviewer understands the paper is about, followed by its strengths and the areas of improvement. A succinct summary (two to three sentences) is ideal.

The brief description highlights what the main points of the paper are. Starting with a summary of the manuscript helps the editor and the authors in framing the comments that the reviewer is making. A short summary also allows the editor to consider the extent to which the reviewer is engaging with the core focus of the submission. On many occasions, a summary that differs from what the authors intend is a signal that a clarification of the focus might be needed from the paper authors. In other occasions, a departure provides an alternative perspective on the possible contribution of the paper.

The strengths and areas of improvement are stated relative to the quality criteria that the journal uses: coherence that allows the literature, theory, methodology, and analysis to come together and lead to warranted claims; insight; anticipated contribution to already-available knowledge within and beyond the discipline; and fit with the journal’s scope. In providing strengths and areas of improvement of the manuscript, the reviewer acknowledges the time and energy that authors have invested in the work, contributing to the fair assessment that is expected of a review. The reviewer also accepts the authors’ perspective even if it is different from their own. The reviewer considers how the authors’, possibly different, approach can contribute to existing discourses in the community. Finally, when providing strengths and areas of improvement, reviewers point out where in the manuscript are those exemplified, adding such specificity.

2.2 Content of the review

We present the elements constituting the content of the review as a list, which was constituted from input by the editors-in-chief, the book review editor, and the associate editors.

  1. 1.

    Contribution. A good review acknowledges explicitly the main contributions of the manuscript by connecting it to current research; such a statement can help authors in situating their contribution in current conversations in the international community. Reviewers are in the best position to do this, given their expertise; they can notice whether the authors have made this connection or whether they have missed key works or discussions that can broaden the contribution beyond their local contexts. This is an important aspect of the journal, which seeks to reach an international community. Reviewers need to think beyond their own interests and perspectives, centering the one proposed by the author, in suggesting connections to current research.

  2. 2.

    Coherence. A good review discusses the coherence of the argument that the authors put forward. There are many ways to address coherence in an empirical paper, by noticing that the research questions align with the study’s theoretical framing and design, the methods chosen to investigate the questions, the analysis performed, the findings, and the conclusions. For a theoretical paper, coherence similarly means alignment among the goal of the paper, the theories articulated, and the arguments made regarding the theories.

  3. 3.

    Theory. A good review discusses the pertinence of the theoretical framing used for the study.

  4. 4.

    Methods. A good review provides an assessment about the methods used in the study; aspects such as type of methodology, participant selection, context in which the study is conducted, processes of data collection and analysis, and steps taken to ensure trustworthiness of the findings (e.g., including reliability of coding or of instruments, or processes to ensure validity of the inferences) are quite important for editors to assess the amount of work that might be needed for the manuscript to be ready for publication. A good review highlights the appropriateness of the methods used and aspects that may cast doubts about the inferences the authors want to make. Assessment of methods can be important even in theoretical papers, as authors explain their approach to developing theory. A good review includes a clear, and well-argued, set of suggestions about how to improve on any of these methodological components.

  5. 5.

    Claims. A good review provides an assessment of the extent to which the claims follow the analysis and the findings. The reviewer will point out specific cases in which there are statements that might not be backed by the analysis or the findings of the analysis or that include interpretations that might be out of the scope.

  6. 6.

    Plausibility. A good review provides an assessment of the plausibility of the interpretations and the extent to which the authors reposition their contribution considering current work. The reviewer can assess how the author has taken up their findings and argued for the contribution they are making.

  7. 7.

    Fit. A good review provides an assessment of the fit with the journal’s scope, noting how the contribution deals with mathematics and education in an international context; the review includes information about how the manuscript fits, or expands, the journal’s scope in ways that the reviewer considers beneficial for the field.

  8. 8.

    Literature. A good review provides suggestions of other research that might inform the work, when such suggestions are in line with and relevant to the authors’ perspectives or when there have been important omissions. Suggestions for relevant research are most appreciated when those support the authors’ argument so that it can be better connected to the international audience; they are least useful when they suggest a paradigm shift for the work.

  9. 9.

    Length. A good review includes ways to make sure that the manuscript stays within the allowed length, including details that are useful to both editors and authors when they come with suggestions for condensing or leaving out current content in the paper.

  10. 10.

    Context. A good review addresses the way in which the context of the study has been described, highlighting the ways in which such content enriches current thinking in the field; this is important because the journal seeks to reach an international audience, and shared meanings cannot be taken for granted.

  11. 11.

    Writing. A good review notices areas in which the writing can be made clearer. ESM receives manuscripts written in English frequently by authors for whom academic English might be a late addition to the other languages that they use in their work. ESM also works with reviewers who are fluent in other languages besides academic English. It is important to have in mind that there are many types of Englishes, and that being a native speaker of one such English is not equivalent to writing proficiently for an academic audience and, conversely, that being a non-native speaker of an English is not equivalent to writing poorly for an academic audience. See also Meaney (2013) and Mesa (2004) for discussions about the use of English in mathematics education research.

2.3 Tone and style of the review

In our discussion about what makes a review good (“helps the editor argue their decision and (...) it helps authors produce a better manuscript”), we propose the following six attributes that, while not related to the content or the structure of the review, contribute significantly to the quality of a review: respectful language, taking authors’ perspective, conciseness, actionable suggestions, and specific assessments. These are proposed in the spirit of reading manuscripts using an asset perspective, even if in the view of the reviewer the manuscript is not publishable in its current state. Each of these attributes contributes to the tone and style that we believe make the review useful.

  1. 1.

    Respectful language. Special attention needs to be devoted to the language used to describe the areas that, in the reviewer’s opinion, deserve attention. The language used needs to be such that the authors will be willing to engage with the suggestions. Reviewers are themselves in a mentorship relationship with the authors: the authors submitting manuscripts to the journal are experts on their own work and the reviewer is a knowledgeable spectator who can provide a different perspective on the manuscript and indicate what could be done to realize the work’s potential. The review uses developmental language, even when the reviewer considers that the manuscript is not ready for publication; by developmental we mean language that upholds the work and at the same time provides, in no ambiguous terms, the areas that need improvement with suggestions for fixing them so that the result is a better, possibly publishable, version. Respect in language also involves awareness of differences across countries and cultures in how criticism is conveyed and the recognition that it may be more difficult to express (and receive) critical points in a language different from one’s native language. When using respectful language, reviewers propose suggestions that uphold, rather than ignore, the authors’ experience in carrying out their work and that are directed at the work, not at the individuals. Reviews that are not respectful make it hard to digest and understand the suggestions and create a negative emotional state that may discourage the author from improving their work.

  2. 2.

    Taking the authors’ perspective. We have mentioned this before; besides their own perspectives, a reviewer takes the authors’ perspective and the vision they had for the study, and thus, the reviewer avoids making suggestions for a different study that the reviewer would rather see. In other words, the suggestions are to be made in light of what the authors intend, rather than proposing directions that might not support a development of those intentions. While comments that suggest further research based on the work are welcome, judging a manuscript against an idealized version of a study that is not the study the author conducted is not useful. The reviewer needs to accept the authors’ perspective (even if they do not agree with it) and be aware that they may have, by the time the manuscript is submitted, little control over the research situation; the review assesses the work on its own terms and uses that perspective to envision a better manuscript.

  3. 3.

    Conciseness. While substantive reviews are appreciated, long reviews that meander or that repeat ideas or that elaborate on peripheral aspects of the work are very difficult to use. After stating a key point, the review explains why it is key, describes how the work addresses it or not, and provides suggestions to make it better.

  4. 4.

    Organization. The review has a structure that makes it easy to follow the points; for example, the reviewer may want to present the most important issues first and then other less important but relevant issues; or they may want to make comments to each of the sections of the manuscript, in the order the author chose.

  5. 5.

    Actionable suggestions. Amidst the criticisms of the manuscript, the reviewer proposes actionable suggestions to address them. While we welcome the full assessment of the quality of the manuscript, concrete ideas for how to address the problematic areas are important. In the content section, we mentioned, as an example, literature that the authors may need to consider to frame their work better. Other concrete ideas include structure and organization of the manuscript, suggestions for alternative analysis, ideas for discussing the findings, or possible contributions that the authors might not have envisioned. In Point 8 under Content of the Review, we also comment on making suggestions for what could be removed from the manuscript to make space for information that the authors have and did not include (this is typical of the methods section).

  6. 6.

    Specific assessments. Useful reviews provide specific rather than just general assessments. While a comment such as “the authors do not include important information in the methods section” could be an appropriate assessment of the manuscript, a good review goes into the specifics of what in the methods section concretely needs more detail.

3 Useful and educative reviews

When we set out to write this editorial, we had in mind several goals. We wanted to:

  • Continue contributing to the transparency of processes of publication in ESM

  • Describe current practices across ESM editors when handling reviews and what they find valuable

  • Present our current thinking about what makes a review useful for the editors and the author

While writing it, we discussed the educative purpose of the reviews for everyone directly involved in reviewing: editors, authors, and reviewers. Editors learn about the submission from different points of view that enhance their own understanding of the work; authors learn about the potential and possibilities of their work in relation to the field; and reviewers learn about current thinking and also about additional perspectives put forth by the other reviewers. Because ESM distributes the reviews along with the decision letter for every manuscript received, reading the letter and what other reviewers said is a practice that reviewers have found beneficial in becoming better at reviewing. The letter is an opportunity to understand how the editor leaned on the reviews to make a decision and give suggestions to the authors; reading the reviews gives the reviewer an opportunity to contrast their own recommendations to those of other experts who reviewed the manuscript, to see whether they are in agreement about the areas that were well done as well as those that needed improvement; the reviewer can also notice whether they missed something that was important in the editor’s decision or something that might not have been as important, in hindsight. The reviewer can also check how the tone and style used differs or aligns with the tone and style used by the other reviewers.

It is important to stress that reviewers’ comments are always taken as suggestions by the editors. The reviewers are advisory to the editors and as such they make suggestions; the editors will determine the relative importance of the comments received across the panel of reviewers in order to write a letter with recommendations for the author that will provide a clear roadmap for improving the manuscript and a rationale for the decision.

In a similar fashion, the handling editor’s letter and the reviewers’ questions and comments are opportunities for the authors to think about their work and how to communicate it to the community served by ESM. While their recommendations are not mandates—rather they are part of a conversation that involves several scholars—it is expected that when authors disagree or do not follow a suggestion made by the editor or a reviewer, they will provide a compelling argument for why that is the case, when they are asked to submit a revised manuscript. When in doubt about how to handle specific recommendations, authors may approach the handling editor for clarification.

In the end, what we assess about our peers’ work is as important as how we assess it. In this editorial, we—as an editorial team—strongly suggest an asset perspective that attends to the potential of the work. A corollary of this perspective is our hope that readers take up invitations to review—as it is upon the expertise and diversity of our reviewing base that the ongoing quality of the work published in ESM rests.