Reviewing systematic literature reviews: ten key questions and criteria for reviewers

Systematic literature review articles are important for synthesizing knowledge in management and business research. However, to date, we lack clear guidelines how to review such articles. This editorial takes the perspective of the reviewer. It presents ten key questions and criteria that reviewers should ask when reviewing systematic literature reviews.

1 3 allow them to address issues that might arise during the review process proactively and consequently avoid them.
We consider SLRs (sometimes also called structured literature reviews) to be a variant of literature reviews, primarily aiming at increasing the transparency of the literature selection process underlying the review. Doing so helps to minimize subjectivity concerning which studies are included in the review and reducing what could be called the sampling error of unsystematic literature reviews. The term systematic thus refers in particular to a structured or systematic literature identification and selection. For reviewers of such SLRs, the following ten questions are crucial.

Does the SLR have an explicit and well-explained research question that is also well-justified?
Every study needs a research question, and that applies to SLRs too. That research question might be broad and topic-based (e.g., What is state-of-the-art in the respective field or literature stream? What do we know about a particular phenomenon?), but could also be more specific (e.g., What is the empirical evidence regarding a specific research question?). Opportunities to frame and justify research questions are endless, but, in all cases, authors should state a clear research question, and reviewers need to check whether the submission provides one. In addition to establishing an explicit research question, authors must justify the choice clearly and convey the potential value of their answer to the research question. It is then the task of the reviewers to assess whether they support that argumentation. In our own experience as reviewers and editors of MRQ, many initial submissions of SLRs lack sufficient justification. However, a literature review is not an end in itself. It needs to be justified like any other academic study. Possible justifications could be phenomenon-based, theory-driven, or guided by practice.

Does the SLR acknowledge previous literature reviews (including meta-analyses)?
Particularly in subject areas with a long history of research, it is to be expected that previous literature reviews exist. Every SLR should acknowledge these previous contributions (and in particular similar SLRs), describe their character (e.g., descriptive, bibliographic, state-of-the-art, or narrative), and explicate why another attempt to synthesize the body of knowledge is necessary. Reasons can be manifold-outdated prior literature reviews might need to be updated, including a different type of publication might potentially allow for new or more complete perspectives (e.g., gray literature vs. journal publications), or perhaps researchers are attempting to answer a hitherto unanswered research question. Reviewers should also check whether the SLR acknowledges published meta-analyses as these share many of the goals of an SLR.

Is the research question correctly translated into a (relevant) set of search terms?
An SLR refers to databases to identify relevant studies and queries them with an appropriate set of search terms. Reviewers must check whether the SLR states its search term transparently, accurately, and ascertain if the search term can guarantee to return the relevant literature to answer the research question. Formulating search terms is essentially the operationalization of the research question, and the success of any SLR stands or falls on it. Denyer and Tranfield (2009) suggest translating a research question with the help of the context-interventions-mechanisms-outcome (CIMO) framework into a set of search terms. Adopting a CIMO framework or a similar alternative also helps to devise the research questions that an SLR could reasonably be expected to answer. The set of search terms should be carefully thought out and not be so narrow that it misses relevant studies but also not so broad that the identified studies do not match the research questions. Reviewers should also check whether the researchers followed an iterative process where the search terms are adapted based on the search process results.

Is the database selection explained and based on a clear rationale?
Different databases of academic literature return different results. Therefore, it is good practice to employ more than one database in the literature search. Triangulation helps compensate for particular disadvantages that naturally limit the output of each individual database. At a minimum, a paper can address triangulation simply by adding more databases (e.g., the more databases, the better); an exemplary SLR will provide tangible reasons for including (or excluding) particular databases. Gusenbauer (2019) and Gusenbauer and Haddaway (2020) evaluate numerous available databases and provide a good starting point for justifying database selection.

Is the literature selection up-to-date?
Authors often submit outdated SLRs to MRQ. Updating a review before the submission is essential, and not something authors should postpone until the later rounds of the review process. Submitting an outdated literature review is usually a reason for a desk rejection, but reviewers should also check that an SLR is not outdated and remains topical throughout the process of revising and resubmitting.

Does the SLR clearly express its inclusion and exclusion criteria?
Quite often, a simple database query returns hundreds (or perhaps thousands) of studies with the potential for inclusion in the review. An SLR will usually report the search funnel and the authors' inclusion and exclusion criteria applied to narrow down a search. Reviewers must check whether those criteria are expressly stated in the paper and examine whether they serve their purpose. The selected inclusion and exclusion criteria depend on the aims and research questions of the SLR: They might be justified in terms of content, method, or publication quality. We recommend using a flow chart to graphically illustrate the selection process. That flow chart should include the initial number of identified studies and the number of studies eliminated at each stage of the literature search process. Further guidance on how to ensure the selection of a final sample is transparent can be found in "Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA)" (Liberati, 2009;Moher, 2009).

Does the SLR include one or more overview tables characterizing each study in the sample?
Another good practice is to include a table summarizing each study from the sample. The table may be incorporated within the paper or, in the case of larger samples, presented as an (online) appendix. Such tables help readers quickly orient themselves in a particular research stream and understand the individual contribution of particular studies they might be interested in. Reviewers will need to check the quality of the information presented in such tables or appendices and whether the authors summarize the right information describing a single study.
Standard columns of the table should include a reference to the individual study, a literal quote (or paraphrase) of its research question, the epistemological character of the study (e.g., conceptual, qualitative, quantitative, or mixed), the specific methodological design employed in the study (in the case of empirical studies), and the main result(s), although those columns might vary to reflect the purpose of the SLR. Depending on the number of studies included in the review, splitting this table into several sub-tables based on sub-topics is recommended. It is also possible to deposit a file in a public (data) repository (for instance, that operated by the Open Science Foundation). A repository offers authors greater flexibility regarding the number of columns and lines to be included, and reviewers can point authors to appropriate repositories.

Does the SLR employ a particular aggregation and presentation method?
It is essential to understand that an SLR is primarily a method to arrive at a suitable sample of studies to be reviewed, but how the selected literature is then analyzed, synthesized, and presented is a decision for individual authors who can choose from among a plethora of methods (see Booth et al. (2012) for an extensive overview of options). Far too often, this is the step where authors abandon methodological considerations. Nevertheless, there are many methods available to ensure this step is conducted in a rigorous and method-led manner, and SLRs employing such methods are more convincing. Such methods might range from a thematic analysis and narrative synthesis through meta-ethnographies to bibliometric approaches (e.g., historiographic mapping). Published MRQ papers illustrate the breadth of available methods. Here, the reviewer's task is to check whether the chosen method was correctly executed and help the authors reach for a more profound and potentially more fruitful analysis. However, the use of particular aggregation and presentation methods should be fully explained and should align with the research question(s) of the SLR. 9. Does the SLR contribute beyond merely offering an analysis of the status quo of the literature?
Synthesis builds on analysis and may come in different forms. Some authors construct integrative models, provide thematic clusters based on bibliometric analyses, or offer creative answers to the research question. Synthesis is an essential step and distinguishes a technically correct but uninformative literature review from an informative one. Literature reviews tend to be among the most often-cited of journal articles, but will only be so if they contribute something unique and provide guidance to the community on what has been done and what is yet to be done.

Does the SLR provide implications for (future) research and practice?
A high-quality synthesis provides a basis for assessing the status quo and can set an agenda for future research. For practitioners, an SLR offers a convenient way to understand what is known or not yet known about a particular phenomenon or problem, and hence, the SLR should make evidence-based suggestions for managerial practice. Authors should not leave this task to readers but proactively outline specific (practice) implications that might speak to consumers, employees, managers, investors, firm owners, policymakers, and any other groups interested in management research. Similarly, for a research audience, every well-executed and well-crafted SLR has the potential to set the agenda for further research. Admittedly, that demands some creativity, as the easy solution to simply point the reader to under-researched areas invites encouraging studies based on the infamous hasnot-been-done-before argument. The onus is on the researcher to identify and justify important avenues for future research, and reviewers should carefully consider whether they support those choices or should highlight alternatives they consider more promising. Authors should include the part of the SLR reflecting future research opportunities in an initial submission, but be aware that the suggestions will also potentially be subject to considerable change as a result of the review process. The length and nature of the future research section also depend on the goal of the SLR. Broader, state-of-the-art SLRs should provide a broad future research agenda, whereas reviews with a narrow research question should have a more focused and precise future research section.
SLRs have an important role in the accumulation of knowledge in management and business research; however, they can only fulfill that role if they are of high quality. This editorial, through its ten questions and criteria for reviewing SLRs, aims to signpost to reviewers the key issues relevant to evaluating the quality of an SLR. We also hope that it will enhance the quality of SLRs submitted to MRQ and other outlets in management research.