Abstract
Systematic literature review articles are important for synthesizing knowledge in management and business research. However, to date, we lack clear guidelines how to review such articles. This editorial takes the perspective of the reviewer. It presents ten key questions and criteria that reviewers should ask when reviewing systematic literature reviews.
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Owing to its focus on systematic literature reviews, meta-analyses, replications, and bibliometric studies, Management Review Quarterly (MRQ) requires submissions to meet specific criteria. Several published MRQ editorials explain those criteria, namely Block and Kuckertz (2018) for replication studies, Block and Fisch (2020) for bibliometric studies, and Fisch and Block (2018) with the extension by Clark (2021) for systematic literature reviews.
We again focus on systematic literature reviews (hereafter SLRs) in this editorial but shift the perspective. Whereas previous editorials addressed (potential) authors of MRQ submissions, this one addresses MRQ’s reviewers. Reviewers for the journal are often selected for their topical expertise and publication record, allowing them to provide the highest quality feedback to authors of SLRs. However, being an experienced researcher in a particular topical domain does not necessarily come with methodological expertise in SLRs. Hence, in this editorial, we highlight ten key questions and criteria that reviewers should consider when evaluating SLRs. Of course, knowledge of these criteria might be helpful for authors as well, as it will allow them to address issues that might arise during the review process proactively and consequently avoid them.
We consider SLRs (sometimes also called structured literature reviews) to be a variant of literature reviews, primarily aiming at increasing the transparency of the literature selection process underlying the review. Doing so helps to minimize subjectivity concerning which studies are included in the review and reducing what could be called the sampling error of unsystematic literature reviews. The term systematic thus refers in particular to a structured or systematic literature identification and selection. For reviewers of such SLRs, the following ten questions are crucial.
1. Does the SLR have an explicit and well-explained research question that is also well-justified?
Every study needs a research question, and that applies to SLRs too. That research question might be broad and topic-based (e.g., What is state-of-the-art in the respective field or literature stream? What do we know about a particular phenomenon?), but could also be more specific (e.g., What is the empirical evidence regarding a specific research question?). Opportunities to frame and justify research questions are endless, but, in all cases, authors should state a clear research question, and reviewers need to check whether the submission provides one. In addition to establishing an explicit research question, authors must justify the choice clearly and convey the potential value of their answer to the research question. It is then the task of the reviewers to assess whether they support that argumentation. In our own experience as reviewers and editors of MRQ, many initial submissions of SLRs lack sufficient justification. However, a literature review is not an end in itself. It needs to be justified like any other academic study. Possible justifications could be phenomenon-based, theory-driven, or guided by practice.
2. Does the SLR acknowledge previous literature reviews (including meta-analyses)?
Particularly in subject areas with a long history of research, it is to be expected that previous literature reviews exist. Every SLR should acknowledge these previous contributions (and in particular similar SLRs), describe their character (e.g., descriptive, bibliographic, state-of-the-art, or narrative), and explicate why another attempt to synthesize the body of knowledge is necessary. Reasons can be manifold—outdated prior literature reviews might need to be updated, including a different type of publication might potentially allow for new or more complete perspectives (e.g., gray literature vs. journal publications), or perhaps researchers are attempting to answer a hitherto unanswered research question. Reviewers should also check whether the SLR acknowledges published meta-analyses as these share many of the goals of an SLR.
3. Is the research question correctly translated into a (relevant) set of search terms?
An SLR refers to databases to identify relevant studies and queries them with an appropriate set of search terms. Reviewers must check whether the SLR states its search term transparently, accurately, and ascertain if the search term can guarantee to return the relevant literature to answer the research question. Formulating search terms is essentially the operationalization of the research question, and the success of any SLR stands or falls on it. Denyer and Tranfield (2009) suggest translating a research question with the help of the context-interventions-mechanisms-outcome (CIMO) framework into a set of search terms. Adopting a CIMO framework or a similar alternative also helps to devise the research questions that an SLR could reasonably be expected to answer. The set of search terms should be carefully thought out and not be so narrow that it misses relevant studies but also not so broad that the identified studies do not match the research questions. Reviewers should also check whether the researchers followed an iterative process where the search terms are adapted based on the search process results.
4. Is the database selection explained and based on a clear rationale?
Different databases of academic literature return different results. Therefore, it is good practice to employ more than one database in the literature search. Triangulation helps compensate for particular disadvantages that naturally limit the output of each individual database. At a minimum, a paper can address triangulation simply by adding more databases (e.g., the more databases, the better); an exemplary SLR will provide tangible reasons for including (or excluding) particular databases. Gusenbauer (2019) and Gusenbauer and Haddaway (2020) evaluate numerous available databases and provide a good starting point for justifying database selection.
5. Is the literature selection up-to-date?
Authors often submit outdated SLRs to MRQ. Updating a review before the submission is essential, and not something authors should postpone until the later rounds of the review process. Submitting an outdated literature review is usually a reason for a desk rejection, but reviewers should also check that an SLR is not outdated and remains topical throughout the process of revising and resubmitting.
6. Does the SLR clearly express its inclusion and exclusion criteria?
Quite often, a simple database query returns hundreds (or perhaps thousands) of studies with the potential for inclusion in the review. An SLR will usually report the search funnel and the authors’ inclusion and exclusion criteria applied to narrow down a search. Reviewers must check whether those criteria are expressly stated in the paper and examine whether they serve their purpose. The selected inclusion and exclusion criteria depend on the aims and research questions of the SLR: They might be justified in terms of content, method, or publication quality. We recommend using a flow chart to graphically illustrate the selection process. That flow chart should include the initial number of identified studies and the number of studies eliminated at each stage of the literature search process. Further guidance on how to ensure the selection of a final sample is transparent can be found in "Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA)" (Liberati, 2009; Moher, 2009).
7. Does the SLR include one or more overview tables characterizing each study in the sample?
Another good practice is to include a table summarizing each study from the sample. The table may be incorporated within the paper or, in the case of larger samples, presented as an (online) appendix. Such tables help readers quickly orient themselves in a particular research stream and understand the individual contribution of particular studies they might be interested in. Reviewers will need to check the quality of the information presented in such tables or appendices and whether the authors summarize the right information describing a single study. Standard columns of the table should include a reference to the individual study, a literal quote (or paraphrase) of its research question, the epistemological character of the study (e.g., conceptual, qualitative, quantitative, or mixed), the specific methodological design employed in the study (in the case of empirical studies), and the main result(s), although those columns might vary to reflect the purpose of the SLR. Depending on the number of studies included in the review, splitting this table into several sub-tables based on sub-topics is recommended. It is also possible to deposit a file in a public (data) repository (for instance, that operated by the Open Science Foundation). A repository offers authors greater flexibility regarding the number of columns and lines to be included, and reviewers can point authors to appropriate repositories.
8. Does the SLR employ a particular aggregation and presentation method?
It is essential to understand that an SLR is primarily a method to arrive at a suitable sample of studies to be reviewed, but how the selected literature is then analyzed, synthesized, and presented is a decision for individual authors who can choose from among a plethora of methods (see Booth et al. (2012) for an extensive overview of options). Far too often, this is the step where authors abandon methodological considerations. Nevertheless, there are many methods available to ensure this step is conducted in a rigorous and method-led manner, and SLRs employing such methods are more convincing. Such methods might range from a thematic analysis and narrative synthesis through meta-ethnographies to bibliometric approaches (e.g., historiographic mapping). Published MRQ papers illustrate the breadth of available methods. Here, the reviewer’s task is to check whether the chosen method was correctly executed and help the authors reach for a more profound and potentially more fruitful analysis. However, the use of particular aggregation and presentation methods should be fully explained and should align with the research question(s) of the SLR.
9. Does the SLR contribute beyond merely offering an analysis of the status quo of the literature?
Synthesis builds on analysis and may come in different forms. Some authors construct integrative models, provide thematic clusters based on bibliometric analyses, or offer creative answers to the research question. Synthesis is an essential step and distinguishes a technically correct but uninformative literature review from an informative one. Literature reviews tend to be among the most often-cited of journal articles, but will only be so if they contribute something unique and provide guidance to the community on what has been done and what is yet to be done.
10. Does the SLR provide implications for (future) research and practice?
A high-quality synthesis provides a basis for assessing the status quo and can set an agenda for future research. For practitioners, an SLR offers a convenient way to understand what is known or not yet known about a particular phenomenon or problem, and hence, the SLR should make evidence-based suggestions for managerial practice. Authors should not leave this task to readers but proactively outline specific (practice) implications that might speak to consumers, employees, managers, investors, firm owners, policymakers, and any other groups interested in management research. Similarly, for a research audience, every well-executed and well-crafted SLR has the potential to set the agenda for further research. Admittedly, that demands some creativity, as the easy solution to simply point the reader to under-researched areas invites encouraging studies based on the infamous has-not-been-done-before argument. The onus is on the researcher to identify and justify important avenues for future research, and reviewers should carefully consider whether they support those choices or should highlight alternatives they consider more promising. Authors should include the part of the SLR reflecting future research opportunities in an initial submission, but be aware that the suggestions will also potentially be subject to considerable change as a result of the review process. The length and nature of the future research section also depend on the goal of the SLR. Broader, state-of-the-art SLRs should provide a broad future research agenda, whereas reviews with a narrow research question should have a more focused and precise future research section.
SLRs have an important role in the accumulation of knowledge in management and business research; however, they can only fulfill that role if they are of high quality. This editorial, through its ten questions and criteria for reviewing SLRs, aims to signpost to reviewers the key issues relevant to evaluating the quality of an SLR. We also hope that it will enhance the quality of SLRs submitted to MRQ and other outlets in management research.
References
Block JH, Fisch C (2020) Eight tips and questions for your bibliographic study in business and management research. Manag Rev Q 70(3):307–312. https://doi.org/10.1007/s11301-020-00188-4
Block J, Kuckertz A (2018) Seven principles of effective replication studies: strengthening the evidence base of management research. Manag Rev Q 68(4):355–359. https://doi.org/10.1007/s11301-018-0149-3
Booth A, Papaioannou D, Sutton A (2012) Systematic Approaches to a Successful Literature Review. Sage Publications, London
Clark WR et al (2021) Extending Fisch and Block’s (2018) tips for a systematic review in management and business literature. Manag Rev Q 71:215–231. https://doi.org/10.1007/s11301-020-00184-8
Denyer D, Tranfield D (2009) Producing a Systematic Review. In: Buchanan DA, Bryman A (eds) The Sage Handbook of Organizational Research Methods. Sage Publications, Thousand Oaks, CA, pp 671–689
Fisch C, Block J (2018) Six tips for your (systematic) literature review in business and management research. Manag Rev Q 68(2):103–106. https://doi.org/10.1007/s11301-018-0142-x
Gusenbauer M (2019) Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics 118:177–214. https://doi.org/10.1007/s11192-018-2958-5
Gusenbauer M, Haddaway NR (2020) Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods 11(2):181–217. https://doi.org/10.1002/jrsm.1378
Liberati A et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. https://doi.org/10.1371/journal.pmed.1000100
Moher D et al (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097
Acknowledgements
We are indebted to Christian Fisch, Mirko Hirschmann, Holger Steinmetz and Rolf Wilmes for sharing their perspectives on the challenges of reviewing SLRs.
Funding
Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kuckertz, A., Block, J. Reviewing systematic literature reviews: ten key questions and criteria for reviewers. Manag Rev Q 71, 519–524 (2021). https://doi.org/10.1007/s11301-021-00228-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11301-021-00228-7