Becoming a great reviewer: Four actionable guidelines

Introduction

Many scholars rise in academic reviewing structures, moving from ad-hoc reviewers, to editorial board members and then to associate editors or editors of reputable scientific journals. In the process, some scholars do not learn to become great reviewers. Consequently we offer four actionable guidelines to help reviewers transform from good to great. First, however, we describe the baseline for ‘good’ reviewing: the necessary elements of good reviewing and some ‘composition-related’ features of good reviews.

Baseline for ‘good’ reviewing

Peer review is central to deciding which scholarly work is actually published in academic journals. Caligiuri and Thomas (2013) identify what differentiates good reviews from poor quality ones, focusing especially on the Journal of International Business Studies (JIBS). They make two recommendations to reviewers that represent a baseline for good reviewing. First, they describe the necessary elements of good reviews, building on a survey with JIBS action editors to identify review features that editors believed were the most helpful to them. Second, they compare characteristics of well-rated reviews with poorly rated ones (with scores assigned systematically by the editors to each review).

The necessary elements of good reviews, as perceived by the JIBS editors, are the following (Caligiuri & Thomas, 2013: 549): the reviewer (1) discloses any potential conflict of interest; (2) declines to undertake a review if she or he feels unqualified to judge; (3) identifies the strengths (as well as the weaknesses) of the manuscript; (4) offers advice on how problems in the manuscript could be addressed, making plausible suggestions for improvements and for alternate ways to analyze the data; (5) provides comments on the manuscript’s overall contribution to the field.

Although all these elements are necessary, JIBS editors benefit most from reviewers’ assessment of the key potential contribution to the field, rather than their lists of shortcomings to be fixed, to secure acceptance. This element is a simple but critical ingredient of good reviewing, because when understood and implemented properly, it transforms the review process from a system that ‘catches’ inadequacies in authors’ works and pushes them to ‘solve’ those issues, toward a more collegial process of social construction of knowledge (Bedeian, 2004). A shift away from mere, supposed quality monitoring (e.g., in the form of reviewers acting as a ‘methods police force’) is also important to safeguard the authors’ authentic voice. As a result, what the authors intended to say is respected, and not diluted or contaminated by a variety of inconsequential changes, made solely to please the editor and reviewers.

Caligiuri and Thomas (2013) also highlight the composition-related characteristics that differentiate good reviews from poor ones. Good reviews: (1) addressed relevant issues more completely and with more depth; (2) provided detailed and constructive suggestions, and additional resources to be consulted by the authors; (3) focused, as stated above, primarily on the manuscript’s substantive contribution; (4) adopted a format that included a set of logically structured (numbered or indexed) comments, to help authors to respond to the comments; and (5) displayed a positive collegial tone rather than adopting the negative tone of a restaurant critic. We think that these elements remain valid today.

Four actionable guidelines to help reviewers become great reviewers

First Guideline for Great Reviewing: Assess Critically the Quality of Theory Development, Including ‘Theory Borrowing’ from Outside of the Field of Study

The modern field of International Business (IB) research, as with any other scholarly field, has its classic references.¹ These are scholarly pieces that self-described IB scholars typically know well and that are foundational to the field. Reviewers of IB research manuscripts should actually know such classic pieces well, and mostly do.

However, what constitutes the important research questions facing IB scholars is in constant flux. As a result, ‘borrowing’ theories from outside of IB, and even from outside of the broad field of management and organizational studies often enables us to strengthen the predictive and explanatory capacity of extant theoretical frameworks, when addressing empirical phenomena in IB.

The field of IB has a long history. In describing the historical development of IB research, Peter Buckley (2002: 366) distinguishes among three key periods. First, postwar IB research focused on the patterns and flows of foreign direct investment (FDI). Second, commencing in the early 1970s, the focus shifted towards explaining the emergence, strategy and organization of multinational enterprises (MNEs); this occurred in parallel with the rise and proliferation of MNEs in the world economy. Third, IB scholars moved in a new direction triggered by the multidimensional phenomenon of globalization and the related ‘new forms’ of international business (Buckley, 2002: 370; Griffith, Cavusgil, & Xu, 2008).

Just as the main themes of IB research have changed over time, so have the theories used to analyze these themes. In its early iterations, IB researchers borrowed heavily from international economics theory and focused on country specific advantages (CSAs), which supposedly drove international trade patterns (Rugman, Verbeke, & Nguyen, 2011); this focus informed the research agenda on FDI stocks and flows. Here, supporting theories from adjacent fields, inter alia, political science, sociology, marketing and cultural studies were also helpful to explain how national differences affected IB.

With the groundbreaking work of Hymer (1960), IB shifted from examining mainly country specific advantages (CSAs) to examining the firm specific advantages (FSA) of the MNE. This shift in focus occurred because of the recognition that micro-level governance and organization matter, and hence new theory was developed in this sphere (Rugman et al., 2011: 756). A subsequent shift from studying only the parent MNE to examining subsidiaries and the entire MNE network (including the complex interactions between dispersed FSAs and CSAs), meant that the field of IB needed to begin to rely on a broad set of theories, both indigenous and exogenous to the field, in order to answer pertinent research questions in the realm of complex resource recombination (Rugman, Verbeke, & Yuan, 2011; Verbeke, 2013).

Conceptual frameworks such as internalization theory, Hofstede’s work-related values, Uppsala School internationalization thinking and analysis of the dark side of MNE activity, the eclectic paradigm, the knowledge-based view of the firm, etc. have become heavily used by scholars in contemporary IB research. Griffith et al. (2008) performed a Delphi study of the most prolific international business scholars to determine future research themes in the field. Their analysis suggested that a wide array of questions was expected to emerge in IB, ranging from management and performance issues inside MNEs, to specific challenges facing firms from (and in) emerging markets, to legal and ethical issues in international business (Griffith et al., 2008: 1226). The increasingly broad scope of research questions suggests that more theoretical frameworks, both indigenous to the IB field and borrowed from other fields of science, will need to be applied in order to address these new, emergent questions. In fact, editorial teams of some IB journals have explicitly embraced the concept of ‘importing’ theory exogenous to IB through their expressed focus on interdisciplinary and multidisciplinary research. Unfortunately, unfettered diversity in theory importing can come with high costs, not only added benefits.

It is here that great reviewing matters: theory development that includes theory borrowing should be conducted cautiously, and its true value added (or lack thereof) to IB should be assessed carefully. Kenworthy and Verbeke (2015) suggest that extensive theory borrowing is reasonable in the early days of a field, but as the field progresses and crafts theoretical advances itself to address phenomena specific to this field, indigenous theories should receive priority over borrowed theories (Kenworthy & Verbeke, 2015: 184). They propose a framework of seven quality tests to help reviewers assess whether borrowed theory is likely to provide value added, beyond the capacity of indigenous theory. Table 1 describes this ‘seven tests’ assessment framework.

Table 1 Seven Tests to Assess the Quality of Theory Borrowing

The first three tests in the framework address the power of the borrowed theory within its original discipline, while the subsequent four tests pertain to its potential contribution to the importing field, in this case IB. The first test simply looks at the predictive power of the theory in its base discipline, by asking whether the theory has the required predictive power demonstrable by testable hypotheses that are significantly supported through statistical analyses. At the very least, the theory should have the capacity to add value to its original field by proposing testable predictions, rather than articulating general arguments. The second test considers the explanatory power of the theory, arguing that the borrowed theory assessed should have significant explanatory power to explain a large number of empirical regularities. In other words, if the generalizability of the theory’s predictions in the base discipline is ambiguous, then application of that theory in the IB context would be ill-advised. Absence of strong competing theories, the third test in the framework, considers whether there are strong, alternative theories that may have a higher explanatory or predictive power than the borrowed theory to explain phenomena in the base discipline. If alternative theories in the original field are at least equally powerful as the theory considered in explaining empirical phenomena, then theory importing in IB should again be approached cautiously.

Kenworthy and Verbeke (2015) argue that these first three tests are ‘necessary’ tests that a theory should pass to be considered a good candidate for borrowing by IB researchers. But four additional ‘sufficient’ conditions should also be met to ensure that value can be derived from theory borrowing by the importing field (2015: 184).

The fourth test assesses whether the issues covered in the borrowed theory “have a clear and intuitively plausible commonality with phenomena in the borrower field (2015: 184)”. This test suggests there should be some shared factors in both fields (such as the variables under study) that can give credence to a supposed predictive and explanatory capacity of the borrowed theory in IB research. The fifth test suggests that the concepts borrowed from the base field, can be linked in a meaningful way with related concepts in IB. Consistency in assumptions (e.g., about the nature of human behavior) is the sixth test. This consistency is critical, since the assumptions of a theory might be ‘self-evident’ in the base field, but intrinsically different from – or in contradiction with – the assumptions prevailing in IB (Kenworthy & Verbeke, 2015: 185). Lastly, the seventh test assesses the knowledge fit to judge the ‘legitimacy’ of the imported theory; this test asks, “Is there strong evidence in IB research that is consistent with – or contradicting – the predictions and propositions of the borrowed theory?” If contradictions abound, then the borrowed theory is unlikely to add much value in IB studies.

Although the ‘seven tests’ framework might be considered an onerous set of tests for reviewers to consider, the field of IB research (as is the case with the broader field of management and organizational studies) is presently inundated with theoretical concepts from completely different (and sometimes very distant) areas of scientific discovery. Great reviewers must assess in a systematic fashion, whether importing a theory carries a strong promise of value added to IB research, or whether it is better to stick with augmenting theories ‘indigenous’ to the field.

Second Guideline for Great Reviewing: Put Effort into Recognizing Where the Paper Has Gone ‘Beyond the State-of-the-Art’

The hallmark of responsible science is combined rigor and relevance. This combination establishes a productive relationship between valid and reliable knowledge and information that is useful to society and (in this case) managerial practice. Mainstream IB scholarship matters, but – contingent upon meeting the first guideline for great reviewing stated above – without deliberate and collective action that promotes variety and inclusivity, the scientific review process may lead to conservatism, incrementalism and homogenization (Rynes, 2006). The ultimate objective of the IB research community of editors, reviewers and authors should be to go beyond the state-of-the-art. There are at least three ways to go beyond the state-of-the-art, and reviewers should put effort into recognizing where this achievement has been realized.

The first approach is to advance significantly thinking about a core issue in the mainstream IB literature, such as the internationalization process and entry strategies, cross-border collaboration and network orchestration, parent-subsidiary linkages, etc., and to provide a fresh perspective. Such inquiries can be pursued by adopting new angles that have not yet been used but reflect an increasingly large part of economic reality, such as digital strategies for global operations, ‘coopetition’ with local and global rivals, global alliance portfolio management, etc.

The second approach is to deploy multilevel and multiple perspectives approaches. Single-level and single perspective analyses have long dominated IB research, yet many IB issues intrinsically involve multiple levels and can benefit from being analyzed through multiple lenses. Analysis at a single level, such as the micro level or macro level, may unfortunately yield an incomplete understanding of what is happening at either level (Petersen, Arregle, & Martin, 2012). For instance, MNEs are now typically prone to participate in international capital markets and to buy minority stocks of distant alliance partners. Beyond the immediate micro-economic implications of such behavior, these firms may now be significantly more vulnerable to particular types of international economic shocks and financial risks (Gilpin, 2002). Analysis of many ‘internal’ international resource allocation decisions can therefore not be divorced from the external forces that are associated with the broader environment, for example, the international financial, regulatory and tax environments.

Adopting multiple perspectives can also enhance our understanding of empirical phenomena. For example, redesigning and implementing international business models typically involve many value chain activities, including research and development (R&D), marketing, finance and accounting, human resources management, and the value chain governance of the firm’s resource portfolio. Unfortunately, even within mainstream IB research, our cumulative understanding of how decision making for individual functions is linked with decision making in other functional areas and with general management, is far from satisfactory (Tsui, Nifadkar, & Ou, 2007), and it is here that reviewers should pay attention to manuscripts that attempt to go beyond the state-of-the-art through multilevel and multiple perspectives thinking, with insights coming from research conducted in multiple subject areas. Multiple perspectives can also imply the comparative analysis of empirical phenomena through alternative conceptual lenses.

The third approach to going beyond the state-of-the-art is to advance the contextualized understanding of different types of international firms, meaning firms originating from – or operating in – different contexts. JIBS has, in the past, strongly supported research that examines the diversity of internationally operating firms, while at the same time upholding high standards of quality, validity and reliability. JIBS reviewers can build upon the journal’s long-standing tradition of embracing analysis of the diversity of the capabilities and strategies of MNEs (large and small) operating in a diverse set of geographic and institutional contexts (regions, countries or industries). This emphasis will only grow in importance in the future, as the world economy is increasingly being shaped by international businesses from emerging economies and by new types of MNEs that compete in creative ways. Context-rich analysis may include the in-depth description of, inter alia: (a) international firms based in different types of economies or different regions or nations; (b) established MNEs entering and competing in new economies, regions or countries; and (c) new types of international operations (e.g., online marketplaces or app-based international new ventures). JIBS reviewers must therefore embrace the broad variety of contexts within which conceptual and empirical contributions might be made, and also pay special attention to comparative studies. Reviewers should welcome articles that engage in contextualization rather than pose questions along the unhelpful lines of “Why Austrian MNEs?” or “Why Cambodian exporters?”.

Great reviewers should welcome ‘grand ideas’ but as noted above, going beyond the state-of-the-art often simply means deploying a new angle to analyze a mainstream IB phenomenon, augmenting prior work through explicitly adding a multilevel or multiple-perspective approach to extant work, or even simply providing enriched contextualization. Here, the extant literature should be used primarily as a guide, not as a blinder (Barkema, Chen, George, Luo & Tsui, 2015).

Of course, going beyond the state-of-the-art on any of the above dimensions should still address an important research question. It has been argued that in management and organization studies, scholars have increasingly focused on technical precision and manageable research projects (Pfeffer, 2007; Tsui, 2007). The IB field has clearly progressed in recent years toward probing big questions and novel ideas, but reviewers should understand the distinction made by Colquitt and Zapata-Phelan (2007) between ‘builders’ (articles generating new concepts, new perspectives or new theories) and ‘expanders’ (articles extending, refining or reformulating existing theories). Here, merely adding a few new moderators or mediators does not typically lead to a sufficiently important contribution warranting publication of a full-length article in JIBS. Similarly, the sole usage of a new or different empirical context, absent a detailed discussion of this new context’s features and the likely implications thereof vis-à-vis context represented in extant work (in terms of predicted impacts on the relevance of constructs or variables, as well as on the relationships between or among these variables) is unlikely to answer an important research question.

Nevertheless, not all JIBS papers are supposed to be ‘builders’. For example, JIBS reviewers should value methodological and empirical contributions, typical for ‘expander’ pieces. Methodological contributions can be made in a variety of forms, as when introducing innovative analytical or statistical methods, new large databases of MNEs, creative measurement designs, advanced multilevel analysis, and thorough comparative studies of different economies, institutions or cultures, to list just a few possibilities. Empirical contributions related to mainstream research questions are to be commended because they allow much needed assessment of the veracity of ‘received’ knowledge. They may confirm or contradict the validity of extant knowledge and conclusions, explore new findings that solve critical puzzles or answer vital questions, and provide finer-grained results that clarify mixed or inconclusive findings on some fundamental IB issues.

The reviewer’s developmental approach to evaluating a manuscript should therefore focus on identifying whether the manuscript has somehow moved the field of IB beyond the state-of-the-art in a broad sense, and if so, how the contribution made can be strengthened further, both in terms of more convincing analysis and clarity of exposition. Of course, all the above requires that the reviewer actually knows well the field of IB research.

Third Guideline for Great Reviewing: Assess Critically, But with an Open Mind and as an Authors’ Resource, Whether the Common Standards of Methodological Rigor Have Been Respected

The development of the IB field has been strongly driven by innovations in research design and methodologies, e.g., as reflected in the now standard approaches to study large-MNE databases and FDI panel datasets. Nevertheless, as is the case in other management and organization studies’ sub-fields, the IB discipline faces the challenges of remaining at par with the methodological standards in adjacent fields for validity, reliability, replicability and generalizability. The IB field can benefit enormously from reviewers who perform the roles of gatekeeper, knowledge promoter and resource for authors, regarding research methods. As gatekeepers, reviewers identify serious methodological problems and limitations. To perform this role reviewers should have both requisite access to critical information underlying empirical analyses and the competence to identify obvious gaps in methodological soundness, and to make constructive suggestions that improve the empirical analysis.

As promoters of good methods, reviewers celebrate those papers that make novel and sound methodological contributions (provided of course that the article contributes substantive value added to the field as well). Such contributions may involve new statistical methods to study large, longitudinal databases, new simulation tools, and new methods for measurement validation. Some innovative methods may originate internally, within the IB field (e.g., instrumentation and measurement equivalence improvement in cross-national, cross-cultural research) while other ones may be imported from other disciplines subject to the first guideline for great reviewing stated above. Latent variable mixture modeling, advanced modeling with longitudinal data, network analysis, moderated mediation and mediated moderation analyses, and computational modeling, to name a few, are among the analytical methods with great application potential in the IB field, especially when deployed in multi-study and multi-country research designs (Franke & Richey, 2010).

As a resource for authors, reviewers should review an article with the goal to improve it. They should not mechanically cite empirical limitations, such as common method variance or possible endogeneity due to cross-sectional data, as the justification for rejecting a study, and disregard the extent to which this limitation is actually serious and could reasonably be addressed in the study at hand. Imagine a study where opportunism’s antecedents are assessed by a survey of small exporting firms from a particular country, with both export performance variables and assessments of opportunism provided by single respondents. A poor review would demand that the authors also survey the actors who supposedly engaged in opportunistic behavior, so as to avoid common method bias. But such a demand is not reasonable. Imagine a medical research study of rape victims that considered the circumstances leading to the rape and the impacts on the victim. Would a medical journal reviewer require that the authors also study the views of the rapists to avoid common method bias? Reviewers unable to escape narrow, mental templates on methodology, will systematically reject excellent papers. In other words, reviewers should be able to contextualize what methodological rigor really means, especially when papers make strong theoretical contributions and authors have made reasonable efforts in terms of robustness assessments, additional tests, as well as ex ante or ex post remedies for acknowledged limitations. As resources for authors, reviewers should ask the following, when judging methodological limitations: (1) How severe are these limitations given the context at hand? (2) How likely is it that these limitations will affect the conclusions reached from the study? (3) How feasible and realistic is it to require researchers to test their hypotheses with a better design or method? (4) How much effort have the authors made to explain and/or to attempt to redress the limitations at hand?

Great reviews do not merely judge the methodological soundness of manuscripts, but suggest actionable ways in which methodological or empirical concerns can be – or should have been – addressed. Authors cannot, however, dodge their own responsibility to address methodological limitations, and they should address upfront their problems as well as possible, before submitting manuscripts. Reviewers are likely to be more receptive and open to submissions from researchers who have cautiously and prudently addressed some common methodological concerns to the extent they can, during the research design and execution stage.

Fourth Guideline for Great Reviewing: Give a Chance to the Analysis of Frontier Phenomena

The second guideline for great reviewing above, recommended identifying at least one dimension in which a paper goes beyond the state-of-the-art. Here, we go one step further. Great reviews, like great theories, must respect ‘out of the box’ thinking that transcends the mainstream. Such papers study ‘frontier phenomena’, ‘outliers’ or the ‘fringe’ in IB (Doh, 2015). While the IB reviewer will generally be familiar with the mainstream of the discipline, she or he might not be familiar with all frontier phenomena. Here again (as with our third guideline), a great reviewer operates as a resource to authors, in helping authors articulate better why the IB readership should be interested in the proposed ‘non-mainstream’ subject matter, and they should help the author position or frame the manuscript to make it clear the article is either a missing piece of the mainstream puzzle, or an addition to it, and is worthy of discussion and publication in a major refereed journal.

The worst assessment of a frontier phenomenon paper is one that simply states, “I must reject your paper because it is descriptive”. Yes, frontier phenomena often require intelligent description before the field can move forward. In this context, not every paper needs to reiterate the same arguments that the mainstream has established. In fact, the mainstream may well be wrong, for example, when driven by ideology in favor of perceived fairness (distributional issues), at the expense of efficiency considerations (as has often been the case with bottom of the pyramid and institutional voids ‘research’). Great reviews pay respect to authors who are trying to move beyond strictly mainstream thinking, and toward a line of inquiry, or a phenomenon, or perhaps an outlier that could change the mainstream. To put it even more bluntly, the reviewer should, where it is reasonable to do so, steer authors away from boring the audience with tried and supposedly ‘true’ messages, and from merely pursuing dustbowl empiricism.²

Doh (2015, citing Von Krogh, Rossi-Lamastra and Haefliger, 2012), notes that often, “research contributions tend to be incremental, marginal, highly theory-and methods-driven and often detached from [a] real-world phenomenon” (p. 609). Great reviews will notice whether the manuscript is either bound by political correctness or dogma associated with the mainstream, or on the contrary exhibits a willingness to rise to the challenge of analyzing new, complex IB phenomena, thereby disputing and possibly refuting the supposed truth associated with the mainstream. Likewise, great reviews will uncover whether the manuscript speaks to the proposed readership with ‘full voice meaningfulness’ (Shapiro, 2016). Shapiro disparages reviewers’ requests to authors to remove references that are anecdotal, book chapters, articles from ‘A-‘or below journals, interviews, metaphors, practitioner journals – in other words, everything except what the reviewer perceives as the mainstream. Without naming (any of the many) journals that advocate such practices, Shapiro suggests that such an approach willfully violates the ‘voice’ of scholars challenging anything close to the frontier of the field or representing outliers or fringe areas. Yet, within any scientific field, progress beyond incremental advances typically comes from movement at the edge, and not from the middle. The sole use of refereed articles published in supposed ‘A’ journals typically eliminates full voice meaningfulness. By contrast, great reviews encourage authors to embrace a myriad of sources of ‘knowing’ about any given phenomenon, and not just those sources that happened to become an article in a top tier journal (where type II errors also abound). The field of IB is fortunate to build upon a vast, dedicated ecosystem of journals, with JIBS being supported by many other outlets that publish mainly high quality and highly innovative work.³ These journals deserve to be cited in JIBS.

Conclusion

It is not easy to become a great reviewer and to behave like one in a consistent fashion, especially when faced with many demands on one’s limited time and intellectual energy. Manuscripts submitted for review can try reviewers’ patience and sometimes act as an irritant. But our four guidelines for great reviewing can be easily adopted, and will hopefully improve further the JIBS reviewing process and ultimately the quality of the research published in the journal. We reiterate the four guidelines for reviewers here, and add as a fifth point our main editorial philosophy principle:

• 1. Assess critically the quality of theory development, including theory borrowing from outside of the broad field of IB studies;

• 2. Put effort into recognizing where the paper has gone ‘beyond the state-of-the-art’;

• 3. Assess critically, but with an open mind and as an authors’ resource, whether the common standards of methodological rigor have been respected;

• 4. Give a chance to the analysis of frontier phenomena;

• 5. As a final point, the main principle of the 2017–2019 editors’ team will be ‘No manuscript left behind’. In simple terms: Reviewers should always put first the interests of the manuscript being assessed.

Notes

• 1 These include, inter alia, Bartlett and Ghoshal (1989), Buckley and Casson (1976), Dunning and Lundan (2nd ed., 2008), Hofstede (1980), Hymer (1960, published in 1976), Johanson and Vahlne (1977), Kogut and Zander (1993), Oviatt and McDougall (1994), Perlmutter (1969), Rugman (1981), Stopford andWells (1972), Teece (1977), Vernon (1966).

• 2 In line with Davis (1971), reviewer engagement often comes from being moved by a manuscript’s ‘wow’ factor, which credibly challenges what has been taken for granted. He claims: “interesting theories are those which deny certain assumptions of their audience, while non-interesting theories are those which affirm certain assumptions of their audience.” (p. 309). The reviewer’s task is then to guide the author toward making obvious the non-obvious, rather than concentrating on the obvious. Mainstream research, while important, reinforces taken-for-granted beliefs in the obvious in any particular field, and good reviews will acknowledge this. If the manuscript is discussing something so obvious, all it does is confirm one of our taken-for granted beliefs, the readership will “respond to it by rejecting its value while affirming its truth…of course” (p. 311). Something that is taken for granted has both a theoretical and practical dimension. A paper that discusses a frontier or an outlier to a taken-for-granted theory, or challenges such theory, will therefore be truly interesting if it addresses both theory and practice.

• 3 The following journals, among other excellent outlets, are part of this dedicated ecoystem: Journal of World Business, Global Strategy Journal, International Business Review, Journal of International Management, Management International Review, Asia Pacific Journal of Management, Management and Organization Review, Multinational Business Review, Cross Cultural and Strategic Management.