The role and impact of reviewers on the marketing discipline

Since John Lynch’s presidential address at the 1998 annual meeting of the Association for Consumer Research (Lynch 1998), a large number of articles have appeared in the marketing literature pertaining to the review process in our field. Almost every new journal editor makes some statement about the standards and etiquette that reviewers should adopt during his or her editorial regime. For some good examples, see Shugan (2003), Desai (2011), and Kumar (2016). Other useful discussions of the review process also exist (e.g., Holbrook’s 1986 paper with seven suggestions for reviewers).

The purpose of this editorial is to give our perspective on (1) how the editorial process has evolved over time, (2) the importance and benefits of the review process to the marketing field, (3) how reviewing fits into career development, and (4) the characteristics of a good review. While it is inevitable that we will cover some ground that has already been tread by previous work, we hope that from the collective 90 years we have spent involved with the review process, we can add a useful perspective to a process that is often dreaded by both authors and reviewers.

Evolution

The (good?) old days

When we entered the field, it was relatively small. For example, Russ Winer’s AMA Doctoral Consortium group had 41 students. Similarly, when Don Lehmann attended the first “open” ACR annual meeting in 1969, there were 90 attendees including quantitative researchers, practitioners, and government officials. Several major marketing journals, including JCR, Marketing Science, JCP, and QME, did not exist and manuscript flow was relatively low. In 1970, when Ralph Day was the editor, JMR had only 21 members of the editorial board,¹ no co-editors, and no associate editors.² In the twentieth century, the editorial process worked as follows:

• Authors snail-mailed 5 hard copies to the editor.

• The editor chose 2–3 reviewers from the editorial board and from an ad hoc list.

• The editor mailed the hard copy papers to the reviewers.

• The reviewers mailed back their reviews.

• The editor made his/her decision and mailed it to the authors as well as copies to the reviewers.

While the reject rate at the top marketing journals has always been high,³ the manuscript flow was lower, editors needed to fill issues, and the prevailing attitude among editors was that they were looking for papers to accept. For example, as late as 2006, JMR and JM were receiving about 250 new manuscripts each year. Others, such as Marketing Letters (which began publishing in 1989), had closer to 100.

Currently

The pressure on editors today is intense. What has led to this?

• The size of the field has increased dramatically. The number of business schools has increased substantially with concomitant increases in marketing faculty positions and doctoral programs. For example, at the 2016 AMA-Sheth Doctoral Consortium, there were 103 students (which was fewer than in 2015), far greater than the 41 at Russ Winer’s consortium.

• Many business schools that were once mainly “teaching” schools now demand some research for promotion and tenure.

• The market, which was once dominated by (and in effect limited to) the U.S. and parts of Europe, is now global with more marketing faculty in the rest of Europe and particularly in the Asia-Pacific region. Editors are seeing papers from universities in countries that were not major sources of manuscripts 10–15 years ago such as China, Turkey, and Iran.

• Business school rankings such as those of the Financial Times include faculty publications in international research journals as a component.

• Some schools supplement salaries with bonuses for publications in top-level international journals.

• Longer doctoral programs and pressure from promotion and tenure committees to have focused research programs have resulted in increased specialization and created more competition at the discipline-based journals.

• Electronic submission systems have reduced the costs of submission and re-submission (although not necessarily the length of the review process).

As a result, in spite of on expansion in the number of journals (Lehmann, 2005), competition for the available journal space has intensified. The number of submissions has increased dramatically to over 600 new and 800+ total submissions per year at some major journals. The journal “system” has responded in two ways. First, the former system of one editor and a review board has been replaced by a three-tier system of an Editor-in-Chief (EIC), a team of associate/senior/area editors who synthesize the reviews and make a recommendation to the EIC/co-editor, and a review board. Several of the top journals have co-editors, who also have decision-making authority on a manuscript in addition to the EIC.

Second, the desk reject rates at all the journals have increased, in several cases to 50%, some of which is due to authors not paying sufficient attention to either journal format or the advice of editors (e.g., Palmatier 2016a, b). Without a high desk rejection rate, it would be nearly impossible to manage the manuscript flow given the number of submissions, the number of rounds of revisions (which many editors attempt to limit to two but which often extend to three or more), and the availability of reviewers with expertise in a particular area who have the capacity and willingness to do the work. Reviewing loads are heavy, and editors, appropriately, try to avoid overloading good reviewers, especially when there is little chance that their efforts will result in a publishable paper in their journal. From the author’s point of view, desk rejections are discouraging and, to some, seem like a violation of “due process.” On the other hand, it is generally better to know sooner rather than later if a paper will be rejected so the author can move on with alternative plans. Desk rejections are also not something new. Robert Ferber (who was the first JMR and second JCR editor) in the 1970’s had three screening editors at JCR whose job was to (a) identify papers that stood little chance of acceptance, and (b) write a short report explaining why along with some suggestions for how to improve the paper.

The end result of all of this is that the review system is under duress. Reviewing in the field of marketing is a volunteer activity⁴ and the top reviewers are already heavily committed. The larger number of papers under review means that editors are both using their current teams more intensively and seeking ad hoc (and in the case of JCR, “training reviewers”, i.e., PhD students) reviewers at a greater rate. It is a struggle to get reviewers to agree to do reviews and an additional challenge to get them to do timely ones. Consequently, editors need to “protect” reviewers and AEs or risk losing them altogether.

This increased load has also created an attitude change among reviewers who often look for reasons to reject rather than to accept papers. Further, reviewers often focus on technical details that may or may not have little to do with the main (substantive) contribution of the paper but are frequently represented as basic requirements for publication (McAlister 2016). The additional layer of review created by the associate editor structure means more time in the system for a given manuscript and more time pressure for timely reviews. It also adds another person who can recommend rejection. Given the tendency to weight negative information more heavily, additional reviews make it harder to get a paper accepted. (Of course the additional review can also both catch errors and make useful suggestions). The use of co-editors creates a potential problem of non-uniform standards as well as increased specialization as co-editors together with AEs and reviewers are chosen to match the paper as closely as possible. This results in papers written to some degree in “code” that are mainly accessible to and read by a small fraction of the field even though they appear in general journals such as JMR. New papers with “out of the box” thinking have a more difficult time getting through the process partly because they don’t fit into any of the narrow paradigms.

Recently, a number of well-documented cases of academic misconduct (ranging from p-hacking to outright fraud) have led journals to adopt new and more stringent rules about data and computer code disclosures when a paper is accepted. This has resulted in reviewers being asked to pay more attention to potential problems, increasing the reviewing burden. It also produces a bit of a “guilty until proven innocent” mentality in the review process. Overall, this has moved the process toward being more adversarial and legalistic (as evidence by long reviews and responses to reviewers).

When faced with two of three negative reviews and a negative reaction from an AE, most editors feel compelled to go with the majority opinion (partly to avoid antagonizing their volunteer reviewers and AEs). There is also another pernicious aspect. Typically, if one of three reviewers recommends acceptance, revisions are only sent back to the two reviewers who didn’t. This in effect loads the jury against acceptance, thus disenfranchising the positive reviewer (whose position is subsequently weighted less heavily).

Importance of the review process

Despite these issues, the peer review process continues to be important and useful. A strong, effective review process helps ensure that papers that are accepted do not have fatal flaws, establishes research standards in the field, and, if operating properly, permits editors to shape the research agenda by accepting papers outside normal boundaries and in areas that might otherwise not receive sufficient attention.

Of course, one does not have to go through the review process. There are a large number of outlets today for non-peer reviewed work. Perhaps the most popular for marketing academics is the Social Science Research Network (SSRN). SSRN essentially replaces the old hard-copy working paper series that business schools used to have. This facilitates the quick dissemination of new ideas and offers authors the increased opportunity to receive comments and citations from other academics. As a result, authors often tout the number of SSRN downloads as an indication of the quality and impact of their work. However, SSRN does not peer review working papers and therefore having a paper in the SSRN database is not a perfect substitute for having one in a peer-reviewed journal or a signal of quality. Similarly, Google Citations include a number of sources, which provides some information but not the more focused view of academic citations captured by ISI, Web of Science.

The review process as part of career development

As noted above, the marketing profession operates on a volunteer basis. Faculty expect to get good reviews when they submit manuscripts and in turn should supply good reviews to other authors. Because becoming a good reviewer is an important part of career development, younger faculty should be willing to review for a journal and many look forward to doing so. (Although as a person becomes more senior, it is “be careful what you wish for” as a good reviewer will wind up with multiple manuscripts on her or his desk.) Younger faculty can be particularly valuable for providing verification that there is nothing “wrong” technically. Importantly, reviewing is more than just “giving back” to the field; it often leads to learning. As reviewers we have been made aware of new areas of research, and research methods, and gained a general appreciation for the quality of work that is being done by our colleagues.

How does reviewing fit into the promotion and tenure process? While having published papers is clearly more important, it is generally assumed that junior faculty will do some reviewing; it would be unusual for a CV to not have a list of journals for which the tenure candidate has reviewed. A strong positive signal is membership on the editorial board of a major journal, but this is uncommon. If non-tenured faculty members are not being asked to do reviews, we recommend that they (or better a well-known senior colleague) contact the editor and/or the AEs and let them know that they are available to do reviews and have expertise in certain areas. While this may appear to be presumptuous, there is a shortage of people able and willing to do reviews, so being proactive is normally welcomed.

Ph.D. students should definitely be asked to assess published and working papers in their doctoral seminars. Whether students should become formally involved with the reviewing process in their doctoral programs is debatable. There is always something amiss even in published papers (no paper written is ever perfect), and tearing papers apart is a normal part of student training. A useful exercise is to provide students with a paper that is (or was) under review at a journal. Since journal editors and authors expect that reviews will be conducted by the faculty members themselves, we do not feel it is appropriate to let doctoral students directly insert their perspectives on a manuscript into the final review. While reviewing is a useful learning experience, in general, students do not have the experience and perspective to understand the extent of the incremental contribution. Students also tend to look for what is wrong with a paper as opposed to what is right/interesting. JCR has formalized this process by allowing for “training reviewers” to write reviews on a paper. However, an important condition is that the students are tutored on how to write a good review. Without that, many doctoral students will focus on a manuscript’s perceived flaws, which is less helpful to editors, the authors of the paper, and the field.

Characteristics of a good review

Provides advice to the editor

One (the?) main purpose of a review is to provide guidance to the AE and ultimately to the editor about the disposition of the manuscript. That is, after all, the raison d’être of the review process. Thus, a good review helps those up the review ladder to either make an accept or reject decision or develop a coherent road map for a revision. As a result, reviewers are encouraged to divide the comments into what the reviewer feels are the most important, major points that affect the “up or down” decision, and those that are more minor, such as editorial issues. It is then up to the editor and AE to determine if the major points made are fixable (which suggests encouraging a revision) or not (i.e., requires rejection).

Where many reviews fall short is in terms of the specificity of their comments. Just to say there is an endogeneity problem is not helpful; it is much more helpful to indicate why the reviewer feels the problem exists, and even better to suggest a solution for it. Similarly, to say that the conceptual development/theory is poor/weak does not indicate where the theory falls short. Also, simply saying that other papers have covered the same ground is less helpful than if the reviewer provides specific citations.

Provides helpful comments to the author

Of course, reviewer comments are also supposed to assist authors in improving their papers whether they are revising them for resubmission to the same journal or not. The idea is that the review process should improve the next version of the paper. Essentially all senior faculty members have had their papers rejected, including, sadly, these authors (multiple times). However, the benefit of having at least three pairs of experienced eyes read a paper and provide comments is enormous.⁵ (In return for this service, authors should not send in partly finished work in the hope that the reviewers will find ways to improve it.) When Preyas Desai was the editor of Marketing Science, he encouraged review teams to use the “reject and resubmit” when appropriate because otherwise other journals would receive the benefits of the improvements that the journal’s review team provided.

A helpful review has the same key property as one giving good advice to the editor and AE: specificity. Authors appreciate a clear roadmap for revisions and AE’s and Editors want key concerns highlighted and distinguished from minor issues.

Helps determine what is a contribution

There are many ways to make a contribution, and a paper does not need to include all of them. Traditionally contributions have been categorized as theoretical, methodological, logical, or substantive. To this we would add “makes one think/inspires future research.” Because no single paper is ever completely definitive, this aspect may be the most important. A considerable amount of qualitative work falls in this category. Finally, there has been an increased focus on work that is helpful to some party, i.e., action research such as that done under the umbrella of TCR (Transformative Consumer Research) or work that impacts practice (e.g., the Lilien practice prize work in marketing science). Before reviewers conclude a paper “makes no significant contribution”, they should make sure that is really true.

Components of a review

Four assessments

A review contains at least four assessments. One is for technical correctness. Here the question should be not whether the paper uses the most advanced (and often costly) approach, but whether the approach is reasonable and likely to produce a good, if not the best, result.

A second is whether the paper is conceptually coherent. This is often operationalized as “does it test a theory?” or unfortunately sometimes “does this test the theory I would test if I were doing this study?” More broadly, the field is in some cases close to strangling itself by insisting that every paper have and test a theory/theoretical basis. A better criterion is “does this paper present a result which makes one think and encourages future research, and did the author make a reasonable attempt to explain/provide a coherent story for the results?” (Interestingly and importantly, the “newer” something is, and therefore potentially more impactful, the less well it will be tied to prior theory precisely because it is new.)

The third area is an assessment of marginal contribution. Here again it is fine to provide your judgement of this as long as you recognize that others may differ in this assessment.

The fourth area is the trickiest. It involves “taste.” Taste colors judgment whether we want it to or not. It is important to identify which of your concerns are about taste, make them explicit, recognize that tastes can differ, and leave it to the editor to decide how to incorporate this into their decision.

Other considerations

Tone

The tone of a review matters. While some authors are “tough” and even enjoy contesting reviews, most are not and do not. Further, most authors want to improve their work if a specific and reasonably doable path to do that is made clear to them. Thus tone matters. Even for work that you feel is well below the (your) standards, that is not justification for belittling the authors or not showing some sympathy to a fellow author who may be just starting out or at an institution without much support for research. Put differently, remember that while you are presumably an expert in some aspects of a paper, you are not “the” expert in all of them. Humility is a plus.

Rigidity

There are few absolutes. Insisting on a particular way of doing things without strong evidence that (a) it is superior in this particular case and (b) doing them that way makes a material difference in the results (i.e., in the first decimal place of an estimate) is counter-productive. If an approach gives materially different results, subsequent replication attempts and meta-analyses will show that. Insisting on a single complex way of doing thing actually retards knowledge development as it perfectly confounds method with the results and precludes variation (e.g., conceptual replications).

Newness/surprise

“New” is a relative term. There is essentially nothing we do that can’t be related to prior work or thinking, whether by marketing scholars, scholars in other disciplines, philosophers, etc. Currently there is a tendency to dismiss a paper that doesn’t have surprising findings as “not a contribution.” This effectively rules out most replications (exact or conceptual) and inhibits future meta-analyses, which are critical for establishing empirical generalizations. Establishing an empirical regularity of an important phenomena is useful even if it is not exciting.

Cleverness/complexity

In the behavioral area, main effect findings are often considered inferior/inadequate when in fact they can be the most meaningful (after all, E = mc² is a pretty simple formula). While it is useful and interesting to look for mediators, moderators, moderated mediation, etc., they should not be requirements for all behavioral papers. Similarly, in the empirical quantitative area, complex estimation methods are not always required or even the most appropriate (McAlister 2016).

Empathy

It is always important to remember that behind every manuscript is many person-hours of hard labor. For empirical work, data collection takes time. Whether it is scraping data from the Internet, administering a survey across multiple countries with suitable translations, running 15 lab studies, or assembling secondary data, the amount of time it takes to create a suitable database can be substantial. Even without data, the author of a theory paper takes many hours to read the literature and develop and hone concepts. Many more hours are spent drafting and revising a manuscript. It is easy as a reviewer to say that more and/or different data or experiments are needed or to make flippant comments that the theory is wrong. Showing a little courtesy and empathy for authors does not hurt.

In addition, careers are at stake. For many academics, getting as few as one or two papers in the top journals can create a tenure case at their schools. While this should never be a criterion for making a recommendation on a manuscript, being careful and diligent in preparing the review as well as following the guidelines noted previously can be enormously helpful, especially to junior colleagues.

Suggestions

• 1. Start every review with “What I like about this paper is” and remember that as you do your review. If you think the paper is good enough to be published, go ahead and say so. A good critical review is not necessarily a negative one.

• 2. Remember the paper is the authors’ paper, not yours, and that their name will appear on it (and not yours). Don’t in effect take over and try to dictate exactly what they “must” do.

• 3. Remember that authors may not respond to every one of your suggestions no matter how great you think they are. They have to respond to several reviewers, an AE, and an Editor which often pull them in different directions.

• 4. Don’t treat the review process as a legal proceeding. A paper can be acceptable even if it doesn’t deal with all your concerns and suggestions. Also try not to introduce new, non-critical suggestions on subsequent rounds of the review process.

• 5. Be timely. It takes you no more time to review paper early than it does late.

• 6. Don’t be upset if editors don’t follow your recommendations. You are an advisor, they are the decision makers.

• 7. Try to complete your review in one sitting. It will save you time remembering where you were and produce a more coherent/consistent review.

• 8. Brevity is key. Indicate the (few) key issues succinctly. Length does not equal quality. Also, clearly separate what you think are major concerns/suggestions from minor “it would be nice to do” points.

• 9. As you review, so shall you be reviewed. Nasty reviews beget more nasty reviews, thoughtful ones encourage thoughtful ones. Which would you prefer to get?

Conclusion

The demand for space in the top marketing journals has increased faster than the supply. As a result, while the so-called “A” journals in marketing have remained fairly constant for a long time, a number of journals such as JAMS have increased dramatically in quality (and in fact are now included in the Financial Times list of top journals). However, due to the growth in the number of marketing scholars around the globe, there is still intense competition for the available journal space. This has significantly increased the pressure on the peer review system since every paper submitted typically needs at least two reviewers, an AE, and an editor to make a decision.

Recruiting enough volunteers to man this system is challenge enough. To make the system work for both the journals and the authors is even harder. There are many critics of the review process; who hasn’t seen criticism of reviewers on Facebook or heard it at a conference? Overall, the system is working fairly well despite its problems. Griping aside, our papers have benefited from the process and, if ultimately published, were better than the versions originally submitted. We also feel that despite the amount of information available about papers and authors on the Internet, the integrity of the review process has remained intact.

The main challenges are really at the individual reviewer level. As authors, we expect to receive fair, helpful reviews from readers who pay careful attention to our work. As reviewers, we are obligated provide nothing less than that. Our observation is that we are lucky in marketing to have a review process that is generally working. Many of us have submitted papers to journals in other fields and have suffered from long review times and cursory or unhelpful reviews. While that happens occasionally in marketing, our experience is that it is not the norm in our field.

We have provided an historical context for the review process in marketing, some observations on it, and some suggestions for how it could be better. Starting with the reviews that we are currently processing, we intend to take our own advice.