Every January, the American Dialect Society (ADS) votes on the “word(s) of the year,” reflecting a word or phrase that has inordinately affected a culture and its citizens (obviously, primarily those in the United States). In 2010, the term earning the crown title was “app,” perhaps to the glee of Steve Jobs and “Macophiles” but chagrin of English professors everywhere. Other winners in recent years have included “tweet” in 2009; “plutoed” (“to demote or devalue someone or something, as happened to the former planet”) in 2006; “truthiness” (“the quality of preferring concepts or facts one wishes to be true, rather than concepts or facts known to be true”) in 2005; and even “weapons of mass destruction” or its abbreviation “WMD” in 2002 (American Dialect Society n.d.).
If the ADS were to poll academicians strictly about academic matters, however, this year’s winner—or at least a worthy candidate—might be “peer review.” It seems we are all talking about it. Or thinking about it. It is, after all, the “golden standard” in academia, the stick (or carrot) by which quality, funding, and promotion and reputation are measured. It is thus a process we never should ignore or take for granted.
But why does it continue to trouble us so? As bioethicist David B. Resnik explains in his recent essay on the subject, it certainly is not new. The process of peer review stems from the “mid 1700s,” when the editors of the Philosophical Transactions of the Royal Society of London “launched a system of peer review to evaluate manuscripts before publication” (Resnik 2011, 24).
“Two centuries went by, however,” Resnik continues, “before the system really caught on” (2011, 24)—and caught on it has. The process should be, by now, trouble-free.
Despite its obvious purposes and advantages, and there are many, there are potential hazards around every corner. But first, the “pros.”
Peer review “serves two distinct functions: It ensures that work is published only if it meets appropriate standards of scholarship and methodology, and it helps authors improve their manuscripts” (Resnik 2011, 24). A paper by Wendy Lipworth, Ian Kerridge, Stacy M. Carter, and Miles Little published in this issue of the Journal of Bioethical Inquiry (JBI) offers a similar initial description, that the “process is well entrenched in academic biomedicine and is used not only as a means of ensuring the quality and dissemination of published material, but also as a means of distinguishing those scientists who deserve academic promotion from those who lack the requisite publishing—and by implication scientific or clinical—skills and dedication” (2011, this issue). As Paul A. Komesaroff, Ian Kerridge, and Wendy Lipworth wrote in a 2008 editorial in the JBI, this “basic goal of reviewing is to facilitate publication of high-quality articles, not to act as a gatekeeper or to protect the interests or reputations of particular journals or institutions” (2008, 3).
But there is more. “Reviewing also has the potential to provide a means for engaging authors and readers in an interactive dialogue around the form and substance of their academic discourses and to encourage conversations about the development and deepening of ideas and practices” (Komesaroff, Kerridge, and Lipworth 2008, 3). This is a mission of the JBI (and many other journals): that is, to act as “a forum for dialogue across conventional academic boundaries” and for discussion and debate of “ethical, cultural and social issues arising in medicine, the health sciences and health care in general … in different geographical and cultural settings” (see “Aims and Scope” at http://www.springer.com/medicine/journal/11673).
In 2006, Nature created a web debate centered around the peer-review process as it began its trial of an “open” method (Greaves et al. 2006), and Eugene Koonin, Laura Landweber, David Lipman, and Ros Dignon discuss whether this and our “immensely powerful means of communication in our information age” might “revitalize a culture of scientific debate” (2006, ¶1 and ¶3).
Lipworth, Kerridge, Carter, and Little found among their study’s participants of reviewers, editors, and authors that not only are the “scientific” goals and processes of peer review valued but also the “social dimensions” such as nurturing colleagues and communities, moral responsibilities to authors, and “social interactions and … the value of collective, rather than isolated, reasoning processes” (2011, this issue).
There is much to be gained by peer review, assuming the system involves integrity, constructive criticism, and efficiency.
What are the costs?
As most of us know, being a peer reviewer requires professional energy and time—and often with little or no credit. Some journals, whether they employ an open or closed system, do acknowledge reviewers outright. This has been a recent discussion point during our monthly JBI board meetings, and a suggestion has been put forth to publish the names of reviewers in the JBI at the end of every year.
This might solve one problem, but a related one remains: do institutions properly credit peer reviewers as part of annual evaluations and/or tenure and promotion? As academicians we receive kudos and other benefits for publishing in peer-reviewed journals. In fact, peer-reviewed publications—particularly in national or international journals—often count more than others such as full-length books, book chapters, and both peer-reviewed and invited presentations. But peer review cannot exist without reviewers, and it seems absurd to be expected to publish in such publications without also being recognized for acting in turn as a reviewer for others.Footnote 1 The payoff for doing so cannot solely be some (unscientific!) karmic sense that another unknown colleague will be willing to do for you in a timely and professional manner as you are for him or her.
Of course, time willing, many of us want to engage in the peer-review process in order to be a part of this scholarly dialogue and for our own edification—institutional remuneration be damned. The invitation to be a peer reviewer, however, typically is unexpected, adding one more time-consuming line-item to an already overburdened to-do list. We must pick and choose. And this poses problems for journals and their editors, who must find ways to increase their reviewer databases, expanding the pool not only for reasons of breadth and depth but also not to exhaust those who have in the past said “yes.” As anyone who has served on a committee or board knows, those who accept such duties and perform them well tend to be tasked with more, and a small portion of professionals engage in the lion’s share of the work. This hardly is a proper “thank you” for those who have served and treated us well; it also does nothing to advance our scholarly communities and dialogues and conversations.
And what of editors’ and peer reviewers’ duties to authors? A letter to the editors by Yatan Pal Singh Balhara in this issue broaches this important question, and the 2008 editorial by Komesaroff, Kerridge, and Lipworth offers certain suggestions, including:
respect for confidentiality, constructive critique, impartiality and integrity, disclosure of dualities and conflicts of interest, and timeliness and responsiveness. It is expected that reviews are honest, courteous, prompt and constructive. Comments should be factual and, where possible, provide constructive suggestions for improvement (2008, 4).
As Balhara (2011) suggests, journal editors also must offer to authors a place at the table in the discussion of their manuscripts, for a one-sided dialogue is no dialogue at all. We at the JBI are continuing to reflect on and revise our policies and, in fact, currently have formed a subcommittee to streamline the process for reviewers and provide them with clear, easy-to-implement guidelines (and perhaps even rubrics) that ensure peer review is professional, constructive, and efficient—for all involved. We will keep you apprised of our efforts as they mature and come to fruition.
Ensuring the efficacy and integrity of the process is crucial. Authors are under “publish or perish” pressure, and the advent of digital media communication technologies, for all their benefits, seems only to have decreased the “allowed” time for research prior to publication. Publications face the same dilemma, with the responsibility of safeguarding quality on the one hand but reducing time-to-publication on the other. Journals are graded not only on this but also on “rejection rate.”
Are we—whether as journal publisher, editor, reviewer, or author—at cross-purposes, even among ourselves? Is the time-push antithetical to our true goals? Does the pressure to maintain a high rejection rate compromise the impartiality of the process and the dissemination of knowledge? Is the peer-review process, often involving two to four reviewers, even scientific? What “kind” of science should it be?
A quote from Robert Smith, former editor of the British Medical Journal, already reiterated in the JBI bears repeating:
we have little evidence on the effectiveness of peer review but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused (cited in Komesaroff et al. 2008, 4).
Resnik (2011) reaches similar conclusions, and Charles Jennings suggests that “[p]eer review is not the one true solution for all time, and given the ever-increasing digitization of scientific communication, it would be foolish to think that no better solution can ever emerge to the problem of filtering scientific information” (2006, ¶16).
What to do? “Pluto” the process? Or continue, as they say, to beat a dead horse? We suspect the conversation about peer review resembles, for better or worse, that of tenure and promotion: the guidelines are always being revised and no one ever has a final word on the matter. This sometimes bodes ill for junior faculty seeking advancement within their institutions, but perhaps there is more optimism regarding the never-ending conversation about peer review. The qualitative study by Lipworth et al. (2011) suggests that many editors, reviewers, and authors give much thought to both the peer-review process itself and the responsibilities, duties, and benefits—scientific and social—associated with it. There exist (heretofore overlooked or unrecognized?) possibilities and opportunities that can be tapped, provided those who participate in peer review have access to resources, support, and recognition of efforts.
We are, to use yet another idiom, in a “pickle.” What we in academia value most (peer-reviewed publications) is only partially rewarded—and difficult to protect in terms of quality and what we understand about “truth” at ever-increasing demands of speed. Thus, we ask you, our readers, authors, and colleagues, for feedback and suggestions. What does the peer-review process mean to you? Are you rewarded for participating in it, no matter your role and, if so, how? How should the process be revised and enhanced? Should it be open or anonymous? Should reviews be published alongside manuscripts, furthering our debate and sense of scholarly exchange? And how should peer reviewers, those often faceless supporters, be thanked for their time and professional expertise?
Academic journals are an important vehicle to facilitate conversations within our large, disparate disciplines and to disseminate credible knowledge about ourselves and the world. Through them, we communicate and better our communities, meet new colleagues and keep in touch with others, collaborate with and bounce ideas off of one another, and grow our own research ideas and agendas. As with conferences, however, there is little time and money to delve into even one journal’s issue, let alone the several we probably wish we had a spare moment to read. Often, we come to journals only when we are searching the literature as background to our own research, that “palpable” work for which we are rewarded. Peer-reviewing sometimes is one of the few chances we get to critically read a paper we might never have otherwise read.
We offer here yet another “two cents” and hope it adds something more than “truthiness” to the latest discussion of peer review. We still submit to the American Dialect Society this as our nomination for 2011’s “word of the year.” And if the ADS fails to agree with us, we suspect there will be many chances to propose it again and again.
A similar trend may be happening regarding academic conferences and the ability or willingness of institutions, particularly during difficult financial times, to compensate faculty with travel funds. Should only those who present research at conferences be remunerated? Should they be given enough funding merely to present and leave or to stay for the conference’s duration? Such limitations possibly diminish academic life and, at some point, we might find we are presenting our peer-reviewed conference papers solely to our fellow panel members.
American Dialect Society. No date. Words of the year: The American Dialect Society annual word(s) of the year. http://www.americandialect.org/index.php/amerdial/categories/C178/ (accessed May 28, 2011).
Balhara, Y.P.S. 2011. Rights of authors: Do we need guidelines for editors as well? Journal of Bioethical Inquiry 8(3). doi:10.1007/s11673-011-9307-1.
Greaves, S., J. Scott, M. Clarke, L. Miller, T. Hannay, A. Thomas, and P. Campbell. 2006. Overview: Nature’s peer review trial. Despite enthusiasm for the concept, open peer review was not widely popular, either among authors or by scientists invited to comment. Nature (December). doi:10.1038/nature05535 (accessed May 28, 2011).
Jennings, C. 2006. Quality and value: The true purpose of peer review. What you can’t measure, you can’t manage: The need for quantitative indicators in peer review. Nature. doi:10.1038/nature05032 (accessed May 28, 2011).
Komesaroff, P.A., I. Kerridge, and W. Lipworth. 2008. The epistemology and ethics of journal reviewing: A second look. Journal of Bioethical Inquiry 5(1): 3–6.
Koonin, E., L. Landweber, D. Lipman, and R. Dignon. 2006. Systems: Reviving a culture of scientific debate: Can “open peer review” work for biologists? Biology Direct is hopeful. Nature. doi:10.1038/nature05005 (accessed May 28, 2011).
Lipworth, W., I. Kerridge, S.M. Carter, and M. Little. 2011. Should biomedical publishing be “opened up”? Toward a values-based peer-review process. Journal of Bioethical Inquiry 8(3). doi:10.1007/s11673-011-9312-4.
Resnik, D.B. 2011. A troubled tradition: It’s time to rebuild trust among authors, editors and peer reviewers. American Scientist 99(1): 24–27.