The controversy

In November 2020, The New York Times reported about the fresh findings of a research study showing that a new nasal spray prevents COVID-19 infection in ferrets (McNeil, 2020). The Times asked experts to chime in and sprinkled the article with short quotations from researchers. Soon, other outlets picked up the news and wrote about the same results. In the fall of 2020, such news was celebrated for good reasons. The New York Times article also mentioned that the study was not yet peer reviewed.

Reading about research findings in the news, we run into different variations of the above expression, telling us that “the results have not been yet peer reviewed.” News articles with such disclaimers often include comments from the authors who conducted the research and sometimes remarks from researchers who were not part of the study (e.g., Beachum et al., 2021; Handy & Jaswal, 2021; Vartabedian, 2020).

Writing to the public about research findings is a positive phenomenon. Public interest in scientific research is welcomed, just like the public’s access to research findings. Much of the scientific work is funded by tax money and is justified by serving the public interest, and that warrants some level of transparency regarding its output (Although it does not license its lay evaluation. Citizens have the right to know how their town spent the money on building a new bridge, but they better not hold a vote on its stress test).

But reporting preliminary research findings to the public can be problematic. In teaching experimental research methods, we tell students that science is a process and that it can generate reliable knowledge if we follow the specific steps of this process. One of these steps is subjecting our work to peer review—letting experts of the same field evaluate the strength of our claims and the value of our work.

What peer review is and what it is not

Peer review is an integral step of the scientific process. Reviewers should be in a position to know if the presented ideas and findings represent a meaningful contribution to the scientific field. Sometimes they don’t or only very marginally and then the expectation is that these works would end up in journals with lower prestige and impact. Besides the level of contribution, reviewers should also be able to point out errors. They can provide useful comments about the methods, notice errors in the study design or statistical analysis, or find that the results are inconclusive. Reviewers may reveal that a different conclusion should be drawn based on the presented results, or that the findings contradict established theories in the literature. Ideally, papers would not get published and become part of scientific knowledge until such errors are addressed.

Manuscripts sometimes don’t pass the peer review test and don’t get published. More often, in case of initial rejection, manuscripts still find their way to get published in other journals. When manuscripts get published in the journal of the authors’ first choice or in a more obscure outlet, the papers are usually different, and hopefully better, than the original submissions. The reasons for the change are the peer reviews and the authors’ compliance with the experts’ suggestions. This change might be minor, like adding a few figures and clearing up confusing language. But it can mean correcting serious methodological and statistical problems, adding studies to present more convincing evidence, or writing a much-modified conclusion at the end of the paper. Ideally, the review process is supportive and the reviewers want to improve the quality of the manuscript.

Peer review does not safeguard against errors, and its quality fluctuates – in a highly non-random way. Journals with higher prestige and impact factor can typically recruit experts who can provide more fundamental and more relevant feedback. Even for these journals, it is not the reviewers’ job to attempt replicating the studies or to rerun the analyses. Reviewers may also be wrong, may not realize errors or flaws in the manuscripts, may make subjective or biased decisions, or just have limited time to accomplish the task. Despite these shortcomings, the scientific community does not have a better solution yet for evaluating and sorting articles for publication purposes (Jennings, 2006). If researchers adhere to the peer review process, they cannot be assured that their findings and conclusion will stand until they see that in the editor’s letter.

If peer review is such an integral part of the scientific process, why don’t researchers simply wait for the peer review to arrive before they start publicizing their findings in the media? The answer is that every involved party is interested in promptly publicizing new research results, even when they are not time sensitive.

Giving information to the public is not giving ground to the media

Research findings can sell well in popular media, but the media’s appetite for science is specific. Null effects, for example, are not exciting. Positive, new, and surprising results get attention, creating incentives that hurt the quality of science (Strack, 2012). Interest in the hot and fresh findings offers researchers the opportunity to be invited and talk about their work in popular and social media prematurely. Such opportunities are well aligned with researchers’ aspiration for recognition from peers and the public for their accomplishments, particularly for being the first to discover something (Bergstrom & West, 2021).

Research institutions and scientific organizations also rely on the media to influence public interest and sway decision-makers in order to gain access to resources. Universities encourage researchers to increase their visibility and young scholars feel a growing pressure to let the world know about their contribution, opinion, or merely their name in the daily news and on social media. Scientific journals also benefit from the media attention that the papers they potentially publish receive, and editors have to have a good sense of predicting the public interest that the submitted manuscripts would generate.

Discussing new findings with and gaining feedback from the research community are useful elements of the scientific endeavor. There are different theoretical models of science communication pertaining to expert-public interaction. In general, these models, as well as policy suggestions, have been shifting from simple dissemination models to a more participatory dialogue that emphasizes the importance of engagement with the public (Bucchi & Trench, 2008), and this shift is partly driven and enabled by electronic communication and storage media (Hurd, 2000). Importantly, these models, especially the more participatory ones, should also increase the public’s trust in science (Lewenstein, 2003).

Of course, the boundary between scientific and public audiences is not crystal clear. Preprint servers, for example, provide access to pre-peer reviewed articles. Authors can deposit their manuscripts on these online archives and distribution sites before peer review and publication, and the servers provide access to these works to the scientific community but also to the public. Given the possible length of the peer review cycles and the value of open science, these sites serve important functions—and, in some fields, are mainstream publication venues (see more about the changing role of preprint servers, e.g., Else, 2020; Gitlin, 2020). Speeding up the process of high-quality peer review could be one way to reduce the reliance on preprint servers. Even if preprint servers make it clear when manuscripts are not peer reviewed (e.g., arXiv, bioRxiv), as researchers and the public more and more often refer to not peer reviewed papers, the value and perceived relevance of peer review will diminish. This reduces the reliability of science.

It doesn’t mean that researchers should not talk about their work to the public until it appears in a journal. They can talk about their ideas and hypotheses and share their experience with and enthusiasm about ongoing projects (many last for years or even decades). But if they do talk about these to the public, researchers and journalists should clearly indicate the status of the scientific information in the publication process. When news articles report about a research paper and say that it is not peer reviewed yet, the readers likely assume that it will be. Moreover, peer review is a technical term of the research community, and lay readers may easily misinterpret it. If anything, researchers should refrain from adding to the already substantial problem of dis- and misinformation.

The nasal spray tested on ferrets turned out to be a promising solution, and the manuscript that reported the findings was accepted for publication after peer review (Vries et al., 2021). But what if it didn’t? Would the same news outlets publish another piece alerting the readers that the peer review found too many issues with the manuscript and it was rejected? Would they issue a statement of correction saying that during the peer review process, the authors corrected their erroneous interpretation of the results? Or they just assume it doesn’t matter anymore because no one remembers yesterday’s news and nobody cares about negative findings (Dumas-Mallet et al., 2017).

Recommendations

What is the solution? Researchers may have it, and it has nothing to do with their level of confidence in their findings (that is often a faulty measure). It is about researchers’ confidence in the scientific method that entails their work’s evaluation by experts in the form of peer review. Before that, they cannot be sure what is their work’s contribution. In order to disseminate reliable scientific knowledge to the public and properly follow our scientific research standards, researchers should be very cautious with contributing to publicizing their pre-peer reviewed research papers. The right protocol should be developed by the scientific community. To start a discussion, here is a possible guideline with two options that researchers may follow.

Option one: As the baseline, researchers should hold on, wait for the editor’s letter, and don’t talk about their findings to the public until the peer review is done.

Option two: If researchers are absolutely certain that the public must know about their work immediately, pre-peer review, then they should communicate its status very clearly and don’t let their hope, confidence, or jargon create any ambiguity. Ideally, a short label expressing the paper’s status in the scientific process would serve this purpose. But that assumes that the public is trained to understand scientific research and the publication process, thus putting the responsibility on the public. Instead, the researchers should bear it. Striking the right balance between transparency and delivering clear messages to the lay public is not trivial. Researchers should say if their work has or has not been submitted to a scientific journal and if it has or has not been accepted for publication. Depending on the manuscript’s submission and review status, they could say, for example, that This article has been submitted to a scientific journal but so far has not been accepted for publication. The phrasing can be adjusted to different scenarios and say if the manuscript has not been submitted to a journal or if it has been accepted already. Authors can also add information about the manuscript’s pre-print status.

The changing fashion and accelerating speed of reporting and publicizing research findings provide us with the opportunity to reconsider the role and importance of the peer review process and the publication practices we follow. There are ways to improve the quality of peer review, maybe even its speed. But as long as we adhere to it, we must take it seriously.