Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review
- 540 Downloads
The peer review process is the gold standard by which academic manuscripts are vetted for publication. However, some investigators have raised concerns regarding its unopposed supremacy, including lack of expediency, susceptibility to editorial bias and statistical limitation due to the small number of reviewers used. Post-publication review—in which the article is assessed by the general readership of the journal instead of a small group of appointed reviewers—could potentially supplement or replace the peer-review process. In this study, we created a computer model to compare the traditional peer-review process to that of post-publication reader review.
We created a mathematical model of the manuscript review process. A hypothetical manuscript was randomly assigned a “true value” representing its intrinsic quality. We modeled a group of three expert peer reviewers and compared it to modeled groups of 10, 20, 50, or 100 reader-reviewers. Reader-reviewers were assumed to be less skillful at reviewing and were thus modeled to be only ¼ as accurate as expert reviewers. Percentage of correct assessments was calculated for each group.
400,000 hypothetical manuscripts were modeled. The accuracy of the reader-reviewer group was inferior to the expert reviewer group in the 10-reviewer trial (93.24% correct vs. 97.67%, p < 0.0001) and the 20-reviewer trial (95.50% correct, p < 0.0001). However, the reader-reviewer group surpassed the expert reviewer group in accuracy when 50 or 100 reader-reviewers were used (97.92 and 99.20% respectively, p < 0.0001).
In a mathematical model of the peer review process, the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size. Further study will be required to determine whether the mathematical assumptions of this model are valid in actual use.
KeywordsPeer review Expert review Post-publication review Computer model
Special Thanks are extended to Dr. David Urbach, MD, MSc, FACS, FRCSC from the Departments of Surgery and Health Policy in the University of Toronto for his assistance with details of the experimental model.
Dr. Herron holds stock options in Hourglass Technology.
- 2.Jefferson T, Rudin M, Brodney Folse S, Davidoff F (2007) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev (2). Article No. MR000016Google Scholar
- 10.Ware M. Peer review: benefits, perceptions and alternatives, Publishing Research Consortium 2008. http://www.publishingresearch.org.uk. Accessed 2 Feb 2011
- 11.ArXiv.org. http://www.arxiv.org. Accessed 2 Feb 2011
- 12.Description of the RAND function in Excel. Article ID 828795, Rev. 6.0. http://support.microsoft.com/kb/828795. Accessed 5 Jan 2011
- 15.Galton F (1907) The ballot box. Nature 75:509Google Scholar
- 16.Surowiecki J (2004) The wisdom of crowds. Random House, New YorkGoogle Scholar
- 17.Nature.com. http://www.nature.com/nature/peerreview/debate/nature05535.html. Accessed 2 Feb 2012
- 18.Hall JC. How to dissect surgical journals. http://www.anzsurg.com/view/0/dissectingSurgicalJournals.html. Accessed 23 Dec 2012