Surgical Endoscopy

, Volume 26, Issue 8, pp 2275–2280 | Cite as

Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review

Article

Abstract

Background

The peer review process is the gold standard by which academic manuscripts are vetted for publication. However, some investigators have raised concerns regarding its unopposed supremacy, including lack of expediency, susceptibility to editorial bias and statistical limitation due to the small number of reviewers used. Post-publication review—in which the article is assessed by the general readership of the journal instead of a small group of appointed reviewers—could potentially supplement or replace the peer-review process. In this study, we created a computer model to compare the traditional peer-review process to that of post-publication reader review.

Methods

We created a mathematical model of the manuscript review process. A hypothetical manuscript was randomly assigned a “true value” representing its intrinsic quality. We modeled a group of three expert peer reviewers and compared it to modeled groups of 10, 20, 50, or 100 reader-reviewers. Reader-reviewers were assumed to be less skillful at reviewing and were thus modeled to be only ¼ as accurate as expert reviewers. Percentage of correct assessments was calculated for each group.

Results

400,000 hypothetical manuscripts were modeled. The accuracy of the reader-reviewer group was inferior to the expert reviewer group in the 10-reviewer trial (93.24% correct vs. 97.67%, p < 0.0001) and the 20-reviewer trial (95.50% correct, p < 0.0001). However, the reader-reviewer group surpassed the expert reviewer group in accuracy when 50 or 100 reader-reviewers were used (97.92 and 99.20% respectively, p < 0.0001).

Conclusions

In a mathematical model of the peer review process, the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size. Further study will be required to determine whether the mathematical assumptions of this model are valid in actual use.

Keywords

Peer review Expert review Post-publication review Computer model 

References

  1. 1.
    Jefferson R, Alderson P, Wager E, Davidoff F (2002) Effects of editorial peer review: a systematic review. JAMA 287:2784–2786PubMedCrossRefGoogle Scholar
  2. 2.
    Jefferson T, Rudin M, Brodney Folse S, Davidoff F (2007) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev (2). Article No. MR000016Google Scholar
  3. 3.
    Cole S, Cole JR, Simon GA (1981) Chance and consensus in peer review. Science 215:881–886CrossRefGoogle Scholar
  4. 4.
    Rothwell PM, Martyn CN (2000) Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone? Brain 123:1964–1969PubMedCrossRefGoogle Scholar
  5. 5.
    Bornmann L, Daniel HD (2008) The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angew Chem Int Ed 47:7173–7178CrossRefGoogle Scholar
  6. 6.
    Bornmann L, Daniel HD (2010) The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal. PLoS One 5(6):e11344PubMedCrossRefGoogle Scholar
  7. 7.
    Bornmann L, Daniel HD (2010) Do author-suggested reviewers rate submissions more favorably than editor-suggested reviewers? A study on Atmospheric Chemistry and Physics. PLoS One 5(10):e13345PubMedCrossRefGoogle Scholar
  8. 8.
    Arms WY (2002) What are the alternatives to peer review? Quality control in scholarly publishing on the web. J Electron Publ 8:1CrossRefGoogle Scholar
  9. 9.
    Mandavilli A (2011) Trial by twitter. Nature 469:286–287PubMedCrossRefGoogle Scholar
  10. 10.
    Ware M. Peer review: benefits, perceptions and alternatives, Publishing Research Consortium 2008. http://www.publishingresearch.org.uk. Accessed 2 Feb 2011
  11. 11.
    ArXiv.org. http://www.arxiv.org. Accessed 2 Feb 2011
  12. 12.
    Description of the RAND function in Excel. Article ID 828795, Rev. 6.0. http://support.microsoft.com/kb/828795. Accessed 5 Jan 2011
  13. 13.
    Schroter S, Black N, Evans S et al (2008) What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 101:507–514PubMedCrossRefGoogle Scholar
  14. 14.
    Galton F (1907) Vox Populi. Nature 75:450–451CrossRefGoogle Scholar
  15. 15.
    Galton F (1907) The ballot box. Nature 75:509Google Scholar
  16. 16.
    Surowiecki J (2004) The wisdom of crowds. Random House, New YorkGoogle Scholar
  17. 17.
  18. 18.
    Hall JC. How to dissect surgical journals. http://www.anzsurg.com/view/0/dissectingSurgicalJournals.html. Accessed 23 Dec 2012
  19. 19.
    Smith R (1997) Peer review: reform or revolution? BMJ 315:759–760PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Mount Sinai School of MedicineNew YorkUSA

Personalised recommendations