Skip to main content
SpringerLink
Log in

Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

The peer review process is the gold standard by which academic manuscripts are vetted for publication. However, some investigators have raised concerns regarding its unopposed supremacy, including lack of expediency, susceptibility to editorial bias and statistical limitation due to the small number of reviewers used. Post-publication review—in which the article is assessed by the general readership of the journal instead of a small group of appointed reviewers—could potentially supplement or replace the peer-review process. In this study, we created a computer model to compare the traditional peer-review process to that of post-publication reader review.

Methods

We created a mathematical model of the manuscript review process. A hypothetical manuscript was randomly assigned a “true value” representing its intrinsic quality. We modeled a group of three expert peer reviewers and compared it to modeled groups of 10, 20, 50, or 100 reader-reviewers. Reader-reviewers were assumed to be less skillful at reviewing and were thus modeled to be only ¼ as accurate as expert reviewers. Percentage of correct assessments was calculated for each group.

Results

400,000 hypothetical manuscripts were modeled. The accuracy of the reader-reviewer group was inferior to the expert reviewer group in the 10-reviewer trial (93.24% correct vs. 97.67%, p < 0.0001) and the 20-reviewer trial (95.50% correct, p < 0.0001). However, the reader-reviewer group surpassed the expert reviewer group in accuracy when 50 or 100 reader-reviewers were used (97.92 and 99.20% respectively, p < 0.0001).

Conclusions

In a mathematical model of the peer review process, the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size. Further study will be required to determine whether the mathematical assumptions of this model are valid in actual use.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. To achieve a random number in a normal distribution with a standard deviation of n, we used the following Excel formula: NORMSINV(RAND()) * n.

References

  1. Jefferson R, Alderson P, Wager E, Davidoff F (2002) Effects of editorial peer review: a systematic review. JAMA 287:2784–2786

    Article  PubMed  Google Scholar 

  2. Jefferson T, Rudin M, Brodney Folse S, Davidoff F (2007) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev (2). Article No. MR000016

  3. Cole S, Cole JR, Simon GA (1981) Chance and consensus in peer review. Science 215:881–886

    Article  Google Scholar 

  4. Rothwell PM, Martyn CN (2000) Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone? Brain 123:1964–1969

    Article  PubMed  Google Scholar 

  5. Bornmann L, Daniel HD (2008) The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angew Chem Int Ed 47:7173–7178

    Article  CAS  Google Scholar 

  6. Bornmann L, Daniel HD (2010) The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal. PLoS One 5(6):e11344

    Article  PubMed  Google Scholar 

  7. Bornmann L, Daniel HD (2010) Do author-suggested reviewers rate submissions more favorably than editor-suggested reviewers? A study on Atmospheric Chemistry and Physics. PLoS One 5(10):e13345

    Article  PubMed  Google Scholar 

  8. Arms WY (2002) What are the alternatives to peer review? Quality control in scholarly publishing on the web. J Electron Publ 8:1

    Article  Google Scholar 

  9. Mandavilli A (2011) Trial by twitter. Nature 469:286–287

    Article  PubMed  CAS  Google Scholar 

  10. Ware M. Peer review: benefits, perceptions and alternatives, Publishing Research Consortium 2008. http://www.publishingresearch.org.uk. Accessed 2 Feb 2011

  11. ArXiv.org. http://www.arxiv.org. Accessed 2 Feb 2011

  12. Description of the RAND function in Excel. Article ID 828795, Rev. 6.0. http://support.microsoft.com/kb/828795. Accessed 5 Jan 2011

  13. Schroter S, Black N, Evans S et al (2008) What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 101:507–514

    Article  PubMed  Google Scholar 

  14. Galton F (1907) Vox Populi. Nature 75:450–451

    Article  Google Scholar 

  15. Galton F (1907) The ballot box. Nature 75:509

    Google Scholar 

  16. Surowiecki J (2004) The wisdom of crowds. Random House, New York

    Google Scholar 

  17. Nature.com. http://www.nature.com/nature/peerreview/debate/nature05535.html. Accessed 2 Feb 2012

  18. Hall JC. How to dissect surgical journals. http://www.anzsurg.com/view/0/dissectingSurgicalJournals.html. Accessed 23 Dec 2012

  19. Smith R (1997) Peer review: reform or revolution? BMJ 315:759–760

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

Special Thanks are extended to Dr. David Urbach, MD, MSc, FACS, FRCSC from the Departments of Surgery and Health Policy in the University of Toronto for his assistance with details of the experimental model.

Disclosure

Dr. Herron holds stock options in Hourglass Technology.

Author information

Authors and Affiliations

  1. Mount Sinai School of Medicine, 1 Gustave L. Levy Place, #1259, New York, NY, 10029, USA

    Daniel M. Herron

Authors
  1. Daniel M. Herron
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel M. Herron.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Herron, D.M. Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review. Surg Endosc 26, 2275–2280 (2012). https://doi.org/10.1007/s00464-012-2171-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-012-2171-1

Keywords

Search

Navigation