Skip to main content

Evaluating the Pros and Cons of Different Peer Review Policies via Simulation

Science and Engineering Ethics volume 22, pages 1073–1094 (2016)Cite this article

Abstract

In the academic world, peer review is one of the major processes in evaluating a scholars contribution. In this study, we are interested in quantifying the merits of different policies in a peer review process, such as single-blind review, double-blind review, and obtaining authors feedback. Currently, insufficient work has been undertaken to evaluate the benefits of different peer review policies. One of the major reasons for this situation is the inability to conduct any empirical study because data are presently unavailable. In this case, a computer simulation is one of the best ways to conduct a study. We perform a series of simulations to study the effects of different policies on a peer review process. In this study, we focus on the peer review process of a typical computer science conference. Our results point to the crucial role of program chairs in determining the quality and diversity of the articles to be accepted for publication. We demonstrate the importance of discussion among reviewers, suggest circumstances in which the double-blind review policy should be adopted, and question the credibility of the authors feedback mechanism. Finally, we stress that randomness plays an important role in the peer review process, and this role cannot be eliminated. Although our model may not capture every component of a peer review process, it covers some of the most essential elements. Thus, even the simulation results clearly cannot be taken as literal descriptions of an actual peer review process. However, we can at least still use them to identify alternative directions for future study.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1

References

  • Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3, 63–84.

    Article  Google Scholar 

  • Arrow, K. J. (1964). Social choice and individual values (2nd ed.). New York: Wiley.

    Google Scholar 

  • Bachand, R. G., & Sawallis, P. P. (2003). Accuracy in the identification of scholarly and peer-reviewed journals and the peer-review process across disciplines. The Serials Librarian, 45(2), 39–59.

    Article  Google Scholar 

  • Barry, B. M., & Hardin, R. (1982). Rational man and irrational society?. Newbury Park, CA: Sage.

    Google Scholar 

  • Billard, L. (1993). Comment. Statistical Science, 8(3), 320–322.

    Article  Google Scholar 

  • Blank, R. M. (1991). The effects of double-blind versus single-blind reviewing: Experimental evidence from the american economic review. American Economic Review, 81(5), 1041–1067.

    Google Scholar 

  • Brams, S. J., & Fishburn, P. C. (1983). America’s unfair elections. Sciences, 23(6), 28–34.

    Article  Google Scholar 

  • Campanario, J. M. (1998). Peer review for journals at it stands today part 1. Science Communication, 19(3), 181–211.

    Article  Google Scholar 

  • Carland, J. A., Carland, J. W., & Aby, C. D. J. (1992). Proposed codification of ethicacy in the publication process. Journal of Business Ethics, 11, 95–104.

    Article  Google Scholar 

  • Ceci, S. J., & Peters, D. P. (1984). How blind is blind review? American Psychologist, 29, 1491–1494.

    Article  Google Scholar 

  • Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., & Waecklerle, J. F. (1998). Masking author identity in peer reviewat factors influence masking success? Journal of the American Medical Association, 280(3), 243–245.

    Article  Google Scholar 

  • Cox, D., Gleser, L., Perlman, M., Reid, N., & Roeder, K. (1993). Report of the ad hoc committee of double-blind refereeing. Statistical Science, 8(3), 310–317.

    Article  Google Scholar 

  • Dalton, M. S. (1995). Refereeing of scholarly works for primary publishing. Annual Review of Information Science and Technology, 30, 213–250.

    Google Scholar 

  • Evans, A. T., McNutt, R. A., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review: A randomized trial. Clinical Research, 38(2), 1497–1525.

    Google Scholar 

  • Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association, 272(2), 143–146.

    Article  Google Scholar 

  • Fletcher, R. H., & Fletcher, S. W. (1997). Evidence for the effectiveness of peer review. Science and Engineering Ethics, 3(1), 35–50.

    Article  Google Scholar 

  • Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., & Nuutinen, A. (2012). Panel discussion does not improve reliability of peer review for medical research grant applications. Journal of Clinical Epidemiology, 65(1), 47–52.

    Article  Google Scholar 

  • Franzini, L. R. (1987). Editors are not blind. American Psychologist, 42, 104.

    Article  Google Scholar 

  • Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawon, E. E. (1994). Effect of institutional prestige on reviewers recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137–138.

    Article  Google Scholar 

  • Genest, C. (1993). Comment. Statistical Science, 8(3), 323–327.

    Article  Google Scholar 

  • Goldberg, P. (1968). Are some women prejudiced against women? Transaction, 5, 28–30.

    Google Scholar 

  • Harnad, S. (1982). Peer commentary on peer review. Behavioral and Brain Sciences, 5(2), 185–186.

    Google Scholar 

  • Hill, S., & Provost, F. (2003). The myth of the double-blind review. SIGKDD Explorations, 2(5), 179–184.

    Article  Google Scholar 

  • Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? Journal of the American Medical Association, 280(3), 240–242.

    Article  Google Scholar 

  • Korngreen, A. (2005). Peer-review system could gain from author feedback. Nature, 438, 282.

    Article  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

    Article  Google Scholar 

  • Link, A. M. (1998). Us and non-us submission: An analysis of review bias. Journal of the American Medical Association, 280(3), 246–247.

    Article  Google Scholar 

  • Madden, S., & DeWitt, D. (2006). Impact of double-blind reviewing on SIGMOD publication rates. SIGMOD Record, 35(2), 29–32.

    Article  Google Scholar 

  • McGiffert, M. (1988). Is justice blind? An inquiry into peer review. Scholarly Publishing, 20(1), 43–48.

    Google Scholar 

  • Perlman, D. (1982). Reviewer bias: Do peters and ceci protest too much? Behavioral and Brain Sciences, 5(2), 231–232.

    Article  Google Scholar 

  • Rennie, D., & Flanagin, A. (1994). The second international congress on peer review in biomedical publication. Journal of the American Medical Association, 272(2), 91.

    Article  Google Scholar 

  • Resnik, D. B., & Elmore, S. A. (2015). Ensuring the quality, fairness, and integrity of journal peer review: A possible role of editors. Science and Engineering Ethics, 1–20. doi:10.1007/s11948-015-9625-5.

  • Snodgrass, R. T. (2003). Developments at TODS. SIGMOD Record, 32(4), 14–15.

    Article  Google Scholar 

  • Snodgrass, R. T. (2006). Single- versus double-blind reviewing: An analysis of the literature. SIGMOD Record, 35(3), 8–21.

    Article  Google Scholar 

  • Spier, R. E., & Poland, G. A. (2013). What is excellent science and how does it relate to what we publish in vaccine? Vaccine, 31(45), 5147–5148.

    Article  Google Scholar 

  • Tung, A. K. H. (2006). Impact of double blind reviewing on SIGMOD publication: A more detail analysis. SIGMOD Record, 35(3), 6–7.

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the Youth Teacher Startup Fund of South China Normal University (No. 14KJ18), and was partially supported by the Natural Science Foundation of China (Grant No. 61402313) and the National High Technology Research and Development Program of China (863, No. 2013AA01A212).

Author information

Affiliations

  1. School of Computer Science, South China Normal University, Guangzhou, China

    Jia Zhu & Chuanhua Xu

  2. Department of Information Technology, Lab Vsio, Hong Kong, China

    Gabriel Fung

  3. School of Decision Sciences, Hang Seng Management College, Hong Kong, China

    Wai Hung Wong

  4. School of Computer Science and Technology, Soochow University, Suzhou, China

    Zhixu Li

Authors
  1. Jia Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabriel Fung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wai Hung Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhixu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chuanhua Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jia Zhu.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhu, J., Fung, G., Wong, W.H. et al. Evaluating the Pros and Cons of Different Peer Review Policies via Simulation. Sci Eng Ethics 22, 1073–1094 (2016). https://doi.org/10.1007/s11948-015-9683-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11948-015-9683-8

Keywords

  • Peer review
  • Social behavior
  • Simulation