Skip to main content

Advertisement

SpringerLink
Log in

Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors

  • Original Paper
  • Published:
Science and Engineering Ethics Aims and scope Submit manuscript

Abstract

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3(1), 63–84.

    Article  Google Scholar 

  • Baethge, C., Franklin, J., & Mertens, S. (2013). Substantial agreement of referee recommendations at a general medical journal–A peer review evaluation at Deutsches Ärzteblatt International. PLoS ONE, 8(5), e61401.

    Article  Google Scholar 

  • Baggs, J. G., Broome, M. E., Dougherty, M. C., Freda, M. C., & Kearney, M. H. (2008). Blinding in peer review: The preferences of reviewers for nursing journals. Journal of Advances in Nursing, 64(2), 131–138.

    Article  Google Scholar 

  • Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and downs of peer review. Advances in Physiology Education, 31(2), 145–152.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniela, H. D. (2010). Reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331.

    Article  Google Scholar 

  • Bornmann, L., Mutza, R., & Daniela, H. D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3), 226–238.

    Article  Google Scholar 

  • Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., & Lortie, C. J. (2009). To name or not to name: the effect of changing author gender on peer review. BioScience, 59(11), 985–989.

    Article  Google Scholar 

  • Budden, A. E., Tregenza, T., Aarssen, L. W., Koricheva, J., Leimu, R., & Lortie, C. J. (2008). Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution, 23(1), 4–6.

    Article  Google Scholar 

  • Butler, D. (2010). Journals step up plagiarism policing. Nature, 466(7303), 167.

    Article  Google Scholar 

  • Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med, 4(1), e40.

    Article  Google Scholar 

  • Callaham, M. L., Wears, R. L., & Waeckerle, J. F. (1998). Effect of attendance at a training session on peer reviewer quality and performance. Annals of Emergency Medicine, 32(3 Pt 1), 318–322.

    Article  Google Scholar 

  • Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Sciences of the United States of America, 108(8), 3157–3162.

    Article  Google Scholar 

  • Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., Waeckerle, J. F., Callaham, M. L., & Rennie, D. (1998). Masking author identity in peer review: What factors influence masking success? PEER Investigators. Journal of the American Medical Association, 280(3), 243–245.

    Article  Google Scholar 

  • Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and U.S. Science Policy. Albany, NY: State University of New York Press.

    Google Scholar 

  • Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–135.

    Article  Google Scholar 

  • Committee on Publication Ethics. (2014). Code of Conduct. http://publicationethics.org/resources/code-conduct. Accessed 31 Oct 2014.

  • Cromey, D. W. (2013). Digital images are data: And should be treated as such. Methods in Molecular Biology, 931, 1–27.

    Article  Google Scholar 

  • Dickersin, K., Min, Y. I., & Meinert, C. L. (1992). Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. Journal of the American Medical Association, 267(3), 374–378.

    Article  Google Scholar 

  • Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & Reporting Bias Group. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PLoS ONE, 8(7), e66844.

    Article  Google Scholar 

  • Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research. Lancet, 337(8746), 867–872.

    Article  Google Scholar 

  • Editors, Nature. (2001). Editorial: Bad peer reviewers. Nature, 413(6852), 93.

    Article  Google Scholar 

  • Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352(6336), 560.

    Article  Google Scholar 

  • Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738.

    Article  Google Scholar 

  • Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association, 272(2), 143–146.

    Article  Google Scholar 

  • Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige on reviewers’ recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137–138.

    Article  Google Scholar 

  • Gasparyan, A. Y., Ayvazyan, L., Akazhanov, N. A., & Kitas, G. D. (2013). Conflicts of interest in biomedical publications: Considerations for authors, peer reviewers, and editors. Croatian Medical Journal, 54(6), 600–608.

    Article  Google Scholar 

  • Gilbert, J. R., Williams, E. S., & Lundberg, G. D. (1994). Is there gender bias in JAMA’s peer-review process? Journal of the American Medical Association, 272(2), 139–142.

    Article  Google Scholar 

  • Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. Journal of the American Medical Association, 280(3), 237–240.

    Article  Google Scholar 

  • Grod, O. N., Budden, A. E., Tregenza, T., Koricheva, J., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2008). Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PLoS ONE, 3(9), e3202.

    Article  Google Scholar 

  • Ho, R. C., Mak, K. K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013). Views on the peer review system of biomedical journals: An online survey of academics from high-ranking universities. BMC Medical Research Methodology, 13, 74.

    Article  Google Scholar 

  • Holy Bible, King James Version. (1991). New York: Random House.

  • Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Medical Education, 12, 83.

    Article  Google Scholar 

  • Hull, D. (1988). Science as a process. Chicago: University of Chicago Press.

    Book  Google Scholar 

  • Hwang, W. S., Roh, S. I., Lee, B. C., Kang, S. K., Kwon, D. K., Kim, S., et al. (2005). Patient-specific embryonic stem cells derived from human SCNT blastocysts. Science, 308(5729), 1777–1783.

    Article  Google Scholar 

  • International Committee of Medical Journal Editors. (2014). Recommendations. http://www.icmje.org. Accessed 31 Oct 2014.

  • Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. Journal of the American Medical Association, 280(3), 240–242.

    Article  Google Scholar 

  • Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010). Editorial peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors care? PLoS ONE, 5(4), e10072.

    Article  Google Scholar 

  • LaFollette, M. (1992). Stealing Into Print: Fraud, plagiarism, and misconduct in scientific publishing. Berkeley, CA: University of California Press.

    Google Scholar 

  • Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.

    Article  Google Scholar 

  • Lee, C. J. (2013). A Kuhnian critique of psychometric research on peer review. Philosophy of Science, 79(5), 859–870.

    Article  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

    Article  Google Scholar 

  • Lloyd, M. E. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis, 23(4), 539–543.

    Article  Google Scholar 

  • Lovejoy, T. I., Revenson, T. A., & France, C. R. (2011). Reviewing manuscripts for peer-review journals: A primer for novice and seasoned reviewers. Annals of Behavioral Medicine, 42(1), 1–13.

    Article  Google Scholar 

  • Mahoney, M. J. (1977). Publication preferences: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.

    Article  Google Scholar 

  • McCullough, J. (1989). First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Science, Technology and Human Values, 14(1), 78–88.

    Article  Google Scholar 

  • McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10), 1371–1376.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.

    Google Scholar 

  • Neff, B. D., & Olden, J. D. (2006). Is peer review a game of chance? BioScience, 56(4), 333–340.

    Article  Google Scholar 

  • Newton, D. P. (2010). Quality and peer review of research: An adjudicating role for editors. Accountability in Research, 17(3), 130–145.

    Article  Google Scholar 

  • Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. Journal of the American Medical Association, 272(2), 149–151.

    Article  Google Scholar 

  • Obokata, H., Wakayama, T., Sasai, Y., Kojima, K., Vacanti, M. P., Niwa, H., et al. (2014). Stimulus-triggered fate conversion of somatic cells into pluripotency. Nature, 505(7485), 641–647.

    Article  Google Scholar 

  • Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., et al. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287(21), 2825–2828.

    Article  Google Scholar 

  • Park, I. U., Peacey, M. W., & Munafò, M. R. (2014). Modelling the effects of subjective and objective decision making in scientific peer review. Nature, 506(7486), 93–96.

    Article  Google Scholar 

  • Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195.

    Article  Google Scholar 

  • Rennie, D. (2003). Misconduct and journal peer review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 118–129). London: BMJ Books.

    Google Scholar 

  • Resch, K. I., Ernst, E., & Garrow, J. (2000). A randomized controlled study of reviewer bias against an unconventional therapy. Journal of the Royal Society of Medicine, 93(4), 164–167.

    Google Scholar 

  • Resnik, D. B. (2011). A troubled tradition. American Scientist, 99(1), 24–28.

  • Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310.

  • Rivara, F. P., Cummings, P., Ringold, S., Bergman, A. B., Joffe, A., & Christakis, D. A. (2007). A comparison of reviewers selected by editors and reviewers suggested by authors. Journal of Pediatrics, 151(2), 202–205.

    Article  Google Scholar 

  • Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., et al. (2006). Effect of blinded peer review on abstract acceptance. Journal of the American Medical Association, 295(14), 1675–1680.

    Article  Google Scholar 

  • Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain, 123(Pt 9), 1964–1969.

    Article  Google Scholar 

  • Rowe, B. H., Strome, T. L., Spooner, C., Blitz, S., Grafstein, E., & Worster, A. (2006). Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Medical Research Methodology, 6, 14.

    Article  Google Scholar 

  • Salasche, S. J. (1997). How to “peer review” a medical journal manuscript. Dermatological Surgery, 23(6), 423–428.

    Article  Google Scholar 

  • Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004). Effects of training on quality of peer review: Randomised controlled trial. British Medical Journal, 328(7441), 673.

    Article  Google Scholar 

  • Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine, 101(10), 507–514.

    Article  Google Scholar 

  • Schroter, S., Tite, L., Hutchings, A., & Black, N. (2006). Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. Journal of the American Medical Association, 295(3), 314–317.

    Article  Google Scholar 

  • Shamoo, A. E., & Resnik, D. B. (2015). Responsible conduct of research (3rd ed.). New York: Oxford University Press.

  • Sismondo, S. (2008). Pharmaceutical company funding and its consequences: A qualitative systematic review. Contempory Clinical Trials, 29(2), 109–113.

    Article  Google Scholar 

  • Smith, R. (1999). Opening up BMJ peer review: A beginning that should lead to complete transparency. British Medical Journal, 318(7175), 4–5.

    Article  Google Scholar 

  • Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182.

    Article  Google Scholar 

  • Stern, J. M., & Simes, R. J. (1997). Publication bias: evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315(7109), 640–645.

    Article  Google Scholar 

  • Strayhorn, J., McDermott, J. F., & Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Acacemy of Child and Adolescent Psychiatry. Journal of the American Acacemy of Child and Adolescent Psychiatry, 150(6), 947–952.

    Google Scholar 

  • Timmer, A., Hilsden, R. J., & Sutherland, L. R. (2001). Determinants of abstract acceptance for the Digestive Diseases Week–A cross sectional study. BMC Medical Research Methodology, 1, 13.

    Article  Google Scholar 

  • Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8), 349–350.

    Article  Google Scholar 

  • van Rooyen, S., Delamothe, T., & Evans, S. J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: Randomised controlled trial. British Medical Journal, 341, c5729.

    Article  Google Scholar 

  • van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal, 318(7175), 23–27.

    Article  Google Scholar 

  • van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review: A randomized trial. Journal of the American Medical Association, 280(3), 234–237.

    Article  Google Scholar 

  • Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., et al. (1998). Ileal–lymphoid–nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet, 351(9103), 637–641.

    Article  Google Scholar 

  • Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled trial. British Journal of Psychiatry, 176, 47–51.

    Article  Google Scholar 

  • Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341–343.

    Article  Google Scholar 

  • Whittaker, R. J., et al. (2008). Journal review and gender equality: A critical comment on Budden et al. Trends in Ecology & Evolution, 23, 478–479.

    Article  Google Scholar 

  • Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting the daylight in reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6, 20.

    Article  Google Scholar 

  • Williams, P., & Wager, E. (2013). Exploring why and how journal editors retract articles: Findings from a qualitative study. Science and Engineering Ethics, 19(1), 1–11.

    Article  Google Scholar 

  • Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.

    Article  Google Scholar 

Download references

Acknowledgments

David B. Resnik is Associate Editor of Accountability in Research. Susan A. Elmore is Editor and Editor-in-Chief of Toxicologic Pathology. This research is supported by the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). It does not represent the views of the NIEHS, NIH, or U.S. government.

Conflict of interest

The authors disclose no conficts of interest.

Author information

Authors and Affiliations

  1. National Institute of Environmental Health Sciences, National Institutes of Health, Box 12233, Mail Drop CU 03, Research Triangle Park, NC, 27709, USA

    David B. Resnik

  2. National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, Research Triangle Park, NC, 27709, USA

    Susan A. Elmore

Authors
  1. David B. Resnik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susan A. Elmore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David B. Resnik.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Resnik, D.B., Elmore, S.A. Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors. Sci Eng Ethics 22, 169–188 (2016). https://doi.org/10.1007/s11948-015-9625-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11948-015-9625-5

Keywords

Search

Navigation