Advertisement

Science and Engineering Ethics

, Volume 22, Issue 1, pp 169–188 | Cite as

Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors

  • David B. ResnikEmail author
  • Susan A. Elmore
Original Paper

Abstract

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making.

Keywords

Peer review Quality Fairness Integrity Ethics Reliability Bias Editors Publication 

Notes

Acknowledgments

David B. Resnik is Associate Editor of Accountability in Research. Susan A. Elmore is Editor and Editor-in-Chief of Toxicologic Pathology. This research is supported by the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). It does not represent the views of the NIEHS, NIH, or U.S. government.

Conflict of interest

The authors disclose no conficts of interest.

References

  1. Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3(1), 63–84.CrossRefGoogle Scholar
  2. Baethge, C., Franklin, J., & Mertens, S. (2013). Substantial agreement of referee recommendations at a general medical journal–A peer review evaluation at Deutsches Ärzteblatt International. PLoS ONE, 8(5), e61401.CrossRefGoogle Scholar
  3. Baggs, J. G., Broome, M. E., Dougherty, M. C., Freda, M. C., & Kearney, M. H. (2008). Blinding in peer review: The preferences of reviewers for nursing journals. Journal of Advances in Nursing, 64(2), 131–138.CrossRefGoogle Scholar
  4. Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and downs of peer review. Advances in Physiology Education, 31(2), 145–152.CrossRefGoogle Scholar
  5. Bornmann, L., Mutz, R., & Daniela, H. D. (2010). Reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331.CrossRefGoogle Scholar
  6. Bornmann, L., Mutza, R., & Daniela, H. D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3), 226–238.CrossRefGoogle Scholar
  7. Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., & Lortie, C. J. (2009). To name or not to name: the effect of changing author gender on peer review. BioScience, 59(11), 985–989.CrossRefGoogle Scholar
  8. Budden, A. E., Tregenza, T., Aarssen, L. W., Koricheva, J., Leimu, R., & Lortie, C. J. (2008). Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution, 23(1), 4–6.CrossRefGoogle Scholar
  9. Butler, D. (2010). Journals step up plagiarism policing. Nature, 466(7303), 167.CrossRefGoogle Scholar
  10. Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med, 4(1), e40.CrossRefGoogle Scholar
  11. Callaham, M. L., Wears, R. L., & Waeckerle, J. F. (1998). Effect of attendance at a training session on peer reviewer quality and performance. Annals of Emergency Medicine, 32(3 Pt 1), 318–322.CrossRefGoogle Scholar
  12. Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Sciences of the United States of America, 108(8), 3157–3162.CrossRefGoogle Scholar
  13. Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., Waeckerle, J. F., Callaham, M. L., & Rennie, D. (1998). Masking author identity in peer review: What factors influence masking success? PEER Investigators. Journal of the American Medical Association, 280(3), 243–245.CrossRefGoogle Scholar
  14. Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and U.S. Science Policy. Albany, NY: State University of New York Press.Google Scholar
  15. Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–135.CrossRefGoogle Scholar
  16. Committee on Publication Ethics. (2014). Code of Conduct. http://publicationethics.org/resources/code-conduct. Accessed 31 Oct 2014.
  17. Cromey, D. W. (2013). Digital images are data: And should be treated as such. Methods in Molecular Biology, 931, 1–27.CrossRefGoogle Scholar
  18. Dickersin, K., Min, Y. I., & Meinert, C. L. (1992). Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. Journal of the American Medical Association, 267(3), 374–378.CrossRefGoogle Scholar
  19. Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & Reporting Bias Group. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PLoS ONE, 8(7), e66844.CrossRefGoogle Scholar
  20. Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research. Lancet, 337(8746), 867–872.CrossRefGoogle Scholar
  21. Editors, Nature. (2001). Editorial: Bad peer reviewers. Nature, 413(6852), 93.CrossRefGoogle Scholar
  22. Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352(6336), 560.CrossRefGoogle Scholar
  23. Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738.CrossRefGoogle Scholar
  24. Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association, 272(2), 143–146.CrossRefGoogle Scholar
  25. Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige on reviewers’ recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137–138.CrossRefGoogle Scholar
  26. Gasparyan, A. Y., Ayvazyan, L., Akazhanov, N. A., & Kitas, G. D. (2013). Conflicts of interest in biomedical publications: Considerations for authors, peer reviewers, and editors. Croatian Medical Journal, 54(6), 600–608.CrossRefGoogle Scholar
  27. Gilbert, J. R., Williams, E. S., & Lundberg, G. D. (1994). Is there gender bias in JAMA’s peer-review process? Journal of the American Medical Association, 272(2), 139–142.CrossRefGoogle Scholar
  28. Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. Journal of the American Medical Association, 280(3), 237–240.CrossRefGoogle Scholar
  29. Grod, O. N., Budden, A. E., Tregenza, T., Koricheva, J., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2008). Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PLoS ONE, 3(9), e3202.CrossRefGoogle Scholar
  30. Ho, R. C., Mak, K. K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013). Views on the peer review system of biomedical journals: An online survey of academics from high-ranking universities. BMC Medical Research Methodology, 13, 74.CrossRefGoogle Scholar
  31. Holy Bible, King James Version. (1991). New York: Random House.Google Scholar
  32. Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Medical Education, 12, 83.CrossRefGoogle Scholar
  33. Hull, D. (1988). Science as a process. Chicago: University of Chicago Press.CrossRefGoogle Scholar
  34. Hwang, W. S., Roh, S. I., Lee, B. C., Kang, S. K., Kwon, D. K., Kim, S., et al. (2005). Patient-specific embryonic stem cells derived from human SCNT blastocysts. Science, 308(5729), 1777–1783.CrossRefGoogle Scholar
  35. International Committee of Medical Journal Editors. (2014). Recommendations. http://www.icmje.org. Accessed 31 Oct 2014.
  36. Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. Journal of the American Medical Association, 280(3), 240–242.CrossRefGoogle Scholar
  37. Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010). Editorial peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors care? PLoS ONE, 5(4), e10072.CrossRefGoogle Scholar
  38. LaFollette, M. (1992). Stealing Into Print: Fraud, plagiarism, and misconduct in scientific publishing. Berkeley, CA: University of California Press.Google Scholar
  39. Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.CrossRefGoogle Scholar
  40. Lee, C. J. (2013). A Kuhnian critique of psychometric research on peer review. Philosophy of Science, 79(5), 859–870.CrossRefGoogle Scholar
  41. Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.CrossRefGoogle Scholar
  42. Lloyd, M. E. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis, 23(4), 539–543.CrossRefGoogle Scholar
  43. Lovejoy, T. I., Revenson, T. A., & France, C. R. (2011). Reviewing manuscripts for peer-review journals: A primer for novice and seasoned reviewers. Annals of Behavioral Medicine, 42(1), 1–13.CrossRefGoogle Scholar
  44. Mahoney, M. J. (1977). Publication preferences: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.CrossRefGoogle Scholar
  45. McCullough, J. (1989). First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Science, Technology and Human Values, 14(1), 78–88.CrossRefGoogle Scholar
  46. McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10), 1371–1376.CrossRefGoogle Scholar
  47. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.Google Scholar
  48. Neff, B. D., & Olden, J. D. (2006). Is peer review a game of chance? BioScience, 56(4), 333–340.CrossRefGoogle Scholar
  49. Newton, D. P. (2010). Quality and peer review of research: An adjudicating role for editors. Accountability in Research, 17(3), 130–145.CrossRefGoogle Scholar
  50. Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. Journal of the American Medical Association, 272(2), 149–151.CrossRefGoogle Scholar
  51. Obokata, H., Wakayama, T., Sasai, Y., Kojima, K., Vacanti, M. P., Niwa, H., et al. (2014). Stimulus-triggered fate conversion of somatic cells into pluripotency. Nature, 505(7485), 641–647.CrossRefGoogle Scholar
  52. Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., et al. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287(21), 2825–2828.CrossRefGoogle Scholar
  53. Park, I. U., Peacey, M. W., & Munafò, M. R. (2014). Modelling the effects of subjective and objective decision making in scientific peer review. Nature, 506(7486), 93–96.CrossRefGoogle Scholar
  54. Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195.CrossRefGoogle Scholar
  55. Rennie, D. (2003). Misconduct and journal peer review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 118–129). London: BMJ Books.Google Scholar
  56. Resch, K. I., Ernst, E., & Garrow, J. (2000). A randomized controlled study of reviewer bias against an unconventional therapy. Journal of the Royal Society of Medicine, 93(4), 164–167.Google Scholar
  57. Resnik, D. B. (2011). A troubled tradition. American Scientist, 99(1), 24–28.Google Scholar
  58. Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310.Google Scholar
  59. Rivara, F. P., Cummings, P., Ringold, S., Bergman, A. B., Joffe, A., & Christakis, D. A. (2007). A comparison of reviewers selected by editors and reviewers suggested by authors. Journal of Pediatrics, 151(2), 202–205.CrossRefGoogle Scholar
  60. Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., et al. (2006). Effect of blinded peer review on abstract acceptance. Journal of the American Medical Association, 295(14), 1675–1680.CrossRefGoogle Scholar
  61. Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain, 123(Pt 9), 1964–1969.CrossRefGoogle Scholar
  62. Rowe, B. H., Strome, T. L., Spooner, C., Blitz, S., Grafstein, E., & Worster, A. (2006). Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Medical Research Methodology, 6, 14.CrossRefGoogle Scholar
  63. Salasche, S. J. (1997). How to “peer review” a medical journal manuscript. Dermatological Surgery, 23(6), 423–428.CrossRefGoogle Scholar
  64. Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004). Effects of training on quality of peer review: Randomised controlled trial. British Medical Journal, 328(7441), 673.CrossRefGoogle Scholar
  65. Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine, 101(10), 507–514.CrossRefGoogle Scholar
  66. Schroter, S., Tite, L., Hutchings, A., & Black, N. (2006). Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. Journal of the American Medical Association, 295(3), 314–317.CrossRefGoogle Scholar
  67. Shamoo, A. E., & Resnik, D. B. (2015). Responsible conduct of research (3rd ed.). New York: Oxford University Press.Google Scholar
  68. Sismondo, S. (2008). Pharmaceutical company funding and its consequences: A qualitative systematic review. Contempory Clinical Trials, 29(2), 109–113.CrossRefGoogle Scholar
  69. Smith, R. (1999). Opening up BMJ peer review: A beginning that should lead to complete transparency. British Medical Journal, 318(7175), 4–5.CrossRefGoogle Scholar
  70. Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182.CrossRefGoogle Scholar
  71. Stern, J. M., & Simes, R. J. (1997). Publication bias: evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315(7109), 640–645.CrossRefGoogle Scholar
  72. Strayhorn, J., McDermott, J. F., & Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Acacemy of Child and Adolescent Psychiatry. Journal of the American Acacemy of Child and Adolescent Psychiatry, 150(6), 947–952.Google Scholar
  73. Timmer, A., Hilsden, R. J., & Sutherland, L. R. (2001). Determinants of abstract acceptance for the Digestive Diseases Week–A cross sectional study. BMC Medical Research Methodology, 1, 13.CrossRefGoogle Scholar
  74. Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8), 349–350.CrossRefGoogle Scholar
  75. van Rooyen, S., Delamothe, T., & Evans, S. J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: Randomised controlled trial. British Medical Journal, 341, c5729.CrossRefGoogle Scholar
  76. van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal, 318(7175), 23–27.CrossRefGoogle Scholar
  77. van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review: A randomized trial. Journal of the American Medical Association, 280(3), 234–237.CrossRefGoogle Scholar
  78. Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., et al. (1998). Ileal–lymphoid–nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet, 351(9103), 637–641.CrossRefGoogle Scholar
  79. Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled trial. British Journal of Psychiatry, 176, 47–51.CrossRefGoogle Scholar
  80. Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341–343.CrossRefGoogle Scholar
  81. Whittaker, R. J., et al. (2008). Journal review and gender equality: A critical comment on Budden et al. Trends in Ecology & Evolution, 23, 478–479.CrossRefGoogle Scholar
  82. Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting the daylight in reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6, 20.CrossRefGoogle Scholar
  83. Williams, P., & Wager, E. (2013). Exploring why and how journal editors retract articles: Findings from a qualitative study. Science and Engineering Ethics, 19(1), 1–11.CrossRefGoogle Scholar
  84. Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht (outside the USA) 2015

Authors and Affiliations

  1. 1.National Institute of Environmental Health SciencesNational Institutes of HealthResearch Triangle ParkUSA
  2. 2.National Toxicology Program, National Institute of Environmental Health SciencesNational Institutes of HealthResearch Triangle ParkUSA

Personalised recommendations