Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process

Abstract

High publication demands and the low acceptance rate of peer review journals place the journal editors and their reviewers in a powerful position. Journal reviewers have a vital role not only in influencing the journal editor's publication decisions, but also in the very nature and direction of scientific research. Because of their influence in peer review outcomes, journal reviewers are aptly described as the “gate keepers of science.” In this article we describe several pitfalls that can impede reviewers' impartial judgement. These include such issues as confirmatory bias, the negative results bias (the file drawer problem), the Matthew effect, the Doctor Fox effect, and gender, race, theoretical orientation, and “political correctness.” We argue that procedures currently used by many professional journals, such as blind or masked review, may not completely alleviate the effects of these pitfalls. Instead, we suggest that increasing reviewers' awareness of the pitfalls, accountability, and vigilance can improve fairness in the peer review process. The ultimate responsibilities belong to the journal editors who are confronted with the difficult task of satisfying journal readers, contributors, reviewers, and owners. We recommend that the journal editors conduct periodic internal and external evaluations of their journals' peer review process and outcomes, with participation of reviewers, contributors, readers and owners.

This is a preview of subscription content, log in to check access.

References

  1. Albert, T. (1997). Why bother with peer review? Lancet 350: 822.

    Article  Google Scholar 

  2. Altman, D.G., Chalmers, I. & Herxheimer. A. (1994). Is there a case for an international medical scientific press council? Journal of the American Medical Association 272: 166–167.

    Article  PubMed  Google Scholar 

  3. Angell, M. (1999). The journal and its owner: Resolving the crisis. New England Journal of Medicine 341: 752.

    Article  PubMed  Google Scholar 

  4. APA (2001). Summary report of journal operations, 2000. American Psychologist 56: 693–694.

    Google Scholar 

  5. Armstrong, S.J. (October 25, 1996). We need to rethink the editorial role of peer reviewers. The Chronicle of Higher Education 43(9): B3.

    Google Scholar 

  6. Bailar, J.C. & Patterson, K. (1985). Journal peer review: The need for a research agenda. New England Journal of Medicine 312: 654–657.

    PubMed  Google Scholar 

  7. Bartko, J.J. (1966). The intra-class correlation coefficient as a measure of reliability. Psychological Reports 19: 3–11.

    PubMed  Google Scholar 

  8. Begley, S. (1992, September 14). Is science censored? Newsweek, p. 63.

  9. Belsky, J. (1986). Infant day care: A cause for concern? Zero to Three 10: 22–24.

    Google Scholar 

  10. Belsky, J. (1987, April). Science, social policy and day care: A personal odyssey. Paper presented at the biennial meeting of the Society for Research in Child Development, New York.

    Google Scholar 

  11. Black, N., Van Rooyen, S. et al. (1998).What makes a good reviewer and a good review for a general medical journal. Journal of the American Medical Association 280: 231–233.

    Article  PubMed  Google Scholar 

  12. Bloom, F.E. (1999). The importance of reviewers. Science 283: 789.

    Article  PubMed  Google Scholar 

  13. Caelleigh, A.S., Hojat, M. et al. (September 2001). Effects of reviewers' gender on assessments of a gender-related standardized manuscript. Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.

    Google Scholar 

  14. Callaham, M.L., Baxt, W.G. et al. (1998). Reliability of editors' subjective quality ratings of peer reviews of manuscripts. Journal of the American medical Association 280: 229–231.

    Article  PubMed  Google Scholar 

  15. Callaham, M.L., Wears, R.L. et al. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Journal of the American Medical Association 280: 254–257.

    Article  PubMed  Google Scholar 

  16. Campion, E.W., Curfman, G.D. & Drazen, J.M. (2000). Tracking the peer-review process. The New England Journal of Medicine 343: 1485–1486.

    Article  PubMed  Google Scholar 

  17. Cho, M.K., Justice, A.C. et al. (1998). Masking author identity in peer review. Journal of the American Medical Association 280: 243–245.

    Article  PubMed  Google Scholar 

  18. Cicchetti, D.V. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35: 300–305.

    Article  Google Scholar 

  19. Colman, A.M. (1979). Editorial role in author-referee disagreements. Bulletin of the British Psychological Society 32: 390–391.

    Google Scholar 

  20. Crandall, R. (1990). Author, reviewer and editorial values: Politics in the working mother research? Journal of Social Behavior and Personality 5: 489–491.

    Google Scholar 

  21. Crandall, R. (1982). Editorial responsibilities in manuscript review. The Behavioral and Brain Sciences 5: 207–208.

    Google Scholar 

  22. Crandall, R. (1978). Interrater agreement on manuscripts is not so bad! American Psychologist 33: 623–624.

    Article  Google Scholar 

  23. Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. The American Sociologist 32: 195–201.

    Google Scholar 

  24. DeGrazia, A. (1963). The scientific reception system and Dr. Velikovsky. American Behavioral Scientist 7: 38–56.

    Google Scholar 

  25. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263: 1385–1389.

    Article  PubMed  Google Scholar 

  26. Dividoff, F., DeAngelis, C.D. et al. (2001). Sponsorship, authorship, and accountability. Journal of American Medical Association 286: 1232–1233.

    Article  Google Scholar 

  27. Drucker, P.F. (1994). Political correctness and American academe. Society 32: 58–63.

    Google Scholar 

  28. Eagley, A.H. (1995a). The science and politics of comparing women and men. American Psychologist 50: 145–158.

    Article  Google Scholar 

  29. Eagley, A.H. (1995b). Reflections on the commentators' views. American Psychologist 50: 169–171.

    Article  Google Scholar 

  30. Fisher, M., Friedman, S.B. & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association 272: 143–146.

    Article  PubMed  Google Scholar 

  31. Fontanarosa, P.B., Glass, R.M. & De Angelis (2000). Thanking authors, peer reviewers, and readers – constancy in a time of change. Journal of the American Medical Association 283: 2016–2017.

    Article  Google Scholar 

  32. Gannon, L., Luchetta, T. et al. (1992). Sex bias in psychological research: Progress or complacency? American Psychologist 47: 389–396.

    Article  PubMed  Google Scholar 

  33. Garfunkel, J.M., Ulshen, M.H. et al. (1994). Effect of institutional prestige on reviewers' recommendations and editorial decisions. Journal of the American Medical Association 272: 137–138.

    Article  PubMed  Google Scholar 

  34. Garrison, G.E. & Kobor, C.P. (2002). Weathering a political storm: A contextual perspective on a psychological research controversy. American Psychologist 57: 165–175.

    PubMed  Google Scholar 

  35. Gilbert, J.R., Williams, E.S. & Lundberg, G.D. (1994). Is there gender bias in JAMA's peer review process? Journal of the American Medical Association 272: 139–142.

    Article  PubMed  Google Scholar 

  36. Godlee, F., Gale, C. Martyn, C.N. (1998). Effect on the quality of peer review of blinding reviewers and asking themto sign their names. Journal of the AmericanMedical Association 280: 237–240.

    Article  Google Scholar 

  37. Goldbeck-Wood, S. (1999). Evidence on peer review – Scientific quality control or smoke screen? British Medical Journal 318: 44–45.

    PubMed  Google Scholar 

  38. Goldberg, P. (1968). Are some women prejudiced against women? Trans-Action 5: 28–30.

    Google Scholar 

  39. Gottfredson, S.D. (1978). Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33: 920–934.

    Article  Google Scholar 

  40. Greenwald, A.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin 82: 1–20.

    Google Scholar 

  41. Hall, J.A. (1979). Author review of reviewers. American Psychologist 34: 798.

    Google Scholar 

  42. Hare-Mustin, R. & Maracek, J. (1988). The meaning of difference: Gender theory, postmodernism, and psychology. American Psychologist 43: 355–364.

    Article  Google Scholar 

  43. Hare-Mustin, R. & Maracek J. (1990). Making a difference: Psychology and the Construction of Gender. New Haven, CT: Yale University Press.

    Google Scholar 

  44. Hendrick, C. (1976). Editorial comment. Personality and Social Psychology Bulletin 2: 207–208.

    Google Scholar 

  45. Herrnstein, R.J. & Murray, C. (1994). The Bell Curve: Intelligence and Class Structure in American Life. New York: The Free Press.

    Google Scholar 

  46. Hoffman, L.W. (1990). Bias and social responsibility in the study of maternal employment. In C.B. Fisher & W.W. Tryon (eds.), Ethics in Applied Developmental Psychology: Emergency Issues in an Emergency Field (Annual Advances in Applied Developmental Psychology, Vol. 4) (pp. 253–271). Norwood, NJ: Ablex.

    Google Scholar 

  47. Hojat, M. (1993a). A mother's love: What children will not receive in day-care centers. (Interview). The Family in America 8: 1–7.

    Google Scholar 

  48. Hojat, M. (1993b). Abandoning research on consequences of nonmaternal care: A disservice to the science. Journal of Social Behavior and Personality 8: 5–8.

    Google Scholar 

  49. Hojat, M. (1993c). The world declaration of the rights of the child: Anticipated challenges. Psychological Reports 72: 1011–1022.

    PubMed  Google Scholar 

  50. Hojat, M. (1990). Can affectional ties be purchased? Comments on working mothers and their families. Journal of Social Behavior and Personality 5: 493–502.

    Google Scholar 

  51. Hojat, M. (1995). Developmental pathways to violence: A psychodynamic paradigm. Peace Psychology Review 1: 176–195.

    Google Scholar 

  52. Hojat, M. An empirical study of possible biases due to reviewers' theoretical perspective, gender, and political correctness of the manuscript in journal peer review process, unpublished manuscript.

  53. Hoover, J.D. & Howard, L.A. (1995). The political correctness controversy revisited: Retreat from argumentation and reaffirmation of critical dialogue. American Behavioral Scientist 38: 963–975.

    Google Scholar 

  54. Horrobin, D.F. (1982). Peer review: A philosophically faulty concept which is proving disastrous for science. The Behavioral and Brain Sciences 5: 217–218.

    Google Scholar 

  55. Horton, R. (1998). The journal ombudsperson: A step toward scientific press oversight. Journal of the American Medical Association 280: 298–299.

    Article  PubMed  Google Scholar 

  56. Horton, R. (1996). The Lancet's ombudsman. Lancet 348: 6.

    Article  Google Scholar 

  57. Howard, L. & Wilkinson, G. (1998). Peer review and editorial decision-making. British Journal of Psychiatry 173: 110–113.

    PubMed  Google Scholar 

  58. Hunt, E. (1971). Psychological publications. American Psychologist 26: 311.

    Google Scholar 

  59. Jefferson, T. & Godlee, F. (eds.) (1999). Peer Review in Health Care. London, UK: British Journal Publishing Group.

    Google Scholar 

  60. Justice, A.C., Cho, M.K. et al. (1998). Does masking author identity improve peer review quality? A randomized controlled trail. Journal of the American Medical Association 280: 240–242.

    Article  PubMed  Google Scholar 

  61. Karen, R. (1994). Becoming Attached. New York: Warner Books.

    Google Scholar 

  62. Kassirer, J.P. & Campion, E.W. (1994). Peer review: Crude and understudied, but indispensable. Journal of the American Medical Association 272: 96–97.

    Article  PubMed  Google Scholar 

  63. Kennedy, D. (2002). Publish or not publish. Science 295: 1793.

    Article  PubMed  Google Scholar 

  64. King, D.W., McDonald, D.D. & Rodereer, M.K. (1981). Scientific Journals in the United States: Their Production, Use, and Economics. Stroudsburg, PA: Hutchinson & Ross.

    Google Scholar 

  65. Laband, D.N. & Piette,M.J. (1994). A citation analysis of the impact of blinded peer review. Journal of the American 272: 147–149.

    Google Scholar 

  66. Lilienfeld, S.O. (2002).When worlds collide: Social science, politics, and the Rind et al. (1998) child sexual abuse meta-analysis. American Psychologist 57: 176–188.

    Google Scholar 

  67. Lock, S. (1985). A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia: ISI Press.

    Google Scholar 

  68. Lock, S. (ed.) (1991). The Future of Medical Journals. Plymouth, UK: British Medical Journal.

    Google Scholar 

  69. Loury, G.C. (1994). Self-censorship in public discourse: A theory of political correctness and related phenomena. Rationality and Society 6: 428–461.

    Google Scholar 

  70. Mahoney, M. (1977). Publication prejudices: An experimental study of confirmatory bias in peer review system. Cognitive Therapy and Research 1: 161–175.

    Article  Google Scholar 

  71. Mahoney, M.J. (1987). Scientific publication and knowledge politics. Journal of Social Behavior and Personality 2(Part 1): 165–176.

    Google Scholar 

  72. McNutt, R.A., Evans, A.T. et al. (1990). The effects of blinding on the quality of peer review. Journal of the American Medical Association 263: 1371–1376.

    Article  PubMed  Google Scholar 

  73. Merton, R.K. (1968). The Matthew effect in science. Science 159: 56–63.

    PubMed  Google Scholar 

  74. Misakian, A.L. & Bero, L.A. (1998). Publication bias and research on passive smoking: Comparison of published and unpublished studies. Journal of the American Medical Association 280: 250–253.

    Article  PubMed  Google Scholar 

  75. Moore, M. (1978). Discrimination or favoritism? Sex bias in book reviews. American Psychologist 33: 936–938.

    Article  Google Scholar 

  76. Naftulin, D.H., Ware, Jr., J.E. & Donnelly, F.A. (1973). The doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education 48: 630–635.

    PubMed  Google Scholar 

  77. Nylenna, M., Riis, P. & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscript. Journal of the American Medical Association 272: 149–151.

    Article  PubMed  Google Scholar 

  78. Olson, C.M. (1990). Peer review of biomedical literature. American Journal of Emergency Medicine 8: 356–358.

    Article  PubMed  Google Scholar 

  79. Over, R. (1982). What is the source of bias in peer review? The Behavioral and Brain Sciences 5: 229–230.

    Google Scholar 

  80. Oxman A.D., Guyatt, G.H. & Singer, J. (1991). Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44: 91–98.

    Article  PubMed  Google Scholar 

  81. Peters, D.P. & Ceci, S.J. (1982). Peer review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5: 187–195.

    Google Scholar 

  82. Pierie, J.P., Walvoort, H.C. & Overbeke, A.J.P. (1996). Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 348: 1480–1483.

    Article  PubMed  Google Scholar 

  83. Popenoe, D. (1993a). American family decline, 1960-1990: A review and appraisal. Journal of Marriage and the Family 55: 527–532.

    Google Scholar 

  84. Popenoe, D. (1993b). The national family wars. Journal of Marriage and the Family 55: 533–553.

    Google Scholar 

  85. Rennie, D. (1998). Peer review in Prague. Journal of the American Medical Association 280: 214–215.

    Article  PubMed  Google Scholar 

  86. Riger, S. (1992). Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist 47: 730–740.

    Article  Google Scholar 

  87. Riis, P. (1992). New paradigm in journalogy. Journal of Internal Medicine 232: 207–213.

    PubMed  Google Scholar 

  88. Rind, B., Tromovich, P. & Bauserman, R. (1998). A meta-analytic examination of assumed properties of child sexual abuse using college samples. Psychological Bulletin 124: 22–53.

    Article  PubMed  Google Scholar 

  89. Rosenberg, R.N. & Anderson, E.R., Jr. (1999). Editorial governance of the Journal of the American Medical Association: A report. Journal of the American Medical Association 281: 2239–2242.

    Article  PubMed  Google Scholar 

  90. Rosenthal, R. (1979). The “file drawer problem” and tolerance of null results. Psychological Bulletin 86: 541–638.

    Google Scholar 

  91. Rubin, P.H. (1994). The assault on the first amendment: Public choice and political correctness. Cato Journal 14: 23–36.

    Google Scholar 

  92. Scarr, S. & Weber, B.L.R. (1978). The reliability of reviewers for the American Psychologist. American Psychologist 33: 935.

    Google Scholar 

  93. Scott, W.A. (1974). Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 31: 799–804.

    Google Scholar 

  94. Sharp, D.W. (1990). What can and should be done to reduce publication bias? Journal of the American Medical Association 263: 1390–1391.

    Article  PubMed  Google Scholar 

  95. Shea, J.A., Caelleigh, A.S. et al. (2001). Review process and publication decision. Academic Medicine 76: 911–916.

    Google Scholar 

  96. Silverstein, L.B. (1991). Transforming the debate about child care and maternal employment. American Psychologist 46: 1025–1032.

    Article  PubMed  Google Scholar 

  97. Smith, R. (1999). Opening of BMJ peer review. British Medical Journal 318: 4–5.

    PubMed  Google Scholar 

  98. Stark-Adamec, C. (1993). Social science and scientific responsibility. Canadian Journal of Physics 71: 192–196.

    Google Scholar 

  99. Stephenson, J. (1997). Medical journals turn gaze inward to examine process of peer review. Journal of the American Medical Association 278: 1389–1391.

    Article  PubMed  Google Scholar 

  100. Sterling, T.D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vise versa. Journal American Statistical Association 54: 30–34.

    Google Scholar 

  101. Susser, M. & Northridge, M.E. (1996). Editor's note: Reviewing for the journal. American Journal of Public Health 86: 161.

    Google Scholar 

  102. Sweitzer, B.J. & Cullen, D.J. (1994). How well does a journal's review process function? A survey of authors' opinion. Journal of the American Medical Association 272: 152–153.

    Article  PubMed  Google Scholar 

  103. Van Rooyen S., Godlee, H. et al. (1998). Effect of open peer review on quality of quality of reviews and reviewers recommendation: A randomised trail. British Medical Journal 317: 23–27.

    Google Scholar 

  104. Van Rooyen, S., Godlee, H. et al. (1998). Effect of blinding and unmaksing on the quality of peer review: A clinical trail. Journal of American Medical Association 280: 234–237.

    Article  Google Scholar 

  105. Ware, J.E. & Williams, R.G. (1975). The Dr. Fox effect: A study of lecturer effectiveness and ratings of instruction. Journal of Medical Education 50: 149–156.

    PubMed  Google Scholar 

  106. Watkins, M.W. (1979). Chance and interrater agreement on manuscript. American Psychologist 34: 796–798.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mohammadreza Hojat.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hojat, M., Gonnella, J.S. & Caelleigh, A.S. Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process. Adv Health Sci Educ Theory Pract 8, 75–96 (2003). https://doi.org/10.1023/A:1022670432373

Download citation

  • gatekeepers of science
  • Journal reviewer guidelines
  • Journal editor's responsibilities
  • peer review
  • publication bias