Advances in Health Sciences Education

, Volume 8, Issue 1, pp 75–96 | Cite as

Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process

  • Mohammadreza HojatEmail author
  • Joseph S. Gonnella
  • Addeane S. Caelleigh


High publication demands and the low acceptance rate of peer review journals place the journal editors and their reviewers in a powerful position. Journal reviewers have a vital role not only in influencing the journal editor's publication decisions, but also in the very nature and direction of scientific research. Because of their influence in peer review outcomes, journal reviewers are aptly described as the “gate keepers of science.” In this article we describe several pitfalls that can impede reviewers' impartial judgement. These include such issues as confirmatory bias, the negative results bias (the file drawer problem), the Matthew effect, the Doctor Fox effect, and gender, race, theoretical orientation, and “political correctness.” We argue that procedures currently used by many professional journals, such as blind or masked review, may not completely alleviate the effects of these pitfalls. Instead, we suggest that increasing reviewers' awareness of the pitfalls, accountability, and vigilance can improve fairness in the peer review process. The ultimate responsibilities belong to the journal editors who are confronted with the difficult task of satisfying journal readers, contributors, reviewers, and owners. We recommend that the journal editors conduct periodic internal and external evaluations of their journals' peer review process and outcomes, with participation of reviewers, contributors, readers and owners.

gatekeepers of science Journal reviewer guidelines Journal editor's responsibilities peer review publication bias 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Albert, T. (1997). Why bother with peer review? Lancet 350: 822.CrossRefGoogle Scholar
  2. Altman, D.G., Chalmers, I. & Herxheimer. A. (1994). Is there a case for an international medical scientific press council? Journal of the American Medical Association 272: 166–167.CrossRefPubMedGoogle Scholar
  3. Angell, M. (1999). The journal and its owner: Resolving the crisis. New England Journal of Medicine 341: 752.CrossRefPubMedGoogle Scholar
  4. APA (2001). Summary report of journal operations, 2000. American Psychologist 56: 693–694.Google Scholar
  5. Armstrong, S.J. (October 25, 1996). We need to rethink the editorial role of peer reviewers. The Chronicle of Higher Education 43(9): B3.Google Scholar
  6. Bailar, J.C. & Patterson, K. (1985). Journal peer review: The need for a research agenda. New England Journal of Medicine 312: 654–657.PubMedGoogle Scholar
  7. Bartko, J.J. (1966). The intra-class correlation coefficient as a measure of reliability. Psychological Reports 19: 3–11.PubMedGoogle Scholar
  8. Begley, S. (1992, September 14). Is science censored? Newsweek, p. 63.Google Scholar
  9. Belsky, J. (1986). Infant day care: A cause for concern? Zero to Three 10: 22–24.Google Scholar
  10. Belsky, J. (1987, April). Science, social policy and day care: A personal odyssey. Paper presented at the biennial meeting of the Society for Research in Child Development, New York.Google Scholar
  11. Black, N., Van Rooyen, S. et al. (1998).What makes a good reviewer and a good review for a general medical journal. Journal of the American Medical Association 280: 231–233.CrossRefPubMedGoogle Scholar
  12. Bloom, F.E. (1999). The importance of reviewers. Science 283: 789.CrossRefPubMedGoogle Scholar
  13. Caelleigh, A.S., Hojat, M. et al. (September 2001). Effects of reviewers' gender on assessments of a gender-related standardized manuscript. Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.Google Scholar
  14. Callaham, M.L., Baxt, W.G. et al. (1998). Reliability of editors' subjective quality ratings of peer reviews of manuscripts. Journal of the American medical Association 280: 229–231.CrossRefPubMedGoogle Scholar
  15. Callaham, M.L., Wears, R.L. et al. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Journal of the American Medical Association 280: 254–257.CrossRefPubMedGoogle Scholar
  16. Campion, E.W., Curfman, G.D. & Drazen, J.M. (2000). Tracking the peer-review process. The New England Journal of Medicine 343: 1485–1486.CrossRefPubMedGoogle Scholar
  17. Cho, M.K., Justice, A.C. et al. (1998). Masking author identity in peer review. Journal of the American Medical Association 280: 243–245.CrossRefPubMedGoogle Scholar
  18. Cicchetti, D.V. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35: 300–305.CrossRefGoogle Scholar
  19. Colman, A.M. (1979). Editorial role in author-referee disagreements. Bulletin of the British Psychological Society 32: 390–391.Google Scholar
  20. Crandall, R. (1990). Author, reviewer and editorial values: Politics in the working mother research? Journal of Social Behavior and Personality 5: 489–491.Google Scholar
  21. Crandall, R. (1982). Editorial responsibilities in manuscript review. The Behavioral and Brain Sciences 5: 207–208.Google Scholar
  22. Crandall, R. (1978). Interrater agreement on manuscripts is not so bad! American Psychologist 33: 623–624.CrossRefGoogle Scholar
  23. Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. The American Sociologist 32: 195–201.Google Scholar
  24. DeGrazia, A. (1963). The scientific reception system and Dr. Velikovsky. American Behavioral Scientist 7: 38–56.Google Scholar
  25. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263: 1385–1389.CrossRefPubMedGoogle Scholar
  26. Dividoff, F., DeAngelis, C.D. et al. (2001). Sponsorship, authorship, and accountability. Journal of American Medical Association 286: 1232–1233.CrossRefGoogle Scholar
  27. Drucker, P.F. (1994). Political correctness and American academe. Society 32: 58–63.Google Scholar
  28. Eagley, A.H. (1995a). The science and politics of comparing women and men. American Psychologist 50: 145–158.CrossRefGoogle Scholar
  29. Eagley, A.H. (1995b). Reflections on the commentators' views. American Psychologist 50: 169–171.CrossRefGoogle Scholar
  30. Fisher, M., Friedman, S.B. & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association 272: 143–146.CrossRefPubMedGoogle Scholar
  31. Fontanarosa, P.B., Glass, R.M. & De Angelis (2000). Thanking authors, peer reviewers, and readers – constancy in a time of change. Journal of the American Medical Association 283: 2016–2017.CrossRefGoogle Scholar
  32. Gannon, L., Luchetta, T. et al. (1992). Sex bias in psychological research: Progress or complacency? American Psychologist 47: 389–396.CrossRefPubMedGoogle Scholar
  33. Garfunkel, J.M., Ulshen, M.H. et al. (1994). Effect of institutional prestige on reviewers' recommendations and editorial decisions. Journal of the American Medical Association 272: 137–138.CrossRefPubMedGoogle Scholar
  34. Garrison, G.E. & Kobor, C.P. (2002). Weathering a political storm: A contextual perspective on a psychological research controversy. American Psychologist 57: 165–175.PubMedGoogle Scholar
  35. Gilbert, J.R., Williams, E.S. & Lundberg, G.D. (1994). Is there gender bias in JAMA's peer review process? Journal of the American Medical Association 272: 139–142.CrossRefPubMedGoogle Scholar
  36. Godlee, F., Gale, C. Martyn, C.N. (1998). Effect on the quality of peer review of blinding reviewers and asking themto sign their names. Journal of the AmericanMedical Association 280: 237–240.CrossRefGoogle Scholar
  37. Goldbeck-Wood, S. (1999). Evidence on peer review – Scientific quality control or smoke screen? British Medical Journal 318: 44–45.PubMedGoogle Scholar
  38. Goldberg, P. (1968). Are some women prejudiced against women? Trans-Action 5: 28–30.Google Scholar
  39. Gottfredson, S.D. (1978). Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33: 920–934.CrossRefGoogle Scholar
  40. Greenwald, A.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin 82: 1–20.Google Scholar
  41. Hall, J.A. (1979). Author review of reviewers. American Psychologist 34: 798.Google Scholar
  42. Hare-Mustin, R. & Maracek, J. (1988). The meaning of difference: Gender theory, postmodernism, and psychology. American Psychologist 43: 355–364.CrossRefGoogle Scholar
  43. Hare-Mustin, R. & Maracek J. (1990). Making a difference: Psychology and the Construction of Gender. New Haven, CT: Yale University Press.Google Scholar
  44. Hendrick, C. (1976). Editorial comment. Personality and Social Psychology Bulletin 2: 207–208.Google Scholar
  45. Herrnstein, R.J. & Murray, C. (1994). The Bell Curve: Intelligence and Class Structure in American Life. New York: The Free Press.Google Scholar
  46. Hoffman, L.W. (1990). Bias and social responsibility in the study of maternal employment. In C.B. Fisher & W.W. Tryon (eds.), Ethics in Applied Developmental Psychology: Emergency Issues in an Emergency Field (Annual Advances in Applied Developmental Psychology, Vol. 4) (pp. 253–271). Norwood, NJ: Ablex.Google Scholar
  47. Hojat, M. (1993a). A mother's love: What children will not receive in day-care centers. (Interview). The Family in America 8: 1–7.Google Scholar
  48. Hojat, M. (1993b). Abandoning research on consequences of nonmaternal care: A disservice to the science. Journal of Social Behavior and Personality 8: 5–8.Google Scholar
  49. Hojat, M. (1993c). The world declaration of the rights of the child: Anticipated challenges. Psychological Reports 72: 1011–1022.PubMedGoogle Scholar
  50. Hojat, M. (1990). Can affectional ties be purchased? Comments on working mothers and their families. Journal of Social Behavior and Personality 5: 493–502.Google Scholar
  51. Hojat, M. (1995). Developmental pathways to violence: A psychodynamic paradigm. Peace Psychology Review 1: 176–195.Google Scholar
  52. Hojat, M. An empirical study of possible biases due to reviewers' theoretical perspective, gender, and political correctness of the manuscript in journal peer review process, unpublished manuscript.Google Scholar
  53. Hoover, J.D. & Howard, L.A. (1995). The political correctness controversy revisited: Retreat from argumentation and reaffirmation of critical dialogue. American Behavioral Scientist 38: 963–975.Google Scholar
  54. Horrobin, D.F. (1982). Peer review: A philosophically faulty concept which is proving disastrous for science. The Behavioral and Brain Sciences 5: 217–218.Google Scholar
  55. Horton, R. (1998). The journal ombudsperson: A step toward scientific press oversight. Journal of the American Medical Association 280: 298–299.CrossRefPubMedGoogle Scholar
  56. Horton, R. (1996). The Lancet's ombudsman. Lancet 348: 6.CrossRefGoogle Scholar
  57. Howard, L. & Wilkinson, G. (1998). Peer review and editorial decision-making. British Journal of Psychiatry 173: 110–113.PubMedGoogle Scholar
  58. Hunt, E. (1971). Psychological publications. American Psychologist 26: 311.Google Scholar
  59. Jefferson, T. & Godlee, F. (eds.) (1999). Peer Review in Health Care. London, UK: British Journal Publishing Group.Google Scholar
  60. Justice, A.C., Cho, M.K. et al. (1998). Does masking author identity improve peer review quality? A randomized controlled trail. Journal of the American Medical Association 280: 240–242.CrossRefPubMedGoogle Scholar
  61. Karen, R. (1994). Becoming Attached. New York: Warner Books.Google Scholar
  62. Kassirer, J.P. & Campion, E.W. (1994). Peer review: Crude and understudied, but indispensable. Journal of the American Medical Association 272: 96–97.CrossRefPubMedGoogle Scholar
  63. Kennedy, D. (2002). Publish or not publish. Science 295: 1793.CrossRefPubMedGoogle Scholar
  64. King, D.W., McDonald, D.D. & Rodereer, M.K. (1981). Scientific Journals in the United States: Their Production, Use, and Economics. Stroudsburg, PA: Hutchinson & Ross.Google Scholar
  65. Laband, D.N. & Piette,M.J. (1994). A citation analysis of the impact of blinded peer review. Journal of the American 272: 147–149.Google Scholar
  66. Lilienfeld, S.O. (2002).When worlds collide: Social science, politics, and the Rind et al. (1998) child sexual abuse meta-analysis. American Psychologist 57: 176–188.Google Scholar
  67. Lock, S. (1985). A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia: ISI Press.Google Scholar
  68. Lock, S. (ed.) (1991). The Future of Medical Journals. Plymouth, UK: British Medical Journal.Google Scholar
  69. Loury, G.C. (1994). Self-censorship in public discourse: A theory of political correctness and related phenomena. Rationality and Society 6: 428–461.Google Scholar
  70. Mahoney, M. (1977). Publication prejudices: An experimental study of confirmatory bias in peer review system. Cognitive Therapy and Research 1: 161–175.CrossRefGoogle Scholar
  71. Mahoney, M.J. (1987). Scientific publication and knowledge politics. Journal of Social Behavior and Personality 2(Part 1): 165–176.Google Scholar
  72. McNutt, R.A., Evans, A.T. et al. (1990). The effects of blinding on the quality of peer review. Journal of the American Medical Association 263: 1371–1376.CrossRefPubMedGoogle Scholar
  73. Merton, R.K. (1968). The Matthew effect in science. Science 159: 56–63.PubMedGoogle Scholar
  74. Misakian, A.L. & Bero, L.A. (1998). Publication bias and research on passive smoking: Comparison of published and unpublished studies. Journal of the American Medical Association 280: 250–253.CrossRefPubMedGoogle Scholar
  75. Moore, M. (1978). Discrimination or favoritism? Sex bias in book reviews. American Psychologist 33: 936–938.CrossRefGoogle Scholar
  76. Naftulin, D.H., Ware, Jr., J.E. & Donnelly, F.A. (1973). The doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education 48: 630–635.PubMedGoogle Scholar
  77. Nylenna, M., Riis, P. & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscript. Journal of the American Medical Association 272: 149–151.CrossRefPubMedGoogle Scholar
  78. Olson, C.M. (1990). Peer review of biomedical literature. American Journal of Emergency Medicine 8: 356–358.CrossRefPubMedGoogle Scholar
  79. Over, R. (1982). What is the source of bias in peer review? The Behavioral and Brain Sciences 5: 229–230.Google Scholar
  80. Oxman A.D., Guyatt, G.H. & Singer, J. (1991). Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44: 91–98.CrossRefPubMedGoogle Scholar
  81. Peters, D.P. & Ceci, S.J. (1982). Peer review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5: 187–195.Google Scholar
  82. Pierie, J.P., Walvoort, H.C. & Overbeke, A.J.P. (1996). Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 348: 1480–1483.CrossRefPubMedGoogle Scholar
  83. Popenoe, D. (1993a). American family decline, 1960-1990: A review and appraisal. Journal of Marriage and the Family 55: 527–532.Google Scholar
  84. Popenoe, D. (1993b). The national family wars. Journal of Marriage and the Family 55: 533–553.Google Scholar
  85. Rennie, D. (1998). Peer review in Prague. Journal of the American Medical Association 280: 214–215.CrossRefPubMedGoogle Scholar
  86. Riger, S. (1992). Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist 47: 730–740.CrossRefGoogle Scholar
  87. Riis, P. (1992). New paradigm in journalogy. Journal of Internal Medicine 232: 207–213.PubMedGoogle Scholar
  88. Rind, B., Tromovich, P. & Bauserman, R. (1998). A meta-analytic examination of assumed properties of child sexual abuse using college samples. Psychological Bulletin 124: 22–53.CrossRefPubMedGoogle Scholar
  89. Rosenberg, R.N. & Anderson, E.R., Jr. (1999). Editorial governance of the Journal of the American Medical Association: A report. Journal of the American Medical Association 281: 2239–2242.CrossRefPubMedGoogle Scholar
  90. Rosenthal, R. (1979). The “file drawer problem” and tolerance of null results. Psychological Bulletin 86: 541–638.Google Scholar
  91. Rubin, P.H. (1994). The assault on the first amendment: Public choice and political correctness. Cato Journal 14: 23–36.Google Scholar
  92. Scarr, S. & Weber, B.L.R. (1978). The reliability of reviewers for the American Psychologist. American Psychologist 33: 935.Google Scholar
  93. Scott, W.A. (1974). Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 31: 799–804.Google Scholar
  94. Sharp, D.W. (1990). What can and should be done to reduce publication bias? Journal of the American Medical Association 263: 1390–1391.CrossRefPubMedGoogle Scholar
  95. Shea, J.A., Caelleigh, A.S. et al. (2001). Review process and publication decision. Academic Medicine 76: 911–916.Google Scholar
  96. Silverstein, L.B. (1991). Transforming the debate about child care and maternal employment. American Psychologist 46: 1025–1032.CrossRefPubMedGoogle Scholar
  97. Smith, R. (1999). Opening of BMJ peer review. British Medical Journal 318: 4–5.PubMedGoogle Scholar
  98. Stark-Adamec, C. (1993). Social science and scientific responsibility. Canadian Journal of Physics 71: 192–196.Google Scholar
  99. Stephenson, J. (1997). Medical journals turn gaze inward to examine process of peer review. Journal of the American Medical Association 278: 1389–1391.CrossRefPubMedGoogle Scholar
  100. Sterling, T.D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vise versa. Journal American Statistical Association 54: 30–34.Google Scholar
  101. Susser, M. & Northridge, M.E. (1996). Editor's note: Reviewing for the journal. American Journal of Public Health 86: 161.Google Scholar
  102. Sweitzer, B.J. & Cullen, D.J. (1994). How well does a journal's review process function? A survey of authors' opinion. Journal of the American Medical Association 272: 152–153.CrossRefPubMedGoogle Scholar
  103. Van Rooyen S., Godlee, H. et al. (1998). Effect of open peer review on quality of quality of reviews and reviewers recommendation: A randomised trail. British Medical Journal 317: 23–27.Google Scholar
  104. Van Rooyen, S., Godlee, H. et al. (1998). Effect of blinding and unmaksing on the quality of peer review: A clinical trail. Journal of American Medical Association 280: 234–237.CrossRefGoogle Scholar
  105. Ware, J.E. & Williams, R.G. (1975). The Dr. Fox effect: A study of lecturer effectiveness and ratings of instruction. Journal of Medical Education 50: 149–156.PubMedGoogle Scholar
  106. Watkins, M.W. (1979). Chance and interrater agreement on manuscript. American Psychologist 34: 796–798.CrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Mohammadreza Hojat
    • 1
    Email author
  • Joseph S. Gonnella
    • 1
  • Addeane S. Caelleigh
    • 2
  1. 1.Center for Research in Medical Education and Health CareJefferson Medical College of Thomas Jefferson UniversityUSA
  2. 2.George Washington University School of MedicineUSA

Personalised recommendations