High publication demands and the low acceptance rate of peer review journals place the journal editors and their reviewers in a powerful position. Journal reviewers have a vital role not only in influencing the journal editor's publication decisions, but also in the very nature and direction of scientific research. Because of their influence in peer review outcomes, journal reviewers are aptly described as the “gate keepers of science.” In this article we describe several pitfalls that can impede reviewers' impartial judgement. These include such issues as confirmatory bias, the negative results bias (the file drawer problem), the Matthew effect, the Doctor Fox effect, and gender, race, theoretical orientation, and “political correctness.” We argue that procedures currently used by many professional journals, such as blind or masked review, may not completely alleviate the effects of these pitfalls. Instead, we suggest that increasing reviewers' awareness of the pitfalls, accountability, and vigilance can improve fairness in the peer review process. The ultimate responsibilities belong to the journal editors who are confronted with the difficult task of satisfying journal readers, contributors, reviewers, and owners. We recommend that the journal editors conduct periodic internal and external evaluations of their journals' peer review process and outcomes, with participation of reviewers, contributors, readers and owners.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Albert, T. (1997). Why bother with peer review? Lancet 350: 822.
Altman, D.G., Chalmers, I. & Herxheimer. A. (1994). Is there a case for an international medical scientific press council? Journal of the American Medical Association 272: 166–167.
Angell, M. (1999). The journal and its owner: Resolving the crisis. New England Journal of Medicine 341: 752.
APA (2001). Summary report of journal operations, 2000. American Psychologist 56: 693–694.
Armstrong, S.J. (October 25, 1996). We need to rethink the editorial role of peer reviewers. The Chronicle of Higher Education 43(9): B3.
Bailar, J.C. & Patterson, K. (1985). Journal peer review: The need for a research agenda. New England Journal of Medicine 312: 654–657.
Bartko, J.J. (1966). The intra-class correlation coefficient as a measure of reliability. Psychological Reports 19: 3–11.
Begley, S. (1992, September 14). Is science censored? Newsweek, p. 63.
Belsky, J. (1986). Infant day care: A cause for concern? Zero to Three 10: 22–24.
Belsky, J. (1987, April). Science, social policy and day care: A personal odyssey. Paper presented at the biennial meeting of the Society for Research in Child Development, New York.
Black, N., Van Rooyen, S. et al. (1998).What makes a good reviewer and a good review for a general medical journal. Journal of the American Medical Association 280: 231–233.
Bloom, F.E. (1999). The importance of reviewers. Science 283: 789.
Caelleigh, A.S., Hojat, M. et al. (September 2001). Effects of reviewers' gender on assessments of a gender-related standardized manuscript. Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.
Callaham, M.L., Baxt, W.G. et al. (1998). Reliability of editors' subjective quality ratings of peer reviews of manuscripts. Journal of the American medical Association 280: 229–231.
Callaham, M.L., Wears, R.L. et al. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Journal of the American Medical Association 280: 254–257.
Campion, E.W., Curfman, G.D. & Drazen, J.M. (2000). Tracking the peer-review process. The New England Journal of Medicine 343: 1485–1486.
Cho, M.K., Justice, A.C. et al. (1998). Masking author identity in peer review. Journal of the American Medical Association 280: 243–245.
Cicchetti, D.V. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35: 300–305.
Colman, A.M. (1979). Editorial role in author-referee disagreements. Bulletin of the British Psychological Society 32: 390–391.
Crandall, R. (1990). Author, reviewer and editorial values: Politics in the working mother research? Journal of Social Behavior and Personality 5: 489–491.
Crandall, R. (1982). Editorial responsibilities in manuscript review. The Behavioral and Brain Sciences 5: 207–208.
Crandall, R. (1978). Interrater agreement on manuscripts is not so bad! American Psychologist 33: 623–624.
Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. The American Sociologist 32: 195–201.
DeGrazia, A. (1963). The scientific reception system and Dr. Velikovsky. American Behavioral Scientist 7: 38–56.
Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263: 1385–1389.
Dividoff, F., DeAngelis, C.D. et al. (2001). Sponsorship, authorship, and accountability. Journal of American Medical Association 286: 1232–1233.
Drucker, P.F. (1994). Political correctness and American academe. Society 32: 58–63.
Eagley, A.H. (1995a). The science and politics of comparing women and men. American Psychologist 50: 145–158.
Eagley, A.H. (1995b). Reflections on the commentators' views. American Psychologist 50: 169–171.
Fisher, M., Friedman, S.B. & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association 272: 143–146.
Fontanarosa, P.B., Glass, R.M. & De Angelis (2000). Thanking authors, peer reviewers, and readers – constancy in a time of change. Journal of the American Medical Association 283: 2016–2017.
Gannon, L., Luchetta, T. et al. (1992). Sex bias in psychological research: Progress or complacency? American Psychologist 47: 389–396.
Garfunkel, J.M., Ulshen, M.H. et al. (1994). Effect of institutional prestige on reviewers' recommendations and editorial decisions. Journal of the American Medical Association 272: 137–138.
Garrison, G.E. & Kobor, C.P. (2002). Weathering a political storm: A contextual perspective on a psychological research controversy. American Psychologist 57: 165–175.
Gilbert, J.R., Williams, E.S. & Lundberg, G.D. (1994). Is there gender bias in JAMA's peer review process? Journal of the American Medical Association 272: 139–142.
Godlee, F., Gale, C. Martyn, C.N. (1998). Effect on the quality of peer review of blinding reviewers and asking themto sign their names. Journal of the AmericanMedical Association 280: 237–240.
Goldbeck-Wood, S. (1999). Evidence on peer review – Scientific quality control or smoke screen? British Medical Journal 318: 44–45.
Goldberg, P. (1968). Are some women prejudiced against women? Trans-Action 5: 28–30.
Gottfredson, S.D. (1978). Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33: 920–934.
Greenwald, A.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin 82: 1–20.
Hall, J.A. (1979). Author review of reviewers. American Psychologist 34: 798.
Hare-Mustin, R. & Maracek, J. (1988). The meaning of difference: Gender theory, postmodernism, and psychology. American Psychologist 43: 355–364.
Hare-Mustin, R. & Maracek J. (1990). Making a difference: Psychology and the Construction of Gender. New Haven, CT: Yale University Press.
Hendrick, C. (1976). Editorial comment. Personality and Social Psychology Bulletin 2: 207–208.
Herrnstein, R.J. & Murray, C. (1994). The Bell Curve: Intelligence and Class Structure in American Life. New York: The Free Press.
Hoffman, L.W. (1990). Bias and social responsibility in the study of maternal employment. In C.B. Fisher & W.W. Tryon (eds.), Ethics in Applied Developmental Psychology: Emergency Issues in an Emergency Field (Annual Advances in Applied Developmental Psychology, Vol. 4) (pp. 253–271). Norwood, NJ: Ablex.
Hojat, M. (1993a). A mother's love: What children will not receive in day-care centers. (Interview). The Family in America 8: 1–7.
Hojat, M. (1993b). Abandoning research on consequences of nonmaternal care: A disservice to the science. Journal of Social Behavior and Personality 8: 5–8.
Hojat, M. (1993c). The world declaration of the rights of the child: Anticipated challenges. Psychological Reports 72: 1011–1022.
Hojat, M. (1990). Can affectional ties be purchased? Comments on working mothers and their families. Journal of Social Behavior and Personality 5: 493–502.
Hojat, M. (1995). Developmental pathways to violence: A psychodynamic paradigm. Peace Psychology Review 1: 176–195.
Hojat, M. An empirical study of possible biases due to reviewers' theoretical perspective, gender, and political correctness of the manuscript in journal peer review process, unpublished manuscript.
Hoover, J.D. & Howard, L.A. (1995). The political correctness controversy revisited: Retreat from argumentation and reaffirmation of critical dialogue. American Behavioral Scientist 38: 963–975.
Horrobin, D.F. (1982). Peer review: A philosophically faulty concept which is proving disastrous for science. The Behavioral and Brain Sciences 5: 217–218.
Horton, R. (1998). The journal ombudsperson: A step toward scientific press oversight. Journal of the American Medical Association 280: 298–299.
Horton, R. (1996). The Lancet's ombudsman. Lancet 348: 6.
Howard, L. & Wilkinson, G. (1998). Peer review and editorial decision-making. British Journal of Psychiatry 173: 110–113.
Hunt, E. (1971). Psychological publications. American Psychologist 26: 311.
Jefferson, T. & Godlee, F. (eds.) (1999). Peer Review in Health Care. London, UK: British Journal Publishing Group.
Justice, A.C., Cho, M.K. et al. (1998). Does masking author identity improve peer review quality? A randomized controlled trail. Journal of the American Medical Association 280: 240–242.
Karen, R. (1994). Becoming Attached. New York: Warner Books.
Kassirer, J.P. & Campion, E.W. (1994). Peer review: Crude and understudied, but indispensable. Journal of the American Medical Association 272: 96–97.
Kennedy, D. (2002). Publish or not publish. Science 295: 1793.
King, D.W., McDonald, D.D. & Rodereer, M.K. (1981). Scientific Journals in the United States: Their Production, Use, and Economics. Stroudsburg, PA: Hutchinson & Ross.
Laband, D.N. & Piette,M.J. (1994). A citation analysis of the impact of blinded peer review. Journal of the American 272: 147–149.
Lilienfeld, S.O. (2002).When worlds collide: Social science, politics, and the Rind et al. (1998) child sexual abuse meta-analysis. American Psychologist 57: 176–188.
Lock, S. (1985). A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia: ISI Press.
Lock, S. (ed.) (1991). The Future of Medical Journals. Plymouth, UK: British Medical Journal.
Loury, G.C. (1994). Self-censorship in public discourse: A theory of political correctness and related phenomena. Rationality and Society 6: 428–461.
Mahoney, M. (1977). Publication prejudices: An experimental study of confirmatory bias in peer review system. Cognitive Therapy and Research 1: 161–175.
Mahoney, M.J. (1987). Scientific publication and knowledge politics. Journal of Social Behavior and Personality 2(Part 1): 165–176.
McNutt, R.A., Evans, A.T. et al. (1990). The effects of blinding on the quality of peer review. Journal of the American Medical Association 263: 1371–1376.
Merton, R.K. (1968). The Matthew effect in science. Science 159: 56–63.
Misakian, A.L. & Bero, L.A. (1998). Publication bias and research on passive smoking: Comparison of published and unpublished studies. Journal of the American Medical Association 280: 250–253.
Moore, M. (1978). Discrimination or favoritism? Sex bias in book reviews. American Psychologist 33: 936–938.
Naftulin, D.H., Ware, Jr., J.E. & Donnelly, F.A. (1973). The doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education 48: 630–635.
Nylenna, M., Riis, P. & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscript. Journal of the American Medical Association 272: 149–151.
Olson, C.M. (1990). Peer review of biomedical literature. American Journal of Emergency Medicine 8: 356–358.
Over, R. (1982). What is the source of bias in peer review? The Behavioral and Brain Sciences 5: 229–230.
Oxman A.D., Guyatt, G.H. & Singer, J. (1991). Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44: 91–98.
Peters, D.P. & Ceci, S.J. (1982). Peer review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5: 187–195.
Pierie, J.P., Walvoort, H.C. & Overbeke, A.J.P. (1996). Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 348: 1480–1483.
Popenoe, D. (1993a). American family decline, 1960-1990: A review and appraisal. Journal of Marriage and the Family 55: 527–532.
Popenoe, D. (1993b). The national family wars. Journal of Marriage and the Family 55: 533–553.
Rennie, D. (1998). Peer review in Prague. Journal of the American Medical Association 280: 214–215.
Riger, S. (1992). Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist 47: 730–740.
Riis, P. (1992). New paradigm in journalogy. Journal of Internal Medicine 232: 207–213.
Rind, B., Tromovich, P. & Bauserman, R. (1998). A meta-analytic examination of assumed properties of child sexual abuse using college samples. Psychological Bulletin 124: 22–53.
Rosenberg, R.N. & Anderson, E.R., Jr. (1999). Editorial governance of the Journal of the American Medical Association: A report. Journal of the American Medical Association 281: 2239–2242.
Rosenthal, R. (1979). The “file drawer problem” and tolerance of null results. Psychological Bulletin 86: 541–638.
Rubin, P.H. (1994). The assault on the first amendment: Public choice and political correctness. Cato Journal 14: 23–36.
Scarr, S. & Weber, B.L.R. (1978). The reliability of reviewers for the American Psychologist. American Psychologist 33: 935.
Scott, W.A. (1974). Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 31: 799–804.
Sharp, D.W. (1990). What can and should be done to reduce publication bias? Journal of the American Medical Association 263: 1390–1391.
Shea, J.A., Caelleigh, A.S. et al. (2001). Review process and publication decision. Academic Medicine 76: 911–916.
Silverstein, L.B. (1991). Transforming the debate about child care and maternal employment. American Psychologist 46: 1025–1032.
Smith, R. (1999). Opening of BMJ peer review. British Medical Journal 318: 4–5.
Stark-Adamec, C. (1993). Social science and scientific responsibility. Canadian Journal of Physics 71: 192–196.
Stephenson, J. (1997). Medical journals turn gaze inward to examine process of peer review. Journal of the American Medical Association 278: 1389–1391.
Sterling, T.D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vise versa. Journal American Statistical Association 54: 30–34.
Susser, M. & Northridge, M.E. (1996). Editor's note: Reviewing for the journal. American Journal of Public Health 86: 161.
Sweitzer, B.J. & Cullen, D.J. (1994). How well does a journal's review process function? A survey of authors' opinion. Journal of the American Medical Association 272: 152–153.
Van Rooyen S., Godlee, H. et al. (1998). Effect of open peer review on quality of quality of reviews and reviewers recommendation: A randomised trail. British Medical Journal 317: 23–27.
Van Rooyen, S., Godlee, H. et al. (1998). Effect of blinding and unmaksing on the quality of peer review: A clinical trail. Journal of American Medical Association 280: 234–237.
Ware, J.E. & Williams, R.G. (1975). The Dr. Fox effect: A study of lecturer effectiveness and ratings of instruction. Journal of Medical Education 50: 149–156.
Watkins, M.W. (1979). Chance and interrater agreement on manuscript. American Psychologist 34: 796–798.
About this article
Cite this article
Hojat, M., Gonnella, J.S. & Caelleigh, A.S. Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process. Adv Health Sci Educ Theory Pract 8, 75–96 (2003). https://doi.org/10.1023/A:1022670432373
- gatekeepers of science
- Journal reviewer guidelines
- Journal editor's responsibilities
- peer review
- publication bias