This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings such as invited papers, early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Abramowitz, S. I., Gomes, B. & Abramowitz, C. V. (1975) Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology 5, No. 3: 187–200. EX
Abrams, P. A. (1991) The predictive ability of peer review of grant proposals: The case of ecology and the US National Science Foundation. Social Studies of Science 21: 111–132. ER
Arkes, H. (1996) The persistence of management folklore. Interfaces 26, No. 4: 42–44.
Armstrong, J. S. (1980) Unintelligible management research and academic prestige. Interfaces 10 (April): 80–86. EXR
Armstrong, J. S. (1982) Barriers to scientific contributions: The author’s formula. The Behavioral and Brain Sciences 5: 197–199. ER
Armstrong, J. S. (1985) Long-Range Forecasting. John Wiley, New York.
Armstrong, J. S. (1996) Management folklore and management science: On portfolio planning, escalation bias, and such (with commentaries). Interfaces 26, No. 4: 25–55.
Armstrong, J. S. & Hubbard, R. (1991) Does the need for agreement among reviewers inhibit the publication of controversial findings? Behavioral and Brain Sciences 14 (March): 136–137. ER
Atkinson, D. R., Furlong, M. J. & Wampold, B. E. (1982) Statistical significance, reviewer evaluations, and the scientific process: Is there a statistically significant relationship? Journal of Counseling Psychology 29, No. 2: 189–194. EX
Bakanic, V., McPhail, C. & Simon, R. J. (1990) If at first you don’t succeed: Review procedures for revised and resubmitted manuscripts. American Sociologist 21, No 4: 373–391. E
Barber, B. (1961) Resistance by scientists to scientific discovery. Science 134: 596–602.
Begg, C. B. & Berlin, J. A. (1988) Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A 151: 419–463. E
Beyer, J. M., Chanove, R. G. & Fox, W. B. (1995) The review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal 38: 1219–1260. ER
Blank, R. M. (1991) The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review. American Economic Review 81: 1041–1067. EXR
Bradley, J. V. (1981) Pernicious publication practices. Bulletin of the Psychonomic Society 18: 31–34. E
Broad, W. & Wade, N. (1982) Betrayers of the Truth. Simon and Schuster, New York.
Campanario, J. M. (1995) On influential books and journal articles initially rejected because of negative referees’ evaluations. Science Communication 16 (March): 304–325. ER
Chalmers, I. (1990) Underreporting research is scientific misconduct. Journal of the American Medical Association 263, No. 10: 1405–1408.
Chalmers, T. C., Frank, C. S. & Reitman, D. (1990) Minimizing the three stages of publication bias. Journal of the American Medical Association 263, No. 10: 1392–1395. E
Christiansen, D. (1978) The perils of publishing. IEEE Spectrum 15, No. 5: 27.
Cho, M. K. & Bero, L. A. (1996) The quality of drug studies published in symposium proceedings. Annals of Internal Medicine 124, No. 5: 485–489. E
Cicchetti, D. V. (1991) The reliability of peer review for manuscript and grant submissions: A cross disciplinary investigation. Behavioral and Brain Sciences 14 (March): 119–186. E
Cohen, J. (1994) The earth is round (p<.05). American Psychologist 49: 997–1003.
Davidson, R. A. (1986) Source of funding and outcome of clinical trials. Journal of General Internal Medicine 1: 155–158. ER
Dewald, W. G., Thursby, J. G. & Anderson, R. G. (1986) Replication in empirical economics: The Journal of Money, Credit, and Banking project. American Economic Review 76: 587–603. E
Eichorn, P. & Yankauer, A. (1987) Do authors check their references? A survey of accuracy of references in three public health journals. American Journal of Public Health 77: 1011–1012. ER
Epstein, W. M. (1990) Confirmational response bias among social work journals. Science, Technology, and Human Values 15: 9–38. EXR
Evans, J. T., Nadjari, H. I. & Burchell, S. A. (1990) Quotational and reference accuracy in surgical journals: A continuing peer review problem. Journal of the American Medical Association 263, No. 10: 1353–1354. E
Fagan, W. T. (1990) To accept or reject: Peer review. Journal of Educational Thought 24: 103–113. ER
Fletcher, R. H. & Fletcher, S. W. (1997) Evidence for the Effectiveness of Peer Review. Science and Engineering Ethics 3: 35–50. EX
Franke, R. H., Edlund, T. W. & Oster, F. (1990) The development of strategic management: Journal quality and article impact. Strategic Management Journal 11: 243–253. ER
Fuess, S. M. (1996) On replication in business and economics research: The QJBE case. Quarterly Journal of Business and Economics 35, No. 2: 3–13. E
Friedman, P. J. (1990) Correcting the literature following fraudulent publication. Journal of the American Medical Association 263: 1416–1419. ER
Gans, J. S. & Shepherd, G. B. (1994) How are the mighty fallen: Rejected classic articles by leading economists. Journal of Economic Perspectives 8, No. 1: 165–179. E
Garcia, J. (1981) Tilting at the paper mills of academe. American Psychologist 36, No. 2: 149–158.
Garfield, E. & Welljams-Dorof, A. (1990) The impact of fraudulent research on the scientific literature. Journal of the American Medical Association 263, No. 10: 1424–1426. E
Goodstein, L. & Brazis, K. (1970) Credibility of psychologists: An empirical study. Psychological Reports 27, No. 3: 835–838. E
Gottfredson, S. D. (1978) Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33: 920–934. EX
Greenwald, A. G. (1975) Consequences of prejudice against the null hypothesis. Psychological Bulletin 82: 1–20. E
Hargens, L. L. (1990) Variation in journal peer review systems: possible causes and consequences. Journal of the American Medical Association 263: 1348–1352.
Harnad, S. (1979) Creative disagreement. The Sciences 19: 18–20.
Hendrick, C. (1976) Editorial comment. Personality and Social Psychology Bulletin 2, No. 3: 207–208.
Hibbitts, B. J. (1996a) Last writes? Re-assessing the law review in the age of cyberspace. http://www.law.pitt.edu/hibbitts/lastrev.htm; version 1.1, June 4, 1996; New York University Law Review 17: 615–688.
Hibbitts, B. J. (1996b) Yesterday once more: Skeptics, scribes and the demise of law reviews. Akron Law Review 30 (forthcoming).
Holub, H. W., Tappeiner, G. & Eberharter, V. (1991) The iron law of important articles. Southern Economic Journal 58: 317–328. ER
Horrobin, D. F. (1982) A philosophically faulty concept which is proving disastrous for science. Behavioral and Brain Sciences 5, No. 2: 217–218.
Horrobin, D. F. (1990) The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association 263 (March 9): 1438–1441.
Horrobin, D. F. (1996) Peer review of research grant application. Lancet (forthcoming).
Hubbard, R. & Armstrong, J. S. (1992) Are null results becoming an endangered species in marketing? Marketing Letters 3: 127–136. ER
Hubbard, R. & Armstrong, J. S. (1994) Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing 11: 233–248. ER
Hubbard, R. & Vetter, D. (1996) An empirical comparison of published replication research in accounting, economics, finance, management and marketing. Journal of Business Research 35: 153–164. ER
Jauch, L. R. & Wall, J. L. (1989) What they do when they get your manuscript: A survey of Academy of Management reviewer practices. Academy of Management Journal 32: 157–173. E
Juhasz, S., Calvert, E., Jackson, T., Kronick, D. A. & Shipton, J. (1975) Acceptance and rejection of manuscripts. IEEE Transactions of Professional Communications PC18: 177–184. E
Kerr, S., Tolliver, J. & Petree, D. (1972) Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20, No. 1: 132–141. ER
King, D. W., McDonald, D. D. & Roderer, N. K. (1981) Scientific Journals in the United States: Their Production, Use, and Economics. Hutchison Ross, Stroudsburg, Pa. E
Kochler, J. J. (1993) The influence of prior beliefs on scientific judgments of evidence quality. Organizational Behavior and Human Decision Processes 56: 28–55. EXR
Kuhn, T. S. (1962) The Structure of Scientific Revolutions. University of Chicago Press, Chicago.
Kupfersmid, J. & Wonderly, D. M. (1994) An Author’s Guide to Publishing Better Articles in Better Journals in the Behavioral Sciences. Clinical Psychology Publishing Co., Brandon, Vermont. ER
Lau, R. R. (1994) An analysis of the accuracy of “trial heat” polls during the 1992 presidential election. Public Opinion Quarterly 58: 2–20.
Lindsey, D. (1978) The Scientific Publication System in Social Science. Jossey Bass, San Francisco. E
Liversidge, A. (1989) PNAS publication of AIDS article spurs debate over peer review. The Scientist 3, No. 7: 4–5, 19.
Lock, S. & Smith, J. (1986) Peer review at work. Scholarly Publishing 17, No. 4: 303–316. E
Lock, S. & Smith, J. (1990) What do peer reviewers do? Journal of the American Medical Association 263, No. 10: 1341–1343. E
Lloyd, M. E. (1990) Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis 23: 539–543. EX
MacNealy, M. S., Speck, B. W. & Clements, N. (1994) Publishing in technical communication journals from the successful author’s point of view. Technical communication 41, No. 2: 240–259. E
Mahoney, M. (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1: 161–175. EXR
Marsh, H. W. & Ball, S. (1989) The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education 57, No. 2: 151–169. E
McCloskey, D. N. & Ziliak, S. T. (1996) The standard error of regressions. Journal of Economic Literature 34 (March): 97–114. E
McNutt, R. A., Evans, A. T., Fletcher, R. H. & Fletcher, S. W. (1990) The effects of blinding on the quality of peer review: A randomized trial. Journal of the American Medical Association 263, No. 10: 1371–1376. EX.
Munley, P. H., Sharkin, B. & Gelso, C. J. (1988) Reviewer ratings and agreement on manuscripts reviewed for the Journal of Counseling Psychology. Journal of Counseling Psychology 35, No. 2: 198–202. E
Murray, G. D. (1988) The task of a statistical referce. British Journal of Surgery 75: 664–667. E
Needleman, H. L. (1992) Salem comes to the National Institutes of Health: Notes from inside the crucible of scientific integrity. Pediatrics 90, No. 6: 977–981.
Newcombe, R. G. (1987) Towards a reduction in publication bias. British Medical Journal 295 (12 September): 656–659.
Patterson, S. C. & Smithey, S. K. (1990) Monitoring scholarly journal publication in political science: The role of the APSR. PS: Political Science and Politics 23: 647–656. ER
Perlman, D. (1982) Reviewer “bias”: Do Peters and Ceci protest too much? The Behavioral and Brain Sciences 5: 231–232. E
Pfeifer, M. P. & Snodgrass, G. L. (1990) The continued use of retracted, invalid scientific literature. Journal of the American Medical Association 263, No. 10: 1420–1423. E
Peters, D. P. & Ceci, S. J. (1982) Peer-review practices of psychology journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5: 187–195. EX
Pressman, S. (1994) Simultaneous multiple journal submissions: The case against. American Journal of Economics and Sociology 53: 316–333.
Rodman, H. & Mancini, J. A. (1977) Errors, manuscripts, and equal treatment. Research in Higher Education 7: 369–374. ER
Roediger, H. L. (1987) The role of journal editors in the scientific process. In: Jackson, D. N. & Rushton, J. P. (eds.), Scientific Excellence. Sage Publications, London.
Rowney, J. A. & Zenisek, T. J. (1980) Manuscript characteristics influencing reviewers’ decisions. Canadian Psychology 21, 17–21. ER
Rushton, J. P. & Ankney, C. D. (1996) Brain size and cognitive ability: Correlations with age, sex, social class, and race. Psychonomic Bulletin and Review 3, No. 1: 21–36.
Salsburg, D. S. (1985) The religion of statistics as practiced in medical journals. American Statistician 39: 220–223. E
Shea, C. (1996) Psychologists debate accuracy of ‘significance test. The Chronicle of Higher Education 42 (August 16): A12 & A17.
Simon, R., Bakanic, V. & McPhail, C. (1986) Who complains to editors and what happens. Sociological Inquiry 56: 259–271. ER
Smart, R. G. (1964) The importance of negative results in psychological research. Canadian Psychologist 5: 225–232. E
Speck, B. W. (1993) Publication Peer Review. Westport, Connecticut.
Spiegel-Rosing, I. (1977) Bibliometric and content analysis. Social Studies of Science 7: 97–113. E
Stamps, A. E. III (1997) Advances in Reer Review Research: An introduction. Science and Engineering Ethics 3: 3–10. ER
Stamps, A. E. III (1997) Using a Dialectical Scientific Brief in Peer Review. Science and Engineering Ethics 3: 85–98.
Sterling, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa. Journal of the American Statistical Association, 54: 30–34. ER
Sterling, T. D., Rosenbaum, W. L. & Weinkam, J. J. (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician 49: 108–112. ER
Stewart, W. W. & Feder, N. (1987) The integrity of the scientific literature. Nature 325: 207–214. ER
Szenberg, M. (1994) Disseminating scholarly output: The case for eliminating the exclusivity of journal submissions. American Journal of Economics and Sociology 53: 303–315.
Taubes, G. (1994) Peer review in cyberspace. Science 266: 967.
Tetlock, P. E. & Kim, J. I. (1987) Accountability and judgment process in a personality prediction task. Journal of Personality and Social Psychology 52: 700–709.
Wade, N. (1976) IQ and heredity: Suspicion of fraud beclouds classic experiment. Science 194: 916–919.
Weiss, D. J. (1989) An experiment in publication: Advance publication review. Applied Psychological Measurement 13: 1–7.
Wilson, J. D. (1978) Peer review and publication. Journal of Clinical Investigation 61, 1697–1701.
Yankauer, A. (1985) Peering at peer review. CBE Views 8, No. 2: 7–10. ER
Yankauer, A. (1990) Who are the peer reviewers and how much do they review? Journal of the American Medical Association 263: 1338–1340. ER
The author, a professor of marketing at the Wharton School since 1968, was a founder editor of the Journal of Forecasting and the International Journal of Forecasting.
This paper is based on a presentation at a workshop, “Advances in Peer Review Research”, American Association for the Advancement of Science Meeting, Baltimore, MD, February 9, 1996.
About this article
Cite this article
Armstrong, J.S. Peer review for journals: Evidence on quality control, fairness, and innovation. SCI ENG ETHICS 3, 63–84 (1997). https://doi.org/10.1007/s11948-997-0017-3