Skip to main content
SpringerLink
Log in

Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system

  • Published:
Science and Engineering Ethics Aims and scope Submit manuscript

Abstract

The documented low levels of reliability of the peer review process present a serious challenge to editors who must often base their publication decisions on conflicting referee recommendations. The purpose of this article is to discuss this process and examine ways to produce a more reliable and useful peer review system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Armstrong, J.S. & Hubbard, R. (1991) Does the need for agreement among reviewers inhibit the publication of controversial findings? Behavioral & Brain Sciences 14:136–137.

    Google Scholar 

  • Bailar, J.C. (1991) Reliability, fairness, objectivity and other inappropriate goals in peer review. Behavioral & Brain Sciences 14:137–138.

    Google Scholar 

  • Cicchetti, D.V. (1976) Assessing inter-rater reliability for rating scales: Resolving some basic issues. British Journal of Psychiatry 129: 452–456.

    Article  Google Scholar 

  • Cicchetti, D.V. (1982) On peer review: “We have met the enemy and he is us.” Behavioral & Brain Sciences 5:205.

    Article  Google Scholar 

  • Cicchetti, D.V. (1991a) The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences 14:119–135.

    Google Scholar 

  • Cicchetti, D.V. (1991b) Author’s response: Reflections from the peer review mirror. Behavioral & Brain Sciences 14:167–186.

    Google Scholar 

  • Cicchetti, D.V. (1997) Peer review: Agreement and disagreement. Behavioral and Brain Sciences. In press.

  • Cicchetti, D.V., & Conn, H.O. (1976) A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Journal of Biology & Medicine 49:373–383.

    Google Scholar 

  • Cicchetti, D.V. & Eron, L.D. (1979) The reliability of manuscript reviewing for the Journal of Abnormal Psychology. Proceedings of the American Statistical Association (Social Statistics Section) 22:596–600.

    Google Scholar 

  • Cicchetti, D.V. & Fleiss, J.L. (1977) Comparison of the null distributions of weighted kappa and the C ordinal statistic. Applied Psychological Methods 1: 195–201.

    Google Scholar 

  • Cicchetti, D.V. & Sparrow, S.S. (1981) Developing criteria for establishing inter-rater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency 86: 127–137.

    Google Scholar 

  • Cicchetti, D.V., Volkmar, F., Klin, A. & Showalter, D. (1995) Diagnosing autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychology, 1: 26–37.

    Google Scholar 

  • Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational & Psychological Measurement 20: 37–46.

    Google Scholar 

  • Cooper, G.S. & Zangwill, L. (1989) An analysis of the quality of research reports in the Journal of General Internal Medicine. Journal of General Internal Medicine 4:232–236.

    Google Scholar 

  • Daniel, H. D. (1993) Guardians of Science: Fairness and reliability of peer review. VHC, New York.

    Google Scholar 

  • Fleiss, J. L. (1975) Measuring agreement between two judges on the presence or absence of a trait. Biometrics 31: 651–659.

    Article  Google Scholar 

  • Fleiss, J. L. (1981) Statistical methods for rates and proportions, Wiley(2nd ed.), New York.

    Google Scholar 

  • Fleiss, J. L. & Cohen, J. (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational & Psychological Measurement 33: 613–619.

    Google Scholar 

  • Fleiss, J. L., Cohen, J. & Everitt, B. S. (1969) Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 72: 323–327.

    Article  Google Scholar 

  • Grove, W. M., Andreason, N. C., McDonald-Scott, P., Keller, M. B. & Shapiro, R. W. (1981) Reliability studies of psychiatric diagnoses: Theory and practice. Archives of General Psychiatry 38:408–413.

    Google Scholar 

  • Hargens, L. L. (1988) Scholarly consensus and journal rejection rates. American Sociological Review 53: 139–151.

    Article  Google Scholar 

  • Hargens, L. L. (1990) Variation in journal peer review systems: possible causes and consequences. Journal of the American Medical Association 263:1348–1352.

    Article  Google Scholar 

  • Ingelfinger, F. J. (1974). Peer review in biomedical publication. American Journal of Medicine 56:686–692.

    Article  Google Scholar 

  • Kemper, K. J., McCarthy, P. L. & Cicchetti, D. V. (1996) Improving participation and interrater agreement in scoring APA abstracts: how well have we succeeded? Archives of Pediatrics and Adolescent Medicine 150: 380–383.

    Google Scholar 

  • Kiesler, C. A. (1991) Confusion between reviewer reliability and wise editorial and funding decisions. Behavioral & Brain Sciences 14:151–152.

    Google Scholar 

  • Lock, S. (1985) A difficult balance: Editorial peer review in medicine. ISI Press.

  • McGraw, K. O. & Wong, S. P. (1996) Forming inferences about some intraclass correlation coefficients. Psychological Methods 1: 30–46.

    Article  Google Scholar 

  • Oxman, A. D., Guyatt, G. H., Singer, J., Goldsmith, C. H., Hutchison, B. G., Milner, R. A. & Streiner, D. L. (1991) Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44: 91–98.

    Article  Google Scholar 

  • Relman, A. S. (1978) Are journals really quality filters? Rockefeller working papers (conference, May 22–23, 1977, New York, NY). Coping with the biomedical literature explosion: A qualitative approach.

  • Rourke, B. P. (1991) Toward openness and fairness in the review process. Behavioral & Brain Sciences 14: 161.

    Google Scholar 

  • Shrout, P. E. & Fleiss, J. L. (1979) Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86: 420–428.

    Article  Google Scholar 

  • Solomon, M. J. & McLeod, R. S. (1993) Clinical studies in surgical journals—have we improved? Diseases of the Colon and Rectum 36: 43–48.

    Article  Google Scholar 

  • Strayhorne, Jr. J., McDermott, Jr., J. F. & Tanguay, P. (1993) An Intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child & Adolescent Psychiatry. American Journal of Psychiatry 150: 947–952.

    Google Scholar 

  • Yalow, R. S. (1978). Radioimmunoassay: A probe for the fine structure of biology systems. In: Les prix nobel en 1977, Nobel Foundation, Almqvist & Wiksell, Stockholm, pp. 243–264.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Yale Child Study Center, 94 Linsley Lake Road, 06471, North Branford, CT, USA

    Domenic V. Cicchetti Ph.D.

Authors
  1. Domenic V. Cicchetti Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

The author is a Senior Research Scientist & Biostatistician, Child Study Center and Department of Psychiatry in the Yale University School of Medicine.

Note: The main points of this report were presented at the Annual Meeting of the American Association for the Advancement of Science (AAAS) in Baltimore, Maryland, on February 9, 1996.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cicchetti, D.V. Referees, editors, and publication practices: Improving the reliability and usefulness of the peer review system. SCI ENG ETHICS 3, 51–62 (1997). https://doi.org/10.1007/s11948-997-0016-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11948-997-0016-4

Keywords

Search

Navigation