Skip to main content

Bias and Groupthink in Science’s Peer-Review System

Abstract

Studies have shown that various types of biases can impact scientific peer review. These biases may contribute to a type of groupthink that can make it difficult to obtain funding or publish innovative or controversial research. The desire to achieve consensus and uniformity within a research group or scientific discipline can make it difficult for individuals to contradict the status quo. This chapter reviews the scientific literature regarding biases in the peer-review system, reflects on the potential impact of bias, and discusses approaches to minimize or control bias in peer review.

Keywords

  • Peer review
  • Scientific journals
  • Publication bias
  • Gender bias

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-36822-7_9
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-36822-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)
Hardcover Book
USD   109.99
Price excludes VAT (USA)

Notes

  1. 1.

    Our definition is loosely inspired by Irvin’s definition (1972) but has been modified so as to apply to the context of science and peer review.

References

  • Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3(1), 63–84.

    CrossRef  Google Scholar 

  • Baggs, J. G., Broome, M. E., Dougherty, M. C., Freda, M. C., & Kearney, M. H. (2008). Blinding in peer review: The preferences of reviewers for nursing journals. Journal of Advances in Nursing, 64(2), 131–138.

    CrossRef  Google Scholar 

  • Barber, B. (1961). Resistance by scientists to scientific discovery. Science, 134(3479), 596–602.

    PubMed  CrossRef  Google Scholar 

  • Bornmann, L., Mutza, R., & Daniela, H. D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3), 226–238.

    CrossRef  Google Scholar 

  • Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., et al. (2009). To name or not to name: The effect of changing author gender on peer review. Bioscience, 59(11), 985–989.

    CrossRef  Google Scholar 

  • Buchen, L. (2009). May 29, 1919: A major eclipse, relatively speaking. Wired, May 29, 2009. Available at: https://www.wired.com/2009/05/dayintech_0529/. Accessed 17 Apr 2017.

  • Campanario, J. M. (2009). Rejecting and resisting Nobel class discoveries: Accounts by Nobel Laureates. Scientometrics, 81(2), 549–565.

    CrossRef  Google Scholar 

  • Card, R. F. (2005). Making sense of the diversity-based legal argument for affirmative action. Public Affairs Quarterly, 19(1), 11–24.

    Google Scholar 

  • Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Sciences of the United States of America, 108(8), 3157–3162.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Chubin, D., & Hackett, E. (1990). Peerless science: Peer review and U.S. science policy. Albany, NY: State University of New York Press.

    Google Scholar 

  • Committee on Science, Engineering, and Public Policy. (2010). Expanding underrepresented minority participation: America’s science and technology talent at the crossroads. Washington, DC: National Academies Press.

    Google Scholar 

  • Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & Reporting Bias Group. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias – An updated review. PLoS One, 8(7), e66844.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research. Lancet, 337(8746), 867–872.

    PubMed  CrossRef  Google Scholar 

  • Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352(6336), 560.

    CrossRef  Google Scholar 

  • Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association, 272(2), 143–146.

    PubMed  CrossRef  Google Scholar 

  • Garfield, E. (1989). Delayed recognition in scientific discovery: Citation frequency analyses aids the search for case histories. Current Contents, 23, 3–9.

    Google Scholar 

  • Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige on reviewers’ recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137–138.

    PubMed  CrossRef  Google Scholar 

  • Ginther, D. K., Haak, L. L., Schaffer, W. T., & Kington, R. (2012). Are race, ethnicity, and medical school affiliation associated with NIH R01 type 1 award probability for physician investigators? Academic Medicine, 87(11), 1516–1524.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Ginther, D. K., Kahn, S., & Schaffer, W. T. (2016). Gender, race/ethnicity, and national institutes of health r01 research awards: Is there evidence of a double bind for women of color? Academic Medicine, 91(8), 1098–1107.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Ginther, D. K., Schaffer, W., Schnel, J., Masimore, B., Liu, F., Haak, L. L., et al. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045), 1015–1019.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. Journal of the American Medical Association, 280(3), 237–240.

    PubMed  CrossRef  Google Scholar 

  • Grod, O. N., Budden, A. E., Tregenza, T., Koricheva, J., Leimu, R., Aarssen, L. W., et al. (2008). Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PLoS One, 3(9), e3202.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Harding, S. (2004). Asocially relevant philosophy of science? Resources from standpoint theory’s controversiality. Hypatia, 19(1), 25–47.

    CrossRef  Google Scholar 

  • Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017). Gender bias in scholarly peer review. In: P. Rodgers (Ed.). eLife, 6, e21718.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Ho, R. C., Mak, K. K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013). Views on the peer review system of biomedical journals: An online survey of academics from high-ranking universities. BMC Medical Research Methodology, 13, 74.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Hunter, J. (2012). Post-publication peer review: Opening up scientific conversation. Frontiers in Computational Neuroscience, 6, 63. (August 30).

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Jang, J., Vannier, M. W., Wang, F., Deng, Y., Ou, F., Bennett, J., et al. (2013). A bibliometric analysis of academic publication and NIH funding. Journal of Informetrics, 7(2), 318–324.

    CrossRef  Google Scholar 

  • Janis, I. L. (1972). Victims of groupthink: A psychological study of foreign policy decisions and fiascoes (2nd ed.). Boston: Houghton Mifflin Company.

    Google Scholar 

  • Janis, I. L. (1982). Groupthink: Psychological studies of policy decisions and fiascos. Boston: Cengage Learning.

    Google Scholar 

  • Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. PEER investigators. Journal of the American Medical Association, 280(3), 240–242.

    PubMed  CrossRef  Google Scholar 

  • Kaatz, A., Lee, Y. G., Potvien, A., Magua, W., Filut, A., Bhattacharya, A., et al. (2016). Analysis of National Institutes of Health R01 application critiques, impact, and criteria scores: Does the sex of the principal investigator make a difference? Academic Medicine, 91(8), 1080–1088.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Kaatz, A., Magua, W., Zimmerman, D. R., & Carnes, M. (2015). A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution. Academic Medicine, 90(1), 69–75.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.

    Google Scholar 

  • Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press.

    CrossRef  Google Scholar 

  • Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.

    CrossRef  Google Scholar 

  • Larivière, V., Ni, C., Gingras, Y., Cronin, B., & Sugimoto, C. (2013). Global gender disparities in science. Nature, 504(7479), 211–213.

    PubMed  CrossRef  Google Scholar 

  • Latour, B., & Woolgar, S. (1979). Laboratory life: The social construction of scientific facts. London: Sage.

    Google Scholar 

  • Lauer, M. S., Krumholz, H. M., & Topol, E. J. (2015). Time for a prepublication culture in clinical research? Lancet, 386(1012), 2447–2449.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

    CrossRef  Google Scholar 

  • Longino, H. (1990). Science as social knowledge. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Lycan, W. G. (1988). Judgement and justification. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Mahoney, M. J. (1977). Publication preferences: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.

    CrossRef  Google Scholar 

  • McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10), 1371–1376.

    PubMed  CrossRef  Google Scholar 

  • Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E., et al. (2016). Bias in research grant evaluation has dire consequences for small universities. PLoS One, 11(6), e0155876.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Nelson, D. J. (2007). A national analysis of minorities in science and engineering faculties at research universities. Norman, OK: University of Oklahoma.

    Google Scholar 

  • Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., et al. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287(21), 2825–2828.

    PubMed  CrossRef  Google Scholar 

  • Planck, M. (1962). Quoted in Kuhn, T. S. 1962). The structure of scientific revolutions (pp. 33–34). Chicago: University of Chicago Press.

    Google Scholar 

  • Pulverer, B. (2016). Preparing for preprints. EMBO Journal, 35(24), 2617–2619.

    PubMed  CrossRef  PubMed Central  Google Scholar 

  • Quine, W. V. (1961). From a logical point of view. New York: Harper and Rowe.

    Google Scholar 

  • Resch, K. I., Ernst, E., & Garrow, J. (2000). A randomized controlled study of reviewer bias against an unconventional therapy. Journal of the Royal Society of Medicine, 93(4), 164–167.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Resnik, D. B. (1994). Methodological conservatism and social epistemology. International Studies in the Philosophy of Science, 8(3), 247–264.

    CrossRef  Google Scholar 

  • Resnik, D. B., & Elmore, S. A. (2016). Ensuring the quality, fairness, and integrity of journal peer review: A possible role for editors. Science and Engineering Ethics, 22(1), 169–188.

    PubMed  CrossRef  Google Scholar 

  • Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., et al. (2006). Effect of blinded peer review on abstract acceptance. Journal of the American Medical Association, 295(14), 1675–1680.

    PubMed  CrossRef  Google Scholar 

  • Shamoo, A. E., & Resnik, D. B. (2015). Responsible conduct of research (3rd ed.). New York: Oxford University Press.

    Google Scholar 

  • Shen, H. (2013). Mind the gender gap. Nature, 495(7439), 22–24.

    PubMed  CrossRef  Google Scholar 

  • Sklar, L. (1975). Methodological conservatism. Philosophical Review, 84(3), 374–400.

    CrossRef  Google Scholar 

  • Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Stent, G. S. (1972). Prematurity and uniqueness in scientific discovery. Scientific American, 227(6), 84–93.

    PubMed  CrossRef  Google Scholar 

  • Stern, J. M., & Simes, R. J. (1997). Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315(7109), 640–645.

    PubMed  CrossRef  PubMed Central  Google Scholar 

  • Teixeira da Silva, J. A., & Dobránszki, J. (2015). Problems with traditional science publishing and finding a wider niche for post-publication peer review. Accountability in Research, 22(1), 22–40.

    PubMed  CrossRef  Google Scholar 

  • Timmer, A., Hilsden, R. J., & Sutherland, L. R. (2001). Determinants of abstract acceptance for the Digestive Diseases Week--a cross sectional study. BMC Medical Research Methodology, 1, 13.

    PubMed  PubMed Central  CrossRef  Google Scholar 

  • Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8), 349–350.

    CrossRef  Google Scholar 

  • van Rooyen, S., Delamothe, T., & Evans, S. J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: Randomised controlled trial. British Medical Journal, 341, c5729.

    PubMed  CrossRef  PubMed Central  Google Scholar 

  • van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal, 318(7175), 23–27.

    PubMed  CrossRef  PubMed Central  Google Scholar 

  • van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review: A randomized trial. Journal of the American Medical Association, 280(3), 234–237.

    PubMed  CrossRef  Google Scholar 

  • Waisbren, S. E., Bowles, H., Hasan, T., Zou, K. H., Emans, S. J., Goldberg, C., et al. (2008). Gender differences in research grant applications and funding outcomes for medical school faculty. Journal of Women’s Health (Larchmont), 17(2), 207–214.

    CrossRef  Google Scholar 

  • Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: A randomised controlled trial. British Journal of Psychiatry, 176, 47–51.

    CrossRef  Google Scholar 

  • Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341–343.

    PubMed  CrossRef  Google Scholar 

Download references

Acknowledgments

This research was supported, in part, by the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH), and the Fonds de Recherche du Québec en Santé (FRQS). This paper does not represent the views of the NIEHS, NIH, the FRQS, or any governmental organization.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to David B. Resnik .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 The Author(s)

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Resnik, D.B., Smith, E.M. (2020). Bias and Groupthink in Science’s Peer-Review System. In: Allen, D., Howell, J. (eds) Groupthink in Science. Springer, Cham. https://doi.org/10.1007/978-3-030-36822-7_9

Download citation