Skip to main content
SpringerLink
Log in

The Dedisciplining of Peer Review

  • Published:
Minerva Aims and scope Submit manuscript

Abstract

The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle with the question of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

Notes

  1. The six agencies: in the US, the National Institutes of Health, National Science Foundation, and National Oceanic and Atmospheric Administration; the Canadian Natural Sciences and Engineering Research Council; the Dutch Technology Foundation; and the European Commission’s 7th Framework Programme. In another essay (Holbrook and Frodeman 2011), we compare the approaches of US NSF and European Commission for integrating societal concerns within peer review.

  2. As Holbrook (2010) notes, the origin of peer review is disputable: it is possible to identify earlier medieval and Islamic instances (Spier 2002); and the origins of grant peer review are much more recent (Hackett and Chubin 2003). The National Advisory Cancer Council, a precursor of the NIH, reviewed applications for the Surgeon General beginning in 1937.

  3. William Blanpied (1999) argues that Bush’s report made the “naive assumption” that a government-funded research agency “could function in virtual isolation from normal political processes…. its failure to recognize the inescapable political character of science disqualify it as an adequate guide to the present.” He further notes that the 1947 Steelman Report Science and Public Policy did not gain acceptance: unlike the laissez-faire model developed by Bush, his report relied on the “quasi-New Deal assumption that the government should allocate financial resources for research in both the public and private sectors on the basis of coordinated planning and rationale [sic] management.” This did not sit well with the new Republican leadership of the 80th Congress.

  4. Francis Collins, NIH Director, cited this kind of gap between basic research and social outcomes in his proposal to create a National Center for Advancing Translational Sciences at NIH (see Kaiser 2011).

  5. A key development here was the landmark whistleblowing essay by anesthesiologist Henry Beecher (1966).

  6. See: http://grants.nih.gov/grants/peer/guidelines_general/Review_Criteria_at_a_glance.pdf.

  7. See: http://grants.nih.gov/grants/peer/hs_review_inst.pdf.

  8. For more evidence of this hybridization consider that at the National Cancer Institute, consumer advocates participate in first stage review, which includes “the patient perspective in the assessment of scientific and technical merit.” See: http://carra.cancer.gov/members/roles/types/peer-review.

  9. Although the term of art in Europe tends to be Good Scientific Practice (GSP).

  10. Some of the information presented here is from phone conversations with NIH employees.

  11. http://grants.nih.gov/grants/peer_review_process.htm. Emphasis added.

  12. See: http://www.youtube.com/watch?v=kfgzdLe92c0.

  13. http://funding.niaid.nih.gov/researchfunding/grant/cycle/pages/part09.aspx#b1. This assessment was also confirmed in a phone interview with a representative from the Center for Scientific Review: by far the major determinant of a proposal’s success is its score in the first stage of peer review.

  14. Available here: http://enhancing-peer-review.nih.gov/faqs.html#1708.

  15. Rothenberg (2010) argues the opposite, that “the general nature of the criteria has not changed over the years.”

  16. The retreat by philosophers of science from social values and political engagement was not only the result of the pull of specialization. As George Reisch (2005) argues, the McCarthy era of post-World War II US politics worked against any philosophers of science engaged in social and political issues.

  17. http://www.nsf.gov/about/history/nsf50/nsf8816.jsp.

  18. Ibid.

  19. In 2010, NSF launched a thorough review of its merit review criteria. As of the fall of 2011, the proposed revision of the broader impacts criterion had elicited widespread criticism within the scientific and science policy communities. See e.g., Frodeman and Holbrook (2011).

  20. See the 2010 NSF Proposal and Award Policies and Procedures Guide, Chapter II. C.1e: http://www.nsf.gov/pubs/policydocs/pappguide/nsf10_1/gpg_2.jsp#IIC1e. Also see the Federal Register notice of August 20, 2009: http://edocket.access.gpo.gov/2009/E9-19930.htm.

  21. But there are also signs that NIH is widening its scope beyond this narrow range. For example, see this new parent funding announcement: http://grants.nih.gov/grants/guide/pa-files/PA-11-180.html.

References

  • Allen, Janice, 2010. ‘The NIH Scientific Review Process.’ CAPR Digital Repository, at http://csid-capr.unt.edu/fedora/repository/capr:1063

  • Ben-David, Joseph. 1984. The Scientist’s Role in Society. Chicago: University of Chicago Press.

  • Benos, Dale, et al, 2007. “The Ups and Downs of Peer Review.” Advances in Physiology Education, vol. 31, pp. 145–152.

  • Beecher, Henry, 1966. “Ethics and Clinical Research.” New England Journal of Medicine, vol. 274, pp. 1354–60.

  • Blanpied, William. 1999. “Science and Public Policy: The Steelman Report and the Politics of Post-World War II Science Policy.” AAAS Science and Technology Policy Yearbook, available at: http://www.aaas.org/spp/yearbook/chap29.htm.

  • Bozeman, Barry, and Craig Boardman. 2009. ‘Broad Impacts and Narrow Perspectives: Passing the Buck on Science and Social Impacts.’ Special Issue, Social Epistemology: US National Science Foundation’s Broader Impacts Criterion. Vo. 23, No. 3–4, Dec. 2009, pp. 183–198.

  • Briggle, Adam. 2005. “Institutional Review Boards,” in Encyclopedia of Science, Technology, and Ethics, Carl Mitcham (ed.), 4 vols. Farmington Hills, MI: Macmillan Reference, 2005, vol. 2, pp. 1024–26.

  • Briggle, Adam. 2010. A Rich Bioethics: Public Policy, Biotechnology, and the Kass Council. Notre Dame, IN: University of Notre Dame Press.

  • Carmichael, Mary, and Sharon Begley. 2010. “Desperately Seeking Cures.” Newsweek, May 15, 2010.

  • Echegoyen, Luis. 2008. Dear Colleague Letter on Broader Impacts Proposal Requirements; at http://www.delta.wisc.edu/events/Workshops/workshop_docs/2008%20workshop%20docs/Dear_Colleague_NSF_Letter.pdf

  • Fitzpatrick, Kathleen. 2010. ‘Peer-to-Peer Review and the Future of Scholarly Authority.’ Social Epistemology, vol. 24, no. 3, pp. 161–180.

  • Frodeman, Robert, and J. Britt Holbrook. 2007. “Science’s Social Effects.” Issues in Science and Technology, Vol. 23, Issue 3 (Spring 2007), pp 28–30: http://www.issues.org/23.3/p_frodeman.html.

  • Frodeman, Robert, and J. Britt Holbrook. 2011. “NSF’s Struggle to Articulate Societal Relevance.” Science, July 8, 2011.

  • Frodeman, Robert, and J. Britt Holbrook, under review. “A Comparative Assessment of Metrics for Societal Impact.” Research Evaluation.

  • Gieryn, Thomas. 1999. Cultural Boundaries of Science: Credibility on the Line. Chicago: University of Chicago Press.

  • Guston, David. 2000. Between Politics and Science: Assuring the Integrity and Productivity of Research. Cambridge: Cambridge University Press.

  • Holbrook, J. Britt. 2005. “Assessing the Science – Society Relation: The Case of the U.S. National Science Foundation’s Second Merit Review Criterion,” in Technology in Society, vol. 27, Issue 4, November 2005, pp. 437–451.

  • Holbrook, J. Britt. 2010. “Peer Review,” in The Oxford Handbook of Interdisciplinarity, Robert Frodeman, (ed.). Oxford: Oxford University Press, 2010, pp. 321–32.

  • Holbrook, J. Britt, and Robert Frodeman. 2011. “Peer Review and Societal Impacts: A Comparison of the US National Science Foundation and the European Commission’s Framework Programmes.” Research Evaluation, Volume 20, Number 3, September 2011, 239–246(8)

  • Hollander, unpublished. “Ethics at the National Science Foundation (NSF): A Brief History of Bureaucratic Struggle.” Available at http://csi.ensmp.fr.

  • Kaiser, Jocelyn. 2011. “Collins Sparks Furor with Proposed NIH Reshuffling.” Science, vol. 331, no. 6016 (28 January), p. 386.

  • Kington, 2007, at http://enhancing-peer-review.nih.gov/meetings/102207-summary.html.

  • Kuhn, Thomas, 1970. “Logic of Discovery or Psychology of Research?” in Criticism and the Growth of Knowledge, edited by I. Lakatos and A. Musgrave (London: Cambridge University Press) pp. 1–23.

  • Latour, Bruno. 1993. We Have Never Been Modern. Harvard University Press.

  • Lightborne, James. 2010. Public Presentation, CAPR Midterm Workshop, Washington, DC, April 22, 2010.

  • Mandel, Richard. 1996. A Half Century of Peer Review, 1946–1996. Bethesda, Md.: Division of Research Grants, National Institutes of Health.

  • NAPA, 2001. National Academy of Public Administration Report on Implementation of new Grant Proposal Guide Requirements Related to the Broader Impacts Criterion. Available at http://www.nsf.gov/publications/pub_summ.jsp?ods_key=iin127.

  • National Academies. 2010. ‘Federal Demonstration Partnership: STAR metrics working Group’. Available at http://sites.nationalacademies.org/PGA/fdp/PGA_057189

  • Peer Review Process. 2011. NIH website, at http://grants.nih.gov/grants/peer_review_process.htm.

  • Pielke, Jr., Roger A., and R. Byerly. 1998. ‘Beyond basic and applied’. Physics Today 51 (2) 42–46, Times Cited: 17, issn: 0031–9228.

  • Reisch, George. 2005. How the Cold War Transformed Philosophy of Science: To the Icy Slopes of Logic. Cambridge University Press: Cambridge.

  • Roberts, Melanie. 2009. “Realizing Societal Benefit from Academic Research: Analysis of the National Science Foundation’s Broader Impacts Criterion.” Special Issue, Social Epistemology: US National Science Foundation’s Broader Impacts Criterion. Vo. 23, No. 3-4, Dec. 2009, pp. 199–219.

  • Rossiter, Margaret W. 1984. ‘The History and Philosophy of Science Program at the National Science Foundation.’ Isis, vol. 75, no. 1, pp. 95–104.

  • Rothenberg, Marc. 2010. ‘Making Judgments about Grant Proposals: A Brief History of the Merit Review Criteria at the National Science Foundation.’ Technology and Innovation, Vol. 12, pp. 189–195.

  • Sarewitz, Daniel. 1996. Frontiers of Illusion: Science, Technology, and the Politics of Progress. Temple University Press.

  • Sarewitz, Daniel. 2000. “Science and Environmental Policy: An Excess of Objectivity,” in Earth Matters: The Earth Sciences, Philosophy, and the Claims of Community, edited by Robert Frodeman (Upper Saddle River, NJ: Prentice Hall) pp. 79–98.

  • Sarewitz, Daniel. April 08, 2003. “Does Science Policy Exist, and If So, Does it Matter?: Some Observations on the U.S. R&D Budget.” Discussion paper. Columbia University’s Earth Institute’s Science, Technology and Global Development Seminar, New York, NY.

  • Schienke, Erich, et al. 2009. “The Role of the National Science Foundation Broader Impacts Criterion in Enhancing Research Ethics Pedagogy.” Special Issue, Social Epistemology: US National Science Foundation’s Broader Impacts Criterion. Vo. 23, No. 3–4, Dec. 2009, pp. 317–336.

  • Shapin, Steven, and Simon Schaffer. 1985. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Princeton, NJ: Princeton University Press.

  • Spier, Ray. 2002. “The History of the Peer-review Process,” TRENDS in Biotechnology, vol. 20, no. 8, pp. 357–358.

Download references

Author information

Authors and Affiliations

  1. Department of Philosophy and Religion Studies, Center for the Study of Interdisciplinarity, University of North Texas, 225 EESAT, Box 310920, Denton, TX, 76203-0920, USA

    Robert Frodeman

  2. Department of Philosophy and Religion Studies, University of North Texas, 1155 Union Circle, #310920, Denton, TX, 76203-5017, USA

    Adam Briggle

Authors
  1. Robert Frodeman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adam Briggle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert Frodeman.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Frodeman, R., Briggle, A. The Dedisciplining of Peer Review. Minerva 50, 3–19 (2012). https://doi.org/10.1007/s11024-012-9192-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11024-012-9192-8

Keywords

Search

Navigation