The “reproducibility crisis” has been one of the most significant stories in science in the past 15 years and has led to significant policy changes across the research landscape. Yet, scandals, irreproducible studies, and cries of crisis have occurred for decades in science. This article seeks to explain why the reproducibility crisis has taken root and become a force in science policy in ways previous crises have not. In short, we argue that it was through the scientific, institutional, and cultural efforts of a group of scientific activists we are calling metascientists. Metascience is a scientific social movement that seeks to use quantification and experimentation to diagnose problems in research practice and improve efficiency. It draws together data scientists, experimental and statistical methodologists, and open science activists into a project with both intellectual and policy dimensions. Metascientists have been remarkably successful at winning grants, motivating news coverage, and changing policies at science agencies, journals, and universities. The social movement lens is useful for understanding the popularization and impact of the reproducibility crisis narrative and suggests ways the institutions of science are adapting to meet a changing political and technological landscape.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Because many of the groups are known by acronyms, the following is a reference list:
COAR- Confederation of Open Access Repositories
COS- Center for Open Science
DARPA- Defense Advanced Research Projects Agency
METRICS- Meta-research Innovation Center at Stanford
SoS- Science of science
SPARC- Scholarly Publishing and Academic Resources Coalition
Penders et al. (2020) suggest a similar typology.
“Open software” has also played a role in metascience, but mainly through the data science/science of science pathway. Specifically, the focus on the openness of code, decentralization, and modifiability of systems that one finds in metascience discussions (e.g., Nosek and Bar-Anan 2012; Ross-Hellauer et al. 2019; Uhlmann et al. 2019) can be traced to the utopian rhetoric of early open software activists (Kelty 2008).
There are, it should be noted, attempts to create some productive dialogue between the quantitative and qualitative traditions of science studies (Bowker 2020; Marres and de Rijcke 2020; Milojevića et al. 2014), some even involving avowed metascientists (e.g., Kang and Evans 2020). However, both self-consciously positioning themselves in reference to a replication crisis and, as we show later, intentionally keeping theoretical complexity at arm’s distance have allowed the field to gain a position in policy that has largely eluded scientometrics and, certainly, qualitative studies of science.
At the Metascience Symposium, DARPA program manager Adam Russell explained that his agency was tasked with synthesizing research to provide guidance for military and political decisions. However, reproducibility problems and the general low quality of scientific research, especially in the social sciences, resulted in a deep skepticism of the published literature. To counter that, the agency has poured tens of millions of dollars into a spate of metascience projects designed to improve the science through a combination of open science initiatives and machine learning.
“Systematizing Confidence in Open Research and Evidence” is an attempt to create an automated system to evaluate behavioral and social science research. “Big Mechanism” seeks to use machine learning to read the academic literature in cancer and develop novel causal models. And, “Automating Scientific Knowledge Extraction” is trying to use artificial intelligence to automatically collect information, extract its useful components, integrate this information into evolving models with the goal of eventually “generating […] machine-generated hypotheses.”
These projects make it clear that the ultimate goal, for at least some metascientists, is not merely the improvement of existing disciplines, but a radical transformation in which disciplinary science feeds into evolving artificial intelligence systems which evaluate, synthesize, and even conduct research.
Historically, the same tension has existed between “domain” experts and the supposedly “domain-free” expertise of software engineers who have attempted to develop automated systems in expert domains (Ribes et al. 2019).
Adut, Ari. 2008. On Scandal: Moral Disturbances in Society, Politics, and Art. Cambridge: Cambridge University Press.
Angrist, Joshua D, and Pischke, Jorn-Steffen. 2010. The credibility revolution in empirical economics: How better research design is taking the con out of economics. The Journal of Economic Perspectives, 24(2): 3–30.
Arribas-Ayllon, Michael, Andrew Bartlett, and Jamie Lewis. 2019. Psychiatric Genetics: From Hereditary Madness to Big Biology. London: Routledge.
Baker, Monya. 2016. Is There a Reproducibility Crisis? Nature 533: 452–454.
Becker, Howard S. 2008. Outsiders. New York: Simon and Schuster.
Begley, C. Glenn, and John P.A. Ioannidis. 2015. Reproducibility in science. Circulation Research 116(1): 116–126. https://doi.org/10.1161/CIRCRESAHA.114.303819.
Begley, C. Glenn, and Lee M. Ellis. 2012. Drug Development: Raise Standards for Preclinical Cancer Research. Nature 483(7391): 531–533.
Belluz, Julia. 2017. Cancer Scientists are Having Trouble Replicating Groundbreaking Research. Vox, Jan. 23. https://www.vox.com/science-and-health/2017/1/23/14324326/replication-science-is-hard. Retrieved 25 June 2020.
Bohlin, Ingemar. 2004. Communication Regimes in Competition: The Current Transition in Scholarly Communication Seen Through the Lens of the Sociology of Technology. Social Studies of Science 34(3): 356–391.
Bourdieu, Pierre. 2004. Science of Science and Reflexivity. Cambridge: Polity.
Bowker, Geoffrey C. 2020. Number or No Numbers in Science Studies. Quantitative Science Studies 1(3): 927–929.
Brown, Phil, Stephen Zavestoski, Sabrina McCormick, Brian Mayer, Rachel Morello-Frosch, and Rebecca Gasior Altman. 2004. Embodied Health Movements: New Approaches to Social Movements in Health. Sociology of Health & Illness 26(1): 50–80.
Bunge, Mario. 1959. Metascientific Inquiries. Springfield: C. C. Thomas.
Chivers, Tom. 2017. These People are Trying to Fix a Huge Problem in Science. Buzzfeed News, Jul. 29. https://www.buzzfeed.com/tomchivers/signifying-nothing. Retrieved 25 June 2020.
Collins, Harry M. 1981. The Place of the Core-Set in Modern Science: Social Contingency with Methodological Propriety. History of Science 19(1): 6–19.
Collins, Harry M. 2019. Forms of Life: The Method and Meaning of Sociology. Cambridge: MIT Press.
COS. 2017. Center for Open Science: Strategic Plan. https://osf.io/x2w9h/. Retrieved 28 June 2020.
De Solla Price, Derek. 1961. Science Since Babylon. New Haven: Yale University Press.
Deterding, Nicole M., and Mary C. Waters. 2018. Flexible Coding of In-depth Interviews: A Twenty-first-century Approach. Sociological Methods and Research. https://doi.org/10.1177/0049124118799377.
Devezer, Berna, Luis G. Nardin, Bert Baumgaertner, and Erkan Ozge Buzbas. 2019. Scientific Discovery in a Model-Centric Framework: Reproducibility, Innovation, and Epistemic Diversity. PLoS ONE 14(5): e0216125.
Epstein, Steven. 2008. Inclusion: The Politics of Difference in Medical Research. Chicago, IL: University of Chicago Press.
Fanelli, Daniele. 2009. How many scientists fabricate and falisify research? A systematic review and meta-analyses of survey data. PLoS ONE 4(5): e5738.
Fanelli, Daniele. 2019. Is Science Really Facing a Reproducibility Crisis, and Do We Need It To? PNAS 115(11): 2628–2631.
Faye, Cathy. 2012. American social psychology: Examining the contours of the 1970s crisis. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences 43(2): 514–521.
Fieldler, Klaus, and Norbert Schwarz. 2016. Questionable Research Practices Revisited. Social Psychological and Personality Science 7(1): 45–52.
Fleck, Ludwik. 1960 (1986). Crisis in Science. In Cognition and Fact: Materials on Ludwik Fleck, eds. R. S. Cohen and T. Schnelle, 153–158. Dordrecht: D. Reidel Publishing Company.
Flis, Ivan. 2019. Psychologists Psychologizing Scientific Psychology: An Epistemological Reading of the Replication Crisis. Theory & Psychology 29(2): 158–181.
Feest, Uljana. 2019. Why Replication Is Overrated. Philosophy of Science 86(5): 895–905.
Fortunato, Santo, Carl T. Bergstrom, Katy Börner, James A. Evans, Dirk Helbing, Staša Milojević, Alexander M. Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, Alessandro Vespignani, Ludo Waltman, Dashun Wang, and Albert-László Barabási. 2018. Science of Science. Science 359(6379): eaao0185.
Freese, Jeremy, and David Peterson. 2018. The Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science. Sociological Theory 36(3): 289–313.
Frickel, Scott, and Neil Gross. 2005. A General Theory of Scientific/Intellectual Movements. American Sociological Review 70(2): 204–232.
Fujimura, Joan H. 1992. Crafting Science: Standardized Packages, Boundary Objects, and ‘Translation.’ In Science as Practice and Culture, ed. A. Pickering, 168–211. Chicago, IL: University of Chicago Press.
Galison, Peter Louis, and David J. Stump, eds. 1996. The Disunity of Science: Boundaries, Contexts, and Power. Redwood City: Stanford University Press.
Gelman, Andrew. 2010. The Experiments are Fascinating. But Nobody can Repeat Them. New York Times, Nov. 19. https://www.nytimes.com/2018/11/19/science/science-research-fraud-reproducibility.html?searchResultPosition=1. Retrieved 25 June 2020.
Gelman, Andrew. 2016. What has happened down here is the winds have changed. Statistical Modeling, Causal Inference, and Social Science. https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/. Retrieved 29 May 2020
Gerson, Elihu M. 2013. Integration of specialties: An institutional and organizational view. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences. 44(4, Part A): 515–524.
Harris, Richard. 2017. Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions. New York: Basic Books.
Ioannidis, John P.A. 2005. Why Most Published Research Findings Are False. PLOS Medicine 2(8): e124.
Ioannidis, John P.A., Daniele Fanelli, Debbie Drake Dunne, and Steven N. Goodman. 2015. Meta-Research: Evaluation and Improvement of Research Methods and Practices. PLOS Biology 13(10): e1002264.
John, Leslie K., George Loewenstein, and Drazen Prelec. 2012. Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling. Psychological Science 32(5): 524–532.
Kang, Donghyun, and James Evans. 2020. Against Method: Exploding the Boundary Between Qualitative and Quantitative Studies of Science. Quantitative Science Studies 1(3): 930–944.
Kelty, Chris. 2008. Two Bits: The Cultural Significance of Free Software. Durham: Duke University Press.
Knorr Cetina, Karin. 1999. Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press.
Leonelli, Sabina. 2016. Data-centric Biology: A Philosophical Study. Chicago: University of Chicago Press.
Marres, Noortje, and Sarah de Rijcke. 2020. From Indicators to Indicating Interdisciplinarity: A Participatory Mapping Methodology for Research Communities In-The-Making. Quantitative Science Studies 1(3): 1041–1055.
McCormick, Sabrina. 2007. Democratizing Science Movements A New Framework for Mobilization and Contestation. Social Studies of Science 37(4): 609–623.
Merton, Robert K. 1973. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press.
Milojevića, Staša, Cassidy R. Sugimoto, Vincent Larivière, Mike Thelwall, and Ying Ding. 2014. The Role of Handbooks in Knowledge Creation and Diffusion: A Case of Science and Technology Studies. Journal of Informetrics 8(3): 693–709.
Mirowski, Philip. 2018. The Future(s) of Open Science. Social Studies of Science 48(2): 171–203.
Morris, Charles. 1946. The Significance of the Unity of Science Movement. Philosophy and Phenomenological Research 6(4): 508–515.
Mullins, Nicholas C. 1972. The development of a scientific specialty: The phage group and the origins of molecular biology. Minerva 10: 51–82.
Munafò, Marcus R., Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, et al. 2017. A Manifesto for Reproducible Science. Nature Human Behaviour 1(1): 0021.
Munafo, Marcus R., et al. 2015. Using Prediction Markets to Forecast Research Rvaluations. Royal Society: Open Science. 2: 150287.
Nelson, Nicole C. 2020. The Methodologists: A Unique Category of Scientific Actors. Engaging Science, Technology, and Society 6: 20–33.
Nosek, Brian A., and Yoav Bar-Anan. 2012. Scientific Utopia: I. Opening Scientific Communication. Psychological Inquiry 23(3): 217–243.
Nosek, Brian A., Jeffrey R. Spies, and Matt Motyl. 2012. Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science 7(6): 615–31.
Nosek, B.A., et al. 2015. Promoting an Open Research Culture. Science 348(6242): 1422–1425.
Open Science Collaboration. 2015. Estimating the Reproducibility of Psychological Science. Science 349(6251): aac4716.
Panofsky, Aaron. 2014. Misbehaving Science: Controversy and the Development of Behavior Genetics. Chicago: University of Chicago Press.
Penders, Bart, J. Britt Holbrook, and Sarah de Rijcke. 2019. Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing. Publications 7(3): 52.
Penders, Bart, Sarah de Rijcke, and J. Britt Holbrook. 2020. Science’s Moral Economy of Repair: Replication and the Circulation of Reference. Accountability in Research 27(2): 107–113.
Peterson, David. 2017. The Depth of Fields: Managing Focus in the Epistemic Subcultures of Mind and Brain Science. Social Studies of Science 47(1): 53–74.
Peterson, David. 2021. The Replication Crisis Won’t be Solved with Broad Brushstrokes. Nature 594(7862): 151.
Peterson, David, and Aaron Panofsky. 2021. Self-Correction in Science: The Diagnostic and Integrative Motives for Replication. Social Studies of Science 51(4): 583–605.
Prinz, Florian, Thomas Schlange, and Khusru Asadullah. 2011. Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets? Nature Reviews Drug Discovery 10(9): 712–712.
Ribes, David, Andrew S. Hoffman, Steven C. Slota, and Geoffrey C. Bowker. 2019. The Logic of Domains. Social Studies of Science 49(3): 281–309.
Ross-Hellauer, Tony, Benedikt Fecher, Kathleen Shearer, and Eloy Rodrigues. 2019. Pubfair: A Framework for Sustainable, Distributed, Open Science Publishing Services. White paper, version 1.
Saltelli, Andrea, and Silvio Funtowicz. 2017. What Is Science’s Crisis Really about? Futures 91: 5–11.
Schooler, Jonathan W. 2014. Metascience Could Rescue the ‘Replication Crisis.’ Nature 515: 9.
Shapin, Steven. 2010. Never Pure: Historical Studies of Science as If It Was Produced by People with Bodies, Situated in Time, Space, Culture, and Society, and Struggling for Credibility and Authority. Baltimore: JHU Press.
Shapin, Steven, and Simon Schaffer. 2017. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Revised ed. edition. Princeton, NJ: Princeton University Press.
Shiffrin, Richard M., Katy Börner, and Stephen M. Stigler. 2018. Scientific Progress despite Irreproducibility: A Seeming Paradox. PNAS 115(11): 2632–2639.
Smaldino, Paul E., and Richard McElreath. 2016. The Natural Selection of Bad Science. The Royal Society: Open Science 3: 160384.
Stehr, Nico, and Peter Weingart, eds. 2000. Practicing Interdisciplinarity. Toronto: University of Toronto Press.
Timmermans, Stefan, and Iddo Tavory. 2012. Theory Construction in Qualitative Research: From Grounded Theory to Abductive Analysis. Sociological Theory 30(3): 167–186.
Uhlmann, Eric Luis, Charles R. Ebersole, Christopher R. Chartier, Timothy M. Errington, Mallory C. Kidwell, Calvin K. Lai, Randy J. McCarthy, Amy Riegelman, Raphael Silberzahn, and Brian A. Nosek. 2019. Scientific Utopia III: Crowdsourcing Science. Perspectives on Psychological Science 14(5): 711–733.
Vazire, Simine. 2018. Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspectives on Psychological Science 13(4): 411–417.
Yong, Ed. 2016. The Inevitable Evolution of Bad Science: A Simulation Shows how the Incentives of Modern Academia Naturally Select for Weaker and Less Reliable Results. The Atlantic, Sept. 21. https://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/. Retrieved 25 June 2020.
This study was funded by NSF Award (Grant No. 1734683) and Templeton World Charity Foundation.
Conflict of interest
The authors have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Peterson, D., Panofsky, A. Metascience as a Scientific Social Movement. Minerva 61, 147–174 (2023). https://doi.org/10.1007/s11024-023-09490-3