Skip to main content
SpringerLink
Log in

Metascience as a Scientific Social Movement

Minerva volume 61, pages 147–174 (2023)Cite this article

Abstract

The “reproducibility crisis” has been one of the most significant stories in science in the past 15 years and has led to significant policy changes across the research landscape. Yet, scandals, irreproducible studies, and cries of crisis have occurred for decades in science. This article seeks to explain why the reproducibility crisis has taken root and become a force in science policy in ways previous crises have not. In short, we argue that it was through the scientific, institutional, and cultural efforts of a group of scientific activists we are calling metascientists. Metascience is a scientific social movement that seeks to use quantification and experimentation to diagnose problems in research practice and improve efficiency. It draws together data scientists, experimental and statistical methodologists, and open science activists into a project with both intellectual and policy dimensions. Metascientists have been remarkably successful at winning grants, motivating news coverage, and changing policies at science agencies, journals, and universities. The social movement lens is useful for understanding the popularization and impact of the reproducibility crisis narrative and suggests ways the institutions of science are adapting to meet a changing political and technological landscape.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1

Notes

  1. Because many of the groups are known by acronyms, the following is a reference list:

    COAR- Confederation of Open Access Repositories

    COS- Center for Open Science

    DARPA- Defense Advanced Research Projects Agency

    METRICS- Meta-research Innovation Center at Stanford

    SoS- Science of science

    SPARC- Scholarly Publishing and Academic Resources Coalition

  2. Penders et al. (2020) suggest a similar typology.

  3. “Open software” has also played a role in metascience, but mainly through the data science/science of science pathway. Specifically, the focus on the openness of code, decentralization, and modifiability of systems that one finds in metascience discussions (e.g., Nosek and Bar-Anan 2012; Ross-Hellauer et al. 2019; Uhlmann et al. 2019) can be traced to the utopian rhetoric of early open software activists (Kelty 2008).

  4. There are, it should be noted, attempts to create some productive dialogue between the quantitative and qualitative traditions of science studies (Bowker 2020; Marres and de Rijcke 2020; Milojevića et al. 2014), some even involving avowed metascientists (e.g., Kang and Evans 2020). However, both self-consciously positioning themselves in reference to a replication crisis and, as we show later, intentionally keeping theoretical complexity at arm’s distance have allowed the field to gain a position in policy that has largely eluded scientometrics and, certainly, qualitative studies of science.

  5. At the Metascience Symposium, DARPA program manager Adam Russell explained that his agency was tasked with synthesizing research to provide guidance for military and political decisions. However, reproducibility problems and the general low quality of scientific research, especially in the social sciences, resulted in a deep skepticism of the published literature. To counter that, the agency has poured tens of millions of dollars into a spate of metascience projects designed to improve the science through a combination of open science initiatives and machine learning.

    “Systematizing Confidence in Open Research and Evidence” is an attempt to create an automated system to evaluate behavioral and social science research. “Big Mechanism” seeks to use machine learning to read the academic literature in cancer and develop novel causal models. And, “Automating Scientific Knowledge Extraction” is trying to use artificial intelligence to automatically collect information, extract its useful components, integrate this information into evolving models with the goal of eventually “generating […] machine-generated hypotheses.”

    These projects make it clear that the ultimate goal, for at least some metascientists, is not merely the improvement of existing disciplines, but a radical transformation in which disciplinary science feeds into evolving artificial intelligence systems which evaluate, synthesize, and even conduct research.

  6. Historically, the same tension has existed between “domain” experts and the supposedly “domain-free” expertise of software engineers who have attempted to develop automated systems in expert domains (Ribes et al. 2019).

References

  • Adut, Ari. 2008. On Scandal: Moral Disturbances in Society, Politics, and Art. Cambridge: Cambridge University Press.

    Google Scholar 

  • Angrist, Joshua D, and Pischke, Jorn-Steffen. 2010. The credibility revolution in empirical economics: How better research design is taking the con out of economics. The Journal of Economic Perspectives, 24(2): 3–30.

  • Arribas-Ayllon, Michael, Andrew Bartlett, and Jamie Lewis. 2019. Psychiatric Genetics: From Hereditary Madness to Big Biology. London: Routledge.

    Book  Google Scholar 

  • Baker, Monya. 2016. Is There a Reproducibility Crisis? Nature 533: 452–454.

    Article  Google Scholar 

  • Becker, Howard S. 2008. Outsiders. New York: Simon and Schuster.

    Google Scholar 

  • Begley, C. Glenn, and John P.A. Ioannidis. 2015. Reproducibility in science. Circulation Research 116(1): 116–126. https://doi.org/10.1161/CIRCRESAHA.114.303819.

    Article  Google Scholar 

  • Begley, C. Glenn, and Lee M. Ellis. 2012. Drug Development: Raise Standards for Preclinical Cancer Research. Nature 483(7391): 531–533.

    Article  Google Scholar 

  • Belluz, Julia. 2017. Cancer Scientists are Having Trouble Replicating Groundbreaking Research. Vox, Jan. 23. https://www.vox.com/science-and-health/2017/1/23/14324326/replication-science-is-hard. Retrieved 25 June 2020.

  • Bohlin, Ingemar. 2004. Communication Regimes in Competition: The Current Transition in Scholarly Communication Seen Through the Lens of the Sociology of Technology. Social Studies of Science 34(3): 356–391.

    Article  Google Scholar 

  • Bourdieu, Pierre. 2004. Science of Science and Reflexivity. Cambridge: Polity.

    Google Scholar 

  • Bowker, Geoffrey C. 2020. Number or No Numbers in Science Studies. Quantitative Science Studies 1(3): 927–929.

    Article  Google Scholar 

  • Brown, Phil, Stephen Zavestoski, Sabrina McCormick, Brian Mayer, Rachel Morello-Frosch, and Rebecca Gasior Altman. 2004. Embodied Health Movements: New Approaches to Social Movements in Health. Sociology of Health & Illness 26(1): 50–80.

    Article  Google Scholar 

  • Bunge, Mario. 1959. Metascientific Inquiries. Springfield: C. C. Thomas.

    Google Scholar 

  • Chivers, Tom. 2017. These People are Trying to Fix a Huge Problem in Science. Buzzfeed News, Jul. 29. https://www.buzzfeed.com/tomchivers/signifying-nothing. Retrieved 25 June 2020.

  • Collins, Harry M. 1981. The Place of the Core-Set in Modern Science: Social Contingency with Methodological Propriety. History of Science 19(1): 6–19.

    Article  Google Scholar 

  • Collins, Harry M. 2019. Forms of Life: The Method and Meaning of Sociology. Cambridge: MIT Press.

    Google Scholar 

  • COS. 2017. Center for Open Science: Strategic Plan. https://osf.io/x2w9h/. Retrieved 28 June 2020.

  • De Solla Price, Derek. 1961. Science Since Babylon. New Haven: Yale University Press.

    Google Scholar 

  • Deterding, Nicole M., and Mary C. Waters. 2018. Flexible Coding of In-depth Interviews: A Twenty-first-century Approach. Sociological Methods and Research. https://doi.org/10.1177/0049124118799377.

    Article  Google Scholar 

  • Devezer, Berna, Luis G. Nardin, Bert Baumgaertner, and Erkan Ozge Buzbas. 2019. Scientific Discovery in a Model-Centric Framework: Reproducibility, Innovation, and Epistemic Diversity. PLoS ONE 14(5): e0216125.

    Article  Google Scholar 

  • Epstein, Steven. 2008. Inclusion: The Politics of Difference in Medical Research. Chicago, IL: University of Chicago Press.

    Google Scholar 

  • Fanelli, Daniele. 2009. How many scientists fabricate and falisify research? A systematic review and meta-analyses of survey data. PLoS ONE 4(5): e5738.

    Article  Google Scholar 

  • Fanelli, Daniele. 2019. Is Science Really Facing a Reproducibility Crisis, and Do We Need It To? PNAS 115(11): 2628–2631.

    Article  Google Scholar 

  • Faye, Cathy. 2012. American social psychology: Examining the contours of the 1970s crisis. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences 43(2): 514–521.

  • Fieldler, Klaus, and Norbert Schwarz. 2016. Questionable Research Practices Revisited. Social Psychological and Personality Science 7(1): 45–52.

    Article  Google Scholar 

  • Fleck, Ludwik. 1960 (1986). Crisis in Science. In Cognition and Fact: Materials on Ludwik Fleck, eds. R. S. Cohen and T. Schnelle, 153–158. Dordrecht: D. Reidel Publishing Company.

  • Flis, Ivan. 2019. Psychologists Psychologizing Scientific Psychology: An Epistemological Reading of the Replication Crisis. Theory & Psychology 29(2): 158–181.

    Article  Google Scholar 

  • Feest, Uljana. 2019. Why Replication Is Overrated. Philosophy of Science 86(5): 895–905.

    Article  Google Scholar 

  • Fortunato, Santo, Carl T. Bergstrom, Katy Börner, James A. Evans, Dirk Helbing, Staša Milojević, Alexander M. Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, Alessandro Vespignani, Ludo Waltman, Dashun Wang, and Albert-László Barabási. 2018. Science of Science. Science 359(6379): eaao0185.

    Article  Google Scholar 

  • Freese, Jeremy, and David Peterson. 2018. The Emergence of Statistical Objectivity: Changing Ideas of Epistemic Vice and Virtue in Science. Sociological Theory 36(3): 289–313.

    Article  Google Scholar 

  • Frickel, Scott, and Neil Gross. 2005. A General Theory of Scientific/Intellectual Movements. American Sociological Review 70(2): 204–232.

    Article  Google Scholar 

  • Fujimura, Joan H. 1992. Crafting Science: Standardized Packages, Boundary Objects, and ‘Translation.’ In Science as Practice and Culture, ed. A. Pickering, 168–211. Chicago, IL: University of Chicago Press.

    Google Scholar 

  • Galison, Peter Louis, and David J. Stump, eds. 1996. The Disunity of Science: Boundaries, Contexts, and Power. Redwood City: Stanford University Press.

    Google Scholar 

  • Gelman, Andrew. 2010. The Experiments are Fascinating. But Nobody can Repeat Them. New York Times, Nov. 19. https://www.nytimes.com/2018/11/19/science/science-research-fraud-reproducibility.html?searchResultPosition=1. Retrieved 25 June 2020.

  • Gelman, Andrew. 2016. What has happened down here is the winds have changed. Statistical Modeling, Causal Inference, and Social Science. https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/. Retrieved 29 May 2020

  • Gerson, Elihu M. 2013. Integration of specialties: An institutional and organizational view. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences. 44(4, Part A): 515–524.

    Article  Google Scholar 

  • Harris, Richard. 2017. Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions. New York: Basic Books.

    Google Scholar 

  • Ioannidis, John P.A. 2005. Why Most Published Research Findings Are False. PLOS Medicine 2(8): e124.

    Article  Google Scholar 

  • Ioannidis, John P.A., Daniele Fanelli, Debbie Drake Dunne, and Steven N. Goodman. 2015. Meta-Research: Evaluation and Improvement of Research Methods and Practices. PLOS Biology 13(10): e1002264.

    Article  Google Scholar 

  • John, Leslie K., George Loewenstein, and Drazen Prelec. 2012. Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling. Psychological Science 32(5): 524–532.

    Article  Google Scholar 

  • Kang, Donghyun, and James Evans. 2020. Against Method: Exploding the Boundary Between Qualitative and Quantitative Studies of Science. Quantitative Science Studies 1(3): 930–944.

    Article  Google Scholar 

  • Kelty, Chris. 2008. Two Bits: The Cultural Significance of Free Software. Durham: Duke University Press.

    Book  Google Scholar 

  • Knorr Cetina, Karin. 1999. Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press.

    Book  Google Scholar 

  • Leonelli, Sabina. 2016. Data-centric Biology: A Philosophical Study. Chicago: University of Chicago Press.

    Book  Google Scholar 

  • Marres, Noortje, and Sarah de Rijcke. 2020. From Indicators to Indicating Interdisciplinarity: A Participatory Mapping Methodology for Research Communities In-The-Making. Quantitative Science Studies 1(3): 1041–1055.

    Article  Google Scholar 

  • McCormick, Sabrina. 2007. Democratizing Science Movements A New Framework for Mobilization and Contestation. Social Studies of Science 37(4): 609–623.

    Article  Google Scholar 

  • Merton, Robert K. 1973. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press.

    Google Scholar 

  • Milojevića, Staša, Cassidy R. Sugimoto, Vincent Larivière, Mike Thelwall, and Ying Ding. 2014. The Role of Handbooks in Knowledge Creation and Diffusion: A Case of Science and Technology Studies. Journal of Informetrics 8(3): 693–709.

    Article  Google Scholar 

  • Mirowski, Philip. 2018. The Future(s) of Open Science. Social Studies of Science 48(2): 171–203.

    Article  Google Scholar 

  • Morris, Charles. 1946. The Significance of the Unity of Science Movement. Philosophy and Phenomenological Research 6(4): 508–515.

    Article  Google Scholar 

  • Mullins, Nicholas C. 1972. The development of a scientific specialty: The phage group and the origins of molecular biology. Minerva 10: 51–82.

    Article  Google Scholar 

  • Munafò, Marcus R., Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, et al. 2017. A Manifesto for Reproducible Science. Nature Human Behaviour 1(1): 0021.

    Article  Google Scholar 

  • Munafo, Marcus R., et al. 2015. Using Prediction Markets to Forecast Research Rvaluations. Royal Society: Open Science. 2: 150287.

    Google Scholar 

  • Nelson, Nicole C. 2020. The Methodologists: A Unique Category of Scientific Actors. Engaging Science, Technology, and Society 6: 20–33.

    Article  Google Scholar 

  • Nosek, Brian A., and Yoav Bar-Anan. 2012. Scientific Utopia: I. Opening Scientific Communication. Psychological Inquiry 23(3): 217–243.

    Article  Google Scholar 

  • Nosek, Brian A., Jeffrey R. Spies, and Matt Motyl. 2012. Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science 7(6): 615–31.

    Article  Google Scholar 

  • Nosek, B.A., et al. 2015. Promoting an Open Research Culture. Science 348(6242): 1422–1425.

    Article  Google Scholar 

  • Open Science Collaboration. 2015. Estimating the Reproducibility of Psychological Science. Science 349(6251): aac4716.

    Article  Google Scholar 

  • Panofsky, Aaron. 2014. Misbehaving Science: Controversy and the Development of Behavior Genetics. Chicago: University of Chicago Press.

    Book  Google Scholar 

  • Penders, Bart, J. Britt Holbrook, and Sarah de Rijcke. 2019. Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing. Publications 7(3): 52.

    Article  Google Scholar 

  • Penders, Bart, Sarah de Rijcke, and J. Britt Holbrook. 2020. Science’s Moral Economy of Repair: Replication and the Circulation of Reference. Accountability in Research 27(2): 107–113.

    Article  Google Scholar 

  • Peterson, David. 2017. The Depth of Fields: Managing Focus in the Epistemic Subcultures of Mind and Brain Science. Social Studies of Science 47(1): 53–74.

    Article  Google Scholar 

  • Peterson, David. 2021. The Replication Crisis Won’t be Solved with Broad Brushstrokes. Nature 594(7862): 151.

    Article  Google Scholar 

  • Peterson, David, and Aaron Panofsky. 2021. Self-Correction in Science: The Diagnostic and Integrative Motives for Replication. Social Studies of Science 51(4): 583–605.

    Article  Google Scholar 

  • Prinz, Florian, Thomas Schlange, and Khusru Asadullah. 2011. Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets? Nature Reviews Drug Discovery 10(9): 712–712.

    Article  Google Scholar 

  • Ribes, David, Andrew S. Hoffman, Steven C. Slota, and Geoffrey C. Bowker. 2019. The Logic of Domains. Social Studies of Science 49(3): 281–309.

    Article  Google Scholar 

  • Ross-Hellauer, Tony, Benedikt Fecher, Kathleen Shearer, and Eloy Rodrigues. 2019. Pubfair: A Framework for Sustainable, Distributed, Open Science Publishing Services. White paper, version 1.

  • Saltelli, Andrea, and Silvio Funtowicz. 2017. What Is Science’s Crisis Really about? Futures 91: 5–11.

    Article  Google Scholar 

  • Schooler, Jonathan W. 2014. Metascience Could Rescue the ‘Replication Crisis.’ Nature 515: 9.

    Article  Google Scholar 

  • Shapin, Steven. 2010. Never Pure: Historical Studies of Science as If It Was Produced by People with Bodies, Situated in Time, Space, Culture, and Society, and Struggling for Credibility and Authority. Baltimore: JHU Press.

    Book  Google Scholar 

  • Shapin, Steven, and Simon Schaffer. 2017. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Revised ed. edition. Princeton, NJ: Princeton University Press.

  • Shiffrin, Richard M., Katy Börner, and Stephen M. Stigler. 2018. Scientific Progress despite Irreproducibility: A Seeming Paradox. PNAS 115(11): 2632–2639.

    Article  Google Scholar 

  • Smaldino, Paul E., and Richard McElreath. 2016. The Natural Selection of Bad Science. The Royal Society: Open Science 3: 160384.

    Google Scholar 

  • Stehr, Nico, and Peter Weingart, eds. 2000. Practicing Interdisciplinarity. Toronto: University of Toronto Press.

    Google Scholar 

  • Timmermans, Stefan, and Iddo Tavory. 2012. Theory Construction in Qualitative Research: From Grounded Theory to Abductive Analysis. Sociological Theory 30(3): 167–186.

    Article  Google Scholar 

  • Uhlmann, Eric Luis, Charles R. Ebersole, Christopher R. Chartier, Timothy M. Errington, Mallory C. Kidwell, Calvin K. Lai, Randy J. McCarthy, Amy Riegelman, Raphael Silberzahn, and Brian A. Nosek. 2019. Scientific Utopia III: Crowdsourcing Science. Perspectives on Psychological Science 14(5): 711–733.

    Article  Google Scholar 

  • Vazire, Simine. 2018. Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspectives on Psychological Science 13(4): 411–417.

    Article  Google Scholar 

  • Yong, Ed. 2016. The Inevitable Evolution of Bad Science: A Simulation Shows how the Incentives of Modern Academia Naturally Select for Weaker and Less Reliable Results. The Atlantic, Sept. 21. https://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/. Retrieved 25 June 2020.

Download references

Funding

This study was funded by NSF Award (Grant No. 1734683) and Templeton World Charity Foundation.

Author information

Authors and Affiliations

  1. Department of Sociology, Stone Hall, Purdue University, 700 State Street, West Lafayette, IN, 47907-2059, USA

    David Peterson

  2. Institute for Society and Genetics, University of California, Los Angeles, Life Sciences Building 3323D, Los Angeles, CA, 90095-7221, USA

    Aaron Panofsky

Authors
  1. David Peterson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aaron Panofsky
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors contributed to data collection and writing. The talks cited are available on https://www.metascience2019.org/. The interviews we collected are not publicly available. The study was ruled exempt from our IRB.

Corresponding author

Correspondence to David Peterson.

Ethics declarations

Conflict of interest

The authors have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peterson, D., Panofsky, A. Metascience as a Scientific Social Movement. Minerva 61, 147–174 (2023). https://doi.org/10.1007/s11024-023-09490-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11024-023-09490-3

Keywords

search

Navigation