In recent years, serious concerns have arisen about reproducibility in science. Estimates of the cost of irreproducible preclinical studies range from 28 billion USD per year in the USA alone (Freedman et al. in PLoS Biol 13(6):e1002165, 2015) to over 200 billion USD per year worldwide (Chalmers and Glasziou in Lancet 374:86–89, 2009). The situation in the social sciences is not very different: Reproducibility in psychological research, for example, has been estimated to be below 50% as well (Open Science Collaboration in Science 349:6251, 2015). Less well studied is the issue of reproducibility of simulation research. A few replication studies of agent-based models, however, suggest the problem for computational modeling may be more severe than for laboratory experiments (Willensky and Rand in JASSS 10(4):2, 2007; Donkin et al. in Environ Model Softw 92:142–151, 2017; Bajracharya and Duboz in: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11, 2013). In this perspective, we discuss problems of reproducibility in agent-based simulations of life and social science problems, drawing on best practices research in computer science and in wet-lab experiment design and execution to suggest some ways to improve simulation research practice.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
An G, Mi Q, Dutta-Moscato J, Vodovotz Y (2009) Agent-based models in translational systems biology. Wiley Interdiscip Rev Sys Bio Med 1(2):159–171
An G, Fitzpatrick B, Christley S, Federico P, Kanarek A, MillerNeilan R, Oremland M, Salinas R, Lenhart S, Laubenbacher R (2017) Optimization and control of agent-based models in biology: a perspective. Bull Math Biol 79(1):63–87
Axtell R, Axelrod R, Epstein JM, Cohen MD (1996) Aligning simulation models: a case study and results. Comput Math Organ Theory 1:123–141
Bajracharya K, Duboz R (2013) Comparison of three agent-based platforms on the basis of a simple epidemiological model (WIP). In: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11
Beck K (2003) Test-Driven Development: By Example. Pearson, Boston
Begley CG, Ellis L (2012) Drug development: raise standards for preclinical research. Nature 483:531–533
Chalmers I, Glasziou P (2009) Avoidable waste in the production and reporting of research evidence. Lancet 374:86–89
Donkin E, Dennis P, Ustalakov A, Warren J, Clare A (2017) Replicating complex agent based models, a formidable task. Environ Model Softw, 92:142–151
Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of reproducibility. PLoS Biol 13(6):e1002165. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165. Accessed 10 June 2015
Grimm V, Revilla E, Berger U, Jeltsch F, Mooij WM, Railsback SF, Weiner J, Wiegand T, DeAngelis DL (2005) Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science 310:987–991
Grimm V, Berger U, DeAngelis DL, Polhill JG, Giskee J, Railsback SF (2010) The ODD protocol: a review and first update. Ecol Model 221:2760–2768
Madeyski L (2010) Test-driven development: an empirical evaluation of agile practice. Springer, Heidelberg
Mäkinen S, Munch J (2014) Effects of test-driven development: a comparative analysis of empirical studies. In: Winkler D, Bifll S, Bergsmann J (eds) Software quality: model-based approaches for advanced software and systems engineering. Springer, Cham
Martin RC (2008) Clean code: a handbook of agile software craftmanship. Pearson, Boston
North M, Macal C (2007) Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. Oxford University Press, Oxford
Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349:6251
Prinz F, Schlange T, Asadullah K (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10:712–713
Railsback S, Grimm V (2012) Agent-based and individual-based modeling: a practical introduction. Princeton University Press, Princeton
Santer TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer, New York
Schnell S (2015) Ten simple rules for a computational biologist’s laboratory notebook. PLoS Comput Biol 11(9):e1004385. https://doi.org/10.1371/journal.pcbi.1004385
Smith, R. (2017). Personal communication
Wilenksy U, Rand W (2007) Making models match: replicating an agent-based model. JASSS 10(4):2
This work was partially supported by National Institute on Drug Abuse grant 1R43DA041760-01.
About this article
Cite this article
Fitzpatrick, B.G. Issues in Reproducible Simulation Research. Bull Math Biol 81, 1–6 (2019). https://doi.org/10.1007/s11538-018-0496-1
- Agent-based models
- Simulation reproducibility
- Test-driven development
- Version control
- Computational lab notebook