Bulletin of Mathematical Biology

, Volume 81, Issue 1, pp 1–6 | Cite as

Issues in Reproducible Simulation Research

  • B. G. FitzpatrickEmail author
Perspectives Article
Part of the following topical collections:
  1. Reproducibility in Computational Biology Regular


In recent years, serious concerns have arisen about reproducibility in science. Estimates of the cost of irreproducible preclinical studies range from 28 billion USD per year in the USA alone (Freedman et al. in PLoS Biol 13(6):e1002165, 2015) to over 200 billion USD per year worldwide (Chalmers and Glasziou in Lancet 374:86–89, 2009). The situation in the social sciences is not very different: Reproducibility in psychological research, for example, has been estimated to be below 50% as well (Open Science Collaboration in Science 349:6251, 2015). Less well studied is the issue of reproducibility of simulation research. A few replication studies of agent-based models, however, suggest the problem for computational modeling may be more severe than for laboratory experiments (Willensky and Rand in JASSS 10(4):2, 2007; Donkin et al. in Environ Model Softw 92:142–151, 2017; Bajracharya and Duboz in: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11, 2013). In this perspective, we discuss problems of reproducibility in agent-based simulations of life and social science problems, drawing on best practices research in computer science and in wet-lab experiment design and execution to suggest some ways to improve simulation research practice.


Agent-based models Simulation reproducibility Validation Test-driven development Version control Computational lab notebook 



This work was partially supported by National Institute on Drug Abuse grant 1R43DA041760-01.


  1. An G, Mi Q, Dutta-Moscato J, Vodovotz Y (2009) Agent-based models in translational systems biology. Wiley Interdiscip Rev Sys Bio Med 1(2):159–171CrossRefGoogle Scholar
  2. An G, Fitzpatrick B, Christley S, Federico P, Kanarek A, MillerNeilan R, Oremland M, Salinas R, Lenhart S, Laubenbacher R (2017) Optimization and control of agent-based models in biology: a perspective. Bull Math Biol 79(1):63–87MathSciNetCrossRefGoogle Scholar
  3. Axtell R, Axelrod R, Epstein JM, Cohen MD (1996) Aligning simulation models: a case study and results. Comput Math Organ Theory 1:123–141CrossRefGoogle Scholar
  4. Bajracharya K, Duboz R (2013) Comparison of three agent-based platforms on the basis of a simple epidemiological model (WIP). In: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11Google Scholar
  5. Beck K (2003) Test-Driven Development: By Example. Pearson, BostonGoogle Scholar
  6. Begley CG, Ellis L (2012) Drug development: raise standards for preclinical research. Nature 483:531–533CrossRefGoogle Scholar
  7. Chalmers I, Glasziou P (2009) Avoidable waste in the production and reporting of research evidence. Lancet 374:86–89CrossRefGoogle Scholar
  8. Donkin E, Dennis P, Ustalakov A, Warren J, Clare A (2017) Replicating complex agent based models, a formidable task. Environ Model Softw, 92:142–151CrossRefGoogle Scholar
  9. Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of reproducibility. PLoS Biol 13(6):e1002165. Accessed 10 June 2015
  10. Grimm V, Revilla E, Berger U, Jeltsch F, Mooij WM, Railsback SF, Weiner J, Wiegand T, DeAngelis DL (2005) Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science 310:987–991CrossRefGoogle Scholar
  11. Grimm V, Berger U, DeAngelis DL, Polhill JG, Giskee J, Railsback SF (2010) The ODD protocol: a review and first update. Ecol Model 221:2760–2768CrossRefGoogle Scholar
  12. Madeyski L (2010) Test-driven development: an empirical evaluation of agile practice. Springer, HeidelbergCrossRefGoogle Scholar
  13. Mäkinen S, Munch J (2014) Effects of test-driven development: a comparative analysis of empirical studies. In: Winkler D, Bifll S, Bergsmann J (eds) Software quality: model-based approaches for advanced software and systems engineering. Springer, ChamGoogle Scholar
  14. Martin RC (2008) Clean code: a handbook of agile software craftmanship. Pearson, BostonGoogle Scholar
  15. North M, Macal C (2007) Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. Oxford University Press, OxfordCrossRefGoogle Scholar
  16. Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349:6251CrossRefGoogle Scholar
  17. Prinz F, Schlange T, Asadullah K (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10:712–713CrossRefGoogle Scholar
  18. Railsback S, Grimm V (2012) Agent-based and individual-based modeling: a practical introduction. Princeton University Press, PrincetonzbMATHGoogle Scholar
  19. Santer TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer, New YorkCrossRefGoogle Scholar
  20. Schnell S (2015) Ten simple rules for a computational biologist’s laboratory notebook. PLoS Comput Biol 11(9):e1004385. CrossRefGoogle Scholar
  21. Smith, R. (2017). Personal communicationGoogle Scholar
  22. Wilenksy U, Rand W (2007) Making models match: replicating an agent-based model. JASSS 10(4):2Google Scholar

Copyright information

© Society for Mathematical Biology 2018

Authors and Affiliations

  1. 1.Tempest Technologies and Department of MathematicsLoyola Marymount UniversityLos AngelesUSA

Personalised recommendations