Issues in Reproducible Simulation Research


In recent years, serious concerns have arisen about reproducibility in science. Estimates of the cost of irreproducible preclinical studies range from 28 billion USD per year in the USA alone (Freedman et al. in PLoS Biol 13(6):e1002165, 2015) to over 200 billion USD per year worldwide (Chalmers and Glasziou in Lancet 374:86–89, 2009). The situation in the social sciences is not very different: Reproducibility in psychological research, for example, has been estimated to be below 50% as well (Open Science Collaboration in Science 349:6251, 2015). Less well studied is the issue of reproducibility of simulation research. A few replication studies of agent-based models, however, suggest the problem for computational modeling may be more severe than for laboratory experiments (Willensky and Rand in JASSS 10(4):2, 2007; Donkin et al. in Environ Model Softw 92:142–151, 2017; Bajracharya and Duboz in: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11, 2013). In this perspective, we discuss problems of reproducibility in agent-based simulations of life and social science problems, drawing on best practices research in computer science and in wet-lab experiment design and execution to suggest some ways to improve simulation research practice.

This is a preview of subscription content, log in to check access.


  1. An G, Mi Q, Dutta-Moscato J, Vodovotz Y (2009) Agent-based models in translational systems biology. Wiley Interdiscip Rev Sys Bio Med 1(2):159–171

    Article  Google Scholar 

  2. An G, Fitzpatrick B, Christley S, Federico P, Kanarek A, MillerNeilan R, Oremland M, Salinas R, Lenhart S, Laubenbacher R (2017) Optimization and control of agent-based models in biology: a perspective. Bull Math Biol 79(1):63–87

    MathSciNet  Article  Google Scholar 

  3. Axtell R, Axelrod R, Epstein JM, Cohen MD (1996) Aligning simulation models: a case study and results. Comput Math Organ Theory 1:123–141

    Article  Google Scholar 

  4. Bajracharya K, Duboz R (2013) Comparison of three agent-based platforms on the basis of a simple epidemiological model (WIP). In: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11

  5. Beck K (2003) Test-Driven Development: By Example. Pearson, Boston

    Google Scholar 

  6. Begley CG, Ellis L (2012) Drug development: raise standards for preclinical research. Nature 483:531–533

    Article  Google Scholar 

  7. Chalmers I, Glasziou P (2009) Avoidable waste in the production and reporting of research evidence. Lancet 374:86–89

    Article  Google Scholar 

  8. Donkin E, Dennis P, Ustalakov A, Warren J, Clare A (2017) Replicating complex agent based models, a formidable task. Environ Model Softw, 92:142–151

    Article  Google Scholar 

  9. Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of reproducibility. PLoS Biol 13(6):e1002165. Accessed 10 June 2015

  10. Grimm V, Revilla E, Berger U, Jeltsch F, Mooij WM, Railsback SF, Weiner J, Wiegand T, DeAngelis DL (2005) Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science 310:987–991

    Article  Google Scholar 

  11. Grimm V, Berger U, DeAngelis DL, Polhill JG, Giskee J, Railsback SF (2010) The ODD protocol: a review and first update. Ecol Model 221:2760–2768

    Article  Google Scholar 

  12. Madeyski L (2010) Test-driven development: an empirical evaluation of agile practice. Springer, Heidelberg

    Google Scholar 

  13. Mäkinen S, Munch J (2014) Effects of test-driven development: a comparative analysis of empirical studies. In: Winkler D, Bifll S, Bergsmann J (eds) Software quality: model-based approaches for advanced software and systems engineering. Springer, Cham

    Google Scholar 

  14. Martin RC (2008) Clean code: a handbook of agile software craftmanship. Pearson, Boston

    Google Scholar 

  15. North M, Macal C (2007) Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. Oxford University Press, Oxford

    Google Scholar 

  16. Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349:6251

    Article  Google Scholar 

  17. Prinz F, Schlange T, Asadullah K (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10:712–713

    Article  Google Scholar 

  18. Railsback S, Grimm V (2012) Agent-based and individual-based modeling: a practical introduction. Princeton University Press, Princeton

    Google Scholar 

  19. Santer TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer, New York

    Google Scholar 

  20. Schnell S (2015) Ten simple rules for a computational biologist’s laboratory notebook. PLoS Comput Biol 11(9):e1004385.

    Article  Google Scholar 

  21. Smith, R. (2017). Personal communication

  22. Wilenksy U, Rand W (2007) Making models match: replicating an agent-based model. JASSS 10(4):2

    Google Scholar 

Download references


This work was partially supported by National Institute on Drug Abuse grant 1R43DA041760-01.

Author information



Corresponding author

Correspondence to B. G. Fitzpatrick.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fitzpatrick, B.G. Issues in Reproducible Simulation Research. Bull Math Biol 81, 1–6 (2019).

Download citation


  • Agent-based models
  • Simulation reproducibility
  • Validation
  • Test-driven development
  • Version control
  • Computational lab notebook