Valid and Reproducible Simulation Studies—Making It Explicit

  • Oliver ReinhardtEmail author
  • Tom Warnke
  • Andreas Ruscheinski
  • Adelinde M. Uhrmacher
Part of the Simulation Foundations, Methods and Applications book series (SFMA)


The validation of complex simulation models is a challenging task. To increase the trust into the model, diverse simulation experiments are executed to explore the behavior of the model and to check its plausibility. Thus, these simulation experiments present an important information about the validity of the model, similarly as the data used for calibration, as input for the model, and for testing its predictiveness. Simulation models are rarely developed from scratch but by reusing existing models, e.g., by extending or composing them, or for cross-validation. These models and their validity provide further details about the validity of a model. Thus, a multitude of artifacts contribute intricately related to the final simulation model and our “gut feelings” about it. To make these artifacts and their relations explicit and accessible, we will apply a declarative formal modeling language, a declarative language for specifying and executing diverse simulation experiments, and a provenance model to relate the diverse artifacts in telling the validation tale of an agent-based migration model.


Validation Multilevel modeling Demography Provenance 



This research was supported by the German Research Foundation (DFG) via research grant UH-66/15 and research grant UH-66/18.


  1. Agha, G., & Palmskog, K. (2018). A survey of statistical model checking. ACM Transactions on Modeling and Computer Simulation, 28(1), 6:1–6:39.Google Scholar
  2. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211.CrossRefGoogle Scholar
  3. Balci, O. (1997). Verification, validation and accreditation of simulation models. In Proceedings of the 1997 winter simulation conference, pp. 135–141. IEEE.Google Scholar
  4. Bandourian, R., McDonald, J., & Turley, R. S. (2002). A comparison of parametric models of income distribution across countries and over time. SSRN Scholarly Paper ID 324900, Social Science Research Network, Rochester, NY, June 2002.Google Scholar
  5. Beauchemin, C. (2015). Migration between Africa and Europe (MAFE): Looking beyond immigration to understand international migration. Population, 70(1), 13–38.CrossRefGoogle Scholar
  6. Bergmann, F. T., Cooper, J., Le Novere, N., Nickerson, D., & Waltemath, D. (2015). Simulation experiment description markup language (SED-ML) Level 1 Version 2. Journal of Integrative Bioinformatics (JIB), 12(2), 119–212.Google Scholar
  7. Bochner, C., Gude, R.,& Schreiber, A. (2008). A python library for provenance recording and querying. Provenance and Annotation of Data and Processes, pp. 229–240.Google Scholar
  8. Coale, A. J., & Mcneil, D. R. (1972). The distribution by age of the frequency of first marriage in a female cohort. Journal of the American Statistical Association, 67(340), 743–749.CrossRefGoogle Scholar
  9. Conte, R., et al. (2012). Manifesto of computational social science. European Physical Journal-Special Topics, 214, 325.Google Scholar
  10. Davies, J. B., Sandström, S., Shorrocks, A. B., & Wolff, E. N. (2011). The level and distribution of global household wealth. The Economic Journal, 121(551), 223–254.CrossRefGoogle Scholar
  11. De Nies, T., et al. (2013). Git2PROV: Exposing version control system content as w3c prov. In Proceedings of the 2013th International Conference on Posters & Demonstrations Track-Volume 1035 (pp. 125–128). CEUR-WS. org.Google Scholar
  12. Ewald, R., & Uhrmacher, A. M. (2014). SESSL: A domain-specific language for simulation experiments. ACM Transactions on Modeling and Computer Simulation, 24(2), 11:1–11:25.Google Scholar
  13. Gray, J., Hilton, J., & Bijak, J. (2017). Choosing the choice: Reflections on modelling decisions and behaviour in demographic agent-based models. Population Studies, 71(sup1), 85–97Google Scholar
  14. Grimm, V., et al. (2006). A standard protocol for describing individual-based and agent-based models. Ecological Modelling, 198(1), 115–126.Google Scholar
  15. Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760–2768.CrossRefGoogle Scholar
  16. Groth, P.,& Moreau, L. (2013). PROV-overview. An overview of the PROV family of documents.Google Scholar
  17. Harris, L. A., et al. (2016). Bionetgen 2.2: Advances in rule-based modeling. Bioinformatics, 32(21), 3366–3368.CrossRefGoogle Scholar
  18. Heligman, L., & Pollard, J. H. (1980). The age pattern of mortality. Journal of the Institute of Actuaries, 107(1), 49–80.CrossRefGoogle Scholar
  19. Helms, T., Himmelspach, J., Maus, C., Röwer, O., Schützel, J., & Uhrmacher, A. M. (2012). Toward a language for the flexible observation of simulations. In Proceedings of the 2012 Winter Simulation Conference, pp. 418:1–418:12. IEEE.Google Scholar
  20. IEEE. (2009). IEEE Standard VHDL Language Reference Manual. IEEE Std 1076-2008 (Revision of IEEE Std 1076-2002) (pp. c1–626).Google Scholar
  21. Klabunde, A., Willekens, F., Zinn, S., & Leuchter, M. (2015). An agent-based decision model of migration, embedded in the life courseModel description in ODD+D format. Max Planck Institute for Demographic Research, Rostock, Germany: Technical report.Google Scholar
  22. Klabunde, A., Zinn, S., Willekens, F., & Leuchter, M. (2016). Multistate modeling extended by behavioral rules (Version 6).
  23. Klabunde, A., Zinn, S., Willekens, F., & Leuchter, M. (2017). Multistate modeling extended by behavioral rules—an example of migration. Population Studies, 71(sup1), 51–67Google Scholar
  24. Klügl, F. (2008). A validation methodology for agent-based simulations. In Proceedings of the 2008 ACM Symposium on Applied Computing, SAC’08 (pp. 39–43). New York: ACM.Google Scholar
  25. Kossow, C., Helms, T., Kreutzer, J. M., Martens, A., & Uhrmacher, A. M. (2016) Evaluating different modeling languages based on a user study. In Proceedings of the 49th Annual Simulation Symposium, ANSS ’16 (pp. 18:1–18:8). Society for Computer Simulation International, San Diego, CA, USA.Google Scholar
  26. Laurent, J., Medina-Abarca, H. F., Boutillier, P., Yang, J.,& Fontana, W. (2018). A trace query language for rule-based models. In Computational Methods in Systems Biology (CMSB 2018), Lecture Notes in Bioinformatics. Cham: Springer.Google Scholar
  27. Leye, S., Himmelspach, J., Uhrmacher, A. M. (2009). A discussion on experimental model validation. In 2009 11th International Conference on Computer Modelling and Simulation (pp. 161–167).Google Scholar
  28. Lim, C., Lu, S., Chebotko, A.,& Fotouhi, F. (2011). OPQL: A first OPM-level query language for scientific workflow provenance. In 2011 IEEE International Conference on Services Computing, pp. 136–143. IEEE.Google Scholar
  29. Lukasiewycz, M., Glaß, M., Reimann, F.,& Teich, J. (2011). Opt4J—a modular framework for meta-heuristic optimization. In Proceedings of the Genetic and Evolutionary Computing Conference (GECCO 2011) (pp. 1723–1730), Dublin, Ireland.Google Scholar
  30. Maler, O., & Nickovic, D. (2004). Monitoring temporal properties of continuous signals. Formal Techniques (pp. 152–166), Modelling and Analysis of Timed and Fault-Tolerant Systems, Lecture Notes in Computer Science. Heidelberg: Springer.Google Scholar
  31. Maus, C., Rybacki, S., & Uhrmacher, A. M. (2011). Rule-based multi-level modeling of cell biological systems. BMC Systems Biology, 5, 166.CrossRefGoogle Scholar
  32. Moreau, L., et al. (2011). The open provenance model core specification (v1. 1). Future Generation Computer Systems, 27(6), 743–756.CrossRefGoogle Scholar
  33. Murta, L., Braganholo, V., Chirigati, F., Koop, D.,& Freire, J. (2014). Noworkflow: Capturing and analyzing provenance of scripts. In International Provenance and Annotation Workshop (pp. 71–83). Springer.Google Scholar
  34. Noble, E. et al. (2012). Linked lives: The Utility of an agent-based approach to modeling partnership and household formation in the context of social care. In Proceedings of the 2012 Winter Simulation Conference, pp. 93:1–93:12. IEEE.Google Scholar
  35. Novère, N. L., Finney, A., Hucka, M., Bhalla, U. S., Campagne, F., Collado-Vides, J., et al. (2005). Minimum information requested in the annotation of biochemical models (MIRIAM). Nature Biotechnology, 23(12), 1509–1515.CrossRefGoogle Scholar
  36. Peng, D., Warnke, T., Haack, F., & Uhrmacher, A. M. (2016). Reusing simulation experiment specifications to support developing models by successive extension. Simulation Modelling Practice and Theory, 68, 33–53.CrossRefGoogle Scholar
  37. Peng, D., Warnke, T., Haack, F., & Uhrmacher, A. M. (2017). Reusing simulation experiment specifications in developing models by successive composition—a case study of the wnt/\(\beta \)-catenin signaling pathway. SIMULATION, 93(8), 659–677.CrossRefGoogle Scholar
  38. Pierce, M. E., Krumme, U., & Uhrmacher, A. M. (2018). Building simulation models of complex ecological systems by successive composition and reusing simulation experiments. In Proceedings of the 2018 Winter Simulation Conference. IEEE.Google Scholar
  39. Rahmandad, H., & Sterman, J. D. (2012). Reporting guidelines for simulation-based research in social sciences. System Dynamics Review, 28(4), 396–411.CrossRefGoogle Scholar
  40. Reinhardt, O., & Uhrmacher, A. M. (2017). An efficient simulation algorithm for continuous-time agent-based linked lives models. In Proceedings of the 50th Annual Simulation Symposium, ANSS’17, pp. 9:1–9:12, San Diego, CA, USA. Society for Computer Simulation International.Google Scholar
  41. Reinhardt, O., Hilton, J., Warnke, T., Bijak, J., & Uhrmacher, A. M. (2018a). Streamlining simulation experiments with agent-based models in demography. Journal of Artificial Societies and Social Simulation, 21(3), 9.Google Scholar
  42. Reinhardt, O., Ruscheinski, A., & Uhrmacher, A. M. (2018b). Odd+p: Complementing the odd protocol with provenance information. In Proceedings of the 2018 Winter Simulation Conference. IEEE.Google Scholar
  43. Rozier, K. Y. (2011). Linear temporal logic symbolic model checking. Computer Science Review, 5(2), 163–203.CrossRefGoogle Scholar
  44. Ruscheinski, A., & Uhrmacher, A. M. (2017). Provenance in modeling and simulation studies-bridging gaps. In Proceedings of the 2017 Winter Simulation Conference. IEEE Press.Google Scholar
  45. Rybacki, S., Leye, S., Himmelspach, J., & Uhrmacher, A. M. (2012). Template and frame based experiment workflows in modeling and simulation software with worms. In 2012 IEEE Eighth World Congress on Services (SERVICES) (pp. 25–32). IEEE.Google Scholar
  46. Scheidegger, C., Koop, D., Santos, E., Vo, H., Callahan, S., Freire, J., et al. (2008). Tackling the provenance challenge one layer at a time. Concurrency and Computation: Practice and Experience, 20(5), 473–483.CrossRefGoogle Scholar
  47. Schützel, J., Peng, D., Uhrmacher, A. M., & Perrone, L. F. (2014). Perspectives on languages for specifying simulation experiments. In Proceedings of the 2014 Winter Simulation Conference (pp. 2836–2847). IEEE Press (2014).Google Scholar
  48. Sheppard,C. J.,& Railsback, S. (2015). Time Extension for NetLogo (Version 1.2) [Software].
  49. Steiniger, A., Uhrmacher, A. M., Zinn, S., Gampe, J.,& Willekens, F. (2014). The role of languages for modeling and simulating continuous-time multi-level models in demography. In Proceedings of the 2014 Winter Simulation Conference, pp. 2978–2989. IEEE.Google Scholar
  50. Troitzsch, K. G. (2004). Validating simulation models. In G. Horton (ed.), 18th European Simulation Multiconference. Networked Simulations and Simulation Networks (pp. 265–270).Google Scholar
  51. Troitzsch, K. G. (2017). Using empirical data for designing, calibrating and validating simulation models. Advances in Social Simulation 2015 (pp. 413–427). Advances in Intelligent Systems and Computing. Cham: Springer.Google Scholar
  52. Uhrmacher, A. M., Brailsford, S., Liu, J., Rabe, M.,& Tolk,A. (2016). Panel—reproducible research in discrete event simulation—a must or rather a maybe? In Proceedings of the 2016 Winter Simulation Conference, pp. 1301–1315. IEEE.Google Scholar
  53. van Deursen, A., Klint, P., & Visser, J. (2000). Domain-specific languages: An annotated bibliography. SIGPLAN Notices, 35(6), 26–36.CrossRefGoogle Scholar
  54. Waltemath, D., Adams, R., Bergmann, F. T., Hucka, M., Kolpakov, F., Miller, A. K., et al. (2011). Reproducible computational biology experiments with SED-ML—the simulation experiment description markup language. BMC Systems Biology, 5(1), 198.CrossRefGoogle Scholar
  55. Warnke, T., Reinhardt, O., Klabunde, A., Willekens, F., & Uhrmacher, A. M. (2017). Modelling and simulating decision processes of linked lives—an approach based on concurrent processes and stochastic race. Population Studies, 71(sup1), 69–83.Google Scholar
  56. Wittgenstein, L. (1922). Tractatus Logico-Philosophicus. London: Routledge & Kegan Paul.zbMATHGoogle Scholar
  57. Zeigler, B. P., Praehofer, H., & Kim, T. G. (2000). Theory of Modeling and Simulation (2nd ed.). San Diego: Academic Press.zbMATHGoogle Scholar
  58. Zinn, S. (2012). A mate-matching algorithm for continuous-time microsimulation models. International Journal of Microsimulation, 5(1), 31–51.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Oliver Reinhardt
    • 1
    Email author
  • Tom Warnke
    • 1
  • Andreas Ruscheinski
    • 1
  • Adelinde M. Uhrmacher
    • 1
  1. 1.University of Rostock, Institute of Computer ScienceRostockGermany

Personalised recommendations