Advertisement

Verifying and Validating Simulations

  • Nuno David
  • Nuno Fachada
  • Agostinho C. Rosa
Chapter
Part of the Understanding Complex Systems book series (UCS)

Abstract

Verification and validation are two important aspects of model building. Verification and validation compare models with observations and descriptions of the problem modelled, which may include other models that have been verified and validated to some level. However, the use of simulation for modelling social complexity is very diverse. Often, verification and validation do not refer to an explicit stage in the simulation development process, but to the modelling process itself, according to good practices and in a way that grants credibility to using the simulation for a specific purpose. One cannot consider verification and validation without considering the purpose of the simulation. This chapter deals with a comprehensive outline of methodological perspectives and practical uses of verification and validation. The problem of evaluating simulations is addressed in four main topics: (1) the meaning of the terms verification and validation in the context of simulating social complexity; (2) types of validation, as well as techniques for validating simulations; (3) model replication and comparison as cornerstones of verification and validation; and (4) the relationship of various validation types and techniques with different modelling strategies.

Notes

Acknowledgements

This work was partially funded by the Fundação para a Ciência e a Tecnologia project UID/EEA/50009/2013.

References

  1. Alberts, S., Keenan, M. K., D’Souza, R. M., & An, G. (2012). Data-parallel techniques for simulating a mega-scale agent-based model of systemic inflammatory response syndrome on graphics processing units. Simulation, 88(8), 895–907. doi:10.1177/0037549711425180, http://journals.sagepub.com/doi/abs/10.1177/0037549711425180 CrossRefGoogle Scholar
  2. Altman, M., Borgman, C., Crosas, M., & Matone, M. (2015). An introduction to the joint principles for data citation. Bulletin of the Association for Information Science and Technology, 41(3), 43–45. doi:10.1002/bult.2015.1720410313, http://onlinelibrary.wiley.com/doi/10.1002/bult.2015.1720410313/abstract CrossRefGoogle Scholar
  3. Amblard, F., Bommel, P., & Rouchier, J. (2007). Assessment and validation of multi-agent models. In Agent-based modelling and simulation in the social and human sciences (pp. 93–116). Oxford: Bardwell Press. http://agritrop.cirad.fr/541339/ Google Scholar
  4. Amorim, R. C., Castro, J. A., Silva, J. Rd., & Ribeiro, C. (2015). A comparative study of platforms for research data management: Interoperability, metadata capabilities and integration potential. In New contributions in information systems and technologies (pp. 101–111). Cham: Springer. doi:10.1007/978-3-319-16486-1_10, https://link.springer.com/chapter/10.1007/978-3-319-16486-1_10 CrossRefGoogle Scholar
  5. Arai, R., & Watanabe, S. (2008). A quantitative method for comparing multi-agent-based simulations in feature space. In Multi-agent-based simulation IX (pp.154–166). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01991-3_12, https://link.springer.com/chapter/10.1007/978-3-642-01991-3_12
  6. Assante, M., Candela, L., Castelli, D., & Tani, A. (2016). Are scientific data repositories coping with research data publishing? Data Science Journal, 15, 6. doi:10.5334/dsj-2016-006, http://datascience.codata.org/articles/10.5334/dsj-2016-006/ CrossRefGoogle Scholar
  7. Axelrod, R. (1993). A Model of the Emergence of New Political Actors. Working paper 93-11-068, Santa Fe Institute. https://www.santafe.edu/research/results/working-papers/a-model-of-the-emergence-of-new-political-actors
  8. Axelrod, R. (1997a). Advancing the art of simulation in the social sciences. In D. R. Conte, P. D. R. Hegselmann, & P. D. P. Terna (Eds.), Simulating Social Phenomena. Lecture notes in economics and mathematical systems (Vol. 456, pp. 21–40). Berlin/Heidelberg: Springer. doi:10.1007/978-3-662-03366-1_2, http://link.springer.com/chapter/10.1007/978-3-662-03366-1_2
  9. Axelrod, R. (1997b). The dissemination of culture: A model with local convergence and global polarization. Journal of Conflict Resolution, 41(2), 203–226. doi:10.1177/0022002797041002001, http://dx.doi.org/10.1177/0022002797041002001
  10. Axtell, R., Axelrod, R., Epstein, J. M., & Cohen, M. D. (1996). Aligning simulation models: A case study and results. Computational & Mathematical Organization Theory, 1(2), 123–141. doi:10.1007/BF01299065, https://link.springer.com/article/10.1007/BF01299065 CrossRefGoogle Scholar
  11. Balci, O., & Sargent, R. G. (1984). Validation of simulation models via simultaneous confidence intervals. American Journal of Mathematical and Management Sciences, 4(3–4), 375–406. doi:10.1080/01966324.1984.10737151, http://dx.doi.org/10.1080/01966324.1984.10737151 CrossRefzbMATHGoogle Scholar
  12. Barreteau, O., Bots, P., Daniell, K., Etienne, M., Perez, P., Barnaud, C., et al. (2017). Participatory approaches. doi:https://doi.org/10.1007/978-3-319-66948-9_12.
  13. Barreteau, O., Bots, P., Daniell, K., Etienne, M., Perez, P., Barnaud, C., et al. (2017). Participatory approaches. In B. Edmonds & R. Meyer (Eds.), Simulating social complexity. Understanding complex systems (2nd ed.). Berlin/Heidelberg: Springer. doi:10.1007/978-3-319-66948-9_12Google Scholar
  14. Boero, R., & Squazzoni, F. (2005). Does empirical embeddedness matter? Methodological issues on agent-based models for analytical social science. Journal of Artificial Societies and Social Simulation, 8(4), 6. http://jasss.soc.surrey.ac.uk/8/4/6.html Google Scholar
  15. Calvez, B., & Hutzler, G. (2005). Automatic tuning of agent-based models using genetic algorithms. In Multi-agent-based simulation VI (pp. 41–57). Berlin/Heidelberg: Springer. doi:10.1007/11734680_4, https://link.springer.com/chapter/10.1007/11734680_4 Google Scholar
  16. Collier, N., & North, M. (2013). Parallel agent-based simulation with repast for high performance computing. Simulation, 89(10), 1215–1235. doi:10.1177/0037549712462620, http://journals.sagepub.com/doi/abs/10.1177/0037549712462620 CrossRefGoogle Scholar
  17. David, N. (2009). Validation and verification in social simulation: patterns and clarification of terminology. In Epistemological aspects of computer simulation in the social sciences (pp. 117–129). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01109-2_9, https://link.springer.com/chapter/10.1007/978-3-642-01109-2_9 CrossRefGoogle Scholar
  18. David, N., Marietto, M. B., Sichman, J. S., & Coelho, H. (2004). The structure and logic of interdisciplinary research in agent-based social simulation. Journal of Artificial Societies and Social Simulation, 7(3), 4. http://jasss.soc.surrey.ac.uk/7/3/4.html Google Scholar
  19. David, N., Sichman, J. S., & Coelho, H. (2005). The logic of the method of agent-based simulation in the social sciences: Empirical and intentional adequacy of computer programs. Journal of Artificial Societies and Social Simulation, 8(4), 2. http://jasss.soc.surrey.ac.uk/8/4/2.html Google Scholar
  20. David, N., Caldas, J. C., & Coelho, H. (2010). Epistemological perspectives on simulation III. Journal of Artificial Societies and Social Simulation, 13(1). doi:10.18564/jasss.1591, http://jasss.soc.surrey.ac.uk/13/1/14.html
  21. Dean, J. S., Gumerman, G. J., Epstein, J. M., Axtell, R. L., Swedlund, A. C., Parker, M. T., et al. (2000). Understanding Anasazi culture change through agent-based modeling. In T. A. Kohler & G. J. Gumerman (Eds.), Dynamics in human and primate societies: Agent-based modeling of social and spatial processes. Santa fe institute studies on the sciences of complexity (pp. 179–205). New York/Oxford: Oxford University Press.Google Scholar
  22. Densmore, O. (2016). AgentScript. http://agentscript.org/ Google Scholar
  23. Edmonds, B., & Hales, D. (2003). Replication, replication and replication: Some hard lessons from model alignment. Journal of Artificial Societies and Social Simulation, 6(4), 11. http://jasss.soc.surrey.ac.uk/6/4/11.html Google Scholar
  24. Edmonds, B. & Moss, S. (2005). From KISS to KIDS—an ‘anti-simplistic’ modelling approach. In: P. Davidsson, B. Logan, & K. Takadama (Eds.), Multi-agent and multi-agent-based simulation (Vol. 3415, pp. 130–144). Berlin/Heidelberg: Springer. doi:10.1007/978-3-540-32243-6_11. http://link.springer.com/10.1007/978-3-540-32243-6_11 CrossRefGoogle Scholar
  25. Epstein, J., & Axtell, R. (1996). Growing artificial societies: Social science from the bottom up. Washington, DC: Brookings Institution Press; Cambridge, MA: MIT Press.Google Scholar
  26. Evans, A., Heppenstall, A., & Birkin, M. (2017). Understanding simulation results. doi: https://doi.org/10.1007/978-3-319-66948-9_10.
  27. Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2015). Towards a standard model for research in agent-based modeling and simulation. PeerJ Computer Science, 1, e36. doi:10.7717/peerj-cs.36, https://peerj.com/articles/cs-36 CrossRefGoogle Scholar
  28. Fachada, N., Rodrigues, J., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2016). micompr: An R package for multivariate independent comparison of observations. The R Journal, 8(2), 405–420. http://journal.r-project.org/archive/2016-2/fachada-rodrigues-lopes-etal.pdf Google Scholar
  29. Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2017a). Parallelization strategies for spatial agent-based models. International Journal of Parallel Programming, 45(3), 449–481.Google Scholar
  30. Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2017b). Model-independent comparison of simulation output. Simulation Modelling Practice and Theory, 72, 131–149. doi: 10.1016/j.simpat.2016.12.013, http://www.sciencedirect.com/science/article/pii/S1569190X16302854
  31. Frank, U., & Troitzsch, K. G. (2005). Epistemological perspectives on simulation. Journal of Artificial Societies and Social Simulation, 8(4), 7. http://jasss.soc.surrey.ac.uk/8/4/7.html Google Scholar
  32. Galán, J. M., Izquierdo, L. R., Izquierdo, S. S., Santos, J. I., Olmo, Rd., & López-Paredes, A. (2017). Checking simulations: Detecting and avoiding errors and artefacts. doi: https://doi.org/10.1007/978-3-319-66948-9_9.
  33. Gilbert, N. (2008). Agent-based models. Thousand Oaks, CA: SAGE. google-Books-ID: Z3cp0ZBK9UsC.Google Scholar
  34. Grimm, V., Polhill, G., & Touza, J. (2017). Documenting social simulation models: The ODD protocol as a standard. doi: https://doi.org/10.1007/978-3-319-66948-9_10.
  35. Gross, D., & Strand, R. (2000). Can agent-based models assist decisions on large-scale practical problems? A philosophical analysis. Complexity, 5(6), 26–33. doi:10.1002/1099-0526(200007/08)5:6¡26::AID-CPLX6¿3.0.CO;2-G, http://onlinelibrary.wiley.com/doi/10.1002/1099-0526(200007/08)5:6<26::AID-CPLX6>3.0.CO;2-G/abstract CrossRefGoogle Scholar
  36. Kratz, J., & Strasser, C. (2014). Data publication consensus and controversies. F1000Research, 3, 94. doi:10.12688/f1000research.3979.3, http://f1000research.com/articles/3-94/v3
  37. Laird, J. E. (2012). The soar cognitive architecture. Cambridge: MIT Press.Google Scholar
  38. Law, A. M. (2015). Simulation modeling and analysis (5th ed.). New York: McGraw Hill Higher Education.Google Scholar
  39. Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., et al. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4), 4.CrossRefGoogle Scholar
  40. McKay, M. D., Beckman, R. J., & Conover, W. J. (1979). Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2), 239–245. doi:10.1080/00401706.1979.10489755, http://dx.doi.org/10.1080/00401706.1979.10489755 MathSciNetzbMATHGoogle Scholar
  41. Merlone, U., Sonnessa, M., & Terna, P. (2008). Horizontal and vertical multiple implementations in a model of industrial districts. Journal of Artificial Societies and Social Simulation 11(2), 5. http://jasss.soc.surrey.ac.uk/11/2/5.html Google Scholar
  42. Miller, J. H. (1998). Active nonlinear tests (ANTs) of complex simulation models. Management Science, 44(6), 820–830. doi:10.1287/mnsc.44.6.820, http://pubsonline.informs.org/doi/abs/10.1287/mnsc.44.6.820 CrossRefzbMATHGoogle Scholar
  43. Miodownik, D., Cartrite, B., & Bhavnani, R. (2010). Between replication and docking: “adaptive agents, political institutions, and civic traditions” revisited. Journal of Artificial Societies and Social Simulation, 13(3), 1.CrossRefGoogle Scholar
  44. Müller, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., et al. (2013), Describing human decisions in agent-based models – ODD + D, an extension of the ODD protocol. Environmental Modelling & Software, 48, 37–48. doi:10.1016/j.envsoft.2013.06.003, http://www.sciencedirect.com/science/article/pii/S1364815213001394 CrossRefGoogle Scholar
  45. Müller, B., Balbi, S., Buchmann, C. M., de Sousa, L., Dressler, G., Groeneveld, J., et al. (2014). Standardised and transparent model descriptions for agent-based models: Current status and prospects. Environmental Modelling & Software, 55, 156–163. doi:10.1016/j.envsoft.2014.01.029, http://www.sciencedirect.com/science/article/pii/S1364815214000395 CrossRefGoogle Scholar
  46. Montgomery, D. C. (2012). Design and analysis of experiments (8th ed.). Hoboken: Wiley.Google Scholar
  47. Moss, S., & Edmonds, B. (2005). Sociology and simulation: Statistical and qualitative cross-validation. American Journal of Sociology, 110(4), 1095–1131. doi:10.1086/427320, http://www.journals.uchicago.edu/doi/abs/10.1086/427320 CrossRefGoogle Scholar
  48. North, M. J., Collier, N. T., Ozik, J., Tatara, E. R., Macal, C. M., Bragen, M., et al. (2013). Complex adaptive systems modeling with Repast Simphony. Complex Adaptive Systems Modeling, 1(1), 3. doi:10.1186/2194-3206-1-3, http://casmodeling.springeropen.com/articles/10.1186/2194-3206-1-3 CrossRefGoogle Scholar
  49. Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. doi:10.1126/science.1213847, http://science.sciencemag.org/content/334/6060/1226 CrossRefGoogle Scholar
  50. Pereda, M., Santos, J. I., & Galan, J. M. (2015). A brief introduction to the use of machine learning techniques in the analysis of agent-based models. SSRN Scholarly Paper ID 2689676. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=2689676 Google Scholar
  51. R Core Team. (2017). R: A language and environment for statistical computing. https://www.R-project.org/
  52. Radax, W., & Rengs, B. (2009). Prospects and pitfalls of statistical testing: Insights from replicating the demographic prisoner’s dilemma. Journal of Artificial Societies and Social Simulation, 13(4), 1.CrossRefGoogle Scholar
  53. Rollins, N. D., Barton, C. M., Bergin, S., Janssen, M. A., & Lee, A. (2014). A Computational Model Library for publishing model documentation and code. Environmental Modelling & Software, 61, 59–64. doi:10.1016/j.envsoft.2014.06.022, http://www.sciencedirect.com/science/article/pii/S1364815214001959 CrossRefGoogle Scholar
  54. Rouchier, J., Cioffi-Revilla, C., Polhill, J. G., & Takadama, K. (2008). Progress in model-to-model analysis. Journal of Artificial Societies and Social Simulation, 11(2), 8. http://jasss.soc.surrey.ac.uk/11/2/8.html Google Scholar
  55. Sargent, R. G. (2013). Verification and validation of simulation models. Journal of Simulation, 7(1), 12–24. doi:10.1057/jos.2012.20, https://link.springer.com/article/10.1057/jos.2012.20 CrossRefGoogle Scholar
  56. Schelling, T. C. (1971). Dynamic models of segregation. The Journal of Mathematical Sociology, 1(2), 143–186. doi:10.1080/0022250X.1971.9989794, http://dx.doi.org/10.1080/0022250X.1971.9989794 CrossRefzbMATHGoogle Scholar
  57. Squazzoni, F. (Ed.). (2009). Epistemological aspects of computer simulation in the social sciences. Lecture notes in computer science (Vol. 5466). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01109-2, http://link.springer.com/10.1007/978-3-642-01109-2
  58. Stonedahl, F., & Wilensky, U. (2010). Finding forms of flocking: Evolutionary search in ABM parameter-spaces. In Multi-agent-based simulation XI (pp. 61–75). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-18345-4_5, https://link.springer.com/chapter/10.1007/978-3-642-18345-4_5 Google Scholar
  59. Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., et al. (2016) Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software, 86, 56–67. doi:10.1016/j.envsoft.2016.09.006, http://www.sciencedirect.com/science/article/pii/S1364815216306041 CrossRefGoogle Scholar
  60. Takadama, K., Suematsu, Y. L., Sugimoto, N., Nawa, N. E., & Shimohara, K. (2003). Cross-element validation in multiagent-based simulation: Switching learning mechanisms in agents. Journal of Artificial Societies and Social Simulation, 6(4), 6. http://jasss.soc.surrey.ac.uk/6/4/6.html Google Scholar
  61. Thiele, J. C., & Grimm, V. (2015). Replicating and breaking models: Good for you and good for ecology. Oikos, 124(6), 691–696. doi:10.1111/oik.02170, http://onlinelibrary.wiley.com/doi/10.1111/oik.02170/abstract CrossRefGoogle Scholar
  62. Troitzsch, K. G. (2004). Validating simulation models. In G. Horton (Ed.), Proceedings of 18th European Simulation Multiconference, ESM 2004 (pp. 265–270). Magdeburg: SCS Publishing House.Google Scholar
  63. Wiersma, W. (2015). AgentBase: Agent based modelling in the browser. http://wybowiersma.net/pub/papers/Wiersma,Wybo,AgentBase_agent_based_modelling_in_the_browser.pdf
  64. Wilensky, U., & Rand, W. (2007). Making models match: Replicating an agent-based model. Journal of Artificial Societies and Social Simulation, 10(4), 2. http://jasss.soc.surrey.ac.uk/10/4/2.html Google Scholar
  65. Will, O., & Hegselmann, R. (2008). A replication that failed – On the computational model in ‘Michael W. Macy and Yoshimichi Sato: Trust, cooperation and market formation in the U.S. and Japan. Proceedings of the national academy of sciences, May 2002’. Journal of Artificial Societies and Social Simulation, 11(3), 3. http://jasss.soc.surrey.ac.uk/11/3/3.html

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.DINÂMIA’CET - ISCTE-IUL - Centre for Socioeconomic and Territorial StudiesISCTE-IUL Instituto Universitário de LisboaLisboaPortugal
  2. 2.Institute for Systems and Robotics (ISR/IST)LARSyS, Instituto Superior TécnicoLisboaPortugal

Personalised recommendations