Abstract
Verification and validation are two important aspects of model building. Verification and validation compare models with observations and descriptions of the problem modelled, which may include other models that have been verified and validated to some level. However, the use of simulation for modelling social complexity is very diverse. Often, verification and validation do not refer to an explicit stage in the simulation development process, but to the modelling process itself, according to good practices and in a way that grants credibility to using the simulation for a specific purpose. One cannot consider verification and validation without considering the purpose of the simulation. This chapter deals with a comprehensive outline of methodological perspectives and practical uses of verification and validation. The problem of evaluating simulations is addressed in four main topics: (1) the meaning of the terms verification and validation in the context of simulating social complexity; (2) types of validation, as well as techniques for validating simulations; (3) model replication and comparison as cornerstones of verification and validation; and (4) the relationship of various validation types and techniques with different modelling strategies.
The original version of this chapter was revised. An erratum to this chapter can be found at https://doi.org/10.1007/978-3-319-66948-9_30
This is a preview of subscription content, log in via an institution to check access.
Notes
- 1.
Numerical simulation refers to simulation for finding solutions to mathematical models, normally for cases in which mathematics does not provide analytical solutions. Technical simulation stands for simulation with numerical models in computational sciences and engineering.
- 2.
Verification in the left quadrant of Fig. 9.1 is sometimes known as “internal validation.”
References
Alberts, S., Keenan, M. K., D’Souza, R. M., & An, G. (2012). Data-parallel techniques for simulating a mega-scale agent-based model of systemic inflammatory response syndrome on graphics processing units. Simulation, 88(8), 895–907. doi:10.1177/0037549711425180, http://journals.sagepub.com/doi/abs/10.1177/0037549711425180
Altman, M., Borgman, C., Crosas, M., & Matone, M. (2015). An introduction to the joint principles for data citation. Bulletin of the Association for Information Science and Technology, 41(3), 43–45. doi:10.1002/bult.2015.1720410313, http://onlinelibrary.wiley.com/doi/10.1002/bult.2015.1720410313/abstract
Amblard, F., Bommel, P., & Rouchier, J. (2007). Assessment and validation of multi-agent models. In Agent-based modelling and simulation in the social and human sciences (pp. 93–116). Oxford: Bardwell Press. http://agritrop.cirad.fr/541339/
Amorim, R. C., Castro, J. A., Silva, J. Rd., & Ribeiro, C. (2015). A comparative study of platforms for research data management: Interoperability, metadata capabilities and integration potential. In New contributions in information systems and technologies (pp. 101–111). Cham: Springer. doi:10.1007/978-3-319-16486-1_10, https://link.springer.com/chapter/10.1007/978-3-319-16486-1_10
Arai, R., & Watanabe, S. (2008). A quantitative method for comparing multi-agent-based simulations in feature space. In Multi-agent-based simulation IX (pp.154–166). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01991-3_12, https://link.springer.com/chapter/10.1007/978-3-642-01991-3_12
Assante, M., Candela, L., Castelli, D., & Tani, A. (2016). Are scientific data repositories coping with research data publishing? Data Science Journal, 15, 6. doi:10.5334/dsj-2016-006, http://datascience.codata.org/articles/10.5334/dsj-2016-006/
Axelrod, R. (1993). A Model of the Emergence of New Political Actors. Working paper 93-11-068, Santa Fe Institute. https://www.santafe.edu/research/results/working-papers/a-model-of-the-emergence-of-new-political-actors
Axelrod, R. (1997a). Advancing the art of simulation in the social sciences. In D. R. Conte, P. D. R. Hegselmann, & P. D. P. Terna (Eds.), Simulating Social Phenomena. Lecture notes in economics and mathematical systems (Vol. 456, pp. 21–40). Berlin/Heidelberg: Springer. doi:10.1007/978-3-662-03366-1_2, http://link.springer.com/chapter/10.1007/978-3-662-03366-1_2
Axelrod, R. (1997b). The dissemination of culture: A model with local convergence and global polarization. Journal of Conflict Resolution, 41(2), 203–226. doi:10.1177/0022002797041002001, http://dx.doi.org/10.1177/0022002797041002001
Axtell, R., Axelrod, R., Epstein, J. M., & Cohen, M. D. (1996). Aligning simulation models: A case study and results. Computational & Mathematical Organization Theory, 1(2), 123–141. doi:10.1007/BF01299065, https://link.springer.com/article/10.1007/BF01299065
Balci, O., & Sargent, R. G. (1984). Validation of simulation models via simultaneous confidence intervals. American Journal of Mathematical and Management Sciences, 4(3–4), 375–406. doi:10.1080/01966324.1984.10737151, http://dx.doi.org/10.1080/01966324.1984.10737151
Barreteau, O., Bots, P., Daniell, K., Etienne, M., Perez, P., Barnaud, C., et al. (2017). Participatory approaches. doi:https://doi.org/10.1007/978-3-319-66948-9_12.
Barreteau, O., Bots, P., Daniell, K., Etienne, M., Perez, P., Barnaud, C., et al. (2017). Participatory approaches. In B. Edmonds & R. Meyer (Eds.), Simulating social complexity. Understanding complex systems (2nd ed.). Berlin/Heidelberg: Springer. doi:10.1007/978-3-319-66948-9_12
Boero, R., & Squazzoni, F. (2005). Does empirical embeddedness matter? Methodological issues on agent-based models for analytical social science. Journal of Artificial Societies and Social Simulation, 8(4), 6. http://jasss.soc.surrey.ac.uk/8/4/6.html
Calvez, B., & Hutzler, G. (2005). Automatic tuning of agent-based models using genetic algorithms. In Multi-agent-based simulation VI (pp. 41–57). Berlin/Heidelberg: Springer. doi:10.1007/11734680_4, https://link.springer.com/chapter/10.1007/11734680_4
Collier, N., & North, M. (2013). Parallel agent-based simulation with repast for high performance computing. Simulation, 89(10), 1215–1235. doi:10.1177/0037549712462620, http://journals.sagepub.com/doi/abs/10.1177/0037549712462620
David, N. (2009). Validation and verification in social simulation: patterns and clarification of terminology. In Epistemological aspects of computer simulation in the social sciences (pp. 117–129). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01109-2_9, https://link.springer.com/chapter/10.1007/978-3-642-01109-2_9
David, N., Marietto, M. B., Sichman, J. S., & Coelho, H. (2004). The structure and logic of interdisciplinary research in agent-based social simulation. Journal of Artificial Societies and Social Simulation, 7(3), 4. http://jasss.soc.surrey.ac.uk/7/3/4.html
David, N., Sichman, J. S., & Coelho, H. (2005). The logic of the method of agent-based simulation in the social sciences: Empirical and intentional adequacy of computer programs. Journal of Artificial Societies and Social Simulation, 8(4), 2. http://jasss.soc.surrey.ac.uk/8/4/2.html
David, N., Caldas, J. C., & Coelho, H. (2010). Epistemological perspectives on simulation III. Journal of Artificial Societies and Social Simulation, 13(1). doi:10.18564/jasss.1591, http://jasss.soc.surrey.ac.uk/13/1/14.html
Dean, J. S., Gumerman, G. J., Epstein, J. M., Axtell, R. L., Swedlund, A. C., Parker, M. T., et al. (2000). Understanding Anasazi culture change through agent-based modeling. In T. A. Kohler & G. J. Gumerman (Eds.), Dynamics in human and primate societies: Agent-based modeling of social and spatial processes. Santa fe institute studies on the sciences of complexity (pp. 179–205). New York/Oxford: Oxford University Press.
Densmore, O. (2016). AgentScript. http://agentscript.org/
Edmonds, B., & Hales, D. (2003). Replication, replication and replication: Some hard lessons from model alignment. Journal of Artificial Societies and Social Simulation, 6(4), 11. http://jasss.soc.surrey.ac.uk/6/4/11.html
Edmonds, B. & Moss, S. (2005). From KISS to KIDS—an ‘anti-simplistic’ modelling approach. In: P. Davidsson, B. Logan, & K. Takadama (Eds.), Multi-agent and multi-agent-based simulation (Vol. 3415, pp. 130–144). Berlin/Heidelberg: Springer. doi:10.1007/978-3-540-32243-6_11. http://link.springer.com/10.1007/978-3-540-32243-6_11
Epstein, J., & Axtell, R. (1996). Growing artificial societies: Social science from the bottom up. Washington, DC: Brookings Institution Press; Cambridge, MA: MIT Press.
Evans, A., Heppenstall, A., & Birkin, M. (2017). Understanding simulation results. doi: https://doi.org/10.1007/978-3-319-66948-9_10.
Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2015). Towards a standard model for research in agent-based modeling and simulation. PeerJ Computer Science, 1, e36. doi:10.7717/peerj-cs.36, https://peerj.com/articles/cs-36
Fachada, N., Rodrigues, J., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2016). micompr: An R package for multivariate independent comparison of observations. The R Journal, 8(2), 405–420. http://journal.r-project.org/archive/2016-2/fachada-rodrigues-lopes-etal.pdf
Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2017a). Parallelization strategies for spatial agent-based models. International Journal of Parallel Programming, 45(3), 449–481.
Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2017b). Model-independent comparison of simulation output. Simulation Modelling Practice and Theory, 72, 131–149. doi:10.1016/j.simpat.2016.12.013, http://www.sciencedirect.com/science/article/pii/S1569190X16302854
Frank, U., & Troitzsch, K. G. (2005). Epistemological perspectives on simulation. Journal of Artificial Societies and Social Simulation, 8(4), 7. http://jasss.soc.surrey.ac.uk/8/4/7.html
Galán, J. M., Izquierdo, L. R., Izquierdo, S. S., Santos, J. I., Olmo, Rd., & López-Paredes, A. (2017). Checking simulations: Detecting and avoiding errors and artefacts. doi: https://doi.org/10.1007/978-3-319-66948-9_9.
Gilbert, N. (2008). Agent-based models. Thousand Oaks, CA: SAGE. google-Books-ID: Z3cp0ZBK9UsC.
Grimm, V., Polhill, G., & Touza, J. (2017). Documenting social simulation models: The ODD protocol as a standard. doi: https://doi.org/10.1007/978-3-319-66948-9_10.
Gross, D., & Strand, R. (2000). Can agent-based models assist decisions on large-scale practical problems? A philosophical analysis. Complexity, 5(6), 26–33. doi:10.1002/1099-0526(200007/08)5:6¡26::AID-CPLX6¿3.0.CO;2-G, http://onlinelibrary.wiley.com/doi/10.1002/1099-0526(200007/08)5:6<26::AID-CPLX6>3.0.CO;2-G/abstract
Kratz, J., & Strasser, C. (2014). Data publication consensus and controversies. F1000Research, 3, 94. doi:10.12688/f1000research.3979.3, http://f1000research.com/articles/3-94/v3
Laird, J. E. (2012). The soar cognitive architecture. Cambridge: MIT Press.
Law, A. M. (2015). Simulation modeling and analysis (5th ed.). New York: McGraw Hill Higher Education.
Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., et al. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4), 4.
McKay, M. D., Beckman, R. J., & Conover, W. J. (1979). Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2), 239–245. doi:10.1080/00401706.1979.10489755, http://dx.doi.org/10.1080/00401706.1979.10489755
Merlone, U., Sonnessa, M., & Terna, P. (2008). Horizontal and vertical multiple implementations in a model of industrial districts. Journal of Artificial Societies and Social Simulation 11(2), 5. http://jasss.soc.surrey.ac.uk/11/2/5.html
Miller, J. H. (1998). Active nonlinear tests (ANTs) of complex simulation models. Management Science, 44(6), 820–830. doi:10.1287/mnsc.44.6.820, http://pubsonline.informs.org/doi/abs/10.1287/mnsc.44.6.820
Miodownik, D., Cartrite, B., & Bhavnani, R. (2010). Between replication and docking: “adaptive agents, political institutions, and civic traditions” revisited. Journal of Artificial Societies and Social Simulation, 13(3), 1.
Müller, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., et al. (2013), Describing human decisions in agent-based models – ODD + D, an extension of the ODD protocol. Environmental Modelling & Software, 48, 37–48. doi:10.1016/j.envsoft.2013.06.003, http://www.sciencedirect.com/science/article/pii/S1364815213001394
Müller, B., Balbi, S., Buchmann, C. M., de Sousa, L., Dressler, G., Groeneveld, J., et al. (2014). Standardised and transparent model descriptions for agent-based models: Current status and prospects. Environmental Modelling & Software, 55, 156–163. doi:10.1016/j.envsoft.2014.01.029, http://www.sciencedirect.com/science/article/pii/S1364815214000395
Montgomery, D. C. (2012). Design and analysis of experiments (8th ed.). Hoboken: Wiley.
Moss, S., & Edmonds, B. (2005). Sociology and simulation: Statistical and qualitative cross-validation. American Journal of Sociology, 110(4), 1095–1131. doi:10.1086/427320, http://www.journals.uchicago.edu/doi/abs/10.1086/427320
North, M. J., Collier, N. T., Ozik, J., Tatara, E. R., Macal, C. M., Bragen, M., et al. (2013). Complex adaptive systems modeling with Repast Simphony. Complex Adaptive Systems Modeling, 1(1), 3. doi:10.1186/2194-3206-1-3, http://casmodeling.springeropen.com/articles/10.1186/2194-3206-1-3
Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. doi:10.1126/science.1213847, http://science.sciencemag.org/content/334/6060/1226
Pereda, M., Santos, J. I., & Galan, J. M. (2015). A brief introduction to the use of machine learning techniques in the analysis of agent-based models. SSRN Scholarly Paper ID 2689676. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=2689676
R Core Team. (2017). R: A language and environment for statistical computing. https://www.R-project.org/
Radax, W., & Rengs, B. (2009). Prospects and pitfalls of statistical testing: Insights from replicating the demographic prisoner’s dilemma. Journal of Artificial Societies and Social Simulation, 13(4), 1.
Rollins, N. D., Barton, C. M., Bergin, S., Janssen, M. A., & Lee, A. (2014). A Computational Model Library for publishing model documentation and code. Environmental Modelling & Software, 61, 59–64. doi:10.1016/j.envsoft.2014.06.022, http://www.sciencedirect.com/science/article/pii/S1364815214001959
Rouchier, J., Cioffi-Revilla, C., Polhill, J. G., & Takadama, K. (2008). Progress in model-to-model analysis. Journal of Artificial Societies and Social Simulation, 11(2), 8. http://jasss.soc.surrey.ac.uk/11/2/8.html
Sargent, R. G. (2013). Verification and validation of simulation models. Journal of Simulation, 7(1), 12–24. doi:10.1057/jos.2012.20, https://link.springer.com/article/10.1057/jos.2012.20
Schelling, T. C. (1971). Dynamic models of segregation. The Journal of Mathematical Sociology, 1(2), 143–186. doi:10.1080/0022250X.1971.9989794, http://dx.doi.org/10.1080/0022250X.1971.9989794
Squazzoni, F. (Ed.). (2009). Epistemological aspects of computer simulation in the social sciences. Lecture notes in computer science (Vol. 5466). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01109-2, http://link.springer.com/10.1007/978-3-642-01109-2
Stonedahl, F., & Wilensky, U. (2010). Finding forms of flocking: Evolutionary search in ABM parameter-spaces. In Multi-agent-based simulation XI (pp. 61–75). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-18345-4_5, https://link.springer.com/chapter/10.1007/978-3-642-18345-4_5
Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., et al. (2016) Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software, 86, 56–67. doi:10.1016/j.envsoft.2016.09.006, http://www.sciencedirect.com/science/article/pii/S1364815216306041
Takadama, K., Suematsu, Y. L., Sugimoto, N., Nawa, N. E., & Shimohara, K. (2003). Cross-element validation in multiagent-based simulation: Switching learning mechanisms in agents. Journal of Artificial Societies and Social Simulation, 6(4), 6. http://jasss.soc.surrey.ac.uk/6/4/6.html
Thiele, J. C., & Grimm, V. (2015). Replicating and breaking models: Good for you and good for ecology. Oikos, 124(6), 691–696. doi:10.1111/oik.02170, http://onlinelibrary.wiley.com/doi/10.1111/oik.02170/abstract
Troitzsch, K. G. (2004). Validating simulation models. In G. Horton (Ed.), Proceedings of 18th European Simulation Multiconference, ESM 2004 (pp. 265–270). Magdeburg: SCS Publishing House.
Wiersma, W. (2015). AgentBase: Agent based modelling in the browser. http://wybowiersma.net/pub/papers/Wiersma,Wybo,AgentBase_agent_based_modelling_in_the_browser.pdf
Wilensky, U. (1999). NetLogo. http://ccl.northwestern.edu/netlogo/
Wilensky, U., & Rand, W. (2007). Making models match: Replicating an agent-based model. Journal of Artificial Societies and Social Simulation, 10(4), 2. http://jasss.soc.surrey.ac.uk/10/4/2.html
Will, O., & Hegselmann, R. (2008). A replication that failed – On the computational model in ‘Michael W. Macy and Yoshimichi Sato: Trust, cooperation and market formation in the U.S. and Japan. Proceedings of the national academy of sciences, May 2002’. Journal of Artificial Societies and Social Simulation, 11(3), 3. http://jasss.soc.surrey.ac.uk/11/3/3.html
Acknowledgements
This work was partially funded by the Fundação para a Ciência e a Tecnologia project UID/EEA/50009/2013.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
David, N., Fachada, N., Rosa, A.C. (2017). Verifying and Validating Simulations. In: Edmonds, B., Meyer, R. (eds) Simulating Social Complexity. Understanding Complex Systems. Springer, Cham. https://doi.org/10.1007/978-3-319-66948-9_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-66948-9_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-66947-2
Online ISBN: 978-3-319-66948-9
eBook Packages: Computer ScienceComputer Science (R0)