Checking Simulations: Detecting and Avoiding Errors and Artefacts

Part of the Understanding Complex Systems book series (UCS)

Why Read This Chapter?

Given the complex and exploratory nature of many agent-based models, checking that the model performs in the manner intended by its designers is a very challenging task. This chapter helps the reader to identify some of the possible types of error and artefact that may appear in the different stages of the modelling process. It will also suggest some activities that can be conducted to detect, and hence avoid, each type.


The aim of this chapter is to provide the reader with a set of concepts and a range of suggested activities that will enhance his or her ability to understand agent-based simulations. To do this in a structured way, we review the main concepts of the methodology (e.g. we provide precise definitions for the terms ‘error’ and ‘artefact’) and establish a general framework that summarises the process of designing, implementing, and using agent-based models. Within this framework we identify the various stages where different types of assumptions are usually made and, consequently, where different types of errors and artefacts may appear. We then propose several activities that can be conducted to detect each type of error and artefact.


Computer Scientist Formal Model Target System Symbolic System Modelling Paradigm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors have benefited from the financial support of the Spanish Ministry of Education and Science (projects DPI2004–06590, DPI2005–05676 and TIN2008–06464–C03–02), the Spanish Ministry for Science and Innovation (CSD2010–00034) within the framework of CONSOLIDER-INGENIO 2010 and of the JCyL (projects VA006B09, BU034A08 and GREX251–2009). We are also very grateful to Nick Gotts, Gary Polhill, Bruce Edmonds and Cesáreo Hernández for many discussions on the philosophy of modelling.


  1. Axelrod RM (1997a) Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P (eds) Simulating social phenomena, vol 456, Lecture notes in economics and mathematical systems. Springer, Berlin, pp 21–40Google Scholar
  2. Axelrod RM (1997b) The dissemination of culture: a model with local convergence and global polarization. J Confl Resolut 41(2):203–226CrossRefGoogle Scholar
  3. Axtell RL (2000) Why agents? On the varied motivations for agent computing in the social sciences. In: Macal CM, Sallach D (eds) Proceedings of the workshop on agent simulation: applications, models, and tools. Argonne National Laboratory, Argonne, pp 3–24Google Scholar
  4. Axtell RL, Epstein JM (1994) Agent based modeling: understanding our creations. The Bulletin of the Santa Fe Institute, Winter, pp 28–32Google Scholar
  5. Bigbee T, Cioffi-Revilla C, Luke S (2007) Replication of sugarscape using MASON. In: Terano T, Kita H, Deguchi H, Kijima K (eds) Agent-based approaches in economic and social complex systems IV: post-proceedings of the AESCS international workshop 2005. Springer, Tokyo, pp 183–190Google Scholar
  6. Bonabeau E (2002) Agent-based modeling: methods and techniques for simulating human systems. Proc Natl Acad Sci U S A 99(2):7280–7287CrossRefGoogle Scholar
  7. Castellano C, Marsili M, Vespignani A (2000) Nonequilibrium phase transition in a model for social influence. Phys Rev Lett 85(16):3536–3539CrossRefGoogle Scholar
  8. Christley S, Xiang X, Madey G (2004) Ontology for agent-based modeling and simulation. In: Macal CM, Sallach D, North MJ (eds) Proceedings of the agent 2004 conference on social dynamics: interaction, reflexivity and emergence. Argonne National laboratory/The University of Chicago, Chicago.
  9. Cioffi-Revilla C (2002) Invariance and universality in social agent-based simulations. Proc Natl Acad Sci U S A 99(3):7314–7316CrossRefGoogle Scholar
  10. Conlisk J (1996) Why bounded rationality? J Econ Lit 34(2):669–700Google Scholar
  11. David N (2013) Validating simulations. Chapter 8 in this volumeGoogle Scholar
  12. Drogoul A, Vanbergue D, Meurisse T (2003) Multi-agent based simulation: where are the agents? In: Sichman JS, Bousquet F, Davidsson P (eds) Proceedings of MABS 2002 multi-agent-based simulation, vol 2581, Lecture notes in computer science. Springer, Bologna, pp 1–15Google Scholar
  13. Edmonds B (2001) The use of models: making MABS actually work. In: Moss S, Davidsson P (eds) Multi-agent-based simulation, vol 1979, Lecture notes in artificial intelligence. Springer, Berlin, pp 15–32CrossRefGoogle Scholar
  14. Edmonds B (2005) Simulation and complexity: how they can relate. In: Feldmann V, Mühlfeld K (eds) Virtual worlds of precision: computer-based simulations in the sciences and social sciences. Lit-Verlag, Münster, pp 5–32Google Scholar
  15. Edmonds B, Hales D (2003) Replication, replication and replication: some hard lessons from model alignment. J Artif Soc Soc Simulat 6(4).
  16. Edmonds B, Hales D (2005) Computational simulation as theoretical experiment. J Math Sociol 29:1–24CrossRefGoogle Scholar
  17. Edwards M, Huet S, Goreaud F, Deffuant G (2003) Comparing an individual-based model of behaviour diffusion with its mean field aggregate approximation. J Artif Soc Soc Simulat 6(4).
  18. Epstein JM (1999) Agent-based computational models and generative social science. Complexity 4(5):41–60MathSciNetCrossRefGoogle Scholar
  19. Epstein JM (2008) Why model? J Artif Soc Soc Simul 11(4).
  20. Epstein JM, Axtell RL (1996) Growing artificial societies: social science from the bottom up. Brookings Institution Press/MIT Press, Cambridge, MAGoogle Scholar
  21. Fensel D (2001) Ontologies: a silver bullet for knowledge management and electronic commerce. Springer, BerlinzbMATHGoogle Scholar
  22. Galán JM, Izquierdo LR (2005) Appearances can be deceiving: lessons learned re-implementing Axelrod’s ‘Evolutionary approach to norms’. J Artif Soc Soc Simulat 8(3).
  23. Galán JM et al (2009) Errors and artefacts in agent-based modelling. J Artif Soc Soc Simulat 12(1).
  24. Gilbert N (1999) Simulation: a new way of doing social science. Am Behav Sci 42(10):1485–1487Google Scholar
  25. Gilbert N (2007) Agent-based models. Sage, LondonGoogle Scholar
  26. Gilbert N, Terna P (2000) How to build and use agent-based models in social science. Mind Soc 1(1):57–72CrossRefGoogle Scholar
  27. Gilbert N, Troitzsch KG (1999) Simulation for the social scientist. Open University Press, BuckinghamGoogle Scholar
  28. Gotts NM, Polhill JG, Adam WJ (2003) Simulation and analysis in agent-based modelling of land use change. In: Online proceedings of the first conference of the European Social Simulation Association, Groningen, 18–21 Sept 2003.
  29. Gruber TR (1993) A translation approach to portable ontology specifications. Knowl Acquis 5(2):199–220CrossRefGoogle Scholar
  30. Hare M, Deadman P (2004) Further towards a taxonomy of agent-based simulation models in environmental management. Math Comput Simulat 64(1):25–40MathSciNetzbMATHCrossRefGoogle Scholar
  31. Hernández C (2004) Herbert A Simon, 1916–2001, y el Futuro de la Ciencia Económica. Revista Europea De Dirección y Economía De La Empresa 13(2):7–23Google Scholar
  32. Heywood JG, Masuda K, Rautmann R, Solonnikov VA (eds) (1990) The Navier–Stokes equations: theory and numerical methods. In: Proceedings of a conference held at Oberwolfach, FRG, 18–24, Sept 1988 (Lecture notes in mathematics), vol 1431. Springer, BerlinGoogle Scholar
  33. Holland JH, Miller JH (1991) Artificial adaptive agents in economic theory. Am Econ Rev 81(2):365–370Google Scholar
  34. Izquierdo LR, Polhill JG (2006) Is your model susceptible to floating point errors? J Artif Soc Soc Simulat 9(4).
  35. Kleijnen JPC (1995) Verification and validation of simulation models. Eur J Oper Res 82(1):145–162zbMATHCrossRefGoogle Scholar
  36. Kleindorfer GB, O’Neill L, Ganeshan R (1998) Validation in simulation: various positions in the philosophy of science. Manage Sci 44(8):1087–1099zbMATHCrossRefGoogle Scholar
  37. Klemm K, Eguíluz V, Toral R, San Miguel M (2003a) Role of dimensionality in Axelrod’s model for the dissemination of culture. Phys A 327:1–5MathSciNetzbMATHCrossRefGoogle Scholar
  38. Klemm K, Eguíluz V, Toral R, San Miguel M (2003b) Global culture: a noise-induced transition in finite systems. Phys Rev E 67(4):045101CrossRefGoogle Scholar
  39. Klemm K, Eguíluz V, Toral R, San Miguel M (2003c) Nonequilibrium transitions in complex networks: a model of social interaction. Phys Rev E 67(2):026120CrossRefGoogle Scholar
  40. Klemm K, Eguíluz V, Toral R, San Miguel M (2005) Globalization, polarization and cultural drift. J Econ Dyn Control 29(1–2):321–334zbMATHCrossRefGoogle Scholar
  41. Kluver J, Stoica C (2003) Simulations of group dynamics with different models. J Artif Soc Soc Simulat 6(4).
  42. Leombruni R, Richiardi M (2005) Why are economists sceptical about agent-based simulations? Phys A 355:103–109MathSciNetCrossRefGoogle Scholar
  43. Moss S (2001) Game theory: limitations and an alternative. J Artif Soc Soc Simulat 4(2).
  44. Moss S (2002) Agent based modelling for integrated assessment. Integr Assess 3(1):63–77CrossRefGoogle Scholar
  45. Moss S, Edmonds B, Wallis S (1997) Validation and verification of computational models with multiple cognitive agents (Report no. 97–25). Centre for Policy Modelling, Manchester.
  46. Ostrom T (1988) Computer simulation: the third symbol system. J Exp Soc Psychol 24(5):381–392CrossRefGoogle Scholar
  47. Parunak HVD, Savit R, Riolo RL (1998) Agent-based modeling vs. equation-based modeling: a case study and users’ guide. In: Sichman JS, Conte R, Gilbert N (eds) Multi-agent systems and agent-based simulation, vol 1534, Lecture notes in artificial intelligence. Springer, Berlin, pp 10–25CrossRefGoogle Scholar
  48. Pavón J, Gómez-Sanz J (2003) Agent oriented software engineering with INGENIAS. In: Marik V, Müller J, Pechoucek M (eds) Multi-agent systems and applications III, 3rd international central and eastern European conference on multi-agent systems, CEEMAS 2003 (Lecture notes in artificial intelligence), vol 2691. Springer, Berlin, pp 394–403Google Scholar
  49. Pignotti E, Edwards P, Preece A, Polhill JG, Gotts NM (2005) Semantic support for computational land-use modelling. In: Proceedings of the 5th international symposium on cluster computing and the grid (CCGRID 2005). IEEE Press, Piscataway, pp 840–847Google Scholar
  50. Polhill JG, Gotts NM (2006) A new approach to modelling frameworks. In: Proceedings of the first world congress on social simulation, vol 1. Kyoto, 21–25 Aug 2006, pp 215–222Google Scholar
  51. Polhill JG, Izquierdo LR (2005) Lessons learned from converting the artificial stock market to interval arithmetic. J Artif Soc Soc Simulat 8(2).
  52. Polhill JG, Izquierdo LR, Gotts NM (2005) The ghost in the model (and other effects of floating point arithmetic). J Artif Soc Soc Simulat 8(1).
  53. Polhill JG, Izquierdo LR, Gotts NM (2006) What every agent based modeller should know about floating point arithmetic. Environ Model Software 21(3):283–309CrossRefGoogle Scholar
  54. Riolo RL, Cohen MD, Axelrod RM (2001) Evolution of cooperation without reciprocity. Nature 411:441–443CrossRefGoogle Scholar
  55. Sakoda JM (1971) The checkerboard model of social interaction. J Math Sociol 1(1):119–132CrossRefGoogle Scholar
  56. Salvi R (2002) The Navier–Stokes equation: theory and numerical methods, Lecture notes in pure and applied mathematics. Marcel Dekker, New YorkzbMATHGoogle Scholar
  57. Sansores C, Pavón J (2005) Agent-based simulation replication: a model driven architecture approach. In: Gelbukh AF, de Albornoz A, Terashima-Marín H (eds) Proceedings of MICAI 2005: advances in artificial intelligence, 4th Mexican international conference on artificial intelligence, Monterrey, 14–18 Nov 2005. Lecture notes in computer science, vol 3789. Springer, Berlin, pp 244–253Google Scholar
  58. Sansores C, Pavón J, Gómez-Sanz J (2006) Visual modeling for complex agent-based simulation systems. In: Sichman JS, Antunes L (eds) Multi-agent-based simulation VI, international workshop, MABS 2005, Utrecht, 25 July 2005, Revised and invited papers. Lecture notes in computer science, vol 3891. Springer, Berlin, pp 174–189Google Scholar
  59. Sargent RG (2003) Verification and validation of simulation models. In: Chick S, Sánchez PJ, Ferrin D, Morrice DJ (eds) Proceedings of the 2003 winter simulation conference. IEEE, Piscataway, pp 37–48Google Scholar
  60. Schelling TC (1971) Dynamic models of segregation. J Math Sociol 1(2):47–186CrossRefGoogle Scholar
  61. Schelling TC (1978) Micromotives and macrobehavior. Norton, New YorkGoogle Scholar
  62. Schmeiser BW (2001) Some myths and common errors in simulation experiments. In: Peters BA, Smith JS, Medeiros DJ, Rohrer MW (eds) Proceedings of the winter simulation conference, vol 1. Arlington, 9–12 Dec 2001, pp 39–46Google Scholar
  63. Takadama K, Suematsu YL, Sugimoto N, Nawa NE, Shimohara K (2003) Cross-element validation in multiagent-based simulation: switching learning mechanisms in agents. J Artif Soc Soc Simulat 6(4).
  64. Taylor AJ (1983) The verification of dynamic simulation models. J Oper Res Soc 34(3):233–242Google Scholar
  65. Xu J, Gao Y, Madey G (2003) A docking experiment: swarm and repast for social network modeling. In: seventh annual swarm researchers conference (SwarmFest 2003), 13–15 Apr 2003, Notre DameGoogle Scholar
  66. Yilmaz L (2006) Validation and verification of social processes within agent-based computational organization models. Comput Math Organ Theory 12(4):283–312MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Departamento de Ingeniería CivilUniversidad de BurgosBurgosSpain
  2. 2.Departamento de Ingeniería CivilUniversidad de BurgosValladolidSpain

Personalised recommendations