Skip to main content
SpringerLink
Log in

Validating Simulation Models: A General Framework and Four Applied Examples

  • Published:
Computational Economics Aims and scope Submit manuscript

Abstract

This paper provides a framework for discussing the empirical validation of simulation models of market phenomena, in particular of agent-based computational economics models. Such validation is difficult, perhaps because of their complexity; moreover, simulations can prove existence, but not in general necessity. The paper highlights the Energy Modeling Forum’s benchmarking studies as an exemplar for simulators. A market of competing coffee brands is used to discuss the purposes and practices of simulation, including explanation. The paper discusses measures of complexity, and derives the functional complexity of an implementation of Schelling’s segregation model. Finally, the paper discusses how courts might be convinced to trust simulation results, especially in merger policy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Arthur, W. B. (2006). Out-of-equilibrium economics and agent-based modeling. In Tesfatsion, Judd, K.L. (2006), pp. 1551–1564.

  • Axelrod, R. (2006). Simulation in the social sciences, in Rennard (2006), pp. 90–100.

  • Axtell R., Axelrod R., Epstein J., Cohen M.D. (1996) Aligning simulation models: A case study and results. Computational and Mathematical Organization Theory 1: 123–141

    Article  Google Scholar 

  • Barreteau, O., and others (2003). Our companion modelling approach. Journal of Artificial Societies and Social Simulation, 6(1), http://jasss.soc.surrey.ac.uk/6/2/1.html (accessed 2006/07/30).

  • Bar-Yam, Y. (1997). Dynamics of complex systems. New York: Perseus Press. http://www.necsi.org/publications/dcs.

  • Bar-Yam, Y. (2003). Unifying principles in complex systems. In M.C. Roco & W.S. Bainbridge (Eds.), Converging technologies for improving human performance: Nanotechnology, biotechnology, information technology and cognitive science (pp. 380–409). New York: Springer. http://www.necsi.org/projects/yaneer/ComplexSystems.pdf (accessed 2006/07/30)

  • Black B., Ayala F.J., Saffran-Brinks C. (1994) Science and the law in the wake of Daubert a new search for scientific knowledge. Texas Law Review 72: 715–751

    Google Scholar 

  • Brenner, T., & Werker, C. (2006). A practical guide to inference in simulation models, Max Planck Institute of Economics, Evolutionary Economics Group, Jena, #0602. https://papers.econ.mpg.de/evo/discussionpapers/2006-02.pdf (accessed 2006/06/23).

  • Bunn D.W., Oliveira F.S. (2003) Evaluating individual market power in electricity markets via agent-based simulation. Annals of Operations Research 121: 57–77

    Article  Google Scholar 

  • Burton R.M. (2003) Computational laboratories for Organizational Science: Questions, validity and docking. Computational & Mathematical Organizational Theory 9: 91–108

    Article  Google Scholar 

  • Cook, C. (2006). Home page: The schelling segregation model demo http://www.econ.iastate.edu/tesfatsi/demos/schelling/schellhp.htm (accessed 2006/07/31).

  • Courtois P.-J. (1985) On time and space decomposition of complex structures. Communications of the ACM 28(6): 590–603

    Article  Google Scholar 

  • Durlauf S. (2005) Complexity and empirical economics. The Economic Journal 115(June): F225–F243

    Article  Google Scholar 

  • Epstein, J. M. (2006). Generative social science: Studies in agent-based Computational modeling. Princeton: P.U.P.

  • Fagiolo, G., Windrum, P., & Moneta, A. (2006). Empirical validation of agent-based models: A critical survey, LEM working paper 2006/14, Pisa, Italy: Laboratory of Economics and Management, Sant’Anna School of Advanced Studies, May. http://www.lem.sssup.it/WPLem/files/2006-14.pdf (accessed 2006/07/30).

  • Forrester J.W. (1961) Industrial dynamics. MIT Press, Camb

    Google Scholar 

  • Frankel, D. M., & Volij, O. (2005). Measuring segregation, mimeo, Iowa State, http://www.econ.iastate.edu/faculty/frankel/segindex_all20.pdf (accessed 2006/07/31)

  • Friedman, M. (1953). Essays in positive economics. University of Chicago Press.

  • Froeb L., Hosken D., Pappalardo J. (2004) Economic research at the FTC: Information, retrospectives and retailing. Review of Industrial Organization 25: 353–374

    Article  Google Scholar 

  • Gilbert N., Troitzsch K.G. (2005) Simulation for the Social Scientist, (2nd ed.). Open University Press, Buckingham

    Google Scholar 

  • Gutowitz, H. (1990). Cellular automata: Theory and experiment. In Proceedings of a workshop sponsored by The Center for Nonlinear Studies. Los Alamos (pp. 7–14). The Hague: North-Holland.

  • Haefner J.W. (2005) Modeling biological systems: Principles and applications (2nd ed.). Springer, New York

    Google Scholar 

  • Huntington H.G., Weyant J.P., Sweeney J.L. (1982) Modeling for insights, not numbers: The experiences of the Energy Modeling Forum. OMEGA: The International Journal of the Management Sciences 10(5): 449–462

    Article  Google Scholar 

  • Judd, K. L. (2006). Computationally intensive analyses in economics, in Tesfatsion and Judd (2006), pp. 881–893.

  • Kaldor N. (1961) Capital accumulation and economic growth. In: Lutz F.A., Hague D.C. (eds). The theory of capital. Macmillan, London, pp. 177–222

    Google Scholar 

  • Klein E.E, Herskovitz P. (2005) Philosophical foundations of computer simulation validation. Simulation & Gaming 36: 303–329

    Article  Google Scholar 

  • Leombruni R., Richiardi M. (2005) Why are economists sceptical about agent-based simulations?. Physica A 355: 103–109

    Article  Google Scholar 

  • Leombruni, R., Richiardi, M., Saam, N. J., & Sonnessa, M. (2006). A common protocol for agent-based social simulation. Journal of Artificial Societies and Social Simulation, 9(1). http://jasss.soc.surrey.ac.uk/9/1/15.html (accessed 2006/06/22).

  • Malerba F., Nelson R., Orsenigo L., Winter S. (1999) History-friendly models of industry evolution: The computer industry. Industrial and Corporate Change 8(1): 3–40

    Article  Google Scholar 

  • Mankin, J. B., O’Neill, R. V., Shugart, H. H., & Rust, B. W. (1977). The importance of validation in ecosystem analysis. In G.S. Innis (Ed.), New directions in the analysis of ecological systems, part 1 simulation council proceedings series (Vol. 5: pp. 63–71). California: Simulation Councils, La Jolla. Reprinted in H.H. Shugart and R.V. O’Neill, eds. Systems ecology Dowden, Hutchinson and Ross, Stroudsburg, Pennsylvania, 1979, pp. 309–317.

  • Manson S.M. (2002) Validation and verification of multi-agent systems. In: Janssen M.A. (eds). Complexity and ecosystem management. Edward Elgar, Cheltenham

    Google Scholar 

  • Marks R.E. (1992) Breeding optimal strategies: Optimal behaviour for oligopolists. Journal of Evolutionary Economics 2: 17–38

    Article  Google Scholar 

  • Marks, R. E. (2006). Market design using agent-based models, in Tesfatsion and Judd (2006), pp. 1339–1380.

  • Marks, R. E., Midgley, D. F., & Cooper, L. G. (2006). Co-evolving better strategies in oligopolistic price wars, in Rennard (2006), pp. 806–821.

  • Midgley D.F., Marks R.E., Cooper L.G. (1997) Breeding competitive strategies. Management Science 43(3): 257–275

    Google Scholar 

  • Midgley, D. F., Marks, R. E., & Kunchamwar, D. (2007). The building and assurance of agent-based models: An example and challenge to the field. Journal of Business Research, 60, 884–893. http://www.agsm.edu.au/~bobm/papers/Midgley-Marks-Kunchamwar.pdf.

  • Miller J.H. (1996) The coevolution of automata in the repeated prisoner’s dilemma. Journal of Economic Behavior and Organizations 29: 87–113

    Article  Google Scholar 

  • Miller J.H. (1998) Active nonlinear tests (ANTs) of complex simulations models. Management Science 44(6): 820–830

    Article  Google Scholar 

  • Moss S., Edmonds B. (2005) Sociology and simulation: Statistical and quantitative cross-validation. American Journal of Sociology 110(4): 1095–1131

    Article  Google Scholar 

  • Rennard J.-P., eds. (2006) Handbook on research on nature-inspired computing for economics and management. Idea Group, Hershey, PA

    Google Scholar 

  • Resnick, M. (1994). Turtles, termites, and traffic jams: Explorations in massively parallel microworlds. MIT Press.

  • Rosen R. (1985) Anticipatory systems: Philosophical, mathematical, and methodological foundations. Pergamon, Oxford

    Google Scholar 

  • Rubinstein, A. (1998). Modeling bounded rationality. MIT Press.

  • Schelling T.C. (1971) Dynamic models of segregation. Journal of Mathematical Psychology 1: 143–186

    Google Scholar 

  • Schelling T.C. (1978) Micromotives and macrobehavior. Norton, New York

    Google Scholar 

  • Schelling, T. C. (2006). Some fun, thirty-five years ago, in Tesfatsion & Judd (2006), pp. 1639–1644.

  • Shannon C.E. (1948) A mathematical theory of communication. Bell Systems Technical Journal 27: 379–423 July, 623 August.

    Google Scholar 

  • Sharpe, W. F. (2007). APSIM: An asset price and portfolio choice simulator. In Investors and markets: Portfolio choices, asset prices and investment advice. Princeton University Press. http://www.stanford.edu/~wfsharpe/apsim/apsim.pdf (accessed 2006/07/30).

  • Simon H.A., Ando A. (1961) Aggregation of variables in dynamic systems. Econometrica 29: 111–138

    Article  Google Scholar 

  • Simon H.A., Wallach D. (1999) Cognitive modeling in perspective. Kognitionswissenschaft 8: 1–4

    Article  Google Scholar 

  • Stephens C.A., Graham A.K., Lyneis J.M. (2005) System dynamics modeling in the legal arena: Meeting the challenges of expert witness admissibility. System Dynamics Review 27(2): 95–122

    Article  Google Scholar 

  • Tesfatsion, L. (2006). Conducting experiments with Chris Cook’s Schelling Demo, mimeo, Iowa State, http://www.econ.iastate.edu/classes/econ308/tesfatsion/segex2.VIITrento.pdf (accessed 2007/05/31).

  • Tesfatsion, L., & Judd, K. L. (Eds.) (2006). In K. Arrow & M. D. Intriligator (Eds.), Handbook of computational economics, volume 2: Agent-based computational economics in the series handbooks in economics. Amsterdam: Elsevier Science.

  • Tukey J.W. (1962) The future of data analysis. Annals of Mathematical Statistics 33: 13–14

    Google Scholar 

  • van Damme, E. E. C., & Pinkse, J. (2005). Merger simulation analysis: An academic perspective, TILEC Discussion paper No. 2005-013 http://ssrn.com/abstract=869737 (accessed 2006/07/30).

  • Vriend N. (2000) An illustration of the essential difference between individual and social learning and its consequences for computational analyses. Journal of Economic Dynamics and Control 24: 1–19

    Article  Google Scholar 

  • Watson J.D., Crick F.H.C. (1953) Molecular structure of nucleic acids: A structure of deoxyribose nucleic acid. Nature 4356: 737–738, April 25.

    Article  Google Scholar 

  • Werden G.J., Froeb L.M., Scheffman D.T. (2004) A Daubert discipline for merger simulation. Antitrust Magazine 18(3): 89–95

    Google Scholar 

  • Weyant, J. P., & Hill, J. (1999). Introduction and overview, The costs of the Kyoto protocol: A multi-model evaluation, The Energy Journal, 20 (Special Issue) vii–xliv, May.

  • Wilensky, U. (1998). NetLogo segregation model. Center for connected learning and computer-based modeling, Northwestern University, Evanston, IL. http://ccl.northwestern.edu/netlogo/models/Segregation.

  • Woodbury, J. R. (Ed.). (2004). Whither merger simulation? The Antitrust Source, May, http://www.abanet.org/antitrust/source/05-04/whither.pdf (accessed 2006/07/30).

Download references

Author information

Authors and Affiliations

  1. Australian Graduate School of Management, Australian School of Business, The University of New South Wales, Sydney, NSW, 2052, Australia

    Robert Ernest Marks

Authors
  1. Robert Ernest Marks
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert Ernest Marks.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Marks, R.E. Validating Simulation Models: A General Framework and Four Applied Examples. Comput Econ 30, 265–290 (2007). https://doi.org/10.1007/s10614-007-9101-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10614-007-9101-7

Keywords

JEL Classifications

Search

Navigation