Since the survey by Windrum et al. (Journal of Artificial Societies and Social Simulation 10:8, 2007), research on empirical validation of agent-based models in economics has made substantial advances, thanks to a constant flow of high-quality contributions. This Chapter attempts to take stock of such recent literature to offer an updated critical review of the existing validation techniques. We sketch a simple theoretical framework that conceptualizes existing validation approaches, which we examine along three different dimensions: (i) comparison between artificial and real-world data; (ii) calibration and estimation of model parameters; and (iii) parameter space exploration. Finally, we discuss open issues in the field of ABM validation and estimation. In particular, we argue that more research efforts should be devoted toward advancing hypothesis testing in ABM, with specific emphasis on model stationarity and ergodicity.
- Agent-based models
- Sensitivity analysis
- Parameter space exploration
This is a preview of subscription content, access via your institution.
Tax calculation will be finalised at checkout
Purchases are for personal use onlyLearn about institutional subscriptions
The validation process might also take different perspectives. In particular, as reported by Burton and Obel (1995), the model’s assumptions and abstractions have to be judged accordingly with the model’s purpose. In this paper, we mostly focus on validation of policy-oriented, descriptive agent-based economic and financial models.
In that there is a major departure with respect to neoclassical models, where the (representative) agent has axiomatic preferences and maximizes some smooth objective function with an easily computable bliss point.
This is also one of the critiques that is usually addressed to ACE. Since ABMs do not stick to some generally accepted axiomatic rule of behavior, they introduce discretionary choices that the modeler shall take. We will see how practitioners have coped with this issue in Sect. 126.96.36.199. A possible solution to discipline the construction phase of an ABMs has been put forward by Grimm et al. (2006) and is called the ODD protocol (from “Overview, Design concepts, and Details”).
In Sect. 31.4.2, we will discuss the tools available for the verification and validation of ABMs.
One can also study the basins of attraction of the dynamical system to study the robustness with respect to initial conditions.
In agent- based modeling, some of the standard validity aspects that are relevant in many fields of numerical simulations are not an issue; for example, systems are always represented in discrete time and, hence, discretization errors are not possible. Further, low emphasis is usually posed on code verification.
See also Secchi and Seri (2017) on the issue of selecting the number of times a computational model should be run.
Level 0 models can be somehow accepted if their aim is merely exploratory rather than descriptive.
See, for example, Dosi et al. (2010, 2013, 2015, 2016a) for replication of business cycle and growth stylized facts; Dosi et al. (2017a) for accounting of labor-market micro and macro regularities; Popoyan et al. (2017) for the reproduction of many credit and interbank market properties; Lamperti et al. (2018a, b) for capturing coevolution of economic fundamentals with energy and emission quantities; Pellizzari and Dal Forno (2007); Leal et al. (2016) for simulating financial market booms and busts.
For a discussion of calibration and testability, see Chap. 40 by Frisch in this volume.
See also Chap. 12 by Marks in this volume.
VAR-LiNGAM stands for Vector Autoregressive Linear Non-Gaussian Acyclic Model.
Alfarano, S., Lux, T., & Wagner, F. (2005). Estimation of agent-based models: The case of an asymmetric herding model. Computational Economics, 26(1), 19–49.
Alfarano, S., Lux, T., & Wagner, F. (2006). Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data. Physica A: Statistical Mechanics and its Applications, 370(1), 38–42.
Anufriev, M., Bao, T., & Tuinstra, J. (2016). Microfoundations for switching behavior in heterogeneous agent models: An experiment. Journal of Economic Behavior & Organization, 129(C):74–99.
Anufriev, M., & Hommes, C. (2012). Evolutionary selection of individual expectations and aggregate outcomes in asset pricing experiments. American Economic Journal: Microeconomics, 4(4), 35–64.
Assenza, T., Delli Gatti, D., & Grazzini, J. (2015). Emergent dynamics of a macroeconomic agent based model with capital and credit. Journal of Economic Dynamics and Control, 50(C):5–28.
Assenza, T., Heemeijer, P., Hommes, C., & Massaro, D. (2013). Individual expectations and aggregate macro behavior. Tinbergen Institute Discussion Papers 13-016/II, Tinbergen Institute.
Axelrod, R. (1997). The complexity of cooperation: Agent-based models of competition and collaboration. Princeton University Press.
Axtell, R. L., & Epstein, J. M. (1994). Agent-based modeling: Understanding our creations. The Bulletin of the Santa Fe Institute, 9(2), 28–32.
Barde, S. (2016a). Direct comparison of agent-based models of herding in financial markets. Journal of Economic Dynamics and Control, 73(C):329–353.
Barde, S. (2016b). A practical, accurate, information criterion for nth order markov processes. Computational Economics, 1–44.
Barde, S., & van der Hoog, S. (2017). An empirical validation protocol for large-scale agent-based models. Studies in Economics 1712, School of Economics, University of Kent.
Bargigli, L., Riccetti, L., Russo, A., & Gallegati, M. (2016). Network calibration and metamodeling of a financial accelerator agent based model. Technical report, Università Politecnica delle Marche.
Battiston, S., Farmer, J. D., Flache, A., Garlaschelli, D., Haldane, A. G., Heesterbeek, H., et al. (2016). Complexity theory and financial regulation. Science, 351(6275), 818–819.
Bianchi, C., Cirillo, P., Gallegati, M., & Vagliasindi, P. (2007). Validating and calibrating agent-based models: A case study. Computational Economics, 30, 245–264.
Bianchi, C., Cirillo, P., Gallegati, M., & Vagliasindi, P. (2008a). Validation in agent-based models: An investigation on the CATS model. Journal of Economic Behavior & Organization, 67, 947–964.
Bianchi, C., Cirillo, P., Gallegati, M., & Vagliasindi, P. A. (2008b). Validation in agent-based models: An investigation on the CATS model. Journal of Economic Behavior & Organization, 67(3–4), 947–964.
Boswijk, H. P., Hommes, C. H., & Manzan, S. (2007). Behavioral heterogeneity in stock prices. Journal of Economic Dynamics and Control, 31(6), 1938–1970.
Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. CRC Press.
Brenner, T., & Werker, C. (2007). A taxonomy of inference in simulation models. Computational Economics, 30(3), 227–244.
Brock, W. A. (1999). Scaling in economics: A reader’s guide. Industrial and Corporate Change, 8(3), 409–446.
Brock, W. A., & Hommes, C. H. (1997). A rational route to randomness. Econometrica, 65(5), 1059–1095.
Brock, W. A., & Hommes, C. H. (1998). Heterogeneous beliefs and routes to chaos in a simple asset pricing model. Journal of Economic Dynamics and Control, 22(8–9), 1235–1274.
Burton, R. M., & Obel, B. (1995). The validity of computational models in organization science: From model realism to purpose of the model. Computational & Mathematical Organization Theory, 1(1), 57–71.
Canova, F., & Sala, L. (2009). Back to square one: Identification issues in DSGE models. Journal of Monetary Economics, 56(4), 431–449.
Chen, S.-H., Chang, C.-L., & Du, Y.-R. (2012). Agent-based economic models and econometrics. The Knowledge Engineering Review, 27(2), 187–219.
Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). ACM.
Chiarella, C., He, X.-Z., & Zwinkels, R. C. (2014). Heterogeneous expectations in asset pricing: Empirical evidence from the S&P500. Journal of Economic Behavior & Organization, 105(C):1–16.
Ciarli, T. (2012). Structural interactions and long run growth: An application of experimental design to agent-based models. Revue de l’OFCE, 124, 295–345.
Dawid, H. & Delli Gatti, H. (2018). Chapter 2 - agent-based macroeconomics. In C. Hommes & B. LeBaron (Eds.), Handbook of computational economics (Vol. 4, pp. 63–156). Elsevier.
Dawid, H., Harting, P., van der Hoog, S., & Neugart, M. (2016). A heterogeneous agent macroeconomic model for policy evaluation: Improving transparency and reproducibility.
de Jong, E., Verschoor, W. F., & Zwinkels, R. C. (2010). Heterogeneity of agents and exchange rate dynamics: Evidence from the EMS. Journal of International Money and Finance, 29(8), 1652–1669.
Del Negro, M., & Schorfheide, F. (2006). How good is what you’ve got? DSGE-VAR as a toolkit for evaluating DSGE models. Economic Review, (Q 2):21–37.
Dieci, R., & He, X.-Z. (2018). Chapter 5 - heterogeneous agent models in finance. In C. Hommes & B. LeBaron (Eds.), Handbook of computational economics (Vol. 4, pp. 257–328). Elsevier.
Dosi, G., Fagiolo, G., Napoletano, M., & Roventini, A. (2013). Income distribution, credit and fiscal policies in an agent-based keynesian model. Journal of Economic Dynamics and Control, 37(8), 1598–1625.
Dosi, G., Fagiolo, G., Napoletano, M., Roventini, A., & Treibich, T. (2015). Fiscal and monetary policies in complex evolving economies. Journal of Economic Dynamics and Control, 52, 166–189.
Dosi, G., Fagiolo, G., & Roventini, A. (2010). Schumpeter meeting keynes: A policy-friendly model of endogenous growth and business cycles. Journal of Economic Dynamics and Control, 34(9), 1748–1767.
Dosi, G., Napoletano, M., Roventini, A., & Treibich, T. (2016a). Micro and macro policies in the Keynes+Schumpeter evolutionary models. Journal of Evolutionary Economics, forthcoming, 1–28.
Dosi, G., Pereira, M., Roventini, A., & Virgilito, M. E. (2017a). When more flexibility yields more fragility: The microfoundations of keynesian aggregate unemployment. Journal of Economic Dynamics & Control, 81, 162–186.
Dosi, G., Pereira, M. C., Roventini, A., & Virgillito, M. E. (2016b). The effects of labour market reforms upon unemployment and income inequalities: An agent based model (LEM Working Papers 2016/27). Scuola Superiore Sant’Anna.
Dosi, G., Pereira, M. C., Roventini, A., & Virgillito, M. E. (2017b). Causes and consequences of hysteresis: Aggregate demand, productivity and employment (LEM Working Papers 2017/07). Scuola Superiore Sant’Anna.
Dosi, G., Pereira, M. C., & Virgillito, M. E. (2017c). On the robustness of the fat-tailed distribution of firm growth rates: A global sensitivity analysis. Journal of Economic Interaction and Coordination, 1–21.
Epstein, J. M., & Axtell, R. (1996). Growing artificial societies: Social science from the bottom up. Brookings Institution Press.
Fabretti, A. (2013). On the problem of calibrating an agent based model for financial markets. Journal of Economic Interaction and Coordination, 8(2), 277–293.
Fagiolo, G., & Dosi, G. (2003). Exploitation, exploration and innovation in a model of endogenous growth with locally interacting agents. Structural Change and Economic Dynamics, 14, 237–273.
Fagiolo, G., & Roventini, A. (2012). Macroeconomic policy in DSGE and agent-based models. Revue de l’OFCE, 0(5), 67–116.
Fagiolo, G., & Roventini, A. (2017). Macroeconomic policy in DSGE and agent-based models redux: New developments and challenges ahead. Journal of Artificial Societies and Social Simulation, 20(1).
Farmer, D. J., & Foley, D. (2009). The economy needs agent-based modelling. Nature, 460, 685–686.
Fernández-Villaverde, J., Ramírez, J. F. R., & Schorfheide, F. (2016). Solution and Estimation Methods for DSGE Models (NBER Working Papers 21862). National Bureau of Economic Research, Inc.
Fernández-Villaverde, J., & Rubio-Ramírez, J. F. (2007). Estimating macroeconomic models: A likelihood approach. Review of Economic Studies, 74(4), 1059–1087.
Franke, R. (2009). Applying the method of simulated moments to estimate a small agent-based asset pricing model. Journal of Empirical Finance, 16(5), 804–815.
Franke, R., & Westerhoff, F. (2012). Structural stochastic volatility in asset pricing dynamics: Estimation and model contest. Journal of Economic Dynamics and Control, 36(8), 1193–1211.
Gaffeo, E., Delli Gatti, D., Desiderio, S., & Gallegati, M. (2008). Adaptive microfoundations for emergent macroeconomics. Eastern Economic Journal, 34(4), 441–463.
Goldbaum, D., & Mizrach, B. (2008). Estimating the intensity of choice in a dynamic mutual fund allocation decision. Journal of Economic Dynamics and Control, 32(12), 3866–3876.
Gourieroux, C., Monfort, A., & Renault, E. (1993). Indirect Inference. Journal of Applied Econometrics, 8(S):85–118.
Grazzini, J., & Richiardi, M. (2015). Estimation of ergodic agent-based models by simulated minimum distance. Journal of Economic Dynamics and Control, 51(C):148–165.
Grazzini, J., Richiardi, M. G., & Tsionas, M. (2017). Bayesian estimation of agent-based models. Journal of Economic Dynamics and Control, 77(C), 26–47.
Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., et al. (2006). A standard protocol for describing individual-based and agent-based models. Ecological modelling, 198(1–2), 115–126.
Grimm, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W. M., Railsback, S. F., et al. (2005). Pattern-oriented modeling of agent-based complex systems: Lessons from ecology. Science, 310(5750), 987–991.
Guerini, M. (2013). Is the friedman rule stabilizing? Some unpleasant results in a heterogeneous expectations framework. Technical report, Department of Economics and Finance Working Papers, Unicatt, Milan.
Guerini, M., & Moneta, A. (2017). A method for agent-based models validation. Journal of Economic Dynamics and Control.
Guerini, M., Napoletano, M., & Roventini, A. (2017). No man is an island: The impact of heterogeneity and local interactions on macroeconomic dynamics. Economic Modelling.
Hansen, L. P., & Heckman, J. J. (1996). The empirical foundations of calibration. The Journal of Economic Perspectives, 10(1), 87–104.
Hassan, S., Pavon, J., & Gilbert, N. (2008). Injecting data into simulation: Can agent-based modelling learn from microsimulation. In World Congress of Social Simulation.
Heine, B.-O., Meyer, M., & Strangfeld, O. (2005). Stylised facts and the contribution of simulation to the economic analysis of budgeting. Journal of Artificial Societies and Social Simulation, 8(4).
Hommes, C. (2011). The heterogeneous expectations hypothesis: Some evidence from the lab. Journal of Economic Dynamics and Control, 35(1), 1–24.
Hommes, C. (2013). Behavioral rationality and heterogeneous expectations in complex economic systems. Number 9781107564978 in Cambridge Books. Cambridge University Press.
Hyvarinen, A., Zhang, K., Shimizu, S., & Hoyer, P. O. (2010). Estimation of a structural vector autoregression model using non-gaussianity. Journal of Machine Learning Research, 11, 1709–1731.
Johansen, S., & Juselius, K. (1990). Maximum likelihood estimation and inference on cointegration. With application to the demand for money. Oxford Bullettin of Economics and Statistics, 52, 169–210.
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263–291.
Kirman, A. (1991). Epidemics of opinion and speculative bubbles in financial markets. In M. Taylor (Ed.), Money and financial markets (pp. 354–368). Blackwell.
Krige, D. G. (1951). A statistical approach to some basic mine valuation problems on the witwatersrand. Journal of the Southern African Institute of Mining and Metallurgy, 52(6), 119–139.
Kukacka, J., & Barunik, J. (2017). Estimation of financial agent-based models with simulated maximum likelihood. Journal of Economic Dynamics and Control, 85(C):21–45.
Lamperti, F. (2018a). Empirical validation of simulated models through the GSL-div: An illustrative application. Journal of Economic Interaction and Coordination, 13(1), 143–171.
Lamperti, F. (2018b). An information theoretic criterion for empirical validation of simulation models. Econometrics and Statistics, 5, 83–106.
Lamperti, F., Dosi, G., Napoletano, M., Roventini, A., & Sapio, A. (2018a). Faraway, so close: Coupled climate and economic dynamics in an agent-based integrated assessment model. Ecological Economics, 150, 315–339.
Lamperti, F., Dosi, G., Napoletano, M., Roventini, A., Sapio, A., et al. (2018b). And then he wasn’t a she: Climate change and green transitions in an agent-based integrated assessment model. Technical report, Laboratory of Economics and Management (LEM), Sant’Anna School of Advanced Studies, Pisa, Italy.
Lamperti, F., Roventini, A., & Sani, A. (2018c). Agent-based model calibration using machine learning surrogates. Journal of Economic Dynamics and Control, 90, 366–389.
Lane, D. A. (1993). Artificial worlds and economics, part II. Journal of Evolutionary Economics, 3(3), 177–197.
Leal, S. J., Napoletano, M., Roventini, A., & Fagiolo, G. (2016). Rock around the clock: An agent-based model of low- and high-frequency trading. Journal of Evolutionary Economics, 26(1), 49–76.
LeBaron, B., & Tesfatsion, L. (2008). Modeling macroeconomies as open-ended dynamic systems of interacting agents. American Economic Review, 98(2), 246–250.
Lee, J.-S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., et al. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4), 4.
Leombruni, R., Richiardi, M., Saam, N. J., & Sonnessa, M. (2006). A common protocol for agent-based social simulation. Journal of Artificial Societies and Social Simulation, 9(1), 15.
Lorscheid, I., Heine, B.-O., & Meyer, M. (2012). Opening the fiblack boxfiof simulations: Increased transparency and effective communication through the systematic design of experiments. Computational and Mathematical Organization Theory, 18(1), 22–62.
Malerba, F., Nelson, R., Orsenigo, L., & Winter, S. (1999). ’History-friendly’ models of industry evolution: The computer industry. Industrial and Corporate Change, 8(1), 3.
Manson, S. (Ed.). (2002). Validation and verification of multi-agent systems, in complexity and ecosystem management. Cheltenham: Edward Elgar.
Marks, R. (2007). Validating simulation models: A general framework and four applied examples. Computational Economics, 30(3), 265–290.
Marks, R. E. (2013). Validation and model selection: Three similarity measures compared. Complexity Economics, 2(1), 41–61.
Marks, R. E. (2018). Pattern-based metrics for validating simulation model output. In C. Beisbart & N. J. Saam (Eds.), Computer simulation validation. Fundamental concepts, methodological frameworks, philosophical perspectives. Springer.
McKay, M. D., Beckman, R. J., & Conover, W. J. (1979). Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2), 239–245.
Metropolis, N., & Ulam, S. (1949). The monte carlo method. Journal of American Statistical Association, 44, 335–341.
Morokoff, W. J., & Caflisch, R. E. (1994). Quasi-random sequences and their discrepancies. SIAM Journal on Scientific Computing, 15(6), 1251–1279.
Paccagnini, A. (2010). DSGE model validation in a bayesian framework: An assessment. MPRA Paper 24509, University Library of Munich, Germany.
Pellizzari, P., & Dal Forno, A. (2007). A comparison of different trading protocols in an agent-based market. Journal of Economic Interaction and Coordination, 2(1), 27–43.
Platt, D., & Gebbie, T. (2016). Can agent-based models probe market microstructure? Papers 1611.08510, arXiv.org.
Popoyan, L., Napoletano, M., & Roventini, A. (2017). Taming macroeconomic instability: Monetary and macro-prudential policy interactions in an agent-based model. Journal of Economic Behavior & Organization, 134(C):117–140.
Recchioni, M. C., Tedeschi, G., & Gallegati, M. (2015). A calibration procedure for analyzing stock price dynamics in an agent-based framework. Journal of Economic Dynamics and Control, 60, 1–25.
Rosen, R. (1985). Anticipatory systems: Philosophical, mathematical, and methodological foundations. Oxford: Pergamon.
Salle, I., & Yıldızoğlu, M. (2014). Efficient sampling and meta-modeling for computational economic models. Computational Economics, 44(4), 507–536.
Schelling, T. C. (1969). Models of segregation. The American Economic Review, 59(2), 488–493.
Schelling, T. C. (1971). Dynamic models of segregation. The Journal of Mathematical Sociology, 1(2), 143–186.
Secchi, D., & Seri, R. (2017). Controlling for false negatives in agent-based models: A review of power analysis in organizational research. Computational and Mathematical Organization Theory, 23(1), 94–121.
Shimizu, S., Hoyer, P. O., Hyvarinen, A., & Kerminen, A. J. (2006). A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7, 2003–2030.
Simon, H. A. (1991). Bounded rationality and organizational learning. Organization Science, 2(1), 125–134.
Spirtes, P., Glymur, C., & Scheines, R. (2000). Causation, prediction, and search. MIT Press.
Tesfatsion, L. (2006). Chapter 16 agent-based computational economics: A constructive approach to economic theory. In Handbook of computational economics, 2 (pp. 831–880).
Thiele, J. C., Kurth, W., & Grimm, V. (2014). Facilitating parameter estimation and sensitivity analysis of agent-based models: A cookbook using NetLogo and R. Journal of Artificial Societies and Social Simulation, 17(3), 11.
Turrell, A. (2016). Agent-based models: Understanding the economy from the bottom up. Quarterly bulletin Q4, Bank of England.
Van Beers, W. C. & Kleijnen, J. P. (2004). Kriging interpolation in simulation: A survey. In Simulation Conference, 2004. Proceedings of the 2004 Winter (vol. 1). IEEE.
Werker, C., & Brenner, T. (2004). Empirical calibration of simulation models 0410. Papers on economics and evolution, Max-Planck-Institut für Ökonomik.
Westerhoff, F. H., & Dieci, R. (2006). The effectiveness of keynes-tobin transaction taxes when heterogeneous agents can trade in different markets: A behavioral finance approach. Journal of Economic Dynamics and Control, 30(2), 293–322.
Windrum, P., Fagiolo, G., & Moneta, A. (2007). Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation, 10(2), 8.
Winker, P., & Gilli, M. (2001). Validation of agent-based models of financial markets. IFAC Proceedings Volumes, 34(20), 401–406.
Winker, P., & Gilli, M. (2004). Applications of optimization heuristics to estimation and modelling problems. Computational Statistics & Data Analysis, 47(2), 211–223.
We gratefully acknowledge the support by the European Union Horizon 2020 research and innovation programme under grant agreement No. 649186 - ISIGrowth. Further, we express our gratitude to Francesca Chiaromonte, Giovanni Dosi, Mauro Napoletano, Marcelo Pereira, Amir Sani, and Maria Enrica Virgillito for helpful comments and discussions on the issues surveyed in this chapter.
Editors and Affiliations
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Fagiolo, G., Guerini, M., Lamperti, F., Moneta, A., Roventini, A. (2019). Validation of Agent-Based Models in Economics and Finance. In: Beisbart, C., Saam, N. (eds) Computer Simulation Validation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-70766-2_31
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70765-5
Online ISBN: 978-3-319-70766-2