Efficient Sampling and Meta-Modeling for Computational Economic Models

Abstract

Extensive exploration of simulation models comes at a high computational cost, all the more when the model involves a lot of parameters. Economists usually rely on random explorations, such as Monte Carlo simulations, and basic econometric modeling to approximate the properties of computational models. This paper aims to provide guidelines for the use of a much more efficient method that combines a parsimonious sampling of the parameter space using a specific design of experiments (DoE), with a well-suited metamodeling method first developed in geostatistics: kriging. We illustrate these guidelines by following them in the analysis of two simple and well known economic models: Nelson and Winter’s industrial dynamics model, and Cournot oligopoly with learning firms. In each case, we show that our DoE experiments can catch the main effects of the parameters on the models’ dynamics with a much lower number of simulations than the Monte-Carlo sampling (e.g. 85 simulations instead of 2,000 in the first case). In the analysis of the second model, we also introduce supplementary numerical tools that may be combined with this method, for characterizing configurations complying with a specific criterion (social optimal, replication of stylized facts, etc.). Our appendix gives an example of the R-project code that can be used to apply this method on other models, in order to encourage other researchers to quickly test this approach on their models.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    See, notably, Besag and Clifford (1991). Applications in economics of this method mostly concern Bayesian estimation of models, see, for instance, Herbst and Schorfheide (2013).

  2. 2.

    More precisely, consider that the null hypothesis is rejected if the observed values of the variable under interest are higher than a given value, at a frequency higher than a given confidence threshold \(\alpha \le 1\). Running simulations until one obtains \(n \cdot \alpha \le n\) observations leading to the rejection of the null hypothesis is shown to have the same power as running \(n\) simulations, and assessing only afterwards whether the null hypothesis has to be rejected or not. The number of simulations can hence be reduced by a factor up to \(\frac{1}{\alpha }\). A small amount of power can further be traded with a larger decrease in the number of simulations, and Silva et al. (2009) provide related estimations of the power loss.

  3. 3.

    Kriging models have been named after Danie G. Krige, a South African mining engineer who developed those models to improve ore evaluation techniques at the Witwatersrand reef complex in South Africa, pioneering the field of geostatistics, see Krige (1951). As for the statistical theory of DoE, it was developed in agriculture in the 1920’s, for real, non-simulated experiments, see Fisher (1935).

  4. 4.

    See, notably, Matheron (1963), Sacks et al. (1989), van Beers and Kleijnen (2004) and Roustant et al. (2010).

  5. 5.

    The estimation of the meta-model is actually done through feasible GLS as the covariance matrix \(C\) is unknown and its parameters have to be estimated, see below.

  6. 6.

    There are two possible triplets of rows (bac, cba, acb) and (abc, cab, bca), each can be permuted in \(3!=6\) different ways, so that one obtains \(2 \times 6=12\) possible configurations.

  7. 7.

    In that case, the meta-model refers to ordinary kriging, and of simple kriging if the mean is known, contrary to universal kriging in the more general case, which is exposed above.

  8. 8.

    Recall that kriging is an exact interpolator, so that the \(R^{2}\) coefficient cannot be computed.

  9. 9.

    In practice, either the factors have a finite set of values, and the ANOVA is performed using the common formula of the multi-variate analysis of variance with discrete factors, see, for instance, Frey and Patil (2002); or the factors are defined over a continuous domain, and the experimental domain has to be discretized in order to apply these formula. In that case, predictions of the response through the meta-model are evaluated over a \(k\)-dimensional grid [see (Welch et al. 1992; Saltelli et al. 1999)].

  10. 10.

    http://ccl.northwestern.edu/netlogo/

  11. 11.

    See also Nelson and Winter (1978) for an extensive presentation and discussion of the model. In this paper, we only use this model as a simple example, in order to apply the method previously developed. We adopt values used in the original model for the parameters that we do not include in our experiments.

  12. 12.

    We also consider a \(10,000\) simulations Monte Carlo sample for robustness checks.

  13. 13.

    R Development Core Team (2013) software can also be used but the package effects, which computes ANOVA marginal effects, is not directly connected to the DiceKriging package, which performs kriging estimation and the modeler has to use the package sensitivity, which delivers less detailed results [see (Roustant et al. 2010)].

  14. 14.

    These figures are built using the principles given in Sub-subsection 2.3.5.

  15. 15.

    We consider that we catch these effects in a robust way if they appear as significant in each of the \(100\) regressions obtained from \(100\) random sets of \(2,000\) simulations.

  16. 16.

    The complete code used in this section is provided in Appendix 4.

  17. 17.

    See Masters (1993) for a general statement, see Yıldızoğlu (2001) and Yıldızoğlu et al. (2012) for the precise description of the learning algorithm.

  18. 18.

    Higher order polynomials would involve too many parameters to be estimated, considering only 33 observations.

  19. 19.

    It should be noted that applying sensitivity analysis to the other forms of kriging identifies the same determinants, which indicates that the overall picture of the meta-model is not sensitive to the specification.

  20. 20.

    See also Salle et al. (2012) for an application of this function to the minimization of a Central Bank’s loss function in a macroeconomic agent-based model.

References

  1. Besag, J., & Clifford, P. (1991). Sequential monte carlo p-values. Biometrika, 78(2), 301–304.

    Article  Google Scholar 

  2. Booth, J., & Butler, R. (1999). An importance sampling algorithm for exact conditional tests in log-linear models. Biometrika, 86(2), 321–332.

    Article  Google Scholar 

  3. Box, G., & Draper, N. (1987). Empirical model building and responses surfaces. New York: Wiley.

    Google Scholar 

  4. Cary, N. (2010). JMP \({\textregistered }\) 9 modeling and multivariate methods. Cary: SAS Institute Inc.

  5. Cioppa, T., (2002). Efficient nearly orthogonal and space-filling experimental designs for high-dimensional complex models. Naval postgraduate school: Doctoral Dissertation in philosophy in operations research.

  6. Durrande, N., Ginsbourger, O., & Roustant, O. (2012). Additive covariance kernels for high-dimensional Gaussian process modeling. Annales de la Faculté des Sciences de Toulouse Tome, 21(3), 481–499.

    Article  Google Scholar 

  7. Fang, K., Lin, D., Winker, P., & Zhang, Y. (2000). Uniform design: theory and application. Technometrics, 42(3), 237–248.

    Article  Google Scholar 

  8. Fisher, R. A. (1935). The design of experiments (9th ed.). New York: Macmillan.

    Google Scholar 

  9. Frey, C., & Patil, S. (2002). Identification and review of sensitivity analysis methods. Risk Analysis, 22(3), 553.

    Article  Google Scholar 

  10. Goupy, J., & Creighton, L. (2007). Introduction to design of experiments with JMP examples (3rd ed.). Cary: SAS Institute Inc.

    Google Scholar 

  11. Herbst, E., & Schorfheide, F. (2013). Sequential monte carlo sampling for DSGE models, working paper 19152. National Bureau of Economic Research.

  12. Iman, R., & Helton, J. (1988). An investigation of uncertainty and sensitivity analysis techniques for computer models. Risk Analysis, 8, 71–90.

    Article  Google Scholar 

  13. Jeong, S., Murayama, M., & Yamamoto, K. (2005). Efficient optimization design method using kriging model. Journal of Aircraft, 42, 413–420.

    Article  Google Scholar 

  14. Jourdan, A. (2005). Planification d’experiences numeriques. Revue MODULAD, 33, 63–73.

    Google Scholar 

  15. Krige, D. G. (1951). A statistical approach to some basic mine valuation problems on the Witwatersrand. Journal of the Chemical, Metallurgical and Mining Society of South Africa, 52(6), 119–139.

    Google Scholar 

  16. Masters, T. (1993). Practical neural network recipes in C++. New York: Academic Press.

    Google Scholar 

  17. Matheron, G. (1963). Principles of geostatistics. Economic Geology, 58, 1246.

    Article  Google Scholar 

  18. Mebane, W. J., & Sekhon, J. (2011). Genetic optimization using derivatives: the rgenoud package for R. Journal of Statistical Software, 42(11), 1–26.

    Google Scholar 

  19. Miller, J., & Page, S. (2007). Complex adaptive systems. Princeton: Princeton University Press.

    Google Scholar 

  20. Nelson, R. R., & Winter, S. G. (1978). Forces generating and limiting concentration under schumpeterian competition. Bell Journal of Economics, 9(2), 524–548.

    Article  Google Scholar 

  21. Nelson, R. R., & Winter, S. G. (1982). The schumpeterian tradeoff revisited. American Economic Review, 72(1), 114–132.

    Google Scholar 

  22. Oeffner, M. (2008). Agent-based Keynesian macroeconomics—an evolutionary model embedded in an agent-based computer simulation. Doctoral dissertation: Bayerische Julius - Maximilians Universitat, Wurzburg.

  23. R Development Core Team. (2013). R: A language and environment for statistical computing, R Foundation for statistical computing, Vienna. ISBN 3-900051-07-0. http://www.R-project.org

  24. Roustant, O., Ginsbourger, D., & Deville, Y. (2010). DiceKriging, diceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. Journal of Statistical Software, 55(2), 100.

    Google Scholar 

  25. Sacks, J., Welch, W., Mitchell, T., & Wynn, H. (1989). Design and analysis of computer experiments. Statistical Science, 4(4), 409.

    Article  Google Scholar 

  26. Salle, I., Sénégas, M., & Yıldızoğlu, M. (2012). How transparent should a Central Bank be? an ABM assessment. avril: mimeo, Bordeaux University.

    Google Scholar 

  27. Saltelli, A., Tarantola, S., & Chan, K. (1999). A quantitative model-independent method for global sensitivity analysis of model output. Technometrics, 41(1), 39–56.

    Article  Google Scholar 

  28. Sanchez, S. M., (2005). Work smarter, not harder: Guidelines for designing simulation experiments, In M. E. Kuhl, N. M. Steiger, F. B. Armstrong, & J. A. Joines (eds.) Proceedings of the 2005 Winter Simulation Conference. Software available at http://harvest.nps.edu/linkedfiles/nolhdesigns_v4.xls

  29. Silva, I., Assuncao, R., & Costa, M. (2009). Power of the sequential monte carlo test. Sequential Analysis, 28(2), 163–174.

    Article  Google Scholar 

  30. Tesfatsion, L., & Judd, K. L. (Eds.). (2006). Handbook of computational economics. Agent-based computational economics (Vol. 2). Amsterdam: North-Holland.

    Google Scholar 

  31. Vallée, T., & Yıldızoğlu, M. (2009). Convergence in the finite Cournot oligopoly with social and individual learning. Journal of Economic Behavior & Organization, 72(2), 670–690.

    Article  Google Scholar 

  32. van Beers, W., & Kleijnen, J. (2004). Kriging interpolation in simulation: a survey. In R. G. Ingalls, M. D. Rossetti, J. S. Smith, & B. A. Peters (Eds.), Handbook of statistics. New York: Elsevier.

    Google Scholar 

  33. Wang, G., & Shan, S. (2007). Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design, 129, 370.

    Article  Google Scholar 

  34. Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J., & Morris, M. D. (1992). Screening, predicting, and computer experiments. Technometrics, 34(1), 15–25.

    Article  Google Scholar 

  35. Ye, K. (1998). Orthogonal column latin hypercubes and their application in computer experiments. Journal of the American Statistical Association, 93(444), 1430–1439.

    Article  Google Scholar 

  36. Yıldızoğlu, M. (2001). Connecting adaptive behaviour and expectations in models of innovation: The potential role of artificial neural networks. European Journal of Economic and Social Systems, 15(3), 51–65.

    Google Scholar 

  37. Yıldızoğlu, M., Sénégas, M.-A., & Zumpe, M. (2012). Learning the optimal buffer-stock consumption rule of Carroll. Macroeconomic Dynamics, 5, 255.

    Google Scholar 

Download references

Acknowledgments

We are grateful to two anonymous referees for their comments and suggestions, and to the participants of the Lipari summer school on “Data mining and modelling of complex techno-socio-economic systems” (July 2012, Italy) for useful comments and discussions. We are responsible for all remaining errors.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Murat Yıldızoğlu.

Appendices

Appendix

A DoE

See appendix Tables 3 and 4.

Table 3 DoE, Nelson and Winter (1982), \(k=6\) factors, \(n=17\) experiments
Table 4 DoE, oligopoly model with learning, \(k=8\) factors, \(n=33\) experiments

R Codes for Kriging

figurea
figureb

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Salle, I., Yıldızoğlu, M. Efficient Sampling and Meta-Modeling for Computational Economic Models. Comput Econ 44, 507–536 (2014). https://doi.org/10.1007/s10614-013-9406-7

Download citation

Keywords

  • Computational economics
  • Exploration of agent-based models
  • Design of experiments
  • Meta-modeling