How Many Times Should One Run a Computational Simulation?
This chapter is an attempt to answer the question “how many runs of a computational simulation should one do,” and it gives an answer by means of statistical analysis. After defining the nature of the problem and which types of simulation are mostly affected by it, the article introduces statistical power analysis as a way to determine the appropriate number of runs. Two examples are then produced using results from an agent-based model. The reader is then guided through the application of this statistical technique and exposed to its limits and potentials.
- Anderson, P. (1972). More is different. Science, 177(4047), 393–396.Google Scholar
- Bardone, E. (2016). Intervening via chance-seeking. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 203–220). New York: Springer.Google Scholar
- Bland, J. M. (2009). The tyranny of power: Is there a better way to calculate sample size? BMJ, 339, b3985.Google Scholar
- Champely, S., Ekstrom, C., Dalgaard, P., Gill, J., Weibelzahl, S., & Rosario, H. D. (2016). Pwr: Basic functions for power analysis.Google Scholar
- Choirat, C., & Seri, R. (2012). Estimation in discrete parameter models. Statistical Science, 27(2), 278–293.Google Scholar
- Coen, C. (2009). Simple but not simpler. Introduction CMOT special issue–simple or realistic. Computational and Mathematical Organization Theory, 15, 1–4.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: LEA.Google Scholar
- Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.Google Scholar
- Cohen, M. D., March, J. G., & Olsen, H. P. (1972). A garbage can model of organizational choice. Administrative Science Quarterly, 17(1), 1–25.Google Scholar
- Davidsson, P., & Verhagen, H. (2017). Types of simulation. doi: https://doi.org/10.1007/978-3-319-66948-9_3.
- de Marchi, S., & Page, S. E. (2014). Agent-based models. Annual Review of Political Science, 17(1), 1–20.Google Scholar
- Edmonds, B., & Meyer, R. (2017). Introduction to the handbook. doi: https://doi.org/10.1007/978-3-319-66948-9_1.
- Edmonds, B., & Moss, S. (2005). From KISS to KIDS — an ‘anti-simplistic’ modelling approach. In P. Davidson (Ed.), Multi agent based simulation. Lecture Notes in Artificial Intelligence (Vol. 3415, pp. 130–144). New York: Springer.Google Scholar
- Erdfelder, E. (1984). Zur Bedeutung und Kontrolle des β-Fehlers bei der inferenzstatistischen Prüfung log-linearer Modelle [The significance and control of the β-error during the inference-statistical examination of the log-linear models]. Zeitschrift für Sozialpsychologie, 15(1), 18–32.Google Scholar
- Fioretti, G. (2016). Emergent organizations. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 19–41). New York: Springer.Google Scholar
- Fioretti, G., & Lomi, A. (2008). An agent-based representation of the garbage can model of organizational choice. Journal of Artificial Societies and Social Simulation, 11(1).Google Scholar
- Herath, D., Secchi, D., & Homberg, F. (2015). Simulating the effects of disorganisation on employee goal setting and task performance. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 63–84). New York: Springer.Google Scholar
- Korn, E. L. (1990). Projecting power from a previous study: Maximum likelihood estimation. The American Statistician, 44(4), 290–292.Google Scholar
- Lakatos, E. (2005). Sample size determination for clinical trials. In Encyclopedia of biostatistics. Hoboken: Wiley.Google Scholar
- Lamperti, F. (2015). An Information Theoretic Criterion for Empirical Validation of Time Series Models. LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant’Anna School of Advanced Studies, Pisa, Italy.Google Scholar
- Liu, T., & Stone, C. C. (2007). Law and statistical disorder: Statistical hypothesis test procedures and the criminal trial analogy. SSRN Scholarly Paper ID 887964, Social Science Research Network, Rochester, NY.Google Scholar
- Ritter, F. E., Schoelles, M. J., Quigley, K. S., & Cousino-Klein, L. (2011). Determining the numbers of simulation runs: Treating simulations as theories by not sampling their behavior. In L. Rothrock & S. Narayanan (Eds.), Human-in-the-loop simulations: Methods and practice (pp. 97–116). London: Springer.CrossRefGoogle Scholar
- Robinson, S. (2014). Simulation. The practice of model development and use (2nd ed.). New York: Palgrave.Google Scholar
- Secchi, D., & Seri, R. (2014). ‘How many times should my simulation run?’ Power analysis for agent-based modeling. In European Academy of Management Annual Conference, Valencia, Spain.Google Scholar
- Simon, H. A. (1976). How complex are complex systems. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association (Vol. 2, pp. 507–522). Baltimore: Philosophy of Science Association.Google Scholar
- Simon, H. A. (1978). Rationality as process and a product of thought. American Economic Review, 68, 1–14.Google Scholar
- Simon, H. A. (1997). Administrative behavior (4th ed.). New York: The Free Press.Google Scholar
- Troitzsch, K. G. (2017). Historical introduction. doi: https://doi.org/10.1007/978-3-319-66948-9_2.
- Wilensky, U. (1999). Netlogo. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL.Google Scholar