How Many Times Should One Run a Computational Simulation?

Chapter
Part of the Understanding Complex Systems book series (UCS)

Abstract

This chapter is an attempt to answer the question “how many runs of a computational simulation should one do,” and it gives an answer by means of statistical analysis. After defining the nature of the problem and which types of simulation are mostly affected by it, the article introduces statistical power analysis as a way to determine the appropriate number of runs. Two examples are then produced using results from an agent-based model. The reader is then guided through the application of this statistical technique and exposed to its limits and potentials.

References

  1. Anderson, P. (1972). More is different. Science, 177(4047), 393–396.Google Scholar
  2. Bardone, E. (2016). Intervening via chance-seeking. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 203–220). New York: Springer.Google Scholar
  3. Bland, J. M. (2009). The tyranny of power: Is there a better way to calculate sample size? BMJ, 339, b3985.Google Scholar
  4. Champely, S., Ekstrom, C., Dalgaard, P., Gill, J., Weibelzahl, S., & Rosario, H. D. (2016). Pwr: Basic functions for power analysis.Google Scholar
  5. Choirat, C., & Seri, R. (2012). Estimation in discrete parameter models. Statistical Science, 27(2), 278–293.Google Scholar
  6. Coen, C. (2009). Simple but not simpler. Introduction CMOT special issue–simple or realistic. Computational and Mathematical Organization Theory, 15, 1–4.Google Scholar
  7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: LEA.Google Scholar
  8. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.Google Scholar
  9. Cohen, M. D., March, J. G., & Olsen, H. P. (1972). A garbage can model of organizational choice. Administrative Science Quarterly, 17(1), 1–25.Google Scholar
  10. Davidsson, P., & Verhagen, H. (2017). Types of simulation. doi: https://doi.org/10.1007/978-3-319-66948-9_3.
  11. de Marchi, S., & Page, S. E. (2014). Agent-based models. Annual Review of Political Science, 17(1), 1–20.Google Scholar
  12. Edmonds, B., & Meyer, R. (2017). Introduction to the handbook. doi: https://doi.org/10.1007/978-3-319-66948-9_1.
  13. Edmonds, B., & Moss, S. (2005). From KISS to KIDS — an ‘anti-simplistic’ modelling approach. In P. Davidson (Ed.), Multi agent based simulation. Lecture Notes in Artificial Intelligence (Vol. 3415, pp. 130–144). New York: Springer.Google Scholar
  14. Erdfelder, E. (1984). Zur Bedeutung und Kontrolle des β-Fehlers bei der inferenzstatistischen Prüfung log-linearer Modelle [The significance and control of the β-error during the inference-statistical examination of the log-linear models]. Zeitschrift für Sozialpsychologie, 15(1), 18–32.Google Scholar
  15. Fioretti, G. (2016). Emergent organizations. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 19–41). New York: Springer.Google Scholar
  16. Fioretti, G., & Lomi, A. (2008). An agent-based representation of the garbage can model of organizational choice. Journal of Artificial Societies and Social Simulation, 11(1).Google Scholar
  17. Fioretti, G., & Lomi, A. (2010). Passing the buck in the garbage can model of organizational choice. Computational and Mathematical Organization Theory, 16(2), 113–143CrossRefGoogle Scholar
  18. Fisher, R. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society. Series B (Methodological), 17(1), 69–78MathSciNetMATHGoogle Scholar
  19. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587–606.CrossRefGoogle Scholar
  20. Gilbert, N., & Terna, P. (2000). How to build and use agent-based models in social science. Mind and Society, 1, 57–72.CrossRefGoogle Scholar
  21. Hahn, G. J., & Meeker, W. Q. (2011). Statistical intervals: A guide for practitioners. Hoboken: Wiley.MATHGoogle Scholar
  22. Heckbert, S. (2013). MayaSim: An agent-based model of the ancient Maya social-ecological system. Journal of Artificial Societies and Social Simulation, 16(4), 11.CrossRefGoogle Scholar
  23. Herath, D., Secchi, D., & Homberg, F. (2015). Simulating the effects of disorganisation on employee goal setting and task performance. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 63–84). New York: Springer.Google Scholar
  24. Herath, D., Costello, J., & Homberg, F. (2017). Team problem solving and motivation under disorganization – an agent-based modeling approach. Team Performance Management, 23(1/2), 46–65.CrossRefGoogle Scholar
  25. Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power. The American Statistician, 55(1), 19–24.MathSciNetCrossRefGoogle Scholar
  26. Kollman, K., Miller, J. H., & Page, S. E. (1992). Adaptive parties in spatial elections. The American Political Science Review, 86(4), 929–937.CrossRefGoogle Scholar
  27. Korn, E. L. (1990). Projecting power from a previous study: Maximum likelihood estimation. The American Statistician, 44(4), 290–292.Google Scholar
  28. Lakatos, E. (2005). Sample size determination for clinical trials. In Encyclopedia of biostatistics. Hoboken: Wiley.Google Scholar
  29. Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701–710.CrossRefGoogle Scholar
  30. Lakens, D. & Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9(3), 278–292.CrossRefGoogle Scholar
  31. Lamperti, F. (2015). An Information Theoretic Criterion for Empirical Validation of Time Series Models. LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant’Anna School of Advanced Studies, Pisa, Italy.Google Scholar
  32. Liu, X. S. (2014). Statistical power analysis for the social and behavioral sciences. New York: Routledge.MATHGoogle Scholar
  33. Liu, T., & Stone, C. C. (2007). Law and statistical disorder: Statistical hypothesis test procedures and the criminal trial analogy. SSRN Scholarly Paper ID 887964, Social Science Research Network, Rochester, NY.Google Scholar
  34. Maggi, E., & Vallino, E. (2016). Understanding urban mobility and the impact of public policies: The role of the agent-based models. Research in Transportation Economics, 55, 50–59.CrossRefGoogle Scholar
  35. Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59(1), 537–563.CrossRefGoogle Scholar
  36. Mayo, D. G. (1992). Did pearson reject the neyman-pearson philosophy of statistics? Synthese, 90(2), 233–262.MathSciNetCrossRefMATHGoogle Scholar
  37. Mungovan, D., Howley, E., & Duggan, J. (2011). The influence of random interactions and decision heuristics on norm evolution in social networks. Computational and Mathematical Organization Theory, 17(2), 152–178.CrossRefGoogle Scholar
  38. Neyman, J. (1950). First course in probability and statistics. New York: Henry Holt and Company.MATHGoogle Scholar
  39. Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20A(1/2), 175–240.CrossRefMATHGoogle Scholar
  40. Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231, 289–337.MATHGoogle Scholar
  41. Pearson, E. S. (1955). Statistical concepts in the relation to reality. Journal of the Royal Statistical Society. Series B (Methodological), 17(2), 204–207.MathSciNetMATHGoogle Scholar
  42. Railsback, S. F., & Grimm, V. (2011). Agent-based and individual-based modeling: A practical introduction (59468th ed.). Princeton: Princeton University Press.MATHGoogle Scholar
  43. Ritter, F. E., Schoelles, M. J., Quigley, K. S., & Cousino-Klein, L. (2011). Determining the numbers of simulation runs: Treating simulations as theories by not sampling their behavior. In L. Rothrock & S. Narayanan (Eds.), Human-in-the-loop simulations: Methods and practice (pp. 97–116). London: Springer.CrossRefGoogle Scholar
  44. Robinson, S. (2014). Simulation. The practice of model development and use (2nd ed.). New York: Palgrave.Google Scholar
  45. Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47(5), 609–612.CrossRefGoogle Scholar
  46. Secchi, D. (2015). A case for agent-based model in organizational behavior and team research. Team Performance Management, 21(1/2), 37–50.CrossRefGoogle Scholar
  47. Secchi, D., & Gullekson, N. (2016). Individual and organizational conditions for the emergence and evolution of bandwagons. Computational and Mathematical Organization Theory, 22(1), 88–133.CrossRefGoogle Scholar
  48. Secchi, D., & Seri, R. (2014). ‘How many times should my simulation run?’ Power analysis for agent-based modeling. In European Academy of Management Annual Conference, Valencia, Spain.Google Scholar
  49. Secchi, D., & Seri, R. (2017). Controlling for ‘false negatives’ in agent-based models: A review of power analysis in organizational research. Computational and Mathematical Organization Theory, 23(1), 94–121.CrossRefGoogle Scholar
  50. Shimazoe, J., & Burton, R. M. (2013). Justification shift and uncertainty: Why are low-probability near misses underrated against organizational routines? Computational and Mathematical Organization Theory, 19(1), 78–100.CrossRefGoogle Scholar
  51. Simon, H. A. (1976). How complex are complex systems. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association (Vol. 2, pp. 507–522). Baltimore: Philosophy of Science Association.Google Scholar
  52. Simon, H. A. (1978). Rationality as process and a product of thought. American Economic Review, 68, 1–14.Google Scholar
  53. Simon, H. A. (1997). Administrative behavior (4th ed.). New York: The Free Press.Google Scholar
  54. Thiele, J., Kurth, W., & Grimm, V. (2015). Facilitating parameter estimation and sensitivity analysis of agent-based models: A cookbook using NetLogo and R. Journal of Artificial Societies and Social Simulation, 17(3), 11.CrossRefGoogle Scholar
  55. Thomsen, S. E. (2016). How docility impacts team efficiency. An agent-based modeling approach. In D. Secchi & M. Neumann (Eds.), Agent-based simulation of organizational behavior. New frontiers of social science research (pp. 159–173). New York: Springer.CrossRefGoogle Scholar
  56. Troitzsch, K. G. (2017). Historical introduction. doi: https://doi.org/10.1007/978-3-319-66948-9_2.
  57. van der Vaart, A. W. (2000). Asymptotic statistics. Cambridge: Cambridge University Press.MATHGoogle Scholar
  58. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values: Context, process, and purpose. American Statistician, 70(2), 129–133.MathSciNetCrossRefGoogle Scholar
  59. Wilensky, U. (1999). Netlogo. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL.Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of InsubriaVareseItaly
  2. 2.University of Southern DenmarkSlagelseDenmark

Personalised recommendations