Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments

  • Iris LorscheidEmail author
  • Bernd-Oliver Heine
  • Matthias Meyer


Many still view simulation models as a black box. This paper argues that perceptions could change if the systematic design of experiments (DOE) for simulation research was fully realized. DOE can increase (1) the transparency of simulation model behavior and (2) the effectiveness of reporting simulation results. Based on DOE principles, we develop a systematic procedure to guide the analysis of simulation models as well as concrete templates for sharing the results. A simulation model investigating the performance of learning algorithms in an economic mechanism design context illustrates our suggestions. Overall, the proposed systematic procedure for applying DOE principles complements current initiatives for a more standardized simulation research process.


Communication Design of experiments Simulation Standards Transparency 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Antony J (2003) Design of experiments for engineers and scientists. Butterworth-Heinemann, Amsterdam Google Scholar
  2. Arifovic J, Ledyard J (2004) Scaling up learning models in public good games. J Public Econ Theory 6:203–238 CrossRefGoogle Scholar
  3. Arnold M, Ponick E, Schenk-Mathes H (2008) Groves mechanism vs. profit sharing for corporate budgeting: An experimental analysis with preplay communication. Eur Account Rev 17:37–63 CrossRefGoogle Scholar
  4. Axelrod R (1997) Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P (eds) Simulating social phenomena. Springer, Berlin, pp 21–40 Google Scholar
  5. Barton R (2004) Designing simulation experiments. In: Proceedings of the winter simulation conference, Washington DC Google Scholar
  6. Bettonvil B, Kleijnen JPC (1996) Searching for important factors in simulation models with many factors: Sequential bifurcation. Eur J Oper Res 96:180–194 CrossRefGoogle Scholar
  7. Box GEP, Hunter JS, Hunter WG (2005) Statistics for experimenters: design, innovation, and discovery. Wiley-Interscience, Hoboken Google Scholar
  8. Camerer C, Ho T (1999) Experience-weighted attraction learning in normal form games. Econometrica 67:827–874 CrossRefGoogle Scholar
  9. Dutta P (1999) Strategies and games: theory and practice. MIT Press, Cambridge Google Scholar
  10. Ewert R, Wagenhofer A (2008) Interne Unternehmensrechnung. Springer, Berlin Google Scholar
  11. Field A, Hole G (2003) How to design and report experiments. SAGE, London Google Scholar
  12. Fisher RA (1971) The design of experiments. Hafner, New York Google Scholar
  13. Gilbert N, Troitzsch K (2005) Simulation for the social scientist. Open University Press, Maidenhead Google Scholar
  14. Gilbert N (2008) Agent-Based Models. SAGE, London Google Scholar
  15. Gode DK, Sunder S (1997) What makes markets allocationally efficient? Q J Econ 112:602 CrossRefGoogle Scholar
  16. Grimm V, Berger U, Bastiansen F, Eliassen S, Ginot V, Giske J, Goss-Custard J, Grand T, Heinz SK, Huse G, Huth A, Jepsen JU, Jørgensen C, Mooij WM, Müller B, Pe’er G, Piou C, Railsback SF, Robbins AM, Robbins MM, Rossmanith E, Rüger N, Strand E, Souissi S, Stillman RA, VabøR, Visser U, DeAngelis DL (2006) A standard protocol for describing individual-based and agent-based models. Ecol Model 198:115–126 CrossRefGoogle Scholar
  17. Grimm V, Berger U, DeAngelis DL, Polhill G, Giske J, Railsback SF (2010) The ODD protocol: a review and first update. Ecol Model 221:2760–2768 CrossRefGoogle Scholar
  18. Groves T (1973) Incentives in teams. Econometrica 41:617–631 CrossRefGoogle Scholar
  19. Groves T, Loeb M (1979) Incentives in a divisionalized firm. Manag Sci 25:221–230 CrossRefGoogle Scholar
  20. Harrison J, Lin Z, Carroll G, Carley K (2007) Simulation modeling in organizational and management research. Acad Manag Rev 32:1229–1245 CrossRefGoogle Scholar
  21. Heine B-O, Meyer M, Strangfeld O (2005) Stylised facts and the contribution of simulation to the economic analysis of budgeting. J Artif Soc Simul 8(4) Google Scholar
  22. Hendricks W, Robey K (1936) The sampling distribution of the coefficient of variation. Ann Math Stat 7:129–132 CrossRefGoogle Scholar
  23. Homburg C, Klarmann M (2003) Empirische Controllingforschung – Anmerkungen aus der Perspektive des Marketing. In: Weber J, Hirsch B (eds) Zur Zukunft der Controllingforschung, Wiesbaden, pp 65–88 CrossRefGoogle Scholar
  24. Kelton WD, Barton RR (2003) Experimental design for simulation. In: Proceedings of the 2003 winter simulation conference, New Orleans, Louisiana Google Scholar
  25. Kleijnen JPC (1998) Experimental design for sensitivity analysis, optimization, and validation of simulation models. In: Banks J (ed) Handbook of simulation: principles, methodology, advances, applications and practice, New York, pp 173–224 Google Scholar
  26. Kleijnen JPC, Sanchez S, Lucas T, Cioppa T (2005) A user’s guide to the brave new world of designing simulation experiments. INFORMS J Comput 17:263–289 CrossRefGoogle Scholar
  27. Kleijnen JPC (2008) Design and analysis of simulation experiments. Springer, New York Google Scholar
  28. Law A (2007) Simulation modeling and analysis. McGraw-Hill, Boston Google Scholar
  29. Malerba F, Nelson R, Orsenigo L, Winter S (1999) History-friendly models of industry evolution: the computer industry. Ind Corp Change 8(1):3–40 CrossRefGoogle Scholar
  30. Manuj I, Mentzer JT, Bowers MR (2009) Improving the rigor of discrete-event simulation in logistics and supply chain research. Int J Phys Distrib Logist Manag 39:172–201 CrossRefGoogle Scholar
  31. Mårtensson A (2003) Managing mission-critical IT in the financial industry. Economic Research Institute, Stockholm School of Economics Google Scholar
  32. Mårtensson A, Mårtensson P (2007) Extending rigor and relevance: towards credible, contributory and communicable research. In: Proceedings of the 15th European conference on information systems, St Gallen Google Scholar
  33. Montgomery DC (2009) Design and analysis of experiments. Wiley, Hoboken Google Scholar
  34. Myerson R (1999) Nash equilibrium and the history of economic theory. J Econ Lit 37:1067–1082 CrossRefGoogle Scholar
  35. North MJ, Macal CM (2007) Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. Oxford University Press, Oxford Google Scholar
  36. Oh RPT, Sanchez SM, Lucas TW, Wan H, Nissen ME (2009) Efficient experimental design tools for exploring large simulation models. Comput Math Organ Theory 15:237–257 CrossRefGoogle Scholar
  37. Peck SL (2004) Simulation as experiment: a philosophical reassessment for biological modeling. Trends Ecol Evol 19:530–534 CrossRefGoogle Scholar
  38. Polhill JG, Parker D, Brown D, Grimm V (2008) Using the ODD protocol for describing three agent-based social simulation models of land-use change. J Artif Soc Simul 11(2).
  39. Raghu TS, Sen PK, Rao HR (2003) Relative performance of incentive mechanisms: computational modeling and simulation of delegated investment decisions. Manag Sci 49:160–178 CrossRefGoogle Scholar
  40. Railsback SF, Grimm V (2011) Agent-based and individual-based modeling: a practical introduction. Princeton University Press, Princeton Google Scholar
  41. Richiardi M, Leombruni R, Saam NJ, Sonnessa M (2006) A common protocol for agent-based social simulation. J Artif Soc Simul 9(1).
  42. Reiss J (2011) A plea for (good) simulations: Nudging economics toward an experimental science. Simul Games 42:243–264 CrossRefGoogle Scholar
  43. Sanchez SM (2006) Work smarter, not harder: Guidelines for designing simulation experiments. In: Proceedings of the 2006 winter simulation conference, Monterey Google Scholar
  44. Sarin R, Vahid F (1999) Payoff assessments without probabilities: a simple dynamic model of choice. Games Econ Behav 28:294–309 CrossRefGoogle Scholar
  45. Saltelli A, Chan K, Scott EM (2000) Sensitivity analysis. Wiley, New York Google Scholar
  46. Saltelli A, Tarantola S, Campolongo F, Ratto M (2004) Sensitivity analysis in practice. Wiley, Chichester Google Scholar
  47. Schmolke A, Thorbek P, DeAngelis DL, Grimm V (2010) Ecological modeling supporting environmental decision making: a strategy for the future. Trends Ecol Evol 25:479–486 CrossRefGoogle Scholar
  48. Siebertz K, Bebber Dv, Hochkirchen T (2010) Statistische Versuchsplanung: Design of Experiments (DoE). Springer, Heidelberg CrossRefGoogle Scholar
  49. Simon H (1973) The organization of complex systems. In: Pattee HH (ed) Hierarchy theory: the challenge of complex systems. Braziller, New York, pp 1–27 Google Scholar
  50. Taber CS, Timpone RJ (1996) Computational modeling. SAGE, Thousand Oaks Google Scholar
  51. Trocine L, Malone LC (2001) An overview of newer, advanced screening methods for the initial phase in an experimental design. In: Proceedings of the 2001 winter simulation conference, Arlington Google Scholar
  52. Wu CFJ, Hamada M (2000) Experiments: planning, analysis, and parameter design optimization. Wiley, New York Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Iris Lorscheid
    • 1
    Email author
  • Bernd-Oliver Heine
    • 1
  • Matthias Meyer
    • 1
  1. 1.Institute of Management Control and AccountingHamburg University of TechnologyHamburgGermany

Personalised recommendations