Design and Analysis of Computational Experiments: Overview

  • Jack P.C. Kleijnen


This chapter presents an overview of the design and analysis of computational experiments with optimization algorithms. It covers classic designs and their corresponding (meta)models; namely, Resolution-III designs including fractional factorial two-level designs for first-order polynomial models, Resolution-IV and Resolution-V designs for two-factor interactions, and designs including central composite designs for second-degree polynomials. It also reviews factor screening in experiments with very many factors, focusing on the sequential bifurcation method. Furthermore, it reviews Kriging models and their designs. Finally, it discusses experiments aimed at the optimization of the parameters of a given optimization algorithm, allowing multiple random experimental outputs. This optimization may use either generalized response surface methodology or Kriging combined with mathematical programming; the discussion also covers Taguchian robust optimization.


Response Surface Methodology Robust Optimization Latin Hypercube Sampling Kriging Model Classic Design 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



Comments by two anonymous referees led to a much improved version of the present chapter.


  1. Adenso-Diaz B, Laguna M (2006) Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research 54(1):99–114zbMATHCrossRefGoogle Scholar
  2. Angün E, Kleijnen J (2009) An asymptotic test of optimality conditions in multiresponse simulation-based optimization, working paperGoogle Scholar
  3. Angün E, Kleijnen J, den Hertog D, Gürkan G (2009) Response surface methodology with stochastic constrains for expensive simulation. Journal of the Operational Research Society 60:735–746zbMATHCrossRefGoogle Scholar
  4. Ankenman B, Nelson B, Staum J (2009) Stochastic kriging for simulation metamodeling. Operations Research (accepted)Google Scholar
  5. Bartz-Beielstein T (2006) Experimental research in evolutionary computation—The new experimentalism. Natural Computing Series, SpringerGoogle Scholar
  6. Bartz-Beielstein T, Preuss M (2010) The future of experimental research. In: Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (eds) Empirical methods for the analysis of optimization algorithms, Springer, pp 17–46Google Scholar
  7. Ben-Tal A, Nemirovski A (2002) Robust optimization: methodology and applications. Mathematical Programming 92(3):353–380CrossRefMathSciNetGoogle Scholar
  8. Bettonvil B, Kleijnen J (1996) Searching for important factors in simulation models with many factors: sequential bifurcation. European Journal of Operational Research 96:180–194CrossRefGoogle Scholar
  9. Bettonvil B, del Castillo E, Kleijnen J (2009) Statistical testing of optimality conditions in multiresponse simulation-based optimization. European Journal of Operational Research 199(2):448–458zbMATHCrossRefMathSciNetGoogle Scholar
  10. Box G, Wilson K (1951) On the experimental attainment of optimum conditions. Journal Royal Statistical Society, Series B 13(1):1–38MathSciNetGoogle Scholar
  11. Cressie N (1993) Statistics for spatial data: revised edition. Wiley, New YorkGoogle Scholar
  12. Dellino G, Kleijnen J, Meloni C (2009) Robust optimization in simulation: Taguchi and response surface methodology. In: Rossini M, Hill R, Johansson B, Dunkin A, Ingalls R (eds) Proceedings of the 2009 Winter Simulation Conference, (accepted)Google Scholar
  13. Den Hertog D, Kleijnen J, Siem A (2006) The correct Kriging variance estimated by bootstrapping. Journal of the Operational Research Society 57(4):400–409zbMATHCrossRefGoogle Scholar
  14. Efron B, Tibshirani R (1993) An introduction to the bootstrap. Chapman & Hall, NYzbMATHGoogle Scholar
  15. Fu M (2008) What you should know about simulation and derivatives. Naval Research Logistics 55:723–736zbMATHCrossRefMathSciNetGoogle Scholar
  16. Kleijnen J (2008) Design and analysis of simulation experiments. SpringerzbMATHGoogle Scholar
  17. Kleijnen J, Sargent R (2000) A methodology for the fitting and validation of metamodels in simulation. European Journal of Operational Research 120(1):14–29zbMATHCrossRefGoogle Scholar
  18. Kleijnen J, Van Beers W (2009) Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations, working paperGoogle Scholar
  19. Kleijnen J,Wan J (2007) Optimization of simulated systems: OptQuest and alternatives. Simulation Modelling Practice and Theory 15:354–362CrossRefGoogle Scholar
  20. Kleijnen J, Sanchez S, Lucas T, Cioppa T (2005) State-of-the-art review: a user’s guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing 17(3):263–289CrossRefGoogle Scholar
  21. Kleijnen J, Bettonvil B, Persson F (2006a) Screening for the important factors in large discrete-event simulation: sequential bifurcation and its applications. In: Dean A, Lewis S (eds) Screening: Methods for experimentation in industry, drug discovery, and genetics, Springer, pp 287–307Google Scholar
  22. Kleijnen J, den Hertog D, Angün E (2006b) Response surface methodology’s steepest ascent and step size revisited: correction. European Journal of Operational Research 170:664–666zbMATHCrossRefGoogle Scholar
  23. Kleijnen J, Van Beers W, Van Nieuwenhuyse I (2010) Constrained optimization in simulation: a novel approach. European Journal of Operational Research 202:164–174zbMATHCrossRefGoogle Scholar
  24. Law A (2007) Simulation modeling and analysis, 4th edn. McGraw-Hill, BostonGoogle Scholar
  25. Lophaven S, Nielsen H, Søndergaard J (2002) DACE—A Matlab Kriging Toolbox. Tech. Rep. IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, Copenhagen, DenmarkGoogle Scholar
  26. Montgomery D (2009) Design and analysis of experiments, 7th edn. Wiley, Hoboken, New JerseyGoogle Scholar
  27. Myers R, Montgomery D (1995) Response surface methodology: process and product optimization using designed experiments. Wiley, NYzbMATHGoogle Scholar
  28. Park S, Fowler J, Mackulak G, Keats J, Carlyle W (2002) D-optimal sequential experiments for generating a simulation-based cycle time-throughput curve. Operations Research 50(6):981–990CrossRefGoogle Scholar
  29. Rajagopalan HK, Vergara FE, Saydam C, Xiao J (2007) Developing effective meta-heuristics for a probabilistic location model via experimental design. European Journal of Operational Research 177(1):83–101, URL http://dx.doi. org/10.1016/j.ejor.2005.11.007zbMATHCrossRefGoogle Scholar
  30. Ridge E, Kudenko D (2007) Screening the parameters affecting heuristic performance. In: Lipson H (ed) GECCO, ACM, p 180, URL http://doi.acm. org/10.1145/1276958.1276994
  31. Ridge E, Kudenko D (2010) Sequential experiment designs for screening and tuning parameters of stochastic heuristics. In: Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (eds) Empirical Methods for the Analysis of Optimization Algorithms, Springer, pp 265–287Google Scholar
  32. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Statistical Science 4(4):409–435zbMATHCrossRefMathSciNetGoogle Scholar
  33. Santner TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. SpringerzbMATHGoogle Scholar
  34. Taguchi G (1987) System of experimental designs. Krauss International, NYGoogle Scholar
  35. Van Beers W, Kleijnen J (2003) Kriging for interpolation in random simulation. Journal of the Operational Research Society (54):255–262zbMATHCrossRefGoogle Scholar
  36. Van Beers W, Kleijnen J (2008) Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping. European Journal of Operational Research 186(3):1099–1113zbMATHCrossRefGoogle Scholar
  37. Wu C, Hamada M (2000) Experiments; planning, analysis, and parameter design optimization. Wiley, NYzbMATHGoogle Scholar
  38. Xu J, Yang F, Wan H (2007) Controlled sequential bifurcation for software reliability study. In: Henderson S, Biller B, Hsieh MH, Shortle J, Tew J, Barton R (eds) Proceedings of the 2007 Winter Simulation Conference, pp 281–288Google Scholar
  39. Yin J, Ng S, Ng K (2008) Kriging model with modified nugget effect. In: Proceedings of the 2008 IEEE International Conference on Industrial Engineering and Engineering Management, pp 1714–1718Google Scholar
  40. Yu HF (2007) Designing a screening experiment with a reciprocal Weibull degradation rate. Computers & Industrial Engineering 52(2):175–191CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Jack P.C. Kleijnen
    • 1
  1. 1.Department of Information ManagementCentER Tilburg UniversityTilburgNetherlands

Personalised recommendations