Journal of Simulation

, Volume 11, Issue 2, pp 137–150 | Cite as

Second-order nearly orthogonal Latin hypercubes for exploring stochastic simulations

  • A D MacCalman
  • H Vieira
  • T Lucas
Original Article


This paper presents new Latin hypercube designs with minimal correlations between all main, quadratic, and two-way interaction effects for a full second-order model. These new designs facilitate exploratory analysis of stochastic simulation models in which there is considerable a priori uncertainty about the forms of the responses. We focus on understanding the underlying complexities of simulated systems by exploring the input variables’ effects on the behavior of simulation responses. These new designs allow us to determine the driving factors, detect interactions between input variables, identify points of diminishing or increasing rates of return, and find thresholds or change points in localized areas. Our proposed designs enable analysts to fit many diverse metamodels to multiple outputs with a single set of runs. Creating these designs is computationally intensive; therefore, several have been cataloged and made available online to experimenters.


simulation design of experiments genetic algorithms Latin hypercube nearly orthogonal designs space-filling designs 



The authors would like to thank our colleagues at the Naval Postgraduate School for their insightful comments. In addition, the authors are grateful to the referees, whose feedback improved the content and clarity of this paper. The research was made possible by a grant from the Office of Naval Research (N0001412WX20823).


  1. Ankenman B, Nelson B and Staum J (2010). Stochastic kriging for simulation metamodeling. Operations Research 58(2): 371–382.CrossRefGoogle Scholar
  2. Atkinson AC, Donev AN and Tobias RD (2007). Optimum Experimental Designs, with SAS. Oxford: Oxford University Press.Google Scholar
  3. Bankes S (1993). Exploratory modeling for policy analysis. Operations Research 41(3): 435–449.CrossRefGoogle Scholar
  4. Barton RR (1998). Simulation metamodels. In: Medeiros DJ, Watson EF, Carson JS, and Manivannan, MS (eds). Proceedings of the 1998 Winter Simulation Conference, Vol. 1, IEEE. Piscataway, NJ, pp 167–174.Google Scholar
  5. Beers V and Kleijnen JPC (2003). Kriging for interpolation in random simulation. Journal of the Operational Research Society 54(3): 255–262.CrossRefGoogle Scholar
  6. Beers V and Kleijnen JPC (2004). Kriging interpolation in simulation: A survey. In Proceedings of the 2004 Winter Simulation Conference. Washington, D.C., Vol. 1, pp 113–112.Google Scholar
  7. Box GEP and Behnken DW (1960). Some new three level designs for the study of quantitative variables. Technometrics 2(4): 455–475.CrossRefGoogle Scholar
  8. Box GEP and Draper NR (1987). Empirical Model-Building and Response Surfaces. Wiley: New York, NY.Google Scholar
  9. Challenor P (2013). Experimental design for the validation of kriging metamodels in computer experiments. Journal of Simulation 7(4): 290–296.CrossRefGoogle Scholar
  10. Cioppa TM and Lucas TW (2007). Efficient nearly orthogonal and space-filling Latin hypercubes. Technometrics 49(1): 45–55.CrossRefGoogle Scholar
  11. Fang KT (1980). The uniform design: Application of number-theoretic methods in experimental design. Acta Mathematicale Applicatae Sinica 3: 363–372.Google Scholar
  12. Fang KT, Lin DKJ, Winker P and Zhang Y (2000). Uniform design: Theory and application. Technometrics 42(3): 237–248.CrossRefGoogle Scholar
  13. Florian A (1992). An efficient sampling scheme: Updated Latin hypercube sampling. Probabilistic Engineering Mechanics 7(2): 123–130.CrossRefGoogle Scholar
  14. Goldberg DE (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. 1st edn, Addison-Wesley Longman Publishing Co: Boston, MA.Google Scholar
  15. Goldfarb HB, Borror CM, Montgomery DC and Anderson-Cook C (2005). Using genetic algorithms to generate mixture-process experimental designs involving control and noise variables. Journal of Quality Technology 37(1): 60–74.Google Scholar
  16. Heredia-Langner A, Carlyle WM, Montgomery DC and Borror CM (2004). Model-robust optimal designs: A genetic algorithm approach. Journal of Quality Technology 35(1): 263–279.Google Scholar
  17. Heredia-Langner A, Carlyle WM, Montgomery DC, Borror CM and Runger GC (2003). Genetic algorithms for the construction of D-optimal designs. Journal of Quality Technology 35(1): 28–46.Google Scholar
  18. Hernandez AS, Lucas TW and Carlyle WM (2012). Constructing nearly orthogonal Latin hypercubes for any nonsaturated run-variable combination. ACM Transactions on Modeling and Computer Simulation 22(20): 4.Google Scholar
  19. Hickernell FJ (1998). A generalized discrepancy and quadrature error bound. Mathematics of Computation 67(221): 299–322.CrossRefGoogle Scholar
  20. Holland JH (1975). Adaptation in Natural and Artificial Systems. The University of Michigan Press: Ann Arbor.Google Scholar
  21. Jin R, Chen W and Sudjianto A (2005). An efficient algorithm for constructing optimal design of computer experiments. Journal of Statistical Planning and Inference 134(1): 268–287.CrossRefGoogle Scholar
  22. Johnson ME, Moore LM and Ylvisaker D (1990). Minimax and maxmin distance design. Journal of Statistical Planning and Inference 26(2): 131–148.CrossRefGoogle Scholar
  23. Jones B and Nachtsheim CJ (2011). A class of three-level designs for definitive screening in the presence of second-order effects. Journal of Quality Technology 43(1): 1–15.Google Scholar
  24. Joseph VR and Hung Y (2008). Orthogonal-maximin Latin hypercube designs. Statistica Sinica 18(1): 171–186.Google Scholar
  25. Kim L and Loh H (2003). Classification trees and bivariate linear discriminant node models. Journal of Graphical and Statistics 12(3): 512–530.CrossRefGoogle Scholar
  26. Kleijnen JPC (2005). An overview of the design and analysis of simulation experiments for sensitivity analysis. European Journal of Operations Research 164(2): 287–300.CrossRefGoogle Scholar
  27. Kleijnen JPC (2008a). Simulation experiments in practice: Statistical design and regression analysis. Journal of Simulation 2(1): 19–27.CrossRefGoogle Scholar
  28. Kleijnen JPC (2008b). Design of experiments: Overview. In Proceedings of the 2008 Winter Simulation Conference. Miami, FL.Google Scholar
  29. Kleijnen JPC, Sanchez SM, Lucas TW and Cioppa TM (2005). A user’s guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing 17(3): 263–289.CrossRefGoogle Scholar
  30. Koehler JR and Owen AB (1996). Computer experiments. Handbook of Statistics 13: 261–308.CrossRefGoogle Scholar
  31. Law A (2007). Simulation Modeling and Analysis. 4th edn, McGraw-Hill: New York.Google Scholar
  32. Liefvendahl M and Stocki R (2006). A study on algorithms for optimization of Latin hypercubes. Journal of Statistical Planning and Inference 136(9): 3231–3247.CrossRefGoogle Scholar
  33. Lucas TW, Kelton WD, Sanchez SM and Sanchez PJ (2015). Changing the paradigm: Simulation, often the method of first resort. Naval Research Logistics 62: 293–303.CrossRefGoogle Scholar
  34. MacCalman AD (2013). Flexible space-filling designs for complex system simulations. Doctoral dissertation. Naval Postgraduate School: Monterey, CA.Google Scholar
  35. McKay MD, Beckman RJ and Conover WJ (1979). A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2): 239–245.Google Scholar
  36. Michalewicz Z and Fogel DB (2010). How to Solve it: Modern Heuristics. Springer-Verlag Berlin: Heidelberg, NY.Google Scholar
  37. Moon H, Santner TJ and Dean A (2011). Algorithms for generating maximin Latin hypercube and orthogonal designs. Journal of Statistical Theory and Practice 5(1): 81–98.CrossRefGoogle Scholar
  38. Morris MD and Mitchell TJ (1995). Exploratory designs for computer experiments. Journal of Statistical Planning and Inference 43(3): 381–402.CrossRefGoogle Scholar
  39. Myers RH, Montgomery DC and Anderson-Cook CM (2009). Response Surface Methodology: Process and Product Optimization using Designed Experiments. 3rd edn, Wiley, Hoboken, NJ.Google Scholar
  40. National Research Council (2006). Defense Modeling, Simulation, and Analysis: Meeting the Challenge Committee on Modeling and Simulation for Defense Transformation. The National Academies Press: Washington, DC.Google Scholar
  41. Owen AB (1994). Controlling correlations. In latin hypercube samples. Journal of the American Statistical Association: Theory and Methods 89(428): 1517–1522.CrossRefGoogle Scholar
  42. Pang FM, Liu Q and Lin DKJ (2009). A construction method for orthogonal Latin hypercube designs with prime power levels. Statist. Sinica 19(4): 1721–1728.Google Scholar
  43. Ryan TP (2007). Modern Experimental Design. 1st edn, Wiley-Interscience: Hoboken, NJ.CrossRefGoogle Scholar
  44. Sacks JW, Welch J, Mitchell TJ and Wynn HP (1989). Design and analysis of computer experiments (includes comments and rejoinder). Statistical Science 4(4): 409–435.CrossRefGoogle Scholar
  45. Sall J (2007). JMP Start Statistics: A Guide to Statistics and Data Analysis Using JMP, Fourth Edition. 4th edn, SAS Publishing, Cary, NC.Google Scholar
  46. Sanchez SM (2008). Better than a petaflop: The power of efficient experimental design. In Proceedings of the 40th Conference on Winter Simulation, pp 73–84.Google Scholar
  47. Sanchez SM, Lucas TW, Sanchez PJ, Nannini CJ and Wan H (2012). Designs for large-scale simulation experiments, with application to defense and homeland security. In: Hinkelmann K (ed) The Design and Analysis of Computer Experiments. Volume 3: Special Designs and Applications. Wiley: Hoboken, NJ, pp 413–441.CrossRefGoogle Scholar
  48. Santner TJ, Williams BJ and Notz WI (2003). The Design and Analysis of Computer Experiments. Springer-Verlag, New York, NY.CrossRefGoogle Scholar
  49. Shewry MC and Wynn HP (1987). Maximum entropy sampling. Journal of Applied Statistics 14(7): 165–170.CrossRefGoogle Scholar
  50. Siggelkow N (2002). Misperceiving interactions among complements and substitutes: Organizational consequences. Management Science 48(7): 900–916.CrossRefGoogle Scholar
  51. Steinberg DM and Lin DKJ (2006). A construction method for orthogonal Latin hypercube designs. Biometrika 93(2): 279–288.CrossRefGoogle Scholar
  52. Sun FS, Liu MQ and Lin DKJ (2009). Construction of orthogonal Latin hypercube designs. Biometrika 96(4): 971–974.CrossRefGoogle Scholar
  53. Vieira JrH (2008). Optimizing stochastic functions using a genetic algorithm: An aeronautic military application. In: Siarry P and Michalewicz Z (eds) Advances in Metaheuristics for Hard Optimization. Springer-Verlag: Berlin, pp 353–363.CrossRefGoogle Scholar
  54. Vieira JrH, Sanchez SM, Kienitz KH and Belderrain MCN (2011). Generating and improving orthogonal designs by using mixed integer programming. European Journal of Operational Research 215(3): 629–638.CrossRefGoogle Scholar
  55. Vieira JrH, Sanchez SM, Kienitz KH and Belderrain MCN (2013). Efficient, nearly orthogonal-and-balanced, mixed designs: An effective way to conduct trade-off analysis via simulation. Journal of Simulation 7(4): 264–275.CrossRefGoogle Scholar
  56. Yang JF, Lin CD, Qian PZG and Lin DKJ (2013). Construction of sliced orthogonal Latin hypercube Designs. Statitsica Sinica 23(3): 1117–1130.Google Scholar
  57. Ye KQ (1998). Orthogonal column Latin hypercubes and their application in computer experiments. Journal of the American Statistical Association: Theory and Methods 93(444): 1430–1439.CrossRefGoogle Scholar

Copyright information

© The Operational Research Society 2016

Authors and Affiliations

  1. 1.United States Military AcademyWest Point, NYUSA
  2. 2.Technological Institute of AeronauticsSão José dos CamposBrazil
  3. 3.Naval Postgraduate SchoolMontereyUSA

Personalised recommendations