Abstract
Computer simulators are often used when it is impossible or infeasible to observe actual systems or processes. These simulators can be very complex, requiring many hours or days for a single simulation, and thus, the number of times we may run the code is small. Running such a simulator at a few chosen input settings comprises a computer experiment. Most of the work done in this area focuses on either estimating the unknown complex input-output relationship or optimizing the output. In this article, we consider the problem of percentile estimation in a computer experiments setting and propose the use of sequential-adaptive designs to estimate percentiles, as opposed to fixed designs. For estimating the value and location of the percentile, we present design criteria that can be used to adaptively select the design sites at which to run the simulator. A comparison of results from using sequential-adaptive designs and fixed designs is presented.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bernardo, M. C., R. Buck, L. Liu, W. A. Nazaret, J. Sacks, and W. J. Welch. 1992. Integrated circuit design optimization using a sequential strategy. IEEE Transa. Computer-Aided Design, 11, 361–372.
Box, G. E. P., W. G. Hunter, and J. S. Hunter. 1978. Statistics for experimenters. New York, NY: John Wiley and Sons.
Chang, P. B., B. J. Williams, T. J. Santner, W. I. Notz, and D. L. Bartel. 1999. Robust optimization of total joint replacements incorporating environmental variables. J. Biomech. Eng., 121, 304–310.
Chang, P. B., B. J. Williams, K. S. B. Bhalla, T. W. Belknap, T. J. Santner, W. I. Notz, and D. L. Bartel. 2001. Design and analysis of robust total joint replacements: Finite element model experiments with environmental variables. J. Biomech. Eng., 123, 239–246.
Currin, C., T. Mitchell, M. Morris, and D. Ylvisaker. 1991. Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments, J. Am. Stat. Assoc., 86, 953–963.
M. E. Johnson, L. M. Moore, and D. Ylvisaker. 1990. Minimax and maximin distance designs, J. Stat. Plann. Inference, 26, 131–148.
D. R. Jones, M. Schonlau, and W. J. Welch. 1998. Efficient global optimization of expensive black-box functions, J. Global Optimization, 13, 455–492.
Kaufman, C. G., D. Bingham, S. Habib, K. Heitman, and J. A. Frieman. 2011. Efficient emulators of computer experiments using compactly supported correlation functions, with an application to cosmology. Ann. Appl. Stat., 8, 2470–2492.
Loeppky, J. L., J. Sacks, and W. J. Welch. 2009. Choosing the sample size of a computer experiment: A practical guide. Technometrics, 51, 366–376.
McKay, M. D., R. J. Beckman, and W. J. Conover. 1979. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21, 239–245.
Morris, M. D., and T. J. Mitchell. 1995. Exploratory designs for computer experiments. J. Stat. Plan. Inference, 43, 38–402.
Oakley, J. 2004. Estimating percentiles of uncertain computer code outputs. Appl. Stat., 53, 83–93.
Ranjan, P., D. Bingham, and G. Michailidis. 2008. Sequential experiment design for contour estimation from complex computer codes. Technometrics, 50, 527–541.
Roy, S. 2008. Sequential-adaptive design of computer experiments for the estimation of percentiles. PhD thesis, Department of Statistics, Ohio State University, Columbus, OH.
Sacks, J., S. B. Schiller, and W. J. Welch. 1989. Designs for computer experiments, Technometrics, 31, 41–47.
Santner, T. J., B. J. Williams, and W. I. Notz. 2003. The design and analysis of computer experiments. New York, NY: Springer-Verlag.
Schonlau, M. 1997. Computer experiments and global optimization. PhD thesis, Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, ON, Canada.
Schonlau, M., W. J. Welch, and D. R. Jones. 1998. Global versus local search in constrained optimization of computer models. N. Dev. Appl. Exp. Design, IMS Lecture Notes Monogr. Ser., 34, 11–25.
Welch, W. J., R. J. Buck, J. Sacks, H. P. Wynn, T. B. Mitchell, and M. D. Morris. 1992. Screening, predicting, and computer experiments. Technometrics, 34, 15–25.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Roy, S., Notz, W.I. Estimating Percentiles in Computer Experiments: A Comparison of Sequential-Adaptive Designs and Fixed Designs. J Stat Theory Pract 8, 12–29 (2014). https://doi.org/10.1080/15598608.2014.840491
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1080/15598608.2014.840491