Abstract
The real utility of simulation lies in comparing different alternatives that might represent competing system designs. Conventional statistical techniques are not directly applicable to the analysis of simulation output data in the evaluation of competing alternatives since the usual assumptions of normality and common variance are difficult to justify in simulation experiments. This paper revisits a known nonparametric test whose application has recently become feasible due to considerable increases in computing power:randomization tests assess the significance of the observed value of the test statistic by evaluating different permutations of the data. The procedure only requires invariance of the data under all permutations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
P.J. Bickel and K.A. Doksum,Mathematical Statistics: Basic Ideas and Selected Topics (Holden-Day, Oakland, 1977).
G.E.P. Box and S.L. Andersen, Permutation theory in the derivation of robust criteria and the study of departures from assumptions, J. Roy. Statist. Soc. Series B 17(1955)1–34.
G.M. Clark, Tutorial: Analysis of simulation output to compare alternatives, in:Proc. Winter Simulation Conf. (1988) pp. 19–24.
G.M. Clark and W. Yang, A Bonferroni selection procedure when using common random numbers with unknown variances, in:Proc. Winter Simulation Conf. (1986) pp. 313–315.
B.J. Cooley and J.W. Cooley, Data analysis for simulation experiments: Application of a distribution-free multiple comparisons procedure, Dec. Sci. 11(1980)482–492.
E.J. Dudewicz, Statistics in simulation: How to design for selecting the best alternative, in:Proc. Winter Conf. (1976).
E.J. Dudewicz and S.R. Dalal, Allocations of observations in ranking and selection with unequal variances, Sankhya-B. 37(1975)28–78.
B. Efron, Bootstrap methods: Another look at the jacknife, Ann. Statist. 7(1979)1–26.
B. Efron and R. Tibshirani, Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy, Statist. Sci. 1(1984)54–77.
R.A. Fisher,Statistical Methods for Research Workers (Oliver & Boyd, Edinburgh, 1925).
R.A. Fisher,The Design of Experiments (Oliver & Boyd, Edinburgh, 1935).
R.V. Foutz, A method for constructing exact tests from test statistics that have unknown null distributions, J. Statist. Comp. Simul. 10(1980)187–193.
D. Goldsman, Ranking and selection in simulation, in:Proc. Winter Simulation Conf. (1983) pp. 387–393.
Y. Hochberg and A.C. Tamhane,Multiple Comparisons Procedures (Wiley, New York, 1987).
W. Hoeffding, The large sample power of tests based on permutations of observations, Ann. Math. Statist. 23(1952)169–192.
J.C. Hsu, Ranking and selection and multiple comparisons with the best, in:Design of Experiments: Ranking and Selection, ed. Santner and Tamhane (Marcel Dekker, New York, 1984) chapter 3.
J.C. Hsu and B.L. Nelson, Optimization over a finite number of system designs with one-stage sampling and multiple comparisons with the best, in:Proc. Winter Simulation Conf. (1988) pp. 451–457.
J.V. Jucker and J.G. Gomez, Policy-comparing simulation experiments: Design and analysis, Dec. Sci. 6(1975)631–645.
J.P.C. Kleijnen,Statistical Tools for Simulation Practitioners (Marcel Dekker, New York, 1987).
A.M. Law and W.D. Kelton,Simulation Modeling and Analysis, 2nd ed. (McGraw-Hill, New York, 1991).
P.A.W. Lewis and A.J. Orav,Simulation Methodology for Statisticians, Operations Analysts, and Engineers, Vol. 1 (Wadsworth & Brooks/Cole, Pacific Grove, 1989).
E.W. Noreen,Computer Intensive Methods for Testing Hypotheses: An Introduction (Wiley, New York, 1989)
M. Pagano and D. Tritchler, On obtaining permutation distributions in polynomial time, J. Amer. Statist. Assoc. 78(1983)435–440.
H. Scheffé,The Analysis of Variance (Wiley, New York, 1959).
C. Spino and M. Pagano, Efficient calculation of the permutation distribution of trimmed means, J. Amer. Statist. Assoc. 86(1991)729–737.
L.E. Toothaker,Multiple Comparisons for Researchers (Sage Pub., Newbury Park, 1991).
D. Tritchler, On inverting permutation tests, J. Amer. Statist Assoc. 79(1984)200–207.
B.L. Welch, The significance of the difference between two means when the population variances are unequal, Biometrika 25(1938)350–362.
W. Yang and B.L. Nelson, Optimization using common random numbers, control variates and multiple comparisons with the best, in:Proc. Winter Simulation Conf. (1989) pp. 444–449.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Yücesan, E. Evaluating alternative system configurations using simulation: A nonparametric approach. Ann Oper Res 53, 471–484 (1994). https://doi.org/10.1007/BF02136839
Issue Date:
DOI: https://doi.org/10.1007/BF02136839