Multiple Comparison Procedures for Determining the Optimal Complexity of a Model
We aim to determine which of a set of competing models is statistically best, that is, on average. A way to define “on average” is to consider the performance of these algorithms averaged over all the training sets that might be drawn from the underlying distribution. When comparing more than two means, an ANOVA F-test tells you whether the means are significantly different, but it does not tell you which means differ from each other. A simple approach is to test each possible difference by a paired t-test. However, the probability of making at least one type I error increases with the number of tests made. Multiple comparison procedures provide different solutions. We discuss these techniques and apply the well known Bonferroni method in order to determine the optimal degree in polynomial fitting and the optimal number of hidden neurons in feedforward neural networks.
KeywordsHide Neuron Honestly Significant Difference Hide Unit Polynomial Fitting Optimal Degree
- 1.Bishop, C. M.: Neural Network for Pattern Recognition. Clarendon Press-Oxford (1995)Google Scholar
- 3.Dean, A., Voss, D.: Design and Analysis of Experiments. Springer Texts in Statistics. Springer-Verlag New York (1999)Google Scholar
- 5.Feelders, A., Verkooijen, W.: On the Statistical Comparison of Inductive Learning Methods. Learning from Data Artificial Intelligence and Statistics V. Springer-Verlag, New York (1996) 271–279Google Scholar
- 6.Hsu, J.C.: Multiple Comparisons: Theory and Methods. Chapman & Hall (1996)Google Scholar
- 7.Jobson, J.D.: Applied Multivariate Data Analysis. Springer Texts in Statistics, Vol 1. Springer-Verlag New York (1991)Google Scholar