Abstract
We aim to determine which of a set of competing models is statistically best, that is, on average. A way to define “on average” is to consider the performance of these algorithms averaged over all the training sets that might be drawn from the underlying distribution. When comparing more than two means, an ANOVA F-test tells you whether the means are significantly different, but it does not tell you which means differ from each other. A simple approach is to test each possible difference by a paired t-test. However, the probability of making at least one type I error increases with the number of tests made. Multiple comparison procedures provide different solutions. We discuss these techniques and apply the well known Bonferroni method in order to determine the optimal degree in polynomial fitting and the optimal number of hidden neurons in feedforward neural networks.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Bishop, C. M.: Neural Network for Pattern Recognition. Clarendon Press-Oxford (1995)
Cobb, G.W.: Introduction to Design and Analysis of Experiments. Springer-Verlag New York (1998)
Dean, A., Voss, D.: Design and Analysis of Experiments. Springer Texts in Statistics. Springer-Verlag New York (1999)
Dietterich, T.G.: Aproximate Statistical Test for Comparing Supervised Classification Learning Algorithms. Neural Computation (1998), Vol. 10, no. 7, 1895–1923
Feelders, A., Verkooijen, W.: On the Statistical Comparison of Inductive Learning Methods. Learning from Data Artificial Intelligence and Statistics V. Springer-Verlag, New York (1996) 271–279
Hsu, J.C.: Multiple Comparisons: Theory and Methods. Chapman & Hall (1996)
Jobson, J.D.: Applied Multivariate Data Analysis. Springer Texts in Statistics, Vol 1. Springer-Verlag New York (1991)
Stone, M.: Cross-validatory Choice and Assesment of Statistical Prediction (with discussion). Journal of the Royal Statistical Society (1974), Series B, 36, 111–147
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Galindo, P.L., Pizarro-Junquera, J., Guerrero, E. (2000). Multiple Comparison Procedures for Determining the Optimal Complexity of a Model. In: Ferri, F.J., Iñesta, J.M., Amin, A., Pudil, P. (eds) Advances in Pattern Recognition. SSPR /SPR 2000. Lecture Notes in Computer Science, vol 1876. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44522-6_82
Download citation
DOI: https://doi.org/10.1007/3-540-44522-6_82
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67946-2
Online ISBN: 978-3-540-44522-7
eBook Packages: Springer Book Archive