Multiple Comparison Procedures for Determining the Optimal Complexity of a Model

  • Pedro L. Galindo
  • Joaquín Pizarro-Junquera
  • Elisa Guerrero
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1876)

Abstract

We aim to determine which of a set of competing models is statistically best, that is, on average. A way to define “on average” is to consider the performance of these algorithms averaged over all the training sets that might be drawn from the underlying distribution. When comparing more than two means, an ANOVA F-test tells you whether the means are significantly different, but it does not tell you which means differ from each other. A simple approach is to test each possible difference by a paired t-test. However, the probability of making at least one type I error increases with the number of tests made. Multiple comparison procedures provide different solutions. We discuss these techniques and apply the well known Bonferroni method in order to determine the optimal degree in polynomial fitting and the optimal number of hidden neurons in feedforward neural networks.

Keywords

Hide Neuron Honestly Significant Difference Hide Unit Polynomial Fitting Optimal Degree 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Bishop, C. M.: Neural Network for Pattern Recognition. Clarendon Press-Oxford (1995)Google Scholar
  2. 2.
    Cobb, G.W.: Introduction to Design and Analysis of Experiments. Springer-Verlag New York (1998)MATHGoogle Scholar
  3. 3.
    Dean, A., Voss, D.: Design and Analysis of Experiments. Springer Texts in Statistics. Springer-Verlag New York (1999)Google Scholar
  4. 4.
    Dietterich, T.G.: Aproximate Statistical Test for Comparing Supervised Classification Learning Algorithms. Neural Computation (1998), Vol. 10, no. 7, 1895–1923CrossRefGoogle Scholar
  5. 5.
    Feelders, A., Verkooijen, W.: On the Statistical Comparison of Inductive Learning Methods. Learning from Data Artificial Intelligence and Statistics V. Springer-Verlag, New York (1996) 271–279Google Scholar
  6. 6.
    Hsu, J.C.: Multiple Comparisons: Theory and Methods. Chapman & Hall (1996)Google Scholar
  7. 7.
    Jobson, J.D.: Applied Multivariate Data Analysis. Springer Texts in Statistics, Vol 1. Springer-Verlag New York (1991)Google Scholar
  8. 8.
    Stone, M.: Cross-validatory Choice and Assesment of Statistical Prediction (with discussion). Journal of the Royal Statistical Society (1974), Series B, 36, 111–147MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2000

Authors and Affiliations

  • Pedro L. Galindo
    • 1
  • Joaquín Pizarro-Junquera
    • 1
  • Elisa Guerrero
    • 1
  1. 1.Dpto. Lenguajes y Sistemas Informáticos Grupo ”Sistemas Inteligentes de Computación”Universidad de Cádiz - CASEMPuerto Real (Cadiz)Spain

Personalised recommendations