Designing Robust Regression Models

Abstract

In this study we focus on the preference among competing models from a family of polynomial regressors. Classical statistics offers a number of wellknown techniques for the selection of models in polynomial regression, namely, Finite Prediction Error (FPE) [1], Akaike’s Information Criterion (AIC) [2], Schwartz’s criterion (SCH) [10] and Generalized Cross Validation (GCV) [4]. Wallace’s Minimum Message Length (MML) principle [16, 17, 18] and also Vapnik’s Structural Risk Minimization (SRM) [11, 12]–based on the classical theory of VC-dimensionality–are plausible additions to this family of modelselection principles. SRM and MML are generic in the sense that they can be applied to any family of models, and similar in their attempt to define a trade-off between the complexity of a given model and its goodness of fit to the data under observation–although they do use different trade-offs, with MML’s being Bayesian and SRM’s being non-Bayesian in principle. Recent empirical evaluations [14, 15] comparing the performance of several methods for polynomial degree selection provide strong evidence in support of the MML and SRM methods over the other techniques.