Model selection bias and Freedman’s paradox Authors Paul M. Lukacs Colorado Division of Wildlife Kenneth P. Burnham U.S. Geological Survey, Colorado Cooperative Fish and Wildlife Research Unit Colorado State University David R. Anderson U.S. Geological Survey, Colorado Cooperative Fish and Wildlife Research Unit Colorado State University Article

First Online: 26 May 2009 Received: 16 October 2008 Revised: 10 February 2009 DOI :
10.1007/s10463-009-0234-4

Cite this article as: Lukacs, P.M., Burnham, K.P. & Anderson, D.R. Ann Inst Stat Math (2010) 62: 117. doi:10.1007/s10463-009-0234-4
Abstract In situations where limited knowledge of a system exists and the ratio of data points to variables is small, variable selection methods can often be misleading. Freedman (Am Stat 37:152–155, 1983) demonstrated how common it is to select completely unrelated variables as highly “significant” when the number of data points is similar in magnitude to the number of variables. A new type of model averaging estimator based on model selection with Akaike’s AIC is used with linear regression to investigate the problems of likely inclusion of spurious effects and model selection bias, the bias introduced while using the data to select a single seemingly “best” model from a (often large) set of models employing many predictor variables. The new model averaging estimator helps reduce these problems and provides confidence interval coverage at the nominal level while traditional stepwise selection has poor inferential properties.

Keywords Akaike’s information criterion Confidence interval coverage Freedman’s paradox Model averaging Model selection bias Model selection uncertainty Multimodel inference Stepwise selection

References Akaike H. (1973) Information theory as an extension of the maximum likelihood principle. In: Petrov B.N., Csaki F. (eds) Second international symposium on information theory. Budapest, Akademiai Kiado, pp 267–281

Akaike H. (1978) On the likelihood of a time series model. The Statistician 27: 217–235

CrossRef MathSciNet Akaike H. (1979) A Bayesian extension of the minimum AIC procedure of autoregressive model fitting. Biometrika 66: 237–242

MATH CrossRef MathSciNet Anderson D.R. (2008) Model based inference in the life sciences: primer on evidence. Springer, New York

MATH CrossRef Buckland S.T., Burnham K.P., Augustin N.H. (1997) Model selection: An integral part of inference. Biometrics 53: 603–618

MATH CrossRef Burnham K.P., Anderson D.R. (2002) Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). Springer, New York

MATH Burnham K.P., Anderson D.R. (2004) Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods and Research 33: 261–304

CrossRef MathSciNet Claeskens G., Hjort N.L. (2008) Model selection and model averaging. Cambridge University Press, New York

MATH Freedman D.A. (1983) A note on screening regression equations. The American Statistician 37: 152–155

CrossRef MathSciNet George E.I., McCulloch R.E. (1993) Variable selection via Gibbs sampling. Journal of the American Statistical Association 88: 881–889

CrossRef Hoeting J.A., Madigan D., Raftery A.E., Volinsky C.T. (1999) Bayesian model averaging: a tutorial (with discussion). Statistical Science 14: 382–417

MATH CrossRef MathSciNet Hurvich C.M., Tsai C.-L. (1989) Regression and time series model selection in small samples. Biometrika 76: 297–307

MATH CrossRef MathSciNet Hurvich C.M., Tsai C.-L. (1990) The impact of model selection on inference in linear regression. The American Statistician 44: 214–217

CrossRef Massart P. (2007) Concentration inequalities and model selection. Springer, Berlin

MATH McQuarrie A.D.R., Tsai C.-L. (1998) Regression and time series model selection. World Scientific Publishing Co., Singapore

MATH Miller A.J. (2002) Subset selection in regression (2nd ed.). Chapman and Hall, New York

MATH Rawlings J.O. (1988) Applied regression analysis: a research tool. Wadsworth, Inc, Belmont

MATH Rencher A.C., Pun F.C. (1980) Inflation of

R
^{2} in best subset regression. Technometrics 22: 49–53

MATH CrossRef SAS Institute, Inc. (2001). SAS version 8.02, Cary, NC.

Sugiura N. (1978) Further analysis of the data by Akaike’s information criterion and the finite corrections. Communications in Statistics, Theory and Methods A7: 13–26

MathSciNet Wheeler M.W. (2009) Comparing model averaging with other model selection strategies for benchmark dose estimation. Environmetrics and Ecological Statistics 16: 37–51

CrossRef Wheeler M.W., Bailer A.J. (2007) Properties of model-averaged BMDLs: a study of model averaging in dichotomous response risk estimation. Risk Analysis 27: 659–670

CrossRef Yang Y. (2007) Prediction/estimation with simple linear models: Is it really simple?. Econometric Theory 23: 1–36

CrossRef MathSciNet © The Institute of Statistical Mathematics, Tokyo 2009