Abstract
We discuss the impact of tuning parameter selection uncertainty in the context of shrinkage estimation and propose a methodology to account for problems arising from this issue: Transferring established concepts from model averaging to shrinkage estimation yields the concept of shrinkage averaging estimation (SAE) which reflects the idea of using weighted combinations of shrinkage estimators with different tuning parameters to improve overall stability, predictive performance and standard errors of shrinkage estimators. Two distinct approaches for an appropriate weight choice, both of which are inspired by concepts from the recent literature of model averaging, are presented: The first approach relates to an optimal weight choice with regard to the predictive performance of the final weighted estimator and its implementation can be realized via quadratic programming. The second approach has a fairly different motivation and considers the construction of weights via a resampling experiment. Focusing on Ridge, Lasso and Random Lasso estimators, the properties of the proposed shrinkage averaging estimators resulting from these strategies are explored by means of Monte-Carlo studies and are compared to traditional approaches where the tuning parameter is simply selected via cross validation criteria. The results show that the proposed SAE methodology can improve an estimators’ overall performance and reveal and incorporate tuning parameter uncertainty. As an illustration, selected methods are applied to some recent data from a study on leadership behavior in life science companies.
Similar content being viewed by others
References
Breiman L., Spector P (1992) Submodel selection and evaluation in regression: the X-random case. Int Stat Rev 60: 291–319
Buchholz A, HollSnder N, Sauerbrei W (2008) On properties of predictors derived with a two-step bootstrap model averaging approach - A simulation study in the linear regression model. Comput Stat Data Anal 52: 2778–2793
Buckland TS, Burnham PK, Augustin HN (1997) Model selection: an integral part of inference. Biometrics 53: 603–618
Burnham K, Anderson D (2002) Model selection and multimodel inference. A practical information-theoretic approach. Springer, New York
Candolo C., Davison AC, DemTtrio CGB (2003) A note on model uncertainty in linear regression. Statistician 52: 165–177
Fan J, Li R (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96: 1348–1360
Golub HG, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21: 215–223
Hansen BE (2007) Least squares model averaging. Econometrica 75: 1175–1189
Hansen BE, Racine J (2011) Jackknife model averaging. J Econ (forthcoming)
Hastie T, Tibsharani R, Friedman J (2001) Elements of statistical learning. Springer, New York
Hjort L, Claeskens G (2003) Frequentist model average estimators. J Am Stat Assoc 98: 879–945
Hjort LN, Claeskens G (2006) Focussed information criteria and model averaging for Cox’s hazard regression model. J Am Stat Assoc 101: 1449–1464
Hoerl A, Kennard R (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12: 55–67
Hoeting JA, Madigan D, Raftery AE, Volinsky CT (1999) Bayesian model averaging: a tutorial. Stat Sci 14: 382–417
Klaußner A (2007) Phasenangepasste Führung von Wachstumsunternehmen (in German). International University Schloss Reichartshausen
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. International Joint Conference on Artificial Intelligence (IJCAI) 1137–1143
Liang H, Zou GH, Wan ATK., Zhang X (2011) Optimal weight choice for frequentist model average estimators. J Am Stat Assoc. doi:10.1198/jasa.2011.tm09478
Magnus JR, Powell O, Prnfer P (2010) A comparison of two model averaging techniques with an application to growth empirics. J Econ 154: 139–153
Oh H, Nychka D, Brown T, Charbonneau P (2004) Analysis of variable stars by robust smoothing. J R Stat Soc C 53: 15–30
Radchenko P, James MG (2008) Variable inclusion and shrinkage algorithms. J Am Stat Assoc 103: 1304–1315
R Development Core Team (2010) R: A language and environment for statistical computing. R foundation for statistical computing. Vienna, Austria, ISBN:3-900051-07-0, http://www.R-project.org
Schomaker M, Wan ATK, Heumann C (2010) Frequentist model averaging with missing observations. Comput Stat Data Anal 54: 3336–3347
Shao J (1997) An asymptotic theory for linear model selection. Stat Sinica 7: 221–264
Stone M (1974) Cross-validatory choice and assessment of statistical predictions. J R Stat Soc B 36: 111–147
Tibsharani R (1996) Regression shrinkage and selection via the Lasso. J R Stat Soc B 58: 267–288
Valdés-Sosa P, Sßnchez-Bornot J, Lage-Castellanos A, Vega-Hernaindez M, Bosch-Bayard J, Melie-Garcia L, Canales-Rodriguez E (2005) Estimating brain functional connectivity with sparse multivariate autoregression. Philos Trans R Soc 360: 969–981
Wan ATK, Zhang X (2009) On the use of model averaging in tourism research. Ann Tour Res 36: 525–532
Wan ATK, Zhang X, Zou HG (2010) Least squares model averaging by Mallows criterion. J Econ 156: 277–283
Wang H, Li B, Leng C (2009a) Shrinkage tuning parameter selection with a diverging number of parameters. J R Stat Soc B 71: 671–683
Wang H, Zhang X, Zou G (2009b) Frequentist model averaging: a review. J Syst Sci Complex 22: 732–748
Wang S, Nan B, Rosset S, Zhu J (2011) Random Lasso. Ann Appl Stat 5: 468–485
Yan J (2007) Enjoy the joy of copulas: with package copula. Journal of Statistical Software 21: 1–21
Yuan Z, Yang Y (2005) Combining linear regression models: when and how?. J Am Stat Assoc 100: 1215–1225
Zhang X., Wan A.T.K., Zhou SZ (2011) Focused information criteria, model selection and model averaging in a Tobit model with a non-zero threshold. J Busi Econ Stat. doi:10.1198/jbes.2011.10075
Zou H, Hastie T (2005) Regularization and variable selection via the Elastic Net. J R Stat Soc B 67: 301–320
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Schomaker, M. Shrinkage averaging estimation. Stat Papers 53, 1015–1034 (2012). https://doi.org/10.1007/s00362-011-0405-2
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00362-011-0405-2
Keywords
- Tuning parameter selection uncertainty
- Model averaging
- Model selection
- Optimal weight choice
- Cross validation