Skip to main content
SpringerLink
Log in

Shrinkage averaging estimation

  • Regular Article
  • Published:
Statistical Papers Aims and scope Submit manuscript

Abstract

We discuss the impact of tuning parameter selection uncertainty in the context of shrinkage estimation and propose a methodology to account for problems arising from this issue: Transferring established concepts from model averaging to shrinkage estimation yields the concept of shrinkage averaging estimation (SAE) which reflects the idea of using weighted combinations of shrinkage estimators with different tuning parameters to improve overall stability, predictive performance and standard errors of shrinkage estimators. Two distinct approaches for an appropriate weight choice, both of which are inspired by concepts from the recent literature of model averaging, are presented: The first approach relates to an optimal weight choice with regard to the predictive performance of the final weighted estimator and its implementation can be realized via quadratic programming. The second approach has a fairly different motivation and considers the construction of weights via a resampling experiment. Focusing on Ridge, Lasso and Random Lasso estimators, the properties of the proposed shrinkage averaging estimators resulting from these strategies are explored by means of Monte-Carlo studies and are compared to traditional approaches where the tuning parameter is simply selected via cross validation criteria. The results show that the proposed SAE methodology can improve an estimators’ overall performance and reveal and incorporate tuning parameter uncertainty. As an illustration, selected methods are applied to some recent data from a study on leadership behavior in life science companies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Breiman L., Spector P (1992) Submodel selection and evaluation in regression: the X-random case. Int Stat Rev 60: 291–319

    Article  Google Scholar 

  • Buchholz A, HollSnder N, Sauerbrei W (2008) On properties of predictors derived with a two-step bootstrap model averaging approach - A simulation study in the linear regression model. Comput Stat Data Anal 52: 2778–2793

    Article  MATH  Google Scholar 

  • Buckland TS, Burnham PK, Augustin HN (1997) Model selection: an integral part of inference. Biometrics 53: 603–618

    Article  MATH  Google Scholar 

  • Burnham K, Anderson D (2002) Model selection and multimodel inference. A practical information-theoretic approach. Springer, New York

    MATH  Google Scholar 

  • Candolo C., Davison AC, DemTtrio CGB (2003) A note on model uncertainty in linear regression. Statistician 52: 165–177

    Article  MathSciNet  Google Scholar 

  • Fan J, Li R (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96: 1348–1360

    Article  MathSciNet  MATH  Google Scholar 

  • Golub HG, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21: 215–223

    Article  MathSciNet  MATH  Google Scholar 

  • Hansen BE (2007) Least squares model averaging. Econometrica 75: 1175–1189

    Article  MathSciNet  MATH  Google Scholar 

  • Hansen BE, Racine J (2011) Jackknife model averaging. J Econ (forthcoming)

  • Hastie T, Tibsharani R, Friedman J (2001) Elements of statistical learning. Springer, New York

    MATH  Google Scholar 

  • Hjort L, Claeskens G (2003) Frequentist model average estimators. J Am Stat Assoc 98: 879–945

    Article  MathSciNet  MATH  Google Scholar 

  • Hjort LN, Claeskens G (2006) Focussed information criteria and model averaging for Cox’s hazard regression model. J Am Stat Assoc 101: 1449–1464

    Article  MathSciNet  MATH  Google Scholar 

  • Hoerl A, Kennard R (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12: 55–67

    Article  MATH  Google Scholar 

  • Hoeting JA, Madigan D, Raftery AE, Volinsky CT (1999) Bayesian model averaging: a tutorial. Stat Sci 14: 382–417

    Article  MathSciNet  MATH  Google Scholar 

  • Klaußner A (2007) Phasenangepasste Führung von Wachstumsunternehmen (in German). International University Schloss Reichartshausen

  • Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. International Joint Conference on Artificial Intelligence (IJCAI) 1137–1143

  • Liang H, Zou GH, Wan ATK., Zhang X (2011) Optimal weight choice for frequentist model average estimators. J Am Stat Assoc. doi:10.1198/jasa.2011.tm09478

  • Magnus JR, Powell O, Prnfer P (2010) A comparison of two model averaging techniques with an application to growth empirics. J Econ 154: 139–153

    Google Scholar 

  • Oh H, Nychka D, Brown T, Charbonneau P (2004) Analysis of variable stars by robust smoothing. J R Stat Soc C 53: 15–30

    Article  MathSciNet  MATH  Google Scholar 

  • Radchenko P, James MG (2008) Variable inclusion and shrinkage algorithms. J Am Stat Assoc 103: 1304–1315

    Article  MathSciNet  MATH  Google Scholar 

  • R Development Core Team (2010) R: A language and environment for statistical computing. R foundation for statistical computing. Vienna, Austria, ISBN:3-900051-07-0, http://www.R-project.org

  • Schomaker M, Wan ATK, Heumann C (2010) Frequentist model averaging with missing observations. Comput Stat Data Anal 54: 3336–3347

    Article  MathSciNet  Google Scholar 

  • Shao J (1997) An asymptotic theory for linear model selection. Stat Sinica 7: 221–264

    MATH  Google Scholar 

  • Stone M (1974) Cross-validatory choice and assessment of statistical predictions. J R Stat Soc B 36: 111–147

    MATH  Google Scholar 

  • Tibsharani R (1996) Regression shrinkage and selection via the Lasso. J R Stat Soc B 58: 267–288

    Google Scholar 

  • Valdés-Sosa P, Sßnchez-Bornot J, Lage-Castellanos A, Vega-Hernaindez M, Bosch-Bayard J, Melie-Garcia L, Canales-Rodriguez E (2005) Estimating brain functional connectivity with sparse multivariate autoregression. Philos Trans R Soc 360: 969–981

    Article  Google Scholar 

  • Wan ATK, Zhang X (2009) On the use of model averaging in tourism research. Ann Tour Res 36: 525–532

    Article  Google Scholar 

  • Wan ATK, Zhang X, Zou HG (2010) Least squares model averaging by Mallows criterion. J Econ 156: 277–283

    MathSciNet  Google Scholar 

  • Wang H, Li B, Leng C (2009a) Shrinkage tuning parameter selection with a diverging number of parameters. J R Stat Soc B 71: 671–683

    Article  MathSciNet  Google Scholar 

  • Wang H, Zhang X, Zou G (2009b) Frequentist model averaging: a review. J Syst Sci Complex 22: 732–748

    Article  MathSciNet  Google Scholar 

  • Wang S, Nan B, Rosset S, Zhu J (2011) Random Lasso. Ann Appl Stat 5: 468–485

    Article  MathSciNet  MATH  Google Scholar 

  • Yan J (2007) Enjoy the joy of copulas: with package copula. Journal of Statistical Software 21: 1–21

    Google Scholar 

  • Yuan Z, Yang Y (2005) Combining linear regression models: when and how?.   J Am Stat Assoc 100: 1215–1225

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang X., Wan A.T.K., Zhou SZ (2011) Focused information criteria, model selection and model averaging in a Tobit model with a non-zero threshold. J Busi Econ Stat. doi:10.1198/jbes.2011.10075

  • Zou H, Hastie T (2005) Regularization and variable selection via the Elastic Net. J R Stat Soc B 67: 301–320

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Infectious Diseases and Epidemiology Research, University of Cape Town, Anzio Road, Observatory, 7925, Cape Town, South Africa

    Michael Schomaker

Authors
  1. Michael Schomaker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Schomaker.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schomaker, M. Shrinkage averaging estimation. Stat Papers 53, 1015–1034 (2012). https://doi.org/10.1007/s00362-011-0405-2

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00362-011-0405-2

Keywords

Mathematics Subject Classification (2000)

Search

Navigation