Skip to main content
SpringerLink
Log in

A general framework for frequentist model averaging

  • Articles
  • Published:
Science China Mathematics Aims and scope Submit manuscript

Abstract

Model selection strategies have been routinely employed to determine a model for data analysis in statistics, and further study and inference then often proceed as though the selected model were the true model that were known a priori. Model averaging approaches, on the other hand, try to combine estimators for a set of candidate models. Specifically, instead of deciding which model is the `right' one, a model averaging approach suggests to fit a set of candidate models and average over the estimators using data adaptive weights. In this paper we establish a general frequentist model averaging framework that does not set any restrictions on the set of candidate models. It broadens the scope of the existing methodologies under the frequentist model averaging development. Assuming the data is from an unknown model, we derive the model averaging estimator and study its limiting distributions and related predictions while taking possible modeling biases into account. We propose a set of optimal weights to combine the individual estimators so that the expected mean squared error of the average estimator is minimized. Simulation studies are conducted to compare the performance of the estimator with that of the existing methods. The results show the benefits of the proposed approach over traditional model selection approaches as well as existing model averaging methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Billingsley P. Probability and Measure. Chichester: John Wiley & Sons, 2008

    MATH  Google Scholar 

  2. Buckland S T, Burnham K P, Augustin N H. Model selection: An integral part of inference. Biometrics, 1997, 53: 603–618

    Article  MATH  Google Scholar 

  3. Claeskens G, Hjort N L. Model selection and model averaging. J Math Psych, 2008, 44: 92–107

    MathSciNet  MATH  Google Scholar 

  4. Danilov D, Magnus J R. Forecast accuracy after pretesting with an application to the stock market. J Forecast, 2004, 23: 251–274

    Article  Google Scholar 

  5. Danilov D, Magnus J R. On the harm that ignoring pretesting can cause. J Econometrics, 2004, 122: 27–46

    Article  MathSciNet  MATH  Google Scholar 

  6. Draper D. Assessment and propagation of model uncertainty. J R Stat Soc Ser B Stat Methodol, 1995, 57: 45–97

    MathSciNet  MATH  Google Scholar 

  7. Gao Y, Zhang X, Wang S, et al. Model averaging based on leave-subject-out cross-validation. J Econometrics, 2016, 192: 139–151

    Article  MathSciNet  MATH  Google Scholar 

  8. Giles D E A, Lieberman O, Giles J A. The optimal size of a preliminary test of linear restrictions in a misspecified regression model. J Amer Statist Assoc, 1992, 87: 1153–1157

    Article  Google Scholar 

  9. Hansen B E. Least squares model averaging. Econometrica, 2007, 75: 1175–1189

    Article  MathSciNet  MATH  Google Scholar 

  10. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference and Prediction. New York: Springer-Verlag, 2009

    Book  MATH  Google Scholar 

  11. Hjort N L, Claeskens G. Frequentist model average estimators. J Amer Statist Assoc, 2003, 98: 879–899

    Article  MathSciNet  MATH  Google Scholar 

  12. Hjort N L, Claeskens G. Focused information criteria and model averaging for the Cox hazard regression model. J Amer Statist Assoc, 2006, 101: 1449–1464

    Article  MathSciNet  MATH  Google Scholar 

  13. Hoeting J A, Madigan D, Raftery A E, et al. Bayesian model averaging. Statist Sci, 1999, 14: 121–149

    MathSciNet  MATH  Google Scholar 

  14. Holland P W, Welsch R E. Robust regression using iteratively reweighted least-squares. Comm Statist Theory Methods, 2007, 6: 813–827

    Article  MATH  Google Scholar 

  15. Hurvich C M, Tsai C-L. Regression and time series model selection in small samples. Biometrika, 1989, 76: 297–307

    Article  MathSciNet  MATH  Google Scholar 

  16. Hurvich C M, Tsai C-L. Bias of the corrected AIC criterion for underfitted regression and time series models. Biometri- ka, 1991, 78: 499–509

    MathSciNet  MATH  Google Scholar 

  17. Karagrigoriou A, Lee S, Mattheou K. A model selection criterion based on the BHHJ measure of divergence. J Statist Plann Inference, 2009, 139: 228–235

    Article  MathSciNet  MATH  Google Scholar 

  18. Lehmann E L. Elements of Large-Sample Theory. Springer Texts in Statistics. New York: Springer-Verlag, 1999

    Book  MATH  Google Scholar 

  19. Lehmann E L, Casella G. Theory of Point Estimation, 2nd ed. Springer Texts in Statistics. New York: Springer-Verlag, 1998

    MATH  Google Scholar 

  20. Liang H, Zou G, Wan A T K, et al. Optimal weight choice for frequentist model average estimators. J Amer Statist Assoc, 2011, 106: 1053–1066

    Article  MathSciNet  MATH  Google Scholar 

  21. Lien D, Shrestha K. Estimating the optimal hedge ratio with focus information criterion. J Futures Markets, 2005, 25: 1011–1024

    Article  Google Scholar 

  22. Madigan D, Raftery A E, York J C, et al. Strategies for graphical model selection. In: Selecting Models from Data: Artificial Intelligence and Statistics IV. Lecture Notes in Statistics, vol. 89. New York: Springer, 1994, 91–100

    Book  MATH  Google Scholar 

  23. Magnus J R, Wan A T K, Zhang X. Weighted average least squares estimation with nonspherical disturbances and an application to the Hong Kong housing market. Comput Statist Data Anal, 2011, 55: 1331–1341

    Article  MathSciNet  MATH  Google Scholar 

  24. Mitra P. Topics in model averaging & toxicity models in combination therapy. PhD Thesis. New Brunswick: Rutgers University, 2015

    Google Scholar 

  25. Pesaran M H, Schleicher C, Zaffaroni P. Model averaging in risk management with an application to futures markets. J Empir Finance, 2009, 16: 280–305

    Article  Google Scholar 

  26. Posada D, Buckley T R. Model selection and model averaging in phylogenetics: Advantages of Akaike information criterion and Bayesian approaches over likelihood ratio tests. Systematic Biology, 2004, 53: 793–808

    Article  Google Scholar 

  27. Raftery A E, Madigan D, Hoeting J A. Bayesian model averaging for linear regression models. J Amer Statist Assoc, 1997, 92: 179–191

    Article  MathSciNet  MATH  Google Scholar 

  28. Stamey T A, Kabalin J N, Ferrari M, et al. Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate, IV: Anti-androgen treated patients. J Urol, 1989, 141: 1088–1090

    Article  Google Scholar 

  29. Thursby J G, Schmidt P. Some properties of tests for specification error in a linear regression model. J Amer Statist Assoc, 1977, 72: 635–641

    Article  MATH  Google Scholar 

  30. Van der Vaart A W. Asymptotic Statistics, Volume 3. Cambridge: Cambridge University Press, 2000

    Google Scholar 

  31. Wan A T K, Zhang X, Zou G. Least squares model averaging by Mallows criterion. J Econometrics, 2010, 156: 277–283

    Article  MathSciNet  MATH  Google Scholar 

  32. Wei Y, McNicholas P D. Mixture model averaging for clustering and classification. Adv Data Anal Classif, 2015, 22: 197–217

    Article  Google Scholar 

  33. Zhang X, Wan A T K, Zhou S Z. Focused information criteria, model selection and model averaging in a tobit model with a non-zero threshold. J Bus Econom Statist, 2012, 30: 132–142

    Article  MathSciNet  Google Scholar 

  34. Zhang X, Yu D, Zou G, et al. Optimal model averaging estimation for generalized linear models and generalized linear mixed-effects models. J Amer Statist Assoc, 2016, 111: 1775–1790

    Article  MathSciNet  Google Scholar 

  35. Zhang X, Zou G, Carroll R J. Model averaging based on Kullback-Leibler distance. Statist Sinica, 2015, 25: 1583–1598

    MathSciNet  MATH  Google Scholar 

  36. Zhang X, Zou G, Liang H. Model averaging and weight choice in linear mixed-effects models. Biometrika, 2014, 101: 205–218

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The work was supported by National Science Foundation of USA (Grant Nos. DMS- 1812048, DMS-1737857, DMS-1513483 and DMS-1418042) and National Natural Science Foundation of China (Grant No. 11529101). This article is a work developed based on the thesis of the first author. The authors wish to use this article to celebrate Professor Lincheng Zhao's 75th birthday and his tremendous and long lasting contribution to statistical research and education in China and around the world. The authors also thank two referees for their valuable suggestions and comments that have helped improve the paper substantially.

Author information

Authors and Affiliations

  1. Department of Statistics and Biostatistics, Rutgers University, Piscataway, NJ, 08854, USA

    Priyam Mitra & Min-ge Xie

  2. Department of Mathematics, City University of Hong Kong, Hong Kong, China

    Heng Lian

  3. Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ, 08540, USA

    Ritwik Mitra

  4. Department of Statistics, George Washington University, Washington, DC, 20052, USA

    Hua Liang

Authors
  1. Priyam Mitra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heng Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ritwik Mitra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hua Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Min-ge Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Min-ge Xie.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mitra, P., Lian, H., Mitra, R. et al. A general framework for frequentist model averaging. Sci. China Math. 62, 205–226 (2019). https://doi.org/10.1007/s11425-018-9403-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11425-018-9403-x

Keywords

MSC(2010)

Search

Navigation