Risk bounds for model selection via penalization
- Andrew BarronAffiliated withDepartment of Statistics, Yale University, P.O. Box 208290, New Haven, CT 06520-8290, USA. e-mail: firstname.lastname@example.org
- , Lucien BirgéAffiliated withURA 1321 “Statistique et modèles aléatoires”, Laboratoire de Probabilités, boîte 188, Université Paris VI, 4 Place Jussieu, F-75252 Paris Cedex 05, France. e-mail: email@example.com
- , Pascal MassartAffiliated withURA 743 “Modélisation stochastique et Statistique”, Bât. 425, Université Paris Sud, Campus d'Orsay, F-91405 Orsay Cedex, France. e-mail: firstname.lastname@example.org
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size.
If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size).
We shall provide various illustrations of our method such as penalized maximum likelihood, projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions.
- Risk bounds for model selection via penalization
Probability Theory and Related Fields
Volume 113, Issue 3 , pp 301-413
- Cover Date
- Print ISSN
- Online ISSN
- Additional Links
- Mathematics subject classifications (1991): Primary 62G05, 62G07; secondary 41A25
- Key words and phrases: Penalization – Model selection – Adaptive estimation – Empirical processes – Sieves – Minimum contrast estimators
- Industry Sectors
- Author Affiliations
- A1. Department of Statistics, Yale University, P.O. Box 208290, New Haven, CT 06520-8290, USA. e-mail: email@example.com, US
- A2. URA 1321 “Statistique et modèles aléatoires”, Laboratoire de Probabilités, boîte 188, Université Paris VI, 4 Place Jussieu, F-75252 Paris Cedex 05, France. e-mail: firstname.lastname@example.org, FR
- A3. URA 743 “Modélisation stochastique et Statistique”, Bât. 425, Université Paris Sud, Campus d'Orsay, F-91405 Orsay Cedex, France. e-mail: email@example.com, FR