Probability Theory and Related Fields

, Volume 113, Issue 3, pp 301–413

Risk bounds for model selection via penalization

  • Andrew Barron
  • Lucien Birgé
  • Pascal Massart

DOI: 10.1007/s004400050210

Cite this article as:
Barron, A., Birgé, L. & Massart, P. Probab. Theory Relat. Fields (1999) 113: 301. doi:10.1007/s004400050210


Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size.

If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size).

We shall provide various illustrations of our method such as penalized maximum likelihood, projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions.

Mathematics subject classifications (1991): Primary 62G05, 62G07; secondary 41A25
Key words and phrases: Penalization – Model selection – Adaptive estimation – Empirical processes – Sieves – Minimum contrast estimators

Copyright information

© Springer-Verlag Berlin Heidelberg 1999

Authors and Affiliations

  • Andrew Barron
    • 1
  • Lucien Birgé
    • 2
  • Pascal Massart
    • 3
  1. 1.Department of Statistics, Yale University, P.O. Box 208290, New Haven, CT 06520-8290, USA. e-mail: barron@stat.yale.eduUS
  2. 2.URA 1321 “Statistique et modèles aléatoires”, Laboratoire de Probabilités, boîte 188, Université Paris VI, 4 Place Jussieu, F-75252 Paris Cedex 05, France. e-mail: lb@ccr.jussieu.frFR
  3. 3.URA 743 “Modélisation stochastique et Statistique”, Bât. 425, Université Paris Sud, Campus d'Orsay, F-91405 Orsay Cedex, France. e-mail: massart@stats.matups.frFR