Aggregation and Sparsity Via ℓ1 Penalized Least Squares

  • Florentina Bunea
  • Alexandre B. Tsybakov
  • Marten H. Wegkamp
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4005)


This paper shows that near optimal rates of aggregation and adaptation to unknown sparsity can be simultaneously achieved via ℓ1 penalized least squares in a nonparametric regression setting. The main tool is a novel oracle inequality on the sum between the empirical squared loss of the penalized least squares estimate and a term reflecting the sparsity of the unknown regression function.


Optimal Rate Reproduce Kernel Hilbert Space Arbitrary Positive Number Oracle Inequality Dantzig Selector 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baraud, Y.: Model selection for regression on a random design. ESAIM Probability & Statistics 7, 127–146 (2002)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Birgé, L.: Model selection for Gaussian regression with random design. Prépublication n. 783, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 - Paris 7 (2002),
  3. 3.
    Birgé, L., Massart, P.: Minimum contrast estimators on sieves: Exponential bounds and rates of convergence. Bernouilli 4, 329–375 (1998)CrossRefMATHGoogle Scholar
  4. 4.
    Bunea, F., Tsybakov, A., Wegkamp, M.H.: Aggregation for Gaussian regression (preprint, 2005),
  5. 5.
    Bunea, F., Wegkamp, M.: Two stage model selection procedures in partially linear regression. The Canadian Journal of Statistics 22, 1–14 (2004)Google Scholar
  6. 6.
    Candes, E., Tao,T.: The Dantzig selector: statistical estimation when p is much larger than n (preprint, 2005)Google Scholar
  7. 7.
    Catoni, O.: Statistical Learning Theory and Stochastic Optimization. In: Ecole d’Eté de Probabilités de Saint-Flour 2001. Lecture Notes in Mathematics, Springer, N.Y (2004)Google Scholar
  8. 8.
    Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425–455 (1994)CrossRefMathSciNetMATHGoogle Scholar
  9. 9.
    Györfi, L., Kohler, M., Krzyżak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, N.Y (2002)CrossRefMATHGoogle Scholar
  10. 10.
    Juditsky, A., Nemirovski, A.: Functional aggregation for nonparametric estimation. Annals of Statistics 28, 681–712 (2000)CrossRefMathSciNetMATHGoogle Scholar
  11. 11.
    Kerkyacharian, G., Picard, D.: Tresholding in learning theory. Prépublication n.1017, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 - Paris 7,
  12. 12.
    Koltchinskii, V.: Model selection and aggregation in sparse classification problems. Oberwolfach Reports: Meeting on Statistical and Probabilistic Methods of Model Selection, (October 2005) (to appear)Google Scholar
  13. 13.
    Nemirovski, A.: Topics in Non-parametric Statistics. In: Ecole d’Eté de Probabilités de Saint-Flour XXVIII. Lecture Notes in Mathematics, vol. 1738, Springer, N.Y (2000)Google Scholar
  14. 14.
    Tibshirani, R.: Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B. 58, 267–288 (1996)MathSciNetMATHGoogle Scholar
  15. 15.
    Tsybakov, A.B.: Optimal rates of aggregation. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS (LNAI), vol. 2777, pp. 303–313. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  16. 16.
    Wegkamp, M.H.: Model selection in nonparametric regression. Annals of Statistics 31, 252–273 (2003)CrossRefMathSciNetMATHGoogle Scholar
  17. 17.
    Yang, Y.: Combining different procedures for adaptive regression. J.of Multivariate Analysis 74, 135–161 (2000)CrossRefMATHGoogle Scholar
  18. 18.
    Yang, Y.: Aggregating regression procedures for a better performance. Bernoulli 10, 25–47 (2004)CrossRefMathSciNetMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Florentina Bunea
    • 1
  • Alexandre B. Tsybakov
    • 2
    • 3
  • Marten H. Wegkamp
    • 1
  1. 1.Department of StatisticsFlorida State UniversityTallahasseeUSA
  2. 2.Laboratoire de Probabilités et Modèles AléatoiresUniversité Paris VIPARISFrance
  3. 3.Institute for Information Transmission ProblemsMoscowRussia

Personalised recommendations