Mathematical Methods of Statistics

, Volume 23, Issue 1, pp 20–37 | Cite as

Concentration inequalities for the exponential weighting method

  • Yu. Golubev
  • D. Ostrovski


The paper is concerned with recovering an unknown vector from noisy data with the help of a family of ordered smoothers [11]. The estimators within this family are aggregated based on the exponential weighting method and the performance of the aggregated estimate is measured by the excess risk controlling deviation of the square losses from the oracle risk. Based on natural statistical properties of ordered smoothers, we propose a novel method for obtaining concentration inequalities for the exponential weighting method.


ordered smoothers exponential weighting unbiased risk estimation oracle inequality concentration inequality 

2010 Mathematics Subject Classification

Primary 62C99 secondary 62C10, 62C20, 62J05. 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    P. Alquier and K. Lounici, “Pac-Bayesian Theorems for Sparse Regression Estimation with Exponential Weights”, Electronic J. Statist, 5, 127–145 (2011).CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    E. Arias-Castro and K. Lounici, “Variable Selection with Exponential Weights and 0-Penalization,” arXiv:1208.2635.Google Scholar
  3. 3.
    H. Akaike, “Information Theory and an Extension of the Maximum Likelihood Principle”, in Proc. 2nd Intern. Symp. Inf. Theory (1973), pp. 267–281.Google Scholar
  4. 4.
    O. Catoni, Statistical Learning Theory and Stochastic Optimization, in Lecture Notes in Math. (Springer, Berlin, 2004), Vol. 1851.Google Scholar
  5. 5.
    E. Chernousova, Yu. Golubev, and E. Krymova, “Ordered Smoothers With Exponential Weighting”, Electronic J. Statist. 7, 2395–2419 (2013).CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    A. Dalayan and J. Salmon, “Sharp Oracle Inequalities for Aggregation of Affine Estimators”, Ann. Statist. 40, 2327–2355 (2012).CrossRefMathSciNetGoogle Scholar
  7. 7.
    A. Demmler and C. Reinsch, “Oscillation Matrices with Spline Smoothing”, Numerische Math. 24, 375–382 (1975).CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Yu. Golubev, “On Universal Oracle Inequalities Related to High Dimensional Linear Models”, Ann. Statist. 38(5), 2751–2780 (2010).CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    G. Golubev, “Exponential Weighting and Oracle Inequalities for Projection Estimates”, Probl. Inform. Transmission 48(3), 269–280 (2012).Google Scholar
  10. 10.
    A. Juditsky and A. Nemirovski, “Functional Aggregation for Nonparametric Regression”, Ann. Statist. 28, 681–712 (2000).CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    A. Kneip, “Ordered Linear Smoothers”, Ann. Statist. 22, 835–866 (1994).CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    G. Lecu’e, “Simultaneous Adaptation to the Margin and to Complexity in Classification”, Ann. Statist. 35, 1698–1721 (2007).CrossRefMathSciNetGoogle Scholar
  13. 13.
    G. Leung and A. Barron, “Information Theory and Mixing Least-Squares Regressions”, IEEE Trans. Inform. Theory 52, 3396–3410 (2006).CrossRefMathSciNetGoogle Scholar
  14. 14.
    C. L. Mallows, “Some Comments on Cp”, Technometrics 15 661–675 (1973).zbMATHGoogle Scholar
  15. 15.
    A. Nemirovski, Topics in Non-Parametric Statistics, in Lectures Notes in Math. (Springer, Berlin, 2000), Vol. 1738.Google Scholar
  16. 16.
    P. Rigollet and A. Tsybakov, “Linear and Convex Aggregation of Density Estimators”, Math. Methods Statist. 16, 260–280 (2007).CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    P. Rigollet and A. Tsybakov, “Sparse Estimation by Exponential Weighting”, Statist. Sci. 27(4), 558–575 (2012).CrossRefMathSciNetGoogle Scholar
  18. 18.
    C. Stein, “Estimation of the Mean of a Multivariate Normal Distribution”, in Proc. Prague Symp. on Asymptotic Statistics (Prague, Czechoslovakia, 1973), pp. 345–381.Google Scholar
  19. 19.
    Y. Yang, “Combining Different Procedures for Adaptive Regression”, J. Multivariate Anal. 74, 135–161 (2000).CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Allerton Press, Inc. 2014

Authors and Affiliations

  1. 1.Aix Marseille Univ.MarseilleFrance
  2. 2.Inst. for Inform. Transmission ProblemsMoscowRussia
  3. 3.Inst. of Physics and TechnologyMoscowRussia

Personalised recommendations