Competing against the Best Nearest Neighbor Filter in Regression
Designing statistical procedures that are provably almost as accurate as the best one in a given family is one of central topics in statistics and learning theory. Oracle inequalities offer then a convenient theoretical framework for evaluating different strategies, which can be roughly classified into two classes: selection and aggregation strategies. The ultimate goal is to design strategies satisfying oracle inequalities with leading constant one and rate-optimal residual term. In many recent papers, this problem is addressed in the case where the aim is to beat the best procedure from a given family of linear smoothers. However, the theory developed so far either does not cover the important case of nearest-neighbor smoothers or provides a suboptimal oracle inequality with a leading constant considerably larger than one. In this paper, we prove a new oracle inequality with leading constant one that is valid under a general assumption on linear smoothers allowing, for instance, to compete against the best nearest-neighbor filters.
Keywordsadaptive smoothing nonparametric regression supervised learning
Unable to display preview. Download preview PDF.
- 1.Arlot, S., Bach, F.: Data-driven calibration of linear estimators with minimal penalties. In: NIPS, pp. 46–54 (2009)Google Scholar
- 4.Baraud, Y., Giraud, C., Huet, S.: Estimator selection in the gaussian setting (2010) (submitted)Google Scholar
- 5.Ben-David, S., Pal, D., Shalev-Shwartz, S.: Agnostic online learning. In: COLT (2009)Google Scholar
- 11.Cornillon, P.-A., Hengartner, N., Matzner-Løber, E.: Recursive bias estimation for multivariate regression smoothers (2009) (submitted)Google Scholar
- 12.Dalalyan, A.S., Salmon, J.: Sharp oracle inequalities for aggregation of affine estimators. technical report, arXiv:1104.3969v2 [math.ST] (2011)Google Scholar
- 14.Dalalyan, A.S., Tsybakov, A.B.: Sparse regression learning by aggregation and Langevin Monte-Carlo. In: COLT (2009)Google Scholar
- 17.Gerchinovitz, S.: Sparsity regret bounds for individual sequences in online linear regression (submitted, 2011)Google Scholar
- 26.Salmon, J., Dalalyan, A.S.: Optimal aggregation of affine estimators. In: COLT (2011)Google Scholar
- 27.Stein, C.M.: Estimation of the mean of a multivariate distribution. In: Proc. Prague Symp. Asymptotic Statist (1973)Google Scholar
- 28.Tsybakov, A.B.: Optimal rates of aggregation. In: COLT, pp. 303–313 (2003)Google Scholar