Random Design Analysis of Ridge Regression
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the “out-of-sample” prediction error, as opposed to the “in-sample” (fixed design) error. The analysis also reveals the effect of errors in the estimated covariance structure, as well as the effect of modeling errors, neither of which effects are present in the fixed design setting. The proofs of the main results are based on a simple decomposition lemma combined with concentration inequalities for random vectors and matrices.
- Ailon, N, Chazelle, B (2009) Approximate nearest neighbors and the fast Johnson–Lindenstrauss transform. SIAM J. Comput. 39: pp. 302-322 CrossRef
- J.-Y. Audibert and O. Catoni. Linear regression through PAC-Bayesian truncation, 2010. arXiv:1010.0072.
- Audibert, J-Y, Catoni, O (2011) Robust linear least squares regression. Ann. Stat. 30: pp. 2766-2794 CrossRef
- Caponnetto, A, Vito, E (2007) Optimal rates for the regularized least-squares algorithm. Found. Comput. Math. 7: pp. 331-368 CrossRef
- O. Catoni. Statistical Learning Theory and Stochastic Optimization, Lectures on Probability and Statistics, Ecole d’Eté de Probabilitiés de Saint-Flour XXXI - 2001, volume 1851 of Lecture Notes in Mathematics. Springer, 2004.
- P. Drineas and M. W. Mahoney. Effective Resistances, Statistical Leverage, and Applications to Linear Equation Solving, 2010. arXiv:1005.3097.
- Drineas, P, Mahoney, MW, Muthukrishnan, S, Sarlós, T (2010) Faster least squares approximation. Numerische Mathematik 117: pp. 219-249 CrossRef
- L. Györfi, M. Kohler, A. Kryżak, and H. Walk. A Distribution-Free Theory of Nonparametric Regression. Springer, 2004.
- Hoerl, AE (1962) Application of ridge analysis to regression problems. Chem. Eng. Prog. 58: pp. 54-59
- R. Horn and C. R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
- D. Hsu, S. M. Kakade, and T. Zhang. A tail inequality for quadratic forms of subgaussian random vectors, 2011. arXiv:1110.2842.
- Hsu, D, Kakade, SM, Zhang, T (2012) Tail inequalities for sums of random matrices that depend on the intrinsic dimension. Electron. Commun. Probab. 17: pp. 1-13
- D. Hsu and S. Sabato. Loss Minimization and Parameter Estimation with Heavy Tails, 2013. arXiv:1307.1827.
- Koltchinskii, V (2006) Local Rademacher complexities and oracle inequalities in risk minimization. Ann. Stat. 34: pp. 2593-2656 CrossRef
- Laurent, B, Massart, P (2000) Adaptive estimation of a quadratic functional by model selection. Ann. Stat. 28: pp. 1302-1338 CrossRef
- E. L. Lehmann and G. Casella. Theory of Point Estimation. Springer, second edition, 1998.
- M. Nussbaum. Minimax risk: Pinsker bound. In S. Kotz, editor, Encyclopedia of Statistical Sciences, Update Volume 3, pages 451–460. Wiley, New York, 1999.
- Rokhlin, V, Tygert, M (2008) A fast randomized algorithm for overdetermined linear least-squares regression. Proc. Natl. Acad. Sci. USA 105: pp. 13212-13217 CrossRef
- Smale, S, Zhou, D-X (2007) Learning theory estimates via integral operators and their approximations. Constr. Approx. 26: pp. 153-172 CrossRef
- I. Steinwart, D. Hush, and C. Scovel. Optimal Rates for Regularized Least Squares Regression. In Proceedings of the 22nd Annual Conference on Learning Theory, pp. 79–93, 2009.
- G. W. Stewart and J.-G. Sun. Matrix Perturbation Theory. Academic Press, 1990.
- Stone, CJ (1982) Optimal global rates of convergence for nonparametric regression. Ann. Stat. 10: pp. 1040-1053 CrossRef
- Zhang, T (2005) Learning bounds for kernel regression using effective data dimensionality. Neural Comput. 17: pp. 2077-2098 CrossRef
- Random Design Analysis of Ridge Regression
Foundations of Computational Mathematics
Volume 14, Issue 3 , pp 569-600
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Linear regression
- Ordinary least squares
- Ridge regression
- Randomized approximation
- Primary 62J07
- Secondary 62J05
- Industry Sectors
- Author Affiliations
- 1. Department of Computer Science, Columbia University, 450 Computer Science Building, 1214 Amsterdam Avenue, Mailcode 0401, New York, NY, 10027-7003, USA
- 2. Microsoft Research, One Memorial Drive, Cambridge, MA, 02142, USA
- 3. Department of Statistics, Rutgers University, 501 Hill Center, 110 Frelinghuysen Road, Piscataway, NJ, 08854, USA