Foundations of Computational Mathematics

, Volume 14, Issue 3, pp 569–600

Random Design Analysis of Ridge Regression

Authors

    • Department of Computer ScienceColumbia University
  • Sham M. Kakade
    • Microsoft Research
  • Tong Zhang
    • Department of StatisticsRutgers University
Article

DOI: 10.1007/s10208-014-9192-1

Cite this article as:
Hsu, D., Kakade, S.M. & Zhang, T. Found Comput Math (2014) 14: 569. doi:10.1007/s10208-014-9192-1

Abstract

This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the “out-of-sample” prediction error, as opposed to the “in-sample” (fixed design) error. The analysis also reveals the effect of errors in the estimated covariance structure, as well as the effect of modeling errors, neither of which effects are present in the fixed design setting. The proofs of the main results are based on a simple decomposition lemma combined with concentration inequalities for random vectors and matrices.

Keywords

Linear regressionOrdinary least squaresRidge regressionRandomized approximation

Mathematics Subject Classification

Primary 62J07Secondary 62J05

Copyright information

© SFoCM 2014