Skip to main content

Strong consistency of Lasso estimators

Abstract

In this paper, we study the strong consistency and rates of convergence of the Lasso estimator. It is shown that when the error variables have a finite mean, the Lasso estimator is strongly consistent, provided the penalty parameter (say, λ n ) is of smaller order than the sample size (say n). We also show that this condition on λ n cannot be relaxed. More specifically, we show that consistency of the Lasso estimators fail in the cases where λ n /nα for some α ∈ (0, ∞]. For error variables with a finite αth moment, 1 < α < 2, we also obtain convergence rates of the Lasso estimator to the true parameter. It is noted that the convergence rates of the Lasso estimators of the non-zero components of the regression parameter vector can be worse than the corresponding least squares estimators. However, when the design matrix satisfies some orthogonality conditions, the Lasso estimators of the zero components are surprisingly accurate; The Lasso recovers the zero components exactly, for large n, almost surely.

This is a preview of subscription content, access via your institution.

References

  • Athreya, K. B. and Lahiri, S. N. (2006). Measure theory and Probability theory. Springer Texts in Statistics, Springer, New York.

    Google Scholar 

  • Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and dantzig selector. Ann. Statist., 37, 1705–1732.

    MathSciNet  MATH  Article  Google Scholar 

  • Drygas, H. (1976). Weak and strong consistency of the least squares estimators in regression models. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete, 34, 119–127.

    MathSciNet  Article  Google Scholar 

  • Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist., 32, 407–499. With discussion, and a rejoinder by the authors.

    MathSciNet  MATH  Article  Google Scholar 

  • Fu, W. J. (1998). Penalized regressions: the bridge versus the lasso. J. Comput. Graph. Statist., 7, 397–416.

    MathSciNet  Article  Google Scholar 

  • Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist., 36, 587–613.

    MathSciNet  MATH  Article  Google Scholar 

  • Knight, K. and Fu, W. (2000). Asymptotics for lasso-type estimators. Ann. Statist., 28, 1356–1378.

    MathSciNet  MATH  Article  Google Scholar 

  • Lai, T. L., Robbins, H. and Wei, C. Z. (1978). Strong consistency of least squares estimates in multiple regression. Proc. Nat. Acad. Sci. U.S.A., 75, 3034–3036.

    MathSciNet  MATH  Article  Google Scholar 

  • Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electron. J. Stat., 2, 90–102.

    MathSciNet  MATH  Article  Google Scholar 

  • Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. Ann. Statist., 34, 1436–1462.

    MathSciNet  MATH  Article  Google Scholar 

  • Meinshausen, N. and Yu, B. (2009). Lasso-type recovery of sparse representations for high-dimensional data. Ann. Statist., 37, 246–270.

    MathSciNet  MATH  Article  Google Scholar 

  • Osborne, M. R., Presnell, B. and Turlach, B. A. (2000). On the LASSO and its dual. J. Comput. Graph. Statist., 9, 319–337.

    MathSciNet  Article  Google Scholar 

  • Rao, C. R. (1973). Linear statistical inference and its applications. John Wiley & Sons, New York London-Sydney, second edition. Wiley Series in Probability and Mathematical Statistics.

    MATH  Book  Google Scholar 

  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc., Ser. B, 58, 267–288.

    MathSciNet  MATH  Google Scholar 

  • Wainwright, M. J. (2006). Sharp thresholds for high-dimensional and noisy recovery of sparsity. Technical report, Dept. of Statistics, UC Berkeley. Preprint available at http://arxiv.org/abs/math/0605740.

  • Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the lasso selection in high-dimensional linear regression. Ann. Statist., 36, 1567–1594.

    MathSciNet  MATH  Article  Google Scholar 

  • Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res., 7, 2541–2563.

    MathSciNet  MATH  Google Scholar 

  • Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc., 101, 1418–1429.

    MathSciNet  MATH  Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to A. Chatterjee.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Chatterjee, A., Lahiri, S.N. Strong consistency of Lasso estimators. Sankhya A 73, 55–78 (2011). https://doi.org/10.1007/s13171-011-0006-0

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13171-011-0006-0

AMS (2000) subject classification

  • Primary 62J07
  • secondary 60F15, 62E20

Keywords and phrases

  • Penalized regression
  • strong law
  • convergence rates