Skip to main content

Sparse Laplacian Shrinkage with the Graphical Lasso Estimator for Regression Problems

Abstract

This paper considers a high-dimensional linear regression problem where there are complex correlation structures among predictors. We propose a graph-constrained regularization procedure, named Sparse Laplacian Shrinkage with the Graphical Lasso Estimator (SLS-GLE). The procedure uses the estimated precision matrix to describe the specific information on the conditional dependence pattern among predictors, and encourages both sparsity on the regression model and the graphical model. We introduce the Laplacian quadratic penalty adopting the graph information, and give detailed discussions on the advantages of using the precision matrix to construct the Laplacian matrix. Theoretical properties and numerical comparisons are presented to show that the proposed method improves both model interpretability and accuracy of estimation. We also apply this method to a financial problem and prove that the proposed procedure is successful in assets selection.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  • Bickel PJ, Ritov Y, Tsybakov AB (2009) Simultaneous analysis of lasso and Dantzig selector. Ann Stat 37(4):1705–1732

    MathSciNet  Article  Google Scholar 

  • Chung F (1997) Spectral graph theory. CBMS regional conference series in mathematics. American Mathematical Society

  • Daye ZJ, Jeng XJ (2009) Shrinkage and model selection with correlated variables via weighted fusion. Comput Stat Data Anal 53(4):1284–1298

    MathSciNet  Article  Google Scholar 

  • Fan J, Peng H (2004) Nonconcave penalized likelihood with a diverging number of parameters. Ann Stat 32(3):928–961

    MathSciNet  Article  Google Scholar 

  • Fan JQ, Li RZ (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96(456):1348–1360

    MathSciNet  Article  Google Scholar 

  • Friedman J, Hastie T, Höfling H, Tibshirani R (2007) Pathwise coordinate optimization. Ann Appl Stat 1(2):302–332

    MathSciNet  Article  Google Scholar 

  • Friedman J, Hastie T, Tibshirani R (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3):432–441

    Article  Google Scholar 

  • Friedman J, Hastie T, Tibshirani RJ (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33(1):1–22

    Article  Google Scholar 

  • Guo JH, Hu JC, Jing BY, Zhang Z (2016) Spline-lasso in high-dimensional linear regression. J Am Stat Assoc 111(513):288–297

    MathSciNet  Article  Google Scholar 

  • Hebiri M, van de Geer S (2011) The smooth-lasso and other \(l1+l2\)-penalized methods. Electron J Stat 5:1184–1226

    MathSciNet  Article  Google Scholar 

  • Huang J, Breheny P, Lee S, Ma S, Zhang CH (2016) The mnet method for variable selection. Stat Sin 26(3):903–923

  • Huang J, Ma S, Li H, Zhang CH (2011) The sparse Laplacian shrinkage estimator for high-dimensional regression. Ann Stat 39(4):2021–2046

    MathSciNet  Article  Google Scholar 

  • Li CY, Li HZ (2008) Network-constrained regularization and variable selection for analysis of genomic data. Bioinformatics 24(9):1175–1182

    Article  Google Scholar 

  • Li CY, Li HZ (2010) Variable selection and regression analysis for graph-structured covariates with an application to genomics. Ann Appl Stat 4(3):1498–1516

    MathSciNet  MATH  Google Scholar 

  • Li Y, Mark B, Raskutti G, Willett R (2018) Graph-based regularization for regression problems with highly-correlated designs. In: 2018 IEEE global conference on signal and information processing (GlobalSIP). IEEE, pp 740–742

  • Mazumder R, Friedman JH, Hastie T (2011) Sparsenet: coordinate descent with nonconvex penalties. J Am Stat Assoc 106(495):1125–1138

    MathSciNet  Article  Google Scholar 

  • Meinshausen N, Yu B (2009) Lasso-type recovery of sparse representations for high-dimensional data. Ann Stat 37(1):246–270

    MathSciNet  Article  Google Scholar 

  • Ravikumar P, Wainwright MJ, Raskutti G, Yu B (2011) High-dimensional covariance estimation by minimizing l1-penalized log-determinant divergence. Electron J Stat 5:935–980

    MathSciNet  Article  Google Scholar 

  • She YY (2010) Sparse regression with exact clustering. Electron J Stat 4:1055–1096

    MathSciNet  Article  Google Scholar 

  • Simon N, Friedman J, Hastie T, Tibshirani R (2013) A sparse-group lasso. J Comput Graph Stat 22(2):231–245

    MathSciNet  Article  Google Scholar 

  • Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc: Ser B 58(1):267–288

    MathSciNet  MATH  Google Scholar 

  • Tibshirani R, Saunders M, Rosset S, Zhu J, Knight K (2005) Sparsity and smoothness via the fused lasso. J R Stat Soc: Ser B 67(1):91–108

    MathSciNet  Article  Google Scholar 

  • Wainwright MJ (2009) Sharp thresholds for noisy and high-dimensional recovery of sparsity using \(l_1\)-constrained quadratic programming (lasso). IEEE Trans Inf Theory 55(5):2183–2202

    Article  Google Scholar 

  • Wu L, Yang Y, Liu H (2014) Nonnegative-lasso and application in index tracking. Comput Stat Data Anal 70:116–126

    MathSciNet  Article  Google Scholar 

  • Yang Y, Yang H (2021) Adaptive and reversed penalty for analysis of high-dimensional correlated data. Appl Math Model 92:63–77

    MathSciNet  Article  Google Scholar 

  • Yuan M, Lin Y (2006) Model selection and estimation in regression with grouped variables. J R Stat Soc: Ser B 68(1):49–67

    MathSciNet  Article  Google Scholar 

  • Yuan M, Lin Y (2007) Model selection and estimation in the gaussian graphical model. Biometrika 94(1):19–35

    MathSciNet  Article  Google Scholar 

  • Zhang B, Horvath S (2005) A general framework for weighted gene co-expression network analysis. Stat Appl Genet Mol Biol 4(1):1–45

    MathSciNet  Article  Google Scholar 

  • Zhang CH (2010) Nearly unbiased variable selection under minimax concave penalty. Ann Stat 38(2):894–942

    MathSciNet  Article  Google Scholar 

  • Zhao P, Yu B (2006) On model selection consistency of lasso. J Mach Learn Res 7:2541–2563

    MathSciNet  MATH  Google Scholar 

  • Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc: Ser B 67(2):301–320

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 12001557, 11671059); and the Youth Talent Development Support Program in Central University of Finance and Economics (QYP202104).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hu Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 194 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Xia, S., Yang, Y. & Yang, H. Sparse Laplacian Shrinkage with the Graphical Lasso Estimator for Regression Problems. TEST 31, 255–277 (2022). https://doi.org/10.1007/s11749-021-00779-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11749-021-00779-7

Keywords

  • Graphical models
  • Laplacian matrix
  • Sparse regression
  • Variable selection

Mathematics Subject Classification

  • 62F12
  • 62J05
  • 62J07