Abstract
Linear regression for high dimensional data is an inherently challenging problem with many solutions generally involving some structural assumption on the model such as lasso’s sparsity in the parameter vector. Considering the random design setting, we apply a different sparsity assumption: sparsity in the covariance or precision matrix of the predictors. Thus, we propose a new regression estimator by first applying methods for estimating a sparse covariance or precision matrix. This matrix is then incorporated into the estimator for the regression parameters. We mainly compare this methodology against the classic ridge or Tikhonov regularization method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The estimator \({X}^\mathrm {T}Y\) occurs in practice in orthogonal experimental designs when X is chosen such that \({X}^\mathrm {T}X=I_p\) assuming \(p<n\). [19].
References
Bickel, P.J., Levina, E.: Covariance regularization by thresholding. The Ann. Stat. 2577–2604 (2008)
Bien, J., Tibshirani, R.J.: Sparse estimation of a covariance matrix. Biometrika 98(4), 807–820 (2011)
Breiman, L., Freedman, D.: How many variables should be entered in a regression equation? J. Am. Stat. Assoc. 78(381), 131–136 (1983)
Bühlmann, P., Van De Geer, S.: Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Science & Business Media (2011)
Cai, T., Liu, W.: Adaptive thresholding for sparse covariance matrix estimation. J. Am. Stat. Assoc. 106(494), 672–684 (2011)
Cai, T., Liu, W., Luo, X.: A constrained l1 minimization approach to sparse precision matrix estimation. J. Am. Stat. Assoc. 106(494), 594–607 (2011)
Cortez, P., Morais, A.D.J.R.: A data mining approach to predict forest fires using meteorological data (2007)
Friedman, J., Hastie, T., Tibshirani, R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3), 432–441 (2008)
Friedman, J., Hastie, T., Tibshirani, R.: Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33(1), 1 (2010)
Hsu, D., Kakade, S.M., Zhang, T.: Random design analysis of ridge regression. In: Conference on Learning Theory, pp. 9–1 (2012)
Jankova, J., Van De Geer, S.: Confidence intervals for high-dimensional inverse covariance estimation. Electron. J. Stat. 9(1), 1205–1229 (2015)
Kashlak, A.B.: Non-asymptotic error controlled sparse high dimensional precision matrix estimation. J. Multi. Anal. 181, 104690 (2021)
Kashlak, A.B.: sparseMatEst: sparse matrix estimation and inference (2019b). https://CRAN.R-project.org/package=sparseMatEst. R package version 1.0.0
Kashlak, A.B., Kong, L.: Nonasymptotic support recovery for high dimensional sparse covariance matrices. Stat. e316 (2020)
Redmond, M., Baveja, A.: A data-driven software tool for enabling cooperative information sharing among police departments. Eur. J. Oper. Res. 141(3), 660–678 (2002)
Rothman, A.J.: Positive definite estimators of large covariance matrices. Biometrika 99(3), 733–740 (2012)
Rothman, A.J., Levina, E., Zhu, J.: Generalized thresholding of large covariance matrices. J. Am. Stat. Assoc. 104(485), 177–186 (2009)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc.: Series B (Methodological) 58(1), 267–288 (1996)
Wu, C.J., Hamada, M.S.: Experiments: Planning, Analysis, and Optimization, vol. 552. Wiley (2011)
Acknowledgements
The authors would like to thank Dr. Xu (Sunny) Wang from Wilfrid Laurier University and Dr. Yan Yuan from the University of Alberta for organizing the special session on Interdisciplinary Data Analysis of High-Dimensional Multimodal Data at AMMCS 2019 where this work was presented. We would also like to thank the comments of the anonymous reviewers who helped improve this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Fang, X., Winter, S., Kashlak, A.B. (2021). Sparse Covariance and Precision Random Design Regression. In: Kilgour, D.M., Kunze, H., Makarov, R., Melnik, R., Wang, X. (eds) Recent Developments in Mathematical, Statistical and Computational Sciences. AMMCS 2019. Springer Proceedings in Mathematics & Statistics, vol 343. Springer, Cham. https://doi.org/10.1007/978-3-030-63591-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-63591-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63590-9
Online ISBN: 978-3-030-63591-6
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)