Abstract
Sparse Group LASSO (SGL) is a regularized model for high-dimensional linear regression problems with grouped covariates. SGL applies \(l_1\) and \(l_2\) penalties on the individual predictors and group predictors, respectively, to guarantee sparse effects both on the inter-group and within-group levels. In this paper, we apply the approximate message passing (AMP) algorithm to efficiently solve the SGL problem under Gaussian random designs. We further use the recently developed state evolution analysis of AMP to derive an asymptotically exact characterization of SGL solution. This allows us to conduct multiple fine-grained statistical analyses of SGL, through which we investigate the effects of the group information and \(\gamma \) (proportion of \(\ell _1\) penalty). With the lens of various performance measures, we show that SGL with small \(\gamma \) benefits significantly from the group information and can outperform other SGL (including LASSO) or regularized models which does not exploit the group information, in terms of the recovery rate of signal, false discovery rate and mean squared error.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We note that \(\mathcal {A}\) is a sufficient but not necessary condition for the state evolution to converge. The reason that we split the analysis at \(\alpha =\mathcal {A}_{\max }\) is because, for \(\alpha > \mathcal {A}_{\max }\), the SGL estimator is 0. Besides, we note that the set \(\mathcal {A}\) only affects the state evolution. Hence when \(\alpha > \mathcal {A}_{\max }\), the calibration is still valid and the mapping between \(\alpha \) and \(\lambda \) is monotone.
References
Bayati, M., Erdogdu, M.A., Montanari, A.: Estimating lasso risk and noise level. In: Advances in Neural Information Processing Systems, pp. 944–952 (2013)
Bayati, M., Lelarge, M., Montanari, A., et al.: Universality in polytope phase transitions and message passing algorithms. Ann. Appl. Probab. 25(2), 753–822 (2015)
Bayati, M., Montanari, A.: The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Trans. Inf. Theory 57(2), 764–785 (2011)
Bayati, M., Montanari, A.: The lasso risk for gaussian matrices. IEEE Trans. Inf. Theory 58(4), 1997–2017 (2011)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)
Berthier, R., Montanari, A., Nguyen, P.M.: State evolution for approximate message passing with non-separable functions. arXiv preprint arXiv:1708.03950 (2017)
Bu, Z., Klusowski, J., Rush, C., Su, W.: Algorithmic analysis and statistical estimation of slope via approximate message passing. arXiv preprint arXiv:1907.07502 (2019)
Celentano, M.: Approximate separability of symmetrically penalized least squares in high dimensions: characterization and consequences. arXiv preprint arXiv:1906.10319 (2019)
Celentano, M., Montanari, A.: Fundamental barriers to high-dimensional regression with convex penalties. arXiv preprint arXiv:1903.10603 (2019)
Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. J. Issued Courant Inst. Math. Sci. 57(11), 1413–1457 (2004)
Donoho, D.L., Maleki, A., Montanari, A.: Message-passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. 106(45), 18914–18919 (2009)
Donoho, D.L., Maleki, A., Montanari, A.: Message passing algorithms for compressed sensing: I. motivation and construction. In: 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo), pp. 1–5. IEEE (2010)
Donoho, D.L., Maleki, A., Montanari, A.: How to design message passing algorithms for compressed sensing. preprint (2011)
Donoho, D.L., Maleki, A., Montanari, A.: The noise-sensitivity phase transition in compressed sensing. IEEE Trans. Inf. Theory 57(10), 6920–6941 (2011)
Foygel, R., Drton, M.: Exact block-wise optimization in group lasso and sparse group lasso for linear regression. arXiv preprint arXiv:1010.3320 (2010)
Friedman, J., Hastie, T., Tibshirani, R.: A note on the group lasso and a sparse group lasso. arXiv preprint arXiv:1001.0736 (2010)
Guo, C., Davies, M.E.: Near optimal compressed sensing without priors: parametric sure approximate message passing. IEEE Trans. Signal Process. 63(8), 2130–2141 (2015)
Hu, H., Lu, Y.M.: Asymptotics and optimal designs of slope for sparse linear regression. In: 2019 IEEE International Symposium on Information Theory (ISIT), pp. 375–379. IEEE (2019)
Ida, Y., Fujiwara, Y., Kashima, H.: Fast sparse group lasso (2019)
Doob, J.L.: Stochastic Processes, vol. 101. Wiley, New York (1953)
Kohavi, R.: Scaling up the accuracy of naive-bayes classifiers: a decision-tree hybrid. KDD 96, 202–207 (1996)
Krzakala, F., Mézard, M., Sausset, F., Sun, Y., Zdeborová, L.: Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices. J. Stat. Mech: Theory Exp. 2012(08), P08009 (2012)
Manoel, A., Krzakala, F., Varoquaux, G., Thirion, B., Zdeborová, L.: Approximate message-passing for convex optimization with non-separable penalties. arXiv preprint arXiv:1809.06304 (2018)
Meier, L., Van De Geer, S., Bühlmann, P.: The group lasso for logistic regression. J. Roy. Stat. Soc. Ser. B (Statistical Methodology) 70(1), 53–71 (2008)
Montanari, A., Eldar, Y., Kutyniok, G.: Graphical models concepts in compressed sensing. Compressed Sensing: Theory and Applications, pp. 394–438 (2012)
Mousavi, A., Maleki, A., Baraniuk, R.G., et al.: Consistent parameter estimation for lasso and approximate message passing. Ann. Stat. 46(1), 119–148 (2018)
Parikh, N., Boyd, S., et al.: Proximal algorithms. Foundations Trends Optimization 1(3), 127–239 (2014)
Puig, A.T., Wiesel, A., Hero, A.O.: A multidimensional shrinkage-thresholding operator. In: 2009 IEEE/SP 15th Workshop on Statistical Signal Processing, pp. 113–116. IEEE (2009)
Rangan, S.: Generalized approximate message passing for estimation with random linear mixing. In: 2011 IEEE International Symposium on Information Theory Proceedings, pp. 2168–2172. IEEE (2011)
Rangan, S., Schniter, P., Fletcher, A.K.: Vector approximate message passing. IEEE Trans. Inf. Theory 65, 6664–6684 (2019)
Rangana, S., Schniterb, P., Fletcherc, A.K., Sarkar, S.: On the convergence of approximate message passing with arbitrary matrices. IEEE Trans. Inf. Theory 65(9), 5339–5351 (2019)
Shi, H.J.M., Tu, S., Xu, Y., Yin, W.: A primer on coordinate descent algorithms. arXiv preprint arXiv:1610.00040 (2016)
Simon, N., Friedman, J., Hastie, T., Tibshirani, R.: A sparse-group lasso. J. Comput. Graph. Stat. 22(2), 231–245 (2013)
Su, W., Bogdan, M., Candes, E., et al.: False discoveries occur early on the lasso path. Ann. Stat. 45(5), 2133–2150 (2017)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc.: Ser. B (Methodol.) 58(1), 267–288 (1996)
Vila, J., Schniter, P.: Expectation-maximization bernoulli-gaussian approximate message passing. In: 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pp. 799–803. IEEE (2011)
Vila, J., Schniter, P., Rangan, S., Krzakala, F., Zdeborová, L.: Adaptive damping and mean removal for the generalized approximate message passing algorithm. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2021–2025. IEEE (2015)
Vila, J.P., Schniter, P.: Expectation-maximization gaussian-mixture approximate message passing. IEEE Trans. Signal Process. 61(19), 4658–4672 (2013)
Yang, Y., Zou, H.: A fast unified algorithm for solving group-lasso penalize learning problems. Stat. Comput. 25(6), 1129–1141 (2014). https://doi.org/10.1007/s11222-014-9498-5
Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. Roy. Stat. Soc. Ser. B (Statistical Methodology) 68(1), 49–67 (2006)
Zou, H.: The adaptive lasso and its oracle properties. J. Am. Stat. Assoc. 101(476), 1418–1429 (2006)
Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. Roy. Stat. Soc. Ser. B (statistical methodology) 67(2), 301–320 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, K., Bu, Z., Xu, S. (2021). Asymptotic Statistical Analysis of Sparse Group LASSO via Approximate Message Passing. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12977. Springer, Cham. https://doi.org/10.1007/978-3-030-86523-8_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-86523-8_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86522-1
Online ISBN: 978-3-030-86523-8
eBook Packages: Computer ScienceComputer Science (R0)
