Folded concave penalized sparse linear regression: sparsity, statistical performance, and algorithmic theory for local solutions
- 975 Downloads
This paper concerns the folded concave penalized sparse linear regression (FCPSLR), a class of popular sparse recovery methods. Although FCPSLR yields desirable recovery performance when solved globally, computing a global solution is NP-complete. Despite some existing statistical performance analyses on local minimizers or on specific FCPSLR-based learning algorithms, it still remains open questions whether local solutions that are known to admit fully polynomial-time approximation schemes (FPTAS) may already be sufficient to ensure the statistical performance, and whether that statistical performance can be non-contingent on the specific designs of computing procedures. To address the questions, this paper presents the following threefold results: (1) Any local solution (stationary point) is a sparse estimator, under some conditions on the parameters of the folded concave penalties. (2) Perhaps more importantly, any local solution satisfying a significant subspace second-order necessary condition (S\(^3\)ONC), which is weaker than the second-order KKT condition, yields a bounded error in approximating the true parameter with high probability. In addition, if the minimal signal strength is sufficient, the S\(^3\)ONC solution likely recovers the oracle solution. This result also explicates that the goal of improving the statistical performance is consistent with the optimization criteria of minimizing the suboptimality gap in solving the non-convex programming formulation of FCPSLR. (3) We apply (2) to the special case of FCPSLR with minimax concave penalty and show that under the restricted eigenvalue condition, any S\(^3\)ONC solution with a better objective value than the Lasso solution entails the strong oracle property. In addition, such a solution generates a model error (ME) comparable to the optimal but exponential-time sparse estimator given a sufficient sample size, while the worst-case ME is comparable to the Lasso in general. Furthermore, to guarantee the S\(^3\)ONC admits FPTAS.
KeywordsSparse recovery Non-convex programming NP-completeness Folded concave penalty Lasso
Mathematics Subject Classification90C26 90C90 62J05 62J07 68Q25
The authors thank the AE and referees for their valuable comments, which significantly improve the paper. This work was supported by Penn State Grace Woodward Collaborative Research Grant, NSF grants CMMI 1300638 and DMS 1512422, NIH grants P50 DA036107 and P50 DA039838, Marcus PSU-Technion Partnership grant, Air Force Office of Scientific Research grant FA9550-12-1-0396, and Mid-Atlantic University Transportation Centers grant. This work was also partially supported by NNSFC grants 11690014 and 11690015. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NSF, the NIDA, the NIH, the AFOSR, the MAUTC or the NNSFC.
- 3.Bian, W., Chen, X.: Optimality conditions and complexity for non-Lipschitz constrained optimization problems. http://www.polyu.edu.hk/ama/staff/xjchen/OCT26 (2014)
- 14.Ge, D., Wang, Z., Ye, Y., Yin, H.: Strong NP-hardness result for regularized \(L_q\)-minimization problems with concave penalty functions. arxiv:1501.00622v1 (2015)
- 16.Hsu, D., Kakade, S.M., Zhang, T.: Random design analysis of ridge regression. arXiv:1106.2363v2. (2014)
- 20.Liu, H., Yao, T., Li, R., Ye, Y.: Electronic Companion to: Folded Concave Penalized Sparse Linear Regression: Sparsity, Statistical Performance, and Algorithmic Theory for Local Solutions (2017)Google Scholar
- 30.Vershynin, R.: How close is the sample covariance matrix to the actual covariance matrix. arXiv:1004.3484v2 (2010)
- 36.Zhang, Y., Wainwright, M.J., Jordan, M.I.: Lower bounds on the performance of polynomial-time algorithms for sparse linear regression. JMLR: Worksh. Conf. Proc. 35, 1–18 (2014)Google Scholar
- 38.Zhou, S.: Restricted Eigenvalue Conditions on Subgaussian Random Matrices. arXiv:0912.4045v2 (2009)