Skip to main content
Log in

Complexity of unconstrained \(L_2-L_p\) minimization

  • Short Communication
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

We consider the unconstrained \(L_q\)-\(L_p\) minimization: find a minimizer of \(\Vert Ax-b\Vert ^q_q+\lambda \Vert x\Vert ^p_p\) for given \(A \in R^{m\times n}\), \(b\in R^m\) and parameters \(\lambda >0\), \(p\in [0, 1)\) and \(q\ge 1\). This problem has been studied extensively in many areas. Especially, for the case when \(q=2\), this problem is known as the \(L_2-L_p\) minimization problem and has found its applications in variable selection problems and sparse least squares fitting for high dimensional data. Theoretical results show that the minimizers of the \(L_q\)-\(L_p\) problem have various attractive features due to the concavity and non-Lipschitzian property of the regularization function \(\Vert \cdot \Vert ^p_p\). In this paper, we show that the \(L_q\)-\(L_p\) minimization problem is strongly NP-hard for any \(p\in [0,1)\) and \(q\ge 1\), including its smoothed version. On the other hand, we show that, by choosing parameters \((p,\lambda )\) carefully, a minimizer, global or local, will have certain desired sparsity. We believe that these results provide new theoretical insights to the studies and applications of the concave regularized optimization problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14, 707–710 (2007)

    Article  Google Scholar 

  2. Chartrand, R., Staneva, V.: Restricted isometry properties and nonconvex compressive sensing. Inverse Probl. 24, 1–14 (2008)

    Article  MathSciNet  Google Scholar 

  3. Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \(\ell _2-\ell _p\) minimization. SIAM J. Sci. Comput. 32, 2832–2852 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  4. Chen, X., Zhou, W.: Convergence of reweighted \(l_1\) minimization algorithms and unique solution of truncated \(l_p\) minimization. Department of Applied Mathematics, The Hong Kong Polytechnic University, Preprint (2010)

  5. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Soc. 96, 1348–1360 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  6. Foucart, S., Lai, M.J.: Sparsest solutions of under-determined linear systems via \(l_q\) minimization for \(0<q\le 1\). Appl. Comput. Harmon. Anal. 26, 395–407 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  7. Frank, I.E., Freidman, J.H.: A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109–148 (1993)

    Article  MATH  Google Scholar 

  8. Garey, M.R., Johnson, D.S.: “Strong” NP-Completeness results: motivation, examples, and implications. J. Assoc. Comput. Mach. 25, 499–508 (1978)

    Article  MATH  MathSciNet  Google Scholar 

  9. Garey, M.R., Johnson, D.S.: Computers and Intractability; A Guide to the Theory of NP-Completeness. W. H. Freeman, New York (1979)

    MATH  Google Scholar 

  10. Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(L_p\) minimization. Math. Program. 129, 285–299 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  11. Huang, J., Horowitz, J.L., Ma, S.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36, 587–613 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  12. Knight, K., Fu, W.J.: Asymptotics for lasso-type estimators. Ann. Stat. 28, 1356–1378 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  13. Lai, M., Wang, Y.: An unconstrained \(l_q\) minimization with \(0 < q< 1\) for sparse solution of under-determined linear systems. SIAM J. Optim. 21, 82–101 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  14. Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  15. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2006)

    MATH  Google Scholar 

  16. Tibshirani, R.: Regression shrinkage and selection via the Lasso. J R. Stat. Soc. B 58, 267–288 (1996)

    MATH  MathSciNet  Google Scholar 

  17. Vazirani, V.: Approximation Algorithms. Springer, Berlin (2003)

    Book  Google Scholar 

Download references

Acknowledgments

The authors would like to thank anonymous referees for their helpful comments. The authors acknowledge the support of the Hong Kong Polytechnic University and Hong Kong Research Grant Council. Research by the second author is supported by National Natural Science Foundation of China under Grants 71001062.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xiaojun Chen.

Appendix

Appendix

1.1 Proof of Lemma 4

First it is easy to see that when \(p=0\), \(g(z)\) has a unique minimizer at \( z = 1\), and the optimal value is \(\frac{1}{2}\). Now we consider the case when \(p\ne 0\). Note that \(g(z)>g(0)=1\) for all \(z<0\), and \(g(z)>g(1)=\frac{1}{2}\) for all \(z>1\). Therefore the minimum point must lie within \([0,1]\).

To optimize \(g(z)\) on \([0,1]\), we check its first derivative

$$\begin{aligned} g^{\prime }(z)=-q(1-z)^{q-1}+\frac{pz^{p-1}}{2}. \end{aligned}$$
(20)

We have \(g^{\prime }(0^+)=+\infty \) and \(g^{\prime }(1)=\frac{p}{2}>0\). Therefore, if function \(g(z)\) has at most two stationary points in (0,1), the first one must be a local maximum and the second one must be the unique global minimum and the minimum value \(c(p,q)\) must be less than \(\frac{1}{2}\).

Now we check the possible stationary points of \(g(z)\). Consider solving \(g^{\prime }(z)=-q(1-z)^{q-1}+\frac{p z^{p-1}}{2}=0\). We get \(z^{1-p}(1-z)^{q-1}=\frac{p}{2q}\).

Define \(h(z)=z^{1-p}(1-z)^{q-1}\). We have

$$\begin{aligned} h^{\prime }(z)=h(z)\left(\frac{1-p}{z}-\frac{q-1}{1-z}\right). \end{aligned}$$

Note that \(\frac{1-p}{z}-\frac{q-1}{1-z}\) is decreasing in \(z\) and must have a root on \((0,1)\). Therefore, there exists a point \(\bar{z}\in (0,1)\) such that \(h^{\prime }(z)>0\) for \(z<\bar{z}\) and \(h^{\prime }(z)<0\) for \(z>\bar{z}\). This implies that \(h(z)=\frac{p}{2q}\) can have at most two solutions in \((0,1)\), i.e., \(g(z)\) can have at most two stationary points. By the previous discussions, the lemma holds. \(\square \)

1.2 Proof of Theorem 6

We again consider the same \(3\)-partition problem, we claim that it can be reduced to a minimization problem in form (6). Again, it suffices to only consider the case when \(\lambda = \frac{1}{2}\) (Here we consider the hardness result for any given \(\epsilon > 0\). Note that after the scaling, \(\epsilon \) may have changed). Consider:

$$\begin{aligned}&\text{ Minimize}_x \quad {\displaystyle P_{\epsilon }(x)=\sum _{j=1}^m\left|\sum _{i=1}^n a_{i}x_{ij}-B\right|^q + \sum _{i=1}^n \left|\sum _{j=1}^m x_{ij}-1\right|^q }\nonumber \\&\qquad \qquad \qquad \quad \quad + \frac{1}{2} \sum _{i=1}^n\sum _{j=1}^m (|x_{ij}|+\epsilon )^p. \end{aligned}$$
(21)

We have

$$\begin{aligned} \text{ Minimize}_x P_{\epsilon }(x)&\ge \text{ Minimize}_x\ \sum _{i=1}^n \left|\sum _{j=1}^m x_{ij}-1\right|^q + \frac{1}{2} \sum _{i=1}^n\sum _{j=1}^m (|x_{ij}|+\epsilon )^p\\&= \sum _{i=1}^n \text{ Minimize}_x \ \left|\sum _{j=1}^m x_{ij}-1\right|^q + \frac{1}{2} \sum _{j=1}^m (|x_{ij}|+\epsilon )^p\\&= n\cdot \text{ Minimize}_{z}\ |1-z|^q+\frac{1}{2} (|z|+\epsilon )^p+ \frac{(m-1)}{2} {\epsilon }^p. \end{aligned}$$

The last equality comes from the submodularity of the function \((x+\epsilon )^p\) and the fact that one can always choose only one of \(x_{ij}\) to be nonzero in each set such that the equality holds. Consider function \(g_{\epsilon }(z)=|1-z|^q+\frac{1}{2} (|z|+\epsilon )^p\). Similar to Lemma 4, one can prove that \(g_{\epsilon }(z)\) has a unique minimizer in \([0,1]\). Denote this minimum value by \(c(p, q, \epsilon )\), we know that \(P_{\epsilon }(x)\ge n c(p, q, \epsilon )\). Then we can argue that the 3-partition problem has a solution if and only if \(P_{\epsilon }(x)= n c(p, q, \epsilon )\). Therefore Theorem 6 holds.\(\square \)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chen, X., Ge, D., Wang, Z. et al. Complexity of unconstrained \(L_2-L_p\) minimization. Math. Program. 143, 371–383 (2014). https://doi.org/10.1007/s10107-012-0613-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-012-0613-0

Keywords

Mathematics Subject Classification (2010)

Navigation