Iterative reweighted methods for \(\ell _1-\ell _p\) minimization

Article
  • 186 Downloads

Abstract

In this paper, we focus on the \(\ell _1-\ell _p\) minimization problem with \(0<p<1\), which is challenging due to the \(\ell _p\) norm being non-Lipschizian. In theory, we derive computable lower bounds for nonzero entries of the generalized first-order stationary points of \(\ell _1-\ell _p\) minimization, and hence of its local minimizers. In algorithms, based on three locally Lipschitz continuous \(\epsilon \)-approximation to \(\ell _p\) norm, we design several iterative reweighted \(\ell _1\) and \(\ell _2\) methods to solve those approximation problems. Furthermore, we show that any accumulation point of the sequence generated by these methods is a generalized first-order stationary point of \(\ell _1-\ell _p\) minimization. This result, in particular, applies to the iterative reweighted \(\ell _1\) methods based on the new Lipschitz continuous \(\epsilon \)-approximation introduced by Lu (Math Program 147(1–2):277–307, 2014), provided that the approximation parameter \(\epsilon \) is below a threshold value. Numerical results are also reported to demonstrate the efficiency of the proposed methods.

Keywords

\(\ell _1-\ell _p\) minimization Generalized first-order stationary point Lower bound Iterative reweighted \(\ell _1\) Iterative reweighted \(\ell _2\) method 

Notes

Acknowledgements

The authors would like to thank the anonymous referees and the associate editor for their numerous insightful comments and suggestions, which have greatly improved the paper. In particular, we would like to thank a referee for clarifying the chain rule for computing the subdifferential of the \(\ell _1\) norm composited with a linear mapping. The work was supported in part by the National Natural Science Foundation of China (11431002, 11671029), the 111 Project of China (B16002), the Ministry of Education of Humanities and Social Science Project (17YJC910005), the Fundamental Research Funds for the Central Universities in UIBE (15YB04) and the Fundamental Research Funds for the Central Universities in BJTU (2016YJS159).

References

  1. 1.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Belloni, A., Chernozhukov, V.: L1-penalized quantile regression in high-dimensional sparse models. Ann. Stat. 39, 82–130 (2011)CrossRefMATHGoogle Scholar
  3. 3.
    Bradic, J., Fan, J., Wang, W.: Penalized composite quasi-likelihood for ultrahigh-dimensional variable selection. J. R. Stat. Soc. Ser. B 73, 325–349 (2011)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bruckstein, A., Donoho, D., Elad, A.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Candès, E., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Candès, E., Walkin, M., Boyd, S.: Enhancing sparsity by reweighted \(l_2\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Chen, X.: Smoothing methods for nonsmooth, nonconvex minimization. Math. Program. 134(1), 71–99 (2012)Google Scholar
  9. 9.
    Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(L_2{-}L_p\) minimization. Math. Program. 143(1–2), 371–383 (2014)Google Scholar
  10. 10.
    Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \(\ell _2{-}\ell _p\) minimization. SIAM J. Sci. Comput. 32, 2832–2852 (2010)Google Scholar
  11. 11.
    Chen, X., Ng, M., Zhang, C.: Nonconvex \(\ell _p\) regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21, 4709–4721 (2012)Google Scholar
  12. 12.
    Chen, X., Zhou, W.: Convergence of Reweighted \(\ell _1\) Minimization Algorithms and Unique Solution of Truncated \(\ell _p\) Minimization. The Hong Kong Polytechnic University, Department of Applied Mathematics (2010)Google Scholar
  13. 13.
    Chen, X., Zhou, W.: Convergence of reweighted \(\ell _1\) minimization algorithm for \(\ell _2{-}\ell _p\) minimization. Comput. Optim. Appl. 59, 47–61 (2014)Google Scholar
  14. 14.
    Daubechies, I., DeVore, R., Fornasier, M., Gunturk, C.: Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63, 1–38 (2010)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Eldar, Y., Kutyniok, G.: Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)CrossRefGoogle Scholar
  17. 17.
    Fan, Y., Fan, J., Barut, E.: Adaptive robust variable selection. Ann. Stat. 42(1), 324–351 (2014)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Foucart, S., Lai, M.: Sparsest solutions of underdetermined linear systems via \(\ell _p\)-minimization for \(0 < q \le 1\). Appl. Comput. Harmon. Anal. 26, 395–407 (2009)Google Scholar
  19. 19.
    Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(\ell _p\) minimization. Math. Program. 21, 1721–1739 (2011)Google Scholar
  20. 20.
    Lai, M., Wang, J.: An unconstrained \(l_q\) minimization with \(0 < q \le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim. 21, 82–101 (2010)Google Scholar
  21. 21.
    Lai, M., Xu, Y., Yin, W.: Improved iteratively reweighted least squares for unconstrained smoothed \(l_q\) minimization. SIAM J. Numer. Anal. 51(2), 927–957 (2013)Google Scholar
  22. 22.
    Lu, Z.: Iterative reweighted minimization methods for \(\ell _p\) regularized unconstrained nonlinear programming. Math. Program. 147(1–2), 277–307 (2014)Google Scholar
  23. 23.
    Mordukhovich, B., Nam, N.: An easy path to convex analysis and applications. Synth. Lect. Math. Stat. 6(2), 1–218 (2013)CrossRefGoogle Scholar
  24. 24.
    Wang, L.: L1 penalized LAD estimator for high dimensional linear regression. J. Multi. Anal. 120, 135–151 (2013)CrossRefMATHGoogle Scholar
  25. 25.
    Wang, L., Wu, Y., Li, R.: Quantile regression for analyzing heterogeneity in ultrahigh dimension. J. Am. Stat. Assoc. 107, 214–222 (2012)CrossRefMATHGoogle Scholar
  26. 26.
    Wright, S., Nowak, R., Figueiredo, M.: Sparse reconstruction by separable approximation. IEEE Trans. Image Process. 57, 2479–2493 (2009)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Xiu, X., Kong, L.: Rank-min-one and sparse tensor decomposition for surveillance video. Pac. J. Optim. 11(2), 403–418 (2015)MathSciNetMATHGoogle Scholar
  28. 28.
    Yin, W., Osher, S., Goldfarb, D., Darbonet, J.: Bregman iterative algorithms for \(\ell _1\)-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1(1), 143–168 (2008)Google Scholar
  29. 29.
    Zhang, L., Li, Y., Kong, L., Zhou, S.: A smoothing iterative method for quantile regression with nonconvex \(\ell _p\) penalty. J. Ind. Manag. Optim. 13(1), 93–112 (2017)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Xianchao Xiu
    • 1
  • Lingchen Kong
    • 1
  • Yan Li
    • 2
  • Houduo Qi
    • 3
  1. 1.Department of Applied MathematicsBeijing Jiaotong UniversityBeijingPeople’s Republic of China
  2. 2.School of Insurance and EconomicsUniversity of International Business and EconomicsBeijingPeople’s Republic of China
  3. 3.School of MathematicsUniversity of SouthamptonSouthamptonUK

Personalised recommendations