Advertisement

A new piecewise quadratic approximation approach for L0 norm minimization problem

  • Qian Li
  • Yanqin Bai
  • Changjun Yu
  • Ya-xiang Yuan
Articles
  • 13 Downloads

Abstract

In this paper, we consider the problem of finding sparse solutions for underdetermined systems of linear equations, which can be formulated as a class of L0 norm minimization problem. By using the least absolute residual approximation, we propose a new piecewise quadratic function to approximate the L0 norm. Then, we develop a piecewise quadratic approximation (PQA) model where the objective function is given by the summation of a smooth non-convex component and a non-smooth convex component. To solve the (PQA) model, we present an algorithm based on the idea of the iterative thresholding algorithm and derive the convergence and the convergence rate. Finally, we carry out a series of numerical experiments to demonstrate the performance of the proposed algorithm for (PQA). We also conduct a phase diagram analysis to further show the superiority of (PQA) over L1 and L1/2 regularizations.

Keywords

sparse optimization non-convex approximation iterative thresholding algorithm 

MSC(2010)

90C30 90C59 90C90 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 11771275).

References

  1. 1.
    Beck A, Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci, 2009, 2: 183–202MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Becker S, Bobin J, Cands E J. NESTA: A fast and accurate first-order method for sparse recovery. SIAM J Imaging Sci, 2011, 4: 1–39MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bioucas-Dias J M, Figueiredo M A. A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans Image Process, 2007, 16: 2992–3004MathSciNetCrossRefGoogle Scholar
  4. 4.
    Blumensath T, Davies M E. Iterative thresholding for sparse approximations. J Fourier Anal Appl, 2008, 14: 629–654MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Blumensath T, Davies M E. Iterative hard thresholding for compressed sensing. Appl Comput Harmon Anal, 2008, 27: 265–274MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Bredies K, Lorenz D A, Reiterer S. Minimization of non-smooth, non-convex functionals by iterative thresholding. J Optim Theory Appl, 2014, 165: 78–112MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Bruckstein A M, Donoho D L, Elad M. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev, 2009, 51: 34–81MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Candes E J, Recht B. Exact matrix completion via convex optimization. Found Comput Math, 2008, 9: 717–772MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Candes E J, Romberg J, Tao T. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans Inform Theory, 2006, 52: 489–509MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Candes E J, Wakin M B, Boyd S P. Enhancing sparsity by reweighted l 1 minimization. J Fourier Anal Appl, 2008, 14: 877–905MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Cao W F, Sun J, Xu Z B. Fast image deconvolution using closed-form thresholding formulas of L q (q = 1/2, 2/3) regularization. J Vis Comm Image Represent, 2013, 24: 1529–1542CrossRefGoogle Scholar
  12. 12.
    Chen S S, Donoho D L, Saunders M A. Atomic decomposition by basis pursuit. SIAM J Sci Comput, 1998, 20: 33–61MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Daubechies I, Defrise M, Christine D M. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm Pure Appl Math, 2004, 57: 1413–1457MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Daubechies I, Devore R, Fornasier M, et al. Iteratively reweighted least squares minimization for sparse recovery. Comm Pure Appl Math, 2010, 63: 1–38MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Devore R, Jawerth B. Image compression through wavelet transform coding. IEEE Trans Inform Theory, 1992, 38: 719–746MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Donoho D L. De-noising by soft-thresholding. IEEE Trans Inform Theory, 1995, 41: 613–627MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Donoho D L. Compressed sensing. IEEE Trans Inform Theory, 2006, 52: 1289–1306MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Efron B, Hastie T. Least angle regression. Ann Statist, 2004, 32: 407–499MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Fan J, Li R. Variable selection via nonconcave penalized likelihood and its oracle properties. J Amer Statist Assoc, 2001, 96: 1348–1360MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Gasso G, Rakotomamonjy A, Canu S. Recovering sparse signals with a certain family of non-convex penalties and DC programming. IEEE Trans Signal Process, 2009, 57: 4686–4698MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Geman D, Reynolds G. Constrained restoration and the recovery of discontinuities. IEEE Trans Pattern Anal Mach Intell, 1992, 14: 367–383CrossRefGoogle Scholar
  22. 22.
    Geman D, Yang C. Nonlinear image recovery with half-quadratic regularization. IEEE Trans Image Process, 1995, 4: 932–946CrossRefGoogle Scholar
  23. 23.
    Gong P H, Zhang C S, Lu Z S, et al. A general iteartive shrinkage and thresholding algorithm for non-convex regularized optimization problems. Proc Int Conf Mach Learn, 2013, 28: 37–45Google Scholar
  24. 24.
    Gorodnitsky I F, Rao B D. Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Trans Signal Process, 1997, 45: 600–616CrossRefGoogle Scholar
  25. 25.
    Hale E T, Yin W, Zhang Y. A fixed-point continuation method for L 1-minimization: Methodology and convergence. SIAM J Optim, 2008, 19: 1107–1130MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Lai M J, Xu Y Y, YinWT. Improved iteratively reweighted least squares for unconstrained smoothed L q minimization. SIAM J Numer Anal, 2013, 51: 927–957MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Lai M J, Yin W T. Augmented L 1 and nuclear-norm models with a globally linearly convergent algorithm. SIAM J Imaging Sci, 2013, 6: 1059–1091MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Moreau J J. Proximite et dualite dans un espace hilbertien. Bull Soc Math France, 1965, 93: 273–299MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Natarajan B K. Sparse approximate solutions to linear systems. SIAM J Comput, 1995, 24: 227–234MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Nesterov Y. Introductory Lectures on Convex Optimization: A Basic Course. Boston: Springer, 2003zbMATHGoogle Scholar
  31. 31.
    Parikh N, Boyd S. Proximal algorithms. Found Trends Optim, 2013, 1: 123–231Google Scholar
  32. 32.
    Qian Y T, Jia S, Zhou J, et al. Hyperspectral unmixing via sparsity-constrained nonnegative matrix factorization. IEEE Trans Geosci Remote Sens, 2011, 49: 4282–4297CrossRefGoogle Scholar
  33. 33.
    Rakotomamonjy A, Flamary R, Gasso G, et al. L p-L q penalty for sparse linear and sparse multiple kernel multitask learning. IEEE Trans Neural Netw, 2011, 22: 1307–1320CrossRefGoogle Scholar
  34. 34.
    Tibshirani R. Regression shrinkage and selection via the LASSO. J R Stat Soc Ser B, 1996, 58: 267–288MathSciNetzbMATHGoogle Scholar
  35. 35.
    Wen B, Chen X J, Pong T K. Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimizaiton porblems. SIAM J Optimi, 2017, 27: 124–145CrossRefzbMATHGoogle Scholar
  36. 36.
    Wu B. High-dimensional analysis on matrix decomposition with applications to correlation matrix estimation in factor models. PhD Thesis. Singapore: National University of Singapore, 2014Google Scholar
  37. 37.
    Xu Z B. Data modeling: Visual psychology approach and L 1/2 regularization theory. In: Proceedings of the Interna-tional Congress of Mathematicians. Berlin: International Mathematical Union, 2010, 3151–3184Google Scholar
  38. 38.
    Xu Z B, Chang X Y, Xu F M, et al. L 1/2 regularization: A thresholding representation theory and a fast solver. IEEE Trans Neural Netw Learn Syst, 2012, 23: 1013–1027CrossRefGoogle Scholar
  39. 39.
    Xu Z B, Guo H L, Wang Y, et al. Representative of L 1/2 regularization among L q (0 < q < 1) regularizations: An experimental study based on phase diagram. Acta Automat Sinica, 2012, 38: 1225–1228MathSciNetGoogle Scholar
  40. 40.
    Xu Z B, Zhang H, Wang Y, et al. L 1/2 regularization. Sci China Inf Sci, 2010, 53: 1159–1169MathSciNetCrossRefGoogle Scholar
  41. 41.
    Yang A Y, Ganesh A, Ma Y. Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell, 2009, 31: 210–227CrossRefGoogle Scholar
  42. 42.
    Yang A Y, Ganesh A, Zhou Z H, et al. Fast L 1-minimization algorithms for robust face recognition. IEEE Trans Image Process, 2013, 22: 3234–3246CrossRefGoogle Scholar
  43. 43.
    Yin P H, Lou Y F, He Q, et al. Minimization of L 1–2 for compressed sensing. SIAM J Sci Comput, 2015, 37: 536–563MathSciNetCrossRefGoogle Scholar
  44. 44.
    Zeng J S, Lin S B, Wang Y, et al. L 1/2 regularization: convergence of iterative half thresholding algorithm. IEEE Trans Signal Process, 2014, 62: 2317–2329MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Zeng J S, Lin S B, Xu Z B. Sparse regularization: Convergence of iterative jumping thresholding algorithm. IEEE Trans Signal Process, 2014, 64: 5106–5118MathSciNetCrossRefGoogle Scholar
  46. 46.
    Zhang C H. Nearly unbiaised variable selection under minimax concave penalty. Ann Statist, 2010, 38: 894–942MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Zhang T. Analysis of multi-stage convex relaxation for sparse regularization. J Mach Learn Res, 2010, 11: 1081–1107MathSciNetzbMATHGoogle Scholar

Copyright information

© Science in China Press and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Qian Li
    • 1
  • Yanqin Bai
    • 1
  • Changjun Yu
    • 1
  • Ya-xiang Yuan
    • 2
  1. 1.Department of MathematicsShanghai UniversityShanghaiChina
  2. 2.Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems ScienceChinese Academy of SciencesBeijingChina

Personalised recommendations