Skip to main content
SpringerLink
Log in

Nonconvex Sorted \(\ell _1\) Minimization for Sparse Approximation

  • Published:
Journal of the Operations Research Society of China Aims and scope Submit manuscript

Abstract

The \(\ell _1\) norm is the tight convex relaxation for the \(\ell _0\) norm and has been successfully applied for recovering sparse signals. However, for problems with fewer samples than required for accurate \(\ell _1\) recovery, one needs to apply nonconvex penalties such as \(\ell _p\) norm. As one method for solving \(\ell _p\) minimization problems, iteratively reweighted \(\ell _1\) minimization updates the weight for each component based on the value of the same component at the previous iteration. It assigns large weights on small components in magnitude and small weights on large components in magnitude. The set of the weights is not fixed, and it makes the analysis of this method difficult. In this paper, we consider a weighted \(\ell _1\) penalty with the set of the weights fixed, and the weights are assigned based on the sort of all the components in magnitude. The smallest weight is assigned to the largest component in magnitude. This new penalty is called nonconvex sorted \(\ell _1\). Then we propose two methods for solving nonconvex sorted \(\ell _1\) minimization problems: iteratively reweighted \(\ell _1\) minimization and iterative sorted thresholding, and prove that both methods will converge to a local minimizer of the nonconvex sorted \(\ell _1\) minimization problems. We also show that both methods are generalizations of iterative support detection and iterative hard thresholding, respectively. The numerical experiments demonstrate the better performance of assigning weights by sort compared to assigning by value.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14(5–6), 629–654 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  2. Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  3. Bogdan, M., van den Berg, E., Su, W., Candes, E.: Statistical estimation and testing via the sorted L1 norm. arXiv:1310.1969 [e-prints] (2013)

  4. Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  5. Bühlmann, P., Van De Geer, S.: Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Berlin (2011)

    Book  Google Scholar 

  6. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MATH  Google Scholar 

  7. Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  8. Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. Signal Process. Lett. IEEE 14(10), 707–710 (2007)

    Article  Google Scholar 

  9. Chartrand, R.: Generalized shrinkage and penalty functions. In: IEEE Global Conference on Signal and Information Processing (2013)

  10. Chartrand, R.: Shrinkage mappings and their induced penalty functions. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Florence, 1026–1029 (2014)

  11. Chartrand, R., Yin, W.: Iteratively reweighted algorithms for compressive sensing. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Washington D.C, pp 3869–3872 (2008)

  12. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    Article  MathSciNet  Google Scholar 

  13. Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(l_2-l_p\) minimization. Math. Program. 143(1–2), 371–383 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  14. Chen, X., Zhou, W.: Convergence of the reweighted \(\ell _1\) minimization algorithm for \(\ell _2\)-\(\ell _p\) minimization. Comput. Optim. Appl. 1–2(59), 47–61 (2013)

  15. Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best \(k\)-term approximation. J. Am. Math. Soc. 22(1), 211–231 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  16. Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55(5), 2230–2249 (2009)

    Article  MathSciNet  Google Scholar 

  17. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)

    Article  MATH  Google Scholar 

  18. Daubechies, I., DeVore, R., Fornasier, M., Güntürk, C.S.: Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1–38 (2010)

    Article  MATH  Google Scholar 

  19. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  20. Foucart, S.: Hard thresholding pursuit: an algorithm for compressive sensing. SIAM J. Numer. Anal. 49(6), 2543–2563 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  21. Foucart, S., Lai, M.J.: Sparsest solutions of underdetermined linear systems via \(\ell _q\)-minimization for \(0 < q \leqslant 1\). Appl. Comput. Harmon. Anal. 26(3), 395–407 (2009)

  22. Gasso, G., Rakotomamonjy, A., Canu, S.: Recovering sparse signals with a certain family of nonconvex penalties and DC programming. IEEE Trans. Signal Process. 57(12), 4686–4698 (2009)

    Article  MathSciNet  Google Scholar 

  23. Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(l_p\) minimization. Math. Program. 129(2), 285–299 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  24. Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)

    Article  Google Scholar 

  25. Huang, X., Liu, Y., Shi, L., Van Huffel, S., Suykens, J.A.K.: Two-level \(\ell _1\) minimization for compressed sensing. Signal Process. 108, 459–475 (2015)

    Article  Google Scholar 

  26. Lai, M.J., Xu, Y., Yin, W.: Improved iteratively reweighted least squares for unconstrained smoothed \(\ell _q\) minimization. SIAM J. Numer. Anal. 51(2), 927–957 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  27. Moody, G.B., Mark, R.G., Goldberger, A.L.: Physionet: a web-based resource for the study of physiologic signals. IEEE Eng. Med. Biol. Mag. 20(3), 70–75 (2001)

    Article  Google Scholar 

  28. Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  29. Needell, D., Tropp, J.A.: Cosamp: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26(3), 301–321 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  30. Saab, R., Yılmaz, Ö.: Sparse recovery by non-convex optimization-instance optimality. Appl. Comput. Harmon. Anal. 29(1), 30–48 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  31. Starck, J.L., Murtagh, F., Fadili, J.M.: Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity. Cambridge University Press, Cambridge (2010)

    Book  Google Scholar 

  32. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996)

    MATH  MathSciNet  Google Scholar 

  33. Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  34. Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53(12), 4655–4666 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  35. Wang, Y., Yin, W.: Sparse signal reconstruction via iterative support detection. SIAM J. Imaging Sci. 3(3), 462–491 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  36. Wright, J., Ma, Y., Mairal, J., Sapiro, G., Huang, T., Yan, S.: Sparse representation for computer vision and pattern recognition. Proc. IEEE 98(6), 1031–1044 (2010)

  37. Xu, Z., Chang, X., Xu, F., Zhang, H.: \(\ell _{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)

    Article  Google Scholar 

  38. Yang, J., Zhang, Y.: Alternating direction algorithms for \(\ell _1\)-problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  39. Yin, P., Lou, Y., He, Q., Xin, J.: Minimizaiton of \(\ell _1-\ell _2\) for compressed sensing. CAM-report 14–01, UCLA (2014)

  40. Zeng, X., Figueiredo, M.: Decreasing weighted sorted \(\ell _1\) regularization. Signal Process. Lett. IEEE 21(10), 1240–1244 (2014)

    Article  Google Scholar 

Download references

Acknowledgments

The authors are grateful to the anonymous reviewers for their helpful comments.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, KU Leuven, 3001, Leuven, Belgium

    Xiao-Lin Huang

  2. Shanghai Key Laboratory for Contemporary Applied Mathematics, School of Mathematical Sciences, Fudan University, Shanghai, 200433, China

    Lei Shi

  3. Department of Mathematics, University of California, Los Angeles, CA, 90095, USA

    Ming Yan

Authors
  1. Xiao-Lin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming Yan.

Additional information

This work is partially supported by European Research Council, the National Natural Science Foundation of China (No. 11201079), the Fundamental Research Funds for the Central Universities of China (Nos. 20520133238 and 20520131169), and the National Natural Science Foundation of United States (Nos. DMS-0748839 and DMS-1317602).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, XL., Shi, L. & Yan, M. Nonconvex Sorted \(\ell _1\) Minimization for Sparse Approximation. J. Oper. Res. Soc. China 3, 207–229 (2015). https://doi.org/10.1007/s40305-014-0069-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40305-014-0069-4

Keywords

Mathematics Subject Classification

Search

Navigation