Nonconvex Sorted \(\ell _1\) Minimization for Sparse Approximation

  • Xiao-Lin Huang
  • Lei Shi
  • Ming YanEmail author


The \(\ell _1\) norm is the tight convex relaxation for the \(\ell _0\) norm and has been successfully applied for recovering sparse signals. However, for problems with fewer samples than required for accurate \(\ell _1\) recovery, one needs to apply nonconvex penalties such as \(\ell _p\) norm. As one method for solving \(\ell _p\) minimization problems, iteratively reweighted \(\ell _1\) minimization updates the weight for each component based on the value of the same component at the previous iteration. It assigns large weights on small components in magnitude and small weights on large components in magnitude. The set of the weights is not fixed, and it makes the analysis of this method difficult. In this paper, we consider a weighted \(\ell _1\) penalty with the set of the weights fixed, and the weights are assigned based on the sort of all the components in magnitude. The smallest weight is assigned to the largest component in magnitude. This new penalty is called nonconvex sorted \(\ell _1\). Then we propose two methods for solving nonconvex sorted \(\ell _1\) minimization problems: iteratively reweighted \(\ell _1\) minimization and iterative sorted thresholding, and prove that both methods will converge to a local minimizer of the nonconvex sorted \(\ell _1\) minimization problems. We also show that both methods are generalizations of iterative support detection and iterative hard thresholding, respectively. The numerical experiments demonstrate the better performance of assigning weights by sort compared to assigning by value.


Iteratively reweighted \(\ell _1\) minimization Iterative sorted thresholding Local minimizer Nonconvex optimization Sparse approximation 

Mathematics Subject Classification

49M37 65K10 90C26 90C52 



The authors are grateful to the anonymous reviewers for their helpful comments.


  1. 1.
    Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14(5–6), 629–654 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Bogdan, M., van den Berg, E., Su, W., Candes, E.: Statistical estimation and testing via the sorted L1 norm. arXiv:1310.1969 [e-prints] (2013)
  4. 4.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Bühlmann, P., Van De Geer, S.: Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Berlin (2011)CrossRefGoogle Scholar
  6. 6.
    Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)CrossRefzbMATHGoogle Scholar
  7. 7.
    Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. Signal Process. Lett. IEEE 14(10), 707–710 (2007)CrossRefGoogle Scholar
  9. 9.
    Chartrand, R.: Generalized shrinkage and penalty functions. In: IEEE Global Conference on Signal and Information Processing (2013)Google Scholar
  10. 10.
    Chartrand, R.: Shrinkage mappings and their induced penalty functions. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Florence, 1026–1029 (2014)Google Scholar
  11. 11.
    Chartrand, R., Yin, W.: Iteratively reweighted algorithms for compressive sensing. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Washington D.C, pp 3869–3872 (2008)Google Scholar
  12. 12.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)CrossRefMathSciNetGoogle Scholar
  13. 13.
    Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(l_2-l_p\) minimization. Math. Program. 143(1–2), 371–383 (2014)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Chen, X., Zhou, W.: Convergence of the reweighted \(\ell _1\) minimization algorithm for \(\ell _2\)-\(\ell _p\) minimization. Comput. Optim. Appl. 1–2(59), 47–61 (2013)Google Scholar
  15. 15.
    Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best \(k\)-term approximation. J. Am. Math. Soc. 22(1), 211–231 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55(5), 2230–2249 (2009)CrossRefMathSciNetGoogle Scholar
  17. 17.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)CrossRefzbMATHGoogle Scholar
  18. 18.
    Daubechies, I., DeVore, R., Fornasier, M., Güntürk, C.S.: Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1–38 (2010)CrossRefzbMATHGoogle Scholar
  19. 19.
    Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Foucart, S.: Hard thresholding pursuit: an algorithm for compressive sensing. SIAM J. Numer. Anal. 49(6), 2543–2563 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Foucart, S., Lai, M.J.: Sparsest solutions of underdetermined linear systems via \(\ell _q\)-minimization for \(0 < q \leqslant 1\). Appl. Comput. Harmon. Anal. 26(3), 395–407 (2009)Google Scholar
  22. 22.
    Gasso, G., Rakotomamonjy, A., Canu, S.: Recovering sparse signals with a certain family of nonconvex penalties and DC programming. IEEE Trans. Signal Process. 57(12), 4686–4698 (2009)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(l_p\) minimization. Math. Program. 129(2), 285–299 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)CrossRefGoogle Scholar
  25. 25.
    Huang, X., Liu, Y., Shi, L., Van Huffel, S., Suykens, J.A.K.: Two-level \(\ell _1\) minimization for compressed sensing. Signal Process. 108, 459–475 (2015)CrossRefGoogle Scholar
  26. 26.
    Lai, M.J., Xu, Y., Yin, W.: Improved iteratively reweighted least squares for unconstrained smoothed \(\ell _q\) minimization. SIAM J. Numer. Anal. 51(2), 927–957 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Moody, G.B., Mark, R.G., Goldberger, A.L.: Physionet: a web-based resource for the study of physiologic signals. IEEE Eng. Med. Biol. Mag. 20(3), 70–75 (2001)CrossRefGoogle Scholar
  28. 28.
    Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)CrossRefzbMATHMathSciNetGoogle Scholar
  29. 29.
    Needell, D., Tropp, J.A.: Cosamp: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26(3), 301–321 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  30. 30.
    Saab, R., Yılmaz, Ö.: Sparse recovery by non-convex optimization-instance optimality. Appl. Comput. Harmon. Anal. 29(1), 30–48 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Starck, J.L., Murtagh, F., Fadili, J.M.: Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity. Cambridge University Press, Cambridge (2010)CrossRefGoogle Scholar
  32. 32.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996)zbMATHMathSciNetGoogle Scholar
  33. 33.
    Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  34. 34.
    Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53(12), 4655–4666 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Wang, Y., Yin, W.: Sparse signal reconstruction via iterative support detection. SIAM J. Imaging Sci. 3(3), 462–491 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  36. 36.
    Wright, J., Ma, Y., Mairal, J., Sapiro, G., Huang, T., Yan, S.: Sparse representation for computer vision and pattern recognition. Proc. IEEE 98(6), 1031–1044 (2010)Google Scholar
  37. 37.
    Xu, Z., Chang, X., Xu, F., Zhang, H.: \(\ell _{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)CrossRefGoogle Scholar
  38. 38.
    Yang, J., Zhang, Y.: Alternating direction algorithms for \(\ell _1\)-problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  39. 39.
    Yin, P., Lou, Y., He, Q., Xin, J.: Minimizaiton of \(\ell _1-\ell _2\) for compressed sensing. CAM-report 14–01, UCLA (2014)Google Scholar
  40. 40.
    Zeng, X., Figueiredo, M.: Decreasing weighted sorted \(\ell _1\) regularization. Signal Process. Lett. IEEE 21(10), 1240–1244 (2014)CrossRefGoogle Scholar

Copyright information

© Operations Research Society of China, Periodicals Agency of Shanghai University, and Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Department of Electrical EngineeringKU LeuvenLeuvenBelgium
  2. 2.Shanghai Key Laboratory for Contemporary Applied Mathematics, School of Mathematical SciencesFudan UniversityShanghai China
  3. 3.Department of MathematicsUniversity of CaliforniaLos AngelesUSA

Personalised recommendations