Skip to main content
SpringerLink
Log in

A smoothing method for sparse optimization over convex sets

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

In this paper, we investigate a class of heuristic schemes to solve the NP-hard problem of minimizing \(\ell _0\)-norm over a convex set. A well-known approximation is to consider the convex problem of minimizing \(\ell _1\)-norm. We are interested in finding improved results in cases where the problem in \(\ell _1\)-norm does not provide an optimal solution to the \(\ell _0\)-norm problem. We consider a relaxation technique using a family of smooth concave functions depending on a parameter. Some other relaxations have already been tried in the literature and the aim of this paper is to provide a more general context. This motivation allows deriving new theoretical results that are valid for general constraint set. We use a homotopy algorithm, starting from a solution to the problem in \(\ell _1\)-norm and ending in a solution of the problem in \(\ell _0\)-norm. The new results are existence of the solutions of the subproblem, convergence of the scheme, a monotonicity of the solutions and an exact penalization theorem independent of the data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdallah, L., Haddou, M., Migot, T.: Solving absolute value equation using complementarity and smoothing functions. J. Comput. Appl. Math. 327, 196–207 (2018)

    MathSciNet  MATH  Google Scholar 

  2. Asif, M.S., Romberg, J.: Sparse recovery of streaming signals using \(\ell _1 \)-homotopy. IEEE Trans. Signal Process. 62(16), 4209–4223 (2014)

    MathSciNet  MATH  Google Scholar 

  3. Auslender, A., Teboulle, M.: Asymptotic Cones and Functions in Optimization and Variational Inequalities. Springer, Berlin (2006)

    MATH  Google Scholar 

  4. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    MathSciNet  MATH  Google Scholar 

  5. Bergounioux, M., Haddou, M.: A new relaxation method for a discrete image restoration problem. J. Convex Anal. 17(3), 421–435 (2010)

    MathSciNet  MATH  Google Scholar 

  6. Bi, S., Liu, X., Pan, S.: Exact penalty decomposition method for zero-norm minimization based on mpec formulation. SIAM J. Sci. Comput. 36(4), A1451–A1477 (2014)

    MathSciNet  MATH  Google Scholar 

  7. Blumensath, T., Davies, M.E.: Gradient pursuits. IEEE Trans. Signal Process. 56(6), 2370–2382 (2008)

    MathSciNet  MATH  Google Scholar 

  8. Bradley, P.S., Mangasarian, O.L.: Feature selection via concave minimization and support vector machines. ICML 98, 82–90 (1998)

    Google Scholar 

  9. Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)

    MathSciNet  MATH  Google Scholar 

  10. Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    MathSciNet  MATH  Google Scholar 

  11. Candès, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)

    MathSciNet  MATH  Google Scholar 

  12. Candes, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)

    MathSciNet  MATH  Google Scholar 

  13. Candes, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)

    MathSciNet  MATH  Google Scholar 

  14. Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14(5), 877–905 (2008)

    MathSciNet  MATH  Google Scholar 

  15. Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007)

    Google Scholar 

  16. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    MathSciNet  MATH  Google Scholar 

  17. Chen, X., Zhou, W.: Convergence of the reweighted \(\ell _1\) minimization algorithm for \(\ell _2-\ell _p\) minimization. Comput. Optim. Appl. 59(1–2), 47–61 (2014)

    MathSciNet  Google Scholar 

  18. Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best \(k\)-term approximation. J. Am. Math. Soc. 22(1), 211–231 (2009)

    MathSciNet  MATH  Google Scholar 

  19. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)

    MathSciNet  MATH  Google Scholar 

  20. Di Lorenzo, D., Liuzzi, G., Rinaldi, F., Schoen, F., Sciandrone, M.: A concave optimization-based approach for sparse portfolio selection. Optim. Methods Softw. 27(6), 983–1000 (2012)

    MathSciNet  MATH  Google Scholar 

  21. Donoho, D. L.: Neighborly polytopes and sparse solutions of underdetermined linear equations. Technical report, Department of Statistics, Stanford University (2004)

  22. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    MathSciNet  MATH  Google Scholar 

  23. Donoho, D.L.: For most large underdetermined systems of linear equations the minimal \(\ell _1\)-norm solution is also the sparsest solution. Commun. Pure Appl. Math. 59(6), 797–829 (2006)

    MathSciNet  MATH  Google Scholar 

  24. Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via \(\ell _1\) minimization. Proc. Natl. Acad. Sci. 100(5), 2197–2202 (2003)

    MathSciNet  MATH  Google Scholar 

  25. Donoho, D.L., Elad, M., Temlyakov, V.N.: Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory 52(1), 6–18 (2006)

    MathSciNet  MATH  Google Scholar 

  26. Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47(7), 2845–2862 (2001)

    MathSciNet  MATH  Google Scholar 

  27. Feng, M., Mitchell, J.E., Pang, J.-S., Shen, X., Wächter, A.: Complementarity formulations of l0-norm optimization problems. Technical report, Technical report, Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY (2013)

  28. Foucart, S., Lai, M.-J.: Sparsest solutions of underdetermined linear systems via \(\ell _q\)-minimization for \(0< q \le 1\). Appl. Comput. Harmonic Anal. 26(3), 395–407 (2009)

    MathSciNet  MATH  Google Scholar 

  29. Friedman, J., Hastie, T., Tibshirani, R.: The elements of statistical learning. Springer series in statistics, vol. 1. Springer, New York (2001)

    MATH  Google Scholar 

  30. Fuchs, J.-J.: On sparse representations in arbitrary redundant bases. IEEE Trans. Inf. Theory 50(6), 1341–1344 (2004)

    MathSciNet  MATH  Google Scholar 

  31. Fung, G.M., Mangasarian, O.L.: Equivalence of minimal \(\ell _0\) and \(\ell _p\) norm solutions of linear equalities, inequalities and linear programs for sufficiently small p. J. Optim. Theory Appl. 151(1), 1–10 (2011)

    MathSciNet  MATH  Google Scholar 

  32. Ge, D., Jiang, X., Ye, Y.: A note on the complexity of l p minimization. Math. Program. 129(2), 285–299 (2011)

    MathSciNet  MATH  Google Scholar 

  33. Gribonval, R., Nielsen, M.: Sparse decomposition in unions of bases. IEEE Trans. Inf. Theory 49(12), 3320–3325 (2003)

    MATH  Google Scholar 

  34. Haddou, M., Maheux, P.: Smoothing methods for nonlinear complementarity problems. J. Optim. Theory Appl. 160(3), 711–729 (2014)

    MathSciNet  MATH  Google Scholar 

  35. Lai, M.-J., Wang, J.: An unconstrained \(\ell _q\) minimization with \(0\le q \le 1\) for sparse solution of underdetermined linear systems. SIAM J. Optim. 21(1), 82–101 (2011)

    MathSciNet  Google Scholar 

  36. Le Thi, H.A., Dinh, T.P., Le, H.M., Vo, X.T.: Dc approximation approaches for sparse optimization. Eur. J. Oper. Res. 244(1), 26–46 (2015)

    MathSciNet  MATH  Google Scholar 

  37. Liuzzi, G., Rinaldi, F.: Solving \(\backslash \ell {\_}0\)-penalized problems with simple constraints via the frank-wolfe reduced dimension method. Optim. Lett. 9(1), 57–74 (2015)

    MathSciNet  MATH  Google Scholar 

  38. Malioutov, D., Aravkin, A.: Iterative log thresholding. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7198–7202. IEEE (2014)

  39. Mallat, S.: A Wavelet Tour of Signal Processing. Academic Press, Cambridge (1999)

    MATH  Google Scholar 

  40. Mohimani, H., Babaie-Zadeh, M., Jutten, C.: A fast approach for overcomplete sparse decomposition based on smoothed norm. IEEE Trans. Signal Process. 57(1), 289–301 (2009)

    MathSciNet  MATH  Google Scholar 

  41. Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)

    MathSciNet  MATH  Google Scholar 

  42. Needell, D.: Noisy signal recovery via iterative reweighted l1-minimization. In: 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers, pp. 113–117. IEEE (2009)

  43. Needell, D., Tropp, J.A.: Cosamp: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmonic Anal. 26(3), 301–321 (2009)

    MathSciNet  MATH  Google Scholar 

  44. Rinaldi, F.: Mathematical programming methods for minimizing the zero-norm over polyhedral sets. PhD thesis, PhD thesis, Sapienza, University of Rome (2009)

  45. Rinaldi, F.: Concave programming for finding sparse solutions to problems with convex constraints. Optim. Methods Softw. 26(6), 971–992 (2011)

    MathSciNet  MATH  Google Scholar 

  46. Rinaldi, F., Schoen, F., Sciandrone, M.: Concave programming for minimizing the zero-norm over polyhedral sets. Comput. Optim. Appl. 46(3), 467–486 (2010)

    MathSciNet  MATH  Google Scholar 

  47. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    MATH  Google Scholar 

  48. Tropp, J., Wright, S.J., et al.: Computational methods for sparse solution of linear inverse problems. Proc. IEEE 98(6), 948–958 (2010)

    Google Scholar 

  49. Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)

    MathSciNet  MATH  Google Scholar 

  50. Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inf. Theory 52(3), 1030–1051 (2006)

    MathSciNet  MATH  Google Scholar 

  51. Voronin, S., Chartrand, R.: A new generalized thresholding algorithm for inverse problems with sparsity constraints. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1636–1640. IEEE (2013)

  52. Weston, J., Elisseeff, A., Schölkopf, B., Tipping, M.: Use of the zero-norm with linear models and kernel methods. J. Mach. Learn. Res. 3(Mar), 1439–1461 (2003)

    MathSciNet  MATH  Google Scholar 

  53. Wipf, D., Nagarajan, S.: Iterative reweighted \(\ell _1 \) and \(\ell _2 \) methods for finding sparse solutions. IEEE J. Sel. Top. Signal Process. 4(2), 317–329 (2010)

    Google Scholar 

  54. Xie, Z., Hu, J.: Rewighted l1-minimization for sparse solutions to underdetermined linear systems. In: 2013 6th International Congress on Image and Signal Processing (CISP), vol. 3, pp. 1660–1664. IEEE (2013)

  55. Zhao, Y.-B.: Rsp-based analysis for sparsest and least \(\ell _1 \)-norm solutions to underdetermined linear systems. IEEE Trans. Signal Process. 61(22), 5777–5788 (2013)

    MathSciNet  MATH  Google Scholar 

  56. Zhao, Y.-B., Kočvara, M.: A new computational method for the sparsest solutions to systems of linear equations. SIAM J. Optim. 25(2), 1110–1134 (2015)

    MathSciNet  MATH  Google Scholar 

  57. Zhao, Y.-B., Luo, Z.-Q.: Constructing new weighted \(\ell _1\)-algorithms for the sparsest points of polyhedral sets. Math. Oper. Res. 42(1), 57–76 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INSA Rennes, CNRS, IRMAR - UMR 6625, Univ Rennes, 35000, Rennes, France

    M. Haddou & T. Migot

Authors
  1. M. Haddou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. Migot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to T. Migot.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haddou, M., Migot, T. A smoothing method for sparse optimization over convex sets. Optim Lett 14, 1053–1069 (2020). https://doi.org/10.1007/s11590-019-01408-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-019-01408-x

Keywords

Search

Navigation