Skip to main content
SpringerLink
Log in

Greedy Approximation in Convex Optimization

  • Published:
Constructive Approximation Aims and scope

Abstract

We study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of a few elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. The problem of approximation of a given element of a Banach space by linear combinations of elements from a given system (dictionary) is well studied in nonlinear approximation theory. At first glance, the settings of approximation and optimization problems are very different. In the approximation problem, an element is given and our task is to find a sparse approximation of it. In optimization theory, an energy function is given and we should find an approximate sparse solution to the minimization problem. It turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular the greedy approximation technique, can be adjusted for finding a sparse solution of an optimization problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bahmani, S., Raj, B., Boufounos, P.: Greedy sparsity-constrained optimization. arXiv:1203.5483v3 [stat.ML]. 6 Jan 2013

  2. Barron, A.R.: Universal approximation bounds for superposition of \(n\) sigmoidal functions. IEEE Trans. Inf. Theory 39, 930–945 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  3. Beck, A., Eldar, Y.C.: Sparsity constrained nonlinear optimization: optimality conditions and algorithms. arXiv:1203.4580v1 [cs.IT]. 20 March 2012

  4. Blumensath, T.: Compressed sensing with nonlinear observations. Preprint 1–9 (2010)

  5. Blumensath, T.: Compressed sensing with nonlinear observations and related nonlinear optimization problems. arXiv:1205.1650v1 [cs.IT]. 8 May 2012

  6. Blumensath, T., Davies, M.E.: Gradient pursuits. IEEE Trans. Signal Process. 56, 2370–2382 (2008)

    Article  MathSciNet  Google Scholar 

  7. Blumensath, T., Davies, M.E.: Stagewise weak gradient pursuits. IEEE Trans. Signal Process. 57, 4333–4346 (2009)

    Article  MathSciNet  Google Scholar 

  8. Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27, 265–274 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  9. Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples. Canadian Mathematical Society, Springer, Berlin (2006)

    Book  Google Scholar 

  10. Borwein, J., Guirao, A.J., Hajek, P., Vanderwerff, J.: Uniformly convex functions an Banach spaces. Proc. Am. Math. Soc. 137(3), 1081–1091 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  11. Chandrasekaran, V., Recht, B., Parrilo, P.A., Willsky, A.S.: The convex geometry of linear inverse problems. In: Proceedings of the 48th Annual Allerton Conference on Communication, Control and Computing, pp. 699–703 (2010)

  12. Clarkson, K.L.: Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Trans. Algorithms 6, 63 (2010)

    Article  MathSciNet  Google Scholar 

  13. Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55, 2230–2249 (2009)

    Article  MathSciNet  Google Scholar 

  14. Demyanov, V.F., Rubinov, A.M.: Approximate Methods in Optimization Problems. Elsevier, Amsterdam (1970)

    Google Scholar 

  15. DeVore, R.A.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998)

    Article  MathSciNet  Google Scholar 

  16. DeVore, R.A., Temlyakov, V.N.: Convex Optimization on Banach Spaces. arXiv:1401.0334v1 [stat.ML]. 1 Jan 2014

  17. Dudik, M., Harchaoui, Z., Malick, J.: Lifted coordinate descent for learning with trace-norm regularization. In: AISTATS (2012)

  18. Dunn, J.C., Harshbarger, S.: Conditional gradient algorithms with open loop step size rules. J. Math. Anal. Appl. 62, 432–444 (1978)

    Article  MATH  MathSciNet  Google Scholar 

  19. Elad, M., Matalon, B., Zibulevsky, M.: Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization. Appl. Comput. Harmonic Anal. 23, 346–367 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  20. Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE Sel. Top. Signal Process. 1, 586–597 (2007)

    Article  Google Scholar 

  21. Frank, M., Wolfe, P.: An algorithm for quadratic programming. Nav. Res. Logist. Q. 3, 95–110 (1956)

    Article  MathSciNet  Google Scholar 

  22. Jaggi, M.: Sparse Convex Optimization Methods for Machine Learning, PhD thesis, ETH Zürich, (2011)

  23. Jaggi, M.: Revisiting Frank-Wolfe: projection-free sparse convex optimization. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia (2013)

  24. Jaggi, M., Sulovský, M.: A Simple Algorithm for Nuclear Norm Regularized Problems. ICML (2010)

  25. Jones, L.K.: A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20, 608–613 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  26. Karmanov, V.G.: Mathematical Programming. Mir Publishers, Moscow (1989)

    MATH  Google Scholar 

  27. Kashin, B.S.: Widths of certain finite-dimensional sets and classes of smooth functions. Izv. Acad. Nauk SSSR Ser. Mat 41, 334–351 (1977)

    MATH  Google Scholar 

  28. Livshitz, E., Temlyakov, V.: Sparse Approximation and Recovery by Greedy Algorithms. arXiv:1303.3595v1 [math.NA]. 14 March 2013

  29. Needell, D., Tropp, J.A.: CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmonic Anal. 26, 301–321 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  30. Needell, D., Vershynin, R.: Uniform uncertainty principle and signal recovery via orthogonal matching pursuit. Found. Comp. Math. 9, 317–334 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  31. Nesterov, Yu.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Boston (2004)

    Book  Google Scholar 

  32. Nguyen, H., Petrova, G.: Greedy Strategies for Convex Optimization. arXiv:1401.1754v1 [math.NA]. 8 Jan 2014

  33. Pshenichnyi, B.N., Danilin, YuM: Numerical Methods in Extremal Problems [in Russian]. Nauka, Moscow (1975)

    Google Scholar 

  34. Shalev-Shwartz, S., Srebro, N., Zhang, T.: Trading accuracy for sparsity in optimization problems with sparsity constrains. SIAM J. Optim. 20(6), 2807–2832 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  35. Temlyakov, V.N.: Weak greedy algorithms. Adv. Comput. Math. 12, 213–227 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  36. Temlyakov, V.N.: Greedy algorithms in Banach spaces. Adv. Comput. Math. 14, 277–292 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  37. Temlyakov, V.N.: Greedy-type approximation in Banach spaces and applications. Constr. Approx. 21, 257–292 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  38. Temlyakov, V.N.: Relaxation in greedy approximation. Constr. Approx. 28, 1–25 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  39. Temlyakov, V.N.: Greedy approximation. Acta Numer. 17, 235–409 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  40. Temlyakov, V.N.: Greedy Approximation. Cambridge University Press, Cambridge (2011)

    Book  MATH  Google Scholar 

  41. Temlyakov, V.N.: Greedy Approximation in Convex Optimization. arXiv:1206.0392v1 [stat.ML]. 2 June 2012 (see also IMI Preprint, 2012:03, 1–25).

  42. Temlyakov, V.N.: Greedy expansions in convex optimization. In: Proceedings of the Steklov Institute of Mathematics, vol. 284, pp. 244–262, (2014) (see also arXiv:1206.0393v1 [stat.ML]. 2 Jun 2012)

  43. Temlyakov, V.N.: Sparse Approximation and Recovery by Greedy Algorithms in Banach Spaces. arXiv:1303.6811v1 [stat.ML]. 27 March 2013

  44. Temlyakov, V.N.: Chebyshev Greedy Algorithm in Convex Optimization. arXiv:1312.1244v1 [stat.ML]. 4 Dec 2013

  45. Tewari, A., Ravikumar, P., Dhillon, I.S.: Greedy Algorithms for Structurally Constrained High Dimensional Problems, prerint 1–10 (2012)

  46. Zhang, T.: Sequential greedy approximation for certain convex optimization problems. IEEE Trans. Inf. Theory 49(3), 682–691 (2003)

    Article  MATH  Google Scholar 

  47. Zhang, T.: Adaptive forward–backward greedy algorithm for learning sparse representations. IEEE Trans. Inf. Theory 57, 4689–4708 (2011)

    Article  Google Scholar 

  48. Zhang, T.: Sparse recovery with orthogonal matching pursuit under RIP. IEEE Trans. Inf. Theory 57, 6215–6221 (2011)

    Article  Google Scholar 

Download references

Acknowledgments

This paper was motivated by the IMA Annual Program Workshop “Machine Learning: Theory and Computation” (March 26–30, 2012), in particular, by talks of Steve Wright and Pradeep Ravikumar. The author is very thankful to Arkadi Nemirovski for an interesting discussion of the results and for his remarks. The author is also very grateful to the referees and the editor whose detailed comments helped to substantially improve the presentation of the results.

Author information

Authors and Affiliations

  1. University of South Carolina, Columbia, SC, USA

    V. N. Temlyakov

  2. Steklov Institute of Mathematics, Moscow, Russia

    V. N. Temlyakov

Authors
  1. V. N. Temlyakov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. N. Temlyakov.

Additional information

Communicated by Joel A. Tropp.

Research was supported by NSF Grant DMS-1160841.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Temlyakov, V.N. Greedy Approximation in Convex Optimization. Constr Approx 41, 269–296 (2015). https://doi.org/10.1007/s00365-014-9272-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00365-014-9272-0

Keywords

Mathematics Subject Classification

Search

Navigation