Dictionary descent in optimization

Abstract

The problem of convex optimization is studied. Usually in convex optimization the minimization is over a d-dimensional domain. Very often the convergence rate of an optimization algorithm depends on the dimension d. The algorithms studied in this paper utilize dictionaries instead of a canonical basis used in the coordinate descent algorithms. We show how this approach allows us to reduce dimensionality of the problem. Also, we investigate which properties of a dictionary are beneficial for the convergence rate of typical greedy-type algorithms.

This is a preview of subscription content, log in to check access.

References

  1. [1]

    J. M. Borwein and A. S. Lewis, Convex Analysis and Nonlinear Optimization. Theory and Examples, CMS Books in Mathematics/Ouvrages de Mathmatiques de la SMC, 3, Springer (New York, 2006).

    Google Scholar 

  2. [2]

    R. A. DeVore, Deterministic Constructions of Compressed Sensing Matrices, J. Complexity, 23 (2007), 918–925.

    MathSciNet  Article  MATH  Google Scholar 

  3. [3]

    R. A. DeVore and V. N. Temlyakov, Convex optimization on Banach spaces, Found. Comput. Math., published online: February 14, 2015, 1–26; arXiv:1401.0334v1, 1 Jan 2014.

    MathSciNet  Google Scholar 

  4. [4]

    I. Dumer, Covering spheres with spheres, Discrete Comput. Geom., 38 (2007), 665–679.

    MathSciNet  Article  MATH  Google Scholar 

  5. [5]

    M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems, IEEE J. Sel. Topics Signal Process., 1 (2007), 586–597.

    Article  Google Scholar 

  6. [6]

    A. C. Gilbert, S. Muthukrishnan and M. J. Strauss, Approximation of functions over redundant dictionaries using coherence, Proc. of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms (Baltimore, MD, 2003), ACM (New York 2003), 243–252.

    Google Scholar 

  7. [7]

    B. S. Kashin, On widths of octahedrons, Uspekhi Mat. Nauk, 30 (1975), 251–252.

    Google Scholar 

  8. [8]

    J. L. Nelson and V. N. Temlyakov, On the size of incoherent systems, J. Approx. Theory, 163 (2011), 1238–1245.

    MathSciNet  Article  MATH  Google Scholar 

  9. [9]

    Yu. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic Publishers (Boston, 2004).

    Google Scholar 

  10. [10]

    C. A. Rogers, Covering a sphere with spheres, Mathematika, 10 (1963), 157–164.

    MathSciNet  Article  MATH  Google Scholar 

  11. [11]

    V. N. Temlyakov, Greedy algorithms in Banach spaces, Adv. Comput. Math., 14 (2001), 277–292.

    MathSciNet  Article  MATH  Google Scholar 

  12. [12]

    V. N. Temlyakov, Greedy approximation, Cambridge University Press (Cambridge, 2011).

    Google Scholar 

  13. [13]

    V. N. Temlyakov, Greedy approximation in convex optimization, Constr. Approx., 41 (2015), 269–296; arXiv: 1206.0392v1 [stat.ML] 2 Jun 2012.

    MathSciNet  Article  MATH  Google Scholar 

  14. [14]

    V. N. Temlyakov, Greedy expansions in convex optimization, Proc. Steklov Inst. Math., 284 (2014), 244–262; arXiv: 1206.0393v1 [stat.ML] 2 Jun 2012.

    MathSciNet  Article  MATH  Google Scholar 

  15. [15]

    V. N. Temlyakov, Chebyshev Greedy Algorithm in convex optimization, IMI Preprint, 2013:08, 1–11; arXiv:1312.1244v1, 4 Dec 2013.

    Google Scholar 

  16. [16]

    V. N. Temlyakov, Sparse approximation and recovery by greedy algorithms in Banach spaces, Forum Math. Sigma, 2 (2014), e12, pp. 26; IMI Preprint, 2013:09,1–27; arXiv:1303.6811v1, 27 Mar 2013.

    MathSciNet  Google Scholar 

  17. [17]

    V. N. Temlyakov, Incoherent systems and covering in finite-dimensional Banach spaces, Mat. Sb., 205 (2014), 97–116.

    MathSciNet  Article  MATH  Google Scholar 

  18. [18]

    J.-L. Verger-Gaugry, Covering a ball with smaller equal balls in Rn, Discrete Comput. Geom., 33 (2005), 143–155.

    MathSciNet  Article  MATH  Google Scholar 

  19. [19]

    T. Zhang, Sparse Recovery with Orthogonal Matching Pursuit under RIP, IEEE Trans. Inf. Theory, 57 (2011), 6215–6221.

    MathSciNet  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to V. N. Temlyakov.

Additional information

Research was supported by NSF grant DMS-1160841.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Temlyakov, V.N. Dictionary descent in optimization. Anal Math 42, 69–89 (2016). https://doi.org/10.1007/s10476-016-0106-0

Download citation

Key words and phrases

  • sparse
  • optimization
  • greedy
  • Banach space
  • convergence rate

Mathematics Subject Classification

  • primary 41A46
  • secondary 65K05, 41A65, 46B20