Mathematical Programming

, Volume 142, Issue 1–2, pp 269–310 | Cite as

Randomized first order algorithms with applications to 1-minimization

  • Anatoli Juditsky
  • Fatma Kılınç Karzan
  • Arkadi Nemirovski
Full Length Paper Series A

Abstract

In this paper we propose randomized first-order algorithms for solving bilinear saddle points problems. Our developments are motivated by the need for sublinear time algorithms to solve large-scale parametric bilinear saddle point problems where cheap online assessment of the solution quality is crucial. We present the theoretical efficiency estimates of our algorithms and discuss a number of applications, primarily to the problem of 1 minimization arising in sparsity-oriented signal processing. We demonstrate, both theoretically and by numerical examples, that when seeking for medium-accuracy solutions of large-scale 1 minimization problems, our randomized algorithms outperform significantly (and progressively as the sizes of the problem grow) the state-of-the art deterministic methods.

Mathematics Subject Classification

90C25 90C47 90C06 65K15 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Azuma K.: Weighted sums of certain dependent random variables. Tökuku Math. J. 19, 357–367 (1967)MathSciNetMATHGoogle Scholar
  2. 2.
    Candès E.J., Tao T.: Decoding by linear programming. IEEE Trans. Inform. Theory 51, 4203–4215 (2006)CrossRefGoogle Scholar
  3. 3.
    Candès E.J.: Compressive sampling. In: Sanz-Solé, M., Soria, J., Varona, J.L., Verdera, J. (eds) International Congress of Mathematicians, Madrid 2006, vol. III, pp. 1437–1452. European Mathematical Society Publishing House, Zurich (2006)Google Scholar
  4. 4.
    Dalalyan A.S., Juditsky A., Spokoiny V.: A new algorithm for estimating the effective dimension-reduction subspace. J. Mach. Learn. Res. 9, 1647–1678 (2008)MathSciNetMATHGoogle Scholar
  5. 5.
    Donoho D., Huo X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inform. Theory 47(7), 2845–2862 (2001)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Dümbgen L., van de Geer S., Veraar M., Wellner J.: Nemirovski’s inequalities revisited. Am. Math. Mon. 117(2), 138–160 (2010)CrossRefMATHGoogle Scholar
  7. 7.
    Grigoriadis M.D., Khachiyan l.G.: A sublinear-time randomized approximation algorithm for matrix games. Oper. Res. Lett. 18, 53–58 (1995)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Hoeffding W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Judistky, A., Nemirovski, A.: Large deviations of vector-valued martingales in 2-smooth normed spaces (2008) E-print: http://www2.isye.gatech.edu/~nemirovs/LargeDevSubmitted.pdf
  10. 10.
    Juditsky, A., Nemirovski, A., Tauvel, C.: Solving variational inequalities with stochastic mirror prox algorithm. Stoch. Syst. 1(1), 17–58 (2011)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Lemaréchal C., Nemirovski A., Nesterov Y.: New variants of bundle methods. Math. Program. 69(1), 111–148 (1995)CrossRefMATHGoogle Scholar
  12. 12.
    Nemirovskii, A., Yudin, D.: Efficient methods for large-scale convex problems. Ekonomika i Matematicheskie Metody (in Russian), 15(1), 135–152 (1979)Google Scholar
  13. 13.
    Nemirovski A.: Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim. 15, 229–251 (2004)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Nemirovski A., Juditsky A., Lan G., Shapiro A.: Stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Nesterov Y.: Smooth minimization of non-smooth functions—CORE Discussion Paper 2003/12, February 2003. Math. Progr. 103, 127–152 (2005)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Pinelis I.: Optimum bounds for the distributions of martingales in banach spaces. Ann. Probab. 22(4), 1679–1706 (1994)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer and Mathematical Optimization Society 2012

Authors and Affiliations

  • Anatoli Juditsky
    • 1
  • Fatma Kılınç Karzan
    • 2
  • Arkadi Nemirovski
    • 3
  1. 1.LJK, Université J. FourierGrenoble Cedex 9France
  2. 2.Carnegie Mellon UniversityPittsburghUSA
  3. 3.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations