Mathematical Programming

, Volume 142, Issue 1–2, pp 269–310 | Cite as

Randomized first order algorithms with applications to 1-minimization

  • Anatoli Juditsky
  • Fatma Kılınç Karzan
  • Arkadi Nemirovski
Full Length Paper Series A


In this paper we propose randomized first-order algorithms for solving bilinear saddle points problems. Our developments are motivated by the need for sublinear time algorithms to solve large-scale parametric bilinear saddle point problems where cheap online assessment of the solution quality is crucial. We present the theoretical efficiency estimates of our algorithms and discuss a number of applications, primarily to the problem of 1 minimization arising in sparsity-oriented signal processing. We demonstrate, both theoretically and by numerical examples, that when seeking for medium-accuracy solutions of large-scale 1 minimization problems, our randomized algorithms outperform significantly (and progressively as the sizes of the problem grow) the state-of-the art deterministic methods.

Mathematics Subject Classification

90C25 90C47 90C06 65K15 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Azuma K.: Weighted sums of certain dependent random variables. Tökuku Math. J. 19, 357–367 (1967)MathSciNetMATHGoogle Scholar
  2. 2.
    Candès E.J., Tao T.: Decoding by linear programming. IEEE Trans. Inform. Theory 51, 4203–4215 (2006)CrossRefGoogle Scholar
  3. 3.
    Candès E.J.: Compressive sampling. In: Sanz-Solé, M., Soria, J., Varona, J.L., Verdera, J. (eds) International Congress of Mathematicians, Madrid 2006, vol. III, pp. 1437–1452. European Mathematical Society Publishing House, Zurich (2006)Google Scholar
  4. 4.
    Dalalyan A.S., Juditsky A., Spokoiny V.: A new algorithm for estimating the effective dimension-reduction subspace. J. Mach. Learn. Res. 9, 1647–1678 (2008)MathSciNetMATHGoogle Scholar
  5. 5.
    Donoho D., Huo X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inform. Theory 47(7), 2845–2862 (2001)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Dümbgen L., van de Geer S., Veraar M., Wellner J.: Nemirovski’s inequalities revisited. Am. Math. Mon. 117(2), 138–160 (2010)CrossRefMATHGoogle Scholar
  7. 7.
    Grigoriadis M.D., Khachiyan l.G.: A sublinear-time randomized approximation algorithm for matrix games. Oper. Res. Lett. 18, 53–58 (1995)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Hoeffding W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Judistky, A., Nemirovski, A.: Large deviations of vector-valued martingales in 2-smooth normed spaces (2008) E-print:
  10. 10.
    Juditsky, A., Nemirovski, A., Tauvel, C.: Solving variational inequalities with stochastic mirror prox algorithm. Stoch. Syst. 1(1), 17–58 (2011)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Lemaréchal C., Nemirovski A., Nesterov Y.: New variants of bundle methods. Math. Program. 69(1), 111–148 (1995)CrossRefMATHGoogle Scholar
  12. 12.
    Nemirovskii, A., Yudin, D.: Efficient methods for large-scale convex problems. Ekonomika i Matematicheskie Metody (in Russian), 15(1), 135–152 (1979)Google Scholar
  13. 13.
    Nemirovski A.: Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim. 15, 229–251 (2004)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Nemirovski A., Juditsky A., Lan G., Shapiro A.: Stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Nesterov Y.: Smooth minimization of non-smooth functions—CORE Discussion Paper 2003/12, February 2003. Math. Progr. 103, 127–152 (2005)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Pinelis I.: Optimum bounds for the distributions of martingales in banach spaces. Ann. Probab. 22(4), 1679–1706 (1994)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer and Mathematical Optimization Society 2012

Authors and Affiliations

  • Anatoli Juditsky
    • 1
  • Fatma Kılınç Karzan
    • 2
  • Arkadi Nemirovski
    • 3
  1. 1.LJK, Université J. FourierGrenoble Cedex 9France
  2. 2.Carnegie Mellon UniversityPittsburghUSA
  3. 3.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations