Skip to main content
SpringerLink
Log in

On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

The proximal point algorithm (PPA) has been well studied in the literature. In particular, its linear convergence rate has been studied by Rockafellar in 1976 under certain condition. We consider a generalized PPA in the generic setting of finding a zero point of a maximal monotone operator, and show that the condition proposed by Rockafellar can also sufficiently ensure the linear convergence rate for this generalized PPA. Indeed we show that these linear convergence rates are optimal. Both the exact and inexact versions of this generalized PPA are discussed. The motivation of considering this generalized PPA is that it includes as special cases the relaxed versions of some splitting methods that are originated from PPA. Thus, linear convergence results of this generalized PPA can be used to better understand the convergence of some widely used algorithms in the literature. We focus on the particular convex minimization context and specify Rockafellar’s condition to see how to ensure the linear convergence rate for some efficient numerical schemes, including the classical augmented Lagrangian method proposed by Hensen and Powell in 1969 and its relaxed version, the original alternating direction method of multipliers (ADMM) by Glowinski and Marrocco in 1975 and its relaxed version (i.e., the generalized ADMM by Eckstein and Bertsekas in 1992). Some refined conditions weaker than existing ones are proposed in these particular contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Auslender, A., Teboulle, M.: Interior projection-like methods for monotone variational inequalities. Math. Program. 104, 39–68 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  2. Auslender, A., Teboulle, M., Ben-Tiba, S.: A logarithmic-quadratic proximal method for variational inequalities. Comput. Optim. Appl. 12, 31–40 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York (1982)

    MATH  Google Scholar 

  4. Blum, E., Oettli, W.: Mathematische Optimierung Grundlagen und Verfahren. Ökonometrie und Unternehmensforschung. Springer, Berlin (1975)

    MATH  Google Scholar 

  5. Boley, D.: Local linear convergence of ADMM on quadratic or linear programs. SIAM J. Optim. 23, 2183–2207 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  6. Burke, J.V., Qian, M.J.: A variable metric proximal point algorithm for monotone operators. SIAM J. Cont. Optim. 37, 353–375 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  7. Cai, X.J., Gu, G., He, B.S., Yuan, X.M.: A relaxed customized proximal point algorithm for separable convex programming. Sci. China Math. 56, 2179–2186 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  8. Corman, E., Yuan, X.M.: A generalized proximal point algorithm and its convergence rate. SIAM J. Optim. 24, 1614–1638 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  9. Davis, D., Yin, W.: Faster convergence rates of relaxed Peaceman-Rachford and ADMM under regularity assumptions, http://arxiv.org/pdf/1407.5210.pdf (2017)

  10. Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66, 889–916 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  11. Douglas, J., Rachford, H.H.: On the numerical solution of the heat conduction problem in 2 and 3 space variables. Trans. Am. Math. Soc. 82, 421–439 (1956)

    Article  MathSciNet  MATH  Google Scholar 

  12. Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal points algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  13. Fang, E.X., Liu, H., He, B.S., Yuan, X.M.: The generalized alternating direction method of multipliers: New theoretical insights and applications. Math. Program. Comput. 7(2), 149–187 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  14. Fukushima, M., Mine, H.: A generalized proximal point algorithm for certain nonconvex minimization problems. Int. J. Syst. Sci. 12, 989–1000 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  15. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrange Methods: Applications to the Solution of Boundary-valued Problems, pp. 299–331. North Holland, Amsterdam (1983)

    Chapter  Google Scholar 

  16. Glowinski, R., Marrocco, A.: Approximation par éléments finis d’ordre un et résolution par pénalisation-dualité d’une classe de problèmes non linéaires, R.A.I.R.O., R2, 41–76 (1975)

  17. Gol’shtein, E.G., Tret’yakov, N.V.: Modified Lagrangian in convex programming and their generalizations. Math. Program. Study 10, 86–97 (1979)

    Article  MathSciNet  Google Scholar 

  18. Güler, O.: On the convergence of the proximal point algorithm for convex minimization. SIAM J. Optim. 1, 403–419 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  19. Güler, O.: New proximal point algorithms for convex minimization. SIAM J. Optim. 2, 649–664 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  20. Han, D.R., Yuan, X.M.: Local linear convergence of the alternating direction method of multipliers for quadratic programs. SIAM J. Numer. Anal. 51, 3446–3457 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  21. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  22. Korpelevich, G.M.: The extragradient method for finding saddle points and other problems. Ekonomika i Matematchskie Metody 12, 747–756 (1976)

    MathSciNet  MATH  Google Scholar 

  23. Liang, J.W., Fadili, J., Peyré, G.: Convergence rates with inexact nonexpansive operators. Math. Progam. 159, 403–434 (2016)

    Article  MATH  Google Scholar 

  24. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  25. Martinet, B.: Regularization d’inequations variationelles par approximations successives. Revue Francaise d’Informatique et de Recherche Opérationelle 4, 154–159 (1970)

    MATH  Google Scholar 

  26. Martinet, B.: Determination approchdée d’un point fixe d’une application pseudo-contractante. C.R. Acad. Sci. Paris 274, 163–165 (1972)

    MathSciNet  MATH  Google Scholar 

  27. Moreau, J.J.: Proximité et dualit ’e dans un espace Hilbertien. Bull. Soc. Math. France 93, 273–299 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  28. Nemirovski, A.: Prox-method with rate of convergence \(O(1/t)\) for variational inequality with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim. 15, 229–251 (2005)

    Article  MATH  Google Scholar 

  29. Nesterov, Y.E.: A method for solving the convex programming problem with convergence rate \(O(1/{k^2})\). Dokl. Akad. Nauk SSSR 269, 543–547 (1983)

    MathSciNet  Google Scholar 

  30. Peaceman, D.H., Rachford, H.H.: The numerical solution of parabolic elliptic differential equations. J. Soc. Indust, Appl. Math. 3, 28–41 (1955)

    Article  MathSciNet  MATH  Google Scholar 

  31. Polyak, B.T.: Introduction to Optimization, Translations Series in Mathematics and Engineering, Optimization Software. Publications Division, New York (1987)

    Google Scholar 

  32. Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969)

    Google Scholar 

  33. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton, N.J. (1970)

    Book  MATH  Google Scholar 

  34. Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 97–116 (1976)

    MathSciNet  MATH  Google Scholar 

  35. Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 877–898 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  36. Shefi, R., Teboulle, M.: Rate of convergence analysis of decmposition methods based on the proximal method of multipliers for convex minimization. SIAM J. Optim. 24, 269–297 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  37. Shen, L., Pan, S.: Linear convergence of the generalized PPA and several splitting methods for the composite inclusion problem. manuscript, (2015)

  38. Teboulle, M.: Convergence of proximal-like algorithms. SIAM J. Optim. 7, 1069–1083 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  39. Xu, H.K.: Iterative algorithms for nonlinear operators. J. Lond. Math. Soc. 66, 240–256 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  40. Yao, Y.Y., Noor, M.A.: On convergence criteria of generalized proximal point algorithms. J. Comput. Appl. Math. 217, 46–55 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  41. Zakon, E.: Mathematical Analysis, vol. I. The Trillia Group, Indiana (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Nanjing University, Nanjing, 210093, China

    Min Tao

  2. Department of Mathematics, Hong Kong Baptist University, Hong Kong, China

    Xiaoming Yuan

Authors
  1. Min Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoming Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoming Yuan.

Additional information

Min Tao was supported by the Natural Science Foundation of China: NSFC-11301280, 11471156. Xiaoming Yuan was supported by the General Research Fund from Hong Kong Research Grants Council: 12300515.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tao, M., Yuan, X. On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm. J Sci Comput 74, 826–850 (2018). https://doi.org/10.1007/s10915-017-0477-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-017-0477-9

Keywords

Search

Navigation