Skip to main content
SpringerLink
Log in

Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

The augmented Lagrangian method (ALM) is one of the most useful methods for constrained optimization. Its convergence has been well established under convexity assumptions or smoothness assumptions, or under both assumptions. ALM may experience oscillations and divergence when the underlying problem is simultaneously nonconvex and nonsmooth. In this paper, we consider the linearly constrained problem with a nonconvex (in particular, weakly convex) and nonsmooth objective. We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are weaker than those in the literature. We call it the Moreau envelope augmented Lagrangian (MEAL) method. We also show that the iteration complexity of MEAL is \(o(\varepsilon ^{-2})\) to yield an \(\varepsilon \)-accurate first-order stationary point. We establish its whole sequence convergence (regardless of the initial guess) and a rate when a Kurdyka–Łojasiewicz property is assumed. Moreover, when the subproblem of MEAL has no closed-form solution and is difficult to solve, we propose two practical variants of MEAL, an inexact version called iMEAL with an approximate proximal update, and a linearized version called LiMEAL for the constrained problem with a composite objective. Their convergence is also established.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Availability of data and materials

Enquiries about data availability should be directed to the authors.

Notes

  1. Locally linear convergence means exponentially fast convergence to a local minimum from a sufficiently close initial point.

References

  1. Andreani, R., Birgin, E.G., Martinez, J.M., Schuverdt, M.L.: On augmented Lagrangian methods with general lower-level constraints. SIAM J. Optim. 18(4), 1286–1309 (2007)

    MathSciNet  MATH  Google Scholar 

  2. Andreani, R., Birgin, E.G., Martinez, J.M., Schuverdt, M.L.: Augmented Lagrangian methods under the constant positive linear dependence constraint qualification. Math. Program. 111, 5–32 (2008)

    MathSciNet  MATH  Google Scholar 

  3. Andreani, R., Birgin, E.G., Martinez, J.M., Schuverdt, M.L.: Second-order negative-curvature methods for box-constrained and general constrained optimization. Comput. Optim. Appl. 45(2), 209–236 (2010)

    MathSciNet  MATH  Google Scholar 

  4. Andreani, R., Fazzio, N., Schuverdt, M.L., Secchin, L.: A sequential optimality condition related to the quasi-normality constraint qualification and its algorithmic consequences. SIAM J. Optim. 29(1), 743–766 (2019)

    MathSciNet  MATH  Google Scholar 

  5. Andreani, R., Secchin, L., Silva, P.: Convergence properties of a second order augmented Lagrangian method for mathematical programs with complementarity constraints. SIAM J. Optim. 28(3), 2574–2600 (2018)

    MathSciNet  MATH  Google Scholar 

  6. Armand, P., Omheni, R.: A globally and quadratically convergent primal-dual augmented Lagrangian algorithm for equality constrained optimization. Optim. Methods Softw. 32(1), 1–21 (2017)

    MathSciNet  MATH  Google Scholar 

  7. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. 116, 5–16 (2009)

    MathSciNet  MATH  Google Scholar 

  8. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods. Math. Program. 137, 91–219 (2013)

    MathSciNet  MATH  Google Scholar 

  9. Bertsekas, D.P.: In: Convergence rate of penalty and multiplier methods, pp. 260–264. SanDiego, California (1973)

  10. Bertsekas, D.P.: On penalty and multiplier methods for constrained minimization. SIAM J. Control. Optim. 14(2), 216–235 (1976)

    MathSciNet  MATH  Google Scholar 

  11. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, London (1982)

    MATH  Google Scholar 

  12. Bian, W., Chen, X., Ye, Y.: Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization. Math. Program. 149(1), 301–327 (2005)

    MathSciNet  MATH  Google Scholar 

  13. Birgin, E.G., Castillo, R., Martinez, J.M.: Numerical comparison of augmented Lagrangian algorithms for nonconvex problems. Comput. Optim. Appl. 31, 31–56 (2005)

    MathSciNet  MATH  Google Scholar 

  14. Birgin, E.G., Floudas, C.A., Martinez, J.M.: Global minimization using an augmented Lagrangian method with variable lower-level constraints. Math. Program. 125, 139–162 (2010)

    MathSciNet  MATH  Google Scholar 

  15. Birgin, E.G., Floudas, C.A., Martinez, J.M.: The boundedness of penalty parameters in an augmented Lagrangian method with constrained subproblems. Optim. Methods Softw. 27(6), 1001–1024 (2012)

    MathSciNet  MATH  Google Scholar 

  16. Birgin, E.G., Haeser, G., Ramos, A.: Augmented Lagrangians with constrained subproblems and convergence to second-order stationary points. Comput. Optim. Appl. 69(1), 51–75 (2018)

    MathSciNet  MATH  Google Scholar 

  17. Birgin, E.G., Martinez, J.M.: Practical Augmented Lagrangian Methods for Constrained Optimization, vol. 10. SIAM, Philadelphia (2014)

    MATH  Google Scholar 

  18. Birgin, E.G., Martinez, J.M.: Complexity and performance of an augmented Lagrangian algorithm. Optim, Methods Softw (2020)

    MATH  Google Scholar 

  19. Bochnak, J., Coste, M., Roy, M.F.: Real Algebraic Geometry, vol. 36. Springer, Berlin (1998)

    MATH  Google Scholar 

  20. Bolte, J., Daniilidis, A., Lewis, A.: The łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17(4), 1205–1223 (2007)

    MATH  Google Scholar 

  21. Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clark subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)

    MathSciNet  MATH  Google Scholar 

  22. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1), 459–494 (2014)

    MathSciNet  MATH  Google Scholar 

  23. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)

    MATH  Google Scholar 

  24. Conn, A.R., Gould, N.I.M., Startenaer, A., Toint, P.L.: Convergence properties of an augmented Lagrangian algorithm for optimization with a combination of general equality and linear constraints. SIAM J. Optim. 6, 674–703 (1996)

    MathSciNet  MATH  Google Scholar 

  25. Conn, A.R., Gould, N.I.M., Toint, P.L.: A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds. SIAM J. Numer. Anal. 28, 545–572 (1991)

    MathSciNet  MATH  Google Scholar 

  26. Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. SIAM, Philadelphia (2000)

    MATH  Google Scholar 

  27. Curtis, F.E., Jiang, H., Robinson, D.P.: An adaptive augmented Lagrangian method for large-scale constrained optimization. Math. Program. 152(1), 201–245 (2015)

    MathSciNet  MATH  Google Scholar 

  28. Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM J. Optim. 29(1), 207–239 (2019)

    MathSciNet  MATH  Google Scholar 

  29. Deng, W., Lai, M.J., Peng, Z., Yin, W.: Parallel multi-block admm with \(o(1/k)\) convergence. J. Sci. Comput. 71, 712–736 (2017)

    MathSciNet  MATH  Google Scholar 

  30. Drusvyatskiy, D.: The proximal point method revisited. SIAG/OPT Views and News 26, 1–8 (2018)

    Google Scholar 

  31. Drusvyatskiy, D., Paquette, C.: Efficiency of minimizing compositions of convex functions and smooth maps. Math. Program. 178, 503–558 (2019)

    MathSciNet  MATH  Google Scholar 

  32. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96, 1348–1360 (2001)

    MathSciNet  MATH  Google Scholar 

  33. Fernadez, D., Solodov, M.V.: Local convergence of exact and inexact augmented Lagrangian methods under the second-order sufficient optimality condition. SIAM J. Optim. 22(2), 384–407 (2012)

    MathSciNet  MATH  Google Scholar 

  34. Grapiglia, G.N., Yuan, Y.X.: On the complexity of an augmented Lagrangian method for nonconvex optimization. ArXiv e-prints (2019)

  35. Haeser, G., Liu, H., Ye, Y.: Optimality condition and complexity analysis for linearly-constrained optimization without differentiability on the boundary. Math. Program. 178, 263–299 (2019)

    MathSciNet  MATH  Google Scholar 

  36. Hajinezhad, D., Hong, M.: Perturbed proximal primal-dual algorithm for nonconvex nonsmooth optimization. Math. Program. 176, 207–245 (2019)

    MathSciNet  MATH  Google Scholar 

  37. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    MathSciNet  MATH  Google Scholar 

  38. Hong, M., Hajinezhad, D., Zhao, M.M.: Prox-pda,: In: The proximal primal-dual algorithm for fast dostributed nonconvex optimization and learning over networks, pp. 1529–1538. , Sydney, Australia (2017)

  39. Jiang, B., Lin, T., Ma, S., Zhang, S.: Structured nonconvex and nonsmooth otpmization: algorithms and iteration complexity analysis. Comput. Optim. Appl. 72(1), 115–157 (2019)

    MathSciNet  MATH  Google Scholar 

  40. Krantz, S., Parks, H.R.: A Primer of Real Analytic Functions, 2nd edn. Birkhauser, Basel (2002)

    MATH  Google Scholar 

  41. Kurdyka, K.: On gradients of functions definable in o-minimal structures. Annales de l’institut Fourier 48(3), 769–783 (1998)

  42. Li, G., Pong, T.K.: Calculus of the exponent of Kurdyka–łojasiewicz inequality and its applications to linear convergence of first-order methods. Found. Comput. Math. 18, 1199–1232 (2018)

    MathSciNet  MATH  Google Scholar 

  43. Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels. In: Les Équations aux dérivées partielles. Éditions du centre National de la Recherche Scientifique, Paris pp. 87–89 (1963)

  44. Łojasiewicz, S.: Sur la geometrie semi-et sous-analytique. Annales de l’institut Fourier 43(5), 1575–1595 (1993)

  45. Mordukhovich, B.S.: Variational Analysis and Generalized Differentiation I: Basic Theory. Springer-Verlag, New York (2006)

    Google Scholar 

  46. Moreau, J.: Proximité et dualité dans un espace hilbertien. Bull. Soc. Math. France 93, 273–299 (1965)

    MathSciNet  MATH  Google Scholar 

  47. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer-Verlag, New York (1999)

    MATH  Google Scholar 

  48. Nouiehed, M., Lee, J.D., Razaviyayn, M.: Convergence to second-order stationary for constrained non-convex optimization. ArXiv e-prints (2018)

  49. Nurminskii, E.A.: The quasigradient method for the solving of the nonlinear programming problems. Cybernetics 9, 145–150 (1973)

  50. O’Neill, M., Wright, S.J.: A log-barrier newton-cg method for bound constrained optimization with complexity guarantees. IMA J. Numer. Anal. 1–38 (2020)

  51. Polyak, B.T., Tretyakov, N.V.: The method of penalty bounds for constrained extremum problems. Zh. Vych Mat i Mat. Fiz, 13:34–46 = U.S.S.R. Computational Mathematics and Mathmatical. Physics 13, 42–58 (1973)

    Google Scholar 

  52. Powell, M.J.D.: In: Optimization, R. Fletcher. (ed.) A method for nonlinear constraints in minimization problems, pp. 283–298. Academic Press, London (1969)

  53. Rockafellar, R.T.: The multiplier method of hestenes and powell applied to convex programming. J. Optim. Theory Appl. 12, 555–562 (1973)

  54. Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1(2), 97–116 (1976)

    MathSciNet  MATH  Google Scholar 

  55. Rockafellar, R.T., Wets, R.J.B.: Variational Analysis. Springer, New York (1997)

    MATH  Google Scholar 

  56. Shiota, M.: Geometry of Subanalytic and Semialgebraic Sets (Progress in Mathematics). Birkhauser, Basel (1997)

    MATH  Google Scholar 

  57. Tretykov, N.Y.: The method of penalty estimates of convex programming. Econ. Math. Methods (Russian) 9, 525–540 (1973)

    MathSciNet  Google Scholar 

  58. Wang, Y., Yin, W., Zeng, J.: Global convergence of admm in nonconvex nonsmooth optimization. J. Sci. Comput. 78, 29–63 (2019)

    MathSciNet  MATH  Google Scholar 

  59. Xie, Y., Wright, S.J.: Complexity of proximal augmented Lagrangian for nonconvex optimalization with nonlinear equality constraints. ArXiv e-prints (2019)

  60. Xu, Y., Yin, W.: A block coordinate descent method for regularized multiconvex optimizaton with applications to nonnegative tensor factorization and completion. SIAM J. Imaging Sci. 6(3), 1758–1789 (2013)

    MathSciNet  MATH  Google Scholar 

  61. Yu, P., Li, G., Pong, T.: Kurdyka–łojasiewicz exponent via inf-projection. Found. Comput. Math. (2021). https://doi.org/10.1007/s10208-021-09528-6

    Article  Google Scholar 

  62. Zeng, J., Lau, T.T.K., Lin, S.B., Yao, Y.: In: Global convergence of block coordinate descent in deep learning. , Long Beach, California (2019) . (PMLR 97)

  63. Zeng, J., Lin, S.B., Yao, Y., Zhou, D.X.: On admm in deep learning: convergence and saturation-avoidance. J. Mach. Learn. Res. 22(199), 1–67 (2021)

    MathSciNet  Google Scholar 

  64. Zeng, J., Yin, W.: On nonconvex descentralized gradient descent. IEEE Trans. Signal Process. 66(11), 2834–2848 (2018)

    MathSciNet  MATH  Google Scholar 

  65. Zhang, C.H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38(2), 894–942 (2010)

    MathSciNet  MATH  Google Scholar 

  66. Zhang, J., Luo, Z.Q.: A global dual error bound and its application to the analysis of linearly constrained nonconvex optimization. ArXiv e-prints (2020)

  67. Zhang, J., Luo, Z.Q.: A proximal alternating direction method of multiplier for linearly constrained nonconvex minimization. SIAM J. Optim. 30(3), 2272–2302 (2020)

    MathSciNet  MATH  Google Scholar 

  68. Zhou, D.X.: Universality of deep convolutional neural networks. Appl. Comput. Harmonic Anal. 48, 787–794 (2020)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China

    Jinshan Zeng

  2. Liu Bie Ju Centre for Mathematical Sciences, City University of Hong Kong, Hong Kong, China

    Jinshan Zeng & Ding-Xuan Zhou

  3. Damo Academy, Alibaba Group US, Bellevue, WA, USA

    Wotao Yin

  4. Department of Mathematics, School of Data Science, City University of Hong Kong, Hong Kong, China

    Ding-Xuan Zhou

Authors
  1. Jinshan Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wotao Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ding-Xuan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wotao Yin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We thank Kaizhao Sun for discussions that help us complete this paper, as well as presenting to us an additional approach to ensure boundedness. The work of J. Zeng is partly supported by National Natural Science Foundation of China (No. 61977038) and the Thousand Talents Plan of Jiangxi Province (No. jxsq2019201124). The work of D.-X. Zhou is partly supported by Research Grants Council of Hong Kong (No. CityU 11307319), Laboratory for AI-powered Financial Technologies, and the Hong Kong Institute for Data Science.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, J., Yin, W. & Zhou, DX. Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints. J Sci Comput 91, 61 (2022). https://doi.org/10.1007/s10915-022-01815-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-022-01815-w

Keywords

Search

Navigation