Advertisement

Local convergence of the Levenberg–Marquardt method under Hölder metric subregularity

Abstract

We describe and analyse Levenberg–Marquardt methods for solving systems of nonlinear equations. More specifically, we propose an adaptive formula for the Levenberg–Marquardt parameter and analyse the local convergence of the method under Hölder metric subregularity of the function defining the equation and Hölder continuity of its gradient mapping. Further, we analyse the local convergence of the method under the additional assumption that the Łojasiewicz gradient inequality holds. We finally report encouraging numerical results confirming the theoretical findings for the problem of computing moiety conserved steady states in biochemical reaction networks. This problem can be cast as finding a solution of a system of nonlinear equations, where the associated mapping satisfies the Łojasiewicz gradient inequality assumption.

This is a preview of subscription content, log in to check access.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

References

  1. 1.

    Ahookhosh, M., Fleming, R.M.T., Vuong, P.T.: Finding zeros of Hölder metrically subregular mappings via globally convergent Levenberg–Marquardt methods, arXiv:1812.00818

  2. 2.

    Aragón Artacho, F.J., Fleming, R.: Globally convergent algorithms for finding zeros of duplomonotone mappings. Optim. Lett. 9(3), 569–584 (2015)

  3. 3.

    Aragón Artacho, F.J., Fleming, R., Vuong, P.T.: Accelerating the DC algorithm for smooth functions. Math. Program. 169B(1), 95–118 (2018)

  4. 4.

    Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program. 116(1-2), 5–16 (2009)

  5. 5.

    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program. 137A(1-2), 91–129 (2013)

  6. 6.

    Behling, R., Iusem, A.: The effect of calmness on the solution set of systems of nonlinear equations. Math. Program. 137A(1-2), 155–165 (2013)

  7. 7.

    Bellavia, S., Cartis, C., Gould, N., Morini, B., Toint, P.L.: Convergence of a regularized Euclidean residual algorithm for nonlinear least squares. SIAM J. Numer. Anal. 48(1), 1–29 (2010)

  8. 8.

    Bellavia, S., Morini, B.: Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA J. Numer. Anal. 35(2), 947–968 (2015)

  9. 9.

    Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optimiz. 17(4), 1205–1223 (2007)

  10. 10.

    Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Lojasiewicz inequalities: Subgradient flows, talweg, convexity. Trans. Amer. Math. Soc. 362(6), 3319–3363 (2010)

  11. 11.

    Cibulka, R., Dontchev, A.L., Kruger, A.Y.: Strong metric subregularity of mappings in variational analysis and optimization. J. Math. Anal. Appl. 457(2), 1247–1282 (2018)

  12. 12.

    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91B(2), 201–213 (2002)

  13. 13.

    Dontchev, A.L., Rockafellar, R.T.: Implicit Functions and Solution Mappings, 2 edn. Springer Series in Operations Research and Financial Engineering. Springer, New York (2014)

  14. 14.

    Eilenberger, G.: Solitons: Mathematical Methods for Physicists. Springer, Berlin (1983)

  15. 15.

    Fan, J.: Convergence rate of the trust region method for nonlinear equations under local error bound condition. Comput. Optim. Appl. 34(2), 215–227 (2006)

  16. 16.

    Fan, J.: The modified Levenberg–Marquardt method for nonlinear equations with cubic convergence. Math. Comput. 81(277), 447–466 (2012)

  17. 17.

    Fan, J., Pan, J.: A note on the Levenberg–Marquardt parameter. Appl. Math. Comput. 207, 351–359 (2009)

  18. 18.

    Fan, J., Yuan, Y.: On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing 74(1), 23–39 (2005)

  19. 19.

    Fischer, A.: Local behavior of an iterative framework for generalized equations with nonisolated solutions. Math. Program. 94B(1), 91–124 (2002)

  20. 20.

    Fischer, A., Herrich, M., Izmailov, A.F., Solodov, M.V.: A globally convergent LP–newton method. SIAM J. Optim. 26(4), 2012–2033 (2015)

  21. 21.

    Fleming, R., Thiele, I.: Mass conserved elementary kinetics is sufficient for the existence of a non-equilibrium steady state concentration. J. Theoret. Biol. 314, 173–181 (2012)

  22. 22.

    Fleming, R.M., Vlassis, N., Thiele, I., Saunders, M.A.: Conditions for duality between fluxes and concentrations in biochemical networks. J. Theoret. Biol. 409, 1–10 (2016)

  23. 23.

    Gevorgyan, A., Poolman, M., Fell, D.: Detection of stoichiometric inconsistencies in biomolecular models. Bioinformatics 24(19), 2245–2251 (2008)

  24. 24.

    Guo, L., Lin, G.H., Ye, J.J.: Solving mathematical programs with equilibrium constraints. J. Optim. Theory Appl. 166(1), 234–256 (2015)

  25. 25.

    Gwoździewicz, J.: The Łojasiewicz exponent of an analytic function at an isolated zero. Comment. Math. Helv. 74(3), 364–375 (1999)

  26. 26.

    Haraldsdóttir, H.S., Fleming, R.M.: Identification of conserved moieties in metabolic networks by graph theoretical analysis of atom transition networks. PLos Comput. Biol. 12(11), e1004,999 (2016)

  27. 27.

    Hasegawa, A.: Plasma Instabilities and Nonlinear Effects. Springer, Berlin (1975)

  28. 28.

    Heirendt, L., et al.: Creation and analysis of biochemical constraint-based models: the COBRA Toolbox v3.0. To appear in Nat. Protoc., https://doi.org/10.1038/s41596-018-0098-2

  29. 29.

    Hoffman, A.: On approximate solutions of systems of linear inequalities. J. Res. Nat. Bur. Standards 49, 263–265 (1952)

  30. 30.

    Izmailov, A.F., Solodov, M.V.: Error bounds for 2-regular mappings with Lipschitzian derivatives and their applications. Math. Program. 89B(3), 413–435 (2001)

  31. 31.

    Izmailov, A.F., Solodov, M.V.: The theory of 2-regularity for mappings with Lipschitzian derivatives and its applications to optimality conditions. Math. Oper. Res. 27(3), 614–635 (2002)

  32. 32.

    Izmailov, A.F., Solodov, M.V.: Newton-Type Methods for Optimization and Variational Problems. Springer, Berlin (2014)

  33. 33.

    Kanzow, C., Yamashita, N., Fukushima, M.: Levenberg–Marquardt methods with strong local convergence properties for solving nonlinear equations with convex constraints. J. Comput. Appl. Math. 172(2), 375–397 (2004)

  34. 34.

    Karas, E.W., Santos, S.A., Svaiter, B.F.: Algebraic rules for computing the regularization parameter of the Levenberg–Marquardt method. Comput. Optim. Appl. 65(3), 723–751 (2016)

  35. 35.

    Kelley, C.: Iterative Methods for Optimization. Frontiers Appl Math, vol. 18. SIAM, Philadelphia (1999)

  36. 36.

    Klamt, S., Haus, U.U., Theis, F.: Hypergraphs and cellular networks. PLoS Comput. Biol. 5(5), e1000,385 (2009)

  37. 37.

    Kruger, A.: Error bounds and Hölder metric subregularity. Set-valued Var. Anal. 23(4), 705–736 (2015)

  38. 38.

    Kurdyka, K., Spodzieja, S.: Separation of real algebraic sets and the Łojasiewicz exponent. Proc. Amer. Math. Soc. 142(9), 3089–3102 (2014)

  39. 39.

    Li, G., Mordukhovich, B.: Hölder metric subregularity with applications to proximal point method. SIAM J. Optim. 22(4), 1655–1684 (2012)

  40. 40.

    Lojasiewicz, S.: Ensembles semi-analytiques université de Gracovie (1965)

  41. 41.

    Ma, C., Jiang, L.: Some research on Levenberg–Marquardt method for the nonlinear equations. Appl. Math. Comput. 184, 1032–1040 (2007)

  42. 42.

    Mordukhovich, B.S.: Variational Analysis and Generalized Differentiation I. Springer, Berlin (2006)

  43. 43.

    Mordukhovich, B.S., Ouyang, W.: Higher-order metric subregularity and its applications. J. Global Optim. 63(4), 777–795 (2015)

  44. 44.

    Moré, J., Garbow, B., Hillstrom, K.: Testing unconstrained optimization software. ACM Trans. Math. Software 7(1), 17–41 (1981)

  45. 45.

    Ngai, H.V.: Global error bounds for systems of convex polynomials over polyhedral constraints. SIAM J. on Optim. 25(1), 521–539 (2015)

  46. 46.

    Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (2006)

  47. 47.

    Ortega, J., Rheinboldt, W.: Iterative solution of nonlinear equations in several variables. Society for industrial and applied mathematics (2000)

  48. 48.

    Pang, J.: Error bounds in mathematical programming. Math. Program. 79B (1–3), 299–332 (1997)

  49. 49.

    Parks, H., Krantz, S.: A Primer of Real Analytic Functions. Birkhäuser, Cambridge (1992)

  50. 50.

    Vui, H.: Global Holderian̈ error bound for nondegenerate polynomials. SIAM J. Optim. 23(2), 917–933 (2013)

  51. 51.

    Whitham, G.B.: Linear and Nonlinear Waves. Wiley, New York (1974)

  52. 52.

    Yamashita, N., Fukushima, M. Alefeld, G., Chen, X. (eds.): On the rate of convergence of the Levenberg–Marquardt method, vol. 15. Springer, Vienna (2001)

  53. 53.

    Yuan, Y.: Recent advances in trust region algorithms. Math. Program. 151B (1), 249–281 (2015)

  54. 54.

    Zhu, X., Lin, G.H.: Improved convergence results for a modified Levenberg–Marquardt method for nonlinear equations and applications in MPCC. Optim. Methods Softw. 31(4), 791–804 (2016)

Download references

Acknowledgements

We would like to thank Mikhail Solodov for suggesting the use of Levenberg–Marquardt methods for solving the system of nonlinear equations arising in biochemical reaction networks. Thanks also go to Michael Saunders for his useful comments on the first version of this manuscript. We are grateful to two anonymous reviewers for their constructive comments, which helped us improving the paper.

Funding

F.J. Aragón was supported by MINECO of Spain and ERDF of EU, as part of the Ramón y Cajal program (RYC-2013-13327) and the I+D grant MTM2014-59179-C2-1-P. M. Ahookhosh, R.M.T. Fleming, and P.T. Vuong were supported by the U.S. Department of Energy, Offices of Advanced Scientific Computing Research and the Biological and Environmental Research as part of the Scientific Discovery Through Advanced Computing program, grant no. DE-SC0010429. P.T. Vuong was also supported by the Austrian Science Fund (FWF), grant M2499-N32.

Author information

Correspondence to Francisco J. Aragón Artacho or Ronan M. T. Fleming.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Russell Luke

Appendix

Appendix

See Tables 1 and 2 for the summary results of the comparisons.

Table 1 Summary of the results of tuning the parameter η for LM-AR with parameters (47) and η ∈ {0.6, 0.7, 0.8, 0.9, 0.99, 0.999, 1} to solve (46) in 20 biological models.
Table 2 Summary of the results of LM-YF, LM-FY, LM-F, and LM-AR with parameters (47) and η = 0.999 for solving (46) in 20 biological models

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ahookhosh, M., Aragón Artacho, F.J., Fleming, R.M.T. et al. Local convergence of the Levenberg–Marquardt method under Hölder metric subregularity. Adv Comput Math 45, 2771–2806 (2019). https://doi.org/10.1007/s10444-019-09708-7

Download citation

Keywords

  • Nonlinear equation
  • Levenberg–Marquardt method
  • Local convergence rate
  • Hölder metric subregularity
  • Łojasiewicz inequality

Mathematics Subject Classification (2010)

  • 65K05
  • 65K10
  • 90C26
  • 92C42