Skip to main content
SpringerLink
Log in

A modified inexact Levenberg–Marquardt method with the descent property for solving nonlinear equations

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

In this work, we propose a modified inexact Levenberg–Marquardt method with the descent property for solving nonlinear equations. A novel feature of the proposed method is that one can directly use the search direction generated by the approach to perform Armijo-type line search once the unit step size is not acceptable. We achieve this via properly controlling the level of inexactness such that the resulting search direction is automatically a descent direction for the merit function. Under the local Lipschitz continuity of the Jacobian, the global convergence of the proposed method is established, and an iteration complexity bound of \(O(1/\epsilon ^2)\) to reach an \(\epsilon \)-stationary solution is proved under some appropriate conditions. Moreover, with the aid of the designed inexactness condition, we establish the local superlinear rate of convergence for the proposed method under the Hölderian continuity of the Jacobian and the Hölderian local error bound condition. For some special parameters, the convergence rate is even quadratic. The numerical experiments on the underdetermined nonlinear equations illustrate the effectiveness and efficiency of the algorithm compared with a previously proposed inexact Levenberg–Marquardt method. Finally, applying it to solve the Tikhonov-regularized logistic regression shows that our proposed method is quite promising.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availibility statement

The datasets tested in this study are openly available in the LIBSVM repository, https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. The codes supporting the numerical experiments are freely available in the Github repository, https://github.com/jhyin-optim/Adaptive_ILM.

References

  1. Kelley, C.T.: Solving Nonlinear Equations with Newton’s Method, Fundamentals of Algorithms. SIAM, Philadelphia (2003)

    Google Scholar 

  2. Fischer, A., Herrich, M., Izmailov, A.F., Solodov, M.V.: A globally convergent LP-Newton method. SIAM J. Optim. 26(4), 2012–2033 (2016)

    MathSciNet  Google Scholar 

  3. Zhu, X.D., Lin, G.H.: Improved convergence results for a modified Levenberg–Marquardt method for nonlinear equations and applications in MPCC. Optim. Methods Softw. 31(4), 791–804 (2016)

    MathSciNet  Google Scholar 

  4. Ahookhosh, M., Artacho, F.J.A., Fleming, R.M., Vuong, P.T.: Local convergence of the Levenberg–Marquardt method under Hölder metric subregularity. Adv. Comput. Math. 45(5), 2771–2806 (2019)

    MathSciNet  Google Scholar 

  5. Ahookhosh, M., Fleming, R.M., Vuong, P.T.: Finding zeros of Hölder metrically subregular mappings via globally convergent Levenberg–Marquardt methods. Optim. Methods Softw. 37(1), 113–149 (2022)

    MathSciNet  Google Scholar 

  6. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944)

    MathSciNet  Google Scholar 

  7. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)

    MathSciNet  Google Scholar 

  8. Yamashita, N., Fukushima, M.: On the rate of convergence of the Levenberg–Marquardt method. In: Topics in Numerical Analysis, pp. 239–249. Springer (2001)

  9. Fan, J.Y.: A modified Levenberg–Marquardt algorithm for singular system of nonlinear equations. J. Comput. Math. 21(5), 625–636 (2003)

    MathSciNet  Google Scholar 

  10. Fan, J.Y., Yuan, Y.X.: On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing 74(1), 23–39 (2005)

    MathSciNet  Google Scholar 

  11. Fan, J.Y.: The modified Levenberg–Marquardt method for nonlinear equations with cubic convergence. Math. Comput. 81(277), 447–466 (2012)

    MathSciNet  Google Scholar 

  12. Yang, X.: A higher-order Levenberg–Marquardt method for nonlinear equations. Appl. Math. Comput. 219(22), 10682–10694 (2013)

    MathSciNet  Google Scholar 

  13. Karas, E.W., Santos, S.A., Svaiter, B.F.: Algebraic rules for computing the regularization parameter of the Levenberg–Marquardt method. Comput. Optim. Appl. 65(3), 723–751 (2016)

    MathSciNet  Google Scholar 

  14. Chen, L.: A modified Levenberg–Marquardt method with line search for nonlinear equations. Comput. Optim. Appl. 65(3), 753–779 (2016)

    MathSciNet  Google Scholar 

  15. Zhao, X., Fan, J.Y.: On the multi-point Levenberg–Marquardt method for singular nonlinear equations. Comput. Appl. Math. 36(1), 203–223 (2017)

    MathSciNet  Google Scholar 

  16. Zhao, R.X., Fan, J.Y.: On a new updating rule of the Levenberg–Marquardt parameter. J. Sci. Comput. 74(2), 1146–1162 (2018)

    MathSciNet  Google Scholar 

  17. Amini, K., Rostami, F., Caristi, G.: An efficient Levenberg–Marquardt method with a new LM parameter for systems of nonlinear equations. Optimization 67(5), 637–650 (2018)

    MathSciNet  Google Scholar 

  18. Fan, J.Y., Huang, J.C., Pan, J.Y.: An adaptive multi-step Levenberg–Marquardt method. J. Sci. Comput. 78(1), 531–548 (2019)

    MathSciNet  Google Scholar 

  19. Behling, R., Gonçalves, D.S., Santos, S.A.: Local convergence analysis of the Levenberg–Marquardt framework for nonzero-residue nonlinear least-squares problems under an error bound condition. J. Optim. Theory Appl. 183(3), 1099–1122 (2019)

    MathSciNet  Google Scholar 

  20. Bergou, E.H., Diouane, Y., Kungurtsev, V.: Convergence and complexity analysis of a Levenberg–Marquardt algorithm for inverse problems. J. Optim. Theory Appl. 185, 927–944 (2020)

    MathSciNet  Google Scholar 

  21. Fischer, A.: Local behavior of an iterative framework for generalized equations with nonisolated solutions. Math. Program. 94(1), 91–124 (2002)

    MathSciNet  Google Scholar 

  22. Guo, L., Lin, G.H., Jane, J.Y.: Solving mathematical programs with equilibrium constraints. J. Optim. Theory Appl. 166(1), 234–256 (2015)

    MathSciNet  Google Scholar 

  23. Wang, H.Y., Fan, J.Y.: Convergence rate of the Levenberg–Marquardt method under Hölderian local error bound. Optim. Methods Softw. 35(4), 767–786 (2020)

    MathSciNet  Google Scholar 

  24. Dan, H., Yamashita, N., Fukushima, M.: Convergence properties of the inexact Levenberg–Marquardt method under local error bound conditions. Optim. Methods Softw. 17(4), 605–626 (2002)

    MathSciNet  Google Scholar 

  25. Fan, J.Y., Pan, J.Y.: Inexact Levenberg–Marquardt method for nonlinear equations. Discrete Contin. Dyn. Syst. B 4(4), 1223–1232 (2004)

    MathSciNet  Google Scholar 

  26. Fan, J.Y., Pan, J.Y.: On the convergence rate of the inexact Levenberg–Marquardt method. J. Ind. Manag. Optim. 7(1), 199–210 (2011)

    MathSciNet  Google Scholar 

  27. Fischer, A., Shukla, P., Wang, M.: On the inexactness level of robust Levenberg–Marquardt methods. Optimization 59(2), 273–287 (2010)

    MathSciNet  Google Scholar 

  28. Bao, J., Yu, C.K.W., Wang, J., Hu, Y., Yao, J.C.: Modified inexact Levenberg–Marquardt methods for solving nonlinear least squares problems. Comput. Optim. Appl. 74(2), 547–582 (2019)

    MathSciNet  Google Scholar 

  29. Wang, H.Y., Fan, J.Y.: Convergence properties of inexact Levenberg–Marquardt method under Hölderian local error bound. J. Ind. Manag. Optim. 17(4), 2265–2275 (2021)

    MathSciNet  Google Scholar 

  30. Huang, J.C., Fan, J.Y.: Global complexity bound of the inexact Levenberg–Marquardt method. J. Oper. Res. Soc. China 6(3), 417–428 (2018)

    MathSciNet  Google Scholar 

  31. Ueda, K., Yamashita, N.: On a global complexity bound of the Levenberg–Marquardt method. J. Optim. Theory Appl. 147(3), 443–453 (2010)

    MathSciNet  Google Scholar 

  32. Ueda, K., Yamashita, N.: Global complexity bound analysis of the Levenberg–Marquardt method for nonsmooth equations and its application to the nonlinear complementarity problem. J. Optim. Theory Appl. 152(2), 450–467 (2012)

    MathSciNet  Google Scholar 

  33. Zhao, R., Fan, J.: Global complexity bound of the Levenberg–Marquardt method. Optim. Methods Softw. 31(4), 805–814 (2016)

    MathSciNet  Google Scholar 

  34. Bellavia, S., Gratton, S., Riccietti, E.: A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients. Numer. Math. 140(3), 791–825 (2018)

    MathSciNet  Google Scholar 

  35. Jian, J., Qin, Y., Yin, J., Ma, G.: Theoretical characteristics and numerical methods for a class of special piecewise quadratic optimization. Appl. Numer. Math. 184, 340–356 (2023)

    MathSciNet  Google Scholar 

  36. Zhou, G., Toh, K.C.: Superlinear convergence of a Newton-type algorithm for monotone equations. J. Optim. Theory Appl. 125(1), 205–221 (2005)

    MathSciNet  Google Scholar 

  37. Mordukhovich, B.S., Yuan, X., Zeng, S., Zhang, J.: A globally convergent proximal Newton-type method in nonsmooth convex optimization. Mathematical Programming, pp. 1–38 (2022)

  38. Solodov, M.V., Svaiter, B.F.: A globally convergent inexact Newton method for systems of monotone equations. In: Fukushima, M., Qi, L. (eds.) Reformulation: Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, Applied Optimization, vol. 22, pp. 355–369. Springer, Boston (1998)

    Google Scholar 

  39. Solodov, M.V., Svaiter, B.F.: A truly globally convergent Newton-type method for the monotone nonlinear complementarity problem. SIAM J. Optim. 10(2), 605–625 (2000)

    MathSciNet  Google Scholar 

  40. Studniarski, M., Ward, D.E.: Weak sharp minima: characterizations and sufficient conditions. SIAM J. Control Optim. 38(1), 219–236 (1999)

    MathSciNet  Google Scholar 

  41. Hu, Y., Li, C., Yang, X.: On convergence rates of linearized proximal algorithms for convex composite optimization with applications. SIAM J. Optim. 26(2), 1207–1235 (2016)

    MathSciNet  Google Scholar 

  42. Bonnans, J.F., Ioffe, A.: Second-order sufficiency and quadratic growth for nonisolated minima. Math. Oper. Res. 20(4), 801–817 (1995)

    MathSciNet  Google Scholar 

  43. Pang, J.S.: Error bounds in mathematical programming. Math. Program. 79(1), 299–332 (1997)

    MathSciNet  Google Scholar 

  44. Hu, Y., Yang, X., Sim, C.K.: Inexact subgradient methods for quasi-convex optimization problems. Eur. J. Oper. Res. 240(2), 315–327 (2015)

    MathSciNet  Google Scholar 

  45. Hu, Y., Li, C., Meng, K., Qin, J., Yang, X.: Group sparse optimization via \(l_{p, q}\) regularization. J. Mach. Learn. Res. 18(1), 960–1011 (2017)

    Google Scholar 

  46. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2010)

    Google Scholar 

  47. Liu, H., Hu, J., Li, Y., Wen, Z.: Optimization: Modeling, Algorithm and Theory. Higher Education Press, Beijing (2020). (in Chinese)

    Google Scholar 

  48. Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (2006)

    Google Scholar 

  49. Moré, J.J., Garbow, B.S., Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. 7(1), 17–41 (1981)

    MathSciNet  Google Scholar 

  50. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)

    MathSciNet  Google Scholar 

  51. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the two anonymous referees and the handling editor for their constructive and pertinent suggestions and remarks that improved the quality of the paper significantly.

Funding

This work was supported by the National Natural Science Foundation of China (12171106), the Natural Science Foundation of Guangxi Province (2023GXNSFBA026029), the National Natural Science Foundation of China (12261008), Research Project of Guangxi Minzu University (2022KJQD03) and the Middle-aged and Young Teachers’ Basic Ability Promotion Project of Guangxi Province (2023KY0168).

Author information

Authors and Affiliations

  1. College of Mathematics and Physics, Center for Applied Mathematics of Guangxi, Guangxi Key Laboratory of Hybrid Computation and IC Design Analysis, Guangxi Minzu University, Nanning, 530006, China

    Jianghua Yin, Jinbao Jian & Guodong Ma

Authors
  1. Jianghua Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinbao Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guodong Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinbao Jian.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yin, J., Jian, J. & Ma, G. A modified inexact Levenberg–Marquardt method with the descent property for solving nonlinear equations. Comput Optim Appl 87, 289–322 (2024). https://doi.org/10.1007/s10589-023-00513-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-023-00513-z

Keywords

Mathematics Subject Classification

Search

Navigation