Skip to main content
SpringerLink
Log in

Modified inexact Levenberg–Marquardt methods for solving nonlinear least squares problems

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

In the present paper, we propose a modified inexact Levenberg–Marquardt method (LMM) and its global version by virtue of Armijo, Wolfe or Goldstein line-search schemes to solve nonlinear least squares problems (NLSP), especially for the underdetermined case. Under a local error bound condition, we show that a sequence generated by the modified inexact LMM converges to a solution superlinearly and even quadratically for some special parameters, which improves the corresponding results of the classical inexact LMM in Dan et al. (Optim Methods Softw 17:605–626, 2002). Furthermore, the quadratical convergence of the global version of the modified inexact LMM is also established. Finally, preliminary numerical experiments on some medium/large scale underdetermined NLSP show that our proposed algorithm outperforms the classical inexact LMM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahookhosh, M., Aragón, F.J., Fleming, R.M.T., Vuongx, P.T.: Local convergence of Levenberg–Marquardt methods under Hölder metric subregularity. arXiv:1703.07461, March 21 (2017)

  2. Armijo, L.: Minimization of functions having continuous partial derivatives. Pac. J. Math. 16, 1–3 (1966)

    Article  MathSciNet  Google Scholar 

  3. Bao, J.F., Li, C., Shen, W.P., Yao, J.C., Guu, S.M.: Approximate Gauss–Newton methods for solving underdetermined nonlinear least squares problems. Appl. Numer. Math. 111, 92–110 (2017)

    Article  MathSciNet  Google Scholar 

  4. Behling, R., Fischer, A., Herrich, M., Iusem, A., Ye, Y.: A Levenberg–Marquardt method with approximate projections. Comput. Optim. Appl. 59, 5–26 (2014)

    Article  MathSciNet  Google Scholar 

  5. Ben-Israel, A.: A Newton–Raphson method for the solution of systems of equations. J. Math. Anal. Appl. 15, 243–252 (1966)

    Article  MathSciNet  Google Scholar 

  6. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1995)

    MATH  Google Scholar 

  7. Björk, A.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)

    Book  Google Scholar 

  8. Bonnans, J.F., Ioffe, A.: Second-order sufficiency and quadratic growth for nonisolated minima. Math. Oper. Res. 20, 801–817 (1995)

    Article  MathSciNet  Google Scholar 

  9. Burke, J.V., Ferris, M.C.: Weak sharp minima in mathematical programming. SIAM J. Control Optim. 31, 1340–1359 (1993)

    Article  MathSciNet  Google Scholar 

  10. Burke, J.V., Ferris, M.C.: A Gauss-Newton method for convex composite optimization. Math. Program. 71, 179–194 (1995)

    MathSciNet  MATH  Google Scholar 

  11. Courtier, P., Thepaut, J.N., Hollingsworth, A.: A strategy for operational implementation of 4D-Var, using an incremental approach. Quart. J. R. Meteorol. Soc. 120, 1367–1387 (1994)

    Article  Google Scholar 

  12. Dan, H., Yamashita, N., Fukushima, M.: Convergence properties of the inexact Levenberg–Marquardt method under local error bound conditions. Optim. Methods Softw. 17, 605–626 (2002)

    Article  MathSciNet  Google Scholar 

  13. Dembo, R.S., Eisenstat, S.C., Steihaug, T.: Inexact Newton methods. SIAM J. Numer. Anal. 19, 400–408 (1982)

    Article  MathSciNet  Google Scholar 

  14. Dennis, J.E., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Englewood Cliffs (1983)

    MATH  Google Scholar 

  15. Deuflhard, P., Heindl, G.: Affine invariant convergence theorems for Newton’s method and extensions to related methods. SIAM J. Numer. Anal. 16(1), 1–10 (1979)

    Article  MathSciNet  Google Scholar 

  16. Fan, J.Y., Pan, J.Y.: Convergence properties of a self-adaptive Levenberg–Marquardt algorithm under local error bound condition. Comput. Optim. Appl. 34, 47–62 (2006)

    Article  MathSciNet  Google Scholar 

  17. Fan, J.Y., Pan, J.Y.: Inexact Levenberg–Marquardt method for nonlinear equations. Discrete Contin. Dyn. Syst. Ser. B 4, 1223–1232 (2004)

    Article  MathSciNet  Google Scholar 

  18. Fan, J.Y., Pan, J.Y.: On the convergence rate of the inexact Levenberg–Marquardt method. J. Ind. Manag. Optim. 7, 199–210 (2011)

    Article  MathSciNet  Google Scholar 

  19. Fan, J.Y., Yuan, Y.X.: On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing 74, 23–39 (2005)

    Article  MathSciNet  Google Scholar 

  20. Fischer, A., Shukla, P.K.: A Levenberg–Marquardt algorithm for unconstrained multicriteria optimization. Oper. Res. Lett. 36, 643–646 (2008)

    Article  MathSciNet  Google Scholar 

  21. Fischer, A., Shukla, P.K., Wang, M.: On the inexactness level of robust Levenberg–Marquardt methods. Optimization 59, 273–287 (2010)

    Article  MathSciNet  Google Scholar 

  22. Guo, L., Lin, G.H., Ye, J.J.: Solving mathematical programs with equilibrium constraints. J. Optim. Theory Appl. 166, 234–256 (2015)

    Article  MathSciNet  Google Scholar 

  23. Hanson, P.C.: Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion. SIAM, Philadelphia (1998)

    Book  Google Scholar 

  24. Häußer, W.M.: A Kantorovich-type convergence analysis for the Gauss–Newton method. Numer. Math. 48, 119–125 (1986)

    Article  MathSciNet  Google Scholar 

  25. Hoffman, A.J.: On approximate solutions of systems of linear inequalities. J. Res. Natl. Bur. Stand. 49, 263–265 (1952)

    Article  MathSciNet  Google Scholar 

  26. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, New York (1985)

    Book  Google Scholar 

  27. Hu, Y.H., Li, C., Meng, K.W., Qin, J., Yang, X.Q.: Group sparse optimization via \(\ell _{p, q}\) regularization. J. Mach. Learn. Res. 18, 1–52 (2017)

    MathSciNet  Google Scholar 

  28. Hu, Y.H., Li, C., Yang, X.Q.: On convergence rates of linearized proximal algorithms for convex composite optimization with applications. SIAM J. Optim. 26, 1207–1235 (2016)

    Article  MathSciNet  Google Scholar 

  29. Hu, Y.H., Yang, X.Q., Sim, C.-K.: Inexact subgradient methods for quasi-convex optimization problems. Eur. J. Oper. Res. 240, 315–327 (2015)

    Article  MathSciNet  Google Scholar 

  30. Kiwiel, K.C., Murty, K.: Convergence of the steepest descent method for minimization quasiconvex functions. J. Optim. Theory Appl. 89(1), 221–226 (1996)

    Article  MathSciNet  Google Scholar 

  31. Lawless, A.S., Gratton, S., Nichols, N.K.: An investigation of incremental 4D-Var using non-tangent linear models. Quart. J. R. Meteorol. Soc. 131, 459–476 (2005)

    Article  Google Scholar 

  32. Levenberg, K.: A method for the solution of certain nonlinear problems in least squares. Quart. Appl. Math. 2, 164–166 (1944)

    Article  MathSciNet  Google Scholar 

  33. Li, G.: On the asymptotically well behaved functions and global error bound for convex polynomials. SIAM J. Optim. 20(4), 1923–1943 (2010)

    Article  MathSciNet  Google Scholar 

  34. Li, G.: Global error bounds for piecewise convex polynomials. Math. Program. 137, 37–64 (2013)

    Article  MathSciNet  Google Scholar 

  35. Li, C., Hu, N.C., Wang, J.H.: Convergence behavior of Gauss–Newton’s method and extensions of the Smale point estimate theory. J. Complex. 26, 268–295 (2010)

    Article  MathSciNet  Google Scholar 

  36. Li, C., Wang, X.: On convergence of the Gauss–Newton method for convex composite optimization. Math. Program. 91, 349–356 (2002)

    Article  MathSciNet  Google Scholar 

  37. Luo, X.D., Luo, Z.Q.: Extension of Hoffman’s error bound to polynomial systems. SIAM J. Optim. 4, 383–392 (1994)

    Article  MathSciNet  Google Scholar 

  38. Luo, Z.Q., Pang, J.S.: Error bounds for analytic systems and their applications. Math. Program. 67, 1–28 (1994)

    Article  MathSciNet  Google Scholar 

  39. Macconi, M., Morini, B., Porcelli, M.: A Gauss-Newton method for solving bound constrained underdetermined nonlinear systems. Optim. Methods Softw. 24, 219–235 (2009)

    Article  MathSciNet  Google Scholar 

  40. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear inequalities. SIAM J. Appl. Math. 11, 431–441 (1963)

    Article  Google Scholar 

  41. Moré, J.J.: The Levenberg–Marquardt algorithm: implementation and theory. In: Watson, G.A. (ed.) Lecture Notes in Mathematics 630: Numerical Analysis, pp. 105–116. Springer, New York (1978)

    Google Scholar 

  42. Moré, J.J., Garbow, B.S., Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. 7, 17–41 (1981)

    Article  MathSciNet  Google Scholar 

  43. Ngai, H.V., Théra, M.: Error bounds for convex differentiable inequality systems in Banach spaces. Math. Program. 104, 465–482 (2005)

    Article  MathSciNet  Google Scholar 

  44. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)

    Book  Google Scholar 

  45. Pang, J.S.: Error bounds in mathematical programming. Math. Program. 79, 299–332 (1997)

    MathSciNet  MATH  Google Scholar 

  46. Porcelli, M.: On the convergence of an inexact Gauss–Newton trust-region method for nonlinear least-squares problems with simple bounds. Optim. Lett. 7, 447–465 (2013)

    Article  MathSciNet  Google Scholar 

  47. Stewart, G.W., Sun, J.G.: Matrix Perturbation Theory. Academic Press, New York (1990)

    MATH  Google Scholar 

  48. Studniarski, M., Ward, D.E.: Weak sharp minima: characterizations and sufficient conditions. SIAM J. Control Optim. 38, 219–236 (1999)

    Article  MathSciNet  Google Scholar 

  49. Sun, W., Yuan, Y.X.: Optimization Theory and Methods: Nonlinear Programming. Springer, New York (2006)

    MATH  Google Scholar 

  50. Wang, J.H., Hu, Y.H., Li, C., Yao, J.-C.: Linear convergence of CQ algorithms and applications in gene regulatory network inference. Inverse Probl. 33, 055017 (2017)

    Article  MathSciNet  Google Scholar 

  51. Yamashita, N., Fukushima, M.: On the rate of convergence of the Levenberg–Marquardt method. Comput. Suppl. 15, 239–249 (2001)

    Article  MathSciNet  Google Scholar 

  52. Yang, W.H.: Error bounds for convex polynomials. SIAM J. Optim. 19, 1633–1647 (2008)

    Article  MathSciNet  Google Scholar 

  53. Ypma, T.J.: Historical development of the Newton–Raphson method. SIAM Rev. 37, 531–551 (1995)

    Article  MathSciNet  Google Scholar 

  54. Zhu, X., Lin, G.H.: Improved convergence results for a modified Levenberg–Marquardt method for nonlinear equations and applications in MPCC. Optim. Methods Softw. 31, 791–804 (2016)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thank sincerely the handling editor and two anonymous referees for their valuable comments and helpful suggestions, which allowed us to improve sharply the original version of this paper.

Author information

Authors and Affiliations

  1. School of Mathematics, Physics and Information Science, Zhejiang Ocean University, Zhoushan, 316022, Zhejiang, People’s Republic of China

    Jifeng Bao

  2. Key Laboratory of Oceanographic Big Data Mining and Application of Zhejiang Province, Zhoushan, 316022, Zhejiang, People’s Republic of China

    Jifeng Bao

  3. Department of Mathematics and Statistics, The Hang Seng University of Hong Kong, Hong Kong, People’s Republic of China

    Carisa Kwok Wai Yu

  4. Department of Mathematics, Zhejiang University of Technology, Hangzhou, 310032, People’s Republic of China

    Jinhua Wang

  5. Shenzhen Key Laboratory of Advanced Machine Learning and Applications, College of Mathematics and Statistics, Shenzhen University, Shenzhen, 518060, People’s Republic of China

    Yaohua Hu

  6. Research Center for Interneural Computing, China Medical University Hospital, China Medical University, Taichung, Taiwan

    Jen-Chih Yao

  7. Department of Mathematics, Hangzhou Normal University, Hangzhou, 311121, People’s Republic of China

    Jinhua Wang

Authors
  1. Jifeng Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carisa Kwok Wai Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinhua Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yaohua Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jen-Chih Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinhua Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Jifeng Bao: This author’s work was supported in part by the National Natural Science Foundation of China (11771398), Zhejiang Provincial Natural Science Foundation of China (LY18A010024) and the Scientific Research Foundation of Zhejiang Ocean University (21065013613). Carisa Kwok Wai Yu: This author’s work was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China (Grant UGC/FDS14/P02/15 and UGC/FDS14/P02/17). Jinhua Wang: This author’s work was supported in part by the National Natural Science Foundation of China (11771397) and by Zhejiang Provincial Natural Science Foundation of China (LY17A010021, LY17A010006). Yaohua Hu: This author’s work was supported in part by the National Natural Science Foundation of China (11601343, 11871347), Natural Science Foundation of Guangdong (2016A030310038), Natural Science Foundation of Shenzhen (JCYJ20170817100950436) and Interdisciplinary Innovation Team of Shenzhen University. Jen-Chih Yao: This author’s work was supported in part by the National Science Council of Taiwan (MOST 105-2115-M-039-002-MY3).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bao, J., Yu, C.K.W., Wang, J. et al. Modified inexact Levenberg–Marquardt methods for solving nonlinear least squares problems. Comput Optim Appl 74, 547–582 (2019). https://doi.org/10.1007/s10589-019-00111-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-019-00111-y

Keywords

Mathematics Subject Classification

Search

Navigation