Numerische Mathematik

, Volume 140, Issue 3, pp 791–825 | Cite as

A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients

  • Stefania Bellavia
  • Serge Gratton
  • Elisa Riccietti


In this paper we consider large scale nonlinear least-squares problems for which function and gradient are evaluated with dynamic accuracy and propose a Levenberg–Marquardt method for solving such problems. More precisely, we consider the case in which the exact function to optimize is not available or its evaluation is computationally demanding, but approximations of it are available at any prescribed accuracy level. The proposed method relies on a control of the accuracy level, and imposes an improvement of function approximations when the accuracy is detected to be too low to proceed with the optimization process. We prove global and local convergence and complexity of our procedure and show encouraging numerical results on test problems arising in data assimilation and machine learning.

Mathematics Subject Classification

65K05 90C30 90C26 90C06 



We thank the authors of [16] for providing us the Matlab code for the data assimilation test problem.


  1. 1.
    Anderson, T.W., Darling, D.A.: A test of goodness of fit. J. Am. Stat. Assoc. 49(268), 765–769 (1954)CrossRefGoogle Scholar
  2. 2.
    Bandeira, A.S., Scheinberg, K., Vicente, L.N.: Convergence of trust-region methods based on probabilistic models. SIAM J. Opt. 24(3), 1238–1264 (2014)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bellavia, S., Morini, B., Riccietti, E.: On an adaptive regularization for ill-posed nonlinear systems and its trust-region implementation. Comput. Opt. Appl. 64(1), 1–30 (2016)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bellavia, S., Riccietti E.: On an elliptical trust-region procedure for Ill-posed nonlinear least-squares problems. J. Optim. Theor. Appl. (2018). MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bergou, E., Gratton, S., Vicente, L.: Levenberg-marquardt methods based on probabilistic gradient models and inexact subproblem solution, with application to data assimilation. SIAM/ASA J. Uncertain. Quantif. 4(1), 924–951 (2016)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Blanchet, J., Cartis, C., Menickelly, M., Scheinberg, K.: Convergence rate analysis of a stochastic trust region method for nonconvex optimization (2016) arXiv preprint arXiv:1609.07428
  7. 7.
    Bollapragada, R., Byrd, R., Nocedal, J.: Exact and inexact subsampled Newton methods for optimization (2016) arXiv preprint arXiv:1609.08502
  8. 8.
    Bottou, L., Curtis, F.E., Nocedal, J.: Optimization methods for large-scale machine learning (2016) arXiv preprint arXiv:1606.04838
  9. 9.
    Causality Workbench Team. A Marketing Dataset. (2008). Accessed 23 Jan 2017
  10. 10.
    Chen, R., Menickelly, M., Scheinberg, K.: Stochastic optimization using a trust-region method and random models. Math. Program. 144, 1–41 (2015)zbMATHGoogle Scholar
  11. 11.
    Conn, A.R., Gould, N.I., Toint, P.L.: Trust Region Methods, vol. 1. SIAM, Philadelphia (2000)CrossRefGoogle Scholar
  12. 12.
    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)CrossRefGoogle Scholar
  13. 13.
    Courtier, P., Thépaut, J.N., Hollingsworth, A.: A strategy for operational implementation of 4d-var, using an incremental approach. Quart. J. R. Meteorol. Soc. 120(519), 1367–1387 (1994)CrossRefGoogle Scholar
  14. 14.
    Friedlander, M.P., Schmidt, M.: Hybrid deterministic-stochastic methods for data fitting. SIAM J. Sci. Comput. 34(3), A1380–A1405 (2012)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Gratton, S., Gürol, S., Toint, P.: Preconditioning and globalizing conjugate gradients in dual space for quadratically penalized nonlinear-least squares problems. Comput. Opt. Appl. 54(1), 1–25 (2013)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Gratton, S., Rincon-Camacho, M., Simon, E., Toint, P.L.: Observation thinning in data assimilation computations. EURO J. Comput. Opt. 3(1), 31–51 (2015)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Hanke, M.: A regularizing Levenberg–Marquardt scheme, with applications to inverse groundwater filtration problems. Inverse Probl. 13(1), 79 (1997)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Kaltenbacher, B., Neubauer, A., Scherzer, O.: Iterative Regularization Methods for Nonlinear Ill-Posed Problems, vol. 6. Walter de Gruyter, Berlin (2008)CrossRefGoogle Scholar
  19. 19.
    Kelley, C.T.: Iterative Methods for Optimization: Matlab Codes. Accessed 12 Mar 2017
  20. 20.
    Krejić, N., Jerinkić, N.K.: Nonmonotone line search methods with variable sample size. Numer. Algorithms 68(4), 711–739 (2015)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Krejić, N., Martínez, J.: Inexact restoration approach for minimization with inexact evaluation of the objective function. Math. Comput. 85(300), 1775–1791 (2016)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course, vol. 87. Springer, Berlin (2013)zbMATHGoogle Scholar
  23. 23.
    Roosta-Khorasani, F., Mahoney, M.W.: Sub-sampled Newton methods 1: globally convergent algorithms (2016) arXiv preprint arXiv:1601.04737
  24. 24.
    Saunders, M.: Systems Optimization Laboratory. Accessed 15 Nov 2016
  25. 25.
    Stephens, M.A.: Edf statistics for goodness of fit and some comparisons. J. Am. Stat. Assoc. 69(347), 730–737 (1974)CrossRefGoogle Scholar
  26. 26.
    Trémolet, Y.: Model-error estimation in 4d-var. Quart. J. R. Meteorol. Soc. 133(626), 1267–1280 (2007)CrossRefGoogle Scholar
  27. 27.
    Weaver, A., Vialard, J., Anderson, D.: Three-and four-dimensional variational assimilation with a general circulation model of the tropical pacific ocean. Part I: formulation, internal diagnostics, and consistency checks. Mon. Weather Rev. 131(7), 1360–1378 (2003)CrossRefGoogle Scholar
  28. 28.
    Wright, S., Nocedal, J.: Numerical optimization. Science 35, 67–68 (1999)zbMATHGoogle Scholar
  29. 29.
    Zhao, R., Fan, J.: Global complexity bound of the Levenberg–Marquardt method. Opt. Methods Softw. 31(4), 805–814 (2016)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Stefania Bellavia
    • 1
  • Serge Gratton
    • 2
  • Elisa Riccietti
    • 2
  1. 1.Dipartimento di Ingegneria IndustrialeUniversità di FirenzeFlorenceItaly
  2. 2.University of Toulouse, IRITToulouse Cedex 7France

Personalised recommendations