Abstract
In this paper, we propose a stochastic Levenberg–Marquardt algorithm based on trust region for stochastic nonlinear least squares problems, where the stochastic Jacobians and gradients are used instead of the exact Jacobians and gradients. We show that the estimates and models of the objective function are probabilistically accurate if the number of samples at each iteration is chosen appropriately. Further, we prove that at least one accumulation point of the sequence generated by the proposed algorithm is a stationary point of the objective function with probability one.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bottou, L., Curtis, F.E., Nocedal, J.: Optimization methods for large-scale machine learning. SIAM Rev. 60(2), 223–311 (2018)
Johnson, R. and Ahang, T.:Accelerating stochastic gradient descent using predictive variance reduction, In: Advances in Neural Information Processing Systems 26, pp. 315–323. Curran Associates, Inc. (2013)
Defazio, A., Bach, F., and Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In: Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pp. 1646–1654. Curran Associates, Inc. (2014)
Byrd, R.H., Hansen, S.L., Nocedal, J., Singer, Y.: A stochastic quasi-Newton method for large-scale optimization. SIAM J. Optim. 26(2), 1008–1031 (2016)
Mokhtari, A., Ribeiro, A.: Res: Regularized stochastic BFGS algorithm. IEEE Trans. Signal Process. 62(23), 6089–6104 (2014)
Da Silva, I.N.; Spatti, D.H.; Flauzino, R.A.; Liboni, L.H.B.; dos Reis Alves, S.F. Artificial Neural Networks: A Practical Course. Springer International Publishing, Basel (2016)
Yu, H., Wilamowski, B.M.: Levenberg-Marquardt training. In: Intelligent systems, pp. 1–12. CRC Press, Boca Raton (2018)
Bellavia, S., Riccietti, E.: On an elliptical trust-region procedure for ill-posed nonlinear leastsquares problems. J. Optim. Theory Appl. 178(3), 824–859 (2018)
Bidabadi, N.: Using a spectral scaling structured BFGS method for constrained nonlinear least squares. Optim. Methods Softw. 34(4), 693–706 (2019)
Gould, N.I.M., Rees, T., Scott, J.A.: Convergence and evaluation-complexity analysis of a regularized tensor-Newton method for solving nonlinear least-squares problems. Comput. Optim. Appl. 73(1), 1–35 (2019)
Yuan, Y.X.: Subspace methods for large scale nonlinear equations and nonlinear least squares. Optim. Eng. 10(2), 207–218 (2009)
Zhang, H.C., Conn, A.R., Scheinberg, K.: A derivative-free algorithm for least squares minimization. SIAM J. Optim. 20(6), 3555–3576 (2010)
Cartis, C., Roberts, L.: A derivative-free Gauss-Newton method. Math. Program. Comput. 11(4), 631–674 (2019)
Cartis, C., Fiala, J., Marteau, B., Rober, L.: Improving the flexibility and robustness of modelbased derivative-free optimization solvers. ACM Trans. Math. Software 45(3), 41 (2019)
Bergou, E., Gratton, S., Vicente, L.N.: Levenberg-Marquardt methods based on probabilistic gradient models and inexact subproblem solution, with application to data assimilation. SIAM/ASA J. Uncertain. Quantif. 4(1), 924–951 (2016)
Zhao, R.X., Fan, J.Y.: Levenberg-Marquardt method based on probabilistic Jacobian models for nonlinear equations. Comput. Optim. Appl. 83, 381–401 (2022)
Conn, A. R., Gould, N. I. M., Toint, Ph. L.: Trust-Region Methods. SIAM, Philadelphia (2000)
Powell, M.J.D.: Convergence properties of a class of minimization algorithms. In: Mangasarian, O.L., Meyer, R.R., Robinson, S.M. (eds.) Nonlinear Programming 2, pp. 1–27. Academic Press, New York (1975)
Çinlar, E., and ðCınlar, E.: Probability and Stochastics. Springer, vol. 261 (2011)
R. Durrett. Probability: theory and examples. Cambridge University Press, Cambridge (2019)
Laha, R.G., Rohatgi, V.K.: Probability Theory. Courier Dover Publications, New York (2020)
Hong, Y., Bergou, H., Doucet, N., Zhang, H., Cranney, J., Ltaief, H., Gratadour, D., Rigaut, F., and Keyes, D. E.: Stochastic Levenberg-Marquardt for solving optimization problems on hardware accelerators. https://repository.kaust.edu.sa/items/60c86d63-8d42-470a-b875-dd88df1980ff (2021)
Bergou, E., Diouane, Y., Kungurtsev V., and Royer, C. W.: A stochastic Levenberg-Marquardt method using random models with application to data assimilation. arXiv:1807.02176v1 (2018)
Moré, J.J., Garbow, B.S., Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. (TOMS) 7(1), 17–41 (1981)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(27), 1–27 (2011)
Author information
Authors and Affiliations
Contributions
J. -Y. Fan made the research plan and W.-Y. Shao performed the research.
Corresponding author
Ethics declarations
Conflict of interest
The authors declared that they have no conflicts of interest to this work.
Additional information
The authors are supported by Shanghai Municipal Science and Technology Key Project (No. 22JC1401500), the National Natural Science Foundation of China (Nos. 1971309 and 12371307), and the Fundamental Research Funds for the Central Universities.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shao, WY., Fan, JY. Global Convergence of a Stochastic Levenberg–Marquardt Algorithm Based on Trust Region. J. Oper. Res. Soc. China (2024). https://doi.org/10.1007/s40305-023-00529-6
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s40305-023-00529-6