Abstract
A simple spectral correction for the Gauss-Newton model applied to nonlinear least squares problems is presented. Such a correction consists in adding a sign-free multiple of the identity to the Hessian of the Gauss-Newton model, being the multiple based on spectral approximations for the Hessians of the residual functions. A detailed local convergence analysis is provided for the resulting method applied to the class of quadratic residual problems. Under mild assumptions, the proposed method is proved to be convergent for problems for which the convergence of the Gauss-Newton method might not be ensured. Moreover, the rate of linear convergence is proved to be better than the Gauss-Newton’s one for a class of non-zero residue problems. These theoretical results are illustrated by numerical examples with quadratic and non-quadratic residual problems.
Similar content being viewed by others
References
Bartholomew-Biggs, M.C.: The estimation of the Hessian matrix in nonlinear least squares problems with non-zero residuals. Math. Program. 12, 67–80 (1977)
Barzilai, J., Borwein, J.: Two-point step size gradient methods. IMA J. Numer. Anal. 8(1), 141–148 (1988). doi:10.1093/imanum/8.1.141
Behling, R., Fischer, A.: A unified local convergence analysis of inexact constrained Levenberg-Marquardt methods. Optim. Lett 6(5), 927–940 (2012). doi:10.1007/s11590-011-0321-3
Bellavia, S., Cartis, C., Gould, N., Morini, B., Toint, Ph.: L.: Convergence of a regularized Euclidean residual algorithm for nonlinear least-squares. SIAM J. Numer. Anal. 48, 1–29 (2010)
Bellavia, S., Morini, B.: Strong local convergence properties of adaptive regularized methods for nonlinear least squares. IMA J. Numer. Anal. 35(2), 947–968 (2015). doi:10.1093/imanum/dru021 [http://imajna.oxfordjournals.org/content/early/2014/05/14/imanum.dru021.abstract]
Birgin, E.G., Martínez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim., 1196–1211 (2000). doi:10.1.1.145.9706
Brown, K.M., Dennis Jr., J.E.: Derivative free analogues of the Levenberg-Marquardt and Gauss algorithms for nonlinear least squares approximation. Numer. Math. 18, 289–297 (1971/72)
Chen, P.: Hessian matrix vs. Gauss-newton Hessian matrix. SIAM J. Numer. Anal 49(4), 1417–1435 (2011). doi:10.1137/100799988
Dennis Jr., J.E.: Non-linear least squares and equations, pp 269–312. Academic Press, London (1977)
Dennis Jr., J.E., Schnabel, R.B.: Numerical methods for unconstrained optimization and nonlinear equations, Classics in Applied Mathematics, vol. 16. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA. doi:10.1137/1.9781611971200, Corrected reprint of the 1983 original (1996)
Dennis Jr., J.E., Song-Bai, S., Vu, P.A.: A memoryless augmented Gauss-Newton method for nonlinear least-squares problems. J. Comput. Math 6(4), 355–374 (1988)
Ferreira, O.P., Gonçalves, M.L.N., Oliveira, P.R.: Local convergence analysis of the Gauss-Newton method under a majorant condition. J. Complex. 27 (1), 111–125 (2011). doi:10.1016/j.jco.2010.09.001
Ferreira, O.P., Gonçalves, M.L.N., Oliveira, P.R.: Convergence of the Gauss-Newton method for convex composite optimization under a majorant condition. SIAM J. Optim. 23(3), 1757–1783 (2013). doi:10.1137/110841606
Gill, P.E., Murray, W.: Algorithms for the solution of the nonlinear least-squares problem. SIAM J. Numer. Anal. 15(5), 977–992 (1978)
Gould, N. I. M.: Ph.l.toint: FILTRANE, a Fortran 95 filter-trust-region package for solving nonlinear least-squares and nonlinear feasibility problems. ACM Trans. Math. Softw. 33(1), 23 (2007)
Gratton, S., Lawless, A.S., Nichols, N.K.: Approximate Gauss-Newton methods for nonlinear least squares problems. SIAM J. Optim 18(1), 106–132 (2007). doi:10.1137/050624935
Kanzow, C., Yamashita, N., Fukushima, M.: Levenberg–marquardt methods with strong local convergence properties for solving nonlinear equations with convex constraints. J. Comput. Appl. Math. 172(2), 375–397 (2004). doi:10.1016/j.cam.2004.02.013. http://www.sciencedirect.com/science/article/pii/ S0377042704001256
Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2, 164–168 (1944)
Marquardt, D.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11(2), 431–441 (1963). doi:10.1137/0111030
McKeown, J.: Specialised versus general purpose algorithms for minimising functions that are sums of squared terms. Math. Program. 9, 57–68 (1975)
Meyer, C.D.: Matrix analysis and applied linear algebra. SIAM philadelphia (2000)
Morrison, D.D.: Methods for nonlinear least squares problems and convergence proofs. In: Lorell, J., Yagi, F. (eds.) Proceedings of the seminar on tracking programs and orbit determination, pp 1–9. Jet Propulsion Laboratory, Pasadena, CA, USA (1960)
Nesterov, Y.: Modified Gauss-Newton scheme with worst-case guarantees for global performance. Optim. Methods Softw. 22, 469–483 (2007)
Nocedal, J., Wright, S.J.: Numerical optimization. Springer Series in Operations Research. Springer, New York (1999). doi:10.1007/b98874
Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7(1), 26–33 (1997)
Salzo, S., Villa, S.: Convergence analysis of a proximal Gauss-Newton method. Comput. Optim. Appl 53, 557–589 (2012). doi:10.1007/s10589-012-9476-9
Smith, F.B., Shanno, D.F.: An improved Marquardt procedure for nonlinear regressions. Technometrics 13(1), 63–74 (1971). doi:10.1080/00401706.1971.10488754
Wilkinson, J.H.: The algebraic eigenvalue problem. Oxford university press new york (1988)
Yuan, Y.X.: Recent advances in numerical methods for nonlinear equations and nonlinear least squares. Numer. Algebra Control Optim. 1(1), 15–34 (2011)
Zhang, H.C., Conn, A.R.: On the local convergence of a derivative-free algorithm for least-squares minimization. Comput. Optim. Appl. 51, 481–507 (2012)
Zhang, H.C., Conn, A.R., Scheinberg, K.: A derivative-free algorithm for the least-squares minimization. SIAM J. Optim. 20, 3555–3576 (2010)
Author information
Authors and Affiliations
Corresponding author
Additional information
Partially supported by CNPq (Grant 304032/2010-7), FAPESP (Grants 2013/05475-7 and 2013/07375-0) and PRONEX-Optimization.
Rights and permissions
About this article
Cite this article
Gonçalves, D.S., Santos, S.A. Local analysis of a spectral correction for the Gauss-Newton model applied to quadratic residual problems. Numer Algor 73, 407–431 (2016). https://doi.org/10.1007/s11075-016-0101-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-016-0101-3