Skip to main content
SpringerLink
Log in

Convergence analysis of a proximal Gauss-Newton method

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

An extension of the Gauss-Newton algorithm is proposed to find local minimizers of penalized nonlinear least squares problems, under generalized Lipschitz assumptions. Convergence results of local type are obtained, as well as an estimate of the radius of the convergence ball. Some applications for solving constrained nonlinear equations are discussed and the numerical performance of the method is assessed on some significant test problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Argyros, I.: On the semilocal convergence of the Gauss-Newton method. Adv. Nonlinear Var. Inequal. 8(2), 93–99 (2005)

    MathSciNet  MATH  Google Scholar 

  2. Argyros, I., Hilout, S.: On the local convergence of the Gauss-Newton method. Punjab Univ. J. Math. 41, 23–33 (2009)

    MathSciNet  MATH  Google Scholar 

  3. Argyros, I., Hilout, S.: On the Gauss-Newton method. J. Appl. Math. Comput. 1–14 (2010)

  4. Argyros, I., Hilout, S.: Extending the applicability of the Gauss-Newton method under average Lipschitz-type conditions. Numer. Algorithms 58(1), 23–52 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bachmayr, M., Burger, M.: Iterative total variation schemes for nonlinear inverse problems. Inverse Probl. 25(10), 105004 (2009)

    Article  MathSciNet  Google Scholar 

  6. Bakushinsky, A.B., Kokurin, M.Y.: Iterative Methods for Approximate Solution of Inverse Problems. Mathematics and Its Applications (New York), vol. 577. Springer, Dordrecht (2004)

    MATH  Google Scholar 

  7. Bellavia, S., Macconi, M., Morini, B.: STRSCNE: a scaled trust-region solver for constrained nonlinear equations. Comput. Optim. Appl. 28(1), 31–50 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  8. Ben-Israel, A.: A modified Newton-Raphson method for the solution of systems of equations. Isr. J. Math. 3, 94–98 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  9. Blaschke, B., Neubauer, A., Scherzer, O.: On convergence rates for the iteratively regularized Gauss-Newton method. IMA J. Numer. Anal. 17(3), 421–436 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  10. Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization. Advanced Books in Mathematics. Canadian Mathematical Society, Ottawa (2000)

    MATH  Google Scholar 

  11. Burke, J.V., Ferris, M.C.: A Gauss-Newton method for convex composite optimization. Math. Program., Ser. A 71(2), 179–194 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  12. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005) (electronic)

    Article  MathSciNet  MATH  Google Scholar 

  13. Dennis, J.E. Jr., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Classics in Applied Mathematics, vol. 16. SIAM, Philadelphia (1996). Corrected reprint of the 1983 original

    Book  MATH  Google Scholar 

  14. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Mathematics and Its Applications, vol. 375 (1996)

    Book  MATH  Google Scholar 

  15. Ferreira, O.P., Svaiter, B.F.: Kantorovich’s majorants principle for Newton’s method. Comput. Optim. Appl. 42(2), 213–229 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  16. Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, West Sussex (2000) pp. xiv+436

    Google Scholar 

  17. Floudas, C.A., Pardalos, P.M.: A Collection of Test Problems for Constrained Global Optimization Algorithms. Lecture Notes in Computer Science, vol. 455. Springer, Berlin (1990)

    Book  MATH  Google Scholar 

  18. Groetsch, C.W.: Generalized Inverses of Linear Operators: Representation and Approximation. Monographs and Textbooks in Pure and Applied Mathematics, vol. 37. Marcel Dekker, New York (1977)

    MATH  Google Scholar 

  19. Häussler, W.M.: A Kantorovich-type convergence analysis for the Gauss-Newton-method. Numer. Math. 48(1), 119–125 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  20. Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms. II. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 306. Springer, Berlin (1993)

    MATH  Google Scholar 

  21. Jin, Q.: A convergence analysis of the iteratively regularized Gauss-Newton method under the Lipschitz condition. Inverse Probl. 24(4), 045002 (2008)

    Article  Google Scholar 

  22. Kaltenbacher, B., Hofmann, B.: Convergence rates for the iteratively regularized Gauss-Newton method in Banach spaces. Inverse Probl. 26(3), 035007 (2010)

    Article  MathSciNet  Google Scholar 

  23. Kanzow, C.: An active set-type Newton method for constrained nonlinear systems. Appl. Optim. 50, 179–200 (2001)

    MathSciNet  Google Scholar 

  24. Kowalik, J., Osborne, M.R.: Methods for Unconstrained Optimization Problems. Elsevier, New York (1969)

    Google Scholar 

  25. Kozakevich, D.N., Martínez, J.M., Santos, S.A.: Solving nonlinear systems of equations with simple constraints. Comput. Appl. Math. 16(3), 215–235 (1997)

    MATH  Google Scholar 

  26. Langer, S.: Investigation of preconditioning techniques for the iteratively regularized Gauss-Newton method for exponentially ill-posed problems. SIAM J. Sci. Comput. 32(5), 2543–2559 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  27. Lewis, A., Wright, S.J.: A proximal method for composite optimization (2008). 0812.0423v1 [math.OC]

  28. Li, C., Hu, N., Wang, J.: Convergence behavior of Gauss-Newton’s method and extensions of the Smale point estimate theory. J. Complex. 26(3), 268–295 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  29. Li, C., Ng, K.F.: Majorizing functions and convergence of the Gauss-Newton method for convex composite optimization. SIAM J. Optim. 18(2), 613–642 (2007) (electronic)

    Article  MathSciNet  MATH  Google Scholar 

  30. Li, C., Wang, X.: On convergence of the Gauss-Newton method for convex composite optimization. Math. Program., Ser. A 91(2), 349–356 (2002)

    Article  MATH  Google Scholar 

  31. Li, C., Wang, X.: Convergence of Newton’s method and uniqueness of the solution of equations in Banach spaces. II. Acta Math. Sin. Engl. Ser. 19(2), 405–412 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  32. Li, C., Zhang, W., Jin, X.: Convergence and uniqueness properties of Gauss-Newton’s method. Comput. Math. Appl. 47(6–7), 1057–1067 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  33. Moreau, J.J.: Fonctions convexes duales et points proximaux dans un espace hilbertien. C. R. Acad. Sci. Paris 255, 2897–2899 (1962)

    MathSciNet  MATH  Google Scholar 

  34. Moreau, J.J.: Propriétés des applications “prox”. C. R. Acad. Sci. Paris 256, 1069–1071 (1963)

    MathSciNet  MATH  Google Scholar 

  35. Moreau, J.J.: Proximité et dualité dans un espace hilbertien. Bull. Soc. Math. Fr. 93, 273–299 (1965)

    MathSciNet  MATH  Google Scholar 

  36. Mosci, S., Rosasco, L., Santoro, M., Verri, A., Villa, S.: Solving structured sparsity regularization with proximal methods. In: Balcázar, J. et al. (eds.) Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Science, vol. 6322, pp. 418–433. Springer, Berlin (2010)

    Chapter  Google Scholar 

  37. Osborne, M.R.: Some aspects of non-linear least squares calculations. In: Lootsma, F. (ed.) Numerical Methods for Non-Linear Optimization, Conf. Univ. Dundee, Dundee, 1971, pp. 171–189. Academic Press, London (1972)

    Google Scholar 

  38. Polyak, B.T.: Introduction to Optimization. Translations Series in Mathematics and Engineering. Optimization Software Inc. Publications Division, New York (1987). Translated from the Russian, with a foreword by Dimitri P. Bertsekas

    Google Scholar 

  39. Ramlau, R., Teschke, G.: Tikhonov replacement functionals for iteratively solving nonlinear operator equations. Inverse Probl. 21(5), 1571–1592 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  40. Ramlau, R., Teschke, G.: A Tikhonov-based projection iteration for nonlinear ill-posed problems with sparsity constraints. Numer. Math. 104(2), 177–203 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  41. Scherzer, O., Grasmair, M., Grossauer, H., Haltmeier, M., Lenzen, F.: Variational Methods in Imaging. Applied Mathematical Sciences, vol. 167. Springer, New York (2009)

    MATH  Google Scholar 

  42. Shacham, M., Brauner, N., Cutlib, M.: A web-based library for testing performance of numerical software for solving nonlinear algebraic equations. Comput. Chem. Eng. 26, 547–554 (2002)

    Article  Google Scholar 

  43. Stewart, G.W.: On the continuity of the generalized inverse. SIAM J. Appl. Math. 17, 33–45 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  44. Ulbrich, M.: Nonmonotone trust-region methods for bound-constrained semismooth equations with applications to nonlinear mixed complementarity problems. SIAM J. Optim. 11(4), 889–917 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  45. Wang, X.: Convergence of Newton’s method and inverse function theorem in Banach space. Math. Comput. 68(225), 169–186 (1999)

    Article  MATH  Google Scholar 

  46. Wang, X.: Convergence of Newton’s method and uniqueness of the solution of equations in Banach space. IMA J. Numer. Anal. 20(1), 123–134 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  47. Wedin, P.: Perturbation theory for pseudo-inverses. BIT Numer. Math. 13, 217–232 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  48. Womersley, R.S.: Local properties of algorithms for minimizing nonsmooth composite functions. Math. Program., Ser. A 32(1), 69–89 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  49. Womersley, R.S., Fletcher, R.: An algorithm for composite nonsmooth optimization problems. J. Optim. Theory Appl. 48(3), 493–523 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  50. Xu, C.: Nonlinear least squares problems. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, pp. 2626–2630. Springer, New York (2009)

    Chapter  Google Scholar 

  51. Xu, C., Li, C.: Convergence criterion of Newton’s method for singular systems with constant rank derivatives. J. Math. Anal. Appl. 345(2), 689–701 (2008)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We are grateful to Alessandro Verri for his constant support and advice. We further thank Curzio Basso for carefully reading our paper.

Author information

Authors and Affiliations

  1. DISI, Università di Genova, Via Dodecaneso 35, 16146, Genova, Italy

    Saverio Salzo & Silvia Villa

  2. DIMA, Università di Genova, Via Dodecaneso 35, 16146, Genova, Italy

    Silvia Villa

Authors
  1. Saverio Salzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Silvia Villa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saverio Salzo.

Appendix

Appendix

In this section we provide the expressions of the test functions for reader’s convenience. All the objective functions are sum of squares

$$\frac{1}{2}\bigl\|{F(x)}\bigr\|^2, \quad F: \mathbb{R}^n\to\mathbb{R}^m$$

where ∥⋅∥ denotes the Euclidean norm in ℝm. We denote by F k :ℝn→ℝ the k-component of the function F for k∈[1,m].

  • Rosenbrock (n=m=2). See [16], p. 7, for instance.

    $$F_1(x) = 10 \bigl(x_2 - x_1^2\bigr), \qquad F_2(x) = 1 - x_1$$

  • Kowalik (n=4,m=11). It is the Enzyme problem given in [24], p. 104.

    $$F_k(x) = V_k - \dfrac{x_1 (y_k^2 + x_2 y_k)}{y_k^2 + x_3 y_k + x_4}$$

    where the vectors of the V k ’s and y k ’s are given in Table 2.

    Table 2 Data for the enzyme problem Kowalik
    Full size table

  • Osborne1 (n=5,m=33). It is the Exponential Fitting problem given in [37], p. 185.

    $$F_k(x) = y_k - \bigl(x_1 + x_2\exp(-x_4 t_k) + x_3 \exp(x_5t_k) \bigr)$$

    where the vectors of the t k ’s and y k ’s are given in Table 3.

    Table 3 Data for the exponential fitting problem Osborne1
    Full size table

  • Osborne2 (n=11,m=65). It is the Fitting Gaussian plus an Exponential Background problem given in [37], p. 186.

    where the t k ’s and y k ’s are given in Table 4.

    Table 4 Data for the problem Osborne2
    Full size table

  • Twoeq6 (n=m=2). It is the Conversion in a chemical reactor problem listed in the NLE library http://www.polymath-software.com/library/problemlist.shtml

    $$\begin{aligned}&F_1(x) = \frac{x_1}{1 - x_1} - 5 \log\biggl(\frac{0.4(1-x_1)}{x_2}\biggr) + 4.45977,\\&F_2(x) = x_2 - (0.4 - 0.5x_1).\end{aligned}$$

  • Teneq1b (n=m=10). It is the Chemical Equilibrium Problem, R=40 problem listed in the NLE library http://www.polymath-software.com/library/problemlist.shtml

    where the constant R=40.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Salzo, S., Villa, S. Convergence analysis of a proximal Gauss-Newton method. Comput Optim Appl 53, 557–589 (2012). https://doi.org/10.1007/s10589-012-9476-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-012-9476-9

Keywords

Search

Navigation