Mathematical Programming

, Volume 71, Issue 2, pp 179–194 | Cite as

A Gauss—Newton method for convex composite optimization

  • J. V. Burke
  • M. C. Ferris


An extension of the Gauss—Newton method for nonlinear equations to convex composite optimization is described and analyzed. Local quadratic convergence is established for the minimization ofh ο F under two conditions, namelyh has a set of weak sharp minima,C, and there is a regular point of the inclusionF(x) ∈ C. This result extends a similar convergence result due to Womersley (this journal, 1985) which employs the assumption of a strongly unique solution of the composite functionh ο F. A backtracking line-search is proposed as a globalization strategy. For this algorithm, a global convergence result is established, with a quadratic rate under the regularity assumption.


Gauss—Newton Convex composite optimization Weak sharp minima Quadratic convergence 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    A. Ben-Israel, “A Newton—Raphson method for the solution of systems of equations,”Journal of Mathematical Analysis and its Applications 15 (1966) 243–252.Google Scholar
  2. [2]
    S.C. Billups and M.C. Ferris, “Solutions to affine generalized equations using proximal mappings,” Mathematical Programming Technical Report 94-15 (Madison, WI, 1994).Google Scholar
  3. [3]
    P.T. Boggs, “The convergence of the Ben-Israel iteration for nonlinear least squares problems,”Mathematics of Computation 30 (1976) 512–522.Google Scholar
  4. [4]
    J.M. Borwein, “Stability and regular points of inequality systems,”Journal of Optimization Theory and Applications 48 (1986) 9–52.Google Scholar
  5. [5]
    J.V. Burke, “Algorithms for solving finite dimensional systems of nonlinear equations and inequalities that have both global and quadratic convergence properties,” Report ANL/MCS-TM-54, Mathematics and Computer Science Division, Argonne National Laboratory (Argonne, IL, 1985).Google Scholar
  6. [6]
    J.V. Burke, “Descent methods for composite nondifferentiable optimization problems,”Mathematical Programming 33 (3) (1985) 260–279.Google Scholar
  7. [7]
    J.V. Burke, “An exact penalization viewpoint of constrained optimization,”SIAM Journal on Control and Optimization 29 (1991) 968–998.Google Scholar
  8. [8]
    J.V. Burke and M.C. Ferris, “Weak sharp minima in mathematical programming,”SIAM Journal on Control and Optimization 31 (1993) 1340–1359.Google Scholar
  9. [9]
    J.V. Burke and R.A. Poliquin, “Optimality conditions for non-finite valued convex composite functions,”Mathematical Programming 57 (1) (1992) 103–120.Google Scholar
  10. [10]
    J.V. Burke and P. Tseng, “A unified analysis of Hoffman's bound via Fenchel duality,”SIAM Journal on Optimization, to appear.Google Scholar
  11. [11]
    L. Cromme, “Strong uniqueness. A far reaching criterion for the convergence analysis of iterative procedures,”Numerische Mathematik 29 (1978) 179–193.Google Scholar
  12. [12]
    R. De Leone and O.L. Mangasarian, “Serial and parallel solution of large scale linear programs by augmented Lagrangian successive overrelaxation,” in: A. Kurzhanski et al., eds.,Optimization, Parallel Processing and Applications, Lecture Notes in Economics and Mathematical Systems, Vol. 304 (Springer, Berlin, 1988) pp. 103–124.Google Scholar
  13. [13]
    J.E. Dennis and R.B. Schnabel,Numerical Methods for Unconstrained Optimizations and Nonlinear Equations (Prentice-Hall, Englewood Cliffs, NJ, 1983).Google Scholar
  14. [14]
    M.C. Ferris, “Weak sharp minima and penalty functions in mathematical programming,” Ph.D. Thesis, University of Cambridge (Cambridge, 1988).Google Scholar
  15. [15]
    R. Fletcher, “Generalized inverse methods for the best least squares solution of non-linear equations,”The Computer Journal 10 (1968) 392–399.Google Scholar
  16. [16]
    R. Fletcher, “Second order correction for nondifferentiable optimization,” in: G.A. Watson, ed.,Numerical Analysis, Lecture Notes in Mathematics, Vol. 912 (Springer, Berlin, 1982) pp. 85–114.Google Scholar
  17. [17]
    R. Fletcher,Practical Methods of Optimization (Wiley, New York, 2nd ed., 1987).Google Scholar
  18. [18]
    U.M. Garcia-Palomares and A. Restuccia, “A global quadratic algorithm for solving a system of mixed equalities and inequalities,”Mathematical Programming 21 (3) (1981) 290–300.Google Scholar
  19. [19]
    K. Jittorntrum and M.R. Osborne, “Strong uniqueness and second order convergence in nonlinear discrete approximation,”Numerische Mathematik 34 (1980) 439–455.Google Scholar
  20. [20]
    K. Levenberg, “A method for the solution of certain nonlinear problems in least squares,”Quarterly Applied Mathematics 2 (1944) 164–168.Google Scholar
  21. [21]
    K. Madsen, “Minimization of nonlinear approximation functions,” Ph.D. Thesis, Institute of Numerical Analysis, Technical University of Denmark (Lyngby, 1985).Google Scholar
  22. [22]
    J. Maguregui, “Regular multivalued functions and algorithmic applications,” Ph.D. Thesis, University of Wisconsin (Madison, WI, 1977).Google Scholar
  23. [23]
    J. Maguregui, “A modified Newton algorithm for functions over convex sets,” in: O.L. Mangasarian, R.R. Meyer and S.M. Robinson, eds.,Nonlinear Programming 3 (Academic Press, New York, 1978) pp. 461–473.Google Scholar
  24. [24]
    O.L. Mangasarian, “Least-norm linear programming solution as an unconstrained minimization problem,”Journal of Mathematical Analysis and Applications 92 (1) (1983) 240–251.Google Scholar
  25. [25]
    O.L. Mangasarian, “Normal solutions of linear programs,”Mathematical Programming Study 22 (1984) 206–216.Google Scholar
  26. [26]
    O.L. Mangasarian and S. Fromovitz, “The Fritz John necessary optimality conditions in the presence of equality and inequality constraints,”Journal of Mathematical Analysis and its Applications 17 (1967) 37–47.Google Scholar
  27. [27]
    D.W. Marquardt, “An algorithm for least-squares estimation of nonlinear parameters,”SIAM Journal of Applied Mathematics 11 (1963) 431–441.Google Scholar
  28. [28]
    J.M. Ortega and W.C. Rheinboldt,Iterative Solution of Nonlinear Equations in Several Variables (Academic Press, New York, 1970).Google Scholar
  29. [29]
    M.R. Osborne and R.S. Womersley, “Strong uniqueness in sequential linear programming,”Journal of the Australian Mathematical Society. Series B 31 (1990) 379–384.Google Scholar
  30. [30]
    B.T. Polyak,Introduction to Optimization (Optimization Software, New York, 1987).Google Scholar
  31. [31]
    M.J.D. Powell, “General algorithm for discrete nonlinear approximation calculations,” in: C.K. Chui, L.L. Schumaker and J.D. Ward, eds.,Approximation Theory IV (Academic Press, New York, 1983) pp. 187–218.Google Scholar
  32. [32]
    H. Rådström, “An embedding theorem for spaces of convex sets,”Proceedings of the American Mathematical Society 3 (1952) 165–169.Google Scholar
  33. [33]
    S.M. Robinson, “Extension of Newton's method to nonlinear functions with values in a cone,”Numerische Mathematik 19 (1972) 341–347.Google Scholar
  34. [34]
    S.M. Robinson, “Normed convex processes,”Transactions of the American Mathematical Society 174 (1972) 127–140.Google Scholar
  35. [35]
    S. Robinson, “Stability theory for systems of inequalities, Part I: linear systems,”SIAM Journal on Numerical Analysis 12 (1975) 754–769.Google Scholar
  36. [36]
    S. Robinson, “Regularity and stability for convex multivalued functions,”Mathematics of Operations Research 1 (1976) 130–143.Google Scholar
  37. [37]
    S. Robinson, “Stability theory for systems of inequalities, Part II: Differentiable nonlinear systems,”SIAM Journal on Numerical Analysis 13 (1976) 497–513.Google Scholar
  38. [38]
    R.T. Rockafellar,Convex Analysis (Princeton University Press, Princeton, NJ, 1970).Google Scholar
  39. [39]
    R.T. Rockafellar, “First- and second-order epi-differentiability in nonlinear programming,”Transactions of the American Mathematical Society 307 (1988) 75–108.Google Scholar
  40. [40]
    R.S. Womersley, “Local properties of algorithms for minimizing nonsmooth composite functions,”Mathematical Programming 32 (1) (1985) 69–89.Google Scholar

Copyright information

© The Mathematical Programming Society, Inc. 1995

Authors and Affiliations

  • J. V. Burke
    • 1
  • M. C. Ferris
    • 2
  1. 1.Department of Mathematics, GN-50University of WashingtonSeattleUnited States
  2. 2.Computer Sciences DepartmentUniversity of WisconsinMadisonUnited States

Personalised recommendations