Mathematical Programming

, Volume 26, Issue 2, pp 190–212 | Cite as

Truncated-newtono algorithms for large-scale unconstrained optimization

  • Ron S. Dembo
  • Trond Steihaug


We present an algorithm for large-scale unconstrained optimization based onNewton's method. In large-scale optimization, solving the Newton equations at each iteration can be expensive and may not be justified when far from a solution. Instead, an inaccurate solution to the Newton equations is computed using a conjugate gradient method. The resulting algorithm is shown to have strong convergence properties and has the unusual feature that the asymptotic convergence rate is a user specified parameter which can be set to anything between linear and quadratic convergence. Some numerical results on a 916 vriable test problem are given. Finally, we contrast the computational behavior of our algorithm with Newton's method and that of a nonlinear conjugate gradient algorithm.

Key words

Unconstrained Optimization Modified Newton Methods Conjugate Gradient Algorithms 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    O. Axelsson, “Solution of linear systems of equations: Iterative methods”, in: V.A. Barker, ed.,Sparse matrix techniques. (Springer-Verlag, New York, 1976) pp. 1–51.Google Scholar
  2. [2]
    R. Chandra, “Conjugate gradient methods for partial differential equations”, Ph.D. dissertation, Yale University (New Haven, CT, 1978), Also available as Department of Computer Science Research Report No. 129.Google Scholar
  3. [3]
    R. Chandra, S.C. Eisenstat and M.H. Schultz, “The modified conjugate residual method for partial differential equations”, in: R. Vichnevetsky, ed.,Advance in computer methods for partial differential equations II, Proceedings of the Second International Symposium on Computer Methods for Partial Differential Equations, Lehigh University, Bethlehem, PA (International Association for Mathematics and Computers in Simulation, June 1977) pp. 13–19.Google Scholar
  4. [4]
    A.R. Curtis, M.J.D. Powell and J.K. Reid, “On the estimation of sparse Jacobian matrices”,Journal of the Institute of Mathematics and its Applications 13 (1974) 117–119.MATHGoogle Scholar
  5. [5]
    R.S. Dembo, S.C. Eisenstat and T. Steihaug, “Inexact Newton methods”,SIAM Journal of Numerical Analysis 19 (1982) 400–408.MATHCrossRefMathSciNetGoogle Scholar
  6. [6]
    R.S. Dembo and J.G. Klincewicz, “A scaled reduced gradient algorithm for network flow problems with convex separable costs”,Mathematical Programming Studies 15 (1981) 125–147.MATHMathSciNetGoogle Scholar
  7. [7]
    R. S. Dembo and T. Steihaug, “A test problem for large-scale unconstrained minimization”, School of Organization and Management, Yale University (New Haven, CT) Working Paper Series B (in preparation).Google Scholar
  8. [8]
    J.E. Dennis Jr. and J.J. Moré, “Quasi-Newton methods, motivation and theory”,SIAM Review 19 (1977) 46–89.MATHCrossRefMathSciNetGoogle Scholar
  9. [9]
    R. Fletcher, “Conjugate gradient methods for indefinite systems”, in: G.A. Watson, ed.,Numerical Analysis, Proceedings of Biennial Conference, Dundee, Scotland, 1975 (Springer-Verlag, New York, 1976) pp. 73–89.CrossRefGoogle Scholar
  10. [10]
    R. Fletcher,Unconstrained optimization (John Wiley and Sons, New York, 1980).MATHGoogle Scholar
  11. [11]
    N.K. Garg and R.A. Tapia, “QDN: A variable storage algorithm for unconstrained optimization”, Technical Report, Department of Mathematical Sciences, Rice University (Houston, TX, 1977).Google Scholar
  12. [12]
    P.E. Gill and W. Murray, “Safeguarded steplength algorithm for optimization using descent methods”, Technical Report NPL NA 37, National Physical Laboratory (1974).Google Scholar
  13. [13]
    P.E. Gill, W. Murray and S.G. Nash, “A conjugate-gradient approach to Newton-type methods”, presented at ORSA/TIMS Joint National Meeting (Colorado Springs, November 1980).Google Scholar
  14. [14]
    P.E. Gill, W. Murray and M.H. WrightPractical optimization (Academic Press, New York, 1981).MATHGoogle Scholar
  15. [15]
    M.R. Hestenes,Conjugate direction methods in optimization (Springer-Verlag New York, 1980).MATHGoogle Scholar
  16. [16]
    M.R. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems”,Journal of Research of the National Bureau of Standards 49 (1952) 409–436.MATHMathSciNetGoogle Scholar
  17. [17]
    D.G. Luenberger, “Hyperbolic pairs in the method of conjugate gradients”,SIAM Journal on Applied Mathematics 17 (1969) 1263–1267.MATHCrossRefMathSciNetGoogle Scholar
  18. [18]
    B. Murtagh and M. Saunders, “Large-scale linearly constrained optimization”,Mathematical Programming 14 (1978) 41–72.MATHCrossRefMathSciNetGoogle Scholar
  19. [19]
    J. Nocedal, “Updating Quasi-Newton matrices with limited storage”,Mathematics of Computation 35 (1980) 773–782.MATHCrossRefMathSciNetGoogle Scholar
  20. [20]
    D.P. O'Leary, “A discrete Newton algorithm for minimizing a function of many variables”,Mathematical Programming 23 (1982) 20–33.MATHCrossRefMathSciNetGoogle Scholar
  21. [21]
    J.M. Ortega and W.C. RheinboldtIterative solution of nonlinear equations in several variables (Academic Press, New York, 1970).MATHGoogle Scholar
  22. [22]
    M.J.D. Powell and Ph.L. Toint, “On the estimation of sparse Hessian matrices”,SIAM Journal on Numerical Analysis 16 (1979) 1060–1074.MATHCrossRefMathSciNetGoogle Scholar
  23. [23]
    D.F. Shanno, “On the convergence of a new conjugate gradient method”,SIAM Journal on Numerical Analysis 15 (1978) 1247–1257.MATHCrossRefMathSciNetGoogle Scholar
  24. [24]
    D.F. Shanno and K.H. Phua, “Algorithm 500: Minimization of unconstrained multivariate functions”,Transactions on Mathematical Software 2 (1976) 87–94.MATHCrossRefGoogle Scholar
  25. [25]
    T. Steihaug, “Quasi-Newton methods for large scale nonlinear problems”, Ph.D. dissertation, Yale University (New Haven, CT, 1980). Also available as Working Paper, Series B No. 49.Google Scholar
  26. [26]
    Ph.L. Toint, “Some numerical results using a sparse matrix updating formula in unconstrained optimization”,Mathematics of Computation 32 (1978) 839–851.MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© North-Holland Publishing Company 1983

Authors and Affiliations

  • Ron S. Dembo
    • 1
  • Trond Steihaug
    • 2
  1. 1.School of Organization and ManagementYale UniversityNew HavenUSA
  2. 2.Department of Mathematical SciencesRice UniversityHoustonUSA

Personalised recommendations