Mathematical Programming

, Volume 63, Issue 1–3, pp 129–156 | Cite as

Representations of quasi-Newton matrices and their use in limited memory methods

  • Richard H. Byrd
  • Jorge Nocedal
  • Robert B. Schnabel


We derive compact representations of BFGS and symmetric rank-one matrices for optimization. These representations allow us to efficiently implement limited memory methods for large constrained optimization problems. In particular, we discuss how to compute projections of limited memory matrices onto subspaces. We also present a compact representation of the matrices generated by Broyden's update for solving systems of nonlinear equations.

Key words

Quasi-Newton method constrained optimization limited memory method large-scale optimization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. J. Barnes, “An algorithm for solving nonlinear equations based on the secant method,”Computer Journal 8 (1965) 66–67.Google Scholar
  2. L. Biegler, J. Nocedal and C. Schmid, “Reduced Hessian methods for large scale constrained optimization,” Technical Report, Department of Electrical Engineering and Computer Science, Northwestern University (Evanston, IL, 1993).Google Scholar
  3. C.G. Broyden, “A class of methods for solving nonlinear simultaneous equations,”Mathematics of Computation 19 (1965) 577–593.Google Scholar
  4. A. Buckley and A. LeNir, “QN-like variable storage conjugate gradients,”Mathematical Programming 27 (1983) 103–119.Google Scholar
  5. R.H. Byrd, P. Lu and J. Nocedal, “A limited memory algorithm for bound constrained optimization,” Technical Report, Department of Electrical Engineering and Computer Science, Northwestern University (Evanston, IL, 1993).Google Scholar
  6. A.R. Conn, N.I.M. Gould and Ph.L. Toint, “Testing a class of methods for solving minimization problems with simple bounds on the variables,”Mathematics of Computation 50/182 (1988) 399–430.Google Scholar
  7. J.E. Dennis Jr. and R.B. Schnabel,Numerical Methods for Unconstrained Optimization and Nonlinear Equations (Prentice-Hall, Englewood Cliffs, NJ, 1983).Google Scholar
  8. R. Fletcher,Practical Methods of Optimization (Wiley, Chichester, 1987, 2nd ed.).Google Scholar
  9. D.M. Gay and R.B. Schnabel, “Solving systems of nonlinear equations by Broyden's method with projected updates,” in: O.L. Mangasarian, R.R. Meyer and S.M. Robinson, eds.,Nonlinear Programming 3 (Academic Press, New York, 1978) pp. 245–281.Google Scholar
  10. J.C. Gilbert and C. Lemaréchal, “Some numerical experiments with variable storage quasi-Newton algorithms,”Mathematical Programming 45 (1989) 407–436.Google Scholar
  11. J.C. Gilbert and J. Nocedal, “The limited memory step computation and automatic differentiation,”Applied Math Letters 6(3) (1993) 47–50.Google Scholar
  12. A. Griewank, “On automatic differentiation,” in: M. Iri and K. Tanabe, eds.,Mathematical Programming (Kluwer Academic Publishers, Tokyo, 1989) pp. 83–107.Google Scholar
  13. H. Fayez Khalfan, “Topics in quasi-Newton methods for unconstrained optimization,” Ph.D. thesis, Department of Mathematics, University of Colorado (Boulder, CO, 1989).Google Scholar
  14. H. Fayez Khalfan, R.H. Byrd, and R.B. Schnabel, “A theoretical and experimental study of the symmetric rank one update,”SIAM Journal on Optimization 3 (1993) 1–24.Google Scholar
  15. D.C. Liu and J. Nocedal, “On the limited memory BFGS method for large scale optimization,”Mathematical Programming 45 (1989) 503–528.Google Scholar
  16. D.Q. Mahidhara and L. Lasdon, “An SQP algorithm for large sparse nonlinear programs,” Technical report, MSIS Department, School of Business Administration, University of Texas (Austin, TX, 1990).Google Scholar
  17. H. Matthies and G. Strang, “The solution of nonlinear finite element equations,”International Journal of Numerical Methods in Engineering 14 (1979) 1613–1626.Google Scholar
  18. J. Nocedal, “Updating quasi-Newton matrices with limited storage,”Mathematics of Computation 35 (1980) 773–782.Google Scholar
  19. J.M. Ortega and W.C. Rheinboldt,Iterative Solution of Nonlinear Equations in Several Variables (Academic Press, New York, 1970).Google Scholar
  20. R.B. Schnabel, “Quasi-Newton methods using multiple secant equations,” Technical Report CU-CS-247-83, Department of Computer Science, University of Colorado (Boulder, CO, 1983).Google Scholar
  21. H.F. Walker, “Implementation of the GMRES method using Householder transformations,”SIAM Journal on Scientific and Statistical Computing 9(1) (1988) 152–163.Google Scholar

Copyright information

© The Mathematical Programming Society, Inc. 1994

Authors and Affiliations

  • Richard H. Byrd
    • 1
  • Jorge Nocedal
    • 2
  • Robert B. Schnabel
    • 1
  1. 1.Computer Science DepartmentUniversity of ColoradoBoulderUSA
  2. 2.Department of Electrical Engineering and Computer ScienceNorthwestern UniversityEvanstonUSA

Personalised recommendations