Conjugate-gradient methods (CG methods) are used to solve large-dimensional problems that arise in computational linear algebra and computational nonlinear optimization. These two subjects share a broad common frontier, and one of the most easily traversed crossing points is via the following simple observation: the problem of solving a system of linear equations Ax = b for the unknown vector x ∈ R n, where A is a positive definite, symmetric matrix and b is a given vector, is mathematically equivalent to finding the minimizing point of the strictly convex quadratic function
The linear CG method for solving the system of linear equations is able to capitalize on this equivalent optimization formulation. It was developed in the pioneering 1952 paper of M.R. Hestenes and E.L. Stiefel [11] who, in turn, cite antecedents in the contributions of several other authors (see [9]). The method fell out of favor with numerical analysts during the 1960s because it did not compete with direct...
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adams, L., and Nazareth, J.L.: Linear and nonlinear conjugate gradient-related methods, SIAM, 1996.
Barrett, R., Berry, M., Chan, T.F., Demmel, J., Donato, J., Dongarra, J., Eijkhout, V., Pozo, R., Romine, C., and Vorst, H.van der: Templates for the solution of linear systems, SIAM, 1993.
Bertsekas, D.P.: Nonlinear programming, second ed., Athena Sci., 1999.
Buckley, A.: ‘Extending the relationship between the conjugate gradient and BFGS algorithms’, Math. Program.15 (1978), 343–348.
Dai, Y.H.: ‘Analyses of nonlinear conjugate gradient methods’, PhD Diss. Inst. Computational Math. Sci./Engin. Computing Chinese Acad. Sci., Beijing, China (1997).
Davidon, W.C.: ‘Variable metric method for minimization’, SIAM J. Optim.1 (1991), 1–17, (Original (with a different preface): Argonne Nat. Lab. Report ANL-5990 (Rev., Argonne, Illinois).
Fenelon, M.C.: ‘Preconditioned conjugate-gradient-type methods for large-scale unconstrained optimization’, PhD Diss. Stanford Univ., Stanford, CA (1981).
Fletcher, R., and Reeves, C.: ‘Function minimization by conjugate gradients’, Computer J.7 (1964), 149–154.
Golub, G.H., and O’Leary, D.P.: ‘Some history of the conjugate gradient and Lanczos algorithms 1948–1976’, SIAM Rev.31 (1989), 50–102.
Hestenes, M.R.: Conjugate direction methods in optimization, Vol. 12 of Appl. Math., Springer, 1980.
Hestenes, M.R., and Stiefel, E.L.: ‘Methods of conjugate gradients for solving linear systems’, J. Res. Nat. Bureau Standards (B)49 (1952), 409–436.
Kolda, T.G., O’Leary, D.P., and Nazareth, J.L.: ‘BFGS with update skipping and varying memory’, SIAM J. Optim.8 (1998), 1060–1083.
Leonard, M.W.: ‘Reduced Hessian quasi-Newton methods for optimization’, PhD Diss. Univ. Calif., San Diego, CA (1995).
Luenberger, D.G.: Linear and nonlinear programming, second ed., Addison-Wesley, 1984.
Nazareth, J.L.: ‘A relationship between the BFGS and conjugate gradient algorithms and its implications for new algorithms’, SIAM J. Numer. Anal.16 (1979), 794–800.
Nazareth, J.L.: ‘The method of successive affine reduction for nonlinear minimization’, Math. Program.35 (1986), 97–109.
Nesterov, Y.E.: ‘A method of solving a convex programming problem with convergence rate O(1/k2)’, Soviet Math. Dokl.27 (1983), 372–376.
Nocedal, J.: ‘Updating quasi-Newton matrices with limited storage’, Math. Comput.35 (1980), 773–782.
Nocedal, J.: ‘Theory of algorithms for unconstrained optimization’, Acta Numer.1 (1992), 199–242.
Polak, E., and Ribière, G.: ‘Note sur la convergence de methode de directions conjuguees’, Revue Franc. Inform. Rech. Oper.16 (1969), 35–43.
Polyak, B.T.: ‘Some methods of speeding up the convergence of iteration methods’, USSR Comput. Math. Math. Phys.4 (1964), 1–17.
Polyak, B.T.: ‘The conjugate gradient method in extremal problems’, USSR Comput. Math. Math. Phys.9 (1969), 94–112.
Powell, M.J.D.: ‘Restart procedures for the conjugate gradient method’, Math. Program.12 (1977), 241–254.
Shah, B.V., Buehler, R.J., and Kempthorne, O.: ‘Some algorithms for minimizing a function of several variables’, J. SIAM12 (1964), 74–91.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Kluwer Academic Publishers
About this entry
Cite this entry
Nazareth, J.L. (2001). Conjugate-Gradient Methods . In: Floudas, C.A., Pardalos, P.M. (eds) Encyclopedia of Optimization. Springer, Boston, MA. https://doi.org/10.1007/0-306-48332-7_69
Download citation
DOI: https://doi.org/10.1007/0-306-48332-7_69
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-7923-6932-5
Online ISBN: 978-0-306-48332-5
eBook Packages: Springer Book Archive