Efficiency Analysis on a Truncated Newton Method with Preconditioned Conjugate Gradient Technique for Optimization

  • J. Z. Zhang
  • N. Y. Deng
  • Z. Z. Wang
Part of the Applied Optimization book series (APOP, volume 82)


It has been shown by a large amount of numerical experiments that among the local algorithms for solving unconstrained optimization problems, the truncated Newton method with preconditioned conjugate gradient (PCG) subiterations is very efficient. In this paper, we investigate its efficiency from theoretical point of view. The question is, compared with Newton’s method with Cholesky factorization, how much it is more efficient in theory. We give a quantitative answer by constructing a truncated Newton method with PCG subiterations -- Algorithm II below. Suppose Newton’s method is convergent with a one-step, Q-order α rate (α≥ 2). We first prove that Algorithm II has the same convergence rate. We then study its average number of arithmetic operations per step and the corresponding number which Newton’s method needs. Ari upper bound for the ratio of these two numbers is obtained. This upper bound is a quantitative estimate of the saving which Algorithm II can achieve from theoretical point of view. Its values, which are listed in the paper, show that the saving is rather remarkable.


unconstrained optimization Newton’s method preconditioned conjugate gradient method efficiency coefficient 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [At 89]
    K. E. Atkinson, An Introduction to numerical analysis, John Wiley and Sons, New York, Second Edition, 1989.Google Scholar
  2. [CGT92]
    A. R. Conn, N. I. M. Gould and Ph. L. Toint, Numerical experiments with the LANCELOT package (Release A) for large-scale nonlinear optimization, Technical Report, pp. 92–075, Rutherford Appleton Laboratory, Chilton, England, 1992.Google Scholar
  3. [DET 82]
    R. Dembo, S. Eisenstat, and T. Steihaug, Inexact Newton method, SIAM Journal on Numerical Analysis, 19 (1982), pp. 400–408.MathSciNetzbMATHGoogle Scholar
  4. [DP 88]
    L. C. W. Dixon and R. C. Price, Numerical experience with the truncated Newton method for unconstrained optimization,Journal of Optimization Theory and Applications, 56(1988), pp. 245–255.MathSciNetzbMATHGoogle Scholar
  5. [DS 83]
    J. E. Dennis and R. B. Schnabel, Numerical methods for unconstrained optimization and nonlinear equations, Prentice Hall, Englewood Cliffs, NJ, 1983.zbMATHGoogle Scholar
  6. [DW 00]
    N. Y. Deng and Z. Z. Wang, Theoretical efficiency of an inexact Newton method, Journal of Optimization Theory and Applications, 105(2000), pp. 97–112.MathSciNetzbMATHGoogle Scholar
  7. [DXZ 93]
    N. Y. Deng, Y. Xiao and F. J. Zhou, Nonmonoton.ic trust region algorithm, Journal of Optimization Theory and Applications, 76(1993), pp. 259–285.MathSciNetzbMATHGoogle Scholar
  8. [EW 96]
    S. C. Eisenstat and H. F. Walker, Choosing the forcing terms in an inexact Newton method, SIAM Journal on Scientific Computing, 17(1996), pp. 3346.MathSciNetGoogle Scholar
  9. [GMW 81]
    P. E. Gill, W. Murray and M. H. Wright, Practical optimization, Academic Press, London and New York, 1981.zbMATHGoogle Scholar
  10. [Ke 95]
    C. T. Kelley, Iterative methods for linear and nonlinear equations,SIAM, Philadelphia, 1995.zbMATHGoogle Scholar
  11. [Na 85]
    S. G. Nash, Preconditioning of truncated-Newton methods, SIAM Journal on Scientific and Statistical Computing, 6(1985), pp. 559–616.MathSciNetGoogle Scholar
  12. [No 96]
    J. Nocedal, Large scale unconstrained optimization, Report-DEECS, Northwestern University, 1996.Google Scholar
  13. [OR70]
    J.M. Ortega and W.C. Rheinboldt, Iterative solution of nonlinear equa-tions in several variables, Academic Press, London, 1970.Google Scholar
  14. Os 60]
    [ Ostrowski, A., Solution of Equations and Systems of Equations, Academic Press, New York, 1960.zbMATHGoogle Scholar
  15. [St 83]
    T. Steihaug, The conjugate gradient method and trust region in large scale optimization,SIAM Journal on Numerical Analysis, 20(1983), pp. 626–637.MathSciNetzbMATHGoogle Scholar
  16. [To 81]
    Ph. L. Toint, Towards an efficient sparsity exploiting Newton method for minimization, in Sparse matrices and their uses (ed. I.S. Duff), Academic Press, London, 1981, pp. 57–88.Google Scholar

Copyright information

© Kluwer Academic Publishers B.V. 2003

Authors and Affiliations

  • J. Z. Zhang
    • 1
  • N. Y. Deng
    • 2
  • Z. Z. Wang
    • 3
  1. 1.Department of MathematicsCity University of Hong KongHong KongChina
  2. 2.China Agricultural UniversityBeijingChina
  3. 3.University of GreenwichUK

Personalised recommendations