# Efficiency Analysis on a Truncated Newton Method with Preconditioned Conjugate Gradient Technique for Optimization

## Abstract

It has been shown by a large amount of numerical experiments that among the local algorithms for solving unconstrained optimization problems, the truncated Newton method with preconditioned conjugate gradient (PCG) subiterations is very efficient. In this paper, we investigate its efficiency from theoretical point of view. The question is, compared with Newton’s method with Cholesky factorization, how much it is more efficient in theory. We give a quantitative answer by constructing a truncated Newton method with PCG subiterations -- Algorithm II below. Suppose Newton’s method is convergent with a one-step, *Q*-order α rate (α≥ 2). We first prove that Algorithm II has the same convergence rate. We then study its average number of arithmetic operations per step and the corresponding number which Newton’s method needs. Ari upper bound for the ratio of these two numbers is obtained. This upper bound is a quantitative estimate of the saving which Algorithm II can achieve from theoretical point of view. Its values, which are listed in the paper, show that the saving is rather remarkable.

## Keywords

unconstrained optimization Newton’s method preconditioned conjugate gradient method efficiency coefficient## Preview

Unable to display preview. Download preview PDF.

## References

- [At 89]K. E. Atkinson, An
*Introduction to numerical analysis*, John Wiley and Sons, New York, Second Edition, 1989.Google Scholar - [CGT92]A. R. Conn, N. I. M. Gould and Ph. L. Toint,
*Numerical experiments with the LANCELOT package (Release A) for large-scale nonlinear optimization*, Technical Report, pp. 92–075, Rutherford Appleton Laboratory, Chilton, England, 1992.Google Scholar - [DET 82]R. Dembo, S. Eisenstat, and T. Steihaug,
*Inexact Newton method*, SIAM Journal on Numerical Analysis, 19 (1982), pp. 400–408.MathSciNetzbMATHGoogle Scholar - [DP 88]L. C. W. Dixon and R. C. Price,
*Numerical experience with the truncated Newton method for unconstrained optimization*,Journal of Optimization Theory and Applications, 56(1988), pp. 245–255.MathSciNetzbMATHGoogle Scholar - [DS 83]J. E. Dennis and R. B. Schnabel,
*Numerical methods for unconstrained optimization and nonlinear equations*, Prentice Hall, Englewood Cliffs, NJ, 1983.zbMATHGoogle Scholar - [DW 00]N. Y. Deng and Z. Z. Wang,
*Theoretical efficiency of an inexact Newton method*, Journal of Optimization Theory and Applications, 105(2000), pp. 97–112.MathSciNetzbMATHGoogle Scholar - [DXZ 93]N. Y. Deng, Y. Xiao and F. J. Zhou,
*Nonmonoton.ic trust region algorithm*, Journal of Optimization Theory and Applications, 76(1993), pp. 259–285.MathSciNetzbMATHGoogle Scholar - [EW 96]S. C. Eisenstat and H. F. Walker,
*Choosing the forcing terms in an inexact Newton method*, SIAM Journal on Scientific Computing, 17(1996), pp. 3346.MathSciNetGoogle Scholar - [GMW 81]P. E. Gill, W. Murray and M. H. Wright,
*Practical optimization*, Academic Press, London and New York, 1981.zbMATHGoogle Scholar - [Ke 95]C. T. Kelley,
*Iterative methods for linear and nonlinear equations*,SIAM, Philadelphia, 1995.zbMATHGoogle Scholar - [Na 85]S. G. Nash,
*Preconditioning of truncated-Newton methods*, SIAM Journal on Scientific and Statistical Computing, 6(1985), pp. 559–616.MathSciNetGoogle Scholar - [No 96]J. Nocedal,
*Large scale unconstrained optimization*, Report-DEECS, Northwestern University, 1996.Google Scholar - [OR70]J.M. Ortega and W.C. Rheinboldt, Iterative solution of nonlinear equa-
*tions in several variables*, Academic Press, London, 1970.Google Scholar - Os 60][ Ostrowski, A.,
*Solution of Equations and Systems of Equations*, Academic Press, New York, 1960.zbMATHGoogle Scholar - [St 83]T. Steihaug, The conjugate gradient method and trust region in large scale
*optimization*,SIAM Journal on Numerical Analysis, 20(1983), pp. 626–637.MathSciNetzbMATHGoogle Scholar - [To 81]Ph. L. Toint, Towards an efficient sparsity exploiting Newton method for
*minimization*, in Sparse matrices and their uses (ed. I.S. Duff), Academic Press, London, 1981, pp. 57–88.Google Scholar