The purpose of this paper is to draw a detailed comparison between Newton's method, as applied to discrete-time, unconstrained optimal control problems, and the second-order method known as differential dynamic programming (DDP). The main outcomes of the comparison are: (i) DDP does not coincide with Newton's method, but (ii) the methods are close enough that they have the same convergence rate, namely, quadratic.
The comparison also reveals some other facts of theoretical and computational interest. For example, the methods differ only in that Newton's method operates on a linear approximation of the state at a certain point at which DDP operates on the exact value. This would suggest that DDP ought to be more accurate, an anticipation borne out in our computational example. Also, the positive definiteness of the Hessian of the objective function is easy to check within the framework of DDP. This enables one to propose a modification of DDP, so that a descent direction is produced at each iteration, regardless of the Hessian.
Nonlinear programming optimal control optimal control algorithms nonlinear dynamics quadratic convergence
Murray, M., andYakowitz, S.,The Application of Optimal Control Methodology to Nonlinear Programming Problems, Mathematical Programming, Vol. 21, pp. 331–347, 1981.MATHCrossRefMathSciNetGoogle Scholar
Yakowitz, S., andRutherford, B.,Computational Aspects of Differential Dynamic Programming, Applied Mathematics and Computation (to appear).Google Scholar
Murray, D. M., andYakowitz, S. J.,Constrained Differential Dynamic Programming, with Application to Multi-Reservoir Control, Water Resource Research, Vol. 15, pp. 1017–1027, 1979.CrossRefGoogle Scholar
Yakowitz, S.,Dynamic Programming Applications in Water Resources, Water Resource Research, Vol. 18, pp. 673–698, 1982.CrossRefGoogle Scholar
Murray, D. M.,Differential Dynamic Programming for the Efficient Solution of Optimal Control Problems, University of Arizona, PhD Thesis, 1978.Google Scholar
Szidarovszky, F., andYakowitz, S.,Principles and Procedures of Numerical Analysis, Plenum Press, New York, New York, 1978.MATHGoogle Scholar
Ortega, J., andRheinboldt, W.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.MATHGoogle Scholar