Automatic Differentiation: Calculation of Newton Steps

Many algorithms for solving optimization problems require the minimization of a merit function, which may be the original objective function, or the solution to sets of simultaneous nonlinear equations which may involve the constraints in the problem. To obtain second order convergence near the solution algorithms to solve both rely on the calculation of Newton steps.

When solving a set of nonlinear equations

the Newton step d at x ⁽⁰⁾, x ∈ R ⁿ, is obtained by solving the linear set of equations

where both the derivatives ∂s _j /∂x _i and the vector function s _j are evaluated at a point, which we will denote by x ⁽⁰⁾.

For convenience we introduce the Jacobian matrix J and write the equation as

When minimizing a function f(x) the Newton equation becomes

where all the derivatives are calculated at a point, again denoted by x ⁽⁰⁾.

In terms of the Hessian, H, and the gradient, g, this can be written

Automatic differentiationcan be used to calculate the gradient, Hessian and Jacobian, but...