Generalized Least Squares and Newton’s Method Algorithms for Nonlinear Root-Solving Applications

Abstract

Many problems in science and engineering must solve nonlinear necessary conditions. For example, a standard problem in optimization involves solving for the roots of nonlinear functions defined by f(x) = 0, where x is the unknown variable. Classically one develops a first-order Taylor series model that defines the necessary condition that must be iteratively refined. The standard assumption is that the correction terms are small. Two classes of problems arise: (1) non-square systems that lead to least-squares solutions, and (2) square systems that are often handled by Newton-like methods. The accuracy of the starting guess impacts the number of iteration cycles required. To handle more nonlinear problems, both approaches are generalized to account for first- through fourth-order approximations. Computational differentiation tools are used for automatically formulating and numerically computing the partial derivatives. Two solution approaches are presented for inverting the tensor-valued necessary condition: (1) an integrated Legendre transformation, homotopy method, and high-order vector reversion of series algorithm; and (2) a computational differentiation-based generalized linear algebra approach. Several numerical examples are presented to demonstrate generalized multilinear Least-Squares and Newton-Raphson Methods. Accelerated convergence rates are demonstrated for scalar and vector root-solving problems. The integration of generalized algorithms and automatic differentiation is expected to have broad potential for impacting the design and use of mathematical programming tools for knowledge discovery applications in science and engineering.

Appendix: Legendre Transformation for Developing a Reversion of Series Solution

All of the nonlinear necessary conditions developed in this work lead to n-tuple sets of algebraic equations. The challenge is that these variables fundamentally lack an independent variable, which can be used for developing an analytic continuation solution for the approximate root solution. An artificial independent variable is introduced into the n-tuple algebraic equations by invoking the following Legendre transformation [28]:

$$ H=x-sx_{0} =0\Rightarrow x=\left. {x_{0} } \right|_{s=1}\, \text{and} \,x=\left. 0 \right|_{s=0} $$

(A.1)

where s∈[1, 0] is the artificial independent variable and H represents a homotopy embedding equation. The equation implies that x = x(s). Repeated differentiation of Eq. A.1, with respect to s, yields the cascade of sensitivity necessary conditions

$$\begin{array}{@{}rcl@{}} H_{,s} =&&\nabla x\cdot x_{,s} -x_{0} =0 \\ H_{,ss} =&&\nabla^{2}x\cdot x_{,s} \cdot x_{,s} +\nabla x\cdot x_{,ss} =0 \\ H_{,sss} =&&\nabla^{3}x\cdot x_{,s} \cdot x_{,s} \cdot x_{,s} +2\nabla^{2}x\cdot x_{,ss} \cdot x_{,s} +\nabla^{2}x\cdot x_{,s} \cdot x_{,ss} +\nabla x\cdot x_{,sss} =0 \\ H_{,ssss} =&&\nabla^{4}x\cdot x_{,s} \cdot x_{,s} \cdot x_{,s} \cdot x_{,s} +3\nabla^{3}x\cdot x_{,ss} \cdot x_{,s} \cdot x_{,s} +2\nabla^{3}x\cdot x_{,s} \cdot x_{,ss} \cdot x_{,s} + \\ \;\;&&\nabla^{3}x\cdot x_{,s} \cdot x_{,s} \cdot x_{,ss} +3\nabla^{2}x\cdot x_{,sss} \cdot x_{,s} +3\nabla^{2}x\cdot x_{,ss} \cdot x_{,ss}\\ &&+\nabla^{2}x\cdot x_{,s} \cdot x_{,sss} +\nabla x\cdot x_{,ssss} \end{array} $$

(A.2)

where ∇x,∇² x,⋯,∇^m x are generated by OCEA, and the unknown rates are presented in the red font. Inverting cascade for the rates yields

$$\begin{array}{@{}rcl@{}} x_{,s} &=&\quad \left({\nabla x} \right)^{-1}x_{0} \\ x_{,ss} &=&-\left({\nabla x} \right)^{-1}\nabla^{2}x\cdot x_{,s} \cdot x_{,s} \\ x_{,sss} &=&-\left({\nabla x} \right)^{-1}\left({\nabla^{3}x\cdot x_{,s} \cdot x_{,s} \cdot x_{,s} +2\nabla^{2}x\cdot x_{,ss} \cdot x_{,s} +\nabla^{2}x\cdot x_{,s} \cdot x_{,ss} } \right) \\ x_{,ssss} &=&-\left({\nabla x} \right)^{-1}\left({\begin{array}{l} \nabla^{4}x\cdot x_{,s} \cdot x_{,s} \cdot x_{,s} \cdot x_{,s} +3\nabla^{3}x\cdot x_{,ss} \cdot x_{,s} \cdot x_{,s} +2\nabla^{3}x\cdot x_{,s} \cdot x_{,ss} \cdot x_{,s} + \\ \nabla^{3}x\cdot x_{,s} \cdot x_{,s} \cdot x_{,ss} +3\nabla^{2}x\cdot x_{,sss} \cdot x_{,s} +3\nabla^{2}x\cdot x_{,ss} \cdot x_{,ss} +\nabla^{2}x\cdot x_{,s} \cdot x_{,sss} \end{array}} \right) \\ \end{array} $$

(A.3)

The solution for x is now analytically continued to fourth order, as a function of the artificial independent variable s, as

$$ x(s)=x_{0} +x_{,s} {\Delta} s+\frac{1}{2!}x_{,ss} {\Delta} s^{2}+\frac{1}{3!}x_{,sss} {\Delta} s^{3}+\frac{1}{4!}x_{,ssss} {\Delta} s^{4} $$

(A.4)

Recalling the homotopy parameters defined in Eq. A.1, it follows that the change in the s-variable is

$$ {\Delta} s=s_{f} -s_{0} =0-1=-1 $$

(A.5)

which permits (A.4) to be expressed as the reversion-of-series solution given by

$$ x=x_{0} -x_{,s} +\frac{1}{2!}x_{,ss} -\frac{1}{3!}x_{,sss} +\frac{1}{4!}x_{,ssss} $$

(A.6)

Equation A.6, when combined with Eqs. A.3 and A.4, makes full use of the sensitivity data recovered by the generalized Matrix-vector solution of Eq. 16. A significant advantage of this approach is that the reversion-of-series approach does not depend on the algebraic complexity of the original nonlinear problem, as well as scaling to arbitrary order expansions.