A Sweeping Gradient Method for Ordinary Differential Equations with Events

Abstract

In this paper, we use the calculus of variations to derive a sensitivity analysis for ordinary differential equations with events. This sweeping gradient method (SGM) requires a forward sweep to evaluate the original model and a backwards sweep of the adjoint to compute the sensitivity. The method is applied to canonical optimal control problems with numerical examples, including the sampled linear quadratic regulator and the optimal time-switching and state-switching for minimum-time transfer of the double integrator. We show that the application of the SGM for these examples matches the gradient determined analytically. Numerical examples are produced using gradient-based optimization algorithms. The emphasis of this work is on modeling considerations for the effective application of this method.

Appendix A: Continuous-Time Linear Quadratic Regulator

1.1 Appendix A.1: Problem Model and Application of the Algorithm

In this section, we apply the sweeping gradient method (SGM) to the continuous-time linear quadratic regulator (LQR) problem. We will consider a linear, time-invariant (LTI) system under full-state feedback. This is represented by the dynamics equations

$$\begin{aligned} \begin{aligned} {\dot{x}}\left( t\right)&=A\,x\left( t\right) +B\,u\left( t\right) ,\\ u\left( t\right)&=-K\,x\left( t\right) , \end{aligned} \end{aligned}$$

(32)

where $x\in {\mathbb {R}}^{n}$ is the system state, $u\in {\mathbb {R}}^{m}$ is the system’s controlled input, with the state matrix $A\in {\mathbb {R}}^{n\times n}$ and input matrix $B\in {\mathbb {R}}^{n\times m}$ taken as problem data. The LQR problem is to optimize the feedback gains $K\in {\mathbb {R}}^{m\times n}$ with respect to the quadratic performance index

$$\begin{aligned} J=\frac{1}{2}\int _{0}^{t_{f}}x^{T}\left( t\right) Q\,x\left( t\right) +u^{T}\left( t\right) R\,u\left( t\right) \ \textrm{d}t \end{aligned}$$

(33)

where $Q\in {\mathbb {R}}^{n\times n}$ and $R\in {\mathbb {R}}^{m\times m}$ are weighting matrices that shape the closed-loop performance (and are typically treated as design parameters), and $t_{f}$ is some specified trajectory duration.

To evaluate the gradient with SGM, we find the required partial derivatives of the dynamics (32) and performance index (33) with respect to the state $x\left( t\right)$ and parameters $\theta ={\text {vec}}\left( K\right) =\left[ K_{1,1},K_{2,1},\ldots ,K_{m,n}\right] ^{T}$.

Under state feedback, the ODE model becomes

$$\begin{aligned} \begin{aligned}{\dot{x}}\left( t\right)&=\left( A-BK\right) x\left( t\right) ,\\ x\left( t_{0}\right)&=x_{0}. \end{aligned} \end{aligned}$$

(34)

The quadratic performance index (33) under state feedback is given by

$$\begin{aligned} J=\frac{1}{2}\int _{0}^{t_{f}}x^{T}\left( t\right) \left( Q+K^{T}\, R\,K\right) x\left( t\right) \,\textrm{d}t. \end{aligned}$$

(35)

Then, the partial derivatives required to find the co-state and evaluate the variational derivative (5) are given by

$$\begin{aligned} \begin{aligned} \frac{\partial f}{\partial x}&=A-BK,\\ \frac{\partial f}{\partial \theta }&=x^{T}\otimes -B,\\ \frac{\partial L}{\partial x}&=\left( Q+K^{T}RK\right) x,\qquad \text {and}\\ \frac{\partial L}{\partial \theta }&=x^{T}\left( x^{T}\otimes K^{T}R\right) , \end{aligned} \end{aligned}$$

where $\otimes$ indicates the Kronecker product. The partial derivatives with respect to the state are standard derivatives of linear and quadratic forms from matrix calculus. The partial derivatives with respect to the parameter vector can be found using the matrix calculus properties of the ${\text {vec}}$ operator or by taking the appropriate element-wise partial derivatives. For example, the partial derivative of the dynamics f with respect to the parameters yields the $n\times nm$ matrix

$$\begin{aligned} \frac{\partial }{\partial \theta }f=\left[ \begin{matrix}x_{1}B_{1,1} &{} x_{1}B_{1,2} &{} \cdots &{} x_{1}B_{1,m} &{} x_{2}B_{1,1} &{} \cdots &{} x_{n}B_{1,1} &{} x_{n}B_{1,2} &{} \cdots &{} x_{n}B_{1,m}\\ x_{1}B_{2,1} &{} x_{1}B_{2,2} &{} \cdots &{} x_{1}B_{2,m} &{} x_{2}B_{2,1} &{} \cdots &{} x_{n}B_{2,1} &{} x_{n}B_{2,2} &{} \cdots &{} x_{n}B_{2,m}\\ \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots &{} \ddots &{} \vdots \\ x_{1}B_{n,1} &{} x_{1}B_{n,2} &{} \cdots &{} x_{1}B_{n,m} &{} x_{2}B_{n,1} &{} \cdots &{} x_{n}B_{n,1} &{} x_{n}B_{n,2} &{} \cdots &{} x_{n}B_{n,m} \end{matrix}\right] . \end{aligned}$$

Using these partial derivatives, the co-state FVP is given by

$$\begin{aligned} \begin{aligned}{\dot{\lambda }}\left( t\right)&=-\left( A-BK\right) ^{T}\lambda \left( t\right) -\left( Q+K^{T}RK\right) x\left( t\right) \\ \lambda \left( t_{f}\right)&=0 \end{aligned} \end{aligned}$$

(36)

and the variational derivative given by

$$\begin{aligned} \frac{\partial {\tilde{J}}}{\partial \theta }=\int _{0}^{t_{f}}\left\{ \lambda ^{T}\left( t\right) \left( x^{T}\left( t\right) \otimes - B\right) +x^{T}\left( t\right) \left( x^{T}\left( t\right) \otimes K^{T}R\right) \right\} \,\textrm{d}t. \end{aligned}$$

(37)

1.2 Appendix A.2: Relationship to Direct Solution

The derivative of the performance index (35) with respect to the feedback gains can also be found directly, first proved by Kalman [46] and more recently analyzed for gradient-based optimization by Fatkhulllin [24] and Bu [11]. For completeness, we include the direct formulation of the derivative and then show it is equivalent to the one shown above.

1.2.1 Appendix A.2.1: Direct Solution

This method uses the fact that the performance index (35) can be expressed as

$$\begin{aligned} J=\frac{1}{2}x^{T}\left( 0\right) X\left( 0\right) x\left( 0\right) , \end{aligned}$$

(38)

where $X$ $\left( t\right)$ is the symmetric solution to the differential Lyapunov equation

$$\begin{aligned} \begin{aligned}{\dot{X}}\left( t\right)&=X\left( t\right) \left( A-BK\right) +\left( A-BK\right) ^{T}X\left( t\right) +Q+K^{T}RK,\\ X\left( t_{f}\right)&=0. \end{aligned} \end{aligned}$$

(39)

The differential of (38) is given by

$$\begin{aligned} \textrm{d}J=\frac{1}{2}x^{T}\left( 0\right) \textrm{d}X\left( 0\right) x\left( 0\right) , \end{aligned}$$

(40)

where $\textrm{d}X\left( t\right)$ satisfies the differential Lyapunov equation

$$\begin{aligned} \begin{aligned}\textrm{d}{\dot{X}}\left( t\right)&=\left( A-BK\right) ^ {T}\textrm{d}X\left( t\right) +\textrm{d}X\left( t\right) \left( A-BK\right) +2dK^{T}\left( RK-B^{T}X\left( t\right) \right) ,\\ \textrm{d}X\left( t_{f}\right)&=0. \end{aligned} \end{aligned}$$

(41)

found by applying the Implicit Function Theorem [44] to (39). The form of the Lyapunov equation (41) allows us to express (40) by the integral

$$\begin{aligned} \textrm{d}J=\int _{0}^{t_{f}}x^{T}\left( t\right) \textrm{d}K^{T}\left( RK-B^ {T}X\left( t\right) \right) x\left( t\right) \,\textrm{d}t. \end{aligned}$$

Since $\textrm{d}J$ is a scalar, we can use the cyclic property of the trace without modifying its value to factor out the $\textrm{d}K^{T}$ from the integral as

$$\begin{aligned} \textrm{d}J=\textrm{d}K^{T}\int _{0}^{t_{f}}\left( RK-B^{T}X\left( t\right) \right) x\left( t\right) x^{T}\left( t\right) \,\textrm{d}t. \end{aligned}$$

This gives us the derivative of the performance index (35) with respect to the gain matrix K as

$$\begin{aligned} \frac{\textrm{d}J}{\textrm{d}K^{T}}=\int _{0}^{t_{f}}\left( RK-B^{T}X\left( t\right) \right) x\left( t\right) x^{T}\left( t\right) \,\textrm{d}t. \end{aligned}$$

(42)

1.2.2 Appendix A.2.2: Equivalence to SGM

The derivative (42) is the same as (37). To see this, we first follow Bryson [9] to assume the form of the co-state as

$$\begin{aligned} \lambda \left( t\right) =S\left( t\right) x\left( t\right) , \end{aligned}$$

(43)

where $S\left( t\right)$ is a yet unknown time-varying matrix. Taking the time derivative of (43) and substituting the dynamics of the state (34) and co-state (36) shows that $S\left( t\right)$ must satisfy the matrix differential equation

$$\begin{aligned} \begin{aligned}{\dot{S}}\left( t\right)&=S\left( t\right) \left( A-BK\right) +\left( A-BK\right) ^{T}S\left( t\right) +Q+K^{T}RK,\\ S\left( t_{f}\right)&=0. \end{aligned} \end{aligned}$$

(44)

Since $S\left( t\right)$ must satisfy the same Lyapunov equation as $X\left( t\right)$ and the solution to the Lyapunov equation is unique, we have $S\left( t\right) =X\left( t\right)$ and $X\left( t\right) x\left( t\right) =\lambda \left( t\right)$.

Fig. 10

Optimization progress for the double integrator numerical example of the continuous-time LQR problem

Fig. 11

Simulated trajectories for the double integrator numerical example of the continuous-time LQR problem

Fig. 12

Optimization path for the double integrator example with background gradient field for the continuous-time LQR problem

Fig. 13

Optimization progress for the linearized cart and pendulum numerical example of the continuous-time LQR problem

Fig. 14

Simulated trajectories for the linearized cart and pendulum numerical example of the continuous-time LQR problem

Next, we will use Roth’s theorem [68], which states for matrices T, U, V of appropriate dimension for valid matrix multiplication,

$$\begin{aligned} {\text {vec}}\left( TUV\right) =\left( V^{T}\otimes T\right) {\text {vec}}\left( U\right) . \end{aligned}$$

We will also use the transpose property of the Kronecker product,

$$\begin{aligned} \left( U\otimes V\right) ^{T}=U^{T}\otimes V^{T}. \end{aligned}$$

Then, we can manipulate (42) by substituting $S\left( t\right)$ for $X\left( t\right)$, distributing the constant matrices into the integral, applying the ${\text {vec}}$ operator, and noting that the ${\text {vec}}$ operator applied to a vector like $x\left( t\right)$ leaves it unchanged, to find

$$\begin{aligned} \frac{\partial J}{\partial {\text {vec}}^{T}K}= & {} \left[ {\text {vec}}\left( \frac{\partial J}{\partial K^{T}}\right) \right] ^{T}=\int _{0}^{t_{f}}x^{T}\left( t\right) \left( x^{T}\left( t\right) \otimes RK\right) \\{} & {} +x^{T}\left( t\right) S\left( t\right) \left( x^{T}\left( t\right) \otimes -B\right) \,\textrm{d}t \end{aligned}$$

as desired.

1.3 Appendix A.3: Numerical Examples

We consider two canonical systems for the LQR numerical examples. These examples are selected to illustrate behavior with different state dimension (and therefore parameter dimension) and stability properties. The first system is a double integrator with state and input matrices

$$\begin{aligned} A=\left[ \begin{matrix}0 &{} 1\\ 0 &{} 0 \end{matrix}\right] \qquad \text {and}\qquad B=\left[ \begin{matrix}0\\ 1 \end{matrix}\right] . \end{aligned}$$

The second system is a linearized cart and pendulum model with state and input matrices

where $m=1$ kg is the mass of the pendulum concentrated at $L=0.5$ m from the cart of mass $M=3$ kg experiencing acceleration due to gravity $g=9.81$ m/s$^{2}$. The state is defined by the translational position of the center of mass of the cart, the pendulum angle from vertical, followed by their respective velocities. The input is modeled as the horizontal force acting on the cart.

For the numerical examples in this and subsequent sections, the original and adjoint systems are solved in sequence using SimuPy [56]. The performance index and its derivative are evaluated by quadrature. The gradient-based optimization is performed using SciPy’s implementation of the conjugate gradient method [80].

For the double integrator, we use $Q=I\in {\mathbb {R}}^{2\times 2}$ and $R=I\in {\mathbb {R}}^{1\times 1}$ for the performance index. The feedback gains are initialized to zero in ${\mathbb {R}}^{1\times 2}$. The initial condition is $x\left( 0\right) =\left[ 1,0.1\right] ^{T}$ and each simulation runs for 32 s. In Fig. 10, we show the progression of several metrics over each iteration of the optimization and establish the colormap for the iteration number shared for each figure associated with a particular example. The color is determined from a perceptually uniform sequential color map from purple to yellow for the optimizer’s iteration number. Red is generally used for “true” solution, in this case the LQR gains found by solving the algebraic Riccati equation (ARE). Figure 10a shows the progression of the performance metric on a log scale with an offset of the best performance, indicated above the vertical axis, to show the progression over a wide dynamic range. In this case, the optimized gains perform marginally better than the LQR solution from the ARE, shown in a dashed red line, which may be an artifact of evaluating performance from an approximate numeric solution of the ODE with a single initial condition. The method could be performed by averaging the results from multiple initial conditions spanning the state space. The performance metric for the final iteration of the optimizer in this example is indicated by both the half-circle and extended bar to indicate the floor of the plot. Figure 10b shows the normalized progression along the line from the initial feedback gains to the LQR gains found by solving the ARE. Figure 10c shows the progression of the norm of the gradient. Figure 11 shows the simulated trajectories of the state, input, and control, respectively, where the color of each curve indicates the iteration of the optimizer matching the colormap of Fig. 10.

Since the double integrator problem has a two-dimensional parameter space, we can visualize the performance and sensitivity of the parameter space in Fig. 12. The value of the performance index is notionally indicated using a grayscale colormap of the circular markers and contour lines. The sensitivity is shown by arrows with lengths proportional to the magnitude of the gradient and orientated antiparallel to the gradient. The figure also shows the gains and sensitivity at each step of the optimization, similar to central path illustrations for interior point methods. The color along the curve indicates iteration number, using the same color map as Fig. 10. The magnitudes of both the performance metric colormap and sensitivity arrow lengths are shown with a log-scale to accommodate the wide dynamic range. The contour lines show the convexity of the performance index. Visual inspection of the gradient field suggests the flow is toward the optimum, as expected.

For the linearized cart and pendulum, we use $Q=I\in {\mathbb {R}}^{4\times 4}$ and $R=I\in {\mathbb {R}}^{1\times 1}$ for the performance index. Since this method assumes a meaningful trajectory $x\left( t\right)$, we initialize the optimization with stable feedback gains by solving the ARE with $Q=0\in {\mathbb {R}}^{4\times 4}$. The initial condition is $x\left( 0\right) =\left[ 1,-1,0,0\right] ^{T}$, and each simulation runs for 32 s. In Fig. 13, we show metrics over each iteration of the optimization and establish the colormap for the iteration number for this example. In this case, the gains found using the ARE marginally outperform the gradient-based optimizer, so the ARE solution forms the floor of Fig. 13a. The final value from the optimization algorithm is indicated with a full circle at the appropriate vertical axis position. Figure 14 shows the trajectories of the state, control, and co-state for each iteration.