On the Existence of Lipschitz Continuous Optimal Feedback Control

Abstract

We consider an optimal control problem involving a nonlinear ODE with control, an integral cost functional, and a control constraint. Our main assumptions include a coercivity condition and the condition that the optimal control is an isolated solution of the variational inequality appearing in the first-order optimality condition. We show that the optimal open-loop control is Lipschitz continuous in time; moreover, we identify the dependence of the Lipschitz constant of the optimal control on the data of the problem. Then, we establish the existence of a Lipschitz continuous optimal feedback control. As an application, we study regularity properties of the optimal value function. A main tool for obtaining these results is the property of uniform strong metric regularity.

1 Introduction

In this paper, we consider an optimal control problem for a time-dependent nonlinear control system over a fixed time interval [0, T] with an integral cost functional. The set of feasible controls consists of all functions in L^∞ (the space of measurable and essentially bounded functions over [0, T]) with values in a given convex and closed set in $\mathbb {R}^{m}$. We assume twice differentiability with respect to the state and the control of the functions involved in the problem and local Lipschitz continuity of these functions together with all their derivatives with respect to all arguments. We also assume the existence of a reference optimal solution. Since the reference optimal control is a function in L^∞, its values can be changed in a subset of [0, T] with Lebesgue measure zero without violating the optimality. In fact, the optimal control is a class of functions that differ from each other on a set of measure zero.

Our first task is to prove that, under an integral coercivity condition at the reference solution, we can select from the class of optimal controls a function which satisfies the first-order optimality condition for all t ∈ [0, T], instead of for almost every (a.e.) t ∈ [0, T]. Then, we show that under a coercivity condition this representative of the optimal controls is Lipschitz continuous with respect to time t ∈ [0, T] provided that it is an isolated solution of the Hamiltonian variational inequality in the first-order optimality condition. Moreover, we establish that the Lipschitz constant of the optimal control depends only on two constants: the coercivity constant and the Lipschitz constant of all functions defining the problem and their first and second derivatives over a bounded set in the space of variables (time, state, control).

The integral coercivity condition is a rather standard assumption in optimal control; the specific condition we use here goes back to the work of Hager [7]. In contrast, the isolatedness condition was introduced only recently in [2, Definition 3.6] in the context of the so-called differential variational inequalities, with the aim to prevent different solution curves from crossing each other. The isolatedness assumption is automatically satisfied when the Hamiltonian has a unique minimizer for each t ∈ [0, T], e.g., when the Hamiltonian is strictly convex. In [2, Theorem 4.1], it was established that if an optimal control $\bar u$ is an isolated solution of the Hamiltonian variational inequality and for each t ∈ [0, T] the mapping defining this variational inequality is strongly metrically regular at $\bar u$ for 0; then, the optimal control $\bar u$ is Lipschitz continuous on [0, T]. We also mention the earlier work [4] in that direction for an optimal control problem with linear dynamics and a strongly convex cost for which strong regularity holds automatically; in fact, only continuity of the optimal control is claimed there but the Lipschitz continuity can be gleaned from the proof. We note that the coercivity condition implies strong metric regularity in the respective function spaces, see [2, Theorem 4.2].

Our next task is to prove the existence of a Lipschitz continuous optimal feedback control. We show that under the coercivity and isolatedness conditions for the optimal control, there exists an optimal feedback control (τ, ξ)↦u^∗(τ, ξ) which is a Lipschitz continuous function; here (τ, ξ) represents the parametrizing pair initial time–initial condition.

Our third and last task is to show that the existence of a Lipschitz continuous optimal feedback control implies that the optimal value function (τ, ξ)↦V (τ, ξ) is differentiable with respect to ξ and its derivative is Lipschitz continuous.

An outline of the paper follows. In Section 2, we introduce the optimal control problem considered and set the stage for the further developments. Section 3 contains preliminary material showing in particular that the optimal control can be redefined on a set of measure zero so that the first-order optimality system holds for all t ∈ [0, T]. Section 4 gives conditions for Lipschitz continuity in time of the optimal open-loop control while Section 5 is devoted to the existence of a Lipschitz continuous optimal feedback control. The last Section 6 applies the latter result to show Lipschitz differentiability of the value function.

2 The Optimal Control Problem

We consider the following optimal control problem:

$$ \min \left\{J(u) := g(x(T)) + {{\int}_{0}^{T}} h(t,x(t),u(t)) \mathrm{d} t \right\}, $$

(1)

subject to

$$ \begin{array}{cc} & \dot x(t) = f\left( t,x(t),u(t)\right), \qquad x(0) = x_{0},\\ & x \in W^{1,\infty}, \quad u \in \mathcal{U} := \{u \in L^{\infty}~:~ u(t) \in U \text{ for a.e. } t \in [0,T] \}, \end{array} $$

(2)

where the state $x(t) \in \mathbb {R}^{n}$, the set U of feasible control values is a closed and convex subset of $\mathbb {R}^{m}$, and the functions $g :\mathbb {R}^{n} \to \mathbb {R}$, $h : [0,T] \times \mathbb {R}^{n} \times \mathbb {R}^{m} \to \mathbb {R}$ and $f : [0,T] \times \mathbb {R}^{n} \times \mathbb {R}^{m} \to \mathbb {R}^{n}$. The final time T and the initial state x₀ are fixed.

Throughout we assume that the function g is twice differentiable and its second derivative is locally Lipschitz continuous, the functions h(t,⋅,⋅) and f(t,⋅,⋅) are two times continuously differentiable (with respect to (x, u)), and these functions, together with all their derivatives, are locally Lipschitz continuous (with respect to (t, x, u)).

We also assume that problem (1)–(2) has a locally optimal solution $(\bar x, \bar u)$. The local optimality is understood in the following way: there exists a number e₀ > 0 such that for every $u \in \mathcal {U}$ with $\|u - \bar u\|_{\infty } \leq e_{0}$ either there is no solution of (2) over [0, T] or such a solution exists and $J(u) \geq J(\bar u)$.

In this paper, we employ the standard function spaces L^∞, L², W^{1, ∞}, W^1,2, all over [0, T]. Specifically, the space of controls u is L^∞, the space of measurable and essentially bounded functions. The state trajectory x is in W^{1, ∞}, the space of Lipschitz continuous functions. For the controls we also use the space L² of measurable square integrable functions, and for the state trajectory x the space W^1,2 such that both x and its derivative $\dot x$ are in L². Furthermore, for an element x of a metric space we denote by IB_a(x) (respectively ${\overset {\circ }{I\!\!B}}_{a}(y)$) the closed (respectively open) ball centered at x with radius a.

Clearly, any feasible control u is actually a class of functions which differ from each other on a set of Lebesgue measure zero. We call any particular function from this class a representative and denote it in the same way, by u.

Introducing the Hamiltonian H(t, x, u, λ) = h(t, x, u) + λ^⊤f(t, x, u), where ^⊤ means transposition, we employ the standard first-order necessary optimality condition (a consequence of the Pontryagin maximum principle) in the form used, e.g., in [7], according to which there exists a Lipschitz continuous function $\bar \lambda : [0,T] \to \mathbb {R}^{n}$ such that the triple $(\bar x, \bar u,\bar \lambda )$ satisfies for a.e. t ∈ [0, T] the following optimality system:

$$ \begin{array}{@{}rcl@{}} && - \dot x(t) + f(t,x(t),u(t)) = 0, \quad x(0) - x_{0} = 0, \\ && \dot \lambda (t) + H_{x}(t,x(t),u(t),\lambda(t)) = 0, \quad \lambda(T) - g_{x}(x(T)) = 0, \\ && H_{u}(t,x(t),u(t),\lambda(t)) + N_{U}(u(t)) \ni 0, \end{array} $$

(3)

where H_x denotes the derivative of H with respect to x, etc., and N_U is the normal cone mapping to the set U defined as

$$ u \mapsto N_{U}(u) = \left\{ \begin{array}{ll} \{y \in \mathbb{R}^{n} \mid \langle y, v-u \rangle \leq 0 \text{ for all } v \in U\} &~ \text{ if } u \in U,\\ \emptyset &~\text{ otherwise.} \end{array}\right. $$

In further lines, we give the following long but important remark, which summarizes various observations that will be used later on.

Remark 1

It is a standard fact that under our assumptions there exist positive reals d₀ and d such that for every $\tilde u \in \mathcal {U}$ with $\| \tilde u - \bar u\|_{\infty } \leq d$ and for every $\xi \in {I\!\!B}_{d_{0}}(x_{0})$ there exists a unique solution $\tilde x$ of the differential equation

$$ \dot x(t) = f(t,x(t),\tilde u(t)) \quad \text{ for a.e. }~t \in [0,T], \quad x(0) = \xi, $$

(4)

which satisfies $\|\tilde x - \bar x\|_{W^{1,\infty }} \leq 1$. Moreover, making d₀ and d smaller if necessary, we obtain that the (unique) solution $\tilde \lambda$ of the linear adjoint equation

$$ \dot \lambda (t) + H_{x}(t,\tilde x(t), \tilde u(t),\lambda(t)) = 0 \quad \text{for a.e. }~t \in [0,T], \quad \lambda(T) = g_{x}(\tilde x(T)) $$

(5)

satisfies $\|\tilde \lambda - \bar \lambda \|_{W^{1,\infty }} \leq 1$. Without loss of generality, we assume that d ≤ 1 and d ≤ e₀, where e₀ appears in the definition of local optimality given in the beginning of this section.

Since $\bar u \in L^{\infty }$, there exists a compact set $\bar U$ such that $\bar u(t) \in \bar U$ for a.e. t ∈ [0, T]. Define the set

$$ {\Omega} = \{(t,x,u,\lambda) ~:~ t \in [0,T],~\text{dist}(u, \bar U) \leq 1,~|x - \bar x(t)| \leq 1,~|\lambda - \bar \lambda(t)| \leq 1\}. $$

Denote by L the Lipschitz constant on Ω of each of the functions f, g, h, f_x, f_u, h_x, h_u, f_xx, f_xu, f_uu, h_xx, h_xu, h_uu, as well as of the functions H, H_x, H_u, H_xx, H_xu, H_uu. Since f and H_x are bounded in Ω, then $\dot {\tilde x}$ and $\dot {\tilde \lambda }$ are also bounded. Make L larger if needed so that for every $\tilde x$ and $\tilde \lambda$ that satisfy (4) and (5), respectively, the functions $(t,v) \mapsto H_{u}(t, \tilde x(t), v, \tilde \lambda (t))$ and $(t,v) \mapsto H_{uu}(t, \tilde x(t), v, \tilde \lambda (t))$ are Lipschitz continuous with constant L in the set $\{ (t,v) ~:~ t \in [0,T],~\text {dist}(v,\bar U) \leq 1\}$. This concludes Remark 1.

To shorten the notations, we skip arguments with “bar”, shifting the “bar” to the functions, e.g., $\bar H(t) := H(t,\bar x(t),\bar u(t),\bar \lambda (t))$, $\bar H(t,u) := H(t,\bar x(t),u,\bar \lambda (t))$, $\bar f(t) := f(t,\bar x(t),\bar u(t))$, $\bar g_{xx} := g_{xx}(\bar x(T))$, etc. Define the matrices

$$ A(t) = \bar f_{x}(t),~~B(t) = \bar f_{u}(t),~~Q(t) = \bar H_{xx}(t),~~S(t) = \bar H_{xu}(t),~~R(t) = \bar H_{uu}(t). $$

Our first main assumption is the following:

• Coercivity: there exists a constant ρ > 0 such that

  $$ \begin{array}{@{}rcl@{}} &&y(T)^{\top} \bar g_{xx} y(T) + {{\int}_{0}^{T}} \left( y(t)^{\top} Q(t)y(t) + w(t)^{\top} R(t)w(t) + 2y(t)^{\top} S(t)w(t)\right) \mathrm{d} t \\ &&\qquad\geq \rho {{\int}_{0}^{T}} |w(t)|^{2} \mathrm{d} t \end{array} $$

  (6)

  for all w ∈ L², y ∈ W^1,2 such that w(t) ∈ U − U, $\dot y(t) = A(t)y(t)+B(t)w(t)$ for a.e. t ∈ [0, T], and y(0) = 0.

The coercivity condition was first used in [7] to show convergence of the multiplier method and later in [5] to establish Lipschitz stability as well as convergence of discrete approximations in optimal control. It can be viewed as a strong second-order sufficient condition in optimal control. Checking this condition would very much depend on the specific problem at hand; sometimes it is enforced numerically by adding penalty terms to the cost. The coercivity condition has also been used for a posteriori numerical verification of optimality after an approximate solution is found.

In the following section, we present some preparatory material. In particular, we show that the coercivity condition implies a pointwise in time coercivity property which plays an important role in further analysis.

3 Preliminaries

Denote by meas(E) the Lebesgue measure of a set E. Let Δ ⊂ [0, T] be a measurable set with meas(Δ) > 0, and let $v: {\Delta } \to \mathbb {R}^{m}$ be a measurable and bounded function. For t ∈Δ denote by V_Δ(v;t) the set of points $w \in \mathbb {R}^{m}$ with the following property: there is a sequence of measurable sets E_k ⊂Δ such that

$$ \text{meas}(E_{k}) > 0, \quad E_{k} \subset [t - 1/k, t + 1/k], \quad \lim\limits_{k \to \infty} \sup\limits_{s \in E_{k}} | v(s) - w | = 0. $$

We denote V_{[0, T]}(v;t) simply by V (v;t).

A point t ∈Δ is said to be essentially non-isolated if for every ε > 0 the set [t − ε, t + ε] ∩Δ is of positive measure.

Lemma 1

Let Δ ⊂ [0, T] be a measurableset and let$v: {\Delta } \to \mathbb {R}^{m}$bea measurable and bounded function. Then, for anyt ∈Δ, thefollowing statements are equivalent:

• (i) V_Δ(v;t) ≠ ∅;

• (ii) t is essentially non-isolated point ofΔ.

Proof

If (i) holds, then the very definition of V_Δ(v;t) implies that t is essentially non-isolated.

Let us pick an essentially non-isolated point t of Δ. Let $K \subset \mathbb {R}^{m}$ be a compact set such that v(s) ∈ K for every s ∈Δ. Take an arbitrary w ∈ K. If for every ε > 0 and every natural number k there exists E_k ⊂ [t − 1/k, t + 1/k] ∩Δ such that meas(E_k) > 0 and $\sup _{s \in E_{k}} |v(s) - w | < \varepsilon$, then w ∈ V_Δ(v;t). If this is not the case, then there exist ε(w) > 0 and a natural number k(w) such that |v(s) − w|≥ ε(w) for a.e. s ∈ [t − 1/k(w), t + 1/k(w)] ∩Δ; that is, $v(s) \not \in {\overset {\circ }{I\!\!B}}_{\varepsilon (w)}(w)$ for a.e. s ∈ [t − 1/k(w), t + 1/k(w)] ∩Δ. If w∉V_Δ(v;t) for every w ∈ K, then, due to the compactness of K, there exist w₁,…, w_r ∈ K such that $K \subset \cup _{i=1}^{r} {\overset {\circ }{I\!\!B}}_{\varepsilon (w_{i})}(w_{i})$. Denote $\bar k := \max \{k(w_{1}), \dots , k(w_{r}) \}$; then, $v(s) \not \in \cup _{i=1}^{r} {\overset {\circ }{I\!\!B}}_{\varepsilon (w_{i})}(w_{i})$ for a.e. $s \in [t - 1/\bar k, t + 1/\bar k]\cap {\Delta }$. This contradicts the essential non-isolatedness of t, since $K \subset \cup _{i=1}^{r} {\overset {\circ }{I\!\!B}}_{\varepsilon (w_{i})}(w_{i})$ and meas$([t - 1/\bar k, t + 1/\bar k]\cap {\Delta }) > 0$. Hence V_Δ(v;t) ≠ ∅ and the proof is complete. □

Taking Δ = [0, T], we obtain that V (v;t) is non-empty for every t ∈ [0, T].

Lemma 2

Letu and$\tilde u$betwo measurable and bounded functions acting from[0, T] to$\mathbb {R}^{m}$, and letu(t) ∈ V (u;t) for everyt ∈ [0, T]. Then, the function$\tilde u$canbe redefined on a set of measure zero in such a waythat$\tilde u(t) \in V(\tilde u;t)$and$|\tilde u(t) - u(t)| \leq \|\tilde u - u\|_{\infty }$foreveryt ∈ [0, T].

Proof

Take an arbitrary t ∈ [0, T]. Consider first the case where both functions u and $\tilde u$ are approximately continuous at t. We recall that u is approximately continuous at t ∈ (0, T) if there exists a measurable set E ⊂ [0, T] containing t such that

$$ \lim\limits_{k \to \infty} 2k \text{meas}\left( E \cap [t - 1/k, t + 1/k]\right) = 1 $$

and the restriction of u to E is continuous. Let $\tilde E$ be the set in the definition of approximate continuity of $\tilde u$ at t ∈ (0, T). Then, the set ${E}_{k}^{\prime } := E \cap \tilde E \cap [t - 1/k, t + 1/k]$ satisfies $\lim _{k \to \infty } 2k \text {meas}({E}_{k}^{\prime }) = 1$. In particular, meas$({E}_{k}^{\prime }) >0$ for all sufficiently large k. Due to the continuity of u and $\tilde u$ on $E \cap \tilde E$, we have

$$ |\tilde u(t) - u(t)| \leq \lim\limits_{k} \left|\frac{1}{\text{meas}({E}_{k}^{\prime})} {\int}_{{E}_{k}^{\prime}} \left( \tilde u(s) - u(s)\right) \mathrm{d} s \right| \leq \|\tilde u - u\|_{\infty}. $$

Moreover, since the sets E_k in the definition of V can be replaced by ${E}_{k}^{\prime }$, we conclude that $\tilde u(t) \in V(\tilde u;t)$.

Now, let t ∈ [0, T] be such that u or $\tilde u$ is not approximately continuous at t, or t equals 0 or T.

We will now redefine $\tilde u(t)$ to fit the claim. It is well known (see, e.g., [8, Theorem 7.54]) that almost all t ∈ [0, T] are points of approximate continuity of both u and $\tilde u$; therefore we need to redefine $\tilde u$ only on a set of measure zero. Note that the sets $V(\tilde u;t)$ are invariant with respect to changes of $\tilde u$ on a set of measure zero.

Denote w := u(t) ∈ V (u;t). Let E_k be the sets in the definition of V. In particular, $\varepsilon _{k} := \sup _{s \in E_{k}} |u(s) - w| \stackrel {k}{\longrightarrow } 0$. Since E_k is of positive measure, it contains an essentially non-isolated point t_k ∈ E_k. According to Lemma 1, there exists $\tilde w_{k} \in V_{E_{k}}(\tilde u; t_{k})$; hence, there exists a sequence $\{{{E}_{k}^{i}}\}_{i}$, ${{E}_{k}^{i}} \subset E_{k}$, such that

$$ \text{meas}\left( {{E}_{k}^{i}}\right) > 0, \quad {{E}_{k}^{i}} \subset \left[t_{k} - 1/i, t_{k} + 1/i\right], \quad {\varepsilon_{k}^{i}} := \sup\limits_{s \in {{E}_{k}^{i}}} |\tilde u(s) - \tilde w_{k}| \stackrel{i}{\longrightarrow} 0. $$

Let $\tilde w$ be a cluster point of the sequence $\{\tilde w_{k}\}$. To show that $\tilde w \in V(\tilde u; t)$, we employ the following argument involving choosing a diagonal sequence. For an arbitrary natural number j, choose k = k_j so large that

$$ |t_{k_{j}} - t| \leq \frac{1}{2j} \quad\text{ and } \quad |\tilde w_{k_{j}} - \tilde w | \leq \frac{1}{j}. $$

Then, choose i = i_j such that

$$ \frac{1}{i_{j}} \leq \frac{1}{2j} \quad\text{ and } \quad \varepsilon_{k_{j}}^{i_{j}} \leq \frac{1}{j}. $$

We have

$$ \begin{array}{@{}rcl@{}} \tilde E_{j} := E_{k_{j}}^{i_{j}} & \subset &\left[t_{k_{j}} - 1/i_{j}, t_{k_{j}} + 1/i_{j} \right] \subset \left[t - |t-t_{k_{j}}| - 1/i_{j}, t + |t-t_{k_{j}}| + 1/i_{j} \right] \\ &\subset& [t - 1/j, t+ 1/j] \end{array} $$

and

$$ \sup\limits_{s \in \tilde E_{j}} |\tilde u(s) - \tilde w | \leq |\tilde w_{k_{j}} - \tilde w| + \sup\limits_{s \in \tilde E_{j}} |\tilde u(s) - \tilde w_{k_{j}} | \leq \frac{1}{j} + \varepsilon_{k_{j}}^{i_{j}} \leq \frac{2}{j}. $$

Taking also into account that meas$(\tilde E_{j}) > 0$, the last two relations imply that $\tilde w \in V(\tilde u; t)$.

For every k and i, we have ${{E}_{k}^{i}} \subset E_{k}$,

$$ \left|\frac{1}{\text{meas}({{E}_{k}^{i}})} {\int}_{{{E}_{k}^{i}}} u(s) \mathrm{d} s - w\right| = \left|\frac{1}{\text{meas}({{E}_{k}^{i}})} {\int}_{{{E}_{k}^{i}}} (u(s) - w )\mathrm{d} s \right| \leq \varepsilon_{k}, $$

and

$$ \left|\frac{1}{\text{meas}({{E}_{k}^{i}})} {\int}_{{{E}_{k}^{i}}} \tilde u(s) \mathrm{d} s - \tilde w_{k} \right| = \left|\frac{1}{\text{meas}({{E}_{k}^{i}})} {\int}_{{{E}_{k}^{i}}} (\tilde u(s) - \tilde w_{k} )\mathrm{d} s \right| \leq {\varepsilon_{k}^{i}}. $$

Hence,

$$ |\tilde w_{k} - w| \leq \left|\frac{1}{\text{meas}({{E}_{k}^{i}})} {\int}_{{{E}_{k}^{i}}} (\tilde u(s) - u(s)) \mathrm{d} s \right| + \varepsilon_{k} + {\varepsilon_{k}^{i}} \leq \| \tilde u - u \|_{\infty} + \varepsilon_{k} + {\varepsilon_{k}^{i}}. $$

Passing to the limit with i and then with k, we obtain $|\tilde w - w| \leq \| \tilde u - u \|_{\infty }$. Then, we redefine $\tilde u(t)$ as $\tilde u(t) = \tilde w$. This completes the proof. □

Corollary 1

Every$v \in \mathcal {U}$canbe redefined on a set of measure zero in such a way thatv(t) ∈ V (v;t) for everyt ∈ [0, T].

For a proof, apply Lemma 2 with $\tilde u = v$ and the constant function u(t) = u for all t ∈ [0, T].

Remark 2

From now on, the element $\bar u \in L^{\infty }$ will be identified with a function (denoted again by $\bar u$) satisfying $\bar u(t) \in V(\bar u;t)$ for every t ∈ [0, T].

Observe that the coercivity condition (6) does not depend on the particular representative of $\bar u$.

Lemma 3

Let the coercivity condition (6) hold, where$\bar u$isidentified as in Remark2. Then,

$$ w^{\top} R(t) w \geq \rho |w|^{2} \quad \text{ for every }~t \in [0,T] ~\text{ and }~w \in U - U. $$

(7)

Proof

Fix an arbitrary t ∈ [0, T]. Since $\bar u(t) \in V(\bar u;t)$, there exists a sequence E_k ⊂ [0, T] such that

$$ \text{meas}(E_{k}) > 0, \quad E_{k} \subset [t - 1/k, t + 1/k], \quad \varepsilon_{k} :=\sup\limits_{s \in E_{k}} | \bar u(s) - \bar u(t) | \to 0. $$

(8)

For an arbitrary w ∈ U − U, we define a function w_k as

$$ w_{k}(s) = \left\{ \begin{array}{ll} w & ~\text{ if } s \in E_{k}, \\ 0 & ~\text{ if } s \not\in E_{k}. \end{array}\right. $$

Using the Cauchy formula for the equation

$$ \dot y_{k}(s) = A(s) y_{k}(s) + B(s) w_{k}(s) \quad \text{ for a.e. }~s \in [0,T], \quad y(0) = 0, $$

we obtain that y_k(s) = 0 for s ∈ [0, t − 1/k] and |y_k(s)|≤ c₁meas(E_k) for s ∈ (t − 1/k, T], where here and further c₁, c₂,… are positive reals independent of k. Then, for the terms involved in (6), we have

$$ \begin{array}{@{}rcl@{}} &&\left|y_{k}(T)^{\top} g_{xx} y_{k}(T)\right| + \left|{{\int}_{0}^{T}} y_{k}(s)^{\top} Q(s)y_{k}(s) \mathrm{d} t \right| \leq c_{2} (\text{meas} (E_{k}))^{2},\\ &&\left| {{\int}_{0}^{T}} y_{k}(s)^{\top} S(s)w_{k}(s) \mathrm{d} t \right| = \left|{\int}_{E_{k}} y_{k}(s)^{\top} S(s) \mathrm{d} s w \right| \leq c_{3} (\text{meas} (E_{k}))^{2},\\ &&{{\int}_{0}^{T}} w_{k}(s)^{\top} R(s) w_{k}(s) \mathrm{d} s = w^{\top} {\int}_{E_{k}} R(s) \mathrm{d} s w,\\ &&{{\int}_{0}^{T}} |w_{k}(s)|^{2} \mathrm{d} s = \text{meas} (E_{k}) |w|^{2}. \end{array} $$

Since $R(s) = \bar H_{uu}(s,\bar u(s))$, using (8), we obtain (see Remark 1) that for s ∈ E_k one has

$$ |R(s) - R(t)| \leq L(|s-t| + \varepsilon_{k}) \leq L(1/k + \varepsilon_{k}) =: \tilde \varepsilon_{k} \to 0. $$

Using the above estimated in (6) and the above five displayed formulas, we obtain

$$ \text{meas} (E_{k}) w^{\top} R(t) w \geq \rho \text{meas} (E_{k}) |w|^{2} - c_{4} (\text{meas} (E_{k}))^{2} - c_{5} \text{meas} (E_{k}) \tilde \varepsilon_{k}. $$

Dividing by meas(E_k) (here we use the first inequality in (8)) and passing to the limit with k, we obtain (7). □

4 Lipschitz Continuity of the Optimal Control

Let us recall the optimality system (3):

$$ \begin{array}{@{}rcl@{}} - \dot x(t) + f(t,x(t),u(t)) &=& 0, \\ x(0) - x_{0} &=& 0, \\ \dot \lambda (t) + H_{x}(t,x(t),u(t),\lambda(t)) &=& 0, \\ \lambda(T) - g_{x}(x(T)) &=& 0, \\ H_{u}(t,x(t),u(t),\lambda(t)) + N_{U}(u(t)) &\ni& 0. \end{array} $$

(9)

Lemma 4

Let the coercivity condition hold. Then, the optimalcontrol$\bar u \in L^{\infty }$hasa representative$\bar u$suchthat the matrix$R(t) = \bar H_{uu}(t,\bar u(t))$satisfies(7) and$(\bar x(t), \bar u(t),\bar \lambda (t))$satisfies(9) for allt ∈ [0, T]. In fact, any representative of the optimal control thatsatisfies$\bar u(t) \in V(\bar u; t)$forallt ∈ [0, T] has this property.

Proof

Let us redefine $\bar u$ so that $\bar u(t) \in V(\bar u;t)$ for all t ∈ [0, T] (see Corollary 1 and Remark 2). Then, according to Lemma 3, the pointwise coercivity condition (7) holds for every t ∈ [0, T].

Fix an arbitrary t ∈ [0, T]. Since $\bar u(t) \in V(\bar u;t)$, there exists a sequence {E_k} of measurable subsets of [0, T] such that (8) holds. Since meas(E_k) > 0 and (9) is satisfied by $(\bar x(t), \bar u(t),\bar \lambda (t))$ almost everywhere, there exists t_k ∈ E_k such that (9) holds for t_k. From (8), we obtain that t_k → t and $\bar u(t_{k}) \to \bar u(t)$. Then, due to the continuity of the function $(t,u) \mapsto H_{u}(t,\bar x(t),u,\bar \lambda (t))$ and the upper semi-continuity of the mapping u↦N_U(u), (9) holds for t as well. □

We recall next the property of strong metric regularity of a general set-valued mapping $\mathcal {F} : \mathcal {Y} \rightrightarrows \mathcal {Z}$, where $\mathcal {Y}$ and $\mathcal {Z}$ are Banach spaces (for more on that, see, e.g., [6, Section 3.7]). A mapping $\mathcal {F}$ is said to be strongly metrically regular at $\hat y$ for $\hat z$ if there exist constants κ ≥ 0, a > 0 and b > 0 such that the truncated inverse mapping

$$ {I\!\!B}_{b}(\hat z) \ni z \mapsto \mathcal{F}^{-1}(z) \cap {I\!\!B}_{a}(\hat y) $$

is single-valued (a function) and Lipschitz continuous on ${I\!\!B}_{b}(\hat z)$. Here $\mathcal {F}^{-1}(z) = \{y \mid z \in \mathcal {F}(y)\}$.

Our further analysis is based on the following version of Robinson’s implicit function theorem. It was first stated as [6, Theorem 5G.3]¹ and then in corrected form as Theorem 3.2 in [2] (see also [3, Theorem 2.3] for a slight extension):

Theorem 1

Leta,b, andκbe positive scalars and let amapping$\mathcal {F}:\mathcal {Y} \rightrightarrows \mathcal {Z}$be strongly metrically regularat$\hat y$for$\hat z$withneighborhoods${I\!\!B}_{a}({\hat y})$and${I\!\!B}_{b}({\hat z})$andconstantκ. Letμ > 0 be suchthatκμ < 1 and letκ^′ > κ/(1 − κμ). Then, forevery positiveαandβsuchthat

$$ \alpha \leq a/2, \quad 2\mu \alpha +2 \beta \leq b \quad\text{ and }\quad 2\kappa^{\prime} \beta \leq \alpha $$

and for every function$g:\mathcal {Y} \to \mathcal {Z}$satisfying

$$ \|g({\hat y})\| \leq \beta \quad \text{ and } \quad \|g({y})-g({y^{\prime}})\| \leq \mu \|y - y^{\prime}\| \quad \text{ for every }~y, y^{\prime} \in {I\!\!B}_{2\alpha}({\hat y}), $$

the mapping$z \mapsto (g+\mathcal {F})^{-1}(z)\cap {I\!\!B}_{\alpha }(\hat y)$is a Lipschitz continuous function on${I\!\!B}_{\beta }(\hat z)$with Lipschitz constantκ^′.

Compared with the standard Robinson’s implicit function theorem, see [6, Theorem 2B.1], Theorem 1 exhibits the fact that everything hinges on the constants involved; that is, the constants of metric regularity of the perturbed mapping $g+\mathcal {F}$ do not depend on the actual perturbations but only on $\|g({\hat y})\|$, the Lipschitz constant of g and the constants of the strong regularity of $\mathcal {F}$. In that sense, Theorem 1 shows strong metric regularity which is uniform with respect to perturbations.

Let us get back to the optimal control problem at hand. If $(t,u) \in \text {cl} \text {gph}(\bar u),$ then there exists a sequence t_k → t such that $\bar u(t_{k}) \to u$. According to (7), we have

$$ w^{\top} \bar H_{uu}\left( t_{k},\bar u(t_{k})\right) w \geq \rho |w|^{2} \quad \text{for every }~w \in U - U. $$

Passing to the limit, we obtain that

$$ w^{\top} \bar H_{uu}(t,u) w \geq \rho |w|^{2} $$

(10)

for every $(t,u) \in \text {cl} \text {gph}(\bar u)$ and every w ∈ U − U. It is well known that the property (10) implies that for every $(t,u) \in \text {cl} \text {gph}(\bar u)$ the mapping

$$ v \mapsto \bar H_{u}(t,u) + \bar H_{uu}(t,u) (v-u) + N_{U}(v) $$

(11)

is strongly metrically regular at u for 0 with constants κ^′ = 1/ρ, a^′ = b^′ = +∞ (that is, with any positive a^′ and b^′), see, e.g., [7, Lemma 1]. Note that these constants are independent of t.

Next, we reformulate, adapted to our notations and needs, a simplified version of Theorem 3.5 in [2], which in turn is a corollary of Theorem 1.

Theorem 2

Assume that for every$(t,u) \in \text {cl} \text {gph}(\bar u)$themapping in (11) is strongly metrically regular atu for0 with constantsκ^′, a^′, b^′that are independent of (t, u). Then, for everyt ∈ [0, T], the mapping$u \mapsto \bar H_{u}(t,u) + N_{U}(u)$isstrongly metrically regular at$\bar u(t)$for0 with any constantsκ, a, b satisfying the inequalities

$$ a \leq \frac{a^{\prime}}{2},~~2 L^{\prime} a \kappa^{\prime} < 1,~~4 L^{\prime} a^{2} < b^{\prime},~~\frac{\kappa^{\prime}}{1- 2 L^{\prime} a \kappa^{\prime}} < \kappa,~~4L^{\prime} a^{2} + 2 b < b^{\prime},~~2\kappa b < a, $$

(12)

whereL^′is a Lipschitz constant of the mapping$u \mapsto \bar H_{uu} (t,u)$ on ${I\!\!B}_{a^{\prime }}(\bar u(t))$, for every t ∈ [0, T].

The conditions (12) are not stated in Theorem 3.5 in [2], but are explicitly written in the beginning of its proof there.

Continuing the analysis of (11), we apply Theorem 2 with a^′ = 1, b^′ = +∞ and κ^′ = 1/ρ, having the inequalities (12) reduced to

$$ a \leq \frac{1}{2}, \quad 2 L a < \rho, \quad \kappa > \frac{1}{\rho - 2 L a}, \quad 2 \kappa b < a, $$

(13)

where now L is the constant from Remark 1.

Remark 3

The important consequence of (13) is that the constants κ, a, b of strong regularity of $u \mapsto \bar H_{u}(t,u) + N_{U}(u)$ at $\bar u(t)$ for 0 can be chosen to depend only on the constant ρ in the coercivity condition (6) and the constant L in Remark 1.

We introduce next our second main assumption:

• Isolatedness: The function $\bar u$ (represented as in Lemma 4) is an isolated solution of the inclusion $\bar H_{u}(t,u) + N_{U}(u) \ni 0$ for all t ∈ [0, T], meaning that there exists a (relatively) open set $\mathcal {O} \subset [0,T] \times \mathbb {R}^{m}$ such that

  $$ \{ (t,u) \in [0,T] \times \mathbb{R}^{m}~:~ \bar H_{u}(t,u) + N_{U}(u) \ni 0 \} \cap \mathcal{O} = \text{gph}(\bar u). $$

  (14)

For example, the isolatedness assumption holds if for every t ∈ [0, T] the inclusion $\bar H_{u}(t,u) + N_{U}(u) \ni 0$ has a unique solution (which has to be $\bar u(t)$). In this case, one can verify the isolatedness condition taking any (relatively) open set $\mathcal {O} \subset [0,T] \times \mathbb {R}^{m}$ containing $\text {gph}(\bar u)$.

Theorem 3

Suppose that the isolatedness assumption (14) and condition (7) hold. Then, the optimalcontrol$\bar u$isLipschitz continuous on [0, T]. Moreover, the Lipschitz constant of$\bar u$dependsonly on the numberρin (7) and the constant L in Remark1.

Proof

The proof is somewhat parallel to the proof of Theorem 3.7 in [2]. Here we use Theorem 2 and (13) instead of the more general Theorem 3.5 in [2] (used in the proof of Theorem 3.7 in [2]), which does not imply the second claim of Theorem 3.

As mentioned around (10), condition (7) implies that for every $(t,u) \in \text {cl} \text {gph}(\bar u)$ the mapping in (11) is strongly metrically regular at u for 0. Then, we can apply Theorem 2. Let the numbers a, b, κ be chosen to satisfy conditions (13), so that for every t ∈ [0, T] the mapping $u \mapsto \bar H_{u}(t,u) + N_{U}(u)$ is strongly metrically regular at $\bar u(t)$ for 0 (see Theorem 2). Let L be the constant in Remark 1; then, the mappings $(t,u) \mapsto \bar H_{u}(t,u)$ and $(t,u) \mapsto \bar H_{uu}(t,u)$ are Lipschitz continuous with constant L on the set $\{ (t,u) ~:~ t \in [0,T],~u \in {I\!\!B}_{a}(\bar u(t))\}$. Without loss of generality we consider $\bar u$ as taking values in the set $\bar U$ in Remark 1; we also recall that a ≤ 1.

Take an arbitrary t ∈ [0, T]. Then, pick α_t < a/2 and then γ_t ∈ (0,1) such that $(\tau ,v) \in \mathcal {O}$ for every τ ∈ [t − γ_t, t + γ_t] ∩ [0, T] and $v \in {I\!\!B}_{\alpha _{t}}(\bar u(t))$, and also

$$ \kappa L \gamma_{t} < 1/2, \quad L (a + 2) \gamma_{t} \leq b, \quad 4 \kappa L \gamma_{t} \leq \alpha_{t} (1 - \kappa L \gamma_{t} ). $$

(15)

For an arbitrary τ ∈ [t − γ_t, t + γ_t] ∩ [0, T] define the mapping $g_{\tau ,t} : U \to \mathbb {R}^{m}$ as

$$ g_{\tau,t}(u) = \bar H_{u}(\tau,u) - \bar H_{u}(t,u), \quad u \in U. $$

Then, we have that

$$ |g_{\tau,t}(\bar u(t))| \leq L |\tau - t| \leq L \gamma_{t}, $$

(16)

and, for any $u,~u^{\prime } \in {I\!\!B}_{a}(\bar u(t))$,

$$ \begin{array}{@{}rcl@{}} |g_{\tau,t}(u) - g_{\tau,t}(u^{\prime})| &=& | \bar H_{u}(\tau,u) - \bar H_{u}(\tau,u^{\prime}) - \bar H_{u}(t,u) + \bar H_{u}(t,u^{\prime}) | \\ &\leq & {{\int}_{0}^{1}} | \bar H_{uu}(\tau,u^{\prime} + s(u-u^{\prime})) - \bar H_{uu}(t,u^{\prime} + s(u-u^{\prime}))|\mathrm{d} s | u - u^{\prime} | \\ &\leq& L \gamma_{t} | u - u^{\prime} |. \end{array} $$

Set

$$ \kappa^{\prime}_{t} = 2 \kappa/(1 - \kappa L \gamma_{t}), \quad \beta_{t} = \mu_{t} = L \gamma_{t}. $$

According to the inequality α_t < a/2, the inequalities in (15), and the definitions of $\kappa ^{\prime }_{t}$, α_t, β_t, the following inequalities are fulfilled (for convenience we skip the subscripts t for a moment):

$$ \mu > 0,~~\kappa \mu < 1,~~ \kappa^{\prime} > \kappa/(1-\kappa \mu),~~\alpha \leq a/2,~~2 \mu \alpha + 2 \beta \leq b,~~2 \kappa^{\prime} \beta \leq \alpha. $$

(17)

Now, we apply Theorem 1. For short, denote $G_{t}(u): = \bar H_{u}(t,u) + N_{U}(u)$. In our context all assumptions of the last theorem are satisfied with g = g_{τ, t}. Thus we obtain that the mapping

$$ {I\!\!B}_{\beta_{t}}(0) \ni z \mapsto (g_{\tau,t} + G_{t})^{-1}(z) \cap {I\!\!B}_{\alpha_{t}}(\bar u(t)) = (G_{\tau})^{-1}(z) \cap {I\!\!B}_{\alpha_{t}}(\bar u(t)) $$

is Lipschitz continuous with Lipschitz constant $\kappa ^{\prime }_{t} = 2 \kappa /(1 - \kappa L \gamma _{t}) \leq 4\kappa$ (see the first inequality in (15)). In particular, there exists a unique $v \in {I\!\!B}_{\alpha _{t}}(\bar u(t))$ such that 0 ∈ G_τ(v). Since τ ∈ [t − γ_t, t + γ_t] ∩ [0, T] and $v \in {I\!\!B}_{\alpha _{t}}(\bar u(t))$, we also have that $(\tau ,v) \in \mathcal {O}$. Due to isolatedness condition, we obtain that $v = \bar u(\tau )$. From (16), we obtain that $g_{\tau ,t}(\bar u(t)) \in {I\!\!B}_{\beta _{t}}(0)$. Thus

$$ \bar u(t) = (g_{\tau,t} + G_{t})^{-1} \left( g_{\tau,t}(\bar u(t))\right) \cap {I\!\!B}_{\alpha}(\bar u(t)). $$

Since $\bar u(\tau ) = (g_{\tau ,t} + G_{t})^{-1} (0) \cap {I\!\!B}_{\alpha }(\bar u(t))$, using (16), we get that

$$ |\bar u(t) - \bar u(\tau) | \leq \kappa^{\prime} |g_{\tau,t}(\bar u(t))| \leq 4\kappa L |t - \tau|. $$

Summarizing, we obtain that for every t ∈ [0, T] there exists a neighborhood (t − γ_t, t + γ_t) ∩ [0, T] in [0, T] in which $\bar u$ is Lipschitz continuous with the same constant 4κL. This implies that $\bar u$ is Lipschitz continuous with the same constant in the whole interval [0, T].

The second claim of the theorem follows from Remark 3 concerning κ. □

The example displayed in Remark 9 in [5] demonstrates that the isolatedness assumption (14) is essential for the Lipschitz continuity of the optimal control shown in Theorem 3. In this example h = (u² − 1)², g = 0, f = 0, $U = \mathbb {R}$, T = 1. Here, for each measurable set Ω ⊂ [0,1] the function defined as u(t) = − 1 for t ∈Ω and u(t) = 1 for t ∈ [0,1] ∖Ω is an optimal control, and the coercivity condition is satisfied. However, the isolatedness condition is satisfied only if the measure of Ω is either zero or 1. In these two cases the optimal control is Lipschitz continuous.

5 Lipschitz Continuous Optimal Feedback Control

In this section, we prove the existence of a Lipschitz continuous locally optimal feedback control for problem (1)–(2). For this purpose we embed the problem into a family of problems by replacing the initial time 0 with any τ ∈ [0, T] and the initial condition x(0) = x₀ with $x(\tau ) = \xi \in \mathbb {R}^{n}$. Denote this new family of problems by P(τ, ξ), so that P(0, x₀) is (1)–(2). Also, denote by J(τ, ξ;u) the value of the objective function of P(τ, ξ) for a control $u \in \mathcal {U}$ being defined as

$$ J(\tau,\xi;u) := g(x(T)) + {\int}_{\tau}^{T} h(t,x(t),u(t)) \mathrm{d} t, $$

where x is the solution of the initial-value problem

$$ \dot x(t) = f(t,x(t),u(t)) \quad \text{for a.e. }~t \in [\tau, T], \quad x(\tau) = \xi. $$

(18)

To set the stage, we give first the following definition which recasts the usual way a locally optimal feedback control is understood. Recall that $(\bar x, \bar u)$ is a locally unique solution of problem (1)–(2).

Definition 1

The function $u^{\ast }: [0,T] \times \mathbb {R}^{n} \to U$ is said to be a locally optimal feedback control around the reference solution pair $(\bar x, \bar u)$ if there exist positive numbers ε₀ and $\bar a$, and a set ${\Gamma } \subset [0,T] \times \mathbb {R}^{n}$ such that

• (i) $\text {gph}(\bar x) + \{0\}\times {I\!\!B}_{\varepsilon _{0}}(0) \subset {\Gamma }$;

• (ii) for every (τ, ξ) ∈Γ the equation

  $$ \dot x(t) = f(t,x(t),u^{\ast}(t,x(t))), \quad x(\tau) = \xi, $$

  (19)

  has a unique absolutely continuous solution $\hat x[\tau ,\xi ]$ on [τ, T] which satisfies $\text {gph}(\hat x[\tau ,\xi ]) \subset {\Gamma }$;

• (iii) the function $\hat u[\tau ,\xi ](\cdot ) := u^{\ast }(\cdot ,\hat x[\tau ,\xi ](\cdot ))$ is measurable, bounded, and satisfies $\|\hat u[\tau ,\xi ] - \bar u\|_{\infty } \leq \bar a$, and $J(\tau ,\xi ;\hat u[\tau ,\xi ]) \leq J(\tau ,\xi ;u)$, where u is any admissible control on [τ, T] with $\| u - \bar u\|_{\infty } \leq \bar a$ for which the corresponding solution x of (19) exists on [τ, T] and satisfies gph(x) ⊂Γ;

• (iv) $u^{\ast }(\cdot ,\bar x(\cdot )) = \bar u(\cdot )$.

The main result of this section follows.

Theorem 4

Let the coercivity condition (6) and the isolatedness condition(14) hold. Then, there exists a locally optimal feedbackcontrol$u^{\ast }: [0,T] \times \mathbb {R}^{n} \to U$around$(\bar x, \bar u)$whichis Lipschitz continuous on the set Γ appearing (together with the positive numbersε₀and$\bar a$)in Definition1.

Let us first sketch the idea of the proof. First, we prove that for ξ close to $\bar x(\tau )$ a unique solution $(\bar x[\tau ,\xi ],\bar u[\tau ,\xi ])$ exists and it is close to the restriction of $(\bar x, \bar u)$ to [τ, T]; moreover, $\bar u[\tau ,\xi ]$ depends in a Lipschitz way on ξ (in the space L^∞). Then, we show that $\bar u[\tau ,\xi ]$ is Lipschitz continuous.

For any τ ∈ [0, T), we define the spaces

$$ Y_{\tau} = W^{1,\infty} \times L^{\infty} \times W^{1,\infty}, \qquad Z_{\tau} = L^{\infty} \times \mathbb{R}^{n} \times L^{\infty} \times \mathbb{R}^{n} \times L^{\infty}, $$

where the time interval for these functional spaces is [τ, T]. It is convenient to define the norm in Y_τ as ∥(x, u, λ)∥ := max{∥x∥_{1, ∞}, ∥u∥_∞, ∥λ∥_{1, ∞}}. For any fixed τ ∈ [0, T), any (locally) optimal solution-multiplier triple y := (x, u, λ) ∈ Y_τ for P($\tau ,\bar x(\tau )$) satisfies the inclusion

$$ F_{\tau}(y) + G_{\tau}(y) \ni 0, $$

(20)

where F_τ : Y_τ → Z_τ and $G_{\tau }: Y_{\tau } \rightrightarrows Z_{\tau }$ are defined as

$$ F_{\tau}(y) = \left( \begin{array}{c} - \dot x + f(\cdot,x,u) \\ x(\tau) - \bar x(\tau) \\ \dot \lambda + H_{x}(\cdot,x,u,\lambda) \\ \lambda(T) - g_{x}(x(T)) \\ H_{u}(\cdot,x,u,\lambda) \end{array} \right), \qquad G_{\tau}(y) = \left( \begin{array}{c} 0 \\ 0 \\ 0 \\ 0 \\ N_{\mathcal{U}}^{\infty} (u) \end{array} \right). $$

Here

$$ N_{\mathcal{U}}^{\infty} (u) := \{ v \in L^{\infty} ~:~ v(t) \in N_{U}(u(t)) ~\text{ for a.e. }~ t \in [\tau,T]\}. $$

By using the superscript ∞ in the notation of the latter set we emphasize that the cone $N_{\mathcal {U}}^{\infty } (u)$ includes only a part of the normal cone $N_{\mathcal {U}}(u)$ which is a subset of the dual space of L^∞; note that the dependence on τ is not indicated.

Proposition 1

Let the coercivity condition (6) hold. Then, the mappingF_τ + G_τis strongly metrically regular at the restrictionof$\bar y:= (\bar x,\bar u,\bar \lambda )$to[τ, T] (denoted in the same way) for 0. Moreover, the constants of strong regularity, callthem$\bar \kappa$, $\bar a$, $\bar b$, can be chosen independent ofτ.

Proof

The strong metric regularity of the mapping F_τ follows from [5, Theorem 5], with the only difference that in [5] there is no terminal term in the cost functional and the functions h and f do not depend on time t. As is well known, under the smoothness conditions imposed the problem with a terminal cost can be transformed into an equivalent problem without a terminal cost. In addition, the time-dependent problem is handled in exactly the same way as the time-invariant; thus, the difference is basically formal. For reader’s convenience, below we outline the proof by highlighting the main steps and utilizing Theorem 1 as a shortcut.

First, observe that the coercivity condition (6) is fulfilled for problem P($\tau ,\bar x(\tau )$) with the same constant ρ for all τ. To show this, it is enough to take w(t) = 0 on [0, τ) in (6). The next step is to linearize the generalized (20) at $\bar y = (\bar x, \bar u, \bar \lambda )$, obtaining

$$ F_{\tau} (\bar y) +{\mathcal{A}}_{\tau}(y- \bar y) + G_{\tau}(y) \ni 0, $$

(21)

where

$$ \mathcal{A}_{\tau} (y) = \left( \begin{array}{c} -\dot x + Ax + Bu\\ x(\tau) - \bar x(\tau) \\ \dot \lambda + Qx +Su \\ \lambda(T) - \bar{g}_{xx}x(T) \\ S^{T}x + Ru \end{array} \right). $$

The strong regularity of the mapping appearing in the linearization (21), say with constants κ, a, b independent of τ is established in [7, Lemma 3] (with the caveat concerning the terminal cost and the dependence on t). Consider the function

$$ g_{\tau}(y) = -F_{\tau}(y) + F_{\tau} (\bar y) + \mathcal{A}_{\tau}(y- \bar y). $$

Then, $g_{\tau }(\bar y) = 0$. Since $\mathcal {A}_{\tau }$ is the strict derivative (in L^∞) of F_τ at $\bar y$, the Lipschitz modulus of g_τ at $\bar y$ is zero. Thus, in the notation of Theorem 1, taking α sufficiently small one can make μ arbitrarily close to zero; furthermore, κ^′ and β could be chosen accordingly to satisfy (17). It remains to put $\bar k = k^{\prime }$, $\bar a = \alpha$, $\bar b = \beta$ and to observe that these constants are independent of τ. □

As a consequence of the last proposition, for any $\xi \in {I\!\!B}_{\bar b}(\bar x(\tau ))$ the inclusion F_τ(y) + G_τ(y) + ζ ∋ 0 with ζ = (0,−ξ,0,0,0) has a unique solution $(\bar x[\tau ,\xi ],\bar u[\tau ,\xi ], \bar \lambda [\tau ,\xi ])$ in ${I\!\!B}_{\bar a}((\bar x,\bar u,\bar \lambda ))$ and it is Lipschitz continuous with respect to $\xi \in {I\!\!B}_{\bar b}(\bar x(\tau ))$ with Lipschitz constant $\bar \kappa$ in the norms of $\mathbb {R}^{n}$ and Y_τ.

Clearly, the constant $\bar b$ can be decreased, if necessary, without affecting the strong regularity property. Then, we may assume that $\bar b > 0$ is chosen so small that the coercivity assumption (6) adapted to problem P(τ, ξ) with $\xi \in {I\!\!B}_{\bar b}(\bar x(\tau ))$ holds with a constant ρ/2 (instead of ρ). Here and further “adapted” means that the matrices A, B, Q, R, S are calculated along $(\bar x[\tau ,\xi ],\bar u[\tau ,\xi ],\bar \lambda [\tau ,\xi ])$ instead of $(\bar x,\bar u,\bar \lambda )$ and the integration in (6) is on [τ, T]. Since under the coercivity assumption, the necessary optimality condition (3) is also sufficient (for local optimality), we obtain the following proposition.

Proposition 2

Let the coercivity condition and isolatedness condition hold. Then, forany$\xi \in {I\!\!B}_{\bar b}(\bar x(\tau ))$, the pair$(\bar x[\tau ,\xi ],\bar u[\tau ,\xi ])$definedin the second to last paragraph is the unique locally optimal solution of problemP(τ, ξ),in the set${I\!\!B}_{\bar a}((\bar x,\bar u))$. Moreover, the function${I\!\!B}_{\bar b}(\bar x(\tau )) \ni \xi \mapsto \bar u[\tau ,\xi ]$isLipschitz continuous in the norm ofL^∞with Lipschitz constant$\bar \kappa$.

It is important to note that assuming $\bar b$ small enough we may guarantee that Remark 1 is still valid with e₀, d₀ and d replaced with e₀/2, d₀/2 and d/2, respectively, and for the interval [τ, T] and the function $\bar u[\tau ,\xi ]$, $\xi \in {I\!\!B}_{\bar b}(\bar x(\tau ))$, instead of [0, T] and $\bar u$. The constant L remains the same.

As already mentioned, the coercivity assumption, adapted to problem P(τ, ξ), would hold (with ρ/2 instead of ρ) provided that $\xi \in {I\!\!B}_{\bar b}(\bar x(\tau ))$. According to Lemma 4, an arbitrary redefinition of $\bar u[\tau ,\xi ]$ on a set of measure zero which satisfies $\bar u[\tau ,\xi ](t) \in V_{[\tau ,T]}(\bar u[\tau ,\xi ];t)$ for every t ∈ [τ, T] (and such exists due to Corollary 1) fulfills the conditions that the matrix R(t) satisfies (7), and $\bar y[\tau ,\xi ]$ satisfies (9) for every t ∈ [τ, T], all adapted to problem P(τ, ξ). Moreover, according to Lemma 2, $\bar u[\tau ,\xi ]$ can be assumed to satisfy $|\bar u[\tau ,\xi ](t) - \bar u(t)| \leq \|\bar u[\tau ,\xi ] - \bar u\|_{\infty }$ for every t ∈ [τ, T].

Lemma 5

Let the coercivity condition and the isolatedness condition hold. Then, thereexists a number$\varepsilon \in (0, \bar b]$suchthat for everyτ ∈ [0, T) and$\xi \in {I\!\!B}_{\varepsilon }(\bar x(\tau ))$thecontrol$\bar u[\tau ,\xi ]$satisfiesthe isolatedness condition (on [τ, T]),namely there exists a (relatively) open set$\mathcal {O} \subset [\tau ,T] \times \mathbb {R}^{m}$suchthat

$$ \begin{array}{@{}rcl@{}} &&\{(t,u) \in [\tau,T] \times \mathbb{R}^{m}~:~ H_{u}(t,\bar x[\tau,\xi](t),u,\bar \lambda[\tau,\xi](t)) + N_{U}(u) \ni 0 \} \cap \mathcal{O}\\ &&= \text{gph}(\bar u[\tau,\xi]). \end{array} $$

(22)

Proof

Let us take ε > 0 so small that

$$ \varepsilon \leq \bar b, \qquad L\left( 2\bar\kappa + \bar L + 1 \right) \varepsilon < \rho, $$

(23)

where $\bar L$ is the Lipschitz constant of $\bar u$ (see Theorem 3). For arbitrarily fixed τ ∈ [0, T) and $\xi \in {I\!\!B}_{\varepsilon }(\bar x(\tau ))$ denote (for short) $\tilde x(t) = \bar x[\tau ,\xi ](t)$, $\tilde u = \bar u[\tau ,\xi ]$ redefined as described before the statement of the lemma, $\tilde \lambda (t) = \bar \lambda [\tau ,\xi ](t)$ and $\tilde y = (\tilde x, \tilde u, \tilde \lambda )$. Also denote $\tilde R(t) = H_{uu}(t,\tilde y(t))$, $\tilde H_{u}(t,u) = H_{u}(t,\tilde x(t),u,\tilde \lambda (t))$. Then, due to the first inequality in (23) and the redefinition of $\tilde u$, we know that for every t ∈ [τ, T]

$$ \begin{array}{@{}rcl@{}} && |\tilde u(t) - \bar u(t)| \leq \| \tilde u - \bar u\|_{\infty}, \quad \tilde H_{u}(t,\tilde u(t)) + N_{U}(\tilde u(t)) \ni 0, \end{array} $$

(24)

$$ \begin{array}{@{}rcl@{}} && w^{\top} \tilde R(t) w \geq \frac{\rho}{2} |w|^{2} \quad \forall w \in U - U. \end{array} $$

(25)

Let us define

$$ \mathcal{O} = \left( \text{gph}(\tilde u) + (-\varepsilon,\varepsilon) \times {\overset{\circ}{I\!\!B}}_{\varepsilon}(0) \right) \cap ([\tau,T) \times \mathbb{R}^{m}), $$

which is relatively open in $[\tau ,T] \times \mathbb {R}^{m}$. We shall prove that the claim of the lemma holds with this set $\mathcal {O}$.

Note that the right side of (22) is contained in the left side; thus, it is sufficient to prove the opposite inclusion. Targeting a contradiction, let us assume that there exists a point

$$ (t_{0},u_{0}) \in \{ (t,u) \in [\tau,T] \times \mathbb{R}^{m} ~:~ \tilde H_{u}(t,u) + N_{U}(u) \ni 0 \} \cap \mathcal{O} $$

(26)

which is not in $\text {gph}(\tilde u)$. Then, $\tilde u(t_{0}) \not =u_{0}$. From (26) and the second relation in (24), we have

$$ \tilde H_{u}(t_{0},u_{0}) + N_{U}(u_{0}) \ni 0, \qquad \tilde H_{u}(t_{0},\tilde u(t_{0})) + N_{U}(\tilde u(t_{0})) \ni 0. $$

From here

$$ \tilde H_{u}(t_{0},u_{0}) (\tilde u(t_{0}) - u_{0}) \geq 0, \qquad \tilde H_{u}(t_{0},\tilde u(t_{0}))(u_{0} - \tilde u(t_{0})) \geq 0, $$

which implies

$$ (\tilde H_{u}(t_{0},u_{0}) - \tilde H_{u}(t_{0},\tilde u(t_{0}))) (u_{0} - \tilde u(t_{0})) \leq 0. $$

Then, using (25) (notice that u₀ ∈ U, since otherwise N_U(u₀) = ∅), we obtain

$$ \begin{array}{@{}rcl@{}} 0 &\geq& \left( \tilde H_{u}(t_{0},u_{0}) - \tilde H_{u}(t_{0},\tilde u(t_{0}))\right) (u_{0} - \tilde u(t_{0})) \\ &=& {{\int}_{0}^{1}} \frac{\mathrm{d}}{\mathrm{d} s} \tilde H_{u}(t_{0}, \tilde u(t_{0}) + s (u_{0} - \tilde u(t_{0}))) \mathrm{d} s (u_{0} - \tilde u(t_{0})) \\ &=& {{\int}_{0}^{1}} (u_{0} - \tilde u(t_{0}))^{\top} \tilde H_{uu}(t_{0}, \tilde u(t_{0}) + s (u_{0} - \tilde u(t_{0}))) (u_{0} - \tilde u(t_{0})) \mathrm{d} s \\ &\geq& {{\int}_{0}^{1}} (u_{0} - \tilde u(t_{0}))^{\top} \tilde R(t_{0}) (u_{0} - \tilde u(t_{0}))^{\top} \mathrm{d} s - {{\int}_{0}^{1}} sL |u_{0} - \tilde u(t_{0})|^{3} \mathrm{d} s\\ &\geq& \frac{\rho}{2} |u_{0} - \tilde u(t_{0})|^{2} - \frac{L}{2} |u_{0} - \tilde u(t_{0})|^{3}. \end{array} $$

Hence,

$$ \rho \leq L |\tilde u(t_{0}) - u_{0}|. $$

Due to the inclusion $(t_{0},u_{0}) \in \mathcal {O}$, there exists $(t_{1},u_{1}) \in \text {gph}(\tilde u)$ such that |t₁ − t₀|≤ ε, |u₁ − u₀|≤ ε. Then, continuing the inequality (26), we obtain

$$ \begin{array}{@{}rcl@{}} \rho &\leq& L\left( |\tilde u(t_{0}) - \bar u(t_{0})| + |\bar u(t_{0}) - \bar u(t_{1})| + |\bar u(t_{1}) - \tilde u(t_{1})| + |\tilde u(t_{1}) - u_{0}|\right) \\ &\leq& L\left( \|\tilde u - \bar u \|_{\infty} + \bar L |t_{0} - t_{1}| + \| \tilde u - \bar u \|_{\infty} + \varepsilon\right) \leq L\left( 2\bar\kappa + \bar L + 1 \right) \varepsilon. \end{array} $$

This last inequality contradicts (23). Hence, (22) holds. □

Having proved that the isolatedness condition is also fulfilled for problem P(τ, ξ), we can apply Theorem 3 to this problem and obtain that the (locally) optimal control $\bar u[\tau ,\xi ]$ is Lipschitz continuous. The Lipschitz constant, $\bar L$, depends on the problem only through the constant ρ (now ρ/2) and the constant L, therefore can be chosen independent of τ and ξ, provided that $|\xi - \bar x(\tau )| \leq \varepsilon$, where ε > 0 is sufficiently small (independent of τ).

Proof

of Theorem 4 Let the number $\varepsilon \in (0, \bar b]$ be the one from Lemma 5. There exists a number ε₀ ∈ (0, ε] such that for every τ ∈ [0, T] and $\xi \in {I\!\!B}_{\varepsilon _{0}}(\bar x(\tau ))$ the corresponding $(\bar x[\tau ,\xi ], \bar u[\tau ,\xi ])$ exists on [τ, T] and satisfies, $| \bar x[\tau ,\xi ](t) - \bar x(t) | \leq \varepsilon$ for every t ∈ [τ, T], and $\|\bar u[\tau ,\xi ] - \bar u \|_{\infty } < \bar a$. Such an ε₀ exists because the mapping ${I\!\!B}_{\bar b} \ni \xi \mapsto (\bar x[\tau ,\xi ],\bar u[\tau ,\xi )]$ is Lipschitz continuous in W^{1, ∞}× L^∞ with constant $\bar \kappa$.

Define the set

$$ {\Gamma} = \{ \bar x[\tau,\xi](t) ~:~ \tau \in [0,T],~\xi \in {I\!\!B}_{\varepsilon_{0}}(\bar x(\tau)),~ t \in [\tau,T]\}. $$

(27)

Clearly, $\text {gph}(\bar x) + \{0\} \times {I\!\!B}_{\varepsilon _{0}}(0) \subset {\Gamma }$. For τ ∈ [0, T] denote

$$ {\Gamma}_{\tau} := \{ \xi ~:~ (\tau,\xi) \in {\Gamma} \}. $$

Since for every τ ∈ [0, T] we have ${\Gamma }_{\tau } \subset B_{\varepsilon }(\bar x(\tau ))$, the function ${\Gamma }_{\tau } \ni \xi \mapsto \bar u[\tau ,\xi ]$ is Lipschitz continuous with Lipschitz constant $\bar \kappa$. Moreover, each of these functions $\bar u[\tau ,\xi ]$ is Lipschitz continuous on [τ, T] with Lipschitz constant $\bar L$. In addition, according to Proposition 2, for every τ ∈ [0, T) and ξ ∈Γ_τ, the function $\bar u[\tau ,\xi ]$ is the unique locally optimal control for the corresponding problem P(τ, ξ) in the set ${I\!\!B}_{\bar a}(\bar u)$.

Now, define feedback control mapping x↦u^∗(⋅, x) as

$$ u^{\ast}(t,x) := \bar u[t,x](t) \quad \text{ for }~(t,x) \in {\Gamma}. $$

(28)

The values $\bar u[t,x](t)$ are well defined since $\bar u[t,x]$ is a (Lipschitz) continuous function. Clearly, all requirements of Definition 1 are satisfied; the last one following from the identity $\bar u[t,\bar x(t)] = \bar u(t)$.

Let us consider two arbitrary pairs (τ, ξ),(s, η) ∈Γ. Due to the Dynamic Programming Principle, for every s ≥ τ, τ, s ∈ [0, T) and every t ∈ [s, T], we have

$$ \bar u[\tau,\xi](t) = \bar u[s,\bar x[\tau,\xi](s)](t). $$

Then,

$$ \begin{array}{@{}rcl@{}} |u^{\ast}(\tau,\xi) - u^{\ast}(s,\eta)| &=& |\bar u[\tau,\xi](\tau) - \bar u[s,\eta](s)| \\ &\leq& | \bar u[\tau,\xi](\tau) - \bar u[\tau,\xi](s) | + | \bar u[\tau,\xi](s) - \bar u[s,\eta](s) | \\ &\leq& \bar L |s-\tau| + | \bar u[s,\bar x[\tau,\xi](s)](s) - \bar u[s,\eta](s) | \\ &\leq& \bar L |s-\tau| + \bar \kappa |\bar x[\tau,\xi](s) - \eta| \\ &\leq& \bar L |s-\tau| + \bar \kappa \left( |\bar x[\tau,\xi](s) - \xi| + | \xi-\eta |\right) \\ &\leq& \bar L |s-\tau| + \bar \kappa \left( M |s-\tau| + | \xi-\eta |\right), \end{array} $$

where M is an upper bound of |f(t, x, u)| in the set Ω defined in Remark 1. This completes the proof of Theorem 4.

□

Remark 4

The last part of the proof and the uniqueness claim in Proposition 2 imply that the function $\hat u[\tau ,\xi ]$ appearing in Definition 1 is the unique locally optimal control for in problem P(τ, ξ) in the set ${I\!\!B}_{\bar a}(\bar u)$.

6 Regularity of the Value Function

In this section, we show that the existence of a Lipschitz continuous optimal feedback established in Theorem 4 implies certain smoothness properties of the value function. In the preceding sections we assume only local optimality at the reference point, see Definition 1. In line with that assumption, we introduce the following definition:

Definition 2

The function $V: [0,T] \times \mathbb {R}^{n} \to \mathbb {R}$ is said to be a local value function of problem (1)–(2) around a reference admissible pair $(\bar x, \bar u)$ if there exist positive numbers ε₀ and $\bar a$, and a set ${\Gamma } \subset [0,T] \times \mathbb {R}^{n}$ such that $\text {gph}(\bar x) + \{0\} \times {I\!\!B}_{\varepsilon _{0}}(0) \subset {\Gamma }$ and for every (τ, ξ) ∈Γ one has

$$ V(\tau,\xi) = \inf\limits_{u} J(\tau,\xi;u), $$

where the infimum of the objective function J(τ, ξ;u) in problem P(τ, ξ) is taken over all admissible pairs (x, u) for which $\|u - \bar u\|_{\infty } \leq \bar a$, (18) has a unique solution x on [τ, T], and gph(x) ⊂Γ.

By this definition, the local value function, with a set Γ and a neighborhood ${I\!\!B}_{\bar a}(\bar u)$, is finite if for every (τ, ξ) ∈Γ there exists at least one admissible pair (x, u) satisfying $\| u - \bar u \|_{\infty } \leq \bar a$ and gph(x) ∈Γ. Clearly, in that case $(\bar x, \bar u)$ is a locally optimal solution.

As in Section 5, we denote Γ_τ := {ξ : (τ, ξ) ∈Γ}. Thus, the condition $\text {gph}(\bar x) + \{0\} \times {I\!\!B}_{\varepsilon _{0}}(0) \subset {\Gamma }$ in Definition 2 means that ${I\!\!B}_{\varepsilon _{0}}(\bar x(t)) \subset {\Gamma }_{t}$ for every t ∈ [0, T]. We also denote $\overset {\circ }{\Gamma } = \{(\tau ,\xi ) ~:~ \tau \in [0,T],~\xi \in \text {int}({\Gamma }_{\tau })\}$.

Theorem 5

Let the coercivity condition and the isolatedness condition hold.Then, problem (1)–(2) has a (finite) local value function V around$(\bar x, \bar u)$(witha set Γ and parametersε₀and$\bar a$);moreoverV (τ,⋅) is differentiable with respect toξwhenever$(\tau ,\xi ) \in \overset {\circ }{\Gamma }$andthe derivativeV_ξis Lipschitz continuous on$\overset {\circ }{\Gamma }$.

Proof

The proof is routine, in principle, but we present it in full, because we deal here with a local value function, which requires some attention to detail. We will prove the theorem with Γ, ε₀ and $\bar a$ as in Theorem 4. Then, there is a locally optimal Lipschitz continuous feedback control u^∗ in the sense of Definition 1, with the corresponding pairs $(\hat x[\tau ,\xi ], \hat u(\tau ,\xi ])$. According to this definition, we have

$$ V(\tau, \xi) = g (\hat x[\tau,\xi](T)) + {\int}_{\tau}^{T} h(s, \hat x[\tau,\xi] (s), \hat u[\tau,\xi](s)) \mathrm{d} s. $$

(29)

First, we prove the following claim.

Claim A: for every $(\tau ,\xi _{0}) \in \overset {\circ }{\Gamma }$ there exists a number δ > 0 such that for every ξ, ξ^′∈ IB_δ(ξ₀) one has $\|\hat u[\tau ,\xi ] - \bar u\|_{\infty } \leq \bar a$ and the initial value problem

$$ \dot x(t) = f(t,x(t),\hat u[\tau,\xi](t)), \quad x(\tau) = \xi^{\prime}, $$

(30)

has a unique solution $x^{\xi ,\xi ^{\prime }}$ on [τ, T] and $\text {gph}(x^{\xi ,\xi ^{\prime }}) \subset {\Gamma }$. Note that $x^{\xi ,\xi } = \hat x[\tau ,\xi ]$.

We recall (see Definition 1) that $\hat x[\tau ,\xi ]$ is the solution of (18) with control $u^{\ast }(t,\hat x[\tau ,\xi ](t))$, where u^∗ is defined in (28). Since x↦u^∗(⋅, x) is Lipschitz continuous, we obtain by a standard argument that for any $(\tau ,\xi _{0}) \in \overset {\circ }{\Gamma }$ there is a constant $\delta _{0} \in (0, \bar a/\bar L]$ such that ${I\!\!B}_{\delta _{0}}(\hat x[\tau ,\xi _{0}](t)) \subset {\Gamma }_{t}$ for every t ∈ [τ, T]. Indeed, due to the Lipschitz continuity of the right-hand side of (18) every solution starting backwards from a point in a sufficiently small neighborhood of $\hat x[\tau ,\xi _{0}](t)$ at time t > τ takes values only in ${I\!\!B}_{\varepsilon _{0}}(\xi _{0})$ at time τ. From the definition of Γ in (27), the graph of each such trajectory in contained in Γ and $\|\hat u[\tau ,\xi ] - \bar u\|_{\infty } \leq \bar a$. Then, Claim A follows from (30) thanks to the Lipschitz continuity of $\hat u[\tau ,\xi ]$ in ξ (Proposition 2). The proof of that last assertion, e.g., by contradiction, is straightforward

In addition, we have the representation

$$ x^{\xi,\xi^{\prime}}(t) - \hat x[\tau,\xi](t) = {\Phi}[\tau,\xi](t,\tau)(\xi^{\prime}-\xi) + o(|\xi^{\prime}-\xi|),\quad t \in [\tau, T], $$

where Φ[τ, ξ](t, s) is the fundamental matrix solution of the linearization of the differential equation in (30) normalized at t = s, that is,

$$ \frac{\partial}{\partial t} {\Phi}(t,s) = f_{x} (t, \bar x[\tau,\xi](t),\bar u[\tau,\xi](t) ) {\Phi}(t,s), \quad {\Phi}(s,s) = \text{ the identity}. $$

In particular, there exists a constant C such that for every ξ, ξ^′∈ IB_δ(ξ₀) one has

$$ \left\|x^{\xi,\xi^{\prime}} - x^{\xi,\xi}\right\|_{\infty} = \left\|x^{\xi,\xi^{\prime}} - \hat x[\tau,\xi]\right\|_{\infty} \leq C |\xi^{\prime} - \xi|. $$

(31)

From Definition 2 and Claim A, we have that

$$ V(\tau,\xi^{\prime}) \leq g \left( x^{\xi,\xi^{\prime}}(T)\right) + {\int}_{\bar \tau}^{T} h\left( s, \hat x^{\xi,\xi^{\prime}}(s), \hat u[\tau,\xi](s)\right) \mathrm{d} s, $$

Utilizing this last inequality in (29), we obtain

$$ \begin{array}{@{}rcl@{}} V(\tau, \xi^{\prime}) - V(\tau,\xi)&\leq& g\left( x^{\xi,\xi^{\prime}} (T)\right) - g(\hat x[\tau,\xi](T)) \\ &&+ {\int}_{\tau}^{T} \left[h(s, x^{\xi,\xi^{\prime}} (s), \hat u[\tau,\xi](s)) - h(s, \hat x[\tau,\xi] (s), \hat u[\tau,\xi](s))\right] \mathrm{d} s \\ & =& g_{x}\left( \tilde x^{T}\right){\Phi}[\tau,\xi](T,\tau)(\xi^{\prime}-\xi) \\ && + {\int}_{\tau}^{T} h_{x}(s, \tilde x (s), \hat u[\tau,\xi](s)) {\Phi}[\tau,\xi](s,\tau)(\xi^{\prime}-\xi) \mathrm{d} s + o(|\xi^{\prime}-\xi|), \end{array} $$

where $\tilde x^{T} \in \text {co}\left \{x^{\xi ,\xi ^{\prime }}(T), \hat x[\tau ,\xi ](T)\right \}$ and $\tilde x(\cdot )$ is a measurable selection of the set-valued map $s \mapsto \text {co} \{x^{\xi ,\xi ^{\prime }}(s), \hat x[\tau ,\xi ](s)\}$. Due to the Lipschitz continuity of g_x and h_x, and (31), we obtain

$$ \begin{array}{@{}rcl@{}} && V(\tau, \xi^{\prime}) - V(\tau,\xi) \leq [g_{x}(\hat x[\tau,\xi](T)) {\Phi}[\tau,\xi](T,\tau) \\ && + {\int}_{\tau}^{T} h_{x}(s, \hat x[\tau,\xi](s), \hat u[\tau,\xi](s)) {\Phi}[\tau,\xi](s,\tau) \mathrm{d} s ] (\xi^{\prime}-\xi) + o(|\xi^{\prime}-\xi|). \end{array} $$

It is well-known that the expression in the brackets equals $(\hat \lambda [\tau ,\xi ](\tau ))^{\top }$, where, as before, $\hat \lambda [\tau ,\xi ]$ is the solution of the adjoint equation (9) for the reference pair $(\hat x[\tau ,\xi ], \hat u[\tau ,\xi ])$ and end-point condition $\hat \lambda (T) = g_{x}(\hat x[\tau ,\xi ](T))$. Hence,

$$ V(\tau, \xi^{\prime}) - V(\tau,\xi) \leq \langle \hat \lambda[\tau,\xi](\tau) , (\xi^{\prime}-\xi) \rangle + o(|\xi^{\prime}-\xi|). $$

(32)

Using this inequality with ξ^′ = ξ₀ and taking into account the Lipschitz continuity of $\hat \lambda [\tau ,\xi ]$ with respect to ξ, we obtain

$$ \begin{array}{@{}rcl@{}} V(\tau, \xi_{0}) - V(\tau,\xi) &\leq& \langle \hat \lambda[\tau,\xi](\tau) , \xi_{0}-\xi \rangle + o(|\xi_{0}-\xi|)\\ &=& \langle \hat \lambda[\tau,\xi_{0}](\tau) , \xi_{0}-\xi \rangle + o(|\xi_{0}-\xi|). \end{array} $$

Using (32) with ξ^′ := ξ and ξ = ξ₀, we obtain

$$ V(\tau, \xi) - V(\tau,\xi_{0}) \leq \langle \hat \lambda[\tau,\xi_{0}](\tau) , \xi-\xi_{0} \rangle + o(|\xi-\xi_{0}|). $$

Combining the last two inequalities, we obtain that V is differentiable with respect to ξ at ξ₀; furthermore, $V_{\xi }(\tau ,\xi _{0}) = \hat \lambda [\tau ,\xi _{0}](\tau )$. The Lipschitz continuity of V_ξ follows from the last expression and the Lipschitz continuity of the function $(\tau ,\xi ) \mapsto \hat \lambda [\tau ,\xi _{0}](\tau )$. □

Observe that Γ, ε₀ and $\bar a$ in this theorem can be taken to be those in the proof of Theorem 4. Also, observe that at the end of the last proof we obtained the equality $V_{\xi }(\tau ,\xi _{0}) = \hat \lambda [\tau ,\xi _{0}](\tau )$, which, as is well known, holds under various sets of assumptions. Moreover, based on Theorem 5, one can verify that if $\bar u$ is a globally optimal solution, then the value function V is a classical solution of the corresponding Hamilton-Jacobi-Bellman equation (see, e.g., [1, Chapter III.3]).