On Nonuniqueness of Solutions of Hamilton–Jacobi–Bellman Equations

An example of a nonunique solution of the Cauchy problem of Hamilton–Jacobi–Bellman (HJB) equation with surprisingly regular Hamiltonian is presented. The Hamiltonian H(t, x, p) is locally Lipschitz continuous with respect to all variables, convex in p and with linear growth with respect to p and x. The HJB equation possesses two distinct lower semicontinuous solutions with the same final conditions; moreover, one of them is the value function of the corresponding Bolza problem. The definition of lower semicontinuous solution was proposed by Frankowska (SIAM J. Control Optim. 31:257–272, 1993) and Barron and Jensen (Commun. Partial Differ. Equ. 15(12):1713–1742, 1990). Using the example an analysis and comparison of assumptions in some uniqueness results in HJB equations is provided.


Introduction
The classical Cauchy problem for the Hamilton-Jacobi-Bellman equation is a partial differential equation with the final condition The HJB equation is related to the value function of the Bolza problem V : [0, T ] × R n → R ∪ {+∞} which is defined as follows: 2) where A[t 0 , T ] denotes the space of absolutely continuous functions from [t 0 , T ] into R n . If the value function is differentiable, it is well-known that it satisfies HJB in the classical sense. However, in many situations the value function is not differentiable. Then the solution of the HJB equation must be defined in nonsmooth sense in such a way that under quite general assumptions on H and g, V is the unique solution of HJB. Since we use the nonsmooth analysis we need a notion of a subgradient. For a vector v ∈ R n and a function f : In 1990 Baron and Jensen [3] and Frankowska [7] introduced extended viscosity solutions to semicontinuous functions for Hamiltonian that is convex in the gradient variable and provided a uniqueness result. Frankowska [7] called these solutions lower semicontinuous solutions. (i) U is lower semicontinuous function and U (T, x) = g(x); (ii) For every (t, x) ∈ dom U , every ( p t , p x ) ∈ ∂U (t, x) the following holds: (1.4) The main goal of the paper is to present an example of two distinct lower semicontinuous solutions of the HJB equation with surprisingly regular Hamiltonian. Understanding the role that the Lipschitz-type condition plays in theorems about uniqueness of solution of HJB is also important. The proposed Hamiltonian H : [0, T ]×R × R → R satisfies the classical assumptions i.e. firstly, it is convex with respect to p, secondly, it increases linearly in p and x, i.e. |H (t, x, p)| 2| p| for any t ∈ [0, T ], x, p ∈ R, thirdly, it is locally Lipschitz continuous, i.e.
where B is the closed unit ball. In addition, we show that one of the indicated lower semicontinuous solutions is the value function given by (1.2). In general, for the uniqueness of HJB solutions, one needs some stronger Lipschitz-type condition, that we shall study further in connection to the results of uniqueness. Frankowska [7] proved that, the value function is the unique lower semicontinuous solution of the HJB equation if the Hamiltonian meets the classical assumptions and it is positively homogeneous in p, i.e. ∀ r >0 H (t, x, r p) = r H(t, x, p). Actually, the result of Frankowska does not require local Lipschitz continuity with respect to the triple (t, x, p). It is enough to assume it is satisfied with respect to state variable x only. The example of nonuniqueness of solution of HJB introduced in the current paper, does not contradict the result of Frankowska as the Hamiltonian in our example, fulfills the classical assumptions, but it is not positively homogeneous in p.
Earlier Ishii [11,Theorem 2.5] and Crandall and Lions [5, Theorem VI.1] had proved the uniqueness of viscosity solutions of HJB in the class of continuous functions. They had assumed instead of linear growth in p and x of Hamiltonian, the following condition that is derived from the optimal control problem. The meaning of (SLC) in the optimal control problems is discussed in the papers of Frankowska and Sedrakyan [8] and Rampazzo [18]. The results in [ In order to understand better the reason for nonuniqueness of the solution of the HJB equation given in our example, we need to recall the Loewen-Rockafellar condition from [12] that is more general version of (SLC). This condition was used by Loewen and Rockafellar [13,14] to study necessary conditions satisfied by optimal solutions of Bolza problem. Galbraith [9] proved that, the value function is the unique lower semicontinuous solution of HJB assuming the Loewen-Rockafellar condition [12]. This uniqueness result has, in general, the same nature as the uniqueness result of Dal Maso and Frankowska [6]. The Hamiltonian from our example of nonuniqueness is constructed using the function ϕ. In Sect. 6 we show that if ϕ is replaced by a function that is sufficiently regular, then we obtain the Hamiltonian that does not satisfy the condition (SLC), while it satisfies the Loewen-Rockafellar condition [12]. Therefore, by virtue of classical results, we are not able to say if after the mentioned change, we get the uniqueness of the solution of HJB or not. However, using the Galbraith result [9], we know that HJB with this Hamiltonian has the unique solution. Thus, the HJB equation can have the unique solution even if the Hamiltonian does not satisfy the condition (SLC). Moreover, the Hamiltonian from our example of nonuniqueness shows the difference between (LLC), (SLC) and Loewen-Rockafellar condition [12]. We know that the condition (LLC) does not guarantee uniqueness of the solution to the HJB equation, so the natural question can be stated-what extra conditions the Hamiltonian should satisfy in order the HJB equation has the unique solution? The answer to this question is obtained when analysing conditions (LLC), (SLC) and Loewen-Rockafellar condition [12]. Namely, some extra conditions in the interdependence between space and subgradient for large values of the subgradient are mandatory (see Sect. 2).
In the literature, an example of nonuniqueness of the solution of the equation of HJB is known. Crandall and Lions in their fundamental article [4] give an example of nonuniqueness of viscosity solution of the transport equation In this example, the function b(·) is bounded and continuous, but is not locally Lipschitz continuous. Therefore the Hamiltonian H (t, x, p) = b(x) p is convex, continuous and satisfies the condition (1.5), but it does not satisfy (LLC). The solutions in the Crandall-Lions example are continuous, and in our example, they are lower semicontinuous. Now we consider an easy example of nonuniqueness of lower semicontinuous solutions of the transport equation x > 1 are lower semicontinuous solutions of the transport equation. In this example, the Hamiltonian H (t, x, p) = x 2 p satisfies (LLC), (SLC) and Loewen-Rockafellar condition [12], but it does not satisfy (1.5) or equivalently it does not have a linear growth in x. It means that these conditions play important roles in the uniqueness of lower semicontinuous solutions of HJB. Notice that Hamiltonians from the above two examples of nonuniqueness do not satisfy the classical assumptions, but the Hamiltonian from our example of nonuniqueness does. In the transport equation we cannot find the example of nonuniqueness as ours. Because on one hand the HJB equation with the Hamiltonian satisfying the classical assumptions and positively homogeneous in p has the unique solution (see [7]). On the other hand conditions (LLC), (SLC) and Loewen-Rockafellar condition [12] are equivalent, if the Hamiltonian is convex in p, positively homogeneous in p and continuous. Therefore, we cannot find differences between them while considering Hamiltonians from the transport equation.
Summarizing, the above examples of nonuniqueness show that the key role in the uniqueness of viscosity solutions of HJB is played by the Lipschitz-type conditions originated in optimization instead of local Lipschitz continuity originated in differential equations theory. Therefore our result is proper for optimization problems.
We have found the example of nonuniqueness presented here while generalizing a result of Plaskacz and Quincampoix [17]. This generalization was published in [16].

Existence and Uniqueness Theorem
In this section we present the well-known theorem on existence and uniqueness of lower semicontinuous solutions of the HJB equation. We discuss results of the paper on the basis of this theorem. First, we introduce basic assumptions on Hamiltonian: Condition (H3) is called a linear growth in p and x of the Hamiltonian. Basing on (H1)-(H3) we can state the following theorem on existence and uniqueness: We shall show that Theorem 2.1 is a particular case of Theorem 2.2 from [9]. To this end, we need to discuss the subgradient characterization of the condition (SLC) and the Loewen-Rockafellar condition from [12]. A subgradient is defined for the function given on the entire Euclidean space (see Definition 1.3), so to use a subgradient of the Lagrangian L we extend it in the following way: Hamiltonian satisfies (H1)-(H3). Moreover, one can prove that the condition (2.1) is a subgradient characterization of Lipschitz continuity of the multifunction (t, x) → epi L(t, x, ·) in the Hausdorff's sense. Besides, it is easy to see that the condition (2.1) implies the following one: For every r > 0 there exists k > 0 such that at every point (t, x, v) (2. 2) The condition (2.2) is a subgradient characterization of the Aubin type continuity of multifunction (t, x) → epi L(t, x, ·). This kind of Aubin continuity was introduced by Loewen and Rockafellar in [12, Definition 2.3, (b)]. However, a subgradient characterization can be found in [10,Proposition 3.4]. Galbraith [9] obtained the uniqueness result assuming convexity of Hamiltonian with respect to p, a mild growth of Hamiltonian with respect to p (see (A1) from [9]) and slightly more general Lipschitz-type condition than (2.2) (see (A2) from [9]). Thus, if we replace the condition (SLC) by (2.2) in Theorem 2.1, then the claim of Theorem 2.1 still holds. In [9,10,[12][13][14] the authors use a limiting subgradient that is larger than the regular subgradient we use. We know that replacing the regular subgradient by the limiting subgradient one the condition (2.2) is unchanged if we assume that Hamiltonian is continuous. We also know that the condition (2.2) implies (LLC) (see [9], Proposition 2.4). Therefore, assuming conditions (H1)-(H3) we obtain Further on, we will show that the implications (2.3) cannot be reversed. Moreover, we prove that the Lipschitz continuity (LLC) raising from differential equations is not sufficient for the uniqueness of lower semicontinuous solution of HJB. However, conditions (SLC) and (2.2) of optimization problems are sufficient for uniqueness.

Example of Hamiltonian
In this section we define a Hamiltonian and discuss its regularity. In the next section we show that the HJB equality with this Hamiltonian does not have the unique lower semicontinuous solution. We define an auxiliary function ϕ : [0, T ] × R → R by the formula ϕ(t, x) = |t − x| exp 2 |t − x| . (3.1) The Hamiltonian H : [0, T ] × R × R → R is defined by the formula

3)
Proof For θ 0 we define an auxiliary function f by the formula f (θ ) = √ θ exp(2 √ θ). We notice that the function f is increasing and f (|t − x|) = ϕ(t, x) for each t ∈ [0, T ], x ∈ R. Furthermore, the function f on [a, b] satisfies Lipschitz continuity if 0 < a < b. We fix t 0 ∈ [0, T ], x 0 ∈ R and p 0 ∈ R, and consider two cases.

By the definition of the Hamiltonian
Case 2 Let p 0 ∈ R and t 0 = x 0 . We define r by the formula r := |t 0 − x 0 |/3, then r |t − x| 5r for each t ∈ B(t 0 , r ), x ∈ B(x 0 , r ). Let l be the Lipschitz constant of the function f on [r, 5r ]. We define k by k := 2 + l/ f 2 (r ).

Proposition 3.2 Lagrangian (3.4) does not satisfy the condition (2.2).
Proof Let the Lagrangian L be given by the formula (3.4).
Therefore, the left and right hand sides of the inequality (2.2) are given by Let t n − x n → 0+ and v n = 2ϕ(t n , x n ). If the inequality (2.2) is satisfied, then for large n ∈ N Multiplying the above inequality by 2(t n − x n ) we have the inequality Passing to the limit, we obtain a contradiction.

An Example of Nonuniqueness
In this section we present two different, bounded, lower semicontinuous solutions of HJB with the Hamiltonian H given by (3.2) and the terminal condition g : R → R given by the formula We are going to define some notions for the whole Euclidean space so we extend

First Solution
Let the function U : [0, T ] × R → R be given by the formula

Theorem 4.1 The function U given by (4.2) is bounded lower semicontinuous solution of the HJB equation with the Hamiltonian (3.2) and the terminal condition (4.1).
Proof It is not difficult to notice that the function U is lower semicontinuous, bounded and U (T, x)=g(x). We will prove that the function U satisfies conditions (1.4). We consider five cases.
Case 5 Let x < t and t ∈ ]0, T ]. Then − p t 0 and − p x = 0 for each That finishes the proof.

Second Solution
Let the function V : [0, T ] × R → R be given by the formula Proof It is not difficult to notice that the function V is lower semicontinuous, bounded and V (T, x) = g(x). We prove that the function V satisfies conditions (1.4). Since ∂ V (t, x) = ∅ for x = t and V (t, x) = U (t, x) for x t and x < 2t − T , then by the Theorem 4.1 it is sufficient to show that V satisfies conditions (1.4), when 2t − T x < t. To do it we consider two cases.

The Value Function
In this section we show that the function V given by (4.3) is the value function, in the sense of definition (1.2), corresponding to the Lagrangian (3.4) and the terminal condition (4.1). To this purpose, we use the methods of [15], whose scheme is as follows: We construct a sequence of Hamiltonians (H n ) n∈N such that H n satisfy (H1)-(H3) together with (SLC) and H n H . Then, by [15,Corollary 3.5], the value functions V n corresponding to HJB with Hamiltonians H n and terminal conditions g n = g converge to the value function V (V n V ). Since Hamiltonians H n additionally, satisfy the condition (SLC), then by Theorem 2.1, the value functions V n are unique lower semicontinuous solutions. Therefore, to find the value function V , one needs to find solutions U n of equations HJB with Hamiltonians H n and the terminal condition g and then take the limit U n = V n → V .

Approximation of the Hamiltonian
Let the function σ : [0, +∞[→ R be given by σ (z) = √ z, and functions σ n : [0, +∞[ → R by σ n (z) = √ z if z 1/n and σ n (z) = 1/ √ n if 0 z < 1/n. We notice that functions σ n satisfy locally the Lipschitz continuity and σ n σ . Using functions σ n we define functions ϕ n : Then functions ϕ n also satisfy locally the Lipschitz continuity and ϕ n ϕ.
It is not difficult to notice that Hamiltonians H n given by (5.1) are continuous, convex with respect to p and have linear growth in p and x, so they satisfy conditions (H1)-(H3). Moreover H n H , because ϕ n ϕ. Similarly, for the proof of Case 2 in Theorem 3.1 we can prove that Hamiltonians (5.1) satisfy the condition (SLC).

Solutions U n U n U n of HJB with H n H n H n
Let U n : [0, T ] × R → R be given by formula: (c) if t − x 1/n and T − 2t + x 1/n, then Proposition 5.1 Let Hamiltonians H n be given by (5.1), the terminal condition g by (4.1), and the function V by (4.3). Then functions U n given by the above formula are bounded lower semicontinuous solutions of equations HJB with Hamiltonians H n and the terminal condition g, moreover U n → V .
Proof It is not difficult to notice that functions U n are bounded ( i.e. −1 U n ( · , · ) 1 ) and U n (T, x) = g(x). We can prove that functions U n are lower semicontinuous on

Regularity of Hamiltonian
Let C be a family of continuous functions ϕ : [0, T ] × R → [0, +∞[ that satisfy the condition ϕ(t, x) = 0 ⇔ t = x. Then it is easy to prove that the Hamiltonian H given by (3.2) with ϕ ∈ C is well-defined and satisfies (H1)-(H3). Next, by L we denote a subfamily of C that contains locally Lipschitz functions. Notice that ϕ given by the formula (3.1) belongs to C, but does not belong to L. The example of a function, that is contained in L, is ϕ(t, x) = |t − x|.
In this section we prove that the Lagrangian L given by (3.4) with ϕ ∈ L satisfies the Loewen-Rockafellar condition (2.2). However, it can be shown easily that its Hamiltonian does not satisfy the Lipschitz-type condition (SLC). It means that we cannot reverse the first implication in (2.3). Moreover, by the result of Galbraith [9] it follows that HJB equation with H given by (3.2) with ϕ ∈ L has the unique solution.
So, it could be said that the nonuniqueness of HJB is due to a particular choice of ϕ ∈ C and that choosing a different finction ϕ ∈ L ⊂ C one can get the unique solution of HJB.
Proof To prove the proposition we consider 3 cases.
So the proposition is proven.

Remark 6.3
The only difference between the assertion of Proposition 6.1 and the condition (A) is contained in modulus. From Propositions 6.1 and 6.2 we obtain that the Lagrangian L given by (3.4) with ϕ ∈ L satisfies the Loewen-Rockafellar condition (2.2).
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.