On nouniqueness of solutions of Hamilton-Jacobi-Bellman equations

An example of a nonunique solution of the Cauchy problem of Hamilton-Jacobi-Bellman (HJB) equation with surprisingly regular Hamiltonian is presented. The Hamiltonian H(t,x,p) is locally Lipschitz continuous with respect to all variables, convex in p and with linear growth with respect to p and x. The HJB equation possesses two distinct lower semicontinuous solutions with the same final conditions; moreover, one of them is the value function of the corresponding Bolza problem. The definition of lower semicontinuous solution was proposed by Barron-Jensen (1990) and Frankowska (1993). Using the example an analysis and comparison of assumptions in some uniqueness results in HJB equations is provided.


Introduction
The classical Cauchy problem for the Hamilton-Jacobi-Bellman equation is a partial differential equation with the final condition If the Hamiltonian H is convex in the gradient variable, then there are relations between solutions of HJB and calculus of variation problems involving a function dual to H. This function, called the Lagrangian and denoted by L, is the Legendre-Fenchel transform of H in its gradient variable: g(x(T )) + T t0 L(t, x(t),ẋ(t)) dt , where A([t 0 , T ], R n ) denotes the space of all absolutely continuous functions from [t 0 , T ] into R n . If the value function is differentiable, it is well-known that it satisfies HJB in the classical sense. However, in many situations the value function is not differentiable. Then the solution of the HJB equation must be defined in nonsmooth sense in such a way that under quite general assumptions on H and g, V is the unique solution of HJB. Since we use the nonsmooth analysis we need a notion of a subgradient. For a vector v ∈ R n and a function f : In 1990 Baron-Jensen [3] and Frankowska [7] introduced extended viscosity solutions to semicontinuous functions for Hamiltonian that is convex in the gradient variable and provided a uniqueness result. Frankowska [7] called these solutions lower semicontinuous solutions.
The main goal of the paper is to present an example of two distinct lower semicontinuous solutions of the HJB equation with surprisingly regular Hamiltonian. Understanding the role that the Lipschitz-type condition plays in theorems about uniqueness of solution of HJB is also important. The proposed Hamiltonian H : [0, T ]×R×R → R satisfies the classical assumptions i.e. firstly, it is convex with respect to p, secondly, it increases linearly in p and x, i.e. |H(t, x, p)| 2|p| for any t ∈ [0, T ], x, p ∈ R, thirdly, it is locally Lipschitz continuous, i.e.
where IB is the closed unit ball. In addition, we show that one of the indicated lower semicontinuous solutions is the value function given by (1.2). In general, for the uniqueness of HJB solutions, one needs some stronger Lipschitz-type condition, that we shall study further in connection to the results of uniqueness. Frankowska [7] proved that, the value function is the unique lower semicontinuous solution of the HJB equation if the Hamiltonian meets the classical assumptions and it is positively homogeneous in p, i.e. ∀ r>0 H(t, x, rp) = rH(t, x, p). Actually, the result of Frankowska does not require local Lipschitz continuity with respect to the triple (t, x, p). It is enough to assume it is satisfied with respect to state variable x only. The example of nonuniqueness of solution of HJB introduced in the current paper, does not contradict the result of Frankowska as the Hamiltonian in our example, fulfills the classical assumptions, but it is not positively homogeneous in p.
Earlier Ishii [11,Thm. 2.5] and Crandall-Lions [5, Thm. VI.1] had proved the uniqueness of viscosity solutions of HJB in the class of continuous functions. They had assumed instead of linear growth in p and x of Hamiltonian, the following condition One can show that if the Hamiltonian is convex in p and possesses a linear growth of the form H(t, x, p) 2(1 + |x|)|p|, then condition (1.5) holds with constant C = 2. Therefore the Hamiltonian from our example of nonuniqueness also satisfies (1.5 that is derived from the optimal control problem. The meaning of (SLC) in the optimal control problems is discussed in the following papers [8,17,18,20] In order to understand better the reason for nonuniqueness of the solution of the HJB equation given in our example, we need to recall the Loewen-Rockafellar condition from [12] that is more general version of (SLC). This condition was used by Loewen-Rockafellar [13,14] to study necessary conditions satisfied by optimal solutions of Bolza problem. Galbraith [9] proved that, the value function is the unique lower semicontinuous solution of HJB assuming the Loewen-Rockafellar condition [12]. This uniqueness result has, in general, the same nature as the uniqueness result of Dal Maso-Frankowska [6]. The Hamiltonian from our example of nonuniqueness is constructed using the function ϕ. In Section 6 we show that if ϕ is replaced by a function that is sufficiently regular, then we obtain the Hamiltonian that does not satisfy the condition (SLC), while it satisfies the Loewen-Rockafellar condition [12]. Therefore, by virtue of classical results, we are not able to say if after the mentioned change, we get the uniqueness of the solution of HJB or not. However, using the Galbraith result [9], we know that HJB with this Hamiltonian has the unique solution. Thus, the HJB equation can have the unique solution even if the Hamiltonian does not satisfy the condition (SLC). Moreover, the Hamiltonian from our example of nonuniqueness shows the difference between (LLC), (SLC) and Loewen-Rockafellar condition [12]. We know that the condition (LLC) does not guarantee uniqueness of the solution to the HJB equation, so the natural question can be stated -what extra conditions the Hamiltonian should satisfy in order the HJB equation has the unique solution? The answer to this question is obtained when analysing conditions (LLC), (SLC) and Loewen-Rockafellar condition [12]. Namely, some extra conditions in the interdependence between space and subgradient for large values of the subgradient are mandatory (see Sect. 2).
In the literature, an example of nonuniqueness of the solution of the equation of HJB is known. Crandall and Lions in their fundamental article [4] give an example of nonuniqueness of viscosity solution of the transport equation In this example, the function b(·) is bounded and continuous, but is not locally Lipschitz continuous. Therefore the Hamiltonian H(t, x, p) = b(x) p is convex, continuous and satisfies the condition (1.5), but it does not satisfy (LLC). The solutions in the Crandall-Lions example are continuous, and in our example, they are lower semicontinuous. Now we consider an easy example of nonuniqueness of lower semicontinuous solutions of the transport equation. Let b(x) = x 2 and g ≡ 0. Then functions V ≡ 0 and U (t, x) = 0, (T − t)x 1, U (t, x) = 1, (T − t)x > 1 are lower semicontinuous solutions of the transport equation. In this example, the Hamiltonian H(t, x, p) = x 2 p satisfies (LLC), (SLC) and Loewen-Rockafellar condition [12], but it does not satisfy (1.5) or equivalently it does not have a linear growth in x. It means that these conditions play important roles in the uniqueness of lower semicontinuous solutions of HJB. Notice that Hamiltonians from the above two examples of nonuniqueness do not satisfy the classical assumptions, but the Hamiltonian from our example of nonuniqueness does. In the transport equation we cannot find the example of nonuniqueness as ours. Because on one hand the HJB equation with the Hamiltonian satisfying the classical assumptions and positively homogeneous in p has the unique solution (see Frankowska [7]). On the other hand conditions (LLC), (SLC) and Loewen-Rockafellar condition [12] are equivalent, if the Hamiltonian is convex in p, positively homogeneous in p and continuous. Therefore, we cannot find differences between them while considering Hamiltonians from the transport equation.
Summarizing, the above examples of nonuniqueness show that the key role in the uniqueness of viscosity solutions of HJB is played by the Lipschitz-type conditions originated in optimization instead of local Lipschitz continuity originated in differential equations theory. Therefore our result is proper for optimization problems.
We have found the example of nonuniqueness presented here while generalizing a result of Plaskacz-Quincampoix [19]. This generalization was published in [16].

Existence and uniqueness theorem
In this section we present the well-known theorem on existence and uniqueness of lower semicontinuous solutions of the HJB equation. We discuss results of the paper on the basis of this theorem. First, we introduce basic assumptions on Hamiltonian: We shall show that Theorem 2.1 is a particular case of Theorem 2.2 from [9]. To this end, we need to discuss the subgradient characterization of the condition (SLC) and the Loewen-Rockafellar condition from [12]. A subgradient is defined for the function given on the entire Euclidean space (see Def. 1.3), so to use a subgradient of the Lagrangian L we extend it in the following way: Hamiltonian satisfies (H1)-(H3). Moreover, one can prove that the condition (2.1) is a subgradient characterization of Lipschitz continuity of the multifunction (t, x) → epi L(t, x, ·) in the Hausdorff's sense. Besides, it is easy to see that the condition (2.1) implies the following one: For every r > 0 there exists k > 0 such that at every point (t, x, v) ∈ R × rIB × R n , every (w 1 , w 2 , p) ∈ ∂L(t, x, v) we have |(w 1 , w 2 )| k(1 + |v| + |L(t, x, v)|)(1 + |p|).
The condition (2.2) is a subgradient characterization of the Aubin type continuity of multifunction (t, x) → epi L(t, x, ·). This kind of Aubin continuity was introduced by Loewen and Rockafellar in [12, Def. 2.3, (b)]. However, a subgradient characterization can be found in [10,Prop. 3.4]. Galbraith [9] obtained the uniqueness result assuming convexity of Hamiltonian with respect to p, a mild growth of Hamiltonian with respect to p (see (A1) from [9]) and slightly more general Lipschitz-type condition than (2.2) (see (A2) from [9]). Thus, if we replace the condition (SLC) by (2.2) in Theorem 2.1, then the claim of Theorem 2.1 still holds. In [9,10,12,13,14] the authors use a limiting subgradient that is larger than the regular subgradient we use. We know that replacing the regular subgradient by the limiting subgradient one the condition (2.2) is unchanged if we assume that Hamiltonian is continuous. We also know that the condition (2.2) implies (LLC) (see [9], Prop. 2.4). Therefore, assuming conditions (H1)-(H3) we obtain Further on, we will show that the implications (2.3) cannot be reversed. Moreover, we prove that the Lipschitz continuity (LLC) raising from differential equations is not sufficient for the uniqueness of lower semicontinuous solution of HJB. However, conditions (SLC) and (2.2) of optimization problems are sufficient for uniqueness.

Example of Hamiltonian
In this section we define a Hamiltonian and discuss its regularity. In the next section we show that the HJB equality with this Hamiltonian does not have the unique lower semicontinuous solution.
We define an auxiliary function ϕ : [0, T ] × R → R by the formula The Hamiltonian H : It is not difficult to see that the Hamiltonian H given by ( Proof. For θ 0 we define an auxiliary function f by the formula f (θ) = √ θ exp(2 √ θ). We notice that the function f is increasing and f (|t − x|) = ϕ(t, x) for each t ∈ [0, T ], x ∈ R. Furthermore, the function f on [a, b] satisfies Lipschitz continuity if 0 < a < b. We fix t 0 ∈ [0, T ], x 0 ∈ R and p 0 ∈ R, and consider two cases.

ARKADIUSZ MISZTELA
The Lagrangian L : [0, T ]×R×R → R ∪ {+∞} given by the formula (1.1) has the following form: Proof. Let the Lagrangian L be given by the formula (3.4).
Therefore, the left and right hand sides of the inequality (2.2) are given by Let t n − x n → 0+ and v n = 2ϕ(t n , x n ). If the inequality (2.2) is satisfied, then for large n ∈ N 2k(1 + ϕ(t n , x n )) 1 + 1 2ϕ(t n , x n ) .
Multiplying the above inequality by 2(t n − x n ) we have the inequality Passing to the limit, we obtain a contradiction.

An example of nonuniqueness
In this section we present two different, bounded, lower semicontinuous solutions of the HJB equation with the Hamiltonian H given by (3.2) and the terminal condition g : R → R given by Now we introduce the lemma that is needed in the proof of the inequality (1.4). For the function f : R n → R ∪ {+∞} and z ∈ dom f we define In this paper we define some notions on the whole Euclidean space, that while applied to U (t, x) given on [0, T ] × R n we extend U (t, x) by setting +∞ for (t, x) / ∈ [0, T ] × R n . In addition to this, we introduce the following notation ψ(t, x) := 1 ϕ(t,x) for t = x. 4.1. First solution. Let the function U : [0, T ] × R → R be given by the formula Proof. It is not difficult to notice that the function U is lower semicontinuous, bounded and U (T, x) = g(x). We will prove that the function U satisfies conditions (1.4). To this end, we consider five cases. Case 1. Let x > t and t ∈ [0, T [. Then the equality ∆U (t, x)(1, 0) = ψ(t, x) holds. Therefore, from the inequality (4.3) we obtain ψ(t, x) p t for each (p t , p x ) ∈ ∂U (t, x). Moreover, ∆U (t, x)(0, 1) = −ψ(t, x) and ∆U (t, x)(0, −1) = ψ(t, x). Therefore, from the inequality (4.3) we have −ψ(t, x) p x and ψ(t, x) −p x for each (p t , p x ) ∈ ∂U (t, x). Since −p t −ψ(t, x) and −p x = ψ(t, x) for each (p t , p x ) ∈ ∂U (t, x), then by the definition of H we have Case 2. Let x > t and t ∈ ]0, T ]. Then the equality ∆U (t, x)(−1, 0) = −ψ(t, x) holds. Therefore, from the inequality (4.3) we have −ψ(t, x) −p t for each (p t , p x ) ∈ ∂U (t, x). Moreover, ∆U (t, x)(0, 1) = −ψ(t, x) and ∆U (t, x)(0, −1) = ψ(t, x). Therefore, from the inequality (4.3) we get −ψ(t, x) p x and ψ(t, x) −p x for each (p t , p x ) ∈ ∂U (t, x). Since −p t −ψ(t, x) and −p x = ψ(t, x) for each (p t , p x ) ∈ ∂U (t, x), then by the definition of H we have Case 3. Let x = t and t ∈ [0, T ]. Then the equality ∆U (t, x)(0, 1) = −∞ holds. If (p t , p x ) ∈ ∂U (t, x), then from the inequality (4.3) we have the contradiction −∞ = ∆U (t, x)(0, 1) p x ∈ R. Therefore ∂U (t, x) = ∅. Case 4. Let x < t and t ∈ [0, T [. Then the equality ∆U (t, x)(1, 0) = 0 holds. Therefore, from the inequality (4.3) we obtain 0 p t for each (p t , p x ) ∈ ∂U (t, x). Moreover, ∆U (t, x)(0, 1) = 0 and ∆U (t, x)(0, −1) = 0. Therefore, from the inequality (4.3) we have 0 p x and 0 −p x for each (p t , p x ) ∈ ∂U (t, x). Since −p t 0 and −p x = 0 for each (p t , p x ) ∈ ∂U (t, x), then by the definition of H we have Case 5. Let x < t and t ∈ ]0, T ]. Then the equality ∆U (t, x)(−1, 0) = 0 holds. Therefore, from the inequality (4.3) we have 0 −p t for each (p t , p x ) ∈ ∂U (t, x). Moreover, ∆U (t, x)(0, 1) = 0 and ∆U (t, x)(0, −1) = 0. Therefore, from the inequality (4.3) we get 0 p x and 0 −p x for each (p t , p x ) ∈ ∂U (t, x). Since −p t 0 and −p x = 0 for each (p t , p x ) ∈ ∂U (t, x), then by the definition of H we have that finishes the proof.  Proof. It is not difficult to notice that the function V is lower semicontinuous, bounded and V (T, x) = g(x). We prove that the function V satisfies conditions (1.4). Since V (t, x) = U (t, x) for x t and x < 2t − T , than by the Theorem 4.2 it is sufficient to show that V satisfies conditions (1.4), when 2t − T x < t. To do it we consider two cases.
, then by the definition of H, that ends the proof.

The value function
In this section we show that the function V given by (4.5) is the value function, in the sense of definition (1.2), corresponding to the Lagrangian (3.4) and the terminal condition (4.1). To this purpose, we use the methods of [15], whose scheme is as follows: We construct a sequence of Hamiltonians (H n ) n∈N such that H n satisfy (H1)-(H3) together with (SLC) and H n ց H. Then, by [15,Cor. 3.5], the value functions V n corresponding to HJB with Hamiltonians H n and terminal conditions g n = g converge to the value function V (V n ր V ). Since Hamiltonians H n additionally, satisfy the condition (SLC), then by Theorem 2.1, the value functions V n are unique lower semicontinuous solutions. Therefore, to find the value function V , one needs to find solutions U n of equations HJB with Hamiltonians H n and the terminal condition g and then take the limit U n = V n → V . Then functions ϕ n also satisfy locally the Lipschitz continuity and ϕ n ց ϕ.
Hamiltonians H n : [0, T ] × R × R → R are defined by the formula . It is not difficult to notice that Hamiltonians H n given by (5.1) are continuous, convex with respect to p and have linear growth in p and x, so they satisfy conditions (H1)-(H3). Moreover H n ց H, because ϕ n ց ϕ. Similarly, for the proof of Case 2 in Theorem 3.1 we can prove that Hamiltonians (5.1) satisfy the condition (SLC).

Solutions U n
U n U n of HJB with H n H n H n . Let U n : [0, T ] × R → R be given by formula: (c) if t − x 1/n and T − 2t + x 1/n, then (d) if t − x 1/n and T − 2t + x 1/n, then (e) if 1/n x − t, then U n (t, x) = exp(−2 √ x − t ) − 1. For x < 2t − T we have U n (t, x) = 1.
Proposition 5.1. Let Hamiltonians H n be given by (5.1), the terminal condition g by (4.1), and the function V by (4.5). Then functions U n given by the above formula are bounded lower semicontinuous solutions of equations HJB with Hamiltonians H n and the terminal condition g, moreover U n → V .

Regularity of Hamiltonian
Let C be a family of continuous functions ϕ : [0, T ] × R → [0, +∞[ that satisfy the condition ϕ(t, x) = 0 ⇔ t = x. Then it is easy to prove that the Hamiltonian H given by (3.2) with ϕ ∈ C is well-defined and satisfies (H1)-(H3). Next, by L we denote a subfamily of C that contains locally Lipschitz functions. Notice that ϕ given by the formula (3.1) belongs to C, but does not belong to L. The example of a function, that is contained in L, is ϕ(t, x) = |t − x|.
In this section we prove that the Lagrangian L given by (3.4) with ϕ ∈ L satisfies the Loewen-Rockafellar condition (2.2). However, it can be shown easily that its Hamiltonian does not satisfy the Lipschitz-type condition (SLC). It means that we cannot reverse the first implication in (2.3). Moreover, by the result of Galbraith [9] it follows that HJB equation with H given by (3.2) with ϕ ∈ L has the unique solution. Remark 6.3. The only difference between the assertion of Proposition 6.1 and the condition (A) is contained in modulus. From Proposition 6.1 and 6.2 we obtain that the Lagrangian L given by (3.4) with ϕ ∈ L satisfies the Loewen-Rockafellar condition (2.2).