On risk-sensitive piecewise deterministic Markov decision processes

We consider a piecewise deterministic Markov decision process, where the expected exponential utility of total (nonnegative) cost is to be minimized. The cost rate, transition rate and post-jump distributions are under control. The state space is Borel, and the transition and cost rates are locally integrable along the drift. Under natural conditions, we establish the optimality equation, justify the value iteration algorithm, and show the existence of a deterministic stationary optimal policy. Applied to special cases, the obtained results already significantly improve some existing results in the literature on finite horizon and infinite horizon discounted risk-sensitive continuous-time Markov decision processes.


Introduction
Since the pioneering work [18], risk-sensitive discrete-time Markov decision processes (DTMDPs) have been studied intensively. Having restricted our attention to total undiscounted or discounted problems, let us mention e.g., [3,6,7,11,12,15,16], most of which deal with the exponential utility, as well as in the present paper. As an application, an open problem in insurance was recently solved in [4] in the framework of risk-sensitive DTMDP. There are notable differences between risk-sensitive and risk-neutral DTMDPs. For instance, in a finite model, i.e., when the state and action spaces are both finite, there is always a deterministic stationary optimal policy in a discounted risk-neutral DTMDP, but not always in a discounted risk-sensitive DTMDP, see [15].
One of the first works on risk-sensitive continuous-time Markov decision processes (CTMDPs) is [21], where only verification theorems were presented. Recently, there have been reviving interests in this topic; see e.g., [8,14,20,24,25,27]. A finite horizon total undiscounted risk-sensitive CTMDP was considered in [14,21,24], whose arguments were summarized as follows. Firstly, the optimality equation is shown to admit a solution out of a small enough class. Secondly, by using the Feynman-Kac formula, this solution is shown to be the value function, and any Markov policy providing the minimizer in the optimality equation is optimal. The proofs of [14,24] reveal that the the main technicalities lie in the first step, for which, the state space was assumed to be denumerable. This assumption is important for the diagonalization argument used in [24], which is an extension of [14] from bounded transition rate to possibly unbounded transition rate, whose growth is bounded by a Lyapunov function. The latter requirement and the boundedness of the cost rate then validate the Feynman-Kac formula applied in the second step. The author of [24] mentioned that it was unclear how to extend his argument to an unbounded cost rate, see Section 7 therein. Following a similar argument as described above, a discounted risk-sensitive CTMDP was also considered in [14], although now the first step becomes, to quote the authors' words (see p.658 therein), "surprisingly far more involved", for which the state space was further assumed to be finite, see Remark 3.6 therein. It is a corollary of the present paper that we significantly weaken the restrictive conditions in [14,24], see Section 3 below.
The present paper is concerned with a risk-sensitive piecewise deterministic Markov decision process (PDMDP), where the expected exponential utility of the total cost is to be minimized. The state space is a general Borel space, the transition and the nonnegative cost rates only need be locally integrable along the drift. A PDMDP is an extension of a CTMDP: now between two consecutive jumps, the process evolves according to deterministic Markov process. For simplicity and to keep the conditions as weak as possible, we do not consider the control on the drift. In spite that there has been a vast literature on PDMDPs; see the well known monographs [9,10] and the references therein, to the best of our knowledge, risk-sensitive PDMDPs have not been systematically studied before.
Our main contributions are the following. We establish the optimality equation satisfied by the value function, justify the value iteration algorithm and show the existence of a deterministic stationary optimal policy. As an application and corollary, finite horizon and infinite horizon discounted risk-sensitive CTMDPs are reformulated as total undiscounted risk-sensitive PDMDPs, and are thus treated in a unified way and under much weaker conditions than in [14,24]. This is possible because we follow a different argument. Namely, we directly show that the value function satisfies the optimality equation, by reducing the total undiscounted risk-sensitive PDMDP to a risk-sensitive DTMDP. This method, without referring to the Feynman-Kac formula, was originally developed by Yushkevich [26] for risk-neutral CTMDPs. Later, it was employed in [1,2,9,10,13,23] for studies of risk-neutral PDMDPs, and in [27] for risk-sensitive CTMDPs. In [8], restricted to stationary policies, the discounted risk-sensitive CTMDP with bounded transition rates was reduced to a DTMDP problem, using the uniformization technique. The induced DTMDP is less standard (with a random cost), and was not further investigated there.
The rest of the paper is organized as follows. In Section 2 we describe the concerned optimal control problem. In Section 3 we present the main results, the proofs of which are postponed to Section 4. We finish the paper with a conclusion in Section 5. Some relevant facts were collected in the appendix for ease of reference.

Model description and problem statement
Notations and conventions. In what follows, B(X) is the Borel σ-algebra of the topological space X, I stands for the indicator function, and δ {x} (·) is the Dirac measure concentrated on the singleton {x}, assumed to be measurable. A measure is σ-additive and [0, ∞]-valued. Below, unless stated otherwise, the term of measurability is always understood in the Borel sense. Throughout this paper, we adopt the conventions of If a mapping f defined on X, and {X i } is a partition of X, then when f is piecewise defined as Let S be a nonempty Borel state space, A be a nonempty Borel action space, and q stand for a signed kernel q(dy|x, a) on B(S) given (x, a) ∈ S × A such that for all Γ S ∈ B(S). Throughout this article we assume that q(·|x, a) is conservative and stable, i.e., where q x (a) := −q({x}|x, a). The signed kernel q is often called the transition rate. Between two consecutive jumps, the state of the process evolves according to a measurable mapping φ from S×[0, ∞) to S, see (5) below. It is assumed that for each and t → φ(x, t) is continuous. Finally let the cost rate c be a [0, ∞)-valued measurable function on S × A. For simplicity, we do not consider the case of different admissible action spaces at different states.
is continuous in a ∈ A.
(b) For each x ∈ S, the (nonnegative) function c(x, a) is lower semicontinuous in a ∈ A.
(c) The action space A is a compact Borel space.
The integrals in the above condition are well defined: the integrands are universally measurable in s ∈ [0, ∞); see Chapter 7 of [5].
Obviously, (t n (ω)) are measurable mappings on (Ω, F). In what follows, we often omit the argument ω ∈ Ω from the presentation for simplicity. Also, we regard x n and θ n+1 as the coordinate variables, and note that the pairs {t n , x n } form a marked point process with the internal history {F t } t≥0 , i.e., the filtration generated by {t n , x n }; see Chapter 4 of [19] for greater details. The marked point process {t n , x n } defines the stochastic process {ξ t , t ≥ 0} on (Ω, F) of interest by where we accept 0 · x := 0 and 1 · x := x for each x ∈ S ∞ , and below we denote S ∞ := S {x ∞ }. A (history-dependent) policy π is given by a sequence (π n ) such that, for each n = 0, 1, 2, . . . , π n (da|x 0 , θ 1 , . . . , x n , s) is a stochastic kernel on A, and for each ω = (x 0 , θ 1 , x 1 , θ 2 , . . . ) ∈ Ω, t > 0, I{t n < t ≤ t n+1 }π n (da|x 0 , θ 1 , . . . , θ n , x n , t − t n ), where a ∞ / ∈ A is some isolated point. A policy π is called Markov if, with slight abuse of notations, π(da|ω, s) = π M (da|ξ s− , s) for some stochastic kernel π M . A Markov policy is further called deterministic if the stochastic kernels π M (da|x, s) = δ {f M (x,s)} (da) for some measurable mapping f M from S × (0, ∞) to A. A policy is called deterministic stationary if for each n = 0, 1, . . . , π n (da|x 0 , θ 1 , . . . , θ n , x n , t − t n ) = δ {f (φ(xn,t−tn))} (da) for some measurable mapping f from S to A. We shall identify such a deterministic stationary policy by the underlying measurable mapping f .
The class of all policies is denoted by Π. Under a fixed policy π = (π n ), for each initial distribution γ on (S, B(S)), by using the Ionescu-Tulcea theorem, one can build a probability measure P π γ on (Ω, F) such that P π γ (x 0 ∈ Γ) = γ(Γ) for each Γ ∈ B(S), and the conditional distribution of (θ n+1 , x n+1 ) with the condition on x 0 , θ 1 , x 1 , . . . , θ n , x n is given on {ω : x n (ω) ∈ S} by ..,θn,xn,s)ds Aq (Γ 2 |φ(x n , t), a)π n (da|x 0 , θ 1 , . . . , θ n , x n , t)dt, A q φ(xn,s) (a)πn(da|x 0 ,θ 1 ,...,θn,xn,s)ds , (7) and given on {ω : . . , θ n , x n ) = 1. Below, when γ is a Dirac measure concentrated at x ∈ S, we use the denotation P π x . Expectations with respect to P π γ and P π x are denoted as E π γ and E π x , respectively. Roughly speaking, the uncontrolled version of the process evolves as follows: given the current state, the process evolves deterministically according to the mapping φ, up to the next jump, taking place after a random time whose distribution is (nonstationary) exponential, and the dynamics continue in the similar manner. A detailed book treatment with many examples of this and more general type of processes, allowing deterministic jumps, can be found in [10].
For each x ∈ S, and policy π = (π n ), defines the concerned performance measure of the policy π ∈ Π given the initial state x ∈ S. Here and below, we put c(x ∞ , a) := 0 for each a ∈ A, and φ(x ∞ , t) = x ∞ for each t ∈ [0, ∞). We are interested in the following optimal control problem for each x ∈ S : Minimize over π ∈ Π: V (x, π).
A policy π * is called optimal if V (x, π * ) = inf π∈Π V (x, π) =: V * (x) for each x ∈ S. The objective of this paper is to show, under the imposed conditions, the existence of a deterministic stationary optimal policy, and to establish the corresponding optimality equation satisfied by the value function V * , together with its value iteration. Evidently, V * (x) ≥ 1 for each x ∈ S. Under the next condition, it will be seen that for each x ∈ S, V * (φ(x, s)) is absolutely continuous in s.
The above condition is mainly assumed for notational convenience. In fact, the main optimality results (such as the existence of a deterministic stationary optimal policy) obtained in this paper can be established without assuming Condition 2.3, at the cost of some additional notations. In a nutshell, one has to consider the setsŜ := {x ∈ S : V * (x) < ∞} and S \Ŝ separately, and note that if x ∈Ŝ, then φ(x, t) ∈Ŝ for each t ∈ [0, ∞). The reasoning presented under Condition 2.3 can be followed in an obvious manner. We formulate the corresponding optimality results in Remarks 3.1 and 3.2 below.

Main statements
We first present the main optimality results concerning problem (8) for the PDMDP model. Their proofs are postponed to the next section.
In particular, V * (φ(x, t)) is absolutely continuous in t for each x ∈ S.
(b) There exists a deterministic stationary optimal policy f , which can be taken as any measurable In particular, V * (φ(x, t)) is absolutely continuous in t for each x ∈Ŝ.
(b) There exists a deterministic stationary optimal policy f , which can be taken as any measurable Next, we present the value iteration algorithm for the value function V * .

Theorem 3.2 Suppose Conditions 2.1, 2.2 and 2.3 are satisfied. Let
Here We can apply our theorems to a special case of a CTMDP. That is, φ(x, t) ≡ x for each x ∈ S. The following α-discounted risk-sensitive CTMDP problem was considered in [14]: Here α > 0 is a fixed constant. In fact, the authors of [14] were restricted to Markov policies, bounded transition and cost rates, i.e., sup x∈S q x < ∞, and sup x∈S,a∈A c(x, a) < ∞, and a finite state space S. These restrictions, e.g., the finiteness of S, were needed for their investigations, see e.g., Remark 3.6 in [14]. Under the compactness-continuity condition (Condition 2.1), it was shown in [14] that there exists an optimal Markov policy for the discounted risk-sensitive CTMDP, and established the optimality equation. By using the theorems presented earlier in this section, we can obtain these optimality results for problem (10) in a much more general setup: the state space S is Borel, there is no boundedness requirement on the transition rate with respect to the state x ∈ S, and the optimality is over the class of history-dependent policies. Furthermore, we let the CTMDP model be nonhomogeneous, i.e., the transition rate q(dy|t, x, a) now is a signed kernel on B(S) from (t, x, a) ∈ [0, ∞) × S × A, satisfying the corresponding version of (3); the notationsq is kept as before, see (2), with the extra argument t in addition to x. Similarly, the nonnegative cost rate c is allowed to be a measurable function on [0, ∞) × S × A.
Corollary 3.1 Consider the α-discounted risk-sensitive (nonhomogeneous) CTMDP problem (10) with c(ξ t , a) being replaced by c(t, ξ t , a). Suppose Then the value function say L * to the α-discounted risk-sensitive CTMDP problem (10) (with c(ξ t , a) being replaced by c(t, ξ t , a)) is given by (c) There exists an optimal deterministsic Markov policy f for the α-discounted risk-sensitive CT-MDP problem (10) (with c(ξ t , a) being replaced by c(t, ξ t , a)). One can take f as any measurable Proof. We prove this by reformulating the nonhomogeneous version of the α-discounted risk-sensitive (nonhomogeneous) CTMDP problem (10) in the form of problem (8) for a PDMDP, which we introduce as follows. We use the notation "hat" to distinguish this model from the original (nonhomogeneous) CTMDP model.
• The action space is the same as in the CTMDP:Â = A.
• The drift is given byφ((t, x), s) := (t + s, x) for each x ∈ S and t, s ≥ 0. Clearly it satisfies the corresponding version of (4).
• The cost rate is given bŷ Now the marked point process {t n ,x n } and controlled processξ t in this PDMDP model is connected to those in the original (nonhomogeneous) CTMDP model, namely (t n , x n ) and ξ t , viat n = t n and x n = (t n , x n ), andξ t = (t, ξ t ). For example, under a fixed strategyπ and initial distributionγ in this PDMDP model, the version of the first equation in (7) now reads on {ω : (da|x 0 ,θ 1 ,...,θn,xn,s)ds × A I{t + t n ∈ Γ 2 }q(Γ 3 |t + t n , x n , a)π n (da|x 0 ,θ 1 , . . . ,θ n ,x n , t)dt, The condition in the previous corollary is much weaker than in [14], and can be further weakened; one only needs the reformulated PDMDP to satisfy Conditions 2.1, 2.2 and 2.3. Moreover, the boundedness of the cost rate c was assumed in the previous corollary only to ensure Condition 2.3 to be satisfied. It can be relaxed if one formulates the previous corollary using the statements in Remarks 3.1 and 3.2.
One can also consider the risk-sensitive nonhomogeneous CTMDP problem on the finite horizon [0, T ] with T > 0 being a fixed constant: where g is a [0, ∞)-valued measurable function; g(x) represents the terminal cost incurred when ξ T = x ∈ S. Let us put g(x ∞ ) := 0. Here α is a fixed nonnegative finite constant. A simpler version of this problem was considered in [24] with α = 0 and a bounded cost rate, where additional restrictions were put on the growth of the transition rate. We can reformulate this problem into the PDMDP problem (8) just as in the above. The only difference is that now we put q (t,x) (a) ≡ 0 for each x ∈ S and t ≥ T, and introduce the following cost rate for each x ∈ S, t ≥ 0 and a ∈ A : if t > T.

Proof of the main statements
For the rest of this paper, it is convenient to introduce the following notations. Let P(A) be the space of probability measures on B(A), endowed with the standard weak topology. For each µ ∈ P(A), Let R denote the set of (Borel) measurable mappings ρ t (da) from t ∈ (0, ∞) → P(A). Here, we do not distinguish two measurable mappings in t ∈ (0, ∞), which coincide almost everywhere with respect to the Lebesgue measure. Let us equip R with the Young topology, which is the weakest topology with respect to which the function ρ ∈ R → Proof. One can legitimately consider the following DTMDP (discrete-time Markov decision process): according to Lemma 2.29 of [9], all the involved mappings are measurable.
• The state space is X : is regarded as an isolated point in X.
• The action space is A := R.
The relevant facts and statements for the DTMDP are included in the Appendix.
One can show that under Conditions 2.1 and 2.2, for each (θ, x) ∈ X, a ∈ A → X f (z)p(dz|(θ, x), a) is continuous for each bounded measurable function f on X; for each (θ, x) ∈ X and (τ, y) ∈ X, a ∈ A → l((θ, x), ρ, (τ, y)) is lower semicontinuous, and A is a compact Borel space. Hence, Condition A.1 for the DTMDP model {X, A, p, l} is satisfied.
The controlled process in the above DTMDP model {X, A, p, l} is denoted by {Y n , n = 0, 1, . . . }, where Y n = (Θ n , X n ), and the controlling process is denoted by {A n , n = 0, 1, . . . }. For n ≥ 1, Θ n and X n correspond to the nth sojourn time and the post-jump state in the PDMDP, Θ 0 is fictitious, and X 0 is the initial state in the PDMDP. Let Σ be the class of all strategies for the DTMDP model {X, A, p, l}, and Σ 0 DM be the class of deterministic Markov strategies in the form σ = (ϕ n ) where ϕ 0 ((θ, x)) does not depend on θ ∈ (0, ∞) for each x ∈ S. We preserve the term of policy for the PDMDP and the term of strategy for the DTMDP.
According to Proposition A.1, the function is the minimal [1, ∞]-valued measurable solution to the optimality equation for each x ∈ S and θ ∈ (0, ∞); this is just (20). Furthermore, by Proposition A.1, there exists a deterministic stationary strategy σ * for the DTMDP such that σ * ((θ, x)) attains the above infimum for each x ∈ S and θ ∈ (0, ∞), and any such strategy σ * verifies Letθ ∈ (0, ∞) be arbitrarily fixed. The function V * ((θ, x)) being measurable in (θ, x) ∈ X, it follows that x ∈ S → V * ((θ, x)) is measurable. The strategy σ * and the constantθ induce a deterministic Markov strategy σ * * = (ϕ n ) ∈ Σ 0 DM , where ϕ 0 ((θ, x)) =: σ * ((θ, x)) for each θ ∈ (0, ∞), x ∈ S, and ϕ n ((θ, x)) := σ((θ, x)) for each n ≥ 1, θ ∈ (0, ∞), x ∈ S. (The control on the isolated point (0, x ∞ ) is irrelevant and we do not specify the definition of the strategy on that point.) This strategy can be identified with a policy π * in the PDMDP, c.f. (6). On the other hand, each policy π = (π n ) can be identified with a deterministic strategy in this DTMDP. Thus, for each x ∈ S. Consequently, the policy π * is optimal, V * (x) = V * ((θ, x)) for each x ∈ S and θ ∈ (0, ∞); recall thatθ was arbitrarily fixed. The statement of this lemma now follows. ✷ The policy π * in the proof of the previous lemma is actually optimal for problem (8). However, it is not necessarily a deterministic nor stationary policy. Also the reduction of the risk-sensitive PDMDP problem (8) to a risk-sensitive problem for the DTMDP model {X, A, p, l} as seen in the proof of the above theorem will be used without special reference in what follows.
In the proof of part (a), it was observed that are absolutely continuous in t and are thus finite for each t ∈ [0, ∞). As in the proof of part (a), similar calculations to those in (14) imply that for each t ∈ [0, ∞), where the last equality is by what was established in part (a). Therefore, for each t ∈ [0, ∞), where the inequality holds because V * (x) ≥ 1 for each x ∈ S. Taking lim t→∞ on the both sides of the previous equality yields: with the inequality following from (15). Hence Here it is clear that s ∈ [0, ∞) → f (φ(x, s)) can be identified as an element of R, denoted asf x . In fact,f x s = δ {f (φ(x,s))} for each s ∈ [0, ∞), whereas x ∈ S →f x ∈ R is measurable. This measurable mapping x ∈ S →f x ∈ R defines a deterministic stationary optimal strategy for the risk-sensitive DTMDP problem (20) by Proposition A.1. It is clear that the measurable mapping x ∈ S → f (x) ∈ A defines an optimal deterministic stationary policy for the PDMDP problem (8).
Finally, we show the remaining part of (a). Let H * be a measurable [1, ∞)-valued function on S such that There exists a measurable mapping h from S to A such that c.f., Proposition D.5 of [17]. It follows that s 0 S H * (y)q(dy|φ(x, τ ), h(φ(x, τ )))dτ is absolutely continuous in s ∈ [0, t] for each t ≥ 0. As in the proof of part (b), and by passing to the lower limit as t → ∞, It remains to refer to Proposition A.1 for that H * (x) ≥ V * (x) for each x ∈ S. ✷ Proof of Theorem 3.2. Let V * 0 (x) := 1 for each x ∈ S. For each n ≥ 0, one can legitimately define Recall that the DTMDP model {X, A, p, l} satisfies Condition A.1, as noted in the proof of Lemma 4.1. Then by Proposition A.1, {V * n } is a monotone nondecreasing sequence of [1, ∞)-valued measurable functions on S such that V * n (x) ↑ V * (x) as n ↑ ∞, for each x ∈ S. Let n ≥ 0 be fixed. As in Lemma 4.3, for each x ∈ S, there is some ρ * ∈ R such that Also the relevant version of Lemma 4.2 holds: for each x ∈ S and ρ ∈ R, Corresponding to (14), we now have where τ ∈ [0, t] → U * n+1 (x, τ ) is integrable and coincides with ∂V * n+1 (φ(x,t)) ∂t almost everywhere, and f is some measurable mapping from S to A, whose existence is guaranteed by Proposition D.5 of [17]. Continued from the above relation, the reasoning in the proof of the first assertion in part (a) of Theorem 3.1 can be followed: eventually we see almost everywhere in τ ∈ [0, t], i.e., the equation . Suppose the recursive definition in (9) is valid up to step n, and V * n (x) = V (n) (x) for each x ∈ S. Consider an arbitrarily fixed [1, ∞)-valued measurable solution V to (18), and let f * be a measurable mapping from S to A such that One can follow the reasoning in the last part of the proof of Theorem 3.1, and see, c.f. (16), φ(x,v)))−c(φ(x,v),f * (φ(x,v))))dv S V * n (y)q(dy|φ(x, s), f * (φ(x, s)))ds +e − ∞ 0 q φ(x,s) (f * (φ(x,s)))ds e ∞ 0 c(φ(x,s),f * (φ(x,s))))ds ≥ inf ρ∈R ∞ 0 e − τ 0 (q φ(x,s) (ρs)−c(φ(x,s),ρs))ds S V * n (y)q(dy|φ(x, τ ), ρ τ ) dτ where the last equality is by (17). Thus, V * n+1 is the minimal [1, ∞)-valued measurable solution to (18), and coincides with V (n+1) . Therefore, by induction V * n = V (n) for each n ≥ 0. It follows now that V (n) (x) ↑ V * (x) as n ↑ ∞ for each x ∈ S. ✷

Conclusion
In this paper, we considered total undiscounted risk-sensitive PDMDP in Borel state and action spaces with a nonnegative cost rate. The transition and cost rates are assumed to be locally integrable along the drift. Under quite natural conditions, we showed that the value function is a solution to the optimality equation, justified the value iteration algorithm, and showed the existence of deterministic stationary optimal policy. As a corollary, the obtained results were applied to improving significantly known results for finite horizon undiscounted and infinite horizon discounted risk-sensitive CTMDP in the literature.

A Appendix
For ease of reference, we present the relevant notations and facts about the risk-sensitive problem for a DTMDP. The proofs of the presented statements can be found in [16] or [27]. Standard description of a DTMDP can be found in e.g., [17,22]. Consider a discrete-time Markov decision process with the following primitives: • X is a nonempty Borel state space.
• A is a nonempty Borel action space.
• p(dy|x, a) is a stochastic kernel on B(X) given (x, a) ∈ X × A.
Let Σ be the space of strategies, and Σ DM be the space of all deterministic strategies for the DTMDP. Let the controlled and controlling processes be denoted by {Y n , n = 0, 1, . . . , ∞} and {A n , n = 0, 1, . . . , ∞}, respectively. The strategic measure of a strategy σ given the initial state x ∈ X is denoted by P σ x . The expectation taken with respect to P σ x is denoted by E σ x . Consider the optimal control problem Minimize over σ : E σ x e ∞ n=0 l(Yn,An,Y n+1 ) =: V(x, σ), x ∈ X.
It is also referred to as the risk-sensitive DTMDP problem. We denote the value function of problem (19) by V * . Then a strategy σ * is called optimal for problem (19) if V(x, σ * ) = V * (x) for each x ∈ X.
Condition A.1 (a) The function l(x, a, y) is lower semicontinuous in a ∈ A for each x, y ∈ X.
(b) For each bounded measurable function f on X and each x ∈ X, X f (y)p(dy|x, a) is continuous in a ∈ A.
(c) The space A is a compact Borel space.
then V * (x) = V(x, ϕ) for each x ∈ X. Then (V (n) (x)) increases to V * (x) for each x ∈ X, where V * is the value function for problem (19). Furthermore, there exists a deterministic stationary strategy ϕ satisfying (21), and so in particular, there exists a deterministic stationary optimal strategy for the risk-sensitive DTMDP problem (19).