1 Introduction

Suppose that the dynamics of some controlled state satisfy

$$\begin{aligned} dX_t = (b_t + B_t X_t - \alpha _t) dt + (c_t + C_t X_t) dW_t, \end{aligned}$$

where W is a one-dimensional Brownian motion, \(\alpha \) is some square-integrable control process and b, B, c, C are real-valued deterministic bounded functions. We consider the problem of minimizing, over all controls \(\alpha \),

$$\begin{aligned} \limsup _{T \rightarrow \infty } \frac{1}{T} E \int _0^T f(s, X_s, \alpha _s) ds, \end{aligned}$$
(0.1)

where f is a quadratic cost function of the form

$$\begin{aligned} f(t,x, a)&= \beta _{xx}(t) x^2 + \beta _x(t) x + \beta _{xa}(t) ax + \beta _{aa}(t) a^2 + \beta _a(t) a + \beta _0(t) \end{aligned}$$

with \(\beta _{xx}, \beta _{x}, \beta _{xa}, \beta _{aa}, \beta _a, \beta _0\) being real-valued, deterministic, left-continuous and bounded functions.

The problem of minimizing (0.1) arises in stylized form, e.g. in applications where an agent aims at keeping a state close to a possibly time-dependent target level, and any adjustment of the state position entails costs depending on the adjustment rate \(\alpha \). We refer to the end of Sect. 1 for a description of some more detailed examples.

The homogeneous problem version, in which \(b,B, c, C, \beta _{xx}, \beta _{x}, \beta _{xa}, \beta _{aa}, \beta _a, \beta _0\) are all constant functions, is already well-studied in the literature, even for a multidimensional generalization (see, e.g., [3]). The focus of the present article lies on the inhomogenity of the setting. Our aim is to provide sufficient conditions for the inhomogeneous problem to be well-posed and to derive an explicit formula for an optimal control.

As is well-known, the solvability of finite-time inhomogeneous linear-quadratic control problems is strongly linked to the solvabilitity of a related Riccati equation (see e.g. [20] and [22]), which in dimension one has the form

$$\begin{aligned} U_t' = \frac{ \left( U_t \right) ^2 }{2 \beta _{aa} (t) } - U_t \left( 2 B_t + \frac{ \beta _{xa}(t) }{ \beta _{aa} (t) } + C_t^2 \right) - 2 \beta _{xx}(t) + \frac{ \beta _{xa}^2 (t) }{2 \beta _{aa} (t)} \end{aligned}$$
(0.2)

(note that U corresponds to 2P in Sect. 2 of [20]). Given a finite time horizon \(T \in (0, \infty )\), a solution of the problem of minimizing \(E\int _0^T f(t, X_t, \alpha _t)dt\) can be expressed in terms of the solution of (0.2) with the terminal condition \(U_T = 0\).

We show that also the problem of minimizing the long-term cost average functional (0.1) can be reduced to the Riccati equation (0.2). The difficulty in the infinite horizon case, however, is that no terminal condition can be imposed. In order to isolate the solution of (0.2) that determines the minimizer of (0.1), we impose the conditions that the solution is non-negative and bounded from above. Probably the most challenging part of the article is to prove that there exists a unique solution of the Riccati equation (0.2) satisfying these boundedness conditions.

Using the unique bounded non-negative solution of (0.2) on \([0, \infty )\) we define a specific control and show, via a classical verification argument, that it is indeed optimal. In contrast to the homogeneous case, the HJB equation characterizing the control problem does depend on time. This goes in line with the fact that the optimally controlled state dynamics are, again in contrast to the homogeneous case, not necessarily ergodic.

There are many articles that solve long-term average cost control problems with time-homogeneous state dynamics. We refer to [19] for an early survey. In homogeneous models the optimally controlled state dynamics usually are ergodic. Therefore, the literature frequently refers to such problems as ergodic control problems. One message of the current paper is that long-term average cost control problems can be well-posed, even without ergodicity of the optimally controlled state.

A fundamental topic in the field of control theory with long-term average cost functionals is the convergence of the HJB equations of the finite time problem version to an ergodic PDE. More precisely, assume that the HJB equation of a finite time control problem is given by

$$\begin{aligned} -\partial _t v - \inf _{a \in A} \left\{ \mathcal {L}^a v + f(t,x,a) \right\} = 0, \end{aligned}$$
(0.3)

where A is the value set of the controls and \(\mathcal {L}^a\) denotes the generator of the controlled state dynamics. There are many contributions providing conditions under which (0.3) transforms into an ergodic PDE of the type

$$\begin{aligned} \eta - \inf _{a \in A} \left\{ \mathcal {L}^a v + {\tilde{f}}(x,a) \right\} = 0 \end{aligned}$$
(0.4)

as the time horizon converges to infinity. Notice that a solution of (0.4) consists of a pair \((\eta , v) \in \mathbb {R}\times \mathcal {C}[0, \infty )\). Usually it is assumed that f does not depend on time. Exceptions are [2, 4] assuming a periodicity in time, and [5] assuming that f depends recursively on the value function divided by time-to-maturity.

[1, 14] consider a homogeneous setting and prove convergence, in some sense, of (0.3) to (0.4) under some state periodicity assumptions. [5, 8, 18] use probabilistic representations in terms of backward stochastic differential equations to establish convergence under dissipativity assumptions guaranteeing that the optimally controlled state is ergodic. [9] consider a system of ergodic BSDEs with dissipative forward part and apply them to a long-term utility maximization problem with regime switching.

We stress that in the present article we do not impose any kind of time periodicity assumption. The only assumption on the state coefficients and the cost coefficients is that they are bounded and left-continuous. As the time horizon converges to infinity, the time dependence in the HJB (0.3) does, in general not disappear, and hence we do not have convergence to (0.4). A time-dependent, but periodic, PDE limit is also described in [4].

Finally, we remark that we do not impose any regularity with respect to time, and hence we can not transform the setting into a 2-dimensional homogeneous setting with time as a new state variable.

2 Main Results

In this section we rigorosly describe the model and summarize our main results.

Let W be a one-dimensional Brownian motion on a probability space \((\Omega , \mathcal {F}, P)\). We denote by \((\mathcal {F}_t)_{t \in [0, \infty )}\) the filtration generated by W, completed by the P-null sets in \(\mathcal {F}\).

Assumption 1.1

Let \(f: [0,\infty ) \times \mathbb {R}\times \mathbb {R}\rightarrow \mathbb {R}\) be of the form

$$\begin{aligned} f(t,x, a)&= \beta _{xx}(t) x^2 + \beta _x(t) x + \beta _{xa}(t) ax + \beta _{aa}(t) a^2 + \beta _a(t) a + \beta _0(t) \end{aligned}$$

and \(b,B,c,C,\beta _{xx}, \beta _{x}, \beta _{xa}, \beta _{aa}, \beta _a, \beta _0: [0,\infty ) \rightarrow \mathbb {R}\) be deterministic, left-continuous and bounded functions. Moreover, assume that

  • \(\det ( \mathcal {H}(f))(t,\cdot ,\cdot ) = 4 \beta _{aa}(t) \beta _{xx}(t) - \beta _{ax}^2(t) \ge \epsilon _1 > 0\) for \(t \in [0, \infty )\) and some constant \(\epsilon _1 > 0\),

  • \(\beta _{aa}(t) \ge \epsilon _2 > 0\) for \(t \in [0, \infty )\) and some constant \(\epsilon _2 > 0\).

By a control process \(\alpha \) we mean a \((\mathcal {F}_t)\)-progressively measurable process \(\alpha \) such that for all \(T \in [0, \infty )\) we have \(\int _0^T \alpha _s^2 ds < \infty \). Given a control \(\alpha \), we assume that state process satisfies the SDE

$$\begin{aligned} dX_{t} = (b_{t} + B_{t} X_{t} -\alpha _{t}) dt + (c_{t} + C_{t} X_{t}) dW_{t}. \end{aligned}$$
(1.1)

Notice that our assumptions imply that for every \(x \in \mathbb {R}\) the SDE (1.1) has a unique solution \(X^{x, \alpha }\) satisfying \(X^{x,\alpha }_0 = x\). Moreover, one can show that for all \(p \in [1, \infty )\) and \(T \in [0, \infty )\) we have \(\sup _{t \in [0,T]} E|X^{x, \alpha }_t|^p < \infty \) (see Sect. 2.5 in [13]).

Given an initial state \(x \in \mathbb {R}\), we say that a control \(\alpha \) is admissible if \(\sup _{t \in [0, \infty )} E[(X^{x, \alpha }_t)^2] < \infty \); and we denote the set of all admissible controls by \(\mathcal {A}(x)\).

For the admissible controls \(\alpha \in \mathcal {A}(x)\) we define the limsup long term average cost functional

$$\begin{aligned} {\bar{J}}(x,\alpha ) = \limsup _{T \rightarrow \infty } E \frac{1}{T}\int _0^T f(s,X^{x, \alpha }_s, \alpha _s) ds. \end{aligned}$$

We now consider the problem of minimizing \({\bar{J}}(x,\alpha )\) among all admissible controls. To this end we introduce the value function

$$\begin{aligned} {\bar{V}}(x) : = \inf _{\alpha \in \mathcal {A}{(x)}} \bar{J} (x, \alpha ), \end{aligned}$$
(1.2)

for all \(x \in \mathbb {R}\). We show below that \({\bar{V}}\) does not depend on x; but since this is a priorily not known, in the definition of \({\bar{V}}\) we add the argument x.

We say that \(\alpha \in \mathcal {A}(x)\) is an optimal control for (1.2), if we have \({\bar{J}}(x, \alpha ) = {\bar{V}}(x)\). Moreover, we say that \(\alpha \in {\mathcal {A}(x)}\) is a closed-loop control if there exists a function \(a:[0, \infty ) \times \mathbb {R}\rightarrow \mathbb {R}\) such that for all \(x \in \mathbb {R}\) the SDE

$$\begin{aligned} dX_t = (b_t+B_t X_t-a(t, X_t)) dt + (c_t + C_t X_t) dW_t \end{aligned}$$
(1.3)

has a unique solution \(X^{x,a}\) and \(\alpha _t = a(t, X^{x,a}_t)\), \(t \in [0, \infty )\).

We now summarize our main results. First, we describe an optimal control and the value function in terms of a solution of the Riccati equation (0.2). We show that there exists a unique initial condition such that equation (0.2) has on \([0, \infty )\) a solution that is bounded from above and bounded from below by 0.

Proposition 1.2

There exists exactly one non-negative and bounded solution of (0.2) on \([0, \infty )\).

The result of Proposition 1.2 is proved in Sect. 2 as a part of Theorem 2.1. In the following we denote by \(U^\infty \) the unique non-negative bounded solution of (0.2) described in Proposition 1.2.

In Sect. 3 we show that there exist constants \(\delta _1, \delta _2 > 0\) such that

$$\begin{aligned} \int _s^t \left( B_r + \frac{ \beta _{xa}(r) - U^\infty _r}{ 2 \beta _{aa}(r) } \right) {\text {d}} r \le - \delta _1 (t-s) + \delta _2 \end{aligned}$$

for all \(0 \le s \le t < \infty \). We can thus define a further bounded process

$$\begin{aligned} \phi ^\infty _t&:= \int _t^\infty \left[ U^\infty _s \left( b_s + c_s C_s + \frac{ \beta _{a}(s) }{ 2 \beta _{aa}(s)} \right) - \frac{ \beta _{a}(s) \beta _{xa}(s) }{ 2 \beta _{aa} (s) } + \beta _{x}(s) \right] \\ {}& \cdot \exp \left( \int _t^s B_r + \frac{ \beta _{xa}(r) - U^\infty _r }{2 \beta _{aa} (r) } {\text {d}} r \right) {\text {d}} s . \end{aligned}$$

We next describe a solution of the long term cost minimization problem in terms of \(U^\infty \) and \(\phi ^\infty \).

Theorem 1.3

The closed-loop control with feedback function

$$\begin{aligned} a^\infty (t,x) = \frac{ \phi ^\infty _t - \beta _a(t) + (U^\infty _t - \beta _{xa}(t) ) x }{ 2 \beta _{aa}(t)} \end{aligned}$$
(1.4)

is an optimal control. Moreover,

$$\begin{aligned} {\bar{V}}(x) = \limsup _{t \rightarrow \infty } \frac{1}{t} \int _0^t \bigg ( \phi ^\infty _s b_s + U^\infty _s \frac{ c_s^2 }{2} + \beta _0 (s) - \frac{ \left( \phi ^\infty _s - \beta _{a} (s) \right) ^2 }{ 4 \beta _{aa}(s) } \bigg ) {\text {d}} s. \end{aligned}$$
(1.5)

Note that (1.5) implies that \({\bar{V}}\) does not depend on x. In the following we therefore omit the argument x and interpret \({\bar{V}}\) as a constant.

Furthermore, the optimal control is not unique. An alteration of the strategy in (1.4) on a compact interval of time would move the process as if having started in another value and generate limited costs. Changing the starting value and modifying costs on a compact interval does not affect the long term average costs \({\bar{V}}\). Thus the alteration is also optimal.

We prove Theorem 1.3 in Sect. 3 as a part of Theorem 3.6. We next proceed by comparing the problem of minimizing \({\bar{J}}(x, \alpha )\) with the problem of minimizing the liminf long term average cost functional

$$\begin{aligned} \underline{J}(x,\alpha ) = \liminf _{T \rightarrow \infty } E \frac{1}{T}\int _0^T f(s,X^{x, \alpha }_s, \alpha _s) ds. \end{aligned}$$
(1.6)

We define also the liminf value

$$\begin{aligned} \underline{V} : = \inf _{\alpha \in \mathcal {A}{(x)}} \underline{J}(x,\alpha ). \end{aligned}$$
(1.7)

One can show that the feedback function (1.4) is also optimal for (1.7) and that \(\underline{V}\) does not depend on x. Moreover, we have

$$\begin{aligned} \underline{V} = \liminf _{t \rightarrow \infty } \frac{1}{t} \int _0^t \bigg ( \phi ^\infty _s b_s + U^\infty _s \frac{ c_s^2 }{2} + \beta _0 (s) - \frac{ \left( \phi ^\infty _s - \beta _{a} (s) \right) ^2 }{ 4 \beta _{aa}(s) } \bigg ) {\text {d}} s. \end{aligned}$$
(1.8)

In general, \(\underline{V}\) is not equal to \({\bar{V}}\). If \(\underline{V} < {\bar{V}}\), then \(X^{\infty ,x}\), the state process controlled with the optimal control \(\alpha ^{\infty ,x}_t = a^\infty (t, X^{\infty ,x}_t)\), is not ergodic, i.e. it does not hold true that the cost time average converges almost surely. More precisely, we have the following.

Proposition 1.4

If \(\underline{V} < {\bar{V}}\), then for all \(x \in \mathbb {R}\) the time average

\(\frac{1}{T}\int _0^T f(s,X^{\infty , x}_s, \alpha ^{\infty ,x}_s) ds\) does not converge a.s., as \(T \rightarrow \infty \).

Proof

We first show that the family \(\frac{1}{T} \int _0^T f(s,X^{\infty , x}_s, \alpha ^{\infty ,x}_s) ds\), \(T \in [0, \infty )\), is uniformly integrable. To this end let \(p \in (1, \infty )\). By Jensen’s inequality we have, for some constant K independent of T,

$$\begin{aligned} E\left[ \left| \frac{1}{T} \int _0^T f(s,X^{\infty , x}_s, \alpha ^{\infty ,x}_s) ds \right| ^p \right]&\le E\left[ \frac{1}{T} \int _0^T |f(s,X^{\infty , x}_s, \alpha ^{\infty ,x}_s)|^p ds \right] \\&\le \left[ \frac{1}{T} \int _0^T K(1 + E|X^{\infty , x}_s|^{2p}) ds \right] \\&\le K(1 + \sup _{s \in [0, \infty )} E|X^{\infty , x}_s|^{2p}). \end{aligned}$$

By Lemma 3.3 below there exists a \(p > 1\) such that \(\sup _{s \in [0, \infty )} E|X^{\infty , x}_s|^{2p} < \infty \). Hence, by the de la Vallee-Poussin theorem, the family \(\frac{1}{T}\int _0^T f(s,X^{\infty , x}_s, \alpha ^{\infty ,x}_s) ds\) is uniformly integrable.

Now suppose that \(\frac{1}{T}\int _0^T f(s,X^{\infty , x}_s, \alpha ^{\infty ,x}_s) ds\) converges a.s. Then, due to uniform integrability, we also have convergence in \(L^1\). This contradicts however that \(\underline{V} < {\bar{V}}\).

\(\square \)

Proposition 1.4 entails, in particular, that if \(\underline{V} < {\bar{V}}\), then the distribution of \(X^{\infty ,x}_t\) does not converge to a stationary distribution, as \(t \rightarrow \infty \).

In the homogeneous case where the drift, diffusion and cost functionals do not depend on t, the optimally controlled state \(X^{\infty ,x}\) is ergodic. The homogeneous case is already well studied in the literature (see e.g. [3]). For the convenience of the reader we briefly explain how our results simplify in the homogeneous case and how they can be extended.

2.1 The Homogeneous Case

Suppose that all modelling functions \(b,B,c,C,\beta _{xx}, \beta _{x}, \beta _{xa}, \beta _{aa}, \beta _a, \beta _0\) are constant. In this case also \(U^\infty \) and \(\phi ^\infty \) are constant; in particular we have

$$\begin{aligned} U^\infty = p + \sqrt{p^2 + q}, \end{aligned}$$
(1.9)

where

$$\begin{aligned} p = 2 B \beta _{aa} + \beta _{xa} + C^2 \beta _{aa} \quad \text { and } \quad q = 4 \beta _{xx} \beta _{aa} - \beta _{xa}^2. \end{aligned}$$

Let \(\kappa = B - \frac{U^\infty - \beta _{xa}}{2 \beta _{aa}} \), and notice that the optimally controlled state \(X^\infty \) satisfies the homogeneous SDE

$$\begin{aligned} dX_t = \left( b - \frac{\phi ^\infty - \beta _{a}}{2 \beta _{aa}} + \kappa X_t \right) dt + (c + C X_t) dW_t. \end{aligned}$$
(1.10)

Assumption 1.1 implies that \(q > 0\). Thus, with (1.9) we get \(U^\infty > p\), and hence

$$\begin{aligned} \kappa < - \frac{C^2}{2}. \end{aligned}$$
(1.11)

Property (1.11), sometimes referred to as dissipativity, guarantees that (1.10) possesses a unique stationary distribution \(\pi \) (see, e.g., Theorem 8.3 in [17]; use for example the Lyapunov function \(W(x) = x^2/2\)). Moreover, if \(X^{\infty , x}\) denotes the solution of (1.10) with initial condition \(x \in \mathbb {R}\), then the distribution of \(X^{\infty , x}_t\) converges to the stationary distribution, as \(t \rightarrow \infty \) (see Remark 8.6 in [17]). This further entails that \(\frac{1}{T}\int _0^T f(X^{\infty , x}_s, a^\infty (s,X^{\infty , x}_s)) ds\) converges a.s. to \(\int f(x, a^{\infty }(x)) \pi (dx)\), as \(T \rightarrow \infty \).

2.2 Dissipativity in the Inhomogeneous Case

Observe that the optimally controlled state \(X^\infty \) satisfies the SDE

$$\begin{aligned} {\text {d}} X_t = \left( b_t - \frac{\phi ^\infty _t - \beta _{a}(t)}{2 \beta _{aa}(t)} + \kappa _t X_t \right) {\text {d}} t + \left( c_t + C_t X_t \right) {\text {d}} W_t , \end{aligned}$$

where \(\kappa _t = B_t - \frac{U^\infty _t - \beta _{xa}(t)}{2 \beta _{aa}(t)}\). By Theorem 2.1 below we obtain that there are constants \(\delta _1,\delta _2 > 0\) such that for all \(0 \le t_1 \le t_2 < \infty \)

$$\begin{aligned} \int _{t_1}^{t_2} \left( \kappa _t + \frac{C_t^2}{2} \right) {\text {d}} t \le - \delta _1 (t_2 - t_1) + \delta _2. \end{aligned}$$

This implies, that for large enough time intervals \([t_1,t_2]\) we have

$$\begin{aligned} \int _{t_1}^{t_2} \kappa _t {\text {d}} t < \int _{t_1}^{t_2} - \frac{C_t^2}{2} {\text {d}} t, \end{aligned}$$

which seems to be a time-average version of the dissipativity condition (1.11).

However, consider \(B_t = 2 \cdot \mathbbm {1}_{\{t \in [0,1) \}}\) and all other parameters to be constant with \(C = 1\), \(\beta _{aa} = \frac{1}{3}\), \(\beta _{xx} = \frac{1}{4}\) and \(b = \beta _{xa} = \beta _x = \beta _a = \beta _0 = 0\). For \(t \ge 1\) we have that 1 is a solution of (0.2), and hence \(U^\infty _t = 1\) for all \(t \ge 1\). Furthermore, \(\kappa _1 = - \frac{3}{2}\). Since \(U^\infty \) is continuous, there is an \(\epsilon >0\) such that for all \(t \in [1-\epsilon ,1)\) we have

$$\begin{aligned} -\frac{1}{2} = -\frac{ C_t^2}{2}< 0 < \kappa _t, \end{aligned}$$

which means that for at least a short time the condition (1.11) is not satisfied.

2.3 A Non-ergodic Example

Example 1.5

Consider the control problem with \(C = 1\), \(\beta _{aa} = \frac{1}{3}\), \(\beta _{xx} = \frac{1}{4}\) and \(b = B = \beta _{xa} = \beta _x = \beta _a = \beta _0 = 0\). Below we define recursively a sequence of increasing times \(0=t_0< t_1< t_2 < \cdots \). Given this sequence we set

$$\begin{aligned} c_t = \left\{ \begin{array}{ll} 1, &{} \text { if } t \in [t_{2k}, t_{2k+1}) \text { for a } k \in \mathbb {N}_0, \\ 2, &{} \text { if } t \in [t_{2k+1}, t_{2k+2}) \text { for a } k \in \mathbb {N}_0. \end{array} \right. \end{aligned}$$

First, observe that the function constant equal to 1 is a solution of (0.2), and hence \(U^\infty = 1\). Suppose that \(t_{2k}\) is defined. Observe that

$$\begin{aligned} \lim _{T \rightarrow \infty } \int _{t_{2k}}^T e^{-\frac{3}{2} (s-t_{2k})} ds = \frac{2}{3}. \end{aligned}$$

Thus, the larger we choose \(t_{2k+1}\), the closer \(\phi ^\infty _{s}\), \(s \in [t_{2k}, (t_{2k} + t_{2k +1})/2]\), gets to \(\frac{2}{3}\). Now we choose \(t_{2k+1}\) such that

$$\begin{aligned} \frac{1}{(t_{2k} + t_{2k +1})/2} \int _0^{(t_{2k} + t_{2k +1})/2} (\frac{1}{2} - \frac{3}{4} \phi ^\infty _s) ds \le \frac{1}{k}. \end{aligned}$$

We next describe how to choose \(t_{2k +2}\). Observe that

$$\begin{aligned} \lim _{T \rightarrow \infty } \int _{t_{2k+1}}^T 2 e^{-\frac{3}{2} (s-t_{2k+1})} ds = \frac{4}{3}. \end{aligned}$$

Therefore, the larger we choose \(t_{2k+2}\), the closer \(\phi ^\infty _{s}\), \(s \in [t_{2k+1}, (t_{2k+1} + t_{2k +2})/2]\), gets to \(\frac{4}{3}\). Now choose \(t_{2k+2}\) such that

$$\begin{aligned} \frac{1}{(t_{2k+1} + t_{2k +2})/2} \int _0^{(t_{2k+1} + t_{2k +2})/2} (2 - \frac{3}{4} \phi ^\infty _s) ds \ge 1-\frac{1}{k}. \end{aligned}$$

We have thus recursively defined the sequence \((t_k)_{k \in \mathbb {N}_0}\).

From (1.5) and (1.8) we now obtain \({\bar{V}} \ge 1\) and \(\underline{V} \le 0\).

2.4 Comparison with the Finite Time Control Problem

The optimal control in (1.4) has a similar form as the corresponing optimal control with a finite time horizon \(T \in (0, \infty )\). Indeed, let \((U^T_t)_{t \in [0, T]}\) be the solution of the Riccati equation (0.2) on [0, T] with terminal condition \(U_T = 0\), and let for all \(t \in [0, T]\)

$$\begin{aligned} \phi ^T_t&= \int _t^T \left[ U^T_s (b_s + c_s C_s) + \beta _{a}(s) \frac{ U^T_s - \beta _{xa}(s) }{ 2 \beta _{aa} (s) } + \beta _{x}(s) \right] \\&\quad \exp \left( \int _t^s B_r + \frac{ \beta _{xa}(r) - U^T_r}{ 2 \beta _{aa}(r) } {\text {d}} r \right) {\text {d}} s . \end{aligned}$$

If we replace \(U^\infty \) with \(U^T\) and \(\phi ^\infty \) with \(\phi ^T\) in (1.4), then we obtain an optimal closed loop control for the problem of minimizing \(E \int _0^T f(t, X^{x, \alpha }_t, \alpha _t)dt\) (see, e.g., Theorem 2.4.3 in [20]). Moreover, one can show that \(U^T\) and \(\phi ^T\) converge to \(U^\infty \) and \(\phi ^\infty \), respectively and hence the optimal feedback function of the finite horizon problem converges to \(a^\infty \) as \(T \rightarrow \infty \) (see Chapter 4 in [6]).

We remark that in the finite time linear quadratic control problem with stochastic (more precisely: progressively measurable) coefficients the equation (0.2) becomes a backward stochastic differential equation (BSDE). If the BSDE is solvable, the control problem is well-posed and its solution can be characterized in terms of the BSDE (see, e.g., [15, 11, 12] and [21]).

2.5 Applications

We close this section by describing some possible applications of the solution of the long-term average cost minimization problem (1.2).

Inventory management. One can think of the state X as the inventory level of some good. Usually a low inventory level entails shortage costs and a high level increases holding costs. Both costs can be taken into accout for by the quadratic dependence of f on x. With the control \(\alpha \) the inventory manager can continuously adjust the inventory level, in both directions. Quadratic dependence of f on \(\alpha \) reflects that level corrections imply costs. The demand of the good and adjustment costs may be subject to seasonal variations and to some long-term trends, allowed for by the time-inhomogenity of the cost function f. In this context (1.4) is the policy that minimizes the long term average inventory costs.

Cash balance management. Companies aim for an optimal cash balance (see, e.g., [7]). On the one hand, they want to avoid being short of cash for meeting obligations. On the other hand they want to avoid holding costs entailed by large cash positions. Any adjustment of the cash position involves transaction costs. The problem of minimizing the long term average overall costs can be formulated as a problem of type (1.2).

Inflation rate regulation. One of the main tasks of central banks is to keep the inflation rate at a healthy level. Both a high inflation rate and deflation can have severe economic implications. To this end there are several tools at the disposal of central banks in oder to affect the inflation rate into either direction, which however also come with side effects of political or economical nature. For example bubbles in stock markets or recessions can be unfavorable outcomes. The parameters of cost of inflation or deflation and the measures against them can change over time with the circumstances. Furthermore, a central bank should aim at presevering a near optimum state for a long time without any visible time horizon, which makes taking the average over time a good target functional.

3 Existence and Uniqueness of \(U^\infty \)

In this section we show the existence and uniqueness of \(U^\infty \), which is defined as the non-negative bounded solution of (0.2). In fact, we show a little more than that, as can be seen in the following theorem, which contains the main result of this section.

Theorem 2.1

Let Assumption 1.1 be fulfilled.

Then there exists exactly one \(u \in \mathbb {R}\) such that (0.2) with the initial condition \(U_0 = u\) has a solution that is on \([0, \infty )\) bounded from below by 0 and bounded form above by \(\hat{U} := \hat{p} + \sqrt{ \hat{p}^2 + \hat{q} }\) , where \(\hat{p} := \sup \limits _{s \in [0,\infty )} \left( 2 B_s \beta _{aa}(s) + \beta _{xa}(s) + C_s^2 \beta _{aa}(s) \right) \) and \(\hat{q} := \sup \limits _{s \in [0,\infty )} \left( 4 \beta _{xx}(s) \beta _{aa} (s) - \beta _{xa}^2 (s) \right) \).

Furthermore, there are constants \(\delta _1, \delta _2 > 0\) such that for any initial value \(U_0\) yielding \(U_r \in [0, \hat{U}]\) for all \(0 \le r \le T < \infty \) we have

$$\begin{aligned}&\int _s^t \left( B_r + \frac{ \beta _{xa}(r) - U_r}{ 2 \beta _{aa}(r) } \right) {\text {d}} r \\&\quad \le \int _s^t \left( B_r + \frac{ \beta _{xa}(r) - U_r}{ 2 \beta _{aa}(r) } + \frac{ C_r^2 }{2} \right) {\text {d}} r \le - \delta _1 (t-s) + \delta _2 \end{aligned}$$

for all \(0\le s \le t \le T\).

We approach this problem by considering a simplified quadratic integral equation, at first for constant and then for piecewise constant parameter functions. Finally we generalize to right-continuous functions and prove Theorem 2.1 via a time-reversal.

Assumption 2.2

Let \(p,q,a: { \mathbb {R}} \rightarrow \mathbb {R}\) be deterministic right-continuous functions such that for all \(s { \in \mathbb {R}}\)

$$\begin{aligned} -\infty< \check{p} \le p_s \le \hat{p}< \infty , \qquad 0< \check{q} \le q_s \le \hat{q}< \infty , d \qquad 0< \check{a} \le a_s \le \hat{a} < \infty \qquad \end{aligned}$$

for constants \(\check{p},\hat{p},\check{q},\hat{q},\check{a},\hat{a} \in \mathbb {R}\).

For the following we define the constants \(\check{Y}:= \check{p} + \sqrt{ \check{p}^2 + \check{q} }\), \( \hat{Y} := \hat{p} + \sqrt{ \hat{p}^2 + \hat{q} }\). Moreover, for all \(t \in \mathbb {R}\) and \(x \in [0,\hat{Y}]\) we define \(Y^{t,x}\) as the solution of the ODE

$$\begin{aligned} y' = - a_t \left( y^2 - 2 p_t y - q_t \right) \end{aligned}$$
(2.1)

on \([t, \infty )\) with initial condition \(Y^{t,x}_t = x\). To shorten notation, we often omit the superscript t and x.

Remark 2.3

Actually it is not necessary for p, q, a to be defined on the complete real axes. Being defined on an interval of the form \((-\infty , K]\) for \(K\in \mathbb {R}\) would suffice. Being right-continuous is also not necessary. What we actually use is firstly in the proof of Proposition 2.10 that p, q, a can be approximated by piecewise constant functions with respect to the \({{\,\mathrm{ess\,sup}\,}}\)-norm and secondly in the proof of Theorem 2.11 that \(Y^{t,x}\) is weakly differentiable with respect to its inital value x. However, for simplicity of argument and notation we assume right-continuity.

Lemma 2.4

Let Assumption 2.2 be fulfilled and \({ t \in \mathbb {R}}\). Then for every starting value \(x \in [0, \hat{Y}]\) Equation (2.1) has a unique solution \(Y^{t,x}\), which is furthermore bounded by

$$\begin{aligned} \min \left\{ x, \check{Y} \right\} \le Y^{t,x}_s \le \hat{Y}, \qquad {\text { for all }s \ge t.} \end{aligned}$$

Proof

We define the auxiliary process \(\tilde{Y}\) as the unique solution of the Lipschitz ODE

$$\begin{aligned} \partial _t \tilde{Y}_s = - a_s \left( \left( \mathcal {T}^{\hat{Y}}_0 (\tilde{Y}_s ) \right) ^2 - 2 p_s \tilde{Y}_s - q_s \right) , \qquad \tilde{Y}_t = Y_t, \end{aligned}$$
(2.2)

where \(\mathcal {T}\) is the truncation operator defined by \(\mathcal {T}_\alpha ^\beta (x) := \max \left( \alpha , \min \left( x, \beta \right) \right) \) for \(\alpha \le \beta \). Observe that for \(\tilde{Y}_s \in [0, \check{Y} )\) we have \(-a_s \big ( \big ( \mathcal {T}^{\hat{Y}}_0 (\tilde{Y}_s ) \big )^2 - 2 p_s \tilde{Y}_s - q_s \big ) > 0\) and for \(\tilde{Y}_s \in [ \hat{Y}, \infty )\) that \(-a_s \big ( \big ( \mathcal {T}^{\hat{Y}}_0 (\tilde{Y}_s ) \big )^2 - 2 p_s \tilde{Y}_s - q_s \big ) \le 0\). Hence, for \(\tilde{Y}_t < \check{Y}\) we have that \(\tilde{Y}_s \ge \tilde{Y}_t\) for all \(s \in [t,\infty )\), since Y is continuous. By the same argument we also obtain for \(\tilde{Y}_t \ge \check{Y}\) that \(\tilde{Y}_s\) cannot reach any value below \(\check{Y}\) and likewise because \(\tilde{Y}_t \le \hat{Y}\) that \(\tilde{Y}_s \le \hat{Y}\). Thus, the truncation of the quadratic term has no consequence and can be omitted without changing the solution. Hence, \(\tilde{Y}\) is also a solution of the untruncated ODE (2.1).

Let Z be an arbitrary solution of (2.1). If \(Z_t\) attains \(\hat{Y}\), then it has a non-positive derivative in that point and hence cannot exceed \(\hat{Y}\). Similarily, if Z attains zero, then it has a non-negative derivative and hence cannot plunge below zero. Consequently, also Z is a solution of the Lipschitz ODE (2.2). However, uniqueness of (2.2) implies \(Z = \tilde{Y}\). \(\square \)

In the following we denote by Y the solution of Equation (2.1).

Remark 2.5

In the proofs of this section we make use of the following hyperbolic identities without explicitly mentioning it:

  • \(\tanh ^{-1}(x) = \frac{1}{2} \ln \left( \frac{ 1 + x }{ 1 - x } \right) \) for \(x \in (-1,1)\),

  • \(\coth ^{-1}(x) = \frac{1}{2} \ln \left( \frac{ x+1 }{ x-1 } \right) \) for \(\vert x \vert > 1\),

  • \(\cosh (\tanh ^{-1}(x)) = ( 1 - x^2 )^{-1/2}\) for \(x \in (-1,1)\),

  • \(\sinh (\coth ^{-1}(x)) = ( 1 - x^2 )^{-1/2}\) for \(x > 1\) .

Lemma 2.6

Let Assumption 2.2 be fulfilled, \({ t \in \mathbb {R}}\) and \(x \in [0, \hat{Y}]\). Furthermore, assume that for some \(s > t\) the functions p, q, a are constant on the interval [t, s), i.e. there are \(\bar{p},\bar{q},\bar{a} \in \mathbb {R}\) such that \(p_r = \bar{p}\), \(q_r = \bar{q}\) and \(a_r = \bar{a}\) for all \(r \in [t,s)\). Then

$$\begin{aligned} Y^{t,x}_r = \left\{ \begin{array}{ll} \bar{p}+\sqrt{\bar{p}^2+\bar{q}} \tanh \left( \bar{a} \sqrt{\bar{p}^2+\bar{q}} (r -t) + \tanh ^{-1} \left( \frac{ x - \bar{p} }{ \sqrt{\bar{p}^2+\bar{q}}} \right) \right) , &{} x \in \big [ 0,\bar{p}+\sqrt{\bar{p}^2+\bar{q}} \big ) \\ \bar{p}+\sqrt{\bar{p}^2+\bar{q}}, &{} x = \bar{p}+\sqrt{\bar{p}^2+\bar{q}} \\ \bar{p}+\sqrt{\bar{p}^2+\bar{q}} \coth \left( \bar{a} \sqrt{\bar{p}^2+\bar{q}} (r -t) + \coth ^{-1} \left( \frac{ x - \bar{p} }{ \sqrt{\bar{p}^2+\bar{q}}} \right) \right) , &{} x \in \big ( \bar{p}+\sqrt{\bar{p}^2+\bar{q}},\infty \big ) \end{array} \right. \end{aligned}$$
(2.3)

for all \(r \in [t,s]\). In particular, \(Y^{t,x}\) is monotone on the interval [t, s].

Proof

Observe that the dynamics of \(Y^{t,x}\) can be reformulated for \(r \in [t,s)\) as the separable ODE

$$\begin{aligned} Y_r' = - \bar{a} \left( \left( Y_r - \bar{p} \right) ^2 - \bar{p}^2 - \bar{q} \right) . \end{aligned}$$

By using the method of separation of variables, the three cases follow. Also, Lemma 2.4 provides uniqueness. The remaining monotonicity follows from the monotonicity of \(\tanh \) and \(\coth \). \(\square \)

The proofs of the following three lemmas are technical and can be found in the appendix.

Lemma 2.7

Let Assumption 2.2 be fulfilled and \([t_1, t_2] \subset { \mathbb {R}}\) with \(t_1 < t_2\). Furthermore, assume that \(x \in [0, \hat{Y}]\) and that the functions p, q, a are constant on the interval \([t_1,t_2)\), i.e. there are \(\bar{p},\bar{q},\bar{a} \in \mathbb {R}\) such that \(p_r = \bar{p}\), \(q_r = \bar{q}\) and \(a_r = \bar{a}\) for all \(r \in [t_1,t_2)\). Then, for \(Y:= Y^{t_1,x}\) and \(t_1 \le t \le s \le t_2\),

$$\begin{aligned} \int _t^s - a_r \left( Y_r - p_r \right) {\text {d}} r = \left\{ \begin{array}{ll} - \bar{a} \sqrt{\bar{p}^2 + \bar{q}} (s - t), &{} Y_t = \bar{p} + \sqrt{\bar{p}^2 + \bar{q}} \\ \frac{1}{2} \ln \left( \frac{ Y_t^2 - 2 \bar{p} Y_t - \bar{q} }{ Y_s^2 - 2 \bar{p} Y_s -\bar{q} } \right) ,&{} Y_t \ne \bar{p} + \sqrt{\bar{p}^2 + \bar{q}} \end{array} \right. \end{aligned}$$
(2.4)

and for \(Y_t \ne \bar{p} + \sqrt{ \bar{p}^2 + \bar{q} }\) we moreover have

$$\begin{aligned} s - t = \frac{ 1 }{ 2 \bar{a} \sqrt{\bar{p}^2 + \bar{q}} } \ln \left( \frac{ \bar{p}^2 + \bar{q} - (Y_t - \bar{p})^2 }{ \bar{p}^2 + \bar{q} - (Y_s - \bar{p})^2 } \right) + \frac{ 1 }{ \bar{a} \sqrt{\bar{p}^2 + \bar{q}} } \ln \left( \frac{ \sqrt{ \bar{p}^2 + \bar{q} } + (Y_s - \bar{p}) }{ \sqrt{ \bar{p}^2 + \bar{q} } + (Y_t - \bar{p}) } \right) . \end{aligned}$$

Proof

See appenix. \(\square \)

Lemma 2.7 gives us the value of the integral in (2.4) when the parameters are constant all the way. Next, we want to find the value of that integral when the process Y goes up and down ending at the value where it started, which we later call an excursion.

Lemma 2.8

Let Assumption 2.2 be fulfilled. Furthermore, let \([t_1,t_2],[t_3,t_4] \subset { \mathbb {R}}\), \(Y_{t_1} = Y_{t_4} \in [0, \hat{Y}]\), \(Y_{t_2} = Y_{t_3} \in [0, \hat{Y}]\) and p, q, a be constant on \([t_1,t_2)\) and also on \([t_3,t_4)\). Then

$$\begin{aligned} \int _{t_1}^{t_2} - a_s ( Y_s - p_s ) {\text {d}} s + \int _{t_3}^{t_4} - a_s ( Y_s - p_s ) {\text {d}} s \le - \min \left\{ \check{a} \sqrt{\check{q}}, \frac{ \hat{a} ( \check{Y}^2 + \check{q} ) }{ 2 \hat{Y} } \right\} ((t_2-t_1) + (t_4-t_3)). \end{aligned}$$

Proof

See appendix. \(\square \)

Lemma 2.9

Let Assumption 2.2 be fulfilled and assume that on the interval \([t_0,t_1)\) with \(\left. { -\infty }< t_0< t_1 < \infty \right. \) the functions p, q, a are constant and \(Y_{t_0} \in [0, \hat{Y}]\). Then

$$\begin{aligned} \int _{t_0}^{t_1} - a_s (Y_s - p_s) {\text {d}} s \le - \check{a} \frac{ \sqrt{\check{q}}}{ \sqrt{2} } (t_1 - t_0 ) + \frac{2 \hat{Y}}{\check{q}} \left( Y_{t_1} - Y_{t_0} \right) \mathbbm {1}_{ \{ Y_{t_1} - Y_{t_0} > 0 \} }. \end{aligned}$$
(2.5)

Proof

See appendix. \(\square \)

Proposition 2.10

Let Assumption 2.2 be fulfilled, \({ -\infty }< t_0 \le t_1 < \infty \) and \(Y_{t_0} \in [0, \hat{Y}]\). Then there exist constants \(\delta _1, \delta _2 > 0\) independent of \(t_0\) and \(t_1\) such that

$$\begin{aligned} \int _{t_0}^{t_1} - a_s ( Y_s - p_s ) {\text {d}} s \le - \delta _1 (t_1-t_0) + \delta _2. \end{aligned}$$

Proof

First, we have a look at the case, where p, q, a are piecewise constant. We split the path of Y into many excursions (as described in Lemma 2.8) and left over time intervals which can not be put together to excursions. Those left over time intervals have to be such that either Y is monotone decreasing or Y is monotone increasing on all of them. Since \(0 \le Y \le \hat{Y}\) (see Lemma 2.4) we get from Lemma 2.9 that the contributions of the left over monotone intervals in the estimate are bounded by \(2 \frac{\hat{Y}}{\check{q}} \hat{Y} =: \delta _2\). Now we set

$$\begin{aligned} \delta _1 := \min \left( \hat{a} \frac{ \check{Y}^2 + \check{q} }{ 2 \hat{Y} }, \frac{ \check{a} \sqrt{ \check{q} }}{ \sqrt{ 2} } \right) , \end{aligned}$$

which is the minimum of the factors that get multiplied with the time increments, given in Lemma 2.8 and Lemma 2.9. Hence, the result holds for all piecewise constant functions p, q, a uniformly.

Since Y depends continuously on a, p and q, for every \(\epsilon _1 > 0\) we can choose piecewise constant approximations \(\tilde{a}\), \(\tilde{p}\), \(\tilde{q}\) fulfilling Assumption 2.2 for the same bounds as a, p, q and generating a \(\tilde{Y}\) such that \(\max \big ( \Vert a - \tilde{a} \Vert _{\infty ,[t_0,t_1]}, \Vert p - \tilde{p} \Vert _{\infty ,[t_0,t_1]}, \Vert q - \tilde{q} \Vert _{\infty ,[t_0,t_1]} \Vert Y - \tilde{Y} \Vert _{\infty ,[t_0,t_1]} \big ) < \epsilon _1.\) Now observe that

$$\begin{aligned}&\left| \int _{t_0}^{t_1} - a_s ( Y_s - p_s ) {\text {d}} s - \int _{t_0}^{t_1} - \tilde{a}_s ( \tilde{Y}_s - \tilde{p}_s ) {\text {d}} s \right| \\&= \left| \int _{t_0}^{t_1} - ( a_s - \tilde{a}_s ) ( \tilde{Y}_s - \tilde{p}_s ) - a_s ( Y_s - \tilde{Y}_s - (p_s - \tilde{p}_s ) ) {\text {d}} s \right| \\&\le \Vert a - \tilde{a} \Vert _\infty \left| \int _{t_0}^{t_1} \tilde{Y}_s - \tilde{p}_s {\text {d}} s \right| + ( \Vert Y - \tilde{Y} \Vert _\infty + \Vert p - \tilde{p} \Vert _\infty ) \int _{t_0}^{t_1} a_s {\text {d}} s \\&\le \Vert a - \tilde{a} \Vert _\infty T ( \hat{Y} + \max \{ \vert \hat{p} \vert , \vert \check{p} \vert \} ) + ( \Vert Y - \tilde{Y} \Vert _\infty + \Vert p - \tilde{p} \Vert _\infty ) T \hat{a}. \end{aligned}$$

Hence, we can choose for every \(\epsilon _2 > 0\) our \(\epsilon _1\) as \(\epsilon _1 = \frac{\epsilon _2}{3T} \frac{ 1 }{ \hat{Y} + \max \{ \vert \hat{p} \vert , \vert \check{p} \vert \} + \hat{a} }\) and obtain

$$\begin{aligned} \left| \int _{t_0}^{t_1} - a_s ( Y_s - p_s ) {\text {d}} s - \int _{t_0}^{t_1} - \tilde{a}_s ( \tilde{Y}_s - \tilde{p}_s ) {\text {d}} s \right| \le \epsilon _1 T ( \hat{Y} + \max \{ \vert \hat{p} \vert , \vert \check{p} \vert \} ) + 2 \epsilon _1 T \hat{a} < \epsilon _2. \end{aligned}$$

Thus, the result for piecewise constant functions holds also true for all allowed functions a, p and q. \(\square \)

Theorem 2.11

Let Assumption 2.2 be fulfilled. Then there are constants \(K_1,K_2 > 0\) such that for all \(x_1,x_2 \in [0, \hat{Y}]\) and all \(-\infty< t \le s < \infty \) we have that

$$\begin{aligned}&\int _{t}^{s} - a_r ( Y^{t,x_1}_r - p_r ) {\text {d}} r \le - K_1 (s-t) + K_2 \quad \text { and } \quad \\&\left| Y^{t,x_1}_s - Y^{t,x_2}_s \right| \le \left| x_1 - x_2 \right| K_1 e^{- K_2 (s - t)} . \end{aligned}$$

Furthermore, there exists a bounded function \(V: \mathbb {R}\rightarrow [0, {\hat{Y}}]\) such that for all \(x \in [0,\hat{Y}]\) and \(s \in \mathbb {R}\)

$$\begin{aligned} \lim _{\begin{array}{c} t \rightarrow - \infty \\ t \le s \end{array}} Y^{t,x}_s = V_s. \end{aligned}$$
(2.6)

Moreover, V is the unique process bounded between 0 and \(\hat{Y}\) and solving Equation (2.1).

Remark 2.12

Equation (2.6) means that V can be interpreted as a pullback attractor. More precisely, in the terminology of nonautonomous dynamical systems, the family of singleton sets \(\{V_t\}\), indexed by \(\mathbb {R}\), is pullback attracting for the dynamical system associated to (2.1) (see e.g. Definition 3.3 in [10]).

Proof of Theorem 2.11

The first of the two inequalities is given by Proposition 2.10. For the other one, by introducing the function \(h(r,x):= - a_r \left( x^2 - 2 p_r x - q_r \right) \) for \((r,x) \in [0,\infty ) \times \mathbb {R}\) and using differentiation in its weak sense, we can write the dynamics of \(Y^{t,x_0}\) as

$$\begin{aligned} \partial _s Y^{t,x_0}_s = h(s,Y^{t,x_0}_s), \quad Y^{t,x_0}_t = x_0. \end{aligned}$$

By standard theory (see e.g. Theorem 1 in Chapter 2.5 of [16]) it is known that \(Y^{t,x_0}\) is also differentiable with respect to its initial value \(x_0\) and that \(\partial _{x_0} Y^{t,x_0}_s\) solves the differential equation

$$\begin{aligned} y'(s) = \partial _{x}{h} ( s, Y^{t,x_0}_s) y(s), \qquad y_t = 1, \end{aligned}$$

which has the solution

$$\begin{aligned} \partial _{x_0} Y^{t,x_0}_s = y(s) = \exp \left( \int _t^s \partial _x{h} (r, Y^{t,x_0}_s) {\text {d}} r \right)&= \exp \left( \int _t^s - 2 a_r \left( Y^{t,x_0}_r - p_r \right) {\text {d}} r \right) . \end{aligned}$$

Therefore,

$$\begin{aligned} \partial _{x_0} Y^{t,x_0}_s \le \exp \left( - 2 \delta _1 (s-t) + 2 \delta _2 \right) \end{aligned}$$

for some constants \(\delta _1, \delta _2 >0\) by Proposition 2.10. Hence,

$$\begin{aligned} \left| Y^{t,x_1}_s - Y^{t,x_2}_s \right|&= \left| \int _{x_2}^{x_1} \partial _x Y^{t,x}_s {\text {d}} x \right| \\&\le \left| \int _{x_2}^{x_1} \exp \left( - 2 \delta _1 (s-t) + 2 \delta _2 \right) {\text {d}} x \right| \\&= \left| x_1 - x_2 \right| \exp \left( - 2 \delta _1 (s-t) + 2 \delta _2 \right) . \end{aligned}$$

Thus, defining \(K_1 := e^{2 \delta _2}\) and \(K_2 := 2 \delta _1\) we obtain the claimed inequalities.

Observe that since \( \left| Y^{t,x_1}_s - Y^{t,x_2}_s \right| \le \left| x_1 - x_2 \right| K_1 e^{- K_2 (s - t)} \) we have that \(Y^{t,x}_s\) is a Cauchy-sequence for decreasing t and hence converges. Note that the limit does not depend on the initial value x. We denote this limit by V:

$$\begin{aligned} V_s = \lim _{\begin{array}{c} t \rightarrow - \infty \\ t \le s \end{array}} Y^{t,x}_s. \end{aligned}$$

Furthermore, due to dominated convergence,

$$\begin{aligned} V_s&= \lim _{\begin{array}{c} t \rightarrow - \infty \\ t \le s \end{array}} Y^{t,x}_s \\&= \lim _{\begin{array}{c} t \rightarrow - \infty \\ t \le u \end{array}} Y^{t,x}_u + \lim _{\begin{array}{c} t \rightarrow - \infty \\ t \le u \end{array}} \int _u^s - a_r \left( \left( Y^{t,x}_r \right) ^2 - 2 p_r Y^{t,x}_r - q_r \right) {\text {d}} r \\&= V_u + \int _u^s - \lim _{\begin{array}{c} t \rightarrow - \infty \\ t \le u \end{array}} a_r \left( \left( Y^{t,x}_r \right) ^2 - 2 p_r Y^{t,x}_r - q_r \right) {\text {d}} r \\&= V_u + \int _u^s - a_r \left( \left( V_r \right) ^2 - 2 p_r V_r - q_r \right) {\text {d}} r, \end{aligned}$$

which means that V solves Equation (2.1).

Assuming that there is another process U bounded between 0 and \(\hat{Y}\) and solving Equation (2.1), we can find for every \(\epsilon >0\) and \(s \in \mathbb {R}\) a real \(t<s\) such that

$$\begin{aligned} \left| V_s - U_s \right| \le \left| Y^{t,V_t}_s - Y^{t,U_t}_s \right| \le \left| V_t - U_t \right| K_1 e^{- K_2 (s - t)} < \epsilon , \end{aligned}$$

which means that V and U are identical. \(\square \)

Now we have all necessary tools in order to prove Theorem 2.1.

Proof of Theorem 2.1

We set

$$\begin{aligned}&\tilde{p}_s := 2 B_s \beta _{aa}(s) + \beta _{xa}(s) + C_s^2 \beta _{aa}(s), \qquad \tilde{q}_s := 4 \beta _{xx}(s) \beta _{aa} (s) - \beta _{xa}^2 (s), \\&\qquad \tilde{a}_s := \frac{1}{2 \beta _{aa} (s) } \end{aligned}$$

and define

$$\begin{aligned} p_s := \left\{ \begin{array}{ll} \tilde{p}_{-s}, &{} s \le 0 \\ \tilde{p}_0, &{} s> 0, \end{array} \right. \qquad q_s := \left\{ \begin{array}{ll} \tilde{q}_{-s}, &{} s \le 0 \\ \tilde{q}_0, &{} s> 0, \end{array} \right. \qquad a_s := \left\{ \begin{array}{ll} \tilde{a}_{-s}, &{} s \le 0 \\ \tilde{a}_0, &{} s > 0 \end{array} \right. \end{aligned}$$

such that p, q, a fulfill Assumpiton 2.2. By Theorem 2.11 there exists a unique process V that solves Equation (2.1) and is bounded by 0 and \(\hat{Y}\). For \(s \in [0,\infty )\) we see that by Theorem 2.11\(U^\infty _s := V_{-s}\) fulfills all the claimed properties. \(\square \)

4 Verification of the Linear-Quadratic Non-ergodic Control

In this section we first prove a verification result, and then apply it in order to prove Theorem 1.3.

Recall the control problem of Sect. 1 with value function (1.2). To shorten notation we abbreviate the drift and the diffusion coefficient in the state dynamics (1.1) by

$$\begin{aligned} \mu (t,x,a)&:= b_t + B_t x - a, \\ \sigma (t,x)&:= c_t + C_t x. \end{aligned}$$

We show that one can characterize the solution of the control problem in terms of the following PDE

$$\begin{aligned} \partial _{t}{\psi }(t,x) + \inf _{a \in \mathbb {R}} \left\{ \mu (t,x,a) \partial _x{\psi }(t,x) + \frac{1}{2} \sigma ^2 (t,x) \partial _{xx}{\psi } (t,x) + f (t,x,a)\right\} = 0. \end{aligned}$$
(3.1)

As a terminal condition we impose that there exists \(\eta \in \mathbb {R}\) such that for all \(x \in \mathbb {R}\) we have

$$\begin{aligned} \limsup _{t \rightarrow \infty } - \frac{\psi (t,x)}{t} = \eta . \end{aligned}$$
(3.2)

Proposition 3.1

a) Let \(\psi \in \mathcal {C}^{1,2}([0, \infty ))\) be a function satisfying (3.1). Suppose that there exists \(\eta \in \mathbb {R}\) such that (3.2) holds true for all \(x \in \mathbb {R}\). Moreover, suppose that there exists \(K \in [0, \infty )\) such that for all \(t \in [0,\infty )\) and \(x \in \mathbb {R}\) we have

$$\begin{aligned} |\psi (t,x) - \psi (t,0)| \le K(1 + |x|^2), \end{aligned}$$
(3.3)

and that also the space derivative \(\partial _x{\psi }\) grows at most polynomially in x, uniformly in t. Then \(\inf _{\alpha \in \mathcal {A}{(x)}} {\bar{J}}(x,\alpha ) \ge \eta \).

b) Let \(a^*(t,x) = (\partial _x \psi (t,x) - \beta _{xa}(t) x - \beta _a(t))/(2 \beta _{aa}(t))\). Suppose that for every \(x \in \mathbb {R}\) the SDE

$$\begin{aligned} dX_t = \mu (t,X_t, a^*(t,X_t)) dt + \sigma (t,X_t) dW_t, \qquad X_0 = x, \end{aligned}$$
(3.4)

possesses a unique solution \(X^{*, x}\) and \(\sup _{t \in [0, \infty )} E[(X^{*, x}_t)^2] < \infty \). Then \(\alpha ^*_t = a^*(t,X^{*,x}_t)\) satisfies

$$\begin{aligned} {\bar{J}}(x,\alpha ^*) = \inf _{\alpha \in \mathcal {A}{(x)}} {\bar{J}}(x,\alpha ) = \eta ; \end{aligned}$$
(3.5)

in particular \(\alpha ^*\) is an optimal control.

Proof

Let \(x \in \mathbb {R}\) and \(\alpha \in \mathcal {A}{(x)}\). We shortly write \(X = X^{x, \alpha }\) in the following. The Ito formula and (3.1) imply

$$\begin{aligned} \psi (T,X_T) - \psi (0,x) &\nonumber \\&= \int _0^T \left( \partial _t{\psi }(t,X_t)+ \mu (t,X_t,\alpha _t) \partial _x{\psi }(t,X_t) + \frac{1}{2} \sigma ^2 (t,X_t) \partial _{xx}{\psi } (t,X_t) \right) dt + M_T \nonumber \\&\ge - \int _0^T f(t,X_t,\alpha _t) dt + M_T , \end{aligned}$$
(3.6)

where \(M_T = \int _0^T \partial _x{\psi }(t,X_t)\sigma (t,X_t) dW_t\). The assumptions on \(\psi \) and on \(\alpha \) entail that \(\int _0^T (\partial _x{\psi }(t,X_t)\sigma (t,X_t))^2 dt < \infty \), and hence \(E(M_T) = 0\). Therefore, taking expectations on both sides of (3.6) and multiplying with \(- \frac{1}{T}\) yields

$$\begin{aligned} \frac{1}{T} E\left( \psi (0,x) - \psi (T,X_T)\right) \le E \frac{1}{T} \int _0^T f(t,X_t,\alpha _t) dt. \end{aligned}$$
(3.7)

Notice that

$$\begin{aligned} \frac{1}{T} E\left( \psi (0,x) - \psi (T,X_T)\right) = \frac{\psi (0,x) - \psi (T,x)}{T} + \frac{\mathbb {E}(\psi (T,x)- \psi (T,X_T))}{T}. \end{aligned}$$
(3.8)

By assumption (3.2), for the first fraction on the RHS of (3.8) we have

$$\begin{aligned} \limsup _{T \rightarrow \infty } \frac{\psi (0,x) - \psi (T,x)}{T} = \eta , \end{aligned}$$

and, since \(\sup _t E(X^2_t) < \infty \), for the second we have

$$\begin{aligned} \limsup _{T} \frac{| E(\psi (T,x)- \psi (T,X_T)|)}{T} \le \limsup _{T}\frac{K(2+|x| + \sup _t E(X^2_t) )}{T} = 0. \end{aligned}$$

Thus, from (3.7) we get

$$\begin{aligned} \eta \le \limsup _{T \rightarrow \infty } E \frac{1}{T} \int _0^T f(t,X_t,\alpha _t) dt = {\bar{J}}(x,\alpha ). \end{aligned}$$

Since \(\alpha \) is chosen arbitrarily, we also have \(\inf _{\alpha \in \mathcal {A}{(x)}} {\bar{J}}(x,\alpha ) \ge \eta \).

Now suppose that (3.4) has a unique solution \(X^* = X^{*, x}\) and that \(\sup _{t \in [0, \infty )} E[(X^{*}_t)^2] < \infty \). Then the control \(\alpha ^*_t = a^*(t,X^{*}_t)\), \(t \ge 0\), belongs to \(\mathcal {A}{(x)}\). Notice that the inequalities (3.6) and (3.7) become equalities if we replace X with \(X^*\). We thus obtain \(\eta = {\bar{J}}(x,\alpha ^*)\). This yields, together with the first part of the proof, the statement (3.5). \(\square \)

A verification result for the liminf cost functional

$$\begin{aligned} \underline{J}(x,\alpha ) = \liminf _{T \rightarrow \infty } E \frac{1}{T}\int _0^T f(s,X^{x, \alpha }_s, \alpha _s) ds , \end{aligned}$$
(3.9)

can be shown similarly. One simply needs to replace the limsup in (3.2) by a liminf.

Remember that \(U^\infty \) is the unique non-negative, bounded solution of (0.2) as described in Theorem 2.1.

Lemma 3.2

Let Assumption 1.1 be fulfilled. Then the process

$$\begin{aligned} \phi ^\infty _t&:= \int _t^\infty \left[ U^\infty _s \left( b_s + c_s C_s + \frac{ \beta _{a}(s) }{ 2 \beta _{aa}(s)} \right) - \frac{ \beta _{a}(s) \beta _{xa}(s) }{ 2 \beta _{aa} (s) } + \beta _{x}(s) \right] \\&\qquad \cdot \,\exp \left( \int _t^s B_r + \frac{ \beta _{xa}(r) - U^\infty _r }{2 \beta _{aa} (r) } {\text {d}} r \right) {\text {d}} s \end{aligned}$$

for \(t \in [0,\infty )\) is well defined and bounded uniformly in time.

Proof

By Theorem 2.1 we obtain

$$\begin{aligned} & \int _t^\infty \left| \left[ U^\infty _s \left( b_s + c_s C_s + \frac{ \beta _{a}(s) }{ 2 \beta _{aa}(s)} \right) - \frac{ \beta _{a}(s) \beta _{xa}(s) }{ 2 \beta _{aa} (s) } + \beta _{x}(s) \right] \right. \\&\left. \exp \left( \int _t^s B_r + \frac{ \beta _{xa}(r) - U^\infty _r }{2 \beta _{aa} (r) } {\text {d}} r \right) \right| {\text {d}} s \\&\le \sup _{ s \in [0,\infty ) } \left[ \hat{U} \left| b_s + c_s C_s + \frac{ \beta _{a}(s) }{ 2 \beta _{aa}(s)} \right| + \left| - \frac{ \beta _{a}(s) \beta _{xa}(s) }{ 2 \beta _{aa} (s) } + \beta _{x}(s) \right| \right] \int _t^\infty e^{ - \delta _1 ( s-t) + \delta _2 } {\text {d}} s \\&= \sup _{ s \in [0,\infty ) } \left[ \hat{U} \left| b_s + c_s C_s + \frac{ \beta _{a}(s) }{ 2 \beta _{aa}(s)} \right| + \left| - \frac{ \beta _{a}(s) \beta _{xa}(s) }{ 2 \beta _{aa} (s) } + \beta _{x}(s) \right| \right] \frac{e^{\delta _2}}{\delta _1} \\&< \infty , \end{aligned}$$

which means that \(\phi ^\infty \) is well defined and bounded. \(\square \)

In the following we use for \(t \in [0, \infty )\), \(x \in \mathbb {R}\) the definitions

$$\begin{aligned} \psi (t,x)&:= \frac{1}{2} U^\infty _t \cdot x^2 + \phi ^\infty _t \cdot x \\&\quad + \int _0^t - \bigg ( \phi ^\infty _s b_s + U^\infty _s \frac{ c_s^2 }{2} + \beta _0 (s) - \frac{ \left( \phi ^\infty _s - \beta _{a} (s) \right) ^2 }{ 4 \beta _{aa}(s) } \bigg ) {\text {d}} s \\ a^\infty (t,x)&:= \frac{ \phi ^\infty _t - \beta _a (t) + ( U^\infty _t - \beta _{xa} (t) ) x }{ 2 \beta _{aa} (t) } \end{aligned}$$

and

$$\begin{aligned} \eta&:= \limsup _{T \rightarrow \infty } \frac{1}{T} \int _0^T \phi ^\infty _s b_s + U^\infty _s \frac{ c_s^2 }{2} + \beta _0 (s) - \frac{ \left( \phi ^\infty _s - \beta _{a} (s) \right) ^2 }{ 4 \beta _{aa}(s) } {\text {d}} s . \end{aligned}$$

Lemma 3.3

Let Assumption 1.1 be fulfilled. Then there exists an \(\epsilon > 0\) such that \(\sup _{t \in [0,\infty )} \mathbb {E}\Big [ \big \vert X^{a^\infty }_t \big \vert ^p \Big ] < \infty \) for every \(p \in (0,2+\epsilon )\) and every initial value \(x_0 \in \mathbb {R}\).

For the proof of Lemma 3.3 we need the following.

Lemma 3.4

Let \(p,q : [0,\infty ) \rightarrow \mathbb {R}\) be measurable and bounded. The integral equation

$$\begin{aligned} h(t) = h(0) + \int _0^t \left[ p(s) \cdot h(s) + q(s) \right] {\text {d}} s, \end{aligned}$$

for \(h(0) \in \mathbb {R}\) and \(t \ge 0\), has the unique, explicit solution

$$\begin{aligned}&h(t) = e^{\int _0^t p(s) {\text {d}} s } \left( h(0) + \int _0^t q(s) e^{- \int _0^s p(r) {\text {d}} r } {\text {d}} s \right) \\&= h(0) e^{\int _0^t p(s) {\text {d}} s } + \int _0^t q(s) e^{\int _s^t p(r) {\text {d}} r } {\text {d}} s . \end{aligned}$$

Proof

That h solves the integral equation is straightforward by weak differentiation. The uniqueness follows since the integral equation is linear in h with bounded coefficients, which makes it a Lipschitz ODE. \(\square \)

Proof of Lemma 3.3

Observe that

$$\begin{aligned}&\mathbb {E}\left[ X^{\alpha ^T}_t \right] = x_0 + \mathbb {E}\left[ \int _0^t \left( b_s + B_s X^{\alpha ^T}_s - \frac{ \phi ^T_s - \beta _a (s) + ( U^T_s - \beta _{xa} (s) ) X^{\alpha ^T}_s }{ 2 \beta _{aa} (s) } \right) {\text {d}} s \right] \\ {}&= x_0 + \int _0^t \left( b_s + \frac{ - \phi ^T_s + \beta _a (s) }{ 2 \beta _{aa} (s) } \right) {\text {d}} s + \int _0^t \left( B_s + \frac{ \beta _{xa}(s) - U^T_s }{ 2 \beta _{aa} (s) } \right) \mathbb {E}\left[ X^{\alpha ^T}_s \right] {\text {d}} s. \end{aligned}$$

By Lemma 3.4 we get

$$\begin{aligned} \mathbb {E}\left[ X^{\alpha ^T}_t \right]&= x_0 e^{\int _0^t B_s + \frac{ \beta _{xa}(s) - U^T_s }{ 2 \beta _{aa} (s) } {\text {d}} s } + \int _0^t \left( b_s + \frac{ - \phi ^T_s + \beta _a (s) }{ 2 \beta _{aa} (s) } \right) e^{\int _s^t B_r + \frac{ \beta _{xa}(r) - U^T_r }{ 2 \beta _{aa} (r) } {\text {d}} r } {\text {d}} s \end{aligned}$$

and hence, using that

$$\begin{aligned} \left| b_s + \frac{ - \phi ^T_s + \beta _a (s) }{ 2 \beta _{aa} (s) } \right| \le \sup _{r \in [0,\infty )} \vert b_r \vert + \frac{ \hat{\phi } + \sup _{r \in [0,\infty )} \vert \beta _a (r) \vert }{ 2 \check{\beta }_{aa} } < \infty , \end{aligned}$$

and Theorem 2.1 we obtain

$$\begin{aligned} \left| \mathbb {E}\left[ X^{\alpha ^\infty }_t \right] \right|&\le \vert x_0 \vert e^{- \delta _1 (t-0) + \delta _2 } + \sup _{r \in [0,\infty )} \left| b_r + \frac{ - \phi ^\infty _r + \beta _a (r) }{ 2 \beta _{aa} (r) } \right| \int _0^t e^{-\delta _1 (t-r) + \delta _2 } {\text {d}} s \\&= \vert x_0 \vert e^{\delta _2} e^{- \delta _1 t } + \left( 1 - e^{- \delta _1 t } \right) \frac{ e^{\delta _2}}{\delta _1} \sup _{r \in [0,\infty )} \left| b_r + \frac{ - \phi ^\infty _r + \beta _a (r) }{ 2 \beta _{aa} (r) } \right| \\&\le \max \left( \vert x_0 \vert e^{\delta _2}, \frac{ e^{\delta _2}}{\delta _1} \left( \sup _{r \in [0,\infty )} \vert b_r \vert + \frac{ \hat{\phi } + \sup _{r \in [0,\infty )} \vert \beta _a (r) \vert }{ 2 \check{\beta }_{aa} } \right) \right) . \end{aligned}$$

Furthermore, using Itô’s formula

$$\begin{aligned}&\mathbb {E}\left[ \left( X^{\alpha ^\infty }_t \right) ^2 \right] \\&\quad = x_0^2 + \mathbb {E}\left[ \int _0^t 2 X^{\alpha ^\infty }_s \left( b_s + B_s X^{\alpha ^\infty }_s - \frac{ \phi ^\infty _s - \beta _a (s) + ( U^\infty _s - \beta _{xa} (s) ) X^{\alpha ^\infty }_s }{ 2 \beta _{aa} (s) } \right) \right. \\&\qquad \left. + \left( c_s + C_s X^{\alpha ^\infty }_s \right) ^2 {\text {d}} s \right] \\&\quad = x_0^2 + \int _0^t c_s^2 + \mathbb {E}\left[ X^{\alpha ^\infty }_s \right] 2 \left( c_s C_s + b_s - \frac{ \phi ^\infty _s - \beta _a (s) }{ 2 \beta _{aa} (s) } \right) \\&\qquad + \mathbb {E}\left[ \left( X^{\alpha ^\infty }_s \right) ^{2} \right] 2 \left( B_s + \frac{ \beta _{xa}(s) - U^\infty _s }{ 2 \beta _{aa} (s) } + \frac{C_s^2 }{2} \right) {\text {d}} s \\&\quad = x_0^2 \exp \left( 2 \int _0^t \left( B_s + \frac{ \beta _{xa}(s) - U^\infty _s }{ 2 \beta _{aa} (s) } \right) + \frac{C_s^2 }{2} {\text {d}} s \right) \\ {}&\qquad + \int _0^t \left\{ c_s^2 + 2 \mathbb {E}\left[ X^{\alpha ^\infty }_s \right] \left( c_s C_s + b_s - \frac{ \phi ^\infty _s - \beta _a (s) }{ 2 \beta _{aa} (s) } \right) \right\} \\&\qquad \cdot \exp \left( 2 \int _s^t \left( B_s + \frac{ \beta _{xa}(s) - U^\infty _s }{ 2 \beta _{aa} (s) } \right) + \frac{C_s^2 }{2} {\text {d}} r \right) {\text {d}} s \end{aligned}$$

due to Lemma 3.4. By Theorem 2.1 we can estimate

$$\begin{aligned} \mathbb {E}\left[ \left( X^{\alpha ^\infty }_t \right) ^2 \right]&\le x_0^2 \exp \left( - 2 \delta _1 t + 2 \delta _2 \right) \\ {}&\quad + \int _0^t \left\{ c_s^2 + 2 \mathbb {E}\left[ X^{\alpha ^\infty }_s \right] \left( c_s C_s + b_s - \frac{ \phi ^\infty _s - \beta _a (s) }{ 2 \beta _{aa} (s) } \right) \right\} \\&\quad \exp \left( - 2 \delta _1 (t-s) + 2 \delta _2 \right) {\text {d}} s \\&\le x_0^2 \exp \left( - 2 \delta _1 t + 2 \delta _2 \right) \\ {}&\quad + \left\{ \sup _{s\in [0,\infty )} c_s^2 + 2 \sup _{s\in [0,\infty )} \mathbb {E}\left[ X^{\alpha ^\infty }_s \right] \left( \sup _{s\in [0,\infty )} \left| c_s C_s \right| \right. \right. \\&\quad \left. \left. + \sup _{s\in [0,\infty )} \left| b_s - \frac{ \phi ^\infty _s - \beta _a (s) }{ 2 \beta _{aa} (s) } \right| \right) \right\} \\&\quad \cdot \frac{ \exp ( 2 \delta _2 ) }{ 2 \delta _1 } \left( 1 - \exp \left( - 2 \delta _1 t \right) \right) \\&< \infty . \end{aligned}$$

With Jensen’s inequality this implies for all \(q \in (0,2]\) that

$$\begin{aligned} \mathbb {E}\left[ \left| X^{\alpha ^\infty }_s \right| ^q \right] \le \mathbb {E}\left[ \left| X^{\alpha ^\infty }_s \right| ^{(q \cdot 2/q)} \right] ^{q/2} \le \mathbb {E}\left[ \left| X^{\alpha ^\infty }_s \right| ^2 \right] \end{aligned}$$

and hence also \(\sup _{s \in [0,\infty )} \mathbb {E}\left[ \left| X^{\alpha ^\infty }_s \right| ^q \right] < \infty \).

Furthermore, for \(2 \le p \in \mathbb {R}\) we analogously obtain

$$\begin{aligned}&\mathbb {E}\left[ \left| X^{\alpha ^\infty }_t \right| ^p \right] \\&\quad \le \vert x_0 \vert ^p \exp \left( - p \left( \delta _1 - \frac{p-2}{2} \sup _{s \in [0,\infty )} C_s^2 \right) t + p \delta _2 \right) \\ {}&\qquad + \left\{ \frac{p^2-p}{2} \sup _{s\in [0,\infty )} \mathbb {E}\left[ \left| X^{\alpha ^T}_s \right| ^{p-2} \right] \sup _{s\in [0,\infty )} c_s^2 \right. \\&\left. \qquad + p \sup _{s\in [0,\infty )} \mathbb {E}\left[ \left| X^{\alpha ^\infty }_s \right| ^{p-1} \right] \left( (p-1) \sup _{s\in [0,\infty )} \left| c_s C_s \right| + \sup _{s\in [0,\infty )} \left| b_s - \frac{ \phi ^\infty _s - \beta _a (s) }{ 2 \beta _{aa} (s) } \right| \right) \right\} \\&\qquad \cdot \frac{ \exp ( p \delta _2 ) }{ p \left( \delta _1 - \frac{p-2}{2} \sup _{s \in [0,\infty )} C_s^2 \right) } \left( 1 - \exp \left( - p \left( \delta _1 - \frac{p-2}{2} \sup _{s \in [0,\infty )} C_s^2 \right) t \right) \right) . \end{aligned}$$

Since \(\delta _1 >0\) and \(\sup _{s \in [0,\infty )} C_s^2 < \infty \), we get that for every p with \(2 \le p < 2+ \frac{2 \delta _1}{\sup _{s \in [0,\infty )} C_s^2}\) that \(\sup _{t \in [0,\infty )} \mathbb {E}\left[ \left| X^{\alpha ^\infty }_t \right| ^p \right] < \infty \). Thus, setting \(\epsilon := \frac{2 \delta _1}{\sup _{s \in [0,\infty )} C_s^2}\) we proved the result. \(\square \)

Lemma 3.5

Let Assumption 1.1 be fulfilled. Then \(\psi \) fulfills Equation (3.1) and \(a^\infty \) is a minimizer for this equation.

Proof

A straightforward calculation yields that

$$\begin{aligned} & \partial _t{\psi }(t,x) + ( \mu (t,x) - a^\infty (t,x)) \partial _x{\psi }(t,x) + \frac{1}{2} \sigma ^2 (t,x) \partial _{xx}{\psi } (t,x) \nonumber \\&= x^2 \left[ - \frac{ \left( U^\infty _t \right) ^2 }{4 \beta _{aa} (t) } - \beta _{xx}(t) + \frac{ \beta _{xa}^2 (t) }{4 \beta _{aa} (t) } \right] \nonumber \\&\qquad + x \left[ - \left( U^\infty _t \frac{ \beta _{a}(t) }{ 2 \beta _{aa}(t)} - \frac{ \beta _{a}(t) \beta _{xa}(t) }{ 2 \beta _{aa} (t) } + {\beta _{x}(t)} \right) + U^\infty _t \left( - \frac{ \phi ^\infty _t - \beta _{a} (t) }{ 2 \beta _{aa}(t)} \right) \right] \nonumber \\&\qquad + \left[ - \beta _0 (t) - \beta _{a}(t) \left( \frac{ \phi ^\infty _t - \beta _{a} (t) }{ 2 \beta _{aa}(t) } \right) - \beta _{aa}(t) \left( \frac{ \phi ^\infty _t - \beta _{a} (t) }{ 2 \beta _{aa}(t) } \right) ^2 \right] \nonumber \\&= - f (t,x,a^\infty (t,x)) . \end{aligned}$$
(3.10)

Next, observe that since f is strictly convex in a and the remainder of Equation (3.1) is affine linear in a, we get that if the derivative with respect to a becomes zero, we are in the unique minimum. Using this, we obtain by

$$\begin{aligned}&\partial _a \left[ \partial _t{\psi }(t,x) + (\mu (t,x) -a ) \partial _x{\psi }(t,x) + \frac{1}{2} \sigma ^2 (t,x) \partial _{xx}{\psi } (t,x) + f(t,x,a) \right] \Bigg \vert _{a=a^\infty (t,x)} \\&= \left[ - \partial _x{\psi }(t,x) + \partial _a{f} (t,x,a) \right] \Big \vert _{a=a^\infty (t,x)} \\&= - U^\infty _t x - \phi ^\infty _t + \beta _{xa}(t) x + \beta _{a}(t) + 2 \beta _{aa}(t) a^\infty (t,x) \\&= 0 \end{aligned}$$

that \(a^\infty \) minimizes Equation (3.1). Therefore, plugging the minimizer \(a^\infty \) into Equation (3.1) and using the result of Equation (3.10) we get

$$\begin{aligned}&\partial _t{\psi }(t,x) + \inf _{a \in \mathbb {R}} \left\{ (\mu (t,x) -a ) \partial _x{\psi }(t,x) + \frac{1}{2} \sigma ^2 (t,x) \partial _{xx}{\psi } (t,x) + f(t,x,a) \right\} \\&= \partial _t{\psi }(t,x) + (\mu (t,x) - a^\infty (t,x) ) \partial _x{\psi }(t,x) + \frac{1}{2} \sigma ^2 (t,x) \partial _{xx}{\psi } (t,x) + f(t,x,a^\infty (t,x)) \\&= 0 . \end{aligned}$$

\(\square \)

Theorem 3.6

Let Assumption 1.1 be fulfilled. Then \(a^\infty \) is an optimal control, \(\psi \) fulfills (3.1) and

$$\begin{aligned} {\bar{J}}(x,a^\infty ) = \inf _{\alpha \in \mathcal {A}{(x)}} {\bar{J}}(x,\alpha ) = \eta . \end{aligned}$$

Proof

We want to apply Proposition 3.1.

Lemma 3.5 already yields that \(\psi \) fulfills (3.1) and that \(a^\infty \) is the corresponding minimizer. Next, observe that

$$\begin{aligned}&\limsup _{t \rightarrow \infty } - \frac{\psi (t,x) }{t} \\&= \limsup _{t \rightarrow \infty } - \frac{ \frac{1}{2} ( U^\infty _t )^2 x^2 + \phi ^\infty _t x + \int _0^t - \Big ( \phi ^\infty _s b_s + U^\infty _s \frac{ c_s^2 }{2} + \beta _0 (s) - \frac{ \left( \phi ^\infty _s - \beta _{a} (s) \right) ^2 }{ 4 \beta _{aa}(s) } \Big ) {\text {d}} s }{t} \\&= \limsup _{t \rightarrow \infty } \frac{1}{t} \int _0^t \phi ^\infty _s b_s + U^\infty _s \frac{ c_s^2 }{2} + \beta _0 (s) - \frac{ \left( \phi ^\infty _s - \beta _{a} (s) \right) ^2 }{ 4 \beta _{aa}(s) } {\text {d}} s \\&= \eta \end{aligned}$$

since \(U^\infty \) and \(\phi ^\infty \) are bounded. Furthermore, for the same reason,

$$\begin{aligned} \left| \psi (t,x) - \psi (t,0) \right|&= \left| \frac{1}{2} ( U^\infty _t )^2 x^2 + \phi ^\infty _t x \right| \\&\le K (1+ \vert x \vert ^2 ) \end{aligned}$$

and \(\partial _x{\psi } (t,x) = U^\infty _t x + \phi ^\infty _t\) is linear in x with an in time uniformly bounded factor.

Finally, Lemma 3.3 gives the bounded second moment of the controlled process, which means that all conditions of Proposition 3.1 are fulfilled, yielding the statement. \(\square \)

5 Conclusion

We have shown that under Assumption (1.1) the problem of minimizing the limsup long-term average cost functional (0.1) is well-posed, and we have described an optimal closed loop control in terms of the unique bounded and non-negative function \(U^\infty \). Some further questions arise naturally.

First, is it possible to extend the results to a multi-dimensional setting? Following the same approach, a multi-dimensional Riccati equation on \([0, \infty )\) has to be studied. Notice that some of the comparison arguments of Sect. 2 can not be simply transferred to a multidimensional setting.

If the drift and diffusion coefficients in the linear state dynamics and the coefficient of the quadratic cost function f are themselves stochastic, then it is natural to assume that the solution of the control problem can be described in terms of a stochastic Riccati equation on \([0, \infty )\). Is it possible to prove existence and uniqueness of a solution and to obtain an optimal control with it?

Finally, we believe that one can generalize the results of Theorem 2.11 to the more general setting, where the derivative of Y is a strictly concave function having a strictly negative and a strictly positive zero. Also the starting value of Y can be generalized to be greater than any negative zero of the derivative of \(Y^{t,x}\). A proof for this claim, using abstract arguments instead of the tedious calculations as presented in Sect. 2, is left for future research.