Optimal Control for a Linear Quadratic Problem with a Stochastic Time Scale

Abstract

We consider a linear-quadratic control problem where a time parameter evolves according to a stochastic time scale. The stochastic time scale is defined via a stochastic process with continuously differentiable paths. We obtain an optimal infinite-time control law under criteria similar to the long-run averages. Some examples of stochastic time scales from various applications have been examined.

APPENDIX

Proof of the Lemma. First, the assertions in the Lemma are proved for the case of Eq. (10) with $A-BR^{-1}B^\mathrm {T}\Pi =-\kappa I$; $\kappa >0$ is a constant, and $I$ is the identity matrix. Then it is shown that the underlying properties of the process $X^{*}_t$, $t\to \infty$, in Eq. (10) do not change for an arbitrary exponentially stable matrix $A-BR^{-1}B^\mathrm {T}\Pi$. Consider the process $X^{*}_t=\hat X_t$, $t\geqslant 0$, with the dynamics

$$ d\hat X_t=-\kappa \alpha _tI \hat X_t+\sqrt {\alpha _t}GdW_t , \quad \hat X_0=0 . $$

Denoting the $i$th component of the process $\hat X_t$, $i=1,\dots ,n$, by $\hat X_{it}$, one can readily obtain the representation $\hat X_{it}=c_{i}M_{it}\left (\left \langle M_{it}\right \rangle +1\right )^{-1/2}$, where the martingale

$$ {M_{it}={c_{i}}^{-1}\int \limits _0^t \sqrt {\alpha _s}\exp {\left \{\int \limits _0^s \kappa \alpha _v dv\right \}}\left (\thinspace \sum _{j=1}^{d} G_{ij}dW_{js}\right )} $$

has the quadratic variation $\langle M_{it}\rangle =\exp {\big \{2\kappa \int \nolimits _0^t \alpha _v dv\big \}}-1$; further, $c_{i}=\big ((2\kappa )^{-1}\sum _{j=1}^{d}G^2_{ij}\big )^{1/2}$, where the $G_{ij}$ are the entries of the matrix $G$ ($i=1,\dots ,n$, $j=1,\dots ,d$); $W_{jt}$ is the $j$th component of the Wiener process $W_t$ ($j=1,\dots ,d$). It was established in [32, Lemma 2.3] that

$$ {\Big \|M_{it}\big (\langle M_{it}\rangle +1\big )^{-1/2}\Big \|\leqslant N(\omega )\sqrt {\ln \ln \big (\langle M_{it}\rangle +e^{e}\big )}},$$

where $N(\omega )\geqslant 0$ is a.s. a finite random variable. Therefore, for the process $\|\hat X_t\|^2$ with probability $1$ one has the relation

$$ \|\hat X_t\|^2\leqslant cN^2(\omega )\ln \left (\int \limits _0^t \alpha _v dv+e\right ) .$$

(A.1)

Then it follows from the estimate (A.1) and the Jensen inequality that

$$ \mathrm {E}\|\hat X_t\|^2\leqslant \tilde c\ln \left (\mathrm {E}\int \limits _0^t\alpha _v dv+e\right ) , $$

(A.2)

where $c$ and $\tilde c$ in (A.1) and (A.2) denote some positive constants whose particular values are inessential and can vary from formula to formula. It can readily be noticed that relation (13) is an obvious consequence of the above representation for the components of $\hat X_t$ and of the law of iterated logarithm for martingales; see, e.g., [33]. Further, we introduce the process $Z_t=X^{*}_t-\hat X_t$ with the dynamic equation

$$ dZ_t=\alpha _t\left (A-BR^{-1}B^\mathrm {T}\Pi \right ) Z_tdt+\alpha _t\left (A-BR^{-1}B^\mathrm {T}\Pi +\kappa I\right )\hat X_t , \quad Z_0=x,$$

which has the solution $Z_t=\Phi (t,0)x+\int \nolimits _0^t\Phi (t,s) \alpha _s(A-BR^{-1}B^{\mathrm {T}}\Pi +\kappa I)\hat X_s ds$. In view of the upper bound for $\|\Phi (t,s)\|$ (see the comment on (11)) and the Cauchy–Schwarz inequality for the process $Z_t$, we have the estimate

$$ \begin {aligned} \|Z_t\|^2&\leqslant 2\kappa _0^2{ \exp {\left \{-2\kappa \int \limits _0^t \alpha _v dv\right \}}}\|x\|^2 \\ &\qquad {}+c\thinspace { \exp {\left \{-\kappa \int \limits _0^t \alpha _v dv\right \}}}\int \limits _0^t \alpha _s{\thinspace \exp {\left \{\kappa \int \limits _0^s \alpha _v dv\right \}}\|\hat X_s\|^2 ds}. \end {aligned} $$

(A.3)

Applying (A.1) to (A.3) gives the relation

$$ \|Z_t\|^2\leqslant 2\kappa _0^2{\thinspace \exp {\left \{-2\kappa \int \limits _0^t \alpha _v dv\right \}}}\|x\|^2+cN^2(\omega ) \ln \left (\int \limits _0^t \alpha _v dv+e\right ).$$

Taking the expectation on both sides in this relation combined with the Jensen inequality leads to the estimate $\mathrm {E}\|Z_t\|^2\leqslant \tilde c+\tilde c \ln \left (\mathrm {E}\int \nolimits _0^t \alpha _v dv+e\right )$; this then implies (12) in the lemma being proved. Inequality (13) is also easy to obtain in a well-known way; see, e.g., the proof in [31, Theorem 2], if one notices that $h_t=\ln \left (\int \nolimits _0^t \alpha _v dv+e\right )$ is a nondecreasing function. Then dividing (A.3) by $h_t$ in the subsequent estimation of the integral on the right-hand side (while using the result for $\|\hat X_t\|^2$) gives a bounded limit. The proof of the lemma is complete.$\quad \blacksquare$

Proof of the Theorem. For $U\in \mathcal {U}$, we write a representation for the difference of cost functionals,

$$ J^{(\alpha )}_T(U^{*})-J^{(\alpha )}_T(U)=2x^\mathrm {T}_T\Pi X^{*}_T-\int \limits _0^T \alpha _t\left (x^\mathrm {T}_tQx_t+u^\mathrm {T}_tRu_t\right )dt-2\int \limits _0^T \sqrt {\alpha _t}x^\mathrm {T}_t\Pi GdW_t , $$

(A.4)

where the variables are $x_t=X^{*}_t-X_t$ and $u_t=U^{*}_t-U_t$ with $dx_t=\alpha _t A x_t dt+\alpha _t B u_t dt$, $x_0=0$. Since the pair $(A,C)$ is observable, it follows that there exists a matrix $F$ such that the matrix $A+FC$ is exponentially stable. Then

$$ {\|x_t\|\leqslant c \exp {\left \{-\bar \kappa \int \limits _0^t \alpha _v dv\right \}}\int \limits _0^t \exp {\left \{\bar \kappa \int \limits _0^{s} \alpha _v dv\right \}}\alpha _s\big (\|Cx_s\|+\|u_s\|\big )ds}$$

for some constant $\bar \kappa >0$. After squaring and applying the Cauchy–Schwarz inequality as well as the conditions $Q=C^{\mathrm {T}}C$, $R>0$, we have

$$ {\|x_t\|^2\leqslant \tilde c \exp {\left \{-\bar \kappa \int \limits _0^t \alpha _v dv\right \}}\int \limits _0^t \exp {\left \{\bar \kappa \int \limits _0^{s} \alpha _v dv\right \}}\alpha _s\left (x^\mathrm {T}_sQx_s+u^{\mathrm {T}}_sRu_s\right )ds}. $$

Further, using integration by parts, we show that

$$ {\int \limits _0^T \|x_t\|^2 ds\leqslant \tilde c\bar \kappa ^{-1} \int \limits _0^T\alpha _t\left (x^\mathrm {T}_tQx_t+u^{\mathrm {T}}_tRu_t\right )dt}. $$

Accordingly, we obtain the estimate

$$ {\|x_T\|^2+\int \limits _0^T \|x_s\|^2 ds\leqslant c_0\int \limits _0^T \alpha _t\left (x^\mathrm {T}_tQx_t+u^\mathrm {T}_tRu_t\right )dt}$$

with $T>0$ and some constant $c_0>0$. Then, considering the elementary inequality $2ab\leqslant a^2\bar c+b^2/\bar c$, which holds for any numbers $a,b$ and $\bar c>0$, the expression on the right-hand side of (A.4) is estimated in the form

$$ J^{(\alpha )}_T(U^{*})-J^{(\alpha )}_T(U)\leqslant c_1 \|X^{*}_T\|^2-c_2\int \limits _0^T \alpha _t\|x_t\|^2 dt-2\int \limits _0^T \sqrt {\alpha _t}x^\mathrm {T}_t\Pi GdW_t ,$$

(A.5)

where $c_1,c_2>0$ are some constants. After taking the expectation on both sides in (A.5) and dividing by ${\mathrm {E}}(\int \nolimits _0^T \alpha _t dt)$, in the limit as $T\to \infty$ we use the result (A.2) in the Lemma; this leads to the relation

$$ \limsup _{T \to \infty }\left ({ \mathrm {E}J^{(\alpha )}_T(U^{*})}\Bigg /{ \mathrm {E}\left (\displaystyle \int \limits _0^T\alpha _t dt\right )}\right )\leqslant \limsup _{T \to \infty }\left ({ \mathrm {E}J^{(\alpha )}_T(U)}\Bigg /{ \mathrm {E}\left (\displaystyle \int \limits _0^T\alpha _t dt\right )}\right ).$$

In the pathwise analysis of (A.5), introducing the notation

$$ {M_T=-2\int \limits _0^T \sqrt {\alpha _t}x^\mathrm {T}_t\Pi GdW_t, } $$

we write an estimate for (A.5) in the form

$$ J^{(\alpha )}_T(U^{*})\leqslant J^{(\alpha )}_T(U)+\mathcal {R_T} ,$$

where $\mathcal {R}_T=-c_3\langle M_T\rangle +M_T$ for some constant $c_3>0$; $\langle M_T\rangle =\int \nolimits _0^T \|\sqrt {\alpha _t}G^{\mathrm {T}}\Pi x_t\|^2 dt$ is the quadratic variation of $M_T$. Note that $\limsup _{T\to \infty }g_T\mathcal {R}_T\leqslant 0$ a.s. for any monotone function $g_T$ with the property $g_T>0$ and $g_T\to \infty$ as $T\to \infty$ (see [34]); in particular, $g_T=\left (\int \nolimits _0^T \alpha _t dt\right )^{-1}$ (see also Assumption $\mathcal {A}$). It is also obvious that (13) implies $\|X^{*}_T\|^2\left (\int \nolimits _0^T \alpha _t dt\right )^{-1} \to 0$ a.s. as $T\to \infty$. Therefore, with probability $1$ we have the relation

$$ \limsup _{T \to \infty }\left ({J^{(\alpha )}_T(U^{*})}\Bigg /{\left (\int \limits _0^T\alpha _t dt\right )}\right )\leqslant \limsup _{T \to \infty }\left ({J^{(\alpha )}_T(U)}\Bigg /{\left (\int \limits _0^T\alpha _t dt\right )}\right ). $$

By the Itô formula, in a standard manner we establish that

$$ {J^{(\alpha )}_T(U^{*})=x^\mathrm {T}\Pi x-(X^{*}_T)^{\mathrm {T}}\Pi X^{*}_T+\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)\int \limits _0^T \alpha _t dt+2\int \limits _0^T \sqrt {\alpha _t}(X^{*}_t)^\mathrm {T}\Pi GdW_t}. $$

Then the value of the criterion in (6) for $U^{*}$ is equal to

$$ {\lim _{T\to \infty }\left \{ \mathrm {E}J^{(\alpha )}_T(U^{*}) \mathrm {E}\left (\int \limits _0^T\alpha _t dt\right )^{-1}\right \}=\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)}.$$

Applying the iterated logarithm law to the martingale $\mathcal {M}_T=\int \nolimits _0^T \sqrt {\alpha _t}(X^{*}_t)^{\mathrm {T}}\Pi GdW_t$ yields the estimate $\|\mathcal {M}_T\|\leqslant c \sqrt {\langle \mathcal {M}_T\rangle \ln \ln \langle \mathcal {M}_T\rangle }$ for large $T$, and the use of (13) allows one to pass to the ineqaulity $\langle \mathcal {M}_T\rangle \leqslant \tilde c \left ( \int \nolimits _0^T \alpha _t dt \right )\ln \left ( \int \nolimits _0^T \alpha _t dt\right )$. Consequently, $\|\mathcal {M}_T\| \left (\int \nolimits _0^T \alpha _t dt\right )^{-1} \to 0$ as $T\to \infty$ a.s.; then

$$ {\lim _{T\to \infty }\left \{J^{(\alpha )}_T(U^{*})\left (\int \limits _0^T\alpha _t dt\right )^{-1}\right \}=\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G) }$$

with probability $1$. The proof of the theorem is complete. $\quad \blacksquare$