Towards scalable synthesis of stochastic control systems

Abstract

Formal synthesis approaches over stochastic systems have received significant attention in the past few years, in view of their ability to provide provably correct controllers for complex logical specifications in an automated fashion. Examples of complex specifications include properties expressed as formulae in linear temporal logic (LTL) or as automata on infinite strings. A general methodology to synthesize controllers for such properties resorts to symbolic models of the given stochastic systems. Symbolic models are finite abstractions of the given concrete systems with the property that a controller designed on the abstraction can be refined (or implemented) into a controller on the original system. Although the recent development of techniques for the construction of symbolic models has been quite encouraging, the general goal of formal synthesis over stochastic control systems is by no means solved. A fundamental issue with the existing techniques is the known “curse of dimensionality,” which is due to the need to discretize state and input sets. Such discretization generally results in an exponential complexity over the number of state and input variables in the concrete system. In this work we propose a novel abstraction technique for incrementally stable stochastic control systems, which does not require state-space discretization but only input set discretization, and that can be potentially more efficient (and thus scalable) than existing approaches. We elucidate the effectiveness of the proposed approach by synthesizing a schedule for the coordination of two traffic lights under some safety and fairness requirements for a road traffic model. Further we argue that this 5-dimensional linear stochastic control system cannot be studied with existing approaches based on state-space discretization due to the very large number of generated discrete states.

Appendix:

Proof Proof of Lemma 2

Let x _q ∈X _q , where x _q =(u ₁,u ₂,…,u _N ), and u _q ∈U _q . Using the definition of $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$, one obtains $\phantom {\dot {i}\!}x^{\prime }_{{\mathsf {q}}}=\left (u_{2},\ldots ,u_{N},u_{{\mathsf {q}}}\right )\in \mathbf {Post}_{u_{{\mathsf {q}}}} (x_{{\mathsf {q}}})$. Since V is a δ-ISS-M _q Lyapunov function for Σ, we have:

$$\begin{array}{@{}rcl@{}} \underline\alpha\left( \left\Vert\overline\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)- \overline{H}_{{\mathsf{q}}}\left( x^{\prime}_{{\mathsf{q}}}\right)\right\Vert^{q}\right)&\leq& V(\overline\xi_{\overline{H}_{{\mathsf{q}}} (x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau), \overline{H}_{{\mathsf{q}}}\left( x^{\prime}_{{\mathsf{q}}}\right))\\ &=&V(\overline\xi_{\overline\xi_{x_{s}x_{{\mathsf{q}}}}(N\tau)u_{{\mathsf{q}}}}(\tau),\overline\xi_{x_{s} x^{\prime}_{{\mathsf{q}}}}(N\tau))\\ &=&V(\overline\xi_{\overline\xi_{x_{s}u_{1}}(\tau)(u_{2},\ldots, u_{N},u_{{\mathsf{q}}})}(N\tau),\overline\xi_{x_{s}(u_{2},\ldots,u_{N},u_{{\mathsf{q}}})}(N\tau))\\ &\leq&\mathsf{e}^{-\kappa N\tau}V(\overline\xi_{x_{s}u_{1}}(\tau),x_{s}). \end{array} $$

(24)

We refer the interested readers to the proof of Theorem 1 in Zamani et al. (2014a) to see how we derived the inequality (24). Hence, one gets

$$\begin{array}{@{}rcl@{}} \Vert\overline\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{H}_{{\mathsf{q}}} \left( x^{\prime}_{{\mathsf{q}}}\right)\Vert\leq(\underline\alpha^{-1}(\mathsf{e}^{-\kappa N\tau}V(\overline\xi_{x_{s}u_{1}}(\tau),x_{s})))^{1/q}, \end{array} $$

(25)

because of $\phantom {\dot {i}\!}\underline \alpha \in \mathcal {K}_{\infty }$. Since the inequality (25) holds for all x _q ∈X _q and u _q ∈U _q , and $\phantom {\dot {i}\!}\underline \alpha \in \mathcal {K}_{\infty }$, inequality (8) holds. □

Proof Proof of Lemma 3

Let x _q ∈X _q , where x _q =(u ₁,u ₂,…,u _N ), and u _q ∈U _q . Using the definition of $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$, one obtains $\phantom {\dot {i}\!}x^{\prime }_{{\mathsf {q}}}=\left (u_{2},\ldots ,u_{N},u_{{\mathsf {q}}}\right )\in \mathbf {Post}_{u_{{\mathsf {q}}}}(x_{{\mathsf {q}}})$. Since Σ is δ-ISS-M _q and using inequality (2), we have:

$$\begin{array}{@{}rcl@{}} \Vert\overline\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{H}_{{\mathsf{q}}} \left( x^{\prime}_{{\mathsf{q}}}\right)\Vert^{q}&=&\Vert\overline\xi_{\overline\xi_{x_{s}x_{{\mathsf{q}}}}(N\tau) u_{{\mathsf{q}}}}(\tau)-\overline\xi_{x_{s}x^{\prime}_{{\mathsf{q}}}}(N\tau)\Vert^{q}\\ &=&\Vert\overline\xi_{\overline\xi_{x_{s}u_{1}}(\tau)(u_{2},\ldots,u_{N},u_{{\mathsf{q}}})}(N\tau)- \overline\xi_{x_{s}(u_{2},\ldots,u_{N},u_{{\mathsf{q}}})}(N\tau)\Vert^{q}\\ &\leq&\beta(\Vert\overline\xi_{x_{s} u_{1}}(\tau)-x_{s}\Vert^{q},N\tau). \end{array} $$

Hence, one gets

$$\begin{array}{@{}rcl@{}} \Vert\overline\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{H}_{{\mathsf{q}}} (x^{\prime}_{{\mathsf{q}}})\Vert\leq(\beta(\Vert\overline\xi_{x_{s}u_{1}}(\tau)-x_{s}\Vert^{q},N\tau))^{1/q}. \end{array} $$

(26)

Since the inequality (26) holds for all x _q ∈X _q and all u _q ∈U _q , and β is a $\phantom {\dot {i}\!}\mathcal {K}_{\infty }$ function with respect to its first argument when the second one is fixed, inequality (10) holds. □

Proof Proof of Theorem 2

We start by proving that R is an ε-approximate simulation relation from S _τ (Σ) to $\phantom {\dot {i}\!}\overline {S}_{\mathsf {q}}({\Sigma })$. Consider any (x _τ ,x _q )∈R. Condition (i) in Definition 5 is satisfied because

$$ (\mathbb{E}[\Vert x_{\tau}-\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})\Vert^{q}])^{\frac{1}{q}}\leq(\underline\alpha^{-1} (\mathbb{E}[V(x_{\tau},\overline{H}_{\mathsf{q}}(x_{\mathsf{q}}))]))^{\frac{1}{q}}\leq\varepsilon. $$

(27)

We used the convexity assumption of $\phantom {\dot {i}\!}\underline \alpha$ and the Jensen inequality (Oksendal 2002) to show the inequalities in Eq. 27. Let us now show that condition (ii) in Definition 5 holds. Consider any υ _τ ∈U _τ . Choose an input u _q ∈U _q satisfying

$$ \Vert \upsilon_{\tau}-u_{{\mathsf{q}}}\Vert_{\infty}=\Vert \upsilon_{\tau}(0)-u_{{\mathsf{q}}}(0)\Vert\leq\mu. $$

(28)

Note that the existence of such u _q is guaranteed by U being a finite union of boxes and by the inequality μ≤s p a n(U) which guarantees that $\phantom {\dot {i}\!}\mathsf {U}\subseteq \bigcup _{p\in [\mathsf {U}]_{\mu }}\mathcal {B}_{{\mu }}(p)$. Consider the transition $\phantom {\dot {i}\!}x_{\tau }\overset {\upsilon _{\tau }}{\underset {\tau }{\longrightarrow }} x^{\prime }_{\tau }=\xi _{x_{\tau }\upsilon _{\tau }}(\tau )$ $\phantom {\dot {i}\!}\mathbb {P}$-a.s. in S _τ (Σ). Since V is a δ-ISS-M _q Lyapunov function for Σ and using inequality (28), we have (cf. equation (3.3) in Zamani et al. 2014a)

$$\begin{array}{@{}rcl@{}} \mathbb{E}[V(x^{\prime}_{\tau},\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))] &\leq& \mathbb{E}[V(x_{\tau},\overline{H}_{{\mathsf{q}}}(x_{q}))] \mathsf{e}^{-\kappa\tau}\\ &&+\frac{1}{\mathsf{e}\kappa} \rho(\|\upsilon_{\tau}-u_{{\mathsf{q}}}\|_{\infty})\leq \underline\alpha\left( \varepsilon^{q}\right) \mathsf{e}^{-\kappa\tau} + \frac{1}{\mathsf{e}\kappa}\rho(\mu). \end{array} $$

(29)

Observe that existence of u _q , by the definition of $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$, implies the existence of $\phantom {\dot {i}\!}x_{{\mathsf {q}}}\overset {u_{{\mathsf {q}}}}{\underset {{\mathsf {q}}}{\longrightarrow }}x^{\prime }_{{\mathsf {q}}}$ in $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$. Using Lemma 1, the concavity of $\phantom {\dot {i}\!}\widehat \gamma$, the Jensen inequality (Oksendal 2002), (9), the inequalities (7), (15), (29), and triangle inequality, we obtain

$$\begin{array}{@{}rcl@{}} \mathbb{E}[V(x^{\prime}_{\tau},\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}))]&=&\mathbb{E}[V(x^{\prime}_{\tau}, \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))+V(x^{\prime}_{\tau}, \overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}))-V(x^{\prime}_{\tau},\xi_{\overline{H}_{{\mathsf{q}}} (x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]\\ &=& \mathbb{E}[V(x^{\prime}_{\tau},\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]+ \mathbb{E}[V(x^{\prime}_{\tau},\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}))-V(x^{\prime}_{\tau}, \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]\\ &\leq&\underline\alpha (\varepsilon^{q})\mathsf{e}^{-\kappa\tau}+\frac{1}{\mathsf{e}\kappa}\rho(\mu)+\mathbb{E}[\widehat\gamma (\Vert\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{H}_{{\mathsf{q}}} (x^{\prime}_{{\mathsf{q}}})\Vert)]\\ &\leq&\underline\alpha(\varepsilon^{q})\mathsf{e}^{-\kappa\tau}+\frac{1}{\mathsf{e}\kappa}\rho(\mu)\\ &&+\widehat\gamma(\mathbb{E}[\Vert\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}} (\tau)-\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)+ \overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)- \overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}})\Vert])\\ &\leq&\underline\alpha(\varepsilon^{q})\mathsf{e}^{-\kappa\tau}+\frac{1}{\mathsf{e}\kappa}\rho(\mu)\\ &&+\widehat\gamma(\mathbb{E}[\Vert\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}} (\tau)-\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)\Vert]+\Vert \overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{H}_{{\mathsf{q}}} (x^{\prime}_{{\mathsf{q}}})\Vert)\\ &\leq&\underline\alpha(\varepsilon^{q})\mathsf{e}^{-\kappa\tau}+\frac{1}{\mathsf{e}\kappa}\rho(\mu)+\widehat \gamma((h_{x_{s}}((N+1)\tau))^{\frac{1}{q}}+\eta)\leq\underline\alpha(\varepsilon^{q}). \end{array} $$

Therefore, we conclude that $\phantom {\dot {i}\!}\left (x^{\prime }_{\tau },x^{\prime }_{{\mathsf {q}}}\right )\in {R}$ and that condition (ii) in Definition 5 holds.

Now we prove that R ⁻¹ is an ε-approximate simulation relation from $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$ to S _τ (Σ). Consider any (x _τ ,x _q )∈R (or equivalently (x _q ,x _τ )∈R ⁻¹). As showed in the first part of the proof, condition (i) in Definition 5 is satisfied. Let us now show that condition (ii) in Definition 5 holds. Consider any u _q ∈U _q . Choose the input υ _τ = u _q and consider $\phantom {\dot {i}\!}x^{\prime }_{\tau }=\xi _{x_{\tau }\upsilon _{\tau }}(\tau )$ $\phantom {\dot {i}\!}\mathbb {P}$-a.s. in S _τ (Σ). Since V is a δ-ISS-M _q Lyapunov function for Σ, one obtains (cf. equation 3.3 in Zamani et al. 2014a):

$$ \mathbb{E}[V(x^{\prime}_{\tau},\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]\leq \mathsf{e}^{-\kappa\tau}\mathbb{E}[V(x_{\tau},\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}}))]\leq \mathsf{e}^{-\kappa\tau}\underline\alpha\left( \varepsilon^{q}\right). $$

(30)

Using Lemma 1, the definition of $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$, the concavity of $\phantom {\dot {i}\!}\widehat \gamma$, the Jensen inequality (Oksendal 2002), (9), the inequalities (7), (15), (30), and triangle inequality, we obtain

$$\begin{array}{@{}rcl@{}} \mathbb{E}[V(x^{\prime}_{\tau},\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}))]&=&\mathbb{E}[V(x^{\prime}_{\tau}, \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))+V(x^{\prime}_{\tau}, \overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}))-V(x^{\prime}_{\tau},\xi_{\overline{H}_{{\mathsf{q}}} (x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]\\ &=& \mathbb{E}[V(x^{\prime}_{\tau},\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]+ \mathbb{E}[V(x^{\prime}_{\tau},\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}))-V(x^{\prime}_{\tau}, \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau))]\\ &\leq&\mathsf{e}^{-\kappa\tau}\underline\alpha(\varepsilon^{q})+\mathbb{E}[\widehat\gamma(\Vert \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{H}_{{\mathsf{q}}} (x^{\prime}_{{\mathsf{q}}})\Vert)]\\ &\leq&\mathsf{e}^{-\kappa\tau}\underline\alpha(\varepsilon^{q})\,+\,\widehat\gamma(\mathbb{E}[\Vert \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{\xi}_{\overline{H}_{{\mathsf{q}}} (x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)+\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}}) u_{{\mathsf{q}}}}(\tau)\,-\,\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}})\Vert])\\ &\leq&\mathsf{e}^{-\kappa\tau}\underline\alpha(\varepsilon^{q})\,+\,\widehat\gamma(\mathbb{E}[\Vert \xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)-\overline{\xi}_{\overline{H}_{{\mathsf{q}}} (x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)\Vert]+\Vert\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}}) u_{{\mathsf{q}}}}(\tau)\,-\,\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}})\Vert)\\ &\leq&\mathsf{e}^{-\kappa\tau}\underline\alpha(\varepsilon^{q})+\widehat\gamma((h_{x_{s}}((N+1)\tau))^{\frac{1}{q}}+ \eta)\leq\underline\alpha(\varepsilon^{q}). \end{array} $$

Therefore, we conclude that $\phantom {\dot {i}\!}(x^{\prime }_{\tau },x^{\prime }_{{\mathsf {q}}})\in {R}$ (or equivalently $\phantom {\dot {i}\!}\left (x^{\prime }_{{\mathsf {q}}},x^{\prime }_{\tau }\right )\in R^{-1}$) and condition (ii) in Definition 5 holds. □

Proof Proof of Theorem 3

We start by proving that R is an ε-approximate simulation relation from S _τ (Σ) to $\phantom {\dot {i}\!}\overline {S}_{\mathsf {q}}({\Sigma })$. Consider any (x _τ ,x _q )∈R. Condition (i) in Definition 5 is satisfied by the definition of R. Let us now show that condition (ii) in Definition 5 holds. Consider any υ _τ ∈U _τ . Choose an input u _q ∈U _q satisfying

$$ \Vert \upsilon_{\tau}-u_{{\mathsf{q}}}\Vert_{\infty}=\Vert \upsilon_{\tau}(0)-u_{{\mathsf{q}}}(0)\Vert\leq\mu. $$

(31)

Note that the existence of such u _q is guaranteed by U being a finite union of boxes and by the inequality μ≤s p a n(U) which guarantees that $\phantom {\dot {i}\!}\mathsf {U}\subseteq \bigcup _{p\in [\mathsf {U}]_{\mu }}\mathcal {B}_{{\mu }}(p)$. Consider the transition $\phantom {\dot {i}\!}x_{\tau }\overset {\upsilon _{\tau }}{\underset {\tau }{\longrightarrow }} x^{\prime }_{\tau }=\xi _{x_{\tau }\upsilon _{\tau }}(\tau )$ $\phantom {\dot {i}\!}\mathbb {P}$-a.s. in S _τ (Σ). It follows from the δ-ISS-M _q assumption on Σ and (31) that:

$$\begin{array}{@{}rcl@{}} \mathbb{E}[\Vert x^{\prime}_{\tau}-\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)\Vert^{q}] &\leq& \beta(\mathbb{E}[\Vert x_{\tau}-\overline{H}_{{\mathsf{q}}}(x_{q})\Vert^{q}],\tau)\\ &&+\gamma(\|\upsilon_{\tau}-u_{{\mathsf{q}}}\|_{\infty})\leq \beta(\varepsilon^{q},\tau) + \gamma(\mu). \end{array} $$

(32)

Existence of u _q , by the definition of $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$, implies the existence of $\phantom {\dot {i}\!}x_{{\mathsf {q}}}\overset {u_{{\mathsf {q}}}}{\underset {{\mathsf {q}}}{\longrightarrow }}x^{\prime }_{{\mathsf {q}}}$ in $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$. Using Eq. 9, the inequalities (5), (17), (32), and triangle inequality, we obtain

$$\begin{array}{@{}rcl@{}} (\mathbb{E}[\Vert x^{\prime}_{\tau}-\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}) \Vert^{q}])^{\frac{1}{q}} &=&(\mathbb{E}[\Vert x^{\prime}_{\tau}-\xi_{\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)+ \xi_{\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)\\ &&-\overline{\xi}_{\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)+ \overline{\xi}_{\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)-\overline{H}_{\mathsf{q}} (x^{\prime}_{\mathsf{q}})\Vert^{q}])^{\frac{1}{q}}\\ &\leq& (\mathbb{E}[\Vert x^{\prime}_{\tau}-\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau) \Vert^{q}])^{\frac{1}{q}}+(\mathbb{E}[\Vert\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau) -\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)\Vert^{q}])^{\frac{1}{q}}\\ &&+(\mathbb{E}[\Vert\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)- \overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}})\Vert^{q}])^{\frac{1}{q}}\\&\leq&(\beta(\varepsilon^{q},\tau) + \gamma(\mu))^{\frac{1}{q}}+(h_{x_{s}}((N+1)\tau))^{\frac{1}{q}}+\eta\leq\varepsilon. \end{array} $$

Therefore, we conclude that $\phantom {\dot {i}\!}\left (x^{\prime }_{\tau },x^{\prime }_{{\mathsf {q}}}\right )\in {R}$ and that condition (ii) in Definition 5 holds.

Now we prove that R ⁻¹ is an ε-approximate simulation relation from $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$ to S _τ (Σ). Consider any (x _τ ,x _q )∈R (or equivalently (x _q ,x _τ )∈R ⁻¹). Condition (i) in Definition 5 is satisfied by the definition of R. Let us now show that condition (ii) in Definition 5 holds. Consider any u _q ∈U _q . Choose the input υ _τ = u _q and consider $\phantom {\dot {i}\!}x^{\prime }_{\tau }=\xi _{x_{\tau }\upsilon _{\tau }}(\tau )$ $\phantom {\dot {i}\!}\mathbb {P}$-a.s. in S _τ (Σ). Since Σ is δ-ISS-M _q , one obtains:

$$ \mathbb{E}[\Vert x^{\prime}_{\tau}-\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}}) u_{{\mathsf{q}}}}(\tau)\Vert^{q}]\leq \beta(\mathbb{E}[\Vert x_{\tau}-\overline{H}_{{\mathsf{q}}} (x_{{\mathsf{q}}})\Vert^{q}],\tau)\leq\beta(\varepsilon^{q},\tau). $$

(33)

Using definition of $\phantom {\dot {i}\!}\overline {S}_{{\mathsf {q}}}({\Sigma })$, (9), the inequalities (5), (17), (33), and the triangle inequality, we obtain

$$\begin{array}{@{}rcl@{}} (\mathbb{E}[\Vert x^{\prime}_{\tau}-\overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}}) \Vert^{q}])^{\frac{1}{q}} &=&(\mathbb{E}[\Vert x^{\prime}_{\tau}-\xi_{\overline{H}_{\mathsf{q}} (x_{\mathsf{q}}) u_{\mathsf{q}}}(\tau)+ \xi_{\overline{H}_{\mathsf{q}} (x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)\\ &&-\overline{\xi}_{\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)+ \overline{\xi}_{\overline{H}_{\mathsf{q}}(x_{\mathsf{q}})u_{\mathsf{q}}}(\tau)-\overline{H}_{\mathsf{q}} (x^{\prime}_{\mathsf{q}})\Vert^{q}])^{\frac{1}{q}}\\ &\leq& (\mathbb{E}[\Vert x^{\prime}_{\tau}-\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau) \Vert^{q}])^{\frac{1}{q}}+(\mathbb{E}[\Vert\xi_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau) -\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)\Vert^{q}])^{\frac{1}{q}}\\ &&+(\mathbb{E}[\Vert\overline{\xi}_{\overline{H}_{{\mathsf{q}}}(x_{{\mathsf{q}}})u_{{\mathsf{q}}}}(\tau)- \overline{H}_{{\mathsf{q}}}(x^{\prime}_{{\mathsf{q}}})\Vert^{q}])^{\frac{1}{q}}\\ &\leq&(\beta(\varepsilon^{q}, \tau))^{\frac{1}{q}}+(h_{x_{s}}((N+1)\tau))^{\frac{1}{q}}+\eta\leq\varepsilon. \end{array} $$

Therefore, we conclude that $\phantom {\dot {i}\!}(x^{\prime }_{\tau },x^{\prime }_{{\mathsf {q}}})\in {R}$ (or equivalently $\phantom {\dot {i}\!}\left (x^{\prime }_{{\mathsf {q}}},x^{\prime }_{\tau }\right )\in R^{-1}$) and condition (ii) in Definition 5 holds. □

Proof Proof of Theorem 10

Denote $\phantom {\dot {i}\!}\hat \theta := \theta - r/2>0$, and $\mathbf {d}_{M}(a):=\left (\frac {1}{M} \sum \limits _{i=1}^{M} \|\xi ^{i}_{x_{s} x_{\mathsf {q}}}-a\|^{q}\right )^{\frac {1}{q}}$ for all $a\in {\mathbb {R}}^{n}$. It follows from Kloeden and Platen (1992, Theorem 4.5.4) that for all p≥1 and $\phantom {\dot {i}\!}a\in {\mathbb {R}}^{n}$

$$\mathbb{E}\left[\|\xi_{x_{s}x_{\mathsf{q}}}(N\tau) - a\|^{p}\right] \leq b(a, p). $$

Since we do not assume that the set of continuous states is bounded, the distance can be any positive real number, and the usual method of applying Hoeffding’s inequality does not work in this case. Instead we use Chernoff-type inequality (e.g. see above formula (1) in Boucheron et al. 2004), which implies that for any a ^′∈A ^r :

$$\mathbb{P}\left( \left|\left( \mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a^{\prime})\right)^{q} - (\mathbf{d}_{M}(a^{\prime}))^{q}\right|\geq \hat\theta\right) \leq \frac{b(a^{\prime}, 2q)}{M\hat\theta^{2}}. $$

Furthermore, since x↦x ^q is Hölder continuous with power q,

$$\mathbb{P}\left( \left|\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a^{\prime}) - \mathbf{d}_{M}(a^{\prime})\right|\geq \hat\theta\right) \leq \frac{b(a^{\prime}, 2q)}{M\hat\theta^{2q}}. $$

Thus, for the union of such events over a ^′∈A ^r, we have

$$ \mathbb{P}\left( \exists a^{\prime}\in A^{r} \text{ s.t. }\left|\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a^{\prime}) - \mathbf{d}_{M}(a^{\prime})\right|\geq \hat\theta\right)\leq \frac{|A^{r}|b(a^{*}, 2q)}{M\hat\theta^{2q}}, $$

(34)

due to the fact that the probability of a union is dominated by the sum of probabilities. Let [⋅]:A→A ^r be any surjective map such that ∥a−[a]∥≤r/2 for all a∈A, i.e. [⋅] chooses an r/2-close point in the grid A ^r. Using this map, we can extrapolate the inequality (34) to the whole set A since

$$\begin{array}{@{}rcl@{}} \left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a) - \mathbf{d}_{M}([a])\right\vert &\leq& \left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a) \,-\, \mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),[a])\right\vert \,+\, \left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),[a])\! - \mathbf{\!d}_{M}([a])\right\vert\\ &\leq& r/2 +\left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),[a]) - \mathbf{d}_{M}([a])\right\vert, \end{array} $$

where we used the fact that |d(H _q (x _q ),a)−d(H _q (x _q ),[a])|≤∥a−[a]∥ by the triangle inequality. As a result, the following inequality holds:

$$\begin{array}{@{}rcl@{}} &&\mathbb{P}\left( \exists a\in A \text{ s.t. }\left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a) - \mathbf{d}_{M}([a])\right\vert\geq \theta\right)\\ &&\leq \mathbb{P}\left( \exists a^{\prime}\in A^{r}\text{ s.t. }\left\vert\mathbf{d}\left( H_{\mathsf{q}}(x_{\mathsf{q}}),a^{\prime}\right) - \mathbf{d}_{M}\left( a^{\prime}\right)\right\vert\geq \hat\theta\right). \end{array} $$

(35)

On the other hand, since for any two functions $\phantom {\dot {i}\!}f,g:A\to {\mathbb {R}}$ it holds that

$$\left\vert\inf_{a\in A}f(a) - \inf_{a\in A}g(a)\right\vert\leq\sup_{a\in A}|f(a) - g(a)|, $$

we obtain that

$$\mathbb{P}\left( \left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),A) - \mathbf{d}^{r}_{M}\right\vert\geq \theta\right)\leq \mathbb{P}\left( \exists a\in A \text{ s.t. }\left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),a) - \mathbf{d}_{M}([a])\right\vert\geq \theta\right). $$

Combining the latter inequality with (34) and (35) yields:

$$\mathbb{P}\left( \left\vert\mathbf{d}(H_{\mathsf{q}}(x_{\mathsf{q}}),A) - \mathbf{d}^{r}_{M}\right\vert\geq \theta\right) \leq \frac{|A^{r}|b(a^{*}, 2q)}{M\hat\theta^{2q}}, $$

and in case M satisfies the assumption of the theorem, the right-hand side is bounded above by π as desired. □