Keywords

figure a
figure b

1 Introduction

Stochastic processes describe phenomena, systems and computations whose behaviour is probabilistic. They are ubiquitous in science and engineering and, in particular, are employed in artificial intelligence and control theory to characterize dynamical models subject to stochastic disturbances, whose correctness is crucial when modelling systems that are deployed to safety-critical environments. Ensuring their correctness with mathematical certainty is an important yet challenging question, in particular for processes with infinite and possibly continuous state spaces. Systems of this kind include sequential decision and planning problems in stochastic environments, auto-regressive time series as well as probabilistic programs, cryptographic protocols, randomised algorithms and much more. Specifications of correctness for complex systems entail complex temporal behaviour, which can be described using linear temporal logic (LTL) or, more generally, \(\omega \)-regular properties.

Probabilistic verification algorithms for finite state systems based on explicit-state techniques or symbolic algorithms based on multi-terminal decision diagrams are inapplicable to systems with enumerably infinite or continuous (i.e. uncountably infinite) state spaces [37, 41]. For stochastic processes with infinite state space, existing methods usually build upon finite abstractions or proof rules based on martingale theory. Finite abstractions first partition the state space into a grid that forms an equivalent (or an approximately equivalent) finite state process, which is then checked using a finite-state verification algorithm [4, 5, 68]. Instead, proof rules directly reduce the verification problem to that of computing proof certificates—known as supermartingale certificates—which are synthesised using constraint solving, guess-and-check procedures, or are learned from data [3, 15, 19]. Proof rules based on supermartingale certificates enable effective verification for infinite-state systems without the intermediate step of computing an abstraction, and have been employed with success in the termination and correctness analysis of probabilistic programs as well as the verification of stochastic dynamical models.

Fig. 1.
figure 1

A simple infinite-state stochastic process over variable \(x \in \mathbb {Z}\). Above, the value of a Streett supermartingale for the reactivity property \(\textsf{GF}(x \text { is even}) \vee \textsf{FG} (x < 0)\).

Full size image

Supermartingale certificates for stochastic models have been developed in the past for specific classes of properties. Previous results introduced proof rules for the almost-sure and the quantitative questions of whether a process eventually hits a target condition (guarantee) [6, 15, 21, 23], always avoids an undesirable condition (safety) [16, 20], and for Boolean combinations of them (obligation), such as reach-avoid specifications [22]. Supermartingale certificates were further generalised to the properties for which the system eventually satisfies a condition permanently (persistence) [8, 17], or hits it infinitely often (recurrence) [17]. Yet, arbitrary Boolean combinations of the latter two, which define the \(\omega \)-regular properties (reactivity) and include LTL [45], are beyond reach for existing techniques. This includes the example of Fig. 1, which exhibits a process over one integer variable x that, when initialised at \(x = 0\), chooses with 0.5 probability to either enumerate the positive numbers or the odd negative numbers. This process satisfies almost surely the property requiring that either x is even infinitely often or that x stays strictly negative from some time onwards; however, as we illustrate in Sect. 6, previous proof rules cannot verify this property.

Notably, reducing \(\omega \)-regular verification to Büchi acceptance does not easily apply to stochastic processes [28, 51, 63]. This is because, to express \(\omega \)-regular as well as LTL properties, this introduces nondeterminism for which standard martingale theory falls short. To reason about \(\omega \)-regular specifications while preserving the probabilistic nature of the system, it is necessary to reason about Rabin, Streett, Muller or Parity acceptance conditions, as the respective automata express \(\omega \)-regular languages in their deterministic form [10, 56].

We introduce a proof rule for the probabilistic verification of Streett acceptance conditions. Our proof rule leverages the Robbins & Siegmund convergence theorem for nonnegative almost supermartingales [9, 55], which we show to characterise the almost-sure acceptance of Streett pairs. A Streett pair (A, B) is satisfied when either A is visited finitely many times or B is visited infinitely often. To conclude that a stochastic process satisfies a Streett pair (A, B) almost surely, we show that it is sufficient to present a nonnegative real function of the state space that strictly decreases in expectation when visiting \(A\setminus B\), possibly increases in expectation when visiting B, and never increases in expectation in any other case. Such functions—which we call Streett supermartingales—constitute formal proof certificates that stochastic processes satisfy Streett pairs almost surely. For example, consider the reactivity property in Fig. 1, which corresponds to the Streett pair where \(A = \{ x \mid x \ge 0\}\) and \(B = \{ x \mid x \text { is even}\}\). A Streett supermartingale for (A, B) is a function \(V :\mathbb {Z}\rightarrow \mathbb {R}_{\ge 0}\) that strictly decreases in expectation when visiting positive odd numbers, possibly increases in expectation when visiting nonnegative even numbers, and does not increase in expectation when visiting negative numbers: a valid Streett supermartingale is the function V(x) that takes value 1 if x is odd and takes value 0 if x is even. Notably, for general Streett acceptance conditions with multiple pairs, it suffices to compute one Streett supermartingale for each pair.

Our result enables effective and automated \(\omega \)-regular and LTL verification and control of discrete-time stochastic dynamical models. We leverage our novel proof rule together with the standard result that deterministic Streett automata (DSA) recognise \(\omega \)-regular languages. Our proof rule readily applies to the synchronous product between a stochastic process and a DSA, where it suffices to compute one Streett supermartingale for each Streett pair together with a supporting invariant, essential to exclude unreachable states for which the specification fails to hold. We provide an automated synthesis algorithm to compute a (1) Streett supermartingale for each pair, (2) a supporting invariant and (3) a control policy simultaneously, with one call to a decision procedure.

We show that for time-homogeneous Markov processes with real-valued state space and piecewise polynomial post-expectation, synthesising Streett supermartingales, supporting invariants and policies with piecewise polynomial template of known degree reduces to quantifier elimination over the first-order theory of the reals with one quantifier alternation. Moreover, we show that synthesising piecewise linear Streett supermartingales and policies, and polyhedral supporting invariants for processes with piecewise linear post-expectation reduces to the first-order existential theory of reals. Finally, we show that when a polyhedral inductive invariant is externally provided, then the synthesis of piecewise linear controllers and Streett supermartingales reduces to quadratically-constrained programming (QCP); furthermore, when the system is autonomous, the sole synthesis of Streett supermartingales reduces to linear programming (LP).

We showcase the practical efficacy of our method on continuous-state probabilistic systems with piecewise affine dynamics, with a prototype implementation. Our implementation is fully automated and capable of synthesizing Streett supermartingale certificates, supporting invariants and control policies simultaneously with a single invocation of a satisfiability modulo theory (SMT) solver. As an experimental benchmark, we consider a collection of \(\omega \)-regular properties ranging over safety, guarantee, recurrence, persistence and reactivity. This demonstrates that our approach is computationally feasible in practice and that it effectively unifies and generalises prior work on supermartingale certificates.

Our contribution is threefold: we present theory, methods, and experiments for a novel approach to automated stochastic \(\omega \)-regular verification and control.

  • Theory We introduce the first supermartingale certificate for full \(\omega \)-regular specifications: the Streett supermartingale. By preserving the probabilistic nature of the model, our proof rule enables effective \(\omega \)-regular verification of infinite state models by reasoning about their post-expectation.

  • Methods We provide sound and complete algorithms for \(\omega \)-regular verification and control based on our proof rule. Our algorithms compute Streett supermartingales, supporting invariants and control policies with known templates, and are complete relative to provided templates and post-expectations.

  • Experiments We have built a prototype showcasing the efficacy of our algorithms on a set of continuous-state probabilistic systems and \(\omega \)-regular properties that include and extend beyond the scope of existing approaches.

Our theoretical contribution applies to any discrete-time deterministic and stochastic dynamical model as well as deterministic and probabilistic programs with discrete and continuous distributions, whose semantics are all special cases of general stochastic processes. Our synthesis algorithm applies to any model whose post-expectation is expressible in piecewise polynomial closed form.

2 Streett Supermartingales

We define stochastic processes on a filtered probability space whose space of outcomes \(\varOmega \) defines an \(\mathcal F\)-measurable space of infinite runs, and \(\{\mathcal{F}_t \}\) is the associated filtration \(\mathcal{F}_t \subseteq \mathcal{F}_{t+1} \subseteq \mathcal{F}\) for all \(t \ge 0\). A discrete-time stochastic process over a \(\varSigma \)-measurable state space S is a sequence \(\{ X_t \}\) with \(X_t :\varOmega \rightarrow S\) that maps every outcome to the state of a trajectory at time t. We say that \(\{X_t\}\) is adapted to \(\{\mathcal{F}_t \}\) if every \(X_t\) is \(\mathcal{F}_t\)-measurable, namely for all \(A \in \varSigma \) it holds that \(X_t^{-1}[A] \in \mathcal{F}_t\). A trajectory \(\tau \) is an infinite sequence of states \(\tau = \tau _0, \tau _1, \tau _2, \dots \) such that \(\tau _t = X_t(\omega )\) for all \(t \ge 0\), for some \(\omega \in \varOmega \). Stochastic processes provide a general characterisation for the semantics of stochastic dynamical models described as stochastic difference equations as well as reactive probabilistic programs that run over infinite time.

Our supermartingale certificate for almost-sure \(\omega \)-regular verification and control of stochastic processes is underpinned by the Robbins & Siegmund theorem for the convergence of nonnegative almost supermartingales.

Theorem 1

(Robbins & Siegmund Convergence Theorem [55]). Let \(\{\mathcal{F}_t \}\) be a filtration and let \(\{V_t \}\), \(\{U_t \}\), and \(\{W_t \}\) be three real-valued nonnegative stochastic processes adapted to \(\{ \mathcal{F}_t \}\). Suppose that, for all \(t \in \mathbb {N}\), the following statement holds almost surely:

$$\begin{aligned} E ({V_{t+1}} \mid { \mathcal {F}_t }) &\le V_t-U_t+W_t. \end{aligned}$$
(1)

Then,

$$\begin{aligned} {{\,\textrm{Pr}\,}}\left( \sum _{t = 0}^\infty U_t < \infty \vee \sum _{t = 0}^\infty W_t = \infty \right) = 1. \end{aligned}$$
(2)

This result generalises the classic convergence theorem for nonnegative supermartingales [54, Theorem 22, p.148], allowing the real-valued process \(\{ V_t \}\) to satisfy the weaker almost-supermartingale condition of Eq. (1) with respect to the two other real-valued processes \(\{ U_t \}\) and \(\{ W_t \}\) [9, 55]. The statement establishes that the event that either series \(\sum _{t = 0}^\infty U_t\) converges or series \(\sum _{t = 0}^\infty W_t\) diverges has probability 1. As we show below, this naturally characterises almost-sure Streett acceptance for general stochastic processes.

A Streett pair (A, B) consists of two measurable regions of the state space \(A,B \in \varSigma \). A trajectory \(\tau = \tau _0, \tau _1, \tau _2, \dots \) satisfies (A, B) if either it visits all states in A finitely many times or it visits any states in B infinitely many times; more formally, \(\tau \) satisfies (A, B) if \(\sum _{t = 0}^\infty \textbf{1}_{A_i}(\tau _t) < \infty \vee \sum _{t = 0}^\infty \textbf{1}_{B_i}(\tau _t) = \infty \), where \(\textbf{1}_\mathcal{S}(\cdot )\) denotes the indicator function of set \(\mathcal S\), which takes value 1 when its argument is a member of \(\mathcal S\) and takes value 0 otherwise. Our result establishes that, to conclude that a stochastic process \(\{ X_t \}\) satisfies (A, B) almost surely, it suffices to present a function V that maps \(\{ X_t \}\) to a nonnegative almost-supermartingale whose expected value decreases strictly when visiting \(A\setminus B\), possibly increases when visiting B, and never increases anywhere else almost surely. We call function V a Streett supermartingale for (A, B).

Theorem 2

(Streett Supermartingales). Let \(\{ X_t\}\) be a stochastic process over state space S and (A, B) be a Streett pair. Suppose that there exists a nonnegative function \(V : S \rightarrow \mathbb {R}_{\ge 0}\) and positive constants \(\epsilon , M > 0\) such that, for all \(t \in \mathbb {N}\), the following condition holds almost surely:

$$\begin{aligned} E[ V(X_{t+1}) \mid \mathcal{F}_t ] \le V(X_t) - \epsilon \cdot \textbf{1}_{A \smallsetminus B}(X_t) + M\cdot \textbf{1}_B(X_t). \end{aligned}$$
(3)

Then, \(\{ X_t \}\) satisfies (A, B) almost surely, i.e.,

$$\begin{aligned} {{\,\textrm{Pr}\,}}\left( \sum _{t = 0}^\infty \textbf{1}_A(X_t) < \infty \vee \sum _{t = 0}^\infty \textbf{1}_B(X_t) = \infty \right) = 1. \end{aligned}$$
(4)
Fig. 2.
figure 2

Intuition for Theorem 2 on exemplar trajectories.

Full size image

Example 1

Figure 2 illustrates Theorem 2 over four exemplar trajectories, with respect to the Streett pair \((\{ s \mid s \text { has label a}\}, \{ s \mid s \text { has label b}\})\). In this example, we illustrate that a Streett supermartingale V—which must be nonnegative—cannot be constructed for the third trajectory, as Eq. (3) requires V to strictly decrease by \(\epsilon \) infinitely many times in the tail behaviour of the trajectory while being never allowed to increase. For all other trajectories instead, a Streett supermartingale V and suitable constants \(\epsilon , M > 0\) exist. In particular, in the first and second trajectories any V is only required to strictly decrease finitely many times. In the fourth trajectory, V is permitted to compensate its requirement to decrease infinitely many times by increasing infinitely many times in the tail behaviour. Notably, the first, the second, and the fourth trajectory are precisely those trajectories that satisfy the specification.    \(\square \)

We provide a specialisation of Theorem 2 (which applies to general stochastic processes) to time-homogeneous Markov processes, whose dynamics only depend on their transition kernel. A transition kernel \(T :S \times \varSigma \rightarrow [0,1]\) gives the probability that the process makes a transition from state \(s \in S\) into the set \(S^\prime \in \varSigma \), independently of time, i.e., for all \(t \in \mathbb {N}\), \(T(X_t, S') = {{\,\textrm{Pr}\,}}(X_{t+1} \in S' \mid \mathcal{F}_t)\). The transition kernel in turn determines the post-expectation \(({{\,\textrm{Post}\,}}h) :S \rightarrow \mathbb {R}\) of any real-valued measurable function \(h :S \rightarrow \mathbb {R}\), defined as the conditional expectation of h after one time step (regardless of absolute time t) as follows:

$$\begin{aligned} {{\,\textrm{Post}\,}}h(X_t) = \int _S h(s)~T(X_t, \textrm{d}s) = E(h(X_{t+1}) \mid \mathcal{F}_t). \end{aligned}$$
(5)

This denotes the expected value of h when evaluated in the subsequent state, given the current state being \(X_t\). For time-homogeneous Markov processes, we establish that to obtain a valid Streett supermartingale it suffices to enforce the requirement of Eq. (3) over \({{\,\textrm{Post}\,}}V\) of a Streett supermartingale V whose domain is restricted to a sufficiently strong supporting invariant I.

Theorem 3

(Supporting Invariants). Let \(\{ X_t \}\) be a time-homogeneous Markov process with initial state \(s_0 \in S\) and transition kernel \(T :S \times \varSigma \rightarrow [0,1]\). Let (A, B) be a Streett pair. Suppose there exists a measurable set \(I \in \varSigma \), a nonnegative function \(V :I \rightarrow \mathbb {R}_{\ge 0}\) and positive constants \(\epsilon ,M > 0\) that satisfy the following five conditions:

$$\begin{aligned} &s_0 \in I\end{aligned}$$
(6)
$$\begin{aligned} &\forall s \in I :T(s, I) = 1\end{aligned}$$
(7)
$$\begin{aligned} &\forall s \in (A \setminus B) \cap I :{{\,\textrm{Post}\,}}V(s) \le V(s) - \epsilon \end{aligned}$$
(8)
$$\begin{aligned} &\forall s \in B \cap I :{{\,\textrm{Post}\,}}V(s) \le V(s) + M\end{aligned}$$
(9)
$$\begin{aligned} &\forall s \in I \setminus (A \cup B) :{{\,\textrm{Post}\,}}V(s) \le V(s) \end{aligned}$$
(10)

Then, V is a Streett supermartingale for (A, B).

Example 2

Consider the time-homogeneous Markov process in Fig. 1 and the LTL property \(\textsf{GF}(x \text { is even})\), corresponding to the Streett pair \((\mathbb {Z}, \{ x \mid x \text { is even}\})\). Provided the supporting invariant \(\{ x \in \mathbb {Z}\mid x > 0\}\), the function that maps the positive even numbers to 0 and the positive odd numbers to 1 is a valid Streett supermartingale if the process is initialised on a positive number. Without a supporting invariant, function V would be required to strictly decrease along all negative numbers, necessarily violating nonnegativity. Notably, the process satisfies \(\textsf{GF}(x \text { is even})\) almost surely only on the positive numbers.    \(\square \)

Finally, a general Streett acceptance condition consists of a finite set of Streett pairs, and a trajectory satisfies the acceptance condition if it satisfies all pairs. To establish that a stochastic process satisfies a general Streett acceptance condition, it suffices to present one Streett supermartingale for each pair.

Theorem 4

Let \(\{ X_t \}\) be a stochastic process and \(\{ (A_i, B_i) :i = 1, \ldots , k\}\) be a Streett acceptance condition. If every Streett pair admits a Streett supermartingale, then \(\{ X_i \}\) satisfies the acceptance condition almost surely:

$$\begin{aligned} {{\,\textrm{Pr}\,}}\left( \bigwedge _{i=1}^k \left( \sum _{t = 0}^\infty \textbf{1}_{A_i}(X_t) < \infty \vee \sum _{t = 0}^\infty \textbf{1}_{B_i}(X_t) = \infty \right) \right) = 1. \end{aligned}$$
(11)

3 Stochastic Omega-Regular Verification and Control

A stochastic dynamical model \(\mathcal M\) over \(\mathbb {R}^n\) consists of an initial state vector \(x_0 \in \mathbb {R}^n\) and a parameterised update function \(f :\mathbb {R}^n \times \mathcal{W} \times K \rightarrow \mathbb {R}^n\) with a space \(\mathcal{W} \) of input disturbances and a space K of control parameters. This defines a time-homogeneous Markov process over the \(\mathcal{B}(\mathbb {R}^n)\)-measurable state space \(\mathbb {R}^n\) given by the following equation:

$$\begin{aligned} X_{t+1}^\mathcal{M} = f(X_t^\mathcal{M}, W_t; \kappa ), \quad X_0^\mathcal{M} = x_0, \end{aligned}$$
(12)

where \(\{ W_t \}\) is a sequence of i.i.d. stochastic input disturbances, each of which draws from the sample space \(\mathcal W\). This assumption restricts our model to time-homogeneous Markov processes, for which Theorem 3 applies. This model subsumes autonomous systems as well as control systems with parameterised policies. For example, a stochastic dynamical model \(f' :\mathbb {R}^n \times \mathcal{U} \times \mathcal{W} \rightarrow \mathbb {R}^n\) with finite or infinite space of control inputs \(\mathcal U\) and a parameterised (memoryless deterministic) policy \(\pi :\mathbb {R}^n \times K \rightarrow \mathcal U\) results in the special case \(f(x, w; \kappa ) = f'(x, \pi (x; \kappa ), w)\). Notably, our model also encompasses finite memory policies with known template and known memory size, for which it is sufficient to add extra state variables and extra input disturbances.

We associate our model with a finite set of observable propositions \(\varPi \) and an observation function \(\langle \!\langle \cdot \rangle \!\rangle :\mathbb {R}^n \rightarrow 2^\varPi \) that maps every state to the set of propositions that hold true in that state. This defines a (measurable) set of traces—the trace language of \(\mathcal M\)—where a trace \(\hat{\tau }\) is an infinite sequence \({\hat{\tau }} = {\hat{\tau }}_0,{\hat{\tau }}_1,{\hat{\tau }}_2, \dots \) where \({\hat{\tau }}_i = \langle \!\langle \tau _i \rangle \!\rangle \) for all \(i \ge 0\), with \(\tau = \tau _0, \tau _1, \tau _2, \dots \) being some trajectory of \(\{ X_t^\mathcal{M} \}\). We treat the question of synthesizing a controller for which \(\mathcal M\) satisfies an LTL formula over atomic propositions in \(\varPi \) or, more generally, satisfies an \(\omega \)-regular property over alphabet \(2^\varPi \) almost surely. For this purpose, we leverage the standard result that deterministic Streett automata (DSA) recognise the \(\omega \)-regular languages. The control synthesis problem amounts to computing a control parameter \(\kappa \in K\) for which the event that the trace language of \(\mathcal M\) is accepted by DSA \(\mathcal A\) has probability 1. The verification problem for autonomous systems or systems with fixed control policy can be simply seen as the special case where K is a singleton.

A deterministic Streett automaton \(\mathcal A\) over alphabet \(2^\varPi \) consists of a finite set of states Q, an initial state \(q_0\), a transition function \(\delta :Q \times 2^\varPi \rightarrow Q\), and a finite set of Streett pairs \(\text{ Acc } = \{(A_1, B_1), \dots , (A_k,B_k)\}\) where \(A_i \subseteq Q\) and \(B_i \subseteq Q\) for all \(i = 1, \dots , k\). The run \(\rho \) of \(\mathcal A\) on input trace \(\hat{\tau }= {\hat{\tau }}_0,{\hat{\tau }}_1,{\hat{\tau }}_2, \dots \) is the infinite sequence of states \(\rho = \rho _0, \rho _1, \rho _2, \dots \) such that \(\rho _0 = q_0\) and \(\rho _{t+1} = \delta (\rho _t, \hat{\tau }_t)\) for all \(t \ge 0\). The automaton accepts \(\hat{\tau }\) if either \(\rho \) visits \(A_i\) finitely many times or \(B_i\) infinitely many times for all \(i = 1, \dots k\), i.e., \(\bigwedge _{i=1}^k \sum _{t=1}^\infty \textbf{1}_{A_i}(\rho _t) < \infty \vee \sum _{t=1}^\infty \textbf{1}_{B_i}(\rho _t) = \infty \). Our approach to probabilistic \(\omega \)-regular verification leverages the fact that a DSA (indeed, any deterministic automaton) recognising the traces of a stochastic process forms in its turn a stochastic process:

$$\begin{aligned} X_{t+1}^\mathcal{A} = \delta (X_t^\mathcal{A}, \langle \!\langle X_{t}^\mathcal{M} \rangle \!\rangle ), \quad X_0^\mathcal{A} = q_0. \end{aligned}$$
(13)

Our approach determines whether \(\{ X_{t}^\mathcal{A} \}\) satisfies the Streett acceptance condition of \(\mathcal A\) with probability 1. We note that Streett automata are dual to Rabin automata, thus any tool to translate an LTL formula \(\varphi \) to a Rabin automaton, equivalently produces a Streett automaton for \(\lnot \varphi \) [30, 39]. Our output can thus be equivalently cast as Rabin acceptance with probability 0.

To determine whether \(\{X^\mathcal{A}_t\}\) satisfies the acceptance condition of \(\mathcal A\), we leverage Theorems 3 and 4 to synthesize a Streett supermartingale for each Streett pair and a supporting invariant over the synchronous product of \(\mathcal M\) and \(\mathcal A\). This is because the process \(\{ X_t^\mathcal{A}\}\) is not time-homogeneous when considered in isolation, as the distribution of next states in the automaton requires information about \(\{ X_t^\mathcal{M}\}\) to be determined. Therefore, we define the product process \(\{ X_{{t} }^{\mathcal{M}\otimes \mathcal{A}} \}\) as \(X_{t}^{\mathcal{M}\otimes \mathcal{A}} = (X_{t}^\mathcal{M}, X_{t}^\mathcal{A})\) for all \(t \in \mathbb {N}\), where \(X_{t+1}^{\mathcal{M}\otimes \mathcal{A}} = (f(X^\mathcal{M}_t, W_t; \kappa ), \delta (X^\mathcal{A}_t, \langle \!\langle X^\mathcal{M}_t \rangle \!\rangle ))\) and \(X_{0}^{\mathcal{M}\otimes \mathcal{A}} = (x_0, q_0)\). We then extend the acceptance condition of \(\mathcal A\) to the product state space \(\mathbb {R}^n \times Q\). Concretely, we define \(\overline{A}_i = \mathbb {R}^n \times A_i\) and \(\overline{B}_i = \mathbb {R}^n \times B_i\) for \(i = 1,\ldots , k\), and we define the acceptance condition of the product process as \(\{ (\overline{A}_1, \overline{B}_1), \ldots , (\overline{A}_k, \overline{B}_k) \}\). Finally, we establish that \(\{ X_{t}^{\mathcal{M}\otimes \mathcal{A}} \}\) satisfies the given acceptance condition almost surely by computing k Streett supermartingales and one supporting invariant over \(\mathbb {R}^n \times Q\).

We assume a known parameterised form for Streett supermartingales and invariant as well as for the control policy (as described above) and, by using Theorems 3 and 4, we express the verification and control problem as the problem of deciding a quantified first-order logic formula. Let \(V :\mathbb {R}^n \times Q \times \varTheta \rightarrow \mathbb {R}_{\ge 0}\) be a parameterised non-negative function of \(\mathbb {R}^n \times Q\) (the Streett supermartingale certificate), with parameter space \(\varTheta \). The post-expectation of V results in the parameterised function \(({{\,\textrm{Post}\,}}V) :\mathbb {R}^n \times Q \times \varTheta \times K \rightarrow \mathbb {R}_{\ge 0}\) over the certificate parameters \(\varTheta \) of V and the control parameters K defined as

$$\begin{aligned} {{\,\textrm{Post}\,}}V(x,q;\theta ,\kappa ) = \int _\mathcal{W} V(f(x,w;\kappa ),\delta (q,\langle \!\langle x \rangle \!\rangle ); \theta ) \Pr (\textrm{d}w) \end{aligned}$$
(14)

To construct our first-order logic decision problem, it is essential to express \({{\,\textrm{Post}\,}}V\) in a symbolic closed-form representation. Notably, computing symbolic closed-form representations for the post-expectation is a general problem in probabilistic verification, for which automated tools exist [35]. Provided that \({{\,\textrm{Post}\,}}V\) is computable, we template k parameterised Streett supermartingale certificates \(V_1, \dots , V_k\) with parameter spaces \(\varTheta _1, \dots , \varTheta _k\) respectively, and template one parameterised invariant predicate \(I :\mathbb {R}^n \times Q \times H \rightarrow \{ \textrm{true}, \textrm{false}\}\) with parameter space H. Then, solving the \(\omega \)-regular control problem with our method amounts to searching for certificate parameters \(\theta _1 \in \varTheta _1, \dots , \theta _k \in \varTheta _k\), invariant parameter \(\eta \in H\), control parameter \(\kappa \in K\) and coefficients \(\epsilon , M > 0\) such that, for every \(i = 1, \dots , k\), the following universally quantified sentences hold:

$$\begin{aligned} &I(x_0, q_0; \eta ) \end{aligned}$$
(15)
$$\begin{aligned} &\forall x \in \mathbb {R}^n, q \in Q, w \in \mathcal {W} :I(x, q; \eta ) \implies I(f(x, w; \kappa ), \delta (q, \langle \!\langle x \rangle \!\rangle ); \eta ) \end{aligned}$$
(16)
$$\begin{aligned} &\forall x \in \mathbb {R}^n, q \in (A_i \setminus B_i) :I(x,q; \eta ) \implies {{\,\textrm{Post}\,}}{V_i} (x, q; \theta _i, \kappa ) \le V_i(x, q; \theta _i) - \epsilon \end{aligned}$$
(17)
$$\begin{aligned} &\forall x \in \mathbb {R}^n, q \in B_i :I(x,q; \eta ) \implies {{\,\textrm{Post}\,}}{V_i} (x, q; \theta _i, \kappa ) \le V_i(x, q; \theta _i) + M \end{aligned}$$
(18)
$$\begin{aligned} &\forall x \in \mathbb {R}^n, q \in Q\setminus (A_i \cup B_i) :I(x,q; \eta )\Rightarrow {{\,\textrm{Post}\,}}{V_i}(x, q; \theta _i, \kappa ) \le V_i(x, q; \theta _i) \end{aligned}$$
(19)
$$\begin{aligned} &\forall x \in \mathbb {R}^n, q \in Q :I(x, q; \eta ) \implies V_i(x, q; \theta _i) \ge 0 \end{aligned}$$
(20)

In particular, Eqs. (15) and (16) respectively indicate the conditions of initiation and consecution for the supporting invariant, yielding a subset of the product space satisfying Eqs. (6) and (7). Equations (17) to (19) indicate the drift conditions, which ensure that \(V_1, \dots , V_k\) satisfy Eqs. (8) to (10) w.r.t. the acceptance conditions extended to the product space. Equation (20) enforces the premise of Theorem 3 that requires V to be non-negative over its domain I.

Fig. 3.
figure 3

Deterministic Streett Automaton for \((x \ge -1) \textsf{UG} (-1 \le x \le 1)\)

Full size image

Example 3

Consider a simple Markov process over one real-valued variable x and control parameter \(\kappa \), described by the following stochastic difference equation:

$$\begin{aligned} x_{t+1} = \kappa \cdot x_t + w_t, \qquad x_0 = 100,\qquad w_t \sim \textrm{Uniform}(-0.1,0.1) \end{aligned}$$
(21)

We wish to synthesise a control parameter \(\kappa \) for which the process satisfies the stabilize-while-avoid property \(\varPhi = (x \ge -1) \textsf{UG} (-1 \le x \le 1)\), which requires the system to avoid \(x < -1\) until it stabilizes within \((-1 \le x \le 1)\). This corresponds to the DSA in Fig. 3, whose states define the necessary drift conditions.

Applying Theorem 3 to the DSA of Fig. 3, we make two observations. Firstly, recalling the intuition of the first trajectory in Fig. 2, we note that the specification is satisfied only if \(q_0\) is visited finitely many times by the product process, and this must be established by a Streett supermartingale that strictly decreases when \(x \ge 1\) and does not increase otherwise. Secondly, recalling the intuition of the third trajectory, we note that such a Streett supermartingale exists only if \(q_2\) is never reached, and this must be established by a supporting invariant. A control parameter for which \(\varPhi \) is satisfied is \(\kappa = 0.5\), and this is established by the following Streett supermartingale and supporting invariant:

$$\begin{aligned} V(x,q) = {\left\{ \begin{array}{ll} x+1&{} \text {if }q = q_0\\ 0&{} \text {otherwise} \end{array}\right. }\qquad I(x,q) = {\left\{ \begin{array}{ll} x \ge -0.2 &{} \text {if }q = q_0\\ -0.2 \le x \le 0.9 &{} \text {if } q= q_1\\ \text {false}&{} \text {if }q = q_2 \end{array}\right. } \end{aligned}$$
(22)

Here, the post-expectation of V results in the function below; note that no term for the stochastic disturbance appears because, in this case, the expected value of w is 0, and so is its contribution to the post-expectation of V:

$$\begin{aligned} {{\,\textrm{Post}\,}}V(x,q) = {\left\{ \begin{array}{ll} 0.5 \cdot x + 1&{}\text {if }x \ge 1 \text { and } q \in \{ q_0, q_1 \}\\ 0 &{}\text {otherwise } \end{array}\right. } \end{aligned}$$
(23)

Altogether, we obtain the following (satisfied) system of universally quantified sentences. The initiation condition (cf. Eq. (15)) results in the following sentence:

$$\begin{aligned} &100 \ge -0.2 \equiv I(x_0, q_0) \end{aligned}$$
(24)

The consecution condition (cf. Eq. (16)) expands into the following three implications, each of which corresponds to a case of I, a feasible transition in the automaton, and a feasible transition of the dynamical model according to the sample space \(\mathcal{W} = [-0.1, 0.1]\) of stochastic disturbances:

$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{x \ge -0.2}_{I(x, q_0)} \wedge \underbrace{x \ge 1}_{\delta (q_0, \cdot ) = q_0} \implies \underbrace{0.5 x + w \ge -0.2}_{I(0.5 x + w, q_0)}\end{aligned}$$
(25)
$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{x \ge -0.2}_{I(x, q_0)} \wedge \underbrace{-1 \le x < 1}_{\delta (q_0, \cdot )=q_1} \implies \underbrace{-0.2 \le 0.5 x + w \le 0.9}_{I(0.5 x + w, q_1)}\end{aligned}$$
(26)
$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{x \ge -0.2}_{I(x, q_0)} \wedge \underbrace{x < -1}_{\delta (q_0, \cdot ) = q_2} \implies \underbrace{\textrm{false}}_{I(0.5 x + w, q_2)}\end{aligned}$$
(27)
$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{-0.2 \le x \le 0.9}_{I(x, q_1)}\wedge \underbrace{x \ge 1}_{\delta (q_1, \cdot ) = q_0} \implies \underbrace{0.5 x + w \ge -0.2}_{I(0.5 x + w, q_0)}\end{aligned}$$
(28)
$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{-0.2 \le x \le 0.9}_{I(x, q_1)} \wedge \underbrace{-1 \le x < 1}_{\delta (q_1, \cdot ) = q_1} \Rightarrow \underbrace{-0.2 \le 0.5 x + w \le 0.9}_{I(0.5 x + w, q_1)}\end{aligned}$$
(29)
$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{-0.2 \le x \le 0.9}_{I(x, q_1)} \wedge \underbrace{x < -1}_{\delta (q_1, \cdot ) = q_2} \implies \underbrace{\textrm{false}}_{I(0.5 x + w, q_2)}\end{aligned}$$
(30)
$$\begin{aligned} &\forall x \in \mathbb {R}, w \in \mathcal{W} :\underbrace{\textrm{false}}_{I(x, q_2)} \wedge \underbrace{\textrm{true}}_{\delta (q_2, \cdot ) = q_2} \implies \underbrace{\textrm{false}}_{I(0.5 x + w, q_2)} \end{aligned}$$
(31)

The strict decrease drift condition associated with \(q_0\) (cf. Eq. (17)) results, with \(\epsilon = 0.5\), in the following two sentences associated with each case of \({{\,\textrm{Post}\,}}V\):

$$\begin{aligned} &\forall x \in \mathbb {R}:\underbrace{x \ge -0.2}_{I(x, q_0)} \wedge (x \ge 1) \implies \underbrace{0.5 \cdot x + 1}_{{{\,\textrm{Post}\,}}V(x, q_0)} ~\le ~ \underbrace{x + 1}_{V(x, q_0)} - \underbrace{0.5}_{\epsilon }\end{aligned}$$
(32)
$$\begin{aligned} &\forall x \in \mathbb {R}:\underbrace{x \ge -0.2}_{I(x, q_0)} \wedge (x < 1) \implies \underbrace{0}_{{{\,\textrm{Post}\,}}V(x, q_0)} ~\le ~ \underbrace{x + 1}_{V(x, q_0)} - \underbrace{0.5}_{\epsilon } \end{aligned}$$
(33)

Similarly, the non-increase drift condition associated with \(q_1\) (cf. Eq. (19)) results in the following two implications:

$$\begin{aligned} &\forall x \in \mathbb {R}:\underbrace{-0.2 \le x \le 0.9}_{I(x, q_1)} \wedge (x \ge 1) \implies \underbrace{0.5 \cdot x + 1}_{{{\,\textrm{Post}\,}}V(x, q_1)} ~\le \underbrace{0}_{V(x, q_1)}\end{aligned}$$
(34)
$$\begin{aligned} &\forall x \in \mathbb {R}:\underbrace{-0.2 \le x \le 0.9}_{I(x, q_1)} \wedge (x < 1) \implies \underbrace{0}_{{{\,\textrm{Post}\,}}V(x, q_1)} \le \underbrace{0}_{V(x, q_1)} \end{aligned}$$
(35)

We note that the invariant in state \(q_1\) is sufficiently strong to exclude the possibility of a transition back to \(q_0\) from \(q_1\) (which is associated with \(x \ge 1\)), making the premise of the implication in Eq. (34) false. The drift condition of \(q_2\) is also trivially satisfied, as the premise of the respective implication is false:

$$\begin{aligned} \forall x \in \mathbb {R} :\underbrace{\textrm{false}}_{I(x, q_2)} \implies \underbrace{0}_{{{\,\textrm{Post}\,}}V(x, q_2)} \le \underbrace{0}_{V(x, q_2)} - \underbrace{0.5}_{\epsilon } \end{aligned}$$
(36)

Finally, the non-negativity condition (cf. Eq. (20)) is trivially satisfied on \(q_1\) and \(q_2\) as \(V(x, q_1) = V(x, q_2) = 0\). For \(q_0\) instead, the condition is the following:

$$\begin{aligned} &\forall x \in \mathbb {R} :\underbrace{x \ge -0.2}_{I(x, q_0)} \implies \underbrace{x + 1}_{V(x, q_0)} \ge 0 \end{aligned}$$
(37)

Notably, every sentence consists of a conjunction of inequalities implying an inequality. As we show in Sect. 4, difference equations, Streett supermartingales and supporting invariants that are piecewise-defined according to a template as in Eq. (22) always result in systems of constraints in this form. This enables effective algorithmic synthesis of Streett supermartingales, supporting invariants and control parameters using symbolic or numerical decision procedures.    \(\square \)

4 Algorithmic Synthesis of Streett Supermartingales

Exhibiting Streett supermartingales and supporting invariants constitutes a witness that the stochastic dynamical model and its control parameter comply with the \(\omega \)-regular property at hand. Under the assumption that these three objects are constrained to be in the form of a template, the verification and control problem is reducible to a decision procedure for quantified first-order formulae. In this section, we define templates that allow effective synthesis using standard symbolic and numerical decision procedures.

We show that under different assumptions and problem settings, the verification and control problem reduces to the following decision procedures:

  • General Control This refers to the general synthesis of a Streett supermartingale, supporting invariant, and control parameters. When these and the associated post-expectation are in piecewise polynomial form (Sect. 4.1), then the synthesis problem is reducible to a quantified formula (with one quantifier alternation) in non-linear real arithmetic (NRA). When they are in piecewise linear form (Sect. 4.2) then the synthesis problem reduces to the existential theory of non-linear real arithmetic (QF_NRA).

  • Shielded Control This refers to the synthesis of a Streett supermartingale and control parameter, given an externally provided inductive invariant. Externally provided invariants are relevant when a shield that ensures the safety of the policy (but not necessarily its liveness) is computed beforehand [7]. This reduces to quadratically constrained programming (QCP) with piecewise linear templates (Sect. 4.3 and Example 5).

  • Verification This refers to the sole synthesis of Streett supermartingales, when the system has a known invariant that is provided a priori. This reduces to linear programming (LP) when templates and post-expectation are piecewise linear (Sect. 4.3 and Example 6).

We introduce a functional template \(F :\mathbb {R}^{N} \times Q \times \varLambda \rightarrow \mathbb {R}\) that maps an N-dimensional real-valued vector, a state of the automaton \(q \in Q\), and a generic template parameter \(\lambda \in \varLambda \) to a real-valued output according to a number of cases, guarded by logical predicates:

$$\begin{aligned} \begin{aligned} F(x,q; \lambda ) &= {\left\{ \begin{array}{ll} g_{1, l+1}(x; \lambda ) &{} \text {if } \bigwedge \nolimits _{i = 1}^{l} g_{1,i}(x; \lambda ) \lesssim _{1,i} 0, \text { and } q \in Q_1\\ &{}\vdots \\ g_{m, l+1}(x; \lambda ) &{} \text {if } \bigwedge \nolimits _{i = 1}^{l} g_{m,i}(x) \lesssim _{m,i} 0, \text { and } q \in Q_m,\\ \end{array}\right. } \end{aligned} \end{aligned}$$
(38)

The value N is a placeholder for either the dimensionality of the state space \(\mathbb {R}^n\), or the joint dimensionality of the system and the stochastic disturbance inputs \(\mathbb {R}^n \times \mathcal{W}\), according to context. The sets \(Q_1, \ldots , Q_m \subseteq Q\) denote constraints on the automaton states and \(\lesssim \) denotes either a strict- or non-strict inequality. This makes the form of Eq. (38) suitable as a template for expressing Streett supermartingales \(V(x, q; \theta ) \equiv F(x, q; \theta )\), supporting invariants \(I(x, q ; \eta ) \equiv \bigwedge [F(x, q; \eta ) \le 0]\), dynamical models \(f(x,w; \kappa ) \equiv F((x, w), -; \kappa )\) as well as the symbolic post-expectation \({{\,\textrm{Post}\,}}V(x, q ; \theta , \kappa ) \equiv F(x, q; \theta , \kappa )\).

Assuming, without loss of generality, that each observable proposition in \(\varPi \) (cf. Sect. 3) corresponds to a single inequality over the state space \(\mathbb {R}^n\), the transition function \(\delta (q, \langle \!\langle x \rangle \!\rangle )\) of the automaton takes the form of template \(D : \mathbb {R}^n \times Q \rightarrow Q\):

$$\begin{aligned} \begin{aligned} D(x, q) &= {\left\{ \begin{array}{ll} q_1' &{} \text {if } \bigwedge \nolimits _{i = 1}^{l} g_{1,i}(x) \lesssim _{1,i} 0, \text { and } q = q_1\\ &{}\vdots \\ q_m' &{} \text {if } \bigwedge \nolimits _{i = 1}^{l} g_{m,i}(x) \lesssim _{m,i} 0, \text { and } q = q_m.\\ \end{array}\right. } \end{aligned} \end{aligned}$$
(39)

where each of the automaton’s transitions corresponds to a case of Eq. (39).

The requirements of Eqs. (15) to (20) reduce to a conjunction of sentences of the form Eq. (40), namely, a universally quantified implication over N-dimensional real-valued variables, where each implication has a premise that is a finite conjunction of inequalities (where L is a placeholder for the number of conjuncts), and a consequent that is a single non-strict inequality:

$$\begin{aligned} \begin{aligned} \forall y \in \mathbb {R}^{N} :\bigwedge _{i = 1}^L g_{i}(y; \lambda ) \lesssim _{i} 0 &\implies g_{L+1}(y; \lambda ) \le 0, \end{aligned} \end{aligned}$$
(40)

This is because our construction only invokes compositions of the templates F and D that produce results that are representable in the form of template F, namely, a piecewise function over \(\mathbb {R}^N \times Q\) with parameters \(\lambda \in \varLambda \). In combination with rewriting at the level of propositional logic, we establish Eq. (40).

Finally, we note that the conjunction of sentences of the form Eq. (40) is existentially quantified over the certificate, invariant and control parameters, as well as the parameters \(\epsilon \) and M, all of which we notationally subsume within \(\lambda \). We now discuss algorithms for finding a satisfying assignment to these existentially quantified parameters under the problem scenarios outlined earlier.

4.1 Piecewise Polynomial Systems and Templates

Under the assumption that all functions g in the templates Eqs. (38) and (39) are polynomials in \( x \in \mathbb {R}^N \) and \( \lambda \in \varLambda \), the synthesis problem reduces to an existentially quantified conjunction of statements in the form of Eq. (40), which are in turn universally quantified implications over polynomial inequalities. This synthesis problem belongs to the first-order theory of nonlinear real arithmetic (NRA) and is decidable using quantifier elimination [24].

4.2 Piecewise Linear Systems and Templates with Parametric Guards

Despite its decidability, the decision procedures for NRA are computationally feasible only for small problems. By making additional assumptions about the system dynamics and templates, we improve the feasibility of the synthesis problem using Farkas’ Lemma. The Farkas’ Lemma [44, p.32 & Table 2.4.1, p.34] states that the following two sentences are equivalent:

$$\begin{aligned} &\forall y \in \mathbb {R}^{N} :Ay \le b \implies c^\textsf{T}y \le d \end{aligned}$$
(41)
$$\begin{aligned} &\exists z \in \mathbb {R}^L_{\ge 0} :\left( \begin{array}{l} A^\textsf{T} z = c\\ \wedge ~b^\textsf{T}z \le d \end{array}\right) \vee \left( \begin{array}{l} A^\textsf{T}z = 0\\ \wedge ~ b^\textsf{T}z < 0 \end{array}\right) \end{aligned}$$
(42)

with z constituting a freshly introduced set of variables. This rewrite eliminates the quantifier alternation and yields a decision problem in the first-order existential theory of non-linear real arithmetic (QF_NRA). In the case where the functions g in Eq. (40) are linear in the variables \(y \in \mathbb {R}^{N}\), and with the help of a technical result that allows strict inequalities in Eq. (40) to be replaced by non-strict inequalities (cf. [20, Lemma 1]), we find that Eq. (40) takes the form of Eq. (41), allowing Farkas’ Lemma to be applied.

Example 4

(General Control). Considering Example 3, suppose we want to synthesise a value for the control parameter \(\kappa \) such that the specification \(\varPhi \) is satisfied almost surely, along with a Streett supermartingale and supporting invariant. For this purpose, we introduce template parameters \(\theta = (\alpha _0, \beta _0, \alpha _1, \beta _1, \alpha _2, \beta _2)\), \(\eta = (\eta _1, \eta _2, \eta _3, \eta _4)\) and template the Streett supermartingale and supporting invariant using the following form:

$$\begin{aligned} \begin{aligned} V(x,q_1; \theta ) &= \alpha _1 \cdot x+ \beta _1 \\ I(x,q_1; \eta ) &= fv (\eta _1 \cdot x \le \eta _2) \wedge (\eta _3 \cdot x \le \eta _4) \end{aligned} \end{aligned}$$
(43)

proceeding analogously for states other than \(q_1\), which yields for \(q \in \{q_0, q_1, q_2\}\) the following expression for \({{\,\textrm{Post}\,}}V\) in terms of the control parameter \(\kappa \):

$$\begin{aligned} \begin{aligned} {{\,\textrm{Post}\,}}V(x, q; \theta , \kappa ) &= {\left\{ \begin{array}{ll} \alpha _0 \kappa \cdot x + \beta _0 &{}\text {if }x \ge 1\\ \alpha _1 \kappa \cdot x + \beta _1 &{}\text {if }-1 \le x < 1\\ \alpha _2 \kappa \cdot x + \beta _2 &{}\text {if } x < -1 \end{array}\right. } \end{aligned} \end{aligned}$$
(44)

Substituting these expressions into Eqs. (15) to (20) results in a conjunction of implications of the form Eq. (40) over inequalities that are linear in the variable \(x \in \mathbb {R}\), but polynomial over the existentially quantified parameters. For example, the non-increasing drift condition associated with \(q_1\) (cf. Eqs. (19), (34) and (35)) corresponds to a number of implications, one for each case of the piecewise-defined \({{\,\textrm{Post}\,}}V(x, q_1; \theta , \kappa )\). Considering the case \(x \ge 1\), we see that the templated implication analogous to Eq. (34) is:

$$\begin{aligned} \begin{aligned} &\forall x \in \mathbb {R}:\underbrace{ \begin{bmatrix} \eta _1 \\ \eta _3 \\ -1 \end{bmatrix} \begin{bmatrix} x \end{bmatrix} \le \begin{bmatrix} \eta _2 \\ \eta _4 \\ -1 \end{bmatrix} }_{I(x, q_1; \eta ) \wedge (x \ge 1)} \implies \underbrace{ \begin{bmatrix} \alpha _0 \kappa - \alpha _1 \end{bmatrix} \begin{bmatrix} x \end{bmatrix} \le \begin{bmatrix} \beta _1 - \beta _0 \end{bmatrix} }_{ {{\,\textrm{Post}\,}}V(x, q_1; \theta , \kappa ) \le V(x,q_1; \theta )} \end{aligned} \end{aligned}$$
(45)

which is in the form of Eq. (41) and yields an existentially quantified disjunction of polynomial inequalities (over the existentially quantified variables, which include the template and control parameters) once rewritten into form Eq. (42), namely a problem in the existential first-order theory of non-linear real arithmetic.    \(\square \)

4.3 Piecewise Linear Systems and Templates with Known Guards

Supposing additionally that an inductive invariant is externally provided, we further improve the computational feasibility of the synthesis problem by reducing it to a quadratically-constrained programming (QCP) problem. In this setting, all inequalities in the premise of Eq. (40) are known linear inequalities of the vector y, and the matrix A and vector b in Eq. (41) are constant (i.e. contain no existentially quantified variables). Therefore, the satisfiability of the premise of Eq. (41) is decidable using linear programming to check whether \(Ay \le b\) admits any solution for y. After removing any implications of the form Eq. (41) which possess an unsatisfiable premise, we may exploit a special case of Farkas’ Lemma that assumes a satisfiable premise [20, Theorem 3]. This version states that if there exists a solution to the system \(Ay \le b\), then the formula Eq. (41) is equivalent to

$$\begin{aligned} \exists z \in \mathbb {R}^L_{\ge 0} :\left( \begin{array}{l} A^\textsf{T} z = c\\ \wedge ~b^\textsf{T}z \le d \end{array}\right) . \end{aligned}$$
(46)

This formula is an existentially quantified conjunction of inequalities, thus transforming the synthesis problem into deciding the satisfiability of a conjunction of polynomial constraints. Such a system of polynomial constraints is reducible to QCP, since higher degree polynomial expressions may be constructed from quadratic constraints by introducing fresh variables. This establishes the reduction to QCP for shielded control when applied to piecewise linear systems and templates, with known invariant. Furthermore, as illustrated in Example 6, if additionally the system is autonomous, the synthesis problem reduces to an LP.

Example 5

(Shielded Control). Continuing from Example 4, we note that if a sufficiently strong invariant is provided a priori (such as that of Eq. (22)), then the synthesis problem reduces to implications of the form Eq. (40) with the property that the linear inequalities occurring within the premise of an implication have constant coefficients. Instead of Eq. (45), for example, we obtain:

$$\begin{aligned} \begin{aligned} &\forall x \in \mathbb {R}:\underbrace{ \begin{bmatrix} 1 \\ -1 \\ -1 \end{bmatrix} \begin{bmatrix} x \end{bmatrix} \le \begin{bmatrix} 0.9 \\ 0.2 \\ -1 \end{bmatrix} }_{I(x, q_1; 1, 0.9, -1, 0.2 ) \wedge (x \ge 1) } \implies \underbrace{ \begin{bmatrix} \alpha _0 \kappa - \alpha _1 \end{bmatrix} \begin{bmatrix} x \end{bmatrix} \le \begin{bmatrix} \beta _1 - \beta _0 \end{bmatrix} }_{ {{\,\textrm{Post}\,}}V(x, q_1; \theta , \kappa ) \le V(x,q_1; \theta )} \end{aligned} \end{aligned}$$
(47)

The premise of Eq. (47) is a known system of linear inequalities, so if its premise is satisfiable (decidable via linear programming) an application of Eq. (46) transforms the synthesis problem into an existentially quantified conjunction of polynomial constraints. The particular constraint Eq. (47) has an unsatisfiable premise, however, and is thus vacuously true.    \(\square \)

Example 6

(Verification). Assuming that \(\kappa = 0.5\), the dynamical model results in an autonomous system, and if a sufficiently strong supporting invariant is provided a priori (as is precisely the case in Example 3), then the implication Eq. (47) becomes:

$$\begin{aligned} \begin{aligned} &\forall x \in \mathbb {R}:\underbrace{ \overbrace{ \begin{bmatrix} 1 \\ -1 \\ -1 \end{bmatrix} }^{A} \begin{bmatrix} x \end{bmatrix} \le \overbrace{ \begin{bmatrix} 0.9 \\ 0.2 \\ -1 \end{bmatrix}}^{b} }_{I(x, q_1; 1, 0.9, -1, 0.2) \wedge (x \ge 1)} \implies \underbrace{ \overbrace{\begin{bmatrix} 0.5 \cdot \alpha _0 - \alpha _1 \end{bmatrix}}^{c^\textsf{T}} \begin{bmatrix} x \end{bmatrix} \le \overbrace{ \begin{bmatrix} \beta _1 - \beta _0 \end{bmatrix} }^{d} }_{ {{\,\textrm{Post}\,}}V(x, q_1; \theta , 0.5) \le V(x,q_1; \theta )} \end{aligned} \end{aligned}$$
(48)

In this case, for an implication with satisfiable premise, we may apply Eq. (46) to obtain an existentially quantified conjunction of inequalities that are linear in x, but further note that matrix A and vector b have constant entries, whereas the vector c and scalar d are linear expressions over template variables. Thus, an application of Eq. (46) generates an existentially quantified conjunction of linear constraints, which is decidable using a linear program.    \(\square \)

5 Experimental Evaluation

We implement our algorithmic technique for the synthesis of Streett supermartingales, supporting invariants, and control policies. Our implementation does not require externally provided invariants, and assumes a template for the Streett supermartingale that is linear in the state variables, and thereby of the form Eq. (38) with a single case for each automaton state. We assume a convex polyhedral template for the supporting invariant, and apply Farkas’ Lemma to produce a decision problem in QF_NRA (Sect. 4.2). In Table 1, we demonstrate examples of \(\omega \)-regular properties and of infinite-state probabilistic systems, with piecewise linear dynamics, certificates and supporting invariants. The Output column of Table 1 describes the synthesis problem (cf. Sect. 4): VIC indicates general control; VC indicates shielded control; VI indicates synthesis of Streett supermartingales and supporting invariants for an autonomous stochastic dynamical model; V indicates verification (namely, synthesis of only Streett supermartingales, using an externally provided invariant).

We use the Spot library [30] to translate the LTL formulae shown in Table 1 into deterministic Streett automata with state-based acceptance conditions. We use SymPy [47] to perform symbolic manipulations and generate the corresponding decision problem, which we solve using an off-the-shelf SMT solver, Z3 [38, 50]. The systems in Table 1 are all infinite-state, namely continuous-state models, with the exception of evenOrNegative that has a countably-infinite state space. The benchmarks make use of discrete random variables, which allows for the post-expectation to be calculated by weighted enumeration of probabilistic choices in the product process (with the exception of evenOrNegative, for which the post-expectation is provided manually). Since our implementation entails deterministic algorithms, we provide the time associated with a single execution owing to negligible variance in these timings.

Table 1. Output of our experiments for a range of infinite-state probabilistic systems and \(\omega \)-regular properties.
Full size table

The benchmark Temperature1 (Table 1) is an instance of a general control problem (Sect. 4) that models an air-conditioned room that dissipates heat to its surroundings (at temperature \(x_\text {ext}\)), with stochastic fluctuations of the room temperature. The state \(x_t \in \mathbb {R}\) is the temperature of the room, \(x_\text {ext} = 280\text {K}\) is the external temperature, and the desired temperature is \(x_\text {set}=295\text {K}\), with \(x_0 = x_\text {ext}\). The system has the following dynamics:

$$\begin{aligned} x_{t+1} = x_{t} - \frac{1}{100} (x_{t} - x_\text {ext}) + (\alpha x_t + \beta ) + \frac{1}{10}(2 w_t - 1), \end{aligned}$$
(49)

with \(w_t \sim \text {Bernoulli}(0.5)\). The dynamics depend upon the parameters \(\alpha , \beta \) of the controller, restricted to \(\alpha , \beta \in [-10, 10]\), to reflect the capabilities of the controller. We define two observations, \(\varPi = \{ \text {Hot}, \text {Cold} \}\) with \(\langle \!\langle x \rangle \!\rangle = \{ \text {Hot} \}\) when the temperature x exceeds \(x_\text {set} + 3\), \(\langle \!\langle x \rangle \!\rangle = \{ \text {Cold} \}\) when the temperature falls below \(x_\text {set} - 3\), and \(\langle \!\langle x \rangle \!\rangle = \emptyset \) otherwise. The specification is \(\textsf{FG}(\lnot \text {Hot} \wedge \lnot \text {Cold})\), namely, that the temperature eventually persists within the interval (292, 298) around \(x_\text {set} = 295\). Our method synthesises a certificate, supporting invariant, and controller with \(\alpha = -1/32, \beta = 4787/512\). The supporting invariant is a conjunction of two linear inequalities at each automaton state.

We next illustrate how shielding improves the efficiency of our synthesis algorithm. Temperature3 involves the same controlled dynamics as Eq. (49), but we add a new observation \(\{ \text {Safe} \}\) associated with the temperature being under 310 K, and aim to satisfy the property \(\textsf{G}(\text {Safe}) \wedge (\textsf{GF}(\text {Cold}) \rightarrow \textsf{GF}(\text {Hot}))\). To synthesise a memoryless controller \(\alpha = -1/64, \beta = 9/2\) for Temperature3 along with a suitable inductive invariant requires a total of 28.58 s (of which the QF_NRA solver requires 23.65 s). In Temperature4 we consider a shielded memoryless controller that ensures the temperature always stays under 310 K:

$$\begin{aligned} \begin{aligned} x_{t+1} &= x_{t} - \frac{1}{100} (x_{t} - x_\text {ext}) + \frac{1}{10}(2 w_t - 1) + {\left\{ \begin{array}{ll} \alpha x_{t} + \beta &{}x_{t} < 305\\ -3 &{}x_{t} \ge 305 \end{array}\right. } \end{aligned} \end{aligned}$$
(50)

and we desire a certified controller for the same reactivity property as in benchmark Temperature3. We constrain \(\alpha ,\beta \in [-5, 5]\) (as modelling assumptions), and we impose \(305 \cdot \alpha + \beta < 5.24\) to ensure that the temperature never exceeds 310K. We provide an invariant a priori, and the resulting QCP is solvable in 0.03 s to obtain \(\alpha =-43/3200, \beta =4\).

To illustrate how our framework is applicable to finite memory controllers, we consider in FinMemoryControl (Table 1) a controller that has one bit of memory (denoted by \(b \in \{0, 1\}\)), which is updated according to the current state x. That is, the system has state space \(\mathbb {R}\times \{ 0, 1 \}\), with update function (cf. Eq. (12)):

$$\begin{aligned} f(x,b,w; \kappa ) = {\left\{ \begin{array}{ll} (x + \alpha + w, 1) &{}\text {if }b = 1 \wedge l\cdot x \le m\\ (x + \alpha + w, 0)&{}\text {if }b = 1 \wedge l \cdot x > m\\ (x + w - 1, 1)&{}\text {if }b = 0 \wedge p \cdot x \le q \\ (x + w - 1, 0) &{}\text {if }b = 0 \wedge p \cdot x > q \end{array}\right. } \end{aligned}$$
(51)

where \(\kappa = ( l, m, p, q, \alpha )\) is the set of control parameters. That is, we wish to synthesise the output of the controller, \(\alpha \), but also the logic that determines how the controller’s memory is to be updated, given the reactivity specification \(\textsf{GF}(x \le 0) \rightarrow \textsf{GF}(x \ge 100)\). This is a decision problem in QF_NRA given that the guards of the template for \(f(x, b, w; \kappa )\) contain template parameters (Sect. 4.2). Our method finds \(\alpha =56, l=1/8,m=14,p=1/2,q=51\).

In summary, we find that our synthesis procedure based on Farkas’ Lemma (Sect. 4.2) allows for the practical synthesis of Streett supermartingales, supporting invariants and control parameters for a range of \(\omega \)-regular properties for infinite (countable/continuous) state piecewise linear probabilistic systems, with our tool terminating in under 30 s for the examples considered. We further illustrate that stronger assumptions (e.g. the external provision of shields or supporting invariants) improve the computational efficiency of control synthesis (cf. Temperature3 vs. Temperature4).

6 Related Work

The verification of finite Markov chains is a classic topic for which automated and scalable algorithmic tools exist [41], which combine graph algorithms and linear algebra to directly compute the probability of satisfying an \(\omega \)-regular specification [10]. This approach exploits the limit behaviour of finite Markov chains, reducing the problem to computing the reachability probabilities of bottom strongly connected components by leveraging the finite graph structure. These techniques do not, however, apply to probabilistic processes over countably infinite or continuous state spaces, which are the focus of this work.

Verification of continuous-state Markov processes has been addressed via two main strategies [42]. The first approximates the continuous-state process with an abstract finite state process (e.g., through state space discretisation) and performs finite-state model checking on the abstraction [29, 58, 62, 69]. The second strategy certifies the property of interest by providing a suitable certificate, using supermartingale theory to analyse Markov chains over general state spaces [48, Chapter 8.4]. This includes supermartingale certificates for specific linear-time properties including almost-sure reachability [15], probabilistic safety [21, 23, 59], reach-avoidance [22], persistence, and recurrence [17, 43]. These rules are justified using martingale theory, including concentration inequalities (e.g. Azuma’s inequality [23]) and the supermartingale convergence theorem [31, Theorem 5.2.9, p.236], with recent order-theoretic justifications [59].

Although prior work introduced supermartingale proof rules for almost-sure persistence and recurrence [17], these are too conservative for general reactivity properties. For instance, in Fig. 1, a reactivity property (a disjunction of persistence and recurrence requirements) holds almost-surely, even though neither disjunct holds almost-surely. Recent work [8] addressed proving \(\omega \)-regular properties with deterministic Streett automata by synthesising a control policy and barrier certificates for the persistence component of each Streett pair. However, as the authors mention, this approach disregards the recurrence component and is thus conservative and mainly suited to safety specifications [8, Section 8.1].

Our approach, by contrast, applies to general Streett pairs, using the Robbins  & Siegmund convergence theorem (Theorem 1) to establish that the disjunction of persistence and recurrence properties in each Streett pair is satisfied with probability one, without requiring either disjunct to hold almost surely. While the Robbins & Siegmund convergence theorem has applications in statistics [9], stochastic optimisation [49, Theorem 17.15], and reinforcement learning [12, Proposition 4.2], we are the first to apply it to derive a supermartingale certificate for Streett conditions and, as a result, \(\omega \)-regular properties.

The algorithmic synthesis of supermartingale certificates for reachability, probabilistic safety, persistence, and recurrence has been addressed for affine programs and certificates using Farkas’ Lemma [6, 15, 20, 23] and for polynomial programs and certificates using Putinar’s Positivstellensatz [19, 21], producing linear- or quadratically-constrained programs, assuming suitably strong inductive invariants are provided a priori. These techniques were first introduced for the synthesis of ranking certificates and invariants for deterministic programs [26, 27]. We apply these techniques to synthesise Streett supermartingales, supporting invariants, and control policies by deciding the satisfiability of a single query in the existential first-order theory of reals (Sect. 5), or by solving a QCP or LP when suitable invariants are externally provided (Sect. 4.3), which may be derived from a shield associated with the controller [7], or when a polyhedral enclosure for the reachable states is known or computed a priori with other methods [25, 32, 46, 53, 57]. Our approach to the joint synthesis of certificates and supporting invariants is in principle applicable to other supermartingale notions studied by prior work [6, 17, 21, 59].

The problem of certified control synthesis in infinite state Markov decision processes (MDPs) has been addressed for specific objectives, such as reachability-reward objectives in countable-state MDPs [11] and reach-avoid specifications for continuous-state MDPs [22, 64, 66], as well as specific infinite-horizon properties [60, 61]. Here, we provide an automated synthesis approach (Sect. 4) applicable to general reactivity properties over continuous-state stochastic processes.

Further automata-theoretic approaches such as recursive Markov chains (RMC) [33, 67] and probabilistic pushdown automata (pPDA) [13, 40, 65] provide means for specifying stochastic processes over countably infinite state spaces. The \(\omega \)-regular model checking problem for these has been studied which, under some restrictions on the model, is decidable [14]. By contrast, our Streett supermartingale theorems (Sect. 2) apply to general stochastic processes (including over uncountably infinite state spaces), though identifying the class of \(\omega \)-regular properties and stochastic processes for which Streett supermartingales are complete (analogous to the notion of positive almost-sure termination [34]) remains an open problem. However, the algorithms in Sect. 4 are relatively complete: if a Streett supermartingale in linear or polynomial form with a known degree exists, our algorithm will compute it.

7 Conclusion

We have introduced the first supermartingale certificate for \(\omega \)-regular properties, by exploiting the Robbins & Siegmund convergence theorem applied to deterministic Streett automata. Our result is the most expressive supermartingale certificate to date, enabling effective almost-sure verification of reactivity properties without requiring each persistence and recurrence component to hold with probability one, as in previous work. We have provided an algorithm to reduce the problem of synthesising Streett supermartingales along with supporting inductive invariants and control policies to symbolic (SMT) and numerical (QCP, LP) decision procedures, and have demonstrated the practical efficacy of our method on several verification and control examples.

Our approach lends itself to extension towards quantitative verification [21, 59], and towards effective algorithmic synthesis of supermartingale certificates via Positivstellensatz [19]. Furthermore, it is open to data-driven techniques along the lines of recent work on neural certificate learning [1,2,3, 18, 22, 36, 52].