CTMCs with Imprecisely Timed Observations

Abstract

Labeled continuous-time Markov chains (CTMCs) describe processes subject to random timing and partial observability. In applications such as runtime monitoring, we must incorporate past observations. The timing of these observations matters but may be uncertain. Thus, we consider a setting in which we are given a sequence of imprecisely timed labels called the evidence. The problem is to compute reachability probabilities, which we condition on this evidence. Our key contribution is a method that solves this problem by unfolding the CTMC states over all possible timings for the evidence. We formalize this unfolding as a Markov decision process (MDP) in which each timing for the evidence is reflected by a scheduler. This MDP has infinitely many states and actions in general, making a direct analysis infeasible. Thus, we abstract the continuous MDP into a finite interval MDP (iMDP) and develop an iterative refinement scheme to upper-bound conditional probabilities in the CTMC. We show the feasibility of our method on several numerical benchmarks and discuss key challenges to further enhance the performance.

This work has been funded by the NWO grant PrimaVera (NWA.1160.18.238), the EU Consolidator Grant 864075 (CAESAR) the EU Horizon 2020 project MISSION (101008233), and the ERC Starting Grant 101077178 (DEUCE).

1 Introduction

Continuous-time Markov chains (CTMCs) are stochastic processes subject to random timing, which are ubiquitous in reliability engineering [48], network processes [33, 35], and systems biology [14, 20]. Here, we consider finite-state labeled CTMCs, which exhibit partial observability through a labeling function, such that analysis can only be done based on observations of the state. Specific techniques such as model checking algorithms compute quantitative aspects of CTMC behavior under the assumption of a static and known initial state [4, 10].

Conditional probabilities In applications such as runtime monitoring [13, 49], we need to analyze an already running system without a static initial state. Instead, we must incorporate past observations, which are given as a sequence of CTMC labels, each of which is observed at a specific time. We call this sequence of timed labels the evidence. We want to incorporate this evidence by conditioning the state of the CTMC on the evidence. For example, “what is the probability of a failure for a production machine (modeled as a CTMC) before time T, given that we have observed particular labels at earlier times $t_1, t_2, \ldots , t_n$?”

Imprecise observation times These conditional probabilities depend on the exact time at which each label was observed. However, in realistic scenarios, the times for the labels in the evidence may not be known precisely. For example, inspections are always done in the first week of a month, but the precise moment of inspection may be unknown. Intuitively, we can interpret such imprecisely timed evidence as a potentially infinite set of (precisely timed) instances of the evidence that vary only in the observation times. For example, an inspection done on “January 2 exactly at noon” is an instance of the imprecise observation time of “the first week of January.” This perspective motivates a robust version of the previous question: “Given the imprecisely timed evidence, what is the maximal probability of a failure before time T over all instances of the evidence?”

Problem statement In this paper, we are given a labeled CTMC together with imprecisely timed evidence. For each instance of the evidence, we can define the probability of reaching a set of target states, conditioned on that evidence. The problem is to compute the supremum over these conditional probabilities for all instances of the evidence. We generalize this problem by considering weighted conditional reachability probabilities (or simply the weighted reachability), where we assign to each state a nonnegative weight. Standard conditional reachability is then a special case with a weight of one for the target states and zero elsewhere.

Contributions Our main contribution is the first method to compute weighted conditional reachability probabilities in CTMCs with imprecisely timed evidence. Our approach consists of the following three steps.

1) Unfolding In Sect. 3, we introduce a method that unfolds the CTMC over all possible timings of the imprecisely timed evidence. We formalize this unfolding as a Markov decision process (MDP) [47], in which the timing imprecision is reflected by nondeterminism. We show that the weighted reachability can be computed via (unconditional) reachability probabilities on a transformed version of this MDP [12, 39]. For the special case of evidence with precise observation times, we obtain a precise solution to the problem that we can directly compute.

2) Abstraction In general, imprecisely timed evidence yields an unfolded MDP with infinitely many states and actions. In Sect. 4, we propose an abstraction of this continuous MDP as a finite interval MDP (iMDP) [27], similar to game-based abstractions [41]. A robust analysis of the iMDP yields upper and lower bounds on the weighted reachability for the CTMC. Moreover, we propose an iterative refinement scheme that converges to the weighted reachability in the limit.

3) Computing bounds in practice In Sect. 5, we use the iMDP abstraction and refinement to obtain sound upper and lower bounds on the weighted reachability in practice. In Sect. 6, we show the feasibility of our method across several numerical benchmarks. Concretely, we show that we obtain reasonably tight bounds on the weighted reachability within a reasonable time. Finally, we discuss the key challenges in further enhancing the performance of our method in Sect. 8.

Related work Closest to our problem are works on model checking CTMCs against deterministic timed automata (DTA) [2, 22, 23]. Evidence can be expressed as a single-clock DTA, and tools such as MC4CSL [1] can calculate the weighted reachability for precise timings. However, for imprecisely timed evidence, checking CTMCs against DTAs yields the sum of probabilities over all instances of the evidence, whereas we are interested in the maximal probability over all instances.

Our setting is also similar to synthesizing timeouts in CTMCs with fixed-delay transitions [9, 15, 42]. Finding optimal timeouts is similar to our objective of finding an instance of the imprecisely timed evidence such that the weighted reachability is maximized. While timeouts can model the time between observations, we consider global observation times, i.e., the time between observations depends on the previous time of observation—which cannot be modeled with timeouts.

We discuss other related work in more detail in Sect. 7.

2 Problem Statement

We recap continuous-time Markov chains (CTMCs) [4, 10] and formalize the problem statement. The set of all probability distributions over a finite set X is denoted as $Dist(X)$. We write tuples $\langle a, b \rangle$ with square brackets, and $\mathbbm {1}_x$ is the indicator function over x, i.e., $\mathbbm {1}_{(y=z)}$ is one if $y=z$ and zero otherwise. We use the standard temporal operators $\lozenge \,$ and $\Box \,$ to denote eventually reaching or always being in a state [11].

Definition 1

(CTMC). A (labeled) continuous-time Markov chain $\mathcal {C}$ is a tuple $\langle S, s_I, \varDelta , E, C, L \rangle$ with a finite set S of states, an initial state $s_I \in S$, a transition matrix $\varDelta :S \rightarrow Dist(S)$, exit-rates $E :S \rightarrow \mathbb {Q}_{\ge 0}$, a finite set of colors C, and a labeling function $L:S \rightarrow C$.

A (timed) CTMC path $\pi = s_0 t_0 s_1 t_2 s_3 \cdots \in \Pi = S \times (\mathbb {R}_{\ge 0} \times S)^*$ is an alternating sequence of states and residence times, where $\varDelta (s_i)(s_{i+1}) > 0 \, \forall i \in \mathbb {N}$. The path $s_0 3 s_1 4 s_2$ means we stayed exactly 3 time units in $s_0$, then transition to $s_1$, where we stayed 4 time units before moving to $s_2$. The CTMC state at time $t \in \mathbb {R}_{\ge 0}$ is denoted by $\pi (t) \in S$, e.g., $\pi (6.2) = s_1$ for the example path above.

An alternative (and equivalent) view of CTMCs is to combine the transition matrix $\varDelta$ and exit-rates E in a transition rate matrix $R :S \times S \rightarrow \mathbb {Q}_{\ge 0}$, where $R(s,s') = \varDelta (s,s') \cdot E(s)$ $\forall s, s' \in S$ [40]. From state $s \in S$, the transient probability distribution $\text {Pr}_s (t) \in Dist(S)$ after time $t \ge 0$ is $\text {Pr}_s (t) = \delta _s \cdot e^{(R - {\textrm{diag}(E)})t}$, where $\delta _s \in \{0,1\}^{|S|}$ is the Dirac distribution for state s, and ${\textrm{diag}(E)}$ is the diagonal matrix with the exit rates E on the diagonal. Thus, the probability of starting in state s and being in state $s' \in S$ after time t is $\Pr _s(t)(s') \in [0,1]$.

Example 1

Consider a simple, single-product inventory where the number of items in stock ranges from 0 to 2, but we can only observe if the inventory is empty or not. This system is modeled by the CTMC shown in Fig. 1a with states $S = \{s_0,s_1,s_2\}$ (modeling the stock) and labels shown by the two colors ( for empty and  for nonempty). The rates at which items arrive and deplete are $R(s_0,s_1) = R(s_1,s_2) = 3$ and $R(s_1,s_0) = R(s_2,s_1) = 2$, respectively.

Fig. 1.

The CTMC (a) for Example 1, (b) the graph for the precise evidence $\rho = \langle t_1, o_1 \rangle , \langle t_2, o_2 \rangle$, and (c) the states of the MDP unfolding defined by Def. 4.

2.1 Problem statement

The key problem we want to solve is to compute reachability probabilities for the CTMC conditioned on a timed sequence of labels, which we call the evidence.

Evidence The evidence $\rho = \langle t_1, o_1 \rangle , \ldots , \langle t_d, o_d \rangle \in (\mathbb {R}_{> 0} \times C)^d$ is a sequence of d times and labels such that $t_i < t_{i+1}$ for all $i \in \{1,\ldots ,d-1\}$. A timed label $\langle t_i, o_i \rangle$ means that at time $t_i$, the CTMC was in a state $s \in S$, that is, $L(s) = o_i$. Since each time $t \in \mathbb {R}_{> 0}$ can only occur once in $\rho$, we overload $\rho$ and denote the evidence at time $t \in \{t_1,\ldots ,t_d\}$ by $\rho (t) = o \in C$, such that $\langle t,o \rangle \in \rho$. While a timed path of a CTMC describes the state at every continuous point in time, the evidence only contains the observations at $d$ points in time. We say that a path $\pi$ is consistent with evidence $\rho$, written as $\pi \models \rho$, if each timed label in $\rho$ matches the label of path $\pi$ at time t, i.e., if $L(\pi (t)) = \rho (t) \, \forall t \in \{t_1,\ldots ,t_d\}$.

Conditional probabilities We want to compute the conditional probability $\mathbb {P}_\mathcal {C}(\pi (t_d) = s ) \mid [\pi \models \rho ])$ that the CTMC $\mathcal {C}$ with initial state $s_I$ generates a path being in state s at time $t_d$, conditioned on the evidence $\rho$. Using Bayes’ rule, we can characterize this conditional probability as follows (assuming $\frac{0}{0} = 0$, for brevity):

$$\begin{aligned} \begin{aligned} \mathbb {P}_\mathcal {C}(\pi (t_d) = s \mid [\pi \models \rho ]) &= \frac{\mathbb {P}_\mathcal {C}([\pi (t_d) = s] \cap [\pi \models \rho ])}{\mathbb {P}_\mathcal {C}(\pi \models \rho )}. \end{aligned} \end{aligned}$$

(1)

Imprecise timings We extend evidence with uncertainty in the timing of each label. The imprecisely timed evidence (or imprecise evidence) $\varOmega = \langle \mathcal {T}_1, o_1 \rangle , \ldots , \langle \mathcal {T}_d, o_d \rangle$ is a sequence of $d$ labels and uncertain timings $\mathcal {T}_i = \cup _{j=1}^q [\,\underline{t}_{j}, \bar{t}_{j}\,]$, with $\underline{t}_j \le \bar{t}_j$ and $q \in \mathbb {N}$. Observe that $\mathcal {T}$ can model both singletons ($\mathcal {T}_i = \{1,2,3\}$) and unions of intervals ($\mathcal {T}_i = [1,1.5] \cup [2,2.5]$). We require that $\max _{t \in \mathcal {T}_i} (t) < \min _{t' \in \mathcal {T}_{i+1}} (t')$ for all $i \in \{1,\ldots ,d-1\}$, i.e., the order of the labels is known, despite the uncertainty in the observation times. Again, we overload notation and denote the evidence at time t by $\varOmega (t) = o$, such that $\exists \langle \mathcal {T},o \rangle \in \varOmega$ with $t \in \mathcal {T}$. Imprecise evidence induces a set of instances of the evidence that only differ in the label times. This set of instances is uncountably infinite whenever one of the imprecise timings $\mathcal {T}$ is a continuous set. Formally, the evidence $\rho = \langle t_1, o_1 \rangle , \ldots , \langle t_d, o_d \rangle$ is an instance of the imprecise evidence $\varOmega$, written as $\rho \in \varOmega$, if $t_i \in \mathcal {T}_i$ for all $i=1,\ldots ,d$.

Example 2

An example of imprecise evidence for the CTMC in Example 1 is . The precise evidence is an instance of $\varOmega$, i.e., $\rho \in \varOmega$. However, and are not, i.e., $\rho ' \notin \varOmega$, $\rho '' \notin \varOmega$, as the timings and labels do not match, respectively.

State-weights Let $w:S \rightarrow \mathbb {R}_{\ge 0}$ be a state-weight function, which assigns to each CTMC state $s \in S$ a non-negative weight. The weight w(s) represents a general measure of risk associated with each state $s \in S$, as used in [39]. For example, w(s) may represent the probability of reaching a set of target states $S_T$ from s within some time horizon $h \ge 0$. We then consider the following problem.

Example 3

For the CTMC in Example 1, consider the state-weight function that assigns to each state the probability of reaching state $s_0$ within time $t=0.1$. Then, the problem above is interpreted as: Given the imprecisely timed evidence $\varOmega$, compute the probability (conditioned on $\varOmega$) of reaching state $s_0$ within time $t=0.1$ (after the end of the evidence).

Our overall workflow to solve Problem 1 is summarized in Fig. 2 and consists of four blocks, which we discuss in Sects. 3 to 5, respectively.

Variations To instead minimize Eq. (2), we would swap every $\inf$ and $\sup$ (and $\max$ and $\min$) in the paper, but our general approach remains the same. Furthermore, by setting $w(s) = 1$ for all $s \in S_T$ and zero otherwise, we can also compute the probability of being in a state in $S_T$ immediately after the evidence. Finally, we remark that Problem 1 only considers events after the end of the evidence. This setting is motivated by applications where the exact system state is not observable, but actual system failures can be observed. Thus, one can typically assume that the system has not failed yet and the problem as formalized in Problem 1 is to predict the conditional probability of a future system failure.

Fig. 2.

Conceptual workflow of our approach for solving Problem 1.

2.2 Interval Markov decision processes

We recap interval MDPs (iMDPs) [27] and define standard MDPs as special case. We denote (i)MDP states by $q \in Q$, whereas CTMC states are denoted $s \in S$.

Definition 2

(iMDP). An interval MDP $\mathcal {I}$ is a tuple $\langle Q, q, A, \mathcal {P} \rangle$, with Q a set of states, $q \in Q$ the initial state, A a set of actions, and where the uncertain transition function $\mathcal {P}:Q \times A \times Q \rightharpoonup \mathbbm {I}\cup \{ [0,0] \}$ is defined over intervals $\mathbbm {I}= \{ [a,b] \mid a,b \in (0,1] \text { and } a \le b \}$. The actions enabled in state $q \in Q$ are $A(q) \subseteq A$.

The assumption that an interval cannot have a lower bound of 0 except the [0, 0] interval is standard, see, e.g., [46, 52]. An MDP is a special case of iMDP, where the upper and lower bounds coincide, i.e., $\mathcal {P}(q,a,q') = [b,b], \, b \in [0,1]$ for all intervals, and each $\mathcal {P}(q,a,\cdot ) \in Dist(Q)$ is a distribution over states. We denote an MDP as $\mathcal {M}= \langle Q,q,A,P \rangle$, with transition function $P :Q \times A \times Q \rightharpoonup [0,1]$. For an MDP $\mathcal {M}$ with transition function P, we write $P \in \mathcal {P}$ if for all $q,q' \in Q$ and $a \in A$ we have $P(q,a,q') \in \mathcal {P}(q,a,q')$ and each $P(q, a, \cdot ) \in Dist(Q)$. Fixing a transition function $P \in \mathcal {P}$ for iMDP $\mathcal {I}$ yields an induced MDP $\mathcal {I}[P]$.

The nondeterminism in an iMDP $\mathcal {I}$ is resolved by a memoryless scheduler $\sigma :Q \rightarrow A$, with $\sigma \in \textrm{Sched}_{\mathcal {I}}$ the set of all schedulers. We denote a finite (i)MDP path by $\xi = q_0, \ldots , q_n \in \Xi _{\mathcal {I}}^\sigma$, where $\Xi _{\mathcal {I}}^\sigma$ is the set of all paths under scheduler $\sigma$. For the Markov chain induced by scheduler $\sigma$ in $\mathcal {I}[P]$, we use the standard probability measure $\mathbb {P}_{\mathcal {I}[P]}^\sigma$ over the smallest sigma-algebra containing the cylinder sets of all finite paths $\xi \in \Xi _{\mathcal {I}}^\sigma$; see, e.g., [11]. If $\textrm{Sched}_{\mathcal {I}}$ is a singleton (i.e., $\mathcal {I}$ has only one scheduler), we omit the script $\sigma$ and simply write $\mathbb {P}_{\mathcal {I}[P]}$ and $\Xi _{\mathcal {I}}$. For MDPs $\mathcal {M}$, we use the analogous notation with subscripts $\mathcal {M}$.

3 Conditional Reachability with Imprecise Evidence

In this section, we treat the first two blocks of Fig. 2. In Sect. 3.1, we unfold the CTMC over the times in the imprecise evidence into an MDP. The main result of this section, Theorem 1, states that the conditional reachability on the CTMC in Problem 1 is equal to the maximal conditional reachability probabilities in the MDP over a subset of schedulers (those that we call consistent; see Def. 5). In Sect. 3.2, we use results from [12] to determine these conditional probabilities via unconditional reachability probabilities on a transformed version of the MDP.

3.1 Unfolding the CTMC into an MDP

We interpret the (precisely timed) evidence $\rho = \langle t_1, o_1 \rangle ,\ldots ,\langle t_d, o_d \rangle$ as a directed graph that encodes the trivial progression over the time steps $t_1,\ldots ,t_d$.

Definition 3

(Evidence graph). An evidence graph $\mathcal {G}= \langle \mathcal {N}, \mathcal {E} \rangle$ is a directed graph where each node $t \in \mathcal {N}\subseteq \mathbb {R}_{> 0}$ is a point in time, and with directed edges $\mathcal {E}\subset \{ t \rightarrow t' : t,t' \in \mathcal {N}\}$, such that $t' > t$ for all $t \rightarrow t' \in \mathcal {E}$.

The graph $\mathcal {G}_\rho = \langle \mathcal {N}_\rho , \mathcal {E}_\rho \rangle$ for the precise evidence $\rho$ has nodes $\mathcal {N}_\rho = \{ 0, t_1, \ldots , t_d, t_\star \}$ and edges $\mathcal {E}_\rho = \{ t_{i-1} \rightarrow t_i : i = 2,\ldots ,d\} \cup \{ 0 \rightarrow t_1, t_d\rightarrow t_\star \}$. As illustrated in Fig. 1b, the graph $\mathcal {G}_\rho$ has exactly one path, which follows the time points $t_1,\ldots ,t_d$ of the evidence $\rho$ itself. Likewise, we model the imprecise evidence $\varOmega$ as a graph $\mathcal {G}_\varOmega$ which is the union of all graphs $\mathcal {G}_\rho$ for all instances $\rho \in \varOmega$, i.e.,

$$\begin{aligned} \mathcal {G}_\varOmega = \langle \mathcal {N}_\varOmega , \mathcal {E}_\varOmega \rangle = \cup _{\rho \in \varOmega } (\mathcal {G}_\rho ) = \langle \cup _{\rho \in \varOmega } (\mathcal {N}_\rho ), \cup _{\rho \in \varOmega } (\mathcal {E}_\rho ) \rangle . \end{aligned}$$

(3)

If $\varOmega$ has infinitely many instances, then $\mathcal {G}_\varOmega$ has infinite branching. Every path $t_0 t_1 \ldots t_dt_\star$ through graph $\mathcal {G}_\varOmega$ corresponds to the time points of the precise evidence $\rho = \langle t_1, o_1 \rangle ,\ldots ,\langle t_d, o_d \rangle \in \varOmega$ (and vice versa).

We denote the successor nodes of $t \in \mathcal {N}$ by $\textsf{post}(t) = \{ t' \in \mathcal {N}: t \rightarrow t' \in \mathcal {E}\}$. For example, the graph in Fig. 1b has $\textsf{post}(0) = t_1$, $\textsf{post}(t_1) = t_2$ and $\textsf{post}(t_2) = t_\star$. We introduce the unfolding operator $\mathcal {M}= \textsf{Unfold}(\mathcal {C}, \mathcal {G})$, which takes a CTMC $\mathcal {C}$ and a graph $\mathcal {G}$, and returns the unfolded MDP $\mathcal {M}$ defined as follows.

Definition 4

(Unfolded MDP). For a CTMC $\mathcal {C}= \langle S, s_I, \varDelta , E, C, L \rangle$ and a graph $\mathcal {G}= \langle \mathcal {N}, \mathcal {E} \rangle$, the unfolded MDP $\textsf{Unfold}(\mathcal {C}, \mathcal {G}) = \langle Q, q_I, A, P \rangle$ has states states $Q = S \times \mathcal {N}$, initial state $q_I = \langle s_I, 0 \rangle$, actions $A = \mathcal {N}$, and transition function P, which is defined for all $\langle s,t \rangle \in Q$, $t' \in \textsf{post}(t)$, $s' \in S$ as

$$\begin{aligned} P\big ( \langle s,t \rangle , t', \langle s',t' \rangle \big ) = {\left\{ \begin{array}{ll} \text {Pr}_s(t' - t)(s') &{} \text {if } t' \ne t_\star , \\ \mathbbm {1}_{(s = s')} &{} \text {if } t' = t_\star , \end{array}\right. } \end{aligned}$$

(4)

The unfolding of the CTMC in Fig. 1a over the graph in Fig. 1b is shown in Fig. 1c. A state $\langle s,t \rangle \in Q$ in the unfolded MDP is interpreted as being in CTMC state $s \in S$ at time t. In state $\langle s,t \rangle$, the set of enabled actions is $A(\langle s,t \rangle ) = \textsf{post}(t) \subset \mathcal {N}$, and taking an action $t' \in \textsf{post}(t)$ corresponds to deterministically jumping to time $t'$. The effect of this action is stochastic and determines the next CTMC state. The transition probability $P( \langle s,t \rangle , t', \langle s',t' \rangle )$ for $t' \ne t_\star$ models the probability of starting in CTMC state $s \in S$ and being in state $s' \in S$ after time $t'-t$ has elapsed, which is precisely the transient probability $\text {Pr}_s(t' - t)(s')$ defined in Sect. 2. Finally, the (terminal) states $\langle s,t_\star \rangle$ for all $s \in S$ are absorbing.

Interpretation of schedulers Every instance $\rho \in \varOmega$ of the imprecise evidence $\varOmega = \langle \mathcal {T}_1, o_1 \rangle , \ldots , \langle \mathcal {T}_d, o_d \rangle$ corresponds to fixing a precise time $t_i \in \mathcal {T}_i$ for all $i=1,\ldots ,d$. For each such $\rho \in \varOmega$, there exists a scheduler $\sigma \in \textrm{Sched}_\mathcal {M}$ for MDP $\mathcal {M}= \textsf{Unfold}(\mathcal {C}, \mathcal {G}_\varOmega )$ that induces a Markov chain which only visits those time points $t_1,\ldots ,t_d$. We call such a scheduler $\sigma$ consistent with the evidence $\rho$.

Definition 5

(Consistent scheduler). A scheduler $\sigma \in \textrm{Sched}_\mathcal {M}$ is consistent with $\rho = \langle t_1, o_1 \rangle , \ldots , \langle t_d, o_d \rangle \in \varOmega$, written as $\sigma \sim \rho$, if for all CTMC states $s \in S$:

$$\begin{aligned} \sigma (\langle s, 0 \rangle ) = t_1, \quad \sigma (\langle s, t_{i} \rangle ) = t_{i+1} \, \forall i \in \{0,\ldots ,d-1\}, \quad \sigma (\langle s, t_d \rangle ) = t_\star . \end{aligned}$$

(5)

We denote the set of all consistent schedulers by $\textrm{Sched}_\mathcal {M}^\textsf{con} \subseteq \textrm{Sched}_\mathcal {M}$.

A consistent scheduler chooses the same action $\sigma (\langle s, t \rangle ) = \sigma (\langle s', t' \rangle )$ in any two MDP states $\langle s, t \rangle , \langle s', t' \rangle \in Q$ for which $t = t'$. There is a one-to-one correspondence between choices $\rho \in \varOmega$ and consistent schedulers: for every $\rho \in \varOmega$, there exists a scheduler $\sigma \in \textrm{Sched}_\mathcal {M}^\textsf{con}$ such that $\sigma \sim \rho$, and vice versa.

Example 4

Consider imprecise evidence for the CTMC in Example 1. A scheduler with $\sigma (\langle s_0, 0.4 \rangle ) = 1.5$, $\sigma (\langle s_1, 0.4 \rangle ) = 1.8$ is inconsistent as it chooses different actions in MDP states with the same time.

Remark 1

The unfolded MDP $\mathcal {M}' = \textsf{Unfold}(\mathcal {C}, \mathcal {G}_\rho )$ for the precise evidence $\rho$ has only a single action enabled in every state (i.e., $\mathcal {M}'$ directly reduces to a discrete-time Markov chain). Hence, $\mathcal {M}'$ has only one scheduler, and $\textrm{Sched}_{\mathcal {M}'}^\textsf{con} = \textrm{Sched}_{\mathcal {M}'}$.

Conditional reachability on unfolded MDP As a main result, we show that $W(\varOmega )$ in Problem 1 can be expressed as maximizing conditional reachability probabilities in the unfolded MDP $\mathcal {M}$ over the consistent schedulers $\textrm{Sched}_\mathcal {M}^\textsf{con} \subset \textrm{Sched}_\mathcal {M}$.

Theorem 1

For a CTMC $\mathcal {C}$ and the imprecise evidence $\varOmega$ with graph $\mathcal {G}_\varOmega$, let $\mathcal {M}= \textsf{Unfold}(\mathcal {C}, \mathcal {G}_\varOmega )$ be the unfolded MDP. Then, using the notation from Sect. 2.2 (for the probability measure $\mathbb {P}_\mathcal {M}^\sigma$ over paths $\xi \in \Xi _\mathcal {M}^\sigma$), Eq. (2) is rewritten as

$$\begin{aligned} \begin{aligned} W(\varOmega ) &= \sup _{\sigma \in \textrm{Sched}_\mathcal {M}^\textsf{con}} \, \sum _{s \in S} \mathbb {P}_\mathcal {M}^\sigma ( \lozenge \, \langle s, t_\star \rangle \mid [\xi \models \rho , \,\, \sigma \sim \rho ]) \cdot w(s). \end{aligned} \end{aligned}$$

(6)

Proof

The proof is in [8, Appendix A] and shows that for every instance $\rho \in \varOmega$, the conditional transient probabilities in the CTMC are equivalent to conditional reachability probabilities in the unfolded MDP under a $\sigma \sim \rho$ consistent to $\rho$.    $\square$

Fig. 3.

The unfolded MDP from Fig. 1c conditioned on different precise evidences. States that do not agree with the evidence are looped back to the initial state.

3.2 Computing conditional probabilities in MDPs

We describe a transformation of the unfolded MDP to compute the conditional reachability probabilities in Eq. (6). Intuitively, we refute all paths through the MDP that do not agree with the labels in the evidence. Specifically, we find the subset of MDP states $Q_\textsf{reset}(\varOmega ) \subset Q$ that disagree with the evidence, defined as

$$\begin{aligned} Q_\textsf{reset}(\varOmega ) = \big \{ \langle s,t \rangle \in Q : \, L(s) \ne \varOmega (t) \big \} \subset Q. \end{aligned}$$

(7)

We reset all states in $Q_\textsf{reset}(\varOmega )$ by adding transitions back to the initial state with probability one. Formally, we define the conditioned MDP $\mathcal {M}_{|\varOmega }$ as follows.

Definition 6

(Conditioned MDP). For $\mathcal {M}= \textsf{Unfold}(\mathcal {C}, \mathcal {G}_\varOmega ) = \langle Q, q_I, A, P \rangle$, the conditioned MDP $\mathcal {M}_{|\varOmega } = \langle Q, q_I, A, P_{|\varOmega } \rangle$ has the same states and actions, but the transition function is defined for all $\langle s,t \rangle \in Q$, $t' \in \textsf{post}(t)$, $s' \in S$ as

$$\begin{aligned} P_{|\varOmega } \big ( \langle s,t \rangle , t', \langle s', t' \rangle \big ) = {\left\{ \begin{array}{ll} P \big ( \langle s,t \rangle , t', \langle s', t' \rangle \big ) \! \! &{} \text {if } \langle s,t \rangle \notin Q_\textsf{reset}(\varOmega ), \\ \mathbbm {1}_{(s' = s_I)} &{} \text {if } \langle s,t \rangle \in Q_\textsf{reset}(\varOmega ). \end{array}\right. } \end{aligned}$$

(8)

Two examples of conditioning on precise evidence are shown in Fig. 3. Compared to Fig. 1c, we removed all probability mass over paths that are not consistent with the evidence and normalized the probabilities for all other paths. The following result from [12] shows that conditional reachabilities in the unfolded MDP are equal to unconditional reachabilities in the conditioned MDP.

Lemma 1

(Thm. 1 in [12]). For the imprecise evidence $\varOmega$, unfolded MDP $\mathcal {M}= \textsf{Unfold}(\mathcal {C}, \mathcal {G}_\varOmega )$, and conditioned MDP $\mathcal {M}_{|\varOmega }$ defined by Def. 6, it holds that

$$\begin{aligned} \mathbb {P}_\mathcal {M}^\sigma ( \lozenge \, \langle s, t_\star \rangle \mid [\xi \models \rho , \,\, \sigma \sim \rho ]) \, = \, \mathbb {P}_{\mathcal {M}_{|\varOmega }}^\sigma ( \lozenge \, \langle s, t_\star \rangle ) \quad \forall \sigma \in \textrm{Sched}_\mathcal {M}\,\, \forall s \in S. \end{aligned}$$

(9)

Finally, combining Lemma 1 with Theorem 1 directly expresses the conditional reachability $W(\varOmega )$ in terms of reachability probabilities on the conditioned MDP.

Theorem 2

Given a CTMC $\mathcal {C}$, a state-weight function $w$, and the imprecisely timed evidence $\varOmega$, let $\mathcal {M}= \textsf{Unfold}(\mathcal {C}, \mathcal {G}_\varOmega )$. Then, it holds that

$$\begin{aligned} \begin{aligned} W(\varOmega ) &= \sup _{\sigma \in \textrm{Sched}_\mathcal {M}^\textsf{con}} \sum _{s \in S} \mathbb {P}^\sigma _{\mathcal {M}_{|\varOmega }}( \lozenge \, \langle s, t_\star \rangle ) \cdot w(s). \end{aligned} \end{aligned}$$

(10)

Solving Problem 1 with precisely timed evidence is now straightforward by solving a finite DTMC, see Remark 1. Furthermore, if the imprecise evidence has finitely many instances, then the MDP is finite. A naive approach to optimize over the consistent schedulers is enumeration, which we discuss in details Sect. 5.

Remark 2

(Variations on Problem 1). With minor modifications to our approach, we can compute, e.g., the likelihood that a CTMC generates precise evidence $\rho$. Concretely, we define a transformed version $\mathcal {M}_\rho$ of the unfolded MDP in which all states in $Q_\textsf{reset}$ are absorbing. We discuss this variation in [8, Appendix C]

4 Abstraction of Conditioned MDPs

For imprecisely timed evidence with infinitely many instances (e.g., imprecise timings over intervals), the conditioned MDP from Sect. 3 has infinitely many states and actions. In this section, we treat block (3) of Fig. 2 and propose an abstraction of this continuous MDP into a finite interval MDP (iMDP). Similar to game-based abstractions [29, 30, 41], we capture abstraction errors as nondeterminism in the transition function of the iMDP. Robust reachability probabilities in the iMDP yield sound bounds on the conditional reachability $W(\varOmega )$. The crux of our abstraction is to create a finite partition of the (infinite) sets of uncertain timings in the evidence, as illustrated by Fig. 4.

Fig. 4.

Two partitions of imprecise evidence $\varOmega = \langle [0.2, 0.8], o_1 \rangle , \langle [1.4, 2.1], o_2 \rangle$. The partition in (a) consists of two elements, such that $\tilde{\mathcal {T}}_1^1 = [0.2, 0.8]$ and $\tilde{\mathcal {T}}_2^1 = [1.4, 2.1]$, where (b) refines this to $\tilde{\mathcal {T}}_1^1 \cup \tilde{\mathcal {T}}_1^2 = [0.2, 0.8]$ and $\tilde{\mathcal {T}}_2^1 \cup \tilde{\mathcal {T}}_2^2 = [1.4, 2.1]$.

Definition 7

(Time partition). A time partition $\varPsi$ of the imprecise evidence $\varOmega = \langle \mathcal {T}_1, o_1 \rangle , \ldots , \langle \mathcal {T}_d, o_d \rangle$ is a set $\varPsi = \cup _{i=1}^d \textsf{partition}(\mathcal {T}_i) \cup \{0, t_\star \}$, where each $\textsf{partition}(\mathcal {T}_i) = \{\mathcal {T}_i^1, \ldots , \mathcal {T}_i^{n_i}\}$ is a finite partition¹ of $\mathcal {T}_i$ into $n_i \in \mathbb {N}$ elements.

With abuse of notation, the element of $\varPsi$ containing time t is $\varPsi (t) \in \varPsi$, and $\varPsi ^{-1}(\psi ) = \{ t : \varPsi (t) = \psi \}$ is the set of times mapping to $\psi \in \varPsi$. As shown by Fig. 4, for each $i \in \{1,\ldots ,d\}$, the sets $\tilde{\mathcal {T}}_i^1, \ldots , \tilde{\mathcal {T}}_i^{n_i}$ are a partition of the set $\mathcal {T}_i$.

To illustrate the abstraction, let $\langle s,t \rangle \xrightarrow {t' : P'} \langle s',t' \rangle$ denote the MDP transition from state $\langle s,t \rangle \in Q$, under action $t' \in A(\langle s,t \rangle )$ to state $\langle s',t' \rangle \in Q$, which has probability $P'$. With this notation, we can express any MDP path as

$$\begin{aligned} \langle s_I, 0 \rangle &\xrightarrow {t : P} {} & {} \langle s,t \rangle {} & {} \xrightarrow {t' : P'} {} & {} \langle s',t' \rangle {} & {} \xrightarrow {t'' : P''} \cdots {} & {} \xrightarrow {t''' : P'''} {} & {} \langle s,t_\star \rangle . \end{aligned}$$

(11)

For every element $\psi \in \varPsi$ of partition $\varPsi$, the abstraction merges all MDP states $\langle s,t \rangle \in Q$ for which the time t belongs to the element $\psi$, that is, for which $t \in \varPsi ^{-1}(\psi )$. Thus, we merge infinitely many MDP states into finitely many abstract states. The MDP path in Eq. (11) matches the next path in the abstraction:

$$\begin{aligned} \langle s_I, 0 \rangle &\xrightarrow {\mathcal {T}: \mathcal {P}} \, {} & {} \langle s,\mathcal {T} \rangle {} & {} \xrightarrow {\mathcal {T}' : \mathcal {P}'} \, {} & {} \langle s',\mathcal {T}' \rangle {} & {} \xrightarrow {\mathcal {T}'' : \mathcal {P}''}\, \cdots {} & {} \xrightarrow {\mathcal {T}''' : \mathcal {P}'''} \, {} & {} \langle s,t_\star \rangle , \end{aligned}$$

(12)

where each $t \in \mathcal {T}$, and each $\mathcal {P}$ is a set of probabilities. The abstraction contains the behavior of the continuous MDP if $P \in \mathcal {P}$ at every step in Eqs. (11) and (12), see, e.g., [38]. The following iMDP abstraction satisfies these requirements.

Definition 8

(iMDP abstraction). For a conditioned MDP $\mathcal {M}_{|\varOmega } = \langle Q, q_I, A, P \rangle$ and a time partition $\varPsi$ of $\varOmega$, the iMDP abstraction $\mathcal {I}= \textsf{Abstract}(\mathcal {M}_{|\varOmega }, \varPsi ) = \langle \tilde{Q}, q_I, \tilde{A}, \mathcal {P} \rangle$, with states $\tilde{Q} = \big \{ \langle s, \varPsi (t) \rangle : \langle s,t \rangle \in Q \big \}$, actions $\tilde{A} = \big \{ \varPsi (t) : t \in A \big \}$, and uncertain transition function $\mathcal {P}$ defined for all $\langle s, \mathcal {T} \rangle , \langle s', \mathcal {T}' \rangle \in \tilde{Q}$ as

$$\begin{aligned} \begin{aligned} \mathcal {P}\big ( \langle s,\mathcal {T} \rangle , \mathcal {T}', \langle s',\mathcal {T}' \rangle \big ) = \textrm{cl}\Big ( \!\!\!\!\! \bigcup _{t \in \varPsi ^{-1}(\mathcal {T}), t' \in \varPsi ^{-1}(\mathcal {T}')} \!\!\!\!\! P\big ( \langle s,t \rangle , t', \langle s', t' \rangle \big )\Big ), \end{aligned} \end{aligned}$$

(13)

where $\textrm{cl}(x) = [\min (x), \max (x)]$ is the interval closure of x.

Fig. 5.

Abstraction of an infinite set of MDP states for all times $t \in [0.2, 0.8]$ into (a) a single iMDP state $\langle s, [0.2, 0.8] \rangle$ with probability intervals that overapproximate the transient distribution (b) as the rectangular set in (c), where the line shows the MDP transition probabilities for all $t \in [0.2,0.8]$. The refinement (d) into two iMDP states $\langle s, [0.2, 0.5] \rangle$ and $\langle s, [0.5, 0.8] \rangle$ splits the approximation of the transient (e) into the two (less conservative) rectangular sets in (f).

An abstraction under the coarse time partition from Fig. 4 is shown in Fig. 5a. The transition probabilities for each MDP state are defined by transient probabilities for the CTMC. Thus, the uncertain transition function $\mathcal {P}$ of the iMDP overapproximates these transient probabilities over a range of times (as shown in Fig. 5b), yielding probability intervals as in Fig. 5c.

Conditional reachability on iMDP We show that the iMDP abstraction can be used to obtain sound upper and lower bounds on the conditional reachability $W(\varOmega )$. Let $W_\mathcal {I}(\tilde{P}, \sigma ) \ge 0$ denote the value for the MDP $\mathcal {I}[\tilde{P}]$ induced by iMDP $\mathcal {I}$ under transition function $\tilde{P}$, and with scheduler $\sigma \in \textrm{Sched}_\mathcal {I}$:

$$\begin{aligned} W_\mathcal {I}(\tilde{P}, \sigma ) {:}{=}\sum _{s \in S} \mathbb {P}_{\mathcal {I}[\tilde{P}]}^\sigma ( \lozenge \, \langle s, t_\star \rangle ) \cdot w(s). \end{aligned}$$

(14)

The next theorem, proven in [8, Appendix B], is the main result of this section.

Theorem 3

Let $\mathcal {I}= \textsf{Abstract}(\mathcal {M}_{|\varOmega }, \varPsi )$ be the iMDP abstraction for a conditioned MDP $\mathcal {M}_{|\varOmega }$ and a time partition $\varPsi$ of $\varOmega$. Then, it holds that

$$\begin{aligned} \begin{aligned} \max _{\sigma \in \textrm{Sched}_{\mathcal {I}}^\textsf{con}} \min _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\sigma ) \le W(\varOmega ) \le \max _{\sigma \in \textrm{Sched}_{\mathcal {I}}^\textsf{con}} \max _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\sigma ). \end{aligned} \end{aligned}$$

(15)

Construction of the iMDP We want to construct the abstract iMDP directly from the CTMC without first constructing the continuous MDP $\mathcal {M}_{|\varOmega }$. Consider computing the probability interval $\mathcal {P}( \langle s,\mathcal {T} \rangle , \mathcal {T}', \langle s',\mathcal {T}' \rangle )$ for the iMDP transition from state $\langle s,\mathcal {T} \rangle$ to $\langle s',\mathcal {T}' \rangle$. This interval is given by the minimum and maximum transient probabilities $\Pr _s(t' - t)(s')$ over all $t \in \mathcal {T}$ and $t' \in \mathcal {T}'$. However, the problem is that the transient probabilities are not monotonic over time in general (see Fig. 5b), so it is unclear how to compute this interval.

Instead, we compute upper and lower bounds for the transient probabilities. Let $\underline{t} = \min (\mathcal {T})$ and $\bar{t} = \max (\mathcal {T})$. An upper bound on the transient probability is given by the probability to reach $s'$ from s at some time $t'-t$, $t \in \mathcal {T}$, $t' \in \mathcal {T}'$:

$$\begin{aligned} \sup _{t \in \mathcal {T}, t' \in \mathcal {T}'} {\Pr }_s(t' - t)(s') \le \sup _{t \in \mathcal {T}, t' \in \mathcal {T}'} \mathbb {P}_{\mathcal {C},s}( \lozenge \, ^{[t, t']} s' ) = \mathbb {P}_{\mathcal {C},s}( \lozenge \, ^{[\underline{t}, \bar{t}']} s' ), \end{aligned}$$

(16)

where $\mathbb {P}_{\mathcal {C},s}$ is the probability measure for the CTMC starting in initial state s, and $\bar{t}' - \underline{t}$ is the maximal time difference. A lower bound is given symmetrically by the transient probability to reach $s'$ in the CTMC at the earliest possible time $\underline{t'} - \bar{t}$ and staying there for the full remaining time $(\bar{t'} - \underline{t}) - (\underline{t}' - \bar{t})$:

$$\begin{aligned} \inf _{t \in \mathcal {T}, t' \in \mathcal {T}'} {\Pr }_s(t' - t)(s') \ge {\Pr }_s(\underline{t'} - \bar{t})(s') \cdot \mathbb {P}_{\mathcal {C},s'}(\Box \, ^{[0, (\bar{t'} - \underline{t}) - (\underline{t}' - \bar{t})]} s'). \end{aligned}$$

(17)

Abstraction refinement

To improve the tightness of the bounds in Theorem 3, we propose a refinement step that splits elements of the time partition $\varPsi$. For example, we may split the single abstract state in Fig. 5a into the two states in Fig. 5d.

Definition 9

(Refinement of time partition). Let $\varPsi$ and $\varPsi '$ be partitions as per Def. 7, for which $|\varPsi '| > |\varPsi |$. We call $\varPsi '$ a refinement of $\varPsi$ if for all $\psi ' \in \varPsi '$, there exists a $\psi \in \varPsi$ such that $\psi ' \subseteq \psi$.

Any refinement $\varPsi$’ of partition $\varPsi$ can be constructed by finitely many splits. We lift the refinement to the iMDP, see also Figs. 5c and 5f. The refined iMDP $\mathcal {I}' = \textsf{Abstract}(\mathcal {M}_{|\varOmega }, \varPsi ')$ has more states and actions, but each union in Eq. (13) is over a smaller set than in iMDP $\textsf{Abstract}(\mathcal {M}_{|\varOmega }, \varPsi )$. Thus, the refinement leads to smaller probability intervals and, in general, to tighter bounds in Theorem 3. Repeatedly refining every element of the partition yields an iMDP with arbitrarily many states and actions and with arbitrarily small probability intervals. Hence, in the limit, we may recover the original continuous MDP by refinements, which also implies that the bounds in Theorem 3 on the refined iMDP converge.

Refinement strategy By splitting every element of the partition $\varPsi$, the number of iMDP states and actions double per iteration, and the number of transitions grows exponentially. Thus, we employ the following guided refinement strategy. At each iteration, we extract the scheduler $\sigma ^\star$ that attains the upper bound in Theorem 3 and determine the set $\tilde{Q}_\textsf{reach}^{\sigma ^\star } \subset \tilde{Q}$ of reachable iMDP states. We only refine the reachable elements $\psi \in \varPsi$, that is, for which there exists a $t \in \psi$ and $s \in S$ such that $\langle s,t \rangle \in \tilde{Q}_\textsf{reach}^{\sigma ^\star }$. Using this guided strategy, we iteratively shrink only the relevant probability intervals, resulting in the same convergence behavior as the naive strategy but without the severe increase in abstraction size.

5 Computing Bounds on the Conditional Reachability

Theorem 3 provides bounds on the conditional reachability $W(\varOmega )$ in Problem 1, but computing these bounds involves optimizing over the subset of consistent schedulers. Recall from Def. 5 that a consistent scheduler chooses the same actions in different states.² As we are not aware of any efficient algorithm to optimize over the consistent schedulers, we compute the following straightforward bounds:

Lemma 2

(Bounds on Problem 1). Let $\mathcal {I}= \textsf{Abstract}(\mathcal {M}_{|\varOmega }, \varPsi )$ be the iMDP abstraction for the unfolded MDP $\mathcal {M}_{|\varOmega }$ and a time partition $\varPsi$. It holds that

$$\begin{aligned} W(\varOmega ) \le \max _{\sigma \in \textrm{Sched}_{\mathcal {I}}^\textsf{con}} \max _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\sigma ) \le \max _{\sigma \in \textrm{Sched}_{\mathcal {I}}} \max _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\sigma ). \end{aligned}$$

(18)

Moreover, any consistent scheduler $\hat{\sigma } \in \textrm{Sched}_\mathcal {I}^\textsf{cons}$ results in a lower bound.

Obtaining lower bounds While we can use any consistent scheduler in Lemma 2 to compute a lower bound on $W(\varOmega )$, we obtain better bounds by modifying a (potentially non-consistent) optimal scheduler $\sigma ^-$ under the worst-case choice of probabilities, i.e., $\sigma ^- = \mathop {\mathrm {arg\,max}}\limits _{\sigma \in \textrm{Sched}_{\mathcal {I}}} \min _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\sigma )$. We check for inconsistency of scheduler $\sigma ^-$ by evaluating the following condition in all pairs of states $\langle s,t \rangle , \langle s',t' \rangle \in \tilde{Q}_\textsf{reach}^{\sigma ^-} \subset \tilde{Q}$ reachable under $\sigma ^-$:

$$\begin{aligned} t = t' \implies \sigma (\langle s,t \rangle ) = \sigma (\langle s',t \rangle ) \quad \forall \langle s,t \rangle , \langle s',t' \rangle \in \tilde{Q}_\textsf{reach}^{\sigma ^-}. \end{aligned}$$

(19)

We remove inconsistencies by changing the action in one of the states to match the others. We take a greedy approach and always adapt to the action chosen most often across all iMDP states $\langle s,t \rangle \in \tilde{Q}$ for the same time t. For example, if $\sigma (\langle s,t \rangle ) = \sigma (\langle s',t \rangle ) \ne \sigma (\langle s'',t \rangle )$, then we only modify $\sigma (\langle s'',t \rangle )$ to match the other actions. Because the set $\tilde{Q}_\textsf{reach}^{\sigma ^-}$ is finite by construction, a finite number of modifications suffices to render any scheduler consistent. The experiments in Sect. 6 show that modifying an inconsistent scheduler yields tighter lower bounds than taking the maximum over many sampled consistent schedulers.

Obtaining upper bounds The set of consistent schedulers is finite but prohibitively large, so enumerating over all consistent schedulers is infeasible. For a sound upper bound, we instead optimize over all schedulers. The experiments in Sect. 6 show that we obtain (relatively) tight bounds. To further refine these upper bounds, the literature suggests another abstraction refinement loop, which can be formulated either directly on the imprecise evidence [21] or on the consistent schedulers [51]. The latter approach leverages the fact that consistent schedulers can also be modeled as searching for (memoryless) schedulers in partially observable MDPs, where the schedulers would only observe the time but not the state. Finally, the hardness of optimizing over consistent schedulers in the iMDP remains open: Classical NP-hardness results for the problems above do not carry over.

6 Numerical Experiments

We implemented our approach in a prototypical Python tool, which is available at https://doi.org/10.5281/zenodo.10438984. The tool builds on top of Storm [34] for the analysis of CTMCs and iMDPs. It takes as input a CTMC $\mathcal {C}$, a property defining the state-weight function $w$, and imprecisely timed evidence $\varOmega$. The tool constructs the abstract iMDP for the coarsest time partition, computing the probability intervals as per Eqs. (16) and (17). The bounds on the conditional reachability in Lemma 2 are computed using robust value iteration. Then, the tool applies guided refinements, as in Sect. 4, and starts a new iteration with the refined partition. After a predefined time limit, the tool returns the lower bound $\underline{W(\varOmega )}$ and upper bound $\overline{W(\varOmega )}$ on the conditional reachability $W(\varOmega )$:

$$\begin{aligned} \underline{W(\varOmega )} = \min _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\hat{\sigma }) \le W(\varOmega ) \le \max _{\sigma \in \textrm{Sched}_{\mathcal {I}}} \max _{\tilde{P} \in \mathcal {P}} W_\mathcal {I}(\tilde{P},\sigma ) = \overline{W(\varOmega )}, \end{aligned}$$

(20)

where the consistent scheduler $\hat{\sigma }$ for the lower bound is obtained by fixing all inconsistencies in the scheduler $\sigma ^-$ defined in Sect. 5. The tool can also compute minimal conditional reachabilities (by swapping all $\min$ and $\max$ operators).

Table 1. Overview of considered benchmarks.

Benchmarks We evaluate our approach on several CTMCs from the literature, creating multiple imprecisely timed evidence for each CTMC. Table 1 lists the evidence length (i.e., the number of observed times and labels), the number of CTMC states and transitions, and the property specifying the state-weight function. More details on the benchmarks are in [8, Appendix D.1], All experiments run on an Intel Core i5 with 8GB RAM, using a time limit of 10 minutes.

Fig. 6.

Results for different CTMCs and different imprecisely timed evidence. The blue lines are the upper bound $\overline{W(\varOmega )}$ (solid) and lower bound $\underline{W(\varOmega )}$ (dashed) on $W(\varOmega )$; red lines show the analogous lower bounds.

Feasibility of our approach We investigate if our approach yields tight bounds on the weighted reachability. Fig. 6 shows the results for each example with different imprecise evidences. The gray area shows the weighted reachabilities (as per Theorem 2) for 500 precisely timed instances $\rho \in \varOmega$ sampled from the imprecise evidence. Recall that the weighted reachability $W(\varOmega )$ is an upper bound to the weighted reachability for each precisely timed evidence $\rho \in \varOmega$. Thus, the upper bound of the gray areas in Fig. 6, indicated as ${W(\varOmega )}'$, is a lower bound of the actual (but unknown) value $W(\varOmega )$. The blue lines are the upper bound $\overline{W(\varOmega )}$ (solid) and lower bound $\underline{W(\varOmega )}$ (dashed) on $W(\varOmega )$ returned by our approach over the runtime (note the log-scale). Similarly, the red lines are the bounds obtained for minimizing the minimal weighted reachability.

Tightness of bounds Fig. 6 shows that we obtain reasonably tight bounds within a minute. In all examples, the lower bound converges close to the maximum of the samples. The improvement is steepest at the start, indicating that the bounds can be quickly improved by only a few refinement steps. In the long run, the improvement of the bounds diminishes, both because each refinement takes longer, and the improvement in each iteration gets smaller.

While not clearly visible in Figs. 6a, the lower bound $\underline{W(\varOmega )}$ (dashed blue line) slightly exceeds the maximal sampled value ${W(\varOmega )}'$ (gray area) in the end. Thus, the lower bound $\underline{W(\varOmega )}$ is closer to the actual weighted reachability $W(\varOmega )$ than the maximal lower bound obtained by sampling. We observed the same results when increasing the number of samples used to compute ${W(\varOmega )}'$ to $10\,000$.

Figs. 6b and 6c show the general benefit of conditioning on evidence. While evidence 1 for AHRS results in a state in which a system failure within the next 50 time units is very likely, a failure conditioned on evidence 2 is very unlikely.

Table 2. Results for all benchmarks (evidence length $|\varOmega |$ is given after the name).

Scalability We investigate the scalability of our approach. Table 2 provides the refinement statistics, bounds, model sizes, and runtimes for all benchmarks. The refinement statistics show the number of iterations (Iter.) and the total number of splits made in the partition. The bounds on $W(\varOmega )$ (which are the solid and dashed blue lines in Fig. 6) and the iMDP sizes are both given for the final iteration. For the timings, we provide the total time (over all iterations) and distinguish between the time spent on unfolding the model, i.e., constructing the iMDP, and analyzing it. Our approach terminates if after an iteration, the total run time so far exceeds the time limit of 10 minutes. The total runtime can, therefore, be significantly longer than 10 minutes.

CTMC size The size of the CTMC has a large impact on the total runtime. For example, for evidence with 4 labels, we can perform up to 27 iterations for Invent (3 CTMC states) but only 6-8 for Ahrs (74 CTMC states). For Polling (576 states) with evidence of length 2, performing 2 iterations takes nearly 50 minutes. The CTMC size affects the unfolding, which requires computing the transient probabilities from all states in one layer to all states in the next one. A clear example is Tandem-1 (120 CTMC states), where nearly all of the runtime is spent on the unfolding. A larger CTMC also leads to more transitions in the iMDP and thus, can increase the analysis time. An example is Polling-1 (576 CTMC states), where most of the runtime is spent in the analysis.

Length of evidence The time per refinement step increases with the length of the evidence. For example, for Invent-4 (with 15 labels), only 7 iterations are performed because the resulting iMDP has 15 layers, so the value iteration becomes the bottleneck (nearly 96% of the runtime for this example is spent on analyzing the iMDP). This is consistent with experiments on unfolded MDPs in [32, 39], where policy iteration-based methods lead to better results.

Caching improves performance To reduce runtimes, we implemented caching in our tool, which allows reusing transient probability computations. For example, if all labels in the evidence have a time interval of the same width (which is the case for Ahrs-1), transient probabilities are the same between layers of the unfolding. Table 1 shows that the unfolding times for Ahrs-1 are indeed lower than for, e.g., Ahrs-3, which has time intervals of different widths.

Likelihood of evidence The size of the iMDP is influenced by the number of CTMC states corresponding to the observed labels. Less likely observations can, therefore, mean that fewer CTMC states need to be considered in each layer. For example, the evidence in Ahrs-2 is 17 times less likely (probability of 0.01, with 569 states) than Ahrs-4 (probability of 0.17, with 4007 states), and as a result the total runtime of Ahrs-2 is less than for Ahrs-4.

7 Related work

Beyond the related work discussed in Sect. 1 on DTAs [2, 22, 23] and synthesis of timeouts [9, 15, 42], the following work is related to ours.

Imprecisely timed evidence can also be expressed via multiphase timed until formulas in continuous-time linear logic [28]. However, similar to DTA, conditioning and computing the maximal weighted reachability are not supported.

Conditional probabilities naturally appear in runtime monitoring [13, 49] and speech recognition [24], and is, e.g., studied for hidden Markov models [50] and MDPs [12, 39]. Approximate model checking of conditional continuous stochastic logic for CTMCs is studied in [25, 26] by means of a product construction formalized as CTMC, but their algorithm is incompatible with imprecise observation times. Conditional sampling in CTMCs is studied by [36], and maximum likelihood inference of paths in CTMCs by [45].

The abstraction of continuous stochastic models into iMDPs is well-studied [43]. Various papers develop abstractions of stochastic hybrid and dynamical systems into iMDPs [6, 7, 19] and relate to early work in [38]. Our abstraction in Sect. 4 is similar to a game-based abstraction, in which the (possibly infinite-state) model is abstracted into a two-player stochastic game [29, 30, 41]. In particular, iMDPs are a special case of a stochastic game in which the actions of the second player in each state only differ in transition probabilities [37, 44]. An interesting extension of our approach is to consider CTMCs with uncertain transition rates, which have recently also been studied extensively, e.g., in [5, 16,17,18, 20, 31].

8 Conclusion

We have presented the first method for computing reachability probabilities in CTMCs that are conditioned on evidence with imprecise observation times. The method combines an unfolding of the problem into an infinite MDP with an iterative abstraction into a finite iMDP. Our experiments have shown the applicability of our method across several benchmarks.

A natural next step is to embed our method in a predictive runtime monitoring framework, which introduces the challenge of running our algorithm in realtime. Another interesting extension is to consider uncertainty in the observed labels. Furthermore, this paper gives rise to four concrete challenges. First, finding better methods to overapproximate the union over MDP probabilities in Eq. (13) may lead to tighter bounds on the weighted reachability. Second, we want to optimize over the consistent schedulers only, potentially via techniques used in [3]. Third, we wish to explore better refinement strategies for the iMDP. The final challenge is to improve the computational performance of our implementation. One promising option to improve performance is to adapt symbolic policy iteration [9], which only considers small sets of candidate actions instead of all actions.