Farkas Certificates and Minimal Witnesses for Probabilistic Reachability Constraints

This paper introduces Farkas certificates for lower and upper bounds on minimal and maximal reachability probabilities in Markov decision processes (MDP), which we derive using an MDP-variant of Farkas’ Lemma. The set of all such certificates is shown to form a polytope whose points correspond to witnessing subsystems of the model and the property. Using this correspondence we can translate the problem of finding minimal witnesses to the problem of finding vertices with a maximal number of zeros. While computing such vertices is computationally hard in general, we derive new heuristics from our formulations that exhibit competitive performance compared to state-of-the-art techniques. As an argument that asymptotically better algorithms cannot be hoped for, we show that the decision version of finding minimal witnesses is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text {NP}}$$\end{document}-complete even for acyclic Markov chains.


Introduction
The goal of program verification is to consolidate the user's trust that a given system works as intended, and if this is not the case, to provide her with useful diagnostic information.Verification tools may, however, contain bugs and so a last grain of insecurity regarding their results always remains.A widely acknowledged approach to overcome this dilemma has been made in the form of certifying algorithms [17,63].These algorithms provide every result with an accompanying certificate, i.e., a token that can be used to verify the result independently and with little ressources.In this way, certificates enable the user (or a third party) to quickly give a mathematically rigorous proof for the correctness of the result irrespective of whether the algorithm itself works correctly.
Counterexamples, i.e. certificates for the violation of a property, can often be obtained as a byproduct of verification procedures.What constitutes a counterexample is highly context-dependent.Finite executions suffice as counterexamples for safety properties and single, possibly infinite, executions are viable counterexamples for LTL [28].Tree-like counterexamples have been considered for fragments of CTL [27].For a probabilistic system M and a linear time property φ, the most prominent notion of counterexample to Pr M (φ) < λ is a set of paths satisfying φ whose probability mass is at least λ (see [1] for a survey).
Another notion of counterexample for probabilistic systems M and properties of the form Pr M (φ) < λ are critical subsystems [1].We adopt the reverse perspective and call a subsystem M of M a witnessing subsystem for the property Pr M (φ) ≥ λ if Pr M (φ) ≥ λ.Small witnessing subsystems offer an insight into what parts of the system are responsible for the satisfaction of the property.Nonetheless, witnessing subsystems can hardly be regarded as viable certificates since verifying Pr M (φ) ≥ λ is as hard as checking Pr M (φ) ≥ λ itself.
In this paper we build a solid bridge between certificates and witnessing subsystems.The systems we consider are modeled as Markov decision processes (MDP), which contain an absorbing goal state representing a desirable outcome.This approach is motivated by the fact that numerous model checking tasks can be reduced to reachability problems [3,30,31,44,72,73].
Using Farkas' Lemma, we introduce certificates for bounds on the minimal and maximal probability to reach the goal state.We show that the set of these certificates forms a polytope and we provide a direct translation of a certificate to a witnessing subsystems for upper bounded threshold properties.Thereby, we bridge the gap between an abstract gadget, serving solely as a proof that the result is correct, and a concrete object, containing crucial diagnostic information about why the result holds.Moreover, our translation reduces the computation of minimal witnessing subsystems to a purely geometric problem, for which we provide and evaluate new exact and heuristic algorithms.
All omitted proofs can be found in the appendix. Contributions.
-Following the concept of certificates in certifying algorithms, we introduce Farkas certificates for reachability problems in MDPs (Table 1).-We give a uniform notion of witnessing subsystem (WS) for Pr max s0 (♦ goal) ≥ λ and Pr min s0 (♦ goal) ≥ λ (Definition 4.1).To the best of our knowledge, witnesses for Pr min s0 (♦ goal) ≥ λ have not been considered previously.-We establish NP-completeness for finding minimal WS even for acyclic discrete time Markov chains (DTMC) (Theorem 4.5).-Our main result establishes a strong connection between the polytopes of Farkas certificates for Pr min s0 (♦ goal) ≥ λ and Pr max s0 (♦ goal) ≥ λ and WS of the same property (Theorem 5.4).In particular, one can read off a minimal WS from a vertex of the polytope with a maximal number of zeros (Corollary 5.5).
-From our polytope characterizations we derive two algorithms for computing minimal WS: one based on vertex enumeration and one based on mixed integer linear programming (Section 6).We also introduce a linear programming based heuristic aimed at computing small WS.We evaluate our approach on DTMC and MDP benchmarks, where particularly our heuristics show competitive results compared to state-of-the-art techniques (Section 7).
Related work.The fundament of certifying algorithms has been surveyed in [63].The idea of "programs that check their work" has also been laid out in [17].In the context of model checking, the most prominent approach for the certification of a positive result has been to construct a proof of the property in the system [15,65,66].Rank-based certificates for the emptiness of a certain automaton [56] can be used to certify positive model checking results.Model checking MDPs in the presence of multiple objectives has been studied in [36,38].Heuristic approaches for computing small witnessing subsystems in DTMCs have been proposed in [5,7,48,50,51] and implemented in the tool Comics [49].Witnessing subsystems in MDPs have been considered in [6,9] and [19], which focuses on succinctly representing witnessing schedulers.The mixed integer linear programming (MILP) formulation of [76,77] allows for an exact computation of minimal witnessing subsystems for the property Pr max s0 (♦ goal) λ.NPcompleteness of computing minimal witnessing subsystems in MDPs was shown in [23], but the exact complexity has, to the best of our knowledge, not been determined for DTMCs (the problem was conjectured to be NP-complete in [76]).
Minimal probabilistic counterexamples given as sets of paths can be computed by reframing the problem as a k-shortest-path problem [42,43].Regular expressions have been considered to succinctly represent the set of paths in [32], and extensions were proposed in [18,75].The tool Dipro [4] computes probabilistic counterexamples, and a translation of these to fault trees was given in [55].Probabilistic counterexamples can be used to automatically guide iterative and refinement-based model checking techniques [22-24, 26, 46, 52].
Farkas' Lemma is a well-known source of certificates for the (in)feasibility of tasks in combinatorial optimization, operations research, and economics, as presented in the detailed historical account given in [69, pp. 209-226] as well as [61,Chapter 2] and [29,64,74].The lecture notes [70] contain a rich variety of applications of linear programming in general and Farkas' Lemma in particular.
A polyhedron is the intersection of finitely many halfspaces, and a polytope is a bounded polyhedron.A face of a polyhedron P is a subset F ⊆ P of the form F = {x ∈ P | a • x = max{a • y | y ∈ P }} for some a ∈ R n .A vertex of P is a face consisting of only one point.
Farkas' Lemma [37] is part of the fundament of polyhedra theory and linear programming.It provides a natural source of certificates showing the infeasibility of a given system of inequalites, or in other words, the emptiness of the polyhedron described by the system.We will use it in the following version.

Markov decision processes.
A Markov decision process (MDP) is a tuple M = (S, Act, ι, P), where S is a finite set of states, Act is a finite set of actions, ι is a probability distribution on S called the initial distribution of M , and P : S × Act ×S → [0, 1] is the transition probability function where we require s ∈S P(s, α, s ) ∈ {0, 1} for all s ∈ S and α ∈ Act.An action α is enabled in state s ∈ S if s ∈S P(s, α, s ) = 1.The set of enabled actions at state s are denoted by Act(s).A path in an MDP M is an infinite sequence s 0 α 0 s 1 α 1 ... such that P(s i , α i , s i+1 ) > 0 for all i ≥ 0. A finite path is a finite sequence π = s 0 α 0 s 1 α 1 ...s n with the same condition for all 0 ≤ i ≤ n − 1.In this case, we define last(π) = s n .Denote by Paths(M) and Paths fin (M) the set of infinite and finite paths in M.
A discrete-time Markov chain (DTMC) is an MDP with a single action, and this action is enabled at every state.If M is a DTMC, then Paths(M) carries a probability measure, where the associated σ-algebra is generated by the cylinder sets Cyl(τ ) = {π ∈ Paths(M) | π has prefix τ } of finite paths τ = s 0 s 1 ...s n in M with probability Pr(Cyl(τ )) = ι(s 0 ) • 0≤i<n P(s i , s i+1 ) (fore more details see [13,Section 10.1]).In the following we denote for a finite set X the set of probability distributions on X by Dist(X).Given µ ∈ Dist(X) let the support of µ be supp A deterministic scheduler is a function S : Paths fin (M) → Act such that S(π) ∈ Act(last(π)) and a randomized scheduler is a function S : Paths fin (M) → Dist(Act) such that supp(S(π)) ⊆ Act(last(π)) for all π ∈ Paths fin (M).Given a deterministic (or randomized) scheduler S, We denote by Pr S the probability measure on infinite S-paths (see [13,Definition 10.92 on page 843] for more details).If we replace ι with the distribution concentrated on state s, then we obtain a probability measure Pr S M,s or short Pr S s on infinite S-paths starting in s.The scheduler is memoryless if S(π) = S(last(π)) for all π ∈ Paths fin (M).We abbreviate memoryless deterministic schedulers as MD-schedulers and memoryless randomized schedulers as MR-schedulers.
The maximum and mininum is indeed attained even by an MD-scheduler [13].
Setting 2.2.Henceforth we will assume that M = (S all , Act, ι, P) has a unique initial state s 0 ∈ S and two distinguished absorbing states fail and goal ∈ S all , i.e., P(goal, α, s) = 0 for all α ∈ Act and s ∈ S all with s = goal, and likewise for fail.Here goal represents a desirable outcome of the modeled system and fail an outcome that is to be avoided.We use the notation S = S all \ {fail, goal}, we assume that every state s ∈ S is reachable from s 0 .We also assume that under every scheduler fail or goal is reachable from any state, i.e., Pr min s (♦(goal ∨ fail)) > 0 for all s ∈ S. If M does not satisfy this condition from the start, we can apply a standard preprocessing step, which is essentially given by taking the MEC quotient of M, see [2,3] and also [25].While it is often easier to verify the condition Pr min s (♦(goal ∨ fail)) > 0, it is in fact equivalent to Pr min s (♦(goal ∨ fail)) = 1 (see Lemma A.1 of the appendix).
Whenever suitable, we denote by M also the set of enabled state-action pairs, i.e., The vectors Pr min (♦ goal) and Pr max (♦ goal) can be characterized using the following linear programs.Although this characterization is well-known, we give a proof in the appendix due to slight differences with the standard literature.
Certificates for universally-quantified statements.In order to deal with the cases (1) and (3), we need the following lemma proved in the appendix.
For the direction from left to right, we take z to be Pr min (♦ goal).The opposite direction follows from Lemma 3.1.
The right hand sides of Corollary 3.2 provide certifying formulations for the threshold problems (1) and (3): to check whether the corresponding threshold statement holds, one must merely check whether z satisfies the inequalities, rather than checking whether Pr min / max s0 (♦ goal) was computed correctly.
Certificates for existentially-quantified statements.In order to find certificates for the remaining two cases (2) and (4), we calculate: For non-strict inequalities, we apply Farkas' Lemma in the opposite direction:

Minimal witnesses for reachability in MDPs
In this section we consider the following problem: Given an MDP M that satisfies the property Pr min M,s0 (♦ goal) ≥ λ (or Pr max M,s0 (♦ goal) ≥ λ), find a small subsystem M of M that still satisfies these thresholds.Such a subsystem is a witness to the satisfaction of the property in M. We first define subsystems and consider different measures of size which we show to be equivalent.Then we deal with the question of finding minimal witnessing subsystems.
Subsystems, witnesses and notions of minimality.Our definition of subsystem is essentially the same to the definition in [76,77]  Intuitively, a subsystem M of M contains a subset of states of M, and a transition of M originating in a state of M remains unchanged in M or is redirected to fail (instead of explicitely redirecting to fail, sub-stochastic distributions are used in [76,77] with the same effect).Definition 4.1 (Subsystem and witness).Let M = (S all , s 0 , Act, P) be an MDP as in Setting 2.2.A subsystem M ⊆ M is an MDP M = (S all , s 0 , Act, P ) with fail, goal ∈ S all ⊆ S all , Act M (s) = Act M (s) for all s ∈ S all , and for all s, t ∈ S all with t = fail and α ∈ Act we have P (s, α, t) > 0 =⇒ P (s, α, t) = P(s, α, t).
We say that the states S all \S all and the transitions (s, α, t) with P(s, α, t) > 0 and P (s, α, t) = 0 have been deleted in M .A witness for Pr Remark 4.2.The condition Act M (s) = Act M (s) ensures that the probability of a deleted transition (s, α, t) is added to (s, α, fail).This is essential for witnesses for Pr min M,s0 (♦ goal) λ as one could otherwise remove entire actions causing low probabilities and obtain greater Pr min in M than in M as a result.For witnesses of Pr max M ,s0 (♦ goal) λ one could delete this condition, thus leading to the notion of [76,77].Example 4.3.Figure 1a depicts an MDP and Figure 1b indicates the subsystem that is obtained by deleting the state t and additionally the transition (u, α, s 0 ).
The following lemma ensures that we can use the subsystems as witnesses for both Pr max M,s0 (♦ goal) λ and Pr min M,s0 (♦ goal) λ.Lemma 4.4.Let M be an MDP as in Setting 2.2 and M ⊆ M. Then: We consider the following notions of minimality for subsystems: (1) State-minimality: |S all | is minimal.
Depending on the situation, one notion might be more suitable than the others.However, in Lemma C.1 of the appendix we show that finding transition-minimal (respectively, size-minimal) witnesses can be reduced to finding state-minimal witnesses with a linear (respectively, quadratic) blow-up.We will therefore restrict ourselves to state-minimality for the rest of this paper.
NP-completeness of finding minimal witnesses for DTMCs.In this section we determine the computational complexity of the witness problem: Given a DTMC M, a positive integer k, and a rational number λ ∈ [0, 1], decide whether there exists a witness M ⊆ M for Pr M,s0 (♦ goal) ≥ λ with at most k states.The corresponding problem for MDPs is known to be NPcomplete [23,77] 1 .In this section we show that the witness problem is even NP-complete for acyclic DTMCs, where acyclicity means that the underlying graph with V = S and E = {(s, t) ∈ S × S | P(s, t) > 0} is acyclic (as before, we take S = S all \{goal, fail}).This answers a conjecture of [76] in the affirmative and also shows NP-completeness of finding minimal witnesses for Pr min M,s0 (♦ goal) ≥ λ.Theorem 4.5.The witness problem is NP-complete for acyclic DTMCs.
Proof (Sketch).An NP-algorithm for the witness problem is given by guessing a set of states of size k and verifying in polynomial time that the corresponding subsystem satisfies Pr M ,s0 (♦ goal) ≥ λ.
For hardness, we give a reduction from the clique problem, which is among Karp's 21 NP-complete problems [53].The idea is the following: Given an instance of the clique problem with graph G = (V, E) and integer k, construct an acyclic Markov chain M with states S = {s 0 } ∪ V ∪ E ∪ {goal, fail} and edges from each vertex v ∈ V to all edges to which it is incident.Then the existence of a k-clique can be reduced to the existence of a "saturated" subsystem in M with k states in V .To check whether the subsystem is saturated, we require it to have more probability than a certain threshold, which depends on k and |V |.Details can be found in the appendix.
Remark 4.6.NP-completeness of transition-minimal and size-minimal versions of the witness problem for acyclic DTMCs follows along the same lines, where only the sizes and thresholds for the subsystems need to be adapted.However, DTMCs whose underlying graph is a tree permit an efficient algorithm for computing minimal witnesses (for the proof see Proposition F.8).
Proposition 4.7.Minimal witnesses in tree-shaped DTMCs can be computed in polynomial time.
Proof (Sketch).The algorithm first transforms the DTMC at hand into a binary (tree-shaped) DTMC, and then works bottom up by storing for each state the highest probability that can be obtained with a subsystem of size k, for all k up to the size of the subtree.

Relating Farkas certificates and minimal witnesses
In this section we establish a strong connection between Farkas certificates on the one hand and witnesses for probabilistic reachability constraints on the other hand.We first note that the set of Farkas certificates for non-strict lower bounds forms a polytope, i.e., a bounded polyhedron.

Lemma 5.1 (Polytopes of Farkas certificates
).Let M = (S all , s 0 , Act, P) be an MDP as in Setting 2.2 and consider A ∈ R M×S and b ∈ R S introduced there.Then for every λ ∈ [0, 1] the polyhedra are both polytopes, called the polytopes of Farkas certificates.
Remark 5.2.For any vector v ∈ R n the support is defined as supp(v) = {i ∈ {1, ..., n} | v i > 0}, and we use analogous notation for vectors in the vector spaces R S and R M .We will henceforth establish a connection between subsystems of M and points in P min (λ) given by taking the support.Thus we may safely restrict our attention to the subpolytope P min ≥0 (λ) = P min (λ) ∩ R S ≥0 .
For our main result below we need some notation.
Notation 5.3.Given an MDP M = (S all , s 0 , Act, P) as in Setting 2.2 and a subset R ⊆ M, where M also denotes the state-action pairs (compare Section 2).We let M R = (S all , s 0 , Act, P ) be the subsystem where, roughly speaking, the state-action pairs in R remain.More precisely, let Theorem 5.4 (Farkas certificates yield witnesses).Let M be an MDP as in Setting 2.2 and λ ∈ [0, 1].Then for a set R ⊆ S the following statements are equivalent: (1) The subsystem M R is a witness for Pr min M,s0 (♦ goal) ≥ λ. (2) There is a point p in P min ≥0 (λ) such that supp(p) ⊆ R.
(3) There is a vertex v of P min ≥0 (λ) such that supp(v) ⊆ R.Moreover, for a set R ⊆ M the following statements are equivalent: (a) The subsystem M R is a witness for Pr max M,s0 (♦ goal) ≥ λ.(b) There is a point p in P max (λ) such that supp(p) ⊆ R. (c) There is a vertex v of P max (λ) such that supp(v) ⊆ R.
Corollary 5.5 (Detecting MWs by vertices of P).Let M = (S all , s 0 , Act, P) be an MDP as in Setting 2.2 and λ ∈ [0, 1].Then a vertex v of P min ≥0 (λ) has a maximal number of zeros among all vertices of P min ≥0 (λ) if and only if M supp(v) is a minimal witness for Pr min s0 (♦ goal) ≥ λ.Dually, a vertex v of P max (λ) has a maximal number of zeros among all vertices of P max (λ) if and only if all of the following hold: (1) M supp(v) = (S all , s 0 , Act, P ) is a minimal witness for Pr max s0 (♦ goal) ≥ λ, (2) for every s ∈ S there is precisely one α ∈ Act(s) with (s, α) ∈ supp(v), (3) the corresponding map S : S → Act is an MD-scheduler on M supp(v) with Pr S s0 (♦ goal) ≥ λ.

Computing witnessing subsystems
In this section we use the results of Section 5 to derive two algorithms for the computation of minimal witnesses for reachability constraints in MDPs.As the problem is NP-hard, we also present a heuristic approach aimed at computing small witnessing subsystems.
Vertex enumeration.Corollary 5.5 gives rise to the following approach of computing minimal witnessing subsystems: enumerate all vertices in the corresponding polytope and choose one with a maximal amount of zeros.Vertex enumeration of polytopes has been studied extensively [11,12,14,20,21,34,35,39,40,62,67] and has been shown to be computationally hard [54, Corollary 2].
First experiments that we have conducted with the SageMath2 toolkit which supports vertex enumeration have not scaled well in the dimension, which in our case is the number of states in the original system.Also, we found no tool support for vertex enumeration that is able to handle sparse matrices, which is essential for bigger benchmarks.
Mixed integer linear programming.An approach that computes minimal witnesses to the threshold problem Pr max s0 (♦ goal) ≥ λ using mixed integer linear programs (MILP) was presented in [76,77].Using the following lemma, we can derive MILP formulations from our polytope formulations.Lemma 6.1.Let P = {x | Ax ≤ b, x ≥ 0} ⊆ R n be a polytope and K ≥ 0 be such that for all p ∈ P and Then a vector (σ, x) is an optimal solution of this MILP if and only if x is a point in P with a maximal number of zeros.
For P min ≥0 (λ) we can use Lemma 3.1 to derive that K = 1 is a viable bound.By invoking again Corollary 5.5, this means that a solution (z, σ) of the MILP encodes a minimal witnessing subsystem in the integral variables σ.This MILP was used in [76,77] for the computation of minimal witnessing subsystems of DTMCs .
An upper bound K as in Lemma 6.1 for P max (λ) can be found in polynomial time by taking the objective value of an optimal solution to the LP max where σ(s, α) are binary integer variables.It was implemented in the tool ltlsubsys.The idea is to directly encode a scheduler in the set of equations Az ≤ b using σ.In [76,77] a number of additional redundant constraints are given to guide the search.In contrast to [76,77] we do not need to handle so-called problematic states, as our precondition Pr min s (♦(goal ∨ fail)) > 0 guarantees that no such states exist.k-step quotient sum (QS k ) heuristics.Approximating the maximal number of zeros in a polytope is computationally hard in general [8].We now derive a heuristic approach for this problem called quotient sum heuristic which is based on iteratively solving LPs over the polytope, where the objective function for each iteration depends on an optimal solution of the previous LP.More precisely, we take o 1 = (1, . . ., 1) and take an optimal solution QS 1 of the LP min o 1 • y s.t.y ∈ P max (λ).Many entries in QS 1 may be small, but still greater than zero.In order to push as many of the small values of QS 1 to zero, we define a new objective function by where C is a value that is greater than any value 1/ QS 1 (i).We now take a solution QS 2 of new LP min o 2 • y = 1≤i≤n y(i) QS 1 (i) s.t.y ∈ P max (λ) and form the next objective function o 3 as in (6.3).Inductively this generates a sequence of objective functions (o k ) k≥1 and corresponding optimal solutions (QS k ) k≥1 in P max / min (λ).By Theorem 5.4 we can construct a witnessing subsystem with as many states as the number of non-zero entries in QS k .

Experiments
In this section we evaluate our MILP formulations and heuristics on a number of DTMC and MDP benchmarks from the Prism benchmark-suite [57,58].We compare our results with the tool Comics [49], which implements heuristic approaches to compute small subsystems for DTMCs.It has two modes: the local search extends a given subsystem by short paths that carry much probability, whereas the global search searches for the next most probable path from the initial state to goal, and adds it to the subsystem.Both approaches iteratively extend a subsystem until it carries more probability than the given threshold and thus have to compute the probability of the subsystem at each iteration.
All computations were performed on a computer with two Intel E5-2680 8 cores at 2.70 GHz running Linux, with a time bound of 30 minutes, a memory bound of 100 GB and with each benchmark instance having access to 4 cores.For the LP and MILP instances we use the Gurobi solver, version 8.1.1 [41].The recorded times of our computations include the construction of the LPs/MILPs and are wall clock times.Pre-processing steps, such as collapsing states that cannot reach goal, are not counted in the time consumption.For Comics, we use the time that is reported as counterexample generation time by the tool.
To validate our implementation, we used Prism to verify that the subsystems that we compute indeed satisfy the probability thresholds.We noticed that for a few instances (< 0.5%) Prism reported a deviation of less than 10 −8 , which can be explained by the fact that both Prism and the solvers that we use rely on floating-point arithmetic, which is approximate by nature.
Our implementation, together with the models we use and benchmark results can be found at https://github.com/simonjantsch/farkas.DTMC benchmarks.As Pr max and Pr min coincide on DTMCs, we can use the heuristics and exact computations derived from either the P max or the P min ≥0 polytope for DTMCs.We use two DTMC benchmarks: a model of the crowds-N -K protocol [68,71] for ensuring anonymous web browsing (with N members and K protocol runs) and a model of the bounded retransmission protocol [33,45] for file transfers (where brp-N -K is the instance with N chunks and K retransmissions).Fig. 3: Comparison of heuristic methods on DTMC benchmarks.
Figure 2 shows the effect of increasing the number of iterations of the QSheuristic for the model crowds-2-8.While the first iteration (taking QS 2 instead of QS 1 ) has an impact on the number of states, more iterations do not improve the result significantly.For QS 1 , the sizes of subsystems increase monotonically with growing λ.Starting with QS 2 the results may, interestingly, have "spikes": increasing λ can lead to smaller subsystems.
Figure 3 shows the results of the QS 2 -heuristic compared to the two modes of Comics for λ that ranges between 0 and the actual reachability probability of the model.A general observation is that the runtime of the QS-heuristic is independent of λ, whereas both modes of Comics use significantly more time with increasing λ.Also, especially for crowds-5-8, one can see that relatively small subsystems are indeed possible even for λ that are close to the actual probability.The exact computations via MILPs hit the timeout for almost all instances in these two models.
In both cases one of our formulations leads to a heuristic that gives monotonically growing subsystems and outperforms both modes in Comics.While QS 2 applied to P min performs better on crowds-5-8 (Figure 3a), it is the other way around on brp-512-2 (Figure 3b).MDP benchmarks.We consider two MDP models: the randomized consensus-N -K protocol of [10,59] (with N processes and a bound K on the random walk) and the CSMA-N -K protocol for data channels [60] (where N is the number of stations, and K is the maximal backoff count).The results of both heuristic and exact computations can be seen in Figure 4 and Figure 5. Whereas the heuristics all needed less than 5 minutes, all MILP instances ran into the timeout except for the ones in Figure 4a.Whenever a MILP instance could not be solved optimally in 30 minutes, we plot both the found upper and lower bound, with the region in between shaded.
The comparison between the MILP formulation that we derived from P max (λ) and the one presented in [76,77] (labeled by ltlsubsys, see also Section 6) showed that both compute comparable upper and lower bounds within the given time (Figure 4b and Figure 5b).In all instances apart from Figure 4b the corresponding QS 2 heuristics performs well and generates subsystems that are as good, or better, than the best upper bounds computed by the MILPs in 30 minutes.As expected, the witnessing subsystems for Pr min s0 (♦ goal) ≥ λ tend to the entire state space as λ tends to the actual value Pr min s0 (♦ goal) (which is 1 in these two models).However, subsystems for Pr max s0 (♦ goal) ≥ λ may be substantially smaller even for large λ.

Conclusion
In this paper we brought together two a priori unrelated notions in the context of probabilistic reachability constraints: on the one hand Farkas certificates, which are vectors satisfying certain linear inequalities that we derive using MDP-specific variants of Farkas' Lemma, and on the other hand witnessing subsystems, which provide insight as to which parts of the system are essential for the satisfaction of the property at hand.This connection reduces the computation of minimal (respectively, small) witnessing subsystems to finding a Farkas certificate with a maximal (respectively, large) number of zeros.Furthermore, it leads to a unified notion of witnessing subsystem for Pr max s0 (♦ goal) ≥ λ and Pr min s0 (♦ goal) ≥ λ.We showed that the decision version of computing minimal witnessing subsystems is NP-complete even for acyclic DTMCs and introduced heuristics for the computation of small witnesses based on Farkas certificates.Experiments of the heuristics exhibited competitive results compared to the approach implemented in Comics and showed that they scale well with the system size and threshold.As expected, computing minimal subsystems using the derived MILP formulations consumed significantly more time than the heuristics and often triggered timeouts.The computations for Pr min s0 (♦ goal) scaled a bit better than the ones for Pr max s0 (♦ goal) (see Figure 4).The upper and lower bounds that were computed in the given time by the new MILP formulation for Pr max s0 (♦ goal) ≥ λ were comparable to known techniques (see Figure 4b and Figure 5b).
As the QS k heuristic may get stuck in local optima, we would like to investigate whether other initial objective functions or updates could lead to better heuristics.Concerning computations of minimal witnessing subsystems, exploring how vertex enumeration techniques could be adapted to the MDP-specific form of the Farkas polytopes is an intersting line of future work.for every t ∈ T , we get z ≤ Qz and by induction z ≤ Q n z .By Lemma A.1 almost every path in M S reaches fail or goal.But Q n (s, t) is the probability to reach t from s in exactly n steps.Therefore we must have Q n → 0 as n → ∞, and in particular z(s ) = z (s ) = 0.
So far, we have argued that the LP (A.1) has a solution, say z * , and that z * (s) = 0 for all s ∈ S with Pr min M,s (♦ goal) = 0.It is easy to see that for all states we must have since we could otherwise increase z * (s) without changing z * elsewhere and obtain a better solution.From [13,Theorem 10.109], it now follows that we must have z * = Pr min (♦ goal).
The argument for the LP describing Pr max (♦ goal) is completely analogous.

B Proofs of Section 3
Lemma 3.1.For the matrix A ∈ R M×S and vector b ∈ R M as in Setting 2.2, we have for all z ∈ R S that Proof.This is a simple consequence of Proposition 2. Thus we get δ • z > δ • Pr min (♦ goal) in contradiction to Proposition 2.3.

Remark B.1 (y-vectors, schedulers and frequencies).
In the equivalences used to derive Proposition 3.3, we could replace z ∈ R S ≥0 with z ∈ R S , leading to the equivalent formulation which is by a variant of Farkas' Lemma (see [69,Corollary 7.1d on p. 89]) equivalent to This shows in total the equivalence and the analogous equivalence with strict inequalities on λ follow similarly using a third version of Farkas' Lemma [69, Corollary 7.1e on p. 89] In Lemma B.2 below we give a hands-on proof for this last equivalence which also provides the following interpretation of a non-negative vector y with yA = δ s0 : Write y = (y s0,α0 , . . ., y s0,αm , . . ., y sn,αm ) and define the MR-scheduler S : S → Dist(Act) by setting for α ∈ Act Lemma B.2.For the matrix A ∈ R M×S and vector b ∈ R M as in Setting 2.2, we have We define an MR-scheduler S of M by S(s, α) = y(s, α) β∈Act(s) y(s, β) for those states s ∈ S for which the denominator is positive.In case that y(s, β) = 0 for all β ∈ Act(s), then we take an arbitrary S(s, •) ∈ Dist(Act(s)).
Let Q be the transition matrix of the DTMC M S induced by S (restricted to S), that is,

Since Pr S
s (♦(goal ∨ fail))) = 1 one sees as in the proof of Proposition 2.3 that Q n → 0 as n → ∞.By invoking the Jordan normal form of Q one deduces from this that Q cannot have (complex) eigenvalues of absolute value greater than or equal to 1.In turn, this implies that the series n≥0 Q n converges and that the limit is the inverse of I − Q.
Let h be the expected frequencies of M S under initial distribution δ s0 , i.e., the solution of where I ∈ R S×S is the identity matrix.Now let us define Then it follows that for all s ∈ S h(s) = We now show that y satisfies y A = δ s0 .By the definition of h, we have By applying (B.2) on the left-hand side and (B.1) on the right, we get which in total is precisely the desired equation of y A = δ s0 .Finally we show that y b ≥ yb by proving the stronger statement that y ≥ y.We first claim for g(s This is verified by the computation which is precisely the assumption yA ≤ δ s0 .Now since g(I − Q) ≤ δ s0 = h(I − Q), we have after multiplying on the right with ( Because of (B.3), this is equivalent to y(s, α) ≤ y (s, α) for all (s, α) ∈ M with g(s) > 0. For those states with g(s) = 0, there is nothing to prove for y ≥ y. and that the probabilities of the paths in these sets is the same in M and M .As minimal and maximal reachability probabilities are attained by an MR-scheduler, we get as a consequence that Pr min M ,s0 (♦ goal) ≤ Pr min M,s0 (♦ goal).For the Pr max M ,s0 (♦ goal) ≤ Pr max M,s0 (♦ goal) one notices that, vice versa, every MR-scheduler S on M can be extended (arbitrarily) to an MR-scheduler S on M and that they satisfy Pr S M ,s0 (♦ goal) ≤ Pr S M,s0 (♦ goal).The rest of the argument is identical.

C Proofs of Section 4
In the following lemma the size of an MDP refers to sum of the number of states and the number of transitions, i.e., triples (s, α, t) with P(s, α, t) > 0.
Lemma C.1 (Reduction to state-minimality).Let M = (S all , s 0 , Act, P) be an MDP as in Setting 2.2.Then there exists an MDP N = (S all , s 0 , Act, P ) such that the transition-minimal (respectively, size-minimal) witnesses of M are in one-to-one correspondence with the state-minimal witnesses of N .The size of N is linear (respectively, quadratic) in the size of M.
Proof.Throughout this proof, we let T denote the transitions of M, i.e., the set of triples with P(s, α, t) > 0. For the reduction from size-minimality to state-minimality, let N be the MDP with states S all = S all T and transitions s α −→ (s, α, t) with probability P(s, α, t) Then there is a bijection between paths in M and paths in N given by s 0 α 0 s 1 α 1 s 2 ... corresponds to s 0 α 0 (s 0 , α 0 , s 1 )α 0 s 1 α 1 (s 1 , α 1 , s 2 )α 1 s 2 ... and this bijection preserves probabilities.This immediately implies that a subsystem of M obtained by deleting states S d and transitions T d is a (minimal) witness if and only if the subsystem of N obtained by deleting states the S d ∪ T d is a (minimal) witness.Clearly, the reduction is linear in size.This finishes the reduction from size-minimality to state-minimality.
For the reduction from transition-minimality, let N be the MDP with states S all = T ∪ {s 0 , goal, fail} and transitions s 0 α −→ (s 0 , α, t) with probability P(s 0 , α, t) Then there is again a probability-preserving bijection between paths in M and paths in N .The rest of the argument is completely analogous.In this case, however, the size of N is quadratic in the size of M since there are O(|T | 2 ) many transitions of the second type in the above list.
Proof.Assume that we are given an instance of the clique problem, i.e., a finite undirected graph G = (V, E) and an integer k ≥ 3 (the cases k < 3 are trivial).Let n = |V |.Consider the DTMC M with states S = {s 0 } ∪ V ∪ E ∪ {fail, goal} and four types of edges, see also Figure 6: s 0 → v for every v ∈ V with probability 1/n; v → {v, w} for every v ∈ V and edge {v, w} ∈ E with probability 1/n; -{v, w} → goal for edge {v, w} ∈ E with probability 1; s → fail for all s ∈ S with remaining probability outgoing from s (provided there is some).
Fig. 6: The acyclic DTMC M reducing the clique problem to the witness problem.The state fail and edges to it are not depicted for simplicity.
We claim that the graph G has a clique with at least k vertices if and only if the MDP M has a witness for Pr M,s0 (♦ goal)

2
. We intend to show that V is a k-clique.Since Pr M ,s0 (♦ goal) ≥ k(k−1) n 2 , we have at least k(k − 1) transitions between states in V and E .Since each state in E has exactly two incoming transitions, we have k(k − 1) ≤ 2b, and therefore a ≤ k.At most a(a−1) 2 states of E have two incoming transitions from V , the others have at most one incoming transition from V .Thus the total number of transitions from where the last step follows from a ≤ k and k ≥ 3.But by assumption, the total number of transitions from V to E is at least k(k − 1), and hence we have everywhere equality in the above computation.This can only happen if 2 and every edge in E has two incoming transitions from V .This forces V to be a k-clique with edge set E .

D Proofs of Section 5
Lemma 5.1 (Polytopes of Farkas certificates).Let M = (S all , s 0 , Act, P) be an MDP as in Setting 2.2 and consider A ∈ R M×S and b ∈ R S introduced there.Then for every λ ∈ [0, 1] the polyhedra are both polytopes, called the polytopes of Farkas certificates.
Proof.This proof has some resemblence with the proof Proposition 2.3.If P min (λ) was unbounded, then there exists z 0 , and hence everywhere equality.This implies that z 1 (t) = 0 for all successors of s 0 and by induction z 1 (t) = 0 for all reachable states t.By assumption all states are reachable from s 0 and hence z 1 = 0, a contradiction.Now fix some arbitrary s max ∈ S max .Then again because of Az 1 ≤ 0 we have for all α ∈ Act(s) and hence everywhere equality.But then t∈S P(s max , α, t) = 1, meaning that there is no transition from s max to fail or goal, and z 1 (t) = m, meaning that the same applies to all successors of s max .This implies by induction that goal and fail are not reachable from s max , a contradiction to the assumption Pr min smax (♦(goal ∨ fail)) > 0.
The argument for P max (λ) is similar: Assume that this polyhedron was unbounded, i.e., there exist y 0 , y 1 ∈ R M with y 1 = 0 such that y 0 + ty 1 ∈ P max (λ) for all t ∈ [0, ∞).Then necessarily y 1 ≥ 0 and y 1 A ≤ 0, and thus also y 1 A(1, ..., 1) ≤ 0. On the other hand, A(1, ..., 1) ≥ 0 and therefore y 1 A(1, ..., 1) ≥ 0. So y 1 A(1, ..., 1) = 0 which means y 1 A = 0. Thus, if y 1 (s, α) > 0, then This implies (D.1) since the right-hand side in both inequalities agree.Hence p ∈ P min ≥0 (λ).( 2) =⇒ (1): Take a point p ∈ P min ≥0 (λ) with supp(p) ⊆ R. Since we may assume that supp(p) = R. Since p(s 0 ) ≥ λ, it suffices to prove which we achieve by invoking Lemma 3.1.We may assume that every state in R is reachable from s 0 in M R , otherwise we restrict to those states.Notice also that for all s ∈ R we have  3): This is a general observation from polytope theory: Take a point As the inequalities x(s, α) ≥ 0 are also part of the description of P and p ∈ P ∩H, the set P ∩ H is a face of P.This face has a vertex v ∈ P ∩ H. Then v is also a vertex of P and we have supp(v) ⊆ supp(p) ⊆ R.
(a) =⇒ (b): Let M R = (S all , s 0 , Act, P ) and denote S = S all \ {goal, fail}.If M R is a witness for Pr max M,s0 (♦ goal) ≥ λ, then there exists an MD-scheduler S : S all → Act such that Pr S M R ,s0 (♦ goal) ≥ λ.Let Q ∈ R S ×S be the transition matrix of M S R , and likewise let c ∈ R S contain the probabilities to go from s to goal in one step in M S R .Recall that for all s ∈ R we have Pr min M R ,s (♦(goal ∨ fail)) = Pr min M,s (♦(goal ∨ fail)) > 0. With the same argument as in the proof of Proposition 2.3 one sees that Q n → 0 as n → ∞.By invoking the Jordan normal form of Q one deduces from this that Q cannot have (complex) eigenvalues of absolute value greater than or equal to 1.In turn, this implies that the series n≥0 Q n converges.
is the probability that a path of length n starting in s 0 ends in s.From this it is easy to see that qc = Pr S M R ,s0 (♦ goal).Now let p ∈ R M be the vector with p(s, α) = q(s) if s ∈ S and S(s) = α, and 0 otherwise.Then clearly supp(p) ⊆ R and we claim that p ∈ P max (λ).Obviously, we have p ≥ 0 and also pb = qc = Pr S M R ,s0 (♦ goal) ≥ λ.Furthermore, This implies pA ≤ δ s0 since for the states s ∈ S \ S there is nothing to show in this inequality.
(b) =⇒ (a): Let p be an element of P max (λ), let U = supp(p) ⊆ R. We will show that M U = (S all , s 0 , Act, P ) is a witness for Pr max M,s0 (♦ goal) ≥ λ, which immediately implies the same for M R .Let A and b be as in Setting 2.2 for M U , and denote S = S all \ {goal, fail}.Let p be the projection of We consider the DTMC M S U and we will show that the probability to reach goal in this DTMC is greater than λ in order to show that M U is a witness.As before denote the transition matrix of this DTMC Q ∈ R S ×S and collect the probabilities to go from a state to goal in one step in c ∈ R S .As in the proof of (a) =⇒ (b) we see that the matrix series n≥0 Q n converges.Also where C ∈ R S ×M U has entries If we let J ∈ R M U ×S be the matrix with entries J((s, α), t) = δ st , then we have the relation A = J − P .Notice also that C • J = I S .This implies with (D.4) that and therefore we have the following telescope sum Putting everything together allows us to calculate (b) ⇐⇒ (c): This is the same argument as for (2) ⇐⇒ (3).
Corollary 5.5 (Detecting MWs by vertices of P).Let M = (S all , s 0 , Act, P) be an MDP as in Setting 2.2 and λ ∈ [0, 1].Then a vertex v of P min ≥0 (λ) has a maximal number of zeros among all vertices of P min ≥0 (λ) if and only if M supp(v) is a minimal witness for Pr min s0 (♦ goal) ≥ λ.Dually, a vertex v of P max (λ) has a maximal number of zeros among all vertices of P max (λ) if and only if all of the following hold: (1) M supp(v) = (S all , s 0 , Act, P ) is a minimal witness for Pr max s0 (♦ goal) ≥ λ, (2) for every s ∈ S there is precisely one α ∈ Act(s) with (s, α) ∈ supp(v), (3) the corresponding map S : S → Act is an MD-scheduler on M supp(v) with Pr S s0 (♦ goal) ≥ λ.
We begin with the 'if' part, and assume that there exists a vertex w of P with a strictly larger number of zeros than v. Since there is for every s ∈ S = supp S (v) only one pair (s, α) in supp(v), this implies that | supp S (w)| < | supp S (v)|.But by Theorem 5.4, (c) =⇒ (a), the subsystem M supp(w) is also a witness, and it would contain a strictly smaller number of states than M supp(v) .Contradiction to the minimality of M supp(v) .
For the 'only if' part, let v be a vertex of P with a maximal number of zeros.Again by Theorem 5.4, (c) =⇒ (a), M supp(v) is a witness.If it was not minimal, then there is a set R ⊆ M such that M R is a witness with a strictly smaller number of states than M supp(v) .Now the proof of Theorem 5.4, (a) =⇒ (b) =⇒ (c) provides a vertex w of P with supp(w) ⊆ R and for every s ∈ supp(w) there is precisely one α ∈ Act such that (s, α) ∈ supp(w).This implies which is a contradiction.Hence M supp(v) is a minimal witness.
Almost the same argument shows that for every s ∈ S there is precisely one α ∈ Act: Otherwise one can again invoke the proof of Theorem 5.4, (a) =⇒ (b) =⇒ (c) applied to M supp(v) and an MD-scheduler S attaining Pr max M supp(v) ,s0 (♦ goal) ≥ λ in order to obtain a vertex of P with a greater number of zeros than v.
Finally, the inequality Pr S M supp(v) ,s0 (♦ goal) ≥ vb ≥ λ follows with the same arguments as in the proof of Theorem 5.4, (b) =⇒ (a), so S is indeed a witnessing scheduler for Pr max s0 (♦ goal) ≥ λ.

E Proofs for Section 6
Lemma 6.1.Let P = {x | Ax ≤ b, x ≥ 0} ⊆ R n be a polytope and K ≥ 0 be such that for all p ∈ P and 1 ≤ i ≤ n we have p(i) ≤ K. Consider the MILP Then a vector (σ, x) is an optimal solution of this MILP if and only if x is a point in P with a maximal number of zeros.
Proof.Suppose that the conditions are satisfied for P = {x | Ax ≤ b, x ≥ 0} ⊆ R n and K ≥ 0.
We first show that for any optimal solution (σ, x) of the MILP, we have x(i) = 0 if and only if σ(i) = 0.If x(i) = 0 and σ(i) = 1, we would get a better solution by setting σ(i) = 0, contradicting the fact that (σ, x) is optimal.If σ(i) = 0, then x(i) must be zero as x(i) ≤ K • σ(i).We write σ(x) for the vector that has a 1 at every position where x is greater than 0, and 0 otherwise.
For every point p ∈ P, (σ(p), p) satisfies the constraints of the MILP.To see that p ≤ K • σ(p), we observe that for all i : K • σ(p)(i) ≤ K, and K was chosen exactly such that for all i : x(i) ≤ K for all x ∈ P. Now we can show the claim: " =⇒ ": Let (σ, x) be an optimal solution of the MILP.Clearly, x ∈ P. By the above argument, σ = σ(x).Suppose that there is another point p in P with more zeros.But then (σ(p), p) is a solution of the MILP and 1 • σ(p) < 1 • σ(x).Hence (σ, x) is not an optimal solution to the MILP, contradicting the assumption.
"⇐=": Let x be a point in P with a maximal number of zeros.Then (σ(x), x) satisfies the constraints of the MILP.Furthermore, it is an optimal solution, as a better solution would contradict the maximality of the number of zeros in x.

F Polynomial algorithm in the tree-shaped case
In this section we show that a minimal witness for the property Pr M (♦ goal) ≥ λ can be computed in polynomial time for tree-shaped DTMCs, given that it exists.Here tree-shaped refers to the property that the underlying graph of M excluding goal and fail, i.e. the graph with vertices V = S and edges E = {(s, t) ∈ S × S | P(s, t) > 0}, is a tree.The algorithm has two steps: first we reduce a tree-shaped DTMC to the special case of a binary tree-shaped DTMC.Then, we provide an algorithm for the binary case whose result can be translated back to the general case.
We consider, as in Setting 2.2, DTMCs with distinguished, absorbing states goal and fail.The predicates acyclic and binary refer to the underlying graph the DTMC excluding goal and fail.
Binarization of Markov chains.We first give a transformation of a Markov chain into a binary Markov chain that preserves the probability to reach goal.Definition F.1 (Binarization).Let M = (S all , s 0 , P) be a DTMC as in Setting 2.2 and < a total order on S such that goal is its minimal element.For every state q ∈ S, let Post(q) = {s ∈ S all | P(q, s) > 0}.We define B(M, <) = (S all , s 0 , P ) by giving a local transformation that is applied to all states q ∈ S all with | Post(q)| > 2. States with less than three successors stay as they are.
Let q ∈ S all , Post(q) = {s 0 , s 1 , . . ., s n } be ordered according to < and define µ i = P(q, s i ) for 0 ≤ i ≤ n.Take n−1 fresh states u 1 , . . ., u n−1 .The new transition probabilities are defined as follows, where we identify q with u 0 : , for 0 ≤ j < n P (u j , u j+1 ) = 1−P (u j , s j ) , for 0 ≤ j < n−1 The condition that goal is the minimal element of < implies for every q ∈ S that if goal ∈ Post(q) = {s 0 , s 1 , . . ., s n }, then goal = s 0 .This makes sure that for every state u ∈ U the probability to reach goal in one step is zero, i.e.P (u, goal) = 0.
The resulting Markov chain is binary by construction and its state space consists of the old states (with adapted outgoing transitions) and the states added by the construction.In what follows, we fix any total order < such that goal is the minimal element of < and write B(M) = B(M, <). Figure 7 shows how the transformation works.Remark F.3.If we start with a tree-shaped Markov chain, the result of the above construction is also tree-shaped.The number of states of B(M) is bounded by the number of states plus the number of transitions of M.
Lemma F.4.There is a probability preserving one-to-one correspondence of paths in M = (S all , s 0 , P) and paths in B(M) = (S all ∪ U, s 0 , P ), where U is the set of all fresh states added by the construction.
Proof.From a path π in B(M) we get a path π in M by removing from π all states in U .We show that Pr B(M) (π) = Pr M (π ).Let S = S all \ {goal, fail}.It suffices to show this for all paths of the form SU + S, so let π = qu 1 . . .u n s with q, s ∈ S, u i ∈ U for 1 ≤ i ≤ n and n ≥ 1.As in Definition F.1, let Post(q) = {s 0 , s 1 , . . .s m } be ordered according to <, with m > n, µ i = P(q, s i ) for all s i ∈ Post(q) and s = s l .By the way the binarization was defined, we get that l ∈ {n, n + 1}.The construction now gives us: By induction, one can see that for all n ≥ 1: and hence Pr B(M) (qu 1 . . .u n s l ) = µ l = P(q, s l ) For the other direction, we observe that given s, q such that s ∈ Post(q) in M, there is a unique path of the form SU * S in B(M) that starts in s and ends in q.By the same reasoning as above, this path has probability P(q, s).Hence, by padding a path π in M with the corresponding states of U for every step, we get a path in B(M) with the same probability.
The following lemma relates witnessing subsystems of M and B(M).
Lemma F.5.Let M = (S all , s 0 , P) be an acyclic Markov chain as in Setting 2.2.The following two statements are equivalent: (1) M has a subsystem M with k reachable states satisfying Pr M (♦ goal) ≥ λ.We construct a subsystem B of B(M) by taking the states S all and adding all states U ⊆ U that lie on some path ((S all ) * U * ) * of B(M).If a state u ∈ U has a transition to a state s ∈ S all \ S all , this transition is redirected to fail.
We get a subsystem B of B(M) with states S all ∪ U .We verify that Pr B (♦ goal) = Pr M (♦ goal) holds, where S = S all \ {goal, fail}: This implies Pr B (♦ goal) ≥ λ.The first and last equivalence use the fact that the Markov chains are acyclic.The second equivalence uses Lemma F.4 and the fact that no state in U has a direct transition to goal.Here, b = (P (q, goal)) q∈S , as in Setting 2.2.
"⇐=": Let B be a subsystem of B with states S ∪ U and Pr B (♦ goal) ≥ λ.Take the subsystem of M induced by S .By the same calculation as above we get Pr M (♦ goal) = Pr B (♦ goal).
Minimal witnessing subsystems for tree-shaped binary DTMCs.Let B = (S all ∪ U, s 0 , P) be an acyclic binary Markov chain such that P(u, goal) = 0 for all u ∈ U and S all ∩ U = ∅.Let S = S all \ {goal, fail} and b = (P(q, goal)) q∈S .We define |q| to be the number of states reachable from q and |q| S = |q| ∩ S.
We now give an algorithm that takes a tree-shaped binary DTMC B, k ∈ N and λ ∈ [0, 1] and computes in polynomial time a subsystem B with k states in S such that Pr B ,s0 (♦ goal) ≥ λ, given that such a subsystem exists.
The idea is to compute a function l q : {0, . . ., |q| S } → [0, 1] for every state q in B with the following interpretation: l q (i) describes how much probability can be achieved in state q with a subsystem that is rooted in q and contains i states in S. If l s0 (k) ≥ λ, we know that there is a subsystem with k states in S with a probability to reach goal of at least λ.In case that we wish to compute the subsystem, we can save the corresponding subsystem for each entry l s0 (k).
We compute l bottom-up as follows: first, l q (0) = 0 for all states q apart from goal, which has l goal (0) = 1.If q ∈ S is a leaf, then l q (1) = b(q) and if q has exactly one successor q , then l q (i + 1) = l q (i), for all 0 ≤ i ≤ |q | S .
Otherwise, suppose that q has two successors q 1 , q 2 with transition probabilities µ 1 , µ 2 .If q ∈ S, we compute for 0 ≤ i < |q| S : For q ∈ U , we do not count the state q in the subsystem and hence we set (for 0 ≤ i ≤ |q|−1): Proposition F.6.Let M = (S all ∪ U, s 0 , P) be an acyclic binary DTMC as in Setting 2.2 and let S = S all \ {goal, fail}.The functions l q : {0, . . ., q |S| } → [0, 1] can be computed in time O(|S ∪ U | 3 ) for every q ∈ S.
Proof.The functions l q are computed bottom-up by using Equations F.1 and F. Proof.(i): Let B be a subsystem with k states in S and Pr B (♦ goal) ≥ λ.As before, let |q| S denote the number of states in S reachable from q ∈ S ∪ U in B .We show for all states in B , by induction on their height, that l q (|q| S ) ≥ Pr B ,q (♦ goal).
(1) Suppose that q is a leaf.If q ∈ U , we have l q (|q| S ) = l q (0) = 0 = Pr B ,q (♦ goal) If q ∈ S, we have l q (|q| S ) = l q (1) = P(q, goal) = Pr B ,q (♦ goal) (2) Suppose that q has two successors q 1 , q 2 that satisfy the property to prove and are reached with probability µ 1 , µ (ii): We show for every state q, by induction on its height, that for every k ∈ {0, . . ., |S|} and θ ∈ [0, 1]: if l q (k) = θ, then we can construct a subsystem Q with root q, k states in S and Pr Q (♦ goal) = θ.
(2) Suppose that q has two successors q 1 , q 2 that satisfy the property to prove and are reached with probability µ 1 , µ 2 .If q ∈ S, we get: As long as k ≤ |q| S the set above is not empty, as we can choose j = |q 1 | S which satisfies the constraints.Let j * be such that the above maximum is obtained, which yields By induction hypothesis we get subsystems of q 1 , q 2 with probability at least l q1 (j * ), l q2 (k−1−j * ) and j * , k−1−j * states in S, which proves the claim.The case for q ∈ U is similar.
As l s0 (k) ≥ λ holds by assumption, we can construct a subsystem of B with k states in S and probability of reaching goal at least λ.
Proof.We describe an algorithm that computes a minimal witnessing subsystem of the tree-shaped DTMC M = (S, s 0 , P) for Pr s0 (♦ goal) ≥ λ.First, we compute the binarization B(M) = (S all ∪U, s 0 , P ) of M as per Definition F.1.The number of states of B(M) is at most 2 • |S|, as M is tree-shaped and hence has at most as many transitions as states.By Lemma F.5, every witnessing subsystem of B(M) for Pr s0 (♦ goal) ≥ λ with k states in S can be mapped to a witnessing subsystem of M for the same property with k reachable states.We compute the function l s0 in polynomial time (Proposition F.6) and choose minimal k such that l s0 (k) ≥ λ.By Lemma F.7, we can compute a witnessing subsystem of B(M) for Pr s0 (♦ goal) ≥ λ with k states in S, and by the same lemma this subsystem is minimal.

Proposition 2 . 3 (
LP characterization, cf.[16, Lemma 8]).Let M be an MDP as in Setting 2.2 and let δ ∈ R n >0 .Then the vectors Pr min (♦ goal) and Pr max (♦ goal) are, respectively, the unique solution of the LPs max δ • z s.t.Az ≤ b and min δ • z s.t.Az ≥ b.

S
(s)(α) = y s,α α∈Act y s,α for those states where the denominator is positive, and S(s) arbitrary otherwise.This method for constructing MR-schedulers is standard, compare for example[47,  Chapter 2]  or[36, Theorem 3.2].Then y s,α is the expected frequency of the pair (s, α) occuring in a random walk on S-paths in M. For the induced DTMC M S it holds that Pr S s0 (♦ goal) = yb, as b contains the probability to move to goal in one step for every state and goal is absorbing.Hence Pr min s0 (♦ goal) ≤ Pr S s0 (♦ goal) = yb ≤ λ which is the property we considered.

2 + 3 2 + 3
states (the 3 accounting for s 0 , fail, and goal).The 'only if' direction is clear.For the reverse direction assume that M has a witness M for Pr M,s0 (♦ goal) ≥ k(k−1) n 2 with at most k := k + k(k−1) states.Denote those states of M lying in V by V and those lying in E by E .Let a = |V | and b = |E |, and recall that a + b ≤ k + k(k−1) be defined similarly.Then A and b are precisely as in Setting 2.2 for the MDP M R .By assumption we have Ap ≤ b and since supp(p) ⊆ R we also have A p ≤ b .We now apply Lemma 3.1 and get p ≤ Pr min M R (♦ goal), and therefore Pr min

Fig. 7 :
Fig. 7: Illustration of the local transformation of Definition F.1 from a state with four successors (a) to its binarization (b).

( 2 )
B(M) = (S all ∪ U, P ) has a subsystem B with reachable states S all ∪ U , where S all ⊆ S all , U ⊆ U , satisfying Pr B (♦ goal) ≥ λ and |S all | = k.Proof." =⇒ ": Let M = (S all , s 0 , P ) be a subsystem of M s.t.|S all | = k and satisfying Pr M (♦ goal) ≥ λ and assume that all states in S all are reachable in M .

2 .
Computing l q (i) requires to compute the biggest out of at most i + 1 values, hence requiring O(|q| S ) computations as i + 1 ≤ |q| S .Hence, computing the vector l q , which has |q| S entries, can be done in O (|q| S )2 time.As this has to be computed for every state in |S ∪ U |, and|q| S ≤ |S ∪ U |, l can be computed in O(|S ∪ U | • |S| 2 ) time.Lemma F.7. Let B = (S all ∪ U, P) be a tree-shaped binary DTMC as in Setting 2.2 such that P(u, goal) = 0 for all states u ∈ U and S all ∩ U = ∅.Then (i) If B has a subsystem B with k states in S and Pr B ,s0 (♦ goal) ≥ λ, then l s0 (k) ≥ λ. (ii) If l s0 (k) = θ, then a subsystem B of B with k states and Pr B ,s0 (♦ goal) = θ can be computed in polynomial time.
Given a state t ∈ S, we let S Pr S s (♦t) and Pr min s (♦t) = min S Pr S s (♦t) denote the maximal and minimal probability to reach t eventually when starting in s and set Pr min (♦t) = (Pr min s (♦t)) s∈S and Pr max (♦t) = (Pr max s The deductions for Pr max (♦ goal) are analogous, so that we get: Proposition 3.3.For ∈ {≥, >} and ∈ {≤, <} we have y ∈ P max (λ) Remark 6.2.To compute minimal witnesses for Pr max s0 (♦ goal) ≥ λ, [76, 77] (witnesses for Pr min s0 (♦ goal) ≥ λ were not considered) propose the MILP with objective: min (s,α)∈M σ(s, α), subject to the conditions