Good-for-MDPs Automata for Probabilistic Analysis and Reinforcement Learning

We characterize the class of nondeterministic \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega $$\end{document}-automata that can be used for the analysis of finite Markov decision processes (MDPs). We call these automata ‘good-for-MDPs’ (GFM). We show that GFM automata are closed under classic simulation as well as under more powerful simulation relations that leverage properties of optimal control strategies for MDPs. This closure enables us to exploit state-space reduction techniques, such as those based on direct and delayed simulation, that guarantee simulation equivalence. We demonstrate the promise of GFM automata by defining a new class of automata with favorable properties—they are Büchi automata with low branching degree obtained through a simple construction—and show that going beyond limit-deterministic automata may significantly benefit reinforcement learning.


Introduction
System specifications are often captured in the form of finite automata over infinite words (ω-automata), which are then used for model checking, synthesis, and learning. Of the commonly-used types of ω-automata, Büchi automata have the simplest acceptance condition, but require nondeterminism to recognize all ω-regular languages. Nondeterministic machines can use unbounded look-ahead to resolve nondeterministic choices. However, important applications-like reactive synthesis or model checking and reinforcement learning (RL) for Markov Decision Process (MDPs [23])-have a game setting, which restrict the resolution of nondeterminism to be based on the past.
Being forced to resolve nondeterminism on the fly, an automaton may end up rejecting words it should accept, so that using it can lead to incorrect results. Due to this difficulty, initial solutions to these problems have been based on deterministic automatausually with Rabin or parity acceptance conditions. For two-player games, Henzinger and Piterman proposed the notion of good-for-games (GFG) automata [15]. These are nondeterministic automata that simulate [21,14,9] a deterministic automaton that recognizes the same language. The existence of a simulation strategy means that nondeterministic choices can be resolved without look-ahead. This work has been supported by the National Natural Science Foundation of China (Grant Nr. 61532019), EPSRC grants EP/M027287/1 and EP/P020909/1, and a CU Boulder RIO grant.
The situation is better in the case of probabilistic model checking, because the game for which a strategy is sought is played on an MDP against "blind nature," rather than against a strategic opponent who may take advantage of the automaton's inability to resolve nondeterminism on the fly. As early as 1985, Vardi noted that probabilistic model checking can be performed with Büchi automata endowed with a limited form of nondeterminism [34]. Limit deterministic Büchi automata (LDBA) [4,11,29] perform no nondeterministic choice after seeing an accepting transition. Still, they recognize all ω-regular languages and are, under mild restrictions [29], suitable for probabilistic model checking. Related Work. The production of deterministic and limit deterministic automata for model checking has been intensively studied [24,22,1,26,33,32,27,29,8,30,20], and several tools are available to produce different types of automata, incl. MoChiBA/Owl [29,30,20], LTL3BA [1], GOAL [33,32], SPOT [8], Rabinizer [19], and Büchifier [16].
So far, only deterministic and a (slightly restricted [29]) class of limit deterministic automata have been considered for probabilistic model checking [34,4,11,29]. Thus, while there have been advances in the efficient production of such automata [11,29,30,20], the consideration of suitable LDBAs by Courcoubetis and Yannakakis in 1988 [3] has been the last time when a fundamental change in the automata foundation of MDP model checking has occurred. Contribution. The simple but effective observation that simulation preserves the suitability for MDPs (for both traditional simulation and the AEC simulation we introduce) extends the class of automata that can be used in the analysis of MDPs. This provides us with three advantages: The first advantage is that we can now use a wealth of simulation based statespace reduction techniques [7,31,10,9] on an automaton A (e.g. an SLDBA) that we would otherwise use for MDP model checking. The second advantage is that we can use A to check if a different language equivalent automaton, such as an NBA B (e.g. an NBA from which A is derived) simulates A. For this second advantage, we can dip into the more powerful class of AEC simulation we define in Section 4 that use properties of winning strategies on finite MDPs. While this is not a complete method for identifying GFM automata, our experimental results indicate that the GFM property is quite frequent for NBAs constructed from random formulas, and can often be established efficiently, while providing a significant statespace reduction and thus offering a significant advantage for model checking.
A third advantage is that we can use the additional flexibility to tailor automata for different applications than model checking, for which specialized automata classes have not yet been developed. We demonstrate this for model-free reinforcement learning (RL). We argue that RL benefits from three properties that are less important in model checking: The first-easy to measure-property is a small number of successors, the second and third, are cautiousness, the scope for making wrong decisions, and forgiveness, the resilience against making wrong decisions, respectively.
A small number of successors is a simple and natural goal for RL, as the lack of an explicit model means that the product space of a model and an automaton cannot be evaluated backwards. In a forward analysis, it matters that nondeterministic choices have to be modeled by enriching the decisions in the MDPs with the choices made by the automaton. For LDBAs constructed from NBAs, this means guessing a suit-able subset of the reachable states when progressing to the deterministic part of the automaton, meaning a number of choices that is exponential in the NBA. We show that we can instead use slim automata in Section 3.2 as a first example of NBAs that are good-for-MDPs, but not limit deterministic. They have the appealing property that their branching degree is at most two, while keeping the Büchi acceptance mechanism that works well with RL [12]. (Slim automata can also be used for model checking, but they don't provide similar advantages over suitable LDBAs there, because the backwards analysis used in model checking makes selecting the correct successor trivial.) Cautiousness and forgiveness are further properties, which are-while harder to quantify-very desirable for RL: LDBAs, for example, suffer from having to make a correct choice when moving into the deterministic part of the automaton, and they have to make this correct choice from a very large set of nondeterministic transitions. While this is unproblematic for standard model checking algorithms that are based on backwards analysis, applications like RL that rely on forward analysis can be badly affected when more (wrong) choices are offered, and when wrong choices cannot be rectified. Cautiousness and forgiveness are a references to this: an automaton is more cautious if it has less scope for making wrong decisions and more forgiving if it allows for correcting previously made decisions (cf. Figure 5 for an example). Our experiments (cf. Section 5) indicate that cautiousness and forgiveness are beneficial for RL. Organization of the Paper. After the preliminaries, we introduce the "good-for-MDP" property (Section 3) and show that it is preserved by simulation, which enables all minimization techniques that offer the simulation property (Section 3.1). In Section 3.2 we use this observation to construct slim automata-NBAs with a branching degree of 2 that are neither limit deterministic nor good-for-games-as an example of a class of automata that becomes available for MDP model checking and RL. We then introduce a more powerful simulation relation, AEC simulation, that suffices to establish that an automaton is good-for-MDPs (Section 4). In Section 5, we evaluate the impact of the contributions of the paper on model checking and reinforcement learning algorithms.

Preliminaries
A nondeterministic Büchi automaton is a tuple A = Σ, Q, q 0 , ∆, Γ , where Σ is a finite alphabet, Q is a finite set of states, q 0 ∈ Q is the initial state, ∆ ⊆ Q × Σ × Q are transitions, and Γ ⊆ Q × Σ × Q is the transition-based acceptance condition.
A run r of A on w ∈ Σ ω is an ω-word r 0 , w 0 , r 1 , w 1 , . . . in (Q×Σ) ω such that r 0 = q 0 and, for i > 0, it is (r i−1 , w i−1 , r i ) ∈ ∆. We write inf(r) for the set of transitions that appear infinitely often in the run r.
The language, L A , of A (or, recognized by A) is the subset of words in Σ ω that have accepting runs in A. A language is ω-regular if it is accepted by a Büchi automaton. An automaton A = Σ, Q, Q 0 , ∆, Γ is deterministic if (q, σ, q ), (q, σ, q ) ∈ ∆ implies q = q . A is complete if, for all σ ∈ Σ and all q ∈ Q, there is a transition (q, σ, q ) ∈ ∆. A word in Σ ω has exactly one run in a deterministic, complete automaton.
A Markov decision process (MDP) M is a tuple (S, A, T, Σ, L) where S is a finite set of states, A is a finite set of actions, T : S × A − D(S), where D(S) is the set of probability distributions over S, is the probabilistic transition (partial) function, Σ is an alphabet, and L : S × A × S → Σ is the labeling function of the set of transitions. For a state s ∈ S, A(s) denotes the set of actions available in s. For states s, s ∈ S and a ∈ A(s), we have that T (s, a)(s ) equals Pr (s |s, a).
A run of M is an ω-word s 0 , a 1 , . . . ∈ S × (A × S) ω such that Pr (s i+1 |s i , a i+1 ) > 0 for all i ≥ 0. A finite run is a finite such sequence. For a run r = s 0 , a 1 , s 1 , . . . we define the corresponding labeled run as L(r) = L(s 0 , a 1 , s 1 ), L(s 1 , a 2 , s 2 ), . . . ∈ Σ ω . We write Ω(M) (Paths(M)) for the set of runs (finite runs) of M and Ω s (M) (Paths s (M)) for the set of runs (finite runs) of M starting from state s. When the MDP is clear from the context we drop the argument M.
A strategy in M is a function µ : Paths → D(A) such that supp(µ(r)) ⊆ A(last(r)), where supp(d) is the support of d and last(r) is the last state of r. Let Ω M µ (s) denote the subset of runs Ω M (s) that correspond to strategy µ and initial state s. Let Π M be the set of all strategies. We say that a strategy µ is pure if µ(r) is a point distribution for all runs r ∈ Paths and we say that µ is positional if last(r) = last(r ) implies µ(r) = µ(r ) for all runs r, r ∈ Paths. The behavior of an MDP M under a strategy µ with starting state s is defined on a probability space (Ω µ s , F µ s , Pr µ s ) over the set of infinite runs of µ from s.

Good-for-MDP (GFM) Automata
Given an MDP M and an automaton A = Σ, Q, q 0 , ∆, Γ , we want to compute an optimal strategy satisfying the objective that the run of M is in the language of A. We define the semantic satisfaction probability for A and a strategy µ from state s as: Finally, Γ × ⊆ (S ×Q)×(A×Q)×(S ×Q) is defined by ((s, q), (a, q ), (s , q )) ∈ Γ × if, and only if, (q, L(s, a, s ), q ) ∈ Γ and T (s, a)(s ) > 0. A strategy µ on the MDP defines a strategy µ × on the product, and vice versa. We define the syntactic satisfaction probabilities as Note that PSyn M A (s) = PSem M A (s) holds for a deterministic A. In general, PSyn M A (s) ≤ PSem M A (s) holds, but equality is not guaranteed because the optimal resolution of nondeterministic choices may require access to future events (see Figure 1). For an automaton to match PSem M A (s 0 ), its nondeterminism is restricted not to rely heavily on the future; rather, it must possible to resolve the nondeterminism on-the-fly. For example, the Büchi automaton presented on the left of Figure 1, which has to guess whether the next symbol is a or b, is not good for MDPs, because the simple Markov chain on the right of Figure 1 does not allow resolution of its nondeterminism on-the-fly.
In the context of RL, techniques based on SLDBAs are particularly useful, because these automata use the Büchi acceptance condition, which can be translated to reachability goals. Good for games and deterministic automata require more complex acceptance conditions, like parity, that do not have a natural translation into rewards [12].
Using SLDBA [4,11,29] has the drawback that they naturally have a high branching degree in the initial part, as they naturally allow for many different transitions to the accepting part of the LDBA. This can be avoided, but to the cost of a blow-up and a more complex construction and data structure [29]. We therefore propose an automata construction that produces NBAs with a small branching degree-it never produces more than two successors. We call these automata slim. The resulting automata are not (normally) limit deterministic, but we show that they are good for MDPs.
Due to technical dependencies we start with presenting a second observation, namely that automata that simulate language equivalent GFM automata are GFM. As a side result, we observe that the same holds for good-for-games automata. The side result is not surprising, as good-for-games automata were defined through simulation of deterministic automata [15]. But, to the best of our knowledge, the observation from Corollary 1 has not been made yet for good-for-games automata.

Simulating GFM
An automaton A simulates an automaton B if the duplicator wins the simulation game. The simulation game is played between a duplicator and a spoiler, who each control a pebble, which they move along the edges of A and B, respectively. The game is started by the spoiler, who places her pebble on an initial state of B. Next, the duplicator puts his pebble on an initial state of A. The two players then take turns, always starting with the spoiler choosing an input letter and a transition for that letter in B, followed by the duplicator choosing a transition for the same letter in A. This way, both players produce an infinite run of their respective automaton. The duplicator has two ways to win a play of the game: if the run of A he constructs is accepting, and if the run the spoiler constructs on B is rejecting. The duplicator wins this game if he has a winning strategy, i.e., a recipe to move his pebble that guarantees that he wins. Such a winning strategy is "good-for-games," as it can only rely on the past. It can be used to transform winning strategies of B, so that, if they were witnessing a good for games property or were good for an MDP, then the resulting strategy for A has the same property.

Lemma 1 (Simulation Properties).
For ω-automata A and B the following holds. (1) and (2) are well known observations. Fact (1) holds because an accepting run of B on a word α can be translated into an accepting run of A on α by using the winning strategy of A in the simulation game. Fact (2) follows immediately from Fact (1). Facts (3) and (4) follow by simulating the behaviour of B on each run.

If
This observation allows us to use a family of state-space reduction techniques, in particular those based on language preserving translations for Büchi automata based on simulation relation [7,31,10,9]. This requires stronger notions of simulations, like direct and delayed simulation [9]. For the deterministic part of an LDBA, one can also use space reduction techniques for DBAs like [25]. Corollary 1. All statespace reduction techniques that turn an NBA A into an NBA B that simulates A preserve GFG and GFM: if A is GFG or GFM, then B is GFG or GFM, respectively.

Constructing Slim GFM Automata
Let us fix Büchi automaton B = Σ, Q, Q 0 , ∆, Γ . We can write ∆ as a function δ : Q × Σ → 2 Q withδ : (q, σ) → {q ∈ Q | (q, σ, q ) ∈ ∆}, which can be lifted to sets, using the deterministic transition function δ : 2 Q × Σ → 2 Q with δ : (S, σ) → q∈Sδ (q, σ). We also define an operator, ndet, that translates deterministic transition functions δ : R × Σ → R to relations, using This is just an easy means to move back and forth between functions and relations, and helps one to visualize the maximal number of successors. We next define the variations of subset and breakpoint constructions that are used to define the well-known limit deterministic GFM automata-which we use in our proofs-and the slim GFM automata we construct. Let 3 Q := (S, S ) | S S ⊆ Q and 3 Q + := (S, S ) | S ⊆ S ⊆ Q . We define the subset notation for the transitions and accepting transitions as We define the raw breakpoint transitions δ R : . In this construction, we follow the set of reachable states (first set) and the states that are reachable while passing at least one of the accepting transitions (second set). To turn this into a breakpoint automaton, we reset the second set to the empty set when it equals the first; the transitions where we reset the second set are exactly the accepting ones. The breakpoint automaton , then there are three cases: Finally, we define transitions ∆ SB ⊆ 2 Q × Σ × 3 Q that lead from a subset to a breakpoint construction, and γ 2,1 : 3 Q × Σ → 3 Q that promote the second set of a breakpoint construction to the first set as follows.
We can now define standard limit deterministic good for MDP automata.
recognizes the same language as B. It is limit deterministic and good for MDPs.
We now show how to construct a slim GFM Büchi automaton.
Theorem 2 (Slim GFM Büchi Automaton). The automaton simulates A. S is slim, language equivalent to B, and good for MDPs.
Proof. S is slim: its set of transitions is the union of two sets of deterministic transitions. We show that S simulates A by defining a strategy in the simulation game, which ensures that, if the spoiler produces a run S 0 . . . S j−1 (S j , S j )(S j+1 , S j+1 ) . . . for A, then the duplicator produces a run (T 0 , T 0 ) . . . (T j−1 , T j−1 )(T j , T j )(T j+1 , T j−1 ) . . . for S, such that (1) S i ⊆ T i holds for all i ∈ ω, and (2) if there are two accepting transitions (S k−1 , S k−1 ), σ k , (S k , S k ) and (S l−1 , S l−1 ), σ l , (S l , S l ) with k < l, there is an k < m ≤ l, such that (T m−1 , T m−1 ), σ m (T m , T m ) is accepting.
To obtain this, we describe a winning strategy for the duplicator while arguing inductively that it mainains (1). Note that (1) holds initially (T 0 = S 0 , induction basis).
Initial Phase: Every move of the spoiler-with some letter σ-that uses a transition from δ S -the subset part of A-is followed by a move from δ B with the same letter σ. When the duplicator follows this strategy the following holds: when, after a pair of moves, the pebble of the spoiler is on state S ⊆ Q, then the pebble of the duplicator is on some state (S, S ). In particular, (1) is preserved during this phase (induction step).
Transition Phase: The one spoiler move-with some letter σ-that uses a transition from ∆ SB -the transition to the breakpoint part of A-is followed by a move from δ B with the same letter σ. When the duplicator follows this strategy, and when, after the pair of moves, the pebble of the spoiler is on state (S, ∅), then the pebble of the duplicator is on some state (T, T ) with S ⊆ T . In particular, (1) is preserved (induction step).
Final Phase: When the spoiler moves from some state (S, S )-with some letter σthat uses a transition from δ B -the breakpoint part of A-to (S,S ), and when the duplicator is in some state (T, T ), then the duplicator does the following. He calculates (T , ∅) = γ 2,1 (T, T ), σ and checks ifS ⊆T holds. IfS ⊆T holds, he plays this transition from γ 2,1 (with the same letter σ). Otherwise, he plays the transition from δ B (with the same letter σ). In either case (1) is preserved (induction step), which closes the inductive argument for (1).
Note that no accepting transition of A is passed in the initial or tansition phase, so the two accepting transitions from (2) must both fall into the final phase.
To show (2), we first observe that S k = ∅, and thus S k ⊆ T k holds. Assuming for contradition that all transitions of S for σ k+1 . . . σ l−1 are non-accepting, we obtainusing (1)-by a straightforward inductive argument that S i ⊆ T i for all i with k≤i<l. (Note that transitions in δ B are accepting when they are also be in γ B .) Using that S l = δ S (S l−1 , σ l ) ∪ γ S (S l−1 , σ l ) ⊆ δ S (T l−1 , σ l ) ∪ γ S (T l−1 , σ l ) holds, the spoiler uses an accepting transition from γ 2,1 in this step.
Using Lemma 1, it now suffices to show that the language of S is included in the language of B. To show this, we simply argue that an accepting run ρ = (Q 0 , Q 0 ), (Q 1 , Q 1 ), (Q 2 , Q 2 ), (Q 3 , Q 3 ), . . . of S on an input word α = σ 0 , σ 1 , σ 2 , . . . can be interpreted as a forest of finitely many finitely branching trees of overall infinite size, where all infinite branches are accepting runs of B. Kőnig's Lemma then proves the existence of an accepting run of B.
This forest is the usual one. The nodes are labeled by states of B, and the roots (level 0) are the initial states of B. 1 ) be the set of positions after accepting transitions in ρ. We define the predecessor function pred : N → I∪{0} with pred : i → max j ∈ I∪{0} | j < i .
We call a node with label q l on level l an end-point if one of the following applies: (1) q l / ∈ Q l or (2) l ∈ I and for all j such that pred(l) ≤ j < l, where q j is the label of the ancestor of this node on level j, we have (q j , σ j , q j+1 ) / ∈ Γ . (1) may only happen after a transition from γ 2,1 has been taken, and the q l is not among the states that is traced henceforth. (2) identifies parts of the run tree that do not contain an accepting transition.
A node labeled with q l on level l that is not an endpoint has δ S (q l , σ l ) children, labeled with the different elements of δ S (q l , σ l ). It is now easy to show by induction over i that the following holds.
1. For all q ∈ Q i , there is a node on level i labeled with q.

For i /
∈ I and q ∈ Q i , there is a node labeled q on level i, a j with pred(i) ≤ j < i, and ancestors on level j and j +1 labeled q j and q j+1 , such that (q j , σ j , q j+1 ) ∈ Γ . (The 'ancestor' on level j + 1 might be the state itself.) For i ∈ I and q ∈ Q i , there is a node labeled q on level i, which is not an end point.
Consequently, the forest is infinite, finitely branching, and finitely rooted, and thus contains an infinite path. By construction, this path is an accepting run of B.
The resulting automata are simple in structure and enable symbolic implementation (See Fig. 2). It cannot be expected that there are much smaller good for MDP automata, as its explicit construction is the only non-polynomial part in model checking MDPs.
Theorem 3. Constructing a GFM Büchi automaton G that recognizes the models of an LTL formula ϕ requires time doubly exponential in ϕ, and constructing a GFM Büchi automaton G that recognizes the language of an NBA B requires time exponential in B.
Proof. As resulting automata are GFM, they can be used to model check MDPs M against this property, with cost polynomial in product of M and G. If G could be produced faster (and if they could, consequently be smaller) than claimed, it will contradict the 2-EXPTIME-and EXPTIME-hardness [4] of these model checking problems.

Accepting End-Component Simulation
An end-component [5,2] of an MDP M is a sub-MDP M of M such that its underlying graph is strongly connected. A maximal end-component is maximal under set-inclusion. Every state of an MDP belongs to at most one maximal end-component.

Theorem 4 (End-Component Properties. Theorem 3.1 and Theorem 4.2 of [5]).
Once an end-component C of an MDP is entered, there is a strategy that visits every state-action combination in C infinitely often with probability 1 and stays in C forever.
For a product MDP, an accepting end-component (AEC) is an end-component that contains some transition in Γ × . There is a positional pure strategy for an AEC C that surely stays in C and almost surely visits a transition in Γ × infinitely often.
For a product MDP, there is a set of disjoint accepting end-components such that, from every state, the maximal probability to reach the union of these accepting endcomponents is the same as the maximal probability to satisfy Γ × . Moreover, this probability can be realized by combining a positional pure (reachability) strategy outside of this union with the aforementioned positional pure strategies for the individual AECs.
Lemma 1 shows that the GFM property is preserved by simulation: For languageequivalent automata A and B, if A simulates B and B is GFM, then A is also GFM. However, a GFM automaton may not simulate a language-equivalent GFM automaton. (See Figure 3.) Therefore we introduce a coarser preorder, Accepting End-Component (AEC) simulation, that exploits the finiteness of the MDP M. We rely on Theorem 4 to focus on positional pure strategies for M × B. Under such strategies, M × B becomes a Markov chain [2] such that almost all its runs have the following properties: With this in mind, we can intuitively ask the spoiler to pick a run through this Markov chain, and to disclose information about this run. Specifically, we can ask her to signal when she has reached an accepting LSCC 5 in the Markov chain, and to provide information about this LSCC, in particular information entailed by the full list of sequences of transitions of some fixed length described above. Runs that can be identified to either not reach an accepting LSCC, to visit transitions not in this list, or to visit only a subset of sequences from this list, form a 0 set. In the simulation game we define below, we make use of this observation to discard such runs. A simulation game can only use the syntactic material of the automata--neither the MDP nor the strategy are available. The information the spoiler may provide cannot explicitly refer to them. What the spoiler may be asked to provide is information on when she has entered an accepting LSCC, and, once she has signaled this, which sequences of length l of automata transitions of B occur in the LSCC. The sequences of automata transitions are simply the projections on the automata transitions from the sequences of transitions of length that occur in the LSCC L. We call this information a gold-brim accepting end-component claim of length , -GAEC claim for short.
The term "gold-brim" in the definition indicates that this is a powerful approach, but not one that can be implemented efficiently. We will define weaker, efficiently implementable notions of accepting end-component claims (AEC claims) later.
The AEC simulation game is very similar to the simulation game of Section 3.1. Both players produce an infinite run of their respective automata. If the spoiler makes an AEC claim, e.g., an -GAEC claim, we say that her run complies with it if, starting with the transition when the AEC claim is made, all states, transitions, or sequences of transitions in the claim appear infinitely often, and all states, transitions, and sequences of transitions the claim excludes do not appear. For an -GAEC claim, this means that all of the sequences of transitions of length in the claim occur infinitely often, and no other sequence of length occurs henceforth.
Thus, like a classic simulation game, an -GAEC simulation game is started by the spoiler, who places her pebble on an initial state of B. Next, the duplicator puts his pebble on an initial state of A. The two players then take turns, always starting with the spoiler choosing an input letter and an according transition from B, followed by the duplicator choosing a transition for the same letter in A.
Different from the classic simulation game, in an -GAEC simulation game, the spoiler has an additional move that she can (and, in order to win, has to) perform once in the game: In addition to choosing a letter and a transition, she can claim that she has reached an accepting end-component, and provide a complete list of sequences of automata transitions of length that can henceforth occur. This store is maintained, and never updated. It has no further effect on the rules of the game: Both players produce an infinite run of their respective automata. The duplicator has four ways to win: 1. if the spoiler never makes an AEC claim, 2. if the run of A he constructs is accepting, 3. if the run the spoiler constructs on B does not comply with the AEC claim, and 4. if the run that the spoiler produces is not accepting.
For -GAEC claims, (4) simply means that the set of transitions defined by the sequences does not satisfy the Büchi, parity, or Rabin acceptance condition.

Theorem 5. [ -GAEC Simulation] If
A and B are language equivalent automata, B is GFM, and there exists an such that A -GAEC simulates B, then A is GFM.
For the proof, we use an arbitrary (but fixed) MDP M, and an arbitrary (but fixed) pure optimal positional strategy µ for M × B, resulting in the Markov chain (M × B) µ . We assume w.l.o.g. that the accepting LSCCs in (M × B) µ are identified, e.g., by a bit.
Let τ be a winning strategy of the duplicator in an -GAEC simulation game. Abusing notation, we let τ • µ denote the finite-memory strategy 6 obtained from µ and τ for M × A, where τ is acting only on the automata part of (M × B), and where the spoiler makes the move to the end-component when she is in some LSCC B of (M × B) µ and gives the full list of sequences of transitions of length that occur in B.
Proof. As B is good for MDPs, we only have to show that the chance of winning in (M × A) τ •µ is at least the chance of winning in (M × B) µ . The chance of winning in (M × B) µ is the chance of reaching an accepting LSCC in (M × B) µ . It is also the chance of reaching an accepting LSCC L ∈ (M × B) µ and, after reaching L, to see exactly the sequences of transitions of length that occur in L, and to see all of them infinitely often.
By construction, τ • µ will translate those runs into accepting runs of (M × A) τ •µ , such that the chance of an accepting run of (M × A) τ •µ is at least the chance of an accepting run of (M × B) µ . As µ is optimal, the chance of winning in M × A is at least the chance of winning in M × B. As B is GFM, this is the chance of M producing a run accepted by B (and thus A) when controlled optimally, which is an upper bound on the chance of winning in M × A.
An -GAEC simulation, especially for large , results in very large state spaces, because the spoiler has to list all sequences of transitions of B of length that will appear infinitely often. No other sequence of length may then appear in the run 7 . This can, of course, be prohibitively expensive.
As a compromise, one can use coarser-grained information at the cost of reducing the duplicator's ability of winning the game. E.g., the spoiler could be asked to only reveal a transition that is repeated infinitely often, plus (when using more powerful acceptance conditions than Büchi), some acceptance information, say the dominating priority in a parity game or a winning Rabin pair. This type of coarse-grained claim can be refined slightly by allowing the duplicator to change at any time the transition that is to appear infinitely often to the transition just used by the spoiler. Generally, we say that an AEC simulation game is any simulation game, where the spoiler provides a list of states, transitions, or sequences of transitions that will occur infinitely often and a list of states, transitions, or sequences of transitions that will not occur in the future when making her AEC claim, and the duplicator may be able to update this list based on his observations, there exists some -GAEC simulation game such that a winning strategy of the spoiler translates into a winning strategy of the spoiler in the AEC simulation game.
The requirement that a winning spoiler strategy translates into a winning spoiler strategy in an -GAEC game entails that AEC simulation games can prove the GFM property.

Corollary 2. [AEC Simulation] If
A and B are language equivalent automata, B is good for MDPs, and A AEC-simulates B, then A is good for MDPs. Of course, for every AEC simulation, one first has to prove that winning strategies for the spoiler translate. We have used two simple variations of the AEC simulation games: accepting transition: the spoiler may only make her AEC claim when taking an accepting transition; this transition-and no other information-is stored, and the spoiler commits to-and commits only to-seeing this transition infinitely often; accepting transition with update: different to the accepting transition AEC simulation game, the duplicator can-but does not have to-update the stored accepting transition whenever the spoiler passes by an accepting transition.
Theorem 6. Both, the accepted transition and the accepted transition with update AEC simulation, can be used to establish the good for MDPs property.
To show this, we describe the strategy translations in accordance with Corollary 2.
Proof. In both cases, the translation of a winning strategy of the spoiler for the 1-GAEC simulation game are straightforward: The spoiler essentially follows her winning strategy from the 1-GAEC simulation game, with the extra rule that she will make her AEC claim to the duplicator on the first accepting transition on or after her AEC claim in the 1-GAEC claim. If the duplicator is allowed to update the transition, this information is ignored by the spoiler-she plays according to her winning strategy from the 1-GAEC simulation game. Naturally, the resulting play will comply with her 1-GAEC claim, and will thus also be winning for the-weaker-AEC claim made to the duplicator.
We use AEC simulation to identify GFM automata among the automata produced (e.g., by SPOT [8]) at the beginning of the transformation. Figure 3 shows an example for which the duplicator wins the AEC simulation game, but loses the ordinary simulation game. Candidates for automata to simulate are, e.g., the slim GFM Büchi automata and the limit deterministic Büchi automata discussed above.

Size of General Büchi Automata for Probabilistic Model Checking
As discussed, automata that simulate slim automata or SLDBAs are good for MDPs. This fact can be used to allow Büchi automata produced from general-purpose tools such as SPOT's [8] ltl2tgba rather than using specialized automata types. Automata produced by such tools are often smaller because such general-purpose tools are highly optimized and not restricted to producing slim or limit deterministic automata. Thus, one produces an arbitrary Büchi automaton using any available method, then transforms this automaton into a slim or limit deterministic automaton, and finally checks whether the original automaton simulates the generated one.
We have evaluated this idea on random LTL formulas produced by SPOT's tool randltl. We have set the tree size, which influences the size of the formulas, to 50, and have produced 1000 formulas with 4 atomic propositions each. We left the other values to their defaults. We have then used SPOT's ltl2tgba (version 2.7) to turn these formulas into non-generalized Büchi automata using default options. Finally, for each automaton, we have used our tool to check whether the automaton simulates a limit deterministic automaton that we produce from this automaton. For comparison, we have also used Owl's [29] tool ltl2ldba (version 19.06.03) to compute limit deterministic nongeneralized Buchi automata. We have also used the option of this tool to compute Büchi automata with a nondeterministic initial part. We used 10 minute timeouts.
Of these 1000 formulas, 315 can be transformed to deterministic Büchi automata. For an additional 103 other automata generated, standard simulation sufficed to show that they are GFM. For a further 11 of them, the simplest AEC simulation (the spoiler chooses an accepting transition to occur infinitely often) sufficed, and another 1 could be classed GFM by allowing the duplicator to update the transition. 501 automata turned out to be nonsimulatable and for 69 we did not get a decision due to a timeout.
For the LTL formulas for which ltl2tgba could not produce deterministic automata, but for which simulation could be shown, the number of states in the generated automata was often lower than the number of states in the automata produced by Owl's tools. On average, the number of states per automaton was ≈15.21 for SPOT's ltl2tgba; while for Owl's ltl2ldba it was ≈46.35. The extended version of this paper [13] contains more details about the evaluation. Let us consider the ratio between the size of automata produced by ltl2tgba and the size of semi-deterministic automata produced by Owl. The average of this number for all automata that are not deterministic and that can be simulated in some way is ≈ 1.0335. This means that on average, for these automata, the semi-deterministic automata are slightly smaller. If we take a look at the first 5 deciles depicted in Fig. 4, we see that there is a large number of formulas for which ltl2tgba and Owl produce automata of the same size. For around 24.3478% of the cases, automata by SPOT are smaller than those produced by Owl (ratio < 1).

GFM Automata and Reinforcement Learning
SLDBAs have been used in [12] for model-free reinforcement learning of ω-regular objectives. While the Büchi acceptance condition allows for a faithful translation of the objective to a scalar reward, the agent has to learn how to control the automaton's nondeterministic choices; that is, the agent has to learn when the SLDBA should cross from the initial component to the accepting component to produce a successful run of a behavior that satisfies the given objective.
Any GFM automaton with a Büchi acceptance condition can be used instead of an SLDBA in the approach of [12]. While in many cases SLDBAs work well, GFM automata that are not limit-deterministic may provide a significant advantage.
Early during training, the agent relies on uniform random choices to discover policies that lead to successful episodes. This includes randomly resolving the automaton nondeterminism. If random choices are unlikely to produce successful runs of the automaton in case of behaviors that should be accepted, learning is hampered because good behaviors are not rewarded. Therefore, GFM automata that are more likely to accept under random choices will result in the agent learning more quickly. We have found the following properties of GFM automata to affect the agent's learning ability. Low branching degree. A low branching degree presents the agent with fewer alternatives, reducing the expected number of trials before the agent finds a good combination of choices. Consider an MDP and an automaton that require a specific sequence of k nondeterministic choices in order for the automaton to accept. If at each choice there are b equiprobable options, the correct sequence is obtained with probability b −k .
Cautiousness. An automaton that enables fewer nondeterministic choices for the same finite input word gives the agent fewer chances to choose wrong. The slim automata construction has the interesting property of "collecting hints of acceptance" before a nondeterministic choice is enabled because S has to be nonempty for a γ 2,1 transition to be present and that requires going through at least one accepting transition. Forgiveness. Mistakes made in resolving nondeterminism may be irrecoverable. This is often true of SLDBAs meant for model checking, in which jumps are made to select a subformula to be eventually satisfied. However, general GFM automata, thanks also to their less constrained structure, may be constructed to "forgive mistakes" by giving more chances of picking a successful run. Figure 5 compares a typical SLDBA to an automaton that is not limit-deterministic and is not produced by the breakpoint construction, but is proved GFM by AEC simulation. This latter automaton has a nondeterministic choice in state q 0 on letter x ∧ ¬y that can be made an unbounded number of times. The agent may choose q 1 repeatedly even if eventually F G x is false and G F y is true. With the SLDBA, on the other hand, there is no room for error. A Case Study. We compared the effectiveness in learning to control a cart-pole model of three automata for the property (F G x) ∨ (G F y) ∧ G safe. The safety component of the objective is to keep the pole balanced and the cart on the track. The left two thirds of the track alternate between x and y at each step. The right third is always labeled y, but in order to reach it, the cart has to cross a barrier, with probability 1/3 of failing.
The three automata are an SLDBA (4 states), a slim automaton (8 states), and a handcrafted forgiving automaton (4 states) similar to the one of Fig. 5. Training of the continuous-statespace model employed PPO [28] as implemented in OpenAI Baselines [6]. Figure 6 shows the learning curves for the three automata averaged over ten runs. They underline the importance of choosing the right automaton in RL. Training parameters, more details on the model, and additional examples can be found in the extended version of this paper [13].

Conclusion
We have defined the class of automata that are good for MDPs-nondeterministic automata that can be used for the analysis of MDPs-and shown it to be closed under different simulation relations. This has multiple favorable implications for model checking and reinforcement learning. Closure under classic simulation opens a rich toolbox of statespace reduction techniques that come in handy to push the boundary of analysis techniques, while the more powerful (and more expensive) AEC simulation has promise to identify source automata that happen to be good for MDPs.
The wider class of GFM automata also shows promise: the slim automata we have defined to tame the branching degree while retaining the desirable Büchi condition for reinforcement learning are able to compete even against optimized SLDBAs.
As outlined in Section 5.2, a low branching degree, cautiousness, and forgiveness make automata particularly well-suited for learning. From a practical point of view, much of the power of this new approach is in harnessing the power of simulation for learning, and forgiveness is closely related to simulation.
The natural follow-up research is to tap the full potential of simulation-based statespace reduction instead of the limited version that we have implemented. Besides using this to get the statespace small-useful for model checking-we will use simulation to construct forgiving automata, which is promising for reinforcement learning.