Software Verification of Hyperproperties Beyond k-Safety

Temporal hyperproperties are system properties that relate multiple execution traces. For (finite-state) hardware, temporal hyperproperties are supported by model checking algorithms, and tools for general temporal logics like HyperLTL exist. For (infinite-state) software, the analysis of temporal hyperproperties has, so far, been limited to $k$-safety properties, i.e., properties that stipulate the absence of a bad interaction between any $k$ traces. In this paper, we present an automated method for the verification of $\forall^k\exists^l$-safety properties in infinite-state systems. A $\forall^k\exists^l$-safety property stipulates that for any $k$ traces, there exist $l$ traces such that the resulting $k+l$ traces do not interact badly. This combination of universal and existential quantification enables us to express many properties beyond $k$-safety, including, for example, generalized non-interference or program refinement. Our method is based on a strategy-based instantiation of existential trace quantification combined with a program reduction, both in the context of a fixed predicate abstraction. Notably, our framework allows for mutual dependence of strategy and reduction.


Introduction
Hyperproperties are system properties that relate multiple execution traces of a system [21] and commonly arise, e.g., in information-flow policies [34], the verification of code optimizations [6], and robustness of software [18]. Consequently, many methods for the automated verification of hyperproperties have been developed [38,40,26,39]. Almost all previous approaches verify a class of hyperproperties called k-safety, i.e., properties that stipulate the absence of a bad interaction between any k traces in the system. For example, we can express a simple form of non-interference as a 2-safety property by stating that any two traces that agree on the low-security inputs should produce the same observable output.
The vast landscape of hyperproperties does, however, stretch far beyond ksafety. The overarching limitation of k-safety (or, more generally, of hypersafety [21]) is an implicit universal quantification over all executions. By contrast, many properties of interest, ranging from applications in information-flow control to robust cleanness, require a combination of universal and existential quantification. For example, consider the reactive program in Figure 1, where N denotes a nondeterministic choice of a natural number. We assume that h, l, and o are a high-security input, a low-security input, and a low-security output, respectively. This program violates the simple 2-safety non-interference property given above as the non-determinism influences the output. Nevertheless, the program is "secure" in the sense that an attacker that observes low-security inputs and outputs cannot deduce information about the high-security input. To capture this formally, we use a relaxed notion of non-interference, in the literature often referred to as generalized non-interference (GNI) [34]. We can, informally, express GNI in a temporal logic as follows: ∀π.∀π .∃π . o π = o π ∧ l π = l π ∧ h π = h π This property requires that for any two traces π, π , there exists some trace π that, globally, agrees with the low-security inputs and outputs on π but the high-security inputs on π . Phrased differently, any observation on the lowsecurity input-output behavior is compatible with every possible high-security input. The program in Figure 1 satisfies GNI. Crucially, GNI is no longer a hypersafety property (and, in particular, no k-safety property for any k) as it requires a combination of universal and existential quantification. Instead, GNI falls in the general class of ∀ * ∃ * -safety properties. Concretely, a ∀ k ∃ l -safety property (using k universal and l existential quantifiers) stipulates that for any k traces, there exist l traces such that the resulting k + l traces do not interact badly. k-safety properties are the special case where l = 0. We study the verification of such properties in infinite-state systems arising, e.g., in software. In contrast to k-safety, where a broad range of methods has been developed [38,26,39,10,40], no method for the automated verification of temporal ∀ * ∃ * properties in infinite-state systems exists (we discuss related approaches in Section 8).

Verification Beyond k-Safety
Our novel verification method is based on a game-based reading of existential quantification combined with the search for a program reduction. The gamebased reading of existential quantification instantiates existential trace quantification with an explicit strategy and constitutes the first practicable method for the verification of ∀ * ∃ * -properties in finite-state systems [22]. Program reductions are a well-established technique to align executions of independent program fragments (such as the individual program copies in a self-composition) to obtain proofs with easier invariants [33,38,26].
So far, both techniques are limited to their respective domain, i.e., the gamebased approach has only been applied to finite-state systems and synchronous specifications, and reductions have (mostly) been used for the verification of ksafety. We combine both techniques yielding an effective (and first) verification technique for hyperproperties beyond k-safety in infinite-state systems arising in software. Notably, our search for reduction and strategy-based instantiation of existential quantification is mutually dependent, i.e., a particular strategy might depend on a particular reduction and vice versa.

Contributions and Structure
The starting point of our work is a new temporal logic called Observation-based HyperLTL (OHyperLTL for short). Our logic extends the existing hyperlogic HyperLTL [20] with capabilities to reason about asynchronous properties (i.e., properties where the individual traces are traversed at different speeds), and to specify properties using assertions from arbitrary background theories (to reason about the infinite domains encountered in software) (Section 4).
To automatically verify ∀ k ∃ l OHyperLTL properties, we combine program reductions with a strategy-based instantiation of existential quantification, both in the context of a fixed predicate abstraction. To facilitate this combination, we first present a game-based approach that automates the search for a reduction. Concretely, we construct an abstract game where a winning strategy for the verifier directly corresponds to a reduction with accompanying proof. As a side product, our game-based interpretation simplifies the search for a reduction in a given predicate abstraction as, e.g., studied by Shemer et al. [38] (Section 5).
Our strategic (game-based) view on reductions allows us to combine them with a game-based instantiation of existential quantification. Here, we view the existentially quantified traces as being constructed by a strategy that, iteratively, reacts to the universally quantified traces. As we phrase both the search for a reduction and the search for existentially quantified traces as a game, we can frame the search for both as a combined abstract game. We prove the soundness of our approach, i.e., a winning strategy for the verifier constitutes both a strategy for the existentially quantified traces and accompanying (mutually dependent) reduction. Despite its finite nature, constructing the abstract game is expensive as it involves many SMT queries. We propose an inner refinement loop that determines the winner of the game (without constructing it explicitly) by computing iterative approximations (Section 6).
We have implemented our verification approach in a prototype tool called HyPA (short for Hyperproperty Verification with Predicate Abstraction) and evaluate HyPA on k-safety properties (that can already be handled by existing methods) and on ∀ * ∃ * -safety benchmarks that cannot be handled by any existing tool (Section 7).
Contributions. In short, our contributions include the following: -We propose a temporal hyperlogic that can specify asynchronous hyperproperties in infinite-state systems; -We propose a game-based interpretation of a reduction (improving and simplifying previous methods for k-safety [38]); -We combine a strategy-based instantiation of existentially quantified traces with the search for a reduction. This yields a flexible (and first) method for the verification of temporal ∀ * ∃ * properties. We propose an iterative method to solve the abstract game that avoids an expensive explicit construction; -We provide and evaluate a prototype implementation of our method.

Overview: Reductions and Quantification as a Game
Our verification approach hinges on the observation that we can express both a reduction and existential trace quantification as a game. In this section, we provide an overview of our game-based interpretations. We begin by outlining our game-based reading of a reduction (illustrating this in the simpler case of ksafety) in Section 2.1 and then extend this to include a game-based interpretation of existential quantification in Section 2.2.

Reductions as a Game
Consider the two programs in Figure 2 and the specification that both programs produce the same output (on initially identical values for x). We can formalize this in our logic OHyperLTL (formally defined in Section 4) as follows: The property states that for all traces π 1 in P1 and π 2 in P2 the LTL specification (x π1 = x π2 ) → (x π1 = x π2 ) holds (where x π refers to the value of x on trace π). Additionally, the observation formula pc = 2 marks the positions at which the LTL property is evaluated: We only observe a trace at steps where pc = 2 (i.e., where the program counter is at the output position).
The verification of our property involves reasoning about two copies of our system (in this case, one of P1 and one of P2) on disjoint state spaces. Consequently, we can interleave the statements of both programs (between two observation points) without affecting the behavior of the individual copies. We refer to each interleaving of both copies as a reduction. The choice of a reduction drastically influences the complexity of the needed invariants [26,38,33]. Given an initial abstraction of the system [29,38], we aim to discover a suitable reduction automatically. Our first observation is that we can phrase the search for a reduction as a game as follows: In each step, the verifier decides on a scheduling (i.e., a non-empty subset M ⊆ {1, 2}) that indicates which of the copies should take a step (i.e., i ∈ M iff copy i should make a program step). Afterward, the refuter can choose an abstract successor state compatible with that scheduling, after which the process repeats. This naturally defines a finite-state two-player safety game that we can solve efficiently. 1 If the verifier wins, a winning strategy (c) Winning strategy for the verifier.  Figure 2c a possible winning strategy for the verifier is given. Each abstract state contains the value of the program counter of both copies (given as the pair at the top) and the predicates that hold in that state. For sake of readability we omit the trace variables and write, e.g., x 1 for x π1 . We mark the initial state with an incoming arrow. The outer label at each state gives the scheduling M ⊆ {1, 2} chosen by the strategy in that state.
directly corresponds to a reduction and accompanying inductive invariant for the safety property within the given abstraction.
For our example, we give (parts of) a possible winning strategy in Figure 2c. In each abstract state, the strategy chooses a scheduling (written next to the state), and all abstract states compatible with that scheduling are listed as successors. Note that whenever the program counter is (2, 2) (i.e., both programs are at their output position), it holds that x 1 = x 2 (as required). The example strategy schedules in lock-step for the most part (by choosing M = {1, 2}) but lets P1 take the inner loop twice, thereby maintaining the linear invariants x 1 = x 2 and y 1 = 2y 2 . In particular, the resulting reduction is property-based [38] as the scheduling is based on the current (abstract) state. Note that the program cannot be verified with only linear invariants in a sequential or parallel (lock-step) reduction.

Beyond k-Safety: Quantification as a Game
We build upon this game-based interpretation of a reduction to move beyond k-safety. As a second example, consider the two programs Q1 and Q2 in Figure 3, where τ denotes a nondeterministic choice of type τ ∈ {N, B}. We wish to check that Q1 refines Q2, i.e., all output behavior of Q1 is also possible in Q2. We can express this in our logic as follows: The property states that for every trace π 1 in Q1 there exists a trace π 2 in Q2 that outputs the same value. The quantifiers range over infinite traces of variable assignments (with infinite domains), making a direct verification of the 1: repeat 2: print(a) 3: x ← N 4: while B do 5: x ← x + 1 6: y ← x 7: while y > 0 do 8: a ← a + x 9: (c) Winning strategy for the verifier. quantifier alternation challenging. In contrast to alternation-free formulas, we cannot reduce the verification to verification on a self composition [8,27]. Instead, we adopt (yet another) game-based interpretation by viewing the existentially quantified traces as being resolved by a strategy (called the witness strategy) [22]. That is, instead of trying to find a witness traces π 2 in Q2 when given the entire trace π 1 , we interpret the ∀∃ property as a game between verifier and refuter. The refuter moves through the state space of Q1 (thereby producing a trace π 1 ), and the verifier reacts to each move by choosing a successor in the state space of Q2 (thereby producing a trace π 2 ). If the verifier can assure that the resulting traces π 1 , π 2 satisfy (a π1 = a π2 ), the ∀∃ property holds. However, this game-based interpretation fails in many instances. There might exist a witness trace π 2 , but the trace cannot be produced by a witness strategy as it requires knowledge of future moves of the refuter. Let us discuss this on the example programs in Figure 3. A simple (informal) solution to construct a witness trace π 2 (when given the entire π 1 ) would be to guarantee that in Q2:4 (meaning location 4 of Q2) and line Q1:6 the value of x in both programs agrees (i.e., x 1 = x 2 holds) and then simply resolve the nondeterminism at Q2:6 with 0. However, to follow this idea, the witness strategy for the verifier, when at Q2:3, would need to know the future value of x 1 when Q1 is at location Q1:6.
Our insight in this paper is that we can turn the strategy-based interpretation of the witness trace π 2 into a useful verification method by combining it with a program reduction. As we express both searches strategically, we can phrase the combined search as a combined game. In particular, both the reduction and the witness strategy are controlled by the verifier and can thus collaborate. In the resulting game, the verifier chooses a scheduling (as in Section 2.1) and, additionally, whenever the existentially quantified copy is scheduled, the verifier also decides on the successor state of that copy. We depict a possible winning strategy in Figure 3c. This strategy formalizes the interplay of reduction and witness strategy. Initially, the verifier only schedules {1} until Q1 has reached program location Q1:6 (at which point the value of x is fixed). Only then does the verifier schedule {2}, at which point the witness strategy can decide on a successor state for Q2. In our case, the strategy chooses a value for x such that x 1 = x 2 holds. As we work in an abstraction of the actual system, we formalize this by restricting the abstract successor states. In particular, in state α 7 the verifier schedules {2} and simultaneously restricts the successors to {α 8 } (i.e., the abstract state where x 1 = x 2 holds), even though abstract state [(6, 4), a 1 = a 2 , x 1 = x 2 ] is also a valid successors under scheduling {2}. We formalize when a restriction is valid in Section 6. The resulting strategy is winning and therefore denotes both a reduction and witness strategy for the existentially quantified copy. Importantly, both reduction and witness strategy are mutually dependent. Our tool HyPA is able to verify both properties (in Figure 2 and Figure 3) in a matter of a few seconds (cf. Section 7).

Preliminaries
We begin by introducing basic preliminaries, including our basic model of computation and background on (finite-state) safety games.
Symbolic Transition Systems. We assume some fixed underlying first-order theory. A symbolic transition system (STS) is a tuple T = (X, init, step) where X is a finite set of variables (possibly sorted), init is a formula over X describing all initial states, and step is a formula over X X (where X := {x | x ∈ X} is the set of primed variables) describing the transitions of the system. A concrete state µ in T is an assignment to the variables in X. We write µ for the assignment over X given by µ (x ) := µ(x). A trace in T is an infinite sequence of assignment µ 0 µ 1 · · · such that µ 0 |= init and for every i ∈ N, µ i µ i+1 |= step. We write Traces(T ) for the set of all traces in T . We can naturally interpret programs as STS by making the program counter explicit.
Formula Transformations. For the remainder of this paper, we fix the set of system variables X. We also fix a finite set of trace variables V = {π 1 , . . . , π k }. For a trace variable π ∈ V we define X π := {x π | x ∈ X} and write X for X π1 ∪ · · · ∪ X π k . For a formula θ over X, we define θ π as the formula over X π obtained by replacing every variable x with x π . Similarly, we define k fresh disjoint copies X = X π1 ∪ · · · ∪ X π k (where X π := {x π | x ∈ X}). For a formula θ over X, we define θ as the formula over X obtained by replacing every variable x π with x π . Safety Games. A safety game is a tuple G = (S SAFE , S REACH , S 0 , T, B) where S = S SAFE s REACH is a set of game states, S 0 ⊆ S a set of initial states, T ⊆ S × S a transition relation, and B ⊆ S a set of bad states. We assume that for every s ∈ S there exists at least one s with (s, s ) ∈ T . States in S SAFE are controlled by player SAFE and those in S REACH by player REACH. A play is an infinite sequence of states s 0 s 1 · · · such that s 0 ∈ S 0 , and (s i , s i+1 ) ∈ T for every i ∈ N. A positional strategy σ for player p ∈ {SAFE, REACH} is a function σ : S p → S such that (s, σ(s)) ∈ T for every s ∈ S p . A play s 0 s 1 · · · is compatible with strategy σ for player p if s i+1 = σ(s i ) whenever s i ∈ S p . The safety player wins G if there is a strategy σ for SAFE such that all σ-compatible plays never visit a state in B. In particular, SAFE needs to win from all initial states.

Observation-based HyperLTL
In this section, we present OHyperLTL (short for observation-based HyperLTL). Our logic builds upon HyperLTL [20], which itself extends linear-time temporal logic (LTL) with explicit trace quantification. In OHyperLTL, we include predicates from the background theory (to reason about infinite variable domains) and explicit observations (to express asynchronous properties). Formulas in OHyperLTL are given by the following grammar: 2 Here π ∈ V is a trace variable, θ is a formula over X, and ξ is a formula over X (called the observation formula). For ease of notation, we assume that all variables in V occur in the quantifier prefix exactly once. We use the standard Boolean connectives ∧, →, ↔, and constants , ⊥, as well as the derived LTL operators eventually φ := U φ, and globally φ := ¬ ¬φ.

Semantics.
A trace t is an infinite sequence µ 0 µ 1 · · · of assignments to X. For i ∈ N, we write t(i) to denote the ith value in t. A trace assignment Π is a partial mapping of trace variables in V to traces. Given a trace assignment Π and i ∈ N, we define Π(i) to be the assignment to X given by Π(i)(x π ) := Π(π)(i)(x), i..e, the value of x π is the value of x on the trace assigned to π. For the LTL body of an OHyperLTL formula, we define: The distinctive feature of OHyperLTL over HyperLTL are the explicit observations. Given an observation formula ξ and trace t, we say that ξ is a valid observation on t (written valid (t, ξ)) if there are infinitely many i ∈ N such that t(i) |= ξ. If valid (t, ξ) holds, we write t ξ for the trace obtained by projecting on those positions i where t(i) |= ξ, i.e., t ξ (i) := t(j) where j is the ith index that satisfies ξ. Given a set of traces T we resolve trace quantification as follows: The semantics mostly agrees with that of HyperLTL [20] but projects each trace to the positions where the observation holds. Given an STS T and OHyperLTL formula ϕ, we write T |= ϕ if ∅ |= Traces(T ) ϕ where ∅ is the empty assignment.
The Power of Observations. The explicit observations in OHyperLTL facilitate the specification of asynchronous hyperproperties, i.e., properties where traces are traversed at different speeds. For the example in Section 2.1, the explicit observations allow us to compare the output of both programs even though the actual step at which the output occurs (in a synchronous semantics) differs between both programs (as P1 takes the inner loop twice as often as P2). As the observations are part of the specification, we can model a broad spectrum of properties ranging, e.g., from timing-insensitive properties (by placing observations only at output locations) to timing-sensitive specifications [28] (by placing observations at closer intervals). Functional (opposed to temporal) k-safety properties specified by pre-and postcondition [10,38,40] can easily be encoded as ∀ k -OHyperLTL properties by placing observations at the start and end of each program. By setting ξ = , i.e., observing every step, we can express synchronous properties. OHyperLTL thus subsumes HyperLTL.
Finite-State Model Checking. Many mechanisms used to express asynchronous hyperproperties render finite-state model checking undecidable [30,16,9]. In contrast, the simple mechanism used in OHyperLTL maintains decidable finite-state model checking. Detailed proofs can be found in the appendix.
Theorem 1. Assume an STS T with finite variable domains and decidable background theory and an OHyperLTL formula ϕ. It is decidable if T |= ϕ.
Proof sketch. Under the assumptions, we can view T as an explicit (instead of symbolic) finite-state transition system. Given an observation formula ξ we can effectively compute an explicit finite-state system T such that This reduces OHyperLTL model checking on T to HyperLTL model checking on T , which is decidable [27].
Note that for infinite-state (symbolic) systems, we cannot effectively compute T as in the proof of Theorem 1. In fact, there may not even exist a system T with the desired property that is expressible in the same background theory.
The finite-state result in Theorem 1 is of little relevance for the present paper. Nevertheless, it indicates that our logic is well suited for verification of infinite-state (software) systems as the (inevitable) undecidability stems from the infinite domains in software programs and not already from the logic itself.
Safety. In this paper, we assume that the hyperproperty is temporally safe [12], i.e., the temporal body of any OHyperLTL formula denotes a safety property.
Note that, as we support quantifier alternation, we can still express hyperliveness properties [21,22]. For example, GNI is both temporally safe and hyperliveness. We model the body of a formula by a symbolic safety automaton [23], which is a tuple A = (Q, q 0 , δ, B) where Q is a finite set of states, q 0 ∈ Q the initial state, B ⊆ Q a set of bad-states, and δ a finite set of automaton edges of the form (q, θ, q ) where q, q ∈ Q are states and θ is a formula over X. Given a trace t over assignments to X, a run of A on t is an infinite sequence of states q 0 q 1 · · · (starting in q 0 ) such that for every i, there exists an edge ( The automaton is deterministic if for every q ∈ Q and every assignments µ to X, there exists exactly one edge (q, θ, q ) ∈ δ with µ |= θ.

Reductions as a Game
After having defined our temporal logic, we turn our attention to the automatic verification of OHyperLTL formulas on STSs. In this section, we begin by formalizing our game-based interpretation of a reduction. To illustrate this, we consider ∀ k OHyperLTL formulas, which, as the body of the formula is a safety property, always denote k-safety properties.
Predicate Abstraction. Our search for a reduction is based in the scope of a fixed predicate abstraction [29,32], i.e., we abstract our system by keeping track of the truth value of a few selected predicates that (ideally) identify properties that are relevant to prove the property in question. Let T = (X, init, step) be an STS and let ϕ = ∀π 1 : ξ 1 . . . ∀π k : ξ k . φ be the (k-safety) OHyperLTL we wish to verify. Let A φ = (Q φ , q φ,0 , δ φ , B φ ) be a deterministic safety automaton for φ. A relational predicate p is a formula over X that identifies a property of the combined state space of k system copies. Let P = {p 1 , . . . , p n } be a finite set of relational predicates. We say a formula over X is expressible in P if it is equivalent to a boolean combination of the predicates in P. We assume that all edge formulas in the automaton A φ , and formulas init πi and (ξ i ) πi for π i ∈ V are expressible in P. Note that we can always add missing predicates to P.
Given the set of predicates P, the state-space of the abstraction w.r.t. P is given by B n , where for each abstract stateŝ ∈ B n , the ith positionŝ[i] ∈ B tracks whether or not predicate p i holds. To simplify notation, we write ite(b, θ, θ ) to be formula θ if b = , and θ otherwise. For each abstract stateŝ ∈ B n , we define ŝ : , ŝ is a formula over X that captures all concrete states that are abstracted toŝ. To incorporate reductions in our abstraction, we parametrize the abstract transition relation by a scheduling M ⊆ {π 1 , . . . , π k }. We lift the step formula from T by defining That is all copies in M take a step while all other copies remain unchanged. Given two abstract statesŝ 1 ,ŝ 2 we say thatŝ 2 is an M -successor ofŝ 1 , written , we can transition fromŝ 1 tô s 2 by only progressing the copies in M .
For an abstract stateŝ, we define obs(ŝ) ∈ B k as the boolean vector that indicates which copy (of π 1 , . . . , π k ) is currently at an observation point, i.e., obs(ŝ)[i] = iff ŝ ∧(ξ i ) πi is satisfiable. Note that as (ξ i ) πi is, by assumption, expressible in P, either all or none of the concrete states in ŝ satisfy (ξ i ) πi .
Game Construction. Building on the parametrized abstract transition relation, we can construct a (finite-state) safety game where winning strategies for the verifier correspond to valid reductions with accompanying proofs. The nodes in our game have two forms: Either they are of the form (ŝ, q, b) whereŝ ∈ B n is an abstract state, q ∈ Q φ a state of the safety automaton, and b ∈ B k a boolean vector indicating which copy has moved since the last automaton step; Or of the form (ŝ, q, b, M ) whereŝ, q, and b are as before and ∅ = M ⊆ {π 1 , . . . , π k } is a scheduling. The initial states are all states (ŝ, q φ,0 , k ) where ŝ ∧ k i=1 init πi is satisfiable (recall that init πi is expressible in P). We mark a state (ŝ, q, b) or (ŝ, q, b, M ) as losing iff q ∈ B φ . For automaton state q ∈ Q φ and abstract stateŝ, we define δ φ (q,ŝ) as the unique state q such that there is an edge (q, θ, q ) ∈ δ φ such that ŝ ∧θ is satisfiable. Uniqueness follows from the assumption that A φ is deterministic and all edge formulas are expressible in P. The transition relation of our game is given by the following rules: In rule (1), we select any scheduling that schedules only copies that have not reached an observation point or have not moved since the last automaton step.
In particular, we cannot schedule any copy that has moved and already reached an observation point. In rule (2), all copies reached an observation point and have moved since the last update (i.e., b = k ) so we progress the automaton and reset b. Lastly, in rule (3), we select an M -successor ofŝ and update b for all copies that take part in the step. In our game, player SAFE takes the role of the verifier, and player REACH that of the refuter. It is the safety player's responsibility to select a scheduling in each step, so we assign nodes of the form (ŝ, q, b) to SAFE. Nodes of the form (ŝ, q, b, M ) are controlled by REACH who can choose an abstract M -successor. Let G ∀ (T ,ϕ,P) be the resulting (finite-state) safety game. A winning strategy for SAFE in G ∀ (T ,ϕ,P) picks, in each abstract state, a valid scheduling that prevents a visit to a losing state. We can thus show: Theorem 2. If player SAFE wins G ∀ (T ,ϕ,P) , then T |= ϕ. Proof sketch. Assume σ is a winning strategy for SAFE in G ∀ (T ,ϕ,P) . Let t 1 , . . . , t k ∈ Traces(T ) be arbitrary. We, iteratively, construct stuttered versions t 1 , . . . , t k of t 1 , . . . , t k by querying σ on abstracted prefixes of t 1 , . . . , t k : Whenever σ schedules copy i we take a proper step on t i ; otherwise we stutter. By construction of G ∀ (T ,ϕ,P) the stuttered traces t 1 , . . . , t k align at observation points. In particular, we have [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ iff [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ. Moreover, the sequence of abstract states in G ∀ (T ,ϕ,P) forms an abstraction of t 1 , . . . , t k and shows that A φ cannot reach a bad state when reading t 1 ξ1 , . . . , t k ξ k (as σ is winning). This already shows that [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ and thus [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ.
As this holds for all traces t 1 , . . . , t k ∈ Traces(T ), we get T |= ϕ as required.
Game Construction and Complexity. If the background theory is decidable, G ∀ (T ,ϕ,P) can be constructed effectively using at most 2 |P|+1 · 2 k queries to an SMT solver. Checking if SAFE wins G ∀ (T ,ϕ,P) can be done with a simple fixpoint computation of the attractor in linear time.
Our game-based method of finding a reduction in a given abstraction is closely related to the notation of a property-directed self-composition [38]. The previously only known algorithm for finding such a reduction is based on an optimized enumeration [38], which, in the worst case, requires O(2 |P|+1 · 2 k ) many enumerations. Our worst-case complexity thus matches the bounds inferred by [38], but avoids the explicit enumeration of reductions (and the concomitant repeated construction of the abstract state-space) and is, as we believe, conceptually simpler to comprehend. Moreover, our game-based technique is the key stepping stone for extending our method beyond k-safety in Section 6.

Verification Beyond k-Safety
Building on the game-based interpretation of a reduction, we extend our verification beyond ∀ * properties to support ∀ * ∃ * properties. We accomplish this by combining the game-based reading of a reduction (as discussed in the previous section) with a game-based reading of existential quantification. For the remainder of this section, fix an STS T = (X, init, step) and let ϕ = ∀π 1 : ξ 1 . . . ∀π l : ξ l .∃π l+1 : ξ l+1 . . . ∃π k : ξ k . φ be the OHyperLTL formula we wish to check, i.e., we universally quantify over l traces followed by an existential quantification over k − l traces. We assume that for every existential quantification ∃π i : ξ i occurring in ϕ, valid (t, ξ i ) holds for every t ∈ Traces(T ) (we discuss this later in Remark 1).

Existential Trace Quantification as a Game
The idea of a game-based verification of ∀ * ∃ * properties is to consider a ∀ * ∃ *property as a game between verifier and refuter [22]. The refuter controls the l universally quantified traces by moving through l copies of the system (thereby producing traces π 1 , . . . , π l ) and the verifier reacts by, incrementally, moving through k − l copies of the system (thereby producing traces π l+1 , . . . , π k ). If the verifier has a strategy that ensures that the resulting traces satisfy φ, T |= ϕ holds. We call such a strategy for the verifier a witness strategy.
We combine this game-based reading of existential quantification with our game-based interpretation of a reduction by, additionally, letting the verifier control the scheduling of the system. When played on the concrete state-space of T the game proceeds in three stages as follows: 1) The verifier selects a valid scheduling M ⊆ {π 1 , . . . , π k }; 2) The refuter selects successor states for all universally quantified copies by fixing an assignment to X π1 , . . . , X π l (only moving copies scheduled by M ); 3) The verifier reacts by choosing successor states for the existentially quantified copies by fixing an assignment to X π l+1 , . . . , X π k (again, only moving copies scheduled by M ). Afterward, the process repeats.
As we work within a fixed abstraction of T , the verifier can, however, not choose concrete successor states directly but only work in the precision captured by the abstraction. Following the general scheme of abstract games, we, therefore, underapproximate the moves available to the verifier [2]. Formally, we abstract the three-stage game outlined before (which was played at the level of concrete states) to a simpler abstract game (consisting of only two stages). In the first stage, the verifier selects both a scheduling M and a restriction on the set of abstract successor states, i.e., a set of abstract states A. In the second stage, the refuter cannot choose any abstract successor state (any M -successor in the terminology from Section 5), but only successors contained in the restriction A.
To guarantee the soundness of this approach, we ensure that the verifier can only pick restrictions that are valid, i.e., restrictions that underapproximate the possibilities of the verifier on the level of concrete states.
Game Construction. We modify our game from Section 5 as follows. States are either of the form (ŝ, q, b) (as in Section 5) or of the form (ŝ, q, b, M, A) whereŝ, q, b, and M are as in Section 5, and A ⊆ B n is a subset of abstract states (the restriction). To reflect the restriction, we modify transition rules (1) and (3). Rule (2) remains unchanged.
In rule (1), the safety player (who, again, takes the role of the verifier) selects both a scheduling M and a restriction A such that validResŝ ,M A holds (which we define later). The reachability player (who takes the role of the refuter) can, in rule (3), select any successor contained in A.
Valid Restriction. The above game construction depends on the definition of validResŝ ,M A . Intuitively, A is a valid restriction if it underapproximates the possibilities of a witness strategy that can pick concrete successor states for all existentially quantified traces. That is, for every concrete state inŝ, a witness strategy (on the level of concrete states) can guarantee a move to a concrete state that is abstracted to an abstract state within A. Formally we define validResŝ ,M A as follows: It expresses that for all concrete states in ŝ (assignments to {X πi } k i=1 ) and for all concrete successor states for the universally quantified copies (assignments to {X πi } l i=1 ), there exist successor states for the existentially quantified copies ({X πi } k i=l+1 ) such that one of the abstract states in A is reached.
where X = {a, x, y}. Here we assume that step := a = a ∧ y = y is the update performed on instruction x ← N from Q2:3 to Q2:4. The above formula is valid.
Correctness. Call the resulting game G ∀∃ (T ,ϕ,P) . The game combines the search for a reduction with that of a witness strategy (both within the precision captured by P). 3 We can show: Theorem 3. If player SAFE wins G ∀∃ (T ,ϕ,P) , then T |= ϕ.
Proof sketch. Let σ be a winning strategy for SAFE in G ∀∃ (T ,ϕ,P) . Let t 1 , . . . , t l ∈ Traces(T ) be arbitrary. We use σ to incrementally construct witness traces t l+1 , . . . , t k by querying σ. In every abstract stateŝ, σ selects a scheduling M and a restriction A such that validResŝ ,M A holds. We plug the current concrete state (reached in our construction of t l+1 , . . . , t k ) into the universal quantification of validResŝ ,M A and get (concrete) witnesses for the existential quantification that, by definition of validResŝ ,M A , are valid successors for the existentially quantified copies in T .

Remark 1.
Recall that we assume that for every existential quantification ∃π i : ξ i occurring in ϕ and all t ∈ Traces(T ), valid (t, ξ i ) holds. This is important to ensure that the safety player (the verifier) cannot avoid observation points forever. We could drop this assumption by strengthening the winning condition in G ∀∃ (T ,ϕ,P) and explicitly state that, in order to win, SAFE needs to visit observations points on existentially quantified traces infinitely many times. Clairvoyance vs. Abstraction. The cooperation between reduction (the ability of the verifier to select schedulings) and witness strategy (the ability to select restrictions on the successor) can be seen as a limited form of prophecy [1,14]. By first scheduling the universal copies, the witness strategy can peek at future moves before committing to a successor state, as we e.g., saw in Figure 3. The "theoretically optimal" reduction is thus a sequential one that first schedules only the universally quantified traces (until an observation point is reached) and thereby provides maximal information for the witness strategy. However, in the context of a fixed abstraction, this reduction is not always optimal. For example, in Figure 3 the strategy schedules the loop in lock-step which is crucial for generating a proof with simple (linear) invariants. In particular, Figure 3 does not admit a witness strategy in the lock-step reduction and does not admit a proof with linear invariants in a sequential reduction. Our verification framework, therefore, strikes a delicate balance between clairvoyance needed by the witness strategy and precision captured in the abstraction, further emphasizing why the searches for reduction and witness strategy need to be mutually dependent. Constructing the game graph of G ∀∃ (T ,ϕ,P) requires the identification of all valid restrictions (of which there are exponentially many in the number of abstract states and thus double exponentially many in the number of predicates) each of which requires to solve a quantified SMT query. We propose a more effective algorithm that solves G ∀∃ (T ,ϕ,P) without constructing it explicitly. Instead, we iteratively refine an abstractionG of G ∀∃ (T ,ϕ,P) . Our method hinges on the following easy observation: SAFE inG we check if all restrictions chosen by σ are valid. If this is the case, σ is also winning for G ∀∃ (T ,ϕ,P) and we can apply Theorem 3. If we find an invalid restriction (ŝ, M, A) used by σ, we refineG by removing not only the restriction (ŝ, M, A) but all (ŝ, M, A ) with A ⊆ A (which is justified by Lemma 1). The algorithm is sketched in Algorithm 1. The subroutine Restrictions(σ) returns all restrictions used by σ, i.e., all tuples (ŝ, M, A) such that σ uses an edge (ŝ, q, b) (ŝ, q, b, M, A) for some q, b. Remove (G, (ŝ, M, A )) removes fromG all edges of the form (ŝ, q, b) (ŝ, q, b, M, A ) for some q, b, and Solve solves a finite-state safety game. To improve the algorithm further, in line 4 we always compute a maximal safety strategy, i.e., a strategy that selects maximal restrictions (w.r.t. ⊆) and therefore allows us to eliminate many invalid restrictions fromG simultaneously. For safety games, there always exists such a maximal winning strategy (see e.g. [11]). Note that whileG is large, solving this finite-state game can be done very efficiently. The running time of solving G ∀∃ (T ,ϕ,P) is dominated by the SMT queries of which our refinement loop, in practice, requires very few. Table 1: Evaluation of HyPA on ksafety instances. We give the size of the abstract game-space (Size), the time taken to compute the abstraction (t abs ), and the overall time taken by HyPA (t). Times are given in seconds.

Instance
Size t abs t When combining Theorem 3 and our iterative solver from Section 6.2 we obtain an algorithm to verify ∀ * ∃ * -safety properties within a given abstraction. We have implemented a prototype of our method in a tool we call HyPA. We use Z3 [35] to discharge SMT queries. The input of our tool is provided as an arbitrary STS in the SMTLIB format [5], making it language independent. In our programs, we make the program counter explicit, allowing us to track predicates locally [31].
Evaluation for k-Safety. As a special case of ∀ * ∃ * properties, HyPA is also applicable to k-safety verification. We collected an exemplifying suite of programs and k-safety properties from the literature [38,26,40,39,40] and manually translated them into STS (this can be automated easily). The results are given in Table 1. As done by Shemer et al. [38], we already provide a set of predicates that is sufficient for some reduction (but not necessarily the lockstep or sequential one), the search for which is then automated by HyPA. Our results Table 2: Evaluation of HyPA on ∀ * ∃ * -safety verification instances. We give the size and construction time of the initial abstraction (Size and t abs ). For both the direct (explicit) and lazy (Algorithm 1) solver we give the time to construct (and solve) the game (t solve ) and the overall time (t = t abs + t solve ). For the lazy solver we, additionally, give the number of refinement iterations (#Ref). Times are given in seconds. TO indicates a timeout after 5 minutes. show the game-based search for a reduction can verify interesting k-safety properties from the literature. We also note that, currently, the vast majority of time is spent on the construction of the abstract system. If we would move to a fixed language, the computation time of the initial abstraction could be reduced by using existing (heavily optimized) abstraction tools [31,17].
Evaluation Beyond k-Safety. The main novelty of HyPA lies in its ability to, for the first time, verify temporal properties beyond k-safety. As none of the existing tools can verify such properties, we compiled a collection of very small example programs and ∀ * ∃ * -safety properties. Additionally, we modified the boolean programs from [13] (where they checked GNI on boolean programs) by including data from infinite domains. The properties we checked range from refinement properties for compiler optimizations, over general refinement of nondeterministic programs, to generalized non-interference. Verification often requires a non-trivial combination of reduction and witness strategy (as the reduction must, e.g., compensate for branches of different lengths). As before, we provide a set of predicates and let HyPA automatically search for a witness strategy with accompanying reduction. We list the results in Table 2. To highlight the effectiveness of our inner refinement loop, we apply both a direct (explicit) construction of G ∀∃ (T ,ϕ,P) and the lazy (iterative) solver in Algorithm 1. Our lazy solver (Algorithm 1) clearly outperforms an explicit construction and is often the only method to solve the game in reasonable time. In particular, we require very few refinement iterations and therefore also few expensive SMT queries. Unsurprisingly, the problem of verifying properties beyond k-safety becomes much more challenging (compared to k-safety verification) as it involves the synthesis of a witness function which is already 2EXPTIME-hard for finite-state systems [36,22]. We emphasize that no other existing tool can verify any of the benchmarks.

Related Work
Asynchronous Hyperproperties. Recently, many logics for the formal specification of asynchronous hyperproperties have been developed [9,30,16,13]. Our logic OHyperLTL is closely related to stuttering HyperLTL (HyperLTL S ) [16]. In HyperLTL S each temporal operator is endowed with a set of temporal formulas Γ and steps where the truth values of all formulas in Γ remain unchanged are ignored during the operator's evaluation. As for most mechanisms used to design asynchronous hyperlogics [9,30,16], finite-state model checking of HyperLTL S is undecidable. By contrast, in OHyperLTL, we always observe the trace at a fixed location, which is key for ensuring decidable finite-state model checking.
k-Safety Verification. The literature on k-safety verification is rich. Many approaches verify k-safety by using a form of self-composition [8,24,19,27] and often employ reductions to obtain compositions that are easier to verify. Our game-based interpretation of a reduction (Section 5) is related to Shemer et al. [38], who study k-safety verification within a given predicate abstraction using an enumeration-based solver (see Section 5 for a discussion). Farzan and Vandikas [26] present a counterexample-guided refinement loop that simultaneously searches for a reduction and a proof. Sousa and Dillig [39] facilitate reductions at the source-code level in program logic. ∀ * ∃ * -Verification. Barthe et al. [7] describe an asymmetric product of the system such that only a subset of the behavior of the second system is preserved, thereby allowing the verification of ∀ * ∃ * properties. Constructing an asymmetric product and verifying its correctness (i.e., showing that the product preserves all behavior of the first, universally quantified, system) is challenging. Unno et al. [40] present a constraint-based approach to verify functional (opposed to temporal) ∀∃ properties in infinite-state systems using an extension of constraint Horn clauses called pfwCHC. The underlying verification approach is orthogonal to ours: pfwCHC allows for a clean separation of the actual verification and verification conditions, whereas our approach combines both. For example, our method can prove the existence of a witness strategy without ever formulating precise constraints on the strategy (which seems challenging). Coenen et al. [22] introduce the game-based reading of existential quantification to verify temporal ∀ * ∃ * properties in a synchronous and finite-state setting. By contrast, our work constitutes the first verification method for temporal ∀ * ∃ * -safety properties in infinite-state systems. The key to our method is a careful integration of reductions which is not possible in a synchronous setting. For finite-state systems (where the abstraction is precise) and synchronous specifications (where we observe every step), our method subsumes the one in [22]. Beutner and Finkbeiner [14] use prophecy variables to ensure that the game-based reading of existential quantification is complete in a finite-state setting. Automatically constructing prophecies for infinite-state systems is interesting future work. Pommellet and Touili [37] study the verification of HyperLTL in infinite-state systems arising from pushdown systems. By contrast, we study verification in infinite-state systems that arise from the infinite variables domains used in software.
Game Solving. Our game-based interpretations are naturally related to infinitestate game solving [25,4,41,15]. State-of-the-art solvers for infinite-state games unroll the game [25], use necessary subgoals to inductively split a game into subgames [4], encode the game as a constraint system [15], and iteratively refine the controllable predecessor operator [41]. We tried to encode our verification approach directly as an infinite-state linear-arithmetic game. However, existing solvers (which, notably, work without a user-provided set of predicates) could not solve the resulting game [25,4]. Our method for encoding the witness strategy using restrictions corresponds to hyper-must edges in general abstract games [2,3]. Our inner refinement loop for solving a game with hyper-must edges without explicitly identifying all edges (Algorithm 1) is thus also applicable in general abstract games.

Conclusion
In this work, we have presented the first verification method for temporal hyperproperties beyond k-safety in infinite-state systems arising in software. Our method is based on a game-based interpretation of reductions and existential quantification and allows for mutual dependence of both. Interesting future directions include the integration of our method in a counter-example guided refinement loop that automatically refines the abstraction and ways to lift the current restriction to temporally safe specifications. Moreover, it is interesting to study if, and to what extent, the numerous other methods developed for ksafety verification of infinite-state systems (apart from reductions) are applicable to the vast landscape of hyperproperties that lies beyond k-safety.
A Proofs for Section 4 Theorem 1. Assume an STS T with finite variable domains and decidable background theory and an OHyperLTL formula ϕ. It is decidable if T |= ϕ.
Proof. Let X be the set of variables in T and let D be the finite domain of the variables (for simplicity we assume that the domain of all variables is the same). An explicit state is then an assignment α : X → D. Let S be the (finite) set of all explicit states. An (explicit) finite-state transition system is a tuple T = (S, S 0 , ρ) where S ⊆ S is a set of explicit states, S 0 ⊆ S is a set of initial states, and ρ ⊆ S × S is the transition relation. The set of traces Traces(T ) is defined as expected.
Under the assumption on T , we can view it as a explicit (and computable) finite-state transition system.
For any observation formula ξ let O(ξ) ⊆ S be the set of all states in which ξ holds (which is computeable).
Given T = (S, S 0 , ρ) and ξ we construct an explicit finite-state transition system T ξ such that Traces( That is all states where ξ holds that are reachable from some state in S 0 in T without passing through another state where ξ holds. Similarly we define ρ := (s, s ) ∈ (O(ξ) ∩ S) 2 | ∃s 0 , s 1 , . . . , s n ∈ S. n ≥ 1 ∧ s 0 = s ∧ s n = s ∧ I.e., there is an edge (s, s ) only if s and s are connected by a path of length at least 1 of unobserved states in S.
As Traces(T ξ ) = { t ξ | t ∈ Traces(T ) ∧ valid (t, ξ)} we can reduce OHyper-LTL model checking on T to HyperLTL model checking on T ξ , which is decidable [27]. Note that a OHyperLTL can use different observation formulas for different quantifiers. In the resulting HyperLTL model checking instance we thus need to resolve different quantifiers on different systems (as, in general, T ξ = T ξ when ξ ≡ ξ ), which is easily done.
The idea is to (implicitly) stutter traces t 1 , . . . , t k between two observation points and compute a pointwise abstraction for these stuttered traces. The stuttering is dictated by σ, i.e., we simulate prefixes in the game and query σ to determine which scheduling to pick. All non-scheduled copies are stuttered. If σ picks a scheduling M , the refuter can pick (in G ∀ (T ,ϕ,P) ) an abstract successor state that is compatible with M . In our simulation we pick the exact abstract states that arises when moving as defined by traces t 1 , . . . , t k , which, by definition of M -successor, is a valid step in the game. This simulation thus gives an abstraction of the stuttered traces which (as σ is winning) avoids a visit to losing states in A φ . In the following we give a more detailed description of this high-level strategy.
Consider the construction in Figure 4. Note that this construction will never finish but allows us to point to key steps showing that [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ. We maintain a concrete state µ i for each copy 1 ≤ i ≤ k, initially set to the initial state according to the fixed traces (line 1). Additionally we maintain a counter c i that tracks at which position of t i the current state is located. It will always be the case that µ i = t i (c i ).
The construction then simulates a play in G ∀ (T ,ϕ,P) using the winning strategy σ to resolve choices made by player SAFE as follows: The simulation starts in state (ŝ, q, b) whereŝ is the initial abstract state based on t 1 , . . . , t k , q the initial state of A φ , and b = k (lines 3 -5). Note that, by construction, this is an initial game state in G ∀ (T ,ϕ,P) . The simulation then continuously advances the play. If b = k and obs(ŝ) = k it applies transition rule (2) in lines 8-12. In particular, the concrete states µ i , counters c i and abstract stateŝ remain unchanged. Otherwise it queries σ on the current state (ŝ, q, b). By transition rule (1), σ can only select a scheduling M (and not change the other state components). In line 14 we write to mark that we do not care about a value. Each trace t i where π i ∈ M then takes a step, c i ← ci for 1 ≤ i ≤ k 10:ŝ ←ŝ 11: q ← δ φ (q,ŝ) 12: b ← ⊥ k 13: else 14: ( , , , M ) ← σ(ŝ, q, b) 15: i.e., we increment c i for those copies, and we update the current state µ i (lines 15 and 16). For non-scheduled copies c i is left unchanged and so is the concrete state µ i . We compute the new abstract stateŝ and update b (as in transition rule (3)).
We can establish a few basic properties: -P1: Whenever the loop is entered, µ 1 ⊗ · · · ⊗ µ k |= ŝ . This follows directly from the construction and the definition of Abstract(·). -P2: Whenever the loop is entered, This follows directly from the construction and the updates performed in lines 15 and 16. -P3: Let (ŝ, q, b) be the current state at the beginning of a loop body and (ŝ , q , b ) after the loop body has executed once. Then, in G ∀ (T ,ϕ,P) , player REACH can force a play from (ŝ, q, b) to (ŝ , q , b ) when the safety player follows strategy σ. In case the conditional in line 7 is taken this is trivial as it directly corresponds to transition rule (2) of the game (the only one that is applicable in that case). The more interesting direction is thus the case where lines 14-19 are executed. So let (ŝ, q, b) be a game state and let µ 1 , . . . , µ k be the current concrete states at the beginning of a loop iteration. By P1 we get that µ 1 ⊗ · · · ⊗ µ k |= ŝ . As (ŝ, q, b, M ) = σ(ŝ, q, b), game state (ŝ, q, b, M ) is reachable under σ (using transition rule (1)). By P2 we have that µ i = t i (c i ) for each i. As each t i is a trace in T we get that t i (c i ) (t i (c i + 1)) |= step. Now µ i = µ i for all non-scheduled copies π i ∈ M , and µ i µ i |= step for all π i ∈ M . It is therefore easy to see thatŝ  So (ŝ, q, b) can take a step to (ŝ, q, b, M ) that is fixed by σ and from (ŝ, q, b, M ) player REACH can move the game to (ŝ , q , b ) as required.
To show that [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ (or equivalently, that the unique run of A φ on t does not visit a bad state) consider the following: We focus on those iterations where the conditional on line 7 is taken. We mark these with a superscript. That is µ j i is the value of µ i when conditional in line 7 was taken for the jth time, and similarly c j i ,ŝ j , and q j (for the values of c i ,ŝ and q, respectively). The important observation is that c j i is now exactly the jth index where t i satisfies the observation formula ξ i , i.e., t i ξi (j) = t i (c j i ). This holds by the design of G ∀ (T ,ϕ,P) , i.e., player SAFE can only schedule copies that have not moved yet or have reached an observation point (via transition rule (1)). We thus get t(j) = t 1 ξ1 (j) ⊗ · · · ⊗ t k ξ k (j) = t 1 (c j 1 ) ⊗ · · · ⊗ t k (c j k ) By P2, t i (c j i ) = µ j i and by P1 µ j 1 ⊗ · · · ⊗ µ j k |= ŝ j . So t(j) |=ŝ j for every j ∈ N, i.e., the sequenceŝ 0 ,ŝ 1 , . . . forms a pointwise abstraction of t. Moreover q 0 , q 1 , . . . is the unique run of A φ onŝ 0 ,ŝ 1 , . . .. As σ is winning and the construction simulates an actual game play allowed by σ (as stated in P3), we get that all states q 0 , q 1 , . . . are not losing. This already concludes that t is accepted by A φ (does not have a rejecting run to a losing state), and so [π 1 → t 1 ξ1 , . . . , π k → t k ξ k ] |= φ as required.
To do so we follow the idea used in the proof of Theorem 2 and query σ to select a scheduling. In addition, we need to actually construct traces t l+1 , . . . , t k . The idea is to use the definition of validResŝ ,M A : In each step, we plug the current concrete states in the universal quantifiers of validResŝ ,M A and obtain a concrete witness for the existentially quantified variables. These correspond exactly to the successor states for the traces t l+1 , . . . , t k . In the following we give a more detailed construction.
For simplicity we assume that there is a unique concrete state µ init that satisfies µ init |= init. Note that this implies that t i (0) = µ init for all 1 ≤ i ≤ l. Letŝ init be the resulting initial abstract state if all k copies are in µ init . Consider the construction in Figure 5 (which, again, does never terminate but allows the construction of witness traces in the limit).
The basic construction is similar to that in the proof of Theorem 2. We again maintainŝ, q, and b and simulate the game using σ to resolve choices made by player SAFE. Additionally, we maintain a trace t i for each l + 1 ≤ i ≤ k, initially set to the length-1-trace consisting of µ init .