On the Separation Question for Tree Languages

We show that the separation property fails for the classes Σ n of the Rabin-Mostowski index hierarchy of alternating automata on infinite trees. This extends our previous result (obtained with Szczepan Hummel) on the failure of the separation property for the class Σ 2 (i.e., for co-Büchi sets). The non-separation result is also adapted to the analogous classes induced by weak alternating automata. To prove our main result, we first consider the Rabin-Mostowski index hierarchy of deterministic automata on infinite words, for which we give a complete answer (generalizing previous results of Selivanov): the separation property holds for Π n and fails for Σ n -classes. The construction invented for words turns out to be useful for trees via a suitable game. It remains open if the separation property holds for all classes Π n of the index hierarchy for tree automata. To give a positive answer it would be enough to show the reduction property of the dual classes—a method well-known in descriptive set theory. We show that it cannot work here, because the reduction property fails for all classes in the index hierarchy.


Introduction
The separation question is whether two disjoint sets A and B can be separated by a set C (i.e., A ⊆ C and B ⊆ C) which is in some sense simpler. Separation is one of the main issues in descriptive set theory. A fundamental result due to Lusin is that two analytic sets can be always separated by a Borel set, but two co-analytic sets in general cannot. The former implies that if a set is simultaneously analytic and co-analytic then it is necessarily Borel, which is the celebrated Suslin Theorem (see, e.g., [8] or [7]).
A well-known fact in automata theory exhibits a similar pattern: if a set of infinite trees as well as its complement are both recognizable by Büchi automata then they are also recognizable by weak alternating automata (weakly recognizable, for short). This result was first proved by Rabin [10] in terms of monadic second-order logic; the automata-theoretic statement was given by Muller, Saoudi, and Schupp [9]. It is not difficult to adapt Rabin's proof to obtain the separation property: any two disjoint Büchi recognizable sets of trees can be separated by a weakly recognizable set (see, e.g., [5]). Quite analogical to the co-analytic case, the separation property fails in general for the dual class of co-Büchi tree languages (i.e., the complements of Büchi recognizable sets). In [5], a pair of such sets is presented that cannot be separated by any Borel set, hence a fortiori by any weakly recognizable set.
A systematic study of the separation property for tree automata has been undertaken by Santocanale and Arnold [11]. They asked if the above-mentioned result of Rabin can be shifted to the higher levels of the index hierarchy of alternating automata with an appropriate generalization of weak recognizability. The question stems naturally from the µ-calculus version of Rabin's result which states that if a tree language is definable both by a Π 2 -term (i.e., with a pattern νµ) and a Σ 2 -term (µν), then it is also definable by an alternation free term, i.e., one in Comp(Π 1 ∪ Σ 1 ) [2]. Somewhat surprisingly, Santocanale and Arnold [11] discovered that the equation Π n ∩ Σ n = Comp(Π n−1 ∪ Σ n−1 ), which amounts to Rabin's result for n = 2, fails for all n ≥ 3. Consequently, it is in general not possible to separate two disjoint sets in the class Σ n by a set in Comp(Π n−1 ∪ Σ n−1 ); similarly for Π n .
There is however another plausible generalisation of Rabin's result suggested by the analogy with descriptive set theory. Letting we may ask if two disjoint sets in a class Σ n can be separated by a set in ∆ n ; a similar question can be stated for Π n . By remarks above, we know that the separation property in this sense holds for Π 2 (Büchi) and fails for Σ 2 (co-Büchi) class, in a perfect analogy with the properties of analytic vs. co-analytic classes in the descriptive set theory 1 .
In the present paper we answer the question negatively for all classes Σ n of the Rabin-Mostowski index hierarchy for alternating automata on infinite trees. (The Σ n -classes correspond to the indices (i, k) with k odd; see the definition below.) By an analogy with the Borel hierarchy [7,8], one is tempted to conjecture that the separation property actually does hold for all classes Π n , but this question seems to be difficult already for n = 3.
To prove our main result, we first study a conceptually simpler case of infinite words and the Rabin-Mostowski index hierarchy of deterministic automata. In this case we give a complete answer: the separation property holds for classes Π n and fails for classes Σ n . The argument is based on a uniform construction of an inseparable pair in each class Σ n . This construction is further used in the case of trees. More specifically, we consider labeled trees whose vertices are divided between two players: Eve and Adam, who wish to form a path in the tree. For a set on infinite words L, we consider the set Win ∃ (L) of those trees where Eve has a strategy to force a path into L. The operation Win ∃ allows us to shift the witness family from words to trees.
It should be noted that in the case of deterministic automata on infinite words, the separation property of the class (1, 2) was proved earlier by Selivanov [12], who also gave a hint [13] how this result can be generalized for all classes Π n .

Index hierarchy
Throughout the paper, ω stands for the set of natural numbers, which we identify with its ordinal type. We also identify a natural number n < ω with the set {0, 1, . . . , n − 1}.
1 However, the classical notation plays a trick here, as the analogy matches the classes Σ 1 1 ∼ Π2 and . . .
We will consider deterministic automata on infinite words and alternating automata on infinite trees. For more background, we refer the reader to a survey by Thomas [14]. A deterministic parity automaton over an input alphabet A can be presented by A = A, Q, q I , Tr, rank , where Q is a finite set of states ranked by the function rank : Q → ω, and Tr : Q×A → Q is a transition function. A run of A on a word u ∈ A ω is a word r ∈ Q ω whose first element r 0 is the initial state q I , and r n+1 = Tr(r n , u n ), for n < ω. It is accepting if the highest rank occurring infinitely often (i.e., lim sup n→∞ rank(r n )) is even. The language L(A) recognized by A consists of those words u ∈ A ω which admit an accepting run. The Rabin-Mostowski index of A is the pair (min rank(Q), max rank(Q)); we may assume without loss of generality that min rank(Q) is 0 or 1. It is useful to partially order the indices as represented on Figure 1. That is, we let (ι, κ) (ι , κ ) if either {ι, . . . , κ} ⊆ {ι , . . . , κ }, or ι = 0, ι = 1, and {ι + 2, . . . , κ + 2} ⊆ {ι , . . . , κ }. We consider the indices (1, κ) and (0, κ − 1) as dual, and let (ι, κ) denote the index dual to (ι, κ). The above ordering induces a hierarchy, that is, if a language L is recognized by an automaton of index (ι, κ) and (ι, κ) (ι , κ ) then L is also recognized by an automaton of index (ι , κ ). Moreover, the hierarchy is strict in the sense that, for any index (ι, κ), there is a language recognized by an automaton of index (ι, κ), but not by any (deterministic) automaton of the dual index (ι, κ) [6,15]. Indeed, the witness can be the parity condition itself: L ι,κ = {u ∈ {ι, . . . , κ} ω : lim sup n→∞ u n is even }. (1) The concept of alternating automaton is best understood via parity games. For the sake of further application, we present them in a more general setting of graph games. A graph game is a perfect information game of two players, say Eve and Adam, where plays may have infinite duration. It can be presented by a tuple Here V ∃ and V ∀ are (disjoint) sets of positions of Eve and Adam, respectively, Move ⊆ V × V is the relation of possible moves, with V = V ∃ ∪ V ∀ , p I ∈ V is a designated initial position, and : V → A is a labelling function, with some alphabet A. These items constitute an arena of the game. Additionally, L ∃ , L ∀ ⊆ A ω are two disjoint sets representing the winning criteria for Eve and Adam, respectively.
The players start a play in the position p I and then move the token according to the relation Move (always to a successor of the current position), thus forming a path in the arena. The move is selected by Eve or Adam, depending on who the owner of the current position is. If a player cannot move, she/he looses. Otherwise, the result of the play is an infinite path v 0 , v 1 , v 2 , . . ., inducing the sequence of labels (v 0 ), (v 1 ), (v 2 ), . . . If this sequence belongs to L ∃ then Eve wins, if it belongs to L ∀ then Adam wins; otherwise there is a draw. We say that Eve wins the game if she has a winning strategy, the similar for Adam.
In the games considered in this paper, we always have L ∀ = A ω − L ∃ , hence a draw will not occur. But it is convenient to consider the winning criteria for both players.
A (full) k-ary tree over a finite alphabet A is a mapping t : k * → A. An alternating parity tree automaton of index (ι, κ) running on such trees can be presented by where Q is a finite set of states with an initial state q I , partitioned into existential states Q ∃ and universal states Q ∀ , δ ⊆ Q × A × {0, 1, . . . , k − 1, ε} × Q is a transition relation, and rank : Q → ω. An input tree t is accepted by A iff Eve has a winning strategy in the parity game Intuitively, the players follow a path in the tree t, additionally annotated by the states. A transition is always selected by the owner of the state. The automaton accepts the tree if Eve can force each such path to be accepting.
The hierarchy of tree languages induced by the indices of alternating parity automata is strict, as showed by Bradfield [4]. An alternative proof of this difficult result was later given [1] based on the Banach Fixed-Point Theorem; Both proofs [1,4] use the same witness family of sets of binary trees defined by parity games. For the sake of further application, we will present this concept in a more general setting.
We consider k-ary trees over an alphabet {∃, ∀} × A. The labels {∃, ∀} are used to partition the nodes of a tree t into positions of Eve and Adam. In the game, the players form a path in t, starting from the root. The next move is selected by Eve or Adam, depending on whether the actual label contains ∃ or ∀. The winning criteria concern the sequence formed by the second components of the labels occurring in the play, which is a word in A ω .
Each language L ⊆ A ω induces two winning criteria: L ∃ = L, and L ∀ = L, which give rise to two games: an L-∃ game, and an L-∀ game. Let us describe formally an L-∃ game over a tree t : k * → {∃, ∀} × A. It is a graph game with the following items: An L-∀ game is defined similarly with the winning criteria L ∀ = L, and L ∃ = L. The set Win ∃ k (L) consists of those trees t, for which Eve has a winning strategy in L-∃-game. The set Win ∀ k (L) consists of those trees t, for which Adam has a winning strategy in L-∀-game. The following can be easily verified.
Fact 1. If a language L of infinite words is recognized by a deterministic automaton of index (ι, κ) then both languages Win ∃ k (L) and Win ∀ k (L) can be recognized by an alternating 2 tree automaton of index (ι, κ).
A family witnessing the strictness of the index hierarchy of alternating tree automata [1,4] consists of the sets of binary trees W i,k , which can be presented by W i,k = Win ∃ 2 (L i,k ). The following Σ/Π terminology for the index hierarchy,motivated by the connection with the µ-calculus (see, e.g., [3]); will be convenient to handle dualities. For each m ≥ 1, we consider two indices: (1, m) and (0, m − 1), and associate the symbol Σ m with this index whose maximum is odd, and Π m with the one whose maximum is even. For example, We will then refer to an automaton of index (ι, κ) as to Σ m or Π m -automaton with an appropriate m.

Deterministic hierarchy over words
In this section we investigate the index hierarchy for deterministic automata on infinite words. A language of infinite words is in the class Σ m if it is recognized by a deterministic Σ m -automaton; similarly for Π m . A language is in the class ∆ m if it is simultaneously Σ m and Π m . We show that the separation property holds for classes Π m and fails for Σ m , for m ≥ 2. In fact, both properties will follow from a single construction (parametrized by m).
Note that we do not consider the classes Σ 1 and Π 1 , which are uninteresting from the point of view of the separation property 3 .
For m ≥ 2, we fix an alphabet Note that max m π is always even. Let I m ⊆ m ω π be the set of infinite words where max m π occurs infinitely often. Let K m be a superset of I m consisting of the words satisfying parity condition, K m = {u ∈ m ω π : lim sup n→∞ u n is even }.
That is, K m coincides with L 1,m or L 0,m−1 of (1), depending on whether m is even or odd. It is straightforward to see that K m is in the class Π m .
In the sequel we consider words over a product alphabet m 2 π . We identify a pair of words u, v ∈ (m ω π ) 2 with a single word over m 2 π in an obvious manner. The subsequent lemma is the heart of our paper.

Lemma 1.
For each m ≥ 2, there exist disjoint sets U 1 , U 2 ⊆ m ω π of class Σ m , satisfying the following: Proof. We first present the construction in the case when m is odd; thus the Σ m -automata have index (1, m), and Π m -automata have index (0, m − 1).
Let P m be an automaton over the alphabet m π with the set of states also equal to m π , and the transition function Tr(q, s) = s, for any q and s. (We leave the remaining items temporarily unspecified.) Let P m × be an automaton over m 2 π which behaves like P m reading only the first component. Similarly, × P m reads only the second component. The automaton for U 2 is defined analogously; the difference concerns only rankings 4 , namely rank(i) = i+1, for all i, and rank(i ) = i+2, for i = 0, 1, . . . , m−2 and rank((m−1) ) = m. Note that the states m − 1 and (m − 1) can be reached only while changing the levels and, whenever it happens, both automata assume the highest odd rank m.
Each word u ∈ m 2 π ω induces the same run in both automata up to the rankings. Clearly, a word u causes infinitely many changes of the level if and only if it contains infinitely many occurrences of m − 1 on both left and right track. By remark above, such a word is accepted by neither of the automata. On the other hand, if the run on u stabilizes on some level then one of the automata necessarily accepts, as the ranks they assume in their runs (after stabilization) differ precisely by 1. This shows that U 1 and U 2 are disjoint and U 1 ∪ U 2 = I m × I m . The inclusion K m × K m ⊆ I m × I m is obvious.
If u ∈ K m × K m then the run on u stabilizes in the upper or lower component (as clearly u ∈ I m × I m ). Then the automaton for U 1 , from some moment on, either reads a word in K m in the upper component or a word in K m in the lower component; in either case it accepts. A similar argument shows the second inclusion, which completes the proof of the lemma in case m is odd. The construction for the case of m even is analogous. We leave it to the reader with Figure 3 as a hint.
The properties of U 1 and U 2 mentioned in Lemma 1 imply a kind of hardness of these sets. Generally, for an automaton A over an alphabet A of some index (ι, κ), let rank A denote the function sending a word u ∈ A ω onto the sequence of ranks assumed by A. More precisely, where r 0 r 1 . . . is the unique run of A on u. If B is another automaton over A with index (ι, κ), we define rank A×B : A ω → {ι, . . . , κ} 2 ω , by

Lemma 2. Let
Proof. Generally, if D is a deterministic parity automaton of index (ι, κ) then, by definition of acceptance, Hence, in our case, u ∈ L(A) ⇒ rank A (u) ∈ K m , and u ∈ L(B) ⇒ rank B (u) ∈ K m . As L(A) and L(B) are disjoint, u ∈ L(A) implies rank A×B (u) ∈ K m × K m , but we know from Lemma 1 that K m × K m ⊆ U 1 . The argument for 2 is similar. Finally, again by disjointness of L(A) and L(B), we have rank A×B (u) ∈ K m × K m , for any u, but we know from Lemma 1 that K m × K m ⊆ U 1 ∪ U 2 , which completes the proof.
We are now ready to state the main result of this section. Recall that the separation property for the class Π 2 was proved earlier by Selivanov [12], who also gave 5 a hint [13] how this result can be generalized for all classes Π n .

Theorem 3. The separation property holds for classes Π m and fails for classes Σ m of the index hierarchy of deterministic word automata.
Proof. We will show that any pair of disjoint languages of class Π m over some finite alphabet A is separable by a language of class ∆ m , whereas this property fails for the pair of sets U 1 , U 2 of Lemma 1 (which are of class Σ m ).
Let A and B be as in Lemma 2. It follows from 1 and 2 that the inverse image of U 1 under the mapping rank A×B , i.e.,

separates L(A) and L(B). Let us see that this set is recognizable by an Σ m -automaton.
For an input u, we just use the automaton for U 1 reading the subsequent values of the function rank A×B ; the construction is straightforward. In a similar vein we can show that rank A×B −1 (U 2 ) is in the class Σ m as well. Clearly these sets are disjoint as U 1 and U 2 are disjoint. But it follows from condition 3 of Lemma 2 that they sum up to A ω , hence they both are of class ∆ m .
To show that U 1 and U 2 are inseparable, we start with the following observation. Suppose that A and B are Π m -automata over the alphabet m 2 π . Then the function rank A×B , which in this case has type rank A×B : m 2 π ω → m 2 π ω , has a fixed point. Indeed, a fixed point f can be defined 6 by an inductive formula (whereTr is the standard extension of Tr from letters to finite words). Now suppose, for the sake of contradiction, that there is a set C ⊆ m ω π of class ∆ m , such that U 1 ⊆ C and U 2 ⊆ C. Let A and B be two automata of class Π m , such that By remark above, the function rank A×B has the fixed point (f 0 , f 1 , . . .). On the other hand, by conditions 1 and 2 of Lemma 2, it reduces C to U 1 ⊆ C, and C to U 2 ⊆ C. This contradiction completes the proof.

Alternating hierarchy over trees
In this section we investigate the Rabin-Mostowski index hierarchy for alternating automata on k-ary trees. A tree language is in the class Σ m if it is recognized by an alternating Σ m -automaton; similarly for Π m . A tree language is in the class ∆ m if it is simultaneously Σ m and Π m . We show that the separation property fails in general for the classes Σ m , for m ≥ 1.
It will be convenient to have some normal form of alternating automata. We call an alternating automaton on k-ary trees, A = A, Q ∃ , Q ∀ , q I , δ, rank , an ∃∀-automaton if it satisfies the following conditions: for any pair (p, s) ∈ Q × A, there are exactly two transitions (p, s, d, q), (p, s, d , q ) ∈ δ. These conditions imply that the graph of the game G(A, t) (see equation (2)) unravels to a full binary tree, where ∃ and ∀ alternate starting with ∃.

Lemma 4.
Any alternating tree automaton can be transformed to an ∃∀-automaton of the same index, recognizing the same language.
A ∀∃-automaton is defined similarly, with the only difference that q I ∈ Q ∀ . Clearly, an analogue of Lemma 4 for ∀∃-automata holds as well.
We are going to define a tree version of the "hard pairs" from Section 3. Let m ≥ 1, and let U 1 and U 2 be the languages defined in the proof of Lemma 1. We let By Fact 1, the sets ∇ 1 and ∇ 2 are of class Σ m . To have some analogue of Lemma 2, we need some analogue of the function rank A×B ; we will define it only for automata in a special form. We call a tree t : The concept of a ∀∃-tree is defined analogously. Note that the trees T (A, t) defined above are ∃∀-trees or ∀∃-trees, whenever A is an ∃∀-automaton or ∀∃-automaton, respectively. At first, we define a product of a binary ∃∀-tree t 1 and a binary ∀∃ tree t 2 as a 4-ary ∃∀-tree t 1 t 2 : 4 * → {∃, ∀} × A × A. It is convenient to fix some bijection 4 ∼ 2 × 2, so that a (finite) word w in 4 * can be identified with a pair of words v, u in 2 * of the same length (such that w i = (v i , u i )); we then use the notation w = (u • v). We then let 7 Now fix k and an alphabet A. Let A and B be two automata on k-ary trees over the alphabet A, both of the class Π m . Assume moreover that A is an ∃∀-automaton and B a ∀∃-automaton. For a tree t : k * → A, consider an ∃∀-tree T (A, t) and a ∀∃-tree T (B, t). We let g A×B (t) = (T (A, t) T (B, t)).
The following is a (partial) analogue of Lemma 2.
7 Figure 4 at the end of the paper shows how from a green ∃∀ tree t1 and a red ∀∃ tree t2 we obtain a blue 4-ary tree t1 t2.

Lemma 5. With the notations above,
Proof. Assume that t ∈ L(A). It implies, that Eve has a winning strategy σ A showing that T (A, t) ∈ W ι,κ . Since L(A) and L(B) are disjoint, Adam has a winning strategy σ B showing that T (B, t) ∈ W ι,κ .
Let σ be the strategy for Eve on the tree T (A, t) T (B, t) which combines σ A and σ B . Namely, σ A , σ B choose one of the two options available for respectively Eve and Adam and σ chooses the uniquely defined combination of these two options. Since σ A leads to a sequence in K m , σ B leads to a sequence in the complement of K m , the resulting sequence defined by σ belongs to K m × K m , in particular it belongs to U 1 .
Assume now that t ∈ L(B). It implies, that Eve has a winning strategy σ B showing that T (B, t) ∈ W ι,κ . Since L(A) and L(B) are disjoint, Adam has a winning strategy σ A showing that T (A, t) ∈ W ι,κ .
Let σ be the strategy for Adam on the tree T (A, t) T (B, t) which combines σ A and σ B . Namely, σ A , σ B choose one of the two options available for respectively Adam and Eve and σ chooses the uniquely defined combination of these two options. Since σ A leads to a sequence in the complement of K m , σ B leads to a sequence in K m , the resulting sequence defined by σ belongs to K m × K m , in particular it belongs to U 2 .
transitions from the state γ A (u 0 ) ↓ 1 on the letter t(γ A (u 0 ) ↓ 2 ). Similarly, there are two transitions in B from the state γ A (v 0 ) ↓ 1 on the letter t(γ B (v 0 ) ↓ 2 ). The vertex t(u 0 • v 0 ) has four successors uniquely defined by these two pairs of transitions. We extend γ A and γ B accordingly and continue building a full 4-ary tree t.
It is easy to verify that the tree t is a fixed point 8 of g A×B . Now suppose, for the sake of contradiction, that there exists a set C of trees over the alphabet {∃, ∀} × (ι, κ) 2 such that C belongs to the class ∆ m , ∇ 1 ⊆ C and ∇ 2 ⊆ C.
Let A and B be two automata of class Π m , such that By the remark above, the function g A×B has the fixed point t. On the other hand, by conditions 1 and 2 of Lemma 5, it reduces C to ∇ 1 ⊆ C, and C to ∇ 2 ⊆ C. This contradiction completes the proof.

Figure 4
Operation on a green ∃∀ tree t1 and a red ∀∃ tree t2 gives the blue 4-ary tree.
The consideration of 4-ary trees in Theorem 6 made the proof more transparent, but the result can be adapted to binary trees as well.

Corollary 7.
There exists a pair of sets of binary trees of class Σ m inseparable by any set of class ∆ m .
It is straightforward to verify that ∇ i = η −1 [V i ], for i = 1, 2. Suppose that V 1 , V 2 are separated by a set C of class ∆ m . It is easy to check that the preimage under the mapping η of a tree language recognized by an alternating automaton of an index (i, n) can be itself recognized by an automaton of the same index. Hence η −1 [C] is in the class ∆ m and separates ∇ 1 and ∇ 2 , which contradicts Theorem 6.