On the Separation Question for Tree Languages
 615 Downloads
Abstract
We show that the separation property fails for the classes Σ _{ n } of the RabinMostowski index hierarchy of alternating automata on infinite trees. This extends our previous result (obtained with Szczepan Hummel) on the failure of the separation property for the class Σ _{2} (i.e., for coBüchi sets). The nonseparation result is also adapted to the analogous classes induced by weak alternating automata.
To prove our main result, we first consider the RabinMostowski index hierarchy of deterministic automata on infinite words, for which we give a complete answer (generalizing previous results of Selivanov): the separation property holds for Π _{ n } and fails for Σ _{ n }classes. The construction invented for words turns out to be useful for trees via a suitable game.
It remains open if the separation property holds for all classes Π _{ n } of the index hierarchy for tree automata. To give a positive answer it would be enough to show the reduction property of the dual classes—a method wellknown in descriptive set theory. We show that it cannot work here, because the reduction property fails for all classes in the index hierarchy.
Keywords
Alternating tree automata Separation property RabinMostowski index1 Introduction
The separation question is whether two disjoint sets A and B can be separated by a set C (i.e., A⊆C and B∩C=∅) which is in some sense simpler. Separation is one of the main issues in descriptive set theory. A fundamental result due to Lusin is that two analytic sets can always be separated by a Borel set, but two coanalytic sets in general cannot. The former implies that if a set is simultaneously analytic and coanalytic then it is necessarily Borel, which is the celebrated Suslin Theorem (see, e.g., [11] or [9]).
A wellknown fact in automata theory exhibits a similar pattern: if a set of infinite trees as well as its complement are both recognizable by Büchi automata then they are also recognizable by weak alternating automata (weakly recognizable, for short). This result was first proved by Rabin [13] in terms of monadic secondorder logic; the automatatheoretic statement was given by Muller, Saoudi, and Schupp [12] in terms of weak alternating automata. It is not difficult to adapt Rabin’s proof to obtain a—slightly stronger—separation property: any two disjoint Büchi recognizable sets of trees can be separated by a weakly recognizable set (see, e.g., [7]). Quite analogical to the coanalytic case, the separation property fails in general for the dual class of coBüchi tree languages (i.e., the complements of Büchi recognizable sets). In [7], a pair of such sets is presented that cannot be separated by any Borel set, hence a fortiori by any weakly recognizable set.
In the present paper we answer the question negatively for all classes Σ _{ n } of the Rabin–Mostowski index hierarchy for alternating automata on infinite trees. (The classes Σ _{ n } correspond to the indices (i,k) with k odd; see the definition in Sect. 2 below.) By an analogy with the Borel hierarchy [9, 11], one is tempted to conjecture that the separation property actually does hold for all classes Π _{ n }, but this question seems to be difficult already for n=3.
To prove our main result, we first study a conceptually simpler case of infinite words and the Rabin–Mostowski index hierarchy of deterministic automata. In this case we give a complete answer: the separation property holds for classes Π _{ n } and fails for classes Σ _{ n }. The argument is based on a uniform construction of an inseparable pair in each class Σ _{ n }. It should be noted that the separation property of the class (1,2) was proved earlier by Selivanov [15], who also gave a hint [16] how this result can be generalized for all classes Π _{ n } in the index hierarchy of deterministic automata on infinite words.
The construction made for words is further used in the case of trees. More specifically, we consider labeled trees whose vertices are divided between two players: Eve and Adam, who wish to form a path in a tree. Given a set on infinite words L, we consider the set Win^{∃}(L) of those trees where Eve has a strategy to force a path into L. The operation Win^{∃} allows us to shift the witness family from words to trees.
The main result is completed by two further observations. We show that the nonseparation result holds for the analogous classes in the index hierarchy of the weak alternating automata. Next, in quest to solve the aforementioned conjecture about the classes Π _{ n }, we investigate reduction property for tree automata. Roughly, this property means that any union A∪B can be refined to a disjoint union A′∪B′ of sets of the same complexity. In descriptive set theory, the reduction property of a class of sets is often used to show the separation property of the dual class (see [9]). We show that this method cannot be used here, because the reduction property fails for all nontrivial classes in the index hierarchy of alternating automata on trees.
The preliminary version of this paper was presented to the conference STACS 2012 [2]. The results about weak automata and about reduction property are new and presented for the first time in this paper.
2 Index Hierarchy
Throughout the paper, ω stands for the set of natural numbers, and notations n∈ω and n<ω are interchangeable. We identify a natural number n<ω with the set {0,1,…,n−1}; in particular 2={0,1}. An alphabet is any set, which is finite but nonempty.
We will consider deterministic automata on infinite words and alternating automata on infinite trees. The latter will be introduced via games. For more background on automata, we refer the reader to a survey by W. Thomas [17].
2.1 Automata on Infinite Words
A deterministic parity automaton on infinite words over an input alphabet A can be presented by \(\mathcal{A} = \langle A, Q,q_{I}, \delta, \mathrm{rank}\rangle\), where Q is a finite set of states ranked by the function rank:Q→ω, and δ:Q×A→Q is a transition function. A run of \(\mathcal{A}\) on a word u∈A ^{ ω } is a word r∈Q ^{ ω } whose first element r _{0} is the initial state q _{ I }, and r _{ n+1}=δ(r _{ n },u _{ n }), for n<ω. It is accepting if the highest rank occurring infinitely often (i.e., lim sup_{ n→∞}rank(r _{ n })) is even. The language \(L(\mathcal{A}) \) recognized by \(\mathcal{A}\) consists of those words u∈A ^{ ω } which admit an accepting run.
To better understand the structure of indices, we can think of a structure 〈{ι,…,κ},≤,Even〉 associated with an index (ι,κ), where Even stands for the evenness predicate. We identify indices whose structures are isomorphic. The ordering of indices corresponds to inclusion of isomorphism types, and the types of dual indices differ only by the monadic predicates, which are complement to each other.
2.2 Games
The players in a graph game start a play in the position p _{ I } and then move the token according to the relation Move (always to a successor of the current position), thus forming a path in the arena. The move is selected by Eve or Adam, depending on who the owner of the current position is. If a player cannot move, she/he looses. Otherwise, the result of the play is an infinite path v _{0},v _{1},v _{2},… , inducing the sequence of labels ℓ(v _{0}),ℓ(v _{1}),ℓ(v _{2}),… . If this sequence belongs to L _{∃} then Eve wins the play, if it belongs to L _{∀} then Adam is the winner; otherwise there is a draw.
In the games considered in this paper, we always have L _{∀}=A ^{ ω }−L _{∃}, hence a draw cannot occur.
Intuitively, a strategy is an oracle that tells the player her/his next move in the play, depending on the current history of the game. A history is any finite path in the arena starting from the initial position, i.e., a sequence π=(v _{0},v _{1},v _{2},…,v _{ k }), with v _{0}=p _{ I }, and (v _{ i },v _{ i+1})∈Move, for i<k. We let last(π)=v _{ k }.
 1.
p _{ I } is the unique history of length 1 belonging to S,
 2.
each history π∈S with last(π)∈V _{∃} has a unique prolongation πw _{ π } in S (where (last(π),w _{ π })∈Move),
 3.
each history π∈S with last(π)∈V _{∀} has a prolongation πw in S, for each w, such that (last(π),w)∈Move,
 4.
S is closed under initial segments.^{2}
A play (finite or infinite) is consistent with a strategy S if all its finite prefixes are in S. A strategy S is winning for Eve, if every play consistent with S is won by Eve. That is, either the play is finite and ends in a position from which Adam has no move, or it is infinite, and the sequence of labels belongs to L _{∃}.
Analogically we define a strategy and a winning strategy for Adam. We say that Eve wins the game \(\mathcal{G}\) if she has a winning strategy in this game, the similar for Adam.
In this paper we will often consider games whose arenas are infinite binary trees. A (full) binary tree over a finite alphabet Alph is a mapping t:2^{∗}→Alph. For a finite alphabet A, we denote by GTr(A) the set of binary trees over the alphabet Alph={∃,∀}×A. Any tree in GTr(A) constitutes an arena, where the labels {∃,∀} induce a partition of the nodes of the tree into positions of Eve and Adam, the moves go to the successors of a current node, and the initial position is the root.
A language L⊆A ^{ ω } can serve as a winning criterion for either of the players. We will therefore consider two classes of games: L∃ games, and L∀ games.
2.3 Automata on Trees
For a finite alphabet A, let Tr(A) denote the set of all binary trees over A (please note the difference with the notation GTr(A) introduced in the previous section).
Intuitively, players in this game follow a path in the tree t, additionally annotated by the states. A transition is always selected by the owner of the state. The automaton accepts the tree if Eve can make sure that the sequence of ranks encountered during an infinite play satisfies the parity condition (and she wins all finite plays if any).
In the present paper, we will consider sets Win^{∃}(L) (and occasionally Win^{∀}(L)), for other sets L recognized by word automata. It is useful to note the following.
Lemma 1
If a language L of infinite words is recognized by a deterministic automaton of index (ι,κ) then both languages Win^{∃}(L) and Win^{∀}(L) can be recognized by an alternating ^{3} tree automaton of index (ι,κ).
Proof
Let \({\mathcal{B}}\) be a deterministic automaton on infinite words of index (ι,κ), such that \(L({\mathcal{B}}) = L\). The alternating tree automaton \({\mathcal{A}}\) accepting Win^{∃}(L) will have three copies of the states of the automaton \({\mathcal{B}}\), say Q ^{∃}, Q ^{∀}, and Q ^{ ε }. The states in Q ^{∃} are existential and the states in Q ^{∀} are universal. We let the states in Q ^{ ε } be also universal (for concreteness), but they will not leave Adam any choice; these states will be used only in εmoves. Let q ^{∇} denote a respective copy of a state q. We let rank(q ^{∇})=rank(q).
The initial state is \(q_{I}^{\varepsilon} \). For any state q and letter a, there are transitions (q ^{ ε },(∃,a),ε,q ^{∃}) and (q ^{ ε },(∀,a),ε,q ^{∀}). Whenever (p,a,q) is a transition in \({\mathcal{B}}\), there are moreover transitions (p ^{∃},(∃,a),0,q ^{ ε }), (p ^{∃},(∃,a),1,q ^{ ε }), and (p ^{∀},(∀,a),0,q ^{ ε }), (p ^{∀},(∀,a),1,q ^{ ε }). Now it can easily be verified that a winning strategy for Eve in the L∃game on t can be transferred to a winning strategy of Eve in the automaton game \(G(\mathcal{A},t) \), and viceversa.
An analogous construction works for Win^{∀}(L) (by interchanging the players). □
We now explain the Σ/Π terminology for the index hierarchy, which we have used informally in Introduction. It originates from the μcalculus (see, e.g., [4]), and will be convenient to handle dualities. For each m≥1, we consider two indices: (1,m) and (0,m−1), and associate the symbol Σ _{ m } with this index whose maximum is odd, and Π _{ m } with the one whose maximum is even. For example, (0,1)≈Σ _{2}, (1,2)≈Π _{2}, (1,3)≈Σ _{3}, (0,2)≈Π _{3}, (1,4)≈Π _{4}, (0,3)≈Σ _{4}, etc. We will then refer to an automaton (of any kind) of index (ι,κ) as to Σ _{ m }automaton or Π _{ m }automaton with an appropriate m.
This terminology extends to classes of languages of infinite words and trees. A language of infinite words is in the class Σ _{ m } if it is recognized by a deterministic Σ _{ m }automaton; similarly for Π _{ m }. A language is in the class Δ_{ m } if it is simultaneously in the classes Σ _{ m } and Π _{ m }.
For trees, we use the analogical terminology for alternating automata.
3 Tinkering with Infinite Words
The following property of languages will be useful. Informally, a language is stuttering if it does not distinguish between a single letter a and an nonempty block a ^{ ℓ }.
Definition 2
 For any infinite sequence of finite words (w _{ n })_{ n<ω },
 For any finite sequence of finite words w _{ n } (n=0,…,k) and any infinite word v,
Directly from the definition we get the following.
Fact 3
If L is stuttering, then \(\overline{L}\) is stuttering too.
Fact 4
Each language L _{ ι,κ } is stuttering.
In the sequel we consider words over a product alphabet E ^{2}. We identify a pair of words 〈(u _{ n }),(v _{ n })〉∈(E ^{ ω })^{2} with a single word over (E ^{2})^{ ω } via a mapping 〈(u _{ n }),(v _{ n })〉↦〈(u _{ n },v _{ n })〉.
If \(\mathcal{B}\) is another deterministic automaton of index (ι,κ) over A, we define \(g_{\mathcal{A}\times\mathcal{B}}: A^{\omega}\to(\{ \iota,\ldots,\kappa \}^{2})^{\omega}\) by \(g_{\mathcal{A}\times\mathcal{B}}(u)= (g_{\mathcal{A}}(u),g_{\mathcal{B}}(u))\).
Lemma 5
Let \(\mathcal{A} \) and \(\mathcal{B} \) be automata of index (ι,κ) over some alphabet A, such that \(L(\mathcal{ A})\cap L(\mathcal{ B}) = \emptyset\), and let L=L _{ ι,κ }.
 1.
if \(u \in L(\mathcal{ A}) \) then \(g_{\mathcal{A} \times\mathcal{B}} (u) \in{L\times{\overline{L}}} \),
 2.
if \(u \in L(\mathcal{ B}) \) then \(g_{\mathcal{A} \times \mathcal{B}} (u) \in{\overline{L}\times L}\),
 3.
\(g_{\mathcal{A} \times\mathcal{B}} (u) \in{\overline {L\times L}} \).
Proof
As mentioned above \(u \in L(\mathcal{ A}) \Rightarrow g_{\mathcal{A}} (u) \in L \), and \(u \notin L(\mathcal{ B}) \Rightarrow g_{\mathcal{B}} (u) \in\overline{L} \). As \(L(\mathcal{ A}) \) and \(L(\mathcal{ B}) \) are disjoint, \(u \in L(\mathcal{ A}) \) implies \(u\notin L(\mathcal{B})\) hence \(g_{\mathcal{A} \times\mathcal{B}} (u) \in L \times\overline{L } \).
The argument for 2 is similar. Finally, again by disjointness of \(L(\mathcal{ A}) \) and \(L(\mathcal{ B}) \), we have \(g_{\mathcal{A} \times \mathcal{B}} (u) \in\overline{L \times L}\), for any u, which completes the proof. □
4 Separation Property of ωLanguages
We recall the main concept of the paper.
Definition 6
A set Z separates a pair of sets X,Y if X⊆Z and Y∩Z=∅ or, symmetrically, Y⊆Z and X∩Z=∅.
We say that the separation property holds for a class Γ _{ m }∈{Σ _{ m },Π _{ m }} if any two disjoint languages^{4} from this class are separable by a language in class Δ_{ m }; otherwise the separation property fails for Γ _{ m }.
In this section we show that the separation property holds for classes Π _{ m } and fails for classes Σ _{ m }, for m≥1. In fact, both properties will follow from a single construction (parametrized by m≥2).
The level 1 is somewhat exceptional. The class Σ _{1} consists of a single language ∅, and the class Π _{1} consists of languages of all words (for each finite alphabet). Therefore the class Δ_{1} is empty. For any alphabet A, there are no disjoint languages in the class Π _{1}, hence the separation property holds trivially. On the other hand, there is a pair of disjoint sets in the class Σ _{1} (namely ∅ and ∅), which is inseparable because the class Δ_{1} is empty. From now on, we will only consider classes with m≥2.
The following is the first nonseparability result.
Lemma 7
Let L=L _{ i,k }⊆{i,…,k}^{ ω } (with ^{5} i∈{0,1}) and let m=k+1−i. Then \(L\times\overline{L}\) and \(\overline{L}\times L\) (which are obviously disjoint) are not separable by a set in Δ_{ m }.
Proof
Suppose to the contrary that there is a set C recognizable by an automaton of index (i,k), such that \(\overline{L}\times L \subseteq C\) and \(L\times\overline{L}\subseteq \overline{C}\).

if u∈C then \(g_{\mathcal{A} \times\mathcal{B}} (u) \in L\times{\overline{L}}\subseteq \overline{C} \),

if \(u \in\overline{C} \) then \(g_{\mathcal{A} \times\mathcal{B}} (u) \in{\overline{L}\times L}\subseteq C\).
For this fixed point f, we have \(f\in C \Rightarrow f\in\overline{C}\) and \(f\in\overline{C} \Rightarrow f\in{C}\), a contradiction which completes the proof. □
The subsequent lemma is the heart of our proof.
Lemma 8
Proof
Each word u∈(E ^{2})^{ ω } induces the same run in both automata up to the rankings. Clearly, a word u causes infinitely many changes of the level if and only if it contains infinitely many occurrences of k on both left and right track if and only if it visits infinitely often a state 〈up,r,k+1〉 and a state 〈down,k+1,s〉. By the definition of the rank functions such a word is accepted by neither of the automata. On the other hand, if the run on u stabilizes on some level then only one of the automata necessarily accepts as the ranks they assume in their runs (after stabilization) differ precisely by 1.
This shows that U _{1} and U _{2} are disjoint and \(U_{1} \cup U_{2} = \overline{I \times I }\). The inclusion \(\overline{L \times L } \subseteq\overline{I \times I }\) is obvious.
If \(u \in L \times\overline{L }\) then the run on u stabilizes in the upper or lower component (as clearly u∉I×I). Then the automaton for U _{1}, from some moment on, either reads a word in L in the upper component or a word in \(\overline{L } \) in the lower component; in either case it accepts. An argument for the inclusion involving U _{2} is similar. □
We are now ready to state the main result of this section. Recall that the separation property for the class Π _{2} was proved earlier by Selivanov [15], who also gave^{7} a hint [16] on how this result can be generalized for all classes Π _{ n }.
Theorem 9
The separation property holds for classes Π _{ m } and fails for classes Σ _{ m } of the index hierarchy of deterministic word automata.
Proof
We will show that any pair of disjoint languages of class Π _{ m } over some finite alphabet A is separable by a language of class Δ_{ m }, whereas this property fails for the pair of sets U _{1},U _{2} constructed in Lemma 8, for any class Σ _{ m }, m≥2.
Now, by Lemma 8, \(L\times{\overline{L}}\subseteq U_{1}\) and \(\overline{L}\times L \subseteq U_{2}\), where L is the parity language in Π _{ m }. If the sets U _{1} and U _{2} in Σ _{ m } were separable by a set in Δ_{ m }, then \(L\times{\overline{L}}\) and \(\overline {L}\times L\) would be separable, a contradiction with Lemma 7. □
5 Tinkering with Trees
5.1 Contractions
Definition 10
In the sequel, we will often use^{8} the celebrated Banach FixedPoint Theorem: If g is a contracting mapping over the same domain Tr(A), it has a (unique) fixed point.
For ι≤κ<ω, we will use an abbreviation GTr({ι,…,κ})=GTr_{ ι,κ }. The following fact, first noted in [1], is very useful in analysis of tree automata.
Theorem 11
[1]
Let \(\mathcal{A}\) be an alternating tree automaton of index (ι,κ) over the alphabet A. There exists a contracting mapping g:Tr(A)→GTr_{ ι,κ } such that \(t\in L(\mathcal{A})\) iff g(t)∈W _{ ι,κ }.
Proof (sketch)
In the first step we replace \({\mathcal{A}}\) with another automaton \({\mathcal{A}}'\) of the same index (ι,κ) such that \(L({\mathcal{A}})=L({\mathcal{A}}')\) and in every state of \({\mathcal{A}}'\) the player has exactly two possible moves. We fix a map h which assigns to a given tree t a tree h(t) of all possible gameplays between Adam and Eve on t on \({\mathcal{A}}'\). That is, for every label of h(t) the first coordinate indicates who the owner of the state is (this is ∃ or ∀) and the second coordinate is the rank of the state q. The sons of a given vertex are induced by transitions from the state q.
This is almost what we need, that is \(t\in L(\mathcal{A})\) if and only if h(t)∈W _{ ι,κ }. In order to make h contracting, we add a dummy initial level to h(t), that is g(t)(ϵ)=g(t)(∃,ι) and g(t)(iv)=h(t)(v) for i=0,1. □
The above theorem also implies its dual variant.
Corollary 12
Let \(\mathcal{A}\) be a tree automaton of index (ι,κ) over the alphabet A. There exists a contracting mapping g:Tr(A)→GTr_{ ι,κ } such that \(t\in L(\mathcal{A})\) iff g(t)∈W ^{∀}(L _{ ι,κ }).
Proof
Since W _{ ι,κ }=W ^{∃}(L _{ ι,κ }), it is enough to compose a contraction from Theorem 11 with a mapping which switches roles between Eve and Adam. □
5.2 Product of Trees
In this section we define the product of two trees in the sets GTr—a crucial technical concept needed later in Sect. 6. In a sense, it is a notion similar to the direct product of infinite words.
As usual we can extend this operation to sets of trees; U⊗V={t⊗s:t∈U,s∈V}.
Lemma 13
If L _{0}⊆A ^{ ω } and L _{1}=B ^{ ω } are stuttering, then Win^{∃}(L _{0})⊗Win^{∃}(L _{1})⊆Win^{∃}(L _{0}×L _{1}) and Win^{∀}(L _{0})⊗Win^{∀}(L _{1})⊆Win^{∀}(L _{0}×L _{1}).
Proof
First let us observe that the sequence of labels in A×B along any infinite path \(d_{1}d'_{1}d_{2}d'_{2}\cdots d_{n}d'_{n}\cdots\) in the product tree t=t _{0}⊗t _{1} is 〈a _{0},b _{0}〉〈a _{0},b _{0}〉〈a _{1},b _{1}〉⋯, where path d _{1} d _{2}⋯d _{ n }⋯ has labels a _{0} a _{1}⋯ in t _{0}, and path \(d'_{1}d'_{2}\cdots d'_{n}\cdots\) has labels b _{0} b _{1}⋯ in t _{1}.
If \(d_{1}d'_{1}d_{2}d'_{2}\cdots\) is any path played along the strategy σ on t then d _{1} d _{2}⋯ is played along σ _{0} in t _{0} and \(d'_{1}d'_{2}\cdots\) is played along σ _{1} in t _{1}.
To prove that Win^{∃}(L _{0})⊗Win^{∃}(L _{1})⊆Win^{∃}(L _{0}×L _{2}), we have to show that if σ _{0} and σ _{1} are winning then so is σ. Let τ be any strategy for Adam and let w=〈a _{0},b _{0}〉〈a _{1},b _{1}〉〈a _{2},b _{2}〉⋯ be the label of the path \(d_{1}d'_{1}d_{2}d'_{2}\cdots\) played along σ and τ. Then a _{0} a _{1} a _{2}⋯ is the label of the path d _{1} d _{2}⋯ played along the winning strategy σ _{0}, thus it belongs to L _{0}, and since L _{0} is stuttering, w _{0}=a _{0} a _{0} a _{1} a _{1}⋯∈L _{0} as well. Similarly w _{1}=b _{0} b _{0} b _{1} b _{1}⋯∈L _{1}. Thus w=w _{0}×w _{1}∈L _{0}×L _{1}, which proves that σ is winning.
The proof for Win^{∀} is very similar. □
6 Inseparable Pairs in Σ_{m}
Recall that, for trees, we use the notation Σ _{ m },Π _{ m } for classes of tree languages recognized by alternating automata of appropriate indices; a tree language is in the class Δ_{ m } if it is simultaneously in the classes Σ _{ m } and Π _{ m }.
We let m≥2; the case of m=1 will be considered in Sect. 6.1. Let \(U^{m}_{1} \) and \(U^{m}_{2} \) be the sets constructed in Lemma 8. By Lemma 1, the sets \(\mathrm{Win}^{\exists}(U^{m}_{1}) \) and \(\mathrm{Win}^{\forall}(U^{m}_{2})\) are in the class Σ _{ m } of the tree automata hierarchy. We will show that they are inseparable by a set in Δ_{ m }. Let us first verify that these two sets are disjoint: if \(t\in\mathrm{Win}^{\exists}(U^{m}_{1})\cap \mathrm{Win}^{\forall}(U^{m}_{2})\) then t has a branch labelled by a word in \(U^{m}_{1}\cap U^{m}_{2}\), a contradiction.
Lemma 14
Proof
Since T is in the class Π _{ m }, there exists (by Theorem 11) a contracting mapping g such that t∈T iff g(t)∈Win^{∃}(L). From Corollary 12 we get a contracting mapping g′ such that t∈T′ iff g′(t)∈Win^{∀}(L).
Since T and T′ are disjoint, t∈T implies t∉T′ and thus g(t)∈Win^{∃}(L) and \(g'(t)\in\mathrm{Win}^{\exists}(\overline{L})\). Hence by Lemma 13 we have \(g(t)\otimes g'(t)\in\mathrm{Win}^{\exists}(L\times\overline{L})\).
Similarly, if t∈T′ then \(g(t)\in\mathrm{Win}^{\forall}(\overline {L})\), g′(t)∈Win^{∀}(L), and \(g(t)\otimes g'(t)\in\mathrm{Win}^{\forall}( \overline{L}\times L)\). Now let us define h by h(t)=g(t)⊗g′(t). It is easy to check that h is contracting. □
Lemma 15
Let (ι,κ) be the index corresponding to the class Π _{ m } and let L=L _{ ι,κ }. Then \(\mathrm{Win}^{\exists}(L\times\overline{L})\) and \(\mathrm {Win}^{\forall}(\overline{L}\times L)\) cannot be separated by a set C∈Δ_{ m }.
Proof
Suppose there is such a set C∈Δ_{ m }. Note that both sets \(C, \overline{C}\) are in Π _{ m }. Then, by the previous lemma with \(T = \overline{C}\), T′=C, there exists a contracting h such that \(t\in\overline{C}\Rightarrow h(t)\in \mathrm{Win}^{\exists}(L\times\overline{L}) \subseteq C\), \(t\in{C}\Rightarrow h(t)\in \mathrm{Win}^{\forall}(\overline{L}\times L) \subseteq\overline{C}\). Since h is contracting, it has a fixed point t=h(t), and for this t, \(t\in\overline{C}\Rightarrow t\in C\), and \(t\in{C}\Rightarrow t \in\overline{C}\), a contradiction. □
Theorem 16
The sets \(\mathrm{Win}^{\exists}(U^{m}_{1})\) and \(\mathrm{Win}^{\forall}(U^{m}_{2})\) are not separable by a set in Δ_{ m }, for m≥2.
Proof
Let (ι,κ) be the index corresponding to the class Π _{ m }. Since, by Lemma 8, \(\mathrm{Win}^{\exists}(L_{\iota,\kappa} \times\overline{L_{\iota,\kappa}})\subseteq\mathrm{Win}^{\exists}(U_{1}^{m})\) and \(\mathrm{Win}^{\forall}(\overline{L_{\iota,\kappa}}\times L_{\iota,\kappa})\subseteq\mathrm{Win}^{\forall}(U_{2}^{m})\), a separator of \(\mathrm{Win}^{\exists}(U_{1}^{m})\) and \(\mathrm{Win}^{\forall}(U_{2}^{m})\) would also separate \(\mathrm{Win}^{\exists}(L_{\iota,\kappa} \times\overline {L_{\iota,\kappa}})\) and \(\mathrm{Win}^{\forall}(\overline{L_{\iota ,\kappa}}\times L_{\iota,\kappa})\), a contradiction with Lemma 15. □
6.1 Level 1
In contrast to deterministic automata for words, the classes Σ _{1} and Π _{1} for alternating automata on trees are not completely trivial. For example, an automaton \(\mathcal{A} \) of index (1,1) may accept a tree t if Eve wins the game \(G(\mathcal{A},t) \) in finite time. Similarly, an automaton of index (0,0) may fail to accept a tree if Adam wins the game in finite time. Not surprisingly, the separation question for classes Σ _{1} and Π _{1} is not difficult to solve. We use, however, different methods than for higher levels, including some basic topology.
Proposition 17
The separation property fails for the class Σ _{1} and holds for the class Π _{1}.
Proof
For the negative result, we construct automata A _{1} and A _{2} as follows. Both automata read the leftmost branch of a given tree over the alphabet {a,b}. A _{1} accepts a tree t if and only if the first appearance of the letter a is at an even level, and A _{2} accepts t iff the first appearance of the letter a is at an odd level. Formally speaking, the automaton A _{1} has two existential states q _{0},q _{1} of rank 1 and one “dead” universal state (with no move) also of rank 1. The initial state is q _{0}. In the game \(G(\mathcal{A},t) \), Eve alternates between her two states q _{0},q _{1} moving to the left when she reads letters b unless she is in the state q _{0} and reads letter a. In this case she moves to the “dead” state of Adam, and thus wins the game. The same description works for A _{2} except that the roles of q _{0} and q _{1} are switched.
Clearly the languages L(A _{1}) and L(A _{2}) are disjoint and of index (1,1). Suppose L(A _{ i }) is contained in a certain language K _{ i } of index (0,0) for i=1,2. We claim that a tree t with every vertex labeled b belongs to K _{1}∩K _{2}. For, let t _{ n } be a tree with every vertex labelled b except of the vertex 0^{ n } (i.e., the nth vertex on the leftmost branch), which is labeled a. Clearly, t is a pointwise limit of trees t _{2n }∈L(A _{1}) (n<ω) and it is also a pointwise limit of trees t _{2n+1}∈L(A _{2}) (n<ω). But, as we have noted above, the sets K _{ i } are closed, and hence t∈K _{1}∩K _{2}. This shows that it is impossible to separate L(A _{1}) and L(A _{2}) by a set in Δ_{1}.
Now let L _{1},L _{2}⊆Tr(Alph) be two disjoint languages recognizable by automata of index (0,0). We will show that they can be separated by a language C in class Δ_{1}.
Since the space is compact and sets U _{ t } are open, one can find t _{0},…,t _{ k }, such that \(L_{1}\subseteq C = U_{t_{0}}\cup\cdots\cup U_{t_{k}}\); by definition C separates L _{1} and L _{2}. We claim that C is in the class Δ_{1}; clearly it is enough to show that this holds for each set of the form (10) (each U _{ t,i } is of such form).
The (0,0) and (1,1) automata recognizing U _{ f } are identical except ranks. We describe them in terms of the respective games. Adam first makes an εmove: he chooses a vertex in the finite set I, say v. Once the decision is made, the automaton moves from the root to v in a deterministic manner and then verifies whether the label t(v) agrees with f(v). If it is the case, the automaton enters a “dead” state of Adam, and Adam looses. Otherwise, it enters a “dead” state of Eve, and Eve looses. Note that any play in this game is finite (bounded by max{v:v∈I}+2), hence the ranks of the states do not matter.
This remark completes the proof. □
Remark

L is simultaneously recognized by an automaton of index (0,0) and by an automaton of index (1,1),

L is closed and open in the above topology,

L is accepted by an alternating automaton B (of any index), such that every gameplay between Adam and Eve in B is finite,

L is accepted by an alternating automaton B, and there is a constant N such that every gameplay between Adam and Eve in B ends in less than N moves.
We now summarize the results of this section, Theorem 16 above, and the results on level 2 mentioned in Introduction [7, 13].
Corollary 18
Consider the RabinMostowski index hierarchy for alternating automata on trees. Separation property fails for all classes Σ _{ m }, m≥1, and holds for the classes Π _{1} and Π _{2}.
The status of the classes Π _{ m }, m≥3, remains open.
7 Weakly Recognizable Tree Languages
A weak alternating parity tree automaton of index (ι,κ) can be presented as an ordinary automaton introduced in Sect. 2.3, subject to an extra monotonicity condition: if (p,a,d,q) is a transition in δ then rank(q)≥rank(p). The RabinMostowski indices of weak automata induce a hierarchy of tree languages similarly as for the ordinary automata; however we will not associate letters Σ/Π with this hierarchy. The strictness of the weak hierarchy has been established prior to the “strong” one by A.W.Mostowski [10]; an alternative proof can be given via the weak versions of languages W _{ ι,κ } (see, e.g., [4]).
Note that the weak automata with indices (0,0) and (1,1) coincide with the ordinary ones, and we have already considered the respective classes in Sect. 6.1.
From now on we fix a pair (ι,κ) (ι<κ<ω) and define M=M _{ ι,κ } to be the set of all non–decreasing sequences in {ι,…,κ}^{ ω }. We let \(\mathbb{A}M\) be the set of all trees in \({\mathcal{T}}_{\iota,\kappa} \), such that the labeling of any path projected at the second component is in M. Let L=L _{ ι,κ }. The mapping g constructed in Theorem 11 has the following property in the weak case (see, e.g., [4] Theorem 8.3.1).
Fact 19
Let T⊆GTr_{ ι,κ } be a weak language of index (ι,κ). Then there exists a contracting mapping g such that \(t\in{\mathcal{T}}\Leftrightarrow g(t)\in\mathrm{Win}^{\exists}(L)\) and moreover, for all t, \(g(t)\in\mathbb{A}M\). The analogous claim holds for Win^{∀}(L).

if t∈T then \(g(t)\in\mathrm{Win}^{\exists}(L)\cap\mathbb{A}M\) and \(g'(t)\in\mathrm{Win}^{\exists}(\overline{L})\cap\mathbb{A}M\); it follows that \(g(t)\otimes g'(t)\in\mathrm{Win}^{\exists}(L\times\overline{L})\cap \mathbb{A}M^{2} \subseteq\mathrm{Win}^{\exists}((L\times\overline{L})\cap M^{2})\);

analogously, if t′∈T′ then \(g(t')\otimes g'(t')\in\mathrm{Win}^{\forall}((\overline{L}\times L)\cap M^{2})\).
Since g⊗g′ is contracting, we get the following, for the same reasons as in Lemma 15.
Fact 20
\(\mathrm{Win}^{\exists}((L\times\overline{L})\cap M^{2})\) and \(\mathrm{Win}^{\forall}((\overline{L}\times L)\cap M^{2})\) cannot be separated by a set C such that C and \(\overline{C}\) are weak languages of index (ι,κ).
From now on we assume that ι=1, hence L=L _{1,κ } and M=M _{1,κ }, with 1<κ<ω. We will prove an analogue of Lemma 8. We call a deterministic automaton on infinite words weak if its transitions do not decrease ranks.
Lemma 21
Proof
Let ⊤={1,…,κ}^{ ω }. For any language Z⊆⊤ and i∈{1,…,κ}, we abbreviate Z _{ i }=Z∩{i,…,κ}^{ ω }.
Let us first consider the case of κ odd.
Let X=L∩M and \(Y = \overline{L} \cap M \).
By definition, the automaton is weak of index (1,κ); We leave it to the reader to check that it recognizes V _{1}. The construction of the automaton for V _{2} is analogous.
An automaton recognizing V _{1} is similar as in the previous case, except for the rank of ♠ which is now κ−1 (which is odd and greater than 1). Moreover, the automaton has a new selflooping state ♡ with the rank κ. The transitions are similar as before whenever monotonicity is preserved. The violation of monotonicity in state \(\overline{i} \) is again punished by entering the state ♠. In contrast, the violation of monotonicity in a state i is not punished at all, but instead it is granted by entering a “winning” state ♡. Again, it is straightforward to verify that the automaton satisfies all requirements. An automaton for V _{2} is analogous.
Now, by reading the proof of Lemma 1, we can easily see if a deterministic automaton recognizing a set of infinite words is weak of index (ι,κ) then the alternating automata recognizing sets Win^{∃}(L) and Win^{∀}(L) are also weak of index (ι,κ). Hence we can note the following.
Fact 22
For languages V _{1},V _{2} constructed in Lemma 21, the sets of trees Win^{∃}(V _{1}) and Win^{∀}(V _{2}) can be recognized by weak alternating automata of index (1,κ).
Thus we obtain the following analogue of
Theorem 23
For every κ≥1, the separation property fails for the class of weak regular languages of index (1,κ).
Proof
For index (1,1), the weak automata coincide with the ordinary ones, and this case has been already settled in Proposition 17.
For κ>1, we claim that the sets Win^{∃}(V _{1}) and Win^{∀}(V _{2}) are inseparable. Indeed, as \(\mathrm {Win}^{\exists}(({L}\times\overline{L})\cap M^{2})\subseteq\mathrm {Win}^{\exists}(V_{1})\) and \(\mathrm{Win}^{\forall}((\overline{L}\times L)\cap M^{2})\subseteq\mathrm{Win}^{\forall}(V_{2})\) (by Lemma 21), a separator of Win^{∃}(V _{1}) and Win^{∀}(V _{2}) would also separate \(\mathrm{Win}^{\exists}((L\times \overline{L})\cap M^{2})\) and \(\mathrm{Win}^{\forall}((\overline{L}\times L)\cap M^{2})\), contradicting Fact 20. □
8 Reduction Property
Sets C,D reduce a given pair of sets A,B if C⊆A, D⊆B, C∪D=A∪B and C∩D=∅. The reduction property holds for a class \({\mathcal{C}}\) of sets, if for every \(A,B\in{\mathcal{C}}\), there exist \(C,D\in{\mathcal{C}}\) which reduce A,B. The reduction property for a class \({\mathcal{C}}\) implies readily the separation property for the dual class \(\{ \overline{C}: C\in{\mathcal{C}}\}\). In settheoretic hierarchies, usually the converse also holds, that is the separation property for a class \({\mathcal{C}}\) implies the reduction property for the dual class (see [9]). We will see that it is not the case for the index hierarchy of the alternating automata on trees.
Theorem 24
The reduction property fails for all classes Σ _{ m } and Π _{ m }, for m≥1.
The rest of this section is devoted to the proof of this theorem. We consider a class Σ _{ m } with m≥2; the proof for Π _{ m } is analogous.
Let us start with a simple combinatorial lemma.
Lemma 25
Proof
(⊆) Let (t′,s)∈Y. Suppose s∈W; then (t,s)∈X. (For, it is in X∪Y because of s, but not in Y because of t.) Now the assumption that t′∈A _{ t } implies that (t′,s)∈X as well. (For, if (t,s)∈a _{ i }×b _{ i } then (t′,s)∈a _{ i }×b _{ i } as well.) But this is impossible as X and Y are disjoint; the contradiction proves that \(s \in\overline{W} \).
The next lemma will introduce a decomposition of a recognizable tree language into “rectangles” in the same class. We fix an alphabet A and a letter d∈A. Recall that d(t _{0},t _{1}) denotes a tree with the root labeled by d, and the left and right subtrees t _{0} and t _{1}, respectively. For sets of trees K,L, let d(K,L)={d(t _{0},t _{1}):t _{0}∈K,t _{1}∈L}. Clearly, if K and L are in the class Σ _{ m }, so is d(K,L).
Let ⊤=Tr(A) stand for all trees over alphabet A.
Lemma 26
Proof
Let \(\mathcal{A}\) be an alternating tree automaton of index (ι,κ) recognizing T (like in (5)). For a state q∈Q, let \(\mathcal{A}_{q} \) be an automaton which differs from \(\mathcal{A}\) only in that its initial state is q.
Let t=d(t _{0},t _{1})∈T, and let S be a winning strategy of Eve in the game \(G(\mathcal{A},t)\) (c.f. Sect. 2.3). Recall that nodes in the arena of this game have form (q,v), where q is a state of the automaton and v is a node of the tree; in particular the initial position of the game is (q _{ I },ε). Let L(t) be the set of those histories in S, where the left successor of the root of the tree t (i.e., the node 0) is reached for the first time (possibly after a sequence of εmoves). That is, π is in L(t) if it is in the form (v _{0},v _{1},…,v _{ k }), where v _{0}=(q _{ I },ε), v _{ k }=(q,0), for some q, and v _{ i }=(p _{ i },ε), for i<k, for some states p _{ i }. Let R(t) be defined similarly for the right subtree.
Let L _{ state }(t)={q:last(π)=(q,0),for some π∈L(t)}, and let R _{ state }(t) be defined similarly. Note that \(t_{0} \in L( \mathcal{A}_{q} ) \), whenever q∈L _{ state }(t), as Eve can use a substrategy of S (starting from an appropriate history) to win the game \(G(\mathcal{A}_{q},t_{0} )\). The similar holds for t _{1} and R _{ state }(t). We claim that a kind of converse also holds: if \(t_{0}' \in\bigcap_{q \in L_{\mathit{state}}(t)} L(\mathcal{A}_{q}) \) and \(t_{1}' \in\bigcap_{q \in L_{\mathit{state}}(t)} R(\mathcal{A}_{q}) \) then \(d (t_{0}' , t_{1}') \in T \). Indeed, for each q∈L _{ state }(t), Eve has a winning strategy S _{ q,0} in the game \(G(\mathcal{A}_{q},t_{0}' )\), and the similar holds for \(t_{1}' \). Hence, Eve can combine these strategies with the strategy S. That is, she plays S until the play reaches a history π∈L(t) with last(π)=(q,0), for some q (or π∈R(t) with last(π)=(q,1)), in which case she switches to the strategy S _{ q,0} (or S _{ q,1}). Symbolic presentation of this new strategy in terms of concatenation of sets of histories is easy but cumbersome, and we omit it here.
We now fix an alphabet A and a tree language W over A belonging to the class Σ _{ m }, but not in Π _{ m }. For concreteness, for ι<κ, we can take the set W _{ ι,κ } for appropriate ι,κ, c.f. Sect. 2.3. For the classes Σ _{1} and Π _{1} (c.f. Sect. 6.1), we leave the construction of the respective examples to the reader. We claim that the pair d(⊤,W),d(W,⊤) is not reducible.
Footnotes
 1.
However, the classical notation plays a trick here, as the analogy matches the classes \({\bf\varSigma}^{1}_{1} \sim\varPi_{2} \) and \({\bf\varPi}^{1}_{1} \sim\varSigma_{2} \).
 2.
Thus S contains no other histories than those mentioned in conditions 1–3.
 3.
In fact, even nondeterministic, but we don’t explore it in this paper.
 4.
Strictly speaking, we assume here that the languages X,Y∈Γ _{ m } are over the same alphabet; which matters in the case of Π _{1} discussed below.
 5.
We will later omit this assumption as the results hold also without it. Clearly the languages L _{ ι,κ } and L _{ ι+2ℓ,κ+2ℓ }, for ℓ≥0, have the same properties.
 6.
The existence and uniqueness of this fixed point can be also inferred from the Banach FixedPoint Theorem, c.f. Sect. 5.1.
 7.
More precisely, that author considered the reduction property for the dual classes Σ _{ m }. See a comment after Proposition 3.5 in [16].
 8.
This makes the proofs simpler, although it would be possible to explicitly construct the fixed points like in the proof of Lemma 7.
 9.
 10.
We allow empty unions/intersections, i.e., ∅ and ⊤.
Notes
Acknowledgements
D. Niwiński wishes to thank Victor Selivanov and Wolfgang Thomas for insightful comments after his talk at the Dagstuhl Seminar 11411 (October 2011), in particular for an encouragement to study the problem for weak automata.
References
 1.Arnold, A.: The μcalculus alternationdepth hierarchy is strict on binary trees. RAIRO Theor. Inform. Appl. 33, 329–339 (1999) CrossRefMATHGoogle Scholar
 2.Arnold, A., Michalewski, H., Niwiński, D.: On the separation question for tree languages. In: STACS, pp. 396–407 (2012) Google Scholar
 3.Arnold, A., Niwiński, D.: Fixed point characterization of weak monadic logic definable sets of trees. In: Nivat, M., Podelski, A. (eds.) Tree Automata and Languages, pp. 159–188. Elsevier, Amsterdam (1992) Google Scholar
 4.Arnold, A., Niwiński, D.: Rudiments of μCalculus. Elsevier Science, Studies in Logic and the Foundations of Mathematics, vol. 146. NorthHolland, Amsterdam (2001) Google Scholar
 5.Bradfield, J.C.: Simplifying the modal mucalculus alternation hierarchy. In: Proc. STACS’98. Lect. Notes Comput. Sci., vol. 1373, pp. 39–49 (1998) Google Scholar
 6.Duparc, J., Murlak, F.: On the topological complexity of weakly recognizable tree languages. In: FCT. LNCS, vol. 4639, pp. 261–273 (2007) Google Scholar
 7.Hummel, S., Michalewski, H., Niwiński, D.: On the Borel inseparability of game tree languages. In: STACS, pp. 565–575 (2009) Google Scholar
 8.Kaminski, M.: A classification of omegaregular languages. Theor. Comput. Sci. 36, 217–229 (1985) MathSciNetCrossRefMATHGoogle Scholar
 9.Kechris, A.S.: Classical Descriptive Set Theory. Springer, New York (1995) CrossRefMATHGoogle Scholar
 10.Mostowski, A.W.: Hierarchies of weak automata and weak monadic formulas. Theor. Comput. Sci. 83, 323–335 (1991) MathSciNetCrossRefMATHGoogle Scholar
 11.Moschovakis, Y.N.: Descriptive Set Theory. NorthHolland, Amsterdam (1980) MATHGoogle Scholar
 12.Muller, D.E., Saoudi, A., Schupp, P.E.: Alternating automata, the weak monadic theory of trees and its complexity. Theor. Comput. Sci. 97(2), 233–244 (1992) MathSciNetCrossRefMATHGoogle Scholar
 13.Rabin, M.O.: Weakly definable relations and special automata. In: BarHillel, Y. (ed.) Mathematical Logic and Foundations of Set Theory, pp. 1–23 (1970) Google Scholar
 14.Santocanale, L., Arnold, A.: Ambiguous classes in μcalculi hierarchies. Theor. Comput. Sci. 333, 265–296 (2005) MathSciNetCrossRefMATHGoogle Scholar
 15.Selivanov, V.: Fine hierarchy of regular ωlanguages. Theor. Comput. Sci. 191, 37–59 (1998) MathSciNetCrossRefMATHGoogle Scholar
 16.Selivanov, V.: Fine hierarchy and mreducibilities theoretical computer science. Theor. Comput. Sci. 405, 116–163 (2008) MathSciNetCrossRefMATHGoogle Scholar
 17.Thomas, W.: Languages, automata, and logic. In: Rozenberg, G., Salomaa, A. (eds.) Handbook of Formal Languages, vol. 3, pp. 389–455. Springer, Berlin (1997) CrossRefGoogle Scholar
 18.Wagner, K.: On ωregular sets. Inf. Control 43, 123–177 (1979) CrossRefMATHGoogle Scholar
Copyright information
Open Access This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.