Orbit closed permutation groups, relation groups, and simple groups

Abstract

A permutation group G on a set $\Omega$ is called orbit closed if every permutation of $\Omega$ preserving the orbits of G in its action on the power set $P(\Omega )$ belongs to G. It is called a relation group if there exists a family $R \subseteq P(\Omega )$ such that G is the group of all permutations preserving R. We prove that if a finite orbit closed permutation group G is simple, or is a subgroup of a simple group, then it is a relation group. This result justifies our general conjecture that with a few exceptions every finite orbit closed group is a relation group. To obtain the result, we prove that most of the finite simple permutation groups are relation groups. We also obtain a complete description of those finite simple permutation groups that have regular sets, and prove that (with one exception) if a finite simple permutation group G is a relation group, then every subgroup of G is a relation group.

1 Introduction

Our research is motivated by a problem that arose originally in applications of permutation groups to computer science. In 1991, a problem concerning parallel complexity of languages led Clote and Kranakis [6] to consider the question which permutation groups can be represented as the symmetry groups of boolean functions.

The symmetry group of a boolean function $F:\{0,1\}^n \rightarrow \{0,1\}$ (also called the invariance group) is the set of all permutations $g$ such that

$$\begin{aligned} F(x_{1g}, x_{2g},\ldots ,x_{ng}) = F(x_1,x_2,\ldots ,x_n) \end{aligned}$$

for all $x_1,x_2,\ldots ,x_n \in \{0,1\}$. Denote by BGR(2) the class of all permutation groups that are the symmetry groups of boolean function. The analogous notation BGR(k) stands for the class of all permutation groups that are the symmetry groups of k-valued boolean function, i.e., those that have the arguments in the set $\{0,1\}$, but take values in the set $\{0,1,\ldots , k-1\}$. Finally, BGR is the union of all BGR(k) for any $k\ge 2$.

In [6], as a key result, a theorem was stated that actually $BGR=BGR(2)$, i.e., that every permutation group that can be represented as the symmetry group of a k-valued boolean function can be represented as the symmetry group of an ordinary (2-valued) boolean function. Yet, in [19], this result was shown to be false. A critical gap in the proof was pointed out, and a counterexample was found. It has been observed that the Klein four group $K_4$ in its natural action belongs to BGR(3), but not to BGR(2).

Considering classes BGR(k), we treat permutation groups up to permutation isomorphism (i.e., up to labeling the points being permuted). Treating n-tuples $(x_1,x_2,\ldots ,x_n) \in \{0,1\}^n$ as the indicator functions of subsets of the set of indices $\{1,2,\ldots ,n\}$, each boolean function $F$ on n variables may be considered as a function $F(x)$ from the family of the subsets $x\subseteq \{1,2,\ldots ,n\}$ to $\{0,1\}$, and this is the way we use in this paper. This yields the correspondence of the boolean functions with hypergraphs (defined as families of subsets) and shows that the class BGR(2) may be viewed as the class of the automorphism groups of hypergraphs (cf. [25]). A family of subsets may be also treated as an unordered relation, and the symmetry group of the corresponding boolean function as the invariance group of such a relation. That is why the members of BGR(2) are called relation groups in [7]. The classes BGR(k) correspond to the automorphism groups of colored hypergraphs or relational structures. In particular, BGR is exactly the class of permutation groups that are orbit closed.

In [7], Dalla Volta and Siemons proved that a finite primitive permutation group is orbit closed if and only if it is a relation group (and that most of the primitive groups are relation groups). The same they did for many classes of finite imprimitive groups. On the other hand, in [7, Corollary 5.3], they stated that there are infinitely many examples of imprimitive permutation groups that are orbit closed but not a relation group. Unfortunately, as it was noted in [13], the proof of this claim contains an error. Given as an example the group $C_3\wr L$ and other examples suggested in the proof are indeed not relation groups, but they are not orbit closed either.

So, the intriguing problem of whether there are only a few exceptions to the general equality that $BGR=BGR(2)$ or whether the classes BGR(k) contain many unknown permutation groups (and hence types of boolean functions), remains open. With the current state of knowledge, our guess is the following.

Conjecture. Every finite orbit closed permutation group, but a few exceptions, is a relation group.

So far, the only known exception is $K_4$.

In [12], we have confirmed this conjecture for abelian permutation groups. In fact, our result includes a larger class of permutation groups, namely all subgroups of direct sums of regular permutation groups. It was proved, in particular, that each member of this class, but listed exceptions, is a relation group.

In this paper, we consider the problem for the other extreme: the class of those permutation groups that are simple as abstract groups. The fact that these groups have no nontrivial normal subgroups, on the one hand, and the classification of finite simple groups, on the other hand, make it feasible to verify the conjecture for this class. In particular, looking for other possible exceptions to the general claim, it is natural to inspect this very class.

It follows from the results of [7] that the only primitive simple groups, other than $A_n$, that are not the relation groups are PSL(2, 8) and $C_5$ in their natural actions. Yet, simple groups also have various imprimitive and intransitive actions, and we wish to establish which of the corresponding imprimitive and intransitive permutation groups are relation groups, and which are not.

At this point, the reader should note that the whole problem is focused, in fact, on the intransitive actions of groups. The main difficulty lies in the intransitive case. Both [12] and the present paper illustrate this.

Note that, in most applications, intransitive actions are often more natural than transitive ones. For instance, most graphs are not vertex transitive and the action of the automorphism group on different orbits may be very different and essential with regard to various graph properties. The same is true about groups of automorphisms of any kind of incidence structure. Now, while there are properties of intransitive permutation groups that can be inferred directly from the properties of their transitive constituents, there are combinatorial properties of intransitive permutation groups that depend also on how the actions on different orbits are related to each other. Orbit closeness and being a relation group are such properties. For example, some permutation groups isomorphic abstractly to the alternating group $A_n$ are orbit closed, while some of them are not (see Theorem 6.1).

The paper is organized as follows. We start with recalling the basic notions in Section 2. In the next section, we deal with transitive imprimitive simple permutation groups. We show that all such groups are relation groups. Intransitive actions of simple groups are described in Proposition 5.1. This is relatively easy to do using the description of the general construction of intransitive permutation groups contained in [19]. This construction and some basic results are recalled for the reader convenience in Section 4. Next, using the classification of the simple groups and results from [2, 7, 19, 21], we prove a number of lemmas on intransitive simple groups, leading to our main result, which is given in Theorem 6.1. It provides a complete description of all simple relation groups. All the remaining simple permutation groups, which includes many intransitive groups abstractly isomorphic to alternating groups, are proved not to be orbit closed. Our research yields also additional results characterizing simple permutation groups that have a regular set (Theorem 6.2), and showing that (with one exception) the property that a simple permutation groups is a relation group is inherited by subgroups (Theorem 6.3).

Throughout, many particular cases arise that require some computer calculations. We have used system GAP 4.10.1. In cases of applying GAP, we try to give only those computational details that seem sufficient for the reader to verify our results, using any computational system for permutation groups.

2 Prerequisites

A permutation group G is a subgroup of the symmetric group $Sym(\Omega )$ on a set $\Omega$. In this paper, all permutation groups are finite and considered up to permutation isomorphism [9, p. 17] (i.e., two groups that differ only in labeling of points are treated as the same). A permutation group G is called trivial, if the only permutation in G is the identity. The symmetric group on n elements is denoted by $S_n = Sym(\{1,2,\ldots ,\})$, and the alternating group on n elements is denoted by $A_n$. The cyclic group generated by the permutation $(1,2,\ldots ,n)$ is denoted by $C_n$. We use standard notation for abstract groups (used in [9]) as the notation for permutation groups in their natural actions. The projective special linear groups are denoted by PSL(d, q) or $L_d(q)$ (depending on the source we refer to). We recall that in their natural actions they are permutation groups on $n= (q^d-1)/(q-1)$ elements.

Following [3], we write permutations on the right, and compose from left to right (as in the notation $\alpha (gh) = (\alpha g)h$). The same notation is applied to the action of permutations on subsets or pairs of elements of the underlying set $\Omega$. Yet, the orbit of a set $x \subseteq \Omega$ in the action of $G \le Sym(\Omega )$ on the subsets of $\Omega$ is denoted by $x^G$.

We view boolean functions $F(x)$ as the functions from the set $P(\Omega )$ of the subsets of a set $\Omega$ to the set $\{0,1\}$. The family of sets x with $F(x)=1$ is considered as an unordered relation R on $\Omega$, and the symmetry group of $F$ is the set of permutations of $\Omega$ preserving R (cf. [7]). We will use the notation $G=\mathcal {G}(R)$ for this group, and refer to R as a defining relation for G. More generally, for a k-valued boolean function $F$, the family ${\mathcal R} = (R_0,\ldots ,R_{k-1})$ of relations $R_i$ defined by $F(x)=i$ may be considered as a relational structure $P(\Omega , \mathcal R)$ (of unordered relations), and the symmetry group of $F$ is the set of permutations of $\Omega$ preserving all the relations in ${\mathcal R}$.

The action of a group $G \le Sym(\Omega )$ on the power set $P(\Omega )$ preserves the cardinalities of the subsets, so it has at least $|\Omega |+1$ orbits. If $G=\mathcal {G}(R)$, then obviously R must be a union of some orbits of G in its action on $P(\Omega )$, and this observation is used in looking for defining relations for a permutation group. Some further observations will also be useful in this regard. Note that it does not matter for the definition whether $\Omega \in R$ or not. Next, if G is transitive, then either all singletons belong to R or no singleton belongs to R. The same concerns the complements of singletons in $\Omega$, because the action of G on a set x and $\Omega \setminus x$ is essentially the same. Note also that a permutation u on $\Omega$ preserves R if and only if it preserves the sets of cardinality k in R, for each $1\le k < \Omega$. The arity of the defining relation R is the set of cardinalities $ar(R) =\{|x| : x\in R\}$. In this connection, recall that a permutation group $G \le Sym(\Omega )$ is k-homogeneous if its action on k-element subsets of $\Omega$ is transitive. It is called set-transitive if it is k-homogeneous for every $k\le |\Omega |$. In this paper, we will make use of the description of the set-transitive permutation groups provided in [2].

Given a permutation group $G \le Sym(\Omega )$, a subset $x\subseteq \Omega$ is called regular, if the (setwise) stabilizer of the set x in G is trivial. In such a case G acts regularly on the orbit $x^G$ in the power set $P(\Omega )$. An existence of a regular set for G is equivalent with having a regular orbit in $P(\Omega )$, that is, an orbit of length |G| ( [5, 21]).

The notions of regular set and regular orbit in the set of subsets are closely connected with that of distinguishing number for groups (see [26]). Recall, that the distinguishing number of a permutation group $G \le Sym(\Omega )$, denoted by D(G), is the least positive integer k such that there exists a partition of $\Omega$ into k parts with the property that only the identity of G stabilizes each part. In particular, $D(G)=2$ if and only if G has a regular set [8].

For other related papers on regular sets and regular orbits, see [1, 11, 16, 20, 22, 24]. For some interesting generalizations see [17, 18].

3 Imprimitive simple groups

Let G be a permutation group that is simple as an abstract group. As we have mentioned in the introduction, if G is in addition primitive, then by [7, Theorem 4.2], except for alternating groups $A_n$, and two additional groups, $C_5$ and PSL(2, 8), each simple primitive group is a relation group. Thus, in this section, as the first step, we focus on transitive simple groups that are imprimitive.

Proposition 3.1

Each simple imprimitive permutation group G has a regular set, and all subgroups of G are relation groups.

As it is well known, for transitive permutation groups G (considered up to permutation isomorphism), it is enough to study faithful actions of G on its cosets. In this paper, by a coset we always mean a right coset. By G/H we denote the set of all (right) cosets of a subgroup H in G. The permutation group given by such an action is denoted by (G, G/H). Note that, if G is simple and H is proper, then each action of G on G/H is faithful and gives rise to a transitive permutation group. It is primitive if and only if H is maximal.

Now, given an abstract simple group G, let $N<G$. Suppose that (G, G/N) is not primitive. Then, there exists a maximal subgroup H of G with $N<H<G$. Thus, the group (G, G/H) is primitive. Elements $g\in G$ act faithfully both on G/N and G/H, and the first action respects H-cosets and determines the second action. We will say that a subset $x \in P(G/N)$ is compatible with the cosets of H, or H-compatible, if, for any $g\in G$, the condition $x \cap Hg\ne \varnothing$ implies $x \supseteq Hg$. For such an x, by $\bar{x}$ we denote the corresponding set of cosets of H.

We start from a technical lemma, which allows us to transfer our results into subgroups of a given group. It is a strengthening of [7, Lemma 3.1].

Lemma 3.2

Let $H\le S_n$ be a permutation group and y a regular set in H. Suppose that $H = \mathcal {G}(R)$ and let $R'$ be the relation obtained from R by deleting the sets of cardinality |y|. Let $G = \mathcal {G}(R')$. If $y^{G} \cap R = \varnothing$, then every subgroup of H is a relation group. In particular, this is so, if $|y|\notin ar(R)$.

Proof

Let $K\le H$, and let $Q= y^K$. We show that $K= \mathcal {G}(R\cup Q)$. Obviously K preserves Q, and as a subgroup of H, it preserves R. Consequently, $K \subseteq \mathcal {G}(R\cup Q)$.

Conversely, let $g\in \mathcal {G}(R\cup Q)$. We show that $g\in K$. Let $R''$ be the family of sets in R of cardinality |y|. Then, $R\cup Q = R' \cup R'' \cup Q$. The cardinalities of sets in $R'$ are different than |y|, so $g$ preserves $R'$. It follows that $g\in G$. Hence, $g$ preserves $y^G$, and since $y^G\cap R = \varnothing$, it preserves $Q \subseteq y^G$. It follows that $g$ preserves $R''$, and therefore, $g$ preserves $R=R'\cup R''$. This implies that $g\in H$. And finally, since y is regular in H, and $g$ preserves $Q=y^K$, it follows that $g\in K$, as required. $\square$

Using the above lemma, we are going to prove that most of simple permutation groups G have the following two properties:

(a) G has a regular set,

(b) each subgroup of G is a relation group.

Such groups could be referred to as hereditarily relation groups with a regular set. In order to shorten repetitive wording, we use the symbol $\mathfrak R^*$ for the class of permutation groups G satisfying (a) and (b). Note that if x is a regular set for G, then it is regular for each subgroup of G. So, if $G\in \mathfrak R^*$, then each subgroup of G belongs to $\mathfrak R^*$, as well.

We prove three reduction lemmas.

Lemma 3.3

Let G be a simple group with subgroups N and H such that $N< H < G$. If (G, G/H) is a relation group and has a regular set, then $(G,G/N)\in \mathfrak R^*$.

Proof

Let $R'$ be a relation on G/H such that $\mathcal {G}(R')=(G,G/H)$. Since (G, G/H) is transitive, we may assume without loss of generality that $Hg\in R'$ for any $g\in G$. Let $R_0$ be the corresponding relation on G/N, that is, consisting of all sets $x\subseteq G/N$ such that there exists a set $x'\in R'$ and $x=\{Ng : g\in G \mathrm{\ and } Hg\in x'\}$. We use it to form a relation R on G/N defining (G, G/N). To this end, let y be an H-compatible subset of G/N such that $\bar{y}$ is a regular set of (G, G/H) and $H \notin \bar{y}$. (Such a set exists, since by assumption, (G, G/H) has a regular set, and the complement of a regular set is regular). We define $R_1$ to consist of all sets of the form $(y\cup \{N\})h$ for $h\in G$. Now, we define $R=R_0 \cup R_1$. Since each set in $R_1$ has cardinality not divisible by [H : N], it follows that $ar(R_0) \cap ar(R_1) = \varnothing$. This means that a permutation u preserves R if and only if u preserves each of $R_0$ and $R_1$.

We show first that $\mathcal {G}(R)\supseteq (G,G/N)$. We need to show that if $x\in R$, then $xg\in R$ for any $g\in G$. Indeed, if $x \in R_0$, then this is so, because $g$ preserves $R'$; if $x= (y\cup \{N\})h\in R_1$ for some $h\in G$, then $xg= (y\cup \{N\})(hg)\in R_1$, as well.

We prove the opposite inclusion . Let $u\in \mathcal {G}(R)$. By definition of R, $u$ is a permutation of G/N preserving the cosets of H, so we may consider its action on the set G/H. In this action, $u$ preserves $R'$ (since in its action on G/N it preserves $R_0$), which means that there exists $g\in (G,G/H)=\mathcal {G}(R')$ whose action on G/H is the same as the action of $u$ on G/H. It is enough to show that the action of $u$ on G/N is the same the action of $g$ on G/N.

Note that $w=ug^{-1}$ acts as the identity on G/H and it preserves the relation R (by the previous inclusion). Let $Nh$ be an arbitrary coset of N with $h\in G$. As $(y\cup \{N\})h\in R$, it follows that $(y\cup \{N\})hw\in R$. Consequently, $(y\cup \{N\})hw= (y\cup \{N\})h'$ for some $h'\in G$. Thus, $yhw\cup \{N\}hw= yh'\cup \{N\}h'$. Since y is regular in (G, G/H), and w acts as the identity on H-cosets, $h=h'$, and consequently, $Nhw= Nh$. Thus, $w$ fixes $Nh$ for any $h\in G$, which means that it acts as the identity on G/N, proving that G is a relation group.

From what we have established it follows also that $y\cup \{N\}$ is a regular set in (G, G/N). Now, the complement of $y\cup \{N\}$ is also regular, and its cardinality is not in ar(R), unless $[H:N]=2$, and $|y\cup \{N\}| =[G:N]/2$. In the latter case $y\cup \{N,Ng'\}$, where $Ng'\notin y$ and $g'\notin N$, is also regular and satisfies the assumptions of Lemma 3.2. Thus, $G\in \mathfrak R^*$, as required. $\square$

In the next lemma, we use the idea of coloring blocks of a partition that appears in the proof of [10, Lemma 2.7].

Lemma 3.4

Let G be a simple group with subgroups N and H such that $N< H < G$. If $[H:N]\ge [G:H]-1$, then $(G,G/N) \in \mathfrak R^*$.

Proof

Denote $[G:H]=n$, and let $s_1,\ldots ,s_{n}$ denote the partition of G/N into sets corresponding to H-cosets of G. Then, $|s_i| = [H:N] \ge n-1$.

Let y be a subset of G/N such that for all $i,j \le n$, if $i\ne j$ then the cardinality $|y\cap s_i|\ne |y\cap s_j|$. Such a set y exists, since $|s_i| =[H:N]\ge n-1$ for all i. We may assume that 1 and [H : N] are among the cardinalities $|y\cap s_i|$, which guarantees that $|y|>|s_i|$. To fix the notation, we assume that it is $s_1$ with $|y\cap s_1|=1$.

We define a relation R as one consisting of all sets $s_i$ and all sets $x= yh$ for any $h\in G$. Note that, since $|y|\ne |s_i|$, the arity $ar(R) = \{|s_i|,|y|\}$.

The most important property of y is that as $|y\cap s_i|$ gives an injective labeling of H-cosets, y may be used to decode the permutations of H-cosets. More precisely, the mapping $\phi : G/H \rightarrow \{0,1,\ldots ,n\}$ which assigns the value $|y\cap s_i|$ to the H-coset corresponding to the set $s_i$, is injective. Hence, the following holds:

$(*)$ if u and w are arbitrary permutations of G/N that preserve the partition into sets $s_i$, then the condition $yu=yw$ implies that u and w have the same induced action on the sets $s_i$ (H-cosets).

Now, similarly as in the previous proof, it is easy to see that $\mathcal {G}(R) \supseteq (G,G/N)$. For the opposite inclusion, assume that $u\in \mathcal {G}(R)$. Then, since $|y|\ne |s_i|$, $yu = yh$ for some $h\in G$. By $(*),$ $u$ acts on H-cosets as $h$. Hence, $w=uh^{-1}$ acts on G/H as the identity and preserves R. Again, all we need to show is that w acts as the identity on G/N.

Suppose to the contrary that there exists a coset $\beta \in G/N$ such that $\beta w\ne \beta$. Let $\alpha$ denote the unique N-coset in $y\cap s_1$, and let $g\in G$ be such that $\alpha g= \beta$. Then, for $v=gwg^{-1}$ we have: (i) $\alpha v\ne \alpha$, (ii) $v$ acts as the identity on H-cosets, and (iii) $v$ preserves R. By (iii), $yv= yh'$ for some $h'\in G$. By (ii) and the definition of y, $yv=y$. Consequently, $v$ fixes $y \cap s_1 = \{\alpha \}$, which is a contradiction with (i). This proves that (G, G/N) is a relation group.

From the proof, it follows that y is a regular set in (G, G/N). We show that there exists a point $\alpha =Ng$ such that $y'=y\cup \{\alpha \}$ is regular, as well. Indeed, if it is possible to add a point to y with keeping the property $(*)$, then we are done. Otherwise, if adding a point always causes that two sets $y\cap s_i$ and $y\cap s_j$, for $i\ne j$, become equinumerous in $y'$ (which happens when $[H:N]=[G:H]-1$), then we can choose an arbitrary pair i, j and add a point in such a way that $|y'\cap s_i|=|y'\cap s_j|=1$, and otherwise, the cardinalities $|y'\cap s_i|$ are pairwise different. Such a set $y'$ is regular in (G, G/N), by the same argument as that applied to y, unless the transposition $(i,j)\in (G,G/N)$. Since the pair i, j can be chosen arbitrarily, (G, G/N) has a regular set of cardinality $|y|+1 \notin R$ (unless $(G,G/N)=S_n$, which is not the case). Applying Lemma 3.2 completes the proof. $\square$

Combining the approaches of the two above lemmas, we may get still something more.

Lemma 3.5

Let G be a simple group with subgroups N and H such that $N< H < G$. If (G, G/H) is a relation group and $[H:N]\ge D(G,G/H)-1$, then (G, G/N) is a relation group and has a regular set. If in addition, D(G, G/H) does not divide [G : H] or $[H:N]> D(G,G/H)-1$, then $(G,G/N)\in \mathfrak R^*$.

Proof

We use the ideas of the previous proof. Denote $d=D(G,G/H)$, and let $s_1,\ldots ,s_{n}$ denote a partition of G/N into sets corresponding to H-cosets of G. Then, $|s_i| = [H:N] \ge d-1$. In addition, we consider the partition $t_1,\ldots , t_{d}$ of $s_1,\ldots , s_{n}$ into d sets corresponding to the distinguishing partition of (G, G/H).

Let $c_1, c_2,\ldots ,c_d$ be a sequence of pairwise different numbers with $c_1=1, c_d=[H:N]$, and $0\le c_i \le [H:N]$, otherwise. The condition $[H:N]\ge D(G,G/H)-1$ guarantees that such a sequence exists. Let y be a subset of G/N such that for each $s_i\in t_j$, $|y\cap s_i|=c_i$. Then, $|y|>|s_i|$.

As in the proof of Lemma 3.3, let $R_0$ be a relation on G/N corresponding to a relation $R'$ on G/H such that $(G,G/H)= \mathcal {G}(R')$. Again, we assume that $Hg\in R'$ for all $g\in G$.

We define $R=R_0\cup R_1$, where $R_1$ consists of all sets $x= yh$ for any $h\in G$. We have $ar(R) = ar(R_0) \cup ar(R_1)$. A problem, comparing with the previous proof, is that this union may not be disjoint. However, all we need is the fact that each permutation v preserving R preserves $R_1$. Indeed, since all $s_i\in R$ and $|y|>|s_i|$, v preserves H-cosets, and since $yh$ are the only sets in R that are not the unions of H-cosets, v preserves $R_1$.

Further, since G acts faithfully on G/H, and $d=D(G,G/H)$ is the distinguishing number for (G, G/H), we have the following property of y (which is enough for our purposes in this case):

$(**)$ if $h\in G$ and $h\ne 1$, then $|y\cap s_i| \ne |yh\cap s_i|$ for some $i\le n$.

Again, it is easy to see that $\mathcal {G}(R) \supseteq (G,G/N)$. For the opposite inclusion, assume that $u\in \mathcal {G}(R)$. Then, since $u$ preserves $R_1$, it acts on H-cosets as some $h\in G$. Hence, $w=uh^{-1}$ acts on G/H as the identity and preserves R. All we need to show is that w acts as the identity on G/N. This part is the same as in the previous proof.

Again, from the proof it follows that y is a regular set in (G, G/N). We wish to modify it so it is still regular, but also satisfies the assumptions of Lemma 3.2. If $[H:N]> D(G,G/H)-1$, then we have room to change one of the numbers $c_i$, $i\ne 1$, so that the resulting set $y'$ has the properties of y, but $|y'|\ne |y|$. If D(G, G/H) does not divide [G : H], then the partition $t_1,\ldots ,t_d$ of $s_1,\ldots ,s_n$ has blocks of different cardinalities, and then it is enough to change the order of $c_1,\ldots ,c_d$ to get regular sets satisfying the assumptions of Lemma 3.2, proving the claim. $\square$

Remark

We do not need it in this paper, but it is worth noting that the three lemmas above hold for arbitrary groups $G> H > N$, as long as both actions on G/H and G/N are faithful, and G is different from $S_n$.

We will combine the above lemmas with what is known on primitive groups. In [21], Seress lists all primitive groups that have no regular set (cf. [7, Theorem 2.2]). Using this list one can distinguish all simple groups in primitive action that have no regular set. In the list below the first entry denotes the degree of the action, and the second—the group itself. If the group is abstractly isomorphic to one of $A_n$ or another group in the list with a different name, then this is indicated. There are no other isomorphisms between the groups in the list except those indicated. (This may be inferred from the classification of the finite simple groups, using, e.g., [9, Appendix A].)

${\mathcal L} = \{(6, L_2(5)\!\cong \! A_5)$, $(7, L_3(2)\!\cong \! L_2(7))$, $(8, L_2(7)\!\cong \! L_3(2))$, $(9, L_2(8))$,

      $(10, L_2(9)\!\cong \! A_6)$, $(11, L_2(11))$, $(11, M_{11})$, $(12, M_{11})$, $(12, M_{12})$,

      $(13, L_3(3))$, $(15, L_4(2) \!\cong \! A_8)$, $(22, M_{22})$, $(23, M_{23})$, $(24, M_{24})\}$.

Combining this with [7, Theorem 4.2], we obtain the following:

Lemma 3.6

Let G be a simple primitive permutation group of degree $n\ge 2$ different from $A_n$. Then, the following hold:

• (i) G has a regular set if and only if $G\notin {\mathcal L}$.

• (ii) G is a relation group if and only if $G\ne C_5, PSL(2,8)$.

Now we prove the main result of this section. We use the notation $G^+$ for the permutation group obtained from G by adding an extra fix point.

Proof of Proposition 3.1

Let G be a simple permutation group that is transitive but not primitive. Then, it is permutation isomorphic to (G, G/N), where N is proper and not maximal subgroup of G. Hence, there exists a maximal subgroup H of G such that $N< H < G$. Since H is maximal, (G, G/H) is primitive. If (G, G/H) is a relation group and has a regular set, then the result holds by Lemma 3.3. Otherwise, (G, G/H) is either $A_n$ or one of the fifteen groups listed in Lemma 3.6.

First, we consider the case when $(G,G/H)=A_n$. It follows that $H=A_{n-1}^+$, $[G:H]=n$, and for $n> 5$, we have $[H:N']=[A_{n-1}^+:N'] \ge n-1$ for any $N' < A_{n-1}^+$. Then, by Lemma 3.4, (G, G/N) is a relation group and has a regular set, as required. If $n=5$, $H=A_4^+$, and $[G:H]=5$. Then, up to conjugation, there are 4 proper subgroups N of H (this can be checked using GAP or other systems for computation in permutation groups). For three of these subgroups, the index $[H:N] \ge 4$. Hence, we may use Lemma 3.4 to infer the required claim. The lemma does not apply only in the case when N is of order 4. In this case, $N=K_4^+$ is the Klein 4-group extended by a one fixed point.

We compute this case directly using GAP. Here (G, G/N) is $A_5$ acting on 15 elements. We can find it in the list of transitive groups as the group T numbered 15T5, generated by permutations

$$\begin{aligned} g&=(1,9,10,3,14)(2,15,7,12,6)(4,5,11,13,8), \mathrm{\ and} \\ h&=(1,4,10)(2,5,8)(3,7,11)(6,9,15)(12,14,13), \end{aligned}$$

on the set $\Omega =\{1,\ldots ,15\}$. It is imprimitive group with blocks

$$\begin{aligned} B=\{\{ 1, 6, 8 \}, \{ 2, 4, 9 \}, \{ 3, 7, 11 \}, \{ 5, 10, 15 \}, \{ 12, 13, 14 \}\}. \end{aligned}$$

These blocks and the orbits of some sets in T are used to define T by a relation. Let $R_1=\{1,2\}^T$, $R_2= (\Omega \setminus \{1,2,4\})^T$, and let $R=B\cup R_1 \cup R_2$. Then, it is not difficult to check that $T= G(R)$. It is enough to observe that $\mathcal {G}(R)$ is contained in $\mathcal {G}(B)$, which is isomorphic to $S_3 \wr S_5$, and to verify computationally that the cardinality of the stabilizer of $R_1 \cup R_2$ in $\mathcal {G}(B)$ is exactly $|A_5|$. In addition, one can easily check that T has regular sets of sizes from 3 to 12. By Lemma 3.2, $T\in \mathfrak R^*$, as required.

(To find regular sets, we have simply asked GAP about the cardinality of the orbits in the action of G on k-element subsets, for each k. If this cardinality is equal to the order of G, it means that each set in the orbit is regular. In other places of this paper, we have used this approach, we just give information about cardinalities of regular sets.)

Now, consider the cases when (G, G/H) is one of the fifteen groups listed in Lemma 3.6. First we assume that (G, G/H) is a relation group, i.e., is different from PSL(2, 8) or $C_5$. We intend to combine Lemma 3.5 with [8, Theorem 1] and Table 2 in Section 3, which shows that $D(G,G/H)\le 4$ for these groups. If $D(G,G/H) =2$ or $D(G,G/H) =3$ and [G : H] is not divisible by 3, then by Lemma 3.5, $(G,G/N)\in \mathfrak R^*$, as required.

If $D(G,G/H) =3$ divides [G : H], then [8, Table 2] shows that (G, G/H) is $M_{11}(12)$ acting on 12 elements or $M_{24}$ in the natural action. For $(G,G/H) = M_{11}(12)$, up to conjugation, there is only one subgroup H of $M_{11}$ of index 12, and all subgroups $N<H$ have index [H : N] at least 11. By Lemma 3.5, it follows that $(G,G/H)\in \mathfrak R^*$. For $(G,G/H) = M_{24}$ there is only one subgroup H of $M_{24}$ of index 24, and all subgroups $N<H$ have index [H : N] at least 23. Again, the claim follows by Lemma 3.5.

For $D(G,G/H) =4$, we have only 3 groups to consider: $PSL(3,2), M_{11}$, and $M_{12}$, and this requires some computation.

Case 1. Let $(G,G/H)=PSL(3,2)$. Then, as H we may take any subgroup of index 7 (since we know that PSL(3, 2) has only one primitive action on 7 elements). Now, for any subgroup N of H of index $[H:N]>2$, by Lemma 3.5, $(G,G/N)\in \mathfrak R^*$, as required. Up to conjugation, there is only one subgroup N of H of index 2. Then, (G, G/N) is a transitive permutation group of degree 14 abstractly isomorphic to PSL(3, 2). This is $T=14\textrm{T}10$ imprimitive group generated by permutations

$$\begin{aligned} g&=(1,5,9,13,3,7,11)(2,6,10,14,4,8,12), \mathrm{\ and} \\ h&= (1,10,6,14,11,9,12)(2,5,8,3,13,7,4), \end{aligned}$$

with blocks

$$\begin{aligned} B = \{\{1,8\},\{2,9\},\{3,10\},\{4,11\},\{5,12\},\{6,13\},\{7,14\}\}. \end{aligned}$$

Now, one can check that for $R=B\cup \{1,2,3,4\}^T$, $T=\mathcal {G}(R)$. Also, it has regular sets of all sizes from 3 to 11, which means that $T\in \mathfrak R^*$, as well.

Case 2. Let $(G,G/H)=M_{11}$. Then, H is the only subgroup of index 11. By Lemma 3.5, for any subgroup N of H of index $[H:N]>3$, $(G,G/N)\in \mathfrak R^*$. Up to conjugation, there is only one subgroup N of H of index less than 4. It is the subgroup N of H of index 2 isomorphic to $A_6$. Then, (G, G/N) is the transitive permutation group $T=22\textrm{T}22$, of degree 22.

We describe how to obtain a relation defining T. First, we take the two-element blocks of imprimitivity and consider the action of $M_{11}$ on these blocks. Now, $M_{11}=\mathcal {G}(R)$ for some relation R (pointed out in [7]). We take the induced relation $R'$ on the two-element blocks of T (obtained by replacing each point in R by two points of the corresponding block). Then, all the sets in $R'$ are of an even cardinality. It remains to check that there exists a 3-element set whose T-orbit together with $R'$ yields a relation generating T (since almost all 3-element sets have this property, we omit the details). In addition, one easily checks that T has regular sets of all sizes from 6 to 18. Since, for example, $7\notin ar(R)$, by Lemma 3.2, $T\in \mathfrak R^*$, as required.

Case 3. Let $(G,G/H)=M_{12}$. Since there is only one action of $M_{12}$ on 12 elements, one may take as H any of the two subgroups of index 12. As before, by Lemma 3.5, for any subgroup N of H of index $[H:N]>4$, $(G,G/N)\in \mathfrak R^*$. Since there is no subgroup N of H of index less than 12, the case is completed.

To complete the proof, it remains to consider two groups that are not relation groups. For $C_5$ we have nothing to prove, since it has no nontrivial subgroups. For $(G,G/H)=PSL(2,8)$, we apply Lemma 3.4, as follows.

First, H is the only subgroup of index 9. By Lemma 3.4, for any subgroup N of H of index $[H:N]\ge 8$, $(G,G/N)\in \mathfrak R^*$. Up to conjugation, there is only one subgroup N of H of index less than 8. This is a group isomorphic abstractly to $C_2^3$. The group $T=(G,G/N)$ is of degree 63, which makes it hard for a direct computation.

We describe how one can modify the proof of Lemma 3.5 to handle this special case. The only assumption that fails to hold is that (G, G/H) is a relation group. So, rather than the group (G, G/H) itself we consider a construction using two-point stabilizer of (G, G/H), which is in this case $G_{\alpha ,\beta } = C_7^{++}$ (the cyclic group on 7 elements with two extra fix points).

So, let $s_1,\ldots ,s_{n}$, as before, be a partition of G/N into the sets corresponding to H-cosets of G, and let $s_1$ and $s_2$ correspond to the H-cosets $\alpha$ and $\beta$. Let $R'$ be a relation on $G/H \setminus (\alpha \cup \beta )$ such that $C_7=\mathcal {G}(R')$ and corresponding to the stabilizer $G_{\alpha ,\beta }$ (see, e.g., [19, p. 385] for such a relation). We construct the corresponding relation on G/N: for each $x\in R'$ by $x'$ we denote the subset of G/N consisting of all cosets $Ng$ such that $Hg\in x$. In addition, we choose three points: $a\in s_1$ and $b,c\in s_2$, and we put $x'' = x'\cup \{a,b,c\}$. By $R''$ we denote the family of all sets $(x'')g$ for all $x\in R'$ and $g\in G$. We put $R_0=R'' \cup \{s_1,\ldots ,s_{n}\}$. The rest of the proof is the same as that of Lemma 3.5. Checking that this construction works is left to the reader. $\square$

4 Subdirect sum of permutation groups

In the next section, we consider intransitive permutation groups that are simple. Here, the main tool is a result describing the construction of intransitive permutation groups in terms of the subdirect sum of permutation groups [19]. In order to make this paper self-contained, we recall here the notion of the subdirect sum and the basic result on it.

Given two permutation groups $G \le Sym(\Omega )$ and $H \le Sym(\Delta )$, the direct sum $G\oplus H$ is the permutation group on the disjoint union $\Omega \cup \Delta$ defined as the set of all permutations $(g,h)$, $g\in G, h\in H$ such that

$$\begin{aligned} \alpha (g,h) = \left\{ \begin{array}{ll} \alpha g, &{} \text{ if } \alpha \in \Omega \\ \alpha h, &{} \text{ if } \alpha \in \Delta \end{array}\right. \end{aligned}$$

Thus, in $G\oplus H$, permutations of G and H act independently in a natural way on the disjoint union of the underlying sets. (Abstractly this is a direct product of groups $G_1$ and $G_2$; we call it a sum to distinguish it from another construction of the direct product of permutation groups acting naturally on $\Omega \times \Delta$.)

Now we define the notion of the subdirect sum following [12] (and the notion of intransitive product in [19]). Let $H_1\lhd \; G_1 \le S_n$ and $H_2\lhd \; G_2\le S_m$ be permutation groups such that $H_1$ and $H_2$ are normal subgroups of $G_1$ and $G_2$, respectively. Suppose, in addition, that factor groups $G_1/H_1$ and $G_2/H_2$ are (abstractly) isomorphic and $\phi : G_1/H_1 \rightarrow G_2/H_2$ is the corresponding isomorphism. Then, by

$$\begin{aligned} G = G_1[H_1] \oplus _\phi G_2[H_2] \end{aligned}$$

we denote the subgroup of $G_1 \oplus G_2$ consisting of all permutations $(g,h)$, $g\in G_1, h\in G_2$, such that $\phi (H_1g) = H_2h$. Each such group will be called a subdirect sum of $G_1$ and $G_2$. (Abstractly, this is a subdirect product of the groups $G_1$ and $G_2$.)

If $H_1=G_1$ and $H_2=G_2$, then $G = G_1 \oplus G_2$ is the usual direct sum of $G_1$ and $G_2$. If $H_1$ and $H_2$ are trivial one-element subgroups, $\phi$ is the isomorphism of $G_1$ onto $G_2$, and the sum is called, in such a case, the parallel sum of $G_1$ and $G_2$. Then, the elements of G are of the form $(g,\phi (g))$, $g\in G_1$, and both the groups act in a parallel manner on their sets via isomorphism $\phi$. In this case, we use the notation $G=G_1||_\phi G_2$, where $\phi$ is an (abstract) isomorphism between $G_1$ and $G_2$, or simply $G=G_1|| G_2$ if there is no need to refer to $\phi$. Note that $G_1$ and $G_2$ need to be abstractly isomorphic, but not necessarily permutation isomorphic, and they may act on sets of different cardinalities.

In the special case when, in addition, $G_1=G_2=G$ and $\phi$ is the identity, we write $G^{(2)}$ for $G||_\phi G$. More generally, for $r\ge 2$, by $G^{(r)}$ we denote the permutation group in which the group G acts in the parallel way (via the identity isomorphisms) on r disjoint copies of a set $\Omega$. This group is called the parallel multiple of G, and its elements are denoted by $g^{(r)}$ with $g\in G$. In particular, we admit $r=1$ and put $G^{(1)}=G$. For example, the cyclic group generated by the permutation $g= (1,2,3)(4,5,6)(7,8,9)$ is permutation isomorphic to the parallel multiple $C_3^{(3)}$.

The main fact established in [19] is that every intransitive group has the form of a subdirect sum, and its components can be easily described.

Theorem 4.1

[19, Theorem 4.1] Let G be an intransitive group acting on a set $\Omega = \Omega _1 \cup \Omega _2$ in such a way that $\Omega _1$ and $\Omega _2$ are disjoint fixed blocks of G. Let $G_1$ and $G_2$ be restrictions of G to the sets $\Omega _1$ and $\Omega _2$, respectively. Let $H_1 \le G_1$ and $H_2 \le G_2$ be the subgroups fixing pointwise $\Omega _2$ and $\Omega _1$, respectively. Then, $H_1$ and $H_2$ are normal subgroups of $G_1$ and $G_2$, respectively, the factor groups $G_1/H_1$ and $G_2/H_2$ are abstractly isomorphic, and

$$\begin{aligned} G = G_1[H_1] \oplus _\phi G_2[H_2], \end{aligned}$$

where $\phi$ is an isomorphism of the factor groups.

In this paper, we also apply other results established in [19], and the reader is referred there in case of any doubts. In particular, recall that alternating groups $A_n$, for $n\ge 3$, are not the symmetry groups of any k-valued boolean function, i.e., they are no orbit closed. Also, the cyclic groups $C_3=A_3, C_4$ and $C_5$ are not orbit closed.

In turn, $S_n\in BGR(2)$ for any $n>1$. Also, for any $k>1$, if groups $G,H\in BGR(k)$, then $G\oplus H \in BGR(k)$, and for any $r\ge 2$, if $H\in BGR(k)$, then $H^{(r)} \in BGR(k)$ [19, Theorems 3.1 and 4.3]. In other words, if G and H are orbit closed, then so are $G\oplus H$ and $H^{(r)}$. These results will be treated in our proofs as known, without further comments.

There is some subtlety regarding the parallel powers we have to be aware of. A well-known fact is that the automorphism group Aut(G) of a group G may have outer automorphisms that are not given by the conjugation action of an element of G. In the case of permutation groups $G\le Sym(\Omega )$, some outer automorphism may still be given by the conjugation action of an element in $Sym(\Omega )$. This corresponds generally to permuting elements of $\Omega$, and such automorphisms are called permutation automorphisms. They form a subgroup of Aut(G), which we denote by PAut(G). Often $PAut(G)=Aut(G)$, but some permutation groups also have other automorphisms, which we will call nonpermutation automorphisms.

Now, if $G = H\oplus _\psi H$, for some permutation group H, where $\psi \in PAut(G)$, then G is permutation isomorphic to $H^{(2)}$ (i.e., $G=H^{(2)}$, according to our convention). If $\psi$ is a nonpermutation automorphism, then $G \ne H^{(2)}$. (More precisely, we should speak here about isomorphisms induced by automorphisms and make distinction between base sets of components, but we assume that this is contained in the notion of the disjoint union, and we will make it explicit only when the need arises.) An example is the alternating group $A_6$ that has a nonpermutation automorphism $\psi$ (cf. [4]). Then, $A_6^{(2)}$ and $A_6 ||_\psi A_6$ are not permutation isomorphic.

5 Intransitive simple groups

Without loss of generality we may assume that permutation groups G we consider have no fixed points. Indeed, if G has fixed points, then $G= G'\oplus I_m$, where $I_m$ is the trivial group on a m-element set and $G'$ has no fixed points. All subgroups $H\le G$ are of the form $H'\oplus I_m$, where $H'\le G'$. Hence, it is easy to see that G is simple if and only if $G'$ is simple. Moreover, by the results in [19] we know that $I_m \in BGR(2)$ for all $m\ge 1$, and $G\in BGR(k)$ whenever $G'\in BGR(k)$. On the other hand, if $G'\notin BGR$, then $G\notin BGR$. So if we prove the dichotomy that either $G'\in BGR(2)$ or $G'\notin BGR$, that is, either $G'$ is a relation group or G is not orbit closed, the results extend immediately on G with fixed points. Thus, in the rest of the paper, it is assumed tacitly that permutation groups in question have no fixed points.

Our first result describes the structure of simple permutation groups.

Proposition 5.1

Let $G = H[H']\oplus _\phi K[K']$ be an intransitive permutation group which has no nontrivial normal subgroups. Then, the groups H and K are simple and isomorphic as abstract groups, and the groups $H'$ and $K'$ are trivial. In particular, G is the parallel sum $G = H||_\phi K$ for some isomorphism $\phi$.

Proof

First note that the group $H'\oplus I_m$ is a normal subgroup of G (since $H'\lhd H$). As, by our general assumption, K is nontrivial, we infer that $H'$ is trivial. Similarly, we observe that $K'$ is trivial. It follows that $\phi$ is an isomorphism between H and K, and G is isomorphic to both H and K. Therefore, H and K are simple. $\square$

The proposition above means that intransitive simple permutation groups with nontrivial orbits always have the form of parallel sums. Some remarks are needed to make a proper use of this result.

Note that (using the inverse isomorphism $\phi ^{-1}$) we see easily that $G_1||_\phi G_2$ and $G_2 ||_{\phi ^{-1}} G_1$ are permutation isomorphic, and since we treat permutation isomorphic groups as identical, the operation of the parallel sum may be considered to be commutative. Further, decomposing each summand step by step we can get a decomposition into transitive components. In particular, in $G=(G_1 ||_\phi G_2)||_\psi G_3$ all the involved groups must be abstractly isomorphic, and G is permutation isomorphic with $G_1 ||_{\phi '} (G_2||_{\psi '} G_3)$, where isomorphisms $\phi '$ and $\psi '$ are suitably determined by $\phi$ and $\psi$. Thus, we may also consider this operation to be associative (up to permutation isomorphism).

So, generally, a simple intransitive permutation group is a parallel sum of two or more transitive components that are all abstractly isomorphic, and the action on the union of orbits is given by a system of suitable isomorphisms between components. (Note that, in general, pointing out a simple abstract group and the cardinalities of the orbits is not enough to determine uniquely, up to permutation isomorphism, the corresponding permutation group. This is so, because some simple groups have nonequivalent actions on a set of a given cardinality For example, projective symplectic group PSp(4, 3) has two nonequivalent actions on the set of cardinality $n=40$.)

The following lemma gives some sufficient conditions for a parallel sum to be a relation group.

Lemma 5.2

Let $G= H ||_\phi K$ be a parallel sum of two groups. If H is a relation group and has a regular set y, then $G\in \mathfrak R^*$.

Proof

Assume that $H\le Sym(\Omega )$ and $K \le Sym(\Delta )$ with $\Delta = \{\alpha _1,\alpha _2,\ldots , \alpha _m\}$. Let $R_0$ be a subset of $P(\Omega )$ such that $\mathcal {G}(R_0)=H$, and denote $\Delta _i = \{\alpha _1,\alpha _2,\ldots , \alpha _i\}$. We define two subsets of $P(\Omega \cup \Delta )$.

$$\begin{aligned} R_1&=\{x\cup \Delta : \; x\subseteq \Omega ,\; |x|= |\Omega |-1\},\\ R_2&=\{ (y\cup \Delta _i)g: \; 0<i<m, \; g\in G \}. \end{aligned}$$

Now, we put $R=R_0\cup R_1\cup R_2$. We show that $G=\mathcal {G}(R)$. First, it is easy to see that each relation $R_0,R_1,R_2$ is preserved by permutations in G, and therefore, $G \subseteq \mathcal {G}(R)$. We prove the opposite inclusion.

Note that sets in $R_0$ are contained in $\Omega$, while each set in $R_1$ or $R_2$ has a nonempty intersection with $\Delta$. Further, each set in $R_1$ contains $\Delta$, which is not the case for any set in $R_2$. It follows that the relations $R_0,R_1,R_2$ are mutually disjoint. Moreover, since the sets in $R_1$ have all cardinality $|\Omega |+|\Delta |-1$, larger than the sets in $R_0$ and $R_2$, it follows that any permutation $g$ preserving R, preserves $R_1$ itself. Since the elements of $R_1$ are the complements of one-element subsets of $\Omega$, this means that each $g\in \mathcal {G}(R)$ preserves the partition into $\Omega$ and $\Delta$. In particular, each such permutation may be presented as $g=hk$ with $h$ and $k$ acting on the sets $\Omega$ and $\Delta$, respectively.

If so, $g$ preserves also $R_0$ and $R_1$, individually. In particular, the action of $\mathcal {G}(R)$ on $\Omega$ is contained in the action of H, i.e., $h\in H$. This proves that $\mathcal {G}(R) \subseteq H \oplus K'$, for some $K'\le Sym(\Delta )$. In order to use Theorem 4.1, we need to show that the pointwise stabilizers of $\Omega$ and $\Delta$ in $\mathcal {G}(R)$ are both trivial.

For the first, suppose that $g=hk\in \mathcal {G}(R)$ fixes each point of $\Omega$. This means that h is the identity, $h=1$. Thus, for each $i<m$, we have that $(y\cup \Delta _i)g= y \cup \Delta _ik$. This must belong to $R_2$ (since $R_2$ is preserved by $g$), which means that $y \cup \Delta _ik= (y\cup \Delta _i)g_i$, for some $g_i\in G$. The only $g_i\in G$ fixing y is $g=1$. Therefore, $\Delta _ik= \Delta _i$ for all $i<m$, which means that $k=1$, as required.

Suppose, in turn, that $g=hk\in \mathcal {G}(R)$ fixes each point $\kappa _i\in \Delta$. Then, for each $i<m$, $(y\cup \Delta _i)g= yh\cup \Delta _i$, and since $R_2$ is preserved by $g$, $yh\cup \Delta _i=(y\cup \Delta _i)g_i$ for some $g_i\in G$. Now, $(y\cup \Delta _i)g_i = yh_i \cup \Delta _ik_i$ for some $h_i\in H$ and $k_i \in K$. Thus, for each $i<m$,

$$\begin{aligned} yh\cup \Delta _i = yh_i \cup \Delta _ik_i. \end{aligned}$$

Since $g_i \in G=H||_\phi K$, $k_i$ is uniquely determined by $h_i$, and since y is regular in H, $g_i=h$, for each $i<m$, and $k_i = k$ does not depend on i. Therefore, for each $i<m$, $\Delta _i = \Delta _ik$ for some $k\in K$. In view of the definition of $\Delta _i$, this implies that $k=k_i=1$. Consequently (since $g_i \in H||_\phi K$), $h_i=h=1$, as required. By Theorem 4.1, $\mathcal {G}(R) = H||_\psi K'$ for some $K'\le Sym(\Delta )$ and some isomorphism $\psi$ between H and $K'$.

As we have proved that $G \subseteq \mathcal {G}(R)$, we have $H||_\phi K \subseteq H||_\psi K'$, which in view of the finiteness of the groups involved (and the fact that $\psi$ and $\phi$ are bijections) implies $K'=K$, proving that G is a relation group.

It remains to prove that G has a suitable regular set, in order to apply Lemma 3.2. First, observe that, since y is regular in H, the complement $\Omega \setminus y$ is also regular in H. It follows that we may assume that $|y|\ne |\Omega |-1$, taking the complement of y in $\Omega$, if necessary. In turn, since y is regular in H, the set $y\cup \Delta$ is regular in $G=H||_\phi K$. Now, the relation $R'$ obtained from R by deleting the sets of cardinality $|y|+|\Delta |$, contains $R_1$, and therefore $G'=\mathcal {G}(R')$ preserves the partition into $\Omega$ and $\Delta$. It follows that the orbit $(y\cup \Delta )^{G'}$ is disjoint with R. Applying Lemma 3.2 completes the proof. $\square$

Below we consider the parallel multiple $A_n^{(r)}$ of the alternating group $A_n$. As we will see, it is a relation group if and only if r is sufficiently large with regard to n.

Lemma 5.3

For every $n\ge 3$ and $r\ge 1$, the parallel multiple $A_n^{(r)}$ is a relation group if and only if $2^r\ge n$. Moreover, if $2^r\ge n$, then $A_n^{(r)} \in \mathfrak R^*$; otherwise, $A_n^{(r)}$ is not orbit closed.

Proof

For $r=1$ the claim is that $A_n$ is not orbit closed, which is the case. So we assume that $r\ge 2$. First we consider the parallel multiple of $S_n$ and construct a relation Q such that $S_n^{(r)} = \mathcal {G}(Q)$. (By [19], we know that $S_n^{(r)}$ is a relation group for any $r\ge 2$, but we need an explicit construction of the set Q to use in the further part of the proof.)

The group $S_n^{(r)}$ acts on a set U consisting of r disjoint copies of $\Omega =\{1,2,\ldots ,n\}$, and for this proof, we may assume $U = \{(i,j) : 1\le i \le r, 1\le j \le n \}$, meaning that, for a fixed i, the elements (i, j) form the i-th copy of $\Omega$. We define a relation $Q\subseteq P(U)$ such that $S_n^{(r)} = \mathcal {G}(Q)$. We put

$$\begin{aligned} Q = \{\{(1,j)\} : 1\le j\le n\} \; \cup \; \{\{(i,j),(i+1,j)\} : 1\le i < r, 1\le j \le n\}. \end{aligned}$$

The first set of the union above contains single elements of the first copy of $\Omega$, while the second set contains suitable unordered pairs.

This guarantees that every permutation in $\mathcal {G}(Q)$ is of the form $g^{(r)}$. Indeed, we need to show that for any i, j and any $h\in \mathcal {G}(Q)$, the image $(i,j)h= (i,jg)$ for some permutation $g$ of $\Omega$ that does not depend on i. Looking for the part containing singletons of Q we infer that $(1,j) = (1,jg)$. Next, for the pairs, we have $\{(1,j),(2,j)\}h= \{(1,jg),(i',j')\}$, and since the only pairs of this form in R are those with $i'=2$ and $j'=jg$, we have $\{(1,j),(2,j)\}h= \{(1,jg),(2,jg)\}$, and consequently, $(2,j)h= (2,jg)$, as required. The same argument works for further pairs $\{(i,j),(i+1,j)\}$ with $i=2,\ldots ,n-1$, proving the claim. Thus, we have shown that $\mathcal {G}(F)\le S_n^{(r)}$. On the other hand, obviously, every permutation of the form $g^{(r)}$ preserves Q, so $\mathcal {G}(Q)= S_n^{(r)}$.

To prove the “if part” of the theorem, assume that $2^r \ge n$. We define a regular set y in $S_n^{(r)}$ such that $y\notin ar(Q)$, and apply Lemma 3.2. To this end, let j(i) denote the i-th bit (from the right) in the binary notation of the number j. Then, we put $(i,j) \in y$ if and only if $j(i)=1$. Thus, the sequences $(1,j),(2,j),\ldots ,(r,j)$ are binary representations of different integers provided $2^r \ge n$.

To see that y is regular, note that applying any permutation $h\in S_n^{(r)}$ to y corresponds to changing positions of the integers $(1,j),(2,j),\ldots ,(r,j)$, in the parallel way for all j. Therefore, all the images $yh$ are different.

Since $ar(Q) = \{1,2\}$ we see that $|y| \notin ar(Q)$, provided $|y|>2$. If this is not the case, we replace y by its complement in U, which is also a regular set, and satisfies the required condition on the cardinality. Thus, by Lemma 3.2, we infer that every subgroup of $S_n^{(r)}$ belongs to $\mathfrak R^*$, as required.

For the “only if” part we show that if $2^r < n$, then $A_n^{(r)}$ is not orbit closed. To this end, it is enough to show that a permutation $g^{(r)}\in S_n^{(r)}$ with $g\in S_n\setminus A_n$ preserves the orbits of $A_n^{(r)}$ in P(U). Let y be any of these orbits. We show that $yg^{(r)} = y$.

As before, consider the sequence $(1,j), (2,j), \dots , (r,j)$ as one determining the binary notation of an r-bit number $m=m_j$ according to the condition: $m(i)=1$ if and only if $(i,j) \in y$. Then, since $2^r < n$, there are two positions $s,t < n$ such that $m_s = m_t$, which means that the corresponding sequences for $j=s,t$ are identical. Consequently, there exists a transposition $u\in S_n$ such that for $w=u^{(r)}$ we have $ywg^{(r)} = yg^{(r)}$. Now, $wg\in A_n^{(r)}$, which means that $ywg^{(r)}=y$, as required. $\square$

In general, if G is a parallel sum of permutation isomorphic components, it needs not to be a parallel multiple. This is so, since permutation groups may have nonpermutation automorphisms. For alternating groups, we have an interesting exception.

Lemma 5.4

Suppose that G is the parallel sum of components permutation isomorphic to a fixed alternating group $A_n$, and $G\ne A_n^{(r)}$ for any $r\ge 1$. Then, $n=6$, $G= A_6^{(r)}||_\psi A_6^{(s)}$ for some $r,s\ge 1$ and G is a relation group. Moreover, $G\in \mathfrak R^*$, with the exception of $G= A_6||_\psi A_6$ which fails to have a regular set.

Proof

It is well known [4] that the only alternating group $A_n$ having a nonpermutation automorphism is $A_6$ and that the index $[Aut(A_6) : PAut(A_6)]=2$, which means that up to permutation automorphisms there is only one nonpermutation automorphism in $A_6$. Hence, $G= A_6^{(r)}$ (which is excluded by our assumption) or $G=A_6^{(r)}||_\psi A_6^{(s)}$. We prove that the latter is a relation group.

First, consider $G= A_6 ||_\psi A_6$. Using GAP, we find a nonpermutation automorphisms $\psi$ of $A_6$ given by the images of generators of $A_6$:

$$\begin{aligned} \big ( (2,3)(4,5) \big )\psi = (2,5)(3,4), \mathrm{\ and \ } \big ( (1,2,3,4)(5,6)\big )\psi = (1,2,3,4)(5,6). \end{aligned}$$

(Note that this mapping cannot be obtained by permuting points). This may be used to form the group $G=A_6||_\psi A_6$ as one generated by

$$\begin{aligned} g= (2,3)(4,5)(2',5')(13',14') \mathrm{\ and }h=(1,2,3,4)(5,6)(1',2',3',4')(5',6') \end{aligned}$$

on the set $\Omega =\Omega _1\cup \Omega _2$ with $\Omega _1 =\{1,\ldots ,6\}$ and $\Omega _2=\{1',\ldots ,6'\}$. It is an exercise left to the reader to check that for $R=\{1,2,3,1'\}^G \cup \{\Omega _1\}$ we have $G=\mathcal {G}(R)$.

Unfortunately, we cannot use Lemma 5.2 to complete the proof, because as one can check, $A_6 ||_\psi A_6$ has no regular set. So we still need to consider the group $G' = A_6^{(2)} ||_\psi A_6$. This group has regular sets of all sizes from 4 to 20, and it is a relation group. The latter can be checked analogously, as the previous case. We take $\Omega ' = \Omega _1\cup \Omega _2\cup \Omega _3$, where $\Omega _3=\{1'',\ldots ,6''\}$. We form a defining relation for $G'$ using the relation R for G. We define $R'=R \cup \{\Omega \} \cup \{\{1',1''\}, \ldots ,\{6',6''\}\}$. It is easy to check that $G'=\mathcal {G}(R')$. Now, using Lemmas 3.2 and 5.2 completes the proof. $\square$

Now we consider the groups from the list $\mathcal L$ introduced in Lemma 3.6. We show that although these groups have no regular sets, their sums have. First we prove the following

Lemma 5.5

For each group H in the list $\mathcal L$, and each $r \ge 2$, $H^{(r)} \in \mathfrak R^*$.

Proof

In view of Lemma 5.2, it is enough to prove the claim for $r=2$. First assume that $H\ne PSL(2,8)$. Then, by Lemma 3.6, H is a relation group. Let n be the degree of H, $\Omega =\{1,2,\ldots ,n\}$, and $\Delta = \{1',2',\ldots ,n'\}$. Then, since H is a relation group, there is a relation $R_1$ on $\Omega$ with sets of cardinality less than n such that $H=\mathcal {G}(R_1)$.

Let Q be the relation defined in the proof of Lemma 5.3 for $r=2$ transferred into the set $\Omega \cup \Delta$ by using the natural bijection. The relation Q is to guarantee the parallel action on the sets $\Omega$ and $\Delta$. Yet, the arities of Q and $R_1$ may not be disjoint. Therefore, we replace Q by the family $R_2$ of the complements of the sets in Q. Then, $ar(R_2) = \{2n-1, 2n-2\}$. Since $2n-2 \ge n$ (as $|\Omega |\ge 2$), $ar(R_1) \cap ar(R_2) = \varnothing$. This guarantees the parallel action on the sets $\Omega$ and $\Delta$, as well. Hence, putting $R=R_1\cup R_2$, it follows that $\mathcal {G}(R)$ preserves both $R_1$ and $R_2$, which means that $\mathcal {G}(R) = H^{(2)}$, as required.

To apply Lemma 3.2, we need to find a regular set y in $H^{(2)}$ of cardinality $n \le |y| \le 2n-3$. Note that if we find a regular set z of cardinality $3\le |z| \le n$, then the set $y = (\Omega \cup \Delta ) \setminus z$ is also regular, and has cardinality $2n-3\ge |y| \ge n$, as required. It follows that all we need is to show that in each case there exists a regular set y of cardinality $3 \le |y| \le 2n-3$. This can be done easily with GAP. To provide easy checking for the reader, we give some details of computations.

Let us start from the group $H=L_2(5)$ acting on $n=6$ elements. To present a regular set for $H^{(2)}$ (that the reader can easily check himself or herself), we need to define the underlying set $\Omega$ and generating permutations for $H^{(2)}$. In each case for $\Omega$, we take $\Omega =\{1,2,\dots ,n\} \cup \{1',2',\ldots ,n'\}$. Then, we find a set of generators for H (one may use, for example, the GAP listing of primitive groups of degree n). In this case, we get $g= (1,2,5)(3,4,6)$ and $h=(3,5)(4,6)$. We form permutations $g^{(2)}$ and $h^{(2)}$ obtaining

$$\begin{aligned} g^{(2)}&= (1,2,5)(3,4,6)(1',2',5')(3',4','6), \\ h^{(2)}&= (3,5)(4,6)(3',5')(4',6'). \end{aligned}$$

Clearly, the above permutations generate $H^{(2)}$. A required regular set is, for instance, $y=\{1,2,3,2',4',6'\}$.

The proof in other cases is the same. The only change is in the choice of permutations $g$ and $h$ generating H, and the regular set y. These details are given in Table 1 for each of the cases $H\ne PSL(2,8)$.

Table 1 Data for the proof of Lemma 5.5

It remains to consider the case when $H=PSL(2,8)$ acting on $n=9$ elements. Since it is not orbit closed, we need to apply another approach. First of all we consider a larger group $S_9^{(2)} \supseteq H^{(2)}$ that is a relation group. We have $S_9^{(2)}= \mathcal {G}(R_2)$, where $R_2$ is a relation defined in the first part of the proof, for $n=9$ in this case.

As before, using GAP we find generators for $H=PSL(2,8)$. We use $g=(1,5,4,2,8,$ 3, 6) and $h=(1,8,6,2,7,3,9).$ As before, we form generators $g^{(2)}$ and $h^{(2)}$ on the set $\{1,\ldots ,9\} \cup \{1',\ldots ,9'\}$ for the group $H^{(2)}$.

Now we look for a set y not only having the trivial stabilizer in $G=H^{(2)}$, but also generating G in $S_9^{(2)}$ in the way explained below. We find that the set $y=\{1,2,3,4,2',3',4',5'\}$ is as required. It has the trivial stabilizer in G, and denoting $R_1 = y^G$, the orbit of y in G, we obtain $G = \mathcal {G}(R)$ for $R=R_1\cup R_2$. We note that $R_1$ and $R_2$ are disjoint (since $ar(R_2)= \{17,16\}$), and the fact that $G = \mathcal {G}(R)$ may be easily checked using GAP.

It remains to observe that the complement of y is also a regular set in G, and having 10 elements, it is disjoint with R. This completes the proof. $\square$

We also need to consider subdirect sums of elements from $\mathcal L$ that arise by using a nonpermutation automorphism. It is easy to check that the groups in $\mathcal L$ having nonpermutation automorphisms are the following:

$$\begin{aligned} {\mathcal L}^* =\{ L_3(2), L_2(11), L_3(3), L_4(2), M_{12} \}. \end{aligned}$$

Moreover, one may also check that the index $[Aut(G):PAut(G)]=2$ in each of these cases. We use this to prove the following.

Lemma 5.6

For a group $H\in \mathcal L$, if G is a parallel sum of $r>1$ copies of H, and G is different from $G=H^{(r)}$, then $G\in \mathfrak R^*$.

Proof

Again, in view of Lemma 5.2, it is enough to prove the claim for $r=2$. In view of the remarks above we may restrict to groups $H\in {\mathcal L}^*$ and to one nonpermutation automorphism $\psi$ in each case. We prove the existence of a relation R such that $G=H||_\psi H=\mathcal {G}(R)$ and it has a regular set y with $|y| \notin ar(R)$. By Lemma 5.2, this implies that $G\in \mathfrak R^*$, as required.

As in the proof of Lemma 5.4, first we find a nonpermutation automorphism of H. For $H=L_3(2)$, using GAP, one can find nonpermutation automorphism $\psi$ given by the images of generators

$$\begin{aligned} \big ( (1,2)(5,7)\big )\psi = (1, 2)(3, 6), \mathrm{\ and \ } \big ( (2,3,4,7)(5,6)\big )\psi = (2, 3, 4, 7)(5, 6). \end{aligned}$$

Hence, the group $G=H||_\psi H$ on the set $\Omega =\{1,\ldots ,7\} \cup \{1',\ldots , 7'\}$ is generated by permutations

$$\begin{aligned} g =(1,2)(5,7)(1',2')(3',6') \mathrm{\ and } h= (2,3,4,7)(5,6)(2',3',4',7')(5',6'). \end{aligned}$$

Now, by Lemma 3.6, there exist a relation $R_0$ on $\{1,\ldots ,7\}$ such that $H=\mathcal {G}(R_0)$. Since H is transitive, one may assume that no singleton belongs to $R_0$. Also we may assume that $\{1,\ldots ,7\}\notin R_0$. Let $R_0'$ be a copy of $R_0$ on $\{1',\ldots ,7'\}$. Let $R=R_0\cup R_0'\cup \{1,\ldots ,7\} \cup \{1,1'\}^G$. It is easy to check that $G=\mathcal {G}(R)$ and that G has regular sets of all sizes from 4 to 10. Since $ar(R) \subseteq \{2,\ldots ,7\}$, the claim follows.

For the next three groups in ${\mathcal L}^*$, the constructions are the same, and the only difference is the nonpermutation automorphism used. Below we give information about regular sets and a nonpermutation automorphism in each of these cases.

a) $L_2(11)$ (regular sets of sizes from 4 to 18):

$$\begin{aligned} \big ((1, 3)(2, 7)(5, 9)(6, 11)\big )\psi&= (1, 4)(2, 3)(5, 10)(9, 11), \\ ( (3, 5, 11)(4, 9, 7)(6, 10, 8))\psi&= (3, 5, 11)(4, 9, 7)(6, 10, 8) \end{aligned}$$

b) $L_3(3)$ (regular sets of sizes from 6 to 20):

$$\begin{aligned} \big ( (3,5,11)(6,7,9)(8,12,13)\big )\psi&= (3, 8, 7)(5, 12, 9)(6, 11, 13)\\ \big ((1,13,7)(2,10,6)(3,5,12)(4,11,9)\big )\psi&= (1, 13, 7)(2, 10, 6)(3, 5, 12)(4, 11, 9) \end{aligned}$$

c) $L_4(2)$ (regular sets of sizes from 6 to 24.):

$\big ( (1, 9, 5, 14, 13, 2, 6)(3, 15, 4, 7, 8, 12, 11) \big )\psi = (1, 4, 2, 14, 13, 7, 8)(3, 10, 15, 9, 5, 6, 12)$

$\big ( (1, 3, 2)(4, 8, 12)(5, 11, 14)(6, 9, 15)(7, 10, 13) \big )\psi =$

$= (1, 2, 3)(4, 14, 10)(5, 12, 9)(6, 13, 11)(7, 15, 8)$

d) For $M_{12}$ the construction as above turned out too complex for computations. We have applied another approach, the results of which can be checked easily using GAP. The group $G = M_{12} ||_\psi M_{12}$ different from $M_{12}^{(2)}$ is generated by

$$\begin{aligned}{} & {} (1,19,17,23,2,4,8,9)(6,16,21,22)(3,12)(5,24,15,7,18,10,20,13), \mathrm{\ and} \\{} & {} \quad (1,23,8,19,9,6,4,21,16,22,2)(3,14,12,7,11,20,13,5,15,24,10) \end{aligned}$$

It is contained in the transitive group T generated by

$$\begin{aligned}{} & {} (1,16,23,19,9,21,2,4)(5,15,12,10,18,24,14,20)(6,8,22,17)(7,13), \mathrm{\ and} \\{} & {} \quad (1,5)(2,11)(3,22)(4,15)(6,10)(7,23)(8,24)(9,14)(12,21)(13,19)(16,20)(17,18), \end{aligned}$$

which in turn, is contained in $M=M_{24}$ generated by

$$\begin{aligned}{} & {} (1,5)(2,14,7,12)(3,21)(4,17,16,11)(6,20,23,22)(9,10,15,13)\mathrm{\ and} \\{} & {} \quad (1,19,15,8,20,23,24,9,14,11,5,10,22,13,2)(3,6,4)(7,16,12,17,18). \end{aligned}$$

Now, the defining relation for G is $R= \{1,2,3,4,5,6\}^M\cup \{1,2,3\}^T, \{1,2\}^G$. A regular set in G is $\{1,2,\ldots ,10\}.$ $\square$

In the next lemma, we consider the last case needed to prove our main result.

Lemma 5.7

For each pair of groups $H \cong K$ in the list $\mathcal L$ that are different as permutation groups, the group $H ||_\phi K \in \mathfrak R^*$.

Proof

Using Lemma 3.6, we infer that there are exactly five possibilities for H and K:

• (i) $n=5, 6$ with $A_5 \cong L_2(5)$,

• (ii) $n=7, 8$ with $L_3(2) \cong L_2(7)$,

• (iii) $n=6, 10$ with $A_6 \cong L_2(9)$,

• (iv) $n=11, 12$ with two actions of $M_{11}$,

• (v) $n=8, 15$ with $A_8 \cong L_4(2)$,

For each pair, we apply the same strategy, similarly to that in the proof of Lemma 5.5. The main difference is that in the case of $G=H|| K$ we have no larger relation group in the form of parallel sum. Instead, we look for the smallest direct sum $H'\oplus K'$ containing H||K such that we may easily see that $H'\oplus K'$ is the relation group.

Let H and K act on disjoint sets $\Omega _1$ and $\Omega _2$, and let $\Omega =\Omega _1\cup \Omega _2$. We take the least groups $H'$ and $K'$ such that $H\le H'\le Sym(\Omega _1)$, $K\le K'\le Sym(\Omega _2)$, and $H',K'$ are relation groups. Let $R_1$ and $R_2$ be relations on $\Omega _1$ and $\Omega _2$ such that $H'= \mathcal {G}(R_1)$ and $K'= \mathcal {G}(R_2)$. As we observed in Section 2, we may assume that $\Omega _1\in R_1$ or not, and $\Omega _2\in R_2$ or not. Without loss of generality, we assume that H and K are ordered so that $|\Omega _1| \ge |\Omega _2|$, and $\Omega _1\in R_1$, while $\Omega _2\notin R_2$.

Then, $H'\oplus K'$ has a defining relation $R' = R_1\cup R_2$. Indeed, if $g\in \mathcal {G}(R')$, then $g$ preserves $\Omega _1$, and therefore it preserves $R_2$, as well. Consequently, $\mathcal {G}(R') \subseteq Sym(\Omega _1) \oplus Sym(\Omega _2)$. Since $R_1$ and $R_2$ are defining relations for $H'$ and $K'$, respectively, it follows that $\mathcal {G}(R) =H'\oplus K'$, as required. Moreover, it is clear that for each $m\in ar(R')$, $m\le |\Omega _1| =\max \{|\Omega _1|,|\Omega _2|\}$.

Next, we look for a set $x\subseteq \Omega$ such that the orbit of x in $G=H|| K$ defines G in $H'\oplus K'$, i.e., such that if we put $R = R'\cup x^G$, then $G=\mathcal {G}(R)$. Having this, the last step is to find a regular set y with $|y|\notin ar(R)$. By the inequality established above, it is enough that $|y|> |\Omega _1|$.

Now, in case (i), we take $G=L_2(5)|| A_5$, $H'=L_2(5)$, and $K'=S_5$. As the generators of $L_2(5)\oplus S_5$ we take $\{1,3,4)(2,5,6), (1,2)(3,4), (7,8,9,10,11), (7,8)\}$. As the generators of $G=L_2(5)||A_5$ (formed similarly as the proof of the previous lemma) we take (1, 3, 4)(2, 5, 6)(8, 9, 11) and (1, 2)(3, 4)(7, 8)(9, 10).

We note that the group $G=L_2(5)|| A_5$ is unique up to permutation isomorphism. This is because $A_5$ has only permutation automorphism (even if $L_2(5)$ has a nonpermutation automorphism). The situation is similar in the remaining cases (one of the groups has only permutation automorphisms), so we make no further mention about it.

We explain the details of finding a set x defining G in $H'\oplus K'$. We make use of the fact that the sets in $R'$ are all contained in one of the orbits defined by $R'$. Therefore, any set that has elements in each of the orbits certainly does not belong to $R'$. By trial and error, using GAP, we check the set $x=\{1,3,7,9\}$. Its cardinality may belong to $ar(R')$, but by the remark above, the orbit $x^G$ is certainly disjoint with $R'$, and therefore to check that $G=\mathcal {G}(R)$, similarly as in the proof of Lemma 3.2, it is enough to check that the stabilizer of the family $x^G$ in $L_2(5)\oplus S_5$ acting on subsets of $\Omega$ is precisely the group G. This is easily done using GAP. Finally, we check that x is actually a regular set in G. Therefore, to apply Lemma 5.2, as a regular set y with $|y|\notin ar(R)$ one may take simply the complement of x in $\Omega$, whose cardinality is $|y|=7 > 6=|\Omega _1|$, as required.

In case (ii), we take $G=L_2(7)||L_3(2)$, $H'=L_2(7)$, and $K'=L_3(2)$. Generally, in other cases we take $H'=H$ and $K'=K$ except when $H=A_n$, in which case we take $H'=S_n$. This guarantees that in each case $H',K'$ are relation groups.

Table 2 Data for the proof of Lemma 5.7

As the generators of $L_2(7)\oplus L_3(2)$, we take the set

$$\begin{aligned} \{(1,6,5)(2,3,7), (9,11,10)(12,15,13), (1,4)(2,7)(3,5)(6,8), (9,12)(14,15) \}, \end{aligned}$$

and the set of two generators for $L_2(7)|| L_3(2)$ is formed using suitable products:

$$\begin{aligned} (1,6,5)(2,3,7)(9,11,10)(12,15,13) \hbox { and } (1,4)(2,7)(3,5)(6,8)(9,12)(14,15). \end{aligned}$$

As the defining set, we take $x=\{1,3,5,10,12,14\}$, and as a regular set y the complement of x in $\Omega$. The other details of computations (to check by the reader) are the same as before. In Table 2, we list the sets of generators and sets x and y for the remaining three cases.

We note that in the last entry the set x is not regular in $L_4(2)||A_8$, and therefore, rather than taking the complement of x, we find another regular set whose complement is as required. (The regular set found is too large for standard GAP 4.10.1 to compute the stabilizer of the family $x^G$ in $L_4(2)||S_8$; so we have changed our approach a little in this very case). $\square$

6 Results

Now, we are ready to state and prove our main results. Since a simple permutation group G may have fixed points, we will consider the restriction $G'$ of G to those points that are not fixed by G.

Theorem 6.1

Let G be a simple permutation group, and $G'$ the restriction of G to the points that are not fixed. Then, G is a relation group if and only if $G'$ is not one of the following:

• (i) $G'=A_n^{(r)}$, where $n\ge 3$ $(n\ne 4)$, $r\ge 1$, and $r< \log _2(n)$;

• (ii) $G'=C_5$ or $G'=PSL(2,8)$.

In cases (i) and (ii) above, G is not orbit closed.

Proof

In the proof, we assume that $G=G'$ has no fixed points. The result extends on groups with fixed points by remarks at beginning of Section 5.

First, if G is primitive, then using [7, Theorem 4.2, Corollary 4.3] we infer that G is a relation group, unless it is $A_n, C_5$ or $L_2(8)$. If $G=A_n$ or $C_5$, then we know it is not orbit closed. If $G=PSL(2,8)$, then it is set-transitive ( [2, 7]), so it is not orbit closed, either.

If G is transitive imprimitive, then the result is by Proposition 3.1: all groups belong to $\mathfrak R^*$.

Thus, we may assume that G is intransitive, and by Proposition 5.1 (and the following remarks), G is a parallel sum of at least two transitive components, all abstractly isomorphic to G.

If none of the components is orbit closed, then all they are permutation isomorphic to one of the groups pointed out above. Hence, either $G=H^{(r)}$ or $G=H^{(r)}||_\psi H^{(s)}$, with $\psi$ induced by a nonpermutation automorphism of H, where $H = A_n (n\ne 4)$, $C_5$, or PSL(2, 8).

In the latter case, since $C_5$ and PSL(2, 8) have only permutation automorphisms, $H=A_n$, and the claim follows by Lemma 5.4. (We note, for future reference, that $G\in \mathfrak R^*$, with the exception of $A_6||_\psi A_6$.)

In the former case, if $H=A_n$, then by Lemma 5.3, G is a relation group if and only if $2^r \ge n$, and otherwise G is not orbit closed. If $H=C_5$, then by [6] (cf. [19, Theorem 6.1]), $H^{r}$ is a relation group for any $r>1$. Finally, if $H=PSL(2,8)$, then by Lemma 5.5, $G = H^{(r)} \in \mathfrak R^*$ for any $r\ge 2$.

Hence, we may assume now that at least one transitive component H of G is a relation group. If H has a regular set y, then by Lemma 5.2, $G\in \mathfrak R^*$. If H has no regular set, and $G=H^{(r)}$ is a parallel multiple, then $G\in \mathfrak R^*$ by Lemma 5.5. If G is a parallel sum of $r>1$ copies of H different from $G=H^{(r)}$, then the same holds by Lemma 5.6.

It remains to consider the situation when G has two transitive components H and K that are not permutation isomorphic, but are abstractly isomorphic, H is a relation group, and H has no regular set. By Proposition 3.1, H is primitive and belongs to the list $\mathcal L$. For the other component K we may assume that either it is not a relation group (in which case the only possibility is $K=A_n$ for some n), or it is a relation group and has no regular set (in which case it appears in the list $\mathcal L$). Looking for the list $\mathcal L$ (preceding Lemma 3.6) we see that there are exactly five possibilities for H and K, and combining Lemmas 5.7 and 5.2 we obtain that, in any case, $G\in \mathfrak R^*$. $\square$

In the proof above, there is a lot of information on regular sets. Some extra arguments yield the complete description.

Theorem 6.2

Let G be a simple permutation group, and $G'$ the restriction of G to the points that are not fixed. Then, G has a regular set if and only if G is not one of the following:

• (i) $G'=A_n^{(r)}$, where $n\ge 3$ $(n\ne 4)$, $r\ge 1$, and $r < \log _2(n-1)$;

• (ii) $G'$ is primitive and appears in the list $\mathcal L$ in Lemma 3.6;

• (iii) $G'=A_6 ||_\psi A_6$ is the exceptional group considered in Lemma 5.4.

Proof

We assume again that $G=G'$ has no fixed points, since it is easy to see that the result extends immediately. The proof follows exactly the proof of the preceding theorem, with one place requiring more consideration.

If G is primitive that the result is by [21] (cf. [7, Theorem 2.2]). If $G=H^{(r)}||_\psi H^{(s)}$, with $\psi$ induced by a nonpermutation automorphism of H, where $H = A_n$, then the claim follows by Lemma 5.4.

If $G=H^{(r)}$ and $H=A_n$, then we need to show, in addition, that for $n=2^r+1$, G has a regular set, and if $2^r < n-1$, then G has no regular set. Indeed, G acting on a set $\Omega$ has a regular set if and only if there is a partition of $\Omega$ into two sets such that no nontrivial permutation in G preserves this partition. Since $A_n^{(r)}$ acts on $\Omega = \Omega _1 \cup \ldots \cup \Omega _r$, where each $\Omega _i$ is a copy of $\{1,\ldots ,n\}$, and the action of $g^{(r)}$ is the same on all copies as the action of $g\in A_n$ on $\{1,\ldots ,n\}$, the union of partitions of all $\Omega _i$ into two sets is equivalent to a partition of $\{1,\ldots ,n\}$ into $2^r$ sets (or less if some partitions of $\Omega _i$ coincide). Now, if $n=2^r+1$, then one can choose partitions of the sets $\Omega _i$ equivalent to a partition of $\{1,\ldots ,n\}$ consisting of a two element set and $n-2$ singletons. Obviously, the only permutation in $A_n$ preserving this partition is the identity. A regular set y in $A_n^{(r)}$ is the union of blocks, one from each partition of $\Omega _i$. If $2^r < n-1$, then any partition of $\{1,\ldots ,n\}$ equivalent to partitions of $\Omega _i$ contains either a block with 3 elements or two 2-element blocks. Then, there exists a nontrivial $g\in A_n$ preserving this partition, and therefore, $A_n^{(r)}$ has no regular set. $\square$

Using the above result, one can determine the distinguishing numbers for simple permutation groups. This was done much earlier for the larger class of almost simple groups in [23], using quite different methods. In turn, these results have been applied in [15] to determine the distinguishing index for graphs with simple automorphism groups.

Finally, we also have a general result concerning subgroups of simple permutation groups. Note that $C_5$ is not orbit closed, and the same concerns its direct sum with m fixed points $C_5\oplus I_m$. Now, $C_5\oplus I_1$ is a subgroup of PSL(2, 5), and more generally $C_5\oplus I_{m+1}$ is a subgroup of $PSL(2,5)\oplus I_m$, $m>0$. This is exceptional with regard to the following.

Theorem 6.3

If a simple permutation group G is a relation group, then every subgroup $H<G$ is a relation group with the exception of $H=C_5\oplus I_{1} < PSL(2,5)$ or more generally, $H=C_5\oplus I_{m+1} < PSL(2,5)\oplus I_{m}$ ($m>0$).

Proof

Again we assume that G has no fixed points and follow the proof of Theorem 6.1. The most involved is now the case of primitive groups. By Lemma 3.1 (ii) [7], if G is a subgroup of group H that has a regular set, is not set-transitive, and is maximal in $S_n$ with respect to this property, then every subgroup of H (and thus G) is a relation group, as required. By [2] (cf. [7, Lemma 2.1]), the only set-transitive simple primitive group not containing $A_n$ is PSL(2, 8), which is not a relation group. So, we need to check those simple groups G that are subgroups of primitive groups that have no regular sets. This requires quite substantial computation, which has been performed in [14]. By the theorem proved in [14], the only subgroup of a simple primitive group which is not a relation group is just $H=C_5\oplus I_1 < PSL(2,5)$.

Further, following the proof of Theorem 6.1, the only case we need still to consider, when it is not proved that $G\in \mathfrak R^*$, are those with $G=(C_5)^{(r)}$ or (in view of Lemma 5.4) $G=A_6 ||_\psi A_6$ with a nonpermutation automorphism $\psi$. In the former case, G has no nontrivial subgroups, so there is nothing to prove. In the latter case, it is enough to observe that $A_6 ||_\psi A_6$ is a subgroup of $M_{12}$, whose all subgroups are relational groups by [14]. $\square$

Remark

In the first part of the proof, for primitive G, we have used ideas of the proof [7, Theorem 4.1]. Yet, there is a mistake in the proof of this theorem in [7]. The argument there works only for $|\Omega |>32$. Nevertheless, the theorem is true as it is shown in [14], where a stronger version of it is proved.