1 Introduction

This paper is motivated by the problem called by Babai [5, p.52] the concrete representation problem. Recall that König’s problem for groups asked which finite groups are the automorphism groups of (simple) graphs. This question, in its abstract version, was quickly answered by Frucht who showed that every group is isomorphic to the automorphism group of some graph. The concrete version, of more combinatorial flavor, asks which finite permutation groups are the automorphism groups of graphs. This problem turns out to be much harder (see [5, 10]). In the abstract version we look for a graph with an arbitrary number of vertices whose automorphism group is isomorphic to a given (abstract) group. In the concrete version we are given a permutation group (GX) acting on a set X of elements and we are looking for a graph (XE) with the same set X of vertices whose automorphisms are precisely the permutations in G.

There are related active areas of research concerning graphical representations of abstract groups, Cayley graphs and automorphism groups of circulant graphs (see [4, 5, 9, 26] for the beginnings, and [1, 2, 7, 8, 17, 27,28,29,30] to mention just a few most recent results). All these are closely connected with the problem considered in this paper. Yet, we must emphasize that we approach the topic from a different point of view. Our main interest is in permutation group structure and classifying permutation groups by their natural representations as the automorphism groups of colored graphs. This relates directly to the direction of research suggested in Wielandt’s [31]. Many natural applications of permutation groups, especially in fields outside mathematics, concern the ways they act rather than their group structure (cf. [21,22,23]). Moreover, we are particularly interested in intransitive permutation groups, as they reflect the way how graphs that fail to be vertex transitive are composed from transitive components. Now, while transitive permutation groups are understood pretty well (mainly due to research connected with the classification of finite simple groups), very little is known, in fact, about intransitive groups and their complex actions on different orbits. Note that most graphs are not vertex transitive and the action of the automorphism group on different orbits may be very different and relate closely to various graph properties.

It is easy to see that some permutation groups, like the alternating groups \(A_n\) on n elements or the groups \(C_n\) generated by the cyclic permutation \((1,2,\ldots ,n)\), are not the groups of automorphisms of any graph (on n elements). For a long time there was no progress in the concrete version of König’s problem. Only research on the so-called Graphical Regular Representation of groups (GRR) followed a more systematic approach to the problem. The final result of this extensive study by Godsil [9], even if it concerned representations of abstract groups, may be interpreted as the description of those regular permutation groups that are the automorphism groups of graphs. The analogous result for automorphism groups of directed graphs has been obtained by Babai [4].

The next natural class of permutation groups to study from this point of view is the class of cyclic permutation groups, that is, those generated by a single permutation. König’s problem for this class turned out not so easy as it could seem at the first sight. After some partial results containing errors and wrong proofs ([24, 25] corrected in [11]), the final result has been obtained only recently [15, 16]. The full description turns out to consist of seven technical conditions concerning possible lengths of the orbits.

To obtain this description, we have applied the aforementioned Wielandt’s approach to start from considering the invariance groups of families of binary relations rather than the automorphism groups of simple graphs. In the language of graphs these are the automorphism groups of (edge) colored graphs. This approach is more natural. So, first, in [15], we have obtained an easy-to-formulate result that a cyclic permutation group G is the automorphism group of a colored graph if and only if for every nontrivial orbit O of G there exists another orbit Q such that \(\gcd (|O|, |Q|) \ge 3\). The proof yields also the result that if a cyclic permutation group is the automorphism group of a colored graph, then it is the automorphism group of a colored graph that uses at most 3 colors. Only then may one consider for which cyclic permutation groups 2 colors suffice, which turns out to have a rather technical solution (see [16]). There are many other results in the area showing that considering edge-colored graphs rather than simple graphs is the right approach (see [21]).

The next natural class to attack in the concrete version of König’s problem is that of abelian permutation groups. The survey [26] reports a result by Zelikovskij [32] where the solution of König’s problem for a large class of abelian permutation groups (namely, those whose order is not divisible by 2, 3 or 5) is provided. There is no English translation of [32], so the survey quotes only the English summary. The restriction means that the lengths of the orbits must be relatively prime to 30, and its apparent aim is to avoid technical complications. Worse, the result as stated is false. In [14] we demonstrate a counterexample and point out the false algebraic assumption used by Zelikovskij in the proof.

In this paper we make the first step to find a correct characterization of the abelian permutation groups that are the automorphism groups of graphs. As before, we start from characterizing those abelian permutation groups that are the automorphism groups of colored graphs and digraphs. Again, it turns out that this can be done in a quite nice way in terms of basic properties of permutation groups.

Our characterizations use a technical notion specific to intransitive permutation groups. For a given permutation group G we say that a permutation \(\sigma \) is 2-orbit compatible with G if for every pair of orbits O and Q of G there is a permutation \(\sigma '\in G\) such that \(\sigma \) and \(\sigma '\) have identical actions on \(O\cup Q\). The group G is 2-orbit-closed if every permutation that is 2-orbit-compatible with G belongs to G. Every transitive permutation group, or having just two orbits, is trivially 2-orbit-closed, but permutation groups containing more than 2 orbits may not be.

It is not difficult to observe that each automorphism group of an edge-colored directed graph is necessarily 2-orbit-closed. In this paper, we prove that for abelian groups this condition is also sufficient: an abelian permutation group A is the automorphism group of an edge-colored directed graph if and only if A is 2-orbit-closed (Theorem 3). Moreover, we prove that an abelian permutation group A is the automorphism group of an edge-colored (simple) graph if and only if A is 2-orbit-closed and satisfies an additional condition concerning groups induced by A on its orbits (Theorem 2). Our main tool in proving these results is the subdirect sum decomposition of intransitive groups, which is recalled for convenience of the reader in Sect. 2.1.

2 Terminology

For standard notions and terminology of permutation groups see, e.g., [6]. We use the notation (GX) to denote a permutation group G acting on a finite set X. Permutation groups are considered up to permutation isomorphism, i.e., two groups that are permutation isomorphic (in the sense of [6, p. 17]) are treated as identical. In particular we usually assume that \(X=\{1,2,\ldots ,n\}\), and by \(S_n\) and \(A_n\) we denote the full symmetric group and the alternating group, respectively, acting on X. By \(C_n\) we denote the subgroup of \(S_n\) generated by the cyclic permutation \((1,2,\ldots ,n)\). By \(I_n\) we denote the trivial permutation group acting on n elements, that is, the subgroup of \(S_n\) containing the identity permutation only.

A k-colored digraph \(\Gamma = (X, \gamma )\) is a set X (of vertices) with a function \(\gamma : X\times X \rightarrow \{0,1,\ldots , k-1\}\). If \(\gamma \) is a function from the unordered pairs \(P_2(X)\) of the points of X to \(\{0,1,\ldots , k-1\}\), then \(\Gamma \) is called a k-colored graph. They may be viewed as the complete digraph or the complete graph, respectively, on a set X, whose edges are colored with at most k different colors. In the case \(k=2\), they may be identified with simple digraphs and graphs, with edges colored 1, and nonedges colored 0.

The i-th degree of a vertex \(x\in X\), denoted \(d_i(x)\), is the number of edges in color i incident with x. The k-tuple \((d_0(x),\ldots ,d_{k-1}(x))\) is referred to as the k-tuple of color degrees of x. In colored digraphs we may distinguish also outdegrees and indegrees, and the corresponding k-tuples.

A permutation \(\sigma \) of X is an automorphism of \(\Gamma =(X,\gamma )\), if it preserves the colors of edges in \(\Gamma \). Obviously, each automorphism preserves also the k-tuples of color degrees of vertices. The automorphisms of \(\Gamma \) form a permutation group, which is denoted \(\text{ Aut }(\Gamma )\). We say also that \(\Gamma \) represents (graphically) the permutation group \(\text{ Aut }(\Gamma )\). We note that not every permutation group is representable by a colored graph or digraph. For example, the alternating group \(A_n\) is not representable for any \(n>3\) (neither by a colored graph nor by a digraph). The cyclic group \(C_n\) is not representable by a colored graph (for any \(n>2\)), but it is representable by a (2-colored) digraph.

Given a permutation group (GX), by \(\text{ Orb }(G,X) = \text{ Orb }(G)\) we denote the colored digraph in which two edges have the same color if and only if they belong to the same orbit of G in its action on \(X\times X\) (i.e., orbital). Similarly, by \(\text{ Orb}^*(G,X) = \text{ Orb}^*(G)\) we denote the colored graph in which two edges have the same color if and only if they belong to the same orbit of G in its action on \(P_2(X)\).

It is easy to see that \(\text{ Aut }(\text{ Orb}^*(G)) \supseteq \text{ Aut }(\text{ Orb }(G))\supseteq G\) as groups of permutations over X. The first group is called the \(2^*\)-closure of G, while the second group is called the 2-closure of G. When a permutation group happens to be equal to its \(2^*\)-closure or 2-closure, then it is called \(2^*\)-closed or 2-closed, respectively. These groups are the largest permutation groups with the given set of orbits on \(P_2(X)\) or on \(X\times X\), respectively (see [5, 31]; according to Wielandt [31] the notion of 2-closure as a tool in the study of permutation groups was introduced by I. Schur.

If G is not \(2^*\)-closed, then G is not the automorphism group of any (colored) graph. Otherwise, there may be various colored graphs \(\Gamma \) such that \(G = \text{ Aut }(\Gamma )\). Yet, each such graph can be obtained from \(\text{ Orb}^*(G)\) by identifying some colors. In particular, if \(G = \text{ Aut }(\Gamma )\) for a simple graph \(\Gamma \), then \(\Gamma \) can be obtained from \(\text{ Orb}^*(G)\) by identifying some colors with 1 (corresponding to edges) and other colors with 0 (nonedges).

We define GR(k) to be the class of all permutation groups that are automorphism groups of colored graphs using at most k colors. The union \(GR = \bigcup _{k\ge 1}GR(k)\) is just the class of \(2^*\)-closed permutation groups. Similarly, we define DGR(k) as the class of all permutation groups that are automorphism groups of colored digraphs using at most k colors. The union \(DGR = \bigcup _{k\ge 1}DGR(k)\) is just the class of 2-closed permutation groups.

While this seems pretty natural topic in the area of graphs and permutation groups not much has been done so far in it. The reason is that, on the one hand, the topic turned out to be rather hard, and on the other hand, the main stream of research in permutation groups was focused so far on delivering tools for the classification of finite simple groups, and this restricted research to transitive groups.

2.1 Subdirect sum decomposition

A natural tool in the study of intransitive permutation groups is the subdirect sum of permutation groups. Given two permutation groups \(G\le S_n\) and \(H\le S_m\), the direct sum \(G\oplus H\) is the permutation group on \(\{1,2,\ldots , n+m\}\) defined as the set of permutations \(\pi =(\sigma ,\tau )\) such that

$$\begin{aligned} \pi (i) = \left\{ \begin{array}{ll} \sigma (i), &{} \text{ if } i\le n\\ n+\tau (i-n), &{} \text{ otherwise. } \end{array}\right. \end{aligned}$$

Thus, in \(G\oplus H\), permutations of G and H act independently in a natural way on a disjoint union of the base sets of the summands.

We introduce the notion of the subdirect sum following [13] (and the notion of intransitive product in [20]). Let \(H_1\triangleleft ; G_1\le S_n\) and \(H_2\triangleleft \; G_2 \le S_m\) be permutation groups such that \(H_1\) and \(H_2\) are normal subgroups of \(G_1\) and \(G_2\), respectively. Suppose, in addition, that factor groups \(G_1/H_1\) and \(G_2/H_2\) are (abstractly) isomorphic and \(\phi : G_1/H_1 \rightarrow G_2/H_2\) is the isomorphism mapping. Then, by

$$\begin{aligned} G = G_1[H_1] \oplus _\phi G_2[H_2] \end{aligned}$$

we denote the subgroup of \(G_1 \oplus G_2\) consisting of all permutations \((\sigma ,\tau )\), \(\sigma \in G_1, \tau \in G_2\) such that \(\phi (\sigma H_1) = \tau H_2\). Each such group will be called a subdirect sum of \(G_1\) and \(G_2\), and denoted briefly \(G_1 \oplus _\phi G_2\) (in this notation the normal subgroups \(H_1\) and \(H_2\) are assumed to be specified in the full description of the isomorphism \(\phi \)).

If \(H_1=G_1\), then \(G_1/H_1\) is a trivial (one-element) group, and consequently, \(G_2/H_2\) must be trivial, which means that we have also \(H_2 = G_2\). Then, \(G = G_1 \oplus G_2\) is the usual direct sum, with \(\phi \) being the mapping between one-element sets. In such special case the subdirect sum \(\oplus _\phi \) will be referred to as trivial. In the case when \(G_1 = G_2 = G\) and \(H_1=H_2\) is the trivial one-element subgroup of G, and \(\phi : G \rightarrow G\) is the identity mapping, the subdirect sum is nontrivial. In this case we call the resulting sum a parallel sum (permutation isomorphic groups \(G_1\) and \(G_2\) act in a parallel manner on their orbits) and denote it briefly G||G. For example, the cyclic group generated by the permutation \(\sigma = (1,2,3)(4,5,6)\) is permutation isomorphic to \(C_3||C_3\).

The main fact established in [20] is that every intransitive group has the form of a subdirect sum, and its components can be easily described. Let G be an intransitive group acting on a set \(X = X_1 \cup X_2\) in such a way that \(X_1\) and \(X_2\) are disjoint fixed blocks for G. Let \(G_1\) and \(G_2\) be restrictions of G to the sets \(X_1\) and \(X_2\), respectively (they are called also constituents). Let \(H_1',H_2' \le G\) be the subgroups fixing pointwise \(X_2\) and \(X_1\), respectively. Let \(H_1 \le G_1\) and \(H_2 \le G_2\) be the restrictions of \(H_1'\) and \(H_2'\) to \(X_1\) and \(X_2\), respectively. Then we have

Theorem 1

[20, Theorem 4.1] If G is a permutation group as described above, then

  1. a)

    \(H_1\) and \(H_2\) are normal subgroups of \(G_1\) and \(G_2\), respectively,

  2. b)

    the factor groups \(G_1/H_1\) and \(G_2/H_2\) are abstractly isomorphic, and

    $$\begin{aligned} G = G_1[H_1] \oplus _\phi G_2[H_2], \end{aligned}$$

    where \(\phi \) is the isomorphism of the factor groups.

3 Preliminary results

First, we establish the representability of regular abelian permutation groups and some other groups connected with the automorphism groups of Cayley graphs. Here, we make use of known results on the so-called Cayley index of abelian groups.

Recall that each regular permutation group may be viewed as the action of an abstract group G on itself given by left multiplication. In such a case we have \(X=G\), and the resulting permutation group will be denoted by (GG), or simply by G, if it is clear from the context that we mean the corresponding regular permutation group. In particular, we use standard notation \(Z_k^m\) and \(Z_k^m \times Z_r^s\) for abstract abelian groups to denote also corresponding permutation groups obtained by the regular action of these groups on themselves (in particular, \(C_n\) and \(Z_n\) denote here the same permutation group).

Given an abstract group G, by \(\text{ Cay }(G)\) we denote the complete directed colored Cayley graph, that is one with all nontrivial elements as generators defining different colors. Observe that \(\text{ Cay }(G) = \text{ Orb }(G,G)\). By \(\text{ Cay}^*(G)\) we denote the complete undirected colored graph obtained from \(\text{ Cay }(G)\) by identifying colors corresponding to g and \(g^{-1}\) for every nontrivial \(g\in G\), and removing the loops. Again, \(\text{ Cay}^*(G) = \text{ Orb}^*(G,G)\).

Now, given a set S of nontrivial elements of G (i.e., different from the identity), by \(\text{ Cay}^*(G;S)\) we define the colored graph obtained from \(\text{ Cay}^*(G)\) by identifying all colors not in S. To admit further identifications, let \(\Pi \) be a partition of S. Then by \(\text{ Cay}^*(G;\Pi )\) we define the colored graph obtained from \(\text{ Cay}^*(G;S)\) by identifying the colors in each block of \(\Pi \). In our notation applied below, \(\Pi \) is written simply by listing its blocks, a block is written in the square brackets, and in the case of a one-element block, brackets are omitted. To make notation as compact as possible we adopt the convention that \(\Pi \) contains only one representative of each pair \(\{g,g^{-1}\}\). In addition, we assume that there are nontrivial pairs \(g,g^{-1}\) not in S, and all elements not in S get color 0 in diagrams represented by nonedges. Then, the graph \(\text{ Cay}^*(G;\Pi )\) is the complete directed graph whose edges are colored with exactly \(|\Pi |+1\) colors.

The following lemma presents the colored graphs whose automorphism groups are \((Z_2^k,Z_2^k)\) for \(k=2,3,4\). (The k-tuples of elements of \(Z_2^k\) are denoted below by corresponding strings of 0’s and 1’s.)

Lemma 1

Each of the following colored graphs represents the regular action of its defining group:

  1. (i)

    \(\text{ Cay}^*(Z_2^2; 10, 01)\),

  2. (ii)

    \(\text{ Cay}^*(Z_2^3; 100, 010, 001)\),

  3. (iii)

    \(\text{ Cay}^*(Z_2^4; 1000, [0100, 1010], [0010, 0001])\)

Proof

We consider the case (iii). Denote the graph by \(\Gamma \). It is pictured in Fig. 1. (Solid, dashed, and dotted lines correspond to colors 1000, 0100, and 0010, respectively). We will speak correspondingly of solid, dashed, and dotted neighbors.

Fig. 1
figure 1

\(\text{ Cay}^*(Z_2^4; 1000, [0100, 1010], [0010, 0001])\)

Since \(\Gamma \) is obtained from a Cayley graph on \(Z_2^4\) by identifying colors, it follows that its automorphism group \(\text{ Aut }(\Gamma )\) contains the regular action of \(Z_2^4\) (which in this notation is given by addition). We need only to prove that \(\text{ Aut }(\Gamma )\) contains no other permutation. Let us consider the stabilizer \(A_0\) of the vertex 0000 in \(\text{ Aut }(\Gamma )\). As the latter is transitive, it is enough to show that \(A_0\) is trivial.

Since the only solid neighbor of 0000 is 1000, \(A_0\) fixes 1000 as well. Further, the only dashed neighbor of 0000 that is a dotted neighbor of 1000 is 1010, while the only solid neighbor of the latter is 0010. Thus the four vertices with coordinates x0y0 are fixed. Considering their dashed neighbors, we see that also each vertex with coordinates x1y0 must be fixed. It follows that the vertices of the cube xyz0 are individually fixed. Considering their dotted neighbors, we conclude that the same holds for the cube xyz1, which completes the proof.

The cases (i) and (ii) are easier and are left to the reader as an exercise. \(\square \)

Now, recall that an abelian permutation group (AX) is transitive if and only if it is regular. It follows that a transitive abelian permutation group A can be identified with the regular action of A (considered as an abstract group) on itself (cf. [6]). In this case, we have a special permutation on A defined by \(\alpha : x\rightarrow x^{-1}\) called the involution. (For properties and a very special role of this permutation see, e.g., [7, 19]). It is easy to observe that the involution preserves the colors of the edges in \(\text{ Cay}^*(A)\). This leads to the well-known fact:

Lemma 2

Le A be a regular abelian permutation group, and \(\alpha \) its involution. If \(\Gamma \) is a colored graph such that \(\text{ Aut }(\Gamma ) \supseteq A\), then \(\alpha \in \text{ Aut }(\Gamma )\).

This is so since \(\text{ Cay}^*(A) = \text{ Orb }(A,A)\), and \(\Gamma \) needs to be a graph obtained from \(\text{ Orb }(A,A)\) by identification of colors.

It follows from this lemma that generally a regular abelian permutation group A does not belong to GR, except for the case when \(\alpha \) is trivial (the identity permutation). This is exactly the case, when \(A=Z_2^n\) for some \(n\ge 0\). It is well known that for \(n\ge 5\), \(Z_2^n\) is representable as the automorphism group of a simple (Cayley) graph (see [18], or claim 1.2 in [19]). Combining this with Lemma 1 we have

Lemma 3

Let A be a regular abelian permutation group. If \(A=Z_2^n\) for some \(n\ge 0\), then \(A\in GR(4)\); otherwise, \(A\notin GR\).

We note that \(Z_2^3\) requires 4 colors, in the sense that there exists no k-colored graph with \(k<4\) whose automorphism group is \(Z_2^3\). The proof of this fact is rather tedious, but one may also check this with a help of computer. We mention it, because it means that the number 4 in the results of this paper cannot be lowered.

The permutation group generated by left translations of a regular abelian group A and its involution \(\alpha \) plays a special role in this paper. We denote it by \(A^+ = \langle A,\alpha \rangle \). We note that if \(\alpha \) is nontrivial, then \(A^+\) is nonabelian. Nevertheless we need knowledge about the representability of such groups, and to establish it, we apply Theorem 1 in [19].

Lemma 4

If A is a regular abelian permutation group, then \(A^+ \in GR(2)\), except for the following groups: \(A = Z_2^2, Z_2^3, Z_2^4, Z_4\times Z_2, Z_4\times Z_2^2, Z_3^2, Z_3^3\), or \(Z_4^2\). In any case, \(A^+\in GR(4)\).

Proof

The first claim follows from [19, Theorem 1] combined with the remark 1.2 preceding this theorem (which adds to the list of exceptions \(Z_2^2\)). For the second claim, the three first cases follow by Lemma 1. For the remaining cases the following five 4-colored graphs of the form \(\text{ Cay}^*(A,P)\) have the automorphism group equal to \(A^+\):

\(\text{ Cay}^*(Z_4\times Z_2; 10, 01)\), \(\text{ Cay}^*(Z_4\times Z_2^2; 100,010,001])\), \(\text{ Cay}^*(Z_3^2; 10, 01,11)\), \(\text{ Cay}^*(Z_3^3; 010,[001,100],[110,101])\), \(\text{ Cay}^*(Z_4^2; 10,01,13).\)

Checking this fact for each of the five graphs is routine, and similar to checking the case (iii) in the proof of Lemma 1. \(\square \)

We note that \(\text{ Cay}^*(Z_3^2; 10, 01,11)\) (pictured in the left-hand side of Fig. 4) is a unique 4-colored graph (in the sense of [12]) with the unique automorphism group \((Z_3^2)^+\). This means that if a colored graph has the automorphism group \((Z_3^2)^+\), then it is color-isomorphic to \(\text{ Cay}^*(Z_3^2; 10, 01,11)\) (i.e., it can be obtained from the latter by suitable renaming vertices and colors). In particular, the number 4 in this lemma cannot be lowered.

We have also two exceptional intransitive abelian permutation groups whose representability (from the point of view of our proof) needs to be established directly. They are two nontrivial subgroups of the direct sum \(Z_3^2\oplus Z_3^2\).

Lemma 5

Let A be a nontrivial subgroup of \(Z_3^2\oplus Z_3^2\) of the form

$$\begin{aligned} A = Z_3^2[H] \oplus _\phi Z_3^2[H], \end{aligned}$$

such that \(H=Z_3\) or \(H=I_9\). Then \(A\in GR(4)\).

Proof

First consider the case when H is a subgroup isomorphic to \(Z_3\). Note that in this case the decomposition formula above describes A uniquely up to permutation isomorphism. Indeed, each subgroup \(Z_3\) of \(Z_3^2\) may be treated as one of the summands of suitably presented \(Z_3^2\), and the isomorphism \(\phi \) between groups isomorphic to \(Z_3\) is unique up to renaming generators of \(Z_3\).

We construct a graph \(\Gamma \) as a suitable composition of two graphs of the form \(\text{ Cay}^*(Z_2^3;\Pi )\). The first component of \(\Gamma \) (corresponding to the first orbit of A) is \(\text{ Cay}^*(Z_2^3;01,11,10)\), and the second one (corresponding to the second orbit of A) is \(\text{ Cay}^*(Z_2^3;10,01)\) (we assume here that the colors 1, 2, 3 are assigned to edges in accordance with the position on the list, so in particular, the edge (00, 01) in the first graph has the same color 1 as the edge (00, 10) in the second graph). To describe the colors of edges between the two components, we assume that the pairs in the second component are denoted with overline (thus, the first component consists of pairs xy, where \(x,y \in \{0,1,2\}\), and the second component consists of analogous overlined pairs \(\overline{xy}\)). Then, we put the color 3 for the edge \((00,\overline{00})\) and the color 1 for the edge \((00,\overline{01})\). This is done under assumption that \(\text{ Aut }(\Gamma ) \supseteq A\), where the first \(Z_3\) subgroup in the decomposition \(A = Z_3^2[Z_3] \oplus _\phi Z_3^2[Z_3]\) is equal to \((Z_3,\{00,10,20\})\), and the second to \((Z_3,\{\overline{00},\overline{10},\overline{20}\})\). This assumption forces the colors for other edges in those orbitals of A that contain the mentioned edges. The remaining edges are colored 0. Note that there is no edge of color 2 between the components.

The graph \(\Gamma \) is illustrated in Fig. 2. The dashed, dotted, and solid lines correspond to colors 1, 2, and 3, respectively. To make the drawing more readable, we have applied the convention that each line between components ending with double arrows corresponds to nine edges in the given color joining each vertex in the horizontal line pointed out by the arrows in the left component with each vertex in the horizontal line pointed out by the arrows in the right component.

Fig. 2
figure 2

\(\text{ Aut }(\Gamma ) = Z_3^2[Z_3] \oplus _\phi Z_3^2[Z_3]\)

Thus we have \(\text{ Aut }(\Gamma ) \supseteq A=Z_3^2[Z_3] \oplus _\phi Z_3^2[Z_3]\), where \(\phi \) is the natural isomorphism between \(Z_3\)-subgroups. Moreover, \(\text{ Aut }(\Gamma )\) preserves the orbits of A, since the quadruples of color degrees in the first component are different from that in the second component.

Now, consider the stabilizer \(S_0\) of point 00 in \(\text{ Aut }(\Gamma )\). To get the equality above, it is enough to show that the cardinality \(|S_0|=3\). Because of the dotted edges (color 2) coming out from 00, \(S_0\) fixes the set \(\{11,22\}\). We show that, actually, \(S_0\) fixes individually points in this pair. Suppose, to the contrary, that \(S_0\) contains an automorphism \(\sigma \) transposing 11 and 22.

Then, \(\sigma \) transposes sets \(\{10,11,12\}\) and \(\{20,21,22\}\), which are disjoint triangles in color 3 (solid line) in the first component. Due to edges in color 3 between the components, \(\sigma \) transposes also the sets: \(\{\overline{01},\overline{11},\overline{21}\}\) and \(\{\overline{02},\overline{12},\overline{22}\}\), which are the triangles in color 1 (dashed lines) in the second component. Similarly, due to edges in color 1 between the components, \(\sigma \) transposes the sets: \(\{\overline{00},\overline{10},\overline{20}\}\) and \(\{\overline{02},\overline{12},\overline{22}\}\), which contradicts the previous claim.

Thus, the set \(\{00,11,22\}\) remains individually fixed in \(S_0\). It is now easy to see that each point in the first component remains fixed under \(S_0\). Consequently, (because of solid edges between the two components) each triangle in color 1 in the second component remains fixed, and (due to the edges colored 2 in the second component), for every automorphism, the permutation in one of these triangles determines permutations in other triangles. Thus, the cardinality of the stabilizer of 00 in \(\text{ Aut }(\Gamma )\) is equal to 3, proving the lemma for \(H=Z_3\).

Consider now the case when \(H=I_9\) is the trivial subgroup of \(Z_3^2\). Then A is the parallel sum \(A = Z_3^2||Z_3^2\). In this case we construct a graph \(\Gamma \) as a combination of \(\text{ Cay}^*(Z_3^2;10,01,11)\) and \(\text{ Cay}^*(Z_3^2;01,10)\). In addition, each edge of the form \((xy,\overline{xy})\) is colored 1, each edge of the form \((xy,\overline{(x+1)y})\), where addition is modulo 3, is colored 2, and the other edges are colored 0. The proof that \(\text{ Aut }(\Gamma ) = Z_3^2||Z_3^2\) is similar to the first case (but simpler), so we leave it to the reader. \(\square \)

In fact, the parallel sum \(Z_3^2||Z_3^2\) is known to belong to GR(2). A suitable construction is contained in the proof of the main result in [3].

4 The structure of abelian permutation groups

From now on (AX) denotes an abelian permutation group on a fixed set X, with orbits \(X_1,\ldots , X_r\). Then, by \(A_i=A|_{X_i}\) we denote the restriction of A to \(X_i\), by \(A^j_i\) the restriction of the pointwise stabilizer of the orbit \(X_j\) to the orbit \(X_i\), and by \(A^*_i\) the restriction of the pointwise stabilizer of the set \(V\setminus X_i\) to the orbit \(X_i\).

Two orbits \(X_i\) and \(X_j\), \(i\ne j\), are called adjacent if the factor group \(A_i/A_i^j\) is not an elementary abelian 2-group. We note that this relation is symmetric. Indeed, the restriction B of A to \(X_i\cup X_j\) can be presented as \(B= A_i[A_i^j] \oplus _\phi A_j[A_j^i]\). This means, in particular, that \(A_i/A_i^j\) is isomorphic to \(A_j/A_j^i\), which implies the claim. Accordingly, an orbit \(X_i\) of A will be called isolated, if it is not adjacent to any orbit \(X_j\), \(j\ne i\).

Let us recall that a permutation \(\sigma \) preserving orbits of A is called 2-orbit-compatible with the permutation group A, if for each pair of orbits \(X_i\) and \(X_j\), \(i\ne j\), the restriction of \(\sigma \) to \(X_i\cup X_j\) belongs to the restriction of the group A to \(X_i\cup X_j\). The group A is 2-orbit-closed if every permutation that is 2-orbit-compatible with A belongs to A. The 2-orbit-closure of A, denoted \(\bar{A}\), is the group consisting of all permutations 2-orbit-compatible with A. Obviously, \(\bar{A}\) has the same orbits as A. Moreover, it has the same restrictions \((\bar{A})_i = A_i\), \((\bar{A})_i^j =A_i^j\), \((\bar{A})_i^* = A_i^*\). In particular, we have the following.

Lemma 6

Let A be a permutation group. Then the following hold

  1. 1.

    A is abelian if and only if \(\bar{A}\) is abelian.

  2. 2.

    An orbit \(X_i\) is isolated in A if and only if \(X_i\) is isolated in \(\bar{A}\).

The notion of 2-orbit closure arises naturally, when one considers automorphism groups of colored graphs and digraphs. All these groups are obviously 2-orbit closed. It is enough to observe that a colored graph (or digraph) \(\Gamma \) has exactly two kinds of edges with regard to its automorphism group: those joining vertices within an orbit of the group and those joining vertices between two different orbits. It is easily seen that a permutation 2-orbit-compatible with \(\text{ Aut }(\Gamma )\) preserves the colors of all edges. The following is an obvious property of 2-orbit-closed groups.

Lemma 7

Let A and B be 2-orbit-closed permutation groups acting on the same set X and having the same orbits. If for any two orbits O and Q, the restriction \(A|_{O\cup Q} = B|_{O\cup Q}\), then \(A=B\).

We have also the following crucial characterization.

Lemma 8

If A is a 2-orbit-closed abelian permutation group, then for every orbit \(X_i\) of A, \(X_i\) is isolated in A if and only if \(A_i/A_i^*\) is an elementary abelian 2-group.

Proof

First observe that for each \(i\le r\), \(A_i^* \subseteq \bigcap _{j\ne i} A_i^j\). We show that for 2-orbit-closed groups the converse inclusion holds, as well. Indeed, suppose that \(\tau \in A_i^j\) for each \(j\ne i\). It follows, that for each \(j\ne i\), there is a permutation \(\sigma _j \in A\), such that its restriction to \(X_i\cup X_j\) is equal to \(\tau \) extended to \(X_j\) by fixing all points in \(X_j\). Consequently, the permutation \(\sigma \) whose restriction to \(X_i\) is equal to \(\tau \) and fixing all points in \(X\setminus X_i\) is 2-orbit-compatible with A and therefore belongs to A. Whence, \(\tau \in A_i^*\), as required.

Now, we prove our claim by contraposition. Suppose that \(A_i/A_i^*\) is not an elementary abelian 2-group, that is, it has an element \(xA_i^*\) of order \(> 2\). This is equivalent to that \(x^2 \notin A_i^*\), which means (by what proved above) that there is \(j\ne i\) such that \(x^2 \notin A_i^j\). The latter is equivalent to \(A_i/A_i^j\) has an element \(xA_i^*\) of order \(> 2\), that is, \(A_i/A_i^j\) is not an elementary abelian 2-group. This means that \(X_i\) is not isolated. These equivalences yield the required result. \(\square \)

Note that the factor group \(A_i/A_i^*\) is an abstract group; we do not define any action of this group. It plays a special role in our main result below.

5 Characterization of \(2^*\)-closed abelian permutation groups

Using definitions formulated at the beginning of Sect. 4, we state our main result.

Theorem 2

Let A be a nontrivial abelian permutation group. Then A is the automorphism group of a colored graph if and only if the following conditions hold

  1. 1.

    A is 2-orbit-closed, and

  2. 2.

    for every orbit \(X_i\) of A, if the factor group \(A_i/A_i^*\) is an elementary abelian 2-group, then so is \(A_i\).

Note that the factor group \(A_i/A_i^*\) is an elementary abelian 2-group (as abstract group) if and only if it is isomorphic to \(Z_2^n\) for some \(n\ge 0\). In turn, the permutation group \(A_i\) (being transitive) is an elementary abelian 2-group if and only if it is permutation isomorphic to the regular action of \(Z_2^n\) for some \(n\ge 0\). Note that this includes trivial cases with \(n=0\). (There are also other permutation groups that are elementary abelian 2-groups, but they are not transitive, and do not apply in our theorem).

The proof of Theorem 2 consists of a number of lemmas. We keep the notation of the previous section. First we prove the “only if” part of the theorem.

Lemma 9

If an abelian permutation group \(A\in GR\), then A satisfies conditions (1) and (2) of Theorem 5.1.

Proof

As we have already noted before Lemma 7, condition (1) obviously holds. For (2), let \(\Gamma \) be a colored graph with \(\text{ Aut }(\Gamma )= A\), and suppose that \(A_i/A_i^*\) is isomorphic to \(Z_2^m\) for some \(m\ge 0\). Since \(A_i\) is abelian and transitive on \(X_i\), it acts regularly on \(X_i\). Therefore \(X_i\) may be identified with \(A_i\) and the action of \(A_i\) with the regular action on itself. In particular, \(A_i^*\) may be considered as a subset of \(X_i\).

For each pair of elements \(x,y\in A_i^*\), there is a permutation \(\sigma \in A\) moving x into y and fixing all the elements outside \(X_i\). Because of commutativity, \(\sigma \) does the same with any pair tx and ty, where \(t\in A_i\) is treated as a permutation on \(A_i\). It follows that the cosets of \(A_i/A_i^*\) have the same property: for each pair of elements xy in the same coset, there is a permutation \(\sigma \in A\) moving x into y and fixing all the elements outside \(X_i\). It follows that for every pair of such elements xy, and every element \(z\notin X_i\), the edges xz and yz in \(\Gamma \) have the same color.

We observe that for each \(x\in X_i\), \(x^{-1}\) is in the same coset as x. Indeed, since \(A_i/A_i^* \cong Z_2^m\), for cosets we have \(xA_i^*xA_i^*= A_i^*\), and by commutativity, \(x^2A_i^*= A_i^*\); hence \(xA_i^* = x^{-1}A_i^*\), as required. Thus, we infer that the edges xz and \(x^{-1}z\) in \(\Gamma \) have the same color, for every element \(z\notin X_i\).

We proceed to show the involution \(\alpha \) in \(X_i=A_i\) treated as a permutation of X (fixing all elements \(x\notin X_i\)) preserves the colors of edges in \(\Gamma \). Indeed, by what established above, it preserves the colors of all edges in \(\Gamma \) that have at most one end in \(X_i\). On the other hand, by Lemma 2 we know that \(\alpha \) preserves the colors of the edges within \(X_i\), which proves the claim.

Consequently, \(\alpha \in A_i\). Since \(A_i\) is regular, it means that \(\alpha \) must be trivial, that is, \(x=x^{-1}\) for all \(x \in A_i\). It follows that \(A_i\) is an elementary abelian 2-group, proving the lemma. \(\square \)

The proof of the “if” part is by induction on the number of orbits in A. Below, we establish the result for two orbits. Note that in this case A is trivially 2-orbit-closed.

Lemma 10

If A is a nontrivial abelian permutation group with two orbits and A satisfies condition (2) of the theorem, then \(A\in GR(4)\).

Proof

Let O and Q be the orbits of A. Let \(A = B[B'] \oplus _\phi C[C']\) be the subdirect decomposition of A with regard to O and Q. Let \(O_1, \ldots , O_r\) be the partition of the orbit O into orbits of \(B'\). Since the actions of B on O is regular (as B is abelian and transitive), the factor group \(B/B'\) acts on the set of orbits \(O_1, \ldots , O_r\) in a regular way. The same is true of the action of \(C/C'\) on the set of orbits \(Q_1,\ldots , Q_s\) of \(C'\) on Q, since \(B/B'\) and \(C/C'\) are isomorphic, \(r=s\). It follows also that \(\phi : B/B' \rightarrow C/C'\) establishes a one-to-one correspondence between the orbits \(O_1, \ldots , O_r\) and \(Q_1,\ldots , Q_r\) so that the action of \(B/B'\) on \(O_1, \ldots , O_r\) is equivalent to the action of \(C/C'\) on \(Q_1,\ldots , Q_r\). After suitable renumbering we may assume that \(\phi (O_i) = Q_i\) for all \(i=1,\ldots ,r\). Moreover, we may assume that the orbits \(O_i\) are identified with the cosets of \(B/B'\), the orbits \(Q_i\) are identified with the cosets of \(C/C'\), and \(B' = O_1\) and \(C' = Q_1\).

We construct a 4-colored graph on the set \(O\cup Q\) of vertices. It consists of two parts: \(\Gamma _1\) on the set of vertices O, and \(\Gamma _2\) on the set of vertices Q. We take \(\Gamma _i\) to be a 4-colored graph given by Lemma 4, such that \(\text{ Aut }(\Gamma _1) = \langle B,\beta \rangle \) and \(\text{ Aut }(\Gamma _2) = \langle C,\gamma \rangle \), where \(\beta \) and \(\gamma \) are corresponding involutions.

For the edges joining O and Q we put colors as follows. First, for each \(i=1,\ldots ,r\), and for all \(y\in O_i\) and all \(z\in Q_i\), the edge yz is colored 1. These edges reflect the one-to-one correspondence between cosets. They guarantee that the regular action on cosets is parallel: if \((\sigma , \tau )\) is a permutation on \(O\cup Q\) preserving the set of these edges (where \(\sigma \) permutes O, and \(\tau \) permutes Q), then \(\sigma (O_i)= O_j\) implies \(\tau (Q_i)= Q_j\) for all \(i,j \le r\). Therefore, in the remaining part of the construction we assume that these edges are the only edges between O and Q colored 1. Note that this guarantees also that, if \(\text{ Aut }(\Gamma ) \subseteq B \oplus C\), then \(\text{ Aut }(\Gamma ) = B[B'] \oplus _\phi C[C'] = A\). So, it remains only to prove that \(\text{ Aut }(\Gamma ) \subseteq B \oplus C\). (This construction will be referred further as joining cosets in parallel manner).

Now the construction differs depending on whether the orbits O and Q are adjacent or not. First we consider the case of adjacent orbits, and define the set of edges between O and Q colored 2. They are chosen to prevent involutions in O and Q.

Since \(B/B'\) is not isomorphic to \(Z_2^n\), it has an element \(xB'\) of order greater than 2. If we would have \(xB' = x^{-1}B'\), then \(x^2B' = B'\), a contradiction. Hence x and \(x^{-1}\) lie in different cosets. We color all the edges between B and \(\phi (xB)\) with the color 2. Moreover, to make sure that \(\text{ Aut }(\Gamma ) \supseteq A\), we put color 2 for all edges between yB and \(\phi (yxB)\) for any \(y\in B\). The remaining edges between X and O are colored 0. Thus, since \(x^{-1}B \ne xB\), the edges between B and \(\phi (x^{-1}B)\) have color 0, while the edges between B and \(\phi (xB)\) have color 2. This ensures that the involution \(\gamma \) does not preserve colors of the edges, and similarly, \(\beta \) does not, either.

Figure 3 illustrates the case for \(B=Z_8, B'=Z_2, C=Z_{12}\), and \(C'=Z_3\); solid lines correspond to color 1, while dotted lines correspond to color 2. The cycles representing \((Z_8)^+\) (on the left, in color 2), and \((Z_{12})^+\) (on the right, in color 1) are drawn in a way grouping vertices corresponding to cosets; this is to make the picture more readable.

Fig. 3
figure 3

Graph \(\Gamma \) with \(\text{ Aut }(\Gamma ) = Z_{8}[Z_2]\oplus _\phi Z_{12}[Z_3]\), consisting of two cycles representing \(Z_{8}\) and \(Z_{12}\), and edges joining the cycles.

To prove that \(\text{ Aut }(\Gamma ) \subseteq B \oplus C\), it remains to show that O is a fixed block for \(\text{ Aut }(\Gamma )\). This may be achieved by suitable rearrangement of colors of edges in \(\Gamma _1\), so that the quadruple of color degrees of vertices in O is different than that in Q (note that because \(\text{ Aut }(\Gamma )\) is transitive on its orbits, this quadruple is the same for all vertices in the given orbit). Such a rearrangement is impossible only in one case, when both graphs \(\Gamma _i\) are isomorphic, all 4 colors are used, and degree in each color is the same. It follows that, according to Lemma 4, this happens only in the case when B and C are isomorphic with \(Z_3^2\). Moreover, it follows that, in such a case, A is a group of the form \(Z_{3}^2[H]\oplus _\phi Z_{3}^2[H]\), and since the orbits are adjacent, \(H = Z_3\) or \(H=I_9\). Yet, by Lemma 5, \(A\in GR(4)\) in such a case, which completes the proof for the adjacent orbits.

Now, assume that O and Q are not adjacent. Then, by condition (2), both B and C are elementary abelian 2-groups of the form \(Z_2^m\). Moreover, since A is nontrivial, at least for one of these group \(m>0\). In this case, the involutions are trivial, so we do not need any special construction to prevent them. Whence, in this case, all the edges between X and O other than the edges guaranteeing parallel action between the cosets are colored 0. Since no group of the form \(Z_2^m\), \(m>0\), has \(4n+1\) elements for any n, as in the previous case, the colors of edges in \(\Gamma _i\) can be rearranged so that to ensure that O is fixed block for \(\text{ Aut }(\Gamma )\). Then the result follows as before, completing the proof. \(\square \)

Remark 1

For future reference note that the edges between the orbits are colored in at most three colors 0, 1, 2. This includes the case covered by Lemma 5.

Remark 2

The assumption is that A is nontrivial is only to exclude the exceptional case of the trivial permutation group acting on exactly 2 elements, which (because of lack of room) is not representable by any 2-element graph.

Now we prove the “if” part.

Lemma 11

If A is a nontrivial abelian permutation group satisfying condition (1) and (2) of the theorem, then \(A\in GR(4)\).

Proof

The proof is by induction on the number of orbits r of A. If \(r=1\), A is transitive, and condition (2) means that A is an elementary abelian 2-group, since in this case \(A_i^* = A\). By Lemma 3\(A\in GR(4)\). If A has 2 orbits, then the result holds by Lemma 10.

Now, suppose that A has \(r>2\) orbits, and the result holds for all groups with the number of orbits less than r.

Consider an arbitrary orbit \(X_i\) and the decomposition of A with regard to this orbit, that is, let \(A = A_i[A_i^*] \oplus _\phi B[B']\), where B is the restriction of A to \(X\setminus X_i\). Since A is nontrivial, we may assume in addition that B is nontrivial (\(A_i\) may happen to be a fixed point). Let \(\bar{B}\) be the 2-orbit closure of B. By Lemma 6, \(\bar{B}\) is abelian, and it has \(r-1\) orbits. Moreover, it satisfies condition (2), providing all isolated orbits in \(\bar{B}\) are those isolated in A. Let us continue under this additional assumption.

Then, by the induction hypothesis, \(\bar{B}\in GR(4)\), and there exists a 4-colored graph \(\Gamma _2\) on the set of vertices \(X\setminus X_i\) representing \(\bar{B}\). We construct a graph \(\Gamma \) on X representing A. Let \(\Gamma _1\) be a 4-colored graph on \(X_i\) representing \((A_i)^+\) (by Lemma 4). We may assume that both the graphs \(\Gamma _1\) and \(\Gamma _2\) are connected in colors 2 and 3 (meaning every two vertices in \(\Gamma _i\) are connected by a path using only edges of color 2 or 3; this may be achieved by a suitable change of colors). For each orbit \(X_j\) of \(\bar{B}\) (that are exactly the orbits of A other than \(X_i\)), we put the edges colored 1 between \(X_i\) and \(X_j\) joining corresponding cosets in parallel manner as in the proof of Lemma 10. The remaining edges joining the vertices of \(X_i\) and \(X\setminus X_i\) are colored 0. Obviously, \(\text{ Aut }(\Gamma ) \supseteq A\), and since \(\bar{B}\) is intransitive, each automorphism of \(\Gamma \) preserves the orbit \(X_i\). Thus, \(\text{ Aut }(\Gamma ) \subseteq (A_i)^+ \oplus \bar{B}\).

We prove that no nontrivial involution on \(X_i\) is admitted, that is, \(\text{ Aut }(\Gamma )\) \(\subseteq A_i \oplus \bar{B})\). Indeed, if \(A_i/A_i^*\) is an elementary abelian 2-group, then by (2), so is \(A_i\), and the involution is trivial. Then \((A_i)^+ = A_i\), and the claim is obvious. So, we may assume that \(A_i/A_i^*\) is not an elementary abelian 2-group. Then by Lemma 8, \(X_i\) is not isolated, which means that there is an orbit \(X_j\), \(j\ne i\), adjacent to \(X_i\). In particular, \(A_i/A_i^j\) is not an elementary abelian 2-group. Consider the restriction of A to \(X_i\cup X_j\), which can be presented in the form \(A_i[A_i^j] \oplus _\phi A_j[A_j^i]\). Similarly as in Lemma 10, we infer that there is \(x\in A_i\) such that x and \(x^{-1}\) lie in different cosets of \(A_i/A_i^j\). Now, \(A_j/A_j^i\) is regular, as it is transitive and abelian, so if an automorphism of \(\Gamma \) fixes \(A_j^i\), it fixes all the cosets of \(A_j/A_j^i\). Because of the edges between \(X_i\) and \(X_j\) guaranteeing a parallel action on cosets, we infer that if an automorphism of \(\Gamma \) fixes \(A_i^j\), then it fixes all the cosets in \(A_i/A_i^j\). Consequently, there is no automorphism of \(\Gamma \) whose restriction to \(X_i\) would be the involution. This proves our claim.

The construction ensures that for any two orbits \(X_j\) and \(X_k\) of A, the restriction of \(\text{ Aut }(\Gamma )\) to \(X_j\cup X_k\) is the same as the restriction of A to \(X_j\cup X_k\). By Lemma 7, \(\text{ Aut }(\Gamma ) = A\), as required.

Thus, we have proved that \(A\in GR(4)\), under conditions \((**)\) that all isolated orbits in \(\bar{B}\) are those isolated in A, and that B has a nontrivial orbit. Consider now the general situation. If there is a trivial orbit (fixed point) in A, we may take this orbit as \(X_i\) above, and the result follows (because conditions \((**)\) are satisfied). Otherwise, if there is any isolated orbit in A, then we may take it as \(X_i\), and again conditions \((**)\) are satisfied, and the result follows. The result also follows in any case when there is an orbit \(X_i\) such that \((**)\) are satisfied. All that remains to consider is the situation when A has an even number r of orbits, all nontrivial, and paired in such a way that for every orbit \(X_i\) there is a unique orbit \(X_j\) in A such that \(X_i\) and \(X_j\) are adjacent.

If \(r=2\), then the result follows by Lemma 10. If \(r\ge 4\), we take a pair of adjacent orbits \(X_i\) and \(X_j\), put \(Y=X_i\cup X_j\), and \(Z=X\setminus Y\), and decompose A with regard to Y and Z: \(A = C[C'] \oplus _\phi B[B']\). Now, the proof is the same as in the case of decomposing with regard to a chosen orbit \(X_i\), with the natural modification for C consisting of two orbits, and using Lemma 10 rather than Lemma 4. This makes the proof simpler, since we may omit the part concerning involutions. An additional case is created for \(n=4\), since then we need to use a more sophisticated coloring of edges to prevent transposing sets Y and Z. In this case C and B consist each of two (adjacent) orbits, and the problem arises when \(\Gamma _1\) and \(\Gamma _2\), representing C and B, respectively, are isomorphic as colored graphs. Then we make use of Remark 1 following the proof of Lemma 10. According to this remark we may assume that in the graph \(\Gamma _1\) representing C the edges between the orbits are in colors 0, 1, 2, while for the graph \(\Gamma _2\) representing B the edges between the orbits are in colors 1, 2, 3. Then \(\Gamma _1\) and \(\Gamma _2\) are no longer isomorphic, and the construction works also in this case. This completes the proof of the lemma. \(\square \)

Now, Theorem 2 follows immediately by Lemmas 9 and 11. In fact, we have proved something more, namely, that in case of abelian permutation groups “four colors suffices.”

Corollary 1

An abelian group A is the automorphism group of a colored graph if and only if it is the automorphism group of a complete graph whose edges are colored in at most 4 colors.

We note that, by the remark following Lemma 3, the bound 4 above is sharp. In fact, using, e.g., the construction of the direct sum with the summand \(Z_2^3\) one may obtain an infinite family of abelian permutation groups requiring 4 colors to be representable by a colored graph.

6 Characterization of 2-closed abelian permutation groups

We show that the notions “2-closed” and “2-orbit-closed” coincide. The result is an essential characterization, since although checking 2-orbit-closure is not easy, checking 2-closure is computationally much harder. It is enough to note that the most elementary definition of 2-closure [5] involves orbitals of a permutation group (and hence, the induced action on the pairs of points), while the definition of 2-orbit-closure refers merely to the basic notion of orbit.

Theorem 3

Let A be an abelian permutation group. Then A is the automorphism group of a colored directed graph if and only if A is 2-orbit-closed.

Proof

To prove this result we follow the approach in the proof of Theorem 2. In fact, the proof is simpler because we can start the induction from \(r=1\) orbits, and there is no problem of involution and special cases, as in the previous proof. In addition, we may omit the assumption about nontriviality of A, since the result holds for trivial groups, as well (in particular, \(I_n\in DGR(2)\) can be represented by the directed graph consisting of one directed path of length \(n-1\)). As the arguments are similar, we give here only a sketch, referring the reader for details to the previous proof.

The “only if part” is immediate by the remark before Lemma 7. We prove the “if part” by induction on the number of orbits r of A. If \(r=1\), A is transitive, and hence regular, and by the result of Babai [4], every nontrivial regular abelian group \(A\in DGR(2)\), except for \(A=Z_2^n\) with \(n=2,3,4\), and \(A= Z_3^2\). In the first case, by Lemma 1, \(Z_2^n \in DGR(4)\) (and again \(Z_2^3\) requires 4 colors; here we have \(x=x^{-1}\) and the cases of directed and undirected graphs are the same). In the second case, \(Z_3^2\in DGR(4)\) by Lemma 4. In fact, it can be easily seen that \(Z_3^2 \in DGR(3)\) (see the right-hand side of Fig. 4).

Fig. 4
figure 4

Simple and directed graphs: \(\text{ Aut }(\Gamma ) = (Z_3^2)^+\) and \(\text{ Aut }(\Delta ) = Z_3^2\)

Now, suppose that A has \(r>1\) orbits, and the result holds for all groups with the number of orbits less than r.

Consider an arbitrary orbit \(X_i\) and the decomposition of A with regard to this orbit, that is, let \(A = A_i[A_i^*] \oplus _\phi B[B']\), where B is the restriction of A to \(X\setminus X_i\). Let \(\bar{B}\) be the 2-orbit closure of B. By Lemma 6, \(\bar{B}\) is abelian, and it has \(r-1\) orbits.

It follows, by the induction hypothesis, that \(\bar{B}\in DGR(4)\), and there exists a 4-colored digraph \(\Gamma _2\) on the set of vertices \(X\setminus X_i\) representing \(\bar{B}\). We construct a digraph \(\Gamma \) on X representing A. Let \(\Gamma _1\) be a 4-colored digraph on \(X_i\) representing \(A_i\) (which exists by the proof for the case \(r=1\)). We may assume that both the graphs \(\Gamma _1\) and \(\Gamma _2\) are connected (as undirected graphs) in colors 2 and 3 (this may be achieved by suitable change of colors). For each orbit \(X_j\) of \(\bar{B}\), we put the edges colored 1 from \(X_i\) to \(X_j\) joining corresponding cosets in parallel manner as in the proof of Lemma 10. We assume that these edges are directed from \(X_i\) to \(X_j\). The remaining edges joining the vertices of \(X_i\) and \(X\setminus X_i\) are colored 0.

Then, obviously, \(\text{ Aut }(\Gamma ) \supseteq A\), Moreover, since \(\Gamma _1\) and \(\Gamma _2\) are connected in colors 2 and 3, and edges in color 1 between \(\Gamma _1\) and \(\Gamma _2\) are directed from \(\Gamma _1\) to \(\Gamma _2\), each automorphism of \(\Gamma \) preserves the orbit \(X_i\). Thus, \(\text{ Aut }(\Gamma ) \subseteq A_i \oplus \bar{B}\).

The construction ensures that for any two orbits \(X_j\) and \(X_k\) of A, the restriction of \(\text{ Aut }(\Gamma )\) to \(X_j\cup X_k\) is the same as the restriction of A to \(X_j\cup X_k\). Hence, by Lemma 7, \(\text{ Aut }(\Gamma ) = A\), as required. This completes the proof. \(\square \)

Again, what we have proved in addition is that “four colors suffices,” and that the bound 4 below is sharp.

Corollary 2

An abelian group A is the automorphism group of a colored directed graph if and only if it is the automorphism group of a complete directed graph (without loops) whose edges are colored in at most 4 colors.