Primitive normalisers in quasipolynomial time

The normaliser problem has as input two subgroups H and K of the symmetric group Sn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {S}_n$$\end{document}, and asks for a generating set for NK(H)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_K(H)$$\end{document}: it is not known to have a subexponential time solution. It is proved in Roney-Dougal and Siccha (Bull Lond Math Soc 52(2):358–366, 2020) that if H is primitive, then the normaliser problem can be solved in quasipolynomial time. We show that for all subgroups H and K of Sn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {S}_n$$\end{document}, in quasipolynomial time, we can decide whether NSn(H)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\mathrm {S}_n}(H)$$\end{document} is primitive, and if so, compute NK(H)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_K(H)$$\end{document}. Hence we reduce the question of whether one can solve the normaliser problem in quasipolynomial time to the case where the normaliser in Sn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {S}_n$$\end{document} is known not to be primitive.


Introduction
The normaliser problem asks for a generating set for N K (H), given subgroups K and H of S n .It is shown in [12] that the problem can be solved in simply exponential time 2 O(n) , but there is no known subexponential solution to the general problem.A permutation group problem P is said to be quasipolynomial if there exists a constant c such that P can be solved in time 2 O(log c n) , where n is the degree of the underlying group or groups.
It is shown in [11] that if H is primitive, then the normaliser problem is quasipolynomial.In this paper, we will show that if N Sn (H) is primitive then the normaliser problem is quasipolynomial.Our main theorem is the following.Theorem 1.1.Let subgroups H = X and K = Y of S n be given.
1. We can decide if N = N Sn (H) is primitive, and if so construct N , in time 2 O(log 3 n) .
2. If N is primitive, then we can compute N K (H) in time 2 O(log 3 n) .
(Throughout the paper we shall assume all generating sets have size at most n: see Lemma 2.3.1).
In fact, we can compute N K (H) in time 2 O(log 3 n) except when: (i) H is intransitive; or (ii) |H| ≥ n 1+⌊log n⌋ but H is not ample (see Definition 3.1); or (iii) |H| < n 1+⌊log n⌋ but H does not have a small base or a small generating set (see Lemma 2.4).In this latter case, we can still compute N K (H) in quasipolynomial time 2 O(log 5 n) , see Proposition 2.5.
Babai in [1] gave a 2 O(log c n) time solution to the string isomorphism problem, and Helfgott in [3] showed that we can take c = 3.The setwise stabiliser problem is a special case of the string isomorphism problem, and was shown in [7] to be polynomial-time equivalent to the intersection problem.Hence to show that N K (H) = N ∩ K can be computed in time 2 O(log 3 n) , it suffices to prove that N can be computed in time 2 O(log 3 n) .
In Section 2, we first present some preliminaries on permutation groups and permutation group algorithms.We then see how we can determine that certain groups H have base and generating set of size O(log n) in quasipolynomial time and prove Proposition 2.5.In Section 3, we introduce the class of ample groups and show that if H is ample then N = N Sn (H) can be computed in quasipolynomial time.Finally these results come together to prove Theorem 1.1.

Preliminaries and small groups
This section first collects background on permutation groups and polynomial time computation and then studies small groups.
Let G ≤ Sym(Ω) and H ≤ Sym(Γ).Then G and H are permutation isomorphic if there exist an isomorphism φ : G → H and a bijection σ : Ω → Γ such that σ(ω g ) = σ(ω) φ(g) for all ω ∈ Ω and g ∈ G.We say that such a pair (φ, σ) is a permutation isomorphism from G to H. We will be using the following key result, proved by Maróti.

Theorem 2.2 ([9]
).Let G be a primitive subgroup of S n .Then at least one of the following holds.
2. There exist m ≥ 5, 1 ≤ k < m/2 and l ≥ 1 such that, up to permutation isomorphism, We shall call these classes Mathieu, large and small respectively.A primitive group is of type PA if it is in product action and the component of the base group is almost simple (see [6]).It follows that a large primitive group is either almost simple (when l = 1) or of type PA (when l > 1).
For G = z 1 , z 2 , . . ., z k ≤ S n and L = y 1 , y 2 , . . ., y k ≤ S m , a homomorphism φ : G → L is given by generator images if it is encoded by a list [z 1 , . . ., z k , y 1 , . . ., y k , φ(z 1 ), . . ., φ(z k )].We shall assume that all homomorphisms are given by generator images, that we have a library of standard representations of all finite simple groups, and that their automorphism groups are known.
The following results are standard (see, for example, [4]).
Lemma 2.3.Given G = Z ≤ S n , the following can be done in time polynomial in |Z| • n.
1. Replace Z by a generating set for G of size at most n; given σ ∈ S n , decide if σ ∈ G; compute |G|; compute the orbits of G; compute the stabiliser in G of any given point; compute an irredundant base for G; decide if G is primitive.
2. Given a map φ : G → L by the images of Z, decide if φ extends to an isomorphism; given an isomorphism φ : G → L, compute φ −1 .
3. Find a minimal normal subgroup of G; compute C G (J) for J G; find generators for the socle soc(G).

Compute the composition factors of G;
decide if G is simple and if so, give an isomorphism from G to a standard representation.
Next, we show that we can find a small base and a small generating set for certain groups H in quasipolynomial time.For a group G ≤ S n , let d(G) and b(G) denote the size of the smallest generating set and base of G, respectively.Lemma 2.4.Let H = X ≤ S n be given.Lastly we observe that the normaliser problem for groups of order less than n 1+⌊log n⌋ can be solved in quasipolynomial time, even if they are not primitive.Proposition 2.5.Let H = X ≤ S n be given.If |H| < n 1+⌊log n⌋ , then N Sn (H), and hence N K (H), can be computed in time 2 O(log 5 n) .Proof.By Lemma 2.3.1, in polynomial time, we can check that |H| < n 1+⌊log n⌋ , compute an irredundant base B for H and remove from X = {x 1 , x 2 , . . ., x s } the generators x i where x i ∈ x 1 , . . ., x i−1 .This gives a base B and a generating set Z for H of size at most log |H| ∈ O(log 2 n).
In [11, proof of Theorem 3.3], it is shown that in time 2 O(|Z||B| log n) , we can construct a set containing all |Z|-tuples of elements of H that are images of Z under conjugation by elements of N Sn (H).By Lemma 2.3.2 and [8, Lemma 3.5], for each such potential image, we can determine a conjugating element σ ∈ N Sn (H) or show that no such σ exists in polynomial time.

Ample groups
In this section, we will introduce ample groups, and show that if N = N Sn (H) is a large primitive group, then H is ample.We then show that in quasipolynomial time, we can decide if a given group is ample and if so compute its normaliser.Finally, we will prove Theorem 1.1.Definition 3.1.We define a subgroup H of S n to be ample if there exist m ≥ 5, 1 ≤ k < m/2 and l ≥ 1 such that soc(H) is permutation isomorphic to A(m, k, l).
Notice that an ample group can be imprimitive.Lemma 3.2.Let H be a subgroup of S n such that N = N Sn (H) is a large primitive group.Then soc(N ) = soc(H), and H is ample.
Proof.We first show that soc(N ) = soc(H).The group soc(H) is characteristic in H, so soc(H) N .A large primitive group is either almost simple or of type PA, and so N has a unique minimal normal subgroup (see [6]).Therefore soc(N ) ≤ soc(H) ≤ H, so soc(N ) H.
To see that soc(H) ≤ soc(N ), let M be a minimal normal subgroup of H. Then either [2,Theorem 4.3B], C N (soc(N )) = 1, a contradiction.Therefore, all minimal normal subgroups of H are contained in soc(N ), hence soc(H) ≤ soc(N ), and so soc(N ) = soc(H).
The largeness of N implies that there exist m, k and l such that soc(N ) is permutation isomorphic to A(m, k, l).Now since soc(H) = soc(N ), the group H is ample.
The following is well known (see [2,Theorem 4.5A], for example).Lemma 3.4.Given H = X ≤ S n , in polynomial time, we can decide if H is ample, and if so output a permutation isomorphism from soc(H) to A(m, k, l), for some m, k and l.
Proof.By Lemma 2.3.3,we can compute a generating set for S := soc(H) in polynomial time.The group S is a direct product of simple groups, so we can decide whether S ∼ = A l m , for some m ≥ 5 and l ≥ 1, by checking if S has l composition factors, each isomorphic to A m .By Lemma 2.3.4,this can be done in polynomial time.
If S is isomorphic to A l m , we next decide if there exists a k such that 1 ≤ k < m/2 and n = m k l .If so, we construct an isomorphism ι : S → A(m, k, l) as follows.Initialise N 1 = S, then for 2 ≤ i ≤ l, we iteratively find a minimal normal subgroup M i of N i and take for each i and so S = M 1 ×M 2 ×. . .×M l .We construct an isomorphism ι : S → A(m, k, l) using an isomorphism from each M i to a direct factor of A(m, k, l), via isomorphisms to a standard copy of A m , in polynomial time by Lemma 2.3.4.
It remains to show how to find a permutation isomorphism between S and A(m, k, l).Let ∆ = [m] l k and let W ≤ Sym(∆) be as in Lemma 3.3.If m = 6 then there exists an involution α such that Aut(A m,k ) = S m,k , α , and we can obtain such an α in polynomial time by Lemma 2.3.4.So in polynomial time, we can write down all 2 l ≤ 2 log n = n coset representatives of W in Aut(A(m, k, l)).We check if S and A(m, k, l) are permutation isomorphic by checking if there exist such a coset representative λ and a bijection σ : {1, 2, . . ., n} → ∆ such that (ιλ, σ) is a permutation isomorphism, in polynomial time by [11,Lemma 2.7].If m = 6 then Aut(A m,k ) = S m,k and so Aut(A(m, k, l)) = W , and we may set λ = 1.
If H is ample and l = 1, then H is almost simple.The next result considers the case where H is ample and l > 1.
Theorem 3.5.Given H = X ≤ S n , we can decide if H is ample and not almost simple, and if so compute N = N Sn (H), in time 2 O(log n log log n) .Proof.By Lemma 3.4, in polynomial time, we can check if H is ample and not almost simple, and if so obtain a permutation isomorphism (φ, σ) from soc(H) to A(m, k, l).
We first show that we can compute a generating set for M = N Sn (soc(H)) of size at most four in polynomial time.Let W be as in Lemma 3.3.The bijection σ −1 induces an isomorphism

Notation 2 . 1 .
Let [m] k denote the set of all k-subsets of {1, 2, . . ., m} with 1 ≤ k ≤ m/2.Let A m,k and S m,k denote A m and S m , acting on [m] k .Let [m] l k denote the set of all l-tuples of [m] k , and let A(m, k, l) be the group (A m,k ) l acting coordinatewise on [m] l k .

Lemma 3 . 3 .
Let W ≤ Sym([m] l k ) be S m,k ≀ S l acting on [m] l k , for some m ≥ 5, 1 ≤ k < m/2 and l ≥ 1.Then the normaliser in Sym([m] l k ) of A(m, k, l) is W .Next we give a polynomial time algorithm to decide whether H is ample.
1.If |H| ≤ n 1+⌊log n⌋ , then in time 2 O(log 3 n) , we can decide if d(H) ≤ ⌈log n⌉, and if so output such a generating set.2. In time 2 O(log 2 n) , we can decide if b(H) ≤ ⌈log n⌉ + 1, and if so output such a base.3.If N = N Sn (H) is a small primitive group, then d(H) ≤ ⌈log n⌉ and b(H) ≤ ⌈log n⌉ + 1. Proof.Part 1: We consider all (⌈log n⌉)-tuples Z of elements of H and decide for each Z if Z = H.The number of such tuples is |H| ⌈log n⌉ ∈ 2 O(log 3 n) .By Lemma 2.3.1, for each such tuple Z, we can decide if Z = H in polynomial time.
Part 2: We consider all (⌈log n⌉ + 1)-tuples B over {1, 2, . . ., n}.For each such B, we check if B is a base of H by checking if H (B) = 1, which can be done in polynomial time by Lemma 2.3.1.Since there are n ⌈log n⌉+1 ∈ 2 O(log 2 n) tuples to consider, the result follows.Part 3: If N is a small primitive group, then H has order at most n 1+⌊log n⌋ .Since H is a normal subgroup of a primitive group, by [5, Theorem 1.1], d(H) ≤ log n or H = S 3 , so d(H) ≤ ⌈log n⌉.By [10], b(H) ≤ b(N ) ≤ ⌈log n⌉ + 1.