## Abstract

Following Babai’s algorithm (Graph isomorphism in quasipolynomial time, arXiv:1512.03547v2, 2016) for the string isomorphism problem, we determine that it is possible to write expressions of short length describing certain permutation cosets, including all permutation subgroups. This is feasible both in the original version of the algorithm and in its CFSG-free version, by Babai (2016, §13.1) and Pyber (A CFSG-free analysis of Babai’s quasipolynomial GI algorithm, arXiv:1605.08266, 2016). The existence of such descriptions gives a weak form of the Cameron–Maróti classification, even without assuming CFSG. This is applicable to proofs of diameter bounds for \(\mathrm {Alt}(n)\) as in Helfgott (Growth in linear algebraic groups and permutation groups: towards a unified perspective, arXiv:1804.03049, 2018): our main result is used in Dona (Towards a CFSG-free diameter bound for \({\mathrm{Alt}}(n)\), arXiv:1810.02710v3, 2018) to free Helfgott’s proof from the use of CFSG. We also thoroughly explicate Babai’s recursion process (as given in Helfgott et al. in Graph isomorphisms in quasi-polynomial time, arXiv:1710.04574, 2017) and obtain explicit constants for the runtime of the algorithm, both with and without the use of CFSG.

This is a preview of subscription content, access via your institution.

## References

- 1.
L. Babai. On the order of uniprimitive permutation groups.

*Ann. of Math. (2)*, 113:553–568, 1981. - 2.
L. Babai. On the order of doubly transitive permutation groups.

*Invent. Math.*, 65:473–484, 1982. - 3.
L. Babai. Graph isomorphism in quasipolynomial time. arXiv:1512.03547v2, 2016.

- 4.
L. Babai, E. M. Luks, and Á. Seress. Permutation groups in NC. In

*STOC ’87 - Proceedings of the 19th Annual ACM Symposium on Theory of Computing*, pages 409–420. Association for Computing Machinery (ACM), 1987. - 5.
L. Babai and Á. Seress. On the diameter of Cayley graphs of the symmetric group.

*J. Combin. Theory Ser. A*, 49(1):175–179, 1988. - 6.
P. J. Cameron. Finite permutation groups and finite simple groups.

*Bull. Lond. Math. Soc.*, 13:1–22, 1981. - 7.
P. J. Cameron.

*Permutation Groups*. Cambridge University Press, Cambridge (UK), 1999. - 8.
D. Dona. Towards a CFSG-free diameter bound for \({{\rm Alt}}(n)\). arXiv:1810.02710v3, 2018.

- 9.
M. Furst, J. Hopcroft, and E. Luks. Polynomial-time algorithms for permutation groups. In

*FOCS ’80 - Proceedings of the 21th Annual IEEE Symposium on Foundations of Computer Science*, pages 36–41. Institute of Electrical and Electronics Engineers (IEEE), 1980. - 10.
H. A. Helfgott, J. Bajpai, and D. Dona. Graph isomorphisms in quasi-polynomial time. arXiv:1710.04574, 2017.

- 11.
H. A. Helfgott. Growth in linear algebraic groups and permutation groups: towards a unified perspective. arXiv:1804.03049, 2018.

- 12.
H. A. Helfgott. Isomorphismes de graphes en temps quasi-polynomial [d’après Babai et Luks, Weisfeiler-Leman, ...] (Exp. no. 1125). In

*Séminaire Bourbaki, Vol. 2016/2017, Exposés 1120-1135*, volume 407 of*Astérisque*, pages 135–182, 2019. - 13.
E. M. Luks. Isomorphism of graphs of bounded valence can be tested in polynomial time.

*J. Comput. System Sci.*, 25:42–65, 1982. - 14.
A. Maróti. On the orders of primitive groups.

*J. Algebra*, 258(2):631–640, 2002. - 15.
J.-P. Massias and G. Robin. Bornes effectives pour certaines fonctions concernant les nombres premiers.

*J. Théor. Nombres Bordeaux*, 8:215–242, 1996. In French. - 16.
L. Pyber. On the orders of doubly transitive permutation groups, elementary estimates.

*J. Combin. Theory Ser. A*, 62:361–366, 1993. - 17.
L. Pyber. A CFSG-free analysis of Babai’s quasipolynomial GI algorithm. arXiv:1605.08266, 2016.

- 18.
H. Robbins. A remark on Stirling’s formula.

*Amer. Math. Monthly*, 62(1):26–29, 1955. - 19.
O. Schreier. Die Untergruppen der freien Gruppen.

*Abh. Math. Semin. Univ. Hambg.*, 5:161–183, 1927. In German. - 20.
Á. Seress.

*Permutation Group Algorithms*. Cambridge University Press, Cambridge (UK), 2003. - 21.
X. Sun and J. Wilmes. Faster canonical forms for primitive coherent configurations. arXiv:1510.02195v2, 2016.

- 22.
H. Wielandt. Abschätzungen für den Grad einer Permutationsgruppe von vorgeschriebenem Transitivitätsgrad.

*Schriften Math. Seminars Inst. Angew. Math. Univ. Berlin*, 2:151–174, 1934. In German. - 23.
R. A. Wilson.

*The Finite Simple Groups*, volume 251 of*Graduate Texts in Mathematics*. Springer, London (UK), 2009. - 24.
B. Weisfeiler and A. Leman. A reduction of a graph to a canonical form and an algebra arising during this reduction.

*Nauchno-Technicheskaya Informatsiya*, 9:12–16, 1968. In Russian.

## Acknowledgements

The author thanks H. A. Helfgott for introducing him to the graph isomorphism problem and for discussions about his paper [12] on the subject.

The author is also grateful to the anonymous referees for their great patience, and for their comments directed at improving the presentation of this paper.

## Author information

### Affiliations

### Corresponding author

## Additional information

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

During his permanence in Göttingen, the author was supported by the European Research Council under Programme H2020-EU.1.1., ERC Grant ID: 648329 (codename GRANT). During his permanence in Jerusalem, he was supported by the Emily Erskine Endowment Fund and by a Postdoctoral Fellowship from the Einstein Institute of Mathematics.

Communicated by Marni Mishna.

## Appendix: Proofs of Combinatorial Lemmas

### Appendix: Proofs of Combinatorial Lemmas

As promised, here are the proofs of the results in Sect. 3.5.

### Proof of Lemma 3.16

From the partition \({\mathcal {C}}=\{A_{i}\}_{i}\) of \(\Gamma \), we construct the partition \({\mathcal {C}}'=\{A_{\mathbf {k}}\}_{\mathbf {k}}\) of \({\mathcal {B}}\) defined as follows: if \(\mathbf {k}=(k_{i})_{i\le |{\mathcal {C}}|}\) is an ordered tuple of integers with \(\sum _{i}k_{i}=k\), then \(A_{\mathbf {k}}=\{B\in {\mathcal {B}}|\forall i(|B\cap A_{i}|=k_{i})\}\). \({\mathcal {C}}'\) is also naturally a coloured partition, as follows. We give the colour of \(A_{i}\) to each \(k_{i}\) inside \(\mathbf {k}\), then we forget the ordering in \(\mathbf {k}\): the resulting multiset of coloured integers is the colour we give to \(A_{\mathbf {k}}\). If *G* respects \({\mathcal {C}}\), then evidently *G* respects \({\mathcal {C}}'\).

Now we prove the claim about the size of the \(A_{\mathbf {k}}\). First, the only \(A_{\mathbf {k}}\) of size 1 are those where \(k_{i}\in \{0,|A_{i}|\}\) for all *i*; for \(k=1\) there are none, so assume \(k>1\). Since \(|A_{i}|\ge 2\), the number of such \(A_{\mathbf {k}}\) is bounded by \(\left( {\begin{array}{c}\lfloor m/2\rfloor \\ \lfloor k/2\rfloor \end{array}}\right) \), which is \(\le \frac{1}{3}\left( {\begin{array}{c}m\\ k\end{array}}\right) \) (for \(m\ge 10\), say). Next, we prove that \(|A_{\mathbf {k}}|\le \frac{2}{3}|{\mathcal {B}}|\).

Fix some \(a>0\), take some \(A=A_{\mathbf {k}}\) for which at least one \(k_{i}=a\), and fix \(A_{0}=A_{i}\) for that particular *i*. If \(a=k\) we are done since \(|A_{0}|\le \frac{2}{3}m\), so we impose \(a<k\).

Assume first \(a>1\). The number of *k*-subsets of \(\Gamma \) intersecting \(A_{0}\) in *a* points is \(\left( {\begin{array}{c}|A_{0}|\\ a\end{array}}\right) \left( {\begin{array}{c}m-|A_{0}|\\ k-a\end{array}}\right) \), so this is an upper bound for |*A*|. It is enough to show that this number is at most \(\frac{2}{3}\left( {\begin{array}{c}m\\ k\end{array}}\right) \) for \(m\ge 100\).

Let us call \(|A_{0}|=\beta m\), where \(\beta \le \frac{2}{3}\). Then one can show

unfolding the binomial coefficients and using \(\beta m-i\le \beta (m-i)\) for all \(0\le i<a\). One verifies directly the statement for \(1<a<k\le 5\), given \(m\ge 100\); suppose then that \(k>5\). The last factor in (6.1) is bounded by \(e^{\frac{2}{\log m}}\), for our choice of *k*. Then we prove

using Robbins’s bound [18] for factorials, and the whole right hand side of (6.1) is bounded by \(\frac{2}{3}\left( {\begin{array}{c}m\\ k\end{array}}\right) \).

Finally, we must cover the case where for \(A_{\mathbf {k}}\) we have \(k_{i}\in \{0,1\}\) for all *i*; there are at least two sets with \(k_{i}=1\), or else we would have \(a=k\). We repeat the steps above, starting from \(\beta _{1}m\beta _{2}m\left( {\begin{array}{c}(1-\beta _{1}-\beta _{2})m\\ k-2\end{array}}\right) \) this time, and we obtain

which is smaller than \(\frac{2}{3}\left( {\begin{array}{c}m\\ k\end{array}}\right) \) for \(k\ge 3\). For \(k=2\), the inequality \(\beta _{1}\beta _{2}m^{2}<\frac{2}{3}\left( {\begin{array}{c}m\\ 2\end{array}}\right) \) is immediate, and we are done. \(\square \)

### Proof of Lemma 3.17

Note that \(2\le k'\le \frac{|\Gamma '|}{2}\) implies \(|\Gamma '|\ge 4\).

Let \(\Delta \) be any orbit of \({\mathcal {B}}\). Any \(x\in \Delta \) is a *k*-set of \(k'\)-sets of elements of \(\Gamma '\), so call \(A(x)\subseteq \Gamma '\) the set of the elements of all the elements of *x*: since the action induced on \(\Delta \) comes from a permutation of \(\Gamma '\), |*A*(*x*)| is constant for all \(x\in \Delta \), say \(|A(x)|=a\). Moreover, for the same reason \(\Delta \) is divided into blocks, one for each possible *A*(*x*): to conclude, we only need to prove that this block system is not trivial unless \(|\Delta |\le \frac{1}{2}|{\mathcal {B}}|\).

Blocks of size 1 mean that each *x* already collects all the possible \(k'\)-subsets of its own *A*(*x*). This means that \(k=\left( {\begin{array}{c}a\\ k'\end{array}}\right) \) and, calling \(|\Gamma '|=m'\), that \(\Delta \) has at most \(\left( {\begin{array}{c}m'\\ a\end{array}}\right) \) elements, one for each *A*(*x*). If \(a=m'\) we are done, so assume \(a<m'\). \({\mathcal {B}}\) has \(\left( {\begin{array}{c}|\Gamma |\\ k\end{array}}\right) \) elements, where \(|\Gamma |=\left( {\begin{array}{c}m'\\ k'\end{array}}\right) \), so it is sufficient to show that

Since \(k\ge 2\) there are at least two distinct \(k'\)-subsets of \(\Gamma '\) forming *A*(*x*), so \(a>k'\) and then \(a\le \left( {\begin{array}{c}a\\ k'\end{array}}\right) \). Verify (6.3) directly for all \(2\le k'<a<m'<12\). Then, (6.3) follows from the bounds \(\left( \frac{x}{y}\right) ^{y}\le \left( {\begin{array}{c}x\\ y\end{array}}\right) \le \left( \frac{ex}{y}\right) ^{y}\) and from the fact that \(m'\ge 12\) implies \(\left( {\begin{array}{c}m'\\ k'\end{array}}\right) ^{a}\ge \left( \frac{11}{2}m'\right) ^{a}>2(em')^{a}\).

Now we assume \(\Delta \) to be a unique block. For \(|\Delta |>\frac{1}{2}|{\mathcal {B}}|\), this implies \(A(x)=\Gamma '\). For each \(x\in \Delta \) and \(\gamma \in \Gamma '\), call \(N(\gamma ,x)=|\{y\in x|y\ni \gamma \}|\): the multiset \(\{N(\gamma ,x)|\gamma \in \Gamma '\}\) is the same for all \(x\in \Delta \), since permutations of \(\Gamma '\) must preserve it.

Suppose first that this multiset has all equal elements, say it is \(\{N,N,N,\ldots ,N\}\) (\(m'\) times). Consider the set \({\mathcal {C}}_{1}\subseteq {\mathcal {B}}\) of all *x* with this multiset, so that \(\Delta \subseteq {\mathcal {C}}_{1}\). Consider also \({\mathcal {C}}_{2}\), containing all *x* with multiset \(\{N+1,\ldots ,N+1,N-1,\ldots ,N-1,N,\ldots ,N\}\), where there are as many \(N+1\) as there are \(N-1\) and the remaining entries (if any) are *N*. Construct the bipartite graph \({\mathcal {C}}_{1}\cup {\mathcal {C}}_{2}\) where \(\{x_{1},x_{2}\}\) is an edge if and only if we can change exactly one \(k'\)-subset inside \(x_{1}\) to obtain \(x_{2}\). Every \(x_{1}\in {\mathcal {C}}_{1}\) has \(k\left( \left( {\begin{array}{c}m'\\ k'\end{array}}\right) -k\right) \ge \left( {\begin{array}{c}m'\\ k'\end{array}}\right) \) neighbours, since we can move each of the \(k'\)-subsets of \(x_{1}\) to any of the \(k'\)-subsets that are not already in \(x_{1}\) and obtain some (distinct) element of \({\mathcal {C}}_{2}\). On the other hand, the number of neighbours of a given \(x_{2}\) is at most \(\left( {\begin{array}{c}m'-2k''\\ k'-k''\end{array}}\right) \), where \(k''\) is the number of \(N+1\) in the multiset of \(x_{2}\): in fact, each \(k'\)-subset that contains all the \(\gamma \) with \(N+1\) can be moved only in one way to produce an element of \({\mathcal {C}}_{1}\). Provided that \(b_{i}\le \frac{1}{2}a_{i}\), having \(a_{1}\le a_{2}\) and \(b_{1}\le b_{2}\) implies \(\left( {\begin{array}{c}a_{1}\\ b_{1}\end{array}}\right) \le \left( {\begin{array}{c}a_{2}\\ b_{2}\end{array}}\right) \); therefore

and since \({\mathcal {C}}_{1}\) and \({\mathcal {C}}_{2}\) are disjoint we obtain \(|\Delta |\le |{\mathcal {C}}_{1}|\le \frac{1}{2}|{\mathcal {B}}|\).

Now suppose that the multiset \(\{N(\gamma ,x)|\gamma \in \Gamma '\}\) has at least two distinct elements; note that \(kk'>m\), from \(A(x)=\Gamma '\) and the fact that equality would imply \(N(\gamma ,x)=1\) regardless of \(\gamma \). Take *N* to be the least frequent element in the multiset, or one of them arbitrarily if more than one exists: say that there are \(k''\) occurrences of *N*, with \(k''\le \frac{m'}{2}<\frac{kk'}{2}\), and call \(A'(x)=\{\gamma |N(\gamma ,x)=N\}\). By definition \(A'(x)\subsetneq \Gamma '\), so there must exist elements *x* with different \(A'(x)\): we group together elements \(x\in {\mathcal {B}}\) based on their \(A'(x)\), and again this forms a system of blocks. The blocks are not the whole \({\mathcal {B}}\) since \(A'(x)\ne \Gamma '\), and one can prove that

similar to how we did for (6.3), excluding that the blocks have size 1. Therefore the new system divides \(\Delta \) nontrivially, and we are done. \(\square \)

## Rights and permissions

## About this article

### Cite this article

Dona, D. On Short Expressions for Cosets of Permutation Subgroups.
*Ann. Comb.* **25, **405–437 (2021). https://doi.org/10.1007/s00026-021-00535-6

Received:

Accepted:

Published:

Issue Date:

### Keywords

- Permutation subgroups
- CFSG
- String isomorphism problem

### Mathematics Subject Classification

- 20B35
- 20E34
- 05E15
- 05C60
- 05C85
- 68Q25