1 Introduction

Throughout this paper, all groups are supposed to be finite and we follow standard notation (e.g. [11] or [14]).

It is widely known that certain properties of either all or some maximal subgroups of a finite group may have a significant impact on the structure of the group. A typical result is, for instance, a Thompson’s classic theorem that establishes the solvability of the groups with an odd order nilpotent maximal subgroup [14, Theorem 10.4.2]. Likewise, the solvability and the structure of Schmidt groups [14, Theorem 9.1.9], i.e., minimal non-nilpotent groups, as well as the classification of minimal non-solvable groups, which arose in a series of papers by Thompson, are renowned examples of how the structure of a group can be affected or is determined by the properties of its maximal subgroups.

Let p be a prime. Recall that a group is said to be p-nilpotent if it has a normal Hall \(p'\)-subgroup. Clearly, p-nilpotency is a local version of nilpotency since a group G is nilpotent if and only if it is p-nilpotent for all primes p. According to a result of Itô, the minimal non-p-nilpotent groups are in fact Schmidt groups, and consequently, are solvable too. This was also extended into the coprime action context in [13]. In 2011, Schmidt groups were also generalized in [12] by demonstrating that groups whose maximal subgroups are either normal or nilpotent are solvable and p-nilpotent for some prime p. The authors have recently extended this result in [1, Corollary 4.2], where they prove the p-solvability of those groups whose maximal subgroups are either normal or p-nilpotent whenever p is odd, by an application of the well-known Glauberman-Thompson normal p-complement theorem [6, Theorem 8.3.1]. However, as shown in [1], this result fails when \(p=2\), indicating that groups whose maximal subgroups are either normal or 2-nilpotent do not have to be 2-solvable, or equivalently, solvable. The main goal of this paper is to show that the non-solvable structure of such groups is extremely limited, and is described as follows.

Theorem A

Let G be a non-solvable finite group and suppose that every maximal subgroup of G is either 2-nilpotent or normal in G. Let S(G) denote the solvable radical of G. Then \(S(G)=\textbf{O}_{2',2}(G)\) and

$$\textbf{O}^{2}(G)/\textbf{O}_{2',2}(G)\cong S\times \ldots \times S,$$

where S is isomorphic to the simple group \(\textrm{PSL}_2(p^{2^a})\), for some odd prime p and \(a\ge 1\), or for some odd prime \(p\equiv \pm 1\) (mod 8) and \(a=0\).

It is obvious that there do not exist non-abelian simple groups whose maximal subgroups are either 2-nilpotent or normal. However, we want to mention that a related result has recently appeared in [5], where it is given a classification of non-abelian simple groups having all subgroups nilpotent or pronormal.

For proving Theorem A, we make use of several results relying on the Classification of Finite Simple Groups, more precisely on the structure of groups whose non-solvable maximal subgroups have prime power index [3, Theorem 1]. We will also need information of the subgroup structure of some specific simple groups and certain families of simple groups of Lie type by appealing to [2, 4, 8].

Typical examples of groups that satisfy the hypotheses of Theorem A corresponding to the distinct possibilities for the parameter a are the almost simple groups \(\textrm{PGL}_2(5^2)\cong \textrm{PSL}_2(5^2).2_1\) and \(\textrm{PGL}_2(17)\cong \textrm{PSL}_2(17).2\), respectively (we follow the notation of [2]). This can be checked with the help of [2] or [15]. An example of a group satisfying the hypotheses of Theorem A that has non-trivial solvable radical is \(G=3.\textrm{M}_{10}\), a central extension of the (non-simple) Mathieu group \(\textrm{M}_{10}\). This group satisfies \(\textbf{O}^2(G)/\textbf{O}_{2',2}(G)=\textrm{PSL}_2(3^2)\). Examples in which the solvable radical is non-central can be easily constructed. It suffices to consider the direct product \(G\times H\), where G is any of the above groups and H is any non-abelian nilpotent group with (odd) order being relatively prime to |G|.

However, when all maximal subgroups of a group are either 2-nilpotent or normal, and we assume, in addition, that it does not have non-trivial quotients that are 2-groups, then we obtain more.

Corollary B

Let G be a finite group such that every maximal subgroup of G is either 2-nilpotent or normal in G. If \(\textbf{O}^2(G)=G\), then G is solvable.

Solvable groups in which every maximal subgroup is 2-nilpotent or normal do not need, of course, to satisfy the above equality \(\textbf{O}^2(G)=G\), and neither need to satisfy \(\textbf{O}_{2',2}(G)=G\), as it happens with the solvable radical in Theorem A. The symmetric group \(\textrm{S}_3\) and the alternating group \(\textrm{A}_4\) are examples showing these assertions respectively. However, we are able to bound the 2-length of such groups.

Theorem C

Let G be a solvable finite group and suppose that every maximal subgroup of G is either 2-nilpotent or normal in G. Then \(\textbf{O}_{2',2,2',2}(G)=G\).

We observe that groups satisfying the hypotheses of Theorem C actually include Schmidt groups, nilpotent groups and odd order groups. Groups satisfying the conditions of Theorem C that do not belong to any of the aforementioned classes are, for instance, the symmetric group \(\textrm{S}_4\) and \(\textrm{GL}_2(3)\).

2 Proofs

Before proving our results, we state the following classification theorem, pointed out in the Introduction, which is crucial in the proof of Theorem A. This result extends Guralnick’s work [7] concerning maximal subgroups of prime power index in simple groups and about groups with all maximal subgroups having prime power index.

Theorem 2.1

[3, Theorem 1] Let G be a non-solvable group in which every non-solvable maximal subgroup has prime power index. Then:

  1. (i)

    the non-abelian composition factors of the group G are pairwise isomorphic and are exhausted by groups from the following list:

    1. (1)

      \(\textrm{PSL}_2(2^p)\), where p is a prime,

    2. (2)

      \(\textrm{PSL}_2(3^p)\), where p is a prime,

    3. (3)

      \(\textrm{PSL}_2(p^{2^a})\), where p is an odd prime and \(a \ge 0\),

    4. (4)

      \(\textrm{Sz}(2^p)\), where p is an odd prime,

    5. (5)

      \(\textrm{PSL}_3(3)\);

  2. (ii)

    for any simple group S from the list in statement \((\textrm{i})\), there exists a group G such that any of its non-solvable maximal subgroups has primary index and \(Soc(G)\cong S\).

In the next lemma, we detail the structure of the Sylow normalizers of \(\textrm{PSL}_2(q)\) with q a prime power for the convenience of readers, as it will be repeatedly used.

Lemma 2.2

Let \(G=\textrm{PSL}_2(q)\), where q is a power of prime p and \(d = (2, q + 1)\). Let r be a prime divisor of |G| and \(R\in \textrm{Syl}_r(G)\).

  1. (1)

    If \(r=p\), then \(\textbf{N}_G(R)=R\rtimes C_{\frac{q-1}{d}}\);

  2. (2)

    If \(2\ne r\mid \frac{q+1}{d}\), then \(\textbf{N}_G(R)=C_{\frac{q+1}{d}}\rtimes C_2\);

  3. (3)

    If \(2\ne r\mid \frac{q-1}{d}\), then \(\textbf{N}_G(R)=C_{\frac{q-1}{d}}\rtimes C_2\);

  4. (4)

    Assume \(p\ne r=2\).

    1. (4.1)

      If \(q\equiv \pm 1\) (mod 8), then \(\textbf{N}_G(R)=R\);

    2. (4.2)

      If \(q\equiv \pm 3\) (mod 8), then \(\textbf{N}_G(R)=(C_2\times C_2)\rtimes C_3\).

Proof

This follows from [8, Theorem 2.8.27]. \(\square \)

We are ready to prove our results.

Proof of Theorem A

Let \(\overline{G}:=G/S(G)> 1\). Assume first that every maximal subgroup of \(\overline{G}\) is 2-nilpotent. Since \(\overline{G}\) cannot be 2-nilpotent because it is non-solvable, then \(\overline{G}\) is a minimal non-2-nilpotent group. But then, by a theorem of Itô, mentioned in the Introduction [8, Theorem IV.5.4], the minimal non-2-nilpotent groups are minimal non-nilpotent groups, so \(\overline{G}\) would be solvable, a contradiction. Therefore, we can assume that \(\overline{G}\) has a maximal subgroup \(\overline{M_1}\) that is not 2-nilpotent, so \(M_1\unlhd G\) by hypothesis. On the other hand, since \(\overline{G}\) is non-solvable, we have that not every maximal subgroup of \(\overline{G}\) can be normal in \(\overline{G}\), otherwise \(\overline{G}\) would be nilpotent. If \(\overline{M_2}\) is one of such subgroups, then again by hypothesis, \(M_2\) must be 2-nilpotent. We conclude that at least both, \(M_1\) and \(M_2\), a maximal subgroup that is normal in G and a 2-nilpotent maximal subgroup exist, and both subgroups contain S(G). In particular, S(G) is 2-nilpotent, and hence S(G) is included in the 2-nilpotent radical of G, that is, in \(\textbf{O}_{2',2}(G)\). The converse containment trivially follows by Feit-Thompson Theorem (on the solvability of odd order groups). Thus, \(\textbf{O}_{2',2}(G)=S(G)\) (possibly trivial), and from now on, without loss of generality, we will assume that \(S(G)=1\).

In the following, we prove that G has a unique minimal normal subgroup N. Assume on the contrary that there is another minimal normal subgroup K of G. Since \(S(G)=1\), we have that K is non-solvable. Now, we take M to be a 2-nilpotent maximal subgroup of G, which we know that does exist. Then \(G=NM\). As \(K\cap N=1\) then

$$K\cong KN/N\le G/N\cong M/(N\cap M),$$

from which we deduce that K is 2-nilpotent, a contradiction. Therefore, the claim is proved.

Write \(N=S_1\times \cdots \times S_n\), where \(S_i\) are isomorphic non-abelian simple groups and \(n\ge 1\). First we prove that G/N is nilpotent. Indeed, if D/N is a maximal subgroup of G/N, then D cannot be 2-nilpotent, so \(D\unlhd G\) by hypothesis, and this is equivalent to say that G/N is nilpotent, as wanted. We assert that G possesses non-solvable maximal subgroups. Observe that the only case in which all maximal subgroups of G are solvable is when G/N is trivial, \(n=1\) and G is a minimal simple group. But in this case G certainly does not satisfy the hypotheses. Therefore, let \(M_0\) be any non-solvable maximal subgroup of G. In particular, it is not 2-nilpotent, so again \(M_0\unlhd G\). From the uniqueness of N it follows that \(N\le M_0\), and then, the nilpotency of G/N implies that \(|G:M_0|\) is prime. In particular, G satisfies the hypotheses of Theorem 2.1, and we conclude that \(S_i\) are all isomorphic to one of the groups, from now on say S, listed in the thesis of that theorem.

In the following, we do a case-by-case analysis of these groups to exclude them all except those of case (3) of Theorem 2.1. We first prove that the normalizer of every Sylow subgroup of S must be 2-nilpotent. Let P be a Sylow p-subgroup of N for an arbitrary prime p dividing |N|. By the Frattini argument, we have \(G=\textbf{N}_G(P)N\), and then we can take a maximal subgroup M of G such that \(\textbf{N}_G(P)\le M\). As M cannot be normal in G, it is 2-nilpotent by hypothesis, so in particular, \(\textbf{N}_{N}(P)=\prod \textbf{N}_S(P_0)\) is 2-nilpotent as well, where \(P_0=P\cap S\in \textrm{Syl}_p(S)\). Thus the assertion is proved.

We start our analysis by assuming that \(S\cong \textrm{PSL}_2(2^q)\) with q prime, and take \(p=2\). Lemma 2.2(1) asserts that if \(P_0\in \textrm{Syl}_2(S)\), then \(\textbf{N}_S(P_0)\cong P_0\rtimes C_{2^q-1}\), which is not 2-nilpotent, contradicting the above paragraph, so this case can be discarded. A similar argument with the prime 2 works to reject the Suzuki group, \(\textrm{Sz}(q)\), with \(q=2^{2n+1}\), because the normalizer of a Sylow 2-subgroup P has the form \(P\rtimes C_{q-1}\) [9, Chap XI, Theorem 3.10].

For the case \(S\cong \textrm{PSL}_3(3)\), we observe that all Sylow normalizers are 2-nilpotent, so we cannot apply the above argument to discard it. Instead, we check in [2] that it has a unique conjugacy class of maximal subgroups of order 24, which are all isomorphic to the symmetric group \(\textrm{S}_4\). Then the direct product of n such subgroups, each one in a different factor \(S_i\), form a conjugacy class of subgroups in N. We stress that there exist other subgroups in \(\textrm{PSL}_3(3)\) that are isomorphic to \(\textrm{S}_4\) out of this single conjugacy class, but they are not maximal. Take \(\textrm{S}_4\cong H_0\) one of those maximal subgroups of S and put \(H=H_0 \times \cdots \times H_0\le N\). We show that the Frattini argument applies to get \(G=\textbf{N}_G(H)N\). Indeed, if \(g\in G\), since G (transitively) permutes the factors \(S_i\), then we have \(H^g=\prod H_0^g\) where each factor belongs to a distinct \(S_i\), is maximal in such \(S_i\) and isomorphic to \(\textrm{S}_4\). Accordingly, \(H^g\) is the direct product of n maximal subgroups isomorphic to \(\textrm{S}_4\), and lies in the above-mentioned conjugacy class of subgroups of N. It follows that \(H^g=H^n\) for some \(n\in N\), and hence \(G=\textbf{N}_G(H)N\), as wanted. Now, let us take a maximal subgroup M of G that contains \(\textbf{N}_G(H)\) because H is not normal in G. Clearly \(N\not \le M\) and thus, by minimality and uniqueness of N, it follows that M cannot be normal in G, so M is 2-nilpotent by hypothesis. Hence \(\textbf{N}_G(H)\) is 2-nilpotent too, contradicting the fact that H is not, so this case can be eliminated.

Suppose now that \(S\cong \textrm{PSL}_2(3^q)\) with q prime, and take \(p=2\). If q is odd, then we certainly have \(3^q\equiv 3\) (mod 8). If \(P_0\in \textrm{Syl}_2(S)\), then \(\textbf{N}_S(P_0)\cong \textrm{A}_4\), the alternating group, by Lemma 2.2(4.1). But \(\textrm{A}_4\) neither is 2-nilpotent, so these cases are excluded too. Therefore, the only case within this family of simple groups that we cannot exclude is when \(q=2\), that is, \(\textrm{PSL}_2(3^2)\), which belongs to case (3) of Theorem 2.1. In fact, we remark that all Sylow normalizers of this group are 2-nilpotent.

Finally, assume that \(S\cong \textrm{PSL}_2(q^{2^a})\) with q an odd prime and \(a\ge 0\). The Sylow normalizers of this group are 2-nilpotent for all primes (see again Lemma 2.2) except for the prime 2 and \(q^{2^a}\equiv \pm 3\) (mod 8); in that case the normalizers of the Sylow 2-subgroups of S are isomorphic to \(\textrm{A}_4\), which is not 2-nilpotent. But note that the above congruence only occurs when \(a=0\) and \(q\equiv \pm 3\) (mod 8), because if \(a\ge 1\), then \(q^{2^a}\equiv 1\) (mod 8) for every odd prime q. Thus, in this family of simple groups, only the case \(a=0\) and \(q\equiv \pm 3\) (mod 8) can be discarded. We want to remark that, according to the list of subgroups of \(\textrm{PSL}_2(q^{2^a})\) with \(a\ge 0\), given in Dickson’s book [4] (see also [10, Theorem 2.1] for a specific list of conjugacy classes of subgroups), we know that \(\textrm{PSL}_2(q^{2^a})\) for each \(a\ge 1\) does not have any single conjugacy class of subgroups (for every isomorphic type) of non-2-nilpotent subgroups. Thus, a similar argument to that of \(\textrm{PSL}_3(3)\) cannot be applied here so as to rule out more groups within this family of simple groups.

This completes our analysis on simple groups, so we have proved that S can only be isomorphic to one of the groups appearing in the statement of the theorem. The rest of the proof consists in proving that G/N is a 2-group.

The uniqueness of N implies that \(\textbf{C}_G(N)=1\), and then

$$N\le G \le \textrm{Aut}(N)= \textrm{Aut}(S)\wr \textrm{S}_n,$$

where \(\textrm{S}_n \) denotes the symmetric group of degree n and \(\textrm{Aut}(S)\wr \textrm{S}_n\) denotes the wreath product. Write \(A=\textrm{Aut}(S)\) and let \(A^*\) be the base group of \(A\wr \textrm{S}_n\). To see that G/N is a 2-group we prove that \((G\cap A^*)/N\) and \(G/(G\cap A^*)\) are both 2-groups. For the first quotient group, we note

$$(G\cap A^*)/N\le \textrm{Out}(S)\times \ldots \times \textrm{Out}(S).$$

As we have proved above that \(S\cong \textrm{PSL}_2(q^{2^a})\) with q odd, then it is known that Out(S) has order \((2, q^{2^a}-1) 2^a= 2^{a+1}\) [2], so our first assertion follows. It remains to show that \(G/(G\cap A^*\)) is a 2-group as well. Notice that

$$G/(G\cap A^*)\cong A^*G/A^*=(A^*G\cap \textrm{S}_n)A^*/A^*\cong A^*G\cap \textrm{S}_n.$$

Suppose that there is a prime \(p\ne 2\) such that p divides \(|A^*G\cap \textrm{S}_n|\) and we seek a contradiction. Note that this assumption implies that \(n\ge 2\) (in fact, \(n\ge 3\)). Let \(P_2\in \textrm{Syl}_2(S)\), so \(P^*=P_2\times \ldots \times P_2\) is a Sylow 2-subgroup of N, and then there exists \(P\in \textrm{Syl}_2(G)\) such that \(P\cap N=P^*\). As we know that G/N is nilpotent, then PN/N is normal in G/N, and this yields to \(G=\textbf{N}_G(P)N\). Now, since \(|G/(G\cap A^*)|\) is divisible by p and \(N\le G\cap A^*\), then there exists a p-element \(x\in G\setminus G\cap A^*\) such that \(x\in \textbf{N}_G(P)\le \textbf{N}_G(P^*)\). Furthermore, we can write \(x= as\), with \(a\in A^*\) and \(1\ne s\in \textrm{S}_n\). It is straightforward that s normalizes \(P^*\), and as a consequence \(a\in \textbf{N}_{A^*}(P^*)=\textbf{N}_A(P_2)\times \ldots \times \textbf{N}_A(P_2)\). Since s permutes non-trivially the direct factors of \(P^*\), we obtain that x does not centralize \(P^*\). On the other hand, P cannot be normal in G, otherwise \(P^*\) would be normal in N. Hence \(\textbf{N}_G(P)\) lies in some maximal subgroup M of G, which cannot be normal in G. By hypothesis, M and then \(\textbf{N}_G(P)\) too, are both 2-nilpotent. Consequently, x should centralize P, and then \(P^*\) as well. This contradiction proves that \(G/(G\cap A^*)\) is a 2-group, so G/N is a 2-group, as wanted. Moreover, we can conclude that \(N=\textbf{O}^2(G)\), and the proof is finished.\(\square \)

Remark

Even though there are multiple examples of groups sayisfying the hypotheses of Theorem A, for distinct simple groups \(\textrm{PSL}_2(p^{2^a})\), we do not know whether the list of simple groups in that theorem is exhaustive.

Proof of Corollary B

Let G be a finite group such that every maximal subgroup of G is either 2-nilpotent or normal in G and suppose further that \(\textbf{O}^2(G)=G\). Assume on the contrary that G is non-solvable. Then Theorem A claims that \(G/S(G)\cong S\times \ldots \times S\), where S(G) is the solvable radical of G and S is one of the simple groups appearing in the statement of that theorem. However, it is clear (by [8, Theorem IV.5.4]) that every non-abelian simple group must possess at least one maximal subgroup that is not 2-nilpotent. In particular, we choose such a subgroup H of S, and put \(H\times S\times \ldots \times S\), which is a maximal subgroup of G that is neither normal nor 2-nilpotent. This contradicts the hypotheses, so G must be solvable. \(\square \)

Proof of Theorem C

Let \(K=\textbf{O}_{2',2}(G)\) and write \(\overline{G}=G/K\). If \(K=G\), then the theorem is already proved, so we will assume \(\overline{G}\ne 1\) and choose \(\overline{N}\) to be a minimal normal subgroup of \(\overline{G}\), which, by solvability, has prime-power order. In addition, N cannot be 2-nilpotent because it contains K properly (the 2-nilpotent radical of G), so \(\overline{N}\) must have odd order, and hence \(N\le \textbf{O}_{2',2,2'}(G)\). We distinguish two possibilities for N. If \(N=G\), then \(\textbf{O}_{2',2,2'}(G)= G\) and we are finished. Thus, we assume that \(N<G\). Now, if M is any maximal subgroup of G containing N, then M cannot be 2-nilpotent, otherwise N would be 2-nilpotent too, a contradiction. Then, by hypothesis \(M\unlhd G\), and this means that every maximal subgroup of G/N is normal, or equivalently, that G/N is nilpotent. From this property, we deduce that \(\textbf{O}_{2',2,2',2}(G)=G\), as required. \(\square \)

Remark

We show that it is not possible to reduce more the conclusion \(\textbf{O}_{2',2,2',2}(G)=G\) in Theorem C. Let us consider the group \(G=C_3. \textrm{S}_4\), that is, the non-split extension by \(C_3\) of \(\textrm{S}_4\) acting via \(\textrm{S}_4/\textrm{A}_4\cong C_2\). In fact, \(G=\textsf{SmallGroup}(72,15)\) taken from the SmallGroups Library of GAP [15]. This group has the following upper \(2' 2\)-series:

$$1< \textbf{O}_{2'}(G){=} C_3< \textbf{O}_{2',2}(G){=}C_2\times C_6<\textbf{O}_{2',2,2'}(G){=}C_3. \textrm{A}_4 <\textbf{O}_{2',2,2',2}(G){=}G.$$

Moreover, the maximal subgroups of G are exactly: either \(C_3. \textrm{A}_4\), which is normal in G, or they are isomorphic to \(D_9\) or \(C_3\rtimes D_4\), which are 2-nilpotent.