On irreducible subgroups of simple algebraic groups

Abstract

Let G be a simple algebraic group over an algebraically closed field K of characteristic $p\geqslant 0$, let H be a proper closed subgroup of G and let V be a nontrivial irreducible KG-module, which is p-restricted, tensor indecomposable and rational. Assume that the restriction of V to H is irreducible. In this paper, we study the triples (G, H, V) of this form when G is a classical group and H is positive-dimensional. Combined with earlier work of Dynkin, Seitz, Testerman and others, our main theorem reduces the problem of classifying the triples (G, H, V) to the case where G is an orthogonal group, V is a spin module and H normalizes an orthogonal decomposition of the natural KG-module.

1 Introduction

Let G be a simple algebraic group defined over an algebraically closed field K, let H be a closed positive-dimensional subgroup of G, and let V be a nontrivial rational irreducible KG-module. We say that (G, H, V) is an irreducible triple if V is irreducible as a KH-module. Triples of this form arise naturally in the investigation of maximal subgroups of classical algebraic groups, and their study can be traced back to work of Dynkin [7] in the 1950s, who considered the special case where H is connected and $K = \mathbb {C}$. In the 1980s, Seitz [17] and Testerman [20] extended the analysis to arbitrary algebraically closed fields (still assuming H is connected), and more recent work of Ghandour [10] has completed the classification of irreducible triples for exceptional groups. Therefore, in this paper we focus on classical groups and their disconnected subgroups.

In [8, 9], Ford determines the irreducible triples (G, H, V) where G is a classical group, H is disconnected, $H^0$ is simple and the composition factors of the restriction $V|_{H^0}$ are p-restricted. Our main aim is to extend Ford’s analysis by removing the restrictive conditions on the structure of $H^0$ and the composition factors of $V|_{H^0}$. The cases for which $V|_{H^0}$ is irreducible are easily deduced from the work of Seitz [17], so we focus on the situation where $V|_{H}$ is irreducible, but $V|_{H^0}$ is reducible. By Clifford theory, the highest weights of the $KH^0$-composition factors of V are H-conjugate and this severely restricts the possibilities for V. Since the triples with H maximal have recently been determined in [3, 4], in this paper we will adopt the following hypothesis:

Hypothesis 1.1

The group G is a simply connected cover of a simple classical algebraic group defined over an algebraically closed field K of characteristic $p \geqslant 0$, H is a closed positive-dimensional subgroup of G, and V is a nontrivial p-restricted irreducible tensor indecomposable rational KG-module such that the following conditions hold:

• H1. $V \ne W^{\tau }$ for any automorphism $\tau$ of G, where W is the natural module;

• H2. HZ(G) / Z(G) is disconnected and non-maximal in G / Z(G).

Let G be a classical group as in Hypothesis 1.1, let n denote the rank of G and let $\{\lambda _1, \ldots , \lambda _n\}$ be a set of fundamental dominant weights for G (we adopt the standard labelling given in Bourbaki [2]). We will write $V_G(\lambda )$ for the irreducible KG-module with highest weight $\lambda$.

Remark 1.2

Condition H1 in Hypothesis 1.1 is equivalent to assuming $V \ne W,W^*$, and also $V \ne V_G(\lambda _3), \, V_G(\lambda _4)$ if $G=D_4$. This hypothesis is unavoidable. For example, we cannot feasibly determine all the simple subgroups of G that act irreducibly on W or $W^*$ (indeed, even the dimensions of the irreducible modules for simple groups are not known, in general). In particular, H1 is a condition adopted in [3, 17].

Suppose G is of type $B_n$ or $D_n$, and let $R(W)=R$ be the radical of the corresponding bilinear form on W (recall that either $R=0$, or $p=2$, $\dim W$ is odd and $\dim R=1$). An orthogonal decomposition of W is a decomposition of the form

$$\begin{aligned} W = W_1 + \cdots + W_t, \end{aligned}$$

where the $W_i$ are pairwise orthogonal subspaces of W. Note that if $W = W_1 + \cdots + W_t$ is such a decomposition, then the $W_i$ are non-degenerate spaces such that $W_i \cap (\sum _{j \ne i}W_j) \subseteq R$ for each i (in particular, if $p \ne 2$ then $W = W_1 + \cdots + W_t$ is a direct sum). We say that a subgroup H of G normalizes such a decomposition if it permutes the $W_i$.

Definition 1.3

Let G be a classical group of type $B_n$ or $D_n$, as in Hypothesis 1.1. A closed subgroup H of G is a decomposition subgroup if one of the following holds:

• (a) H normalizes an orthogonal decomposition $W = W_1 + \cdots + W_t$; or

• (b) $(G,p)=(D_n,2)$, H stabilizes a 1-dimensional non-singular subspace U of W, and H normalizes an orthogonal decomposition of the natural module for the stabilizer $G_U=B_{n-1}$.

Similarly, if $(G,p)=(C_n,2)$ then H is a decomposition subgroup of G if it is the image of a decomposition subgroup of the dual group $\widetilde{G}=B_n$ with respect to a bijective morphism $\varphi :\widetilde{G} \rightarrow G$.

Theorem 1.4

Suppose G, H and $V = V_G(\lambda )$ satisfy the conditions in Hypothesis 1.1, and assume that $V|_{H^0}$ is reducible. Then $V|_{H}$ is irreducible only if one of the following holds:

• (a) (G, H, V) is one of the cases in Table 1; or

• (b) G is of type $B_n$ or $D_n$ (or type $C_n$ if $p=2$), V is a spin module and H is a decomposition subgroup.

Moreover, if (a) holds then $V|_{H}$ is irreducible.

Table 1 The irreducible triples (G, H, V) in Theorem 1.4

Remark 1.5

In case (i) of Table 1, $T_n$ denotes a maximal torus of G. In all cases, $H^0$ is the connected component of a maximal subgroup of G, with the exception of the cases labelled (ii) and (iii), where H is contained in a subgroup $D_4<G$. Also note that in cases (iii)–(vi), $W|_{H^0}$ is the tensor product of the natural modules of the simple components of $H^0$. In case (ii), H is the image of a subgroup $C_1^3.X<C_4$ as in (iii), under an isogeny $\varphi :C_4 \rightarrow B_4$. In cases (v) and (vi), we record H and V up to $\mathrm{Aut}(G)$-conjugacy (so in case (vi) for example, if $\tilde{H}$ is the image of H under a graph automorphism of G, then $V_G(\lambda _8)$ is an irreducible $K\tilde{H}$-module). Finally, let us note that the situation in part (b) of Theorem 1.4 is very special and we refer the reader to Sect. 7 for further details.

Remark 1.6

If $V|_{H^0}$ is irreducible, then $(G,H^0,V)$ is one of the cases in [17, Table 1] and

$$\begin{aligned} H^0 \leqslant H \leqslant N_G(H^0) \quad \text{ and } \quad C_G(H^0) \leqslant Z(G). \end{aligned}$$

More precisely, we are either in the situation described in part (b) of Theorem 1.4, or in terms of Seitz’s notation in [17, Table 1], one of the following holds (modulo scalars):

• (a) $N_G(H^0) = H^0$;

• (b) $N_G(H^0) = H^0.2$ and $(G,H^0,V)$ is one of the cases labelled I$_4$, I$_5$, I$_6$ (with $n=3$ in the notation of [17, Table 1]), II$_1$, S$_1$, S$_7$, MR$_1$, MR$_4$;

• (c) $N_G(H^0) = H^0.\mathrm{Sym}_3=D_4.\mathrm{Sym}_3$ and $(G,H^0,V)$ is the case labelled S$_8$.

Here we refer the reader to the proof of [3, Theorem 2.5.1] to see that $N_G(H^0) = H^0.2$ in the case labelled S$_7$, and that $N_G(H^0)=H^0.\mathrm{Sym}_3$ in case S$_8$.

Let us briefly describe our approach to the proof of Theorem 1.4. Suppose $V|_{H^0}$ is reducible. If H is a maximal subgroup of G then (G, H, V) can be read off from the main theorems of [3, 4], so let us assume $H< M < G$ with M maximal. Since $V|_{H}$ is irreducible, it follows that $V|_{M}$ is also irreducible and we can consider the possibilities for the irreducible triple (G, M, V), which are determined in [17] (if M is connected) and [3, 4] (if M is disconnected). We can then proceed by studying the possible embeddings of H in M.

Our next result is a combination of the main theorems in [3, 4, 17], together with Theorem 1.4. Note that we assume $n \geqslant 3$ if $G=B_n$, $n \geqslant 2$ if $G=C_n$, and $n \geqslant 4$ if $G=D_n$.

Corollary 1.7

Let G be a simply connected cover of a simple classical algebraic group over an algebraically closed field K of characteristic $p \geqslant 0$. Let H be a positive-dimensional closed subgroup of G, and let $V=V_G(\lambda )$ be a nontrivial p-restricted irreducible tensor indecomposable rational KG-module such that $V|_{H}$ is irreducible. Then one of the following holds:

• (a) $V=W^{\tau }$ for some automorphism $\tau$ of G, where W is the natural module;

• (b) G is of type $B_n$ or $D_n$ (or type $C_n$ if $p=2$) and V is a spin module;

• (c) (G, H, V) is recorded in Table 2.

Table 2 Positive-dimensional irreducible subgroups of classical groups

Remark 1.8

Let us make some comments on the cases in Table 2:

• (i) In Table 2, we write X for $\mathrm{Sym}_3$ or $Z_3$, Z for $Z_2$ or 1, and Y denotes any k-transitive subgroup of $\mathrm{Sym}_{n+1}$.

• (ii) For G of type $A_n$, the highest weight $\lambda$ is recorded up to conjugacy by a graph automorphism of G (this is consistent with [17, Table 1]). For instance, in the first row of the table, we have $G = A_n$, $H=T_n.Y$ and $\lambda =\lambda _k$ with $2 \leqslant k \leqslant (n+1)/2$. By applying a suitable graph automorphism, we see that $V_G(\lambda _k)|_{H}$ is irreducible for all $2 \leqslant k \leqslant n-1$.

• (iii) In the fourth column, we describe the restriction of $\lambda$ to a suitable maximal torus of the derived subgroup $[H^0,H^0]$ (this is denoted by $\lambda |_{H^0}$), in terms of highest weights for the simple components. The one exception is the case $(G,H)=(A_n,T_n.Y)$, where $H^0=T_n$ is a maximal torus of G and thus $[H^0,H^0]=1$.

• (iv) In the fifth column, $\kappa$ denotes the number of $KH^0$-composition factors of $V|_{H^0}$.

• (v) In the final column, we record various conditions on G, H and $\lambda$ that are necessary and sufficient for the irreducibility of $V|_{H}$. In a few cases, the conditions for irreducibility are rather complicated and so we record them here (the example in case (b) was discovered by Ford in [8]):

  • (a) $(G,H,\lambda ) = (A_{2m-1},C_m,a\lambda _k+b\lambda _{k+1})$: $1 \leqslant k < m$; $a+b=p-1$; $p \ne 2$; $a \ne 0$ if $k=m-1$.

  • (b) $(G,H,\lambda ) = (B_n,D_n.2,\sum _{i}a_i\lambda _i)$: $p \ne 2$; $a_n=1$; if $a_i,a_j \ne 0$, where $i<j<n$ and $a_k=0$ for all $i<k<j$, then $a_i+a_j \equiv i-j (\mathrm{mod} p)$; if $i<n$ is maximal such that $a_i \ne 0$, then $2a_i \equiv -2(n-i)-1 (\mathrm{mod} p)$.

  • (c) $(G,H,\lambda ) = (C_{2m},C_m^2.2,\lambda _{2m-1}+a\lambda _{2m})$: $0 \leqslant a < p$; $(m,a) \ne (1,0)$; $2a+3 \equiv 0 (\mathrm{mod} p)$.

  • (d) $(G,H,\lambda ) = (D_n,B_{n-1}, a\lambda _k+b\lambda _{n-i})$: $1 \leqslant k \leqslant n-2$; $i \in \{0,1\}$; $ab \ne 0$; $a+b+n-k-1 \equiv 0 (\mathrm{mod} p)$.

Our final result concerns chains of irreducibly acting subgroups. Let G and V be given as in Hypothesis 1.1 and write $\ell =\ell (G,V)$ for the length of the longest chain of closed positive-dimensional subgroups

$$\begin{aligned} H_{\ell }<H_{\ell -1}< \cdots< H_2 < H_1 = G \end{aligned}$$

such that $V|_{H_{\ell }}$ is irreducible. We call such a sequence of subgroups an irreducible chain. If G is an orthogonal group (or a symplectic group with $p=2$) and V is a spin module, then $\ell (G,V)$ can be arbitrarily large, and it is easy to see that the same is true if $V=W$ or $W^*$.

Theorem 1.9

Suppose G and V satisfy the conditions in Hypothesis 1.1 and assume V is not a spin module. Then either $\ell (G,V) \leqslant 5$, or G is of type $A_n$ and $\lambda \in \{\lambda _2, \lambda _3, \lambda _{n-2}, \lambda _{n-1}\}$.

The upper bound in this theorem is best possible. In fact, if we exclude the exceptional cases then either $\ell (G,V) \leqslant 4$, or $G \in \{B_4,C_4\}$, $\lambda =\lambda _3$, $p=2$ and

$$\begin{aligned} A_1^3.Z_3< A_1^3.\mathrm{Sym}_3< D_4< D_4.2 < G \end{aligned}$$

is an irreducible chain of length 5 (see Theorem 8.1 for a more precise statement). The exceptions with $G=A_n$ are genuine in the strong sense that $\ell (G,V)$ can be arbitrarily large. We refer the reader to Sect. 8 for further details.

2 Preliminaries

2.1 Notation and terminology

Most of our notation is fairly standard. As in Hypothesis 1.1, let G be a simply connected cover of a simple classical algebraic group, which is defined over an algebraically closed field K of characteristic $p \geqslant 0$. Fix a Borel subgroup $B=UT$ of G, where T is a maximal torus of G and U is the unipotent radical of B. Let $\Pi (G) =\{\alpha _1, \ldots , \alpha _{n}\}$ be the corresponding base of the root system $\Sigma (G)$ of G, where n denotes the rank of G. Let $\{\lambda _1, \ldots , \lambda _{n}\}$ be the fundamental dominant weights for T corresponding to $\Pi (G)$.

There is a bijection between the set of dominant weights of G and the set of isomorphism classes of irreducible KG-modules; if $\lambda$ is a dominant weight then we use $V_G(\lambda )$ to denote the unique irreducible KG-module with highest weight $\lambda$. We also recall that if $p>0$ then a dominant weight $\lambda = \sum _{i}a_i\lambda _i$ is p-restricted if $a_i<p$ for all i. By Steinberg’s tensor product theorem, every irreducible KG-module decomposes in a unique way as a tensor product $V_0 \otimes V_1^{\sigma _p} \otimes \cdots \otimes V_r^{\sigma _{p^r}}$, where $V_i$ is a p-restricted irreducible KG-module, $\sigma _{p^i}:G \rightarrow G$ is a standard Frobenius morphism (with $\sigma _{p^i}=1$ if $p=0$), and $V_i^{\sigma _{p^i}}$ (which we will also denote by $V_i^{(p^i)}$) is the KG-module obtained by preceding the action of G on $V_i$ by the endomorphism $\sigma _{p^i}$. It is convenient to say that every dominant weight is p-restricted if $p=0$.

In addition, $\mathrm{Lie}(G)$ denotes the Lie algebra of G, and $U_{\alpha } = \{x_{\alpha }(t) {\mid } t \in K\}$ is the root subgroup of G corresponding to a root $\alpha \in \Sigma (G)$. If $x \in G$ then $t_x : G \rightarrow G$ is the inner automorphism of G induced by conjugation by x, so $t_x(g) = xgx^{-1}$ for all $g \in G$. We write $T_i$ for an i-dimensional torus. If H is a closed positive-dimensional subgroup of G and $T_{H^0}$ is a maximal torus of $[H^0,H^0]$ contained in T, then we abuse notation by writing $\mu |_{H^0}$ to denote the restriction of a T-weight $\mu$ to the subtorus $T_{H^0}$. We define a partial order $\preccurlyeq$ on the set of weights for T, where $\mu \preccurlyeq \nu$ if and only if $\mu = \nu - \sum _{i=1}^{n}c_i\alpha _i$ for some non-negative integers $c_i$ (in this situation, we say that $\mu$ is under $\nu$). Finally, we set $\mathbb {N}_0 = \mathbb {N} \cup \{0\}$, we write $\mathrm{Sym}_{n}$ and $\mathrm{Alt}_{n}$ for the symmetric and alternating groups of degree n, and we denote a cyclic group of order m by $Z_m$ (or just m).

Recall that a map $\varphi :G_1 \rightarrow G_2$ of algebraic groups is a morphism if it is a group homomorphism that is also a morphism of the underlying varieties. In particular, it is important to note that an injective morphism does not necessarily induce an isomorphism $G_1 \cong \varphi (G_1)$ of algebraic groups. If $G_1$ is a semisimple algebraic group with root system $\Phi$, then we will say that $G_1$ is of type $\Phi$ (and we will sometimes denote this by writing $G_1 = \Phi$). For example, $\mathrm{SL}_{2}(K)$ and $\mathrm{PGL}_{2}(K)$ are both simple algebraic groups of type $A_1 = B_1 = C_1$, and $\mathrm{PSp}_{4}(K)$ and $\mathrm{SO}_{5}(K)$ are both of type $B_2=C_2$. Finally, note that if H is a closed positive-dimensional subgroup of an algebraic group G, and $\varphi :H \rightarrow G$ is the inclusion map, then the differential $d\varphi :\mathrm{Lie}(H) \rightarrow \mathrm{Lie}(G)$ is an injective Lie algebra homomorphism (since $\varphi :H \rightarrow \varphi (H)$ is an isomorphism of algebraic groups).

2.2 Diagonal embeddings

Let G / Z be a central product, where $G = G_1 \times \cdots \times G_t$ and $Z \leqslant Z(G)$. A subgroup H / Z of G / Z is a subdirect product if each of the projection maps $\pi _i:H \rightarrow G_i$ is surjective. In the context of algebraic groups, the related notion of a diagonally embedded subgroup is defined as follows:

Definition 2.1

Let H be a closed subgroup of $G = G_1\times \cdots \times G_t$ where the $G_i$ are isomorphic simply connected simple algebraic groups. We say that H is diagonally embedded in G if each projection $\pi _i: H \rightarrow G_i$ is a bijective morphism. Note that we do not require each projection map $\pi _i$ to induce an isomorphism $H \cong G_i$ of algebraic groups.

The next lemma is a well known result of Steinberg (see [18, Theorem 30] and [19, 10.13]), which describes the bijective endomorphisms of a simple algebraic group. Here $t_x$ and $\sigma _q$ are defined as in Sect. 2.1, and we adopt Steinberg’s definition of a graph automorphism of a simple algebraic group G (see [18, Sect. 10]). In particular, a graph automorphism is an isomorphism of algebraic groups unless $(G,p) = (C_2,2)$, $(G_2,3)$ or $(F_4,2)$.

Lemma 2.2

Let G be a simple algebraic group over an algebraically closed field of characteristic $p \geqslant 0$. Let $\varphi :G \rightarrow G$ be a bijective morphism. Then $\varphi = t_{x}\sigma _q\gamma ^k$ for some $x \in G$, p-power q and integer $k \in \{0,1\}$, where $\gamma$ is a graph automorphism of G. Moreover, if G is classical and $(G,p) \ne (C_2,2)$, then $\varphi$ is an isomorphism of algebraic groups if and only if $\sigma _q=1$.

Lemma 2.3

Let $\varphi :H \rightarrow G$ be a surjective morphism of algebraic groups and let $d\varphi :\mathrm{Lie}(H) \rightarrow \mathrm{Lie}(G)$ be the corresponding differential map. Then $d\varphi (\mathrm{Lie}(H))$ is a KG-submodule of $\mathrm{Lie}(G)$, and hence also an ideal of $\mathrm{Lie}(G)$.

Proof

Let $\mathrm{Ad}_G: G \rightarrow \mathrm{GL}(\mathrm{Lie}(G))$ be the adjoint representation of G. We must consider $\mathrm{Ad}_G(g)(d\varphi (X))$, for $g\in G$ and $X \in \mathrm{Lie}(H)$. As above, let $t_g:G\rightarrow G$ denote conjugation by g. Then $\mathrm{Ad}_G(g)(d\varphi (X)) = (dt_g\circ d\varphi )(X) = d(t_g\circ \varphi )(X)$. Since $\varphi$ is surjective, $g = \varphi (h)$ for some $h\in H$, so we have

$$\begin{aligned}&\mathrm{Ad}_G(g)(d\varphi (X))=d(t_{\varphi (h)}\circ \varphi )(X) = d(\varphi \circ t_h)(X)\\&\quad = d\varphi \circ \mathrm{Ad}_H(h)(X)\in d\varphi (\mathrm{Lie}(H)). \end{aligned}$$

Therefore, $d\varphi (\mathrm{Lie}(H))$ is $\mathrm{Ad}_G$-invariant and hence a KG-submodule of $\mathrm{Lie}(G)$.

Finally, let V be a KG-module with corresponding representation $\rho :G\rightarrow \mathrm{GL}(V)$, and let S be a G-invariant subspace of V. Then S is invariant under the action of $d\rho (\mathrm{Lie}(G))$. We conclude that $d\varphi (\mathrm{Lie}(H))$ is an ideal of $\mathrm{Lie}(G)$. $\square$

Recall that a morphism $\varphi :H \rightarrow G$ of algebraic groups is an isogeny if it is surjective with finite kernel. If such a map exists, we say that H is isogenous to G (this is not a symmetric relation).

Lemma 2.4

Let G be a simply connected simple classical algebraic group of rank m over an algebraically closed field K of characteristic $p \geqslant 0$, let H be a connected algebraic group and let $\varphi :H \rightarrow G$ be an isogeny. Then $\varphi$ is a bijection. Moreover, if $d\varphi \ne 0$ then either $\varphi$ is an isomorphism of algebraic groups, or one of the following holds:

• (i) G and H are both of type $A_m$, with p dividing $m+1$;

• (ii) G and H are both of type $B_m$, $C_m$ or $D_m$, with $p=2$;

• (iii) (G, H) is of type $(B_m,C_m)$ or $(C_m,B_m)$, with $p=2$.

Proof

First we claim that H is also a simple group of rank m. Clearly, if N is a proper nontrivial connected normal subgroup of H, then $\varphi (N)$ is a proper nontrivial connected normal subgroup of G, which is not possible since G is simple. Therefore, H is simple. If $T_H$ is a maximal torus of H, then $\varphi (T_H)$ is a maximal torus of G (see [1, Proposition 11.14], for example), and $\dim T_H = \dim \varphi (T_H)$. Therefore, H has rank m. Now, by comparing dimensions, we deduce that G and H have the same root system, unless $(G,H) = (B_m,C_m)$ or $(C_m,B_m)$. Note that if $p \ne 2$ and $G=B_m$ then H is also of type $B_m$ because an isogeny from $B_m$ to $C_m$ only exists when $p=2$. Similarly, if $p \ne 2$ and $G=C_m$ then H is of type $C_m$.

To see that $\varphi$ is a bijection (of abstract groups), first observe that $\ker (\varphi ) \leqslant Z(H)$ since H is simple, so the claim is trivial if $p=2$ and $H= B_m$ or $C_m$. Now assume $p \ne 2$ if $H=B_m$ or $C_m$. As above, G and H have the same root system. In particular, if $H_{\mathrm{sc}}$ denotes the simply connected group with the same root system as H, then $H_{\mathrm{sc}}$ and G are isomorphic algebraic groups (this follows from the classification of simple algebraic groups over K, using the fact that G is simply connected). Set $\psi = \varphi \circ \pi$, where $\pi :H_{\mathrm{sc}} \rightarrow H$ is the natural isogeny. Then $\psi : H_{\mathrm{sc}} \rightarrow G$ is an isogeny with kernel $L \leqslant Z(H_{\mathrm{sc}})$, so $H_{\mathrm{sc}}/L \cong G$ as abstract groups. In particular, $Z(H_{\mathrm{sc}}/L) \cong Z(G) \cong Z(H_{\mathrm{sc}})$, so $L=1$ is the only possibility. Therefore $\psi$ is injective, and thus $\varphi$ is also injective. We conclude that $\varphi$ is a bijection.

To complete the proof, we may assume that $d\varphi \ne 0$ and (G, H, p) is not one of the cases labelled (i)–(iii) in the statement of the lemma. As above, G and H are both simple groups of the same type and rank. Since $\mathrm{Lie}(H)$ is simple (see [11, Table 1]), it follows that $d\varphi$ is an isomorphism of Lie algebras and thus $\varphi$ is an isomorphism of algebraic groups. $\square$

Lemma 2.5

Let J be a closed connected subdirect product of $G_1\times G_2$, where $G_1$ and $G_2$ are isomorphic simply connected simple classical algebraic groups. Then J is diagonally embedded in $G_1 \times G_2$, and either $J=G_1 \times G_2$ or $J \cong G_1$ as algebraic groups.

Proof

Let $\pi _i:J \rightarrow G_i$ be the i-th projection map and set $L = \ker (d\pi _1) \cap \ker (d\pi _2)$. Note that $L = 0$ since J is a closed positive-dimensional subgroup of $G_1 \times G_2$. Without loss of generality, we will assume that $d\pi _1 \ne 0$.

First assume $\ker (\pi _1)$ is infinite. Since $\pi _2$ is injective on $\ker (\pi _1)$, we have $\dim \ker (\pi _1) = \dim \pi _2(\ker (\pi _1))$. Moreover, the surjectivity of $\pi _2$ implies that $\pi _2(\ker (\pi _1))$ is an infinite normal subgroup of $G_2$, so the simplicity of $G_2$ implies that $\pi _2(\ker (\pi _1)) = G_2$ and thus $\dim \ker (\pi _1) = \dim G_2$. Therefore, $\dim J = \dim G_1+ \dim G_2$ and we conclude that $J=G_1\times G_2$.

For the remainder, we may assume that $\ker (\pi _1)$ is finite, so $\pi _1$ is an isogeny and Lemma 2.4 implies that $\pi _1$ is a bijection and either $J\cong G_1$, or $(G_1,J)$ is one of the cases labelled (i)–(iii). In particular, J is simple and $\ker (\pi _2)$ is finite. By a further application of Lemma 2.4, we see that $\pi _2$ is also a bijection and thus J is diagonally embedded. To complete the proof it remains to show that $J \cong G_1$ as algebraic groups. Seeking a contradiction, let us assume that $\ker (d\pi _i) \ne 0$ for $i = 1,2$, so $\mathrm{im}(d\pi _i)$ is a proper non-zero ideal of $\mathrm{Lie}(G_i)$ (see Lemma 2.3).

First assume $p=2$ and $(G_1,J)$ is of type $(B_m,B_m)$, $(C_m,C_m)$, $(B_m,C_m)$ or $(C_m,B_m)$. For $m \geqslant 2$, the ideal structure of $\mathrm{Lie}(J)$ is described in [6, Sect. 5]. Excluding the case where $J = C_m$ is simply connected and $m \geqslant 3$ is odd, we observe that $\mathrm{Lie}(J)$ has an irreducible socle S (as a KJ-module), which immediately implies that L contains S. This is a contradiction, since $L=0$. Now assume $J = C_m$ is simply connected and $m \geqslant 3$ is odd. If $G_1=C_m$ then $J \cong G_1$ since $G_1$ is simply connected, so let us assume $G_1=B_m$. The socle of $\mathrm{Lie}(J)$ is of the form $Z \oplus M$, where $Z=Z(\mathrm{Lie}(J))$ is 1-dimensional and M is a nontrivial irreducible module. Without loss of generality, we may assume that M is not contained in $\ker (d\pi _1)$, which implies that $\ker (d\pi _1) = Z$. Therefore $\mathrm{im}(d\pi _1)$ is an ideal of $\mathrm{Lie}(G_1)$ of codimension 1, but this is not compatible with the ideal structure described in [6]. Therefore, once again we have reached a contradiction. Finally, if $J = B_1=C_1$ is adjoint then $\mathrm{Lie}(J)$ has an irreducible socle and we can repeat the argument given above.

Next suppose that $G_1$ and J are both of type $A_m$, where p divides $m+1$. We may assume that $m \geqslant 2$. Seeking a contradiction, suppose that J is not simply connected. By inspecting [11, Table 1] we deduce that $\mathrm{im}(d\pi _i) = Z(\mathrm{Lie}(G_i))$ and $\mathrm{ker}(d\pi _i)$ is the commutator subalgebra of $\mathrm{Lie}(J)$ for $i=1,2$. But this implies that $L\ne 0$, which is a contradiction. To complete the proof, we may assume that $G_1$ and J are both of type $D_m$, with $p=2$. If m is odd then we can repeat the previous argument, using [11, Table 1], so let us assume m is even. If J is not simply connected then $\mathrm{Lie}(J)$ has an irreducible socle S (see [6, Sect. 5]), which must be contained in L. Once again, this is a contradiction. $\square$

The next result is a natural generalization of Lemma 2.5.

Proposition 2.6

Let J be a closed connected subdirect product of $G_1\times \dots \times G_t$, where the $G_i$ are isomorphic simply connected simple classical algebraic groups. Then the following hold:

• (i) J is semisimple.

• (ii) There exists a positive integer $r\leqslant t$ such that $J=J_1\cdots J_r$, where each $J_i$ is isomorphic to $G_1$.

• (iii) There exist integers $0=t_0< t_1< t_2< \dots < t_r = t$ such that $J_{i}$ is diagonally embedded in $G_{t_{i-1}+1}\times \cdots \times G_{t_i}$.

Proof

We use induction on t, noting that the case $t=1$ is trivial, and Lemma 2.5 handles the case $t=2$. Let us assume $t \geqslant 3$, and let $\pi _i:J \rightarrow G_i$ be the i-th projection map. As in the proof of the previous lemma, we may assume that $d\pi _1 \ne 0$.

If $\ker (\pi _i)$ is finite for any i, then $\dim J = \dim G_1$ and thus $\ker (\pi _1)$ is also finite. Then by arguing as in the proof of Lemma 2.5, we deduce that J is diagonally embedded in $G_1 \times \cdots \times G_t$ and $J \cong G_1$. For the remainder, we may assume that $\ker (\pi _i)$ is infinite for all i. Since $\pi _i(R_u(J))$ is a proper normal subgroup of $\pi _i(J) = G_i$ for each i, it follows that J is reductive. Similarly, by considering $\pi _i(Z(J))$, we deduce that Z(J) is finite and thus J is semisimple. In particular, we may write

$$\begin{aligned} J = J_1 \cdots J_r, \end{aligned}$$

where each $J_i$ is simple. Note that $\pi _i(J_j)$ is a connected normal subgroup of $G_i$ for all i, j, so $\pi _i(J_j)=1$ or $G_i$. Let $\sigma$ be the projection map

$$\begin{aligned} \sigma :J\rightarrow G_2\times \cdots \times G_t. \end{aligned}$$

We now consider two cases.

Case 1. $\ker (\sigma )$ is infinite.

First assume $\ker (\sigma )$ is infinite, so $\ker (\sigma )^0$ is a connected positive-dimensional normal subgroup of J. By relabelling the $J_i$, if necessary, we may assume that

$$\begin{aligned} \ker (\sigma )^0 = J_1 \cdots J_a \end{aligned}$$

for some $a \in \{1, \ldots , r\}$ (see [13, Theorem 27.5(c)], for example). Now $\pi _1(J_i) = G_1$ for all $1 \leqslant i \leqslant a$, so the injectivity of $\pi _1$ on $\ker (\sigma )^0$, together with the simplicity of $G_1$, implies that $a=1$ and $J_1$ is of type $G_1$. In particular, $r>1$ since we are assuming that $\ker (\pi _1)$ is infinite. Also note that $\pi _i(J_2 \cdots J_r) = G_i$ for all $i \geqslant 2$.

We claim that $\pi _1(J_i)=1$ for all $2 \leqslant i \leqslant r$, so $J_2 \cdots J_r \leqslant G_2 \times \cdots \times G_t$ is a subdirect product and the result follows by induction. To justify the claim, let $i \in \{2, \ldots , r\}$ and consider $\pi _1|_{J_1J_i}: J_1J_i \rightarrow G_1$. Since $\pi _1(J_1)=G_1$, we have $\pi _1(J_1J_i)=G_1$. Now $\ker (\pi _1|_{J_1J_i})^0$ is a connected normal subgroup of $J_1J_i$, so $\ker (\pi _1|_{J_1J_i})^0=1, J_1, J_i$ or $J_1J_i$. It is easy to see that $\ker (\pi _1|_{J_1J_i})^0=J_i$ is the only possibility, so $\pi _1(J_i)=1$ as claimed.

Case 2. $\ker (\sigma )$ is finite.

To complete the proof, we may assume that $\ker (\sigma )$ is finite. Now $J/\ker (\sigma )$ is connected and reductive, and it is isomorphic to a subdirect product of $G_2 \times \cdots \times G_t$. By induction, there exists $s \in \{1, \ldots , t-1\}$ such that $J/\ker (\sigma ) = L_1\cdots L_{s}$ and $L_i \cong G_1$ for each i. But $J=J_1 \cdots J_r$ and the $J_i$ are simple, so $r=s$ and $\dim J_i = \dim G_1$ for all i (indeed, $J_i$ is isogenous to $G_1$), hence $\dim J= r \dim G_1$. Note that $r>1$ since $\ker (\pi _1)$ is infinite.

Since $\pi _1: J \rightarrow G_1$ is surjective, we have $\dim \ker (\pi _1) = (r-1)\dim G_1$. Moreover, by relabelling the $J_i$ if necessary, we may assume that $\ker (\pi _1)^0= J_2\cdots J_r$ (see [13, Theorem 27.5(c)]). Therefore, $\pi _1(J_1) = G_1$ and $J_2 \cdots J_r \leqslant G_2\times \cdots \times G_t$. By a further relabelling, we may assume that there exists an integer $b \in \{1, \ldots , t\}$ such that $\pi _i(J_1)=G_i$ for $1\leqslant i \leqslant b$, and $\pi _i(J_1)=1$ if $b<i\leqslant t$.

First we claim that $b<t$. Seeking a contradiction, suppose that $b=t$, so $\pi _i(J_1)=G_i$ for all i. Let $j \in \{2, \ldots , r\}$ and consider $\pi _i|_{J_1J_j}: J_1J_j \rightarrow G_i$. By arguing as above, we deduce that $\ker (\pi _i|_{J_1J_j})^0=J_j$, so $\pi _i(J_j) = 1$ for all i. Therefore $J_j=1$ and thus $r=1$, which is a contradiction.

Since $b<t$ and $J_1 \leqslant G_1 \times \cdots \times G_b$ is a subdirect product, by induction we deduce that $J_1$ is diagonally embedded in $G_1 \times \cdots \times G_b$ and $J_1 \cong G_1$. If we fix $i \in \{1, \ldots , b\}$ and $j \in \{2, \ldots , r\}$, then $\ker (\pi _i|_{J_1J_j})^0=J_j$ and thus $J_2 \cdots J_r \leqslant \ker (\pi _i)$. Therefore, $J_2 \cdots J_r \leqslant G_{b+1} \times \cdots \times G_t$ is a subdirect product (since $\pi _i(J)=G_i$ and $\pi _i(J_1) = 1$ for all $i>b$) and the result follows by induction. $\square$

2.3 Irreducible triples

Define G, H and V as in Hypothesis 1.1. The next result records a basic observation (see [4, Remark 2]).

Lemma 2.7

If $V|_{H}$ is irreducible, then H does not normalize a nontrivial connected unipotent subgroup of G. In particular, $H^0$ is reductive.

Suppose $V|_{H}$ is irreducible, but $V|_{H^0}$ is reducible. Then Clifford theory implies that

$$\begin{aligned} V|_{H^0}=V_1 \oplus \cdots \oplus V_m, \end{aligned}$$

(1)

where $m \geqslant 2$ and the $V_i$ are irreducible $KH^0$-modules that are transitively permuted under the induced action of $H/H^0$.

Remark 2.8

Since the irreducibility of $V|_{H}$ implies that $H^0$ is reductive, we have $H^0 = JZ(H^0)$ where $J = [H^0,H^0]$ is the derived subgroup of $H^0$. Now $Z(H^0)$ acts as scalars on the $V_i$ so that they are also irreducible upon restriction to J. In particular, the irreducibility of $V|_{H}$ implies that the KJ-composition factors of $V|_{J}$ are transitively permuted under the induced action of $H/H^0$.

If the $V_i$ in (1) are isomorphic as $KH^0$-modules, then $V|_{H^0}$ is said to be homogeneous. For example, $V|_{H^0}$ is homogeneous if $N_G(H^0) = H^0C_G(H^0)$. The following result is [3, Proposition 2.6.2].

Proposition 2.9

If H is a cyclic extension of $H^0$, then the irreducible $KH^0$-modules $V_i$ in (1) are pairwise non-isomorphic. In particular, $V|_{H^0}$ is not homogeneous.

We will also need the following lemma.

Lemma 2.10

Let $V_1$ and $V_2$ be p-restricted irreducible KG-modules and set $V= V_1 \otimes V_2$. Then one of the following holds:

• (i) V is irreducible, $G = B_n$ or $C_n$, $p = 2$ and $V_1$ and $V_2$ can be arranged so that $V_i = V_G(\mu _i)$ and $\mu _1$ (respectively $\mu _2$) has support on the short (respectively, long) roots;

• (ii) V has non-isomorphic composition factors.

Proof

By [17, (1.6)], V is irreducible if and only if G and V satisfy the conditions in (i), so let us assume V is reducible. Write $V_i = V_G(\mu _i)$ and note that V has a composition factor of highest weight $\mu = \mu _1+\mu _2$ occurring with multiplicity 1. Then any other composition factor has highest weight $\nu \ne \mu$ and the result follows. $\square$

3 Subgroup structure

3.1 A reduction theorem

Let G be a simple classical algebraic group with natural module W. Following [14, Sect. 1], we introduce six natural, or geometric, collections of closed subgroups, labelled $\mathcal {C}_i$ for $1 \leqslant i \leqslant 6$, and we set $\mathcal {C}=\bigcup _{i}\mathcal {C}_i$. These subgroups are defined in terms of the underlying geometry of W, and a rough description of the subgroups in each $\mathcal {C}_i$ collection is given in Table 3 (note that the subgroups in the collection $\mathcal {C}_5$ are finite). There are two types of subgroups in the $\mathcal {C}_4$ collection (indicated by the two rows in Table 3); following [4], we write $\mathcal {C}_4 = \mathcal {C}_4(i) \cup \mathcal {C}_4(ii)$ accordingly. The following result is [14, Theorem 1] (we use the term non-geometric for the subgroups arising in part (ii)).

Table 3 The $\mathcal {C}_i$ collections

Theorem 3.1

Let G be a simple classical algebraic group with natural module W, and let H be a closed subgroup of G. Then one of the following holds:

• (i) H is contained in a member of $\mathcal {C}$;

• (ii) modulo scalars, H is almost simple and E(H) (the unique quasisimple normal subgroup of H) is irreducible on W. Further, if $G=\mathrm{SL}(W)$ then E(H) does not fix a non-degenerate form on W. In addition, if H is infinite then $E(H)=H^0$ is tensor indecomposable on W.

3.2 Geometric subgroups of $\mathrm{GO}(W)$

In our inductive proof of Theorem 1.4, we will need to consider the subgroup structure of $G = \mathrm{GO}(W)$, which is the full isometry group of a non-degenerate quadratic form on W. Here $\dim W = 2n \geqslant 6$ and thus $G^0 = \mathrm{SO}(W)$ is a simple group of type $D_n$. The notion of a geometric subgroup extends naturally to G and we can define the subgroup collections $\mathcal {C}_1, \ldots , \mathcal {C}_6$ as above. It is straightforward to check that the proof of the main theorem of [14] extends to this slightly more general situation (see [14, Theorem 1$'$]), and thus Theorem 3.1 holds. In particular, any subgroup of G that is not contained in a geometric subgroup is said to be non-geometric, and these subgroups satisfy the conditions described in part (ii) of Theorem 3.1.

In the proofs of Propositions 5.10 and 5.19, we need information on the maximal non-parabolic geometric subgroups of G. This is given in the following proposition.

Proposition 3.2

Let M be a positive-dimensional non-parabolic geometric subgroup of $G=\mathrm{GO}(W) = D_n.2$, which is not contained in $G^0$. Then the possibilities for M are recorded in Table 4.

Table 4 The non-parabolic geometric subgroups $M<D_n.2$ with $M \not \leqslant D_n$

Proof

Here M is a disconnected $\mathcal {C}_i$-subgroup of G, where $i \in \{1,2,3,4\}$ (recall that the subgroups in $\mathcal {C}_5$ are finite, and there are no $\mathcal {C}_6$-subgroups in orthogonal groups). The structure of M is easily determined from its geometric description (see [4, Sect. 2.5], for example), and it is straightforward to determine whether or not M is contained in $G^0$.

For example, if $M \in \mathcal {C}_1$ then $M = G_U$ is the stabilizer of a subspace U of W (the natural KG-module), and one of the following holds (recall that M is non-parabolic, so U is not totally singular):

• (a) U is non-degenerate and $\dim U$ is even;

• (b) U is non-degenerate, $\dim U$ is odd and $p \ne 2$;

• (c) U is non-singular, $\dim U = 1$ and $p=2$.

In (a) and (b), $M = \mathrm{GO}(U) \times \mathrm{GO}(U^{\perp })$ is not contained in $G^0$. Similarly, in (c), $M = B_{n-1} \times 2$ (up to isomorphism) is not in $G^0$. These are the cases labelled (i), (ii) and (iii) in Table 4.

Next suppose M is a $\mathcal {C}_4(i)$ tensor product subgroup of type $\mathrm{SO}(W_1) \otimes \mathrm{SO}(W_2)$, where $W= W_1 \otimes W_2$, $\dim W_i=a_i$, $a_1 \ne a_2$ and $p \ne 2$. Note that

$$\begin{aligned} M = \mathrm{GO}(W_1) \circ \mathrm{GO}(W_2) = (\mathrm{SO}(W_1) \circ \mathrm{SO}(W_2)).\langle x_1, x_2 \rangle \end{aligned}$$

is a central product and the $x_i$ are certain involutions. More precisely, if $a_1$ and $a_2$ are both even, then we may assume that $x_1$ acts as a reflection on $W_1$ and centralizes $W_2$ (and vice versa for $x_2$). Therefore, $x_1, x_2 \in \mathrm{SO}(W)$ and thus $M<G^0$. On the other hand, if $a_1$ is odd (so $a_2$ is even) then we can choose $x_1$ so that it acts as $-1$ on $W_1$ and centralizes $W_2$, and $x_2$ is defined as above. Here $x_1 \in G^0$ but $x_2 \not \in G^0$, whence M is not contained in $G^0$. This is the case labelled (vii) in Table 4.

The other cases are similar. For instance, suppose M is a $\mathcal {C}_3$-subgroup. Geometrically, M is the stabilizer of a decomposition $W = U_1 \oplus U_2$, where $U_1$ and $U_2$ are maximal totally singular subspaces of W, so $\dim U_1=n$ and $M = \mathrm{GL}(U_1).2$. Now $G^0$ contains an element interchanging $U_1$ and $U_2$ if and only if n is even, so M is contained in $G^0$ if and only if n is even, and this explains the parity condition on n recorded in Table 4 (see case (vi)). $\square$

Proposition 3.3

Let M be one of the subgroups of $G=D_n.2$ listed in Table 4, and assume $n \geqslant 3$ and $p \ne 2$. Set $V=V_{G^0}(\lambda )$, where one of the following holds:

• (a) $\lambda = \lambda _{n-1}+\lambda _{n}$;

• (b) $\lambda = \lambda _{k}$, where $1<k<n-1$.

Then V extends to a representation of G, and $V|_{M}$ is reducible.

Proof

First observe that $\lambda$ is fixed under the induced action of an involutory graph automorphism of $G^0$ on the set of T-weights of G (where T is a maximal torus of $G^0$), so the representation $V=V_{G^0}(\lambda )$ does indeed extend to a representation of $G=G^0.2$. We will deal in turn with each of the relevant cases in Table 4 (note that case (iii) is not applicable, since we are assuming that $p \ne 2$). Seeking a contradiction, let us assume that $V|_{M}$ is irreducible. By Clifford theory (see Sect. 2.3), the $KM^0$-composition factors of V are transitively permuted under the induced action of $M/M^0$.

Case 1. M is a $\mathcal {C}_1$-subgroup of type $D_lD_{n-l}.2^2$.

Here $1 \leqslant l < n/2$ and $M^0=M_1M_2$, where $M_1 = D_l$ and $M_2 = D_{n-l}$. We will inspect the proof of [4, Lemma 3.2.3]. By [17], $V|_{M^0}$ is reducible and thus Clifford theory implies that there are either two or four $KM^0$-composition factors (since $|M:M^0|=4$).

First assume $l=1$, so $M = M^0\langle \tau _1,\tau _2\rangle$ where $\tau _1$ is an involution inverting the 1-dimensional torus $M_1$, and $\tau _2$ is an involutory graph automorphism of $M_2 = D_{n-1}$. We may assume that $M_2 = \langle U_{\pm \alpha _2}, \ldots , U_{\pm \alpha _n}\rangle$. Note that $M_1$ acts as scalars on the $KM^0$-composition factors of V, each of which is an irreducible $KM_2$-module. If there are exactly two $KM^0$-composition factors of V then the argument in the proof of [4, Lemma 3.2.3] goes through unchanged (the details are given in the proof of [4, Lemma 3.2.2]), and the result follows immediately.

Similar reasoning applies if there are four composition factors. By Clifford theory, if $\nu$ is the highest weight of a $KM^0$-composition factor, then $\nu |_{M_2} = \lambda |_{M_2}$ or $(\tau _2 \cdot \lambda )|_{M_2}$ (here $\nu |_{M_2}$ denotes the restriction of $\nu$ to a suitable maximal torus of $M_2$ contained in T, and similarly for $\lambda |_{M_2}$ and $(\tau _2 \cdot \lambda )|_{M_2}$). However, $\mu = \lambda -\alpha _1-\alpha _2-\cdots -\alpha _{n-1} = -\lambda _1+2\lambda _{n}$ affords the highest weight of a $KM^0$-composition factor in case (a), but clearly $\mu |_{M_2}$ is not conjugate to $\lambda |_{M_2}$. Case (b) is entirely similar, using $\mu = \lambda -\alpha _1-\cdots -\alpha _k$. Therefore, in both cases we have reached a contradiction.

Now assume $l \geqslant 2$. As noted in the proof of [4, Lemma 3.2.3], up to conjugacy we have

$$\begin{aligned} \begin{aligned} M_{1}&=\langle U_{\pm \alpha _{1}},\ldots ,U_{\pm \alpha _{l-1}},U_{\pm (\alpha _{l-1}+2(\alpha _l+\cdots +\alpha _{n-2})+\alpha _{n-1}+\alpha _{n})} \rangle \\ M_{2}&=\langle U_{\pm \alpha _{l+1}},\ldots ,U_{\pm \alpha _{n}} \rangle \end{aligned} \end{aligned}$$

(2)

and $M=M^0\langle \tau _1,\tau _2 \rangle$, where $\tau _1$ and $\tau _2$ act as involutory graph automorphisms on $M_1$ and $M_2$, respectively. Let $\{\omega _{1,1},\ldots ,\omega _{1,l}\}$ and $\{\omega _{2,1},\ldots ,\omega _{2,n-l}\}$ be the fundamental dominant weights corresponding to the above bases of the root systems of $M_1$ and $M_2$, respectively (here $\tau _1$ acts as a transposition on $\{\omega _{1,1},\ldots ,\omega _{1,l}\}$, interchanging $\omega _{1,l-1}$ and $\omega _{1,l}$, and similarly $\tau _2$ acts on $\{\omega _{2,1},\ldots ,\omega _{2,n-l}\}$ by interchanging the weights $\omega _{2,n-l-1}$ and $\omega _{2,n-l}$). Note that if $\mu = \sum _{i=1}^{n}{b_i\lambda _i}$ is a weight for T then

$$\begin{aligned} \begin{aligned} \mu |_{M^0} =&\sum _{i=1}^{l-1} b_{i}\omega _{1,i}+(b_{l-1}+2b_{l} +\dots +2b_{n-2}+b_{n-1}+b_{n})\omega _{1,l} \\&+\sum _{i=1}^{n-l} b_{l+i}\omega _{2,i}. \end{aligned} \end{aligned}$$

(3)

Consider case (a). Here $\mu = \lambda - \alpha _{l}-\alpha _{l+1}-\cdots - \alpha _{n-2}-\alpha _{n-1}$ affords the highest weight of a $KM^0$-composition factor (see the proof of [4, Lemma 3.2.3]) and we calculate that

$$\begin{aligned} \mu |_{M^0} = \omega _{1,l-1}+\omega _{1,l}+2\omega _{2,n-l},\quad \lambda |_{M^0} = 2\omega _{1,l}+\omega _{2,n-l-1}+\omega _{2,n-l}. \end{aligned}$$

In particular, we observe that

$$\begin{aligned} \mu |_{M^0} \not \in \{\lambda |_{M^0},(\tau _1 \cdot \lambda )|_{M^0},(\tau _2 \cdot \lambda )|_{M^0}, (\tau _1\tau _2 \cdot \lambda )|_{M^0}\}, \end{aligned}$$

so $\mu |_{M^0}$ is not M-conjugate to $\lambda |_{M^0}$. This is a contradiction.

In case (b) we have $\lambda =\lambda _k$, where $1<k<n-1$, and thus

$$\begin{aligned} \lambda |_{M^0} = \left\{ \begin{array}{ll} \omega _{1,k} &{} \quad 1<k<l-1 \\ \omega _{1,l-1}+\omega _{1,l} &{}\quad k=l-1 \\ 2\omega _{1,l} &{}\quad k=l \\ 2\omega _{1,l}+\omega _{2,k-l} &{}\quad l<k<n-1. \end{array}\right. \end{aligned}$$

Set

$$\begin{aligned} \mu = \left\{ \begin{array}{ll} \lambda -\alpha _{k} - \alpha _{k+1} - \cdots - \alpha _{l} &{}\quad 1<k<l \\ \lambda -\alpha _{l} &{}\quad k=l \\ \lambda -\alpha _{l} - \alpha _{l+1} - \cdots - \alpha _{k} &{}\quad l<k<n-1. \end{array}\right. \end{aligned}$$

Then $\mu$ affords the highest weight of a $KM^0$-composition factor, and in each case it is easy to check that $\mu |_{M^0}$ is not M-conjugate to $\lambda |_{M^0}$. For example, suppose $1<k<l-1$. Then $\mu = \lambda _{k-1}-\lambda _{l}+\lambda _{l+1}$ and thus $\mu |_{M^0} = \omega _{1,k-1}+\omega _{2,1}$, which is not conjugate to $\lambda |_{M^0} = \omega _{1,k}$. The other cases are similar.

Case 2. M is a $\mathcal {C}_1$-subgroup of type $B_lB_{n-l-1} \times 2^2$.

Here $0\leqslant l<n/2$ and $M^0=M_1M_2$, where $M_1=B_{l}$ and $M_2=B_{n-l-1}$. Let $\{\beta _1,\ldots ,\beta _l\}$ and $\{\gamma _1,\ldots ,\gamma _{n-l-1}\}$ be bases of the root systems of $M_1$ and $M_2$, respectively, and let $\{\eta _1,\dots ,\eta _l\}$ and $\{\nu _1,\dots ,\nu _{n-l-1}\}$ be the corresponding fundamental dominant weights. Then up to conjugacy, we may assume that the simple root elements of $M_1$ and $M_2$ are as follows

$$\begin{aligned} x_{\beta _i}(t) = \left\{ \begin{array}{ll} x_{\alpha _i}(t) &{}\quad 1\leqslant i<l \\ x_{\delta }(t)x_{\epsilon }(t) &{}\quad i = l \end{array}\right. \end{aligned}$$

$$\begin{aligned} x_{\gamma _j}(t) = \left\{ \begin{array}{ll} x_{\alpha _{l+j}}(t) &{}\quad 1\leqslant j< n-l-1 \\ x_{\alpha _{n-1}}(t)x_{\alpha _n}(t) &{}\quad j = n-l-1 \end{array}\right. \end{aligned}$$

for all $t \in K$, where $\delta = \alpha _l+\alpha _{l+1}+\cdots +\alpha _{n-1}$ and $\epsilon = \alpha _l+\alpha _{l+1}+\cdots +\alpha _{n-2}+\alpha _n$ (see [21, Claim 8], for example). Note that $V|_{M^0}$ is homogeneous.

First consider (a). In terms of the above notation, we calculate that $\lambda |_{M^0}=2\eta _l+2\nu _{n-l-1}$. By considering the restrictions $\alpha _i|_{M^0}$, we see that $\lambda -\alpha _{n-1}$ and $\lambda -\alpha _n$ both restrict to the weight $\lambda |_{M^0}-\gamma _{n-l-1}$. This weight has multiplicity 1 in the $KM^0$-composition factor of V afforded by $\lambda$. Moreover, one checks that $\lambda$ is the only T-weight $\mu$ in V such that $\lambda |_{M^0}-\gamma _{n-l-1} \preccurlyeq \mu |_{M^0}$ and $\lambda |_{M^0}-\gamma _{n-l-1} \ne \mu |_{M^0}$, so there must be a $KM^0$-composition factor with highest weight $\lambda |_{M^0}-\gamma _{n-l-1}$. However, this contradicts the homogeneity of $V|_{M^0}$.

Now consider (b). Suppose $k\leqslant l$, so $l>0$ and $\lambda |_{M_2}$ is trivial. Now the weight $\mu = \lambda -\alpha _k-\cdots -\alpha _l$ affords the highest weight of a $KM^0$-composition factor of V, but $\mu |_{M_2}$ is nontrivial and this contradicts the homogeneity of $V|_{M^0}$. Now assume $k>l$. Here $\lambda |_{M^0}=2\eta _l+\nu _{k-l+1}$. However, the weight $\mu = \lambda -\alpha _l-\alpha _{l+1}-\cdots -\alpha _k$ affords the highest weight of a $KM^0$-composition factor of V and

$$\begin{aligned} \mu |_{M^0} = \left\{ \begin{array}{ll} \eta _{l-1}+\nu _{k-l+2} &{}\quad k<n-2 \\ \eta _{l-1}+2\nu _{n-l-1} &{}\quad k=n-2 \end{array}\right. \end{aligned}$$

Once again, this contradicts the homogeneity of $V|_{M^0}$.

Case 3. M is a $\mathcal {C}_2$-subgroup of type $(2^t \times B_l^t).\mathrm{Sym}_t$.

Here $2n=(2l+1)t$, $l \geqslant 1$ and $t \geqslant 2$ is even. Note that the conclusion to [4, Lemma 4.2.1] still applies in this situation.

First consider (a). If $l=1$ then the argument in the third paragraph on [4, p. 48] applies, and we reach a contradiction via [4, Lemma 4.2.1]. Next suppose $(l,t)=(2,2)$, so $n=5$. Here we argue as in the third to last paragraph on [4, p. 50]. (Alternatively, in the notation of [4, Lemma 4.3.8], note that $\dim V = \dim V_{D_5}(\lambda ) = 210$ and $\lambda |_{M^0} = 2\omega _{1,2}+2\omega _{2,2}$, so the $KM^0$-composition factor afforded by $\lambda$ has dimension $10^2=100$, which does not divide $\dim V$.)  Finally, if $l \geqslant 2$ and $(l,t) \ne (2,2)$ then we can argue as in the third to last paragraph on [4, p. 52] (again, we get a contradiction via [4, Lemma 4.2.1]). Case (b) is similar and we omit the details.

Case 4. M is a $\mathcal {C}_2$-subgroup of type $(D_l^t.2^t).\mathrm{Sym}_t$.

Here $n=lt$, $l \geqslant 1$ and $t \geqslant 2$. As in the previous case, note that the conclusion to [4, Lemma 4.2.1] still applies.

If $l=1$ then $M = N_{G}(T)$ is the normalizer of a maximal torus and this case is ruled out as in the first paragraph in the proof of [4, Lemma 4.3.8]. Now assume $l \geqslant 2$. Consider case (a). If $l \geqslant 3$ then we can argue as on [4, p. 56] to rule out this case, and as noted in the penultimate paragraph on [4, p. 57], the same argument also applies if $l=2$. Case (b) is very similar: if $l \geqslant 3$ then we argue as on [4, p. 55] (we repeatedly apply [4, Lemma 4.2.1]), and for $l=2$ we note that the argument on [4, p. 57] goes through unchanged.

Case 5. M is a $\mathcal {C}_3$-subgroup of type $A_{n-1}T_1.2$.

Here $n \geqslant 3$ is odd. Set $L = (M^0)'=A_{n-1}$ and note that $V|_{L}$ has exactly two composition factors. We may assume that $L = \langle U_{\pm \alpha _1}, \ldots , U_{\pm \alpha _{n-1}}\rangle$. As in the proof of [4, Lemma 3.2.2], let $V_j$ be the sum of the T-weight spaces (T a maximal torus of $G^0=D_n$) in $V=V_{G^0}(\lambda )$ of the form $\lambda - \sum _{i=1}^{n-1}c_i\alpha _i - j \alpha _{n}$, $j \in \mathbb {N}_0$. Since $V_j$ is L-stable, every T-weight of V is of this form, with $j=0$ or 1 (by [16, Theorem 1] and saturation; see [12, Sect. 13.4]). In particular, $w_0\lambda = -(\tau \cdot \lambda )$ is of this form (where $w_0$ is the longest word in the Weyl group of $G^0$, and $\tau$ is an involutory graph automorphism of $G^0$ that interchanges the weights $\lambda _{n-1}$ and $\lambda _n$). Therefore, if we write $\lambda = \sum _{i=1}^{n}a_i\lambda _i$ then

$$\begin{aligned} 2\lambda - (a_{n-1}-a_n)(\lambda _{n-1}-\lambda _n) = \sum _{i=1}^{n}c_i\alpha _i \end{aligned}$$

and $c_n \in \{0,1\}$. By expressing the $\lambda _i$ in terms of the $\alpha _i$, we deduce that either $\lambda =\lambda _1$, or $n=3$ and $\lambda = \lambda _2$ or $\lambda _3$. This immediately eliminates cases (a) and (b).

Case 6. M is a $\mathcal {C}_4(i)$-subgroup of type $B_aD_b.2$.

Here $n = (2a+1)b$ and $M^0=M_1M_2$ is semisimple, where $M_1$ is of type $B_a$ and $M_2$ is of type $D_b$. Note that the embedding of M in G is via a tensor product action on the natural KG-module W. Write $M=M^0\langle \sigma \rangle$, where $\sigma$ induces a graph automorphism on $M_2$ and centralizes $M_1$.

Let $\Pi (M_1)=\{\beta _1, \ldots ,\beta _a\}$ and $\Pi (M_2)=\{\gamma _1,\ldots ,\gamma _b\}$ be bases of the root systems $\Sigma (M_1)$ and $\Sigma (M_2)$, respectively. Now W restricts to $M_1$ as 2b copies of the natural module for $M_1$, and hence up to conjugacy, we may assume that $M_1$ lies in the subgroup

$$\begin{aligned} \langle U_{\pm \alpha _i}{\mid } (2a+1)j+1\leqslant i< (2a+1)(j+1), \, 0\leqslant j < b \rangle , \end{aligned}$$

the derived subgroup of an $A_{2a}\times \cdots \times A_{2a}$ (b copies) Levi subgroup of G. The projection of $M_1$ into each of the factors of this group is the natural embedding of a group of type $B_a$ in $A_{2a}$. We may assume that

$$\begin{aligned} \alpha _{(2a+1)j+i}|_{M^0}=\beta _i \quad \text{ for } \text{ all }~0\leqslant j < b, 1\leqslant i\leqslant a \end{aligned}$$

and

$$\begin{aligned} \alpha _{(2a+1)j+a+i}|_{M^0}=\beta _{a-i+1} \quad \text{ for } \text{ all }~0\leqslant j< b, 1\leqslant i\leqslant a, \end{aligned}$$

where $\alpha |_{M^0}$ denotes the restriction of $\alpha$ to a maximal torus $T_{M^0} < T$ of $M^0$.

Let P be the parabolic subgroup of $M^0$, which contains the opposite Borel subgroup, with Levi factor $M_1T_{M^0}$. We may assume that P is contained in the parabolic subgroup of G, which contains the opposite Borel subgroup of G, whose Levi factor has derived subgroup as given above. By comparing the flags of commutator subspaces of W with respect to the two unipotent radicals, we are able to determine the restrictions of sufficiently many T-weights to $T_{M^0}$ in order to deduce the restrictions of the remaining simple roots. We get

$$\begin{aligned} \alpha _{(2a+1)j}|_{M^0}=\gamma _j-\beta _0,\quad \alpha _{(2a+1)b}|_{M^0}= \gamma _b-\gamma _{b-1}-(\beta _0-\beta _1), \end{aligned}$$

where $1\leqslant j< b$ and $\beta _0=2\sum _{i=1}^a \beta _i$.

By [17], $V|_{M^0}$ is reducible, so Clifford theory implies that V has precisely two $KM^0$-composition factors, with highest weights $\lambda |_{M^0}$ and $(\sigma \cdot \lambda )|_{M^0}$. In particular, if we set $T_{M_1}=T_{M^0} \cap M_1$, then every $KM_1$-composition factor of V has highest weight $\lambda |_{M_1}$, whence every $T_{M_1}$-weight of V is of the form

$$\begin{aligned} \lambda |_{M_1}-\sum _{j=1}^{a}n_j\beta _j,\quad \text{ for } \text{ some }~n_j\in \mathbb {N}_0. \end{aligned}$$

(4)

However, in case (a) we find that the weight $\lambda -\alpha _n$ restricts to $\lambda |_{M_1} + \beta _0-\beta _1$, which contradicts (4). In (b), choose $1\leqslant i<b$ such that $|(2a+1)i-k|$ is minimal, and set $\mu = \lambda -a_k-\alpha _{k+1}-\cdots -\alpha _{(2a+1)i}$ if $k\leqslant (2a+1)i$, otherwise set $\mu = \lambda -\alpha _{(2a+1)i}-\alpha _{(2a+1)i+1}-\cdots -\alpha _k$. Then

$$\begin{aligned} \mu |_{M_1} = (\lambda - r-(\gamma _i-\beta _0))| _{M_1}= (\lambda - r+\beta _0)|_{M_1}, \end{aligned}$$

where either $r=0$ or r is a positive root of $M_1$. Once again, this contradicts (4).

This completes the proof of Proposition 3.3. $\square$

Proposition 3.4

Let M be one of the subgroups of $G=D_n.2$ listed in Table 4, and assume $p= 2$. Set $V=V_{G^0}(\lambda )$, where

$$\begin{aligned} \lambda = \sum _{i=1}^{n-2}a_i\lambda _i+a_{n-1}(\lambda _{n-1}+\lambda _n) \end{aligned}$$

is p-restricted. Assume that V is nontrivial and $V \ne W,W^*$. Then V extends to a representation of G, and $V|_{M}$ is reducible.

Proof

As before, $\lambda$ is fixed by an involutory graph automorphism of $G^0$, so V extends to a representation of $G=G^0.2$. Seeking a contradiction, let us assume that $V|_{M}$ is irreducible. There are four cases to consider.

Case 1. M is a $\mathcal {C}_1$-subgroup of type $D_lD_{n-l}.2^2$.

Here $M^0=M_1M_2$, where $M_1=D_l$, $M_2=D_{n-l}$ and $1 \leqslant l < n/2$. First assume $l=1$. As in the proof of the previous proposition, we may assume that $M_2=\langle U_{\pm \alpha _{2}},\ldots ,U_{\pm \alpha _{n}} \rangle$, and by arguing as in the proof of [4, Lemma 3.2.3] we quickly reduce to the case where $V|_{M^0}$ has exactly four composition factors. Let k be minimal such that $a_k\ne 0$. Then $\lambda -\alpha _1-\cdots -\alpha _k$ affords the highest weight of a $KM_2$-composition factor of V, which is not conjugate (via a graph automorphism of $M_2$) to the composition factor afforded by $\lambda$. This contradiction eliminates the case $l=1$.

Now assume $l \geqslant 2$. As before, $M=M^0\langle \gamma _1,\gamma _2 \rangle$, where $\gamma _1$ and $\gamma _2$ act as involutory graph automorphisms on $M_1$ and $M_2$, respectively, and we may assume that $M_1$ and $M_2$ are as given in (2). Let $\{\omega _{1,1},\ldots ,\omega _{1,l}\}$ and $\{\omega _{2,1},\ldots ,\omega _{2,n-l}\}$ be the fundamental dominant weights corresponding to the bases $\Pi (M_1)$ and $\Pi (M_2)$ in (2), respectively. In view of (3), it is easy to see that $\lambda |_{M^0} = (\gamma _2\cdot \lambda )|_{M^0}$ and

$$\begin{aligned} (\gamma _1\cdot \lambda )|_{M^0}= & {} (\gamma _1\gamma _2\cdot \lambda )|_{M^0} \\= & {} \lambda |_{M^0}+\left( 2(a_{l}+ \cdots +a_{n-2})+a_{n-1}+a_{n}\right) (\omega _{1,l-1}-\omega _{1,l}). \end{aligned}$$

Now, by arguing as in the proof of [4, Lemma 3.2.3] we quickly reduce to the case $\lambda = a_{n-1}(\lambda _{n-1}+\lambda _{n})$. By inspecting (2), we see that $\mu = \lambda - \alpha _{l} - \alpha _{l+1} - \cdots - \alpha _{n-2} - \alpha _{n-1}$ affords the highest weight of a $KM^0$-composition factor, but this is not conjugate to $\lambda |_{M^0}$ since $\mu |_{M^0} = \lambda |_{M^0} + \omega _{1,l-1} - \omega _{1,l} - \omega _{2,n-l-1} + \omega _{2,n-l}$.

Case 2. M is a $\mathcal {C}_2$-subgroup of type $B_{n-1} \times 2$.

Write $M = M^0 \times \langle z \rangle$. By [17], $V|_{M^0}$ is reducible and thus $V|_{M^0} = V_1 \oplus V_2$, where $V_1$ and $V_2$ are irreducible $KM^0$-modules. Since z is central, it follows that $V|_{M^0}$ is homogeneous, but this is ruled out by Proposition 2.9.

Case 3. M is a $\mathcal {C}_2$-subgroup of type $(D_l^t.2^t).\mathrm{Sym}_t$.

Here $n=lt$, where $l \geqslant 1$ and $t \geqslant 2$. The case $l=1$ can be ruled out by arguing as in the first paragraph in the proof of [4, Lemma 4.3.8]. If $l \geqslant 3$ then by arguing as in the proof of [4, Lemma 4.3.8] (see [4, p. 55]) we reduce to the case $\lambda = \lambda _{n-1}+\lambda _n$, and this possibility is ruled out by the argument in the penultimate paragraph on [4, p. 56]. Finally, if $l=2$ then we quickly get down to the cases $\lambda \in \{\lambda _1+\lambda _{n-1}+\lambda _n, \lambda _{n-1}+\lambda _n\}$. The case $\lambda = \lambda _{n-1}+\lambda _n$ is ruled out as in loc. cit., and the other case is eliminated by arguing as in the final paragraph in the proof of [4, Lemma 4.3.8].

Case 4. M is a $\mathcal {C}_3$-subgroup of type $A_{n-1}T_1.2$.

Here $n \geqslant 3$ is odd and the argument given in the analysis of Case 5 in the proof of Proposition 3.3 can be applied.

This completes the proof of Proposition 3.4. $\square$

4 Geometric subgroups: the connected case

Suppose G, H and $V=V_G(\lambda )$ satisfy the conditions in Hypothesis 1.1. In this section and the next, we will establish Theorem 1.4 when H is contained in a geometric maximal subgroup M of G, excluding the special situation described in part (b) of Theorem 1.4. Assume $V|_{H}$ is irreducible, so $H^0$ is reductive by Lemma 2.7. Our first task is to determine the possibilities for the irreducible triple (G, M, V). If M is connected then we can read off the relevant cases by applying [17, Theorem 1]; the cases that arise are recorded in Table 5. Similarly, if M is disconnected, we can appeal to the main theorem of [4], which yields the list of cases given in Table 6 (in the first line of the table, T denotes a maximal torus of G).

Remark 4.1

It is worth noting that the cases listed in [17, Table 1] are recorded in terms of the image of the underlying representation $\varphi :G \rightarrow \mathrm{GL}(V)$, so Seitz’s table gives $(\varphi (G),\varphi (M),V)$, rather than (G, M, V). For instance, at the level of subgroups, the cases labelled S$_3$ and S$_4$ in [17, Table 1] correspond to irreducible triples with $(G,M) = (C_3,G_2)$ or $(B_3,G_2)$, but only the former possibility is listed in [17, Table 1] because in both cases the image $\varphi (G)$ is of type $C_3$. We also observe that there are certain maximal rank configurations arising in [17, Theorem 4.1] which are not listed [17, Table 1]. For example, referring to the case labelled MR$_1$ in [17, Table 1] (so $p=3$), Theorem 4.1 of [17] implies that the short (respectively, long) root $A_2$ in $G_2$ acts irreducibly on any p-restricted $G_2$-module whose highest weight has support on the short (respectively, long) roots, but only the former is listed in the table. In addition, we also note that the highest weights in [17, Table 1] are only given up to conjugacy by a graph automorphism, and we adopt the same convention in Table 5. Note that if the graph automorphism introduces a Frobenius twist on the module, then we will list the irreducible action on the corresponding p-restricted module.

Remark 4.2

As noted in Remark 1.2, if W denotes the natural KG-module then any irreducible triple (G, M, V) with $V = W^{\tau }$ (for some $\tau \in \mathrm{Aut}(G)$) is also excluded in [17, Table 1]. In particular, Seitz does not list the cases $(G,M) = (D_4,A_2)$ (with $p \ne 3$) and $(B_2,A_1)$ ($p \ne 2,3$), with V a spin module for G. For example, the spin module for $B_2$ is 4-dimensional, and it corresponds to the natural symplectic representation of $C_2$.

Remark 4.3

The triples of the form (G, M, V), where $(G,p) = (B_n,2)$ and M is a disconnected geometric maximal subgroup, are not stated explicitly in [4], but they are easily determined from the relevant list of cases in [4, Table 1] for the corresponding dual group of type $C_n$. The only possibilities are $M=B_l^t.\mathrm{Sym}_t$ (a $\mathcal {C}_2$-subgroup) with $\lambda =\lambda _n$, or $M=D_n.2$ and $\lambda =\lambda _n$ or $\sum _{i<n}a_i\lambda _i$. Note that G acts reducibly on W, so there are no triples involving non-geometric subgroups.

Table 5 M is a connected geometric subgroup

Table 6 M is a disconnected geometric subgroup

Remark 4.4

Let us make a few comments on the cases in Tables 5 and 6:

• (a) In case (ii) in Table 5 we have $G=A_n$ and $M=C_m$, where $n=2m-1$ and $m \geqslant 2$. Moreover, $\lambda =a\lambda _k+b\lambda _{k+1}$, where $1 \leqslant k < m$, $a+b=p-1>1$ and $a \ne 0$ if $k=m-1$. In particular, $p \ne 2$.

• (b) Note that H is a decomposition subgroup in case (v) of Table 5, so we may assume that $k \geqslant 2$ (if $k=1$ then V is a spin module).

• (c) In case (vi) in Table 5 we have $G=D_n$, $M=B_{m}$ (with $n=m+1 \geqslant 4$, $p \ne 2$) and $\lambda =b\lambda _k+a\lambda _{n-1}$, where $1 \leqslant k < n-1$, $ab \ne 0$ and $a+b+n-1-k \equiv 0 (\mathrm{mod} p)$.

• (d) In Table 5, following [17, Table 1], for G of type $A_n$ we record the highest weight $\lambda$ up to conjugacy by a graph automorphism.

• (e) Consider case (iv) in Table 6, where $G=B_n$ and $M=D_n.2$ is a $\mathcal {C}_1$-subgroup. Here $p \ne 2$ and the conditions on the highest weight $\lambda =\sum _{i=1}^{n}a_i\lambda _i$ are given in part (b) of Remark 1.8(v). Since H is a decomposition subgroup, we may assume that $a_i \ne 0$ for some $i<n$.

• (f) In case (vi) in Table 6 we have $G=C_n$ and $M=C_m^2.2$ is a $\mathcal {C}_2$-subgroup, where $n=2m$. Moreover, $\lambda =\lambda _{n-1}+a\lambda _n$, where $0 \leqslant a<p$ and $2a+3 \equiv 0 (\mathrm{mod} p)$. In particular, $p \ne 2$.

• (g) In cases (xi), (xii) and (xiii) we record M and V up to $\mathrm{Aut}(G)$-conjugacy. For instance, in case (xi), if $\tilde{M}$ denotes the image of M under an appropriate triality graph automorphism of G then $(G,\tilde{M},V_G(\lambda _1+\lambda _3))$ is an irreducible triple.

The main result of this section is the following.

Theorem 4.5

Let G, H and V be given as in Hypothesis 1.1, and assume $H<M<G$ where M is a connected geometric maximal subgroup of G. Then $V|_{H}$ is reducible.

Proof

Write $V=V_G(\lambda )$. The possibilities for (G, M, V) are given in Table 5; in each case M is a simple group of rank m. Let $\{\eta _1, \ldots , \eta _m\}$ be a set of fundamental dominant weights for M. Seeking a contradiction, let us assume that $V|_{H}$ is irreducible.

First consider case (i) in Table 5 (this is the case labelled I$_1$ in [17, Table 1]). Here $G=A_{n}$ and $M=C_m$ is a $\mathcal {C}_6$-subgroup of G, where $n=2m-1$, $m \geqslant 2$ and $p \ne 2$ (this is the natural embedding $\mathrm{Sp}(W) < \mathrm{SL}(W)$). Note that $\lambda =k\lambda _1$, $k \geqslant 2$ and $V|_{M} = V_M(k\eta _1)$ (see [17, Table 1]). Let J be a maximal subgroup of M containing H, so

$$\begin{aligned} H \leqslant J<M<G. \end{aligned}$$

We consider the irreducible triple $(M,J,V_M(k\eta _1))$. If $V|_{J^0}$ is irreducible then the triple $(M,J^0,V_M(k\eta _1))$ has to be in [17, Table 1], but it is easy to check that there are no compatible examples. Therefore J is disconnected and $V|_{J^0}$ is reducible. In this situation, $(M,J,V_M(k\eta _1))$ must be one of the triples arising in the main theorems of [3, 4], but once again we find that there are no such triples. We conclude that $V|_{H}$ is reducible in case (i).

The other cases in Table 5 are very similar, although some extra care is required in case (vi). Here $G=D_n$, $M=B_{n-1}$ is a $\mathcal {C}_1$-subgroup (we can view M as the stabilizer in G of a 1-dimensional non-degenerate subspace of W), and $\lambda =b\lambda _k+a\lambda _{n-1}$ satisfies the following conditions:

$$\begin{aligned} 1 \leqslant k < n-1, \quad a,b \ne 0, \quad a+b+n-1-k \equiv 0 (\mathrm{mod} p) \end{aligned}$$

(5)

(see case IV$_1'$ in [17, Table 1]). We note that $V|_{M} = V_M(b\eta _k+a\eta _{n-1})$. As before, let J be a maximal subgroup of M containing H, and consider the irreducible triple $(M,J,V_M(b\eta _k+a\eta _{n-1}))$.

If $V|_{J^0}$ is irreducible then by inspecting [17, Table 1] we see that the only possibility is the case labelled III$_1'$, where $M=B_3$ (so $n=4$), $J=G_2$, $k=2$ and $a+2b+2 \equiv 0 (\mathrm{mod} p)$. By (5), we also have $a+b+1 \equiv 0 (\mathrm{mod} p)$, so p divides $b+1$, and thus p divides a, which is a contradiction since the highest weight $\lambda = b\lambda _k+a\lambda _{n-1}$ is p-restricted. Therefore J is disconnected and $V|_{J^0}$ is reducible. We are now in a position to apply the main theorems in [3, 4]. We deduce that the only possibility is the configuration found by Ford, with $J=D_{n-1}.2$ (see the case labelled $\mathrm{U}_2$ in [8, Table II]; also see Remark 4.4(e)). Since the highest weight of $V|_{M}$ is $b\eta _k+a\eta _{n-1}$, we must have $a=1$ and $2a \equiv -2(n-1-k)-1 (\mathrm{mod} p)$. But it is easy to see that this congruence condition is incompatible with the congruence condition in (5). $\square$

5 Geometric subgroups: the disconnected case

The main result of this section is the following, which completes the proof of Theorem 1.4 when H is contained in a maximal geometric subgroup of G.

Theorem 5.1

Let G, H and $V=V_G(\lambda )$ be given as in the statement of Theorem 1.4, and assume that $H<M<G$ where M is a disconnected geometric maximal subgroup of G. Then $V|_{H}$ is irreducible if and only if (G, H, V) is one of the cases recorded in Table 7.

Table 7 The irreducible triples (G, H, V) in Theorem 5.1

Remark 5.2

Suppose that $H<M<G$, where (G, M, V) is the case labelled (vii) in Table 6, so $G=C_n$, $M=D_n.2$ and $p=2$. In Proposition 5.19 we deduce that $V|_H$ is irreducible if and only if $(n,p)=(4,2)$, $\lambda =\lambda _3$ and $H=C_1^3.X$ with $X=Z_3$ or $\mathrm{Sym}_3$, and so this establishes Theorem 5.1 in this situation. The result for case (vii) will be obtained from the result for (v) via an isogeny. That is, $V|_{H}$ is irreducible if and only if $(n,p)=(4,2)$, $\lambda =\lambda _3$ and $H=B_1^3.Z_3$ or $B_1^3.\mathrm{Sym}_3$. Therefore, for the remainder of this section we will exclude case (v) in Table 6 from our analysis.

We begin with a couple of preliminary lemmas. Our first result will play an important role in the analysis of cases (vii), (x), (xi) and (xii) in Table 6.

Lemma 5.3

Let G be a simple classical algebraic group with natural module W, and let H be a closed positive-dimensional subgroup of G such that $W|_{H^0}$ is reducible. Then there exists a geometric maximal subgroup M of G such that

• (i) $H \leqslant M$, and

• (ii) M does not normalize any decomposition of the form $W = W_1 \otimes \cdots \otimes W_t$, where $t \geqslant 3$ and the $W_i$ are equidimensional.

Proof

This follows from the proof of [14, Theorem 1$'$]. In particular, we refer the reader to the proofs of Lemmas 3.1, 3.2 and 3.3 in [14]. $\square$

We will also need the following lemma when dealing with cases (ii) and (viii) in Table 6.

Lemma 5.4

Let $G=A_n$ or $D_n$, and set $\mu _2 = \gamma \cdot \mu _1$, where $\mu _1=\sum _{i}a_i\lambda _i$ is a T-weight of G and $\gamma$ is an involutory graph automorphism of G. If we write $\mu _2-\mu _1 = \sum _{i}c_i\alpha _i$, then each $c_i$ is non-negative if and only if $\mu _1 = \mu _2$.

Proof

First assume $G=A_n$. We may assume that $\gamma$ interchanges the fundamental dominant weights $\lambda _i$ and $\lambda _{n+1-i}$ ($1 \leqslant i \leqslant n/2$), so $\mu _2= \sum _{i} a_{n+1-i} \lambda _i$. Set $\ell =\lfloor n/2 \rfloor$. Since

$$\begin{aligned} \lambda _i=\frac{1}{n+1}\left( \sum _{j=1}^{i-1}j(n+1-i)\alpha _j+\sum _{j=i}^n i(n+1-j)\alpha _j\right) \end{aligned}$$

(see [12, Table 1], for example) it follows that

$$\begin{aligned} \mu _2-\mu _1&= \frac{1}{n+1}\sum _{i=1}^n (a_{n+1-i}-a_i)\left( \sum _{j=1}^{i-1}j(n+1-i)\alpha _j+\sum _{j=i}^n i(n+1-j)\alpha _j\right) \\&= \frac{1}{n+1}\left[ \sum _{j=1}^{n}(n+1-j)\alpha _j \sum _{i=1}^j i(a_{n+1-i}-a_i)\right. \\&\left. \quad \; + \sum _{j=1}^{n-1}j\alpha _j \sum _{i=j+1}^n (n+1-i)(a_{n+1-i}-a_i)\right] ; \end{aligned}$$

so for $1\leqslant j \leqslant \ell$ we have

$$\begin{aligned} c_j\!=\! \frac{1}{n+1}(n+1-2j)\sum _{i=1}^{j-1} i(a_{n+1-i}-a_i)\!+\!\frac{1}{n+1}j\sum _{i=j}^\ell (n\!+\!1-2i)(a_{n+1-i}-a_i) \end{aligned}$$

and if $\ell +1 \leqslant j \leqslant n$ then

$$\begin{aligned} c_j= & {} \frac{1}{n+1}(n+1-2j)\sum _{i=1}^{n-j} i(a_{n+1-i}-a_i)+\frac{1}{n+1}(n+1-j)\\&\times \sum _{i=n+1-j}^\ell (n+1-2i)(a_{i}-a_{n+1-i}). \end{aligned}$$

We deduce that $c_j+c_{n+1-j}=0$ for all $1 \leqslant j \leqslant \ell$. In addition, if n is odd then $c_{\ell +1}=0$. The result follows.

The case $G=D_n$ is very similar. Here we may assume that $\gamma$ interchanges $\lambda _{n-1}$ and $\lambda _n$, so $\mu _2-\mu _1=(a_n-a_{n-1})(\lambda _{n-1}-\lambda _n)$ and the desired result follows. $\square$

For the remainder of this section, we will assume that (G, H, V) is given as in the statement of Theorem 1.4, so the conditions in Hypothesis 1.1 are satisfied. We will deal with each of the cases in Table 6 in turn, excluding case (v) as explained in Remark 5.2.

5.1 Proof of Theorem 5.1, part I

In this section, we will establish Theorem 5.1 in the case where $H<M<G$ and (G, M, V) is one of the cases labelled (i), (ii), (iii), (iv), (vi), (viii), (ix) or (xiii) in Table 6. The remaining cases, labelled (vii), (x), (xi) and (xii), will be handled in Sect. 5.2.

Proposition 5.5

Suppose $H<M<G$ and (G, M, V) is the case labelled (i) in Table 6, so $G=A_n$, $M=N_G(T)=T.\mathrm{Sym}_{n+1}$ is the normalizer of a maximal torus T of G, and $V=V_G(\lambda _k)$ with $1<k<n$. Then $V|_{H}$ is irreducible if and only if $H=T.X$ and $X < \mathrm{Sym}_{n+1}$ is $\ell$-transitive, where $\ell = \min \{k,n+1-k\}$.

Proof

Here $V=V_G(\lambda _k)=\Lambda ^k(W)$ is the k-th exterior power of the natural KG-module W, and by duality we may assume that $1<k \leqslant (n+1)/2$. Let $\mathcal {W}(G)=N_G(T)/T = \mathrm{Sym}_{n+1}$ be the Weyl group of G.

Set $S=H^0$ and note that $S \leqslant T$ is a subtorus. Let $\Lambda _S(W)$ and $\Lambda _S(V)$ be the set of S-weights of W and V, respectively, so

$$\begin{aligned} W|_{S} = \bigoplus _{\mu \in \Lambda _S(W)} W_{\mu },\quad V|_{S} = \bigoplus _{\mu \in \Lambda _S(V)} V_{\mu } \end{aligned}$$

where $W_{\mu }$ is the $\mu$-weight space of W, and similarly $V_{\mu }$ is the $\mu$-weight space of V. There is a natural action of $N_G(S)$ on $\Lambda _S(W)$ and $\Lambda _S(V)$ given by $(x \cdot \mu )(s) = \mu ( xsx^{-1})$. In particular, $N_G(S)$ permutes the S-weight spaces on W and V.

First assume $S=T$. Here the S-weight spaces on W and V are 1-dimensional, and $V|_{H}$ is irreducible if and only if $H/T \leqslant \mathcal {W}(G)$ acts transitively on $\Lambda _S(V)$. This is equivalent to the condition that H / T is a k-transitive subgroup of $\mathcal {W}(G)=\mathrm{Sym}_{n+1}$. Indeed, we note that the S-weight vectors on V are of the form $w_{i_1} \wedge \cdots \wedge w_{i_k}$, where the $i_j$ are distinct and $\{w_1, \ldots , w_{n+1}\}$ is a basis of W consisting of S-weight vectors. This gives the desired result when $S=T$, so for the remainder let us assume that S is a proper subtorus of T.

Seeking a contradiction, suppose $V|_{H}$ is irreducible. Now $H \leqslant N_G(S)$ (since $S=H^0$) and thus $V|_{N_G(S)}$ is irreducible. In particular, $N_G(S)$ acts irreducibly on W (otherwise $N_G(S)$ lies in a parabolic subgroup of G, which would imply $V|_{H}$ is reducible), so $N_G(S)$ must transitively permute the set of S-weight spaces on W. Therefore, these S-weight spaces are equidimensional, whence $N_G(S) \leqslant J<G$, where J is a $\mathcal {C}_2$-subgroup of G. More precisely, J is the normalizer in G of the direct sum decomposition $\bigoplus _{\mu \in \Lambda _S(W)}W_{\mu }$. If we now consider the irreducible triple (G, J, V) then the main theorem of [4] implies that the S-weight spaces on W are 1-dimensional, so S is a regular torus. In particular,

$$\begin{aligned} S \leqslant C_G(S)=T \leqslant N_G(S) \leqslant N_G(T)=M \end{aligned}$$

and we define

$$\begin{aligned} \mathcal {W}(S) := N_G(S)/C_G(S) = N_G(S)/T \leqslant N_G(T)/T = \mathcal {W}(G). \end{aligned}$$

As noted above, $\mathcal {W}(S)$ permutes the S-weight spaces $V_{\mu }$, and the irreducibility of $V|_{N_G(S)}$ implies that this action is transitive. In particular, the S-weight spaces on V are equidimensional. In fact, we claim that they are 1-dimensional. To see this, let d denote the dimension of S and fix a basis $\{w_1, \ldots , w_{n+1}\}$ of W comprising S-weight vectors. Then there exist integers $c_{i,j}$, $1 \leqslant i \leqslant d$, $1 \leqslant j \leqslant n+1$ such that

$$\begin{aligned} (s_1, \ldots , s_d) \cdot w_j = \left( s_1^{c_{1,j}}s_2^{c_{2,j}} \ldots s_d^{c_{d,j}}\right) w_j \end{aligned}$$

for all $(s_1, \ldots , s_d) \in (K^*)^d \cong S$ and all $1 \leqslant j \leqslant n+1$. Without loss of generality, we may assume that the $w_j$ are ordered so that the d-tuples

$$\begin{aligned} (c_{1,1}, \ldots , c_{1,d}), \ldots , (c_{n+1,1}, \ldots , c_{n+1,d}) \end{aligned}$$

(6)

are in lexicographic order. (Note that these d-tuples are distinct since the S-weight spaces on W are 1-dimensional.) Then $w_1 \wedge \cdots \wedge w_k \in V$ is an S-weight vector of weight

$$\begin{aligned} s_1^{\sum _{i=1}^{k}c_{1,i}}s_2^{\sum _{i=1}^{k}c_{2,i}} \cdots s_d^{\sum _{i=1}^{k}c_{d,i}}. \end{aligned}$$

In view of the lexicographic ordering of the tuples in (6), it follows that this S-weight has multiplicity 1, and this justifies the claim.

As previously observed, the irreducibility of $V|_{N_G(S)}$ now implies that $\mathcal {W}(S) \leqslant \mathrm{Sym}_{n+1}$ is k-transitive, so to complete the proof of the proposition, it suffices to show that $\mathcal {W}(S)$ is not 2-transitive.

To see this, first let c be the codimension of S in T and let $X(T) \cong \mathbb {Z}^n$ and $X(S) \cong \mathbb {Z}^{n-c}$ be the corresponding character groups. The sublattice $S^{\perp }$ is defined by

$$\begin{aligned} S^{\bot }=\{\gamma \in X(T) \mid \gamma |_{S}=1\} \cong \mathbb {Z}^c \end{aligned}$$

and we set

$$\begin{aligned} X(T)_{\mathbb {R}} = X(T) \otimes _{\mathbb {Z}} \mathbb {R},\quad S^{\perp }_{\mathbb {R}} = S^{\perp } \otimes _{\mathbb {Z}} \mathbb {R}. \end{aligned}$$

Now $\mathcal {W}(G)$ acts faithfully on $X(T)_{\mathbb {R}}$, and $\mathcal {W}(S)=N_G(S)/T$ stabilizes the c-dimensional subspace $S^{\perp }_{\mathbb {R}}$. Let P be the pointwise stabilizer of $S^{\perp }_{\mathbb {R}}$ in $\mathcal {W}(G)$. By [15, Corollary A.29], P is a parabolic subgroup of $\mathcal {W}(G)=\mathrm{Sym}_{n+1}$, so it is a direct product of smaller degree symmetric groups. In particular, P is intransitive. Finally, we observe that $\mathcal {W}(S)$ normalizes P (since it stabilizes $S^{\perp }_{\mathbb {R}}$), so the intransitivity of P implies that $\mathcal {W}(S)$ is either intransitive, or transitive and imprimitive. In particular, $\mathcal {W}(S)$ is not 2-transitive. $\square$

Proposition 5.6

Suppose $H<M<G$ and (G, M, V) is the case labelled (iv) in Table 6. Then $V|_{H}$ is reducible.

Proof

Here $G=B_n$ and $M=D_n.2$, where $n \geqslant 3$ and $p \ne 2$. We have

$$\begin{aligned} H< M=D_n.2 < G = B_n \end{aligned}$$

and $V=V_G(\lambda )$, where the highest weight $\lambda = \sum _{i}a_i\lambda _i$ satisfies the conditions recorded in Remark 4.4(e). This is the case labelled $\mathrm{U}_2$ in [8, Table II]. In particular, we note that $a_n=1$ and $V|_{M^0}$ has exactly two composition factors, say $V|_{M^0} = V_1 \oplus V_2$, where $V_i$ has highest weight $\mu _i$, and

$$\begin{aligned} \mu _1= & {} \sum _{i=1}^{n-2}a_i\eta _i + a_{n-1}\eta _{n-1}+(a_{n-1}+1)\eta _n,\\ \mu _2= & {} \sum _{i=1}^{n-2}a_i\eta _i + (a_{n-1}+1)\eta _{n-1}+a_{n-1}\eta _n. \end{aligned}$$

(with respect to fundamental dominant weights $\{\eta _1, \ldots , \eta _n\}$ for $M^0=D_n$). As noted in Remark 4.4(e), we may assume that $a_i \ne 0$ for some $i<n$.

Seeking a contradiction, let us assume that $V|_{H}$ is irreducible, so $H \not \leqslant M^0$ since $V|_{M^0}$ is reducible. Set $H_1 = H \cap M^0$ and let J be a maximal subgroup of $M^0$ that contains $H_1$. Then $H=H_1.2$ and the irreducibility of $V|_{H}$ implies that $V_1|_{H_1}$ and $V_2|_{H_1}$ are irreducible, so $V_1|_{J}$ and $V_2|_{J}$ are also irreducible.

We can now consider the irreducible triple $(M^0,J,V_1)$, which must be one of the cases recorded in [3, 4, 17]. Given the conditions on $\lambda$ (in particular, the fact that $a_n=1$ and $a_i \ne 0$ for some $i<n$), it is easy to see that there are no compatible examples in [3, 4]. The only possible example in [17, Table1] is the case labelled IV$_1'$, with $J=B_{n-1}$, $a=1$ and $b \ne 0$. However, we claim that the conditions in this configuration are incompatible with those that are given in Remark 4.4(e). Indeed, we have $a_{n-1}=0$ and there is a unique $k<n-1$ with $a_k \ne 0$. In case IV$_1'$ we have $a_k+n-k \equiv 0 (\mathrm{mod} p)$ and the conditions in Remark 4.4(e) yield $2a_k \equiv -2(n-k)-1 (\mathrm{mod} p)$. If both conditions hold, then p divides $a_k+n-k$ and $1+2a_k+2(n-k)$, so p divides $1+a_k+n-k$. Clearly, this is impossible. $\square$

Proposition 5.7

Suppose $H<M<G$ and (G, M, V) is the case labelled (vi) in Table 6. Then $V|_{H}$ is reducible.

Proof

Here $G= C_n$, $M=C_m^2.2$ is a $\mathcal {C}_2$-subgroup and $\lambda =\lambda _{n-1}+a\lambda _n$, where $n=2m$, $0\leqslant a<p$ and $2a+3 \equiv 0 (\mathrm{mod} p)$. In particular, note that $p \ne 2$ and $a<p-1$. Seeking a contradiction, let us assume that $V|_{H}$ is irreducible. Set $M^0=C_m^2=M_1M_2$ (a direct product of two simply connected groups of type $C_m$) and let $\{\omega _{i,1}, \ldots , \omega _{i,m}\}$ be fundamental dominant weights for $M_i$.

As recorded in [4, Table 4.2], we have $V|_{M^0} = V_1 \oplus V_2$ where

$$\begin{aligned} V_1&= V_{M_1}((a+1)\omega _{1,m}) \otimes V_{M_2}(\omega _{2,m-1}+a\omega _{2,m}) \\ V_2&= V_{M_1}(\omega _{1,m-1}+a\omega _{1,m}) \otimes V_{M_2}((a+1)\omega _{2,m}). \end{aligned}$$

Set $H_1 = H \cap M^0$ and note that $H_1^0 = H^0$ and $H=H_1.2$. In particular, since $V|_{H}$ is irreducible it follows that $V|_{H_1}$ has exactly two composition factors, namely

$$\begin{aligned} V|_{H_1} = V_1|_{H_1} \oplus V_2|_{H_1}. \end{aligned}$$

Note that $H^0$ is a proper subgroup of $M^0$ since we are assuming that H is disconnected and non-maximal. Let $\pi _i : H_1 \rightarrow M_i$ be the i-th projection map and note that $\ker (d\pi _1) \cap \ker (d\pi _2) = 0$ since $H_1$ is a closed positive-dimensional subgroup of $M^0$.

Claim. $\pi _1(H_1)$ and $\pi _2(H_1)$ are infinite.

Seeking a contradiction, suppose that $\pi _1(H_1)$ is finite, in which case $\pi _2(H_1)$ is infinite since H is positive-dimensional. The finiteness of $\pi _1(H_1)$ implies that $d\pi _1 = 0$, so $\ker (d\pi _2)=0$ since $\ker (d\pi _1) \cap \ker (d\pi _2) = 0$. Also note that $\ker (\pi _1)$ is a closed subgroup of finite index in $H_1$, so $H_1^0 \leqslant \ker (\pi _1)$ and thus $\pi _2|_{H^0}: H^0 \rightarrow M_2$ is injective.

Next we claim that $\pi _2$ is surjective. Suppose otherwise. Then there exists a positive-dimensional maximal subgroup $J_2$ of $M_2$ such that $\pi _2(H_1) \leqslant J_2< M_2$. The irreducibility of $V_1|_{H_1}$ and $V_2|_{H_1}$ implies that $\pi _2(H_1)$ acts irreducibly on the $KM_2$-modules with highest weights $\omega _{2,m-1}+a\omega _{2,m}$ and $(a+1)\omega _{2,m}$, so we can consider the irreducible triples

$$\begin{aligned} (M_2,J_2, V_{M_2}(\omega _{2,m-1}+a\omega _{2,m})),\quad (M_2,J_2, V_{M_2}((a+1)\omega _{2,m})). \end{aligned}$$

By inspecting [17, Table 1] we see that there are no compatible examples with $J_2$ connected. Similarly, by applying the main theorems in [3, 4], there are no examples with $J_2$ disconnected. This is a contradiction, hence $\pi _2$ is surjective.

It follows that $\pi _2(H_1^0)=M_2$, so $\pi _2|_{H^0}: H^0 \rightarrow M_2$ is a bijective morphism. Moreover, $\ker (d(\pi _2|_{H^0}))=0$ since $\ker (d\pi _2)=0$, so $d(\pi _2|_{H^0})$ is an isomorphism of Lie algebras and thus $\pi _2|_{H^0}$ is an isomorphism of algebraic groups. In particular, $H^0$ is a simply connected group of type $C_m$. By Lemma 2.2 we may write $\pi _2|_{H^0}=t_{x}$ for some $x\in H^0$, where $t_x$ is an inner automorphism (conjugation by x). In addition, note that $H \leqslant N_G(H^0)=H^0C_G(H^0)$ and thus $V|_{H^0}$ is homogeneous.

Let $\{\eta _1,\ldots , \eta _m\}$ be a set of fundamental dominant weights for $H^0$. Then $V|_{H^0}$ has composition factors isomorphic to $V_{H^0}(\eta _{m-1}+a\eta _m)$ and $V_{H^0}((a+1)\eta _m)$, which contradicts the fact that $V|_{H^0}$ is homogeneous. We conclude that $\pi _1(H_1)$ is infinite, and similarly $\pi _2(H_1)$ is also infinite.

Claim. $\pi _1$ and $\pi _2$ are surjective.

Seeking a contradiction, suppose $\pi _1$ is not surjective. Since $\pi _1(H_1)$ is infinite, there exists a positive-dimensional maximal subgroup $J_1$ of $M_1$ such that $\pi _1(H_1) \leqslant J_1<M_1$ and we can consider the irreducible triples

$$\begin{aligned} (M_1,J_1, V_{M_1}((a+1)\omega _{1,m})), \quad (M_1,J_1, V_{M_1}(\omega _{1,m-1}+a\omega _{1,m})). \end{aligned}$$

As before, we find that there are no compatible examples, which is a contradiction and thus $\pi _1$ is surjective. An entirely similar argument shows that $\pi _2$ is also surjective.

By the previous claim, it follows that $\pi _i(H^0)=M_i$ for $i=1,2$, so $H^0$ is a subdirect product of the direct product $M^0=M_1M_2$. By applying Lemma 2.5, noting that $H^0 <M^0$, we deduce that $H^0 \cong M_1$ is diagonally embedded in $M_1M_2$, so we may write

$$\begin{aligned} H^0=\{(\tau _1(x), \tau _2(x)) {\mid } x \in \mathrm{Sp}_{2m}(K)\}, \end{aligned}$$

where $\tau _i: \mathrm{Sp}_{2m}(K)\rightarrow M_i$ is a bijective morphism. By appealing to Lemma 2.2, we may write $\tau _i=t_{x_i}\sigma _{q_i}$ for some $x_i \in H^0$ and p-power $q_i$ (where $\sigma _{q_i}$ is a standard field automorphism), and once again we note that $V|_{H^0}$ is homogeneous. Note that at least one $q_i$ is equal to 1 (since $H^0$ is a closed subgroup $M^0$); without loss of generality we will assume $q_2=1$.

Let $\{\eta _1,\dots , \eta _m\}$ be a set of fundamental dominant weights for $H^0$. Then

$$\begin{aligned} V_1|_{H^0} = V_{H^0}((a+1)\eta _m)^{(q_1)} \otimes V_{H^0}(\eta _{m-1}+a\eta _{m}) \end{aligned}$$

and

$$\begin{aligned} V_2|_{H^0} = V_{H^0}(\eta _{m-1}+a\eta _{m})^{(q_1)} \otimes V_{H^0}((a+1)\eta _m), \end{aligned}$$

so $V|_{H^0}$ has composition factors with highest weights

$$\begin{aligned} \eta _{m-1}+((a+1)q_1+a)\eta _m,\quad q_1\eta _{m-1}+(aq_1+a+1)\eta _m. \end{aligned}$$

Since $V|_{H^0}$ is homogeneous, these highest weights must be equal and thus $q_1=1$. Now $p<a-1$ so the modules $V_{H^0}((a+1)\eta _m)$ and $V_{H^0}(\eta _{m-1}+a\eta _{m})$ are p-restricted and thus Lemma 2.10 implies that $V|_{H^0}$ is not homogeneous. This is a contradiction.$\square$

Proposition 5.8

Suppose $H<M<G$ and (G, M, V) is the case labelled (viii) in Table 6. Then $V|_{H}$ is reducible.

Proof

Here $G=D_n$ and $M=(D_m^2.2).2$ is a $\mathcal {C}_2$-subgroup, where $n=2m$, $m\geqslant 3$ is odd and $p=2$. Moreover, $V=V_G(\lambda )$ where $\lambda =\lambda _1+\lambda _{n-1}$ or $\lambda _1+\lambda _n$ (see Table 6); without loss of generality, we will fix $\lambda = \lambda _1+\lambda _{n-1}$. Seeking a contradiction, let us assume that $V|_{H}$ is irreducible.

Write $M^0=D_m^2=M_1M_2$ and let $\{\omega _{i,1}, \ldots , \omega _{i,m}\}$ be fundamental dominant weights for $M_i$. Then [4, Table 4.2] indicates that $V|_{M^0} = V_1 \oplus V_2 \oplus V_3 \oplus V_4$, where

$$\begin{aligned} V_1&= V_{M_1}(\omega _{1,1}+\omega _{1,m}) \otimes V_{M_2}(\omega _{2,m-1}) \\ V_2&= V_{M_1}(\omega _{1,1}+\omega _{1,m-1}) \otimes V_{M_2}(\omega _{2,m}) \\ V_3&= V_{M_1}(\omega _{1,m-1}) \otimes V_{M_2}(\omega _{2,1}+\omega _{2,m}) \\ V_4&= V_{M_1}(\omega _{1,m}) \otimes V_{M_2}(\omega _{2,1}+\omega _{2,m-1}) \end{aligned}$$

Set $H_1 = H \cap M^0$ and note that $H_1^0 = H^0$ and $|H:H_1|=4$. Indeed, $H/H_1$ is isomorphic to a subgroup of $M/M^0$ and thus $|H:H_1| \leqslant 4$, but $V|_{H_1}$ has at least four composition factors and thus the irreducibility of $V|_H$ implies that $|H:H_1|=4$. Therefore, $V|_{H_1}$ has exactly four composition factors, namely

$$\begin{aligned} V|_{H_1}=V_1|_{H_1}\oplus V_2|_{H_1}\oplus V_3|_{H_1}\oplus V_4|_{H_1}. \end{aligned}$$

In order to proceed as in the proof of the previous proposition, we need to slightly modify our set-up. Indeed, G is the simply connected group of type $D_n$, so $M^0=M_1M_2$ is a central product of spin groups of type $D_m$. Since the $KM^0$-module W lifts to a representation $\rho : L \rightarrow \mathrm{GL}(W)$, where $L=L_1L_2$ is the direct product of two simply connected groups of type $D_m$, we have $M^0 = L/Y$ where $Y = \ker (\rho )$. In particular, there exist subgroups $R \leqslant R_1 \leqslant L$ such that $H_1=R_1/Y$ and $H^0=R/Y$. Note that $H_1/H^0 \cong R_1/R$ and $R^0=R_1^0$. Let $\pi _i: R_1 \rightarrow L_i$ be the i-th projection map and note that $\ker (d\pi _1) \cap \ker (d\pi _2) = 0$ since $R_1$ is a closed positive-dimensional subgroup of L.

The $KM^0$-module $V_i$ lifts to a representation $\rho _i: L\rightarrow \mathrm{GL}(V_i)$, so we can consider $V_i|_{R_1}$. The irreducibility of $V_i|_{H_1}$ implies that $V_i|_{R_1}$ is also irreducible, whence $\pi _1(R_1)$ is irreducible on each of the $KL_1$-modules

$$\begin{aligned} V_{L_1}(\omega _{1,1}+\omega _{1,m}),\quad V_{L_1}(\omega _{1,1}+\omega _{1,m-1}),\quad V_{L_1}(\omega _{1,m-1}),\quad V_{L_1}(\omega _{1,m}), \end{aligned}$$

(7)

and similarly $\pi _2(R_1)$ is irreducible on the $KL_2$-modules

$$\begin{aligned} V_{L_2}(\omega _{2,m-1}),\quad V_{L_2}(\omega _{2,m}),\quad V_{L_2}(\omega _{2,1}+\omega _{2,m}),\quad V_{L_2}(\omega _{2,1}+\omega _{2,m-1}). \end{aligned}$$

(8)

Claim. $\pi _1(R_1)$ and $\pi _2(R_1)$ are infinite.

We proceed as in the proof of Proposition 5.7. Suppose $\pi _1(R_1)$ is finite. Then $\pi _2(R_1)$ is infinite, $\ker (d\pi _2)=0$ and $R_1^0 \leqslant \ker (\pi _1)$, so $\pi _2|_{R^0}: R^0\rightarrow L_2$ is injective.

Suppose $\pi _2$ is not surjective. Then there exists a positive-dimensional maximal subgroup $J_2$ of $L_2$ such that $\pi _2(R_1) \leqslant J_2 <L_2$, and we can consider the irreducible triples $(L_2,J_2,U)$ for the four $KL_2$-modules U in (8). By applying the main theorems of [3, 4, 17] we find that there are no compatible examples (note that in the case labelled IV$_1'$ in [17, Table 1], we require the parameters to be $a=b=k=1$, hence the given congruence condition implies that m is even, which is false). This is a contradiction, hence $\pi _2$ is surjective.

It follows that $\pi _2(R_1^0)=L_2$ and $\pi _2|_{R^0}: R^0\rightarrow L_2$ is a bijective morphism. Furthermore, $\ker (d(\pi _2|_{R^0}))=0$ so $d(\pi _2|_{R^0})$ is an isomorphism and thus $\pi _2|_{R^0}$ is an isomorphism of algebraic groups. In particular, $R^0$ is simply connected of type $D_m$. By Lemma 2.2, we may write $\pi _2|_{R^0}=t_{x}\gamma ^{k}$ for some $x \in R^0$ and integer $k \in \{0,1\}$, where $\gamma$ is an involutory graph automorphism (note that $m \ne 4$, so a triality automorphism does not arise here). Now $H \leqslant N_G(H^0)$ induces algebraic group automorphisms of $H^0$ that permute the $KR^0$-composition factors of V, so $V|_{R^0}$ has the following homogeneity property:

$$\begin{aligned} \begin{array}{l} { Either\, V|_{R^0}~is~homogeneous,~or~the~homogeneous~components~of~V|_{R^0}~are} \\ { conjugate~under~an~involutory~graph~automorphism~of~R^0.} \end{array} \end{aligned}$$

(9)

Let $\{\eta _1,\ldots , \eta _m\}$ be fundamental dominant weights for $R^0$. Then $V|_{R^0}$ has composition factors isomorphic to $V_{R^0}(\eta _{m-1})$, $V_{R^0}(\eta _{m})$, $V_{R^0}(\eta _1 + \eta _{m})$ and $V_{R^0}(\eta _1 + \eta _{m-1})$, but this is incompatible with (9). Therefore $\pi _1(R_1)$ is infinite, and similarly $\pi _2(R_1)$ is also infinite.

Claim. $\pi _1$ and $\pi _2$ are surjective.

Suppose $\pi _1$ is not surjective. Then there exists a positive-dimensional maximal subgroup $J_1$ of $L_1$ such that $\pi _1(R_1) \leqslant J_1<L_1$ and we can consider the irreducible triples $(L_1,J_1, U)$ for the four $KL_1$-modules U in (7). We have already noted that there are no compatible examples and thus $\pi _1$ is surjective. Similarly, $\pi _2$ is surjective.

We have $\pi _i(R^0)=L_i$ for $i=1,2$, so $R^0$ is a subdirect product of $L=L_1L_2$ and thus Lemma 2.5 implies that either $R^0 =L$, or $R^0\cong L_1$ is diagonally embedded in $L_1L_2$. If $R^0=L$ then $H^0=M^0$ and the irreducibility of $V|_{H}$ implies that $H=M$, which is false. Therefore $R^0$ is diagonally embedded, so

$$\begin{aligned} R^0 = \{(\tau _1(x), \tau _2(x)) {\mid } x \in \mathrm{Spin}_{2m}(K)\} \end{aligned}$$

and $\tau _i: \mathrm{Spin}_{2m}(K)\rightarrow L_i$ is a bijective morphism. In particular, we may write $\tau _i=t_{x_i}\sigma _{q_i}\gamma ^{k_i}$ for some $x_i \in R^0$, p-power $q_i$ and $k_i\in \{0,1\}$ (see Lemma 2.2). Again, we observe that (9) holds. Since $R^0$ is a closed subgroup of L, it follows that at least one $q_i$ is equal to 1. We may assume $q_2=1$.

Let $\{\eta _1,\ldots , \eta _m\}$ be a set of fundamental dominant weights for $R^0$. By considering the restriction of V to $R^0$, we deduce that $V_i|_{R^0}$ has a composition factor with highest weight $\mu _i$ as follows:

$(k_1,k_2)$	$\mu _1$	$\mu _2$	$\mu _3$	$\mu _4$
(0, 0)	$q_1\eta _{1}+\eta _{m-1}+q_1\eta _m$	$q_1\eta _{1}+q_1\eta _{m-1}+\eta _m$	$\eta _{1}+q_1\eta _{m-1}+\eta _m$	$\eta _{1}+\eta _{m-1}+q_1\eta _m$
(1, 0)	$q_1\eta _{1}+(q_1+1)\eta _{m-1}$	$q_1\eta _{1}+(q_1+1)\eta _m$	$\eta _{1}+(q_1+1)\eta _m$	$\eta _{1}+(q_1+1)\eta _{m-1}$
(0, 1)	$q_1\eta _{1}+(q_1+1)\eta _m$	$q_1\eta _{1}+(q_1+1)\eta _{m-1}$	$\eta _{1}+(q_1+1)\eta _{m-1}$	$\eta _{1}+(q_1+1)\eta _m$
(1, 1)	$q_1\eta _{1}+q_1\eta _{m-1}+\eta _m$	$q_1\eta _{1}+\eta _{m-1}+q_1\eta _m$	$\eta _{1}+\eta _{m-1}+q_1\eta _m$	$\eta _{1}+q_1\eta _{m-1}+\eta _m$

In view of (9), we deduce that $q_1=1$ in all four cases. By applying Lemma 2.10 it follows that $V|_{R^0}$ is not homogeneous. More precisely, $\mu _1$ affords the highest weight of a composition factor of $V_1|_{R^0}$ and if $\nu$ is the highest weight of any other composition factor of $V_1|_{R^0}$, then $\nu \ne \mu _1$ and $\nu \preccurlyeq \mu _1$. However, in view of Lemma 5.4, this is incompatible with (9). $\square$

Proposition 5.9

Suppose $H<M<G$ and (G, M, V) is the case labelled (ii) in Table 6. Then $V|_{H}$ is reducible.

Proof

Here $G=A_n$ and $M=A_m^2.2$ is a $\mathcal {C}_4(ii)$-subgroup, where $n=m(m+2)$, $p \ne 2$ and $m \geqslant 2$. Moreover, $V=V_G(\lambda )$ and $\lambda =\lambda _2$ or $\lambda _{n-1}$. By duality, we may assume that $V=V_G(\lambda _2) = \Lambda ^2(W)$. Seeking a contradiction, let us assume that $V|_{H}$ is irreducible.

Write $M^0 = A_m^2 = M_1M_2$ and note that this is a central product. Let $\{\omega _{i,1}, \ldots , \omega _{i,m}\}$ be fundamental dominant weights for $M_i$. As recorded in [4, Table 6.2], we have $V|_{M^0} = V_1 \oplus V_2$, where

$$\begin{aligned} V_1 = V_{M_1}(\omega _{1,2}) \otimes V_{M_2}(2\omega _{2,1}),\quad V_2 = V_{M_1}(2\omega _{1,1}) \otimes V_{M_2}(\omega _{2,2}). \end{aligned}$$

Set $H_1 = H \cap M^0$ and note that $H_1^0 = H^0$ and $H=H_1.2$. Since $V|_{H}$ is irreducible it follows that $V|_{H_1}$ has exactly two composition factors, namely

$$\begin{aligned} V|_{H_1} = V_1|_{H_1} \oplus V_2|_{H_1}. \end{aligned}$$

Note that $H^0<M^0$ since H is disconnected and non-maximal.

As in the proof of Proposition 5.8, we need to modify this initial set-up in order to proceed as we did in the proof of Proposition 5.7. Since W is a $KM^0$-module, it lifts to a representation $\rho : L \rightarrow \mathrm{GL}(W)$, where $L=L_1L_2$ is the direct product of two simply connected groups $A_m = \mathrm{SL}_{m+1}(K)$. Then $M^0 = L/Y$, where $Y = \ker (\rho )$, and so there exist subgroups $R \leqslant R_1 \leqslant L$ such that $H_1 = R_1/Y$ and $H^0=R/Y$. Note that $H_1/H^0 \cong R_1/R$ is finite, so $R^0=R_1^0$. Let $\pi _i : R_1 \rightarrow L_i$ be the i-th projection map and observe that $\ker (d\pi _1) \cap \ker (d\pi _2) = 0$.

Since the $KM^0$-module $V_i$ lifts to a representation $\rho _i : L \rightarrow \mathrm{GL}(V_i)$, we can consider the restriction of $V_i$ to $R_1$. The irreducibility of $V_1|_{H_1}$ and $V_2|_{H_1}$ implies that $V_1|_{R_1}$ and $V_2|_{R_1}$ are also irreducible, whence $\pi _1(R_1)$ is irreducible on the $KL_1$-modules $V_{L_1}(\omega _{1,2})$ and $V_{L_1}(2\omega _{1,1})$, and $\pi _2(R_1)$ acts irreducibly on $V_{L_2}(2\omega _{2,1})$ and $V_{L_2}(\omega _{2,2})$.

Claim. $\pi _1(R_1)$ and $\pi _2(R_1)$ are infinite.

We proceed as in the proof of Proposition 5.7; the details are very similar. Suppose that $\pi _1(R_1)$ is finite, so $\ker (d\pi _2)=0$. Then $\pi _2(R_1)$ has to be infinite since H (and thus $H_1$, and also $R_1$) is infinite. Since $\ker (\pi _1) \leqslant R_1$ has finite index, it follows that $R_1^0 \leqslant \ker (\pi _1)$ and thus $\pi _2|_{R^0}: R^0 \rightarrow L_2$ is injective.

Suppose $\pi _2$ is not surjective. Then there exists a positive-dimensional maximal subgroup $J_2$ of $L_2$ such that $\pi _2(R_1) \leqslant J_2< L_2$. As noted above, $\pi _2(R_1)$ acts irreducibly on $V_{L_2}(2\omega _{2,1})$ and $V_{L_2}(\omega _{2,2})$, so we may consider the irreducible triples $(L_2,J_2, V_{L_2}(2\omega _{2,1}))$ and $(L_2,J_2, V_{L_2}(\omega _{2,2}))$. In the usual way, by inspecting [3, 4, 17], we deduce that there are no compatible examples, whence $\pi _2$ is surjective.

Therefore $\pi _2(R_1^0)=L_2$ and thus $\pi _2|_{R^0}: R^0 \rightarrow L_2$ is a bijective morphism. Moreover, $\ker (d(\pi _2|_{R^0}))=0$ so $d(\pi _2|_{R^0})$ is an isomorphism and thus $\pi _2|_{R^0}$ is an isomorphism of algebraic groups. By Lemma 2.2, we can write $\pi _2|_{R^0}=t_{x}\gamma ^{k}$ for some $x \in R^0$ and integer $k\in \{0,1\}$, where $\gamma$ is a graph automorphism. Note that (9) holds.

Let $\{\eta _1,\ldots , \eta _m\}$ be a set of fundamental dominant weights for $R^0$. Then $V|_{R^0}$ has composition factors isomorphic to $V_{R^0}(2\eta _1)$ and $V_{R^0}(\eta _2)$ if $k=0$, and $V_{R^0}(2\eta _m)$ and $V_{R^0}(\eta _{m-1})$ if $k=1$. But the corresponding highest weights are incompatible with (9), so we have reached a contradiction. We conclude that $\pi _1(R_1)$ is infinite, and similarly $\pi _2(R_1)$ is also infinite.

Claim. $\pi _1$ and $\pi _2$ are surjective.

Suppose $\pi _1$ is not surjective. Then there exists a positive-dimensional maximal subgroup $J_1$ of $L_1$ such that $\pi _1(R_1) \leqslant J_1<L_1$ and we can consider the irreducible triples $(L_1,J_1, V_{L_1}(2\omega _{1,1}))$ and $(L_1,J_1, V_{L_1}(\omega _{1,2}))$. As above, there are no compatible examples and thus $\pi _1$ is surjective. An entirely similar argument shows that $\pi _2$ is also surjective.

Now $\pi _i(R^0)=L_i$ for $i=1,2$, so $R^0$ is a subdirect product of $L=L_1L_2$ and thus Lemma 2.5 implies that $R^0 \cong L_1$ is diagonally embedded in $L_1L_2$ (note that $H^0<M^0$, so $R^0 < L$). Therefore

$$\begin{aligned} R^0=\{(\tau _1(x), \tau _2(x)) {\mid } x \in \mathrm{SL}_{m+1}(K)\}, \end{aligned}$$

where $\tau _i: \mathrm{SL}_{m+1}(K)\rightarrow L_i$ is a bijective morphism. As before, we may write $\tau _i=t_{x_i}\sigma _{q_i}\gamma ^{k_i}$ for some $x_i \in R^0$, p-power $q_i$ and $k_i\in \{0,1\}$. Note that (9) holds. As before, we may assume that $q_2=1$.

Let $\{\eta _1,\dots , \eta _m\}$ be a set of fundamental dominant weights for $R^0$. Now $V|_{R^0} = V_1|_{R^0} \oplus V_2|_{R^0}$ and we calculate that $V|_{R^0}$ has composition factors with the following highest weights $\mu _1$ and $\mu _2$:

$(k_1,k_2)$	$\mu _1$	$\mu _2$
(0, 0)	$2\eta _{1}+q_1\eta _2$	$2q_1\eta _{1}+\eta _2$
(1, 0)	$2\eta _{1}+q_1\eta _{m-1}$	$\eta _{2}+2q_1\eta _m$
(0, 1)	$q_1\eta _{2}+2\eta _m$	$2q_1\eta _{1}+\eta _{m-1}$
(1, 1)	$q_1\eta _{m-1}+2\eta _m$	$\eta _{m-1}+2q_1\eta _m$

In all four cases, (9) implies that $q_1=1$.

Now $V|_{R^0}$ is non-homogeneous by Lemma 2.10. More precisely, $V_1|_{R^0}$ has a composition factor of highest weight $\mu _1$ as in the table (with $q_1=1$), occurring with multiplicity 1. If $\nu$ denotes the highest weight of any other composition factor of $V_1|_{R^0}$, then $\nu \ne \mu _1$ and $\nu \preccurlyeq \mu _1$ (so $\mu _1 - \nu = \sum _{i}c_i\alpha _i$ for some $c_i \in \mathbb {N}_0$). Therefore, Lemma 5.4 implies that $V|_{R^0}$ does not satisfy the homogeneity condition in (9) and this final contradiction completes the proof of the proposition. $\square$

Proposition 5.10

Suppose $H<M<G$ and (G, M, V) is the case labelled (iii) in Table 6. Then $V|_{H}$ is reducible.

Proof

Here $G=A_n$, $V=V_G(\lambda _k)$ with $1< k < n$, and $M=D_m.2$ is a $\mathcal {C}_6$-subgroup with $n=2m-1$, $m \geqslant 2$ and $p \ne 2$. Let $\{\eta _1, \ldots , \eta _m\}$ be a set of fundamental dominant weights for $M^0=D_m$. There are three separate cases to deal with here, depending on the value of k (by duality, we may assume that $2 \leqslant k \leqslant m$):

• (a) $k=m$: $V|_{M^0}=V_1 \oplus V_2$ is reducible, where $V_1$ and $V_2$ have highest weights $2\eta _{m-1}$ and $2\eta _m$, respectively (see [4, Table 3.2]).

• (b) $k=m-1$: $V|_{M^0}$ is irreducible, with highest weight $\eta _{m-1}+\eta _m$ (see case I$_5$ in [17, Table 1]). Note that $m \geqslant 3$.

• (c) $2 \leqslant k<m-1$: $V|_{M^0}$ is irreducible, with highest weight $\eta _{k}$ (see case I$_4$ in [17, Table 1]). Note that $m \geqslant 4$.

Seeking a contradiction, let us assume that $V|_{H}$ is irreducible.

First assume that (a) holds. Note that $H \not \leqslant M^0$ since $V|_{M^0}$ is reducible. To begin with, let us assume $m=2$. Here $G=A_3$ and $H^0<M^0=A_1A_1$, so

$$\begin{aligned} H^0 \in \{T_2, A_1, A_1T_1\} \end{aligned}$$

(recall that $H^0$ is reductive; see Lemma 2.7). Also note that $V=\Lambda ^2(W)$, where W is the natural KG-module. We claim that $H^0=A_1$. To see this, suppose $S \leqslant H^0$ is a central torus. Then $H \leqslant N_G(S)$ and thus the set of fixed points of S on V is H-invariant. But S lies in a maximal torus of $M^0$, which has nontrivial fixed points on V, so this contradicts the irreducibility of $V|_{H}$. This justifies the claim. Therefore

$$\begin{aligned} H \leqslant N_G(H^0) = H^0C_G(H^0) \end{aligned}$$

and thus $V|_{H^0}$ is homogeneous.

By considering the possible embeddings of $H^0$ in $M^0$, it follows that $W|_{H^0}$ is the two-fold tensor product $U\otimes U$, where U is the natural $KH^0$-module. Hence, $W|_{H^0} = W_1\oplus W_2$, where $W_1 = V_{H^0}(2\omega )$ and $W_2 = V_{H^0}(0\omega )$ is the trivial irreducible $KH^0$-module (here $\omega$ is the fundamental dominant weight for $H^0$). Since $\dim W_1 \ne \dim W_2$ we deduce that $W|_{N_G(H^0)}$ is reducible and thus $N_G(H^0)$ lies in a parabolic subgroup of G. This contradicts the irreducibility of $V|_{H}$.

Now assume $m \geqslant 3$. Set $H_1 = H \cap M^0$ and note that $H = H_1.2$, so $V|_{H_1}$ has exactly two composition factors, namely $V_1|_{H_1}$ and $V_2|_{H_1}$. Note that $H_1<M^0$ since we are assuming that H is disconnected and non-maximal. Let J be a maximal subgroup of $M^0$ that contains $H_1$, so we have

$$\begin{aligned} H_1 = H \cap M^0 \leqslant J < M^0 = D_m. \end{aligned}$$

We consider the irreducible triples $(M^0,J,V_1)$ and $(M^0,J,V_2)$, where $V_1 = V_{M^0}(2\eta _{m-1})$ and $V_2 = V_{M^0}(2\eta _{m})$. By inspecting [17, Table 1], and using the main theorems of [3, 4], we deduce that $J=B_{m-1}$ is the only possibility and

$$\begin{aligned} V_1|_{J} = V_2|_{J} = V_{B_{m-1}}(2\xi _{m-1}) \end{aligned}$$

(where $\{\xi _1, \ldots , \xi _{m-1}\}$ are fundamental dominant weights for $B_{m-1}$); see case IV$_1$ in [17, Table 1]. Note that if $H_1 = J = B_{m-1}$ then the two $KH^0$-composition factors of $V|_{H^0}$ (namely $V_1|_{H^0}$ and $V_2|_{H^0}$) are isomorphic, but this is ruled out by Proposition 2.9. Therefore $H_1$ is a proper subgroup of J, so let L be a maximal subgroup of J that contains $H_1$, in which case

$$\begin{aligned} H_1 = H \cap M^0 \leqslant L< J = B_{m-1} < M^0=D_m. \end{aligned}$$

We now consider the irreducible triple $(J,L,V_{B_{m-1}}(2\xi _{m-1}))$. In the usual way, by inspecting [3, 4, 17], we deduce that there are no compatible configurations and this completes the analysis of case (a).

Next consider case (b). First assume $H \leqslant M^0$. Since we are assuming H is disconnected and non-maximal, it follows that $H \leqslant J<M^0$ for some maximal subgroup J of $M^0$, and we may consider the irreducible triple $(M^0,J,V_{M^0}(\eta _{m-1}+\eta _m))$. By inspecting [3, 4, 17], it is easy to check that there are no compatible examples. In the same way, we deduce that $H \not \leqslant M^0$ in case (c).

Finally, let us consider cases (b) and (c), with $H \not \leqslant M^0$. Let J be a maximal subgroup of $M = D_m.2$ such that

$$\begin{aligned} H \leqslant J < M = D_m.2 = \mathrm{GO}(W). \end{aligned}$$

Note that J is disconnected, and J is either geometric or non-geometric (as described in Sect. 3.2). Given the highest weight of $V|_{M^0}$, we can rule out the latter possibility by applying [3, Theorem 3], so we may assume J is geometric. (Note that we can appeal to [3, Theorem 3] since $V|_{M^0}$ is irreducible.) The possibilities for J are determined in Proposition 3.2 and they are listed in Table 4. We now apply Proposition 3.3, which implies that $V|_{J}$ is reducible. This final contradiction completes the proof of the proposition. $\square$

Proposition 5.11

Suppose $H<M<G$ and (G, M, V) is one of the cases labelled (ix) or (xiii) in Table 6. Then $V|_{H}$ is reducible.

Proof

First consider the case labelled (xiii). Here $G=D_8$, $M=C_2^2.2$ is a $\mathcal {C}_4(ii)$-subgroup, $p \ne 5$ and $V=V_G(\lambda _7)$. Seeking a contradiction, let us assume that $V|_{H}$ is irreducible. We proceed as in the proof of Proposition 5.8.

Write $M^0=M_1M_2$, which is a central product of two simply connected groups of type $C_2$, and let $\{\omega _{1,1},\omega _{1,2}\}$ and $\{\omega _{2,1},\omega _{2,2}\}$ be fundamental dominant weights for $M_1$ and $M_2$, respectively. As recorded in [4, Table 6.2], we have $V|_{M^0} = V_1 \oplus V_2$, where

$$\begin{aligned} V_1 = V_{M_1}(\omega _{1,1}) \otimes V_{M_2}(\omega _{2,1}+\omega _{2,2}),\quad V_2 = V_{M_1}(\omega _{1,1}+\omega _{1,2}) \otimes V_{M_2}(\omega _{2,1}). \end{aligned}$$

Set $H_1 = H \cap M^0$, so $H=H_1.2$ and $H_1^0 = H^0$. Since $V|_{H}$ is irreducible it follows that $V|_{H_1}$ has exactly two composition factors, namely $V_1|_{H_1}$ and $V_2|_{H_1}$.

The $KM^0$-module W lifts to a representation $\rho :L \rightarrow \mathrm{GL}(W)$, where $L = L_1L_2$ is a direct product of two simply connected groups of type $C_2$, so $M^0 = L/Y$ where $Y = \ker (\rho )$. Since $H^0 \leqslant H_1 \leqslant M^0$, there exist subgroups $R \leqslant R_1 \leqslant L$ such that $H_1 = R_1/Y$ and $H^0 = R/Y$. Note that $R^0 = R_1^0$. Let $\pi _i : R_1 \rightarrow L_i$ be the i-th projection map and observe that $\ker (d\pi _1) \cap \ker (d\pi _2) = 0$. Since the $KM^0$-module $V_i$ lifts to a representation $\rho _i:L \rightarrow \mathrm{GL}(V_i)$, we can consider $V_{i}|_{R_1}$. The irreducibility of $V_i|_{H_1}$ implies that $V_i|_{R_1}$ is also irreducible, so we deduce that $\pi _1(R_1)$ acts irreducibly on the $KL_1$-modules $V_{L_1}(\omega _{1,1})$ and $V_{L_1}(\omega _{1,1}+\omega _{1,2})$, and similarly, $\pi _2(R_1)$ is irreducible on $V_{L_2}(\omega _{2,1})$ and $V_{L_2}(\omega _{2,1}+\omega _{2,2})$.

Claim. $\pi _1(R_1)$ and $\pi _2(R_1)$ are infinite.

We proceed as in the previous cases. Suppose $\pi _1(R_1)$ is finite. Then $\pi _2(R_1)$ is infinite, $\ker (d\pi _2)=0$ and $R_1^0 \leqslant \ker (\pi _1)$, so $\pi _2|_{R^0}: R^0 \rightarrow L_2$ is injective.

Suppose $\pi _2$ is not surjective. Then there exists a positive-dimensional maximal subgroup $J_2$ of $L_2$ such that $\pi _2(R_1) \leqslant J_2< L_2$, and we can consider the irreducible triples $(L_2,J_2, V_{L_2}(\omega _{2,1}))$ and $(L_2,J_2, V_{L_2}(\omega _{2,1}+\omega _{2,2}))$. By inspecting [3, 4, 17] we find that there are no compatible examples, which is a contradiction and thus $\pi _2$ is surjective. Therefore $\pi _2(R_1^0)=L_2$ and thus $\pi _2|_{R^0}: R^0 \rightarrow L_2$ is a bijective morphism. Since $\ker (d(\pi _2|_{R^0}))=0$ we deduce that $\pi _2|_{R^0}$ is an isomorphism of algebraic groups, so by Lemma 2.2 we can write $\pi _2|_{R^0}=t_{x}$ for some $x \in R^0$. If $\{\eta _1,\eta _2\}$ is a set of fundamental dominant weights for $R^0$, then $V|_{R^0}$ has composition factors isomorphic to $V_{R^0}(\eta _1)$ and $V_{R^0}(\eta _1+\eta _2)$, but $V|_{R^0}$ is homogeneous since $N_G(H^0) = H^0C_G(H^0)$, so this is a contradiction. We conclude that $\pi _1(R_1)$ is infinite, and similarly $\pi _2(R_1)$ is infinite.

Claim. $\pi _1$ and $\pi _2$ are surjective.

Suppose $\pi _1$ is not surjective. Then there exists a positive-dimensional maximal subgroup $J_1$ of $L_1$ such that $\pi _1(R_1) \leqslant J_1<L_1$, and we can consider the irreducible triples $(L_1,J_1, V_{L_1}(\omega _{1,1}))$ and $(L_1,J_1, V_{L_1}(\omega _{1,1}+\omega _{1,2}))$. As noted above, there are no compatible examples, so $\pi _1$ must be surjective and an entirely similar argument shows that $\pi _2$ is also surjective.

We have $\pi _i(R^0)=L_i$ for $i=1,2$, so $R^0$ is a subdirect product of $L=L_1L_2$ and thus Lemma 2.5 implies that either $R^0=L$, or $R^0 \cong L_1$ is simply connected and diagonally embedded in L. If $R^0=L$ then $H^0=M^0$ and thus $H=M$ (since H is disconnected), which is false. Therefore, $R^0 \cong L_1$ is diagonally embedded and thus

$$\begin{aligned} R^0=\{(\tau _1(x), \tau _2(x)) {\mid } x \in \mathrm{Sp}_{4}(K) \} \end{aligned}$$

where each $\tau _i: \mathrm{Sp}_{4}(K) \rightarrow L_i$ is a bijective morphism. By Lemma 2.2 we may write $\tau _i=t_{x_i}\sigma _{q_i}\gamma ^{k_i}$ for some $x_i \in R^0$, p-power $q_i$ and $k_i\in \{0,1\}$, where $\gamma$ is a graph automorphism of $C_2$ if $p=2$, otherwise $\gamma =1$. We may assume that $q_2=1$. Since $N_G(H^0) =H^0C_G(H^0)$, it follows that $V|_{R^0}$ is homogeneous.

As above, let $\{\eta _1,\eta _2\}$ be a set of fundamental dominant weights for $R^0$. First assume that $p \ne 2$, so $(k_1,k_2) = (0,0)$. Then $V|_{R^0}$ has composition factors with highest weights $(q_1+1)\eta _1+\eta _2$ and $(q_1+1)\eta _1+q_1\eta _2$, and the homogeneity of $V|_{R^0}$ implies that $q_1=1$. But Lemma 2.10 implies that $V|_{R^0}$ is non-homogeneous, so we have reached a contradiction.

Now assume $p=2$. Here $V|_{R^0}$ has composition factors with highest weights $\mu _1$ and $\mu _2$ as follows:

$(k_1,k_2)$	$\mu _1$	$\mu _2$
(0, 0)	$(q_1+1)\eta _1+\eta _2$	$(q_1+1)\eta _1+q_1\eta _2$
(1, 0)	$\eta _1+2(q_1+1)\eta _2$	$(q_1+1)\eta _1+q_1\eta _2$
(0, 1)	$(q_1+1)\eta _1+\eta _2$	$q_1\eta _1+2(q_1+1)\eta _2$
(1, 1)	$\eta _1+2(q_1+1)\eta _2$	$q_1\eta _1+2(q_1+1)\eta _2$

Since $V|_{R^0}$ is homogeneous, we deduce that $k_1=k_2$ and $q_1=1$.

If $(k_1,k_2)=(0,0)$ then $V_1|_{R^0} = V_{R^0}(\eta _1)\otimes V_{R^0}(\eta _1+\eta _2)$ and thus Lemma 2.10 contradicts the homogeneity of $V|_{R^0}$. Similarly, if $(k_1,k_2) = (1,1)$ then

$$\begin{aligned} V_1|_{R^0} = V_{R^0}(2\eta _2)\otimes V_{R^0}(\eta _1+2\eta _2)&\cong V_{R^0}(\eta _2)^{(2)} \otimes V_{R^0}(\eta _1)\otimes V_{R^0}(\eta _2)^{(2)} \\&\cong (V_{R^0}(\eta _2)\otimes V_{R^0}(\eta _2))^{(2)}\otimes V_{R^0}(\eta _1) \end{aligned}$$

Now $V_{R^0}(\eta _2)\otimes V_{R^0}(\eta _2)$ has composition factors with highest weights $2\eta _2$ and $2\eta _1$, whence $V_1|_{R^0}$ has composition factors with highest weights $\eta _1+4\eta _2$ and $5\eta _1$. This final contradiction completes the analysis of case (xiii) in Table 6.

The case labelled (ix) is similar (and easier). Here $G=B_4$, $V=V_G(\lambda _4)$ and $M=B_1^2.2$ is a $\mathcal {C}_4(ii)$-subgroup, where $p \ne 2,3$. The connected component $M^0=M_1M_2$ is a central product of two simply connected groups of type $B_1$, and we note that

$$\begin{aligned} V|_{M^0}=V_1 \oplus V_2 = (V_{M_1}(\omega _1) \otimes V_{M_2}(3\omega _2)) \oplus (V_{M_1}(3\omega _1) \otimes V_{M_2}(\omega _2)) \end{aligned}$$

(see [4, Table 6.2]), where $\omega _1$ and $\omega _2$ are fundamental dominant weights for $M_1$ and $M_2$, respectively. We leave the remaining details to the reader. $\square$

5.2 Proof of Theorem 5.1, part II

In order to complete the proof of Theorem 5.1, it remains to deal with the cases labelled (vii), (x), (xi) and (xii) in Table 6.

Remark 5.12

Suppose that $V|_{H}$ is irreducible, where $H<M<G$ and (G, M, V) is one of the cases (x), (xi) or (xii). Here M is the normalizer of a tensor product decomposition $W = W_1 \otimes \cdots \otimes W_t$ of the natural KG-module, with $t=3$ or 4. Therefore, by combining Lemma 5.3 with our earlier work in Sects. 4 and 5.1, we deduce that $W|_{H^0}$ is irreducible. Indeed, if $W|_{H^0}$ is reducible then Lemma 5.3 implies that we may replace M by some other geometric maximal subgroup of G that does not normalize such a decomposition, in which case our earlier work implies that $V|_{H}$ is reducible.

In order to deal with cases (x) and (xi), we first establish some preliminary reductions.

Lemma 5.13

Let $G=C_4$ and let $H<G$ be a closed positive-dimensional subgroup that is contained in a $\mathcal {C}_4(ii)$-subgroup $M=C_1^3.\mathrm{Sym}_3$ of G. Set $V=V_G(\lambda )$, where $\lambda = \lambda _2$ and $p \ne 2$, or $\lambda =\lambda _3$ and $p \ne 3$. If $V|_{H}$ is irreducible, then $H^0$ is a subdirect product of $M^0=C_1^3$.

Proof

Here $M^0 = C_1^3=M_1M_2M_3$ is a central product of three simply connected groups of type $C_1$. Let $\omega _i$ be the fundamental dominant weight for $M_i$, and note that

$$\begin{aligned} V|_{M^0} = V_1 \oplus V_2 \oplus V_3, \end{aligned}$$

(10)

where

$$\begin{aligned} \begin{aligned} V_1&= V_{M_1}(2\omega _{1}) \otimes V_{M_2}(2\omega _{2})\otimes V_{M_3}(0\omega _3)\\ V_2&= V_{M_1}(2\omega _{1}) \otimes V_{M_2}(0\omega _{2})\otimes V_{M_3}(2\omega _3)\\ V_3&= V_{M_1}(0\omega _{1}) \otimes V_{M_2}(2\omega _{2})\otimes V_{M_3}(2\omega _3) \end{aligned} \end{aligned}$$

(11)

if $\lambda =\lambda _2$, and

$$\begin{aligned} \begin{aligned} V_1&= V_{M_1}(\omega _{1}) \otimes V_{M_2}(\omega _{2})\otimes V_{M_3}(3\omega _3)\\ V_2&= V_{M_1}(\omega _{1}) \otimes V_{M_2}(3\omega _{2})\otimes V_{M_3}(\omega _3)\\ V_3&= V_{M_1}(3\omega _{1}) \otimes V_{M_2}(\omega _{2})\otimes V_{M_3}(\omega _3) \end{aligned} \end{aligned}$$

(12)

if $\lambda =\lambda _3$ (see [4, Table 6.2]). As noted in Remark 5.12, the irreducibility of $V|_{H}$ implies that $W|_{H^0}$ is irreducible, where W denotes the natural KG-module.

As a $KM^0$-module, W lifts to a representation $\rho : L \rightarrow \mathrm{GL}(W)$, where $L=L_1L_2L_3$ is the direct product of three simply connected groups of type $C_1$. Then $M^0 = L/Y$, where $Y = \ker (\rho )$, and there exists a subgroup R of L such that $H^0=R/Y$.

We need to show that $R^0$ is a subdirect product of L. Note that the irreducibility of $W|_{H^0}$ implies that $W|_{R^0}$ is also irreducible. Set $J=[R^0,R^0]$ and recall that $H^0$ is reductive (see Lemma 2.7), so $R^0$ is reductive and thus

$$\begin{aligned} J \in \{C_1^3, C_1^2,C_1,1\}. \end{aligned}$$

If $J=C_1^3$ then we are done, so assume otherwise. If $J=1$ then $R^0$ is a torus, contradicting the irreducibility of $W|_{R^0}$. Finally, suppose $J=C_1^2$ or $C_1$. Since $R^0$ is the product of J and a central torus, the irreducibility of $W|_{R^0}$ implies that $W|_{J}$ is irreducible. This immediately implies that the projection maps $\pi _i:J \rightarrow L_i$ are surjective, so $R^0$ is a subdirect product of L as required. $\square$

Lemma 5.14

Suppose $H<M<G$ and (G, M, V) is the case labelled (x) in Table 6, where $V=V_G(\lambda _2)$ and $p \ne 2$. If $V|_{H}$ is irreducible, then $H^0=M^0$.

Proof

As in the previous lemma, $G=C_4$ and $M^0 = C_1^3=M_1M_2M_3$ is a central product of simply connected groups of type $C_1$. Define $L=L_1L_2L_3$, Y and R as above, so $Y \leqslant Z(L)$, $M^0=L/Y$ and $H^0 = R/Y$. Note that $V|_{M^0} = V_1 \oplus V_2 \oplus V_3$, where the $V_i$ are given in (11). Also recall that the irreducibility of $V|_{H}$ implies that $W|_{H^0}$ is also irreducible (see Remark 5.12).

By Lemma 5.13, R is a subdirect product of $L_1L_2L_3$, so Proposition 2.6 implies that $R^0$ is isomorphic to a commuting product of simple groups of type $C_1$. If $R^0$ is of type $C_1^3$ then $H^0=M^0$ and we are done, so let us assume that $R^0$ is of type $C_1$ or $C_1^2$.

Suppose $R^0$ is of type $C_1$. Let $\eta _1$ be the fundamental dominant weight for $R^0$. By Proposition 2.6, $R^0 \cong L_1$ is simply connected and diagonally embedded in L, so we may write

$$\begin{aligned} R^0=\{(\tau _1(x), \tau _2(x), \tau _3(x)) {\mid } x \in \mathrm{Sp}_{2}(K)\} \end{aligned}$$

where $\tau _i: \mathrm{Sp}_{2}(K)\rightarrow L_i$ is a bijective morphism. By Lemma 2.2, $\tau _i=t_{x_i}\sigma _{q_i}$ for some $x_i \in R^0$ and p-power $q_i$, and we may assume that $q_3=1$ (since $R^0$ is a closed subgroup of L). Then $V|_{R^0}$ has composition factors with highest weights $(2q_1+2q_2)\eta _1$, $(2q_1+2)\eta _1$ and $(2q_2+2)\eta _1$. Since $V|_{R^0}$ is homogeneous (note that $N_G(H^0) = H^0C_G(H^0)$), it follows that $q_1=q_2=1$. But now Lemma 2.10 contradicts the homogeneity of $V|_{R^0}$.

Finally, let us assume that $R^0 = R_1R_2$ is of type $C_1^2$. Let $\{\eta _{1}, \eta _{2}\}$ be fundamental dominant weights for $R^0$. Once again, Proposition 2.6 implies that $R_1$ and $R_2$ are simply connected groups of type $C_1$, and without loss of generality we may assume that

$$\begin{aligned} R_1=\{(\tau _{1}(x),1,1) {\mid } x \in \mathrm{Sp}_2(K)\},\quad R_2=\{(1,\tau _{2}(x),\tau _{3}(x)) {\mid } x \in \mathrm{Sp}_2(K)\} \end{aligned}$$

where $\tau _{i}: \mathrm{Sp}_2(K)\rightarrow L_i$ is a bijective morphism. As before, we may write $\tau _{i}=t_{x_{i}}\sigma _{q_{i}}$, so $V|_{R^0}$ has composition factors with highest weights

$$\begin{aligned} 2q_{1}\eta _1+2q_{2}\eta _2, \quad 2q_{1}\eta _1+2q_{3}\eta _2, \quad 2(q_{2}+q_{3})\eta _2. \end{aligned}$$

(13)

Since $R^0$ is a closed subgroup of L, at least one $q_i$ is equal to 1. By considering $N_G(H^0)$, it follows that $V|_{R^0}$ is either homogeneous, or the homogeneous components of $V|_{R^0}$ are conjugate under an involutory automorphism of $R^0$ interchanging $R_1$ and $R_2$. However, this observation is incompatible with the weights recorded in (13). This is a contradiction. $\square$

The next lemma gives an analogous reduction for $V=V_G(\lambda _3)$ in case (x) in Table 6. Note that we include the additional case $p=2$, which will be needed in Propositions 5.17 and 5.19.

Lemma 5.15

Suppose $H<M<G$ with $G=C_4$, $M = C_1^3.\mathrm{Sym}_3$ and $p \ne 3$. Set $V=V_G(\lambda _3)$ and assume that $V|_{H}$ is irreducible. Then $H^0=M^0$.

Proof

This is entirely similar to the proof of Lemma 5.14, and we omit the details. In particular, we note that there are no additional difficulties when $p=2$. $\square$

We are now in a position to settle cases (x) and (xi).

Proposition 5.16

Suppose $H<M<G$ and (G, M, V) is the case labelled (x) in Table 6, so $G=C_4$, $M=C_1^3.\mathrm{Sym}_3$ and $V=V_G(\lambda )$, where $\lambda =\lambda _2$ or $\lambda _3$. Then $V|_{H}$ is irreducible if and only if $H=C_1^3.Z_3$.

Proof

If $V|_{H}$ is irreducible, then Lemmas 5.14 and 5.15 imply that $H^0=M^0$. Therefore, H transitively permutes the $V_i$ in (10), so $H/H^0 \leqslant \mathrm{Sym}_3$ is transitive. Since $H<M$, we conclude that $H=C_1^3.Z_3$ is the only possibility. $\square$

Proposition 5.17

Suppose $H<M<G$ and (G, M, V) is the case labelled (xi) in Table 6, so $G=D_4$, $M=C_1^3.\mathrm{Sym}_3$, $p=2$ and $V=V_G(\lambda )$, where $\lambda \in \{\lambda _1+\lambda _4, \lambda _3+\lambda _4\}$. Then $V|_{H}$ is irreducible if and only if $H=C_1^3.Z_3$.

Proof

Write $M^0=C_1^3=M_1M_2M_3$ and let $\omega _i$ be the fundamental dominant weight for $M_i$. Note that

$$\begin{aligned} H< M = C_1^3.\mathrm{Sym}_3< G=D_4< N = C_4. \end{aligned}$$

Suppose that $V|_{H}$ is irreducible.

First consider the case $\lambda =\lambda _3+\lambda _4$. Here $V|_{M^0} = V_1 \oplus V_2 \oplus V_3$ and (12) holds (see [4, Table 6.2]). Let $\{\xi _1,\xi _2,\xi _3,\xi _4\}$ be fundamental dominant weights for N. Since V is the restriction of the KN-module $V_N(\xi _3)$ to G (see the case labelled $\mathrm{MR}_{4}$ in [17, Table 1]), Lemma 5.15 implies that $H^0=M^0$ and thus $H=C_1^3.Z_3$ is the only possibility. An entirely similar argument applies if $\lambda =\lambda _1+\lambda _4$, and once again we deduce that $H=C_1^3.Z_3$. $\square$

To complete the proof of Theorem 5.1 it remains to consider cases (vii) and (xii) in Table 6. First we establish an important reduction for case (vii).

Lemma 5.18

Suppose $H<M<G$ and (G, M, V) is the case labelled (vii) in Table 6, so $G=C_n$, $M=D_n.2$, $p=2$ and $\lambda =\sum _{i=1}^{n-1}a_i\lambda _i$. If $V|_{H}$ is irreducible, then $n=4$, $\lambda = \lambda _3$ and $H < M^0$.

Proof

Here $M=D_n.2$ is a $\mathcal {C}_6$-subgroup of G, where $G=C_n$, $n \geqslant 2$ and $p=2$. Set $V=V_G(\lambda )$ and let $\{\eta _1, \ldots , \eta _n\}$ be a set of fundamental dominant weights for $M^0=D_n$. We have $\lambda =\sum _{i=1}^{n-1}a_i\lambda _i$ and $V|_{M^0}$ is irreducible with highest weight

$$\begin{aligned} \mu = \sum _{i=1}^{n-2}a_i\eta _i+a_{n-1}(\eta _{n-1}+\eta _n) \end{aligned}$$

(see case MR$_4$ in [17, Table 1]).

Note that $n \geqslant 3$ since we are assuming that $\lambda$ is nontrivial, p-restricted and $V \ne W$. Suppose that $V|_{H}$ is irreducible.

First assume $H \not \leqslant M^0$. As in the proof of Proposition 5.10, let J be a maximal subgroup of $M=D_n.2=\mathrm{GO}(W)$ containing H. Then J is disconnected, (M, J, V) is an irreducible triple, and $V|_{M^0} = V_{M^0}(\mu )$. By applying [3, Theorem 3], we deduce that J is a geometric subgroup of M, so the possibilities for J are listed in Table 4. By applying Proposition 3.4, we conclude that $V|_{J}$ is reducible, which is a contradiction.

Now suppose $H \leqslant M^0$. Let J be a maximal subgroup of $M^0$ containing H (note that $H <M^0$ since we are assuming H is disconnected). Then $V|_{J}$ is irreducible and we can consider the possibilities for the irreducible triple $(M^0,J,V|_{M^0})$. By inspecting [3, 4, 17], given the highest weight of $V|_{M^0}$, we quickly deduce that $n=4$ is the only possibility (note that if $n=3$ then the highest weight of $V|_{M^0}$ has at least two non-zero coefficients and it is easy to check that there are no compatible examples), $J=C_1^3.\mathrm{Sym}_3$ is a $\mathcal {C}_4(ii)$-subgroup of $M^0=D_4$ and $V|_{M^0} = V_{M^0}(\eta _3+\eta _4)$, so $\lambda =\lambda _3$. $\square$

Proposition 5.19

Suppose $H<M<G$ and (G, M, V) is the case labelled (vii) in Table 6, so $G=C_n$, $M=D_n.2$, $p=2$ and $\lambda =\sum _{i=1}^{n-1}a_i\lambda _i$. Then $V|_{H}$ is irreducible if and only if $n=4$, $\lambda =\lambda _3$ and $H= C_1^3.Z_3$ or $C_1^3.\mathrm{Sym}_3$.

Proof

Suppose that $V|_{H}$ is irreducible. By the proof of the previous lemma, $n=4$, $\lambda = \lambda _3$ and

$$\begin{aligned} H \leqslant J = C_1^3.\mathrm{Sym}_3< M^0 = D_4 < G = C_4. \end{aligned}$$

In addition, if $\{\eta _1, \ldots , \eta _4\}$ are fundamental dominant weights for $M^0 = D_4$, then $V|_{M^0} = V_{M^0}(\eta _3+\eta _4)$. Therefore, we have now reduced the problem to the case numbered (xi) in Table 6, which was handled in Proposition 5.17. In particular, we conclude that $H=C_1^3.Z_3$ or $C_1^3.\mathrm{Sym}_3$, as required. $\square$

Remark 5.20

As noted in Remark 5.2, Proposition 5.19 implies that $V|_{H}$ is irreducible in case (v) if and only if $G=B_4$, $p=2$, $\lambda =\lambda _3$ and $H=B_1^3.Z_3$ or $B_1^3.\mathrm{Sym}_3$.

To complete the proof of Theorem 5.1, it remains to deal with the case labelled (xii) in Table 6.

Proposition 5.21

Suppose $H<M<G$ and (G, M, V) is the case labelled (xii) in Table 6, so $G=D_8$, $M=C_1^4.\mathrm{Sym}_4$, $p \ne 3$ and $V=V_G(\lambda )$, where $\lambda =\lambda _7$. Then $V|_{H}$ is irreducible if and only if $H=C_1^4.X$, where $X < \mathrm{Sym}_4$ is transitive.

Proof

Write $M^0 = C_1^4=M_1M_2M_3M_4$, which is a central product of simply connected groups of type $C_1$, and let $\omega _{i}$ be the fundamental dominant weight for $M_i$. Then [4, Table 6.2] indicates that $V|_{M^0} = V_1 \oplus V_2 \oplus V_3\oplus V_4$, where

$$\begin{aligned} \begin{aligned} V_1&= V_{M_1}(\omega _{1}) \otimes V_{M_2}(\omega _{2})\otimes V_{M_3}(\omega _3)\otimes V_{M_4}(3\omega _{4})\\ V_2&= V_{M_1}(\omega _{1}) \otimes V_{M_2}(\omega _{2})\otimes V_{M_3}(3\omega _3)\otimes V_{M_4}(\omega _{4})\\ V_3&= V_{M_1}(\omega _{1}) \otimes V_{M_2}(3\omega _{2})\otimes V_{M_3}(\omega _3)\otimes V_{M_4}(\omega _{4})\\ V_4&= V_{M_1}(3\omega _{1}) \otimes V_{M_2}(\omega _{2})\otimes V_{M_3}(\omega _3)\otimes V_{M_4}(\omega _{4}) \end{aligned} \end{aligned}$$

(14)

Assume that $V|_{H}$ is irreducible, so $W|_{H^0}$ is also irreducible (see Remark 5.12), where W is the natural KG-module.

Since W is a $KM^0$-module, it lifts to a representation $\rho : L \rightarrow \mathrm{GL}(W)$, where $L=L_1L_2L_3L_4$ is the direct product of simply connected groups of type $C_1$. Then $M^0 = L/Y$, where $Y = \ker (\rho )$, and there exists a subgroup R of L such that $H^0=R{/}Y$.

Claim. $H^0$ is a subdirect product of $M^0$.

We need to show that $R^0$ is a subdirect product of L. To do this, we proceed as in the proof of Lemma 5.13; the argument is very similar (using the irreducibility of $W|_{H^0}$) and we omit the details.

Claim. $H^0=M^0$.

Since $R^0$ is a subdirect product of L, Proposition 2.6 implies that $R^0$ is isomorphic to a commuting product of simply connected groups of type $C_1$. If $R^0$ is of type $C_1^4$ then $H^0=M^0$, so we may assume that $R^0$ is of type $C_1, C_1^2$ or $C_1^3$.

Suppose $R^0$ is of type $C_1$. Let $\eta$ be the fundamental dominant weight for $R^0$. By Proposition 2.6, we may write

$$\begin{aligned} R^0=\{(\tau _1(x), \tau _2(x), \tau _3(x), \tau _4(x)) {\mid } x \in \mathrm{Sp}_{2}(K) \}, \end{aligned}$$

where $\tau _i: \mathrm{Sp}_{2}(K) \rightarrow L_i$ is a bijective morphism. As before, Lemma 2.2 implies that $\tau _i=t_{x_i}\sigma _{q_i}$ for some $x_i \in R^0$ and p-power $q_i$, and we may assume that $q_4=1$. Then $V|_{R^0}$ has composition factors with highest weights

$$\begin{aligned}&\displaystyle (q_1+q_2+q_3+3)\eta , \quad (q_1+q_2+3q_3+1)\eta ,\\&\displaystyle (q_1+3q_2+q_3+1)\eta ,\quad (3q_1+q_2+q_3+1)\eta . \end{aligned}$$

Since $N_G(H^0) = H^0C_G(H^0)$, it follows that $V|_{R^0}$ is homogeneous, so $q_i=1$ for all i. But Lemma 2.10 implies that $V|_{R^0}$ is non-homogeneous, so we have reached a contradiction. Note that if $p=2$ then one of the tensor factors in $V_i|_{R^0}$ is non-restricted, but we can still argue as in the proof of Lemma 2.10 by considering the tensor product of the restricted factors.

Next suppose $R^0=R_1R_2$, where each $R_i \cong C_1$ is simply connected. Let $\{\eta _{1}, \eta _{2}\}$ be fundamental dominant weights for $R^0$. In view of Proposition 2.6, we may assume that either

$$\begin{aligned} \begin{aligned} R_1&=\{(\tau _{1}(x),1,1,1) {\mid } x \in \mathrm{Sp}_2(K)\},\\ R_2&=\{(1,\tau _{2}(x),\tau _{3}(x),\tau _{4}(x)) {\mid } x \in \mathrm{Sp}_2(K)\} \end{aligned} \end{aligned}$$

(15)

or

$$\begin{aligned} \begin{aligned} R_1&=\{(\tau '_{1}(x),\tau '_{2}(x),1,1) {\mid } x \in \mathrm{Sp}_2(K)\},\\ R_2&=\{(1,1,\tau '_{3}(x),\tau '_{4}(x)) {\mid } x \in \mathrm{Sp}_2(K)\} \end{aligned} \end{aligned}$$

(16)

where $\tau _i, \tau _i'$ are bijective morphisms from $\mathrm{Sp}_{2}(K)$ to $L_i$. Note that $V|_{R^0}$ is either homogeneous, or the homogeneous components of $V|_{R^0}$ are conjugate under an involutory automorphism of $R^0$ that interchanges $R_1$ and $R_2$.

First assume (15) holds. As above, we may write $\tau _{i}=t_{x_{i}}\sigma _{q_{i}}$, so $V|_{R^0}$ has composition factors with highest weights $q_{1}\eta _1+ (q_{2}+q_{3}+3q_{4})\eta _2$, $q_{1}\eta _1+ (q_{2}+3q_{3}+q_{4})\eta _2$, $q_{1}\eta _1+(3q_{2}+q_{3}+q_{4})\eta _2$ and $3q_{1}\eta _1+ (q_{2}+q_{3}+q_{4})\eta _2$, and at least one $q_i$ is equal to 1. But this contradicts the above homogeneity properties of $V|_{R^0}$. Similarly, suppose (16) holds and write $\tau _{i}'=t_{x_{i}'}\sigma _{q_{i}'}$, where at least one $q_i'$ is equal to 1. Then $V|_{R^0}$ has composition factors with highest weights $(q'_{1}+q'_{2})\eta _1 +(q'_{3}+3q'_4)\eta _2$, $(q'_{1}+q'_{2})\eta _1+ (3q'_{3}+q'_4)\eta _2$, $(q'_{1}+3q'_{2})\eta _1+ (q'_{3}+q'_4)\eta _2$ and $(3q'_{1}+q'_{2})\eta _1+ (q'_{3}+3q'_4)\eta _2$, and we deduce that $q'_i=1$ for all i. Then $V|_{R^0}$ has composition factors with highest weights $2\eta _1+4\eta _2$ and $2\eta _1+ 2\eta _2$, but once again this is incompatible with the observed homogeneity properties of $V|_{R^0}$.

Finally, let us assume $R^0 = R_1R_2R_3$ is of type $C_1^3$. Let $\{\eta _{1}, \eta _{2},\eta _3\}$ be fundamental dominant weights for $R^0$. In view of Proposition 2.6, we may assume that

$$\begin{aligned} R_1&= \{(\tau _{1}(x),\tau _{2}(x),1,1) {\mid } x \in \mathrm{Sp}_2(K)\} \\ R_2&= \{(1,1,\tau _{3}(x),1) {\mid } x \in \mathrm{Sp}_2(K)\} \\ R_3&= \{(1,1,1,\tau _{4}(x)) {\mid } x \in \mathrm{Sp}_2(K)\} \end{aligned}$$

where $\tau _{i}=t_{x_{i}}\sigma _{q_{i}}$ as above. Note that $V|_{R^0}$ is either homogeneous, or any two homogeneous components of $V|_{R^0}$ are conjugate via an automorphism of $R^0$ of order 2 or 3, induced by a suitable permutation of the three factors of $R^0$. However, $V|_{R^0}$ has composition factors with highest weights $(q_{1}+q_{2})\eta _1+ q_{3}\eta _2+ 3q_{4}\eta _3$, $(q_{1}+q_{2})\eta _1+ 3q_{3}\eta _2+ q_{4}\eta _3$, $(q_{1}+3q_{2})\eta _1+ q_{3}\eta _2+ q_{4}\eta _3$ and $(3q_{1}+q_{2})\eta _1+ q_{3}\eta _2+ q_{4}\eta _3$, so $V|_{R^0}$ does not have the stated homogeneity property. This is a contradiction.

We have now reduced to the case $H^0=M^0=C_1^4$. From the above description of $V|_{M^0}$ it is clear that $V|_{H}$ is irreducible if and only if $H=C_1^4.X$, where $X < \mathrm{Sym}_4$ is transitive. The result follows.$\square$

This completes the proof of Theorem 5.1.

6 Non-geometric subgroups

In order to complete the proof of Theorem 1.4, it remains to determine the irreducible triples (G, H, V) satisfying Hypothesis 1.1, where $V|_{H^0}$ is reducible and H is not contained in a geometric subgroup of G. Here the latter condition implies that H is one of the non-geometric subgroups that arise in part (ii) of Theorem 3.1 in Sect. 3. In particular, $W|_{H^0}$ is irreducible and tensor indecomposable, so we can apply the main theorem of [3]. (Note that if $(G,p)=(C_n,2)$ and H fixes a non-degenerate quadratic form on W, then H is contained in a geometric $\mathcal {C}_6$-subgroup $D_n.2<C_n$, which is a situation we dealt with in Proposition 5.19.)

Theorem 6.1

Let G, H and $V=V_G(\lambda )$ be given as in Hypothesis 1.1, and assume that H is not contained in a geometric subgroup of G. Then $V|_{H}$ is reducible.

Proof

By [3, Theorem 1], the only possibility is the case $(G,H,\lambda )=(C_{10},A_5.2, \lambda _3)$ with $p \ne 2,3$. However, we claim that H is a maximal subgroup, so this example does not satisfy the conditions in Hypothesis 1.1. To see this, let $\{\eta _1, \ldots , \eta _5\}$ be a set of fundamental dominant weights for $H^0$ and note that $W=V_{H^0}(\eta _3)$ (where W is the natural KG-module). Suppose H is non-maximal, say $H<M<G$ with M maximal. Then M is non-geometric, so $M^0$ is a simple group that acts irreducibly on W and thus $(M^0,H^0,W)$ is an irreducible triple. By inspecting [17, Table 1], we see that there are no compatible examples. This is a contradiction, so H is maximal as claimed. $\square$

In view of Theorems 4.5, 5.1 and 6.1, the proof of Theorem 1.4 is complete.

7 Spin modules

In this section, we briefly consider the special case arising in part (b) of Theorem 1.4, where G is a simply connected group of type $B_n$ or $D_n$ (or type $C_n$ if $p=2$), V is a spin module and H is a decomposition subgroup of G, as defined in the introduction. Recall that H normalizes an orthogonal decomposition

$$\begin{aligned} W = W_1 + \cdots + W_t \end{aligned}$$

of the natural KG-module W, where the $W_i$ are pairwise orthogonal subspaces.

Our goal here is simply to highlight the difference between this very specific situation and the general case we have studied in Sects. 4, 5 and 6. We will do this by establishing a preliminary result (see Proposition 7.4); a detailed analysis of spin modules and decomposition subgroups will be given in a forthcoming paper.

Let G be a simply connected simple algebraic group of type $B_n$ or $D_n$ over an algebraically closed field K of characteristic p. For convenience, we will assume that $p \ne 2$. Let W be the natural KG-module. As before, fix a set of simple roots $\{\alpha _1, \ldots ,\alpha _n\}$ and fundamental dominant weights $\{\lambda _1, \ldots , \lambda _n\}$ for G. We will assume that $n \geqslant 3$ if $G=B_n$ and $n \geqslant 5$ if $G=D_n$ (note that the spin modules for $D_4$ are excluded in Hypothesis 1.1; see Remark 1.2). We may write $\overline{G} = G/Z = \mathrm{SO}(W)$, where $Z \leqslant Z(G)$. Similarly, for a subgroup J of G we set $\overline{J} = JZ/Z \leqslant \overline{G}$.

Let V be a spin module for G. In terms of highest weights, either $V=V_{G}(\lambda _n)$, or $G=D_n$ and $V=V_G(\lambda _{n-1})$ (in the latter case, note that $V_G(\lambda _{n-1})=V_G(\lambda _{n})^{\tau }$, where $\tau$ is a graph automorphism of G). The next result is well known (see [4, Lemma 2.3.2] for a proof).

Lemma 7.1

$\dim V_{B_n}(\lambda _n)=2^n$ and $\dim V_{D_n}(\lambda _{n-1}) = \dim V_{D_n}(\lambda _{n}) = 2^{n-1}$.

Let $W=W_1 \perp W_2$ be an orthogonal decomposition, where $W_1$ and $W_2$ are non-degenerate subspaces with $\dim W_i \geqslant 3$. Let H be the stabilizer in G of this decomposition, so $H = H^0.2$ and $H^0$ is a central product of two simply connected orthogonal groups. More precisely,

$$\begin{aligned} \overline{H} = \overline{G}_{W_1} \cap \overline{G}_{W_2} = (\mathrm{GO}(W_1) \times \mathrm{GO}(W_2)) \cap \overline{G} = \overline{H}^0.2 \end{aligned}$$

and $\overline{H}^0 = \mathrm{SO}(W_1) \times \mathrm{SO}(W_2)$ is semisimple.

Proposition 7.2

Let V be a spin module for G and let H be the stabilizer in G of an orthogonal decomposition $W = W_1 \perp W_2$ as above. Then $V|_{H}$ is irreducible. Moreover, each $KH^0$-composition factor of V is a tensor product of spin modules for both orthogonal factors of $H^0$.

Proof

If $G=B_n$ then H / Z(G) is a disconnected subgroup in the collection $\mathcal {C}_1$ of geometric maximal subgroups of G, and the result follows immediately from [4, Proposition 3.1.1]. Now assume $G=D_n$. If $W_1$ is odd-dimensional, then H / Z(G) is connected and Seitz’s main theorem [17, Theorem 1] implies that $V|_{H^0}$ is an irreducible tensor product of appropriate spin modules (see the cases labelled $\mathrm{IV}_{1}$ (with $k=1$) and $\mathrm{IV}_{2}$ in [17, Table 1]). Finally, suppose $W_1$ is even-dimensional. If $\dim W_1 \ne n$, then H / Z(G) is a disconnected $\mathcal {C}_1$-subgroup and [4, Proposition 3.1.1] applies. If $\dim W_1 = n$ is even then H is properly contained in a $\mathcal {C}_2$-subgroup of G (namely, the full normalizer in G of the orthogonal decomposition); the proof of [4, Lemma 3.2.3] goes through unchanged, and the result follows. $\square$

Now consider an orthogonal decomposition of the form

$$\begin{aligned} W = W_1 \perp \cdots \perp W_t, \end{aligned}$$

(17)

where $t \geqslant 2$ and each $W_i$ is a non-degenerate subspace with $\dim W_i \geqslant 3$. Let H be the stabilizer in G of this decomposition, so

$$\begin{aligned} \overline{H} = \bigcap _{i=1}^{t}\overline{G}_{W_i} = (\mathrm{GO}(W_1) \times \cdots \times \mathrm{GO}(W_t)) \cap \overline{G} = \overline{H}^0.2^{t-1} \end{aligned}$$

If $t=2$ then $V|_{H}$ is irreducible by Proposition 7.2, so let us assume $t \geqslant 3$. We claim that $V|_{H}$ is still irreducible. To see this, we first handle the special case where the $W_i$ are equidimensional.

Lemma 7.3

If $\dim W_i = d \geqslant 3$ for all i, then $V|_{H}$ is irreducible.

Proof

If $d=2l+1$ is odd, then $H = 2^{t-1} \times B_l^t$ and $H.\mathrm{Sym}_t$ is a $\mathcal {C}_2$-subgroup of G. Here the proof of [4, Lemma 4.3.2] goes through unchanged (the symmetric group $\mathrm{Sym}_t$ in the $\mathcal {C}_2$-subgroup plays no role in the argument) and we deduce that $V|_{H}$ is irreducible.

Now assume $d=2l$ is even, so $H=D_l^t.2^{t-1}$ and we may write $H^0 = X_1 \cdots X_t$, where each $X_i = D_l$ is simply connected. Here the elementary abelian 2-group $2^{t-1}$ is generated by involutions $z_1, \ldots , z_{t-1}$, where $z_i$ acts as a graph automorphism on $X_i$ and $X_{i+1}$, and centralizes the remaining factors of $H^0$. Now $V_1 \otimes \cdots \otimes V_t$ is a composition factor of $V|_{H^0}$, where each $V_i$ is a spin module for $X_i$. By repeatedly applying the $z_i \in H$ to conjugate this composition factor, we deduce that $V|_{H^0}$ has at least $2^{t-1}$ distinct, H-conjugate $KH^0$-composition factors. Since

$$\begin{aligned} 2^{t-1} \cdot \dim (V_1 \otimes \cdots \otimes V_t) = 2^{t-1} \cdot 2^{t(l-1)} = 2^{tl-1} = \dim V \end{aligned}$$

(see Lemma 7.1) we conclude that $V|_{H}$ is irreducible. $\square$

We can now establish our main result for spin modules and decomposition subgroups.

Proposition 7.4

Let H be the stabilizer in G of the decomposition in (17), and assume $\dim W_i \geqslant 3$ for each i. Then $V|_{H}$ is irreducible.

Proof

Let $d_1, \ldots , d_s$ be the distinct dimensions of the summands in (17), and let $a_i$ be the number of summands of dimension $d_i$. If $s>1$ then we may assume that $d_i < d_{i+1}$ for all $1 \leqslant i <s$. We may re-order and re-label the subspaces in (17) so that

$$\begin{aligned} W = (W_{1,1} \perp \cdots \perp W_{1,a_1}) \perp \cdots \perp (W_{s,1} \perp \cdots \perp W_{s,a_s}), \end{aligned}$$

(18)

where $\dim W_{i,j} = d_i$ for all i, j. Then

$$\begin{aligned} \overline{H} = (\mathrm{GO}(W_{1,1}) \times \cdots \times \mathrm{GO}(W_{1,a_1}) \times \cdots \times \mathrm{GO}(W_{s,1}) \times \cdots \times \mathrm{GO}(W_{s,a_s})) \cap \overline{G}. \end{aligned}$$

We proceed by induction on s.

The base case $s=1$ was handled in Lemma 7.3, so let us assume $s>1$. Set

$$\begin{aligned} U_1 = (W_{1,1} \perp \cdots \perp W_{1,a_1}) \perp \cdots \perp (W_{s-1,1} \perp \cdots \perp W_{s-1,a_{s-1}}) \end{aligned}$$

and

$$\begin{aligned} U_2 = W_{s,1} \perp \cdots \perp W_{s,a_s}, \end{aligned}$$

so $W = U_1 \perp U_2$ and $H \leqslant M$, where

$$\begin{aligned} \overline{M} = (\mathrm{GO}(U_1) \times \mathrm{GO}(U_2)) \cap \overline{G} = \overline{M}^0.2. \end{aligned}$$

Write $M^0 = M_1M_2$, where $M_1$ and $M_2$ are simply connected orthogonal groups with natural modules $U_1$ and $U_2$, respectively. Let $H_1 = H \cap M^0$ and note that $H=H_1.2 \not \leqslant M^0$.

Here $M^0=M_1M_2$ is a central product. The $KM^0$-module W lifts to a representation $\rho : L \rightarrow \mathrm{GL}(W)$, where $L=L_1 \times L_2$ is the direct product of two simply connected orthogonal groups with $L_i \cong M_i$, so $M^0 = L/Y$ where $Y = \ker (\rho )$. In particular, there exist subgroups $R \leqslant R_1 \leqslant L$ such that $H_1=R_1/Y$ and $H^0=R/Y$. Note that $H_1/H^0 \cong R_1/R$ and $R^0=R_1^0$ (since $H^0 = H_1^0$). Let $\pi _i: R_1 \rightarrow L_i$ be the i-th projection map and note that $R_1 = \pi _1(R_1) \times \pi _2(R_1)$. There are several cases to consider.

First assume $U_1$ and $U_2$ are odd-dimensional. Here [17, Table 1] indicates that $V|_{M^0} = V_1 \otimes V_2$ is irreducible, where $V_i$ is the spin module for $M_i$. The $KM^0$-module V lifts to a representation $\varphi :L \rightarrow \mathrm{GL}(V)$, so we can consider $V|_{R_1}$. By induction, $V_i|_{\pi _i(R_1)}$ is irreducible for $i=1,2$, so $V|_{R_1}$ is irreducible and thus $V|_{H_1}$ is also irreducible. The result follows.

Next suppose $\dim U_1$ is even and $\dim U_2$ is odd, so [4, Proposition 3.1.1] implies that $V|_{M^0}$ has exactly two composition factors, namely

$$\begin{aligned} V|_{M^0} = (V_1 \otimes V_2) \oplus (V_1' \otimes V_2), \end{aligned}$$

where $V_1$ and $V_1'$ are the two spin modules for $M_1$, and $V_2$ is the spin module for $M_2$. Here the $KM^0$-modules $V_1 \otimes V_2$ and $V_1' \otimes V_2$ lift to representations $\varphi : L \rightarrow \mathrm{GL}(V_1 \otimes V_2)$ and $\varphi ' : L \rightarrow \mathrm{GL}(V_1' \otimes V_2)$, so we can consider $(V_1 \otimes V_2)|_{R_1}$ and $(V_1' \otimes V_2)|_{R_1}$. By induction, $V_2|_{\pi _2(R_1)}$ is irreducible, and $\pi _1(R_1)$ acts irreducibly on $V_1$ and $V_1'$. Therefore, $V|_{R_1}$ has precisely two composition factors, which are interchanged by an element in $R_1.2$ that acts as a graph automorphism on $L_1$ and centralizes $L_2$. Therefore, $V|_{H}$ is irreducible. An entirely similar argument applies if $\dim U_1$ is odd and $\dim U_2$ is even.

Finally, suppose $U_1$ and $U_2$ are both even-dimensional. Here

$$\begin{aligned} V|_{M^0} = (V_1 \otimes V_2) \oplus \left( V_1' \otimes V_2'\right) \end{aligned}$$

where $V_i$ and $V_i'$ are the two spin modules for $M_i$, and the inductive hypothesis implies that $\pi _i(R_1)$ acts irreducibly on $V_i$ and $V_i'$, for $i=1,2$. As before, it follows that $V|_{R_1}$ has exactly two composition factors, which are interchanged by an element in $R_1.2$ that acts simultaneously as a graph automorphism on both $L_1$ and $L_2$. Therefore, the two $KH_1$-composition factors of V are H-conjugate, and thus $V|_{H}$ is irreducible. $\square$

In view of the previous proposition, we can easily construct chains of positive-dimensional closed subgroups

$$\begin{aligned} H_k< H_{k-1}< \cdots< H_1 <G \end{aligned}$$

such that $V|_{H_i}$ is irreducible for all i. For instance, take any sequence of successive refinements of a fixed orthogonal decomposition of W such that each refinement is also an orthogonal decomposition that only contains subspaces of dimension at least three. Then the stabilizers in G of these decompositions form a chain of subgroups with the desired irreducibility property. In particular, such a chain can be arbitrarily long. This is in stark contrast to the general situation, where the length of an irreducible chain is at most five (see Theorem 1.9, which will be proved in the next section).

8 Irreducible chains

In this final section we prove Theorem 1.9. Recall that if G is a simple algebraic group and $V=V_G(\lambda )$ is a nontrivial p-restricted irreducible KG-module, then we write $\ell =\ell (G,V)$ for the length of the longest chain of closed positive-dimensional subgroups

$$\begin{aligned} H_{\ell }<H_{\ell -1}< \cdots< H_2 < H_1 = G \end{aligned}$$

such that $V|_{H_{\ell }}$ is irreducible.

As noted in the previous section, if G is an orthogonal group (or a symplectic group with $p=2$) and V is a spin module, then $\ell (G,V)$ can be arbitrarily large (one can simply take an appropriate chain of decomposition subgroups, for example). Similarly, if $V=W$ or $W^*$ (where W is the natural KG-module) then $\ell (G,V)$ is unbounded. For instance, if $G=\mathrm{Sp}_{2n}(K)$ then set $H_{i} = \mathrm{Sp}_{2}(K) \wr X_i$, where $X_i \leqslant \mathrm{Sym}_n$ is transitive. The transitivity of $X_i$ implies that $W|_{H_i}$ is irreducible, and it is easy to see that if n is sufficiently large then we can find arbitrarily long chains of transitive subgroups

$$\begin{aligned} X_i< X_{i-1}< \cdots < X_1 = \mathrm{Sym}_n. \end{aligned}$$

In fact, if we choose n appropriately, then we may assume that each $X_i$ is 3-transitive (see Remark 8.2 below).

Now let us assume that $V \ne W,W^*$, and also assume that V is not a spin module. In this situation, it is natural to ask whether or not $\ell (G,V)$ is bounded above by an absolute constant. Our main theorem is the following, which immediately yields Theorem 1.9. (Note that in Table 8, T is a maximal torus of G and $\mathrm{M}_{n}$ is the simple Mathieu group of degree n).

Theorem 8.1

Let G be a simply connected cover of a simple classical algebraic group with natural module W. Let $V=V_G(\lambda )$ be a p-restricted irreducible KG-module, where $V \ne W,W^*$ and V is not a spin module. Then either

• (i) $\ell (G,V) \leqslant 5$; or

• (ii) $G=A_n$ and $\lambda \in \{\lambda _2, \lambda _3, \lambda _{n-2}, \lambda _{n-1}\}$.

More precisely, excluding the cases in (ii), we have $\ell (G,V) \leqslant 3$, unless (G, V) is one of the cases listed in Table 8.

Table 8 The irreducible chains in Theorem 8.1

Remark 8.2

The cases in part (ii) of Theorem 8.1 are genuine exceptions; for suitable values of n, $\ell (G,V)$ can be arbitrarily large. By duality, we only need to consider the cases $\lambda =\lambda _2$ and $\lambda _3$. Recall that if $H=T_n.X <G$, then $V_G(\lambda _3)|_{H}$ is irreducible if and only if $X \leqslant \mathrm{Sym}_{n+1}$ is 3-transitive (and similarly, X has to be 2-transitive if $\lambda =\lambda _2$); see Proposition 5.5. Suppose $n=q=2^e$ for some positive integer $e \geqslant 2$. The finite simple group $\mathrm{PSL}_{2}(q)$ has a faithful 3-transitive action on the projective line $\mathbb {F}_{q} \cup \{\infty \}$, which extends to a faithful action of its automorphism group $\mathrm{P\Gamma L}_{2}(q) = \mathrm{PSL}_{2}(q).Z_e$. Therefore, $\mathrm{PSL}_{2}(q).d \leqslant \mathrm{P\Gamma L}_{2}(q)$ is a 3-transitive subgroup of $\mathrm{Sym}_{n+1}$ for every divisor d of e. In particular, by choosing e appropriately we can construct arbitrarily long chains of 3-transitive subgroups of $\mathrm{Sym}_{n+1}$, and each of the corresponding subgroups $T_n.X < G$ acts irreducibly on $V_G(\lambda _2)$ and $V_G(\lambda _3)$.

Proof of Theorem 8.1

The proof is a combination of the main theorems in [3, 4, 17], together with Theorem 1.4. To illustrate the general approach, we will consider the case $G=A_n$. Set $V = V_G(\lambda )$. In order to prove the theorem, we may assume that

$$\begin{aligned} \lambda \not \in \{\lambda _1,\lambda _2,\lambda _3,\lambda _{n-2},\lambda _{n-1},\lambda _n\} \end{aligned}$$

(19)

and $\ell (G,V) \geqslant 4$, so there is an irreducible chain $H_4< H_3< H_2 < H_1 = G$.

Consider the irreducible triple $(G,H_4,V)$. First assume that $H_4$ is disconnected and $V|_{H_4^0}$ is reducible. Then $(G,H_4,V)$ must be one of the irreducible triples arising in Theorem 1.4, so $H_4 = T.X_4$ and $\lambda = \lambda _k$, where $X_4<\mathrm{Sym}_{n+1}$ is s-transitive and $s = \max \{k, n+1-k\} \geqslant 4$. Moreover, $H_3 = T.X_3$ and $H_2 = T.X_2$, where $X_3<X_2 \leqslant \mathrm{Sym}_{n+1}$ are also s-transitive groups of degree $n+1$.

Using the classification of finite simple groups, it can be shown that $\mathrm{Sym}_{n+1}$ and the alternating group $\mathrm{Alt}_{n+1}$ (for $n \geqslant 5$) are the only 4-transitive groups of degree $n+1$, unless $n \in \{10,11,22,23\}$ in which case the Mathieu group $\mathrm{M}_{n+1}$ is also 4-transitive (see [5, Theorem 4.11], for example). Similarly, if $t \geqslant 5$ then $\mathrm{Sym}_{n+1}$ and $\mathrm{Alt}_{n+1}$ (for $n \geqslant t+1$) are the only t-transitive groups of degree $n+1$, with the single exception of $\mathrm{M}_{24}$ when $t=5$ and $n = 23$. Since $s \geqslant 4$, it follows that either $s =4$ and $n \in \{10,11,22,23\}$, or $s=5$ and $n=23$. In each case, $X_4 = \mathrm{M}_{n+1}$, $X_3 = \mathrm{Alt}_{n+1}$, $X_2 = \mathrm{Sym}_{n+1}$ and no proper positive-dimensional subgroup of $H_4$ acts irreducibly on V. These special cases are recorded in Table 8.

To complete the proof of Theorem 8.1 for $G=A_n$, we may assume that $(G,H_4,V)$ is an irreducible triple with $H_4$ connected. The possibilities are recorded in [17, Table 1], and in view of (19) we see that the relevant cases therein are labelled

$$\begin{aligned} \mathrm{I}_1, \mathrm{I}'_{1}, \mathrm{I}_2, \mathrm{I}_3, \mathrm{I}_4, \mathrm{I}_5, \mathrm{I}_{12}. \end{aligned}$$

(20)

Consider the irreducible triple $(G,H_3,V)$. If $H_3$ is connected, then $(G,H_3,V)$ also corresponds to one of the cases in (20), but it is routine to check that this collection of cases does not contain a pair of triples $(G,H_4,V)$ and $(G,H_3,V)$ with $H_4<H_3$. Finally, suppose that $H_3$ is disconnected. The connectivity of $H_4$ implies that $H_4 \leqslant H_3^0$, so $V|_{H_3^0}$ is irreducible and we have an irreducible chain

$$\begin{aligned} H_4 \leqslant H_3^0< H_3< H_2 < H_1 = G. \end{aligned}$$

By the argument above, we have $H_4 = H_3^0$ and thus $H_4 < H_3 \leqslant N_G(H_4)$. It is now easy to see that $(G,H_4,V)$ must correspond to the case I$_4$ or I$_5$ in [17, Table 1], so n is odd, $H_4=D_{(n+1)/2}$ and $H_3 = D_{(n+1)/2}.2$. But $D_{(n+1)/2}.2<A_n$ is a maximal subgroup, so we have reached a contradiction.

This completes the proof of Theorem 8.1 for $G=A_n$. The other cases are similar, and we leave it to the reader to check the details.$\square$

This completes the proof of Theorem 1.9.