Pseudo-Riemannian Geodesic Orbit Nilmanifolds of Signature $\varvec{(n-2,2)}$

Abstract

The geodesic orbit property is useful and interesting in itself, and it plays a key role in Riemannian geometry. It implies homogeneity and has important classes of Riemannian manifolds as special cases. Those classes include weakly symmetric Riemannian manifolds and naturally reductive Riemannian manifolds. The corresponding results for indefinite metric manifolds are much more delicate than in Riemannian signature, but in the last few years important corresponding structural results were proved for geodesic orbit Lorentz manifolds. Here we extend Riemannian and Lorentz results to trans-Lorentz nilmanifolds. Those are the geodesic orbit pseudo Riemannian manifolds $M = G/H$ of signature $(n-2,2)$ such that a nilpotent analytic subgroup of G is transitive on M. For that we suppose that there is a reductive decomposition ${\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {n}}\text { (vector space direct sum) with } [{\mathfrak {h}},{\mathfrak {n}}] \subset {\mathfrak {n}}$ and ${\mathfrak {n}}$ nilpotent. When the metric is nondegenerate on $[{\mathfrak {n}},{\mathfrak {n}}]$ we show that ${\mathfrak {n}}$ is abelian or 2-step nilpotent. That is the same result as for geodesic orbit Riemannian and Lorentz nilmanifolds. When the metric is degenerate on $[{\mathfrak {n}},{\mathfrak {n}}]$ we show that ${\mathfrak {n}}$ is a double extension of a geodesic orbit nilmanifold of either Riemannian or Lorentz signature.

1 Introduction and Statement of Results

A pseudo-Riemannian manifold $(M,ds^2)$ is called a geodesic orbit manifold (or a manifold with homogeneous geodesics, or simply a GO manifold), if every geodesic of M is an orbit of a 1-parameter subgroup of the full isometry group $I(M) = I(M,ds^2)$. One loses no generality if one replaces I(M) by its identity component $I^0(M)$. If G is a transitive Lie subgroup of $I^0(M)$, so $(M,ds^2) = (G/H,ds^2)$ where H is an isotropy subgroup of G, and if every geodesic of M is an orbit of a 1-parameter subgroup of G, then we say that $(M,ds^2)$ is a G-geodesic orbit manifold, or a G-GO manifold. Clearly every G-GO manifold is a GO manifold, but not vice versa. The class of geodesic orbit manifolds includes (but is not limited to) symmetric spaces, weakly symmetric spaces, normal and generalized normal homogeneous spaces, and naturally reductive spaces. For the current state of knowledge in the theory of Riemannian geodesic orbit manifolds we refer the reader to [2] and its bibliography.

In this paper, we study the GO condition for pseudo-Riemannian nilmanifolds $(N,ds^2)$, relative to subgroups $G \subset I(N)$ of the form , where H is an isotropy subgroup. Most of our results apply to the case where $(N,ds^2)$ is a trans-Lorentz manifold, that is, the signature of $ds^2$ is $(n-2,2)$, where $n = \dim N$.

Our results for G-GO manifolds $(M,ds^2) = (G/H,ds^2)$ require the coset space G/H to be reductive. In other words, they make use of an ${\text {Ad}}_G(H)$-invariant decomposition ${\mathfrak {g}}= {\mathfrak {m}}\oplus {\mathfrak {h}}$. Very few structural results are known for indefinite metric GO manifolds that are not reductive, and we always assume that G/H is reductive (see the discussion below).

The GO condition for reductive spaces is well known:

Geodesic Lemma [8] Let $(M,ds^2)=G/H$ be a reductive pseudo-Riemannian homogeneous space, with the corresponding reductive decomposition ${\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {m}}$. Then M is a G-geodesic orbit space if and only if, for any $T \in {\mathfrak {m}}$, there exist $A = A(T) \in {\mathfrak {h}}$ and $k=k(T) \in {\mathbb {R}}$ such that if $T' \in {\mathfrak {m}}$ then

$$\begin{aligned} \langle [T+A,T']_{\mathfrak {m}},T\rangle = k \langle T,T'\rangle , \end{aligned}$$

(1)

where $\langle \cdot ,\cdot \rangle$ denotes the inner product on ${\mathfrak {m}}$ defined by $ds^2$, and the subscript ${}_{\mathfrak {m}}$ in (1) means taking the ${\mathfrak {m}}$-component in ${\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {m}}$.

Note that $k(T)=0$ unless T is a null vector [substitute $T'=T$ in (1)].

Recall that a pseudo-Riemannian nilmanifold is a pseudo-Riemannian manifold admitting a transitive nilpotent Lie group of isometries. In the Riemannian case, the full isometry group of a nilmanifold $(N,ds^2)$, where N is a transitive nilpotent group of isometries, is the semidirect product , where H is the group of all isometric automorphisms of $(N,ds^2)$ [13, Theorem 4.2]. In other words, N is the nilradical of I(N). In the pseudo-Riemannian cases, I(N) might still contain and yet be strictly larger. In indefinite signatures of metric a nilmanifold is not necessarily reductive as a coset space of I(N), and even when it is, N does not have to be a normal subgroup of I(N). Here the GO condition does not rescue us, for there exist 4-dimensional, Lorentz GO nilmanifolds that are reductive relative to I(N), but for which N is not an ideal in I(N) [7, Sect. 3]. Moreover, already in dimension 4 (the lowest dimension for homogeneous pseudo-Riemannian spaces G/H with H connected that are not reductive), every non-reductive space is a GO manifold when we make a correct choice of parameters [3, Theorem 4.1]. These results explain (and motivate) our study of G-GO nilmanifolds , where N is nilpotent, and N is the maximal connected subgroup of isometric automorphisms of $(N,ds^2)$ (although our results remain valid for a smaller subgroup H). Given a reductive G-GO trans-Lorentz nilmanifolds $(G/H,ds^2)$, where , with N nilpotent, and the corresponding reductive decomposition ${\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {n}}$ at the level of Lie algebras, we denote $\langle \cdot ,\cdot \rangle$ the inner product on ${\mathfrak {n}}$ induced by $ds^2$, and by $\langle \cdot ,\cdot \rangle '$, the restriction of $\langle \cdot ,\cdot \rangle$ to the derived algebra ${\mathfrak {n}}'=[{\mathfrak {n}},{\mathfrak {n}}]$.

The structure of the paper is as follows. Section 2 contains the proof of the first main theorem:

Theorem 1

Let $(M = G/H, ds^2)$ be a connected trans-Lorentz G-geodesic orbit nilmanifold where $,$ with N nilpotent. Let $\langle \cdot ,\cdot \rangle$ denote the inner product on ${\mathfrak {n}}$ induced by $ds^2$. If $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ is nondegenerate, then N is either abelian or 2-step nilpotent.

Remark 1

There are very many connected trans-Lorentz G-geodesic orbit nilmanifolds as in Theorem 1. They are real forms of the complexifications of Riemannian GO spaces. See [4, Proposition 4.3 and Corollary 5.4] for the collection and [14] for the fact that those real forms are GO. $\diamondsuit$

Theorem 1 extends the results of [9, Theorem 2.2] (for the Riemannian signature) and of [11, Theorem 2] and [5, Theorem 7] (for the Lorentz signature) to the trans-Lorentz case. Our proof of Theorem 1 is split into two parts, given in Sects. 2.1 and 2.2, depending on the signature of $\langle \cdot ,\cdot \rangle '$. In Sect. 2.3 we give an example which shows that the results of Theorem 1 and [11, Theorem 2] are “almost” tight in the sense of the signature: there is a G-GO nilmanifold of signature (8, 4), with the Lorentz derived algebra, which is 4-step nilpotent.

In Sect. 3 we extend the result of [11, Theorem 3] (for the Lorentz signature) to the trans-Lorentz [signature $(n-2,2)$] setting.

In Theorem 2, stated just below, we prove that if the restriction $\langle \cdot ,\cdot \rangle '$ is degenerate then ${\mathfrak {n}}$ can be obtained by the double extension procedure from a metric Lie algebra of either a Riemannian or Lorentz nilmanifold (by metric Lie algebra we mean a Lie algebra with an inner product). The double extension construction (which is explained in Sect. 3) is a useful tool in pseudo-Riemannian homogeneous geometry, in particular in the theory of bi-invariant metrics (see the recent survey [12]) and in the context of GO nilmanifolds [11, Sect. 4]. The precise result is

Theorem 2

Let $(M = G/H, ds^2)$ be a connected trans-Lorentz G-geodesic orbit nilmanifold where $,$ with N nilpotent. Let $\langle \cdot ,\cdot \rangle$ denote the inner product on ${\mathfrak {n}}$ induced by $ds^2$. If $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ is degenerate, then $({\mathfrak {n}}, \langle \cdot ,\cdot \rangle )$ is a either a 2-dimensional double extension of a metric Lie algebra corresponding to a Lorentz GO nilmanifold, or a 4-dimensional double extension of a metric Lie algebra corresponding to a Riemannian GO nilmanifold.

Remark 2

The Lorentz GO nilmanifold in Theorem 2 is geodesic orbit relative to the group $G_0$, which is the semidirect product of the group of parallel translations and pseudo orthogonal automorphisms, as constructed in Lemma 6, and hence by the results of [11] is either at most 2-step nilpotent, or is by itself obtained from a 2-dimensional double extension of a metric Lie algebra of a Riemannian nilmanifold (which must be at most 2-step nilpotent by [9, Theorem 2.2]). As the composition of two repeated 2-dimensional double extensions is equivalent to a single 4-dimensional double extension, we deduce that in the assumptions of Theorem 2, the Lie algebra of the nilmanifold M is obtained either by a 2-dimensional double extension of a Lorentz Lie algebra ${\mathfrak {m}}_0$ or by a 4-dimensional double extension of a Riemannian Lie algebra ${\mathfrak {m}}_0$, where in both cases, ${\mathfrak {m}}_0$ is at most 2-step nilpotent.

Note that even a 2-dimensional GO double extension of an abelian definite Lie algebra can be of an arbitrarily high step, as shown in [11, Sect. 5]. $\diamondsuit$

The authors have no competing interests to declare that are relevant to the content of this article.

2 Proof of Theorem 1: If $ds^2|_{[{\mathfrak {n}},{\mathfrak {n}}]}$ is Nondegenerate Then ${\mathfrak {n}}$ is Either Abelian or 2-Step Nilpotent

Given a reductive homogeneous pseudo-Riemannian manifold $(G/H,ds^2)$, where , with N nilpotent, we identify ${\mathfrak {n}}= \textrm{Lie}(N)$ with the tangent space to G/H at 1N. Let $\langle \cdot ,\cdot \rangle$ be the inner product on ${\mathfrak {n}}$ induces by $ds^2$, and denote ${\mathfrak {n}}'=[{\mathfrak {n}},{\mathfrak {n}}]$.

Assume that the restriction $\langle \cdot ,\cdot \rangle '$ of the inner product $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}'$ is nondegenerate. Denote ${\mathfrak {v}}=({\mathfrak {n}}')^\perp$; note that ${\mathfrak {n}}$ is the direct orthogonal sum of ${\mathfrak {n}}'$ and ${\mathfrak {v}}$, and both subspaces ${\mathfrak {n}}'$ and ${\mathfrak {v}}$ of ${\mathfrak {n}}$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant.

Remark 3

Note that if $V_1$ and $V_2$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspaces of ${\mathfrak {n}}$, then each of the following subspaces is also ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant:

$$\begin{aligned} V_1^\perp , \quad V_1 + V_2, \quad V_1 \cap V_2, \quad [V_1, V_2], \quad \{X \in {\mathfrak {n}}\, : \, [X, V_1] \subset V_2\}. \end{aligned}$$

In particular, the centraliser and the normaliser of an ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspace of ${\mathfrak {n}}$ are themselves ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant. $\diamondsuit$

Let $(G/H,ds^2)$ be G-geodesic orbit. Following the first steps in the proof of [5, Theorem 7] and of [11, Theorem 1], we take $T=X+Y$ and $T' = X'+Y'$, where $X,X' \in {\mathfrak {n}}'$, $Y,Y' \in {\mathfrak {v}}$, and T is non-null in (1). Then $k(T)=0$ and there exists $A=A(X,Y) \in {\mathfrak {h}}$ such that

$$\begin{aligned} \langle [A,X'],X\rangle + \langle [A,Y'],Y\rangle + \langle [X,X']+[Y,X']+[X,Y']+[Y,Y'],X\rangle =0. \end{aligned}$$

(2)

Taking $Y'=Y, X'=0$ we obtain, by continuity,

$$\begin{aligned} \langle [Y,X],X\rangle =0, \quad \text {for all } Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'. \end{aligned}$$

(3)

As ${\mathfrak {v}}$ generates ${\mathfrak {n}}$ it follows that

$$\begin{aligned} \langle [T,X],X\rangle =0, \quad \text {for all } T \in {\mathfrak {n}}, \, X \in {\mathfrak {n}}'. \end{aligned}$$

(4)

Remark 4

Note that if $\langle \cdot ,\cdot \rangle '$ is definite, Eq. (4) implies $[{\mathfrak {n}},{\mathfrak {n}}']=0$, and so ${\mathfrak {n}}$ is at most 2-step nilpotent, regardless of the signature of $\langle \cdot ,\cdot \rangle$. In the context of Theorem 1 we can therefore assume that $\langle \cdot ,\cdot \rangle '$ is indefinite. $\diamondsuit$

Separating the $X'$- and the $Y'$-components in (2) and using (3) and (4), we find that for all $X \in {\mathfrak {n}}'$ and $Y \in {\mathfrak {v}}$ with $X+Y$ non-null, there exists $A=A(X,Y) \in {\mathfrak {h}}$ such that for all $X' \in {\mathfrak {n}}', \, Y' \in {\mathfrak {v}}$,

$$\begin{aligned} \langle [A,Y],Y' \rangle= & {} \langle [Y,Y'],X \rangle , \end{aligned}$$

(5)

$$\begin{aligned} {}[A+Y,X]= & {} 0. \end{aligned}$$

(6)

Denote ${\mathfrak {s}}:= \mathfrak {so}({\mathfrak {n}}', \langle \cdot ,\cdot \rangle ') \subset \mathfrak {gl}({\mathfrak {n}}')$, the algebra of skew-symmetric endomorphisms relative to the restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}'$. By (4) ${\mathfrak {k}}:= {\text {ad}}_{\mathfrak {g}}({\mathfrak {n}})|_{{\mathfrak {n}}'}$ is a subalgebra of ${\mathfrak {s}}$ consisting of nilpotent endomorphisms. In fact, the map $\phi : {\mathfrak {n}}\rightarrow {\mathfrak {k}}$ defined by $\phi (T) = {\text {ad}}(T)|_{{\mathfrak {n}}'}$ for $T \in {\mathfrak {n}}$ is a Lie algebra homomorphism. Using Engel’s Theorem, ${\mathfrak {k}}$ is triangular. Thus it is conjugate by an inner automorphism [10, Theorem 2.1] to a subalgebra of the nilpotent part ${\mathfrak {u}}$ of an Iwasawa decomposition ${\mathfrak {s}}={\mathfrak {t}}\oplus {\mathfrak {a}}\oplus {\mathfrak {u}}$. In the following we may (and do) assume ${\mathfrak {k}}\subset {\mathfrak {u}}$.

In view of Remark 4, to prove Theorem 1 we need to consider two cases: when the restrictions of $\langle \cdot ,\cdot \rangle$ to both ${\mathfrak {n}}'$ and ${\mathfrak {v}}$ are Lorentz, and when restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}$ is trans Lorentz and the restriction to ${\mathfrak {v}}$ is definite.

We consider these two cases separately in the following two subsections. The proof of Theorem 1 will follow from Propositions 1 and 2.

2.1 Both ${\mathfrak {n}}'$ and ${\mathfrak {v}}$ Are Lorentz

In this subsection we additionally assume, in the assumptions of Theorem 1, that the restrictions of $\langle \cdot ,\cdot \rangle$ to both ${\mathfrak {n}}'$ and ${\mathfrak {v}}$ are both Lorentz. We prove the following.

Proposition 1

Let $(M = G/H, ds^2)$ be a connected pseudo-Riemannian G-geodesic orbit nilmanifold where with N nilpotent. If the restrictions of $\langle \cdot ,\cdot \rangle$ to both ${\mathfrak {n}}'$ and ${\mathfrak {v}}$ are of Lorentz signature, then N is either abelian or 2-step nilpotent.

Proof

Denote $m = \dim {\mathfrak {n}}'$ (note that $m \ge 2$). We adopt the notation and will use the facts stated at the start of this section.

The subalgebra ${\mathfrak {s}}=\mathfrak {so}({\mathfrak {n}}',\langle \cdot ,\cdot \rangle ') \subset \mathfrak {gl}({\mathfrak {n}}')$ of skew-symmetric endomorphisms of $\langle \cdot ,\cdot \rangle '$ is isomorphic to $\mathfrak {so}(m -1, 1)$. We can choose a basis $\{e_1, \dots , e_m\}$ for ${\mathfrak {n}}'$ relative to which the restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}'$ and the nilpotent part ${\mathfrak {u}}$ of the Iwasawa decomposition of ${\mathfrak {s}}$ are given by the following matrices:

$$\begin{aligned} \langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'} = \left( {\begin{matrix} 0 &{} 0 &{} 1 \\ 0 &{} I_{m-2} &{} 0 \\ 1 &{} 0 &{} 0 \end{matrix}} \right) \quad \text {and}\quad {\mathfrak {u}}= \left\{ \left( {\begin{matrix} 0 &{} 0 &{} 0 \\ u &{} 0_{m-2} &{} 0 \\ 0 &{} -u^t &{} 0 \end{matrix}}\right) \, : \, u \in {\mathbb {R}}^{m-2} \right\} . \end{aligned}$$

(7)

As ${\mathfrak {k}}\subset {\mathfrak {u}}$, we obtain a linear map $\Phi : {\mathfrak {n}}\rightarrow {\text {Span}}(e_2, \dots , e_{m-1})$ such that, for all $T \in {\mathfrak {n}}$,

$$\begin{aligned}{}[T, e_1] = \Phi T,\quad [T, e_m] = 0,\quad \text {and}\quad [T, e_i] = -\langle \Phi T,e_i\rangle e_m\quad \text {for } 2 \le i \le m-1. \end{aligned}$$

(8)

As ${\mathfrak {u}}$ (and hence ${\mathfrak {k}}$) is abelian, we obtain $[[{\mathfrak {v}},{\mathfrak {v}}],{\mathfrak {n}}']=0$. Since ${\mathfrak {v}}$ generates ${\mathfrak {n}}$, we obtain $[[{\mathfrak {n}},{\mathfrak {n}}],{\mathfrak {n}}']=0$, which implies that ${\mathfrak {n}}'$ is abelian.

Introduce the 2-forms $\omega _i \in \Lambda ^2({\mathfrak {n}})$ by

$$\begin{aligned}{}[T_1,T_2]= \sum \limits _{i=1}^{m} \omega _i(T_1,T_2)e_i\quad \text {for } T_1, T_2 \in {\mathfrak {n}}. \end{aligned}$$

(9)

From (7) and (8) we have

$$\begin{aligned} \omega _1(T_1,T_2)=\langle [T_1,T_2],e_m\rangle , \quad \text {and} \quad \omega _1({\mathfrak {n}},{\mathfrak {n}}')=0. \end{aligned}$$

(10)

As $\omega _1$ cannot be zero (since $e_1 \in {\mathfrak {n}}'$) we obtain $\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0$.

Using (8) and (9), the Jacobi identity gives

$$\begin{aligned} \sigma \Big (\omega _1(T_1,T_2)\Phi T_3 + \sum \limits _{i=2}^{m-1} \omega _i(T_1,T_2)\langle \Phi T_3,e_i\rangle e_m\Big )=0, \end{aligned}$$

(11)

where $\sigma$ denotes the cyclic permutation of $T_1,T_2,T_3 \in {\mathfrak {n}}$.

Consider the following two subspaces of ${\mathfrak {n}}$:

$$\begin{aligned}{} & {} {\mathfrak {c}}= {\text {Ker}}\Phi ( = \{T \in {\mathfrak {n}}\, : \, [T, {\mathfrak {n}}'] = 0\}), \nonumber \\ {}{} & {} {\mathfrak {q}}= \{T \in {\mathfrak {n}}\, : \omega _1(T,{\mathfrak {n}}) ( = \langle [T, {\mathfrak {n}}],e_m\rangle ) = 0\} \end{aligned}$$

(12)

[the fact that ${\text {Ker}}\Phi$ is the centraliser of ${\mathfrak {n}}'$ in ${\mathfrak {n}}$ follows from (8)]. By [11, Theorem 1(b)] we can (and will) assume that ${\mathfrak {c}}$ is degenerate. Furthermore, we can assume that ${\mathfrak {c}}\ne {\mathfrak {n}}$ (equivalently, $\Phi \ne 0$), as otherwise the algebra ${\mathfrak {n}}$ is 2-step nilpotent by (8).

In these notations and assumptions, we have the following.

Lemma 1

• (a) ${\mathfrak {n}}' \subset {\mathfrak {q}}\subset {\mathfrak {c}},$ and both ${\mathfrak {q}}$ and ${\mathfrak {c}}$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant ideals of ${\mathfrak {n}}$. Moreover, $[{\mathfrak {c}},{\mathfrak {c}}] \subset {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'$.

• (b) ${\text {codim}}{\mathfrak {q}}= 2$ and ${\text {codim}}{\mathfrak {c}}\in \{1,2\}$ (equivalently, ${\text {rk}}\, \Phi \in \{1,2\}$).

• (c) Let $e \in {\mathfrak {c}}\cap {\mathfrak {c}}^\perp$ be a nonzero (necessarily null) vector. Then $e \in {\mathfrak {v}},$ the line ${\mathbb {R}}e$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant and $[e,e^\perp ]= 0$.

• (d) Let $f \in {\mathfrak {v}}$ be a null vector such that $f \notin e^\perp$ and $\langle f,e\rangle =1$. Then $[{\mathfrak {h}},[f,e]]=0$.

• (e) $[f,e] \in {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'$ and $e \in {\mathfrak {q}}$.

Proof

(a) The fact that ${\mathfrak {n}}' \subset {\mathfrak {q}}$ follows from (10). Furthermore, taking $T_1 \in {\mathfrak {q}}$ in (11) we obtain $\omega _1(T_2,T_3) \Phi (T_1)=0$ which implies ${\mathfrak {q}}\subset {\mathfrak {c}}$. As both ${\mathfrak {q}}$ and ${\mathfrak {c}}$ contain ${\mathfrak {n}}'$, they are ideals of ${\mathfrak {n}}$. The fact that ${\mathfrak {c}}$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant follows from Remark 3. Moreover, by Remark 3, ${\mathfrak {q}}$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant provided ${\mathbb {R}}e_m$ is. To see the latter, we note that $[{\mathfrak {v}}, {\mathfrak {n}}']$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant. From (8) (and the fact that $\Phi \ne 0$), we have $e_m \in [{\mathfrak {v}}, {\mathfrak {n}}'] \subset {\text {Span}}(e_2, \dots , e_m)$. Then $[{\mathfrak {v}}, {\mathfrak {n}}'] \cap ([{\mathfrak {v}}, {\mathfrak {n}}'])^\perp = {\mathbb {R}}e_m$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, by Remark 3. Finally, the fact that $[{\mathfrak {c}},{\mathfrak {c}}] \subset {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'$ follows from the Jacobi identity, as $[{\mathfrak {c}},{\mathfrak {n}}']=0$.

(b) If ${\text {rk}}\Phi \ge 3$, then for almost all triples $T_1,T_2,T_3 \in {\mathfrak {n}}$, the vectors $\Phi T_1, \Phi T_2, \Phi T_3 \in {\text {Span}}(e_2, \dots , e_{m-1})$ are linearly independent, and so $\omega _1=0$ by (11). This is a contradiction, so ${\text {rk}}\Phi \le 2$. As $\Phi \ne 0$ we obtain ${\text {codim}}{\mathfrak {c}}\, (= {\text {rk}}\Phi ) \in \{1,2\}$. Now as ${\mathfrak {q}}\subset {\mathfrak {c}}$ by (a) and as ${\mathfrak {q}}$ is the null space of the skew-symmetric form $\omega _1$, the codimension of ${\mathfrak {q}}$ must be a positive even number. Furthermore, from (a) we have $\omega _1({\mathfrak {c}},{\mathfrak {c}})=0$. If ${\text {codim}}{\mathfrak {c}}= 1$, this implies ${\text {codim}}{\mathfrak {q}}=2$. If ${\text {rk}}\Phi =2$, we take $T_1,T_2 \in {\mathfrak {n}}$ in (11) such that the vectors $\Phi T_1, \Phi T_2 \in {\text {Span}}(e_2, \dots , e_{m-1})$ are linearly independent and take $T_3 \in {\mathfrak {c}}$. We obtain $\omega _1(T_1,{\mathfrak {c}})=\omega _1(T_2,{\mathfrak {c}})=0$. As ${\text {Span}}(T_1,T_2) \oplus {\mathfrak {c}}= {\mathfrak {n}}$ we get $\omega _1({\mathfrak {n}}, {\mathfrak {c}})=0$, and so ${\mathfrak {c}}\subset {\mathfrak {q}}$ by (12) which implies ${\mathfrak {c}}={\mathfrak {q}}$ by (a).

(c) As ${\mathfrak {c}}$ is degenerate, the space ${\mathfrak {c}}\cap {\mathfrak {c}}^\perp$ has dimension 1 and is spanned by a (nonzero) null vector e. As ${\mathfrak {n}}' \subset {\mathfrak {c}}$ by (a) we obtain $e \in {\mathfrak {v}}$. The subspace ${\mathbb {R}}e$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant as ${\mathfrak {c}}$ is (and by Remark 3). Then (5) with $Y=e, \, Y' \in e^\perp \cap {\mathfrak {v}}$ implies that $\langle [e,e^\perp \cap {\mathfrak {v}}],X\rangle =0$ for all $X \in {\mathfrak {n}}'$ such that $e+X$ is non-null. This gives $[e,e^\perp \cap {\mathfrak {v}}] = 0$. As $e \in {\mathfrak {c}}$ we have $[e, {\mathfrak {n}}'] = 0$, and so $[e,e^\perp ] = 0$.

(d) Choose $f \in {\mathfrak {v}}$ to be a null vector such that $f \notin {\mathfrak {c}}$ and $\langle f,e\rangle =1$ (this choice is not unique). Let $A \in {\mathfrak {h}}$. By (c), ${\mathbb {R}}e$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, and so $[A,e]=ae$, for some $a \in {\mathbb {R}}$, and so $\langle [A,f],e\rangle =-a$. As $e^\perp \oplus {\mathbb {R}}f = {\mathfrak {n}}$, we have $[A,f] + af \in e^\perp$. Then $[[A,f],e] = -a [f,e]$ since $[e,e^\perp ] = 0$ by (c). As $[f,[A,e]]=a[f,e]$, the claim follows.

(e) Choose f as in (d). Then $e^\perp \oplus {\mathbb {R}}f = {\mathfrak {n}}$, and so from (c) we have $[e, {\mathfrak {n}}] \subset {\mathbb {R}}[f,e]$. Then from (d) and (6), with $X=[f,e]$, we obtain $[Y,[f,e]]=0$, for all $Y \in {\mathfrak {v}}$ such that $Y+[f,e]$ is non-null, and hence for all $Y \in {\mathfrak {v}}$. As ${\mathfrak {n}}'$ is abelian, we obtain $[f,e] \in {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'$. But from (8), ${\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}' \subset {\text {Span}}(e_2, \dots , e_m)$ (as $\Phi \ne 0$), and so $\langle [f,e],e_m\rangle =0$ by (7). So $\langle [{\mathfrak {n}},e],e_m\rangle =0$ and the claim follows by (12). $\square$

Now choose $e \in {\mathfrak {c}}\cap {\mathfrak {c}}^\perp$ as in Lemma 1(c) and choose $f \in {\mathfrak {v}}\setminus e^\perp$ as in Lemma 1(d). By Lemma 1(a) we have ${\mathfrak {n}}' \subset {\mathfrak {q}}\subset {\mathfrak {c}}\subset e^\perp$, and by Lemma 1(b), ${\text {codim}}{\mathfrak {q}}= 2$ (and then either ${\mathfrak {c}}= {\mathfrak {q}}$ or ${\mathfrak {c}}=e^\perp$). As f is not contained in $e^\perp$, and hence in ${\mathfrak {c}}$, we have $\Phi f \ne 0$ by (8). Without loss of generality (scaling f and e and specifying the orthonormal basis $\{e_2, \dots , e_{m-1}\}$) we can assume that $\Phi f = e_2$, and so by (8),

$$\begin{aligned}{}[f,e_1]=e_2, \quad [f,e_2] = - e_m,\quad [f,e_i]=0 \quad \text {for } i > 2. \end{aligned}$$

(13)

Note that with this choice of the basis, ${\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}' \subset {\text {Span}}(e_3, \dots , e_m)$.

Moreover, from Lemma 1(e) (and Lemma 1(a)), the 2-dimensional subspace ${\mathfrak {q}}^\perp$ contains e and lies in ${\mathfrak {v}}$. Thus we have ${\mathfrak {q}}^\perp = {\text {Span}}(e, Y_0)$ for some $Y_0 \notin {\mathfrak {q}}, \, Y_0 \perp e$, with $\langle Y_0,Y_0\rangle \ne 0$. Note that $e^\perp = {\mathbb {R}}Y_0 \oplus {\mathfrak {q}}$, and so ${\mathbb {R}}f \oplus {\mathbb {R}}Y_0 \oplus {\mathfrak {q}}= {\mathfrak {n}}$. As $\omega _1({\mathfrak {q}}, {\mathfrak {n}}) = 0$ and $\omega _1 \ne 0$, we must have $\omega _1(f,Y_0) \ne 0$. Denote $\kappa _i=\omega _i(f,Y_0)$, so that $[f,Y_0] = \sum _{i=1}^m \kappa _i e_i$ [by (9)], with $\kappa _1 \ne 0$.

As $Y_0 \perp {\mathfrak {q}}$, from (5) with $Y=Y_0$ and $Y' \in {\mathfrak {q}}\cap {\mathfrak {v}}$ we obtain $\langle [Y_0,Y'],X\rangle =0$, for all $Y' \in {\mathfrak {q}}\cap {\mathfrak {v}}$ and all $X \in {\mathfrak {n}}'$ such that $Y_0+X$ is non-null, which implies $[Y_0,{\mathfrak {q}}\cap {\mathfrak {v}}] = 0$. Moreover, as ${\mathfrak {q}}^\perp$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, for any $A \in {\mathfrak {h}}$ we have $[A,Y_0] \in {\text {Span}}(e, Y_0)$, and so $[A,Y_0]= \mu e$, for some $\mu \in {\mathbb {R}}$, since $Y_0$ is non-null and $Y_0 \perp e$.

Then from (6) with $X=[f,Y_0]$ we obtain

$$\begin{aligned}{}[[A,f],Y_0]+[f,[A,Y_0]]+[Y,[f,Y_0]]=0, \end{aligned}$$

(14)

for all $Y \in {\mathfrak {v}}$ such that $Y+[f,Y_0]$ is non-null [note that here we have a particular $A=A(X,Y) \in {\mathfrak {h}}$]. Let $[A,f]= a f + b Y_0 + Y'$, where $Y' \in {\mathfrak {q}}\cap {\mathfrak {v}}$. Then $[[A,f],Y_0] = a [f,Y_0]$ as $[Y_0,{\mathfrak {q}}\cap {\mathfrak {v}}] = 0$. Furthermore, $[f,[A,Y_0]] = \mu [f,e]$. By Lemma 1(e) we have $[f,e] \in {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'$, and so $[f,e] \in {\text {Span}}(e_3, \dots , e_m)$. Take $Y = f + \lambda e$ in (14), where $\lambda \in {\mathbb {R}}$ is chosen in such a way that $Y+X=f+\lambda e + [f,Y_0]$ is non-null. As $e \in {\mathfrak {c}}$ we have $[e,[f,Y_0]]=0$, and so we obtain $a [f,Y_0] + [f,[f,Y_0]] \in {\text {Span}}(e_3, \dots , e_m)$. Substituting $[f, Y_0]= \sum _{i=1}^m \kappa _i e_i$ and using (13) we get $a(\kappa _1 e_1 + \kappa _2 e_2) + \kappa _1 e_2 = 0$ which implies $\kappa _1 = 0$, a contradiction. $\square$

2.2 Trans-Lorentz ${\mathfrak {n}}'$, definite ${\mathfrak {v}}$

In this subsection we consider the last remaining case in the proof of Theorem 1. We prove the following.

Proposition 2

Let $(M = G/H, ds^2)$ be a connected pseudo-Riemannian G-geodesic orbit nilmanifold where with N nilpotent. Denote ${\mathfrak {n}}'=[{\mathfrak {n}},{\mathfrak {n}}]$ and ${\mathfrak {v}}=({\mathfrak {n}}')^\perp$. If the restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}'$ is trans-Lorentz, and the restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {v}}$ is definite, then N is either abelian or 2-step nilpotent.

Proof

Denote $m = \dim {\mathfrak {n}}'$ (we can assume that $m \ge 4$, as otherwise the claim follows from [11, Theorem 1(b)]). We adopt the notation and will use the facts stated at the start of the section.

The subalgebra ${\mathfrak {s}}=\mathfrak {so}({\mathfrak {n}}',\langle \cdot ,\cdot \rangle ') \subset \mathfrak {gl}({\mathfrak {n}}')$ of skew-symmetric endomorphisms of $\langle \cdot ,\cdot \rangle '$ is isomorphic to $\mathfrak {so}(m-2, 2)$. We can choose a basis $\{e_1, \dots , e_m\}$ for ${\mathfrak {n}}'$ relative to which the restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}'$ and the nilpotent part ${\mathfrak {u}}$ of the Iwasawa decomposition of ${\mathfrak {s}}$ are given by the following matrices:

$$\begin{aligned}{} & {} \langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'} = \left( {\begin{matrix} 0_2 &{} 0 &{} I_2 \\ 0 &{} I_{m-4} &{} 0 \\ I_2 &{} 0 &{} 0_2 \end{matrix}} \right) \quad \text {and}\nonumber \\{} & {} {\mathfrak {u}}= \left\{ \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ \alpha &{} 0 &{} 0 &{} 0 &{} 0 \\ u &{} v &{} 0_{m-4} &{} 0 &{} 0 \\ 0 &{} \beta &{} -u^t &{} 0 &{} -\alpha \\ -\beta &{} 0 &{} -v^t &{} 0 &{} 0 \end{matrix}}\right) \, : \, u, v \in {\mathbb {R}}^{m-4}, \, \alpha , \beta \in {\mathbb {R}}\right\} . \end{aligned}$$

(15)

The homomorphism $\phi : {\mathfrak {n}}\rightarrow {\mathfrak {u}}$ [given by $\phi (T) = {\text {ad}}(T)|_{{\mathfrak {n}}'}$ for $T \in {\mathfrak {n}}$] defines linear maps $U, V:{\mathfrak {n}}\rightarrow {\mathbb {R}}^{m-4}={\text {Span}}(e_3, \dots , e_{m-2})$ and vectors $a, b \in {\mathfrak {n}}$ such that for $T \in {\mathfrak {n}}$, the corresponding entries of the matrix $\phi (T)$ in the notation of (15) are given by $u =UT, \, v =VT, \, \alpha =\langle a,T\rangle$ and $\beta =\langle b,T\rangle$.

Lemma 2

We have $\phi ({\mathfrak {n}}')=0,$ and so the subalgebra ${\mathfrak {n}}'$ is abelian.

Proof

Clearly $\phi ({\mathfrak {n}}') \subset [{\mathfrak {u}}, {\mathfrak {u}}]$. From (15), the subalgebra $[{\mathfrak {u}},{\mathfrak {u}}]$ is the subspace of elements of ${\mathfrak {u}}$ [as given in (15)] with $v=0$ and $\alpha =0$, so that for $X \in {\mathfrak {n}}'$, we have $VX=0$ and $\langle a, X\rangle =0$. For $X' \in {\mathfrak {n}}$ we have $0= [X,X'] + [X',X] = \phi (X)X' + \phi (X')X$ which gives $x_1' UX + x_1 UX' = 0$ and $x_1'\langle b, X\rangle + x_1\langle b, X'\rangle =0$, where $x_1$ and $x_1'$ are the $e_1$-components of the vectors X and $X'$, respectively. This implies $U{\mathfrak {n}}'=0$ and $\langle b,{\mathfrak {n}}'\rangle =0$, so $\phi ({\mathfrak {n}}')= 0$, as required. But $\phi (T)X=[T,X]$ for $T \in {\mathfrak {n}}$ and $X \in {\mathfrak {n}}'$, and the second claim follows. $\square$

From Lemma 2 it follows that the subalgebra ${\mathfrak {k}}=\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}$ is abelian. It is not hard to see using the root decomposition of ${\mathfrak {u}}$ relative to the abelian subalgebra ${\mathfrak {a}}\subset {\mathfrak {s}}$ in the Iwasawa decomposition (or to calculate directly), that the algebra ${\mathfrak {u}}$ contains three different maximal abelian subalgebras given below [in the notation of (15)]:

• (i) ${\mathfrak {u}}_1 = \{Q \in {\mathfrak {u}}\, : \, v=0\}$.

• (ii) ${\mathfrak {u}}_2 = {\mathbb {R}}\left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ u_1 &{} v_1 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0_{m-5} &{} 0 &{} 0 \\ 0 &{} 0 &{} -u_1 &{} 0 &{} 0 &{} -1 \\ 0 &{} 0 &{} -v_1 &{} 0 &{} 0 &{} 0 \end{matrix}}\right) \oplus \left\{ \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ w &{} 0 &{} 0 &{} 0_{m-5} &{} 0 &{} 0 \\ 0 &{} \beta &{} 0 &{} - w^t &{} 0 &{} 0 \\ -\beta &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \end{matrix}}\right) \, : \, w \in {\mathbb {R}}^{m-5}, \, \beta \in {\mathbb {R}}\right\}$, where $v_1 \ne 0$ [up to specifying a basis in ${\mathbb {R}}^{m-5}={\text {Span}}(e_4, \dots , e_{m-2})$].

• (iii) ${\mathfrak {u}}_3$ is a maximal abelian subalgebra of the Heisenberg algebra $\left\{ \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0\\ u &{} v &{} 0_{m-4} &{} 0 &{} 0 \\ 0 &{} \beta &{} - u^t &{} 0 &{} 0 \\ -\beta &{} 0 &{} -v^t &{} 0 &{} 0 \end{matrix}}\right) \, : \, u,v \in {\mathbb {R}}^{m-4}, \, \beta \in {\mathbb {R}}\right\} \subset {\mathfrak {u}}$.

We will consider these three cases separately, but following the same pattern. Let ${\mathfrak {c}}= {\text {Ker}}\phi \cap {\mathfrak {v}}$ be the centraliser of ${\mathfrak {n}}'$ in ${\mathfrak {v}}$, and ${\mathfrak {c}}^\perp$ be its orthogonal complement in ${\mathfrak {v}}$. By Remark 3, both subspaces ${\mathfrak {c}}$ and ${\mathfrak {c}}^\perp$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant. We will always assume that the subspace ${\mathfrak {c}}^\perp$ is non-trivial (equivalently, $\phi \ne 0$), for otherwise ${\mathfrak {n}}$ is at most 2-step nilpotent.

Denote $\pi : {\mathfrak {h}}\rightarrow \mathfrak {so}({\mathfrak {c}}^\perp )$ the restriction of the representation of ${\mathfrak {h}}$ to ${\mathfrak {c}}^\perp$, so that $\pi (A)Y=[A,Y]$ for $A \in {\mathfrak {h}}$ and $Y \in {\mathfrak {c}}^\perp$.

Lemma 3

• (a) If $L \subset {\mathfrak {c}}^\perp$ is an ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspace, then $[L, L^\perp ]=0$.

• (b) The subspace ${\mathfrak {c}}^\perp$ has no 1-dimensional ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspaces.

• (c) If $L \subset {\mathfrak {c}}^\perp$ is a 2-dimensional ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspace, then $[L,L] \subset {\mathfrak {z}}({\mathfrak {n}}),$ where ${\mathfrak {z}}({\mathfrak {n}})$ is the centre of ${\mathfrak {n}}$.

• (d) If the subalgebra $\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )$ is abelian, then $[{\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ] \subset {\mathfrak {z}}({\mathfrak {n}})$.

Proof

Assertion (a) follows from (5) if we take $Y \in L$ and $Y' \in L^\perp$.

For assertion (b), suppose that for a nonzero $Y \in {\mathfrak {c}}^\perp$, the space ${\mathbb {R}}\, Y$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant. Then $[{\mathfrak {h}}, Y]=0$, and so $[Y, {\mathfrak {v}}]=0$ by (6). As ${\mathfrak {v}}$ generates ${\mathfrak {n}}$, we obtain $[Y, {\mathfrak {n}}]=0$, and in particular, $\phi (Y)=0$ contradicting the fact that $Y \in {\mathfrak {c}}^\perp \setminus \{0\}$.

For assertion (c), suppose that $L={\text {Span}}(Y_1,Y_2) \subset {\mathfrak {c}}^\perp$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, with the vectors $Y_1$ and $Y_2$ being orthonormal. Then for any $A \in {\mathfrak {h}}$, we obtain that $AY_1$ is a multiple of $Y_2$, and $AY_2$ is a multiple of $Y_1$. Hence $AX=0$, where $X=[Y_1,Y_2]$, and so $[X,{\mathfrak {v}}]=0$, by (6). As ${\mathfrak {v}}$ generates ${\mathfrak {n}}$, the subspace $[L,L]={\mathbb {R}}X$ lies in the centre of ${\mathfrak {n}}$.

For assertion (d), suppose that the subalgebra $\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )$, is abelian. Then ${\mathfrak {c}}^\perp$ is the direct, orthogonal sum of ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspaces of dimension 1 or 2 each. But by assertion (b), there can be no 1-dimensional subspaces, and then the claim follows from assertions (c) and (a). $\square$

Introduce the 2-forms $\omega _i \in \Lambda ^2({\mathfrak {v}})$ by

$$\begin{aligned}{}[Y_1,Y_2]= \sum \limits _{i=1}^{m} \omega _i(Y_1,Y_2)e_i\quad \text {for } Y_1, Y_2 \in {\mathfrak {v}}. \end{aligned}$$

(16)

Then

$$\begin{aligned}{}[Y_3,[Y_1,Y_2]] = \sum \limits _{i=1}^{m} \omega _i(Y_1,Y_2) \phi (Y_3) e_i. \end{aligned}$$

(17)

From Lemma 2 (and the fact that $[{\mathfrak {c}},{\mathfrak {n}}']=0$) we have ${\mathfrak {n}}'=[{\mathfrak {v}},{\mathfrak {v}}] + [{\mathfrak {v}}, {\mathfrak {n}}'] = [{\mathfrak {v}},{\mathfrak {v}}] + \phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}'$.

We now separately consider three cases for $\phi ({\mathfrak {n}})$ as given above.

Case (i): $\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_1$. Then $\phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}' \subset {\text {Span}}(e_2, \dots ,e_m)$. As $e_1 \in {\mathfrak {n}}'= [{\mathfrak {v}},{\mathfrak {v}}] + \phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}'$, we obtain $\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0$. The Jacobi identity gives $\sigma (\omega _1(Y_1,Y_2)(\langle a, Y_3\rangle e_2 + UY_3 - \langle b,Y_3\rangle )=0$, where $\sigma$ denotes the cyclic permutation of $Y_1,Y_2,Y_3 \in {\mathfrak {v}}$, which can be written as

$$\begin{aligned} \sigma (\omega _1(Y_1,Y_2)\phi (Y_3))=0. \end{aligned}$$

(18)

Taking $Y_1, Y_2 \in {\mathfrak {c}}$ and $Y_3 \in {\mathfrak {c}}^\perp$ we obtain $\omega _1({\mathfrak {c}},{\mathfrak {c}}) = 0$. By Lemma 3(a) we have $[{\mathfrak {c}},{\mathfrak {c}}^\perp ]=0$, and so by (16) we obtain that also $\omega _1({\mathfrak {c}},{\mathfrak {c}}^\perp ) = 0$. As $\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0$, we deduce that $\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ) \ne 0$. Now if ${\text {rk}}\phi (=\dim {\mathfrak {c}}^\perp ) > 2$, then the elements $\phi (Y_1), \, \phi (Y_2)$ and $\phi (Y_3)$ are linearly independent for almost all triples of vectors $Y_1, Y_2, Y_3 \in {\mathfrak {v}}$, and so (18) implies $\omega _1({\mathfrak {v}},{\mathfrak {v}}) = 0$, a contradiction. By Lemma 3(b), we have $\dim {\mathfrak {c}}^\perp > 1$, and so the only remaining possibility is $\dim {\mathfrak {c}}^\perp = 2$. But then from Lemma 3(c) we obtain $[{\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ] \subset {\mathfrak {z}}({\mathfrak {n}})$. Taking $Y_1, Y_2, Y_3 \in {\mathfrak {c}}^\perp$ in Eq. (17), we get $\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=0$, a contradiction.

Case (ii): $\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_2$. For $Y \in {\mathfrak {v}}$, we have

$$\begin{aligned} \phi (Y)= \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ \langle a,Y\rangle &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ \lambda \langle a,Y\rangle &{} \mu \langle a,Y\rangle &{} 0 &{} 0 &{} 0 &{} 0 \\ WY &{} 0 &{} 0 &{} 0_{m-5} &{} 0 &{} 0 \\ 0 &{} \langle b,Y\rangle &{} -\lambda \langle a,Y\rangle &{} -(WY)^t &{} 0 &{} -\langle a,Y\rangle \\ -\langle b,Y\rangle &{} 0 &{} -\mu \langle a,Y\rangle &{} 0 &{} 0 &{} 0 \end{matrix}}\right) , \end{aligned}$$

(19)

where $\lambda , \mu \in {\mathbb {R}}$ and $W: {\mathfrak {v}}\rightarrow {\mathbb {R}}^{m-5}={\text {Span}}(e_4, \dots , e_{m-2})$. We can assume that $a \ne 0$ and $\mu \ne 0$, for otherwise $\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_1$.

Arguing similarly to the previous case, we see that $\phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}' \subset {\text {Span}}(e_2, \dots ,e_m)$, and so we must have $\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0$. From the Jacobi identity we obtain

$$\begin{aligned} \sigma (\omega _1(Y_1,Y_2)(\langle a,Y_3\rangle e_2 +WY_3))=0, \end{aligned}$$

(20)

where $\sigma$ denotes the cyclic permutation of $Y_1,Y_2,Y_3 \in {\mathfrak {v}}$. From (20) with $Y_1, Y_2 \in {\mathfrak {c}}$ and $Y_3 \in {\mathfrak {c}}^\perp$ we obtain $\omega _1({\mathfrak {c}},{\mathfrak {c}}) = 0$. Furthermore, we have $[{\mathfrak {c}},{\mathfrak {c}}^\perp ]=0$ by Lemma 3(a), and so $\omega _1({\mathfrak {c}},{\mathfrak {c}}^\perp ) = 0$ by (16). It follows that $\omega _1({\mathfrak {c}}, {\mathfrak {v}})=0$, and so we must have $\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ) \ne 0$, as $\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0$.

From the $e_2$-component of (20) we obtain $\omega _1 \wedge \alpha = 0$, where $\alpha$ is the 1-form on ${\mathfrak {v}}$ defined by $\alpha (Y)=\langle a,Y\rangle$. By generalised Cartan’s Lemma [1, Lemma 1] we get $\omega _1 = \gamma \wedge \alpha$ for some 1-form $\gamma \in {\mathfrak {v}}^*$. As $\omega _1 \ne 0$, the 1-form $\gamma$ is not a multiple of $\alpha$. Moreover, as $\omega _1({\mathfrak {c}}, {\mathfrak {v}})=0$, both the vector a and the vector $c \in {\mathfrak {v}}$ dual to $\gamma$ lie in ${\mathfrak {c}}^\perp$. Taking the inner product of (20) with $e_s, \, s=4, \dots , m-2$, we find that $W^t e_s \in {\text {Span}}(a,c)$. Hence in the matrix $\phi (Y)$ given in (19), for all $Y \in ({\text {Span}}(a,c))^\perp \cap {\mathfrak {v}}$, we have $WY=\langle a,Y\rangle =0$. We first suppose that $b \in {\text {Span}}(a,c)$. Then ${\mathfrak {c}}^\perp \subset {\text {Span}}(a,c)$, and as $\dim {\mathfrak {c}}^\perp > 1$ by Lemma 3(b), we deduce that $\dim {\mathfrak {c}}^\perp = 2$ and hence, that $[{\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ]$ lies in the centre of ${\mathfrak {n}}$, by Lemma 3(c). But now if we take $Y_1, Y_2, Y_3 \in {\mathfrak {c}}^\perp$ with $\langle a,Y_3\rangle \ne 0$ in (17), then from the $e_2$-component we get $\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=0$ which is a contradiction.

We therefore suppose that $b \notin {\text {Span}}(a,c)$, and so ${\text {Span}}(a,b) \subset {\mathfrak {c}}^\perp \subset {\text {Span}}(a,b,c)$ from (19). Note that the subspace $[{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']]$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by Remark 3. As $[Y,{\mathfrak {n}}']=\phi (Y){\mathfrak {n}}'$ for $Y \in {\mathfrak {v}}$, Eq. (19) gives that the subspace $[{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']]$ lies in ${\text {Span}}(e_3, \dots , e_m)$. Moreover, as $\phi (Y)^2e_3=\mu \langle a,Y\rangle e_{m-1}$ and $\phi (Y)^2e_2 = -\mu \langle a,Y\rangle (\lambda \langle a,Y\rangle e_{m-1} + \mu \langle a,Y\rangle e_m)$, and as $a \ne 0$ and $\mu \ne 0$, we obtain that the subspace $[{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']]$ contains ${\text {Span}}(e_{m-1}, e_m)$. But then $[{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']] \cap ([{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']])^\perp = {\text {Span}}(e_{m-1}, e_m)$, and so the subspace ${\text {Span}}(e_{m-1}, e_m)$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by Remark 3. Then the subspace $\{Y \in {\mathfrak {c}}^\perp \, : \, [Y,{\mathfrak {n}}'] \subset {\text {Span}}(e_{m-1}, e_m)\}$ is also ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by Remark 3. But the latter subspace is given by $\{Y \in {\mathfrak {c}}^\perp \, : \, WY=0, \, \langle a,Y\rangle =0\} = {\mathfrak {c}}^\perp \cap ({\text {Span}}(a,c))^\perp ={\mathbb {R}}Y_0$, where $Y_0 \ne 0$ is the component of b orthogonal to ${\text {Span}}(a,c)$. This gives a 1-dimensional ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspace of ${\mathfrak {c}}^\perp$, in contradiction with Lemma 3(b).

Case (iii): $\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_3$. This is the most involved case. For $Y \in {\mathfrak {v}}$, we have

$$\begin{aligned} \phi (Y)= \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ UY &{} VY &{} &{} 0_{m-4} &{} 0 &{} 0 \\ 0 &{} \langle b,Y\rangle &{} -(UY)^t &{} 0 &{} 0 \\ -\langle b,Y\rangle &{} 0 &{} -(VY)^t &{} 0 &{} 0 \end{matrix}}\right) , \end{aligned}$$

(21)

where $b \in {\mathfrak {v}}$, and $U,V: {\mathfrak {v}}\rightarrow {\mathbb {R}}^{m-4}={\text {Span}}(e_3, \dots , e_{m-2})$ are such that for all $Y_1, Y_2 \in {\mathfrak {v}}$ we have

$$\begin{aligned} \langle UY_1,VY_2\rangle =\langle UY_2,VY_1\rangle \quad (\text {equivalently, the matrix} U^tV \text {is symmetric}). \end{aligned}$$

(22)

The following lemma sorts out the “non-generic” cases.

Lemma 4

Suppose $\phi ({\mathfrak {n}}) \not \subset {\mathfrak {u}}_1$ and $\phi ({\mathfrak {n}}) \not \subset {\mathfrak {u}}_2$ [up to specifying the basic vectors $e_1,e_2,e_{m-1},e_m,$ but keeping the form of $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ given in (15)]. We have the following.

• (a) For almost all $Y \in {\mathfrak {v}},$ the vectors UY and VY are linearly independent.

• (b) The subspace ${\text {Span}}(e_{m-1}, e_m)$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant.

• (c) ${\mathfrak {c}}={\text {Ker}}U \cap {\text {Ker}}V,\; {\mathfrak {c}}^\perp =({\text {Ker}}U \cap {\text {Ker}}V)^\perp ,$ and so for almost all $Y \in {\mathfrak {c}}^\perp ,$ the vectors UY and VY are linearly independent.

• (d) For almost all $e \in {\text {Span}}(e_3, \dots , e_{m-2}),$ the 1-forms $\xi _e$ and $\eta _e$ on ${\mathfrak {v}}$ defined by $\xi _e(Y)=\langle UY,e\rangle$ and $\eta _e(Y)=\langle VY,e\rangle$ are linearly independent.

Proof

We cannot have $U=V=0$, as otherwise the subspace ${\mathfrak {c}}^\perp$ is at most 1-dimensional, in contradiction with Lemma 3(b). Furthermore, if U and V are proportional, we can specify the vectors $e_1,e_2,e_{m-1},e_m$ [without changing the form of $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ given in (15)] in such a way that $V=0$ hence obtaining $\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_1$. We can therefore assume that U and V are not proportional.

We use the following well known fact. If $F_1, F_2: {\mathbb {R}}^p \rightarrow {\mathbb {R}}^q$ are linear maps such that ${\text {rk}}(F_1x,F_2x) \le 1$, for all $x \in {\mathbb {R}}^p$, then either $F_1$ and $F_2$ are proportional, or there exist $w \in {\mathbb {R}}^q$ and $\lambda _1, \lambda _2 \in ({\mathbb {R}}^p)^*$ such that $F_1x=\lambda _1(x)w$ and $F_2x=\lambda _2(x)w$, for all $x \in {\mathbb {R}}^p$.

For assertion (a), we apply the above fact to U and V. As we assume that U and V are not proportional, we obtain that $UY=\lambda _1(Y)w,\; VY=\lambda _2(Y)w$ for non-proportional 1-forms $\lambda _1, \lambda _2 \in {\mathfrak {v}}^*$ and for some $w \ne 0$. But this leads to a contradiction with (22).

To prove assertion (b) we note that from assertion (a) and from (21) it follows that ${\text {Span}}(e_{m-1}, e_m) \subset [{\mathfrak {v}},{\mathfrak {n}}'] \subset {\text {Span}}(e_3, \dots , e_m)$, and so $[{\mathfrak {v}},{\mathfrak {n}}'] \cap ([{\mathfrak {v}},{\mathfrak {n}}'])^\perp ={\text {Span}}(e_{m-1}, e_m)$. Hence the subspace ${\text {Span}}(e_{m-1}, e_m)$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by Remark 3.

For assertion (c) we note that from (21) we have ${\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp + {\mathbb {R}}b$. But if $b \notin ({\text {Ker}}U \cap {\text {Ker}}V)^\perp$, then the subspace $\{Y \in {\mathfrak {c}}^\perp \, : \, [Y,{\mathfrak {n}}'] \subset {\text {Span}}(e_{m-1}, e_m)\}=\{Y \in {\mathfrak {c}}^\perp \, : \, UY=VY=0\}$ is 1-dimensional and is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by assertion (b) and Remark 3, in contradiction with Lemma 3(b). It follows that $b \in ({\text {Ker}}U \cap {\text {Ker}}V)^\perp$, and so ${\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp$ and ${\mathfrak {c}}= {\text {Ker}}U \cap {\text {Ker}}V$, as required. Then assertion (a) implies that for almost all $Y \in {\mathfrak {c}}^\perp$, we have ${\text {rk}}(UY|VY) = 2$.

For assertion (d), we apply the above linear-algebraic fact to the conjugates of U and V. As we assume U and V to be not proportional, the condition that the 1-forms $\xi _e$ and $\eta _e$ on ${\mathfrak {v}}$ are linearly dependent for all $e \in {\text {Span}}(e_3, \dots , e_{m-2})$ would imply the existence of $w \in {\mathfrak {v}}\setminus \{0\}$ and $e,e' \in {\text {Span}}(e_3, \dots , e_{m-2})$ such that $UY=\langle w,Y\rangle e$ and $VY=\langle w,Y\rangle e'$, for all $Y \in {\mathfrak {v}}$. But then by assertion (c), ${\mathfrak {c}}^\perp = {\mathbb {R}}w$ which contradicts Lemma 3(b). $\square$

As Cases (i) and (ii) have been already understood, for the rest of the proof we will assume that the conditions of Lemma 4 are satisfied.

As $\phi ({\mathfrak {v}}){\mathfrak {n}}' \subset {\text {Span}}(e_3, \dots , e_m)$, in order to have both $e_1$ and $e_2$ in ${\mathfrak {n}}'$, we need the 2-forms $\omega _1, \omega _2 \in \Lambda ^2({\mathfrak {v}})$ defined by (16) to be linearly independent. From the Jacobi identity we obtain

$$\begin{aligned} \sigma (\omega _1(Y_1,Y_2) UY_3 + \omega _2(Y_1,Y_2) VY_3)=0, \end{aligned}$$

(23)

where $\sigma$ denotes the cyclic permutation of $Y_1,Y_2,Y_3 \in {\mathfrak {v}}$. Taking $Y_1,Y_2 \in {\mathfrak {c}}$ and $Y_3 \in {\mathfrak {c}}^\perp$ in such a way that ${\text {rk}}(UY_3|VY_3) = 2$ we obtain $\omega _1({\mathfrak {c}},{\mathfrak {c}})=\omega _2({\mathfrak {c}},{\mathfrak {c}})=0$. As we also have $\omega _1({\mathfrak {c}},{\mathfrak {c}}^\perp )=\omega _2({\mathfrak {c}},{\mathfrak {c}}^\perp )=0$ by Lemma 3(a) and (16), we obtain $\omega _1({\mathfrak {c}},{\mathfrak {v}})=\omega _2({\mathfrak {c}},{\mathfrak {v}})=0$, and hence the restrictions of $\omega _1$ and $\omega _2$ to ${\mathfrak {c}}^\perp$ must be linearly independent.

Lemma 5

In the assumptions of Lemma 4 we have the following.

• (a) $\dim {\mathfrak {c}}^\perp \le 4$.

• (b) Introduce $K_1, K_2 \in \mathfrak {so}({\mathfrak {c}}^\perp )$ by $\langle K_iY,Y'\rangle =\omega _i(Y,Y')$ for $Y, Y' \in {\mathfrak {c}}^\perp$ and $i=1,2$. Denote $S={\text {Span}}(K_1, K_2) \subset \mathfrak {so}({\mathfrak {c}}^\perp )$ (note that $\dim S = 2$). Then the subalgebra $\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )$ normalises the subspace S.

Proof

For assertion (a), consider the pencil $\mu _1 \omega _1 + \mu _2 \omega _2 \subset \Lambda ^2({\mathfrak {v}})$, where $\mu _1, \mu _2 \in {\mathbb {R}}$. Suppose at least one element of this pencil has rank greater than or equal to 4. Specifying the vectors $e_1,e_2,e_{m-1},e_m$ we can assume, without loss of generality, that ${\text {rk}}\omega _1 \ge 4$.

Let ${\mathcal {U}}\subset {\text {Span}}(e_3, \dots , e_{m-2})$ be the subset of those vectors e for which the 1-forms $\xi _e, \eta _e \in {\mathfrak {v}}^*$ are linearly independent. By Lemma 4(d), the subset ${\mathcal {U}}$ is open and dense in ${\text {Span}}(e_3, \dots , e_{m-2})$. Taking the inner product of (23) with $e \in {\mathcal {U}}$ we obtain $\omega _1 \wedge \xi _e + \omega _2 \wedge \eta _e=0$. By generalised Cartan’s Lemma [1, Lemma 1], there exist 1-forms $\gamma _{11}, \gamma _{12}= \gamma _{21}, \gamma _{22} \in {\mathfrak {v}}^*$ such that $\omega _1 = \gamma _{11} \wedge \xi _e + \gamma _{12} \wedge \eta _e$ and $\omega _2 = \gamma _{21} \wedge \xi _e + \gamma _{22} \wedge \eta _e$. In particular, ${\text {rk}}\omega _1 \le 4$, and so ${\text {rk}}\omega _1 = 4$ by our assumption. Let $L_1=\{Y \in {\mathfrak {v}}\, : i_Y (\omega _1)=0\}$. Then $L_1$ has codimension 4, and $\xi _e(L_1)= \eta _e(L_1)=0$, for all $e \in {\mathcal {U}}$, and hence for all $e \in {\mathfrak {v}}$. It follows that $UL_1=VL_1=0$, and so by Lemma 4(c), ${\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \subset L_1^\perp$ which implies $\dim {\mathfrak {c}}^\perp \le 4$.

Now suppose that ${\text {rk}}(\mu _1 \omega _1 + \mu _2 \omega _2) < 4$, for all $\mu _1, \mu _2 \in {\mathbb {R}}$. As the rank is always even and as $\omega _1$ and $\omega _2$ are linearly independent, we obtain ${\text {rk}}(\mu _1 \omega _1 + \mu _2 \omega _2) = 2$, for all $(\mu _1, \mu _2) \in {\mathbb {R}}^2\setminus \{(0,0)\}$. Then it is easy to see that there exist three linearly independent 1-forms $\zeta _1, \zeta _2, \zeta _3 \in {\mathfrak {v}}^*$ such that $\omega _1=\zeta _1 \wedge \zeta _3$ and $\omega _2=\zeta _2 \wedge \zeta _3$. From (23) we get $\omega _1 \wedge \xi _e + \omega _2 \wedge \eta _e=0$, for all $e \in {\text {Span}}(e_3, \dots , e_{m-2})$ which gives $\zeta _1 \wedge \zeta _3 \wedge \xi _e + \zeta _2 \wedge \zeta _3 \wedge \eta _e=0$. It follows that $\zeta _1 \wedge \zeta _2 \wedge \zeta _3 \wedge \xi _e=0$, and so $\xi _e \in {\text {Span}}(\zeta _1, \zeta _2, \zeta _3)$, and similarly $\eta _e \in {\text {Span}}(\zeta _1, \zeta _2, \zeta _3)$, for all $e \in {\text {Span}}(e_3, \dots , e_{m-2})$. But then the common kernel $L_2$ of the 1-forms $\zeta _1, \zeta _2, \zeta _3$ has codimension 3 and lies in the kernel of both U and V. It follows that ${\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \subset L_2^\perp$ and so $\dim {\mathfrak {c}}^\perp \le 3$.

For assertion (b), we first note that the subspace ${\text {Span}}(e_3, \dots ,e_m)=({\text {Span}}(e_{m-1},e_m))^\perp$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by Lemma 4(b) and Remark 3. Now let $A \in {\mathfrak {h}}$ and $Y,Y' \in {\mathfrak {c}}^\perp$. Then by (9), the component of the vector $[AY,Y']+[Y,AY']$ lying in ${\text {Span}}(e_1,e_2)$ equals $(\omega _1(AY,Y')+\omega _1(Y,AY'))e_1+(\omega _2(AY,Y')+\omega _2(Y,AY'))e_2=\langle [K_1,\pi (A)] Y,Y'\rangle e_1 + \langle [K_2,\pi (A)] Y,Y'\rangle e_2$. As ${\text {Span}}(e_3, \dots ,e_m)$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, the component of the vector $A[Y,Y']$ lying in ${\text {Span}}(e_1,e_2)$ equals the component of the vector $A(\omega _1(Y,Y')e_1+\omega _2(Y,Y')e_2)$ lying in ${\text {Span}}(e_1,e_2)$, which is $\omega _1(Y,Y') (A_{11}e_1+A_{12}e_2) + \omega _2(Y,Y') (A_{21}e_1+A_{22}e_2)= \langle (A_{11}K_1+A_{21}K_2)Y,Y'\rangle e_1 + \langle (A_{12}K_1+A_{22}K_2)Y,Y'\rangle e_2$, where $A_{ij}$ denotes the corresponding entry of the matrix of $({\text {ad}}(A))|_{{\mathfrak {n}}'}$ relative to the basis $\{e_1,e_2, \dots , e_m\}$. We deduce that $[K_1,\pi (A)], [K_2,\pi (A)] \in {\text {Span}}(K_1,K_2)$, as required. $\square$

By Lemmas 3(b) and  5(a), we have $2 \le \dim {\mathfrak {c}}^\perp \le 3$. Moreover, by Lemma 5(b), the subalgebra $\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )$ normalises the 2-dimensional subspace $S={\text {Span}}(K_1,K_2) \!\subset \mathfrak {so}({\mathfrak {c}}^\perp )$. If $\pi ({\mathfrak {h}})$ is abelian, then by Lemma 3(d), we obtain $[{\mathfrak {c}}^\perp , {\mathfrak {c}}^\perp ] \subset {\mathfrak {z}}({\mathfrak {n}})$. Then taking $Y_1,Y_2,Y_3 \in {\mathfrak {c}}^\perp$ in (17) we get $\sum \nolimits _{i=1}^{m} \omega _i(Y_1,Y_2) \phi (Y_3) e_i=0$ which by (21) gives $\omega _1(Y_1,Y_2)UY_3 + \omega _2(Y_1,Y_2)VY_3 =0$. As by Lemma 4(c), the vectors $UY_3$ and $VY_3$ are linearly independent for almost all $Y_3 \in {\mathfrak {c}}^\perp$, we deduce that $\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=\omega _2({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=0$, a contradiction. So the subalgebra $\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )$ is non-abelian. It is easy to see that the only possible case when the normaliser of a two-dimensional subspace S of a subalgebra $\mathfrak {so}({\mathfrak {c}}^\perp ),\; \dim {\mathfrak {c}}^\perp \in \{2,3,4\}$, is non-abelian is the following: $\dim {\mathfrak {c}}^\perp = 4$, so that $\mathfrak {so}({\mathfrak {c}}^\perp ) = \mathfrak {so}(4) = \mathfrak {so}(3) \oplus \mathfrak {so}(3)$ (direct sum of ideals), and then S lies in one of the two $\mathfrak {so}(3)$-components.

In this last remaining case, denote ${\tilde{\omega }}_j \in \Lambda ^2({\mathfrak {c}}^\perp ), \; j=1,2$, the restriction of the 2-form $\omega _j \in \Lambda ^2({\mathfrak {v}})$ to ${\mathfrak {c}}^\perp$, and for $e \in {\text {Span}}(e_3, \dots , e_{m-2})$, denote ${\tilde{\xi }}_e,{\tilde{\eta }}_e \in ({\mathfrak {c}}^\perp )^*$ the restrictions of the 1-forms $\xi _e,\eta _e \in {\mathfrak {v}}^*$, respectively (so that for $Y \in {\mathfrak {c}}^\perp$ we have ${\tilde{\xi }}_e(Y)=\langle UY,e\rangle$ and ${\tilde{\eta }}_e(Y)=\langle VY,e\rangle$). Restricting Eq. (23) to ${\mathfrak {c}}^\perp$ [note that $U{\mathfrak {c}}=V{\mathfrak {c}}=0$ and $\omega _1({\mathfrak {c}},{\mathfrak {v}})=\omega _2({\mathfrak {c}},{\mathfrak {v}})=0$ anyway], we obtain ${\tilde{\omega }}_1 \wedge {\tilde{\xi }}_e + {\tilde{\omega }}_2 \wedge {\tilde{\eta }}_e=0$, for all $e \in {\text {Span}}(e_3, \dots , e_{m-2})$. Applying the Hodge star operator (and noting that the dual vectors to ${\tilde{\xi }}_e$ and ${\tilde{\eta }}_e$ are $U^te, V^te \in {\mathfrak {c}}^\perp$, respectively) gives $i_{U^te}({\star }{\tilde{\omega }}_1)+i_{V^te}({\star }{\tilde{\omega }}_2)=0$. Let ${\star }K_j \in \mathfrak {so}({\mathfrak {c}}^\perp ), \; j=1,2$, be defined by $\langle ({\star }K_j) Y,Y'\rangle ={\star }{\tilde{\omega }}_j(Y,Y')$ for $Y,Y' \in {\mathfrak {c}}^\perp$. Then from the latter equation we obtain $\langle ({\star }K_1)U^te,Y\rangle +\langle ({\star }K_2)V^te,Y\rangle =0$, for all $Y \in {\mathfrak {c}}^\perp$ and all $e \in {\text {Span}}(e_3, \dots , e_{m-2})$. This is equivalent to

$$\begin{aligned} {\tilde{U}} ({\star }K_1) + {\tilde{V}} ({\star }K_2) = 0, \end{aligned}$$

(24)

where ${\tilde{U}}$ and ${\tilde{V}}$ are the restrictions of U and V to ${\mathfrak {c}}^\perp$, respectively. Note that $K_1$ and $K_2$ are linearly independent and belong to the same $\mathfrak {so}(3)$-component of the algebra $\mathfrak {so}({\mathfrak {c}}^\perp )=\mathfrak {so}(3) \oplus \mathfrak {so}(3)$ (direct sum of ideals). As these components are ${\star }$-invariant, we obtain that ${\star }K_1$ and ${\star }K_2$ are also linearly independent and belong to the same $\mathfrak {so}(3)$-component. This implies that $({\star }K_2)^2=\mu {\text {Id}}$ for some $\mu < 0$ and that $({\star }K_1) ({\star }K_2) = \nu {\text {Id}}+ K_3$, where $\nu \in {\mathbb {R}}$, and $K_3 \ne 0$ belongs to the same $\mathfrak {so}(3)$-component of $\mathfrak {so}({\mathfrak {c}}^\perp )$ as ${\star }K_1$ and ${\star }K_2$. In particular, $\det K_3 \ne 0$. We now multiply (24) by ${\tilde{U}}^t$ on the left and by ${\star }K_2$ on the right. We get ${\tilde{U}}^t{\tilde{U}}(\nu {\text {Id}}+ K_3)+\mu {\tilde{U}}^t{\tilde{V}}=0$. But $U^tV$ is symmetric by (22), and so ${\tilde{U}}^t{\tilde{V}}$ is symmetric (as $U{\mathfrak {c}}=V{\mathfrak {c}}=0$) which implies that the $4 \times 4$ matrix ${\tilde{U}}^t{\tilde{U}} K_3$ is also symmetric. Choosing a basis for ${\mathfrak {c}}^\perp$ which diagonalises the semi-definite matrix ${\tilde{U}}^t{\tilde{U}}$ we find that ${\tilde{U}}^t{\tilde{U}} K_3$ can be symmetric only when ${\tilde{U}}^t{\tilde{U}} K_3=0$. But as $\det K_3 \ne 0$, this implies ${\tilde{U}}=0$, that is, $U{\mathfrak {c}}^\perp =0$, which in combination with $U{\mathfrak {c}}=0$ gives $U=0$. This is a contradiction with Lemma 4(a) which completes the proof of the proposition and of Theorem 1. $\square$

2.3 Example

The following example shows that Theorem 1, concerning the 2-step property of ${\mathfrak {n}}$ for the case when the derived algebra ${\mathfrak {n}}'$ is nondegenerate, is “almost" tight in terms of the signature. We construct a nilpotent, metric Lie algebra $({\mathfrak {n}},\langle \cdot ,\cdot \rangle )$ with the following properties:

• $\dim {\mathfrak {n}}= 12,\; \dim {\mathfrak {n}}'=4, \, \dim {\mathfrak {v}}= 8$.

• ${\mathfrak {n}}'$ is Lorentz and ${\mathfrak {v}}$ is of signature (5, 3), so that ${\mathfrak {n}}$ is of signature (8, 4).

• ${\mathfrak {n}}$ is 4-step nilpotent (and ${\mathfrak {n}}'$ is abelian).

• $({\mathfrak {n}},\langle \cdot ,\cdot \rangle )$ is G-geodesic orbit (for G as in Theorem 1).

We define ${\mathfrak {n}}= {\mathfrak {v}}\oplus {\mathfrak {n}}'$, where $\dim {\mathfrak {n}}'=4,\; \dim {\mathfrak {v}}= 8$. We have a basis $\{e_1, e_2,e_3,e_4\}$ for ${\mathfrak {n}}'$, and a basis $\{f_1, \dots , f_8\}$ for ${\mathfrak {v}}$. The inner product $\langle \cdot ,\cdot \rangle$ is defined in such a way that ${\mathfrak {v}}\perp {\mathfrak {n}}'$, and

$$\begin{aligned} \langle \cdot ,\cdot \rangle |_{{\mathfrak {v}}} = \left( \begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} I_2 \\ 0 &{} 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} I_2 &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 &{} 0 &{} 0 \\ I_2 &{} 0 &{} 0 &{} 0 &{} 0 \end{matrix} \right) \quad \text {and}\quad \langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'} = \left( \begin{matrix} 0 &{} 0 &{} 1 \\ 0 &{} I_2 &{} 0 \\ 1 &{} 0 &{} 0 \end{matrix} \right) . \end{aligned}$$

The Lie bracket is defined as follows:

$$\begin{aligned}{} & {} [f_1, e_1]=e_2,\quad [f_2, e_1]=e_3,\quad [f_1, e_2]=[f_2, e_3]=-e_4,\\{} & {} {}[f_1,f_2]=e_1,\quad [f_1,f_6]=e_2,\quad [f_2,f_6]=e_3,\quad [f_1, f_4]=[f_2, f_5]=e_4. \end{aligned}$$

It is not hard to see that the algebra ${\mathfrak {n}}'$ so defined is 4-step nilpotent (in fact, if we disregard the inner product, our algebra ${\mathfrak {n}}'$ is the direct sum of the 6-dimensional ideal ${\text {Span}}(f_1,f_2,e_1,e_2,e_3,e_4)$ (the algebra $L_{6,21}(1)$ in [6]) and the 6-dimensional abelian ideal ${\text {Span}}(f_3, f_4+e_2, f_5+e_3, f_6-e_1, f_7, f_8)$).

We now define, for every $T=X+Y$, where $X=\sum _{i=1}^{4} x_i e_i \in {\mathfrak {n}}', \; Y=\sum _{j=1}^8 y_j f_j \in {\mathfrak {v}}$, the linear operator ${\mathcal {A}}$ on ${\mathfrak {n}}$ such that ${\mathcal {A}}{\mathfrak {n}}' \subset {\mathfrak {n}}',\, {\mathcal {A}}{\mathfrak {v}}\subset {\mathfrak {v}}$, and relative to the chosen bases for ${\mathfrak {v}}$ and ${\mathfrak {n}}'$,

$$\begin{aligned} {\mathcal {A}}|_{{\mathfrak {v}}}=\left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0\\ x_2+y_4 &{} x_3+y_5 &{} 0 &{} -y_1 &{} -y_2 &{} 0 &{} 0 &{} 0 \\ x_1-y_6 &{} 0 &{} 0 &{} 0 &{} 0 &{} y_1 &{} 0 &{} 0 \\ 0 &{} x_1-y_6 &{} 0 &{} 0 &{} 0 &{} y_2 &{} 0 &{} 0 \\ y_2 &{} -y_1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} y_3-x_4 &{} -y_2 &{} y_6-x_1 &{} 0 &{} -x_2-y_4 &{} 0 &{} 0 \\ x_4-y_3 &{} 0 &{} y_1 &{} 0 &{} y_6-x_1 &{} -x_3-y_5 &{} 0 &{} 0 \\ \end{matrix}} \right) \quad \text {and}\quad {\mathcal {A}}|_{{\mathfrak {n}}'}=\left( \begin{matrix} 0 &{} 0 &{} 0 &{} 0 \\ -y_1 &{} 0 &{} 0 &{} 0 \\ -y_2 &{} 0 &{} 0 &{} 0 \\ 0 &{} y_1 &{} y_2 &{} 0 \end{matrix} \right) \, . \end{aligned}$$

A direct calculation shows that ${\mathcal {A}}$ so defined is a skew-symmetric derivation, and that the GO equation $\langle {\mathcal {A}}T'+[T,T'],T\rangle = 0$ [see (1)] is satisfied, for all $T' \in {\mathfrak {n}}$. As ${\mathcal {A}}$ depends linearly on T, one may expect the algebra $({\mathfrak {n}},\langle \cdot ,\cdot \rangle )$ to be even naturally reductive. $\diamondsuit$

This example also shows that a pseudo-Riemannian G-GO nilmanifold with nondegenerate derived algebra loses the property of being 2-step nilpotent already when $\langle \cdot ,\cdot \rangle '$ is Lorentz (for the case when $\langle \cdot ,\cdot \rangle '$ is definite, see Remark 4).

3 Proof of Theorem 2: If $ds^2|_{[{\mathfrak {n}},{\mathfrak {n}}]}$ is Degenerate Then ${\mathfrak {n}}$ is a Double Extension

In this section we consider the case when the restriction of the inner product $\langle \cdot ,\cdot \rangle$ to the derived algebra ${\mathfrak {n}}'$ is degenerate.

We start with the following Lemma.

Lemma 6

Let $(M = G/H, ds^2)$ be a connected pseudo-Riemannian G-geodesic orbit nilmanifold where $,$ with N nilpotent. Let $\langle \cdot ,\cdot \rangle$ denote the inner product on ${\mathfrak {n}}$ induced by $ds^2$. Suppose $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ is degenerate. Let ${\mathfrak {m}}_1$ and ${\mathfrak {e}}$ be subspaces of ${\mathfrak {n}}$ with the following properties:

• (i) ${\mathfrak {e}}\subset {\mathfrak {n}}' \subset {\mathfrak {m}}_1$ (so that, in particular, ${\mathfrak {m}}_1$ is an ideal of ${\mathfrak {n}}$);

• (ii) both ${\mathfrak {m}}_1$ and ${\mathfrak {e}}$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant;

• (iii) $\langle {\mathfrak {e}}, {\mathfrak {m}}_1\rangle = 0$ and $[{\mathfrak {e}},{\mathfrak {m}}_1]=0;$

• (iv) $\dim {\mathfrak {m}}_1 + \dim {\mathfrak {e}}= \dim {\mathfrak {n}}$.

Define the metric nilpotent Lie algebra ${\mathfrak {m}}_0 = {\mathfrak {m}}_1/{\mathfrak {e}}$ with the inner product $\langle \cdot ,\cdot \rangle _0$ induced from ${\mathfrak {m}}_1$ [this is well-defined by (iii)], and the pseudo-Riemannian nilmanifold $(M_0 = G_0/H_0, ds_0^2),$ where $,$ with $N_0$ the (simply connected) Lie group whose Lie algebra is ${\mathfrak {m}}_0,$ $ds_0^2$ is the left-invariant metric on $M_0$ defined by $\langle \cdot ,\cdot \rangle _0,$ and $H_0$ is the maximal connected group of pseudo-orthogonal automorphisms of $\langle \cdot ,\cdot \rangle _0$.

Then $(M_0, ds_0^2)$ is a $G_0$-GO pseudo-Riemannian nilmanifold.

Note that the fact that the inner product $\langle \cdot ,\cdot \rangle _0$ as constructed in Lemma 6 is nondegenerate follows from assumption (iv).

If we denote $m_0=\dim {\mathfrak {e}}$, then the signature of the metric $ds_0^2$ is $(p-m_0,q-m_0)$, where (p, q) is the signature of $ds^2$. In the settings of Lemma 6, we say that the metric Lie algebra $({\mathfrak {n}},\langle \cdot ,\cdot \rangle )$ is a $2m_0$-dimensional double extension of the metric Lie algebra $({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)$. Informally, to get $({\mathfrak {n}},\langle \cdot ,\cdot \rangle )$, we first take the central extension of $({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)$ by ${\mathfrak {e}}$ and then the extension of the resulting Lie algebra ${\mathfrak {m}}_1$ by $m_0$-dimensional space of derivations.

Proof of Lemma 6

To check the $G_0$-GO property for $({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)$ we need to choose (an arbitrary, but fixed) linear complement to ${\mathfrak {e}}$ in ${\mathfrak {m}}_1$, which, with some abuse of notation, we will still denote ${\mathfrak {m}}_0$. Then ${\mathfrak {m}}_1 = {\mathfrak {m}}_0 \oplus {\mathfrak {e}}$. For $X, Y \in {\mathfrak {m}}_0$ we have $\langle X,Y\rangle _0=\langle X,Y\rangle$, as ${\mathfrak {e}}\perp {\mathfrak {m}}_1$ by assumption (iii). We define the Lie bracket $[\cdot ,\cdot ]_0$ on ${\mathfrak {m}}_0$ by $[X,Y]_0=[X,Y]_{{\mathfrak {m}}_0}$, for $X, Y \in {\mathfrak {m}}_0$. It is easy to see that $({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)$ is isomorphic to the quotient algebra ${\mathfrak {m}}_1/{\mathfrak {e}}$.

Let $X \in {\mathfrak {m}}_0$. By the Geodesic Lemma, there exist $A(X) \in {\mathfrak {h}}$ and $k(X) \in {\mathbb {R}}$ such that for all $Y \in {\mathfrak {m}}_0$ we have $\langle [X+A(X),Y],X\rangle = k(X) \langle X,Y\rangle$. By assumption (ii) we have $[A(X),Y] \in {\mathfrak {m}}_1 \, (= {\mathfrak {m}}_0 \oplus {\mathfrak {e}})$, and so we can define an endomorphism D(X) of ${\mathfrak {m}}_0$ by the formula $D(X)Y = [A(X),Y]_{{\mathfrak {m}}_0}$. As ${\text {ad}}_{\mathfrak {g}}(A(X))$ is skew-symmetric and ${\mathfrak {e}}\perp {\mathfrak {m}}_1$, the endomorphism D(X) is skew-symmetric relative to $\langle \cdot ,\cdot \rangle _0$. To see that D(X) is a derivation of the Lie algebra $({\mathfrak {m}}_0, [\cdot ,\cdot ]_0)$ we write, for $Y_1, Y_2 \in {\mathfrak {m}}_0$,

$$\begin{aligned} 0&= ([A(X),[Y_1,Y_2]]-[[A(X),Y_1],Y_2]-[Y_1,[A(X),Y_2]])_{{\mathfrak {m}}_0}\\&= ([A(X),[Y_1,Y_2]])_{{\mathfrak {m}}_0}-([[A(X),Y_1],Y_2])_{{\mathfrak {m}}_0}-([Y_1,[A(X),Y_2]])_{{\mathfrak {m}}_0}\\&= ([A(X),[Y_1,Y_2]_0])_{{\mathfrak {m}}_0}-([D(X)Y_1,Y_2])_{{\mathfrak {m}}_0}-([Y_1,D(X)Y_2])_{{\mathfrak {m}}_0}\\&= D(X)([Y_1,Y_2]_0)-[D(X)Y_1,Y_2]_0-[Y_1,D(X)Y_2]_0, \end{aligned}$$

where in the third line, we used the fact that ${\mathfrak {e}}$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, by assumption (ii).

It follows that there exists $A_0(X) \in {\mathfrak {h}}_0$, where ${\mathfrak {h}}_0$ is the Lie algebra of $H_0$, such that $D(X)Y = [A_0(X),Y]_0$, for all $X,Y \in {\mathfrak {m}}_0$. Then from assumption (iii) and the fact that $\langle [X+A(X),Y],X\rangle = k(X) \langle X,Y\rangle$ it follows that $\langle [X+A_0(X),Y]_0,X\rangle _0 = k(X) \langle X,Y\rangle _0$, for all $X,Y \in {\mathfrak {m}}_0$, as required by the Geodesic Lemma. $\square$

First suppose that $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ has degeneracy 1 and is semidefinite. Denote ${\mathfrak {v}}= ({\mathfrak {n}}')^\perp$ and choose a vector e such that ${\mathfrak {n}}' \cap {\mathfrak {v}}={\mathbb {R}}e$. Denote ${\mathfrak {m}}_1={\mathfrak {n}}' + {\mathfrak {v}}$. Note that $e^\perp = {\mathfrak {m}}_1$ and that all four subspaces ${\mathbb {R}}e, {\mathfrak {n}}', {\mathfrak {v}}$ and ${\mathfrak {m}}_1$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, by Remark 3. To be able to apply Lemma 6 (with ${\mathfrak {e}}={\mathbb {R}}e$) we only need to check that $[e, {\mathfrak {m}}_1]=0$. Taking $T'=e$ and $T = X+Y \in {\mathfrak {m}}_1$ in (1), where $X \in {\mathfrak {n}}'$ and $Y \in {\mathfrak {v}}$, we obtain $\langle [e,X+Y],X\rangle =0$, and so $\langle [e,X],X\rangle =\langle [e,Y],X\rangle =0$, for all $X \in {\mathfrak {n}}', \, Y \in {\mathfrak {v}}$. From the second equation it follows that [e, Y] is a multiple of e, and hence $[e,Y]=0$, for all $Y \in {\mathfrak {v}}$, as ${\text {ad}}_{{\mathfrak {n}}}(Y)$ is nilpotent. From the first equation we also obtain that [e, X] is a multiple of e, as ${\text {ad}}_{{\mathfrak {n}}'}e$ is nilpotent and skew-symmetric and $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ has degeneracy 1 and is semidefinite. Then $[e,X]=0$, for all $X \in {\mathfrak {n}}'$, as ${\text {ad}}_{{\mathfrak {n}}}(X)$ is nilpotent. Thus $[e,{\mathfrak {m}}_1]=0$, and the claim follows from Lemma 6, with ${\mathfrak {e}}={\mathbb {R}}e$.

Next suppose that $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ has degeneracy 2 and is semidefinite. Denote ${\mathfrak {v}}= ({\mathfrak {n}}')^\perp , \; {\mathfrak {o}}={\mathfrak {n}}' \cap {\mathfrak {v}}$ and ${\mathfrak {s}}={\mathfrak {n}}' + {\mathfrak {v}}$. The restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {s}}$ has degeneracy 2 and is semidefinite. We have $\dim {\mathfrak {o}}= {\text {codim}}{\mathfrak {s}}= 2$ and ${\mathfrak {o}}^\perp = {\mathfrak {s}}$. Moreover, all four subspaces ${\mathfrak {o}}, {\mathfrak {n}}', {\mathfrak {v}}$ and ${\mathfrak {s}}$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, by Remark 3. If $[{\mathfrak {o}}, {\mathfrak {s}}]=0$, we can directly apply Lemma 6 with ${\mathfrak {m}}_1={\mathfrak {s}}$ and ${\mathfrak {e}}= {\mathfrak {o}}$, and the claim follows. We therefore assume that $[{\mathfrak {o}}, {\mathfrak {s}}] \ne 0$. Taking $T'=e \in {\mathfrak {o}}$ and $T \in {\mathfrak {s}}$ in (1) we obtain $\langle [e,T],T\rangle =0$. As the restriction of the inner product to ${\mathfrak {s}}$ is semidefinite, of degeneracy 2 (and ${\mathfrak {s}}^\perp = {\mathfrak {o}}$), and ${\text {ad}}_{{\mathfrak {s}}}e$ is both skew-symmetric and nilpotent, we obtain $[e,T] \subset {\mathfrak {o}}$, for all $e \in {\mathfrak {o}}$ and $T \in {\mathfrak {s}}$, and hence $[{\mathfrak {s}},{\mathfrak {o}}] \subset {\mathfrak {o}}$. We obtain a nilpotent representation of the (nilpotent) algebra ${\mathfrak {s}}$ on the 2-dimensional space ${\mathfrak {o}}$. By Engel’s Theorem, we can find a basis $\{e_1,e_2\}$ for ${\mathfrak {o}}$ such that $[{\mathfrak {s}}, e_2]=0$ and $[T,e_1]=\lambda (T)e_2$, for all $T \in {\mathfrak {s}}$, where $\lambda \in {\mathfrak {s}}^*$. As we have assumed that $[{\mathfrak {o}}, {\mathfrak {s}}] \ne 0$, the 1-form $\lambda$ is nonzero (but note that $\lambda ({\mathfrak {o}})=0$). Then $[{\mathfrak {o}}, {\mathfrak {s}}] = {\mathbb {R}}e_2$, and so by Remark 3, the subspace ${\mathbb {R}}e_2$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant. Choose two vectors $f_1, f_2 \in {\mathfrak {n}}$ such that ${\text {Span}}(f_1,f_2) \oplus {\mathfrak {s}}= {\mathfrak {n}}$, and $\langle f_i,f_j\rangle =0,\; \langle f_i,e_j\rangle =\delta _{ij}$, for $i,j=1,2$. We claim that the assumptions of Lemma 6 are satisfied with ${\mathfrak {e}}= {\mathbb {R}}e_2$ and ${\mathfrak {m}}_1 = {\mathbb {R}}f_1 \oplus {\mathfrak {s}}$. Indeed, assumptions (i) and (iv) are obviously true, and for assumption (ii) we note that ${\mathfrak {m}}_1=({\mathbb {R}}e_2)^\perp$ by construction, and hence ${\mathfrak {m}}_1$ is ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant by Remark 3, as ${\mathbb {R}}e_2$ is. It remains to show that $[{\mathfrak {m}}_1,e_2]=0$. As we already know that $[{\mathfrak {s}}, e_2]=0$, it suffices to show that $[f_1,e_2]=0$. Taking $T'=e_2$ and $T =\xi f_1+X\in {\mathfrak {m}}_1$, where $X \in {\mathfrak {s}}, \xi \in {\mathbb {R}}$, in (1) (and using the fact that ${\mathbb {R}}e_2$ and ${\mathfrak {m}}_1$ are orthogonal, ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspaces) we obtain $\langle [f_1,e_2],\xi f_1+X\rangle =0$, for all $X \in {\mathfrak {s}}, \xi \in {\mathbb {R}}$. It follows that $[f_1,e_2]$ is a multiple of $e_2$, which must be zero, by nilpotency. The claim now follows from Lemma 6.

The last case to consider is the one when $\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'}$ has degeneracy 1 and index 1. This is the most involved case. As above, we denote ${\mathfrak {v}}= ({\mathfrak {n}}')^\perp$ and choose a vector e such that ${\mathfrak {n}}' \cap {\mathfrak {v}}={\mathbb {R}}e$. Denote ${\mathfrak {m}}_1={\mathfrak {n}}' + {\mathfrak {v}}$, so that $e^\perp = {\mathfrak {m}}_1$. The subspaces ${\mathbb {R}}e, {\mathfrak {n}}', {\mathfrak {v}}$ and ${\mathfrak {m}}_1$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant, by Remark 3. We claim that the assumptions of Lemma 6 are satisfied with ${\mathfrak {e}}={\mathbb {R}}e$. It is easy to see that the only fact we need to establish is that $[e, {\mathfrak {m}}_1]=0$.

The proof is completed by the following proposition.

Proposition 3

In the above notation, the vector e lies in the centre of ${\mathfrak {m}}_1$.

Proof

Denote $m = \dim {\mathfrak {n}}'$. Seeking a contradiction we assume that $[{\mathfrak {m}}_1,e] \ne 0$.

Let ${\mathfrak {f}}=\{T \in {\mathfrak {n}}\, : \, \langle [T,X],X\rangle =0, \text { for all } X \in {\mathfrak {n}}'\}$. It is easy to see that ${\mathfrak {f}}$ is a subalgebra of ${\mathfrak {n}}$.

Lemma 7

In the above notation, the following holds.

• (a) ${\mathfrak {v}}\subset {\mathfrak {f}}$ and $[{\mathfrak {f}},e]=0$ (so, in particular, $[e,{\mathfrak {v}}]=0$).

• (b) There exists a hyperplane ${\mathfrak {n}}_0 \subset {\mathfrak {n}}',$ with ${\mathfrak {n}}_0 \oplus {\mathbb {R}}e = {\mathfrak {n}}',$ and a basis $\{e_1, \dots , e_{m-1}\}$ for ${\mathfrak {n}}_0$ such that relative to the basis $\{e_1, \dots , e_{m-1},e\}$ for ${\mathfrak {n}}',$ we have

  $$\begin{aligned}{} & {} \langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}'} = \left( {\begin{matrix} 0 &{} 0 &{} 1 &{} 0\\ 0 &{} I_{m-3} &{} 0 &{} 0 \\ 1 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 \end{matrix}} \right) , \quad {\text {ad}}_{{\mathfrak {n}}'}T = \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 \\ VT &{} 0_{m-3} &{} 0 &{} 0 \\ 0 &{} -(VT)^t &{} 0 &{} 0 \\ a(T) &{} (WT)^t &{} 0 &{} 0 \end{matrix}}\right) \quad \text {and }\nonumber \\ {}{} & {} {\text {ad}}_{{\mathfrak {n}}'}e = \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 \\ u &{} 0_{m-3} &{} 0 &{} 0 \\ 0 &{} -u^t &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 \end{matrix}}\right) , \end{aligned}$$

  (25)

  for all $T \in {\mathfrak {f}},$ where $u \in L:={\text {Span}}(e_2, \dots , e_{m-2}),\, u \ne 0, \, a \in {\mathfrak {f}}^*$ and $V,W: {\mathfrak {f}}\rightarrow L$ are linear maps. In particular, $[{\mathfrak {f}},e_{m-1}]=0$.

• (c) The subspaces ${\text {Span}}(u, e_{m-1})$ and ${\mathbb {R}}e_{m-1}$ are ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant. Moreover, for any $A \in {\mathfrak {h}},$ we have $[A,e]=\alpha (A)e, \, [A,u] = \beta (A) e_{m-1}, \, [A,e_{m-1}] = \gamma (A)e_{m-1},$ for some $\alpha ,\beta ,\gamma \in {\mathfrak {h}}^*$.

Proof

For assertion (a), the fact that $[{\mathfrak {f}},e]=0$ easily follows: for all $T \in {\mathfrak {f}}$ and $X \in {\mathfrak {n}}'$, we have $0=\langle [T,X],e\rangle =-\langle [T,e],X\rangle$. Therefore [T, e] is a multiple of e, which must be zero as ${\text {ad}}_{{\mathfrak {n}}'}T$ is nilpotent.

To see that ${\mathfrak {v}}\subset {\mathfrak {f}}$, take $T'=Y \in {\mathfrak {v}}$ and $T=X \in {\mathfrak {n}}'$ in (1). As ${\mathfrak {v}}$ and ${\mathfrak {n}}'$ are orthogonal, ${\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})$-invariant subspaces, se obtain $\langle [X,Y],X\rangle =0$, as required.

For assertion (b), we note that the subspace $[e,{\mathfrak {n}}'] \subset {\mathfrak {n}}'$ does not contain ${\mathbb {R}}e$ (indeed, for no $X \in {\mathfrak {n}}'$ we can have $[X,e]=e$, as ${\text {ad}}_{{\mathfrak {n}}'}X$ is nilpotent). Choose a linear complement ${\mathfrak {n}}_0$ to ${\mathbb {R}}e$ in ${\mathfrak {n}}'$ in such a way that ${\mathfrak {n}}_0 \supset [e,{\mathfrak {n}}']$. The restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}_0$ is nondegenerate and is of Lorentz signature. For $T \in {\mathfrak {f}}$ we define the endomorphism $\phi _T$ of ${\mathfrak {n}}_0$ by $[T,X]=\phi _T X + \mu (X)e$, for $X \in {\mathfrak {n}}_0$. For every $T \in {\mathfrak {f}}$, the endomorphism $\phi _T$ is skew-symmetric (as $\langle e,{\mathfrak {n}}'\rangle =0$) and nilpotent (as $[{\mathfrak {f}},e] = 0$ by assertion (a)). Moreover, the map $\phi : {\mathfrak {f}}\rightarrow \mathfrak {so}({\mathfrak {n}}_0,\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}_0})$ sending T to $\phi _T$ is a Lie algebra homomorphism (as $[{\mathfrak {f}},e] = 0$). Considering the Iwasawa decomposition of the Lie algebra $\mathfrak {so}(m -2, 1)=\mathfrak {so}({\mathfrak {n}}_0,\langle \cdot ,\cdot \rangle |_{{\mathfrak {n}}_0})$, by the argument similar to that in the proof of Proposition 1 [see Eqs. (7) and (8)] we can construct a basis $\{e_1, \dots , e_{m-1}\}$ for ${\mathfrak {n}}_0$ such that the restriction of $\langle \cdot ,\cdot \rangle$ to ${\mathfrak {n}}'$ relative to the basis $\{e_1, \dots , e_{m-1},e\}$ for ${\mathfrak {n}}_0$ has the form as given in (25), and moreover, there is a linear map $V: {\mathfrak {f}}\rightarrow L \,(={\text {Span}}(e_2, \dots , e_{m-2})$ such that for all $T \in {\mathfrak {f}}$ we have $\phi _T e_1 = VT, \; \phi _T e_{m-1}=0$ and $\phi _T e_i = -\langle VT,e_i\rangle e_{m-1}$, for $i=2, \dots , m-2$.

It follows that for some linear map $W: {\mathfrak {f}}\rightarrow L$ and linear forms $a, b \in {\mathfrak {f}}^*$, we have

$$\begin{aligned}{} & {} {}[T, e_1] = VT + a(T)e, \quad [T, e_{m-1}]=b(T)e, \nonumber \\{} & {} {}[T, e_i] = -\langle VT,e_i\rangle e_{m-1} + \langle WT,e_i\rangle e, \quad i=2, \dots , e_{m-2}, \end{aligned}$$

(26)

for all $T \in {\mathfrak {f}}$. In particular, taking $T=e$ and using the fact that $[e,{\mathfrak {n}}_0] \subset {\mathfrak {n}}_0$ (by construction of ${\mathfrak {n}}_0$) we obtain $We=0$ and $a(e)=b(e)=0$. Thus ${\text {ad}}_{{\mathfrak {n}}'}e$ has the form as given in (25), where we denote $u=Ve \in L$. Then from (26) we obtain $[e,u]=-\Vert u\Vert ^2e_{m-1}$, and so $[T,[e,u]]=-\Vert u\Vert ^2 b(T) e$, for all $T \in {\mathfrak {f}}$. But $[T,e]=0$ by assertion (a), which gives $[[T,u],e]= \Vert u\Vert ^2 b(T) e$. As ${\text {ad}}_{{\mathfrak {n}}'}[T,u]$ is nilpotent we get $\Vert u\Vert ^2 b(T) = 0$, for all $T \in {\mathfrak {f}}$. If $b \ne 0$ we get $u=0$ (as the restriction of $\langle \cdot ,\cdot \rangle$ to L is definite), and so $[e,{\mathfrak {n}}']=0$. As $[e,{\mathfrak {v}}]=0$ by assertion (a) we obtain $[e,{\mathfrak {m}}_1]=0$ contradicting our assumption. Therefore $b=0$, and then Eq. (26) imply that ${\text {ad}}_{{\mathfrak {n}}'}T$ has the form given in (25).

The last statement in assertion (b) follows from (25).

For assertion (c), we note that from (25) we obtain $[e,{\mathfrak {n}}']={\text {Span}}(u, e_{m-1})$ and $[e,[e,{\mathfrak {n}}']]={\mathbb {R}}e_{m-1}$, and so the first claim follows from Remark 3. Then the second claim also follows (note that the u-component of [A, u] vanishes as $\langle [A,u],u\rangle =0$ and u lies in the subspace L with a definite inner product). $\square$

Let now $f \notin {\mathfrak {m}}_1$ be a null vector such that $f \perp {\mathfrak {n}}_0$ and $\langle f,e\rangle =1$ (the choice of such an f is not unique); note that ${\mathfrak {m}}_1 \oplus {\mathbb {R}}f = {\mathfrak {n}}$.

Lemma 8

The following holds:

$$\begin{aligned}&{}[{\mathfrak {h}},[f,e]]=0, \end{aligned}$$

(27)

$$\begin{aligned}&{}[e,[f,{\mathfrak {v}}]]=[[f,e],{\mathfrak {v}}]=0, \end{aligned}$$

(28)

$$\begin{aligned}&{}[{\mathfrak {v}}, [f,[f,e]]] = 0, \end{aligned}$$

(29)

$$\begin{aligned}&{}[f,[f,[f,e]]] = 0. \end{aligned}$$

(30)

Proof

From Lemma 7(c), for any $A \in {\mathfrak {h}}$, we have $\langle [A,f],e\rangle = -\langle [A,e],f\rangle = -\alpha (A)$, $\langle [A,f],e_{m-1}\rangle = -\langle [A,e_{m-1}],f\rangle = 0$ and $\langle [A,f],u\rangle = -\langle [A,u],f\rangle = 0$, as $e_{m-1} \in {\mathfrak {n}}_0 \subset f^\perp$. It follows that $[A,f]=-\alpha (A)f + Y + X$, where $Y \in {\mathfrak {v}}$ and $X \in ({\text {Span}}(e_{m-1},u))^\perp \cap {\mathfrak {n}}'$. Then $[A,[f,e]]=[[A,f],e]+[f,[A,e]]=[-\alpha (A)f + Y + X,e]+[f, \alpha (A)e]=0$, as $[Y,e]=0$ by Lemma 7(a) and $[X,e]=0$ by (25). This proves (27).

Take in (1) $T'=[f,e]$ and a non-null vector $T=\mu f + Y + X$, where $Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'$ and $\mu \in {\mathbb {R}}$. As $[A,[f,e]]=0$ by (27) and ${\mathfrak {v}}\perp {\mathfrak {n}}'$ we obtain $\langle [\mu f + Y + X, [f,e]], \mu f + X\rangle =0$, for all $Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'$ and $\mu \in {\mathbb {R}}$, by continuity, from which we get

$$\begin{aligned}{} & {} \langle [Y, [f,e]], X\rangle =0, \quad \langle [X, [f,e]], X\rangle =0, \quad \langle [f, [f,e]], f\rangle =0, \nonumber \\{} & {} \langle [f, [f,e]], X\rangle +\langle [X, [f,e]], f\rangle =0, \end{aligned}$$

(31)

for all $Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'$.

The first equation of (31) implies that [Y, [f, e]] is a multiple of e. But $[Y, [f,e]]=[[Y, f],e]$, as $[Y,e]=0$ by Lemma 7(a), and so $[Y, [f,e]]=[[Y, f],e]=0$, as ${\text {ad}}_{{\mathfrak {n}}'}[Y,f]$ is nilpotent. This proves (28).

We now consider the last equation of (31). We have $\langle [X, [f,e]], f\rangle =\langle -[e, [X,f]] - [f,[e,X]], f\rangle$. As $[e,{\mathfrak {n}}'] \subset {\mathfrak {n}}_0$ (by construction of ${\mathfrak {n}}_0$) and $f \perp {\mathfrak {n}}_0$, we have $\langle [e, [X,f]], f\rangle =0$. Taking $X=x_1e_1+{\tilde{x}}+x_{m-1}e_{m-1}+xe \in {\mathfrak {n}}$, where $x_1, x_{m-1},x \in {\mathbb {R}}$ and ${\tilde{x}} \in L\,(={\text {Span}}(e_2,\dots ,e_{m-2}))$ we obtain from (25) that $[e,X] = x_1 u +\langle {\tilde{x}},u\rangle e_{m-1}$, which gives $\langle [X, [f,e]], f\rangle = -\langle [f,[e,X]], f\rangle = -\langle [f,x_1 u +\langle {\tilde{x}},u\rangle e_{m-1}],f\rangle = -x_1 \langle [f,u],f\rangle - \langle {\tilde{x}},u\rangle \langle [f,e_{m-1}],f\rangle$. From the last equation of (31) we obtain $[f,[f,e]]=\langle [f,u],f\rangle e_{m-1} + \langle [f,e_{m-1}],f\rangle u + \eta e$, for some $\eta \in {\mathbb {R}}$. But then $\eta = 0$ from the third equation of (31). Moreover, as $e \in {\mathfrak {v}}$, we get $[e,[f,e]]=0$ by (28), which implies $[e,[f,[f,e]]]=0$. Substituting the above expression for [f, [f, e]] and using (25) we find

$$\begin{aligned} \langle [f,e_{m-1}],f\rangle =0, \quad \text {and so } [f,[f,e]]=\langle [f,u],f\rangle e_{m-1}. \end{aligned}$$

(32)

But from Lemma 7(a), (b) we have ${\mathfrak {v}}\subset {\mathfrak {f}}$ and $[{\mathfrak {f}},e_{m-1}]=0$ which implies $[[f,[f,e]],{\mathfrak {v}}]=0$, by the second equation of (32) . This establishes (29).

From the second equation of (31) we obtain that $[f,e] \in {\mathfrak {f}}$, and so ${\text {ad}}_{{\mathfrak {n}}'}[f,e]$ has the form given in (25), in particular, $[[f,e],u] \in {\text {Span}}(e_{m-1},e)$. As $[e,[f,u]] \in [e,{\mathfrak {n}}'] = {\text {Span}}(u, e_{m-1})$ (by (25)), we obtain $[f,[e,u]] = [[f,e],u] + [e,[f,u]] \in {\text {Span}}(u,e_{m-1},e)$. But $[e,u]=-\Vert u\Vert ^2e_{m-1}$ by (25), and so we obtain $[f,e_{m-1}]= \rho _1 u +\rho _2 e_{m-1} + \rho _3 e$, for some $\rho _1, \rho _2, \rho _3 \in {\mathbb {R}}$. Then from the first equation of (32) we get $\rho _3=0$. Moreover, as $[f,e] \in {\mathfrak {f}}$, from (25) we find $[[f,e],e_{m-1}]=0$ which implies $[e,[f,e_{m-1}]]=0$ (since $[e,e_{m-1}]=0$ by (25)). From the expression for $[f,e_{m-1}]$ above we obtain $[e,\rho _1 u +\rho _2 e_{m-1}]=0$ which implies $\rho _1=0$, again by (25). Therefore $[f,e_{m-1}]= \rho _2 e_{m-1}$ which gives $[f,e_{m-1}]=0$, by nilpotency. Now Eq. (30) follows from the second equation of (32). $\square$

We can now complete the proof of the proposition. We have ${\mathfrak {n}}= {\mathbb {R}}f \oplus {\mathfrak {m}}_1 = {\mathbb {R}}f \oplus ({\mathfrak {v}}+ {\mathfrak {n}}') = ({\mathbb {R}}f \oplus {\mathfrak {v}}) + {\mathfrak {n}}'$. It follows that the subspace ${\mathfrak {V}}={\mathbb {R}}f \oplus {\mathfrak {v}}$ contains some linear complement to ${\mathfrak {n}}'$ in ${\mathfrak {n}}$, and hence generates ${\mathfrak {n}}$. Then ${\mathfrak {n}}= {\mathfrak {V}}+ [{\mathfrak {V}},{\mathfrak {V}}]+[{\mathfrak {V}},[{\mathfrak {V}},{\mathfrak {V}}]]+ \cdots ,$ and so ${\mathfrak {n}}'=[{\mathfrak {V}},{\mathfrak {V}}]+[{\mathfrak {V}},[{\mathfrak {V}},{\mathfrak {V}}]]+ \cdots$ As we already know that $[e,{\mathfrak {v}}]=0$ [by Lemma 7(a)], to show that $[e,{\mathfrak {m}}_1]=0$ it suffices to prove that $[e,{\mathfrak {n}}']=0$, that is, to prove that $[e,[T_1,[T_2,[\dots ,[T_{r-1},T_r]\dots ]]]]=0$, where $r \ge 2$, and where, for every $i=1, \dots , r$, we have either $T_i=f$ or $T_i \in {\mathfrak {v}}$. The proof goes by induction by $r \ge 2$. If $r=2$ the claim follows from the facts that $[e,{\mathfrak {v}}]=0$ and that $[e,[f,{\mathfrak {v}}]]=0$ [by (28)]. Suppose $r > 2$. If $T_1 \in {\mathfrak {v}}$, then the claim follows by the induction assumption from the fact that $[e,T_1]=0$. Suppose $T_1=f$. Then by the induction assumption it suffices to prove that $[[e,f],[T_2,[T_3,[\dots ,[T_{r-1},T_r]\dots ]]]]=0$. If $T_2 \in {\mathfrak {v}}$, the claim follows from the fact that $[e,[f,{\mathfrak {v}}]]=0$ [by (28)] and the induction assumption [or from (29) if $r=3$]. Suppose $T_2=f$. Then it suffices to prove that $[[[e,f],f],[T_3,[\dots ,[T_{r-1},T_r]\dots ]]=0$. But $[[[e,f],f],f]=0$ by (30) and $[[[e,f],f],{\mathfrak {v}}]=0$ by (29). It follows that $[[[e,f],f],T_i]=0$, for all $i=3, \dots , r$, which completes the proof of Proposition 3. $\square$

With Proposition 3, application of Lemma 6 completes the proof of Theorem 2.