Abstract
The geodesic orbit property is useful and interesting in itself, and it plays a key role in Riemannian geometry. It implies homogeneity and has important classes of Riemannian manifolds as special cases. Those classes include weakly symmetric Riemannian manifolds and naturally reductive Riemannian manifolds. The corresponding results for indefinite metric manifolds are much more delicate than in Riemannian signature, but in the last few years important corresponding structural results were proved for geodesic orbit Lorentz manifolds. Here we extend Riemannian and Lorentz results to transLorentz nilmanifolds. Those are the geodesic orbit pseudo Riemannian manifolds \(M = G/H\) of signature \((n2,2)\) such that a nilpotent analytic subgroup of G is transitive on M. For that we suppose that there is a reductive decomposition \({\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {n}}\text { (vector space direct sum) with } [{\mathfrak {h}},{\mathfrak {n}}] \subset {\mathfrak {n}}\) and \({\mathfrak {n}}\) nilpotent. When the metric is nondegenerate on \([{\mathfrak {n}},{\mathfrak {n}}]\) we show that \({\mathfrak {n}}\) is abelian or 2step nilpotent. That is the same result as for geodesic orbit Riemannian and Lorentz nilmanifolds. When the metric is degenerate on \([{\mathfrak {n}},{\mathfrak {n}}]\) we show that \({\mathfrak {n}}\) is a double extension of a geodesic orbit nilmanifold of either Riemannian or Lorentz signature.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction and Statement of Results
A pseudoRiemannian manifold \((M,ds^2)\) is called a geodesic orbit manifold (or a manifold with homogeneous geodesics, or simply a GO manifold), if every geodesic of M is an orbit of a 1parameter subgroup of the full isometry group \(I(M) = I(M,ds^2)\). One loses no generality if one replaces I(M) by its identity component \(I^0(M)\). If G is a transitive Lie subgroup of \(I^0(M)\), so \((M,ds^2) = (G/H,ds^2)\) where H is an isotropy subgroup of G, and if every geodesic of M is an orbit of a 1parameter subgroup of G, then we say that \((M,ds^2)\) is a Ggeodesic orbit manifold, or a GGO manifold. Clearly every GGO manifold is a GO manifold, but not vice versa. The class of geodesic orbit manifolds includes (but is not limited to) symmetric spaces, weakly symmetric spaces, normal and generalized normal homogeneous spaces, and naturally reductive spaces. For the current state of knowledge in the theory of Riemannian geodesic orbit manifolds we refer the reader to [2] and its bibliography.
In this paper, we study the GO condition for pseudoRiemannian nilmanifolds \((N,ds^2)\), relative to subgroups \(G \subset I(N)\) of the form , where H is an isotropy subgroup. Most of our results apply to the case where \((N,ds^2)\) is a transLorentz manifold, that is, the signature of \(ds^2\) is \((n2,2)\), where \(n = \dim N\).
Our results for GGO manifolds \((M,ds^2) = (G/H,ds^2)\) require the coset space G/H to be reductive. In other words, they make use of an \({\text {Ad}}_G(H)\)invariant decomposition \({\mathfrak {g}}= {\mathfrak {m}}\oplus {\mathfrak {h}}\). Very few structural results are known for indefinite metric GO manifolds that are not reductive, and we always assume that G/H is reductive (see the discussion below).
The GO condition for reductive spaces is well known:
Geodesic Lemma [8] Let \((M,ds^2)=G/H\) be a reductive pseudoRiemannian homogeneous space, with the corresponding reductive decomposition \({\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {m}}\). Then M is a Ggeodesic orbit space if and only if, for any \(T \in {\mathfrak {m}}\), there exist \(A = A(T) \in {\mathfrak {h}}\) and \(k=k(T) \in {\mathbb {R}}\) such that if \(T' \in {\mathfrak {m}}\) then
where \(\langle \cdot ,\cdot \rangle \) denotes the inner product on \({\mathfrak {m}}\) defined by \(ds^2\), and the subscript \({}_{\mathfrak {m}}\) in (1) means taking the \({\mathfrak {m}}\)component in \({\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {m}}\).
Note that \(k(T)=0\) unless T is a null vector [substitute \(T'=T\) in (1)].
Recall that a pseudoRiemannian nilmanifold is a pseudoRiemannian manifold admitting a transitive nilpotent Lie group of isometries. In the Riemannian case, the full isometry group of a nilmanifold \((N,ds^2)\), where N is a transitive nilpotent group of isometries, is the semidirect product , where H is the group of all isometric automorphisms of \((N,ds^2)\) [13, Theorem 4.2]. In other words, N is the nilradical of I(N). In the pseudoRiemannian cases, I(N) might still contain and yet be strictly larger. In indefinite signatures of metric a nilmanifold is not necessarily reductive as a coset space of I(N), and even when it is, N does not have to be a normal subgroup of I(N). Here the GO condition does not rescue us, for there exist 4dimensional, Lorentz GO nilmanifolds that are reductive relative to I(N), but for which N is not an ideal in I(N) [7, Sect. 3]. Moreover, already in dimension 4 (the lowest dimension for homogeneous pseudoRiemannian spaces G/H with H connected that are not reductive), every nonreductive space is a GO manifold when we make a correct choice of parameters [3, Theorem 4.1]. These results explain (and motivate) our study of GGO nilmanifolds , where N is nilpotent, and N is the maximal connected subgroup of isometric automorphisms of \((N,ds^2)\) (although our results remain valid for a smaller subgroup H). Given a reductive GGO transLorentz nilmanifolds \((G/H,ds^2)\), where , with N nilpotent, and the corresponding reductive decomposition \({\mathfrak {g}}= {\mathfrak {h}}\oplus {\mathfrak {n}}\) at the level of Lie algebras, we denote \(\langle \cdot ,\cdot \rangle \) the inner product on \({\mathfrak {n}}\) induced by \(ds^2\), and by \(\langle \cdot ,\cdot \rangle '\), the restriction of \(\langle \cdot ,\cdot \rangle \) to the derived algebra \({\mathfrak {n}}'=[{\mathfrak {n}},{\mathfrak {n}}]\).
The structure of the paper is as follows. Section 2 contains the proof of the first main theorem:
Theorem 1
Let \((M = G/H, ds^2)\) be a connected transLorentz Ggeodesic orbit nilmanifold where \(,\) with N nilpotent. Let \(\langle \cdot ,\cdot \rangle \) denote the inner product on \({\mathfrak {n}}\) induced by \(ds^2\). If \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) is nondegenerate, then N is either abelian or 2step nilpotent.
Remark 1
There are very many connected transLorentz Ggeodesic orbit nilmanifolds as in Theorem 1. They are real forms of the complexifications of Riemannian GO spaces. See [4, Proposition 4.3 and Corollary 5.4] for the collection and [14] for the fact that those real forms are GO. \(\diamondsuit \)
Theorem 1 extends the results of [9, Theorem 2.2] (for the Riemannian signature) and of [11, Theorem 2] and [5, Theorem 7] (for the Lorentz signature) to the transLorentz case. Our proof of Theorem 1 is split into two parts, given in Sects. 2.1 and 2.2, depending on the signature of \(\langle \cdot ,\cdot \rangle '\). In Sect. 2.3 we give an example which shows that the results of Theorem 1 and [11, Theorem 2] are “almost” tight in the sense of the signature: there is a GGO nilmanifold of signature (8, 4), with the Lorentz derived algebra, which is 4step nilpotent.
In Sect. 3 we extend the result of [11, Theorem 3] (for the Lorentz signature) to the transLorentz [signature \((n2,2)\)] setting.
In Theorem 2, stated just below, we prove that if the restriction \(\langle \cdot ,\cdot \rangle '\) is degenerate then \({\mathfrak {n}}\) can be obtained by the double extension procedure from a metric Lie algebra of either a Riemannian or Lorentz nilmanifold (by metric Lie algebra we mean a Lie algebra with an inner product). The double extension construction (which is explained in Sect. 3) is a useful tool in pseudoRiemannian homogeneous geometry, in particular in the theory of biinvariant metrics (see the recent survey [12]) and in the context of GO nilmanifolds [11, Sect. 4]. The precise result is
Theorem 2
Let \((M = G/H, ds^2)\) be a connected transLorentz Ggeodesic orbit nilmanifold where \(,\) with N nilpotent. Let \(\langle \cdot ,\cdot \rangle \) denote the inner product on \({\mathfrak {n}}\) induced by \(ds^2\). If \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) is degenerate, then \(({\mathfrak {n}}, \langle \cdot ,\cdot \rangle )\) is a either a 2dimensional double extension of a metric Lie algebra corresponding to a Lorentz GO nilmanifold, or a 4dimensional double extension of a metric Lie algebra corresponding to a Riemannian GO nilmanifold.
Remark 2
The Lorentz GO nilmanifold in Theorem 2 is geodesic orbit relative to the group \(G_0\), which is the semidirect product of the group of parallel translations and pseudo orthogonal automorphisms, as constructed in Lemma 6, and hence by the results of [11] is either at most 2step nilpotent, or is by itself obtained from a 2dimensional double extension of a metric Lie algebra of a Riemannian nilmanifold (which must be at most 2step nilpotent by [9, Theorem 2.2]). As the composition of two repeated 2dimensional double extensions is equivalent to a single 4dimensional double extension, we deduce that in the assumptions of Theorem 2, the Lie algebra of the nilmanifold M is obtained either by a 2dimensional double extension of a Lorentz Lie algebra \({\mathfrak {m}}_0\) or by a 4dimensional double extension of a Riemannian Lie algebra \({\mathfrak {m}}_0\), where in both cases, \({\mathfrak {m}}_0\) is at most 2step nilpotent.
Note that even a 2dimensional GO double extension of an abelian definite Lie algebra can be of an arbitrarily high step, as shown in [11, Sect. 5]. \(\diamondsuit \)
The authors have no competing interests to declare that are relevant to the content of this article.
2 Proof of Theorem 1: If \(ds^2_{[{\mathfrak {n}},{\mathfrak {n}}]}\) is Nondegenerate Then \({\mathfrak {n}}\) is Either Abelian or 2Step Nilpotent
Given a reductive homogeneous pseudoRiemannian manifold \((G/H,ds^2)\), where , with N nilpotent, we identify \({\mathfrak {n}}= \textrm{Lie}(N)\) with the tangent space to G/H at 1N. Let \(\langle \cdot ,\cdot \rangle \) be the inner product on \({\mathfrak {n}}\) induces by \(ds^2\), and denote \({\mathfrak {n}}'=[{\mathfrak {n}},{\mathfrak {n}}]\).
Assume that the restriction \(\langle \cdot ,\cdot \rangle '\) of the inner product \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}'\) is nondegenerate. Denote \({\mathfrak {v}}=({\mathfrak {n}}')^\perp \); note that \({\mathfrak {n}}\) is the direct orthogonal sum of \({\mathfrak {n}}'\) and \({\mathfrak {v}}\), and both subspaces \({\mathfrak {n}}'\) and \({\mathfrak {v}}\) of \({\mathfrak {n}}\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant.
Remark 3
Note that if \(V_1\) and \(V_2\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspaces of \({\mathfrak {n}}\), then each of the following subspaces is also \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant:
In particular, the centraliser and the normaliser of an \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspace of \({\mathfrak {n}}\) are themselves \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant. \(\diamondsuit \)
Let \((G/H,ds^2)\) be Ggeodesic orbit. Following the first steps in the proof of [5, Theorem 7] and of [11, Theorem 1], we take \(T=X+Y\) and \(T' = X'+Y'\), where \(X,X' \in {\mathfrak {n}}'\), \(Y,Y' \in {\mathfrak {v}}\), and T is nonnull in (1). Then \(k(T)=0\) and there exists \(A=A(X,Y) \in {\mathfrak {h}}\) such that
Taking \(Y'=Y, X'=0\) we obtain, by continuity,
As \({\mathfrak {v}}\) generates \({\mathfrak {n}}\) it follows that
Remark 4
Note that if \(\langle \cdot ,\cdot \rangle '\) is definite, Eq. (4) implies \([{\mathfrak {n}},{\mathfrak {n}}']=0\), and so \({\mathfrak {n}}\) is at most 2step nilpotent, regardless of the signature of \(\langle \cdot ,\cdot \rangle \). In the context of Theorem 1 we can therefore assume that \(\langle \cdot ,\cdot \rangle '\) is indefinite. \(\diamondsuit \)
Separating the \(X'\) and the \(Y'\)components in (2) and using (3) and (4), we find that for all \(X \in {\mathfrak {n}}'\) and \(Y \in {\mathfrak {v}}\) with \(X+Y\) nonnull, there exists \(A=A(X,Y) \in {\mathfrak {h}}\) such that for all \(X' \in {\mathfrak {n}}', \, Y' \in {\mathfrak {v}}\),
Denote \({\mathfrak {s}}:= \mathfrak {so}({\mathfrak {n}}', \langle \cdot ,\cdot \rangle ') \subset \mathfrak {gl}({\mathfrak {n}}')\), the algebra of skewsymmetric endomorphisms relative to the restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}'\). By (4) \({\mathfrak {k}}:= {\text {ad}}_{\mathfrak {g}}({\mathfrak {n}})_{{\mathfrak {n}}'}\) is a subalgebra of \({\mathfrak {s}}\) consisting of nilpotent endomorphisms. In fact, the map \(\phi : {\mathfrak {n}}\rightarrow {\mathfrak {k}}\) defined by \(\phi (T) = {\text {ad}}(T)_{{\mathfrak {n}}'}\) for \(T \in {\mathfrak {n}}\) is a Lie algebra homomorphism. Using Engel’s Theorem, \({\mathfrak {k}}\) is triangular. Thus it is conjugate by an inner automorphism [10, Theorem 2.1] to a subalgebra of the nilpotent part \({\mathfrak {u}}\) of an Iwasawa decomposition \({\mathfrak {s}}={\mathfrak {t}}\oplus {\mathfrak {a}}\oplus {\mathfrak {u}}\). In the following we may (and do) assume \({\mathfrak {k}}\subset {\mathfrak {u}}\).
In view of Remark 4, to prove Theorem 1 we need to consider two cases: when the restrictions of \(\langle \cdot ,\cdot \rangle \) to both \({\mathfrak {n}}'\) and \({\mathfrak {v}}\) are Lorentz, and when restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}\) is trans Lorentz and the restriction to \({\mathfrak {v}}\) is definite.
We consider these two cases separately in the following two subsections. The proof of Theorem 1 will follow from Propositions 1 and 2.
2.1 Both \({\mathfrak {n}}'\) and \({\mathfrak {v}}\) Are Lorentz
In this subsection we additionally assume, in the assumptions of Theorem 1, that the restrictions of \(\langle \cdot ,\cdot \rangle \) to both \({\mathfrak {n}}'\) and \({\mathfrak {v}}\) are both Lorentz. We prove the following.
Proposition 1
Let \((M = G/H, ds^2)\) be a connected pseudoRiemannian Ggeodesic orbit nilmanifold where with N nilpotent. If the restrictions of \(\langle \cdot ,\cdot \rangle \) to both \({\mathfrak {n}}'\) and \({\mathfrak {v}}\) are of Lorentz signature, then N is either abelian or 2step nilpotent.
Proof
Denote \(m = \dim {\mathfrak {n}}'\) (note that \(m \ge 2\)). We adopt the notation and will use the facts stated at the start of this section.
The subalgebra \({\mathfrak {s}}=\mathfrak {so}({\mathfrak {n}}',\langle \cdot ,\cdot \rangle ') \subset \mathfrak {gl}({\mathfrak {n}}')\) of skewsymmetric endomorphisms of \(\langle \cdot ,\cdot \rangle '\) is isomorphic to \(\mathfrak {so}(m 1, 1)\). We can choose a basis \(\{e_1, \dots , e_m\}\) for \({\mathfrak {n}}'\) relative to which the restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}'\) and the nilpotent part \({\mathfrak {u}}\) of the Iwasawa decomposition of \({\mathfrak {s}}\) are given by the following matrices:
As \({\mathfrak {k}}\subset {\mathfrak {u}}\), we obtain a linear map \(\Phi : {\mathfrak {n}}\rightarrow {\text {Span}}(e_2, \dots , e_{m1})\) such that, for all \(T \in {\mathfrak {n}}\),
As \({\mathfrak {u}}\) (and hence \({\mathfrak {k}}\)) is abelian, we obtain \([[{\mathfrak {v}},{\mathfrak {v}}],{\mathfrak {n}}']=0\). Since \({\mathfrak {v}}\) generates \({\mathfrak {n}}\), we obtain \([[{\mathfrak {n}},{\mathfrak {n}}],{\mathfrak {n}}']=0\), which implies that \({\mathfrak {n}}'\) is abelian.
Introduce the 2forms \(\omega _i \in \Lambda ^2({\mathfrak {n}})\) by
As \(\omega _1\) cannot be zero (since \(e_1 \in {\mathfrak {n}}'\)) we obtain \(\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0\).
Using (8) and (9), the Jacobi identity gives
where \(\sigma \) denotes the cyclic permutation of \(T_1,T_2,T_3 \in {\mathfrak {n}}\).
Consider the following two subspaces of \({\mathfrak {n}}\):
[the fact that \({\text {Ker}}\Phi \) is the centraliser of \({\mathfrak {n}}'\) in \({\mathfrak {n}}\) follows from (8)]. By [11, Theorem 1(b)] we can (and will) assume that \({\mathfrak {c}}\) is degenerate. Furthermore, we can assume that \({\mathfrak {c}}\ne {\mathfrak {n}}\) (equivalently, \(\Phi \ne 0\)), as otherwise the algebra \({\mathfrak {n}}\) is 2step nilpotent by (8).
In these notations and assumptions, we have the following.
Lemma 1

(a)
\({\mathfrak {n}}' \subset {\mathfrak {q}}\subset {\mathfrak {c}},\) and both \({\mathfrak {q}}\) and \({\mathfrak {c}}\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant ideals of \({\mathfrak {n}}\). Moreover, \([{\mathfrak {c}},{\mathfrak {c}}] \subset {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'\).

(b)
\({\text {codim}}{\mathfrak {q}}= 2\) and \({\text {codim}}{\mathfrak {c}}\in \{1,2\}\) (equivalently, \({\text {rk}}\, \Phi \in \{1,2\}\)).

(c)
Let \(e \in {\mathfrak {c}}\cap {\mathfrak {c}}^\perp \) be a nonzero (necessarily null) vector. Then \(e \in {\mathfrak {v}},\) the line \({\mathbb {R}}e\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant and \([e,e^\perp ]= 0\).

(d)
Let \(f \in {\mathfrak {v}}\) be a null vector such that \(f \notin e^\perp \) and \(\langle f,e\rangle =1\). Then \([{\mathfrak {h}},[f,e]]=0\).

(e)
\([f,e] \in {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'\) and \(e \in {\mathfrak {q}}\).
Proof
(a) The fact that \({\mathfrak {n}}' \subset {\mathfrak {q}}\) follows from (10). Furthermore, taking \(T_1 \in {\mathfrak {q}}\) in (11) we obtain \(\omega _1(T_2,T_3) \Phi (T_1)=0\) which implies \({\mathfrak {q}}\subset {\mathfrak {c}}\). As both \({\mathfrak {q}}\) and \({\mathfrak {c}}\) contain \({\mathfrak {n}}'\), they are ideals of \({\mathfrak {n}}\). The fact that \({\mathfrak {c}}\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant follows from Remark 3. Moreover, by Remark 3, \({\mathfrak {q}}\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant provided \({\mathbb {R}}e_m\) is. To see the latter, we note that \([{\mathfrak {v}}, {\mathfrak {n}}']\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant. From (8) (and the fact that \(\Phi \ne 0\)), we have \(e_m \in [{\mathfrak {v}}, {\mathfrak {n}}'] \subset {\text {Span}}(e_2, \dots , e_m)\). Then \([{\mathfrak {v}}, {\mathfrak {n}}'] \cap ([{\mathfrak {v}}, {\mathfrak {n}}'])^\perp = {\mathbb {R}}e_m\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, by Remark 3. Finally, the fact that \([{\mathfrak {c}},{\mathfrak {c}}] \subset {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'\) follows from the Jacobi identity, as \([{\mathfrak {c}},{\mathfrak {n}}']=0\).
(b) If \({\text {rk}}\Phi \ge 3\), then for almost all triples \(T_1,T_2,T_3 \in {\mathfrak {n}}\), the vectors \(\Phi T_1, \Phi T_2, \Phi T_3 \in {\text {Span}}(e_2, \dots , e_{m1})\) are linearly independent, and so \(\omega _1=0\) by (11). This is a contradiction, so \({\text {rk}}\Phi \le 2\). As \(\Phi \ne 0\) we obtain \({\text {codim}}{\mathfrak {c}}\, (= {\text {rk}}\Phi ) \in \{1,2\}\). Now as \({\mathfrak {q}}\subset {\mathfrak {c}}\) by (a) and as \({\mathfrak {q}}\) is the null space of the skewsymmetric form \(\omega _1\), the codimension of \({\mathfrak {q}}\) must be a positive even number. Furthermore, from (a) we have \(\omega _1({\mathfrak {c}},{\mathfrak {c}})=0\). If \({\text {codim}}{\mathfrak {c}}= 1\), this implies \({\text {codim}}{\mathfrak {q}}=2\). If \({\text {rk}}\Phi =2\), we take \(T_1,T_2 \in {\mathfrak {n}}\) in (11) such that the vectors \(\Phi T_1, \Phi T_2 \in {\text {Span}}(e_2, \dots , e_{m1})\) are linearly independent and take \(T_3 \in {\mathfrak {c}}\). We obtain \(\omega _1(T_1,{\mathfrak {c}})=\omega _1(T_2,{\mathfrak {c}})=0\). As \({\text {Span}}(T_1,T_2) \oplus {\mathfrak {c}}= {\mathfrak {n}}\) we get \(\omega _1({\mathfrak {n}}, {\mathfrak {c}})=0\), and so \({\mathfrak {c}}\subset {\mathfrak {q}}\) by (12) which implies \({\mathfrak {c}}={\mathfrak {q}}\) by (a).
(c) As \({\mathfrak {c}}\) is degenerate, the space \({\mathfrak {c}}\cap {\mathfrak {c}}^\perp \) has dimension 1 and is spanned by a (nonzero) null vector e. As \({\mathfrak {n}}' \subset {\mathfrak {c}}\) by (a) we obtain \(e \in {\mathfrak {v}}\). The subspace \({\mathbb {R}}e\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant as \({\mathfrak {c}}\) is (and by Remark 3). Then (5) with \(Y=e, \, Y' \in e^\perp \cap {\mathfrak {v}}\) implies that \(\langle [e,e^\perp \cap {\mathfrak {v}}],X\rangle =0\) for all \(X \in {\mathfrak {n}}'\) such that \(e+X\) is nonnull. This gives \([e,e^\perp \cap {\mathfrak {v}}] = 0\). As \(e \in {\mathfrak {c}}\) we have \([e, {\mathfrak {n}}'] = 0\), and so \([e,e^\perp ] = 0\).
(d) Choose \(f \in {\mathfrak {v}}\) to be a null vector such that \(f \notin {\mathfrak {c}}\) and \(\langle f,e\rangle =1\) (this choice is not unique). Let \(A \in {\mathfrak {h}}\). By (c), \({\mathbb {R}}e\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, and so \([A,e]=ae\), for some \(a \in {\mathbb {R}}\), and so \(\langle [A,f],e\rangle =a\). As \(e^\perp \oplus {\mathbb {R}}f = {\mathfrak {n}}\), we have \([A,f] + af \in e^\perp \). Then \([[A,f],e] = a [f,e]\) since \([e,e^\perp ] = 0\) by (c). As \([f,[A,e]]=a[f,e]\), the claim follows.
(e) Choose f as in (d). Then \(e^\perp \oplus {\mathbb {R}}f = {\mathfrak {n}}\), and so from (c) we have \([e, {\mathfrak {n}}] \subset {\mathbb {R}}[f,e]\). Then from (d) and (6), with \(X=[f,e]\), we obtain \([Y,[f,e]]=0\), for all \(Y \in {\mathfrak {v}}\) such that \(Y+[f,e]\) is nonnull, and hence for all \(Y \in {\mathfrak {v}}\). As \({\mathfrak {n}}'\) is abelian, we obtain \([f,e] \in {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'\). But from (8), \({\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}' \subset {\text {Span}}(e_2, \dots , e_m)\) (as \(\Phi \ne 0\)), and so \(\langle [f,e],e_m\rangle =0\) by (7). So \(\langle [{\mathfrak {n}},e],e_m\rangle =0\) and the claim follows by (12). \(\square \)
Now choose \(e \in {\mathfrak {c}}\cap {\mathfrak {c}}^\perp \) as in Lemma 1(c) and choose \(f \in {\mathfrak {v}}\setminus e^\perp \) as in Lemma 1(d). By Lemma 1(a) we have \({\mathfrak {n}}' \subset {\mathfrak {q}}\subset {\mathfrak {c}}\subset e^\perp \), and by Lemma 1(b), \({\text {codim}}{\mathfrak {q}}= 2\) (and then either \({\mathfrak {c}}= {\mathfrak {q}}\) or \({\mathfrak {c}}=e^\perp \)). As f is not contained in \(e^\perp \), and hence in \({\mathfrak {c}}\), we have \(\Phi f \ne 0\) by (8). Without loss of generality (scaling f and e and specifying the orthonormal basis \(\{e_2, \dots , e_{m1}\}\)) we can assume that \(\Phi f = e_2\), and so by (8),
Note that with this choice of the basis, \({\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}' \subset {\text {Span}}(e_3, \dots , e_m)\).
Moreover, from Lemma 1(e) (and Lemma 1(a)), the 2dimensional subspace \({\mathfrak {q}}^\perp \) contains e and lies in \({\mathfrak {v}}\). Thus we have \({\mathfrak {q}}^\perp = {\text {Span}}(e, Y_0)\) for some \(Y_0 \notin {\mathfrak {q}}, \, Y_0 \perp e\), with \(\langle Y_0,Y_0\rangle \ne 0\). Note that \(e^\perp = {\mathbb {R}}Y_0 \oplus {\mathfrak {q}}\), and so \({\mathbb {R}}f \oplus {\mathbb {R}}Y_0 \oplus {\mathfrak {q}}= {\mathfrak {n}}\). As \(\omega _1({\mathfrak {q}}, {\mathfrak {n}}) = 0\) and \(\omega _1 \ne 0\), we must have \(\omega _1(f,Y_0) \ne 0\). Denote \(\kappa _i=\omega _i(f,Y_0)\), so that \([f,Y_0] = \sum _{i=1}^m \kappa _i e_i\) [by (9)], with \(\kappa _1 \ne 0\).
As \(Y_0 \perp {\mathfrak {q}}\), from (5) with \(Y=Y_0\) and \(Y' \in {\mathfrak {q}}\cap {\mathfrak {v}}\) we obtain \(\langle [Y_0,Y'],X\rangle =0\), for all \(Y' \in {\mathfrak {q}}\cap {\mathfrak {v}}\) and all \(X \in {\mathfrak {n}}'\) such that \(Y_0+X\) is nonnull, which implies \([Y_0,{\mathfrak {q}}\cap {\mathfrak {v}}] = 0\). Moreover, as \({\mathfrak {q}}^\perp \) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, for any \(A \in {\mathfrak {h}}\) we have \([A,Y_0] \in {\text {Span}}(e, Y_0)\), and so \([A,Y_0]= \mu e\), for some \(\mu \in {\mathbb {R}}\), since \(Y_0\) is nonnull and \(Y_0 \perp e\).
Then from (6) with \(X=[f,Y_0]\) we obtain
for all \(Y \in {\mathfrak {v}}\) such that \(Y+[f,Y_0]\) is nonnull [note that here we have a particular \(A=A(X,Y) \in {\mathfrak {h}}\)]. Let \([A,f]= a f + b Y_0 + Y'\), where \(Y' \in {\mathfrak {q}}\cap {\mathfrak {v}}\). Then \([[A,f],Y_0] = a [f,Y_0]\) as \([Y_0,{\mathfrak {q}}\cap {\mathfrak {v}}] = 0\). Furthermore, \([f,[A,Y_0]] = \mu [f,e]\). By Lemma 1(e) we have \([f,e] \in {\mathfrak {z}}({\mathfrak {n}}) \cap {\mathfrak {n}}'\), and so \([f,e] \in {\text {Span}}(e_3, \dots , e_m)\). Take \(Y = f + \lambda e\) in (14), where \(\lambda \in {\mathbb {R}}\) is chosen in such a way that \(Y+X=f+\lambda e + [f,Y_0]\) is nonnull. As \(e \in {\mathfrak {c}}\) we have \([e,[f,Y_0]]=0\), and so we obtain \(a [f,Y_0] + [f,[f,Y_0]] \in {\text {Span}}(e_3, \dots , e_m)\). Substituting \([f, Y_0]= \sum _{i=1}^m \kappa _i e_i\) and using (13) we get \(a(\kappa _1 e_1 + \kappa _2 e_2) + \kappa _1 e_2 = 0\) which implies \(\kappa _1 = 0\), a contradiction. \(\square \)
2.2 TransLorentz \({\mathfrak {n}}'\), definite \({\mathfrak {v}}\)
In this subsection we consider the last remaining case in the proof of Theorem 1. We prove the following.
Proposition 2
Let \((M = G/H, ds^2)\) be a connected pseudoRiemannian Ggeodesic orbit nilmanifold where with N nilpotent. Denote \({\mathfrak {n}}'=[{\mathfrak {n}},{\mathfrak {n}}]\) and \({\mathfrak {v}}=({\mathfrak {n}}')^\perp \). If the restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}'\) is transLorentz, and the restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {v}}\) is definite, then N is either abelian or 2step nilpotent.
Proof
Denote \(m = \dim {\mathfrak {n}}'\) (we can assume that \(m \ge 4\), as otherwise the claim follows from [11, Theorem 1(b)]). We adopt the notation and will use the facts stated at the start of the section.
The subalgebra \({\mathfrak {s}}=\mathfrak {so}({\mathfrak {n}}',\langle \cdot ,\cdot \rangle ') \subset \mathfrak {gl}({\mathfrak {n}}')\) of skewsymmetric endomorphisms of \(\langle \cdot ,\cdot \rangle '\) is isomorphic to \(\mathfrak {so}(m2, 2)\). We can choose a basis \(\{e_1, \dots , e_m\}\) for \({\mathfrak {n}}'\) relative to which the restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}'\) and the nilpotent part \({\mathfrak {u}}\) of the Iwasawa decomposition of \({\mathfrak {s}}\) are given by the following matrices:
The homomorphism \(\phi : {\mathfrak {n}}\rightarrow {\mathfrak {u}}\) [given by \(\phi (T) = {\text {ad}}(T)_{{\mathfrak {n}}'}\) for \(T \in {\mathfrak {n}}\)] defines linear maps \(U, V:{\mathfrak {n}}\rightarrow {\mathbb {R}}^{m4}={\text {Span}}(e_3, \dots , e_{m2})\) and vectors \(a, b \in {\mathfrak {n}}\) such that for \(T \in {\mathfrak {n}}\), the corresponding entries of the matrix \(\phi (T)\) in the notation of (15) are given by \(u =UT, \, v =VT, \, \alpha =\langle a,T\rangle \) and \(\beta =\langle b,T\rangle \).
Lemma 2
We have \(\phi ({\mathfrak {n}}')=0,\) and so the subalgebra \({\mathfrak {n}}'\) is abelian.
Proof
Clearly \(\phi ({\mathfrak {n}}') \subset [{\mathfrak {u}}, {\mathfrak {u}}]\). From (15), the subalgebra \([{\mathfrak {u}},{\mathfrak {u}}]\) is the subspace of elements of \({\mathfrak {u}}\) [as given in (15)] with \(v=0\) and \(\alpha =0\), so that for \(X \in {\mathfrak {n}}'\), we have \(VX=0\) and \(\langle a, X\rangle =0\). For \(X' \in {\mathfrak {n}}\) we have \(0= [X,X'] + [X',X] = \phi (X)X' + \phi (X')X\) which gives \(x_1' UX + x_1 UX' = 0\) and \(x_1'\langle b, X\rangle + x_1\langle b, X'\rangle =0\), where \(x_1\) and \(x_1'\) are the \(e_1\)components of the vectors X and \(X'\), respectively. This implies \(U{\mathfrak {n}}'=0\) and \(\langle b,{\mathfrak {n}}'\rangle =0\), so \(\phi ({\mathfrak {n}}')= 0\), as required. But \(\phi (T)X=[T,X]\) for \(T \in {\mathfrak {n}}\) and \(X \in {\mathfrak {n}}'\), and the second claim follows. \(\square \)
From Lemma 2 it follows that the subalgebra \({\mathfrak {k}}=\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}\) is abelian. It is not hard to see using the root decomposition of \({\mathfrak {u}}\) relative to the abelian subalgebra \({\mathfrak {a}}\subset {\mathfrak {s}}\) in the Iwasawa decomposition (or to calculate directly), that the algebra \({\mathfrak {u}}\) contains three different maximal abelian subalgebras given below [in the notation of (15)]:

(i)
\({\mathfrak {u}}_1 = \{Q \in {\mathfrak {u}}\, : \, v=0\}\).

(ii)
\({\mathfrak {u}}_2 = {\mathbb {R}}\left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ u_1 &{} v_1 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0_{m5} &{} 0 &{} 0 \\ 0 &{} 0 &{} u_1 &{} 0 &{} 0 &{} 1 \\ 0 &{} 0 &{} v_1 &{} 0 &{} 0 &{} 0 \end{matrix}}\right) \oplus \left\{ \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ w &{} 0 &{} 0 &{} 0_{m5} &{} 0 &{} 0 \\ 0 &{} \beta &{} 0 &{}  w^t &{} 0 &{} 0 \\ \beta &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \end{matrix}}\right) \, : \, w \in {\mathbb {R}}^{m5}, \, \beta \in {\mathbb {R}}\right\} \), where \(v_1 \ne 0\) [up to specifying a basis in \({\mathbb {R}}^{m5}={\text {Span}}(e_4, \dots , e_{m2})\)].

(iii)
\({\mathfrak {u}}_3\) is a maximal abelian subalgebra of the Heisenberg algebra \(\left\{ \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0\\ u &{} v &{} 0_{m4} &{} 0 &{} 0 \\ 0 &{} \beta &{}  u^t &{} 0 &{} 0 \\ \beta &{} 0 &{} v^t &{} 0 &{} 0 \end{matrix}}\right) \, : \, u,v \in {\mathbb {R}}^{m4}, \, \beta \in {\mathbb {R}}\right\} \subset {\mathfrak {u}}\).
We will consider these three cases separately, but following the same pattern. Let \({\mathfrak {c}}= {\text {Ker}}\phi \cap {\mathfrak {v}}\) be the centraliser of \({\mathfrak {n}}'\) in \({\mathfrak {v}}\), and \({\mathfrak {c}}^\perp \) be its orthogonal complement in \({\mathfrak {v}}\). By Remark 3, both subspaces \({\mathfrak {c}}\) and \({\mathfrak {c}}^\perp \) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant. We will always assume that the subspace \({\mathfrak {c}}^\perp \) is nontrivial (equivalently, \(\phi \ne 0\)), for otherwise \({\mathfrak {n}}\) is at most 2step nilpotent.
Denote \(\pi : {\mathfrak {h}}\rightarrow \mathfrak {so}({\mathfrak {c}}^\perp )\) the restriction of the representation of \({\mathfrak {h}}\) to \({\mathfrak {c}}^\perp \), so that \(\pi (A)Y=[A,Y]\) for \(A \in {\mathfrak {h}}\) and \(Y \in {\mathfrak {c}}^\perp \).
Lemma 3

(a)
If \(L \subset {\mathfrak {c}}^\perp \) is an \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspace, then \([L, L^\perp ]=0\).

(b)
The subspace \({\mathfrak {c}}^\perp \) has no 1dimensional \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspaces.

(c)
If \(L \subset {\mathfrak {c}}^\perp \) is a 2dimensional \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspace, then \([L,L] \subset {\mathfrak {z}}({\mathfrak {n}}),\) where \({\mathfrak {z}}({\mathfrak {n}})\) is the centre of \({\mathfrak {n}}\).

(d)
If the subalgebra \(\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )\) is abelian, then \([{\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ] \subset {\mathfrak {z}}({\mathfrak {n}})\).
Proof
Assertion (a) follows from (5) if we take \(Y \in L\) and \(Y' \in L^\perp \).
For assertion (b), suppose that for a nonzero \(Y \in {\mathfrak {c}}^\perp \), the space \({\mathbb {R}}\, Y\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant. Then \([{\mathfrak {h}}, Y]=0\), and so \([Y, {\mathfrak {v}}]=0\) by (6). As \({\mathfrak {v}}\) generates \({\mathfrak {n}}\), we obtain \([Y, {\mathfrak {n}}]=0\), and in particular, \(\phi (Y)=0\) contradicting the fact that \(Y \in {\mathfrak {c}}^\perp \setminus \{0\}\).
For assertion (c), suppose that \(L={\text {Span}}(Y_1,Y_2) \subset {\mathfrak {c}}^\perp \) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, with the vectors \(Y_1\) and \(Y_2\) being orthonormal. Then for any \(A \in {\mathfrak {h}}\), we obtain that \(AY_1\) is a multiple of \(Y_2\), and \(AY_2\) is a multiple of \(Y_1\). Hence \(AX=0\), where \(X=[Y_1,Y_2]\), and so \([X,{\mathfrak {v}}]=0\), by (6). As \({\mathfrak {v}}\) generates \({\mathfrak {n}}\), the subspace \([L,L]={\mathbb {R}}X\) lies in the centre of \({\mathfrak {n}}\).
For assertion (d), suppose that the subalgebra \(\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )\), is abelian. Then \({\mathfrak {c}}^\perp \) is the direct, orthogonal sum of \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspaces of dimension 1 or 2 each. But by assertion (b), there can be no 1dimensional subspaces, and then the claim follows from assertions (c) and (a). \(\square \)
Introduce the 2forms \(\omega _i \in \Lambda ^2({\mathfrak {v}})\) by
Then
From Lemma 2 (and the fact that \([{\mathfrak {c}},{\mathfrak {n}}']=0\)) we have \({\mathfrak {n}}'=[{\mathfrak {v}},{\mathfrak {v}}] + [{\mathfrak {v}}, {\mathfrak {n}}'] = [{\mathfrak {v}},{\mathfrak {v}}] + \phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}'\).
We now separately consider three cases for \(\phi ({\mathfrak {n}})\) as given above.
Case (i): \(\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_1\). Then \(\phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}' \subset {\text {Span}}(e_2, \dots ,e_m)\). As \(e_1 \in {\mathfrak {n}}'= [{\mathfrak {v}},{\mathfrak {v}}] + \phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}'\), we obtain \(\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0\). The Jacobi identity gives \(\sigma (\omega _1(Y_1,Y_2)(\langle a, Y_3\rangle e_2 + UY_3  \langle b,Y_3\rangle )=0\), where \(\sigma \) denotes the cyclic permutation of \(Y_1,Y_2,Y_3 \in {\mathfrak {v}}\), which can be written as
Taking \(Y_1, Y_2 \in {\mathfrak {c}}\) and \(Y_3 \in {\mathfrak {c}}^\perp \) we obtain \(\omega _1({\mathfrak {c}},{\mathfrak {c}}) = 0\). By Lemma 3(a) we have \([{\mathfrak {c}},{\mathfrak {c}}^\perp ]=0\), and so by (16) we obtain that also \(\omega _1({\mathfrak {c}},{\mathfrak {c}}^\perp ) = 0\). As \(\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0\), we deduce that \(\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ) \ne 0\). Now if \({\text {rk}}\phi (=\dim {\mathfrak {c}}^\perp ) > 2\), then the elements \(\phi (Y_1), \, \phi (Y_2)\) and \(\phi (Y_3)\) are linearly independent for almost all triples of vectors \(Y_1, Y_2, Y_3 \in {\mathfrak {v}}\), and so (18) implies \(\omega _1({\mathfrak {v}},{\mathfrak {v}}) = 0\), a contradiction. By Lemma 3(b), we have \(\dim {\mathfrak {c}}^\perp > 1\), and so the only remaining possibility is \(\dim {\mathfrak {c}}^\perp = 2\). But then from Lemma 3(c) we obtain \([{\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ] \subset {\mathfrak {z}}({\mathfrak {n}})\). Taking \(Y_1, Y_2, Y_3 \in {\mathfrak {c}}^\perp \) in Eq. (17), we get \(\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=0\), a contradiction.
Case (ii): \(\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_2\). For \(Y \in {\mathfrak {v}}\), we have
where \(\lambda , \mu \in {\mathbb {R}}\) and \(W: {\mathfrak {v}}\rightarrow {\mathbb {R}}^{m5}={\text {Span}}(e_4, \dots , e_{m2})\). We can assume that \(a \ne 0\) and \(\mu \ne 0\), for otherwise \(\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_1\).
Arguing similarly to the previous case, we see that \(\phi ({\mathfrak {c}}^\perp ){\mathfrak {n}}' \subset {\text {Span}}(e_2, \dots ,e_m)\), and so we must have \(\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0\). From the Jacobi identity we obtain
where \(\sigma \) denotes the cyclic permutation of \(Y_1,Y_2,Y_3 \in {\mathfrak {v}}\). From (20) with \(Y_1, Y_2 \in {\mathfrak {c}}\) and \(Y_3 \in {\mathfrak {c}}^\perp \) we obtain \(\omega _1({\mathfrak {c}},{\mathfrak {c}}) = 0\). Furthermore, we have \([{\mathfrak {c}},{\mathfrak {c}}^\perp ]=0\) by Lemma 3(a), and so \(\omega _1({\mathfrak {c}},{\mathfrak {c}}^\perp ) = 0\) by (16). It follows that \(\omega _1({\mathfrak {c}}, {\mathfrak {v}})=0\), and so we must have \(\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ) \ne 0\), as \(\omega _1({\mathfrak {v}},{\mathfrak {v}}) \ne 0\).
From the \(e_2\)component of (20) we obtain \(\omega _1 \wedge \alpha = 0\), where \(\alpha \) is the 1form on \({\mathfrak {v}}\) defined by \(\alpha (Y)=\langle a,Y\rangle \). By generalised Cartan’s Lemma [1, Lemma 1] we get \(\omega _1 = \gamma \wedge \alpha \) for some 1form \(\gamma \in {\mathfrak {v}}^*\). As \(\omega _1 \ne 0\), the 1form \(\gamma \) is not a multiple of \(\alpha \). Moreover, as \(\omega _1({\mathfrak {c}}, {\mathfrak {v}})=0\), both the vector a and the vector \(c \in {\mathfrak {v}}\) dual to \(\gamma \) lie in \({\mathfrak {c}}^\perp \). Taking the inner product of (20) with \(e_s, \, s=4, \dots , m2\), we find that \(W^t e_s \in {\text {Span}}(a,c)\). Hence in the matrix \(\phi (Y)\) given in (19), for all \(Y \in ({\text {Span}}(a,c))^\perp \cap {\mathfrak {v}}\), we have \(WY=\langle a,Y\rangle =0\). We first suppose that \(b \in {\text {Span}}(a,c)\). Then \({\mathfrak {c}}^\perp \subset {\text {Span}}(a,c)\), and as \(\dim {\mathfrak {c}}^\perp > 1\) by Lemma 3(b), we deduce that \(\dim {\mathfrak {c}}^\perp = 2\) and hence, that \([{\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp ]\) lies in the centre of \({\mathfrak {n}}\), by Lemma 3(c). But now if we take \(Y_1, Y_2, Y_3 \in {\mathfrak {c}}^\perp \) with \(\langle a,Y_3\rangle \ne 0\) in (17), then from the \(e_2\)component we get \(\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=0\) which is a contradiction.
We therefore suppose that \(b \notin {\text {Span}}(a,c)\), and so \({\text {Span}}(a,b) \subset {\mathfrak {c}}^\perp \subset {\text {Span}}(a,b,c)\) from (19). Note that the subspace \([{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']]\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by Remark 3. As \([Y,{\mathfrak {n}}']=\phi (Y){\mathfrak {n}}'\) for \(Y \in {\mathfrak {v}}\), Eq. (19) gives that the subspace \([{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']]\) lies in \({\text {Span}}(e_3, \dots , e_m)\). Moreover, as \(\phi (Y)^2e_3=\mu \langle a,Y\rangle e_{m1}\) and \(\phi (Y)^2e_2 = \mu \langle a,Y\rangle (\lambda \langle a,Y\rangle e_{m1} + \mu \langle a,Y\rangle e_m)\), and as \(a \ne 0\) and \(\mu \ne 0\), we obtain that the subspace \([{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']]\) contains \({\text {Span}}(e_{m1}, e_m)\). But then \([{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']] \cap ([{\mathfrak {v}},[{\mathfrak {v}},{\mathfrak {n}}']])^\perp = {\text {Span}}(e_{m1}, e_m)\), and so the subspace \({\text {Span}}(e_{m1}, e_m)\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by Remark 3. Then the subspace \(\{Y \in {\mathfrak {c}}^\perp \, : \, [Y,{\mathfrak {n}}'] \subset {\text {Span}}(e_{m1}, e_m)\}\) is also \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by Remark 3. But the latter subspace is given by \(\{Y \in {\mathfrak {c}}^\perp \, : \, WY=0, \, \langle a,Y\rangle =0\} = {\mathfrak {c}}^\perp \cap ({\text {Span}}(a,c))^\perp ={\mathbb {R}}Y_0\), where \(Y_0 \ne 0\) is the component of b orthogonal to \({\text {Span}}(a,c)\). This gives a 1dimensional \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspace of \({\mathfrak {c}}^\perp \), in contradiction with Lemma 3(b).
Case (iii): \(\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_3\). This is the most involved case. For \(Y \in {\mathfrak {v}}\), we have
where \(b \in {\mathfrak {v}}\), and \(U,V: {\mathfrak {v}}\rightarrow {\mathbb {R}}^{m4}={\text {Span}}(e_3, \dots , e_{m2})\) are such that for all \(Y_1, Y_2 \in {\mathfrak {v}}\) we have
The following lemma sorts out the “nongeneric” cases.
Lemma 4
Suppose \(\phi ({\mathfrak {n}}) \not \subset {\mathfrak {u}}_1\) and \(\phi ({\mathfrak {n}}) \not \subset {\mathfrak {u}}_2\) [up to specifying the basic vectors \(e_1,e_2,e_{m1},e_m,\) but keeping the form of \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) given in (15)]. We have the following.

(a)
For almost all \(Y \in {\mathfrak {v}},\) the vectors UY and VY are linearly independent.

(b)
The subspace \({\text {Span}}(e_{m1}, e_m)\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant.

(c)
\({\mathfrak {c}}={\text {Ker}}U \cap {\text {Ker}}V,\; {\mathfrak {c}}^\perp =({\text {Ker}}U \cap {\text {Ker}}V)^\perp ,\) and so for almost all \(Y \in {\mathfrak {c}}^\perp ,\) the vectors UY and VY are linearly independent.

(d)
For almost all \(e \in {\text {Span}}(e_3, \dots , e_{m2}),\) the 1forms \(\xi _e\) and \(\eta _e\) on \({\mathfrak {v}}\) defined by \(\xi _e(Y)=\langle UY,e\rangle \) and \(\eta _e(Y)=\langle VY,e\rangle \) are linearly independent.
Proof
We cannot have \(U=V=0\), as otherwise the subspace \({\mathfrak {c}}^\perp \) is at most 1dimensional, in contradiction with Lemma 3(b). Furthermore, if U and V are proportional, we can specify the vectors \(e_1,e_2,e_{m1},e_m\) [without changing the form of \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) given in (15)] in such a way that \(V=0\) hence obtaining \(\phi ({\mathfrak {n}}) \subset {\mathfrak {u}}_1\). We can therefore assume that U and V are not proportional.
We use the following well known fact. If \(F_1, F_2: {\mathbb {R}}^p \rightarrow {\mathbb {R}}^q\) are linear maps such that \({\text {rk}}(F_1x,F_2x) \le 1\), for all \(x \in {\mathbb {R}}^p\), then either \(F_1\) and \(F_2\) are proportional, or there exist \(w \in {\mathbb {R}}^q\) and \(\lambda _1, \lambda _2 \in ({\mathbb {R}}^p)^*\) such that \(F_1x=\lambda _1(x)w\) and \(F_2x=\lambda _2(x)w\), for all \(x \in {\mathbb {R}}^p\).
For assertion (a), we apply the above fact to U and V. As we assume that U and V are not proportional, we obtain that \(UY=\lambda _1(Y)w,\; VY=\lambda _2(Y)w\) for nonproportional 1forms \(\lambda _1, \lambda _2 \in {\mathfrak {v}}^*\) and for some \(w \ne 0\). But this leads to a contradiction with (22).
To prove assertion (b) we note that from assertion (a) and from (21) it follows that \({\text {Span}}(e_{m1}, e_m) \subset [{\mathfrak {v}},{\mathfrak {n}}'] \subset {\text {Span}}(e_3, \dots , e_m)\), and so \([{\mathfrak {v}},{\mathfrak {n}}'] \cap ([{\mathfrak {v}},{\mathfrak {n}}'])^\perp ={\text {Span}}(e_{m1}, e_m)\). Hence the subspace \({\text {Span}}(e_{m1}, e_m)\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by Remark 3.
For assertion (c) we note that from (21) we have \({\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp + {\mathbb {R}}b\). But if \(b \notin ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \), then the subspace \(\{Y \in {\mathfrak {c}}^\perp \, : \, [Y,{\mathfrak {n}}'] \subset {\text {Span}}(e_{m1}, e_m)\}=\{Y \in {\mathfrak {c}}^\perp \, : \, UY=VY=0\}\) is 1dimensional and is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by assertion (b) and Remark 3, in contradiction with Lemma 3(b). It follows that \(b \in ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \), and so \({\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \) and \({\mathfrak {c}}= {\text {Ker}}U \cap {\text {Ker}}V\), as required. Then assertion (a) implies that for almost all \(Y \in {\mathfrak {c}}^\perp \), we have \({\text {rk}}(UYVY) = 2\).
For assertion (d), we apply the above linearalgebraic fact to the conjugates of U and V. As we assume U and V to be not proportional, the condition that the 1forms \(\xi _e\) and \(\eta _e\) on \({\mathfrak {v}}\) are linearly dependent for all \(e \in {\text {Span}}(e_3, \dots , e_{m2})\) would imply the existence of \(w \in {\mathfrak {v}}\setminus \{0\}\) and \(e,e' \in {\text {Span}}(e_3, \dots , e_{m2})\) such that \(UY=\langle w,Y\rangle e\) and \(VY=\langle w,Y\rangle e'\), for all \(Y \in {\mathfrak {v}}\). But then by assertion (c), \({\mathfrak {c}}^\perp = {\mathbb {R}}w\) which contradicts Lemma 3(b). \(\square \)
As Cases (i) and (ii) have been already understood, for the rest of the proof we will assume that the conditions of Lemma 4 are satisfied.
As \(\phi ({\mathfrak {v}}){\mathfrak {n}}' \subset {\text {Span}}(e_3, \dots , e_m)\), in order to have both \(e_1\) and \(e_2\) in \({\mathfrak {n}}'\), we need the 2forms \(\omega _1, \omega _2 \in \Lambda ^2({\mathfrak {v}})\) defined by (16) to be linearly independent. From the Jacobi identity we obtain
where \(\sigma \) denotes the cyclic permutation of \(Y_1,Y_2,Y_3 \in {\mathfrak {v}}\). Taking \(Y_1,Y_2 \in {\mathfrak {c}}\) and \(Y_3 \in {\mathfrak {c}}^\perp \) in such a way that \({\text {rk}}(UY_3VY_3) = 2\) we obtain \(\omega _1({\mathfrak {c}},{\mathfrak {c}})=\omega _2({\mathfrak {c}},{\mathfrak {c}})=0\). As we also have \(\omega _1({\mathfrak {c}},{\mathfrak {c}}^\perp )=\omega _2({\mathfrak {c}},{\mathfrak {c}}^\perp )=0\) by Lemma 3(a) and (16), we obtain \(\omega _1({\mathfrak {c}},{\mathfrak {v}})=\omega _2({\mathfrak {c}},{\mathfrak {v}})=0\), and hence the restrictions of \(\omega _1\) and \(\omega _2\) to \({\mathfrak {c}}^\perp \) must be linearly independent.
Lemma 5
In the assumptions of Lemma 4 we have the following.

(a)
\(\dim {\mathfrak {c}}^\perp \le 4\).

(b)
Introduce \(K_1, K_2 \in \mathfrak {so}({\mathfrak {c}}^\perp )\) by \(\langle K_iY,Y'\rangle =\omega _i(Y,Y')\) for \(Y, Y' \in {\mathfrak {c}}^\perp \) and \(i=1,2\). Denote \(S={\text {Span}}(K_1, K_2) \subset \mathfrak {so}({\mathfrak {c}}^\perp )\) (note that \(\dim S = 2\)). Then the subalgebra \(\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )\) normalises the subspace S.
Proof
For assertion (a), consider the pencil \(\mu _1 \omega _1 + \mu _2 \omega _2 \subset \Lambda ^2({\mathfrak {v}})\), where \(\mu _1, \mu _2 \in {\mathbb {R}}\). Suppose at least one element of this pencil has rank greater than or equal to 4. Specifying the vectors \(e_1,e_2,e_{m1},e_m\) we can assume, without loss of generality, that \({\text {rk}}\omega _1 \ge 4\).
Let \({\mathcal {U}}\subset {\text {Span}}(e_3, \dots , e_{m2})\) be the subset of those vectors e for which the 1forms \(\xi _e, \eta _e \in {\mathfrak {v}}^*\) are linearly independent. By Lemma 4(d), the subset \({\mathcal {U}}\) is open and dense in \({\text {Span}}(e_3, \dots , e_{m2})\). Taking the inner product of (23) with \(e \in {\mathcal {U}}\) we obtain \(\omega _1 \wedge \xi _e + \omega _2 \wedge \eta _e=0\). By generalised Cartan’s Lemma [1, Lemma 1], there exist 1forms \(\gamma _{11}, \gamma _{12}= \gamma _{21}, \gamma _{22} \in {\mathfrak {v}}^*\) such that \(\omega _1 = \gamma _{11} \wedge \xi _e + \gamma _{12} \wedge \eta _e\) and \(\omega _2 = \gamma _{21} \wedge \xi _e + \gamma _{22} \wedge \eta _e\). In particular, \({\text {rk}}\omega _1 \le 4\), and so \({\text {rk}}\omega _1 = 4\) by our assumption. Let \(L_1=\{Y \in {\mathfrak {v}}\, : i_Y (\omega _1)=0\}\). Then \(L_1\) has codimension 4, and \(\xi _e(L_1)= \eta _e(L_1)=0\), for all \(e \in {\mathcal {U}}\), and hence for all \(e \in {\mathfrak {v}}\). It follows that \(UL_1=VL_1=0\), and so by Lemma 4(c), \({\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \subset L_1^\perp \) which implies \(\dim {\mathfrak {c}}^\perp \le 4\).
Now suppose that \({\text {rk}}(\mu _1 \omega _1 + \mu _2 \omega _2) < 4\), for all \(\mu _1, \mu _2 \in {\mathbb {R}}\). As the rank is always even and as \(\omega _1\) and \(\omega _2\) are linearly independent, we obtain \({\text {rk}}(\mu _1 \omega _1 + \mu _2 \omega _2) = 2\), for all \((\mu _1, \mu _2) \in {\mathbb {R}}^2\setminus \{(0,0)\}\). Then it is easy to see that there exist three linearly independent 1forms \(\zeta _1, \zeta _2, \zeta _3 \in {\mathfrak {v}}^*\) such that \(\omega _1=\zeta _1 \wedge \zeta _3\) and \(\omega _2=\zeta _2 \wedge \zeta _3\). From (23) we get \(\omega _1 \wedge \xi _e + \omega _2 \wedge \eta _e=0\), for all \(e \in {\text {Span}}(e_3, \dots , e_{m2})\) which gives \(\zeta _1 \wedge \zeta _3 \wedge \xi _e + \zeta _2 \wedge \zeta _3 \wedge \eta _e=0\). It follows that \(\zeta _1 \wedge \zeta _2 \wedge \zeta _3 \wedge \xi _e=0\), and so \(\xi _e \in {\text {Span}}(\zeta _1, \zeta _2, \zeta _3)\), and similarly \(\eta _e \in {\text {Span}}(\zeta _1, \zeta _2, \zeta _3)\), for all \(e \in {\text {Span}}(e_3, \dots , e_{m2})\). But then the common kernel \(L_2\) of the 1forms \(\zeta _1, \zeta _2, \zeta _3\) has codimension 3 and lies in the kernel of both U and V. It follows that \({\mathfrak {c}}^\perp = ({\text {Ker}}U \cap {\text {Ker}}V)^\perp \subset L_2^\perp \) and so \(\dim {\mathfrak {c}}^\perp \le 3\).
For assertion (b), we first note that the subspace \({\text {Span}}(e_3, \dots ,e_m)=({\text {Span}}(e_{m1},e_m))^\perp \) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by Lemma 4(b) and Remark 3. Now let \(A \in {\mathfrak {h}}\) and \(Y,Y' \in {\mathfrak {c}}^\perp \). Then by (9), the component of the vector \([AY,Y']+[Y,AY']\) lying in \({\text {Span}}(e_1,e_2)\) equals \((\omega _1(AY,Y')+\omega _1(Y,AY'))e_1+(\omega _2(AY,Y')+\omega _2(Y,AY'))e_2=\langle [K_1,\pi (A)] Y,Y'\rangle e_1 + \langle [K_2,\pi (A)] Y,Y'\rangle e_2\). As \({\text {Span}}(e_3, \dots ,e_m)\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, the component of the vector \(A[Y,Y']\) lying in \({\text {Span}}(e_1,e_2)\) equals the component of the vector \(A(\omega _1(Y,Y')e_1+\omega _2(Y,Y')e_2)\) lying in \({\text {Span}}(e_1,e_2)\), which is \(\omega _1(Y,Y') (A_{11}e_1+A_{12}e_2) + \omega _2(Y,Y') (A_{21}e_1+A_{22}e_2)= \langle (A_{11}K_1+A_{21}K_2)Y,Y'\rangle e_1 + \langle (A_{12}K_1+A_{22}K_2)Y,Y'\rangle e_2\), where \(A_{ij}\) denotes the corresponding entry of the matrix of \(({\text {ad}}(A))_{{\mathfrak {n}}'}\) relative to the basis \(\{e_1,e_2, \dots , e_m\}\). We deduce that \([K_1,\pi (A)], [K_2,\pi (A)] \in {\text {Span}}(K_1,K_2)\), as required. \(\square \)
By Lemmas 3(b) and 5(a), we have \(2 \le \dim {\mathfrak {c}}^\perp \le 3\). Moreover, by Lemma 5(b), the subalgebra \(\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )\) normalises the 2dimensional subspace \(S={\text {Span}}(K_1,K_2) \!\subset \mathfrak {so}({\mathfrak {c}}^\perp )\). If \(\pi ({\mathfrak {h}})\) is abelian, then by Lemma 3(d), we obtain \([{\mathfrak {c}}^\perp , {\mathfrak {c}}^\perp ] \subset {\mathfrak {z}}({\mathfrak {n}})\). Then taking \(Y_1,Y_2,Y_3 \in {\mathfrak {c}}^\perp \) in (17) we get \(\sum \nolimits _{i=1}^{m} \omega _i(Y_1,Y_2) \phi (Y_3) e_i=0\) which by (21) gives \(\omega _1(Y_1,Y_2)UY_3 + \omega _2(Y_1,Y_2)VY_3 =0\). As by Lemma 4(c), the vectors \(UY_3\) and \(VY_3\) are linearly independent for almost all \(Y_3 \in {\mathfrak {c}}^\perp \), we deduce that \(\omega _1({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=\omega _2({\mathfrak {c}}^\perp ,{\mathfrak {c}}^\perp )=0\), a contradiction. So the subalgebra \(\pi ({\mathfrak {h}}) \subset \mathfrak {so}({\mathfrak {c}}^\perp )\) is nonabelian. It is easy to see that the only possible case when the normaliser of a twodimensional subspace S of a subalgebra \(\mathfrak {so}({\mathfrak {c}}^\perp ),\; \dim {\mathfrak {c}}^\perp \in \{2,3,4\}\), is nonabelian is the following: \(\dim {\mathfrak {c}}^\perp = 4\), so that \(\mathfrak {so}({\mathfrak {c}}^\perp ) = \mathfrak {so}(4) = \mathfrak {so}(3) \oplus \mathfrak {so}(3)\) (direct sum of ideals), and then S lies in one of the two \(\mathfrak {so}(3)\)components.
In this last remaining case, denote \({\tilde{\omega }}_j \in \Lambda ^2({\mathfrak {c}}^\perp ), \; j=1,2\), the restriction of the 2form \(\omega _j \in \Lambda ^2({\mathfrak {v}})\) to \({\mathfrak {c}}^\perp \), and for \(e \in {\text {Span}}(e_3, \dots , e_{m2})\), denote \({\tilde{\xi }}_e,{\tilde{\eta }}_e \in ({\mathfrak {c}}^\perp )^*\) the restrictions of the 1forms \(\xi _e,\eta _e \in {\mathfrak {v}}^*\), respectively (so that for \(Y \in {\mathfrak {c}}^\perp \) we have \({\tilde{\xi }}_e(Y)=\langle UY,e\rangle \) and \({\tilde{\eta }}_e(Y)=\langle VY,e\rangle \)). Restricting Eq. (23) to \({\mathfrak {c}}^\perp \) [note that \(U{\mathfrak {c}}=V{\mathfrak {c}}=0\) and \(\omega _1({\mathfrak {c}},{\mathfrak {v}})=\omega _2({\mathfrak {c}},{\mathfrak {v}})=0\) anyway], we obtain \({\tilde{\omega }}_1 \wedge {\tilde{\xi }}_e + {\tilde{\omega }}_2 \wedge {\tilde{\eta }}_e=0\), for all \(e \in {\text {Span}}(e_3, \dots , e_{m2})\). Applying the Hodge star operator (and noting that the dual vectors to \({\tilde{\xi }}_e\) and \({\tilde{\eta }}_e\) are \(U^te, V^te \in {\mathfrak {c}}^\perp \), respectively) gives \(i_{U^te}({\star }{\tilde{\omega }}_1)+i_{V^te}({\star }{\tilde{\omega }}_2)=0\). Let \({\star }K_j \in \mathfrak {so}({\mathfrak {c}}^\perp ), \; j=1,2\), be defined by \(\langle ({\star }K_j) Y,Y'\rangle ={\star }{\tilde{\omega }}_j(Y,Y')\) for \(Y,Y' \in {\mathfrak {c}}^\perp \). Then from the latter equation we obtain \(\langle ({\star }K_1)U^te,Y\rangle +\langle ({\star }K_2)V^te,Y\rangle =0\), for all \(Y \in {\mathfrak {c}}^\perp \) and all \(e \in {\text {Span}}(e_3, \dots , e_{m2})\). This is equivalent to
where \({\tilde{U}}\) and \({\tilde{V}}\) are the restrictions of U and V to \({\mathfrak {c}}^\perp \), respectively. Note that \(K_1\) and \(K_2\) are linearly independent and belong to the same \(\mathfrak {so}(3)\)component of the algebra \(\mathfrak {so}({\mathfrak {c}}^\perp )=\mathfrak {so}(3) \oplus \mathfrak {so}(3)\) (direct sum of ideals). As these components are \({\star }\)invariant, we obtain that \({\star }K_1\) and \({\star }K_2\) are also linearly independent and belong to the same \(\mathfrak {so}(3)\)component. This implies that \(({\star }K_2)^2=\mu {\text {Id}}\) for some \(\mu < 0\) and that \(({\star }K_1) ({\star }K_2) = \nu {\text {Id}}+ K_3\), where \(\nu \in {\mathbb {R}}\), and \(K_3 \ne 0\) belongs to the same \(\mathfrak {so}(3)\)component of \(\mathfrak {so}({\mathfrak {c}}^\perp )\) as \({\star }K_1\) and \({\star }K_2\). In particular, \(\det K_3 \ne 0\). We now multiply (24) by \({\tilde{U}}^t\) on the left and by \({\star }K_2\) on the right. We get \({\tilde{U}}^t{\tilde{U}}(\nu {\text {Id}}+ K_3)+\mu {\tilde{U}}^t{\tilde{V}}=0\). But \(U^tV\) is symmetric by (22), and so \({\tilde{U}}^t{\tilde{V}}\) is symmetric (as \(U{\mathfrak {c}}=V{\mathfrak {c}}=0\)) which implies that the \(4 \times 4\) matrix \({\tilde{U}}^t{\tilde{U}} K_3\) is also symmetric. Choosing a basis for \({\mathfrak {c}}^\perp \) which diagonalises the semidefinite matrix \({\tilde{U}}^t{\tilde{U}}\) we find that \({\tilde{U}}^t{\tilde{U}} K_3\) can be symmetric only when \({\tilde{U}}^t{\tilde{U}} K_3=0\). But as \(\det K_3 \ne 0\), this implies \({\tilde{U}}=0\), that is, \(U{\mathfrak {c}}^\perp =0\), which in combination with \(U{\mathfrak {c}}=0\) gives \(U=0\). This is a contradiction with Lemma 4(a) which completes the proof of the proposition and of Theorem 1. \(\square \)
2.3 Example
The following example shows that Theorem 1, concerning the 2step property of \({\mathfrak {n}}\) for the case when the derived algebra \({\mathfrak {n}}'\) is nondegenerate, is “almost" tight in terms of the signature. We construct a nilpotent, metric Lie algebra \(({\mathfrak {n}},\langle \cdot ,\cdot \rangle )\) with the following properties:

\(\dim {\mathfrak {n}}= 12,\; \dim {\mathfrak {n}}'=4, \, \dim {\mathfrak {v}}= 8\).

\({\mathfrak {n}}'\) is Lorentz and \({\mathfrak {v}}\) is of signature (5, 3), so that \({\mathfrak {n}}\) is of signature (8, 4).

\({\mathfrak {n}}\) is 4step nilpotent (and \({\mathfrak {n}}'\) is abelian).

\(({\mathfrak {n}},\langle \cdot ,\cdot \rangle )\) is Ggeodesic orbit (for G as in Theorem 1).
We define \({\mathfrak {n}}= {\mathfrak {v}}\oplus {\mathfrak {n}}'\), where \(\dim {\mathfrak {n}}'=4,\; \dim {\mathfrak {v}}= 8\). We have a basis \(\{e_1, e_2,e_3,e_4\}\) for \({\mathfrak {n}}'\), and a basis \(\{f_1, \dots , f_8\}\) for \({\mathfrak {v}}\). The inner product \(\langle \cdot ,\cdot \rangle \) is defined in such a way that \({\mathfrak {v}}\perp {\mathfrak {n}}'\), and
The Lie bracket is defined as follows:
It is not hard to see that the algebra \({\mathfrak {n}}'\) so defined is 4step nilpotent (in fact, if we disregard the inner product, our algebra \({\mathfrak {n}}'\) is the direct sum of the 6dimensional ideal \({\text {Span}}(f_1,f_2,e_1,e_2,e_3,e_4)\) (the algebra \(L_{6,21}(1)\) in [6]) and the 6dimensional abelian ideal \({\text {Span}}(f_3, f_4+e_2, f_5+e_3, f_6e_1, f_7, f_8)\)).
We now define, for every \(T=X+Y\), where \(X=\sum _{i=1}^{4} x_i e_i \in {\mathfrak {n}}', \; Y=\sum _{j=1}^8 y_j f_j \in {\mathfrak {v}}\), the linear operator \({\mathcal {A}}\) on \({\mathfrak {n}}\) such that \({\mathcal {A}}{\mathfrak {n}}' \subset {\mathfrak {n}}',\, {\mathcal {A}}{\mathfrak {v}}\subset {\mathfrak {v}}\), and relative to the chosen bases for \({\mathfrak {v}}\) and \({\mathfrak {n}}'\),
A direct calculation shows that \({\mathcal {A}}\) so defined is a skewsymmetric derivation, and that the GO equation \(\langle {\mathcal {A}}T'+[T,T'],T\rangle = 0\) [see (1)] is satisfied, for all \(T' \in {\mathfrak {n}}\). As \({\mathcal {A}}\) depends linearly on T, one may expect the algebra \(({\mathfrak {n}},\langle \cdot ,\cdot \rangle )\) to be even naturally reductive. \(\diamondsuit \)
This example also shows that a pseudoRiemannian GGO nilmanifold with nondegenerate derived algebra loses the property of being 2step nilpotent already when \(\langle \cdot ,\cdot \rangle '\) is Lorentz (for the case when \(\langle \cdot ,\cdot \rangle '\) is definite, see Remark 4).
3 Proof of Theorem 2: If \(ds^2_{[{\mathfrak {n}},{\mathfrak {n}}]}\) is Degenerate Then \({\mathfrak {n}}\) is a Double Extension
In this section we consider the case when the restriction of the inner product \(\langle \cdot ,\cdot \rangle \) to the derived algebra \({\mathfrak {n}}'\) is degenerate.
We start with the following Lemma.
Lemma 6
Let \((M = G/H, ds^2)\) be a connected pseudoRiemannian Ggeodesic orbit nilmanifold where \(,\) with N nilpotent. Let \(\langle \cdot ,\cdot \rangle \) denote the inner product on \({\mathfrak {n}}\) induced by \(ds^2\). Suppose \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) is degenerate. Let \({\mathfrak {m}}_1\) and \({\mathfrak {e}}\) be subspaces of \({\mathfrak {n}}\) with the following properties:

(i)
\({\mathfrak {e}}\subset {\mathfrak {n}}' \subset {\mathfrak {m}}_1\) (so that, in particular, \({\mathfrak {m}}_1\) is an ideal of \({\mathfrak {n}}\));

(ii)
both \({\mathfrak {m}}_1\) and \({\mathfrak {e}}\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant;

(iii)
\(\langle {\mathfrak {e}}, {\mathfrak {m}}_1\rangle = 0\) and \([{\mathfrak {e}},{\mathfrak {m}}_1]=0;\)

(iv)
\(\dim {\mathfrak {m}}_1 + \dim {\mathfrak {e}}= \dim {\mathfrak {n}}\).
Define the metric nilpotent Lie algebra \({\mathfrak {m}}_0 = {\mathfrak {m}}_1/{\mathfrak {e}}\) with the inner product \(\langle \cdot ,\cdot \rangle _0\) induced from \({\mathfrak {m}}_1\) [this is welldefined by (iii)], and the pseudoRiemannian nilmanifold \((M_0 = G_0/H_0, ds_0^2),\) where \(,\) with \(N_0\) the (simply connected) Lie group whose Lie algebra is \({\mathfrak {m}}_0,\) \(ds_0^2\) is the leftinvariant metric on \(M_0\) defined by \(\langle \cdot ,\cdot \rangle _0,\) and \(H_0\) is the maximal connected group of pseudoorthogonal automorphisms of \(\langle \cdot ,\cdot \rangle _0\).
Then \((M_0, ds_0^2)\) is a \(G_0\)GO pseudoRiemannian nilmanifold.
Note that the fact that the inner product \(\langle \cdot ,\cdot \rangle _0\) as constructed in Lemma 6 is nondegenerate follows from assumption (iv).
If we denote \(m_0=\dim {\mathfrak {e}}\), then the signature of the metric \(ds_0^2\) is \((pm_0,qm_0)\), where (p, q) is the signature of \(ds^2\). In the settings of Lemma 6, we say that the metric Lie algebra \(({\mathfrak {n}},\langle \cdot ,\cdot \rangle )\) is a \(2m_0\)dimensional double extension of the metric Lie algebra \(({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)\). Informally, to get \(({\mathfrak {n}},\langle \cdot ,\cdot \rangle )\), we first take the central extension of \(({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)\) by \({\mathfrak {e}}\) and then the extension of the resulting Lie algebra \({\mathfrak {m}}_1\) by \(m_0\)dimensional space of derivations.
Proof of Lemma 6
To check the \(G_0\)GO property for \(({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)\) we need to choose (an arbitrary, but fixed) linear complement to \({\mathfrak {e}}\) in \({\mathfrak {m}}_1\), which, with some abuse of notation, we will still denote \({\mathfrak {m}}_0\). Then \({\mathfrak {m}}_1 = {\mathfrak {m}}_0 \oplus {\mathfrak {e}}\). For \(X, Y \in {\mathfrak {m}}_0\) we have \(\langle X,Y\rangle _0=\langle X,Y\rangle \), as \({\mathfrak {e}}\perp {\mathfrak {m}}_1\) by assumption (iii). We define the Lie bracket \([\cdot ,\cdot ]_0\) on \({\mathfrak {m}}_0\) by \([X,Y]_0=[X,Y]_{{\mathfrak {m}}_0}\), for \(X, Y \in {\mathfrak {m}}_0\). It is easy to see that \(({\mathfrak {m}}_0,\langle \cdot ,\cdot \rangle _0)\) is isomorphic to the quotient algebra \({\mathfrak {m}}_1/{\mathfrak {e}}\).
Let \(X \in {\mathfrak {m}}_0\). By the Geodesic Lemma, there exist \(A(X) \in {\mathfrak {h}}\) and \(k(X) \in {\mathbb {R}}\) such that for all \(Y \in {\mathfrak {m}}_0\) we have \(\langle [X+A(X),Y],X\rangle = k(X) \langle X,Y\rangle \). By assumption (ii) we have \([A(X),Y] \in {\mathfrak {m}}_1 \, (= {\mathfrak {m}}_0 \oplus {\mathfrak {e}})\), and so we can define an endomorphism D(X) of \({\mathfrak {m}}_0\) by the formula \(D(X)Y = [A(X),Y]_{{\mathfrak {m}}_0}\). As \({\text {ad}}_{\mathfrak {g}}(A(X))\) is skewsymmetric and \({\mathfrak {e}}\perp {\mathfrak {m}}_1\), the endomorphism D(X) is skewsymmetric relative to \(\langle \cdot ,\cdot \rangle _0\). To see that D(X) is a derivation of the Lie algebra \(({\mathfrak {m}}_0, [\cdot ,\cdot ]_0)\) we write, for \(Y_1, Y_2 \in {\mathfrak {m}}_0\),
where in the third line, we used the fact that \({\mathfrak {e}}\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, by assumption (ii).
It follows that there exists \(A_0(X) \in {\mathfrak {h}}_0\), where \({\mathfrak {h}}_0\) is the Lie algebra of \(H_0\), such that \(D(X)Y = [A_0(X),Y]_0\), for all \(X,Y \in {\mathfrak {m}}_0\). Then from assumption (iii) and the fact that \(\langle [X+A(X),Y],X\rangle = k(X) \langle X,Y\rangle \) it follows that \(\langle [X+A_0(X),Y]_0,X\rangle _0 = k(X) \langle X,Y\rangle _0\), for all \(X,Y \in {\mathfrak {m}}_0\), as required by the Geodesic Lemma. \(\square \)
First suppose that \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) has degeneracy 1 and is semidefinite. Denote \({\mathfrak {v}}= ({\mathfrak {n}}')^\perp \) and choose a vector e such that \({\mathfrak {n}}' \cap {\mathfrak {v}}={\mathbb {R}}e\). Denote \({\mathfrak {m}}_1={\mathfrak {n}}' + {\mathfrak {v}}\). Note that \(e^\perp = {\mathfrak {m}}_1\) and that all four subspaces \({\mathbb {R}}e, {\mathfrak {n}}', {\mathfrak {v}}\) and \({\mathfrak {m}}_1\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, by Remark 3. To be able to apply Lemma 6 (with \({\mathfrak {e}}={\mathbb {R}}e\)) we only need to check that \([e, {\mathfrak {m}}_1]=0\). Taking \(T'=e\) and \(T = X+Y \in {\mathfrak {m}}_1\) in (1), where \(X \in {\mathfrak {n}}'\) and \(Y \in {\mathfrak {v}}\), we obtain \(\langle [e,X+Y],X\rangle =0\), and so \(\langle [e,X],X\rangle =\langle [e,Y],X\rangle =0\), for all \(X \in {\mathfrak {n}}', \, Y \in {\mathfrak {v}}\). From the second equation it follows that [e, Y] is a multiple of e, and hence \([e,Y]=0\), for all \(Y \in {\mathfrak {v}}\), as \({\text {ad}}_{{\mathfrak {n}}}(Y)\) is nilpotent. From the first equation we also obtain that [e, X] is a multiple of e, as \({\text {ad}}_{{\mathfrak {n}}'}e\) is nilpotent and skewsymmetric and \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) has degeneracy 1 and is semidefinite. Then \([e,X]=0\), for all \(X \in {\mathfrak {n}}'\), as \({\text {ad}}_{{\mathfrak {n}}}(X)\) is nilpotent. Thus \([e,{\mathfrak {m}}_1]=0\), and the claim follows from Lemma 6, with \({\mathfrak {e}}={\mathbb {R}}e\).
Next suppose that \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) has degeneracy 2 and is semidefinite. Denote \({\mathfrak {v}}= ({\mathfrak {n}}')^\perp , \; {\mathfrak {o}}={\mathfrak {n}}' \cap {\mathfrak {v}}\) and \({\mathfrak {s}}={\mathfrak {n}}' + {\mathfrak {v}}\). The restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {s}}\) has degeneracy 2 and is semidefinite. We have \(\dim {\mathfrak {o}}= {\text {codim}}{\mathfrak {s}}= 2\) and \({\mathfrak {o}}^\perp = {\mathfrak {s}}\). Moreover, all four subspaces \({\mathfrak {o}}, {\mathfrak {n}}', {\mathfrak {v}}\) and \({\mathfrak {s}}\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, by Remark 3. If \([{\mathfrak {o}}, {\mathfrak {s}}]=0\), we can directly apply Lemma 6 with \({\mathfrak {m}}_1={\mathfrak {s}}\) and \({\mathfrak {e}}= {\mathfrak {o}}\), and the claim follows. We therefore assume that \([{\mathfrak {o}}, {\mathfrak {s}}] \ne 0\). Taking \(T'=e \in {\mathfrak {o}}\) and \(T \in {\mathfrak {s}}\) in (1) we obtain \(\langle [e,T],T\rangle =0\). As the restriction of the inner product to \({\mathfrak {s}}\) is semidefinite, of degeneracy 2 (and \({\mathfrak {s}}^\perp = {\mathfrak {o}}\)), and \({\text {ad}}_{{\mathfrak {s}}}e\) is both skewsymmetric and nilpotent, we obtain \([e,T] \subset {\mathfrak {o}}\), for all \(e \in {\mathfrak {o}}\) and \(T \in {\mathfrak {s}}\), and hence \([{\mathfrak {s}},{\mathfrak {o}}] \subset {\mathfrak {o}}\). We obtain a nilpotent representation of the (nilpotent) algebra \({\mathfrak {s}}\) on the 2dimensional space \({\mathfrak {o}}\). By Engel’s Theorem, we can find a basis \(\{e_1,e_2\}\) for \({\mathfrak {o}}\) such that \([{\mathfrak {s}}, e_2]=0\) and \([T,e_1]=\lambda (T)e_2\), for all \(T \in {\mathfrak {s}}\), where \(\lambda \in {\mathfrak {s}}^*\). As we have assumed that \([{\mathfrak {o}}, {\mathfrak {s}}] \ne 0\), the 1form \(\lambda \) is nonzero (but note that \(\lambda ({\mathfrak {o}})=0\)). Then \([{\mathfrak {o}}, {\mathfrak {s}}] = {\mathbb {R}}e_2\), and so by Remark 3, the subspace \({\mathbb {R}}e_2\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant. Choose two vectors \(f_1, f_2 \in {\mathfrak {n}}\) such that \({\text {Span}}(f_1,f_2) \oplus {\mathfrak {s}}= {\mathfrak {n}}\), and \(\langle f_i,f_j\rangle =0,\; \langle f_i,e_j\rangle =\delta _{ij}\), for \(i,j=1,2\). We claim that the assumptions of Lemma 6 are satisfied with \({\mathfrak {e}}= {\mathbb {R}}e_2\) and \({\mathfrak {m}}_1 = {\mathbb {R}}f_1 \oplus {\mathfrak {s}}\). Indeed, assumptions (i) and (iv) are obviously true, and for assumption (ii) we note that \({\mathfrak {m}}_1=({\mathbb {R}}e_2)^\perp \) by construction, and hence \({\mathfrak {m}}_1\) is \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant by Remark 3, as \({\mathbb {R}}e_2\) is. It remains to show that \([{\mathfrak {m}}_1,e_2]=0\). As we already know that \([{\mathfrak {s}}, e_2]=0\), it suffices to show that \([f_1,e_2]=0\). Taking \(T'=e_2\) and \(T =\xi f_1+X\in {\mathfrak {m}}_1\), where \(X \in {\mathfrak {s}}, \xi \in {\mathbb {R}}\), in (1) (and using the fact that \({\mathbb {R}}e_2\) and \({\mathfrak {m}}_1\) are orthogonal, \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspaces) we obtain \(\langle [f_1,e_2],\xi f_1+X\rangle =0\), for all \(X \in {\mathfrak {s}}, \xi \in {\mathbb {R}}\). It follows that \([f_1,e_2]\) is a multiple of \(e_2\), which must be zero, by nilpotency. The claim now follows from Lemma 6.
The last case to consider is the one when \(\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'}\) has degeneracy 1 and index 1. This is the most involved case. As above, we denote \({\mathfrak {v}}= ({\mathfrak {n}}')^\perp \) and choose a vector e such that \({\mathfrak {n}}' \cap {\mathfrak {v}}={\mathbb {R}}e\). Denote \({\mathfrak {m}}_1={\mathfrak {n}}' + {\mathfrak {v}}\), so that \(e^\perp = {\mathfrak {m}}_1\). The subspaces \({\mathbb {R}}e, {\mathfrak {n}}', {\mathfrak {v}}\) and \({\mathfrak {m}}_1\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant, by Remark 3. We claim that the assumptions of Lemma 6 are satisfied with \({\mathfrak {e}}={\mathbb {R}}e\). It is easy to see that the only fact we need to establish is that \([e, {\mathfrak {m}}_1]=0\).
The proof is completed by the following proposition.
Proposition 3
In the above notation, the vector e lies in the centre of \({\mathfrak {m}}_1\).
Proof
Denote \(m = \dim {\mathfrak {n}}'\). Seeking a contradiction we assume that \([{\mathfrak {m}}_1,e] \ne 0\).
Let \({\mathfrak {f}}=\{T \in {\mathfrak {n}}\, : \, \langle [T,X],X\rangle =0, \text { for all } X \in {\mathfrak {n}}'\}\). It is easy to see that \({\mathfrak {f}}\) is a subalgebra of \({\mathfrak {n}}\).
Lemma 7
In the above notation, the following holds.

(a)
\({\mathfrak {v}}\subset {\mathfrak {f}}\) and \([{\mathfrak {f}},e]=0\) (so, in particular, \([e,{\mathfrak {v}}]=0\)).

(b)
There exists a hyperplane \({\mathfrak {n}}_0 \subset {\mathfrak {n}}',\) with \({\mathfrak {n}}_0 \oplus {\mathbb {R}}e = {\mathfrak {n}}',\) and a basis \(\{e_1, \dots , e_{m1}\}\) for \({\mathfrak {n}}_0\) such that relative to the basis \(\{e_1, \dots , e_{m1},e\}\) for \({\mathfrak {n}}',\) we have
$$\begin{aligned}{} & {} \langle \cdot ,\cdot \rangle _{{\mathfrak {n}}'} = \left( {\begin{matrix} 0 &{} 0 &{} 1 &{} 0\\ 0 &{} I_{m3} &{} 0 &{} 0 \\ 1 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 \end{matrix}} \right) , \quad {\text {ad}}_{{\mathfrak {n}}'}T = \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 \\ VT &{} 0_{m3} &{} 0 &{} 0 \\ 0 &{} (VT)^t &{} 0 &{} 0 \\ a(T) &{} (WT)^t &{} 0 &{} 0 \end{matrix}}\right) \quad \text {and }\nonumber \\ {}{} & {} {\text {ad}}_{{\mathfrak {n}}'}e = \left( {\begin{matrix} 0 &{} 0 &{} 0 &{} 0 \\ u &{} 0_{m3} &{} 0 &{} 0 \\ 0 &{} u^t &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 \end{matrix}}\right) , \end{aligned}$$(25)for all \(T \in {\mathfrak {f}},\) where \(u \in L:={\text {Span}}(e_2, \dots , e_{m2}),\, u \ne 0, \, a \in {\mathfrak {f}}^*\) and \(V,W: {\mathfrak {f}}\rightarrow L\) are linear maps. In particular, \([{\mathfrak {f}},e_{m1}]=0\).

(c)
The subspaces \({\text {Span}}(u, e_{m1})\) and \({\mathbb {R}}e_{m1}\) are \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant. Moreover, for any \(A \in {\mathfrak {h}},\) we have \([A,e]=\alpha (A)e, \, [A,u] = \beta (A) e_{m1}, \, [A,e_{m1}] = \gamma (A)e_{m1},\) for some \(\alpha ,\beta ,\gamma \in {\mathfrak {h}}^*\).
Proof
For assertion (a), the fact that \([{\mathfrak {f}},e]=0\) easily follows: for all \(T \in {\mathfrak {f}}\) and \(X \in {\mathfrak {n}}'\), we have \(0=\langle [T,X],e\rangle =\langle [T,e],X\rangle \). Therefore [T, e] is a multiple of e, which must be zero as \({\text {ad}}_{{\mathfrak {n}}'}T\) is nilpotent.
To see that \({\mathfrak {v}}\subset {\mathfrak {f}}\), take \(T'=Y \in {\mathfrak {v}}\) and \(T=X \in {\mathfrak {n}}'\) in (1). As \({\mathfrak {v}}\) and \({\mathfrak {n}}'\) are orthogonal, \({\text {ad}}_{\mathfrak {g}}({\mathfrak {h}})\)invariant subspaces, se obtain \(\langle [X,Y],X\rangle =0\), as required.
For assertion (b), we note that the subspace \([e,{\mathfrak {n}}'] \subset {\mathfrak {n}}'\) does not contain \({\mathbb {R}}e\) (indeed, for no \(X \in {\mathfrak {n}}'\) we can have \([X,e]=e\), as \({\text {ad}}_{{\mathfrak {n}}'}X\) is nilpotent). Choose a linear complement \({\mathfrak {n}}_0\) to \({\mathbb {R}}e\) in \({\mathfrak {n}}'\) in such a way that \({\mathfrak {n}}_0 \supset [e,{\mathfrak {n}}']\). The restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}_0\) is nondegenerate and is of Lorentz signature. For \(T \in {\mathfrak {f}}\) we define the endomorphism \(\phi _T\) of \({\mathfrak {n}}_0\) by \([T,X]=\phi _T X + \mu (X)e\), for \(X \in {\mathfrak {n}}_0\). For every \(T \in {\mathfrak {f}}\), the endomorphism \(\phi _T\) is skewsymmetric (as \(\langle e,{\mathfrak {n}}'\rangle =0\)) and nilpotent (as \([{\mathfrak {f}},e] = 0\) by assertion (a)). Moreover, the map \(\phi : {\mathfrak {f}}\rightarrow \mathfrak {so}({\mathfrak {n}}_0,\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}_0})\) sending T to \(\phi _T\) is a Lie algebra homomorphism (as \([{\mathfrak {f}},e] = 0\)). Considering the Iwasawa decomposition of the Lie algebra \(\mathfrak {so}(m 2, 1)=\mathfrak {so}({\mathfrak {n}}_0,\langle \cdot ,\cdot \rangle _{{\mathfrak {n}}_0})\), by the argument similar to that in the proof of Proposition 1 [see Eqs. (7) and (8)] we can construct a basis \(\{e_1, \dots , e_{m1}\}\) for \({\mathfrak {n}}_0\) such that the restriction of \(\langle \cdot ,\cdot \rangle \) to \({\mathfrak {n}}'\) relative to the basis \(\{e_1, \dots , e_{m1},e\}\) for \({\mathfrak {n}}_0\) has the form as given in (25), and moreover, there is a linear map \(V: {\mathfrak {f}}\rightarrow L \,(={\text {Span}}(e_2, \dots , e_{m2})\) such that for all \(T \in {\mathfrak {f}}\) we have \(\phi _T e_1 = VT, \; \phi _T e_{m1}=0\) and \(\phi _T e_i = \langle VT,e_i\rangle e_{m1}\), for \(i=2, \dots , m2\).
It follows that for some linear map \(W: {\mathfrak {f}}\rightarrow L\) and linear forms \(a, b \in {\mathfrak {f}}^*\), we have
for all \(T \in {\mathfrak {f}}\). In particular, taking \(T=e\) and using the fact that \([e,{\mathfrak {n}}_0] \subset {\mathfrak {n}}_0\) (by construction of \({\mathfrak {n}}_0\)) we obtain \(We=0\) and \(a(e)=b(e)=0\). Thus \({\text {ad}}_{{\mathfrak {n}}'}e\) has the form as given in (25), where we denote \(u=Ve \in L\). Then from (26) we obtain \([e,u]=\Vert u\Vert ^2e_{m1}\), and so \([T,[e,u]]=\Vert u\Vert ^2 b(T) e\), for all \(T \in {\mathfrak {f}}\). But \([T,e]=0\) by assertion (a), which gives \([[T,u],e]= \Vert u\Vert ^2 b(T) e\). As \({\text {ad}}_{{\mathfrak {n}}'}[T,u]\) is nilpotent we get \(\Vert u\Vert ^2 b(T) = 0\), for all \(T \in {\mathfrak {f}}\). If \(b \ne 0\) we get \(u=0\) (as the restriction of \(\langle \cdot ,\cdot \rangle \) to L is definite), and so \([e,{\mathfrak {n}}']=0\). As \([e,{\mathfrak {v}}]=0\) by assertion (a) we obtain \([e,{\mathfrak {m}}_1]=0\) contradicting our assumption. Therefore \(b=0\), and then Eq. (26) imply that \({\text {ad}}_{{\mathfrak {n}}'}T\) has the form given in (25).
The last statement in assertion (b) follows from (25).
For assertion (c), we note that from (25) we obtain \([e,{\mathfrak {n}}']={\text {Span}}(u, e_{m1})\) and \([e,[e,{\mathfrak {n}}']]={\mathbb {R}}e_{m1}\), and so the first claim follows from Remark 3. Then the second claim also follows (note that the ucomponent of [A, u] vanishes as \(\langle [A,u],u\rangle =0\) and u lies in the subspace L with a definite inner product). \(\square \)
Let now \(f \notin {\mathfrak {m}}_1\) be a null vector such that \(f \perp {\mathfrak {n}}_0\) and \(\langle f,e\rangle =1\) (the choice of such an f is not unique); note that \({\mathfrak {m}}_1 \oplus {\mathbb {R}}f = {\mathfrak {n}}\).
Lemma 8
The following holds:
Proof
From Lemma 7(c), for any \(A \in {\mathfrak {h}}\), we have \(\langle [A,f],e\rangle = \langle [A,e],f\rangle = \alpha (A)\), \(\langle [A,f],e_{m1}\rangle = \langle [A,e_{m1}],f\rangle = 0\) and \(\langle [A,f],u\rangle = \langle [A,u],f\rangle = 0\), as \(e_{m1} \in {\mathfrak {n}}_0 \subset f^\perp \). It follows that \([A,f]=\alpha (A)f + Y + X\), where \(Y \in {\mathfrak {v}}\) and \(X \in ({\text {Span}}(e_{m1},u))^\perp \cap {\mathfrak {n}}'\). Then \([A,[f,e]]=[[A,f],e]+[f,[A,e]]=[\alpha (A)f + Y + X,e]+[f, \alpha (A)e]=0\), as \([Y,e]=0\) by Lemma 7(a) and \([X,e]=0\) by (25). This proves (27).
Take in (1) \(T'=[f,e]\) and a nonnull vector \(T=\mu f + Y + X\), where \(Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'\) and \(\mu \in {\mathbb {R}}\). As \([A,[f,e]]=0\) by (27) and \({\mathfrak {v}}\perp {\mathfrak {n}}'\) we obtain \(\langle [\mu f + Y + X, [f,e]], \mu f + X\rangle =0\), for all \(Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'\) and \(\mu \in {\mathbb {R}}\), by continuity, from which we get
for all \(Y \in {\mathfrak {v}}, \, X \in {\mathfrak {n}}'\).
The first equation of (31) implies that [Y, [f, e]] is a multiple of e. But \([Y, [f,e]]=[[Y, f],e]\), as \([Y,e]=0\) by Lemma 7(a), and so \([Y, [f,e]]=[[Y, f],e]=0\), as \({\text {ad}}_{{\mathfrak {n}}'}[Y,f]\) is nilpotent. This proves (28).
We now consider the last equation of (31). We have \(\langle [X, [f,e]], f\rangle =\langle [e, [X,f]]  [f,[e,X]], f\rangle \). As \([e,{\mathfrak {n}}'] \subset {\mathfrak {n}}_0\) (by construction of \({\mathfrak {n}}_0\)) and \(f \perp {\mathfrak {n}}_0\), we have \(\langle [e, [X,f]], f\rangle =0\). Taking \(X=x_1e_1+{\tilde{x}}+x_{m1}e_{m1}+xe \in {\mathfrak {n}}\), where \(x_1, x_{m1},x \in {\mathbb {R}}\) and \({\tilde{x}} \in L\,(={\text {Span}}(e_2,\dots ,e_{m2}))\) we obtain from (25) that \([e,X] = x_1 u +\langle {\tilde{x}},u\rangle e_{m1}\), which gives \(\langle [X, [f,e]], f\rangle = \langle [f,[e,X]], f\rangle = \langle [f,x_1 u +\langle {\tilde{x}},u\rangle e_{m1}],f\rangle = x_1 \langle [f,u],f\rangle  \langle {\tilde{x}},u\rangle \langle [f,e_{m1}],f\rangle \). From the last equation of (31) we obtain \([f,[f,e]]=\langle [f,u],f\rangle e_{m1} + \langle [f,e_{m1}],f\rangle u + \eta e\), for some \(\eta \in {\mathbb {R}}\). But then \(\eta = 0\) from the third equation of (31). Moreover, as \(e \in {\mathfrak {v}}\), we get \([e,[f,e]]=0\) by (28), which implies \([e,[f,[f,e]]]=0\). Substituting the above expression for [f, [f, e]] and using (25) we find
But from Lemma 7(a), (b) we have \({\mathfrak {v}}\subset {\mathfrak {f}}\) and \([{\mathfrak {f}},e_{m1}]=0\) which implies \([[f,[f,e]],{\mathfrak {v}}]=0\), by the second equation of (32) . This establishes (29).
From the second equation of (31) we obtain that \([f,e] \in {\mathfrak {f}}\), and so \({\text {ad}}_{{\mathfrak {n}}'}[f,e]\) has the form given in (25), in particular, \([[f,e],u] \in {\text {Span}}(e_{m1},e)\). As \([e,[f,u]] \in [e,{\mathfrak {n}}'] = {\text {Span}}(u, e_{m1})\) (by (25)), we obtain \([f,[e,u]] = [[f,e],u] + [e,[f,u]] \in {\text {Span}}(u,e_{m1},e)\). But \([e,u]=\Vert u\Vert ^2e_{m1}\) by (25), and so we obtain \([f,e_{m1}]= \rho _1 u +\rho _2 e_{m1} + \rho _3 e\), for some \(\rho _1, \rho _2, \rho _3 \in {\mathbb {R}}\). Then from the first equation of (32) we get \(\rho _3=0\). Moreover, as \([f,e] \in {\mathfrak {f}}\), from (25) we find \([[f,e],e_{m1}]=0\) which implies \([e,[f,e_{m1}]]=0\) (since \([e,e_{m1}]=0\) by (25)). From the expression for \([f,e_{m1}]\) above we obtain \([e,\rho _1 u +\rho _2 e_{m1}]=0\) which implies \(\rho _1=0\), again by (25). Therefore \([f,e_{m1}]= \rho _2 e_{m1}\) which gives \([f,e_{m1}]=0\), by nilpotency. Now Eq. (30) follows from the second equation of (32). \(\square \)
We can now complete the proof of the proposition. We have \({\mathfrak {n}}= {\mathbb {R}}f \oplus {\mathfrak {m}}_1 = {\mathbb {R}}f \oplus ({\mathfrak {v}}+ {\mathfrak {n}}') = ({\mathbb {R}}f \oplus {\mathfrak {v}}) + {\mathfrak {n}}'\). It follows that the subspace \({\mathfrak {V}}={\mathbb {R}}f \oplus {\mathfrak {v}}\) contains some linear complement to \({\mathfrak {n}}'\) in \({\mathfrak {n}}\), and hence generates \({\mathfrak {n}}\). Then \({\mathfrak {n}}= {\mathfrak {V}}+ [{\mathfrak {V}},{\mathfrak {V}}]+[{\mathfrak {V}},[{\mathfrak {V}},{\mathfrak {V}}]]+ \cdots ,\) and so \({\mathfrak {n}}'=[{\mathfrak {V}},{\mathfrak {V}}]+[{\mathfrak {V}},[{\mathfrak {V}},{\mathfrak {V}}]]+ \cdots \) As we already know that \([e,{\mathfrak {v}}]=0\) [by Lemma 7(a)], to show that \([e,{\mathfrak {m}}_1]=0\) it suffices to prove that \([e,{\mathfrak {n}}']=0\), that is, to prove that \([e,[T_1,[T_2,[\dots ,[T_{r1},T_r]\dots ]]]]=0\), where \(r \ge 2\), and where, for every \(i=1, \dots , r\), we have either \(T_i=f\) or \(T_i \in {\mathfrak {v}}\). The proof goes by induction by \(r \ge 2\). If \(r=2\) the claim follows from the facts that \([e,{\mathfrak {v}}]=0\) and that \([e,[f,{\mathfrak {v}}]]=0\) [by (28)]. Suppose \(r > 2\). If \(T_1 \in {\mathfrak {v}}\), then the claim follows by the induction assumption from the fact that \([e,T_1]=0\). Suppose \(T_1=f\). Then by the induction assumption it suffices to prove that \([[e,f],[T_2,[T_3,[\dots ,[T_{r1},T_r]\dots ]]]]=0\). If \(T_2 \in {\mathfrak {v}}\), the claim follows from the fact that \([e,[f,{\mathfrak {v}}]]=0\) [by (28)] and the induction assumption [or from (29) if \(r=3\)]. Suppose \(T_2=f\). Then it suffices to prove that \([[[e,f],f],[T_3,[\dots ,[T_{r1},T_r]\dots ]]=0\). But \([[[e,f],f],f]=0\) by (30) and \([[[e,f],f],{\mathfrak {v}}]=0\) by (29). It follows that \([[[e,f],f],T_i]=0\), for all \(i=3, \dots , r\), which completes the proof of Proposition 3. \(\square \)
With Proposition 3, application of Lemma 6 completes the proof of Theorem 2.
References
Agaoka, J.: On a generalization of Cartan’s lemma. J. Algebra 127, 470–507 (1989)
Berestovskii, V.N., Nikonorov, Yu.G.: Riemannian Manifolds and Homogeneous Geodesics. Springer Monographs in Mathematics, Springer, Cham (2020)
Calvaruso, G., Fino, A., Zaeim, A.: Homogeneous geodesics of nonreductive homogeneous pseudoRiemannian 4manifolds. Bull. Braz. Math. Soc. 46, 23–64 (2015)
Chen, Z., Wolf, J.A.: Weakly symmetric pseudoRiemannian nilmanifolds. J. Differ. Geom. 121, 541–572 (2022)
Chen, Z., Wolf, J.A., Zhang, S.: On the geodesic orbit property for Lorentz manifolds. J. Geom. Anal. 32, Paper No. 81, 14 (2022)
de Graaf, W.: Classification of 6dimensional nilpotent Lie algebras over fields of characteristic not 2. J. Algebra 309, 640–653 (2007)
del Barco, V., Ovando, G.: Isometric actions on pseudoRiemannian nilmanifolds. Ann. Glob. Anal. Geom. 45, 95–110 (2014)
Dušek, Z., Kowalski, O.: Lightlike homogeneous geodesics and the geodesic lemma for any signature. Publ. Math. Debrecen 71, 245–252 (2007)
Gordon, C.S.: Homogeneous Riemannian manifolds whose geodesics are orbits. In: “Topics in Geometry, in Memory of Joseph D’Atri”, Progress in Nonlinear Differential Equations, vol. 20, pp. 155–174. Birkhäuser, Basel (1996)
Mostow, G.D.: On maximal subgroups of real Lie groups. Ann. Math. (2) 74, 503–517 (1961)
Nikolayevsky, Y., Wolf, J.A.: The structure of geodesic orbit Lorentz nilmanifolds. J. Geom. Anal. 33, 82 (2023). https://doi.org/10.1007/s12220022011344
Ovando, G.P.: Lie algebras with adinvariant metrics: a surveyguide. Rend. Semin. Mat. Univ. Politec. Torino 74, 243–268 (2016)
Wolf, J.A.: On locally symmetric spaces of nonnegative curvature and certain other locally homogeneous spaces. Commun. Math. Helv. 37, 265–295 (1963)
Wolf, J.A.: Families of geodesic orbit spaces and related pseudoRiemannian manifolds. J. Geom. Anal. 33, 214 (2023)
Acknowledgements
ZC was partially supported by NNSF of China (11931009 and 12131012) and Guangdong Basic and Applied Basic Research Foundation (2023A1515010001). YN was partially supported by ARC Discovery Grant DP210100951. JW was partially supported by a Simons Foundation Grant. SZ was partially supported by the National Natural Science Foundation of China (No. 12201358), Natural Science Foundation of Shandong Province (No. ZR2021QA051).
Funding
Open Access funding enabled and organized by CAUL and its Member Institutions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Joe Wolf, one of the authors of this paper, our colleague, mentor and dear friend passed away before the paper was submitted. We were blessed to work with and learn from him. We dedicate this paper to his memory. Zhiqi Chen, Yuri Nikolayevsky, Shaoxiang Zhang.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Chen, Z., Nikolayevsky, Y., Wolf, J.A. et al. PseudoRiemannian Geodesic Orbit Nilmanifolds of Signature \(\varvec{(n2,2)}\). J Geom Anal 34, 132 (2024). https://doi.org/10.1007/s12220024015799
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12220024015799
Keywords
 Pseudo Riemannian nilmanifold
 Geodesic orbit manifold
 Twostep nilpotent
 Double extension
 Lorentz and trans Lorentz nilmanifolds