Classification of homogeneous hypersurfaces in some noncompact symmetric spaces of rank two

Abstract

We classify, up to isometric congruence, the homogeneous hypersurfaces in the Riemannian symmetric spaces $\textrm{SL}(3,\mathbb {H})/\textrm{Sp}(3), \textrm{SO}(5,\mathbb {C})/\textrm{SO}(5),$ and $\textrm{Gr}^*(2,\mathbb {C}^{n+4}) = \textrm{SU}(n+2,2)/\textrm{S}(\textrm{U}(n+2)\textrm{U}(2)), \, n \geqslant 1$.

1 Introduction

A properly embedded connected hypersurface S in a complete connected Riemannian manifold M is called homogeneous if the subgroup of isometries of M preserving S acts transitively on S. A proper isometric action of a connected Lie group G on M is said to be of cohomogeneity one if it has a codimension one orbit, and two such actions are called orbit equivalent if there is an isometry of M mapping the orbits of one action onto the orbits of the other. Nonsingular orbits of cohomogeneity-one actions are homogeneous hypersurfaces and, conversely, every homogeneous hypersurface arises in this way. Since a cohomogeneity-one action is uniquely determined by any of its orbits, the problem of classification of homogeneous hypersurfaces up to isometric congruence is equivalent to that of cohomogeneity-one actions up to orbit equivalence. Such actions have been in the limelight because they can be used to construct various geometric structures on manifolds, like Einstein metrics or metrics with special holonomy. In the context of Riemannian symmetric spaces, they also reveal new connections between geometric and algebraic properties of the space as well as provide the chief example of polar and hyperpolar actions.

The classification of cohomogeneity-one actions on irreducible symmetric spaces of compact type was obtained by Kollross (see [15] and further references there). The reducible case is still open. In the noncompact rank-one situation, the classification was obtained by Berndt, Tamaru, and Brück for real and complex hyperbolic spaces and the Cayley hyperbolic plane [2, 8] and by Díaz-Ramos, Domínguez-Vázquez, and Rodríguez-Vázquez for quaternionic hyperbolic spaces [11]. The higher rank noncompact case, however, presents a significantly more difficult problem. It is a standard fact about cohomogeneity-one actions on Hadamard manifolds that any such action has at most one singular orbit (and no exceptional orbits). Berndt and Tamaru classified cohomogeneity-one actions on irreducible symmetric spaces of noncompact type that have either no singular orbits [6] or a totally geodesic singular orbit [7]. They later went on to prove that any cohomogeneity-one action on such a space M that has a non-totally-geodesic singular orbit must arise from one of two new methods proposed in [9]: the canonical extension and the nilpotent construction. The canonical extension method allows one to obtain cohomogeneity-one actions on M from those on certain totally geodesic submanifolds of M called boundary components. These submanifolds are noncompact symmetric spaces of lower rank, which, in theory, allows one to proceed by induction on rank. The nilpotent construction method reduces the search for cohomogeneity-one actions to a certain problem in representation theory of reductive Lie groups. This method is arguably more involved and has so far produced only two new examples not arising through any other technique. It is noteworthy that both of them are actions on exceptional symmetric spaces related to the exceptional Lie group $G_2$.

Using the above results, Berndt and Tamaru obtained the explicit classification of cohomogeneity-one actions on symmetric spaces $\textrm{SL}(3,\mathbb {R})/\textrm{SO}(3)$ and $G_2^2/\textrm{SO}(4)$ [9]. Later, Berndt and Domínguez-Vázquez refined the nilpotent construction method, which allowed them to obtain the classification on $G_2(\mathbb {C})/G_2, \textrm{SL}(3,\mathbb {C})/\textrm{SU}(3),$ and $\textrm{Gr}^*(2, \mathbb {R}^{n+4}) = \textrm{SO}^0(n+2,2)/\textrm{SO}(n+2)\textrm{SO}(2), \, n \geqslant 1$ [5].

The purpose of the present article is to coalesce ideas from these two papers as well as develop some new computational techniques to obtain the explicit classification of cohomogeneity-one actions—and thus homogeneous hypersurfaces—on several noncompact symmetric spaces of rank 2 that have not been considered yet, namely on $\textrm{SL}(3,\mathbb {H})/\textrm{Sp}(3), \textrm{SO}(5,\mathbb {C})/\textrm{SO}(5),$ and $\textrm{Gr}^*(2,\mathbb {C}^{n+4}) = \textrm{SU}(n+2,2)/\textrm{S}(\textrm{U}(n+2)\textrm{U}(2)), n \geqslant 1$. The first of them is the quaternionic analog of the spaces $\textrm{SL}(3,\mathbb {R})/\textrm{SO}(3)$ and $\textrm{SL}(3,\mathbb {C})/\textrm{SU}(3)$ that have already been dealt with. In order to handle this space, we have to use the classification of so-called protohomogeneous subspaces of $\mathbb {H}^n$ that was recently obtained in [11]. The space $\textrm{Gr}^*(2,\mathbb {C}^{n+4})$ is the complex analog of the real noncompact Grassmannian of two-planes studied by Berndt and Domínguez-Vázquez. It is also the noncompact dual of the ordinary Grassmannian of complex two-planes, a space whose differential geometry was studied in depth by Berndt in [1]. We adapt some results from that paper to deal with the congruence problem for actions on $\textrm{Gr}^*(2,\mathbb {C}^{n+4})$. Finally, $\textrm{SO}(5,\mathbb {C})/\textrm{SO}(5)$ is a space with root system of type $B_2$ dual to the compact symmetric space $\textrm{Spin}(5)$. The nilpotent construction problem is not so onerous for this space, so we handle it using rather elementary methods and no auxiliary results. The main result of the paper is that the nilpotent construction method yields no new actions on these spaces. This provides further evidence to the hypothesis made by Berndt and Tamaru in [9] that the nilpotent construction only yields new actions for exceptional symmetric spaces, and perhaps only those related to $G_2$.

There are still a few noncompact symmetric spaces of rank 2 for which the classification of cohomogeneity-one actions remains open, namely $E_6^{-26}/F_4$ (type $A_2$), $\textrm{Sp}(2,2)/\textrm{Sp}(2)\textrm{Sp}(2)$ (type $C_2$), $\textrm{Gr}^*(2, \mathbb {H}^{n+4}), E_6^{-14}/\textrm{Spin}(10)\textrm{U}(1),$ and $\textrm{SO}(5,\mathbb {H})/\textrm{U}(5)$ (all three of type $(BC)_2$). Unsurprisingly, the main obstacle to the classification for all of them is the complexity of the nilpotent construction problem. For example, for $E_6^{-26}/F_4$ it involves dealing with the 16-dimensional spin representation of $\textrm{Spin}(9)$, while for $\textrm{Sp}(2,2)/\textrm{Sp}(2)\textrm{Sp}(2)$ one has a 10-dimensional representation of the Lorentz group $\textrm{SO}^0(5,1)$. Undoubtedly, novel conceptual methods are needed if one hopes to solve the classification problem for all symmetric spaces of noncompact type.

The paper is organized as follows. Section 2 reviews the terminology, notation, and previous results needed to understand the rest of the paper. We also introduce some new terminology helpful for studying the nilpotent construction problem (Sect. 2.3). Apart from that, we correct a small conceptual error that occurred in [9] and a few other publications of its authors (Sect. 2.4) and prove a useful result for noncompact rank-1 symmetric spaces allowing one to represent cohomogeneity-one actions with a singular orbit on them in a uniform manner (Sect. 2.5). Sections 3, 4, and 5 are each dedicated to one of the three spaces in question.

2 Preliminaries

This section serves a number of purposes. First, it is intended to establish the terminology and notation necessary to understand Berndt and Tamaru’s classification results as well as the rest of the paper. Second, it provides a slightly more general perspective on the nilpotent construction. Finally, it introduces a few useful results that will aid in describing certain cohomogeneity-one actions later in the paper. We follow the notation established in [5, 9] and refer to the same articles as well as [12, 14] for more detailed expositions.

2.1 Parabolic subalgebras and subgroups

Given a symmetric space M, one usually represents it as a quotient G/K, where $G = I^0(M)$ and K is the isotropy subgroup of G at any point. In practice, however, many symmetric spaces are given as quotients G/K, where the Riemannian symmetric pair (G, K) is not effective, i.e., the action of G on G/K is not effective, so we will allow some degree of freedom as well.

Let (G, K) be a Riemannian symmetric pair of noncompact type. Let $(\mathfrak {{g}}, \uptheta )$ denote the corresponding orthogonal symmetric Lie algebra. Recall that we have:

• (1) G and $\mathfrak {{g}}$ are semisimple.

• (2) $\uptheta$ is uniquely determined by K, it is a Cartan involution on $\mathfrak {{g}}$ and gives a Cartan decomposition $\mathfrak {{g}} = \mathfrak {{k}} \oplus \mathfrak {{p}}$ with $\mathfrak {{k}} = \textrm{Lie} (K)$.

• (3) $\uptheta$ admits a unique lift to G, which we denote by $\Uptheta$ and sometimes call a global Cartan involution.

• (4) $G^\Uptheta = K$.

• (5) $Z(G) \subseteq K$ is a discrete subgroup.

We assume that $\mathfrak {{k}}$ contains no nontrivial ideals of $\mathfrak {{g}}$ or, equivalently, that $\mathfrak {{g}}$ has no compact ideals. Denote $M = G/K$ and $o = eK \in M$. Under our assumption, the subgroup of elements of G that act trivially on M is precisely Z(G). We also assume Z(G) is finite, which is the same as to ask that K is compact. In this case, K is a maximal compact subgroup of G. If $\uppi _o :G \rightarrow M$ stands for the orbit map at o, then $d(\uppi _o)_e :\mathfrak {{g}} \rightarrow T_oM$ restricts to an isomorphism $\mathfrak {{p}} {\mathop {\sim }\limits ^{\rightarrow }}T_oM$, and this is an isomorphism between the adjoint representation of K on $\mathfrak {{p}}$ and the isotropy representation of K on $T_oM$. Although any K-invariant inner product on $\mathfrak {{p}}$ gives a G-invariant Riemannian metric on M and turns M into a symmetric space of noncompact type, we take it to be the restriction of the Killing form B of $\mathfrak {{g}}$ to $\mathfrak {{p}}$ by default.¹ We then have a morphism of Lie groups $G \twoheadrightarrow I^0(M)$, and this is a local isomorphism with kernel Z(G). We endow $\mathfrak {{g}}$ with the inner product $B_\uptheta (X,Y) = -B(X, \uptheta Y)$. This inner product coincides with B on $\mathfrak {{p}}$ and equals $-B$ on $\mathfrak {{k}}$, and the Cartan decomposition is orthogonal with respect to both $B_\uptheta$ and B. Whenever we talk about lengths and angles in $\mathfrak {{g}}$, it is with respect to $B_\uptheta$ unless otherwise specified. If we have subspaces $U \subset V \subseteq \mathfrak {{g}}$, we let $V \ominus U$ stand for the orthogonal complement of U in V. Furthermore, if we have a direct sum decomposition $V = U \oplus W$ such that U and W are orthogonal to each other, we sometimes stress it by writing $V = U \oplus ^\perp W$. Observe that $\uptheta$ is orthogonal and self-adjoint with respect to both B and $B_\uptheta$. The adjoint representation of G is orthogonal with respect to B but not $B_\uptheta$. The adjoint representation of K on $\mathfrak {{g}}$, however, is orthogonal with respect to $B_\uptheta$ because it preserve the Cartan decomposition and thus commutes with $\uptheta$.

Pick a maximal abelian subspace $\mathfrak {{a}}$ of $\mathfrak {{p}}$ and denote $r = \dim \mathfrak {{a}} = \textrm{rk} (M)$. Let $A \subseteq G$ be the connected abelian Lie subgroup with $\textrm{Lie} (A) = \mathfrak {{a}}$. The totally geodesic submanifold $A \cdot o \simeq \mathbb {E}^r \subseteq M$ is a maximal flat. Having fixed $\mathfrak {{a}}$, we have the corresponding root system $\Upsigma \subset \mathfrak {{a}}^*$ and restricted root space decomposition $\mathfrak {{g}} = \mathfrak {{g}}_0 \oplus \bigoplus _{\upalpha \in \Upsigma } \mathfrak {{g}}_\upalpha$. All summands in this decomposition are mutually orthogonal. We also have an orthogonal decomposition $\mathfrak {{g}}_0 = \mathfrak {{k}}_0 \oplus \mathfrak {{a}}$, where $\mathfrak {{k}}_0 = Z_\mathfrak {{k}}(\mathfrak {{a}}) = N_\mathfrak {{k}}(\mathfrak {{a}})$. The Dynkin diagram² of $\Upsigma$ will be denoted by $\textrm{DD}_M$. Pick a subset of positive roots $\Upsigma ^+ \subset \Upsigma$ with $\Uplambda = \left\{ {\upalpha _1,\ldots ,\upalpha _r} \right\} \subseteq \Upsigma ^+$ the corresponding set of simple roots and write $\mathfrak {{n}} = \bigoplus _{\upalpha \in \Upsigma ^+} \mathfrak {{g}}_\upalpha$. This is a nilpotent Lie subalgebra and it gives rise to a connected nilpotent Lie subgroup $N \subset G$. We have a vector space decomposition $\mathfrak {{g}} = \mathfrak {{k}} \oplus \mathfrak {{a}} \oplus \mathfrak {{n}}$ known as the Iwasawa decomposition and its global counterpart $G = KAN$ (which is just a diffeomorphism). In particular, both A and N are closed and simply connected and AN is a solvable subgroup of G acting simply transitively on M. Consequently, M can be realized as a solvable subgroup of G with a suitable left-invariant metric, although this of course depends on many choices made along the way. The orbit $N \cdot o$ of N is a properly embedded submanifold of M called a horocycle.

For each $\upalpha \in \Upsigma$ we define

$$\begin{aligned}&\mathfrak {{k}}_\upalpha = \mathfrak {{k}} \cap (\mathfrak {{g}}_\upalpha \oplus \mathfrak {{g}}_{-\upalpha }) = \left\{ {X + \uptheta X \mid X \in \mathfrak {{g}}_\upalpha } \right\} , \\&\mathfrak {{p}}_\upalpha = \mathfrak {{p}} \cap (\mathfrak {{g}}_\upalpha \oplus \mathfrak {{g}}_{-\upalpha }) = \left\{ {X - \uptheta X \mid X \in \mathfrak {{g}}_\upalpha } \right\} , \end{aligned}$$

so $\mathfrak {{k}}_\upalpha \oplus \mathfrak {{p}}_\upalpha = \mathfrak {{g}}_\upalpha \oplus \mathfrak {{g}}_{-\upalpha }$.

When restricted to $\mathfrak {{a}}$, $B_\uptheta$ gives rise to an isomorphism $\mathfrak {{a}} {\mathop {\sim }\limits ^{\rightarrow }}\mathfrak {{a}}^*$. We carry $B_\uptheta$ along this isomorphism to get an inner product on $\mathfrak {{a}}^*$. Given $\upalpha \in \Upsigma$, we write $H_\upalpha$ for the corresponding vector in $\mathfrak {{a}}$, so $\langle {H_\upalpha } | {H_\upbeta } \rangle = \langle {H_\upalpha } , {\upbeta } \rangle = \langle {\upalpha } , {H_\upbeta } \rangle = \langle {\upalpha } | {\upbeta } \rangle$. We also let $H^1, \ldots , H^r$ stand for the basis of $\mathfrak {{a}}$ dual to $\upalpha _1, \ldots , \upalpha _r$, so we have $\langle {H_i} , {\upalpha _j} \rangle = \langle {H^i} | {H_{\upalpha _j}} \rangle = \updelta _{ij}$.

Conjugacy classes of parabolic subalgebras of $\mathfrak {{g}}$ are parametrized by subsets of $\Uplambda$ modulo the action of $\textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$ and thus allow description within the framework of the root space decomposition. We will confine ourselves to maximal proper parabolic subalgebras only, i.e., those corresponding to subsets of cardinality $r-1$. Take any $j \in \left\{ {1,\ldots ,r} \right\}$, and denote $\Upphi _j = \Uplambda \ \left\{ {\upalpha _j} \right\}$. We write $\Upsigma _j$ for the root subsystem of $\Upsigma$ generated by $\Upphi _j$ and $\Upsigma _j^+ = \Upsigma _j \cap \Upsigma ^+$. Define $\mathfrak {{l}}_j = \mathfrak {{g}}_0 \oplus \bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha$ and $\mathfrak {{n}}_j = \bigoplus _{\upalpha \in \Upsigma ^+ \ \Upsigma _j^+} \mathfrak {{g}}_\upalpha$. These subalgebras of $\mathfrak {{g}}$ are reductive and nilpotent, respectively. One can easily check that $[\mathfrak {{l}}_j, \mathfrak {{n}}_j] \subseteq \mathfrak {{n}}_j$, so $\mathfrak {{q}}_j = \mathfrak {{l}}_j \oplus \mathfrak {{n}}_j$ is a Lie subalgebra of $\mathfrak {{g}}$. In fact, it is a maximal proper parabolic subalgebra of $\mathfrak {{g}}$, and the semidirect sum decomposition

is called the Chevalley decomposition of $\mathfrak {{q}}_j$. Every maximal proper parabolic subalgebra of $\mathfrak {{g}}$ is $\textrm{Inn}(\mathfrak {{g}})$-conjugate to $\mathfrak {{q}}_j$ for a unique $j \in \left\{ {1,\ldots ,r} \right\}$, and two subalgebras $\mathfrak {{q}}_j$ and $\mathfrak {{q}}_{i}$ are $\textrm{Aut} (\mathfrak {{g}})$-conjugate if and only if there exists an automorphism of $\textrm{DD}_M$ mapping $\upalpha _j$ to $\upalpha _i$. Next, write $\mathfrak {{a}}_j = \bigcap _{\upalpha \in \Upphi _j} \ker \upalpha = \mathbb {R}H^j$ and $\mathfrak {{a}}^j = \mathfrak {{a}} \ominus \mathfrak {{a}}_j = \bigoplus _{\upalpha \in \Upphi _j} \mathbb {R}H_\upalpha$. It is straightforward to check that $\mathfrak {{l}}_j = Z_{\mathfrak {{g}}}(\mathfrak {{a}}_j) = N_{\mathfrak {{g}}}(\mathfrak {{a}}_j)$. Define $\mathfrak {{m}}_j = \mathfrak {{l}}_j \ominus \mathfrak {{a}}_j$. This is also a reductive Lie subalgebra, and the Chevalley decomposition subdivides further into

, which is called the Langlands decomposition of $\mathfrak {{q}}_j$. Let us write $\mathfrak {{m}}_j = \mathfrak {{z}}_j \oplus \mathfrak {{g}}_j$ for the (unique) Levi decomposition of $\mathfrak {{m}}_j$. Here $\mathfrak {{z}}_j = \mathfrak {{z}}(\mathfrak {{m}}_j) \subseteq \mathfrak {{k}}_0$, while $\mathfrak {{g}}_j = [\mathfrak {{m}}_j, \mathfrak {{m}}_j] = [\mathfrak {{l}}_j, \mathfrak {{l}}_j]$ is semisimple. One readily sees that $\mathfrak {{g}}_j = (\mathfrak {{k}}_0 \ominus \mathfrak {{z}}_j) \oplus \mathfrak {{a}}^j \oplus \bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha$ and that it is a $\uptheta$-stable subalgebra of $\mathfrak {{g}}$. What is more, $\uptheta$ restricts to a Cartan involution of $\mathfrak {{g}}_j$. We denote $\mathfrak {{b}}_j = \mathfrak {{g}}_j \cap \mathfrak {{p}}$. Note that $\mathfrak {{a}}^j$ is a maximal abelian subspace of $\mathfrak {{b}}_j$ and the corresponding restricted root system for $\mathfrak {{g}}_j$ is $\Upsigma _j$. Finally, we define $\mathfrak {{k}}_j = \mathfrak {{q}}_j \cap \mathfrak {{k}} = \mathfrak {{m}}_j \cap \mathfrak {{k}}$.

Now we look at what this plethora of subalgebras and decompositions produces when lifted to G. First things first, we have simply connected closed Lie subgroups $A_j$ and $N_j$ of G corresponding to $\mathfrak {{a}}_j$ and $\mathfrak {{n}}_j$, respectively. By design, $A_j$ is abelian and $N_j$ is nilpotent. Define $L_j = Z_G(\mathfrak {{a}}_j)$. This is a (possibly disconnected) closed reductive Lie subgroup of G with Lie algebra $\mathfrak {{l}}_j$. The product $Q_j = L_j N_j \subseteq G$ is a closed Lie subgroup with Lie algebra $\mathfrak {{q}}_j$, and this is a maximal proper parabolic subgroup of G. It is not hard to see that $L_j$ normalizes $N_j$ and they intersect trivially, so we actually have a semidirect product $Q_j = L_j \ltimes N_j$, which is the global version of the Chevalley decomposition. Next, let $G_j$ be the connected Lie subgroup corresponding to $\mathfrak {{g}}_j$, write $K_j = L_j \cap K$, and define $M_j = K_j G_j$. These are all closed subgroups, $K_j$ is compact, $M_j$ is reductive, $\textrm{Lie} (K_j) = \mathfrak {{k}}_j$, and $\textrm{Lie} (M_j) = \mathfrak {{m}}_j$. We have inclusions $M_j \subset L_j \subset Q_j$, and one can show that $K_j$ is a maximal compact subgroup in all three of these groups. Finally, let $Z_j$ stand for the center of $M_j$. This is a compact subgroup of $K_j$ with Lie algebra $\mathfrak {{z}}_j$. The subgroups $M_j$ and $A_j$ commute and intersect trivially, hence we have a direct product decomposition $L_j = M_j \times A_j$, which, when plugged into the global Chevalley decomposition, induces the global version of the Langlands decomposition: $Q_j = M_j \times A_j \ltimes N_j$.

Eventually, we turn our attention to M. The parabolic subgroup $Q_j$ acts transitively on M with isotropy subgroup $K_j$ at o. The subgroups $A_j$ and $N_j$ give us orbits $A_j \cdot o \simeq \mathbb {{E}}$, which is a geodesic lying in the maximal flat $A \cdot o$, and $N_j \cdot o$, which a properly embedded submanifold of the horocycle $N \cdot o$. We have a Lie triple system $\mathfrak {{b}}_j \subseteq \mathfrak {{p}}$, which corresponds to a totally geodesic properly embedded submanifold $B_j = G_j \cdot o = M_j \cdot o \subseteq M$ often called a boundary component of M in the context of the maximal Satake compactification of M (see, e.g., [10]). The submanifold $B_j$ is itself a symmetric space of noncompact type. It has rank $r-1$ and can be represented, for example, by the Riemannian symmetric pair $(M_j^0,K_j^0)$. We also have a totally geodesic submanifold $F_j = L_j \cdot o \simeq B_j \times (A_j \cdot o)$ corresponding to the Lie triple system $\mathfrak {{b}}_j \oplus \mathfrak {{a}}_j \subseteq \mathfrak {{p}}$ (note that $F_j$ is a symmetric space and $F_j \simeq B_j \times (A_j \cdot o)$ is its decomposition into the Riemannian product of its noncompact and Euclidean parts). Finally, we can form a commutative diagram

The diffeomorphism $B_j \times A_j \times N_j \simeq M$ is called a horospherical decomposition of M.

2.2 Classes of cohomogeneity-one actions

Now we describe the different types of cohomogeneity-one actions that appear in the structure results of Berndt and Tamaru.

Let $\ell \subseteq \mathfrak {{a}}$ be a one-dimensional subspace. Then, the connected Lie subgroup $H_\ell$ of G with Lie algebra $\mathfrak {{h}}_\ell = (\mathfrak {{a}} \ominus \ell ) \oplus \mathfrak {{n}}$ is closed and its action on M has cohomogeneity one and no singular orbits (in other words, its orbits form a Riemannian foliation of codimension one). Moreover, all of its orbits are isometrically congruent to each other. If T is an automorphism of the Dynkin diagram $\textrm{DD}_M$ and $\widehat{T}$ is the orthogonal transformation of $\mathfrak {{a}}$ induced by T, then the actions of $H_\ell$ and $H_{\widehat{T}(\ell )}$ are orbit equivalent. Conversely, if $\ell$ and $\ell '$ are one-dimensional subspaces of $\mathfrak {{a}}$ such that the actions of $H_\ell$ and $H_{\ell '}$ are orbit equivalent, then these subspaces differ by $\widehat{T}$ for some $T \in \textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$. See [6, 18] for more details.

Let $\ell \subseteq \mathfrak {{g}}_{\upalpha _i}$ be a one-dimensional subspace, $1 \leqslant i \leqslant r$. Then, the connected Lie subgroup $H_i$ of G with Lie algebra $\mathfrak {{h}}_i = \mathfrak {{a}} \oplus (\mathfrak {{n}} \ominus \ell )$ is closed and its action on M has cohomogeneity one, no singular orbits, and exactly one minimal orbit. Were we to choose another line $\ell '$ in $\mathfrak {{g}}_{\upalpha _i}$, the resulting action would be orbit equivalent to that for $\ell$. If $T \in \textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$ is some automorphism mapping $\upalpha _i$ to $\upalpha _j$, the actions of $H_i$ and $H_j$ are orbit equivalent. Conversely, if we have $1 \leqslant i,j \leqslant r$ such that the actions of $H_i$ and $H_j$ are orbit equivalent, then $\upalpha _i$ and $\upalpha _j$ differ by some automorphism of $\textrm{DD}_M$. Again, see [6, 18] for details.

Let L be a maximal proper connected reductive Lie subgroup of G (it is automatically closed). Let $H \subseteq L$ be a closed connected subgroup acting on M with cohomogeneity one. Then, H and L have the same orbits (and thus are trivially orbit equivalent), and their orbit through o is totally geodesic. Moreover, unless M is isometric to a real hyperbolic space, this orbit is singular. Such actions were explicitly classified in [7] for M irreducible. In a nutshell, the classification goes as follows. A properly embedded connected submanifold F of M is called reflective if the geodesic reflection in F is a well-defined isometry of M (equivalently, if F is a connected component of the fixed point set of an involutive isometry of M). A reflective submanifold F is necessarily totally geodesic and any of its normal spaces is itself tangent to a totally geodesic submanifold, which we sloppily denote³ by $F^\perp$. If F is reflective and $F^\perp$ is of rank 1 (hence a hyperbolic space), then F is a totally geodesic singular orbit of a cohomogeneity-one action. Conversely, a totally geodesic singular orbit F of a cohomogeneity-one action is reflective with $F^\perp$ of rank one, apart from 5 non-reflective examples, all of which are mysteriously related to the exceptional Lie group $G_2$ (and none of which will appear in this paper so they are of no concern to us). What Berndt and Tamaru did in [7] is they classified reflective submanifolds F with $\textrm{rk} (F^\perp ) = 1$ for each irreducible M of noncompact type.

Now fix $j \in \left\{ {1,\ldots ,r} \right\}$. Let $H_j$ be a closed connected subgroup of $I(B_j)$ acting on the boundary component $B_j$ with cohomogeneity one. Write $\mathfrak {{i}}(B_j)$ for the Lie algebra of $I(B_j)$ and $\mathfrak {{h}}_j \subseteq \mathfrak {{i}}(B_j)$ for that of $H_j$. Note that as a symmetric space, $B_j$ can be represented by a Riemannian symmetric pair $(G_j, G_j \cap K)$. In particular, we have a morphism $G_j \twoheadrightarrow I^0(B_j)$, which induces a Lie algebra homomorphism $\mathfrak {{g}}_j \twoheadrightarrow \mathfrak {{i}}(B_j)$. It may fail to be an isomorphism, for the action $G_j \curvearrowright B_j$ does not have to be effective. Nonetheless, $\mathfrak {{g}}_j$ is semisimple, so there is a unique ideal in it complementary to the kernel of the map $\mathfrak {{g}}_j \twoheadrightarrow \mathfrak {{i}}(B_j)$, hence we can identify $\mathfrak {{i}}(B_j)$ with this ideal (see Sect. 2.4 for a more explicit description of this ideal and discussion of the non-effectiveness of $G_j \curvearrowright B_j$). But now we have $\mathfrak {{h}}_j \subseteq \mathfrak {{i}}(B_j) \hookrightarrow \mathfrak {{g}}_j \subseteq \mathfrak {{m}}_j$ as well as the Langlands decomposition

, so we can form a Lie subalgebra

. The corresponding connected Lie subgroup $H_j^\Uplambda \subseteq Q_j$ is closed and acts on M with cohomogeneity one. This action is called the canonical extension of the action $H_j \curvearrowright B_j$, and the group $H_j^\Uplambda$ itself is called the canonical extension of $H_j$. The action of $H_j^\Uplambda$ on M has a singular orbit if and only if the action of $H_j$ on $B_j$ has one, and the codimension of the former in M equals the codimension of the latter in $B_j$. Note that if we are given two such subgroups $H_j, H'_j \subseteq I^0(B_j)$ whose actions on $B_j$ are orbit equivalent, the resulting actions of $H_j^\Uplambda$ and ${H'_j}^\Uplambda$ on M may fail to be orbit equivalent (see the example after Proposition 4.2 in [9]). However, if the actions of $H_j$ and $H'_j$ are orbit equivalent by some $f \in I^0(B_j)$ (and not just $I(B_j)$), the actions of $H_j^\Uplambda$ and ${H'_j}^\Uplambda$ will in fact be orbit equivalent. For this reason we introduce the following useful notion: two isometric actions on a Riemannian manifold Q are called strongly orbit equivalent if their orbits can be identified by some isometry $f \in I^0(Q)$. We stress that $I^0(B_j)$ is not a subgroup of $G_j$ or $M_j$ but a quotient thereof, so it is necessary to embed $\mathfrak {{i}}(B_j)$ into $\mathfrak {{g}}_j$ for the above construction to work, as the sum

does not make sense a priori (we are pointing out this small issue because it was not taken into account in the description of the canonical extension method in [5, 9]). We refer to [9] for more details.

Finally, we describe the nilpotent construction method. Once again, fix $j \in \left\{ {1,\ldots ,r} \right\}$. The nilpotent Lie algebra $\mathfrak {{n}}_j$ is graded:

$$\begin{aligned} \mathfrak {{n}}_j = \bigoplus _{\upnu = 1}^k \mathfrak {{n}}_j^\upnu , \quad \mathfrak {{n}}_j^\upnu = \bigoplus _{\upalpha (H^j) = \upnu } \mathfrak {{g}}_\upalpha , \quad k = \updelta (H^j), \end{aligned}$$

where $\updelta \in \Upsigma ^+$ is the highest root. The adjoint actions of $L_j$ on $\mathfrak {{n}}_j$ respects this grading. Let $\mathfrak {{v}} \subseteq \mathfrak {{n}}_j^1$ be a linear subspace of dimension at least 2. One can easily see that $\mathfrak {{n}}_{j,\mathfrak {{v}}} = \mathfrak {{n}}_j \ominus \mathfrak {{v}}$ is a Lie subalgebra. Its corresponding connected Lie subgroup of G is closed and will be denoted by $N_{j, \mathfrak {{v}}}$. We also have $N_{\mathfrak {{l}}_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) = N_{\mathfrak {{m}}_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) \oplus \mathfrak {{a}}_j$, which implies $N_{L_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) = N_{M_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) \times A_j$. Furthermore, $N_{\mathfrak {{l}}_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) = \uptheta N_{\mathfrak {{l}}_j}(\mathfrak {{v}})$ and thus $N^0_{L_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) = \Uptheta N^0_{L_j}(\mathfrak {{v}})$, and the same remains true with $\mathfrak {{l}}_j$ and $L_j$ replaced with $\mathfrak {{m}}_j$ and $M_j$, respectively. Define a closed connected Lie subgroup $H_{j, \mathfrak {{v}}} = N^0_{L_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) \ltimes N_{j,\mathfrak {{v}}} \subset L_j \ltimes N_j = Q_j$. It was shown in [5, 9] that the following assumptions are equivalent:

• (i) $F_j \subseteq H_{j, \mathfrak {{v}}} \cdot o$.

• (ii) $N^0_{L_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}})$ acts transitively on $F_j$.

• (iii) $N^0_{M_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}})$ acts transitively on $B_j$.

• (iv) The image of the projection of $N_{\mathfrak {{l}}_j}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ along $\mathfrak {{k}}$ equals $\mathfrak {{b}}_j \oplus \mathfrak {{a}}_j$.

• (v) The image of the projection of $N_{\mathfrak {{m}}_j}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ along $\mathfrak {{k}}$ equals $\mathfrak {{b}}_j$.

Now, suppose the following two assumptions, which we sometimes call ’assumptions (1) and (2) of the nilpotent construction’, are satisfied:

• (1) The image of the projection of $N_{\mathfrak {{m}}_j}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ along $\mathfrak {{k}}$ equals $\mathfrak {{b}}_j$, and

• (2) $N_{K_j}(\mathfrak {{n}}_{j,\mathfrak {{v}}}) = N_{K_j}(\mathfrak {{v}})$ acts transitively on the unit sphere in $\mathfrak {{v}}$.

In this case, $H_{j, \mathfrak {{v}}}$ acts on M with cohomogeneity one and its orbit though o is singular of codimension equal to $\dim \mathfrak {{v}}$. What is more, if $\mathfrak {{v}}' \subseteq \mathfrak {{n}}_j^1$ is another such subspace that differs from $\mathfrak {{v}}$ by $\textrm{Ad} (k)$ for some $k \in K_j$, then $\mathfrak {{v}}$ satisfies the above assumptions if and only if $\mathfrak {{v}}'$ does, and, if they do, the actions $H_{j, \mathfrak {{v}}}$ and $H_{j, \mathfrak {{v}}'}$ are orbit equivalent. See [5, 9] for details.

Now we formulate the main result from [9], which guarantees that all cohomogeneity-one actions on irreducible symmetric spaces of noncompact type can be obtained by one of the five methods described above.

Theorem 2.1

Let $M = G/K$ be an irreducible Riemannian symmetric space of noncompact type and rank r, and let H be a closed connected subgroup of G acting on M with cohomogeneity one. Then, one of the following statements holds:

• (1) The orbits of H form a Riemannian foliation and exactly one of the following two cases holds:

  • (i) The action of H is orbit equivalent to the action of $H_\ell$ for some one-dimensional linear subspace $\ell \subseteq \mathfrak {{a}}$, where $\ell$ is determined uniquely up to the action of $\textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$ on $\mathbb {{P}}\mathfrak {{a}}$.

  • (ii) The action of H is orbit equivalent to the action of $H_i$ for some $i \in \left\{ {1,\ldots ,r} \right\}$, where i is determined uniquely up to the action of $\textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$ on $\left\{ {1,\ldots ,r} \right\}$.

• (2) There exists exactly one singular orbit and one of the following two cases holds:

  • (i) H is contained in a maximal proper connected reductive Lie subgroup L of G, H, and L have the same orbits, and the singular orbit is totally geodesic and isometrically congruent to one of the submanifolds described in Theorems 3.3 and 4.2 in [7].

  • (ii) H is contained in a maximal proper parabolic subgroup $Q_j$ of G (for suitable choices of $o \in M, \mathfrak {{a}} \subseteq \mathfrak {{p}},$ and $\Upsigma ^+ \subset \Upsigma$) for some $j \in \left\{ {1, \ldots , r} \right\}$ and one of the following two subcases holds:

    • (a) The action of H is orbit equivalent to the canonical extension of a cohomogeneity-one action with a singular orbit on the boundary component $B_j$ of M.

    • (b) The action of H is orbit equivalent to the action of a group $H_{j, \mathfrak {{v}}}$ obtained by nilpotent construction applied to some subspace $\mathfrak {{v}} \subseteq \mathfrak {{n}}_j^1$ with $\dim \mathfrak {{v}} \geqslant 2$ satisfying both assumptions of the construction.

Remarks

• 1. Here is how one goes about explicitly classifying cohomogeneity-one actions on a given irreducible symmetric space of noncompact type. Actions without singular orbits and with a totally geodesic singular orbit correspond to (1) and (2)(i), respectively, and we already have their explicit classification, so we only need to classify actions with a non-totally-geodesic singular orbit. To this end, a priori, we need to find all actions arising through the canonical extension and nilpotent construction methods for each $j \in \left\{ {1,\ldots ,r} \right\}$. However, the problem is alleviated by the fact that if there exists an automorphism of $\textrm{DD}_M$ mapping $\upalpha _i$ to $\upalpha _j$, then we need only deal with these two constructions for i. This follows from the fact that any $T \in \textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$ gives an automorphism of $\Upsigma$ as a root system, and any automorphism of $\Upsigma$ is given by $({\left. \textrm{Ad} (k) \right| _{\mathfrak {{a}}}})^*$, where $k \in N_{\widetilde{K}}(\mathfrak {{a}})$ and $\widetilde{K}$ is the isotropy subgroup of I(M) at o. If we start with $T \in \textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$ mapping $\upalpha _i$ to $\upalpha _j$ and find $k \in N_{\widetilde{K}}(\mathfrak {{a}})$ that gives the corresponding automorphism of $\Upsigma$, then k maps $B_j$ onto $B_i$, $\mathfrak {{n}}_j$ onto $\mathfrak {{n}}_i$, etc., so it provides an orbit equivalence between actions canonically extended from $B_j$ and $B_i$, and the same for the nilpotent construction (this argument remains valid if M is reducible provided that the Riemannian metric comes from the Killing form). Finding all actions arising by canonical extension from $B_j$ entails knowing the classification of cohomogeneity-one actions on $B_j$ up to strong orbit equivalence. Since $B_j$ has rank lower than M, an inductive algorithm is sometimes possible (the main problem here is that $B_j$ may happen to be reducible,⁴ and the classification of cohomogeneity-one actions on reducible spaces has not been fully developed yet). Finding all actions arising from the nilpotent construction basically boils down to a problem in representation theory of reductive Lie groups. We will discuss this in a bit more detail in the next subsection. Having found all such actions for all j, one also needs to tell which of them are orbit equivalent or have a totally geodesic singular orbit.

• 2. Cases (2)(i), (2)(ii)(a), and (2)(ii)(b) may overlap. For example, an action obtained by nilpotent construction for some j may have a totally geodesic singular orbit or be orbit equivalent to some action canonically extended from some $B_i$. We will see examples of such behavior later in the paper.

• 3. It is usually fairly straightforward to find what the Lie algebras $\mathfrak {{m}}_j$ and $\mathfrak {{k}}_j$ and thus the groups $M_j^0$ and $K_j^0$ look like. It may be a bit harder, however, to do so for the full groups $M_j$ and $K_j$. In practical terms, it means that in the nilpotent construction it may be easier to find all subspaces $\mathfrak {{v}} \subseteq \mathfrak {{n}}_j^1$ satisfying the two assumptions up to $K_j^0$-congruence but not $K_j$-congruence. To make things worse, even if two subspaces $\mathfrak {{v}}, \mathfrak {{v}}' \subseteq \mathfrak {{n}}_j^1$ are not $K_j$-congruent, they may still, in theory, produce orbit equivalent actions, so we may end up with a larger list of actions and have to discard some of them. However, this scarcely poses an actual problem, for the reality is that whenever we see an action arising by nilpotent construction, chances are it either has a totally geodesic singular orbit or arises by canonical extension from some boundary component of M.

• 4. Finally, note that if M has rank 1 and is, therefore, isometric to a hyperbolic space over $\mathbb {R}, \mathbb {C}, \mathbb {H},$ or $\mathbb {{O}}$, then it has $\Uplambda = \left\{ {\upalpha _1} \right\}$ and the boundary component $B_1$ is a point. For this reason, the canonical extension method produces exactly one cohomogeneity-one action, which is nothing but the action of the isotropy subgroup K. Moreover, assumption (1) of the nilpotent construction is trivially satisfied for every $\mathfrak {{v}} \subseteq \mathfrak {{n}}_1^1$. Historically, the rank-1 case had mostly been handled before the advent of the canonical extension and nilpotent construction methods (see [2, 8]), although the case of $\mathbb {H}H^n$ stayed unresolved until recently (see [11]).

Theorem 2.1 gives the classification of connected properly embedded homogeneous hypersurfaces in irreducible symmetric spaces of noncompact type up to isometric congruence, which justifies the title of the present paper. Indeed, given a connected complete Riemannian manifold M, connected properly embedded homogeneous hypersurfaces in M are the same as nonsingular orbits of proper isometric cohomogeneity-one actions on M by connected Lie groups. An orbit equivalence between two such actions provides an isometric congruence between their corresponding orbits by design. Conversely, let H be a connected Lie group acting properly, isometrically, and with cohomogeneity one on M, and let $F \subseteq M$ be any orbit of H. Then, all the other orbits are uniquely determined by F as the connected components of the sets $F_r = \left\{ {x \in M \mid \textrm{dist}(x,F) = r} \right\} , \; r > 0$. (Informally, these are equidistant ’tubes’ of various radii around F; $F_r$ is connected when F is nonprincipal but may have two connected components otherwise.) Consequently, if we are given another such action $H' \curvearrowright M$ and an orbit $F' \subseteq M$ of $H'$, an isometry $f \in I(M)$ mapping F onto $F'$ must map $F_r$ onto $F'_r$ for each $r > 0$, so f is an orbit equivalence between the actions of H and $H'$. If F is a singular orbit and there are no other singular orbits (which is always the case when M is a noncompact symmetric space, see [2, Section 2]), any orbit equivalence of the action of H with itself preserves F and thus each $F_r$, which means that no two distinct orbits of H are congruent. Finally, considering the case of cohomogeneity-one actions without singular orbits, that is, cases (1)(i) and (1)(ii) in Theorem 2.1, it was shown in [6] that all orbits of $H_\ell$ are congruent, whereas $H_i$ has a unique minimal orbit F, and $F_r$ consists of two orbits (for each $r>0$) that are congruent to each other and to no other orbits. Note that, given a cohomogeneity-one action $H \curvearrowright M$ and its principal orbit $F \subseteq M$, the singular points of the action are precisely the focal points of F.

Altogether, the classification of homogeneous hypersurfaces in an irreducible noncompact symmetric space M goes like this. Every connected properly embedded homogeneous hypersurface in M is isometrically congruent to a principal orbit of a cohomogeneity-one action on M of one of the 5 types described in Theorem 2.1. Congruent hypersurfaces correspond to orbit equivalent actions. Each $H_\ell$ ($\ell \in \mathbb {{P}}\mathfrak {{a}}/\textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$) in (1)(i) gives precisely one congruence class of homogeneous hypersurfaces, each $H_i$ ($i \in \left\{ {1, \ldots , r} \right\} /\textrm{Aut} ^\textrm{w}(\textrm{DD}_M)$) in (1)(ii) gives a one-parameter family of classes parametrized by $t \geqslant 0$, and each action in (2) gives a one-parameter family parametrized by $t>0$.

2.3 Generalizing the nilpotent construction problem

Since the nilpotent construction is so involved and presents one of the main hindrances to the classification of cohomogeneity-one actions on irreducible symmetric spaces of noncompact type, it is worth looking at it from a slightly more general perspective and introducing some new terminology to talk about it more easily.

Let G be a reductive Lie group with Lie algebra $\mathfrak {{g}}$, a maximal compact subgroup $K \subseteq G$, and a Cartan decomposition $\mathfrak {{g}} = \mathfrak {{k}} \oplus \mathfrak {{p}}$ (we are using the notation from [14, Section VII.2] in this subsection). Let V be a finite-dimensional real representation of G. Fix some K-invariant inner product on V and let $\mathfrak {{v}} \subseteq V$ be a subspace with $\dim \mathfrak {{v}} \geqslant 2$. Inspired by [11], we call $\mathfrak {{v}}$ protohomogeneous if there exists a subgroup of K preserving $\mathfrak {{v}}$ and acting transitively on the unit sphere in $\mathfrak {{v}}$. We call $\mathfrak {{v}}$ admissible if the projection of $N_{\mathfrak {{g}}}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ along $\mathfrak {{k}}$ is onto, i.e., its image equals the whole $\mathfrak {{p}}$. The problem, which we call the generalized nilpotent construction problem, is to classify subspaces of V that are both protohomogeneous and admissible up to the action of K. One readily sees how this generalizes the nilpotent construction problem by taking $G = M_j$ (or $L_j$), $K = K_j, \mathfrak {{g}} = \mathfrak {{m}}_j = \mathfrak {{k}}_j \oplus \mathfrak {{b}}_j$ (or $\mathfrak {{l}}_j = \mathfrak {{k}}_j \oplus (\mathfrak {{b}}_j \oplus \mathfrak {{a}}_j)$), and $V = \mathfrak {{n}}_j^1$. Admissibility and protohomogeneity become assumptions (1) and (2) of the nilpotent construction, respectively. Note that every subspace of V is trivially admissible when G is compact.

This problem is solved for a few representations. For instance, Berndt and Brück handled the cases $(G, V) = (\textrm{SO}(n), \mathbb {R}^n), (\textrm{U}(n), \mathbb {C}^n), (\textrm{Spin}(7), \text {the spin representation} \; \mathbb {R}^8)$ in [2], while Díaz-Ramos et al. dealt with $(G,V) = (\textrm{Sp}(n)\textrm{Sp}(1), \mathbb {H}^n)$ in [11]. See also [5, 9] for some examples with G noncompact. The latter of these two papers contains also a general result on the nilpotent construction problem that facilitates the search for admissible and protohomogeneous subspaces for certain noncompact symmetric spaces and choices of j (see Proposition 5 there), and it may, in theory, be possible to extend this result to the generalized nilpotent construction problem. When dealing with the nilpotent construction on some rank-2 spaces later in the paper, we will solve this problem for a few more representations. We want to stress, however, that almost all attempts to solve the problem have so far been ad hoc, and it does not seem feasible to proceed like this in the future. The more complex the representation of G on V is, the more difficult the problem becomes (for instance, it took the authors of [11] an entire paper to solve the problem for $\textrm{Sp}(n)\textrm{Sp}(1) \curvearrowright \mathbb {H}^n$!). We believe that a more holistic and general approach to the problem is necessary if one wants to solve the nilpotent construction problem for all symmetric spaces of noncompact type.

2.4 Error correction

Before proceeding to the classification of cohomogeneity-one actions on concrete symmetric spaces, we would like to point out a minor error in [3, Paragraph 13.2], [4, Section 3], and [9, Section 2] and correct it.

It is stated in the above works that $\mathfrak {{g}}_j \cap \mathfrak {{k}} = [\mathfrak {{b}}_j, \mathfrak {{b}}_j]$. It is not true in general. For example, if we take $\mathfrak {{g}} = \mathfrak {{sl}}(3, \mathbb {H})$ with $\uptheta (X) = X^* = -\bar{X}$, $\mathfrak {{a}} = \left\{ {\text {real traceless diagonal matrices}} \right\}$, and $\upalpha _j$ any of the two simple roots of $\Upsigma$, then $\mathfrak {{g}}_j \simeq \mathfrak {{sl}}(2, \mathbb {H}) \oplus \mathfrak {{sp}}(1)$, so any Cartan involution on it will be trivial on the compact summand $\mathfrak {{sp}}(1)$. Consequently, we have $[\mathfrak {{b}}_j, \mathfrak {{b}}_j] \oplus \mathfrak {{b}}_j = \mathfrak {{sl}}(2, \mathbb {H})$ but $\mathfrak {{g}}_j \cap \mathfrak {{k}} = [\mathfrak {{b}}_j, \mathfrak {{b}}_j] \oplus \mathfrak {{sp}}(1)$. The reason for this issue is that in general $\mathfrak {{g}}_j$ may have nonzero compact ideals. In fact, we have the following

Proposition 2.2

Let $\mathfrak {{g}}$ be a semisimple Lie algebra with a Cartan involution $\uptheta$, and let $\mathfrak {{g}} = \bigoplus _{\upmu = 1}^k \mathfrak {{g}}_\upmu \oplus \bigoplus _{\upnu = k+1}^n \mathfrak {{g}}_\upnu$ be its decomposition into simple ideals, where the first k ideals are noncompact and the rest are compact. Then, $\uptheta$ respects this decomposition and is the identity on the compact ideals. Moreover, if we write $\mathfrak {{g}}_\upmu = \mathfrak {{k}}_\upmu \oplus \mathfrak {{p}}_\upmu$ for the induced Cartan decomposition of each factor, then

$$\begin{aligned} {[}\mathfrak {{p}}, \mathfrak {{p}}] = \bigoplus _{\upmu =1}^k [\mathfrak {{p}}_\upmu , \mathfrak {{p}}_\upmu ] = \bigoplus _{\upmu =1}^k \mathfrak {{k}}_\upmu . \end{aligned}$$

Proof

The only thing which is not straightforward here is that $[\mathfrak {{p}}_\upmu , \mathfrak {{p}}_\upmu ] = \mathfrak {{k}}_\upmu$ for $1 \leqslant \upmu \leqslant k$. One possible way to show this is explained in [14, Section VI, problems 22-24].

Here and in the future we refer to the sum of all (non)compact ideals of a semisimple Lie algebra as its (non)compact part.

Corollary 2.1

In Proposition 2.2, the compact part $\bigoplus _{\upnu = k+1}^n \mathfrak {{g}}_\upnu$ of $\mathfrak {{g}}$ can be described as $Z_\mathfrak {{k}}(\mathfrak {{p}}) = Z_\mathfrak {{g}}(\mathfrak {{p}})$.

Proof

For each $\upmu \in \left\{ {1, \ldots , k} \right\}$, if X lies in $Z_{\mathfrak {{g}}_\upmu }(\mathfrak {{p}}_\upmu )$, then it also commutes with $[\mathfrak {{p}}_\upmu , \mathfrak {{p}}_\upmu ] = \mathfrak {{k}}_\upmu$ and hence lies in the center of $\mathfrak {{g}}_\upmu$, which is zero. The rest follows.

This issue with $\mathfrak {{g}}_j$ also means that the action of $G_j$ on the boundary component $B_j$ may fail badly to be effective: if there is a nonzero compact ideal in $\mathfrak {{g}}_j$, its corresponding Lie subgroup will act trivially on $B_j$. So what we need is to exclude the compact ideals of $\mathfrak {{g}}_j$ from consideration. We will give an explicit description of the noncompact part of $\mathfrak {{g}}_j$ in terms of the restricted root space decomposition of $\mathfrak {{g}}$.

Define $\varvec{\widetilde{\mathfrak {{g}}}_j}$ to be Lie subalgebra of $\mathfrak {{g}}$ generated by $\mathfrak {{g}}_\upalpha$’s with $\upalpha \in \Upsigma _j$. Clearly, $\widetilde{\mathfrak {{g}}}_j \subseteq \mathfrak {{g}}_j$. Since any $X \in \mathfrak {{g}}_\upalpha$ satisfies $[X, \uptheta X] = -||X||^2 H_\upalpha$, we have $\mathfrak {{a}}^j \subseteq \widetilde{\mathfrak {{g}}}_j$, so we can write $\widetilde{\mathfrak {{g}}}_j = (\widetilde{\mathfrak {{g}}}_j \cap \mathfrak {{k}}_0) \oplus \mathfrak {{a}}^j \oplus \bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha$. Observe also that $(\widetilde{\mathfrak {{g}}}_j \cap \mathfrak {{k}}_0) \oplus \mathfrak {{a}}^j = \sum _{\upalpha \in \Upsigma _j^+} [\mathfrak {{g}}_\upalpha , \mathfrak {{g}}_{-\upalpha }]$.

Proposition 2.3

The noncompact part of $\mathfrak {{g}}_j$ is precisely $\widetilde{\mathfrak {{g}}}_j$, while the compact part is $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) \ominus \mathfrak {{z}}_j$. The compact and noncompact parts of $\mathfrak {{g}}_j$ are mutually orthogonal (with respect to $B_\uptheta$).

Proof

We start by looking at $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j)$ and proving a number of alternative expressions for it, namely $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) = Z_{\mathfrak {{k}}_0}(\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha ) = Z_{\mathfrak {{k}}_0}(\widetilde{\mathfrak {{g}}}_j) = Z_{\mathfrak {{m}}_j}(\mathfrak {{b}}_j) = Z_{\mathfrak {{m}}_j}(\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha ) = Z_{\mathfrak {{m}}_j}(\widetilde{\mathfrak {{g}}}_j)$. It suffices to show that the smallest of these six centralizers, $Z_{\mathfrak {{k}}_0}(\widetilde{\mathfrak {{g}}}_j)$, coincides with the two largest ones, $Z_{\mathfrak {{m}}_j}(\mathfrak {{b}}_j)$ and $Z_{\mathfrak {{m}}_j}(\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha )$.

Take $X \in Z_{\mathfrak {{m}}_j}(\mathfrak {{b}}_j)$. Since $\mathfrak {{a}}^j \subseteq \mathfrak {{b}}_j$, X cannot have nonzero components in any of $\mathfrak {{g}}_\upalpha$, $\upalpha \in \Upsigma _j$, so $X \in \mathfrak {{k}}_0 \oplus \mathfrak {{a}}^j = \mathfrak {{g}}_0 \cap \mathfrak {{m}}_j$. Let $Y \in \mathfrak {{g}}_\upalpha$, $\upalpha \in \Upsigma _j$. Then, $Y - \uptheta Y \in \mathfrak {{b}}_j$, so $[X, Y - \uptheta Y] = 0$, which means that $[X, Y] = [X, \uptheta Y]$. But $[X, Y] \in \mathfrak {{g}}_\upalpha$, whereas $[X, \uptheta Y] \in \mathfrak {{g}}_{-\upalpha }$, so both are zero, which implies that X commutes with $\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha$ and hence with the subalgebra generated by it, which is $\widetilde{\mathfrak {{g}}}_j$. Finally, if X had a nonzero component in $\mathfrak {{a}}^j$, it would not commute with some $\mathfrak {{g}}_\upalpha$, $\upalpha \in \Upsigma _j$. We conclude that $X \in Z_{\mathfrak {{k}}_0}(\widetilde{\mathfrak {{g}}}_j)$, so $Z_{\mathfrak {{m}}_j}(\mathfrak {{b}}_j) = Z_{\mathfrak {{k}}_0}(\widetilde{\mathfrak {{g}}}_j)$. Similar arguments show that $Z_{\mathfrak {{m}}_j}(\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha ) = Z_{\mathfrak {{k}}_0}(\widetilde{\mathfrak {{g}}}_j)$.

Now we deal with the compact part of $\mathfrak {{g}}_j$. According to Corollary 2.1, it coincides with $Z_{\mathfrak {{g}}_j}(\mathfrak {{b}}_j)$. We claim that $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j)$ decomposes as an orthogonal direct sum $\mathfrak {{z}}_j \oplus Z_{\mathfrak {{g}}_j}(\mathfrak {{b}}_j)$. Indeed, it follows immediately from the fact that $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) = Z_{\mathfrak {{m}}_j}(\mathfrak {{b}}_j)$ and that $\mathfrak {{m}}_j = \mathfrak {{z}}_j \oplus ^\perp \mathfrak {{g}}_j$. The assertion in the proposition about the compact part follows.

Now we have to show that the noncompact part of $\mathfrak {{g}}_j$ is $\widetilde{\mathfrak {{g}}}_j$ and that it is orthogonal to the compact part. First, we show that $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) \perp \widetilde{\mathfrak {{g}}}_j$. Indeed, any X from the centralizer is already orthogonal to $\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha$, and if Y, Z lie in the latter, we have $\langle {X} | {[Y,Z]} \rangle = -B(X, [Y,Z]) = -B([X,Y],Z) = 0$. Since $\widetilde{\mathfrak {{g}}}_j$ is spanned as a vector space by $\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha$ and $[\bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha , \bigoplus _{\upalpha \in \Upsigma _j} \mathfrak {{g}}_\upalpha ]$, X is orthogonal to $\widetilde{\mathfrak {{g}}}_j$. Next, assume $X \in \mathfrak {{k}}_0$ is orthogonal to $\widetilde{\mathfrak {{g}}}_j$. We want to show that it lies in $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j)$, that is, commutes with all $\mathfrak {{g}}_\upalpha$’s, $\upalpha \in \Upsigma _j$. Given $Y \in \mathfrak {{g}}_\upalpha$, $Z = [X,Y] \in \mathfrak {{g}}_\upalpha$, and $||Z||^2 = -B([X,Y], \uptheta Z) = -B(X, [Y, \uptheta Z]) = \langle {X} | {[Y, \uptheta Z]} \rangle = 0$, since $[Y, \uptheta Z]$ lies in $\widetilde{\mathfrak {{g}}}_j$. We deduce that $\mathfrak {{g}}_j = (Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) \ominus \mathfrak {{z}}_j) \oplus ^\perp \widetilde{\mathfrak {{g}}}_j$.

The fact that $\widetilde{\mathfrak {{g}}}_j$ is actually an ideal of $\mathfrak {{g}}_j$ is obvious because it has a complementary ideal $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) \ominus \mathfrak {{z}}_j$ with which it commutes.

Remark 2.4

The same is true when we work with arbitrary parabolic subalgebras, not necessarily maximal ones. Instead of taking $\Upphi _j = \Uplambda \ \left\{ {\upalpha _j} \right\}$, one takes arbitrary subsets $\Upphi \subseteq \Uplambda$ and defines $\mathfrak {{q}}_\Upphi$ and other related subalgebras in a similar fashion. The semisimple subalgebra $\mathfrak {{g}}_\Upphi$ may contain compact ideals, so the equality $[\mathfrak {{b}}_\Upphi , \mathfrak {{b}}_\Upphi ] = \mathfrak {{g}}_\Upphi \cap \mathfrak {{k}}$ does not hold in general. But the above proof applies almost verbatim in this more general situation and yields a handy description of the compact and noncompact parts of $\mathfrak {{g}}_\Upphi$.

Define $\varvec{\widetilde{G}_j}$ to be the connected Lie subgroup of G corresponding to $\widetilde{\mathfrak {{g}}}_j$. Since $\widetilde{G}_j$ is semisimple and G has finite center, it follows from [16, Section 6] that $\widetilde{G}_j$ is closed. The intersection $\widetilde{G}_j \cap K$ is then compact, so $(\widetilde{G}_j, \widetilde{G}_j \cap K)$ is a Riemannian symmetric pair of noncompact type whose corresponding symmetric space is $B_j$. It follows that $\widetilde{G}_j \cap K$ is connected and the center of $\widetilde{G}_j$ is finite and contained in $\widetilde{G}_j \cap K$. Therefore, we have a finite covering $\widetilde{G}_j \twoheadrightarrow \widetilde{G}_j/Z(\widetilde{G}_j) \cong I^0(B_j)$, so the action $G_j \curvearrowright B_j$ is ’almost’ effective and $\widetilde{\mathfrak {{g}}}_j \cong \mathfrak {{i}}(B_j)$. This identification of $\mathfrak {{i}}(B_j)$ with an ideal in $\mathfrak {{g}}_j$ is precisely what we did in the canonical extension method. Note also that since we have $\mathfrak {{m}}_j = Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) \oplus \widetilde{\mathfrak {{g}}}_j$ and the first summand is precisely the kernel of the representation of $\mathfrak {{m}}_j$ on $\mathfrak {{b}}_j$, we can take the Lie algebra $\mathfrak {{h}}_j^\Uplambda$ in the canonical extension to be not

but

, where $\widehat{\mathfrak {{h}}}_j \subseteq \mathfrak {{m}}_j$ is any Lie subalgebra the image of whose projection to $\widetilde{\mathfrak {{g}}}_j$ along $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j)$ is $\mathfrak {{h}}_j$ and whose corresponding connected Lie subgroup of $M_j$ is closed (e.g., we can take $\widehat{\mathfrak {{h}}}_j = Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_j) \oplus \mathfrak {{h}}_j$). The resulting action on M would still have the same orbits. This observation will let us describe canonically extended actions more neatly.

2.5 Rank-one case

Here, we prove a small technical result pertaining to cohomogeneity-one actions on rank-1 noncompact symmetric spaces, i.e., on hyperbolic spaces over finite-dimensional real normed division algebras. It will allow us to describe actions on rank-2 spaces canonically extended from their boundary components in a relatively nice and uniform way. Note that if M is a noncompact symmetric space of rank 1, we have $\Uplambda = \left\{ {\upalpha _1} \right\}$ and $\mathfrak {{m}}_1 = \mathfrak {{k}}_1 = \mathfrak {{k}}_0$, so $M_1$ is compact and every subspace of $\mathfrak {{n}}_1^1$ is automatically admissible.

Proposition 2.5

Let M be a symmetric space of noncompact type and rank 1.

• (1) If $\mathfrak {{v}} \subseteq \mathfrak {{n}}_1^1 = \mathfrak {{g}}_{\upalpha _1}$ is a protohomogeneous subspace, then $\mathfrak {{h}}_\mathfrak {{v}} = N_\mathfrak {{k}}(\mathfrak {{a}} \oplus \mathfrak {{n}}_{1,\mathfrak {{v}}}) \oplus \mathfrak {{a}} \oplus \mathfrak {{n}}_{1,\mathfrak {{v}}}$ is a Lie subalgebra of $\mathfrak {{g}}$ whose corresponding connected Lie subgroup is closed, acts on M with cohomogeneity one, and in fact has the same orbits as $H_{1, \mathfrak {{v}}}$.

• (2) Let $V \subseteq \mathfrak {{p}}$ be a Lie triple system corresponding to a totally geodesic orbit S of some cohomogeneity-one action on M, assume $\mathfrak {{a}} \subseteq V$, and let $\mathfrak {{h}}$ stand for the preimage of V under the isomorphism $\mathfrak {{a}} \oplus \mathfrak {{n}} {\mathop {\sim }\limits ^{\rightarrow }}\mathfrak {{p}}$. Then, $N_\mathfrak {{k}}(\mathfrak {{h}}) \oplus \mathfrak {{h}}$ is a Lie subalgebra of $\mathfrak {{g}}$ whose corresponding connected Lie subgroup is closed, acts on M with cohomogeneity one, and has S as its orbit.

Proof

For the first part, note that $\mathfrak {{h}}_{1, \mathfrak {{v}}} = N_{\mathfrak {{k}}_0}(\mathfrak {{n}}_{1,\mathfrak {{v}}}) \oplus \mathfrak {{a}} \oplus \mathfrak {{n}}_{1,\mathfrak {{v}}}$, so it is contained in $\mathfrak {{h}}_\mathfrak {{v}}$ (which is clearly a Lie subalgebra) and has the same projection in $\mathfrak {{p}}$ as $\mathfrak {{h}}_\mathfrak {{v}}$. Since we already know that $H_{1, \mathfrak {{v}}}$ acts with cohomogeneity one, it follows that its orbits coincide with those of the Lie subgroup corresponding to $\mathfrak {{h}}_\mathfrak {{v}}$. The latter is easily seen to be $N^0_K(A N_{1,\mathfrak {{v}}}) A N_{1,\mathfrak {{v}}}$, which is closed by the global Iwasawa decomposition.

For the second part, our assumption implies $\mathfrak {{a}} \subseteq \mathfrak {{h}}$, so we can write $\mathfrak {{h}} = \mathfrak {{a}} \oplus \mathfrak {{h}}_n$, where $\mathfrak {{h}}_n = \mathfrak {{h}} \cap \mathfrak {{n}}$. Assume for a moment that we know that $\mathfrak {{h}}$ is a Lie subalgebra of $\mathfrak {{a}} \oplus \mathfrak {{n}}$ and write H for its connected Lie subgroup. Clearly, $\mathfrak {{h}}_n$ is then also a Lie subalgebra. Its corresponding Lie subgroup $H_n$ of N is closed because the exponential map of N is a diffeomorphism. We see that $H = AH_n$ and hence it is also a closed subgroup. Under our assumption, $N_\mathfrak {{k}}(\mathfrak {{h}}) \oplus \mathfrak {{h}}$ is trivially a Lie subalgebra of $\mathfrak {{g}}$ and its Lie subgroup is $N_K^0(H)H$, which is closed. Finally, $N_\mathfrak {{k}}(\mathfrak {{h}}) = N_\mathfrak {{k}}(V)$, and the fact that S is an orbit of some cohomogeneity-one action implies that the slice representation of $N^0_K(H)H$ at o is of cohomogeneity one, hence so is the action of $N^0_K(H)H$ on M.

It follows that we only need to prove that $\mathfrak {{h}}$ is a Lie subalgebra of $\mathfrak {{a}} \oplus \mathfrak {{n}}$ and its corresponding Lie subgroup H has S as an orbit (a priori, we only know that its orbit through o is of the same dimension and touches S at o tangentially), which we are going to do in a rather ad hoc fashion. Consider first the special case when our totally geodesic orbit is

$$\begin{aligned} S' = {\left\{ \begin{array}{ll} \mathbb {R}H^k \; (0 \leqslant k \leqslant n-1), \quad &{}\text {if} \; M = \mathbb {R}H^n, \\ \mathbb {C}H^k \; (0 \leqslant k \leqslant n-1) \; \text {or} \; \mathbb {R}H^n, \quad &{}\text {if} \; M = \mathbb {C}H^n, \\ \mathbb {H}H^k \; (0 \leqslant k \leqslant n-1) \; \text {or} \; \mathbb {C}H^n, \quad &{}\text {if} \; M = \mathbb {H}H^n, \\ \mathbb {O}H^1 \; \text {or} \; \mathbb {H}H^2, \quad &{}\text {if} \; M = \mathbb {O}H^2, \end{array}\right. } \end{aligned}$$

(2.1)

where all these totally geodesic submanifolds are embedded into the corresponding hyperbolic space in a standard way. We are going to write $\mathfrak {{h}}'$ and $\mathfrak {{h}}'_n$ instead of $\mathfrak {{h}}$ and $\mathfrak {{h}}_n$ here to avoid notational confusion later on. Note that $S'$ is an orbit of the Lie subgroup

$$\begin{aligned} G' = {\left\{ \begin{array}{ll} \textrm{SO}^0(k,1) \; (0 \leqslant k \leqslant n-1), \quad &{}\text {if} \; M = \mathbb {R}H^n, \\ \textrm{SU}(k,1) \; (0 \leqslant k \leqslant n-1) \; \text {or} \; \textrm{SO}^0(n,1), \quad &{}\text {if} \; M = \mathbb {C}H^n, \\ \textrm{Sp}(k,1) \; (0 \leqslant k \leqslant n-1) \; \text {or} \; \textrm{SU}(n,1), \quad &{}\text {if} \; M = \mathbb {H}H^n, \\ \textrm{SO}^0(8,1) \; \text {or} \; \textrm{Sp}(2,1), \quad &{}\text {if} \; M = \mathbb {O}H^2, \end{array}\right. } \end{aligned}$$

also embedded into the corresponding G in a standard way. But $G'$ is $\Uptheta$-stable and V coincides with $\mathfrak {{p}}' = \mathfrak {{g}}' \cap \mathfrak {{p}}$, where $\mathfrak {{g}}' = \textrm{Lie} (G')$. Moreover, $\mathfrak {{a}} \subseteq \mathfrak {{p}}'$ is a maximal abelian subspace, while $\Upsigma ' \cap \Upsigma ^+$ is a choice of positive restricted roots for $\mathfrak {{g}}'$. It all implies that $\mathfrak {{h}}' = \mathfrak {{a}} \oplus \mathfrak {{h}}'_n$ coincides with $\mathfrak {{a}} \oplus \mathfrak {{n}}'$, where $\mathfrak {{n}}'$ is the sum of positive root subspaces of $\mathfrak {{g}}'$. But the latter sum is a Lie subalgebra of $\mathfrak {{a}} \oplus \mathfrak {{n}}$. What is more, since $S'$ is an orbit of $G'$, it is also an orbit of $H' = AH'_n = AN' \subseteq G'$, so $S'$ chosen in such a special way satisfies the assertions of (2).

Totally geodesic submanifolds in (2.1) exhaust the list of totally geodesic orbits of cohomogeneity-one actions on hyperbolic spaces up to isometric congruence (see [2, Theorem 1]). In other words, given an arbitrary S as in part (2) of the proposition, it is congruent to some $S'$ as in (2.1), for which the assertions of (2) are satisfied. Now, $S'$ is a symmetric space of rank 1, so $I^0(S')$ acts homogeneously and isotropically on $S'$. Moreover, every element of $I^0(S')$ can be extended to an isometry of M. This is because $S'$ is a semisimple and totally geodesic complete submanifold of M (see the argument in [7, Proposition 3.2] or [12, Theorem V.4.1(i)]). It follows that we can take our congruence $k \in I(M)$ between S and $S'$ to preserve o and $\mathfrak {{a}}$ and even fix $\mathfrak {{a}}$ pointwise, i.e., to lie in $Z_{\widetilde{K}}(\mathfrak {{a}})$, where $\widetilde{K}$ is the isotropy subgroup of I(M) at o. But such k preserves both $\mathfrak {{a}} \oplus \mathfrak {{n}}$ and $\mathfrak {{p}}$ and is a Lie algebra automorphism of the former. Since it maps V onto $V'$, it must map $\mathfrak {{h}}$ onto $\mathfrak {{h}}'$, so $\mathfrak {{h}}$ is a Lie subalgebra. Moreover, $S'$ is an orbit of $H'$, which means that S must be an orbit of H, which completes the proof.

3 Classification of cohomogeneity-one actions on $\textrm{SL}(3, \mathbb {H})/\textrm{Sp}(3)$

In this section we classify, up to orbit equivalence, cohomogeneity-one actions on the noncompact symmetric space $\textrm{SL}(3, \mathbb {H})/\textrm{Sp}(3)$.

The symmetric space $M = \textrm{SL}(3, \mathbb {H})/\textrm{Sp}(3)$ is irreducible of rank 2 and dimension 14, and its root system is $A_2$. It is the quaternionic analog of the $A_2$-type spaces $\textrm{SL}(3, \mathbb {R})/\textrm{SO}(3)$ and $\textrm{SL}(3, \mathbb {C})/\textrm{SU}(3)$, the explicit classification of cohomogeneity-one actions on which has already been obtained in [5, 9], respectively. Let $\Uplambda = \left\{ {\upalpha _1, \upalpha _2} \right\}$. Then, $\Upsigma ^+ = \left\{ {\upalpha _1, \upalpha _2, \upalpha _1 + \upalpha _2} \right\}$ and all the roots have multiplicity 4. The Lie subalgebra $\mathfrak {{k}}_0$ is isomorphic to $\mathfrak {{sp}}(1)^{\oplus 3}$.

Theorem 3.1

Let $H \subset G = \textrm{SL}(3, \mathbb {H})$ be a closed connected subgroup acting on M with cohomogeneity one. Then, its action is orbit equivalent to exactly one of the following:

• (1) The action of the connected Lie subgroup $H_\ell$ of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_\ell = (\mathfrak {{a}} \ominus \ell ) \oplus \mathfrak {{n}}, \end{aligned}$$

  where $\ell$ is a one-dimensional linear subspace of $\mathfrak {{a}}$. Its orbits are all isometrically congruent to each other and form a Riemannian foliation on M.

• (2) The action of the connected Lie subgroup $H_1$ of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_1 = \mathfrak {{a}} \oplus (\mathfrak {{n}} \ominus \ell _1), \end{aligned}$$

  where $\ell _1$ is any one-dimensional linear subspace of $\mathfrak {{g}}_{\upalpha _1}$. Its orbits form a Riemannian foliation on M and there is exactly one minimal orbit.

• (3) The action of the subgroup $L_1 = L_1^0$ of G. It has a 6-dimensional totally geodesic singular orbit $F_1 \simeq \mathbb {R}H^5 \times \mathbb {E}$.

• (4) The action of the subgroup $\textrm{SL}(3,\mathbb {C})$ of G embedded in a standard way. It has an 8-dimensional totally geodesic singular orbit isometric to $\textrm{SL}(3, \mathbb {C})/\textrm{SU}(3)$.

• (5) The action of the connected Lie subgroup $H^\Uplambda _{1,k}, k \in \left\{ {0,1,2,3} \right\}$, of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_{1,k}^\Uplambda = N_{\mathfrak {{k}}_1}(\mathfrak {{w}}) \oplus \mathfrak {{w}} \oplus \mathfrak {{a}}_1 \oplus \mathfrak {{n}}_1, \end{aligned}$$

  where $\mathfrak {{w}}$ is a k-dimensional subspace of $\mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2}$ containing $\mathfrak {{a}}^1$ (unless $k = 0$). Here, $N_{\mathfrak {{k}}_1}(\mathfrak {{w}}) \simeq \mathfrak {{sp}}(1) \oplus \mathfrak {{so}}(5-k) \oplus \mathfrak {{so}}(k)$, where the first summand is $Z_{\mathfrak {{k}}_0}(\mathfrak {{b}}_1)$ and the rest is the normalizer of $\mathfrak {{w}}$ in the Lie algebra $\widetilde{\mathfrak {{g}}}_1 \cap \mathfrak {{k}} \simeq \mathfrak {{so}}(5)$ of the isotropy subgroup of the isometry group of the boundary component $B_1 \simeq \mathbb {R}H^5$. This action has a minimal.⁵ singular orbit of codimension $5-k$ and can be obtained by canonical extension of the cohomogeneity-one action on $B_1$ with $\mathbb {R}H^k$ as a totally geodesic singular orbit.

Proof

We consider different cases of Theorem 2.1. If the orbits of H form a foliation, we get the actions in (1) and (2). The actions in (3) and (4) are the only ones that have a totally geodesic singular orbit according to [7]. Here, we need to check that the slice representations of $L_1$ and $\textrm{SL}(3, \mathbb {C})$ are transitive on spheres to make sure that these two groups do indeed act on M with cohomogeneity one. It is not hard to show that $K_1 \simeq \textrm{Sp}(2) \times \textrm{Sp}(1)$ (in particular, $L_1$ is connected) and that the slice representation of $L_1$ at o is equivalent to the standard representation of $\textrm{Sp}(2) \times \textrm{Sp}(1)$ on $\mathbb {H}^2$, which is transitive on the unit sphere. Modulo $\mathbb {{Z}}/2\mathbb {{Z}}$, this is the isotropy representation of the quaternionic hyperbolic plane, which should not be surprising because $F_1^\perp \simeq \mathbb {H}H^2$. What for $\textrm{SL}(3, \mathbb {C})$, its stabilizer at o is $\textrm{SU}(3)$ and its slice representation at o can be easily computed to be of cohomogeneity one. In fact, it is equivalent to the tautological representation $\mathbb {C}^3$ of $\textrm{SU}(3)$, which is just evidence of the fact that $(\textrm{SL}(3,\mathbb {C}) \cdot o)^\perp \simeq \mathbb {C}H^3$.

Now we determine the actions induced by canonical extension. Since the root system of M is $A_2$, the boundary components $B_1$ and $B_2$ are isometrically congruent (see [19, Proposition 28]), so it suffices to consider only actions arising from $B_1$. The boundary component is isometric to $\textrm{SL}(2, \mathbb {H})/\textrm{Sp}(2) \simeq \textrm{SO}^0(5,1)/\textrm{SO}(5) \cong \mathbb {R}H^5$. There are, up to strong orbit equivalence, exactly 4 cohomogeneity-one actions with a singular orbit on $\mathbb {R}H^5$. According to Proposition 2.5 and the observation at the end of Sect. 2.4, the actions in (5) are the canonical extensions of these 4 actions on $B_1$. None of them are congruent to the actions in (3) and (4) because their singular orbits have different dimensions.

We are left to investigate actions arising from the nilpotent construction. Again, we need only consider one of the indices $\left\{ {1,2} \right\}$, but this time we choose $j = 2$ (this choice is strategical as will become evident later on). We have $\mathfrak {{n}}_2 = \mathfrak {{n}}_2^1 = \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \simeq \mathbb {H}^2$ and $K_2 \simeq \textrm{Sp}(2) \times \textrm{Sp}(1)$. The representation of $K_2$ on $\mathfrak {{n}}_2$ is equivalent to the standard representation of $\textrm{Sp}(2) \times \textrm{Sp}(1)$ on $\mathbb {H}^2$:

$$\begin{aligned} (A, q) \cdot \begin{bmatrix} x \\ y \end{bmatrix} = A \begin{bmatrix} xq^{-1} \\ yq^{-1} \end{bmatrix}, \end{aligned}$$

where A is a quaternionic-unitary $2 \times 2$ matrix.

We first filter out subspaces of $\mathfrak {{n}}_2$ that are not protohomogeneous. This problem was recently solved in greater generality for the standard action of $\textrm{Sp}(n)\textrm{Sp}(1)$ on $\mathbb {{H}}^n$ by Díaz-Ramos, Domínguez-Vázquez, and Rodríguez-Vázquez in [11]. The authors explicitly classified protohomogeneous subspaces of $\mathbb {{H}}^n$ up to the action of $\textrm{Sp}(n)\textrm{Sp}(1)$ in terms of their quaternionic Kähler angle—the quaternionic analog of the Kähler angle of a real subspace of a Hilbert space first introduced by Berndt and Brück in [2]. We recall its definition via the following lemma, which was essentially proven in [2]:

Lemma 3.2

Let $\mathfrak {{v}} \subseteq \mathbb {{H}}^n$ be a real subspace, and let $v \in \mathfrak {{v}}$ be a nonzero vector. There exists a canonical basis $(J_1, J_2, J_3)$ of the quaternionic structure $\mathcal {J}$ on $\mathbb {{H}}^n$ and a uniquely defined triple $(\upvarphi _1, \upvarphi _2, \upvarphi _3) \in [0, \frac{\uppi }{2}]^3$ such that:

• (i) $\upvarphi _i$ is the Kähler angle of v with respect to $J_i$ (that is, the angle between $J_i v$ and $\mathfrak {{v}}$) for each $i \in \left\{ {1,2,3} \right\}$.

• (ii) $\langle {\textrm{pr}_\mathfrak {{v}} \circ J_i (v)} | {\textrm{pr}_\mathfrak {{v}} \circ J_j (v)} \rangle = 0$ for each $i \ne j$, where $\textrm{pr}_\mathfrak {{v}}$ is the orthogonal projector onto $\mathfrak {{v}}$ in $\mathbb {{H}}^n$.

• (iii) $\upvarphi _1 \leqslant \upvarphi _2 \leqslant \upvarphi _3$.

• (iv) $\upvarphi _1$ is minimal and $\upvarphi _3$ is maximal among the Kähler angles of v with respect to all nonzero elements of $\mathcal {J}$.

In fact, $(J_1, J_2, J_3)$ is a basis for $\mathcal {J}$ that diagonalizes the symmetric bilinear form

$$\begin{aligned} \mathcal {J} \times \mathcal {J} \rightarrow \mathbb {R}, \quad (J, J') \mapsto \langle {\textrm{pr}_\mathfrak {{v}} \circ J (v)} | {\textrm{pr}_\mathfrak {{v}} \circ J' (v)} \rangle \end{aligned}$$

with $\cos ^2(\upvarphi _i)||v||^2, \; i \in \left\{ {1,2,3} \right\} ,$ on the diagonal. Such a triple $(\upvarphi _1, \upvarphi _2, \upvarphi _3)$ is called the quaternionic Kähler angle of $\mathfrak {{v}}$ with respect to v.

The quaternionic Kähler angle of $\mathfrak {{v}}$ in general depends on v. However, we have the following fact (see [11, Lemma 2.4] for a proof):

Lemma 3.3

If $\mathfrak {{v}}$ is a protohomogeneous subspace of $\mathbb {{H}}^n$, then it has constant quaternionic Kähler angle, i.e., it does not depend on the choice of $v \in \mathfrak {{v}} \ \left\{ {0} \right\}$. In that case, we call $(\upvarphi _1, \upvarphi _2, \upvarphi _3)$ simply the quaternionic Kähler angle of $\mathfrak {{v}}$.

The classification of protohomogeneous subspaces of $\mathbb {{H}}^n$ obtained in [11, Theorem A] shows that a k-dimensional protohomogeneous subspace $\mathfrak {{v}} \subseteq \mathbb {{H}}^n$ is ‘almost always’ determined by its quaternionic Kähler angle up to $\textrm{Sp}(n)\textrm{Sp}(1)$: the only exception is when $k \leqslant n$ is congruent to 0 or $3 \; (\textrm{mod} 4)$, in which case there may be (at most) two noncongruent protohomogeneous subspaces of dimension k with the same quaternionic Kähler angle. Since in our case $n=2$, this uncertainty does not concern us. From the table in the aforementioned classification theorem, we see that there are the following possibilities for $\mathfrak {{v}} \subseteq \mathfrak {{n}}_2$:

• Case 1: $\dim \mathfrak {{v}} = 2$ and the quaternionic Kähler angle of $\mathfrak {{v}}$ is $(\upvarphi , \frac{\uppi }{2}, \frac{\uppi }{2})$ for some $\upvarphi \in [0, \frac{\uppi }{2}]$.

• Case 2: $\dim \mathfrak {{v}} = 3$ and the quaternionic Kähler angle of $\mathfrak {{v}}$ is $(\upvarphi , \upvarphi , \frac{\uppi }{2})$ for some $\upvarphi \in \left\{ {0, \frac{\uppi }{3}} \right\}$.

• Case 3: $\dim \mathfrak {{v}} = 4$ and the quaternionic Kähler angle of $\mathfrak {{v}}$ is $(0, \upvarphi , \upvarphi )$ for some $\upvarphi \in [0, \frac{\uppi }{2}]$.

We start our investigation with case 1. First, assume that $\upvarphi = 0$. In that case, $\mathfrak {{v}}$ is a totally complex subspace of $\mathfrak {{n}}_2$ and we can take it to be $\mathbb {C}j E_{23}$ so that $\mathfrak {{n}}_{2, \mathfrak {{v}}} = \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathbb {C}E_{23}$. Now we need to check whether $\mathfrak {{v}}$ is admissible. To this end, we have to compute $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$ and see if its projection in $\mathfrak {{p}}$ is the whole $\mathfrak {{b}}_2$ or a proper subspace of it. Observe that

$$\begin{aligned} \mathfrak {{m}}_2 = \left\{ \left. \begin{pmatrix} p_{11} &{} p_{12} &{} 0 \\ p_{21} &{} p_{22} &{} 0 \\ 0 &{} 0 &{} q \end{pmatrix} \right| \textrm{Re}(p_{11}) + \textrm{Re}(p_{22}) = 0, \textrm{Re}(q) = 0 \right\} \simeq \mathfrak {{sl}}(2, \mathbb {H}) \oplus \mathfrak {{sp}}(1). \end{aligned}$$

By commuting such matrices with $jE_{23}$ and $kE_{23}$, one readily sees that

$$\begin{aligned} N_{\mathfrak {{m}}_2}(\mathfrak {{v}}) = \left\{ \left. \begin{pmatrix} p_{11} &{} 0 &{} 0 \\ p_{21} &{} p_{22} &{} 0 \\ 0 &{} 0 &{} q \end{pmatrix} \right| \textrm{Re}(p_{11}) + \textrm{Re}(p_{22}) = 0, p_{22} \in \mathbb {C}, q \in \mathbb {R}i \right\} . \end{aligned}$$

The projection of this in $\mathfrak {{p}}$ is clearly equal to $\mathfrak {{b}}_2$, so $\mathfrak {{v}} = \mathbb {C}j E_{23} \subseteq \mathfrak {{n}}_2$ produces a cohomogeneity-one action. The corresponding Lie algebra is

$$\begin{aligned} \mathfrak {{h}}_{2, \mathfrak {{v}}} = (\mathbb {R}i E_{11} \oplus (\textrm{Im}\mathbb {{H}}) E_{22} \oplus \mathbb {R}i E_{33}) \oplus \mathfrak {{a}} \oplus \mathfrak {{g}}_{\upalpha _1} \oplus \mathbb {C}E_{23} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}, \end{aligned}$$

(3.1)

where the first summand in the parentheses is $\mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{k}} = \mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{k}}_0$. We claim that this action is orbit equivalent to the canonical extension of a cohomogeneity-one action on $B_1$ with a singular orbit of codimension 2. Indeed, in the notation of (5), we take $k=3$ and $\mathfrak {{w}} = \mathfrak {{a}}^1 \oplus \mathbb {C}E_{23}$. Then, we have:

$$\begin{aligned} \mathfrak {{h}}_{1, 3}^\Uplambda = N_{\mathfrak {{k}}_1}(\mathfrak {{w}}) \oplus \mathfrak {{a}} \oplus \mathfrak {{g}}_{\upalpha _1} \oplus \mathbb {C}E_{23} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}. \end{aligned}$$

(3.2)

Observe that $\mathfrak {{h}}_{2, \mathfrak {{v}}}$ and $\mathfrak {{h}}_{1, 3}^\Uplambda$ are Lie subalgebras of the parabolic subalgebra $\mathfrak {{q}}_1$, and they both sit nicely within the Langlands decomposition $\mathfrak {{q}}_1 = \mathfrak {{m}}_1 \oplus \mathfrak {{a}}_1 \oplus \mathfrak {{n}}_1$: $\mathfrak {{h}}_{2, \mathfrak {{v}}} = (\mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{m}}_1) \oplus (\mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{a}}_1) \oplus (\mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{n}}_1)$, and the same for $\mathfrak {{h}}_{1, 3}^\Uplambda$. It follows that, in terms of the horospherical decomposition $M = B_1 \times A_1 \times N_1$, the singular orbits of the actions of $H_{2, \mathfrak {{v}}}$ and $H_{1,3}^\Uplambda$ are $((H_{2, \mathfrak {{v}}} \cap M_1) \cdot o) \times A_1 \times N_1$ and $((H_{1,3}^\Uplambda \cap M_1) \cdot o) \times A_1 \times N_1$, respectively. The first factors here are the singular orbits of the cohomogeneity-one actions of $H_{2, \mathfrak {{v}}} \cap M_1$ and $H_{1,3}^\Uplambda \cap M_1$ on $B_1$, respectively. Looking at the decompositions (3.1) and (3.2), it is clear that these singular orbits coincide, since they both correspond to the Lie triple system $\mathfrak {{a}}^1 \oplus \left\{ {\uplambda E_{23} + \bar{\uplambda }E_{32} \mid \uplambda \in \mathbb {C}} \right\}$. Since the singular orbits of $H_{2, \mathfrak {{v}}}$ and $H_{1,3}^\Uplambda$ coincide, these groups have the same orbits, for all the other orbits are just equidistant tubes around the singular one.

Next, assume that $\upvarphi = \frac{\uppi }{2}$. Then, $\mathfrak {{v}}$ is a totally real subspace of $\mathfrak {{n}}_2$, so, acting by $K_2$ if necessary, we may assume $\mathfrak {{v}} = \mathbb {R}E_{13} \oplus \mathbb {R}E_{23}$. Simple computations reveal that a matrix $X \in \mathfrak {{m}}_2$ normalizing $\mathfrak {{v}}$ must have $p_{12}, p_{21} \in \mathbb {R}$, which means that the image of the projection of $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ will be a proper subspace of $\mathfrak {{b}}_2$, so such $\mathfrak {{v}}$ is not an admissible subspace.

We are left to consider the case $\upvarphi \in (0, \frac{\uppi }{2})$. Here $\mathfrak {{v}}$ is a subspace of constant Kähler angle $\upvarphi$ inside a totally complex subspace of $\mathfrak {{n}}_2$ of real dimension 4. Without loss of generality, we choose the latter to be $\mathbb {C}E_{13} \oplus \mathbb {C}E_{23}$. Then, we can take $\mathfrak {{v}}$ to be the real span of $E_{23}$ and $i \cos \upvarphi E_{23} + i \sin \upvarphi E_{13}$ (see [2, Proposition 7]). By commuting elements of $\mathfrak {{m}}_2$ with these two vectors and solving simple systems of linear equations, one gets, among other things, that $\textrm{Re}(p_{12}) = 0$ and $\textrm{Re}(p_{21}) = 2 \textrm{Re}(p_{11}) \cot \upvarphi$. It implies that the image of the projection of $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ is contained in $\mathfrak {{b}}_2 \cap \left\{ {\textrm{Re}(p_{11}) \cot \upvarphi - \textrm{Re}(p_{12}) = 0} \right\}$, which is a linear hyperplane in $\mathfrak {{b}}_2$. Therefore, this $\mathfrak {{v}}$ is not admissible either.

In the second case, consider $\mathfrak {{v}}$ with a quaternionic Kähler angle $(0, 0, \frac{\uppi }{2})$ first. Such $\mathfrak {{v}}$ can be described as $(\textrm{Im}\mathbb {{H}}) v$ for some nonzero $v \in \mathfrak {{n}}_2$. We may assume without loss of generality that $v = E_{23}$. One then easily computes:

$$\begin{aligned} N_{\mathfrak {{m}}_2}(\mathfrak {{v}}) = \left\{ \left. \begin{pmatrix} p_{11} &{} 0 &{} 0 \\ p_{21} &{} p_{22} &{} 0 \\ 0 &{} 0 &{} q \end{pmatrix} \right| \textrm{Re}(p_{11}) + \textrm{Re}(p_{22}) = 0, q = \textrm{Im}(p_{22}) \right\} . \end{aligned}$$

The image of the projection of that to $\mathfrak {{p}}$ is the whole of $\mathfrak {{b}}_2$, so this $\mathfrak {{v}}$ is admissible. We have:

$$\begin{aligned} \mathfrak {{h}}_{2, \mathfrak {{v}}} = ((\textrm{Im}\mathbb {{H}}) E_{11} \oplus (\textrm{Im}\mathbb {{H}})(E_{22} + E_{33})) \oplus \mathfrak {{a}} \oplus \mathfrak {{g}}_{\upalpha _1} \oplus \mathbb {R}E_{23} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}, \end{aligned}$$

where the first summand in the parentheses is $\mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{k}} = \mathfrak {{h}}_{2, \mathfrak {{v}}} \cap \mathfrak {{k}}_0 \simeq \mathfrak {{sp}}(1) \oplus \mathfrak {{sp}}(1)$. In a similar vein to what we did in case 1 with $\upvarphi = 0$, one can show that the orbits of $H_{2, \mathfrak {{v}}}$ coincide with the orbits of $H_{1,2}^\Uplambda$ if we take $\mathfrak {{w}} = \mathfrak {{a}}^1 \oplus \mathbb {R}E_{23}$. Therefore, this $\mathfrak {{v}}$ does not produce a new action.

Now let $\mathfrak {{v}}$ be of quaternionic Kähler angle $(\frac{\uppi }{3}, \frac{\uppi }{3}, \frac{\uppi }{2})$. It follows from Proposition 5.3 and Remark 5.4 in [11] that $\mathfrak {{v}} = \textrm{span} \left\{ {E_{23}, iE_{23} + i\sqrt{3}E_{13}, jE_{23} - j\sqrt{3}E_{13}} \right\}$ does the trick. Simple calculations show that an element of $\mathfrak {{m}}_2$ normalizing $\mathfrak {{v}}$ must have $\textrm{Re}(p_{11}) = \textrm{Re}(p_{22}) = 0$, so the image of the projection of $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ will be a proper subspace of $\mathfrak {{b}}_2$, hence this $\mathfrak {{v}}$ is not admissible.

Finally, assume $\mathfrak {{v}}$ is 4-dimensional. First, let its quaternionic Kähler angle be (0, 0, 0), that is, let it be a quaternionic line in $\mathfrak {{m}}_2$. Without loss of generality, we choose $\mathfrak {{v}} = \mathbb {{H}}E_{23} = \mathfrak {{g}}_{\upalpha _2}$. One immediately sees that $N_{\mathfrak {{m}}_2}(\mathfrak {{v}}) = \mathfrak {{m}}_2 \ominus \mathfrak {{g}}_{\upalpha _1}$, whose projection in $\mathfrak {{p}}$ is the whole $\mathfrak {{b}}_2$, so this $\mathfrak {{v}}$ is admissible. We have:

$$\begin{aligned} \mathfrak {{h}}_{2, \mathfrak {{v}}} = \mathfrak {{g}}_0 \oplus \mathfrak {{g}}_{\upalpha _1} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}. \end{aligned}$$

But this Lie subalgebra coincides with $\mathfrak {{h}}_{1,1}^\Uplambda$ if we take $\mathfrak {{w}} = \mathfrak {{a}}^1$. Consequently, $\mathfrak {{v}}$ does not give a new action.

Now let the quaternionic Kähler angle of $\mathfrak {{v}}$ be $(0, \frac{\uppi }{2}, \frac{\uppi }{2})$. Then, $\mathfrak {{v}}$ is a totally complex subspace, hence we may assume $\mathfrak {{v}} = \mathbb {C}E_{13} \oplus \mathbb {C}E_{23}$. Any element of $\mathfrak {{m}}_2$ normalizing such a subspace must have $p_{12}, p_{21} \in \mathbb {C}$, so the image of the projection of $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$ to $\mathfrak {{p}}$ is smaller than $\mathfrak {{b}}_2$ and $\mathfrak {{v}}$ is not admissible.

Finally, let $\mathfrak {{v}}$ be of quaternionic Kähler angle $(0, \upvarphi , \upvarphi ), \upvarphi \in (0, \frac{\uppi }{2})$. According to [2] (see the discussion before Theorem 5 there), we can take $\mathfrak {{v}}$ to be spanned by $E_{23}, iE_{23}, j\cos \upvarphi E_{23} + j\sin \upvarphi E_{13},$ and $k\cos \upvarphi E_{23} + k\sin \upvarphi E_{13}$. In a similar fashion to what we had in case 1, by solving the system of linear equations defining $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$, one gets—among other things—the same two linear dependencies $\textrm{Re}(p_{12}) = 0$ and $\textrm{Re}(p_{21}) = 2 \textrm{Re}(p_{11}) \cot \upvarphi$. Therefore, the projection of $N_{\mathfrak {{m}}_2}(\mathfrak {{v}})$ in $\mathfrak {{p}}$ is yet again contained in $\mathfrak {{b}}_2 \cap \left\{ {\textrm{Re}(p_{11}) \cot \upvarphi - \textrm{Re}(p_{12}) = 0} \right\}$ and $\mathfrak {{v}}$ fails to be admissible. This finishes the proof of Theorem 3.1.

4 Classification of cohomogeneity-one actions on $\textrm{SO}(5, \mathbb {C})/\textrm{SO}(5)$

In this section we classify, up to orbit equivalence, cohomogeneity-one actions on the noncompact dual of the compact Lie group $\textrm{Spin}(5)$.

The symmetric space $\textrm{SO}(5, \mathbb {C})/\textrm{SO}(5)$ is irreducible of rank 2 and dimension 10. Its root system is $B_2$ and can be identified with the root system of the complex simple Lie algebra $\mathfrak {{so}}(5, \mathbb {C})$. Consequently, all the root spaces have complex dimension 1. Let $\Uplambda = \left\{ {\upalpha _1, \upalpha _2} \right\}$, where $\upalpha _1$ is the longest root. Then, $\Upsigma ^+ = \left\{ {\upalpha _1, \upalpha _2, \upalpha _1 + \upalpha _2, \upalpha _1 + 2\upalpha _2} \right\}$. We also have $\mathfrak {{k}}_0 = i\mathfrak {{a}} = \mathbb {R}i H_{\upalpha _1} \oplus ^\perp \mathbb {R}i H^2 = \mathbb {R}i H^1 \oplus ^\perp \mathbb {R}i H_{\upalpha _2} \simeq \mathfrak {{u}}(1) \oplus \mathfrak {{u}}(1)$, where i the complex structure of $\mathfrak {{so}}(5,\mathbb {C})$. The fact that $\upalpha _1$ and $\upalpha _2$ have different lengths implies that the corresponding boundary components $B_1 \simeq \mathbb {R}H^3$ and $B_2 \simeq \mathbb {R}H^3$ have different sectional curvatures ($- ||\upalpha _1||^2$ and $-||\upalpha _2||^2$, respectively) and thus are not congruent (which reflects the asymmetry of the Dynkin diagram $B_2$).

Theorem 4.1

Let $H \subset G = \textrm{SO}(5,\mathbb {C})$ be a closed connected subgroup acting on M with cohomogeneity one. Then, its action is orbit equivalent to exactly one of the following:

• (1) The action of the connected Lie subgroup $H_\ell$ of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_\ell = (\mathfrak {{a}} \ominus \ell ) \oplus \mathfrak {{n}}, \end{aligned}$$

  where $\ell$ is a one-dimensional linear subspace of $\mathfrak {{a}}$. Its orbits are all isometrically congruent to each other and form a Riemannian foliation on M.

• (2) The action of the connected Lie subgroup $H_i, i \in \left\{ {1,2} \right\} ,$ of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_i = \mathfrak {{a}} \oplus (\mathfrak {{n}} \ominus \ell _i), \end{aligned}$$

  where $\ell _i$ is any one-dimensional linear subspace of $\mathfrak {{g}}_{\upalpha _i}$. Its orbits form a Riemannian foliation on M and there is exactly one minimal orbit.

• (3) The action of the subgroup $\textrm{SO}(4, \mathbb {C})$ of G embedded in a standard way. It has a 6-dimensional totally geodesic singular orbit isometric to $\textrm{SO}(4, \mathbb {C})/\textrm{SO}(4) \simeq \textrm{SL}(2, \mathbb {C})/\textrm{SU}(2) \times \textrm{SL}(2, \mathbb {C})/\textrm{SU}(2) \simeq \mathbb {R}H^3 \times \mathbb {R}H^3$.

• (4) The action of the connected Lie subgroup $H^\Uplambda _{j,0}, j \in \left\{ {1,2} \right\}$, of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_{j,0}^\Uplambda = \mathfrak {{k}}_j \oplus \mathfrak {{a}}_j \oplus \mathfrak {{n}}_j. \end{aligned}$$

  This action has a minimal singular orbit of codimension 3 and can be obtained by canonical extension of the cohomogeneity-one action on $B_j$ with a single point as a singular orbit.

• (5) The action of the connected Lie subgroup $H^\Uplambda _{j,1}, j \in \left\{ {1,2} \right\}$, of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_{j,1}^\Uplambda = \mathfrak {{k}}_0 \oplus \mathfrak {{a}} \oplus \mathfrak {{n}}_j, \end{aligned}$$

  This action has a minimal singular orbit of codimension 2 and can be obtained by canonical extension of the cohomogeneity-one action on $B_j$ with a geodesic as a singular orbit.

Proof

We consider different cases of Theorem 2.1. If the orbits of H form a foliation, we get the actions in (1) and (2). The action in (3) is the only one that has a totally geodesic singular orbit according to [7]. The stabilizer of $\textrm{SO}(4,\mathbb {C})$ at o is $\textrm{SO}(4)$, whose slice representation at o is equivalent to the tautological one (we have $(\textrm{SO}(4,\mathbb {C}) \cdot o)^\perp \simeq \mathbb {R}H^4$), hence $\textrm{SO}(4,\mathbb {C})$ does indeed act with cohomogeneity one. It is also worth noting that we can take

$$\begin{aligned} \mathfrak {{so}}(4, \mathbb {C}) =\mathfrak {{g}}_0 \oplus \bigoplus _{\begin{array}{c} \upalpha \in \Upsigma \\ \text {long root} \end{array}} \mathfrak {{g}}_{\upalpha }. \end{aligned}$$

The long roots in $B_2$ form a root system $A_1 \sqcup A_1$ sitting inside $B_2$, and the hyperbolic spaces in the decomposition $\textrm{SO}(4, \mathbb {C})/\textrm{SO}(4) \simeq \mathbb {R}H^3 \times \mathbb {R}H^3$ have the same curvature.

Now we determine the actions arising via the canonical extension method. Each boundary component $B_j$ is isometric to the real hyperbolic space $\mathbb {R}H^3$ (with different curvatures depending on j), which has precisely two cohomogeneity-one actions with a singular orbit up to strong orbit equivalence. One of them is the action of the (restricted) isotropy subgroup of $\mathbb {R}H^3$, which has a single point as a singular orbit and whose canonical extension is described in (4). The other one has a geodesic as a singular orbit and, by Proposition 2.5 and the observation made at the end of Sect. 2.4, is given by the connected Lie subgroup corresponding to $N_{\mathfrak {{k}}_j}(\mathfrak {{a}}^j) \oplus \mathfrak {{a}}^j = \mathfrak {{k}}_0 \oplus \mathfrak {{a}}^j$. Its canonical extension is given in (5). Observe that the actions in (4) and (5) are not orbit equivalent to each other because the normal spaces of their singular orbits are tangent to the corresponding boundary components and thus have different sectional curvatures.

Now we proceed to the main part, namely to the nilpotent construction. Since the Dynkin diagram $B_2$ is asymmetric, we have to consider two cases.

Nilpotent construction with $j=2$. In this case we have:

$$\begin{aligned} \mathfrak {{n}}_2^1&= \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \simeq \mathbb {C}^2, \\ \mathfrak {{l}}_2&= \mathfrak {{g}}_{-\upalpha _1} \oplus \mathfrak {{g}}_0 \oplus \mathfrak {{g}}_{\upalpha _1} = \mathfrak {{g}}_2 \oplus \mathfrak {{z}}_2 \oplus \mathfrak {{a}}_2 \simeq \mathfrak {{sl}}(2, \mathbb {C}) \oplus \mathfrak {{u}}(1) \oplus \mathfrak {{u}}(1) \simeq \mathfrak {{gl}}(2, \mathbb {C}), \\ \mathfrak {{k}}_2&= \mathfrak {{k}}_0 \oplus \mathfrak {{k}}_{\upalpha _1} = (\mathfrak {{g}}_2 \cap \mathfrak {{k}}) \oplus \mathfrak {{z}}_2 \simeq \mathfrak {{su}}(2) \oplus \mathfrak {{u}}(1) \simeq \mathfrak {{u}}(2). \end{aligned}$$

The adjoint representation of $\mathfrak {{g}}_2$ on $\mathfrak {{n}}_2^1$ is a nontrivial complex representation, hence it is equivalent to the irreducible representation of $\mathfrak {{sl}}(2, \mathbb {C})$ on $\mathbb {C}^2$. The adjoint representation of $\mathfrak {{a}}_2 \oplus \mathfrak {{z}}_2 = \mathbb {C}H^2$ on $\mathfrak {{n}}_2^1$ is the standard complex representation by scalars because $H^2$ acts on $\mathfrak {{n}}_2^1$ by multiplication by $\langle {\upalpha _2} , {H^2} \rangle = 1$. It follows that the adjoint representation of $\mathfrak {{l}}_2$ (respectively, $\mathfrak {{k}}_2$) on $\mathfrak {{n}}_2^1$ is equivalent to the tautological representation of $\mathfrak {{gl}}(2, \mathbb {C})$ (respectively, $\mathfrak {{u}}(2)$) on $\mathbb {C}^2$. Since $\mathfrak {{l}}_2$ is a $\uptheta$-stable Lie subalgebra of $\mathfrak {{g}}$, we may assume $\uptheta$ corresponds to the standard Cartan involution on $\mathfrak {{gl}}(2, \mathbb {C})$ (minus the adjoint of a matrix). Therefore, the problem of finding admissible and protohomogeneous subspaces of the $L^0_2$-module $\mathfrak {{n}}_2^1$ is equivalent to the analogous problem for the tautological representation of $\textrm{GL}(2, \mathbb {C})$.

Let $\mathfrak {{v}} \subseteq \mathfrak {{n}}_2^1$ be a linear subspace of dimension at least 2. According to Lemma 1 and Proposition 7 in [2], $\mathfrak {{v}}$ is protohomogeneous if and only if it has constant Kähler angle. In particular, it must be even-dimensional. If $\mathfrak {{v}} = \mathfrak {{n}}_2^1$, then it is trivially admissible and we have $\mathfrak {{h}}_{2, \mathfrak {{v}}} = \mathfrak {{l}}_2 \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2} = \mathfrak {{so}}(4, \mathbb {C}) \ominus \mathfrak {{g}}_{-\upalpha _1 - 2\upalpha _2}$. Since we know that $H_{2, \mathfrak {{v}}}$ acts with cohomogeneity one, it follows that it has the same orbits as $\textrm{SO}(4, \mathbb {C})$, so its action has already been taken into account in (3). Now suppose $\mathfrak {{v}}$ has real dimension 2. It was shown in [5, Theorem 6] that such a subspace is admissible if and only if its Kähler angle is zero, i.e., if it is a complex subspace. Up to the action of $K_2^0 \simeq \textrm{U}(2)$, we may assume $\mathfrak {{v}} = \mathfrak {{g}}_{\upalpha _2}$. We have

$$\begin{aligned} \mathfrak {{h}}_{2, \mathfrak {{v}}} = \mathfrak {{l}}_2 \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2} = \mathfrak {{g}}_{-\upalpha _1} \oplus \mathfrak {{g}}_0 \oplus \mathfrak {{n}}_1. \end{aligned}$$

If we look back at (5), we see that $\mathfrak {{h}}_{1,1}^\Uplambda$ is a Lie subalgebra of $\mathfrak {{h}}_{2, \mathfrak {{v}}}$. Since $H_{1,1}^\Uplambda$ acts with cohomogeneity one, its orbits coincide with the orbits of $H_{2, \mathfrak {{v}}}$. Altogether, we see that the nilpotent construction method with $j=2$ does not give rise to any new actions.

Nilpotent construction with $j=1$. In this case we have:

$$\begin{aligned} \mathfrak {{n}}_1^1&= \mathfrak {{n}}_1 = \mathfrak {{g}}_{\upalpha _1} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2} \simeq \mathbb {C}^3, \\ \mathfrak {{l}}_1&= \mathfrak {{g}}_{-\upalpha _2} \oplus \mathfrak {{g}}_0 \oplus \mathfrak {{g}}_{\upalpha _2} = \mathfrak {{g}}_1 \oplus \mathfrak {{z}}_1 \oplus \mathfrak {{a}}_1 \simeq \mathfrak {{sl}}(2, \mathbb {C}) \oplus \mathfrak {{u}}(1) \oplus \mathfrak {{u}}(1) \simeq \mathfrak {{gl}}(2, \mathbb {C}), \\ \mathfrak {{k}}_2&= \mathfrak {{k}}_0 \oplus \mathfrak {{k}}_{\upalpha _2} = (\mathfrak {{g}}_1 \cap \mathfrak {{k}}) \oplus \mathfrak {{z}}_1 \simeq \mathfrak {{su}}(2) \oplus \mathfrak {{u}}(1) \simeq \mathfrak {{u}}(2). \end{aligned}$$

We fix an isomorphism between $\mathfrak {{sl}}(2, \mathbb {C})$ and $\mathfrak {{g}}_1$ by sending the standard basis e, f, h of the former to $X, \uptheta X, h_{\upalpha _2} \in \mathfrak {{g}}_1$, where $X \in \mathfrak {{g}}_{\upalpha _2}, \uptheta X \in \mathfrak {{g}}_{-\upalpha _2}$, and $h_{\upalpha _2} = \frac{2}{||\upalpha _2||^2} H_{\upalpha _2}$. Observe that $\mathfrak {{n}}_1$ is an $\upalpha _2$-string, so it is an irreducible complex representation of $\mathfrak {{g}}_1 \simeq \mathfrak {{sl}}(2, \mathbb {C})$. Since it is 3-dimensional, it is isomorphic to the adjoint representation of $\mathfrak {{sl}}(2, \mathbb {C})$. Now, $\mathfrak {{sl}}(2, \mathbb {C})$ is simple, hence it is also simple over $\mathbb {R}$, which means that the representation of $\mathfrak {{g}}_1$ on $\mathfrak {{n}}_1$ is irreducible as a real representation. An alternative description of this representation is the second symmetric power of the tautological representation of $\mathfrak {{sl}}(2, \mathbb {C})$. The adjoint representation of $\mathfrak {{a}}_1 \oplus \mathfrak {{z}}_1 = \mathbb {C}H^1$ on $\mathfrak {{n}}_1^1$ is the standard complex representation by scalars because $H^1$ acts on $\mathfrak {{n}}_1$ by multiplication by $\langle {\upalpha _1} , {H^1} \rangle = 1$. Consequently, we can describe the adjoint representation of $\mathfrak {{l}}_1 \simeq \mathfrak {{sl}}(2, \mathbb {C}) \oplus \mathbb {C}$ (respectively, $\mathfrak {{k}}_1 \simeq \mathfrak {{su}}(2) \oplus \mathfrak {{u}}(1)$) on $\mathfrak {{n}}_1$ as the exterior tensor product $\uprho _3 \otimes \upsigma$, where $\uprho _3$ is the irreducible 3-dimensional complex representation of $\mathfrak {{sl}}(2, \mathbb {C})$ (respectively, of $\mathfrak {{su}}(2)$) and $\upsigma$ is the tautological representation of $\mathbb {C}$ (respectively, $\mathfrak {{u}}(1)$) on $\mathbb {C}$.

Now we need to determine—up to $K_1^0 \simeq \textrm{U}(2)$—all subspaces $\mathfrak {{v}}$ of $\mathfrak {{n}}_1$ which are both protohomogeneous and admissible. We begin by borrowing an argument from [5, Proposition 6]. The case $\mathfrak {{v}} = \mathfrak {{n}}_1$ can be excluded straight away since $\textrm{U}(2)$ cannot act transitively on the 5-sphere. According to [5, Proposition 5], we may assume that $N_{\mathfrak {{m}}_1}(\mathfrak {{v}}) = \uptheta N_{\mathfrak {{m}}_1}(\mathfrak {{n}}_{1, \mathfrak {{v}}})$ is contained in $(\mathfrak {{g}}_1 \cap (\mathfrak {{g}}_0 \oplus \mathfrak {{n}})) \oplus \mathfrak {{z}}_1 = \mathfrak {{k}}_0 \oplus \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2}$. In that case, $N_{\mathfrak {{k}}_1}(\mathfrak {{v}})$ is contained in $\mathfrak {{k}}_0 \simeq \mathfrak {{u}}(1) \oplus \mathfrak {{u}}(1)$. But then $\mathfrak {{v}}$ must be 2-dimensional, for $K_0^0 \simeq \textrm{U}(1) \times \textrm{U}(1)$ cannot act transitively on any sphere of dimension greater than one. We will show that—under the assumption $N_{\mathfrak {{m}}_1}(\mathfrak {{v}}) \subseteq \mathfrak {{k}}_0 \oplus \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2}$—the only option for $\mathfrak {{v}}$ is $\mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2}$.

Take nonzero vectors $e_j \in \mathfrak {{g}}_{\upalpha _1 + j\upalpha _2}, j \in \left\{ {0,1,2} \right\} ,$ such that $e_j = \textrm{ad} (X) e_{j-1}$. Let $v = \sum _{j=0}^2 z_j e_j \in \mathfrak {{v}}$ be a nonzero vector. A generic element of $\mathfrak {{k}}_0 \oplus \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2}$ can be written as

$$\begin{aligned} Y = (a + ib) X + c h_{\upalpha _2} + i(d h_{\upalpha _2} + e H^1), \end{aligned}$$

where a, b, c, d,  and e are some real numbers. If $\mathfrak {{v}}$ lies in $\mathfrak {{g}}_{\upalpha _1} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}$, then a and b must be zero for Y to normalize $\mathfrak {{v}}$, which means that the projection of $N_{\mathfrak {{m}}_1}(\mathfrak {{v}})$ in $\mathfrak {{p}}$ will be smaller than $\mathfrak {{b}}_1$ in that case. Hence, we may assume that $z_2 \ne 0$ and, for the sake of contradiction, that either $z_0$ or $z_1$ is also nonzero. Observe that $\textrm{ad} (h_{\upalpha _2})$ acts diagonally on $\mathfrak {{n}}_1 = \mathfrak {{g}}_{\upalpha _1} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2}$ with eigenvalues $-2, 0,$ and 2, whereas $\textrm{ad} (H^1)$ acts as the identity on the whole $\mathfrak {{n}}_1$. Consequently, $i(d h_{\upalpha _2} + e H^1)$ acts on $\mathfrak {{n}}_1$ diagonally with eigenvalues $i(e-2d), ie, i(e+2d)$. It implies that $N_{\mathfrak {{k}}_1}(\mathfrak {{v}}) \subseteq \mathfrak {{k}}_0$ must be one-dimensional (otherwise it would be the whole $\mathfrak {{k}}_0$ and $\mathfrak {{v}}$ would have to be of dimension at least 3). In this situation, $N_{\mathfrak {{k}}_1}(\mathfrak {{v}})$ is spanned by some $i(d_0 h_{\upalpha _2} + e_0 H^1)$ and $\mathfrak {{v}} = \mathbb {R} v \oplus \mathbb {R} \textrm{ad} (i(d_0 h_{\upalpha _2} + e_0 H^1)) v$. In order for $\textrm{ad} (Y)$ to normalize $\mathfrak {{v}}$, $\textrm{ad} (Y) v$ must be a linear combination of v and $\textrm{ad} (i(d_0 h_{\upalpha _2} + e_0 H^1)) v$, which boils down to the following system of equations:

$$\begin{aligned} {\left\{ \begin{array}{ll} (-2c + i(e-2d))z_1 = \uplambda z_1 + i\upmu (e_0 - 2d_0)z_1, \\ (a+ib)z_1 + i e z_2 = \uplambda z_2 + i\upmu e_0 z_2, \\ (a+ib)z_2 + (2c + i(e+2d))z_3 = \uplambda z_3 + i\upmu (e_0 + 2d_0)z_3. \end{array}\right. } \end{aligned}$$

First assume that $z_1 \ne 0$. The first equation then implies $\uplambda = -2c$, which, when substituted into the second one, gives

$$\begin{aligned} a+ib = \frac{z_2}{z_1}(-2c + i(\upmu e_0 - e)). \end{aligned}$$

By writing $\dfrac{z_2}{z_1} = f + ig$, we obtain

$$\begin{aligned} {\left\{ \begin{array}{ll} a = -2fc - g(\upmu e_0 - e), \\ b = -2gc + f(\upmu e_0 - e). \end{array}\right. } \end{aligned}$$

Whatever f and g are, we get a linear dependency on a, b, e, which means that $\mathfrak {{v}}$ cannot be admissible. In case $z_1 = 0$ but $z_2 \ne 0$, the second equation of our system implies $\uplambda = 0$, which transforms the third equation into

$$\begin{aligned} a+ib = \frac{z_3}{z_2}(-2c + i(\upmu (e_0 + 2d_0) - e - 2d)). \end{aligned}$$

In a similar way, we get a linear dependency on a, b, e.

The upshot of the above argument is that, up to $K_1^0$, $\mathfrak {{v}}$ must be equal to $\mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2}$. Observe that this is indeed an admissible and protohomogeneous subspace. Since the representation of $\textrm{SU}(2) \subset K_1^0$ on $\mathfrak {{n}}_1$ is equivalent to the 3-dimensional complex irreducible representation of $\textrm{SU}(2)$, there exists an element of $K_1^0$ that maps $\mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2}$ onto $\mathfrak {{g}}_{\upalpha _1}$ (in terms of the representation on the space of quadratic polynomials, we can take the special unitary matrix $\left[ {\begin{matrix} 0 &{} -1 \\ 1 &{} 0 \end{matrix}} \right]$, which induces $y^2 \mapsto x^2$). We have:

$$\begin{aligned} \mathfrak {{n}}_{1, \mathfrak {{g}}_{\upalpha _1}}&= \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2}, \\ N_{\mathfrak {{m}}_1}(\mathfrak {{n}}_{1, \mathfrak {{g}}_{\upalpha _1}})&= \uptheta N_{\mathfrak {{m}}_1}(\mathfrak {{g}}_{\upalpha _1}) = \mathfrak {{k}}_0 \oplus \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2}, \\ \mathfrak {{h}}_{1, \mathfrak {{g}}_{\upalpha _1}}&= N_{\mathfrak {{m}}_1}(\mathfrak {{n}}_{1, \mathfrak {{g}}_{\upalpha _1}}) \oplus \mathfrak {{a}}_1 \oplus \mathfrak {{n}}_{1, \mathfrak {{g}}_{\upalpha _1}} = \mathfrak {{g}}_0 \oplus \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2} = \mathfrak {{g}}_0 \oplus \mathfrak {{n}}_2. \end{aligned}$$

But looking at the actions given in (5), one sees that $\mathfrak {{h}}_{1, \mathfrak {{g}}_{\upalpha _1}} = \mathfrak {{h}}_{2,1}^\Uplambda$, which means that the actions of $H_{2,1}^\Uplambda$ and $H_{1, \mathfrak {{g}}_{\upalpha _1}}$ have the same orbits. We conclude that the nilpotent construction method does not give any new actions for M.

5 Classification of cohomogeneity-one actions on the noncompact complex two-plane Grassmannians

In this section we classify, up to orbit equivalence, cohomogeneity-one actions on the noncompact complex Grassmann manifolds $\textrm{Gr}^*(2, \mathbb {C}^{n+4}) = \textrm{SU}(n+2,2)/\textrm{S}(\textrm{U}(n+2)\textrm{U}(2)), n \geqslant 1$.

The symmetric space $M = \textrm{SU}(n+2,2)/\textrm{S}(\textrm{U}(n+2)\textrm{U}(2))$ is irreducible of rank 2 and dimension $4n+8$, and its root system is $BC_2$. This is the complex analog of the symmetric space $\textrm{Gr}^*(2, \mathbb {R}^{n+4}) = \textrm{SO}^0(n+2,2)/\textrm{SO}(n+2)\textrm{SO}(2)$ of type $B_2$, the classification of cohomogeneity-one actions on which, as we mentioned earlier, was obtained by Berndt and Domínguez-Vázquez in [5]. If we choose simple roots $\upalpha _1$ and $\upalpha _2$ such that $2\upalpha _2$ is also a root, then we have $\Upsigma ^+ = \left\{ {\upalpha _1, \upalpha _2, 2\upalpha _2, \upalpha _1 + \upalpha _2, \upalpha _1 + 2\upalpha _2, 2\upalpha _1 + 2\upalpha _2} \right\}$, where $\upalpha _1$ and $\upalpha _1 + 2\upalpha _2$ have multiplicity 2, $\upalpha _2$ and $\upalpha _1 + \upalpha _2$ have multiplicity 2n, and $2\upalpha _2$ and $2\upalpha _1 + 2\upalpha _2$ have multiplicity 1. The Lie algebra $\mathfrak {{k}}_0$ is isomorphic to $\mathfrak {{u}}(n) \oplus \mathfrak {{u}}(1)$. Observe that M is a Hermitian symmetric space and it also has a quaternion-Kähler structure. Let I and $\mathcal {J} \subseteq \textrm{End}(TM)$ denote the almost complex and quaternion-Kähler structures of M, respectively (the latter is a rank-3 subbundle of the endomorphism bundle of TM). We will use the interplay between them to distinguish between non-equivalent cohomogeneity-one actions on M. The boundary components $B_1$ and $B_2$ are isometric to $\mathbb {C}H^{n+1}$ and $\mathbb {R}H^3$, respectively. Therefore, for the first time, we are encountering a noncompact rank-2 symmetric space containing a boundary component not isometric to a real hyperbolic space. The reason why this is special is because, unlike real hyperbolic spaces, complex hyperbolic spaces have a nondiscrete moduli space of cohomogeneity-one actions with a singular orbit, so the canonical extension method, when applied to $B_1$, will produce a one-parameter family of cohomogeneity-one actions with a singular orbit on M. In order to be able to formulate the theorem, we need to know what the almost complex structure I on M looks like in terms of the restricted root space decomposition of $\mathfrak {{g}}$. With respect to $I_o$, $\mathfrak {{p}}_{\upalpha _2}$ and $\mathfrak {{p}}_{\upalpha _1 + \upalpha _2}$ are complex subspaces of $\mathfrak {{p}}$. Moreover, $I_o$ swaps $\mathfrak {{p}}_{\upalpha _1}$ and $\mathfrak {{p}}_{\upalpha _1 + 2\upalpha _2}$, so each of them is a totally real subspace. Finally, $I_o$ sends $\mathbb {R}H_{\upalpha _2}$ to $\mathfrak {{p}}_{2\upalpha _2}$ and $\mathbb {R}H^1$ to $\mathfrak {{p}}_{2\upalpha _1 + 2\upalpha _2}$. We pull this complex structure back to $\mathfrak {{a}} \oplus \mathfrak {{n}}$ along the isomorphism $\mathfrak {{a}} \oplus \mathfrak {{n}} {\mathop {\sim }\limits ^{\rightarrow }}\mathfrak {{p}}$. Note that $B_1$ is a complex submanifold of M, whereas $B_2$ is a totally real one.

Theorem 5.1

Let $H \subset G = \textrm{SU}(n+2,2)$ be a closed connected subgroup acting on M with cohomogeneity one. Then, its action is orbit equivalent to exactly one of the following:

• (1) The action of the connected Lie subgroup $H_\ell$ of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_\ell = (\mathfrak {{a}} \ominus \ell ) \oplus \mathfrak {{n}}, \end{aligned}$$

  where $\ell$ is a one-dimensional linear subspace of $\mathfrak {{a}}$. Its orbits are all isometrically congruent to each other and form a Riemannian foliation on M.

• (2) The action of the connected Lie subgroup $H_i, i \in \left\{ {1,2} \right\} ,$ of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_i = \mathfrak {{a}} \oplus (\mathfrak {{n}} \ominus \ell _i), \end{aligned}$$

  where $\ell _i$ is any one-dimensional linear subspace of $\mathfrak {{g}}_{\upalpha _i}$. Its orbits form a Riemannian foliation on M and there is exactly one minimal orbit.

• (3) The action of the subgroup $\textrm{SU}(n+1,2)$ of G embedded in a standard way. It has a totally geodesic singular orbit of codimension 4 isometric to $\textrm{Gr}^*(2, \mathbb {C}^{n+3}) = \textrm{SU}(n+1,2)/\textrm{S}(\textrm{U}(n+1)\textrm{U}(2))$. This orbit is a complex/quaternionic submanifold with respect to the complex/quaternion-Kähler structures of M.

• (4) The action of the subgroup $\textrm{SU}(n+2,1)$ of G embedded in a standard way. It has a totally geodesic singular orbit of dimension $2n+4$ isometric to $\textrm{Gr}^*(1, \mathbb {C}^{n+3}) = \mathbb {C}H^{n+2} = \textrm{SU}(n+2,1)/\textrm{S}(\textrm{U}(n+2)\textrm{U}(1))$. This orbit is a complex/totally complex submanifold with respect to the complex/quaternion-Kähler structures of M.

• (5) In case $n = 2m \geqslant 2$, the action of the subgroup $\textrm{Sp}(m+1,1)$ of G embedded in a standard way. It has a totally geodesic singular orbit of dimension $2n + 4$ isometric to $\textrm{Gr}^*(1, \mathbb {H}^{m+2}) = \mathbb {H}H^{m+1} = \textrm{Sp}(m+1,1)/\textrm{Sp}(m+1)\textrm{Sp}(1)$. This orbit is a totally real/quaternionic submanifold with respect to the complex/quaternion-Kähler structures of M.

• (6) The action of the connected Lie subgroup $H^\Uplambda _{2,k}, k \in \left\{ {0,1} \right\}$, of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_{2,0}^\Uplambda = \mathfrak {{k}}_2 \oplus \mathfrak {{a}}_2 \oplus \mathfrak {{n}}_2, \quad \mathfrak {{h}}_{2,1}^\Uplambda = \mathfrak {{k}}_0 \oplus \mathfrak {{a}} \oplus \mathfrak {{n}}_2. \end{aligned}$$

  This action has a minimal singular orbit of codimension $3-k$ and can be obtained by canonical extension of the cohomogeneity-one action on $B_2 \simeq \mathbb {R}H^3$ with a single point ($k=0$) or geodesic ($k=1$) as a singular orbit.

• (7) The action of the connected Lie subgroup $H_{1, (\upvarphi , k)}^\Uplambda$, $\upvarphi \in [0, \frac{\uppi }{2}]$, of G with Lie algebra

  $$\begin{aligned} \mathfrak {{h}}_{1, (\upvarphi , k)}^\Uplambda = N_{\mathfrak {k}_1}(\mathfrak {w}) \oplus \mathfrak {{w}} \oplus \mathfrak {{a}}_1 \oplus \mathfrak {{n}}_1, \end{aligned}$$

  where $\mathfrak {{w}} \subseteq \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{2\upalpha _2}$ is the orthogonal complement of a subspace $\mathfrak {{w}}^\perp \subseteq \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{2\upalpha _2}$ of dimension k and constant Kähler angle $\upvarphi$, and

  • (a) For $\upvarphi = 0$, $k \in \left\{ {2, 4, \ldots , 2n+2} \right\}$, and $\mathfrak {{w}}^\perp$ is required to lie in $\mathfrak {{g}}_{\upalpha _2}$ unless $k = 2n+2$. In this case, $\mathfrak {{w}}^\perp$ is just a complex subspace of complex dimension k/2;

  • (b) For $\upvarphi = \frac{\uppi }{2}$, $k \in \left\{ {2, 3, \ldots , n+1} \right\}$, and $\mathfrak {{w}}^\perp$ is required to lie in $\mathfrak {{g}}_{\upalpha _2}$ unless $k = n+1$, in which case it is required to contain $\mathfrak {{g}}_{2\upalpha _2}$. In this case, $\mathfrak {{w}}^\perp$ is a totally real subspace;

  • (c) For $\upvarphi \in (0, \frac{\uppi }{2})$, $k \in \left\{ {2, 4, \ldots , 2\lfloor \frac{n}{2} \rfloor } \right\}$, and $\mathfrak {{w}}^\perp$ is required to lie in $\mathfrak {{g}}_{\upalpha _2}$.

  This action has a minimal singular orbit of codimension k and can be obtained by canonical extension of a cohomogeneity-one action on $B_1 \simeq \mathbb {C}H^{n+1}$ with a singular orbit.

Proof

We consider different cases of Theorem 2.1. If the orbits of H form a foliation, we get the actions in (1) and (2).

The actions in (3), (4), and (5) are the only ones with a totally geodesic singular orbit according to [7]. Let us describe these three orbits in a bit more detail. Recall that M can be thought of as the set of 2-dimensional complex subspaces of $\mathbb {C}^{n+4}$ on which the restriction of the standard indefinite Hermitian form of signature $(n+2,2)$ is negative definite. The group $G = \textrm{SU}(n+2,2)$ acts transitively on that set and the stabilizer of $o = \langle e_{n+3}, e_{n+4} \rangle _\mathbb {C}$ is precisely $K = \textrm{S}(\textrm{U}(n+2)\textrm{U}(2))$.

Consider the subset $S_3$ of M consisting of those 2-dimensional subspaces that lie in the complex hyperplane $\langle e_2, \ldots , e_{n+4} \rangle _\mathbb {C}$. This subset is $\textrm{Gr}^*(2,\mathbb {C}^{n+3})$ embedded totally geodesically into M. The subgroup $\textrm{SU}(n+1,2) \subset G$ of elements fixing the first basis vector $e_1$ preserves $S_3$ and acts transitively on it. Its isotropy subgroup at o is $\textrm{S}(\textrm{U}(n+1)\textrm{U}(2))$, whose subgroup $\textrm{U}(2)$ acts transitively on the unit sphere in $N_o S_3$, so $\textrm{SU}(n+1,2)$ does indeed act on M with cohomogeneity one and has $S_3$ as its singular orbit (we have $S_3^\perp \simeq \mathbb {C}H^2$).

Similarly, define $S_4$ to be the subset of M consisting of those 2-dimensional subspaces of $\mathbb {C}^{n+4}$ that contain the last basis vector $e_{n+4}$. These are in bijective correspondence with the complex lines in the hyperplane $\langle e_1, \ldots , e_{n+3} \rangle _\mathbb {C}$ on which the restriction of the standard indefinite Hermitian form of signature $(n+2,1)$ is negative definite, so $S_4$ is actually $\textrm{Gr}^*(1, \mathbb {C}^{n+3}) = \mathbb {C}H^{n+2}$ embedded totally geodesically into M. The subgroup $\textrm{SU}(n+2,1) \subset G$ of elements fixing $e_{n+4}$ preserves $S_4$ and acts transitively on it. Its isotropy subgroup at o is $\textrm{S}(\textrm{U}(n+2)\textrm{U}(1))$ and its slice representation at o is equivalent to the tautological representation $\textrm{U}(n+2) \curvearrowright \mathbb {C}^{n+2}$, also known as the isotropy representation of $\mathbb {C}H^{n+2}$, which is a reflection of the fact that $S_4^\perp \simeq \mathbb {C}H^{n+2}$. It is not hard to show that $S_4^\perp$ is a reflective submanifold of M. But now observe that $(S_4^\perp )^\perp = S_4$ is of rank 1, so, according to Sect. 2.2, $S_4^\perp$ is also a totally geodesic singular orbit of some cohomogeneity-one action. We claim that this action is orbit equivalent to the action of $\textrm{SU}(n+2,1)$, i.e., that $S_4$ and $S_4^\perp$ are congruent. Indeed, $S_4$ is easily seen to be a totally complex submanifold with respect to the quaternion-Kähler structure of M, which means that there exists a one-dimensional subspace $\mathcal {J}'_o \subset \mathcal {J}_o$ preserving $T_o S_4$ and $J(T_o S_4) = N_o S_4 = T_o S_4^\perp$ for each $J \in \mathcal {J}_o$ orthogonal to $\mathcal {J}'_o$. Each such J of square $-1$ can be realized as the differential at o of some $k \in K$ (see the discussion of the quaternion-Kähler structure of M below), so k is an isometry mapping $S_4$ onto $S_4^\perp$. Note that the geodesic reflection in $S_4$ is a holomorphic involutive isometry. The submanifold $S_4$ is called a complex form of M (see [20]).

Finally, if $n=2m \geqslant 2$, consider the standard indefinite quaternionic Hermitian form H on $\mathbb {H}^{m+2}$ of signature $(m+1,1)$ and identify $\mathbb {H}^{m+2}$ with $\mathbb {C}^{2\,m+4}$ in a standard way. Note that if we write $H = h - \omega j$, where $h, \omega :\mathbb {H}^{m+2} \times \mathbb {H}^{m+2} \rightarrow \mathbb {C}$, then h is precisely our indefinite Hermitian form on $\mathbb {C}^{n+4}$. Consider the subset $S_5$ of M consisting of the quaternionic lines in $\mathbb {C}^{n+4}$ the restriction of H to which is negative definite. This subset is $\textrm{Gr}^*(1, \mathbb {H}^{m+2}) = \mathbb {H}H^{m+1}$ embedded totally geodesically into M. The group $\textrm{Sp}(m+1,1)$ sits naturally inside G and preserves $S_5$. Its isotropy subgroup at o is $\textrm{Sp}(m+1)\textrm{Sp}(1)$, whose slice representation at o is equivalent to the standard representation on $\mathbb {H}^{m+1}$. Just like before, we have $S_5^\perp \simeq \mathbb {H}H^{m+1} \simeq S_5$, and these two submanifolds are in fact congruent. Their tangent spaces at o are mapped to each other by $I_o$ because $S_5$ is totally real. But just as was the case with the almost complex structures in $\mathcal {J}_o$, $I_o$ also comes from K, so the corresponding $k \in K$ maps $S_5$ onto $S_5^\perp$. The geodesic reflection in $S_5$ is an anti-holomorphic involutive isometry, for which reason $S_5$ is called a real form of M (see [13]).

Now we proceed to the canonical extension method. There are two strong orbit equivalence classes of cohomogeneity-one actions on $B_2 \simeq \mathbb {R}H^3$ with a singular orbit, whose extensions are described in (6). Now consider $B_1 \simeq \mathbb {C}H^{n+1}$. All cohomogeneity-one actions on $B_1$ with a non-totally-geodesic singular orbit come from the nilpotent construction method. The nilpotent construction representation for $\mathbb {C}H^{n+1}$ is equivalent to the tautological representation $\textrm{U}(n) \curvearrowright \mathbb {C}^n$. We know that if two protohomogeneous subspaces of $\mathfrak {{g}}_{\upalpha _2} \simeq \mathbb {C}^n$ are $\textrm{U}(n)$-congruent, they produce strongly orbit equivalent cohomogeneity-one actions on $\mathbb {C}H^{n+1}$. It was shown in [8, Theorem 4.1(ii)] that the converse is true. To be precise, the following are equivalent for protohomogeneous subspaces $\mathfrak {{v}}, \mathfrak {{v}}' \subseteq \mathfrak {{g}}_{\upalpha _2}$:

• (i) The cohomogeneity-one actions on $\mathbb {C}H^{n+1}$ produced by $\mathfrak {{v}}$ and $\mathfrak {{v}}'$ are orbit equivalent.

• (ii) The cohomogeneity-one actions on $\mathbb {C}H^{n+1}$ produced by $\mathfrak {{v}}$ and $\mathfrak {{v}}'$ are strongly orbit equivalent.

• (iii) $\mathfrak {{v}}$ and $\mathfrak {{v}}'$ are $\textrm{U}(n)$-congruent.

So what we really need is the classification of protohomogeneous subspaces of $\mathbb {C}^n$ with respect to $\textrm{U}(n)$. This was carried out by Berndt and Brück in [2]. They showed that a subspace of $\mathbb {C}^n$ is protohomogeneous if and only if it has constant Kähler angle, and two such subspaces are $\textrm{U}(n)$-congruent if and only if they have the same dimension and Kähler angle. Together with the classification of cohomogeneity-one actions on $\mathbb {C}H^{n+1}$ with a totally geodesic singular orbit (see [2, Theorem 1]), Proposition 2.5 and the observation at the end of Sect. 2.4 imply that the actions in (7) exhaust the list of actions coming from those on $B_1$ by canonical extension. No two actions from (7) are mutually orbit equivalent by design: the normal spaces of their singular orbits differ either in dimension or (constant) Kähler angle. Here we use the fact that every isometry of M is either holomorphic or anti-holomorphic so it preserves the Kähler angles of tangent subspaces (holomorphic and anti-holomorphic isometries constitute the two connected components of I(M)).

There is one action in (6) that may, in theory, have a totally geodesic singular orbit, namely the action of $H_{1,(0,4)}^\Uplambda$. Its orbit $S = H_{1,(0,4)}^\Uplambda \cdot o$ is a complex submanifold of M of codimension 4. But so is the singular orbit of the action of $\textrm{SU}(n+1,2) \subset G$ given in (3), so the action of $H_{1,(0,4)}^\Uplambda$ may be orbit equivalent to that of $\textrm{SU}(n+1,2)$. To show that this is not the case, we have to prove that S is not totally geodesic. To this end, we need to show the second fundamental form $I I$ of S is nonzero. Note that

$$\begin{aligned} T_oS = \textrm{pr}_\mathfrak {{p}}(\mathfrak {{w}}) \oplus \mathfrak {{a}}_1 \oplus \mathfrak {{n}}_1, \quad N_oS = \textrm{pr}_\mathfrak {{p}}(\mathfrak {{w}}^\perp ). \end{aligned}$$

Recall that $\mathfrak {{w}}^\perp$ is contained in $\mathfrak {{g}}_{\upalpha _2}$ if $n>1$ and coincides with $\mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{2\upalpha _2}$ if $n=1$. Since $[\mathfrak {{g}}_{\upalpha _1 + \upalpha _2}, \mathfrak {{g}}_{-\upalpha _1}] = \mathfrak {{g}}_{\upalpha _2}$, we can find $X \in \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}$ and $Y \in \mathfrak {{g}}_{-\upalpha _1}$ such that $[X,Y] \in \mathfrak {{g}}_{\upalpha _2} \ \mathfrak {{w}}$. Note that $X - \uptheta X$ and $Y - \uptheta Y$ lie in $T_oS$. Since S is an orbit of $H_{1,(0,4)}^\Uplambda$ and the Riemannian metric on M comes from the Killing form of $\mathfrak {{g}}$, we have $I I(X - \uptheta X, Y - \uptheta Y) = \textrm{pr}_{N_oS}([Z, Y - \uptheta Y])$, where $Z \in \mathfrak {{k}}$ is any vector such that $X - \uptheta X + Z \in \mathfrak {{h}}_{1,(0,4)}^\Uplambda$ (see, for example,⁶ [17]). We pick $Z = X + \uptheta X$, which gives

$$\begin{aligned} I I(X - \uptheta X, Y - \uptheta Y) = \textrm{pr}_{N_oS}([X + \uptheta X, Y - \uptheta Y]) = \textrm{pr}_{N_oS}([X, Y] - \uptheta [X,Y]) \ne 0, \end{aligned}$$

which implies that the fundamental form of S is nonzero at o and S is not totally geodesic.

Now we need to prove that the actions in (6) and (7) are mutually non-equivalent. We are going to show that, given an action from (6) with a singular orbit S and one from (7) with a singular orbit $S'$ of the same dimension as S, normal spaces to S and $S'$ have either different Kähler angles or vectors of different holomorphic sectional curvatures. Computing all such sectional curvatures by hand would be a daunting task, so we will employ a different strategy and leverage the abundance of geometric structures on M. In [1], Berndt studied the complex Grassmannian of two-planes $\textrm{Gr}(2, \mathbb {C}^{n+4})$, which is the compact dual of our symmetric space M, so we denote it by $M^*$. Since holonomy is preserved under duality, $M^*$ is a Hermitian symmetric space and it also has a quaternion-Kähler structure. Write $I^*$ for the almost complex structure of $M^*$ and $\mathcal {J}^*$ for the 3-dimensional subbundle of $\textrm{End}(T M^*)$ given by the quaternion-Kähler structure of $M^*$. Berndt showed that holomorphic sectional curvatures of $M^*$ lie between 4 and 8 after a suitable rescaling of the Riemannian metric. Moreover, he proved that, given any $p \in M^*$ and a nonzero $X \in T_p M^*$, the following are equivalent:

• (i) X is singular.

• (ii) $I_p^* X \perp \mathcal {J}^*_p X$ or $I_p^* X \in \mathcal {J}^*_p X$.

• (iii) The holomorphic sectional curvature determined by X is either minimal or maximal (that is, either 4 or 8).

Recall that X is called singular if it is tangent to more than one maximal flat in $M^*$. It is equivalent to asking that X is a singular point with respect to the restricted isotropy representation of $M^*$, i.e., that the $\textrm{S}(\textrm{U}(n+2)\textrm{U}(2))$-orbit of X in $T_p M^*$ is singular. From this second description, one sees that the set of regular (= nonsingular) vectors in $T_p M^*$ is connected, open, and dense. In (ii) and (iii), $I^*_p X \perp \mathcal {J}^*_p X$ corresponds to $K_\textrm{hol}(X) = 4$, while $I_p^* X \in \mathcal {J}_p^*X$ corresponds to $K_\textrm{hol}(X) = 8$. So this result allows us to divide singular tangent vectors to $M^*$ into two groups depending on their holomorphic sectional curvature. We temporarily call a singular vector X of type A if $I_p^* X \perp \mathcal {J}_p^* X$ and of type B if $I_p^* X \in \mathcal {J}_p^* X$. Since duality preserves holonomy and isotropy subgroups and changes the sign of sectional curvatures, the whole discussion applies to M with ’minimal’ interchanged with ’maximal’ in (iii) and 4 and 8 replaced with $-4c$ and $-8c$, respectively (our normalization of the Riemannian metric may differ from the one in [1]). Now, the fact that is of paramount importance for us and that allows to use this interplay between the two structures of M is that isometries preserve not only the set of singular vectors but their type. Indeed, any isometry is either holomorphic or anti-holomorphic, so it preserves holomorphic sectional curvatures and hence the type of a singular vector.

We already know what the complex structure of M looks like with respect to the restricted root space decomposition, so now we need to describe the quaternion-Kähler structure. We have $\mathfrak {{g}} = \mathfrak {{su}}(n+2,2)$, and its Cartan decomposition is:

$$\begin{aligned} \mathfrak {{k}} = \left\{ \left. \left[ \begin{array}{c | c} \begin{array}{c c c} &{} &{} \\ &{} \mathfrak {{u}}(n+2) &{} \\ &{} &{} \end{array} &{} 0 \\ \hline 0 &{} \mathfrak {{u}}(2) \end{array} \right] \right| \textrm{tr}= 0 \right\} , \quad \mathfrak {{p}} = \left\{ \left[ \begin{array}{c | c} \begin{array}{c c c} &{} &{} \\ &{} 0 &{} \\ &{} &{} \end{array} &{} B \\ \hline B^* &{} 0 \end{array} \right] \right\} , \end{aligned}$$

where B runs through $\textrm{Mat}((n+2) \times 2, \mathbb {C})$. The restricted isotropy representation of M is just the adjoint representation of $K = \textrm{S}(\textrm{U}(n+2)\textrm{U}(2))$ on $\mathfrak {{p}}$. Note that if we identify $\mathfrak {{p}}$ with $\mathbb {C}^{n+2} \oplus \mathbb {C}^{n+2} \cong \mathbb {C}^{n+2} \otimes \mathbb {C}^2$ in the obvious way, then the restricted isotropy representation is isomorphic to the restriction of the external tensor product representation $\uprho _{n+2} \otimes \uprho _2$ of $\textrm{U}(n+2) \times \textrm{U}(2)$ to K (here $\uprho _k$ is the tautological representation of $\textrm{U}(k)$ on $\mathbb {C}^k$). We have a normal subgroup $\textrm{Sp}(1) \simeq \textrm{SU}(2) \trianglelefteq K$, and its representation on $\mathfrak {{p}}$ is just $1 \otimes \uprho _2$, so it extends to a unital $\mathbb {R}$-algebra homomorphism $\mathbb {H}\rightarrow \textrm{End}(\mathfrak {{p}})$, i.e., a quaternionic structure on $\mathfrak {{p}}$. The fact that $\textrm{SU}(2)$ is a normal subgroup of $K = \textrm{Hol}(M,o)$ means that this structure is holonomy-invariant and thus gives a parallel quaternionic structure on M, while $\textrm{SU}(2) \subseteq \textrm{O}(\mathfrak {{p}})$ means that this structure is compatible with the Riemannian metric and therefore makes M into a quaternion-Kähler manifold. Note that—by construction—every almost complex structure J in $\mathcal {J}_o$ lies in the image of the isotropy representation, i.e., it is the differential at o of some isometry $k \in K$. We have already used this fact before when we described the totally geodesic singular orbits of the actions in (3)–(5).

Let us take $\mathfrak {{a}} = \mathbb {R}(E_{n+2,n+3} + E_{n+3,n+2}) \oplus \mathbb {R}(E_{n+1,n+4} + E_{n+4,n+1})$. The restricted root space decomposition of $\mathfrak {{g}}$ with respect to such a choice of $\mathfrak {{a}}$ is explicitly described in Knapp [14, p. 371, Example 2]. Using that, as well as our description of the quaternion-Kähler structure, one easily sees how $\mathcal {J}_o \subseteq \textrm{End}(\mathfrak {{p}})$ acts on different root vectors. Note also that the almost complex structure on M is given at the point o simply by

$$\begin{aligned} I_o :\; \left[ \begin{array}{c | c} 0 &{} B \\ \hline B^* &{} 0 \end{array} \right] \mapsto \left[ \begin{array}{c | c} 0 &{} iB \\ \hline -i B^* &{} 0 \end{array} \right] . \end{aligned}$$

Finally, observe that $\mathfrak {{su}}(n+2,2)$ is a real form of $\mathfrak {{sl}}(n+4, \mathbb {C})$, so the Killing form B of $\mathfrak {{g}}$ is just the restriction of the Killing form of $\mathfrak {{sl}}(n+4, \mathbb {C})$, which is $(2n+8)\textrm{tr}$. Now we are ready to search for singular vectors of types A and B in $\mathfrak {{p}}$. For instance, one can compute that $\mathcal {J}_o$ sends $\mathfrak {{p}}_{2\upalpha _2}$ onto $\mathbb {R}H_{\upalpha _2} \oplus \mathbb {C}(E_{n+1,n+3} + E_{n+3,n+1})$, while $I_o$ sends it onto $\mathbb {R}H_{\upalpha _2}$, so we deduce that $\mathfrak {{p}}_{2\upalpha _2}$ consists entirely of singular vectors of type B. In a similar fashion, one calculates that

• $\mathfrak {{p}}_{\upalpha _1}$ and $\mathfrak {{p}}_{\upalpha _1 + 2\upalpha _2}$ consist of singular vectors of type A, while

• $\mathfrak {{p}}_{2\upalpha _1 + 2\upalpha _2}, \mathfrak {{p}}_{\upalpha _2}, \mathfrak {{p}}_{\upalpha _1 + \upalpha _2}, \mathbb {R}H^1,$ and $\mathbb {R}H_{\upalpha _2}$ all consist of singular vectors of type B.

Consider the action of $H_{2,1}^\Uplambda$. The normal space of its singular orbit at o is $\mathfrak {{p}}_{\upalpha _1}$, which is a totally real subspace of $\mathfrak {{p}}$. The only action in (7) the normal spaces of whose singular orbit are totally real of dimension 2 is the action of $H_{1,(\pi /2,2)}^\Uplambda$. Whatever n is, $\mathfrak {{w}}^\perp$ intersects $\mathfrak {{g}}_{\upalpha _2}$ and thus $N_o(H_{1,(\pi /2,2)}^\Uplambda \cdot o) = \textrm{pr}_\mathfrak {{p}}(\mathfrak {{w}}^\perp )$ contains a singular vector of type B, whereas $N_o(H_{2,1}^\Uplambda \cdot o) = \mathfrak {{p}}_{\upalpha _1}$ consists entirely of singular vectors of type A. Therefore, there cannot exist an orbit equivalence between these two actions. Now take the action of $H_{2,0}^\Uplambda$. The normal space of its singular orbit at o is $T_o B_2 = \mathbb {R}H_{\upalpha _1} \oplus \mathfrak {{p}}_{\upalpha _1}$, which is totally real. Again, the only action in (7) the normal spaces of whose singular orbit are totally real of dimension 3 is the action of $H_{1,(\pi /2,3)}^\Uplambda$ ($n \geqslant 2$). Regardless of the value of n, $\dim (\mathfrak {{w}}^\perp \cap \mathfrak {{g}}_{\upalpha _2}) \geqslant 2$, so $N_o(H_{1,(\pi /2,3)}^\Uplambda \cdot o) = \textrm{pr}_\mathfrak {{p}}(\mathfrak {{w}}^\perp )$ contains a 2-dimensional subspace L of singular vectors of type B, while $N_o(H_{2,0}^\Uplambda \cdot o)$ contains $\mathfrak {{p}}_{\upalpha _1}$, a 2-dimensional subspace of singular vectors of type A. If there was an orbit equivalence between these two actions, there would be one fixing o, which would have to send L onto a plane in $N_o(H_{2,0}^\Uplambda \cdot o)$ overlapping with $\mathfrak {{p}}_{\upalpha _1}$, which is impossible. Consequently, no action in (6) is orbit equivalent to an action in (7).

Finally, we proceed to the nilpotent construction method. To distinguish between the Euclidean and Hermitian inner products on $\mathfrak {{a}} \oplus \mathfrak {{n}}$, we will be adding the letter H when talking about the latter. For example, $v \perp _H w$ means orthogonality with respect to the Hermitian inner product.

Nilpotent construction with $j=2$. In this case we have:

$$\begin{aligned} \mathfrak {{n}}_2^1&= \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \simeq \mathbb {C}^n \oplus \mathbb {C}^n \cong \mathbb {C}^n \otimes \mathbb {C}^2, \\ \mathfrak {{m}}_2&= \mathfrak {{z}}_2 \oplus \mathfrak {{g}}_2 = \mathfrak {{u}}(n) \oplus \mathfrak {{so}}(3,1) \simeq \mathfrak {{u}}(n) \oplus \mathfrak {{sl}}(2,\mathbb {C}), \\ \mathfrak {{k}}_2&= \mathfrak {{u}}(n) \oplus \mathfrak {{su}}(2). \end{aligned}$$

It is not hard to show that the representation of $\mathfrak {{m}}_2$ on $\mathfrak {{n}}_2^1$ is equivalent to the external tensor product representation of $\mathfrak {{u}}(n) \oplus \mathfrak {{sl}}(2,\mathbb {C})$ on $\mathbb {C}^n \otimes \mathbb {C}^2$. Consequently, the representation of $K_2^0 \simeq \textrm{U}(n) \times \textrm{SU}(2) \simeq \textrm{S}(\textrm{U}(n)\textrm{U}(2))$ on $\mathfrak {{n}}_2^1$ is equivalent to the isotropy representation of $\textrm{Gr}^*(2, \mathbb {C}^{n+2}) = \textrm{SU}(n,2)/\textrm{S}(\textrm{U}(n)\textrm{U}(2))$. Pick some orthonormal bases (with respect to the Hermitian inner product) $e_1, \ldots , e_n \in \mathbb {C}^n$ and $f_1, f_2 \in \mathbb {C}^2$. Observe that when we think of $K_2^0 \curvearrowright \mathfrak {{n}}_2^1$ as the isotropy representation of $\textrm{Gr}^*(2, \mathbb {C}^{n+2})$, the two-dimensional subspace $\mathbb {R}(e_1 \otimes f_1) \oplus \mathbb {R}(e_2 \otimes f_2)$ is a maximal flat. This observation will prove useful later on. Now, let $\mathfrak {{v}} \subseteq \mathfrak {{n}}_2^1$ be an admissible and protohomogeneous subspace and let $v = v_1 + v_2 \in \mathfrak {{v}}, v_1 \in \mathfrak {{g}}_{\upalpha _2}, v_2 \in \mathfrak {{g}}_{\upalpha _1 + \upalpha _2}$. First, mimicking the proof of Theorem 8 in [5], we prove the following

Lemma 5.2

Let $T = \left[ {\begin{matrix} 0 &{} -1 \\ 1 &{} 0 \end{matrix}} \right] \in \mathfrak {{so}}(2) \subseteq \mathfrak {{su}}(2) \subseteq \mathfrak {{m}}_2$. If $[T, v_1] \perp _H v_2$, then either $v_1 = 0$ or $v_2 = 0$.

Proof of the lemma

Under the assumption $[T, v_1] \perp _H v_2$, we do not lose generality by taking $v_1 = r e_1 \otimes f_1$ and $v_2 = s e_2 \otimes f_2$ for some $r,s \in \mathbb {R}$. By protohomogeneity of $\mathfrak {{v}}$, we have:

$$\begin{aligned} \mathfrak {{v}} = \mathbb {R}(r e_1 \otimes f_1 + s e_2 \otimes f_2) \oplus ^\perp N_{\mathfrak {{k}}_2}(\mathfrak {{v}})(r e_1 \otimes f_1 + s e_2 \otimes f_2). \end{aligned}$$

Note that our assumption implies that the second summand here is actually orthogonal to both $e_1 \otimes f_1$ and $e_2 \otimes f_2$ (with respect to the Euclidean inner product). Now let $S + A \in N_{\mathfrak {{m}}_2}(\mathfrak {{v}}), S \in \mathfrak {{u}}(n), A = \left[ {\begin{matrix} x &{} y \\ z &{} -x \end{matrix}} \right] \in \mathfrak {{sl}}(2, \mathbb {C})$. We compute:

$$\begin{aligned}&(S + A)(r e_1 \otimes f_1 + s e_2 \otimes f_2) \\&\quad = rS e_1 \otimes f_1 + sS e_2 \otimes f_2 + r e_1 \otimes (x f_1 + z f_2) + s e_2 \otimes (y f_1 - x f_2) \\&\quad = r \textrm{Re}(x) e_1 \otimes f_1 - s \textrm{Re}(x) e_2 \otimes f_2 + (\text {terms in} \; (r e_1 \otimes f_1 + s e_2 \otimes f_2)^\perp ). \end{aligned}$$

We see that $r \textrm{Re}(x) e_1 \otimes f_1 - s \textrm{Re}(x) e_2 \otimes f_2$ must lie in $\mathbb {R}(r e_1 \otimes f_1 + s e_2 \otimes f_2)$, which is only possible when either r or s is zero.

Since $\mathbb {R}(e_1 \otimes f_1) \oplus \mathbb {R}(e_2 \otimes f_2)$ is a maximal flat, it intersects the isotropy orbit of each vector, i.e., there exists some $k \in K_2^0$ such that $\textrm{Ad} (k)v \in \mathbb {R}(e_1 \otimes f_1) \oplus \mathbb {R}(e_2 \otimes f_2)$. It then follows from Lemma 5.2 that $\textrm{Ad} (k)v$ is proportional to either $e_1 \otimes f_1$ or $e_2 \otimes f_2$. Applying $T \in K_2^0$ if needed, we may assume the former is the case, so we simply assume $e_1 \otimes f_1 \in \mathfrak {{v}}$.

Since $\mathfrak {{v}}$ is protohomogeneous, we have $\mathfrak {{v}} \subseteq \textrm{span} \left\{ {K_2^0 \cdot (e_1 \otimes f_1)} \right\} = \mathfrak {{g}}_{\upalpha _2} \oplus \mathbb {C}(e_1 \otimes f_2)$. We decompose $\mathfrak {{su}}(2)$ as a vector space into two pieces:

$$\begin{aligned} \ell = \left\{ \left. \begin{bmatrix} ia &{} 0 \\ 0 &{} -ia \end{bmatrix} \right| a \in \mathbb {R}\right\} , \quad \ell ^\perp = \left\{ \left. \begin{bmatrix} 0 &{} -\overline{z} \\ z &{} 0 \end{bmatrix} \right| z \in \mathbb {C}\right\} , \end{aligned}$$

so $\mathfrak {{su}}(2) = \ell \oplus \ell ^\perp$. Clearly, $\mathfrak {{v}} \subseteq \mathfrak {{g}}_{\upalpha _2} \Leftrightarrow N_{\mathfrak {{k}}_2}(\mathfrak {{v}}) \subseteq \mathfrak {{u}}(n) \oplus \ell$. One can easily see that every subspace $\mathfrak {{v}}$ of $\mathfrak {{g}}_{\upalpha _2}$ is automatically admissible. Now, we may regard such $\mathfrak {{v}}$ as lying in the isometry Lie algebra $\mathfrak {{g}}_1 = \widetilde{\mathfrak {{g}}}_1$ of $B_1 \simeq \mathbb {C}H^{n+1}$, and $\mathfrak {{v}}$ is protohomogeneous in $\mathfrak {{n}}_2^1 \curvearrowleft M_2$ if and only if it is such in $\mathfrak {{g}}_{\upalpha _2} \curvearrowleft \textrm{U}(n)$ (which just means that it has constant Kähler angle). Moreover, the canonical extension of the resulting cohomogeneity-one action on $B_1$ has the same orbits as the action obtained from $\mathfrak {{v}}$ by nilpotent construction performed on M. So such $\mathfrak {{v}}$’s produce no new actions and we may assume $\mathfrak {{v}}$ does not lie in $\mathfrak {{g}}_{\upalpha _2}$ and thus $N_{\mathfrak {{k}}_2}(\mathfrak {{v}})$ has a nonzero projection in $\ell ^\perp$. Let $S + A \in N_{\mathfrak {{k}}_2}(\mathfrak {{v}}), S = (s_{ij})_{i,j=1}^n, A = \left[ {\begin{matrix} ia &{} -\overline{z} \\ z &{} -ia \end{matrix}} \right] , z \ne 0$. We compute:

In order for this to lie in $\mathfrak {{v}} \subseteq \mathfrak {{g}}_{\upalpha _2} \oplus \mathbb {C}(e_1 \otimes f_2)$, we must have $s_{i1} = 0$ for $2 \leqslant i \leqslant n$, which means that $(ia + s_{11})e_1 \otimes f_1 + ze_1 \otimes f_2 \in \mathfrak {{v}}$. But this forces $N_{\mathfrak {{k}}_2}(\mathfrak {{v}})$ to lie inside $\mathfrak {{u}}(1) \oplus \mathfrak {{su}}(2) \subseteq \mathfrak {{u}}(n) \oplus \mathfrak {{su}}(2)$, for otherwise we would have some $S + A \in N_{\mathfrak {{k}}_2}(\mathfrak {{v}}), S \in \mathfrak {{u}}(n) \ \mathfrak {{u}}(1),$ moving $(ia + s_{11})e_1 \otimes f_1 + ze_1 \otimes f_2$ out of $\mathfrak {{g}}_{\upalpha _2} \oplus \mathbb {C}(e_1 \otimes f_2)$. The upshot of all this is that $\mathfrak {{v}} \subseteq \mathbb {C}(e_1 \otimes f_1) \oplus \mathbb {C}(e_1 \otimes f_2)$ and $N_{\mathfrak {{m}}_2}(\mathfrak {{v}}) \subseteq \mathfrak {{u}}(1) \oplus \mathfrak {{sl}}(2, \mathbb {C})$ (and we still assume $e_1 \otimes f_1 \in \mathfrak {{v}}$). So we have reduced our problem to looking for admissible and protohomogeneous subspaces of $\mathbb {C}^2$ with respect to the tautological representation of $\mathfrak {{gl}}(2, \mathbb {C})$. Protohomogeneity singles out precisely subspaces of constant Kähler angle $\upvarphi$. We consider three cases:

Case 1: $\upvarphi = 0$. Since we assume $\mathfrak {{v}} \nsubseteq \mathfrak {{g}}_{\upalpha _2}$, we must have $\mathfrak {{v}} = \mathbb {C}(e_1 \otimes f_1) \oplus \mathbb {C}(e_1 \otimes f_2)$. But then we get a cohomogeneity-one action with a totally geodesic singular orbit isometric to $\textrm{Gr}^*(2, \mathbb {C}^{n+3})$, see (3).

Case 2: $\upvarphi = \frac{\uppi }{2}$. Without loss of generality, we pick $\mathfrak {{v}} = \mathbb {R}(e_1 \otimes f_1) \oplus \mathbb {R}(e_1 \otimes f_2)$. In this case, $N_{\mathfrak {{gl}}(2,\mathbb {C})}(\mathfrak {{v}}) = \mathfrak {{gl}}(2, \mathbb {R})$, so $\mathfrak {{v}}$ fails to be admissible.

Case 3: $\upvarphi \in (0, \frac{\uppi }{2})$. According to [2, Proposition 7], we can take $\mathfrak {{v}}$ to be the span of (1, 0) and $(i\cos \upvarphi , i\sin \upvarphi )$. If we write the coordinates on $\mathbb {C}^2$ as $z_1$ and $z_2$, then $\mathfrak {{v}}$ is cut out by the equations $\textrm{Re}(z_2) = 0$ and $\textrm{Im}(z_2) = \textrm{Im}(z_1) \tan \upvarphi$. We need to check whether the projection of $N_{\mathfrak {{sl}}(2,\mathbb {C})}(\mathfrak {{v}})$ to the space of Hermitian traceless matrices is onto. A matrix $\left[ {\begin{matrix} x &{} y \\ z &{} -x \end{matrix}} \right] \in \mathfrak {{sl}}(2, \mathbb {C})$ normalizing $\mathfrak {{v}}$ is subject to the following equations:

$$\begin{aligned} {\left\{ \begin{array}{ll} \textrm{Re}(z) = 0, \\ \textrm{Im}(z) = \textrm{Im}(x) \tan \upvarphi , \\ -\textrm{Im}(z) \cos \upvarphi + \textrm{Im}(x) \sin \upvarphi = 0, \\ \textrm{Re}(z) \cos \upvarphi - \textrm{Re}(x) \sin \upvarphi = (\textrm{Re}(x) \cos \upvarphi + \textrm{Re}(y) \sin \upvarphi ) \tan \upvarphi . \end{array}\right. } \end{aligned}$$

The second and third equations are the same, so, simplifying, we are left with:

$$\begin{aligned} {\left\{ \begin{array}{ll} \textrm{Re}(z) = 0, \\ \textrm{Im}(z) = \textrm{Im}(x) \tan \upvarphi , \\ \textrm{Re}(x) = - \textrm{Re}(y) \dfrac{\tan \upvarphi }{2}. \end{array}\right. } \end{aligned}$$

These equations cut out a 3-dimensional subspace inside $\mathfrak {{sl}}(2, \mathbb {C})$. Its projection to the space of Hermitian traceless matrices is onto precisely when it does not intersect $\mathfrak {{su}}(2)$. But these equations have a nontrivial solution in $\mathfrak {{su}}(2)$, namely $\left[ {\begin{matrix} i &{} -i\tan \upvarphi \\ i\tan \upvarphi &{} -i \end{matrix}} \right]$, so $\mathfrak {{v}}$ is not admissible.

Nilpotent construction with $j=1$. In this case we have:

$$\begin{aligned} \mathfrak {{n}}_1^1&= \mathfrak {{g}}_{\upalpha _1} \oplus \mathfrak {{g}}_{\upalpha _1 + \upalpha _2} \oplus \mathfrak {{g}}_{\upalpha _1 + 2\upalpha _2} \simeq \mathbb {C}^{n+2}, \\ \mathfrak {{m}}_1&= \mathfrak {{g}}_1 \oplus \mathfrak {{z}}_1 = \mathfrak {{su}}(n+1,1) \oplus \mathfrak {{u}}(1) \simeq \mathfrak {{u}}(n+1,1), \\ \mathfrak {{k}}_1&= \mathfrak {{s}}(\mathfrak {{u}}(n+1) \oplus \mathfrak {{u}}(1)) \oplus \mathfrak {{u}}(1) \simeq \mathfrak {{u}}(n+1) \oplus \mathfrak {{u}}(1). \end{aligned}$$

The representation of $\mathfrak {{m}}_1$ on $\mathfrak {{n}}_1^1$ can be shown to be equivalent to the tautological representation of $\mathfrak {{u}}(n+1,1)$. We write $\mathbb {C}^{n+2} = \mathbb {C}^{n+1} \oplus \mathbb {C}$ and notice that $\mathfrak {{k}}_1$ preserves the two summands. Consequently, a protohomogeneous subspace $\mathfrak {{v}} \subseteq \mathbb {C}^{n+2}$ intersecting nontrivially with $\mathbb {C}^{n+1}$ (which is always the case when $\dim \mathfrak {{v}} \geqslant 3$) must lie there entirely, in which case admissibility cannot be achieved. So we only need to consider 2-dimensional subspaces transversal to $\mathbb {C}^{n+1}$. Take any such $\mathfrak {{v}}$. We can apply the same argument to its intersection with $\mathbb {C}\subseteq \mathbb {C}^{n+2}$, so it must be trivial as well. In other words, the projection $\textrm{pr}_{\mathbb {C}^{n+1}}(\mathfrak {{v}})$ is two-dimensional. Observe that this projection is protohomogeneous in $\mathbb {C}^{n+1}$ with respect to $\textrm{U}(n+1)$ and thus has constant Kähler angle $\upvarphi$. Again, according to [2, Proposition 7], we may assume

$$\begin{aligned} \mathfrak {{v}} = \langle v_1, v_2 \rangle _\mathbb {R}= \langle e_1 + ae_{n+2}, i\cos \upvarphi e_1 + i\sin \upvarphi e_2 + be_{n+2} \rangle _\mathbb {R}, \; a,b \ne 0. \end{aligned}$$

Moreover, acting by $\textrm{U}(1)$ in the last coordinate, we can take $a > 0$. Finally, since $K_1^0$ acts transitively on the unit circle in $\mathfrak {{v}}$, we must have $|b| = a$. Consequently, we can confine ourselves to working in $\mathbb {C}e_1 \oplus \mathbb {C}e_2 \oplus \mathbb {C}e_{n+2}$. Let us denote the corresponding coordinates by x, y,  and z, respectively. Note that $\textrm{Im}(b) \ne 0$, for otherwise we would have a nontrivial intersection $\mathfrak {{v}} \cap \mathbb {C}^{n+1}$. We see that $\mathfrak {{v}}$ is cut out by the equations

$$\begin{aligned} {\left\{ \begin{array}{ll} \textrm{Re}(y) = 0, \\ \textrm{Im}(x) \sin \upvarphi = \textrm{Im}(y) \cos \upvarphi , \\ \textrm{Im}(b) \textrm{Im}(x) = \textrm{Im}(z) \cos \upvarphi , \\ a \textrm{Re}(x) = \textrm{Re}(z) - \dfrac{\textrm{Re}(c)}{\textrm{Im}(c)}\textrm{Im}(z). \end{array}\right. } \end{aligned}$$

(5.1)

The subspace $\mathfrak {{v}}$ is admissible if and only if for each $w \in \mathbb {C}^{n+1}$ there exists

$$\begin{aligned} S = \left[ \begin{array}{c | c} \begin{array}{c c c} &{} &{} \\ &{} A &{} \\ &{} &{} \end{array} &{} w \\ \hline w^* &{} c \end{array} \right] \in \mathfrak {{u}}(n+1,1) \end{aligned}$$

preserving $\mathfrak {{v}}$. The fact that S lies in $\mathfrak {{u}}(n+1,1)$ means that $A \in \mathfrak {{u}}(n+1)$ and $\textrm{Re}(c) = 0$. Applying S to $(1,0,\ldots ,a)$ and $(i\cos \upvarphi ,i\sin \upvarphi ,0,\ldots ,b)$, we get vectors

$$\begin{aligned} \left[ \begin{gathered} a_{11} + aw_1 \\ a_{21} + aw_2 \\ \cdots \\ \overline{w}_1 + iac \end{gathered} \right] \quad \text {and} \quad \left[ \begin{gathered} ia_{11}\cos \upvarphi + ia_{12}\sin \upvarphi + w_1 b \\ ia_{21}\cos \upvarphi + ia_{22}\sin \upvarphi + w_2 b \\ \cdots \\ i\overline{w}_1 \cos \upvarphi + i\overline{w}_2 \sin \upvarphi + ibc \end{gathered} \right] . \end{aligned}$$

Note that $a_{12} = \overline{a}_{21}$ and $\textrm{Re}(a_{11}) = \textrm{Re}(a_{22}) = 0$. In order to lie in $\mathfrak {{v}}$, both $S(v_1)$ and $S(v_2)$ must satisfy equations (5.1) (and all their other coordinates should be 0, but we do not care about that), which altogether gives a system of 8 equations, and we are left with a rather daunting and tedious task of solving it. We are not going to write all these equations here; instead, let us label them (i) through (viii), where equations (i) to (iv) are for $S(v_1)$ substituted into (5.1) and (v) to (viii) are for $S(v_2)$ substituted there. Fortunately, there is no need to solve the entire system. Indeed, provided that $\upvarphi \ne 0$, use (i) to express $\textrm{Re}(a_{21})$ in terms of $\textrm{Re}(w_2)$ and plug the result into (vi) to get a linear dependency on the real and imaginary parts of $w_1$ and $w_2$. Therefore, such $\mathfrak {{v}}$ cannot be admissible.

Now let $\upvarphi = 0$, which just means that $\textrm{pr}_{\mathbb {C}^{n+1}}(\mathfrak {{v}})$ is a complex line. The first two equations in (5.1) then simply mean that $y = 0$. Assume first that $\textrm{Re}(b) \ne 0$. Use equations (iii) and (iv) to express $a_{11}$ and c in terms of $w_1$. Equation (vii) will tell us that $a \textrm{Im}(b) = 1$. Inserting it all into (viii) yields a linear dependency on $\textrm{Re}(w_1)$ and $\textrm{Im}(w_1)$, which implies that the coefficients in this dependency must be zero. But one can easily compute that the coefficient of $\textrm{Im}(w_1)$ is $(a-\frac{1}{a})^2 + \textrm{Re}(b)^2$, which cannot be zero. So we end up with the case $\textrm{Re}(b) = 0$. Equation (iv) yields $a = 1$ and thus $b = i$, so $\mathfrak {{v}}$ becomes the complex line spanned inside $\mathbb {C}^{n+2}$ by $e_1 + e_{n+2}$. It is not hard to verify that this subspace is admissible and protohomogeneous. Now we want to see what this $\mathfrak {{v}}$ looks like with respect to the restricted root space decomposition. To this end, we need to have an explicit identification between $\mathfrak {{m}}_1$ and $\mathfrak {{u}}(n+1,1)$ and also between $\mathfrak {{n}}_1^1$ and $\mathbb {C}^{n+2}$. We have:

$$\begin{aligned} \mathfrak {{m}}_1 = \left\{ \left. \left[ \begin{array}{c | c} \begin{array}{c c c} &{} &{} \\ &{} A &{} \\ &{} &{} \end{array} &{} \begin{array}{c c c} &{} &{} \\ 0 &{} 0 &{} u \\ &{} &{} \end{array} \\ \hline \begin{array}{c c c} &{} 0 &{} \\ &{} 0 &{} \\ &{} u^* &{} \end{array} &{} \begin{array}{c c c} ib &{} &{} \\ &{} ib &{} \\ &{} &{} ia \end{array} \end{array} \right] \right| A \in \mathfrak {{u}}(n+1), u \in \mathbb {C}^{n+1}, a,b \in \mathbb {R}, \textrm{tr}= 0 \right\} , \end{aligned}$$

where u and $u^*$ are regarded as column and row vectors, respectively. Similarly,

$$\begin{aligned} \mathfrak {{n}}_1^1 = \left\{ \left. \left[ \begin{array}{c | c} \begin{array}{c c c} &{} &{} \\ &{} 0 &{} \\ &{} &{} \end{array} &{} \begin{array}{c c c} &{} &{} \\ -v &{} v &{} 0 \\ &{} &{} \end{array} \\ \hline \begin{array}{c c c} &{} v^* &{} \\ &{} v^* &{} \\ &{} 0 &{} \end{array} &{} \begin{array}{c c c} 0 &{} 0 &{} -\overline{w} \\ 0 &{} 0 &{} -\overline{w} \\ -w &{} w &{} 0 \end{array} \end{array} \right] \right| v \in \mathbb {C}^{n+1}, w \in \mathbb {C}\right\} , \end{aligned}$$

where the top left zero block is of the size $(n+1) \times (n+1)$ and $\pm v$ and $v^*$ are again regarded as column and row vectors, respectively. Now, sending the matrix in the definition of $\mathfrak {{m}}_1$ to

$$\begin{aligned} \left[ \begin{array}{c | c} \begin{array}{c c c} &{} &{} \\ &{} A-ibE &{} \\ &{} &{} \end{array} &{} u \\ \hline u^* &{} i(a+b) \end{array} \right] \end{aligned}$$

gives a Lie algebra isomorphism $\mathfrak {{m}}_1 {\mathop {\sim }\limits ^{\rightarrow }}\mathfrak {{u}}(n+1,1)$. Similarly, sending the matrix in the definition of $\mathfrak {{n}}_1^1$ to (v, w) (here v is a row vector) gives an $\mathbb {R}$-linear isomorphism $\mathfrak {{n}}_1^1 {\mathop {\sim }\limits ^{\rightarrow }}\mathbb {C}^{n+2}$. Moreover, under these isomorphisms, the adjoint representation of $\mathfrak {{m}}_1$ on $\mathfrak {{n}}_1^1$ becomes the tautological representation of $\mathfrak {{u}}(n+1,1)$. Note, however, that the isomorphism $\mathfrak {{n}}_1^1 {\mathop {\sim }\limits ^{\rightarrow }}\mathbb {C}^{n+2}$ is not $\mathbb {C}$-linear with respect to the complex structure on $\mathfrak {{n}}_1^1$ induced from $\mathfrak {{a}} \oplus \mathfrak {{n}}$! It is $\mathbb {C}$-linear in v but $\mathbb {C}$-antilinear in w. This subtlety is important if we want to transfer our subspace $\mathfrak {{v}}$ from $\mathbb {C}^{n+2}$ to $\mathfrak {{n}}_1^1$: while it is a complex line in $\mathbb {C}^{n+2}$, it will become a totally real 2-dimensional subspace in $\mathfrak {{n}}_1^1$. In fact, if we modify $\mathfrak {{v}}$ slightly by an element from $\textrm{U}(n+1,1)$ to make it a complex line spanned by $(0,\ldots ,0,1,1)$, then the subspace of $\mathfrak {{n}}_1^1$ it corresponds to is precisely $\mathfrak {{g}}_{\upalpha _1}$, which can be easily seen from our description of the isomorphism above. We thus have a cohomogeneity-one action given by the Lie subgroup $H_{1,\mathfrak {{g}}_{\upalpha _1}}$, whose Lie algebra is

$$\begin{aligned} \mathfrak {{h}}_{1, \mathfrak {{g}}_{\upalpha _1}}&= N_{\mathfrak {{m}}_1}(\mathfrak {{n_1}} \ominus \mathfrak {{g}}_{\upalpha _1}) \oplus \mathfrak {{a}}^1 \oplus (\mathfrak {{n_1}} \ominus \mathfrak {{g}}_{\upalpha _1}) \\&= (\mathfrak {{k}}_0 \oplus \mathfrak {{a}}^1 \oplus \mathfrak {{g}}_{\upalpha _2} \oplus \mathfrak {{g}}_{2\upalpha _2}) \oplus \mathfrak {{a}}^1 \oplus (\mathfrak {{n_1}} \ominus \mathfrak {{g}}_{\upalpha _1}) \\&= \mathfrak {{k}}_0 \oplus \mathfrak {{a}} \oplus \mathfrak {{n}}_2 = \mathfrak {{h}}_{2,1}^\Uplambda , \end{aligned}$$

so $H_{1,\mathfrak {{g}}_{\upalpha _1}} = H_{2,1}^\Uplambda$, and this action was taken into account in (6). Altogether, we conclude that the nilpotent construction does not yield any new actions for M.