Nilpotent Lie Groups: Fourier Inversion and Prime Ideals

Abstract

We establish a Fourier inversion theorem for general connected, simply connected nilpotent Lie groups $G= \hbox {exp}({\mathfrak {g}})$ by showing that operator fields defined on suitable sub-manifolds of ${\mathfrak {g}}^*$ are images of Schwartz functions under the Fourier transform. As an application of this result, we provide a complete characterisation of a large class of invariant prime closed two-sided ideals of $L^1(G)$ as kernels of sets of irreducible representations of G.

1 Introduction

For a connected, simply connected, nilpotent Lie group G, the description of its spectrum and the Fourier inversion theorem are due to Kirillov [3], who showed that the dual space $\widehat{G}$ of G is in one-to-one correspondence with the space ${\mathfrak {g}}^*/G$ of co-adjoint orbits of G. R. Howe proved in [2] that for every irreducible unitary representation $(\pi ,{\mathcal H}_\pi )$ of G and every smooth linear operator a on ${\mathcal H}_\pi$ there exists a Schwartz function $f_a$ on G such that $\pi (f_a)= a$. He also showed that the mapping $a \mapsto f_a$ is linear and continuous with respect to the Fréchet topology of the space ${\mathcal B}^\infty ({\mathcal H}_{\pi })$ of smooth linear operators on ${\mathcal H}_\pi$ and the Fréchet topology on the space ${\mathcal S}(G)$ of Schwartz functions on G. In [12], N. Pedersen gave a precise construction on the map $a \mapsto f_a$, for a fixed representation, using the trace function.

In this paper, we study a general version of the Fourier inversion theorem for nilpotent Lie groups. More precisely, we generalise the result of Howe’s mentioned above by constructing a continuous retract from the space of adapted smooth kernel functions defined on a smooth G-invariant sub-manifold M of ${\mathfrak {g}}^*$ and supported in a subset $G\cdot {\mathcal M}$ of M, where ${\mathcal M}$ is a relatively compact open subset of M, into the space ${\mathcal S}(G)$. Note that since we work with various representations at the same time, we cannot apply the construction of Pedersen’s, unless the manifold is an open subset of ${\mathfrak {g}}^*$. The main difficulty with this retract construction is that the spectrum of nilpotent Lie groups is not Hausdorff, thus given a smooth operator field defined on a small (Hausdorff) region in the spectrum, one cannot take an extension as in the usual locally compact Hausdorff space case. We will apply the variable group techniques developed in [11] to prove this result, which we call the Retract Theorem, by induction on the length $\vert I\vert$ of the largest index set I for which $({\mathcal B}\times {\mathfrak {g}}^*)_I\cap M\ne \emptyset$ (defined in Sect. 2.4).

Once we have the Retract Theorem, we can apply it to study the $\mathbf{G}$-prime ideals of the Banach algebra $L^{1}{(G)}$. Here $\mathbf{G}$ denotes a Lie subgroup of the automorphism group of G with the property that the $\mathbf{G}$-orbits in ${\mathfrak {g}}^*$ are all locally closed. The Retract Theorem implies that the Schwartz functions contained in the kernel of a $\mathbf{G}$-orbit $\Omega$ in $\widehat{G}$ are dense in the $L^{1}{(G)}$-kernel of $\Omega$. Using the methods in [7], it follows that every $\mathbf{G}$-prime ideal in $L^{1}{(G)}$ is the kernel of such a $\mathbf{G}$-orbit $\Omega$. This result can be used, for instance, in the study of bounded irreducible representations $(\pi ,X)$ of a Lie group $\mathbf{G}$ on a Banach space X. Restricting the representation $\pi$ to the nilradical G, one obtains the kernel $\hbox {ker}(\pi _{\vert G})$ of $\pi _{\vert G}$ in the algebra $L^1(G)$. The ideal $\hbox {ker}(\pi _{\vert G})$ is then $\mathbf{G}$-prime. If $\hbox {ker}(\pi _{\vert G})$ is given as the kernel in $L^1(G)$ of a $\mathbf{G}$-orbit ${\mathbf{G}}\cdot \pi _0 \subset \widehat{G}$ for some $\pi _0 \in \widehat{G}$, then one can use $\pi _0$ to make an analysis of $\pi$ as Mackey did in the case of unitary representations.

Prime ideals in $L^1(G)$ have been studied by various authors in different settings. For connected, simply connected, nilpotent Lie groups, J. Ludwig showed in [6] that the closed prime ideals of $L^1( G )$ coincide with the kernels of the irreducible unitary representations. In 1984, D. Poguntke studied the action of an abelian compact group K on a nilpotent Lie group [13] and characterised the K-prime ideals as kernels of K-orbits. In [4], R. Lahiani and C. Molitor-Braun identified the K-prime ideals with hull contained in the generic part of the dual space of G for a general compact Lie subgroup K of the automorphism group of G. In [7] and [8], it was shown that for an exponential Lie group $\mathbf{G}$, the $\mathbf{G}$-prime ideals are also kernels of $\mathbf{G}$-orbits. In this way the bounded irreducible Banach space representations of an exponential Lie group could be determined.

The paper is organised in the following way: in Sect. 2 we recall the definition of induced representations and of kernel functions, we explain the notion of variable nilpotent Lie groups and their Lie algebras, of index sets for co-adjoint orbits and of adapted kernel functions on a G-invariant sub-manifold of ${\mathfrak {g}}^*$. In Sect. 3, we state our main theorem of the paper, the Retract Theorem, and in Sect. 4 we present the proof of the theorem, dividing it into several steps. As an application of the Retract Theorem, in the last section (Sect. 5) we show that every $\mathbf{G}$-prime ideal in $L^{1}{(G)}$ is the kernel of a $\mathbf{G}$-orbit.

2 Notations and Generalities

2.1 Representations and Kernel Functions

Let $G=\hbox {exp}({\mathfrak {g}})$ be a connected, simply connected, nilpotent Lie group and ${\mathfrak {g}}$ be its Lie algebra. All the irreducible unitary representations of G (and hence of $L^1(G)$) are obtained (up to equivalence) in the following way: Let $l\in {\mathfrak {g}}^*$ and ${\mathfrak {p}}={\mathfrak {p}}(l)$ be an arbitrary polarisation of l in ${\mathfrak {g}}$ (a maximal isotropic subalgebra of ${\mathfrak {g}}$ for the bilinear form $(X,Y)\mapsto \langle l,[X,Y]\rangle$). Let $P(l)=\hbox {exp}({\mathfrak {p}}(l))$. The induced representation denoted by $\pi _l:= \text {ind}^G_{P(l)}\chi _l$ on the Hilbert space $\mathfrak {H}_l$,

$$\begin{aligned} \mathfrak {H}_l=L^2(G/P(l), \chi _l):= & {} \{\xi :G\rightarrow {\mathbb C}; \, \xi \text { measurable}, \\&\xi (gp)=\chi _l(p^{-1})\xi (g), g\in G, p\in P(l),\\ \nonumber&\Vert \xi \Vert _{2} ^2= \int _{G/P(l)}\vert \xi (g)\vert ^2d\dot{g} <\infty \}, \end{aligned}$$

where $d\dot{g}$ is the invariant measure on G / P(l), is unitary and irreducible. Here $\chi _l$ is the character defined on P(l) by $\chi _l(g) = e^{-i \langle l,\log g \rangle }$ for all $g\in P(l)$. Two different polarisations for the same l give equivalent representations. The same is true for the case of two linear forms l and $l'$ belonging to the same co-adjoint orbit.

One particular way to obtain a polarisation is the following: Let $\{Z_1, \dots , Z_n\}$ denote a Jordan-Hölder basis of ${\mathfrak {g}}$, for $1\le k\le n$, let ${\mathfrak {g}}_k:=\text {span}\{Z_k, \dots , Z_n\}$ be the linear span of $Z_k, \dots , Z_n$ and $l_k = l\vert _{{\mathfrak {g}}_k}$ for all $l\in {\mathfrak {g}}^*$. The polarisation ${\mathfrak {p}}(l)_{\mathcal Z}={\mathfrak {p}}(l):=\sum _{k=1}^n {\mathfrak {g}}_k(l_k)$ of l in ${\mathfrak {g}}$, with ${\mathfrak {g}}_k(l_k):= \{U\in {\mathfrak {g}}_k; \langle l,[U,{\mathfrak {g}}_k] \rangle \equiv 0\}$, is called the Vergne polarisation at l with respect to the basis $Z_1, \dots , Z_n$. We refer to [1] for more details on the theory of irreducible representations of nilpotent Lie groups.

Let $\pi _l=\text {ind}_{P(l)}^G \chi _l$. The corresponding representation of $L^1(G)$, also denoted by $\pi _l$, is obtained via the formula $\pi _l(f)\xi := \int _G f(x) \big (\pi _l(x)\xi \big )dx$, for all $\xi \in \mathfrak {H}_l$. If $f\in {L^1(G)}$, then $\pi _l(f)$ is a kernel operator, i.e. it is of the form

$$\begin{aligned} \big (\pi _l(f)\xi \big )(g) = \int _{G/P(l)} F(l,g,u) \xi (u)du, \end{aligned}$$

where F is the operator kernel given by

$$\begin{aligned} F(l,g,u)=\int _{P(l)}f(ghu^{-1})\chi _l(h)dh \quad \text {for } \, g,u\in G. \end{aligned}$$

If f is a Schwartz function, then the kernel function F belongs to ${\mathcal C}^{\infty }$ and satisfies the covariance relation

$$\begin{aligned} F(l, gh, g'h')= \overline{\chi _l(h)} \chi _l(h') F(l, g, g') \quad \text {for } \, h,h' \in P(l) \text { and } g,g' \in G, \end{aligned}$$

and is a Schwartz function on $G/P(l) \times G/P(l)$.

2.2 Group Actions

Let $G=\hbox {exp}({\mathfrak {g}})$ be a connected, simply connected, nilpotent Lie group and A be a Lie subgroup of the automorphism group $\text {Aut}(G)$ of G acting smoothly on G. This action will be denoted by

$$\begin{aligned} \begin{array}{ccc} \nonumber A \times G &{}\mapsto &{}G\\ \nonumber (a,x) &{}\mapsto &{}a\cdot x. \end{array} \end{aligned}$$

The action of A on G induces naturally actions of A on ${\mathfrak {g}}$, ${\mathfrak {g}}^*$, $\widehat{G}$, $L^1(G)$, and on ${\mathcal S}(G)$. These group actions will lead to examples for our retract theory and provide an important application of retracts.

2.3 Variable Lie Algebras and Groups

We will prove our main theorem by induction; in our proofs, new parameters and new variations will appear. This may be handled most easily by the concept of variable Lie structures. Such structures were already considered in [5, 11, 10] and [9], among others.

Definition 2.3.1

• 1 Let ${\mathfrak {g}}$ be a real vector space of finite dimension n and ${\mathcal B}$ be an arbitrary nonempty set. We say that $( {\mathcal B},{\mathfrak {g}})$ is a variable (nilpotent) Lie algebra if

  • (a) For every $\beta \in {\mathcal B}$, there exists a Lie bracket $[ \cdot , \cdot ]_{\beta }$ defined on ${\mathfrak {g}}$ such that ${\mathfrak {g}}_{\beta }:=({\mathfrak {g}}, [ \cdot , \cdot ]_{\beta })$ is a nilpotent Lie algebra.

  • (b) There exists a fixed basis ${\mathcal Z}={\mathcal Z}^0=\{ Z_1=Z_1^0, \dots , Z_n=Z_n^0\}$ of ${\mathfrak {g}}$ such that the structure constants $a^k_{ij}(\beta )$ defined by

    $$\begin{aligned}{}[Z_i, Z_j]_{\beta } := \sum _{k=1}^n a^k_{ij}(\beta ) Z_k \end{aligned}$$

    satisfy the following property: For all $\beta \in {\mathcal B}$ and $k\le \max \{i, j\}$, $a^k_{ij}(\beta )=0$. This means that $\{Z_1, \dots , Z_n\}$ is a Jordan-Hölder basis for ${\mathfrak {g}}_{\beta }=({\mathfrak {g}}, [\cdot , \cdot ]_{\beta })$.

• 2 Assume that ${\mathcal B}$ is a smooth manifold. If the structure constants $a^k_{ij}(\beta )$ vary smoothly on ${\mathcal B}$, we say that $({\mathcal B}, {\mathfrak {g}})$ is a smooth variable (nilpotent) Lie algebra.

We will denote $({\mathcal B}, {\mathfrak {g}}) = ({\mathfrak {g}}, [\cdot , \cdot ]_{\beta })_{\beta \in {\mathcal B}}$ for the variable Lie algebra.

For the rest of the paper we will assume that all variable Lie algebras are smooth. If ${\mathcal B}$ is reduced to a singleton, we have in fact no dependency on $\beta$ in ${\mathcal B}$ but a fixed Lie algebra. To each variable Lie algebra, we associate a variable Lie group $G_{\beta }$. The variable Lie group ${\mathbb G}:=(G_{\beta })_{\beta }$ may be identified with the collection of Lie algebras $({\mathfrak {g}}, [\cdot , \cdot ]_{\beta })_{\beta }$ equipped with the corresponding Campbell-Baker-Hausdorff multiplications. If ${\mathbb G}=(G_{\beta })_{\beta }$ is a (smooth) variable Lie group endowed with a fixed Jordan-Hölder basis, then the corresponding Vergne polarisations, induced representations and operator kernels all depend on $\beta \in {\mathcal B}$ and $l\in {\mathfrak {g}}^*$.

2.4 Ludwig-Zahir Indices

Let $({\mathcal B}, {\mathfrak {g}})$ be a smooth variable Lie algebra. We assume that ${\mathfrak {g}}$ is equipped with a fixed basis ${\mathcal Z}={\mathcal Z}^0=\{Z_1=Z_1^0, \dots , Z_n=Z_n^0\}$, which is a Jordan-Hölder basis for every $({\mathfrak {g}}, [\cdot , \cdot ]_{\beta })$.

Let $(\beta , l)\in {\mathcal B} \times {\mathfrak {g}}^*$. The Ludwig-Zahir indices $I(\beta , l)$ defined in [11] can be obtained in the following way: Let ${\mathfrak {g}}_{\beta }(l):= \{ U\in {\mathfrak {g}}; \langle l,[U,{\mathfrak {g}}]_{\beta } \rangle \equiv 0\}$ be the stabiliser of l in ${\mathfrak {g}}_{\beta }=({\mathfrak {g}}, [\cdot , \cdot ]_{\beta })$ and let ${\mathfrak {a}}_{\beta }(l)$ be the maximal ideal contained in ${\mathfrak {g}}_{\beta }(l)$. If ${\mathfrak {a}}_{\beta }(l)={\mathfrak {g}}_{\beta }(l)={\mathfrak {g}}$, then $\chi _{(\beta ,l)}(x):= e^{-i \langle l, \log _{\beta }x \rangle }$ is a character on $G_{\beta }$ and nothing has to be done. In this case, there are no Ludwig-Zahir indices, i.e. $I(\beta , l)=\emptyset$. Otherwise, let

$$\begin{aligned} j_1(\beta ,l)= & {} \max \{j\in \{1, \dots ,n\};~Z_j^0 \not \in {\mathfrak {a}}_{\beta }(l)\}, \, \text { and}\\ k_1(\beta ,l)= & {} \max \{ k\in \{1, \dots , n\};~\langle l, [Z^0_{j_1(\beta ,l)}, Z^0_k]_{\beta }\rangle \ne 0\}. \end{aligned}$$

We let

$$\begin{aligned} X_1(\beta ,l)&:&=Z^0_{k_1(\beta ,l)}, \\ Y_1(\beta , l)&:&=Z^0_{j_1(\beta ,l)}, \\ Z_1(\beta , l)&:&= [Z^0_{k_1(\beta , l)},Z^0_{j_1(\beta ,l)}] _\beta , \, \text { and} \\ c(\beta ,l)&:&=\langle {l},{Z_1(\beta ,l)}\rangle . \end{aligned}$$

We then consider

$$\begin{aligned} {\mathfrak {g}}_1(\beta ,l):= \{U \in {\mathfrak {g}}; ~\langle l,[U,Y_1(\beta ,l)]_\beta \rangle =0\} \end{aligned}$$

(2.1)

which is an ideal of co-dimension one in ${\mathfrak {g}}_{\beta }$.

A Jordan-Hölder basis of $({\mathfrak {g}}_1(\beta ,l), [\cdot , \cdot ]_{\beta })$ is given by ${\mathcal Z}^1(\beta ,l)=\{ Z^1_i(\beta ,l);~i \ne k_1(\beta ,l)\}$ with

$$\begin{aligned} Z^1_i(\beta ,l) := Z_i^0 -\frac{ {\langle l,[Z^0_i, Y_1(\beta ,l)]_{\beta }\rangle }}{c(\beta ,l)} X_1{(\beta ,l)}, \quad i\ne k_1(\beta ,l). \end{aligned}$$

(2.2)

One sees that $Z^1_i(\beta ,l) = Z^0_i$, if $i>k_1(\beta ,l)$. As previously we may now compute the indices $j_2(\beta ,l), k_2(\beta ,l)$ of $l_1:=l\vert _{{\mathfrak {g}}_1(\beta ,l)}$ with respect to this new basis and construct the corresponding subalgebra ${\mathfrak {g}}_2(\beta ,l)$ with its associated basis $\{Z^2_i(\beta ,l); ~i \ne k_1(\beta ,l), k_2(\beta ,l)\}$. This procedure stops after a finite number r of steps. Let

$$\begin{aligned} I_{\mathcal Z}(\beta ,l) = I(\beta , l)=\big ((j_1(\beta ,l), k_1(\beta ,l)), \dots , (j_r(\beta ,l), k_r(\beta ,l)) \big ), \end{aligned}$$

which is called the Ludwig-Zahir index of l in ${\mathfrak {g}}_{\beta }$ with respect to the basis $\{Z_1, \dots , Z_n\}$. The construction in [11] shows that the final subalgebra ${\mathfrak {g}}_r(\beta ,l)$ obtained by this construction coincides with the Vergne polarisation of l in ${\mathfrak {g}}_{\beta }$ with respect to the basis ${\mathcal Z}^0$ (see also [9, 10]). Note that the length $\vert I\vert =2r$ of the index set $I= I(\beta , l)$ gives us the dimension of the co-adjoint orbit $\mathrm{Ad}^*(G_\beta )\ell$. The vectors $Y_1(\beta ,l),\cdots , Y_r(\beta ,l)$ together with the stabiliser ${\mathfrak {g}}_\beta (l)$ of l in ${\mathfrak {g}}_\beta$ span the polarisation ${\mathfrak {p}}_\beta (l)={\mathfrak {g}}_r(\beta ,l)$ and

$$\begin{aligned} {\mathfrak {g}}=\oplus _{i=1}^r {\mathbb R}X_i(\beta ,l)\oplus _{i=1}^r {\mathbb R}Y_i(\beta ,l)\oplus {\mathfrak {g}}_\beta (l). \end{aligned}$$

Let us introduce the following notations: For any index set $I\in ({\mathbb N}^2)^r \equiv {\mathbb N}^{2r}$ with $r=0,\cdots , \mathrm{dim}({\mathfrak {g}})/2$, we let

$$\begin{aligned}&({\mathcal B}\times {\mathfrak {g}}^*)_I:=\{(\beta ,l)\in {\mathcal B}\times {\mathfrak {g}}^*; I(\beta ,l)=I\} \quad \text { and }\\&({\mathcal B}\times {\mathfrak {g}}^*)_I \cap \Sigma _I := \{(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_I; \, l(Z_{j_i})=l(Z_{k_i})=0 \text { for } 1 \le i \le r\}. \end{aligned}$$

This last line corresponds to the Pukanszky section associated to the index I. In fact, in [9] it was proved that the indices $j_s(\beta ,l), k_s(\beta ,l)$ coincide with the Pukanszky indices of the given layer (if one does not make any distinction between the j’s and the k’s). For many I’s, the subset $({\mathcal B}\times {\mathfrak {g}}^*)_I$ is empty. Hence it is reasonable to define

$$\begin{aligned} {\mathcal I}:=\Big \{I\in \bigcup _{j=0}^{\mathrm{dim}({\mathfrak {g}})/2} ({\mathbb N}^2)^{j}; \, ({\mathcal B}\times {\mathfrak {g}}^*)_I\ne \emptyset \Big \} \quad \text {and} \quad {\mathcal B}\times {\mathfrak {g}}^*= \dot{\bigcup }_{I\in {\mathcal I}}({\mathcal B}\times {\mathfrak {g}}^*)_I. \end{aligned}$$

This gives a partition of ${\mathcal B}\times {\mathfrak {g}}^*$ into the different layers $({\mathcal B}\times {\mathfrak {g}}^*)_I$. The set ${\mathcal I}$ may be ordered lexicographically: if $I=\{(j_1,k_1),\cdots , (j_r,k_r)\},I'=\{(j_1',k_1'),\cdots , (j_{r'}',k_{r'}')\}\in {\mathcal I}$, we say that $I<I'$ if either $2r=\vert I\vert < \vert I'\vert = 2r'$ or there exists $a\in \{ 1, \dots , r\}$ such that

$$\begin{aligned} ( j_s, k_s) = (j_s', k_s') \text { if } s<a \text { and } ( j_a, k_a) < (j_a', k_a'), \end{aligned}$$

which means that

$$\begin{aligned} \text {either } j_a< j_a' \text { or } (j_a = j_a' \text { and } k_a < k_a'). \end{aligned}$$

This allows us to define

$$\begin{aligned} ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I} := \{(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_J; \,J \le I\}= \bigcup _{J\le I}({\mathcal B}\times {\mathfrak {g}}^*)_J. \end{aligned}$$

By induction on the length of the index sets, it is easy to see that for every $I\in {\mathcal I}$ there exists a smooth function $P_I$ on ${\mathcal B}\times {\mathfrak {g}}^*$, which is polynomial in l for fixed $\beta \in {\mathcal B}$ such that

$$\begin{aligned} ({\mathcal B}\times {\mathfrak {g}}^*)_I=\{(\beta ,l); P_{I'}(\beta ,l)=0 \text { for }I'>I \text { and } P_I(\beta ,l)\ne 0\}. \end{aligned}$$

(2.3)

2.5 Co-adjoint Orbits

For any index set I, we consider the subspace ${\mathfrak {s}}_I$ of ${\mathfrak {g}}^*$ which is given by

$$\begin{aligned} {\mathfrak {s}}_I=\text {span} \{Z_j^*;~ j\in I\}. \end{aligned}$$

For each $\beta \in {\mathcal B}$, let

$$\begin{aligned} \Sigma _{\beta ,I}:=\{(\beta ,l)\in (\{\beta \}\times {\mathfrak {g}}^*)_I;~ l\in {\mathfrak {s}}_I \}. \end{aligned}$$

Then $\Sigma _{\beta ,I}$ is locally closed in ${\mathfrak {s}}_I$, since we have the smooth functions $P_{I'},I'\in {\mathcal I}$, defined on ${\mathcal B}\times {\mathfrak {g}}^*$ as in (2.3).

Let $d:=\vert I\vert$. For $l\in {\mathfrak {g}}^*$, let

$$\begin{aligned} \Omega _{\beta ,l}=\{Ad_\beta ^*(g)l; \, g\in G\} \end{aligned}$$

be the $G_\beta$-orbit of l. Then

$$\begin{aligned} \mathrm{dim}(\Omega _{\beta ,l})=d \quad \text {for} \quad l\in ({\mathcal B}\times {\mathfrak {g}}^*)_I. \end{aligned}$$

There exist functions $p_j:({\mathcal B}\times {\mathfrak {g}}^*)_I\times {\mathbb R}^d\rightarrow {\mathbb R}, j=1,\cdots , n$, which are rational in $l\in {\mathfrak {g}}^*$ and polynomial in $z\in {\mathbb R}^d$ for fixed $\beta \in {\mathcal B}$ such that for every $(\beta , l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_I$,

$$\begin{aligned} \Omega _{\beta ,l}=\Big \{\sum _{i=1}^n p_{i}(\beta ,l,z) Z_i^*; \, z\in {\mathbb R}^d\Big \}. \end{aligned}$$

Furthermore if we write $I=\{i_1<\cdots <i_d\}$, then

$$\begin{aligned} p_{i_j}(\beta ,l,z)= z_j \quad \text {for} \, \ j=1,\cdots ,d, \end{aligned}$$

and for $i\not \in I$, we have

$$\begin{aligned} p_{i}(\beta ,l,z)= \langle l ,Z_i\rangle +p'_i(\beta ,l,z_1,\cdots , z_j), \quad i_j<i<i_{j+1}. \end{aligned}$$

Definition 2.5.1

A subset M of ${\mathcal B}\times {\mathfrak {g}}^*$ is called G-invariant if for every $(\beta ,l)\in M$ the element $g\cdot (\beta ,l):=(\beta , \mathrm{Ad}^*_\beta (g)l)$ is also contained in M.

2.6 Schwartz Functions

Let $r \in {\mathbb N}$, we define the space of (generalised) Schwartz functions ${\mathcal S}( {\mathbb R}^r, {\mathcal B}, G) \equiv {\mathcal S}( {\mathbb R}^r, {\mathcal B}, {\mathfrak {g}}) \equiv {\mathcal S}( {\mathbb R}^r, {\mathcal B}, {\mathbb R}^n)$ to be the set of all functions f from ${\mathbb R}^r \times {\mathcal B} \times G$ to ${\mathbb C}$ such that the function $\tilde{f}$ defined by

$$\begin{aligned} \tilde{f}(\alpha ,\beta ,(x_1, \dots , x_n)):= f(\alpha ,\beta , \hbox {exp}_{\beta }(x_1Z_1+ \cdots + x_nZ_n)) \quad \text {for } \, \alpha \in {\mathbb R}^r, \beta \in {\mathcal B}\end{aligned}$$

is smooth on ${\mathbb R}^r \times {{\mathcal B}} \times {\mathbb R}^n$ and that

$$\begin{aligned} \Vert \tilde{f}\Vert _{K, T_1, \dots , T_s, A_1, A_2, B_1, B_2}= & {} \sup _{\beta \in K; \alpha \in {\mathbb R}^r; x\in {\mathbb R}^n} \Big [\sup _{\vert r_i\vert \le A_i; \vert s_j\vert \le B_j; i,j\in \{1,2\}} \vert \alpha ^{r_1} x^{s_1} \\&{ }&\quad \quad \quad \quad T_1 T_2 \cdots T_s \frac{\partial ^{r_2}}{\partial \alpha ^{r_2}} \frac{\partial ^{s_2}}{\partial x^{s_2}} \tilde{f}(\alpha ,\beta , (x_1, \dots , x_n)) \vert \Big ] \\ \nonumber< & {} + \infty , \end{aligned}$$

for any compact subset K of ${{\mathcal B}}$, any finite collection $T_1, \dots , T_s$ of smooth vector fields defined on the manifold ${{\mathcal B}}$, and any $A_1, A_2, B_1, B_2 \in {\mathbb N}$. The function space ${\mathcal S}({\mathbb R}^r, {\mathcal B}, G)$ is equipped with the topology defined by the collection of all these semi-norms. One may of course also use coordinates of the second kind to define the semi-norms on ${\mathcal S}({\mathbb R}^r, {\mathcal B}, G)$. Note that the space ${\mathcal S}({\mathbb R}^r, {\mathcal B}, G)$ does not depend on the choice of the Jordan-Hölder basis.

2.7 Kernel Functions

Let S be a subset of ${\mathcal B}\times {\mathfrak {g}}^*$ and L be a smooth manifold. We say that a mapping $F:S\rightarrow L$ is smooth, if the restriction of F to any smooth manifold N contained in S is smooth.

Let ${\mathcal B}\times {\mathfrak {g}}^*$ be a smooth variable nilpotent Lie group with Jordan-Hölder basis ${\mathcal Z}$. For any $(\beta ,l)\in {\mathcal B}\times {\mathfrak {g}}^*$, denote the Vergne polarisation ${\mathfrak {p}}_{\beta ,l}$ at $(\beta ,l)$ associated to ${\mathcal Z}$. We put $\pi (\beta ,l):=\text {ind}^G_{P(\beta ,l)} \chi _l$, with $P(\beta ,l):=\text {exp}_{\beta } {\mathfrak {p}}(\beta ,l)$, for the corresponding family of induced unitary representations. Then the mapping $(\beta ,l)\mapsto {\mathfrak {p}}(\beta ,l)$ is smooth on each subset $({\mathcal B}\times {\mathfrak {g}}^*)_I$. For each index set I with length $d_I$ and $(\beta ,l)\in {\mathcal B}\times {\mathfrak {g}}^*$, choose a Malcev basis $R(\beta ,l)=\{R_1(\beta ,l),\cdots , R_{d_I}(\beta ,l) \}$ of ${\mathfrak {g}}$ relative to ${\mathfrak {p}}(\beta ,l)$, such that the mappings $(\beta ,l)\mapsto R(\beta ,l)$ are smooth on the different layers $({\mathcal B}\times {\mathfrak {g}}^* )_I$.

Definition 2.7.1

Let M be any smooth G-invariant manifold of ${\mathcal B}\times {\mathfrak {g}}^*$ and let $r\in {\mathbb N}$. We denote by ${\mathcal D}^c_{M,r}$ the space of all functions $F: {\mathbb R}^r \times { M} \times G \times G \rightarrow {\mathbb C}$ satisfying the following conditions.

• 1. F satisfies the covariance condition for every $(\beta ,l)\in M$ with respect to ${\mathfrak {p}}(\beta ,l)$, i.e.

  $$\begin{aligned} F(\alpha , (\beta ,l), x \cdot _{\beta }p, y\cdot _{\beta }q) = \overline{\chi _l(p)} \chi _l(q) F(\alpha ,(\beta ,l), x,y)) \end{aligned}$$

  for all $\alpha \in {\mathbb R}^r$, $p,q \in P(\beta ,l)$ and $x,y\in G$.

• 2. The function F satisfies the following compatibility condition

  $$\begin{aligned} F(\alpha , (\beta , \text {Ad}^*_{\beta }(g)l), x,y)=F(\alpha , (\beta ,l), x\cdot _{\beta } g, y \cdot _{\beta } g), \end{aligned}$$

  for $\alpha \in {\mathbb R}^r, (\beta ,l)\in M$ and $x,y,g\in G$. This compatibility condition reflects the unitary equivalence of the representations $\pi _{(\beta ,l)}$ and $\pi _{(\beta , \text {Ad}^*_{\beta }(g)l)}$.

• 3. The support of F in $(\beta ,l)$ is compact modulo G, i.e. there exists a compact subset C of M such that $F(\cdot , (\beta ,l), \cdot , \cdot )$ is 0 outside the subset of $G\cdot {C}$.

• 4. The function F has the Schwartz space property, i.e. for any $I\in {\mathcal I}$ the function $F_{\vert {\mathbb R}^r\times M\cap ({\mathcal B}\times {\mathfrak {s}}_I)\times G\times G }$ is smooth and that

  $$\begin{aligned} \Vert F \Vert _{ D, A_1, A_2, B_1, B_2, C_1, C_2}:= & {} \sup _{(\beta ,l) \in M, \alpha \in {\mathbb R}^r, x,x'\in {\mathbb R}^r} \Big [ \sup _{ \vert r_i\vert \le A_i, \vert s_j\vert \le B_j, \vert t_k\vert \le C_k; i, j, k \in \{1,2\}} \vert \alpha ^{r_1} x^{s_1} (x')^{t_1} \nonumber \\&{ }&D_{(\beta ,l)} \frac{\partial ^{r_2}}{\partial \alpha ^{r_2}} \frac{\partial ^{s_2}}{\partial x^{s_2}} \frac{\partial ^{t_2}}{\partial (x')^{t_2}} \tilde{F}(\alpha , (\beta ,l), x, x')\vert \Big ]< \infty , \end{aligned}$$

  (2.4)

  where

  $$\begin{aligned} \tilde{F}(\alpha , (\beta , l), x, x'):= & {} F(\alpha , (\beta ,l), \hbox {exp}_{\beta }(x_1R_1) \cdots \hbox {exp}_{\beta }(x_rR_r), \hbox {exp}_{\beta }(x'_1R_1)\\&\cdots \hbox {exp}_{\beta }(x'_rR_r)), \end{aligned}$$

  for any smooth differential operator $D=D_{(\beta ,l)}$ on the manifold M, and any $A_1, A_2$, $B_1, B_2$, $C_1, C_2 \in {\mathbb N}$.

The space ${\mathcal D}^c_{M,r}$ will be equipped with the topology defined by the collection of all these semi-norms. This does of course not depend on the choice of the smooth Malcev basis of ${\mathfrak {g}}$ with respect to the smooth family of Vergne polarisations.

Definition 2.7.2

Let $M\subset {\mathcal B}\times {\mathfrak {g}}^*$. A field $F=(F(\beta ,l))_{{(\beta ,l)}\in M}$ of kernel functions is called adapted if it satisfies the conditions in Definition 2.7.1.

For an adapted field of kernel functions F on M, denote by $op_F$ the field of smooth operators defined through their kernel functions. For $(\beta ,l)\in M$, the operator $op_{F(\beta ,l)}$ acts on the space $L^2(G/P(\beta ,l),\chi _{(\beta ,l)})$ in the following way:

$$\begin{aligned} op_{F(\beta ,l)}\xi (g)=\int _{G/P(\beta ,l)}F(\beta ,l)(g,x)\xi (x)d\dot{x}. \end{aligned}$$

Remarks 2.7.2.1

• a) If we impose the condition that the support of $(\beta ,l)$ be contained in the set $G\cdot C_0$ for a fixed subset $C_0$ of M, then we will denote the space of kernel functions by ${\mathcal D}^{C_0}_{M}$.

• b) One has a similar definition of the kernel functions if one takes another smooth family of polarisations together with a smooth family of Malcev bases.

3 The Retract Theorem

In this section, we state the main theorem of the paper which will be proved in the next section.

Theorem 3.1

Let ${\mathcal B}\times G$ be a smooth variable nilpotent Lie group, $I=\{(j_1,k_1)< \cdots < (j_r, k_r)\}$ be an index set and let M be a smooth G-invariant sub-manifold of ${\mathcal B}\times {\mathfrak {g}}^*$ contained in $({\mathcal B}\times {\mathfrak {g}}^*)_{\le I}$ such that $M_I:=M\cap ({\mathcal B}\times {\mathfrak {g}}^*)_{I}\ne \emptyset$. Let $\pi (\beta ,l)$ be defined as previously from the smooth family of Vergne polarisations for $(\beta ,l) \in M$. Then there exists an open nonempty relatively compact subset ${\mathcal M}\subset M_I$ with closure $\overline{\mathcal M}$ contained in $M_I$ such that the following holds: For any adapted kernel function $F \in {\mathcal D}^{\mathcal M}_{M}$, there is a function f in the Schwartz space ${\mathcal S}({\mathbb R}^r,{\mathcal B}, G)$ such that $\pi _{(\beta ,l)}(f(\alpha ,\beta ,\cdot ))$ has $F(\alpha , (\beta ,l), \cdot , \cdot )$ as an operator kernel for all $(\alpha , (\beta ,l))\in {\mathbb R}^r \times { M}$. Moreover the mapping $F\mapsto f$ is continuous with respect to the corresponding function space topologies.

If the variation is trivial, then we get the following theorem.

Theorem 3.2

Let ${\mathfrak {g}}$ be a nilpotent Lie algebra with Jordan-Hölder basis ${\mathcal Z}$. Let M be a smooth G-invariant sub-manifold of ${\mathfrak {g}}^*$ and $I:=\max \{J\in {\mathcal I}_{\mathcal Z}: M\cap {\mathfrak {g}}^*_J\ne \emptyset \}$. Let $\pi _l=\pi (l)$ be defined as previously from the smooth family of Vergne polarisations for $l\in M$. Then there exists an open, relatively compact nonempty subset ${\mathcal M}\subset {\mathfrak {g}}^*_{I}$ of M such that ${\mathcal M} \subset \overline{\mathcal M}\subset M_I$, $\overline{\mathcal M}$ is compact and that the following holds: For any kernel function $F \in {\mathcal D}^{\mathcal M}_{M}$, there is a function f in the Schwartz space ${\mathcal S}(G)$ such that $\pi _{l}(f)$ has $F( l, \cdot , \cdot )$ as an operator kernel for all $l \in {M}$. Moreover, the Schwartz function f may be constructed such that the mapping $F\mapsto f$ is continuous with respect to the corresponding function space topologies.

Remark 3.2.1

If M is contained in ${\mathfrak {g}}^*_{I_{max}}$, where $I_{max}$ is the maximal index set in ${\mathcal I}$, then we have the following (well known) result.

Theorem 3.3

Let ${\mathcal B}\times G$ be a simply connected, connected smooth variable nilpotent Lie group and $M=({\mathcal B}\times {\mathfrak {g}}^*)_{gen}:=({\mathcal B}\times {\mathfrak {g}}^*)_{I_{max}}$ be the space of generic co-adjoint orbits. Let ${\mathcal M}$ be an open relatively compact subset of M such that $\overline{{\mathcal M}}\subset M.$ For every adapted field of kernel functions $F\in {\mathcal D}^{\mathcal M}_M$, there exists a unique Schwartz function $f=R(F):G\rightarrow {\mathbb C}$ such that

$$\begin{aligned} \pi _{(\beta ,l)}(f)=op_{F(\beta ,l)} \, \text { for any } \, (\beta ,l)\in {\mathcal B}\times {\mathfrak {g}}^*, \end{aligned}$$

and the mapping $F\mapsto R(F)$ is continuous.

Proof

It suffices to apply the Fourier inversion formula. For each $F\in {\mathcal D}^{\mathcal M}_M$, let

$$\begin{aligned} f(\beta ,g)=R(F)(\beta ,g):=\int _{\Sigma _{\beta ,I_{max}}}{\text{ t }r}(\pi _{(\beta ,l)}(g)\circ op_{F(\beta ,l)})\vert P_a(\beta ,l)\vert dl,\, g\in G, \end{aligned}$$

where $P_a(\beta ,l)$ is the Pfaffian of the polynomial $Q(l)=\det {(\langle l,[Z_i,Z_j]_\beta \rangle _{i,j\in I_{max}}})$. It follows from [11] that the function f is Schwartz and the Fourier inversion theorem tells us that $\pi _{(\beta ,l)}(f)=op_{F(\beta ,l)}$ for any $(\beta ,l)\in {\mathcal B}\times {\mathfrak {g}}^*$. $\square$

4 Proof of the Retract Theorem

The proof of Theorem 3.1 proceeds by induction on the length $\vert I\vert$ of the largest index set I for which $({\mathcal B}\times {\mathfrak {g}}^*)_I\cap M\ne \emptyset$ and it will be done in several steps.

4.1 The case $I=\emptyset$

Suppose that all the elements $(\beta ,l)\in M$ are characters of ${\mathfrak {g}}_\beta$, which means that their index sets are empty.

Let us replace the variable group $({\mathcal B},G)$ by the group $({\mathcal C},G)$, where ${\mathcal C}={\mathcal B}$ as a manifold, and the multiplications coming from ${\mathcal C}$ are abelian, i.e. $[U,V]_\gamma =0$ for every $U,V\in {\mathfrak {g}}$ and $\gamma \in {\mathcal C}$. We identify now the group G with its Lie algebra and then $U\cdot _\gamma V=U+V$ for every $U,V\in {\mathfrak {g}}$ and $\gamma \in {\mathcal C}$. This also means that $\chi _l$ is a character on $G_{\gamma }= \text {exp}_{\gamma } {\mathfrak {g}}$, for all $(\gamma ,l) \in {\mathcal C}\times {\mathfrak {g}}^*$. Now take ${\mathcal M}=M$. Let $F \in {\mathcal S}({\mathbb R}^r\times M)$ be a kernel function with compact support in the variables $(\gamma ,l)$. As ${\mathbb R}^r \times M$ is a sub-manifold of ${\mathbb R}^r \times {\mathcal C}\times {\mathfrak {g}}^*$, the function F may be extended to a Schwartz function $\widetilde{F}$ (in the sense of Sect. 2.6 and 2.7) on ${\mathbb R}^r \times {\mathcal C}\times {\mathfrak {g}}^*$ with compact support in the variables $(\gamma ,l)$. Let $f := (2\pi )^n{\mathcal F}^{-1}_3 \widetilde{F}$, where ${\mathcal F}^{-1}_3$ denotes the partial inverse Fourier transform in the variable l which is the third variable in ${\mathbb R}^r \times {\mathcal C}\times {\mathfrak {g}}^*$. Then $f \in {\mathcal S}({\mathbb R}^r \times {\mathcal C}\times {\mathfrak {g}}^*)$. For all $(\alpha ,(\gamma ,l)) \in {\mathbb R}^r \times {M}$, we have

$$\begin{aligned} \pi _{(\gamma ,l)} \big ( f(\alpha ,\gamma , \cdot ) \big )= & {} \widehat{f}^{\,\,3}(\alpha , \gamma ,l) \\= & {} (2\pi )^n {\mathcal F}_3 {\mathcal F}_3^{-1} F(\alpha ,(\gamma , l))\\= & {} F(\alpha , (\gamma ,l)). \end{aligned}$$

In particular, $\pi _{(\gamma ,l)}(f(\alpha ,\gamma , \cdot )) =0$ if $(\alpha , (\gamma ,l)) \in {\mathbb R}^r \times (M\setminus {\mathcal C})$. The continuity of the map $F \mapsto f$ is obvious. This proves the first step in the induction procedure.

4.2 Reducing ${\mathcal B}$

There are two cases where we can reduce the manifold ${\mathcal B}$.

• 1. Suppose that there exists a smooth function $\varphi :{\mathcal B}\rightarrow {\mathbb R}_+$ which is not constant on the subset ${\mathcal B}_M:=p_{\mathcal B}(M)$, where $p_{\mathcal B}:{\mathcal B}\times {\mathfrak {g}}^*\rightarrow {\mathcal B}$ is the canonical projection. Let $\beta _0\in {\mathcal B}$ such that $\varphi (\beta _0)\in ]a,b[$ for some $b>a>0$ and let ${\mathcal B}_0:=\{\beta \in {\mathcal B}; \frac{a}{2}< \varphi (\beta )<2b\}$ and $M_0:=\{(\beta ,l)\in M; \beta \in {\mathcal B}_0\}$. Suppose that the theorem holds for the pair $({\mathcal B}_0,M_0)$. Let us show that the result remains true for the pair $({\mathcal B},M)$. Let ${\mathcal M}_0$ be an open relatively compact subset as in the theorem for $({\mathcal B}_0,M_0 )$. We let ${\mathcal M}:=\{(\beta ,l)\in M; {a}<\varphi (\beta )<b\}\cap {\mathcal M}_0$. We will show that ${\mathcal M}$ works for $({\mathcal B},M )$. Note that since ${\mathcal M}_0$ is open in $M_0$, we have that ${\mathcal M}$ is open in M. Let F be a kernel function defined on ${\mathbb R}^{r} \times M \times G \times G$ such that its support in $(\beta ,l)$ is contained in $G\cdot {\mathcal M}\subset M_0$. By assumption, there exists $f\in {\mathcal S}({\mathbb R}^{r} \times {\mathcal B}_0 \times G)$ such that $\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))$ admits $F(\cdot , (\beta ,l),\cdot , \cdot )$ as an operator kernel if $(\beta ,l) \in M_0$. In particular, $\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))=0$ if $(\beta ,l) \in M_0 \setminus G\cdot {\mathcal M}_0$. As ${\mathcal B}_0$ is a sub-manifold of ${\mathcal B}$, we may extend f to a function in ${\mathcal S}({\mathbb R}^{r} \times {\mathcal B}\times G)$ which we denote also by f. Choose $\vartheta \in {\mathcal C}^{\infty }_c({\mathbb R})$ with $0\le \vartheta \le 1$, $\vartheta \equiv 1$ on [a, b] and $\vartheta \equiv 0$ on $[0, \frac{a}{2}]\cup [2b, +\infty [$. We define $\phi \in {\mathcal C}^{\infty }(M)$ by $\phi (\beta ,l):= \vartheta (\varphi (\beta ))$. Then $\phi \equiv 1$ on $G\cdot {\mathcal M}_0$ and $\phi \equiv 0$ on $M\setminus G\cdot {\mathcal M}_0$. By taking $g:= \phi \cdot f$, we have that $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))=\phi (\beta , l) \cdot \pi _{(\beta ,l)}(f(\cdot ,\beta ,\cdot ))$. If $(\beta ,l) \in {\mathcal M}\subset {\mathcal M}_0$, then $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))=\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))$ and it admits $F(\cdot , (\beta ,l), \cdot , \cdot )$ as an operator kernel. If $(\beta ,l) \in M_0\setminus G\cdot {{\mathcal M}_0}$, then $\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))=0$ and $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))=0$. If $(\beta ,l) \in M \setminus M_0$, then $\varphi (\beta )\in [0, \frac{a}{2}]\cup [2b, +\infty [$, hence $\phi (\beta ,l)=0$ and so $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))=0$. Therefore the result is true for the function g.

• 2. If there exists a smooth sub-manifold ${\mathcal B}_0$ of ${\mathcal B}$ such that $p_{\mathcal B}(M)\subset {\mathcal B}_0$, then we can apply our theorem to the pair $({\mathcal B}_0,M)$. Since every smooth function $f_0$ on ${\mathcal B}_0\times G$ can be extended to a smooth function f on ${\mathcal B}\times G$, the Retract Theorem also holds for $({\mathcal B},M)$.

Remark 4.2.1

Let ${\mathcal B}$ and M be given as in the statement of the theorem. Let

$$\begin{aligned} p_{\mathcal B}: M \rightarrow {\mathcal B};\quad p_{\mathcal B}(\beta ,l)=\beta , \end{aligned}$$

be the canonical projection. If we denote by $M^{max}$ the subset of M consisting of all $(\beta ,l)\in M$ for which the rank of $dp_{\mathcal B}(\beta ,l)$ is maximal, then $M^{max}$ is open in M and the subset $p_{\mathcal B}(M^{max})$ of ${\mathcal B}$ is a smooth sub-manifold of ${\mathcal B}$. If $p_{\mathcal B}(M^{max})$ contains at least two elements, by the reasoning in Sect. 4.2, using a non-constant smooth function $\varphi _0$ on $p_{\mathcal B}(M^{max})$, which can be extended to a smooth function $\varphi$ of ${\mathcal B}$, we reduce ${\mathcal B}$ to ${\mathcal B}^{max}$ and we can always assume in this way that $p_{\mathcal B}(M)$ is a smooth sub-manifold of ${\mathcal B}$. If $p_{\mathcal B}(M^{max})$ is a singleton $\{\beta _0\}$, then $M=M^{max}$ and $p_{\mathcal B}(M)$ is obviously a smooth sub-manifold of ${\mathcal B}$.

4.3 Reducing to smoothly varying subspaces depending on ${\mathcal B}$

Let $M \subset {\mathcal B}\times {\mathfrak {g}}^*$ be a smooth G-invariant sub-manifold of ${\mathcal B}\times {\mathfrak {g}}^*$. Let us fix the largest index

$$\begin{aligned} I_M=I=\big ( (j_1, k_1), \cdots , (j_r, k_r)\big )=(j_1,k_1)\times I_1, \end{aligned}$$

where $I_1=((j_2,k_2),\cdots , (j_r,k_r))$ is the index set of $(\beta , l|_{g^1(\beta , l)})$, such that the open subset $M_I:=({\mathcal B}\times {\mathfrak {g}}^*)_I \cap M$ of M is nonempty. Let $p_{\mathcal B}:M\rightarrow {\mathcal B}; (\beta , l)\mapsto \beta$, be the projection onto the first variable and set

$$\begin{aligned} {\mathcal B}_M:=p_{\mathcal B}(M), \end{aligned}$$

which is a smooth sub-manifold of ${\mathcal B}$ by Remark 4.2.1.

Let ${\mathfrak {c}}_1:={\mathfrak {g}}_{j_{1}+1}= \text {span}\{Z_{j_1+1}, \dots , Z_n\}\subset {\mathfrak {g}}$ and let

$$\begin{aligned} {\mathfrak {n}}_\beta ^1:= [{\mathfrak {g}},{\mathfrak {c}}_1]_\beta +[Z_{j_1}, {\mathfrak {g}}_{k_1+1}]_\beta \subset {\mathfrak {c}}_1 \, \text { for }\, \beta \in {\mathcal B}. \end{aligned}$$

(4.5)

Then, by the definition of the indices $(j_1, k_1)$, we have

$$\begin{aligned} {\mathfrak {n}}_\beta ^1\subset \hbox {ker}(l)\cap {\mathfrak {c}}_1\subset {\mathfrak {a}}_\beta (l) \, \text { if } \, (\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^* )_{\le I}. \end{aligned}$$

(4.6)

It is easy to see that ${\mathfrak {n}}^1_{\beta }$ is an ideal in ${\mathfrak {g}}$. Let

$$\begin{aligned} Z_\beta :=[Z_{k_1},Z_{j_1}]_\beta \, \text { for }\, \beta \in {\mathcal B}. \end{aligned}$$

We fix a scalar product $\langle {\cdot },{\cdot }\rangle$ on ${\mathfrak {g}}$ such that $\{Z_1, \dots , Z_n\}$ is an orthonormal basis and we identify ${\mathfrak {c}}_1^*$ with ${\mathfrak {c}}_1$ by identifying $\sum _{r=j_1+1}^n a_rZ^*_r \in {\mathfrak {c}}_1^*$ with the element $\sum ^n_{r=j_1+1} a_r Z_r$ of ${\mathfrak {c}}_1$. Denote by $\Vert \cdot \Vert _2$ the Euclidean norm on ${\mathfrak {c}}_1$ (and hence on ${\mathfrak {c}}_1^*$) with respect to the given scalar product. We also identify

$$\begin{aligned} ({\mathfrak {n}}^1_{\beta })^{\perp } := \{q \in {\mathfrak {c}}_1^*; ~\langle q, {\mathfrak {n}}^1_{\beta }\rangle =\{0\}\} \end{aligned}$$

with a subspace of ${\mathfrak {c}}_1$. For all $\beta \in {\mathcal B}$, we write ${\mathfrak {c}}_1 = {\mathfrak {n}}^1_{\beta } \oplus ({\mathfrak {n}}^1_{\beta })^{\perp }$ and define $p_{\beta }$ to be the orthogonal projection of ${\mathfrak {c}}_1$ onto $({\mathfrak {n}}^1_{\beta })^{\perp }$. For each $\beta \in {\mathcal B}$, a generating subset of ${\mathfrak {n}}^1_{\beta }$ is given by

$$\begin{aligned} V(\beta )= & {} \{v_1(\beta ),\cdots , v_s(\beta )\}\\:= & {} \{[Z_a,Z_{a'}]_\beta ; a=1,\cdots , n, a' = j_{1}+1,\cdots ,n\}\\&\cup \{[Z_b,Z_{j_1}]_\beta ; b=k_1+1,\cdots , n\}. \end{aligned}$$

Let

$$\begin{aligned} a_{j,j'}(\beta ):=\langle {v_j(\beta )},{v_{j'}(\beta )}\rangle \, \text { for } \, {1\le j,j'\le s}. \end{aligned}$$

Fix $0\le k\le s$, let ${\mathcal I}_k=\{J\subset \{1,\cdots , s\}; \vert J\vert =k\}$ and for $\beta \in {\mathcal B}$, let

$$\begin{aligned} h_k(\beta ):= \sum _{J\in {\mathcal I}_k}\det {\big (\big (a_{j,j'}(\beta )\big )_{j,j'\in J}}\big )^2. \end{aligned}$$

It is easy to check that

$$\begin{aligned} h_k(\beta ) \ne 0\Leftrightarrow & {} v_1(\beta ), \dots , v_s(\beta ) \text { have at least rank } k,\\ h_k(\beta )=0\Leftrightarrow & {} v_1(\beta ), \dots , v_s(\beta ) \text { have rank }r <k. \end{aligned}$$

Let $n_1\in {\mathbb N}$ and put $f_0:= h_{n_1+1}$ and $f_1:= h_{n_1}$. Let

$$\begin{aligned} {\mathcal B}^{\le n_1}= & {} \{ \beta \in {\mathcal B}; \, f_0(\beta )=0\},\\ {\mathcal B}^{\ge n_1}= & {} \{\beta \in {\mathcal B}; \, f_1(\beta )\ne 0\},\\ {\mathcal B}^{n_1}= & {} \{\beta \in {\mathcal B}; \, f_0(\beta )=0 \text { and } f_1(\beta ) \ne 0\}. \end{aligned}$$

One sees that ${\mathcal B}^{\ge n_1}$ is open in ${\mathcal B}$, and hence is a sub-manifold of ${\mathcal B}$. Again, according to the reduction argument in Sect. 4.2 we can assume that $f_1(\beta )\ne 0$ for all $\beta \in {\mathcal B}$. On the other hand, let $n_1:= \max _{\beta \in {\mathcal B}_M}\mathrm{dim}({\mathfrak {n}}^1_\beta )$, then we have

$$\begin{aligned} {\mathcal B}^{\le n_1}:= & {} \{\beta \in {\mathcal B}; \, \mathrm{dim}({\mathfrak {n}}^1_\beta )\le n_1 \},\\ {\mathcal B}^{n_1}:= & {} \{\beta \in {\mathcal B}; \, \mathrm{dim}({\mathfrak {n}}^1_\beta )=n_1 \},\\ {\mathcal B}^{\ge n_1}:= & {} \{\beta \in {\mathcal B}; \, \mathrm{dim}({\mathfrak {n}}^1_\beta ) \ge n_1\}. \end{aligned}$$

Note that if we want ${\mathfrak {n}}^1_{\beta }$ to be of fixed dimension and to have ${\mathfrak {n}}^1_{\beta }$, $({\mathfrak {n}}^1_{\beta })^{\perp }$ and $p_{\beta }$ to vary smoothly with respect to $\beta$, we must restrict to ${\mathcal B}^{n_1}$. But in general ${\mathcal B}^{n_1}$ is not a sub-manifold of ${\mathcal B}$. Therefore we must find a smooth sub-manifold inside ${\mathcal B}^{n_1}$ containing an open subset of the smooth manifold ${\mathcal B}_M=p_{\mathcal B}(M)$. We have to distinguish the following two cases:

Case 1: If the differential $df_0$ is not identically zero on ${\mathcal B}_{M}$, we may define

$$\begin{aligned} {\mathcal B}^{max}_M:=\{\beta \in {\mathcal B}_M; \, df_0(\beta )\ne 0\} \, \, \text { and } \, {\mathcal B}^{max}:= \{\beta \in {\mathcal B}; \, df_0(\beta )\ne 0\}. \end{aligned}$$

By assumption, ${\mathcal B}^{max}$ is a nonempty open subset of ${\mathcal B}$.

Case 2: Assume that $df_0$ is identically zero on ${\mathcal B}_{M}$.

If $\mathrm{dim}( {\mathcal B}_M)< \mathrm{dim}({\mathcal B})$, we may build a function $\gamma \in {\mathcal C}^{\infty }({\mathcal B})$ such that $\gamma \equiv 0$ on ${\mathcal B}_{M}$ and $d\gamma$ is not identically zero on ${\mathcal B}_M$. We put $\widetilde{f}_0=f_0 + \gamma$. Then $\widetilde{f}_0 \equiv 0$ on ${\mathcal B}_{M}$ and $d\widetilde{f}_0$ is not identically zero on ${{\mathcal B}_M}$. We then define

$$\begin{aligned} {\mathcal B}^{max}_M:= \{ \beta \in {\mathcal B}_M; \, d\widetilde{f}_0(\beta )\ne 0\},\, {\mathcal B}^{max}:= \{ \beta \in {\mathcal B}; \, \widetilde{f}_0(\beta )=0, d\widetilde{f}_0(\beta )\ne 0\}. \end{aligned}$$

By the construction of $\widetilde{f}_0$, we have again that ${\mathcal B}^{max}$ is an open subset of ${\mathcal B}$ and ${\mathcal B}^{max}_M\subset {\mathcal B}^{max}$.

If $\mathrm{dim}( {\mathcal B}_M)=\mathrm{dim}({\mathcal B})$, then ${\mathcal B}_M$ is open in ${\mathcal B}$ and we take a smooth function $\tilde{f}\ne 0$ in ${\mathcal B}$ supported on ${\mathcal B}_M$. Let

$$\begin{aligned} {\mathcal B}^{max}:= \{ \beta \in {\mathcal B}; \, \widetilde{f}(\beta )\ne 0\} \, \text { and } \, {\mathcal B}^{max}_M:= \{ \beta \in {\mathcal B}_M; \, \widetilde{f}(\beta )\ne 0\}. \end{aligned}$$

In both cases, the ideals ${\mathfrak {n}}_\beta ^1$ vary smoothly on the smooth sub-manifold ${\mathcal B}^{max}$ of ${\mathcal B}$, since $\mathrm{dim}({\mathfrak {n}}_\beta ^1)=n_1$ on ${\mathcal B}^{max}_M$. The projection $p_{\beta }$ also varies smoothly on ${\mathcal B}^{max}$.

Remark 4.3.1

According to Remark 4.2.1, we can now assume that

$$\begin{aligned} {\mathcal B}={\mathcal B}^\mathrm{max}. \end{aligned}$$

Furthermore, since the function $\beta \mapsto \Vert p_\beta (Z_\beta )\Vert _{2} ^2$ is smooth on ${\mathcal B}$, we can take $\beta ^0\in {\mathcal B}$ and $0< \delta< R< \infty$ such that $\delta< \Vert p_{\beta ^0}(Z_{\beta ^0})\Vert _{2} < R$ and by using the reduction argument, we can then assume that the number $\Vert p_\beta (Z_\beta )\Vert _{2}$ is contained in the interval $[\delta ,R]$ for any $\beta \in {\mathcal B}$.

4.3.1 On the Manifold M

Let us focus on the manifold M again. Let $(\beta _0, l_0) \in M$ be fixed, but arbitrary. There exist $0< \delta< R< \infty$ such that

$$\begin{aligned} 0<\delta<\min \{\vert \langle l_0, Z_{\beta _0} \rangle \vert , \Vert p_{\beta _0} (Z_{\beta _0}) \Vert _2\}< \max \{\vert \langle l_0, Z_{\beta _0} \rangle \vert , \Vert p_{\beta _0} (Z_{\beta _0}) \Vert _2\}< R. \end{aligned}$$

This is due to the fact that $M \subset ({\mathcal B}\times {\mathfrak {g}}^*)_I$. According to Remark 4.3.1 we can now assume that

$$\begin{aligned} 0<\delta<\min \{\Vert p_{\beta } (Z_{\beta }) \Vert _2\}< \max \{\Vert p_{\beta } (Z_{\beta }) \Vert _2\}< R \end{aligned}$$

for all $\beta \in {\mathcal B}$. We define

$$\begin{aligned} M^{\delta , R}= & {} M^{ red}\\:= & {} \{(\beta ,l)\in M; \, 0<\delta<\min \{\vert \langle l, Z_{\beta } \rangle \vert \}< \max \{\vert \langle l, Z_{\beta } \rangle \vert \} <R\}. \end{aligned}$$

Obviously, $M^{ red}$ is open in M and thus is a smooth sub-manifold of M. On the other hand, we define

$$\begin{aligned} ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I, R, \delta }= & {} \Big \{(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I}; \\&{ }\quad \frac{1}{2} \delta< \min \{ \vert \langle l, Z_{\beta } \rangle \vert \}< \max \{ \vert \langle l, Z_{\beta } \rangle \vert \} < \frac{3}{2}R\Big \}. \end{aligned}$$

4.3.2 Reducing M

Now we claim that if the Retract Theorem holds for $({\mathcal B}, M^{red})$, then it remains true for $({\mathcal B}, M)$.

Assume that the result is true for $({\mathcal B}, M^{ red})$. Let ${\mathcal M}$ be the open subset in $M^{ red}$ given by the assumption. We will show that one may take the same manifold ${\mathcal M}$ for $({\mathcal B}, M)$ such that the theorem remains true for $({\mathcal B}, M)$. As $M^{red}$ is open in M, the set ${\mathcal M}$ also has a nonempty interior in M. Moreover, $p_{{\mathcal B}}(\mathcal M) \subset p_{\mathcal B}(M^{red}) \subset {\mathcal B}^{n_1} \subset {\mathcal B}^{\ge n_1}$. Let $\emptyset \ne C \subset {\mathcal M}$ be compact and let F be a kernel function defined on ${\mathbb R}^{r}\times M\times G\times G$ whose support in $(\beta ,l)$ is contained in $G\cdot C$. The restriction of F to ${\mathbb R}^{r} \times M^{red} \times G \times G$ is a kernel function for $({\mathcal B}, M^{red})$.

By assumption, there exists $f \in {\mathcal S}({\mathbb R}^{r} \times {\mathcal B}\times G)$ such that $\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))$ admits $F(\cdot , (\beta ,l), \cdot , \cdot )$ as an operator kernel if $(\beta ,l) \in M^{red}$. In particular, $\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))=0$ if $(\beta ,l) \in M^{red} \setminus C$. As $\emptyset \ne C \subset {\mathcal M}$ is compact, there exist $\delta _1, R_1\in {\mathbb R}_+$ such that

$$\begin{aligned} 0<\delta<\delta _1\le & {} \min \{\vert \langle l, Z_{\beta }\rangle \vert \} \\\le & {} \max \{\vert \langle l, Z_{\beta }\rangle \vert \} \\\le & {} R_1<R \end{aligned}$$

for all $(\beta ,l) \in C$, as $C \subset {\mathcal M} \subset M \subset ({\mathcal B}\times {\mathfrak {g}}^*)_I$. Let $u\in {\mathcal C}_c^{\infty }({\mathbb R})$ be odd such that $u\equiv 1$ on $[\delta _1, R_1]$ and $u\equiv 0$ on $[0,\delta ] \cup [R, +\infty [$. There exists $\chi \in {\mathcal S}({\mathbb R})$ such that $\widehat{\chi }=u$. Let us define a function $\psi$ on $\bigcup _{\beta }\{\beta \}\times \hbox {exp}({\mathbb R}Z_{\beta })$ by $\psi (\beta , \hbox {exp}(sZ_{\beta })):= \chi (s)$. For $(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_J$ with $J\le I$, we have $Z_{\beta } \in {\mathfrak {a}}_{\beta }(l) \subset {\mathfrak {g}}_{\beta }(l)$ and $\pi _{({\beta },l)}\vert _{{\mathbb R}Z_{\beta }}(\psi (\beta , \cdot ))= {\widehat{\chi }}(\langle l, Z_{\beta } \rangle ){\mathbb {I}}_{{\mathfrak {H}}(\beta ,l)} = u(\langle l, Z_{\beta } \rangle ){\mathbb {I}}_{{\mathfrak {H}}(\beta ,l)}$.

We define a function g on ${\mathbb R}^r \times {\mathcal B}\times G$ by

$$\begin{aligned} g(\cdot , \beta , \cdot ):= f(\cdot , \beta , \cdot ) * \psi (\beta , \cdot ). \end{aligned}$$

This implies that

$$\begin{aligned} \pi _{(\beta ,l)}(g(\cdot , \beta , \cdot )) = u(\langle l, Z_{\beta } \rangle ) \pi _{(\beta ,l)}(f(\cdot , \beta , \cdot )). \end{aligned}$$

If $(\beta ,l)\in C$, then $u(\langle l, Z_{\beta }\rangle )=1$ and $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))= \pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))$ admits $F(\cdot , (\beta ,l), \cdot , \cdot )$ as an operator kernel. If $(\beta ,l) \in M ^{red}\setminus C$, then $\pi _{(\beta ,l)}(f(\cdot , \beta , \cdot ))=0$, hence $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))=0$ and $F(\cdot , (\beta ,l), \cdot , \cdot )=0$. If $(\beta ,l) \in M\setminus M^{red}$, then $\vert \langle l,Z_{\beta } \rangle \vert \notin [\delta , R]$, i.e. $u(\langle l, Z_{\beta } \rangle )=0$, which implies that $\pi _{(\beta ,l)}(g(\cdot , \beta , \cdot ))=0$. Hence, the mapping $F\mapsto g$ satisfies the property of the retract for $({\mathcal B}, M)$.

4.4 Construction of a New Variable Group

We start this section with an example which will demonstrate the use of a variable group and its variable algebra in the retract construction.

4.4.1 An Example

In this subsection, we will consider the free two-step nilpotent Lie group on four generators.

Let ${\mathfrak {g}}={\mathfrak {f}}_{4,2}$ be the free two-step nilpotent Lie algebra with four generators. This algebra has a Jordan-Hölder basis $\{Y_1,\cdots , Y_{10}\}$, where

$$\begin{aligned} \,[ Y_1,Y_2]= Y_5,&[ Y_2,Y_3]=Y_8,\\ \,[ Y_1,Y_3]= Y_6,&[ Y_2,Y_4]=Y_9,\\ \,[ Y_1,Y_4]= Y_7,&[ Y_3,Y_4]=Y_{10}. \end{aligned}$$

The centre ${\mathfrak {z}}$ of ${\mathfrak {g}}$ is the subspace ${\mathfrak {z}}=\text {span}\{{Y_j\vert \, j=5, \cdots , 10}\}$. Let $G= F_{4,2}= \hbox {exp}({\mathfrak {f}}_{4,2})$ be the simply connected nilpotent Lie group with Lie algebra ${\mathfrak {f}}_{4,2}$.

The co-adjoint orbits and the Pukanszky layers of the orbit spaces of the group G have been determined in the paper [4]. The non-generic index sets are given by the conditions

$$\begin{aligned} I=\{(i,j)\vert 1\le i<j\le 4\} \end{aligned}$$

and

$$\begin{aligned} {\mathfrak {g}}^*_{(i,j)}=\{l\in {\mathfrak {g}}^*\vert \langle l,[ Y_i,Y_j]\rangle =\langle l,Y_k\rangle \ne 0, \langle l,Y_{k'}\rangle =0 \text { for } k'>k \}, \end{aligned}$$

where $Y_k= [Y_i, Y_j]$. Consider now the layer ${\mathfrak {g}}^*_{(3,4)}$, it is given precisely by

$$\begin{aligned} {\mathfrak {g}}^*_{(3,4)}= \{l= \sum _{i=1}^{10} a_i Y_i^* \in {\mathfrak {g}}^* \vert \, a_5 a_{10}+ a_8 a_7- a_6 a_9= 0, a_{10} \ne 0\}. \end{aligned}$$

Hence the section $\Sigma _{(3,4)}$ of ${\mathfrak {g}}^*_{(3,4)}$ is given by

$$\begin{aligned} \Sigma _{(3,4)}= & {} \bigg \{ a_1 Y_1^* +a_2 Y_2^*\\&+\;\bigg (\frac{a_6 a_9- a_7 a_8}{a_{10}} \bigg )Y_5^*+\sum _{i=6}^{10}a_iY_i^*\vert a_1,a_2,a_i \in {\mathbb R}, i=6,\cdots , 10 \bigg \}. \end{aligned}$$

The co-oadjoint orbit of $l\in \Sigma _{(3,4)}$ is the subset

$$\begin{aligned} \Omega _l= & {} \bigg \{\bigg (a_1+\frac{a_6}{a_{10}}u_2+\frac{a_7}{a_{10}}u_1 \bigg )Y_1^*+ \bigg (a_2+\frac{a_8}{a_{10}}u_2+\frac{a_9}{a_{10}}u_1 \bigg )Y_2^*+u_1Y_3^*+u_2 Y_4^*+ \\&\bigg (\frac{a_6 a_9- a_7 a_8}{a_{10}} \bigg )Y_5^*+\sum _{i=6}^{10}a_iY_i^*\vert u_1,u_2\in {\mathbb R}\bigg \}. \end{aligned}$$

For $l\in {\mathfrak {g}}^*_{(3,4)}$, the stabiliser ${\mathfrak {g}}(l)$ of l is spanned by

$$\begin{aligned} S_1(q):= Y_1- \frac{a_7}{a_{10}}Y_3+ \frac{a_6}{a_{10}}Y_4, \ S_2(q):= Y_2- \frac{a_9}{a_{10}}Y_3+ \frac{a_8}{a_{10}}Y_4 \text { and the center }{\mathfrak {z}}, \end{aligned}$$

for $q:=l_{\vert {\mathfrak {z}}}$, and the Vergne polarisation ${\mathfrak {p}}(l)$ of l is ${\mathfrak {g}}(l)+ {\mathbb R}Y_4$. By the construction in Sect. 2.4, we obtain the new basis vectors of ${\mathfrak {g}}$ adapted to ${\mathfrak {p}}(l)$ as the following:

$$\begin{aligned} Z_{1}(l)= & {} Y_3, \\ Z_2(l)= & {} Y_1- \frac{a_7}{a_{10}}Y_3+\frac{a_6}{a_{10}}Y_4, \\ Z_3(l)= & {} Y_2- \frac{a_9}{a_{10}}Y_3+\frac{a_8}{a_{10}}Y_4, \\ Z_k(l)= & {} Y_k \, \text { for } k= 4, \ldots , 10. \end{aligned}$$

Let ${\mathcal B}= \{(a_5, \ldots , a_{10})\in {\mathbb R}^6\vert \, a_5 a_{10}+ a_8 a_7- a_6 a_9= 0, a_{10} \ne 0\}$. Now we define the new variable Lie algebra $({\mathfrak {g}}, {\mathcal B})$ by fixing a basis $\{Z_1, \ldots , Z_{10}\} \subseteq {\mathfrak {g}}\simeq {\mathbb R}^{10}$ and we equip ${\mathbb R}^{10}$ with the brackets $[\cdot , \cdot ]_q$, for $q\in {\mathcal B}$, satisfying the following:

$$\begin{aligned} \,[ Z_1, Z_2]_q= & {} [Y_3, Y_1- \frac{a_7}{a_{10}} Y_3+\frac{a_6}{a_{10}}Y_4]= - Y_6+\frac{a_6}{a_{10}}Y_{10}, \\ \,[Z_1, Z_3]_q= & {} [Y_3, Y_2- \frac{a_9}{a_{10}} Y_3+\frac{a_8}{a_{10}}Y_{10}]= - Y_8+\frac{a_8}{a_{10}}Y_{10}, \\ \,[ Z_1, Z_4]_q= & {} Y_{10}, \\ \,[ Z_2, Z_3]_q= & {} Y_5- \frac{a_9}{a_{10}}Y_6+ (\frac{a_7}{a_{10}}+\frac{a_8}{a_{10}})Y_8-\frac{a_6}{a_{10}}Y_9+(\frac{a_6a_{10}}{a_{10}^2})Y_{10}, \\ \,[ Z_2, Z_4]_q= & {} Y_7- \frac{a_7}{a_{10}} Y_{10}, \\ \,[ Z_3, Z_4]_q= & {} Y_9- \frac{a_9}{a_{10}} Y_{10}. \end{aligned}$$

For any $q=(a_5,\cdots , a_{10})\in {\mathcal B}$, the linear functional $l_{q}$ on ${\mathfrak {g}}^*$ is defined by

$$\begin{aligned} l_q(Z_1)= & {} a_3,\\ l_q(Z_2)= & {} \langle l,Y_1- \frac{a_7}{a_{10}}Y_3+\frac{a_6}{a_{10}}Y_4\rangle =a_1- \frac{a_3a_7}{a_{10}}+\frac{a_4a_6}{a_{10}}, \\ l_q(Z_3)= & {} \langle l,Y_2- \frac{a_9}{a_{10}}Y_3+\frac{a_8}{a_{10}}Y_4\rangle =a_2- \frac{a_3a_9}{a_{10}}+\frac{a_4a_8}{a_{10}}, \\ l_q(Z_j)= & {} a_j \, \text { for } j= 4, \ldots , 10. \end{aligned}$$

Let $({\mathfrak {g}}_1(q), {\mathcal B})$ be the subalgebra of $({\mathfrak {g}}, {\mathcal B})$ given by $\mathrm{span}\{Z_2, \ldots , Z_{10}\}$. We have then $[{\mathfrak {g}}_1, {\mathfrak {g}}_1]_q \subset {\mathfrak {g}}_1$. By the construction of variable Lie algebras given in Sect. 2.3, we have $({\mathfrak {g}}_1(q), [\cdot , \cdot ]_q)\simeq ({\mathfrak {p}}(l_q), [\cdot , \cdot ])$ and hence $\langle l_q, [{\mathfrak {g}}_1, {\mathfrak {g}}_1]_q \rangle = 0$. Note that we will prove the retract theorem by induction on the dimension of the variable group. The induction procedure transforms a given variable Lie algebra $({\mathfrak {g}}, {\mathcal B})$ into a new variable one $({\mathfrak {g}}_1, {\mathcal B}_1)$ of lower dimension in a consistent manner.

Let M be a smooth G-invariant manifold of ${\mathfrak {g}}^*$ contained in $\bigcup _{(i, j)\le (3,4)} {\mathfrak {g}}^*_{(i,j)}$ such that $M\cap {\mathfrak {g}}^*_{(3,4)}\ne \emptyset$. Choose $\delta >0$ so that

$$\begin{aligned} M^{\text {red}}:= \{l\in M \vert \, \vert \langle l, Y_{10}\rangle \vert > \delta \}\cap \Sigma _{(3,4)} \end{aligned}$$

is non-empty. For $l\in M^{\text {red}}$, let

$$\begin{aligned} \pi _l:=\hbox {ind}_{P(l)}^{G} \chi _l. \end{aligned}$$

For $l\in \Sigma _{(3,4)}$, we define also the diffeomorphism $\Phi _l$ from ${\mathbb R}\times {\mathbb R}^9$ onto G by

$$\begin{aligned} \Phi _l(y_3, r):=\hbox {exp}(y_3 Y_3)\cdot \hbox {exp}(\sum _{j=2}^{10} z_j Z_j(l)) \, \text { for } \, (y_3, r)\in {\mathbb R}\times {\mathbb R}^9. \end{aligned}$$

Then for any $u, v\in {\mathbb R}^{10}$ and $l\in \Sigma _{(3,4)}$, we have that

$$\begin{aligned} \Phi _l(u\cdot _l v)= \Phi _l(u)\cdot \Phi _l(v). \end{aligned}$$

For $f\in {\mathcal S}(G), y_3\in {\mathbb R}, l\in \Sigma _{(3,4)}$ and $q=l_{\vert {\mathfrak {z}}}$, let

$$\begin{aligned} \hat{f}(l,y_3,q):= \widehat{f\circ \Phi _l}^2(y_3, {l_q}_{\vert {\mathbb R}^9}), \end{aligned}$$

where $\hat{h}^2$ is defined for all $h\in C^\infty ({\mathfrak {g}}^*_{(3,4)}, {\mathcal S}({\mathbb R}^{10}) )$ by

$$\begin{aligned} \hat{h} ^2(l, y_3, r):= \int _{{\mathbb R}^9} e^{-ir(u)}h(l,y_3,u)du, \ r\in ({\mathbb R}^9)^*, y_3\in {\mathbb R}, l\in \Sigma _{(3,4)}. \end{aligned}$$

Then for $\xi \in L^2({\mathbb R})$ and $u\in {\mathbb R}$, we have that

$$\begin{aligned} \pi _l(f)\xi (u)= & {} \int _{\mathbb R}\hat{f}^2 \bigg (l,u-y_3, \bigg (a_1- \frac{a_3a_7}{a_{10}}+\frac{a_4a_6}{a_{10}},a_2- \frac{a_3a_9}{a_{10}}+\frac{a_4a_8}{a_{10}},a_4\\&+ y_3 a_{10}, l_{\vert {\mathfrak {z}}}\bigg ) \bigg )\xi (y_3)dy_3. \end{aligned}$$

We must find a smooth sub-manifold ${\mathcal M}$ inside $M^{\text {red}}$ such that for a given smooth compactly supported operator field F(l) on ${\mathcal M}$, we have a Schwartz function f defined on G such that

$$\begin{aligned} \hat{f}^2 \bigg (l,u-y_3, \bigg (a_1- \frac{a_3a_7}{a_{10}}+\frac{a_4a_6}{a_{10}},a_2- \frac{a_3a_9}{a_{10}}+\frac{a_4a_8}{a_{10}},a_4+ y_3 a_{10}, l_{\vert {\mathfrak {z}}}\bigg ) \bigg ){=} k_F(l, u,y_3) \end{aligned}$$

for all $u,y_3\in {\mathbb R}$ and l in ${\mathcal M}$ (here $k_F$ denotes the kernel function of the operator F(l)), and $\pi _{l'}(f)= 0$ for all $l'$ in ${\mathfrak {g}}^*_{(i,j)}$ with $(i, j)< (3,4)$.

For simplicity of the notations, let

$$\begin{aligned} {\mathfrak {g}}_1^0:=\text {span}\{Z_1,Z_2,{\mathfrak {z}}\}\subset {\mathfrak {g}}_1. \end{aligned}$$

Then we can consider ${\mathcal M}$ to be a smooth sub-manifold of $({\mathfrak {g}}_1^0)^*$. The construction of the manifold ${\mathcal M}$ and of the retract function in the general case will be shown in Sect. 4.5.

In our example, we take a smooth kernel function $k(l,u,v), l\in {\mathcal M}$ and $u, v\in {\mathbb R}$, compactly supported in ${\mathcal M}$ and we extend it to a smooth function K(l, u, v) on $({\mathfrak {g}}_1^0)^*$ which is compactly supported in $l\in ({\mathfrak {g}}_1^0)^*$, vanishes for $\vert l(Y_{10})\vert \le \delta$ and is Schwartz in the variables (u, v). We can then find a function $h\in {\mathcal S}({\mathfrak {g}}_1^0\times {\mathbb R}^2)$ such that

$$\begin{aligned} \hat{h}^{{\mathfrak {g}}_1^0}(l,u,v)={\vert l(Y_{10})\vert }K(l,u,v) \, \text { for } u,v \in {\mathbb R}, l\in ({\mathfrak {g}}_1^0)^*. \end{aligned}$$

Now let

$$\begin{aligned} \tilde{h}(q, g_1, u,v):=\int _{{\mathbb R}\times {\mathfrak {z}}^*}h(g_1+y Y_4 + V, u,v) e^{-i\langle q,V\rangle }dy dV \, \text { for } q\in {\mathfrak {z}}^*, g_1\in {\mathfrak {g}}_1. \end{aligned}$$

Define the function $\tilde{f}: G\times {\mathfrak {z}}^*\rightarrow {\mathbb C}$ by

$$\begin{aligned}&\tilde{f}(\hbox {exp}( y_3 Y_3)\cdot \hbox {exp}(s_1 S_1(q)+s_2 S_2(q)) \cdot \hbox {exp}(y_4 Y_4) \cdot \hbox {exp}(Z), q) :=\\&\quad \quad e^{i q(Z)}\int _{\mathbb R}\tilde{h}(q, \mathrm{Ad}_q(\hbox {exp}(v Y_3))({s_1 S_1(q)+s_2 S_2(q)}), y_3,v)e^{-i a_{10} y_4 v}dv. \end{aligned}$$

Finally let

$$\begin{aligned} f(g):=\int _{{\mathfrak {z}}^*}\tilde{f}(g,q)dq \quad \text { for } g\in G. \end{aligned}$$

Then the function f has the required properties (by the computations in Sect. 4.5).

4.4.2 The Mapping $\alpha (\beta ,l)$

For $(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I}$, we have seen in (4.6) that ${\mathfrak {n}}^1_{\beta } \subset \hbox {ker}(l)$. Let $q:=l\vert _{{\mathfrak {c}}_1} \in ({\mathfrak {n}}^1_{\beta })^{\perp }$ and $\langle l, Z_{\beta }\rangle = \langle l, p_{\beta }(Z_{\beta })\rangle = \langle q, p_{\beta }(Z_{\beta })\rangle$. For $(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I, R, \delta }$, we have that $\vert \langle l, [Z_{k_1}, Z_{j_1}]_{\beta } \rangle \vert = \vert \langle l, Z_{\beta } \rangle \vert> \frac{\delta }{2}>0$ implies $\langle l, [Z_{k_1}, Z_{j_1}]_{\beta } \rangle \ne 0$ and $j_1(\beta ,l)=j_1$, $k_1(\beta ,l)=k_1$.

Take an odd function $\varphi \in C^\infty ({\mathbb R})$ with the properties that $\varphi (s)=0$ for $0\le s<\delta /4 \text { and } s>2 R$, $1>\varphi (s)>0$ for $s\in ]\delta /4,\delta /2[ \cup ]3R/2,2R[$ and $\varphi (s)=1$ for $3 R/2\ge s \ge \delta /2$. For every $(\beta ,q)\in {\mathcal B}\times {\mathfrak {c}}_1^*$, we construct the vector $\alpha (\beta ,q)\in {\mathfrak {c}}_1\simeq ({\mathfrak {c}}_1)^* \$ by

$$\begin{aligned} \alpha (\beta ,q):= & {} \varphi (\Vert p_\beta (Z_\beta )\Vert _{2} )\varphi ({\langle {q},{p_\beta (Z_\beta )}\rangle }) p_\beta (q) \nonumber \\&+\big (1-\varphi (\Vert p_\beta (Z_\beta )\Vert _{2} )\varphi (\vert {\langle {q},{p_\beta (Z_\beta )}\rangle }\vert )\big )p_\beta (Z_\beta ). \end{aligned}$$

(4.7)

By the construction, $\alpha (\beta ,q)\in ({\mathfrak {n}}^1_\beta )^\perp \subset {\mathfrak {c}}_1^* \equiv {\mathfrak {c}}_1$ for every $(\beta ,q)\in {\mathcal B}\times {\mathfrak {c}}_1^*$. On the other hand, for $(\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta }$ and $q=l_{\vert {\mathfrak {c}}_1}$, we have that

$$\begin{aligned} \alpha (\beta ,q)= & {} \varphi (\Vert p_\beta (Z_\beta )\Vert _{2} )\varphi ({\langle {q},{p_\beta (Z_\beta )}\rangle }) p_\beta (q)\nonumber \\&+ (1-\varphi (\Vert p_\beta (Z_\beta )\Vert _{2} )\varphi (\vert {\langle {q},{p_\beta (Z_\beta )}\rangle }\vert ))p_\beta (Z_\beta )\nonumber \\= & {} \pm p_\beta (q)+(1-1)p_\beta (Z_\beta )\nonumber \\= & {} \pm q. \end{aligned}$$

(4.8)

This is due to the fact that $p_{\beta }(q)=q$ as ${\mathfrak {n}}^1_{\beta } \subset \hbox {ker}(q)$ for $q=l\vert _{{\mathfrak {c}}_1}$, if $(\beta ,l) \in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I}$. We will show that

$$\begin{aligned} \langle {\alpha (\beta ,q)},{Z_\beta }\rangle= & {} \varphi (\Vert p_\beta (Z_\beta )\Vert _{2} )\varphi ({\langle {q},{p_\beta (Z_\beta )}\rangle }) \langle {p_\beta (q)},{Z_\beta }\rangle \\&+(1-\varphi (\Vert p_\beta (Z_\beta )\Vert _{2} )\varphi (\vert {\langle {q},{p_\beta (Z_\beta )}\rangle }\vert ))\Vert p_\beta (Z_\beta )\Vert _{2} ^2> 0 \end{aligned}$$

on ${\mathcal B}\times {\mathfrak {c}}_1^*$. In fact, let us first notice that $\langle p_{\beta }(q), Z_{\beta }\rangle = \langle q, p_{\beta }(Z_{\beta })\rangle$. As $\varphi$ is an odd function and $\varphi \ge 0$ on ${\mathbb R}_+$, we have

$$\begin{aligned} A:= \varphi (\Vert p_{\beta }(Z_{\beta })\Vert _2) \varphi (\langle q, p_{\beta }(Z_{\beta })\rangle ) \langle p_{\beta }(q), Z_{\beta }\rangle \ge 0. \end{aligned}$$

Since $0\le \varphi \le 1$ on ${\mathbb R}_+$,

$$\begin{aligned} B:= \big (1- \varphi (\Vert p_{\beta }(Z_{\beta })\Vert _2) \varphi (\vert \langle q, p_{\beta }(Z_{\beta })\rangle \vert ) \big ) \Vert p_{\beta }(Z_{\beta })\Vert ^2_2 \ge 0. \end{aligned}$$

If none of the $\varphi (\cdot )$’s is equal to zero and if $\langle p_{\beta }(q), Z_{\beta }\rangle \ne 0$, then $A>0$. If $\langle p_{\beta }(q), Z_{\beta }\rangle =0$, then $\varphi (\vert \langle p_{\beta }(q), Z_{\beta }\rangle \vert )=0$ and thus $B>0$, as by Remark 4.3.1$\Vert p_{\beta }(Z_{\beta })\Vert _2>0$. If one of the $\varphi (\cdot )$’s is equal to zero, then again $B>0$.

For $(\beta ,q)\in {\mathcal B}\times {\mathfrak {c}}_1^*$, let

$$\begin{aligned} \delta (\beta ,q):=\mathrm{{ad}}_\beta ^*(Z_{j_1})\alpha (\beta ,q)\in {\mathfrak {g}}^*. \end{aligned}$$

We have that

$$\begin{aligned} \langle {\delta (\beta ,q)},{Z_{k_1}}\rangle =\langle {\alpha (\beta ,q)},{Z_\beta }\rangle > 0 \end{aligned}$$

and

$$\begin{aligned} \langle {\delta (\beta ,q)},{[{\mathfrak {g}},{\mathfrak {g}}] _\beta }\rangle= & {} \langle {\alpha (\beta ,q)},{[[{\mathfrak {g}},{\mathfrak {g}}]_\beta ,Z_{j_1}] _\beta } \rangle \subset \langle {\alpha (\beta ,q)},{[{\mathfrak {g}},[{\mathfrak {g}},Z_{j_1} ]_\beta ] _\beta }\rangle \\&\subset \langle {\alpha (\beta ,q)},{[{\mathfrak {g}},{\mathfrak {c}}_1] _\beta }\rangle \ \subset \langle {\alpha (\beta ,q)},{{\mathfrak {n}}^1_\beta }\rangle =\{0\}, \end{aligned}$$

by the definition of $\alpha (\beta ,q)$ in (4.7). This means that $\delta (\beta ,q)$ is an algebra homomorphism of ${\mathfrak {g}}_\beta =({\mathfrak {g}}, [ \cdot , \cdot ]_{\beta })$ which does not vanish at the vector $Z_{k_1}$. Hence the subspace

$$\begin{aligned} {\mathfrak {g}}^1(\beta ,q):=\hbox {ker}(\delta (\beta ,q)) \end{aligned}$$

is an ideal of ${\mathfrak {g}}_\beta$ of co-dimension one and

$$\begin{aligned} {\mathfrak {g}}={\mathbb R}Z_{k_1}\oplus {\mathfrak {g}}^1(\beta ,q). \end{aligned}$$

(4.9)

Furthermore ${\mathfrak {g}}^1(\beta ,q)$ contains ${\mathfrak {c}}_1$ for any $(\beta ,q) \in {\mathcal B}\times {\mathfrak {c}}_1^*$. In fact,

$$\begin{aligned} \langle \delta (\beta ,q),{\mathfrak {c}}_1\rangle = \langle \alpha (\beta ,q), [{\mathfrak {c}}_1, Z_{j_1}]_{\beta }\rangle =0 \end{aligned}$$

as $\alpha (\beta ,q) \in ({\mathfrak {n}}^1_{\beta })^{\perp }$ and $[{\mathfrak {c}}_1, Z_{j_1}]_{\beta } \subset {\mathfrak {n}}^1_{\beta }$.

4.4.3 The New Variable Group $({\mathcal B}_1,G_1)$

In order to construct a new variation in the induction procedure, we put

$$\begin{aligned} {\mathcal B}_1:= {\mathcal B}\times {\mathbb R}\times {\mathfrak {c}}_1^*. \end{aligned}$$

For $(\beta ,y,q)\in {\mathcal B}_1$, we define a Jordan-Hölder basis

$$\begin{aligned} \tilde{{\mathcal Z}}^1(\beta ,y,q)=\{\tilde{Z}^1_1(\beta ,q),\cdots , \tilde{Z}^1_{{n-1}}(\beta ,q)\} \end{aligned}$$

of ${\mathfrak {g}}_1(\beta ,y,q)=\hbox {ker}(\delta (\beta ,q))$ by

$$\begin{aligned} \alpha ^{\beta ,y,q}_{k}=\alpha ^{\beta ,q}_{k}:=\frac{\langle {\alpha (\beta ,q)},{[Z_k,Z_{j_1}]_\beta }\rangle }{\langle {\alpha (\beta ,q)},{Z_\beta }\rangle } \end{aligned}$$

and

$$\begin{aligned} \widetilde{{\mathcal Z}}^1(\beta ,y,q)=\widetilde{{\mathcal Z}}^1(\beta ,q):= & {} \{ Z_1-\alpha ^{\beta ,q}_{1} Z_{k_1}, \cdots , Z_{k_1-1}-\alpha ^{\beta ,q}_{k_1-1} Z_{k_1}, Z_{k_1+1},\cdots , Z_n\}\\= & {} \{\tilde{Z}^1_1(\beta ,q),\cdots , \tilde{Z}^1_{n-1}(\beta ,q)\}.\ \end{aligned}$$

In particular, for $(\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I, R, \delta }$ we have by (2.1) that

$$\begin{aligned} {\mathfrak {g}}_1(\beta ,y,l\vert _{{\mathfrak {c}}_1}) = {\mathfrak {g}}_1(\beta ,l)={\mathfrak {g}}^1(\beta , l_{\vert {\mathfrak {c}}_1}). \end{aligned}$$

In fact, in this case $j_1(\beta ,l) = j_1$, $k_1(\beta ,l)=k_1$ and

$$\begin{aligned} {\mathfrak {g}}_1(\beta ,y, l\vert _{{\mathfrak {c}}_1}) {=} \{U \in {\mathfrak {g}}; \langle \delta (\beta , l\vert _{{\mathfrak {c}}_1}),U\rangle = 0\}{=} \{U\in {\mathfrak {g}}; \langle l, [U,Z_{j_1}]_{\beta }\rangle =0 \}= {\mathfrak {g}}_1(\beta ,l), \end{aligned}$$

as $\alpha (\beta , l\vert _{{\mathfrak {c}}_1})=\varepsilon \cdot l\vert _{{\mathfrak {c}}_1}$ with $\varepsilon =\pm 1$ provided $(\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I, R, \delta }$.

For each k, we also have that

$$\begin{aligned} \alpha _k^{\beta ,q}=\frac{\langle {l,[Z_k,Z_{j_1}]_\beta }\rangle }{\langle {l},{Z_\beta }\rangle }. \end{aligned}$$

(4.10)

This new basis $\widetilde{{\mathcal Z}}^1(\beta ,y,q)$ coincides then, up to normalisation, with the basis obtained in Sect. 2.4, both procedures and bases generate the same indices. Furthermore by (4.10), for $(\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta }$, we have

$$\begin{aligned} \tilde{\mathcal Z}^1(\beta ,y,l_1)={\mathcal Z}^1(\beta ,l), \end{aligned}$$

(4.11)

where ${\mathcal Z}^1(\beta ,l)$ is defined in Sect. 2.4 and $l_1= l\vert _{{\mathfrak {c}}_1}$.

For any $(\beta , y, q) \in {\mathcal B}_1$, let us write

$$\begin{aligned}{}[\tilde{Z}^1_u(\beta ,q),\tilde{Z}^1_v(\beta ,q)] =\sum _{w=1}^{n-1} \gamma (\beta ,q)^{u,v}_w \tilde{Z}^1_w(\beta ,q) \quad \text {for } \, u<v \text { in }\{1,\cdots ,n-1 \}. \end{aligned}$$

We obtain in this way a new variable Lie algebra $({\mathcal B}_{1}, {\mathfrak {g}}_1)$, where

$$\begin{aligned} {\mathfrak {g}}_1={\mathbb R}^{n-1},\ {\mathcal B}_1={\mathcal B}\times {\mathbb R}\times {\mathfrak {c}}_1^* \end{aligned}$$

and the canonical basis ${\mathcal Z}^1=\{Z^1_1,\cdots , Z^1_{n-1}\}$ of ${\mathfrak {g}}_1$ satisfies, by definition,

$$\begin{aligned}{}[Z^1_u,Z^1_v] _{(\beta ,q)}=\sum _{w=1}^{n-1} \gamma (\beta ,q)^{u,v}_w Z^1_{w} \quad \text { for } \, u<v \text { in } \{1,\cdots ,n-1 \}. \end{aligned}$$

This means that the new variable Lie algebra $({\mathcal B}_1, {\mathfrak {g}}_1)$ with ${\mathfrak {g}}_1 \equiv {\mathfrak {g}}^1(\beta ,q)$ is defined such that $({\mathfrak {g}}_1, [\cdot , \cdot ]_{(\beta ,q)}) \equiv ({\mathfrak {g}}^1(\beta ,q), [\cdot , \cdot ]_{(\beta ,q)})$.

Given $(\beta ,l) \in {\mathcal B}\times {\mathfrak {g}}^*$, let us define $l_1$ in ${\mathfrak {g}}_1^*$ by $l_1(Z^1_i) := l( \tilde{Z}^1_i(\beta ,q))$ for all $i\in \{1, \dots , n-1\}$. One has $l_1(Z^1_i)= l(Z_{i+1})$ if $i \ge k_1$. We also define a map

$$\begin{aligned} \nonumber \iota _1:{\mathcal B}\times {\mathfrak {g}}^*\rightarrow & {} {\mathcal B}_1 \times {\mathfrak {g}}_1^*\\ (\beta ,l)\mapsto & {} ( (\beta , \langle l,Z_{k_1}\rangle ,\alpha (\beta , l_{\vert {\mathfrak {c}}_1})), l_1), \end{aligned}$$

(4.12)

where $l_1\equiv l_{\vert {\mathfrak {g}}^1(\beta ,l_{\vert {\mathfrak {c}}_1})}$. We see that $\iota _1$ is obviously smooth, injective and is a diffeomorphism onto its image.

Using (4.9) we can identify every $l\in {\mathfrak {g}}^*$ with the pair $(v,l_1)$ where $v:= \langle l, Z_{k_1}\rangle$ and $l_1:=l_{\vert {\mathfrak {g}}_1}\equiv l_{\vert {\mathfrak {g}}^1(\beta ,l_{\vert {\mathfrak {c}}_1})}$. We can then transfer the natural action of G on ${\mathcal B}\times {\mathfrak {g}}^*$ to ${\mathcal B}_1\times {\mathfrak {g}}_1^*$ using the mapping $\iota _1$. This gives us

$$\begin{aligned} g\cdot ((\beta ,v,q),l_1)= ((\beta ,v+\langle \mathrm{Ad}^*_\beta (g)l_1,Z_{k_1}\rangle ,\mathrm{Ad}_\beta ^*(g)q),\mathrm{Ad}^*_\beta (g)l_1). \end{aligned}$$

We have automatically the relation

$$\begin{aligned} \iota _1(g\cdot (\beta ,l))=g\cdot (\iota _1(\beta ,l)) \end{aligned}$$

for any $g\in G$ and $(\beta ,l)\in {\mathcal B}\times {\mathfrak {g}}^*$.

Consider now the smooth manifold

$$\begin{aligned} ({\mathcal B}\times {\mathfrak {g}}^*)^0_{\le I, R, \delta } :=\{(\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta }; \langle l,Z_{k_1}\rangle =\langle l,Z_{j_1}\rangle =0\}. \end{aligned}$$

Obviously the smooth manifold $({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta }$ is diffeomorphic with the manifold ${\mathbb R}^2\times ({\mathcal B}\times {\mathfrak {g}}^*)^0_{\le I,R,\delta }$. The mapping

$$\begin{aligned} \Phi :{\mathbb R}^2\times ({\mathcal B}\times {\mathfrak {g}}^*)^0_{\le I,R,\delta } \rightarrow ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta } \end{aligned}$$

given by

$$\begin{aligned} \Phi (s,t,(\beta ,l)):=\Big (\beta , \mathrm{Ad}^*(\hbox {exp}(\frac{s}{\langle l,Z_\beta \rangle } Z_{j_1})\hbox {exp}(\frac{t}{\langle l,Z_\beta \rangle } Z_{k_1}))l\Big ) \end{aligned}$$

is such a diffeomorphism. Hence every smooth G-invariant sub-manifold M of $({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta }$ can be decomposed into a direct product of ${\mathbb R}^2$ with the smooth manifold $M^0$, where

$$\begin{aligned} M^0:=\{(\beta ,l)\in M; \langle l,Z_{k_1}\rangle =\langle l,Z_{j_1}\rangle = 0\}. \end{aligned}$$

For $(\beta ,l) \in {\mathcal B}\times {\mathfrak {g}}^*$, one has $l_1(Z^1_i)=l(Z_i)$ if $i<k_1$ and $l_1(Z^1_i)=l(Z_{i+1})$ if $i\ge k_1$. We remark that for $(\beta ,l)$ and $(\beta ,l')$ in M with $\iota _1(\beta ,l)=\iota _1(\beta ,l')$ we have that l and $l'$ have the same restriction to ${\mathfrak {g}}_1(\beta ,l)={\mathfrak {g}}_1(\beta ,l')$, so they are on the same co-adjoint orbit and $l'=\mathrm{Ad}^*(y)l$ for some $y\in P(\beta ,l)$ and hence

$$\begin{aligned} F(\beta ,l)=F(\beta ,l') \end{aligned}$$

by the conditions on the operator fields defined over M given in Definition 2.7.1.

We denote the new variable Lie group by $\mathbb G_1=({\mathcal B}_1, G_1)$, where $G_1=(\text {exp}_{\beta _1} {\mathfrak {g}}_1)_{\beta _1 \in {\mathcal B}_1}$ and $\text {exp}_{\beta _1} {\mathfrak {g}}_1$ is the connected, simply connected, nilpotent Lie group associated to the Lie algebra $({\mathfrak {g}}_1, [\cdot , \cdot ]_{\beta _1})$.

4.5 Induction Step

To simplify notations, from now on we will omit the subscript $\beta$ in the notations of the multiplication and the exponential map, unless the subscript is crucial for the understanding. There are two preliminary steps to check.

4.5.1 Induction Hypothesis

In this subsection, we will prove the result for $({\mathcal B}, M)$ using induction. Let $M_1=\iota _1(M)$, where $\iota _1$ is constructed in (4.12). Let us recall that $l_1=l_1(\beta , l\vert _{{\mathfrak {c}}_1}) \equiv l\vert _{{\mathfrak {g}}_1(\beta , l\vert _{{\mathfrak {c}}_1})}$ for $(\beta ,l)\in ({\mathcal B}\times {\mathfrak {g}}^*)_{\le I}$. The Vergne polarisation ${\mathfrak {p}}(\beta ,l)$ for l in $({\mathfrak {g}},[\cdot , \cdot ]_{\beta })$, obtained by the procedure of Ludwig-Zahir (see [9, 11]), is also the Vergne polarisation for $l_1$ in $({\mathfrak {g}}_1, [\cdot , \cdot ]_{\beta })$. Let us denote by $P(\beta ,l)=\text {exp}_{\beta } {\mathfrak {p}}(\beta ,l))$ the corresponding subgroup. The associated induced representations will be denoted by $\pi _{(\beta ,l)} := \text {ind}^G_{P(\beta ,l)} \chi _l$, respectively, $\tilde{\pi }_{((\beta , l\vert _{{\mathfrak {c}}_1}), l_1)}:=\text {ind}^{G_1}_{P(\beta ,l)} \chi _{l_1}$. Then $\pi _{(\beta ,l)} \cong \text {ind}^G_{G_1} \tilde{\pi }_{((\beta , l\vert _{{\mathfrak {c}}_1}), l_1)}$, as usual.

Since M is G-invariant, the manifold $M_1=\iota _1(M)$ is also G-invariant in ${\mathcal B}_1\times {\mathfrak {g}}_1^*$. Hence we can write $M_1$ as a direct product manifold ${\mathbb R}^2\times M_1^0$, where

$$\begin{aligned} M_1^0:=\{((\beta , 0, q),l_1); \langle l_1,Z_{j_1}\rangle =0\} \end{aligned}$$

is $G_1$-invariant. Note that $M_1^0$ is contained in $({\mathcal B}_1\times {\mathfrak {g}}^*_1 )_{I_1}$ and for every $((\beta ,v,l_{\vert {\mathfrak {c}}_1}),l_1)\in M_1$ we have that $\infty> R>\vert \langle l_1,Z_\beta \rangle \vert>\delta >0$. The induction hypothesis in ${\mathcal B}_1\times {\mathfrak {g}}_1^*$ and $M_1^0 \subset ({\mathcal B}_1\times {\mathfrak {g}}_1^*)_{I_1}$ gives us an open relatively compact non-empty subset ${\mathcal M}_1^0$ of $M_1^0$ with the required properties of the theorem.

We choose now a relatively compact open subset ${\mathcal M}_1$ of $M_1$ such that $\overline{{\mathcal M}_1}\subset M_1$ and ${\mathcal M}_1$ is contained in $G\cdot {\mathcal M}_1^0$. Let

$$\begin{aligned} {\mathcal M}:=\iota _1^{-1}({\mathcal M}_1) \quad \text {and} \quad {\mathcal M}^0:=\iota _1^{-1}({\mathcal M}_1^0). \end{aligned}$$

Then ${\mathcal M}$ is non-empty open with its closure $\overline{{\mathcal M}}$ contained in M and ${\mathcal M}$ is contained in $G\cdot {\mathcal M}_0$. We take a kernel function $F\in {\mathcal D}^c_{M}$ such that its support is contained in ${\mathbb R}^r\times G\cdot {\mathcal M}\times G\times G$.

Given the kernel function F, we will now define a kernel function for the variable group $({\mathcal B}_1, G_1)$. For simplicity, we will omit the subscripts $\beta$ or $(\beta , v,l\vert _{{\mathfrak {c}}_1})$ in the notations of the multiplication and the exponential map, and we will identify $g_1, g_1'\in {\mathbb G}_1 =\big ((G_1)_{(\beta _1)}\big )_{\beta _1\in {\mathcal B}_1} \equiv {\mathfrak {g}}_1$ with the corresponding elements in $G_1$. In the following computations, the parameters $\beta$ and $(\beta ,v, l\vert _{{\mathfrak {c}}_1})$ will indicate how to multiply group elements or how to decompose smoothly the group elements, even if it is not marked explicitly. For $\iota _1(\beta ,l)=((\beta ,\langle l,Z_{k_1}\rangle ,l_{\vert {\mathfrak {c}}_1}), l_1)\in M_1$, we put

$$\begin{aligned}&F_1(\alpha , u, t, ((\beta ,\langle l,Z_{k_1}\rangle ,l_{\vert {\mathfrak {c}}_1}), l_1), g_1, g'_1) := \\&\quad \quad (2 \pi )^{n-j_1+1} \cdot \vert c(\beta , l) \vert \cdot F(\alpha , (\beta ,l), \hbox {exp}((u+t)X) \cdot g_1, \hbox {exp}(tX) \cdot g'_1), \end{aligned}$$

for $\alpha \in {\mathbb R}^r, u,t\in {\mathbb R}$ and $g_1,g_1'\in G_1$, where $c{(\beta ,l)}:= \langle l, [Z_{k_1}, Z_{j_1}]_{\beta }\rangle \ne 0$ and $X=Z_{k_1}$. This function $F_1$ has its support $S_1:=\iota _1(S)$ contained in $G\cdot {\mathcal M}_1$, and belongs to ${\mathcal D}^c_{M_1}$. The operator field $F_1$ is smooth on $M_1$, since the mappings F and c are both smooth. By the induction hypothesis, there exists $h \in {\mathcal S}( {\mathbb R}^{r+2}, {\mathcal B}_1, G_1)$ such that $\tilde{\pi }_{((\beta _1, l\vert _{{\mathfrak {c}}_1}), l_1)} \big ( h(\alpha , u, t, \beta _1, \cdot )\big )$ admits $F_1(\alpha , u, t, (\beta _1, l_1), \cdot , \cdot )$ as an operator kernel for all $(\beta _1, l_1) \in {M}_1^0$. The construction of the retract function f will now be done in several steps.

4.5.2 Definition of the Retract Function on the Original Group

For $(\beta , v, q)\in {\mathcal B}_1$, let us first define $\widetilde{h}$ by

$$\begin{aligned} \widetilde{h}(\alpha , u, t, (\beta ,v,q), g_1):= & {} \int _{{\mathbb R}} \int _{{\mathfrak {c}}_1} h(\alpha , u, t, (\beta ,v,q), g_1 \cdot \hbox {exp}(yY)\\&\cdot \hbox {exp}(Z)) e^{-iq(Z)} dZ dy, \end{aligned}$$

where $Y=Z_{j_1}$ and $Z=Z_\beta =[X,Y]_\beta$ with $X= Z_{k_1}$. The integral converges, as h is Schwartz in ${\mathfrak {g}}_1$ (for fixed $\beta _1$), and it is rapidly decreasing in $q\in ({\mathfrak {c}}_1)^*$, because it is a Fourier transform in Z. For all $(\beta ,v,q) \in {\mathcal B}\times {\mathbb R}\times {\mathfrak {c}}_1^*$, we then define

$$\begin{aligned}&\widetilde{f}(\alpha , (\beta ,v,q) , \hbox {exp}(uX) \cdot g_1 \cdot \hbox {exp}(yY) \cdot \hbox {exp}(Z))\\&\quad = e^{iq(Z)} \int _{{\mathbb R}} \widetilde{h}(\alpha , u, t, (\beta ,v,q), g_1^{-t}) e^{-ityq([X,Y]_{\beta })}dt \end{aligned}$$

with $g= \hbox {exp}(uX) \cdot g_1$ and $g_1^{-t}:= \hbox {exp}(-tX) \cdot g_1 \cdot \hbox {exp}(tX)$. The function $\widetilde{f}$ is smooth on ${\mathbb R}^r \times ({\mathcal B}\times {\mathfrak {c}}_1^*)\times G$. As $\widetilde{f}$ is of rapid decrease in $q\in {\mathfrak {c}}_1^*$ by construction, we may define f by

$$\begin{aligned} f(\alpha ,\beta , g):= \int _{{\mathfrak {c}}_1^*} \widetilde{f}(\alpha , (\beta ,0, q),g)dq, \quad \alpha \in {\mathbb R}^r, \beta \in {\mathcal B}, g\in G. \end{aligned}$$

One can see that $f\in {\mathcal S}({\mathbb R}^r, {\mathcal B}, G)$ (in the sense of Sect. 2.6).

4.5.3 The Retract Property

Let us now compute $\pi _{(\beta ,l)}\big (f(\alpha ,\beta ,\cdot ) \big )$ for $(\beta ,l)\in M$. Since the manifold M is contained in $({\mathcal B}\times {\mathfrak {g}}^*)_{\le I,R,\delta }$, we have that ${\mathfrak {c}}_1 \subset {\mathfrak {a}}_{\beta }(l)\subset {\mathfrak {g}}_{\beta }(l)$. If we identify $\hbox {exp}({\mathfrak {c}}_1)$ and ${\mathfrak {c}}_1$, as well as $\hbox {exp}(Z)$ and Z, for any function $\xi (\beta ) \in {\mathfrak {H}}_{(\beta ,l)}$ (the representation space of $\pi _{(\beta ,l)}$) and any $\tilde{g} \in G$, we have that

$$\begin{aligned}&\Big ( \pi _{(\beta ,l)} \big (f(\alpha ,\beta ,\cdot ) \big ) \xi (\beta ) \Big )(\tilde{g})\\&\quad = \int _{G_{\beta }/{\mathfrak {c}}_1} \int _{{\mathfrak {c}}_1} f(\alpha , \beta , g \cdot Z) \Big ( \pi _{(\beta ,l)}(g) \pi _{(\beta ,l)} (Z) \xi (\beta ) \Big )(\tilde{g}) dZ d\dot{g}\\&\quad = \int _{G_{\beta }/{\mathfrak {c}}_1} \int _{{\mathfrak {c}}_1} f(\alpha ,\beta , g \cdot Z) e^{-il(Z)} \Big ( \pi _{(\beta ,l)} (g) \xi (\beta ) \Big ) (\tilde{g}) dZ d\dot{g}\\&\quad = \int _{G_{\beta }/{\mathfrak {c}}_1} \int _{{\mathfrak {c}}_1} \int _{{\mathfrak {c}}_1^*} \tilde{f}(\alpha , (\beta ,0,q), g \cdot Z) e^{-il(Z)} \Big (\pi _{(\beta ,l)}(g) \xi (\beta ) \Big ) (\tilde{g}) dq dZ d\dot{g}\\&\quad = \int _{G_{\beta }/{\mathfrak {c}}_1} \int _{{\mathfrak {c}}_1} \int _{{\mathfrak {c}}_1^*} \tilde{f}(\alpha , (\beta ,0,q),g) e^{iq(Z)} e^{-il(Z)} \Big (\pi _{(\beta ,l)}(g) \xi (\beta )\Big ) (\tilde{g}) dq dZ d\dot{g}\\&\quad = \Big ( \frac{1}{2\pi }\Big )^{n-j_1}\int _{G_{\beta }/{\mathfrak {c}}_1} \tilde{f}(\alpha , (\beta , 0, l\vert _{{\mathfrak {c}}_1}), g) \Big (\pi _{(\beta ,l)}(g) \xi (\beta ) \Big ) (\tilde{g}) d\dot{g}. \end{aligned}$$

We use the following smooth decomposition: $X=Z_{k_1}, {\mathfrak {g}}_1={\mathfrak {g}}_1(\beta , l)$ which gives us

$$\begin{aligned} \tilde{g} = \hbox {exp}(sX) \cdot \tilde{g}_1 \quad \text {with} \quad s=s(\tilde{g}, \beta , l\vert _{{\mathfrak {c}}_1}), \tilde{g}_1= \tilde{g}_1(\tilde{g}, \beta , l\vert _{{\mathfrak {c}}_1}). \end{aligned}$$

We then obtain (using the fact that ${\mathfrak {c}}_1\subset {\mathfrak {a}}_\beta (l)$ for all our $(\beta ,l)$’s) that:

$$\begin{aligned}&(2\pi )^{n-j_1} \cdot \Big (\pi _{(\beta ,l)} \big (f(\alpha ,\beta , \cdot ) \big )\xi (\beta ) \Big )(\hbox {exp}(sX) \cdot \tilde{g}_1) \\&\quad =~ \int _{G_{\beta }/{\mathfrak {c}}_1} \tilde{f}(\alpha , (\beta , 0, l\vert _{{\mathfrak {c}}_1}), g) \Big (\pi _{(\beta ,l)}(g)\xi (\beta ) \Big )(\hbox {exp}(sX) \cdot \tilde{g}_1)d\dot{g}\\&\quad =~ \int _{{\mathbb R}} \int _{G_1/{\mathfrak {c}}_1} \tilde{f}(\alpha , (\beta , 0, l\vert _{{\mathfrak {c}}_1}), \hbox {exp}(uX)\cdot g_1)\cdot \xi (\beta )(g_1^{-1}\\&\qquad \cdot \hbox {exp}((s-u)X) \cdot \tilde{g}_1) d\dot{g}_1 du\\&\quad =~ \int _{{\mathbb R}} \int _{G_1/{\mathfrak {c}}_1} \tilde{f}(\alpha , (\beta , 0, l\vert _{{\mathfrak {c}}_1}), \hbox {exp}(uX) \cdot g_1)\cdot \\&\quad ~~~~~\tilde{\xi }(\beta , 0, l\vert _{z})(s-u)\Big (\big (\hbox {exp}(-(s-u)X) \cdot g_1 \cdot \hbox {exp}((s-u)X) \big )^{-1} \cdot \tilde{g}_1 \Big ) d\dot{g}_1 du\\&\quad \text {with } \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(v)(g_1):= \xi (\beta )(\hbox {exp}(vX)\cdot g_1)\\&\quad = \int _{{\mathbb R}} \int _{G_1/{\mathfrak {c}}_1} \tilde{f}(\alpha ,(\beta , 0, l\vert _{{\mathfrak {c}}_1}), \hbox {exp}((s-r)X)\cdot g_1) \cdot \\&~~~~~~~~\tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big (\big (\hbox {exp}(-rX) \cdot g_1 \cdot \hbox {exp}(rX) \big )^{-1} \cdot \tilde{g}_1 \Big )d\dot{g}_1 dr \quad (\text {with } s- u= r)\\&\quad =~ \int _{{\mathbb R}} \int _{G_1/{\mathfrak {c}}_1} \tilde{f}(\alpha , (\beta ,0, l\vert _{{\mathfrak {c}}_1}), \hbox {exp}((s-r)X) \cdot g_1^r) \cdot \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r)(g_1^{-1} \cdot \tilde{g}_1) d\dot{g}_1 dr\\&\quad \text {with }g_1^r=\hbox {exp}(rX) \cdot g_1 \cdot \hbox {exp}(-rX)\\&\quad =~ \int _{{\mathbb R}} \int _{G_1/{\mathfrak {c}}_1} \tilde{f}(\alpha , (\beta , 0, l\vert _{{\mathfrak {c}}_1}),\\&\qquad \hbox {exp}((s-r)X) \cdot g_1^r) \Big (\tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)}(g_1)\tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big ) (\tilde{g}_1) d\dot{g}_1 dr\\&\quad \text {with } l_1\equiv l\vert _{{\mathfrak {g}}_1}\\&\quad =~ \int _{{\mathbb R}} \int _{G_1/{\mathfrak {c}}_1} \int _{{\mathbb R}} \tilde{h}(\alpha , s-r, t,(\beta , 0, l\vert _{{\mathfrak {c}}_1}), g_1^{r-t})\\&\qquad \Big ( \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)}(g_1) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big )(\tilde{g}_1) dt d\dot{g}_1 dr \\&\quad =~ \int _{{\mathbb R}} \int _{G_1/\hbox {exp}({\mathbb R}Y)\cdot {\mathfrak {c}}_1} \int _{{\mathbb R}} \int _{{\mathbb R}} \tilde{h}(\alpha , s-r, t, (\beta , 0, l\vert _{{\mathfrak {c}}_1}), w_1^{r-t}\cdot (\hbox {exp}(yY)^{r-t}) \\&\qquad \Big ( \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)} (w_1)\tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big )(\tilde{g}_1) dt dy d\dot{w}_1 dr\\&\qquad (\text {as } l(Y)=0,\text { for } l_1 \equiv l\vert _{{\mathfrak {g}}_1(\beta , 0, l\vert _{{\mathfrak {c}}_1})})\\&\quad =~\int _{{\mathbb R}} \int _{G_1/\hbox {exp}({\mathbb R}Y)\cdot {\mathfrak {c}}_1} \int _{{\mathbb R}} \int _{{\mathbb R}} \tilde{h}(\alpha , s-r,t, (\beta , 0, l\vert _{{\mathfrak {c}}_1}), w_1^{r-t} \\&\qquad \cdot \hbox {exp}(yY)) \cdot e^{i(r-t) c(\beta , l)y} \Big ( \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)} (w_1) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big )(\tilde{g}_1) dtdyd\dot{w}_1 dr\\&\quad (\text {as }\big (\hbox {exp}(yY)\big )^{r-t}= \hbox {exp}(yY + y(r-t)[Z_{k_1}, Z_{j_1}]_{\beta }), \, \text {by the covariance of } \tilde{h}, \text {with} \\&\qquad c(\beta , l)= \langle l, [Z_{k_1}, Z_{j_1}]_{\beta }\rangle \ne 0)\\&\quad =~\frac{1}{\vert c(\beta , l)\vert } \int _{{\mathbb R}} \int _{G_1/\hbox {exp}({\mathbb R}Y)\cdot {\mathfrak {c}}_1} \int _{{\mathbb R}} \int _{{\mathbb R}} \tilde{h}(\alpha , s-r, t, (\beta , 0, l\vert _{{\mathfrak {c}}_1}), w_1^{r-t}\\&\qquad \cdot \hbox {exp}(c(\beta ,l)^{-1} \tilde{y}Y) ) e^{ir\tilde{y}} e^{-it \tilde{y}} \\&\qquad \Big ( \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)} (w_1) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big ) (\tilde{g}_1) dt d\tilde{y} d\dot{w}_1 dr \quad (\text {with } \tilde{y} = c(\beta ,l) y)\\ \end{aligned}$$

$$\begin{aligned}&\quad = \frac{1}{\vert c(\beta , l)\vert } \int _{{\mathbb R}} \int _{G_1/\hbox {exp}({\mathbb R}Y)\cdot {\mathfrak {c}}_1} \int _{{\mathbb R}} \int _{{\mathbb R}} \int _{{\mathbb R}} \int _{{\mathfrak {c}}_1}h(\alpha , s-r, t, (\beta , 0, l\vert _{{\mathfrak {c}}_1}), w_1^{r-t} \\&\qquad \cdot \hbox {exp}(c(\beta , l)^{-1} \tilde{y}Y) \cdot \hbox {exp}(y'Y) \cdot Z)\cdot e^{-il(Z)} e^{ir\tilde{y}} e^{-it \tilde{y}}\\&\qquad \Big ( \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)}(w_1) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big ) (\tilde{g}_1) dZ dy' dt d\tilde{y} d\dot{w}_1 dr\\&\quad =~\frac{1}{\vert c(\beta , l)\vert } \int _{{\mathbb R}} \int _{G_1/\hbox {exp}({\mathbb R}Y)\cdot {\mathfrak {c}}_1} \int _{{\mathbb R}} \int _{{\mathbb R}} \int _{{\mathbb R}} \int _{{\mathfrak {c}}_1} h(\alpha , s-r, t, (\beta , 0, l\vert _{{\mathfrak {c}}_1}), w_1^{r-t}\\&\qquad \cdot \hbox {exp}(y''Y) \cdot Z)\\&\qquad \cdot e^{-il(Z)} e^{ir\tilde{y}} e^{-it \tilde{y}} \Big ( \tilde{\pi }_{(\beta _1, l_1)}(w_1) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(r) \Big ) (\tilde{g}_1) dZ dy'' dt d\tilde{y} d\dot{w}_1 dr\\&\qquad (\text {for } y''= y'+ c(\beta , l)^{-1} \tilde{y})\\&\quad =~\frac{1}{2\pi } \cdot \frac{1}{\vert c(\beta , l)\vert } \int _{{\mathbb R}} \int _{G_1/\hbox {exp}({\mathbb R}Y)\cdot {\mathfrak {c}}_1} \int _{{\mathbb R}} \int _{{\mathfrak {c}}_1} h(\alpha , s-t, t,(\beta , 0, l\vert _{{\mathfrak {c}}_1}), \\&\qquad w_1\cdot \hbox {exp}(y''Y) \cdot Z)\\&\qquad \cdot e^{-il(Z)} \Big ( \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)}(w_1) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(t) \Big ) (\tilde{g}_1) dZ dy'' dt d\dot{w}_1\\&\quad =\frac{1}{2\pi }\cdot \frac{1}{\vert c(\beta , l)\vert } \int _{{\mathbb R}} \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)} \big ( h(\alpha , s{-}t, t, (\beta , 0, l\vert _{{\mathfrak {c}}_1}), \cdot ) \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(t) \big ) ( \tilde{g}_1) dt\\&\quad \text { as } e^{-il(Z)} \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)} (w_1) = \tilde{\pi }_{((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1)} (w_1 \cdot \hbox {exp}(y''Y)\cdot Z). \end{aligned}$$

Let us finish the computation for $(\beta , l) \in M$. It suffices to take $(\beta ,l)\in M^{0}$. Then $((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1) \in M_1^0$ and by the induction hypothesis,

$$\begin{aligned}&\Big ( \pi _{(\beta ,l)} \big (f(\alpha ,\beta , \cdot ) \big ) \xi (\beta ) \Big )(\tilde{g}) \\ { }&\quad =~\Big ( \pi _{(\beta ,l)} \big (f(\alpha ,\beta , \cdot ) \big )\xi (\beta ) \Big )(\hbox {exp}(sX) \cdot \tilde{g}_1) \\&\quad =~ \Big (\frac{1}{2\pi }\Big )^{n-j_1+1} \cdot \frac{1}{\vert c(\beta , l)\vert } \int _{{\mathbb R}} \int _{G_1/P(\beta ,l)} F_1(\alpha , s-t, t, ((\beta , 0, l\vert _{{\mathfrak {c}}_1}), l_1), \tilde{g}_1, g_1)\\&\qquad \tilde{\xi }(\beta ,0,l\vert _{{\mathfrak {c}}_1})(t)(g_1)d\dot{g}_1 dt\\&\quad =~ \int _{{\mathbb R}} \int _{G_1/P(\beta ,l)} F(\alpha ,(\beta ,l), \hbox {exp}(sX) \cdot \tilde{g}_1, \hbox {exp}(tX) \cdot g_1)\xi (\beta )(\hbox {exp}(tX)\cdot g_1) dg_1 dt \\&\quad =~ \int _{G/P(\beta ,l)} F(\alpha ,(\beta ,l), \tilde{g}, g)\xi (\beta )(g) dg. \end{aligned}$$

Hence for every $(\beta ,l)\in M$, we have the required result.

The algorithm used to build the retract function f respects the semi-norms defining the topology of our function spaces. So the retract map $F\mapsto f$ is continuous.

5 $\mathbf{G}$-prime ideals in $L^1(G)$

In this section, we will study the structure of the A-prime ideals in $L^1(G)$ by using the Retract Theorem.

5.1 A Retract Defined on Closed Orbits

Let $\mathbf{G}$ be a Lie group of automorphisms of a connected, simply connected, nilpotent Lie group $G=\hbox {exp}({\mathfrak {g}})$ containing the inner automorphisms of G. For instance take any simply connected Lie group $\mathbf{G}$ and let G be the nilradical of $\mathbf{G}$.

Let $l_0\in {\mathfrak {g}}^*$ be fixed, we consider the orbit $\Omega =\Omega _{l_0}:= {\mathbf{G}} \cdot l_0$ in ${\mathfrak {g}}^*$, let $O=O_{l_0}$ be the G-orbit of $l_0$. We assume that $\Omega$ is locally closed in ${\mathfrak {g}}^*$. In particular we can write

$$\begin{aligned} \Omega =\overline{\Omega }\cap U, \end{aligned}$$

where $\overline{\Omega }$ denotes the closure of $\Omega$ in ${\mathfrak {g}}^*$ and U is a $\mathbf{G}$-invariant open subset of ${\mathfrak {g}}^*$. It is then a $\mathbf{G}$-invariant smooth sub-manifold of ${\mathfrak {g}}^*$ diffeomorphic to the manifold $\mathbf G / \mathbf{G }_{l_0}$, where $\mathbf G _{l_0}$ denotes the stabiliser $\mathbf G _{l_0}:= \{\alpha \in \mathbf G ; \alpha \cdot l_0=l_0\}$. The $\mathbf{G}$-orbit ${\mathbf{G}}\cdot (G\cdot l_0)$ in the orbit space ${\mathfrak {g}}^*/G$ is then locally closed and homeomorphic to the quotient ${\mathbf{G}}/{\mathbf{G}}_{O}$, where ${\mathbf{G}}_O$ is the stabiliser of the set O in $\mathbf{G}$. In fact, we have that ${\mathbf{G}}_O=G \cdot {\mathbf{G}}_{l_0}$.

For a Jordan-Hölder basis ${\mathcal Z}=\{Z_1,\cdots , Z_n\}$ of ${\mathfrak {g}}$ and $\mathbf{g}\in {\mathbf{G}}$, let

$$\begin{aligned} \mathbf{g} \cdot {\mathcal Z}:=\{\mathrm{Ad}(\mathbf{g})Z_1,\cdots , \mathrm{Ad}(\mathbf{g})Z_n\}, \end{aligned}$$

which is again a Jordan-Hölder basis of ${\mathfrak {g}}$. For every index set I, we have the following relation (see [9]):

$$\begin{aligned} \mathrm{Ad}^*(\mathbf{g}){\mathfrak {g}}^*_{I,\mathbf{g}\cdot {\mathcal Z}}={\mathfrak {g}}^*_{I,{\mathcal Z}},\quad \mathbf{g}\in {\mathbf{G}}. \end{aligned}$$

(5.13)

For an index set I and a Jordan-Hölder basis ${\mathcal Z}$ of ${\mathfrak {g}}$, recall that

$$\begin{aligned} {\mathfrak {s}}_I:=\sum _{i\in I} {\mathbb R}Z_{i}^*, \quad \Sigma _{I,{\mathcal Z}}:={\mathfrak {s}}_I\cap {\mathfrak {g}}^*_{I,{\mathcal Z}}, \end{aligned}$$

and the mapping $E_{I}:{\mathbb R}^d\times \Sigma _{I,{\mathcal Z}}\rightarrow {\mathfrak {g}}^*_{I,{\mathcal Z}}$ is given by

$$\begin{aligned} E_I(s_1,t_1,\cdots , s_r,t_r; l):= \mathrm{Ad}^*(\hbox {exp}(s_1Z_{j_1})\hbox {exp}(t_1 Z_{k_1})\cdots \hbox {exp}(s_r Z_{j_r})\hbox {exp}(t_r Z_{k_r})) l. \end{aligned}$$

We have that $E_I$ is a bijection and $E_I({\mathbb R}^d\times \{l\})$ is the G-orbit of l. Let

$$\begin{aligned} \Upsilon : {\mathfrak {g}}^*_{I,{\mathcal Z}}\rightarrow \Sigma _{I,{\mathcal Z}}; \quad \Upsilon (l):=\mathrm{Ad}^*({\mathbf{G}})l\cap \Sigma _{I,{\mathcal Z}}= p_{\Sigma _{I,{\mathcal Z}}}(E_I^{-1}(l)), \end{aligned}$$

where $p_{\Sigma _{I,{\mathcal Z}}}$ is the projection of ${\mathbb R}^d\times \Sigma _{I,{\mathcal Z}}$ onto $\Sigma _{I,{\mathcal Z}}$.

For the orbit $\Omega$, we need to construct a finite partition of unity $(\psi _i)_{i\in \Gamma }$ consisting of smooth G-invariant functions $\psi _i: \Omega \rightarrow {\mathbb R}_+$ such that for every $i\in \Gamma$ the support of each function $\psi _i$ is contained in an open subset of ${\mathfrak {g}}^*_{I,\mathbf{g_i}\cdot {\mathcal Z}}$ for some $\mathbf{g}_i\in {\mathbf{G}}$. In order to do that let $\varphi : {\mathbb R}\rightarrow {\mathbb R}_+$ be a smooth function with compact support and vanish in a neighbourhood of 0. We define a function $\psi : {\mathfrak {g}}^*_{\le I}\rightarrow {\mathbb R}_+$ by

$$\begin{aligned} \psi (l):= \varphi (P_I(\Upsilon (l))) \, \text { if } l\in {\mathfrak {g}}^*_I \quad \text {and} \quad \psi (l):= 0 \, \text { if } l\in {\mathfrak {g}}^*_{<I}, \end{aligned}$$

where $P_I$ is a smooth function on ${\mathcal B}\times {\mathfrak {g}}^*$ defined in Sect. 2.4. We see that $\psi$ is smooth (since $\varphi$ vanishes in a neighbourhood of 0) and is G-invariant by the construction. Let

$$\begin{aligned} U_{I, {\mathcal Z}}:= \{l\in \Omega ; \psi (l)\ne 0\}. \end{aligned}$$

Now assume that ${\mathfrak {g}}^*_I= {\mathfrak {g}}^*_{I,{\mathcal Z}}$ be the maximal layer with respect to ${\mathcal Z}$ such that $\Omega \cap {\mathfrak {g}}^*_{I,{\mathcal Z}}\ne \emptyset$. We have that $\Omega \cap {\mathfrak {g}}^*_{I,\mathbf{g}\cdot {\mathcal Z}}\ne \emptyset$ but $\Omega \cap {\mathfrak {g}}^*_{I',\mathbf{g}\cdot {\mathcal Z}}= \emptyset$ for $\mathbf{g}\in {\mathbf{G}}$ and $I'> I$. Moreover, $U_{I,{\mathcal Z}}$ is a non-empty open subset of $\Omega$ contained in ${\mathfrak {g}}^*_{I}$ and

$$\begin{aligned} \Omega \subset \bigcup _{\mathbf{g\in G}}\mathrm{Ad}^*(\mathbf{g})U_{I, {\mathcal Z}}. \end{aligned}$$

Let C be a compact subset of ${\mathfrak {g}}^*$ contained in $\Omega$, then there exists a finite subset $\Gamma \subset \mathbf{G}$ such that

$$\begin{aligned} C\subset \bigcup _{\mathbf{g}\in \Gamma }\mathrm{Ad}^*(\mathbf{g})U_{I,{\mathcal Z}}. \end{aligned}$$

Hence there is a finite partition of unity $(\psi _i)_{i}$ consisting of smooth G-invariant functions $\psi _i: \Omega \rightarrow {\mathbb R}_+$ such that the support of each function $\psi _i$ is contained in $\mathrm{Ad}^*(\mathbf{g}_i)U_{I,{\mathcal Z}}\subset {\mathfrak {g}}^*_{I,\mathbf{g}_i\cdot {\mathcal Z}}$ for every $\mathbf{g}_i\in \Gamma$.

Suppose we have a smooth adapted operator field F on $\Omega$ supported on $G\cdot C$, we can write

$$\begin{aligned} F=\sum _{i\in \Gamma } \psi _i F. \end{aligned}$$

According to the Retract Theorem, for each $i\in \Gamma$, there is a (retract) Schwartz function $f_i$ on G such that

$$\begin{aligned} \pi _{l}(f_i)=op_{\psi _i F(l)} \end{aligned}$$

for every $l\in \Omega$. Let $f:= \sum _{i\in \Gamma } f_i$, we have that

$$\begin{aligned} \pi _{l}(f)= & {} \sum _{i\in \Gamma } \pi _{l}(f_i)\\ \nonumber= & {} \sum _{i\in \Gamma }\psi _i op_{F(l)}\\ \nonumber= & {} op_{F(l)}. \end{aligned}$$

This is, for every smooth adapted kernel function supported on $G\cdot C$, we build a retract function.

5.2 $\mathbf{G}$-Prime Ideals

As an application, we show that every $\mathbf{G}$-prime ideal in $L^1(G)$ is the kernel of a $\mathbf{G}$-orbit. Let us first recall the definition of $\mathbf{G}$-prime ideals.

Definition 5.2.1

A two-sided closed ideal $\mathfrak {I}$ in $L^{1}{(G)}$ is called $\mathbf{G}$-prime, if $\mathfrak {I}$ is $\mathbf{G}$-invariant and if, for all $\mathbf{G}$-invariant two-sided ideals $\mathfrak {I}_1$ and $\mathfrak {I}_2$ of $L^1(G)$, the following implication holds

$$\begin{aligned} \mathfrak {I}_1 * \mathfrak {I}_2 \subset \mathfrak {I} \Rightarrow \mathfrak {I}_1 \subset \mathfrak {I} \, \text { or } \, \mathfrak {I}_2 \subset \mathfrak {I}. \end{aligned}$$

Denote by $Prim^*(G)$ the collection of all the kernels of irreducible unitary representations of $L^1(G)$. For a closed subset C of $Prim^*(G)$, let

$$\begin{aligned} \hbox {ker}(C):= \bigcap _{P\in C}P. \end{aligned}$$

For a subset $\mathfrak {I}$ of $L^{1}{(G)}$, denote by $h(\mathfrak {I})$ the subset

$$\begin{aligned} h(\mathfrak {I}):= \{P\in Prim^*(G); \mathfrak {I} \subset P\}. \end{aligned}$$

The set $h(\mathfrak {I})$ is then closed in $Prim^*(G)$ with respect to the Fell topology.

We have the following result for $\mathbf{G}$-prime ideals of $L^1(G)$ which can be viewed as an application of the Retract Theorem.

Theorem 5.1

Let G be a simply connected, connected nilpotent Lie group and let $\mathbf{G}$ be a Lie group of automorphisms of G containing the inner automorphisms, which acts smoothly on the group G, such that every $\mathbf{G}$-orbit in ${\mathfrak {g}}^*$ is locally closed. If $\mathfrak {I}$ is a proper $\mathbf{G}$-prime ideal of $L^1(G)$, then there exists an $\mathbf{G}$-orbit $\Omega _{l_0}$ in ${\mathfrak {g}}^*$ such that

$$\begin{aligned} \mathfrak {I}= \hbox {ker}(\Omega _{l_0}). \end{aligned}$$

Moreover, the kernel of each $\mathbf{G}$-orbit is a $\mathbf{G}$-prime ideal.

Proof

For any ${\mathbf{G}}$-orbit $\Omega$ in ${\mathfrak {g}}^*$, the Retract Theorem tells us that the Schwartz functions contained in $\hbox {ker}(\Omega )$ are dense in $\hbox {ker}(\Omega )$ (see [7, proof of Proposition 4.1] and [4]). From the proof of [8, Theorem 1.2.12], it follows that the hull of a prime ideal $\mathfrak {I}$ is the closure of a ${\mathbf{G}}$-orbit in $Prim^*(G)\simeq \widehat{G}$. On the other hand, the density of ${\mathcal S}(G)\cap \hbox {ker}(\Omega )$ implies that $\hbox {ker}(\Omega )^N$ is contained in the minimal ideal $J(\Omega )$ with hull $\Omega$ for some $N\in {\mathbb N}$. This tells us that $\hbox {ker}(\Omega )^N\subset J(\Omega )\subset \mathfrak {I}$, since the minimal ideal with hull $\Omega$ is contained in every ideal with hull $\Omega$. Since $\mathfrak {I}$ is ${\mathbf{G}}$-prime, we have that $\mathfrak {I}=\hbox {ker}(\Omega )$.

Obviously the ideal $\hbox {ker}(\Omega )$ is ${\mathbf{G}}$-prime for any $\mathbf G$-orbit $\Omega$ in ${\mathfrak {g}}^*$. To see this, let $\mathfrak {I}_1$ and $\mathfrak {I}_2$ be two ${\mathbf{G}}$-invariant ideals of $L^{1}{(G)}$ such that $\mathfrak {I}_1*\mathfrak {I}_2\subset \hbox {ker}(\Omega )$. This means that $\mathfrak {I}_1*\mathfrak {I}_2\subset \hbox {ker}(\Omega )\subset \hbox {ker}(\pi _l)$ for some $l \in \Omega$. We have then either $\mathfrak {I}_1$ or $\mathfrak {I}_2$ is contained in $\hbox {ker}(\pi _l)$, since $\pi _l$ is irreducible. But if $\mathfrak {I}_1$ is contained in $\hbox {ker}(\pi _l)$, it is also contained in $\hbox {ker}( \pi _{k\cdot l})$ since $\mathfrak {I}_1$ is $\mathbf{G}$-invariant. Hence $\mathfrak {I}_1\subset \hbox {ker}(\Omega )$ and the proof is thus complete. $\square$