A reduction principle for Fourier coefficients of automorphic forms

Abstract

We consider a special class of unipotent periods for automorphic forms on a finite cover of a reductive adelic group $\mathbf {G}(\mathbb {A}_\mathbb {K})$, which we refer to as Fourier coefficients associated to the data of a ‘Whittaker pair’. We describe a quasi-order on Fourier coefficients, and an algorithm that gives an explicit formula for any coefficient in terms of integrals and sums involving higher coefficients. The maximal elements for the quasi-order are ‘Levi-distinguished’ Fourier coefficients, which correspond to taking the constant term along the unipotent radical of a parabolic subgroup, and then further taking a Fourier coefficient with respect to a ${\mathbb K}$-distinguished nilpotent orbit in the Levi quotient. Thus one can express any Fourier coefficient, including the form itself, in terms of higher Levi-distinguished coefficients. In companion papers we use this result to determine explicit Fourier expansions of minimal and next-to-minimal automorphic forms on split simply-laced reductive groups, and to obtain Euler product decompositions of certain Fourier coefficients.

1 Introduction

1.1 Main results

In this paper we establish a reduction principle for unipotent periods of an automorphic form on a reductive algebraic group $\mathbf{G}$ defined over a number field ${\mathbb K}$. Such periods generalize Fourier coefficients for classical modular forms, and are important because they often encode quantities of arithmetic interest. Our motivation comes in part from string theory, where certain so-called non-perturbative effects can be expressed in terms of unipotent periods [23, 24, 35]. This is explored further in a companion paper [21] that uses the results of the present paper in an essential way.

We work in the useful generality of a pair $(G,\Gamma ),$ where G is a finite central extension of the adele group $\mathbf{G}(\mathbb {A}) = \mathbf{G}(\mathbb {A}_{\mathbb K})$, and $\Gamma \subset G$ is a discrete subgroup on which the covering map restricts to an isomorphism with $\mathbf{G}({\mathbb K})$. This class includes the important central extensions defined in [8]. Also in this case, by [32, Appendix I], the covering map has a canonical splitting over every unipotent subgroup $N = \mathbf{N}(\mathbb {A})$, which we will use to identify N as a subgroup of G. If $\eta$ is a left $\Gamma$-invariant function on G, and $\chi$ is a character of N that is trivial on $N\cap \Gamma$, then the $(N,\chi )$-unipotent period of $\eta$ is the integral

$$\begin{aligned} \mathcal {U}_{(N,\chi )}[\eta ](g)=\int _{[N]}\eta (ng)\chi (n)^{-1}dn, \quad [N]:=({N\cap \Gamma })\backslash N. \end{aligned}$$

(1.1)

Since [N] is compact, the period integral converges under mild conditions on $\eta$; although it is quite unlikely that one can say anything useful in the generality of any N and $\chi$.

In this paper we consider a special class of unipotent periods ${\mathcal {F}}_{S,\varphi }$ associated to certain pairs $(S,\varphi )\in {\mathfrak {g}}\times {\mathfrak {g}}^*$, where ${\mathfrak {g}}={\mathfrak {g}}({\mathbb K})$ is the Lie algebra of ${\mathbf {G}}({\mathbb K})$. We refer to these as $(S,\varphi )$-Fourier coefficients, or simply as Fourier coefficients. As we explain in the next section, this class includes the unipotent periods studied in the theory of automorphic forms [14,15,16,17, 29], which we call neutral Fourier coefficients, and also those arising in string theory [13, 24, 35], which we call parabolic Fourier coefficients.

Our main result is a reduction algorithm that allows us to express a given Fourier coefficient in terms of others, which are higher with respect to a natural quasi-order $\succcurlyeq$. This is the more important direction for applications. It is much easier to write a Fourier coefficient in terms of lower coefficients; such expressions tend to be simpler but less useful — see Proposition 4.3.3 and its proof below, which uses the techniques of [19].

We now give a quick sketch of the essential ideas. We say $S\in {\mathfrak {g}}$ is ${\mathbb Q}$-semisimple if ${\text {ad}}(S)$ acts semisimply on ${\mathfrak {g}}$ with eigenvalues in ${\mathbb Q}$, and we write ${\mathfrak {g}}^S_{{\lambda }}$ for the $\lambda$-eigenspace. We set

$$\begin{aligned} {\mathfrak {g}}^S_{>{\lambda }}=\textstyle {\bigoplus \nolimits }_{\mu >{\lambda }}{\mathfrak {g}}^S_{\mu },\quad {\mathfrak {g}}^S_{\ge {\lambda }}=\textstyle {\bigoplus \nolimits }_{\mu \ge {\lambda }}{\mathfrak {g}}^S_{\mu }, \quad { {etc.}}, \end{aligned}$$

with similar notation for ${\mathfrak {g}}^*$. We say $(S,\varphi )\in {\mathfrak {g}}\times {\mathfrak {g}}^*$ is a Whittaker pair if S is ${\mathbb Q}$-semisimple and $\varphi \in ({\mathfrak {g}}^*)^S_{-2}$. In this case $\varphi$ is given by the Killing form pairing with a unique nilpotent element $f_\varphi$ in ${\mathfrak {g}}^S_{-2}$. We write ${\mathfrak {g}}_{\varphi } \subset {\mathfrak {g}}$ for the stabilizer of $\varphi$, and we set

$$\begin{aligned} {\mathfrak {n}}_{S,\varphi } = {\mathfrak {g}}^S_{> 1}\oplus ({\mathfrak {g}}^S_{1}\cap {\mathfrak {g}}_{\varphi }). \end{aligned}$$

(1.2)

Then ${\mathfrak {n}}_{S,\varphi }$ is a nilpotent Lie algebra and $\varphi \vert _{{\mathfrak {n}}_{S,\varphi }}$ is a Lie algebra character. By exponentiation we get a unipotent subgroup $N_{S,\varphi }\subset G$ and a character $\chi _{S,\varphi }$, and we define

$$\begin{aligned} {\mathcal {F}}_{S,\varphi }[\eta ]=\mathcal {U}_{(N_{S,\varphi },\chi _{S,\varphi })}[\eta ]. \end{aligned}$$

(1.3)

We now define a $\Gamma$-invariant quasi-order on Whittaker pairs. If $\varpi =(H,\varphi )$ and $\varpi '=(H',\varphi ')$ are Whittaker pairs, then we write $\varpi \succcurlyeq \varpi '$, first in the following two basic cases:

• (i) if $\varphi '=\varphi$, H and $H'$ commute, and we have

  $$\begin{aligned} {\mathfrak {g}}_{\varphi }\cap {\mathfrak {g}}^H_{\ge 1} \subseteq {\mathfrak {g}}^{H-H'}_{\ge 0} \end{aligned}$$

  (1.4)

• (ii) if $H=H'$, and there is a parabolic subalgebra $\mathfrak {p}=\mathfrak {l}+{\mathfrak {n}}$ defined over ${\mathbb K}$ such that

  $$\begin{aligned} H, f_{\varphi '} \in \mathfrak {l},\quad f_\varphi -f_{\varphi '} \in {\mathfrak {n}}. \end{aligned}$$

  (1.5)

More generally, we write $\varpi \succcurlyeq \varpi '$ if there is a sequence $\varpi _1,\ldots ,\varpi _n$ with $\varpi _1=\varpi$, $\varpi _n=\varpi '$, and elements $\gamma _1,\ldots ,\gamma _{n-1}$ in $\Gamma$ such that $\varpi _i \succcurlyeq \gamma _i\cdot \varpi _{i+1}$ in the sense of (i) or (ii).

We will also consider separately the quasi-orders given by (i) and (ii). Thus we will write

$$\begin{aligned} H\succcurlyeq _\varphi H' \text { if } (H,\varphi ) \succcurlyeq (H',\varphi );\quad \text { and }\quad \varphi \succcurlyeq _H \varphi ' \text { if } (H,\varphi ) \succcurlyeq (H,\varphi '). \end{aligned}$$

(1.6)

We say that a pair $(H,\varphi )$ is Levi-distinguished if H is maximal with respect to $\succcurlyeq _\varphi$. In this case, ${\mathcal {F}}_{H,\varphi }$ reduces to a $\mathbb {K}$-distinguished coefficient on a Levi quotient of a parabolic subgroup, by first taking the constant term along the unipotent radical—see Sect. 2.3 for details.

Algorithm A, our main result, described in Sect. 4, relates the Fourier coefficients for two pairs $(S,\varphi )\succcurlyeq (H,\varphi )$ and gives an explicit formula of the form

$$\begin{aligned} {\mathcal {F}}_{H,\varphi }[\eta ]= {\mathcal {M}}_H^S({\mathcal {F}}_{S,\varphi }[\eta ]) + \text {higher terms}. \end{aligned}$$

(1.7)

Here the “main” term ${\mathcal {M}}_H^S$ is a discrete sum of integral transforms that we describe shortly. The “higher” terms are similar expressions, detailed in Sect. 4, involving coefficients ${\mathcal {F}}_{H',\varphi '}$ such that ${(H',\varphi ')} \succcurlyeq {(H,\varphi )}$ but $\varphi '$ is not in the $\Gamma$-orbit of $\varphi$, which implies that the closure of the $\mathbf{G}({\mathbb C})$-orbit of $\varphi '$ properly contains that of $\varphi$. By iteration, we deduce that

• (a) any ${\mathcal {F}}_{H,\varphi }$ can be expressed via Levi-distinguished ${\mathcal {F}}_{S,\psi }$ satisfying $(S,\psi )\succcurlyeq (H,\varphi )$.

• (b) in particular, $\eta ={\mathcal {F}}_{0,0}[\eta ]$ can be expressed in terms of Levi-distinguished ${\mathcal {F}}_{S,\psi }$.

• (c) if $\eta$ is cuspidal (see Sect. 4.4) it can be expressed in terms of ${\mathbb K}$-distinguished $\mathcal {F}_{S,\psi }$.

We now define ${\mathcal {M}}_H^S$. For this we write $(\mathfrak {g}^H_{>\lambda })^S_\mu ={\mathfrak {g}}^H_{>{\lambda }}\cap {\mathfrak {g}}^S_\mu$ etc., and we set

$$\begin{aligned} {\mathfrak {u}}:= ({\mathfrak {g}}^{H}_{>1})^S_1, \quad {\mathfrak {v}}:= ({\mathfrak {g}}_{> 1}^{H})^S_{<1}, \quad {\mathfrak {w}}:=({\mathfrak {g}}^{H}_{1})^S_{<1}. \end{aligned}$$

(1.8)

We regard these as nilpotent subquotients of the Lie algebra ${\mathfrak {g}}$ via the identifications

$$\begin{aligned} {\mathfrak {u}}\cong ({\mathfrak {g}}^{H}_{>1})^{S}_{\ge 1}/({\mathfrak {g}}^{H}_{>1})^{S}_{>1}, \; {\mathfrak {v}}\cong {\mathfrak {g}}^H_{> 1}/({\mathfrak {g}}^H_{> 1})^{S}_{\ge 1}, \;{\mathfrak {w}}\cong {\mathfrak {g}}^H_{\ge 1}/\left( {\mathfrak {g}}^H_{> 1}+ ({\mathfrak {g}}^H_{\ge 1})^S_{\ge 1}\right) , \end{aligned}$$

(1.9)

and we define corresponding subquotients of the group G as follows:

$$\begin{aligned} U={\text {Exp}}({\mathfrak {u}}\otimes _\mathbb {K} \mathbb {A}),\quad V={\text {Exp}}({\mathfrak {v}}\otimes _\mathbb {K} \mathbb {A}),\quad \Omega ={\text {Exp}}({\mathfrak {w}}). \end{aligned}$$

(1.10)

We note that $\Omega$ is a discrete group since ${\mathfrak {w}}\subset {\mathfrak {g}}= {\mathfrak {g}}({\mathbb K})$ is a lattice. We now set

$$\begin{aligned} {\mathcal {M}}_H^S({\mathcal {F}}_{S,\varphi }[\eta ])(g) = \sum _{w\in \Omega }\, \int _{V} \int _{ {[ U]}} {\mathcal {F}}_{S,\varphi }[\eta ](wvug) \, dudv \qquad g \in G . \end{aligned}$$

(1.11)

There is an important special case where the higher terms in (1.7) vanish. We write $\psi \succ \varphi$ if $\psi \succcurlyeq _H \varphi$ for some H, but $\psi \not \in \Gamma \cdot \varphi$, and we say $\varphi$ is in the Whittaker support $\mathrm{WS}(\eta )$ of $\eta$ if ${\mathcal {F}}_{S,\varphi }[\eta ]\ne 0$ for some S, but if $\psi \succ \varphi$ then ${\mathcal {F}}_{S,\psi }[\eta ]= 0$ for all S.

Theorem B

(see Sect. 4.2) If $\varphi$ is in $\mathrm{WS}(\eta )$ and $(S,\varphi )\succcurlyeq (H,\varphi )$ then ${\mathcal {F}}_{H,\varphi }[\eta ] ={\mathcal {M}}_H^S({\mathcal {F}}_{S,\varphi }[\eta ]).$

1.2 Classes of Fourier coefficients and further results

The most natural way to complete a nilpotent $\varphi \in {\mathfrak {g}}^*$ to a Whittaker pair is provided by the Jacobson–Morozov theorem. Specifically, $\varphi$ is given by the Killing pairing with a unique nilpotent $f\in {\mathfrak {g}},$ and f in turn can be completed to an $\mathfrak {sl_2}$-triple (e, h, f). Then $(h,\varphi )$ is a Whittaker pair. We will call such pairs, and the corresponding coefficients, neutral. Such coefficients were extensively studied in [14,15,16,17, 29], where the name “Fourier coefficient” was reserved exclusively for this case.

Many applications use the class of parabolic Fourier coefficients, which are period integrals with respect to a character of the unipotent radical U of a parabolic subgroup $P=LU$. Despite their prevalence in the literature, these have in general not been evaluated in terms of known functions on $\mathbf {G}(\mathbb {A})$.

An important subclass of parabolic Fourier coefficients are those with respect to maximal unipotent subgroups, which we call Whittaker coefficients. When the automorphic form is spherical and the unipotent character is non-degenerate these factorize over primes and the non-archimedean factors can be explicitly computed using the Casselman–Shalika formula [10]. By Lemma 2.3.9 below these Whittaker coefficients are precisely the Levi-distinguished Fourier coefficients corresponding to nilpotent orbits that are principal in Levi subalgebras defined over ${\mathbb K}$. We will call such orbits PL-orbits (see Sect. 2.2 below for more details).

It follows from [19] that for any Whittaker pair $(H,\varphi )$ there exists $h\in {\mathfrak {g}}$ such that $(h,\varphi )$ is a neutral pair and $H\succcurlyeq _\varphi h$ (see Corollary 3.2.2 below). Moreover, by [19, Theorem C], if ${\mathcal {F}}_{h,\varphi }[\eta ]\equiv 0$ for some neutral pair $(h,\varphi )$ then ${\mathcal {F}}_{S,\varphi }[\eta ]\equiv 0$ for any Whittaker pair $(S,\varphi )$ with the same nilpotent element $\varphi$. Thus, neutral coefficients determine the Whittaker support ${\text {WS}}(\eta )$ and are often used to define it. More generally, if $S\succcurlyeq _\varphi H$ then ${\mathcal {F}}_{S,\varphi }$ can be obtained from ${\mathcal {F}}_{H,\varphi }$ by an integral transform (that can also involve discrete summation), see Proposition 4.3.3 below.

As an illustrative example we may take the coefficient $B={\mathcal {F}}_{S,\varphi }$ to be a Whittaker coefficient with a character supported only on the exponentiated root space of a single simple root, and the coefficient $C={\mathcal {F}}_{H,\varphi }$ to be a Fourier coefficient with respect to the unipotent radical of the standard maximal parabolic subgroup obtained from the same root, together with a restriction of the same character. Then, B can be obtained as a period integral of C over the quotient of the unipotent groups. Theorem B and Algorithm A allow us to go in the opposite direction and obtain C from B and from Levi-distinguished coefficients corresponding to higher orbits.

Let us discuss the relations between these classes of Fourier coefficients referring to Sect. 2 for more details. By Corollary 4.4.6 below, for any Whittaker pair $(H,\varphi )$, there exists an S such that $(S,\varphi )$ is a Levi-distinguished Whittaker pair, and $S\succcurlyeq _\varphi H$. In Lemma B.7 we show that Levi-distinguished Whittaker pairs have maximal dimension of ${\mathfrak {n}}_{S,\varphi }$ among all Whittaker pairs with the same $\varphi$. By Lemma 2.3.9 below, if $\varphi$ is a principal nilpotent in L then ${\mathcal {F}}_{H,\varphi }$ is a Whittaker coefficient. To summarize, for any orbit $\mathcal {O}$ we have that

$$\begin{aligned} \text {neutral} \; \preccurlyeq \; \text {any} \; \preccurlyeq \; \text {Levi-distinguished} \; \supseteq \text {Whittaker}. \end{aligned}$$

(1.12)

In particular, for the zero orbit $\{0\}$, the neutral coefficient is the identity map ${\mathcal {F}}_{0,0}[\eta ]=\eta$, while the Whittaker coefficient is the constant term map. From this summary we conclude that if all Fourier coefficients corresponding to non-PL orbits identically vanish then Algorithm A allows us to express $\eta$, and all its Fourier coefficients, through its Whittaker coefficients.

In the case $G={\text {GL}}_n(\mathbb {A})$ this generalizes the classical result by Piatetski-Shapiro and Shalika [34, 37] that expresses every cuspidal form through its Whittaker coefficients with respect to non-degenerate characters. We explain how Algorithm A allows us to reproduce this result in Sect. 5.3 below.

1.3 Related works

Our main tool is a generalization of the deformation technique of [18, 19], which in turn builds on the root-exchange method of [15, 17].

There are three crucial differences between the approach in the current paper and that of [18]: we consider automorphic forms (rather than mostly local representations), we consider the general case of the relation $S\succcurlyeq _{\varphi } H$ (rather than requiring H to be neutral), and we give explicit formulas relating various Fourier coefficients, while [18] concentrates on vanishing properties. A special case of Algorithm A in which $(H,\varphi )$ is neutral can be established using the technique of [18]. One could combine this with a similar, but easier, statement (Proposition 4.3.3 below) to relate any ${\mathcal {F}}_{H,\varphi }$ to any ${\mathcal {F}}_{S,\varphi }$. However this would be less useful than Algorithm A, which provides more compact expressions since it proceeds directly from $(H,\varphi )$ to $(S,\varphi )$ without a detour through neutral coefficients.

One can show that for split simply-laced groups the so-called minimal and next-to-minimal orbits are always PL. Thus Algorithm A and Theorem B give formulas for automorphic forms attached to the minimal and next-to-minimal representations of simply-laced groups, as well as all their Fourier coefficients, in terms of their Whittaker coefficients. We develop these formulas in a companion paper [21]. Another application of Theorem B is to deduce that certain Fourier coefficients are Eulerian [20].

A Fourier expansion for the discrete spectrum of ${\text {GL}}_n(\mathbb {A})$ is provided in [28], generalizing [34, 37]. In Sect. 5.3 below we apply Algorithm A to provide a further generalization.

1.4 Structure of the paper

In Sect. 2 we give the definitions of the notions mentioned above, as well as of Whittaker triples and quasi-Fourier coefficients. These are technical notions defined in [18] and widely used in the current paper as well.

In Sect. 3 we relate Fourier and quasi-Fourier coefficients corresponding to different Whittaker pairs and triples. To do that we further develop the deformation technique of [18, 19], making it more general, more explicit, and better adapted to the global case.

Section 4 contains the main results of the paper. In Sect. 4.1 we describe Algorithm A. The algorithm proceeds by deforming a Whittaker pair $(H,\varphi )$ to a bigger pair $(S,\varphi )$ along a straight line $H+t(S-H)$ with $t\in [0,1]$ in the Cartan subalgebra. At certain critical values t, additional quasi-Fourier coefficients (associated with higher orbits) are generated. These can be rewritten in terms of higher Fourier coefficients by proceeding in a straight line away from a neutral element, see Algorithm 4.1.1. The final result consists of the main term for $(S,\varphi )$ together with these higher Fourier coefficients. In Sect. 4.2 we prove that the algorithm is correct and terminates in a finite number of steps. We also derive Theorem B. In Sect. 4.3 we prove additional results, including a formula for ${\mathcal {F}}_{S,\varphi }$ in terms of ${\mathcal {F}}_{H,\varphi }$ (this direction is opposite to that of Theorem B). In Sect. 4.4 we give a constructive proof that for any Whittaker pair $(H,\varphi )$, there exists an S such that $(S,\varphi )$ is a Levi-distinguished Whittaker pair, and $S\succcurlyeq _\varphi H$.

In Sect. 5 we provide explicit examples in the cases ${\text {SL}}_4, {\text {GL}}_n, {\text {Sp}}_4$ and Heisenberg parabolics of arbitrary simply-laced groups.

Two appendices contain proofs of geometric lemmas on PL-orbits and on our order relation on rational nilpotent orbits.

2 Preliminaries on nilpotent orbits and Whittaker pairs

In this section, we fix basic notation for Whittaker pairs and Fourier coefficients. We also introduce some new preparatory results.

2.1 Whittaker pairs

As in the introduction, let ${\mathbb K}$ be a number field, $\mathfrak {o}$ its ring of integers, and let $\mathbb {A}=\mathbb {A}_{{\mathbb K}}$ be its ring of adeles. Let $\mathbb {T} = \{z \in \mathbb {C} : \left| {z}\right| = 1\}$ and fix a non-trivial additive character $\chi : \mathbb {A}\rightarrow \mathbb {T}$, which is trivial on ${\mathbb K}$. Then $\chi$ defines an isomorphism between $\mathbb {A}$ and the character group $\hat{\mathbb {A}} := {\text {Hom}}(\mathbb {A}, \mathbb {T})$ via the map $a\mapsto \chi _{a}$, where $\chi _{a}(b)=\chi (ab)$ for $a,b\in \mathbb {A}$. Furthermore, this isomorphism restricts to an isomorphism

$$\begin{aligned} \widehat{\mathbb {A}/{\mathbb K}} := {\text {Hom}}(\mathbb {A}/\mathbb {K}, \mathbb {T}) \cong \{r\in \hat{\mathbb {A}}\, : \,r|_{{\mathbb K}}\equiv 1\}=\{\chi _{a} : a \in {\mathbb K}\}\cong {\mathbb K}\, , \end{aligned}$$

(2.1)

which means that we may parametrize characters on $\mathbb {A}$ trivial on $\mathbb {K}$ by elements in $\mathbb {K}$.

Let $\mathbf{G}$ be a reductive group defined over ${\mathbb K}$, $\mathbf{G}(\mathbb {A})$ the group of adelic points of $\mathbf{G}$ and let G be a finite central extension of $\mathbf{G}(\mathbb {A})$. That is,

$$\begin{aligned} 1 \rightarrow C \rightarrow G \xrightarrow {{\text {pr}}} \mathbf {G}(\mathbb {A}) \rightarrow 1 \end{aligned}$$

(2.2)

for some finite group C.

We assume that there exists a section $\mathbf{G}({\mathbb K})\rightarrow G$ of the projection ${\text {pr}} : G {\twoheadrightarrow }\mathbf{G}(\mathbb {A})$. Fix such a section and denote its image in G by $\Gamma$. By [32, Appendix I], the cover $G{\twoheadrightarrow }\mathbf{G}(\mathbb {A})$ canonically splits over unipotent subgroups, and thus we will consider such subgroups as subgroups of G. Let ${\mathfrak {g}}({\mathbb K})$ denote the Lie algebra of $\mathbf{G}({\mathbb K}) \cong \Gamma$ which we will often abbreviate to $\mathfrak {g}$. Let $\mathfrak {v}$ be a nilpotent subalgebra of ${\mathfrak {g}}$ and let ${\mathfrak {v}}(\mathbb {A}):={\mathfrak {v}}\otimes _{{\mathbb K}}\mathbb {A}$ be its adelization. As in the introduction, we denote by ${\text {Exp}}({\mathfrak {v}})$ the unipotent subgroup of $\Gamma$ obtained by exponentiation of ${\mathfrak {v}}$ using the above split over unipotent subgroups, and we denote by $V:={\text {Exp}}({\mathfrak {v}}(\mathbb {A}))$ the unipotent subgroup of G obtained by exponentiation of ${\mathfrak {v}}(\mathbb {A})$. We note that ${\text {Exp}}(\mathfrak {v}) = V \cap \Gamma$ and for convenience we will denote by [V] the compact quotient $(V\cap \Gamma )\backslash V$.

To conveniently describe different unipotent subgroups of G and characters on these subgroups we introduce the following notion.

Definition 2.1.1

A Whittaker pair is an ordered pair $(S,\varphi )\in {\mathfrak {g}}\times {\mathfrak {g}}^*$ such that S is a rational semi-simple element (that is, a semi-simple element for which the eigenvalues of the adjoint action are in ${\mathbb Q}$), and ${\text {ad}}^*(S)\varphi =-2\varphi$.

We will often identify $\varphi \in {\mathfrak {g}}^*$ with its dual nilpotent element $f = f_\varphi \in \mathfrak {g}$ with respect to the Killing form $\langle \, , \, \rangle$. We will say that $\varphi$ is nilpotent if $f_\varphi$ is a nilpotent element of ${\mathfrak {g}}$. Equivalently, $\varphi \in {\mathfrak {g}}^*$ is nilpotent if and only if the Zariski closure of its coadjoint orbit includes zero. For example, if $(S,\varphi )$ is a Whittaker pair then $\varphi$ is nilpotent.

For any rational semi-simple $S\in {\mathfrak {g}}$ and $\lambda \in {\mathbb Q}$ we introduce the following notation

$$\begin{aligned} {\mathfrak {g}}_{\lambda }^{S}:=\{X\in {\mathfrak {g}}\, : \, [S,X]=\lambda X\}, \qquad {\mathfrak {g}}_{> \lambda }^{S}:=\bigoplus _{\begin{array}{c} \mu \in {\mathbb Q}\\ \mu>\lambda \end{array}}{\mathfrak {g}}_{\mu }^S, \qquad {\mathfrak {g}}_{\ge \lambda }^S :={\mathfrak {g}}_{\lambda }^{S}\oplus {\mathfrak {g}}_{>\lambda }^S\, , \end{aligned}$$

(2.3)

and analogously for ${\mathfrak {g}}_{< \lambda }^{S}$ and ${\mathfrak {g}}_{\le \lambda }^{S}$, with a similar use of notation for ${\mathfrak {g}}^*$.

For any $\varphi \in {\mathfrak {g}}^*$ let $\mathfrak {g}_\varphi$ be the centralizer of $\varphi$ in $\mathfrak {g}$ under the coadjoint action and define an anti-symmetric form $\omega _{\varphi } : \mathfrak {g} \times \mathfrak {g} \rightarrow \mathbb {K}$ by $\omega _{\varphi }(X,Y)=\varphi ([X,Y])$. We extend $\varphi$ and $\omega _\varphi$ to a functional and an anti-symmetric form on $\mathfrak {g}(\mathbb {A})$ respectively by linearity. Given a Whittaker pair $(S,\varphi ) \in {\mathfrak {g}}\times {\mathfrak {g}}^*$, we let $\mathfrak {u} := \mathfrak {g}^S_{\ge 1}$ and define

$$\begin{aligned} {\mathfrak {n}}_{S,\varphi }:=\{ X \in \mathfrak {u} \,:\, \omega _\varphi (X,Y)=0 \,\,\text {for all }Y\in \mathfrak {u}\} \quad \text {and} \quad N_{S,\varphi }:={\text {Exp}}\bigl ( {\mathfrak {n}}_{S,\varphi }(\mathbb {A}) \bigr ) \end{aligned}$$

(2.4)

which, by Lemma 3.2.5 below, can also be written as

$$\begin{aligned} \mathfrak {n}_{S,\varphi } = \mathfrak {g}^S_{>1} \oplus (\mathfrak {g}^S_1 \cap \mathfrak {g}_\varphi ) . \end{aligned}$$

(2.5)

Note that ${\mathfrak {n}}_{S,\varphi }$ is an ideal in $\mathfrak {u}$ with abelian quotient, and that $\varphi$ defines a character of ${\mathfrak {n}}_{S,\varphi }$.

We define a corresponding character $\chi _\varphi$ on $N_{S,\varphi }$, trivial on $N_{S,\varphi } \cap \Gamma$, by

$$\begin{aligned} \chi _\varphi (n) := \chi (\varphi (\log n)) = \chi (\langle f_\varphi , \log n\rangle ) \, . \end{aligned}$$

(2.6)

More generally, let $\mathfrak {r}\subseteq {\mathfrak {u}}$ be any isotropic subspace (not necessarily maximal) with respect to $\omega _\varphi |_{{\mathfrak {u}}}$, that includes $\mathfrak {n}_{S,\varphi }$. Note that ${\mathfrak {n}}_{S,\varphi } \subseteq \mathfrak {r} \subseteq {\mathfrak {u}}$, and that ${\mathfrak {n}}_{S,\varphi }$ and $\mathfrak {r}$ are ideals in ${\mathfrak {u}}$. Let $R={\text {Exp}}\bigl (\mathfrak {r}(\mathbb {A})\bigr )$. Then $\chi _{\varphi }^R : R \rightarrow \mathbb {T}$ defined by $\chi _\varphi ^R(r)=\chi (\varphi (\log r))$ is a character of R trivial on $R \cap \Gamma$. Indeed, since $\mathfrak {r}$ is isotropic, we have that $\omega _\varphi |_{\mathfrak {r(\mathbb {A})}} = 0$ and thus $\chi _\varphi ^R \in {\text {Hom}}(R, \mathbb {T})$, and $\varphi (X) \in \mathbb {K}$ for $X \in \mathfrak {r}(\mathbb {K})$.

Definition 2.1.2

We call a function on G an automorphic function if it satisfies the following properties:

• (i) invariant under the left action of $\Gamma$,

• (ii) finite under the right action of the preimage in G of $\prod _{\text {finite }\nu }{} \mathbf{G}(\mathfrak {o}_\nu ),$ and

• (iii) smooth when restricted to the preimage in G of $\prod _{\text {infinite }\nu }{} \mathbf{G}({\mathbb K}_\nu )$.

We denote the space of all automorphic functions by $C^{\infty }(\Gamma \backslash G)$.

Definition 2.1.3

Let $(S,\varphi )$ be a Whittaker pair for ${\mathfrak {g}}$ and let $R, N_{S,\varphi },\chi _{\varphi }$ and $\chi _{\varphi }^{R}$ be as above. For an automorphic function $\eta$, we define the Fourier coefficient of $\eta$ with respect to the pair $(S,\varphi )$ to be

$$\begin{aligned} {\mathcal {F}}_{S,\varphi }[\eta ](g):=\mathop \int \limits _{[N_{S,\varphi }]}\eta (ng)\, \chi _{\varphi }(n)^{-1}\, dn. \end{aligned}$$

(2.7)

We also define its R-Fourier coefficient to be the function

$$\begin{aligned} {\mathcal {F}}_{S,\varphi }^{R}[\eta ](g):=\mathop \int \limits _{[R]}\eta (rg)\, \chi _{\varphi }^{R}(r)^{-1}\, dr. \end{aligned}$$

(2.8)

Observe that if $\pi$ denotes a subrepresentation of $C^{\infty }(\Gamma \backslash G)$ that contains $\eta$ then ${\mathcal {F}}_{S,\varphi }[\eta ]$ and ${\mathcal {F}}_{S,\varphi }^{R}[\eta ]$ are matrix coefficients corresponding to the vector $\eta \in \pi$ and the functional on the space of automorphic functions defined by the integrals above.

Note that $\mathcal {F}_{0,0}[\eta ]=\eta$. For a general Whittaker pair $(S,\varphi ),$ $\mathcal {F}_{S,\varphi }[\eta ](g)$ is a smooth function on G in the above sense, but is not invariant under $\Gamma$ any more. On the other hand, its restriction to the joint centralizer $G_{S,\varphi }$ of S and $\varphi$ is left $G_{S,\varphi }\cap \Gamma$-invariant. As shown in [14], if $\eta$ is also $\mathfrak {z}$-finite and has moderate growth, then the restriction of $\eta$ to $G_{S,\varphi }$ still has moderate growth, but may stop being $\mathfrak {z}$-finite.

Remark 2.1.4

In [19, Sect. 6] the integrals (2.7) and (2.8) above are called Whittaker–Fourier coefficients, but in this paper we call them Fourier coefficients for short.

Definition 2.1.5

A Whittaker pair $(h,\varphi )$ is called neutral if either $(h,\varphi )=(0,0)$, or h and the Lie algebra element $f = f_\varphi$ that corresponds to $\varphi$ under the Killing form pairing can be completed to an -triple (e, h, f). Equivalently, $(h, \varphi )$ is called neutral if the map $X \mapsto {\text {ad}}^*(X)\varphi$ defines an epimorphism ${\mathfrak {g}}^h_{0}{\twoheadrightarrow }({\mathfrak {g}}^*)^h_{-2}$, and h can be completed to an -triple. For more details on -triples over arbitrary fields of characteristic zero see [7, Sect. 11].

Definition 2.1.6

We say that $(S,\varphi ,\varphi ')$ is a Whittaker triple if $(S,\varphi )$ is a Whittaker pair and $\varphi '\in ({\mathfrak {g}}^*)^{S}_{>-2}$.

For a Whittaker triple $(S,\varphi ,\varphi ')$, let U, R,  and $N_{S,\varphi }$ be as in Definition 2.1.3. Note that $\varphi +\varphi '$ defines a character of $\mathfrak {r}$. Extend it by linearity to a character of $\mathfrak {r}(\mathbb {A})$ and define an automorphic character $\chi _{\varphi +\varphi '}$ of R by $\chi _{\varphi +\varphi '}^{R}(\exp X):=\chi (\varphi (X)+\varphi '(X))$. For an example for this notation see Sect. 5.2 below.

Definition 2.1.7

For an automorphic function $\eta$, we define its $(S,\varphi ,\varphi ')$-quasi Fourier coefficient to be the function

$$\begin{aligned} {\mathcal {F}}_{S,\varphi ,\varphi '}[\eta ](g):=\mathop \int \limits _{[N_{S,\varphi }]}\chi _{\varphi +\varphi '}(n)^{-1}\eta (ng)dn. \end{aligned}$$

(2.9)

We also define its $(S,\varphi ,\varphi ',R)$-quasi Fourier coefficient to be the function

$$\begin{aligned} {\mathcal {F}}_{S,\varphi ,\varphi '}^{R}[\eta ](g):=\mathop \int \limits _{[R]}\chi _{\varphi +\varphi '}^{R}(r)^{-1}\eta (rg)dr. \end{aligned}$$

(2.10)

Definition 2.1.8

We call a ${\mathbb K}$-subgroup of $\mathbf{G}$ a split torus of rank m if it is isomorphic as a ${\mathbb K}$-subgroup to ${\text {GL}}_1^m$. We call a Lie subalgebra $\mathfrak {l}\subseteq {\mathfrak {g}}$ a ${\mathbb K}$-Levi subalgebra if it is the centralizer of a split torus.

Remark 2.1.9

We note that the Lie algebra of any split torus is spanned by rational semisimple elements. Consequently, a subalgebra of $\mathfrak {l}\subseteq {\mathfrak {g}}$ is a ${\mathbb K}$-Levi subalgebra if and only if it is the centralizer of a rational semisimple element of ${\mathfrak {g}}$. Another equivalent condition is that $\mathfrak {l}$ is the Lie algebra of a Levi subgroup of a parabolic subgroup of G defined over ${\mathbb K}$.

For convenience, we fix a complex embedding $\sigma :{\mathbb K}{\hookrightarrow }{\mathbb C}$, which allows us to map a $\Gamma$-orbit $\mathcal {O}$ in $\mathfrak {g}$ to a $\mathbf {G}(\mathbb {C})$-orbit in $\mathfrak {g}(\mathbb {C}) := \mathfrak {g} \otimes _{\sigma (\mathbb {K})} \mathbb {C}$. One can show, using [12], that the complex orbit corresponding to ${\mathcal {O}}$ does not depend on $\sigma$. However, we will not need this fact.

Definition 2.1.10

Let $(H,\varphi )$ and $(S,\varphi )$ be two Whittaker pairs with the same $\varphi$. We say that $(H,\varphi )$ dominates $(S,\varphi )$ if H and S commute and

$$\begin{aligned} {\mathfrak {g}}_{\varphi }\cap {\mathfrak {g}}^H_{\ge 1}\subseteq {\mathfrak {g}}^{S-H}_{\ge 0}\,. \end{aligned}$$

(2.11)

This relation is denoted $S \succcurlyeq _{\varphi } H$ in the introduction.

2.2 Principal nilpotent elements, PL elements and standard Whittaker pairs

We introduce some notions for coadjoint nilpotent orbits under the action of $\Gamma$. For general results on nilpotent orbits over algebraically closed fields see [9, 11].

Definition 2.2.1

We say that a nilpotent orbit under $\Gamma$ in ${\mathfrak {g}}^*$ is principal if it is Zariski dense in the nilpotent cone $\mathcal {N}({\mathfrak {g}}^*)$. We say that $\varphi \in {\mathfrak {g}}^*$ is a principal nilpotent element if its orbit is principal.

We say that a nilpotent $\varphi \in {\mathfrak {g}}^*$ is principal in a Levi (or PL for short) if there exists a ${\mathbb K}$-Levi subalgebra $\mathfrak {l}\subset {\mathfrak {g}}$ and a nilpotent element $f\in \mathfrak {l}$ such that the Killing form pairing with f defines $\varphi$ in ${\mathfrak {g}}^*$, and a principal nilpotent element of $\mathfrak {l}^*$. We call a nilpotent $\Gamma$-orbit in ${\mathfrak {g}}^*$ a PL-orbit if it consists of PL elements.

We remark that if G is quasi-split then a nilpotent element $\varphi \in {\mathfrak {g}}^*$ is principal if and only if it is regular, i.e. the dimension of its centralizer equals the rank of ${\mathfrak {g}}$.

Lemma 2.2.2

Let ${\mathfrak {n}}$ be the nilpotent radical of the Lie algebra of a minimal parabolic subgroup $P_0$. Then ${\mathfrak {n}}$ intersects any nilpotent orbit under $\Gamma$ in ${\mathfrak {g}}$.

Proof

Let $f\in {\mathfrak {g}}$ be nilpotent, and complete to an ${\mathfrak {sl}}_2$-triple (e, h, f). Then h defines a parabolic subgroup P which then includes a minimal parabolic $Q_0$. Then $Q_0$ is conjugate to $P_0$ under $\Gamma$ (see [5, Thm. 4.13(b)]). Since f lies in the nilpotent radical of the Lie algebra of P, its conjugate will lie in ${\mathfrak {n}}$. $\square$

Definition 2.2.3

We say that a Whittaker pair $(S,\varphi )$ is standard if ${\mathfrak {n}}_{S,\varphi }$ is the nilpotent radical of the Lie algebra of a minimal parabolic subgroup of G. In this case we will call the Fourier coefficient ${\mathcal {F}}_{S,\varphi }$ a Whittaker coefficient.

Remark 2.2.4

In [19, Sect. 6] the Whittaker coefficients are called principal degenerate Whittaker–Fourier coefficients.

Corollary 2.2.5

A nilpotent element $\varphi \in {\mathfrak {g}}^*$ is principal if and only if it can be completed to a neutral standard Whittaker pair.

Proof

Let h complete $\varphi$ to a neutral standard Whittaker pair. Then ${\mathfrak {n}}_{h,\varphi }={\mathfrak {g}}^h_{>1}={\mathfrak {g}}^h_{>0}$ is the nilpotent radical of the Lie algebra of a minimal parabolic subgroup, and thus so is ${\mathfrak {n}}:={\mathfrak {g}}^h_{<1}$. Let $f\in {\mathfrak {n}}$ define $\varphi$ through the Killing form pairing. Then we have $[f,{\mathfrak {g}}^h_{\le 0}]={\mathfrak {n}}$ and thus $\Gamma f\cap {\mathfrak {n}}$ is Zariski open, and thus Zariski dense, in ${\mathfrak {n}}$. The statement follows now from Lemma 2.2.2.

Conversely, let $\varphi \in {\mathfrak {g}}^*$ be a principal nilpotent, and let (e, h, f) be an ${\mathfrak {sl}}_2$-triple such that f defines $\varphi$ via the Killing form. Let ${\mathcal {O}}$ denote the complex orbit of f and $\bar{\mathcal {O}}$ denote its Zariski closure. Then $\bar{\mathcal {O}}= {\mathcal {N}}({\mathfrak {g}})\supset {\mathfrak {g}}^h_{<0}$. Thus ${\mathcal {O}}$ is the Richardson orbit for ${\mathfrak {g}}^h_{\le 0}$, and thus $\dim {\mathcal {O}}=2\dim {\mathfrak {g}}^h_{<0}$. Now suppose by way of contradiction that the pair $(h,\varphi )$ is not standard. Then ${\mathfrak {g}}^h_{\le 0}$ is not a minimal ${\mathbb K}$-parabolic subalgebra, i.e. there exists a smaller ${\mathbb K}$-parabolic subalgebra $\mathfrak {p}$ with nilpotent radical ${\mathfrak {n}}\supsetneq {\mathfrak {g}}^h_{<0}$. But ${\mathfrak {n}}\subset {\mathcal {N}}= \bar{\mathcal {O}}$, and thus ${\mathcal {O}}$ is a Richardson orbit for $\mathfrak {p}$, thus $\dim {\mathcal {O}}= 2\dim {\mathfrak {n}}>2\dim {\mathfrak {g}}^h_{<0}=\dim {\mathcal {O}}$ which is a contradiction. $\square$

Corollary 2.2.6

A nilpotent $\varphi \in {\mathfrak {g}}^*$ is PL if and only if it can be completed to a standard Whittaker pair $(S,\varphi )$.

Proof

Let $(S,\varphi )$ be a standard Whittaker pair. Then $S=h+Z$ where $(h,\varphi )$ is neutral and commutes with Z. Then Z defines a Levi subalgebra $\mathfrak {l}$, and the Whittaker pair $(h,\varphi )$ is neutral and standard in $\mathfrak {l}$. By Corollary 2.2.5, $\varphi$ is principal in $\mathfrak {l}$.

Conversely, if $\varphi$ is principal in $\mathfrak {l}$ and Z defines $\mathfrak {l}$ we let $S:=TZ+h$ for $T\in {\mathbb Q}_{>0}$ big enough. Then $(S,\varphi )$ is a standard Whittaker pair. $\square$

Let us remark that in [19] a different definition of principal and PL elements was given. The following lemma states the equivalence of the definitions.

Lemma 2.2.7

Let $\varphi \in {\mathfrak {g}}^*$ be nilpotent. Then

• (i) $\varphi$ is PL if and only if there exist a maximal split toral subalgebra ${\mathfrak {a}}$ of $\mathfrak {g}$ and a choice of associated simple roots $\Pi$ such that $\varphi \in \bigoplus _{\alpha _i \in \Pi } \mathfrak {g}^*_{\alpha _i}$, where $\mathfrak {g}^*_{\alpha _i}$ denotes the dual of the root space ${\mathfrak {g}}_{{\alpha }_i}$.

• (ii) If $\varphi \in \bigoplus _{\alpha _i \in \Pi } \mathfrak {g}^*_{\alpha _i}$ then $\varphi$ is principal in the Levi subalgebra given by those simple roots ${\alpha }_i$ for which the projection of $\varphi$ to $\mathfrak {g}^*_{\alpha _i}$ is non-zero.

Proof

Let $\sum _{{\alpha }_i\in \Pi } {\mathfrak {g}}^{\times }_{{\alpha }_i}\subset \bigoplus _{\alpha _i \in \Pi } \mathfrak {g}^*_{\alpha _i}$ denote the subset of vectors with all projections non-zero. It is enough to show that $\varphi \in {\mathfrak {g}}^*$ is principal if and only if there exist $({\mathfrak {a}},\Pi )$ as above such that $\varphi \in \sum _{{\alpha }_i\in \Pi } {\mathfrak {g}}^{\times }_{{\alpha }_i}$.

To show that, assume first that $\varphi$ is principal. Then, by Corollary 2.2.5, $\varphi$ can be completed to a neutral standard pair $(h,\varphi )$. Then h defines a torus and simple roots, and we have $\varphi \in \sum _{{\alpha }_i\in \Pi } {\mathfrak {g}}^{\times }_{{\alpha }_i}$. Conversely, give ${\mathfrak {a}}$ and $\Pi$ as above, we let $h:=\sum c_i {\alpha }_i^\vee$, where ${\alpha }_i^\vee$ are the coroots given by scalar product with ${\alpha }_i$, and $c_i$ are chosen such that $\varphi \in ({\mathfrak {g}}^*)^h_{-2}$. Then $(h,\varphi )$ is a standard Whittaker pair. Moreover, $\varphi$ is a generic element of $({\mathfrak {g}}^*)^h_{-2}$ and thus the Jacobson–Morozov theory implies that $(h,\varphi )$ is a neutral pair. $\square$

Remark 2.2.8

Note that for $G={\text {GL}}_n(\mathbb {A})$ all orbits $\mathcal {O}$ are PL-orbits. In general this is, however, not the case, see Appendix A for details.

2.3 Levi-distinguished Fourier coefficients

Definition 2.3.1

We say that a nilpotent $f \in {\mathfrak {g}}$ is ${\mathbb K}$-distinguished, if it does not belong to a proper ${\mathbb K}$-Levi subalgebra $\mathfrak {l}\subsetneq {\mathfrak {g}}$. In this case we will also say that $\varphi \in {\mathfrak {g}}^*$ given by the Killing form pairing with f is ${\mathbb K}$-distinguished. We will also say that the orbit of $\varphi$ is ${\mathbb K}$-distinguished.

Example 2.3.2

The nilpotent orbits for $\mathrm {Sp}_{2n}({\mathbb C})$ are given by partitions of 2n such that odd parts have even multiplicity. Each such orbit, except the zero one, decomposes to infinitely many $\mathrm {Sp}_{2n}({\mathbb Q})$-orbits — one for each collection of equivalence classes of quadratic forms $Q_1,\dots ,Q_k$ of dimensions $m_1,\dots m_k$ where k is the number of even parts in the partition and $m_1,\dots m_k$ are the multiplicities of these parts. A complex orbit is distinguished over $\mathbb {C}$ (i.e. does not intersect a proper Levi subalgebra) if and only if all parts have multiplicity one (and thus there are no odd parts). To see the “only if” part note that if the partition includes a part k with multiplicity two then the orbit intersects the Levi ${\text {GL}}_k \times \mathrm {Sp}_{2(n-k)}$. If k is odd then this Levi is defined over ${\mathbb Q}$ and thus all ${\mathbb Q}$-distinguished orbits correspond to totally even partitions. If k is even then this Levi is defined over ${\mathbb Q}$ if and only if the quadratic form on the multiplicity space of k is not anisotropic. Thus, we obtain that a necessary condition for an orbit ${\mathcal {O}}$ to be ${\mathbb Q}$-distinguished is that its partition ${\lambda }({\mathcal {O}})$ is totally even, a sufficient condition is that ${\lambda }({\mathcal {O}})$ is multiplicity free, and for totally even partitions with multiplicities there are infinitely many ${\mathbb Q}$-distinguished orbits and at least one not ${\mathbb Q}$-distinguished. For example, for the partition (4, 2) all orbits in $\mathfrak {sp}_{6}({\mathbb Q})$ are ${\mathbb Q}$-distinguished, for the partition $2^3$ some orbits are ${\mathbb Q}$-distinguished and some are not, and all other partitions do not correspond to ${\mathbb Q}$-distinguished orbits.

Lemma 2.3.3

Every principal nilpotent element is ${\mathbb K}$-distinguished.

Proof

Let $f\in {\mathfrak {g}}$ define a principal nilpotent element via the Killing form. Suppose the contrary: f lies in a proper ${\mathbb K}$-Levi subalgebra $\mathfrak {l}$ of ${\mathfrak {g}}$. Let $Z\in {\mathfrak {g}}$ be a rational semi-simple element such that $\mathfrak {l}= \mathfrak {g}^Z_0$. Complete f to an ${\mathfrak {sl}}_2$-triple $\gamma :=(e,h,f)$ in $\mathfrak {l}$. Then ${\text {ad}}(Z)$ acts by a scalar on every irreducible submodule of the adjoint action of $\gamma$ on ${\mathfrak {g}}$. Since $\mathfrak {l}\ne {\mathfrak {g}}$, there exists an irreducible submodule V on which ${\text {ad}}(Z)$ acts by a negative scalar $-c$. Let v be a highest weight vector of V of weight d, and let $S:=h+c^{-1}(d+2)Z$. Then $v+f\in {\mathfrak {g}}^S_{-2}$ and thus $v+f$ is nilpotent. Since f is principal, $v+f$ lies in the Zariski closure of $\Gamma f$. On the other hand, $v+f$ belongs to the affine space $f+{\mathfrak {g}}^e$, which is called the Slodowy slice to $\Gamma f$ at f, and is transversal to $\Gamma f$, contradicting the assumption that $v+f$ lies in the Zariski closure of $\Gamma f$. $\square$

Lemma 2.3.4

Let $f\in {\mathfrak {g}}$ be nilpotent. Then all ${\mathbb K}$-Levi subalgebras $\mathfrak {l}\subseteq {\mathfrak {g}}$ such that $f \in \mathfrak {l}$ and f is ${\mathbb K}$-distinguished in $\mathfrak {l}$ are conjugate by the centralizer of f.

Proof

Complete f to an -triple $\gamma :=(e,h,f)$ and denote its centralizer by $G_{\gamma }$. Let us show that all ${\mathbb K}$-Levi subalgebras $\mathfrak {l}$ of ${\mathfrak {g}}$ that contain $\gamma$ and in which f is distinguished are conjugate by $G_\gamma$. Let $\mathfrak {l}$ be such a subalgebra, $L\subseteq G$ be the corresponding Levi subgroup, and let C denote the maximal split torus of the center of L. Then C is a split torus in $G_{\gamma }$. Let us show that it is a maximal split torus. Let $T\supseteq C$ be a larger split torus in $G_{\gamma }$. Then, the centralizer of T in ${\mathfrak {g}}$ is a ${\mathbb K}$-Levi subalgebra that lies in $\mathfrak {l}$ and includes $\gamma$, and thus is equal to $\mathfrak {l}$. Thus $T=C$.

Since $\mathfrak {l}$ is the centralizer of T in G, T is a maximal split torus of $G_{\gamma }$, and all maximal split tori of reductive groups are conjugate (see [3, 15.14]), we get that all the choices of L are conjugate.

Since all the choices of $\gamma$ are conjugate by the centralizer of f, the lemma follows. $\square$

Definition 2.3.5

Let $Z\in {\mathfrak {g}}$ be a rational-semisimple element and $\mathfrak {l}$ denote its centralizer. Let $(h,\varphi )$ be a neutral Whittaker pair for $\mathfrak {l}$, such that the orbit of $\varphi$ in $\mathfrak {l}^*$ is ${\mathbb K}$-distinguished. We say that the Whittaker pair $(h+Z,\varphi )$ is Levi-distinguished if

$$\begin{aligned} {\mathfrak {g}}^{h+Z}_{> 1}={\mathfrak {g}}^{h+Z}_{\ge 2}= {\mathfrak {g}}^{Z}_{>0}\oplus \mathfrak {l}^{h}_{\ge 2}\text { and }{\mathfrak {g}}^{h+Z}_{ 1}=\mathfrak {l}^{h}_{1}. \end{aligned}$$

(2.12)

In this case we also say that the Fourier coefficient ${\mathcal {F}}_{h+Z,\varphi }$ is Levi-distinguished.

Remark 2.3.6

Let $(h,\varphi )$ be a neutral Whittaker pair for ${\mathfrak {g}}$. If $\varphi$ is ${\mathbb K}$-distinguished then ${\mathcal {F}}_{h,\varphi }$ is a Levi-distinguished Fourier coefficient. If a rational semi-simple Z commutes with h and with $\varphi$, and $\varphi$ is ${\mathbb K}$-distinguished in $\mathfrak {l}:={\mathfrak {g}}^Z_0$ then ${\mathcal {F}}_{h+TZ,\varphi }$ is a Levi-distinguished Fourier coefficient for any T bigger than $m/M+1$, where m is the maximal eigenvalue of h and M is the minimal positive eigenvalue of Z. See also Lemma 4.4.5 for further discussion.

Lemma 2.3.7

([19, Lemma 3.0.2]) For any Whittaker pair $(H,\varphi )$ there exists $Z\in {\mathfrak {g}}^{H}_0$ such that $(H-Z,\varphi )$ is a neutral Whittaker pair.

Remark 2.3.8

In [19] the lemma is proven over a local field, but the proof only used the Jacobson–Morozov theorem, that holds over arbitrary fields of characteristic zero.

Lemma 2.3.9

For any Whittaker pair $(H,\varphi )$, the following are equivalent:

• (i) $(H,\varphi )$ is standard pair

• (ii) $(H,\varphi )$ is a Levi-distinguished Fourier coefficient, and $\varphi$ is a PL nilpotent.

Proof

First let $(H,\varphi )$ be a standard pair. Then by Lemma 2.3.7, H can be decomposed as $H=h+Z$ where $(h,\varphi )$ is a neutral pair and Z commutes with h and with $\varphi$. Let $\mathfrak {l}$ and L denote the centralizers of Z in ${\mathfrak {g}}$ and G, and $N:=N_{H,\varphi }$. Then N is the unipotent radical of a minimal parabolic subgroup of G, and L is a Levi subgroup of G. Thus, $N\cap L$ is the unipotent radical of a minimal parabolic subgroup of L. The Lie algebra of $N\cap L$ is ${\mathfrak {n}}_{H,\varphi }\cap {\mathfrak {g}}^Z_0={\mathfrak {g}}^{h}_{\ge 1}\cap {\mathfrak {g}}^Z_0$. Thus, ${\text {Exp}}({\mathfrak {g}}^h_{\le -1}\cap {\mathfrak {g}}^Z_0)$ is the unipotent radical of a minimal parabolic subgroup of L. Since $\varphi$ is given by Killing form pairing with $f\in {\mathfrak {g}}^h_{\le -1}\cap {\mathfrak {g}}^Z_0$, we get by Corollary 2.2.5 that $\varphi$ is principal in $\mathfrak {l}$. Replacing Z by tZ with t large enough, we obtain that $(H,\varphi )$ is a Levi-distinguished pair.

Now, assume that $\varphi$ is a PL nilpotent, and let $(H, \varphi ) = (h+Z,\varphi )$ be a Levi-distinguished pair. Let $\mathfrak {l}={\mathfrak {g}}^Z_0$ be the corresponding Levi subalgebra, and let $f=f_{\varphi }$ be the element of ${\mathfrak {g}}$ that defines $\varphi$. Since f is distinguished in $\mathfrak {l}$, and principal in some Levi subalgebra, Lemmas 2.3.3 and 2.3.4 imply that f is principal in $\mathfrak {l}$. Thus, ${\mathfrak {n}}_{H,\varphi }\cap \mathfrak {l}$ is the nilpotent radical of the Lie algebra of a minimal parabolic subgroup of L and thus ${\mathfrak {n}}_{H,\varphi }={\mathfrak {n}}_{H,\varphi }\cap \mathfrak {l}\oplus {\mathfrak {g}}^Z_{>0}$ the nilpotent radical of the Lie algebra of a minimal parabolic subgroup of G. Thus $(H,\varphi )$ is standard Whittaker pair. $\square$

Lemma 2.3.10

Let $(H,\varphi )$ and $(S,\varphi )$ be Levi-distinguished Whittaker pairs with the same $\varphi$. Then $\dim {\mathfrak {n}}_{S,\varphi }=\dim {\mathfrak {n}}_{H,\varphi }$

Proof

By the definition of Levi-distinguished Whittaker pair, there exists a decomposition $H=h+Z$ such that $\varphi$ is given by the Killing form pairing with a distinguished element $f\in \mathfrak {l}={\mathfrak {g}}^Z_0$ and (2.12) holds. Let $S=h'+Z'$ be a decomposition for S satisfying the same properties, and let $\mathfrak {l}':={\mathfrak {g}}^{Z'}_0$. By Lemma 2.3.4, $\mathfrak {l}$ and $\mathfrak {l}'$ are conjugate by the centralizer of f, and thus we can assume that $\mathfrak {l}=\mathfrak {l}'$. By the Jacobson–Morozov theorem, h and $h'$ are conjugate by the centralizer of f in $\mathfrak {l}$. Now,

$$\begin{aligned} \dim {\mathfrak {n}}_{H,\varphi }=\dim \mathfrak {l}^h_{\ge 2}+\dim {\mathfrak {g}}^Z_{>0}=\dim \mathfrak {l}^h_{\ge 2}+(\dim {\mathfrak {g}}-\dim \mathfrak {l})/2= \dim {\mathfrak {n}}_{S,\varphi }\,. \end{aligned}$$

(2.13)

$\square$

In Lemma B.7 (ii) below we show that Levi-distinguished pairs have maximal dimension of ${\mathfrak {n}}_{S,\varphi }$ among all Whittaker pairs with nilpotent element $\varphi$.

2.4 Order on nilpotent orbits and Whittaker support

Definition 2.4.1

We define a partial order on nilpotent orbits in ${\mathfrak {g}}^* = {\mathfrak {g}}^*({\mathbb K})$ to be the transitive closure of the following relation R: $({\mathcal {O}},{\mathcal {O}}')\in R$ if there exist $\varphi \in {\mathcal {O}},$ rational semi-simple $H,Z\in {\mathfrak {g}}$, and $\varphi '\in ({\mathfrak {g}}^*)^Z_{>0 } \cap ({\mathfrak {g}}^*)^H_{-2}$ such that $\varphi \in ({\mathfrak {g}}^*)^Z_0 \cap ({\mathfrak {g}}^*)^H_{-2 }$, $[H,Z]=0,$ and $\varphi +\varphi '\in {\mathcal {O}}'$.

In the notation of the introduction, the conditions in the definition read $\varphi +\varphi '\succcurlyeq _H \varphi$, the parabolic subalgebra in (1.5) being ${\mathfrak {g}}^Z_{\ge 0}$. In Appendix B, we study these rational orbits in more detail. In particular, in Corollary B.3 we prove that this is indeed a partial order, i.e. that R is anti-symmetric. We will thus denote ${\mathcal {O}}\le {\mathcal {O}}'$ (or ${\mathcal {O}}'\ge {\mathcal {O}}$) if $({\mathcal {O}},{\mathcal {O}}')$ lies in the transitive closure of R, and ${\mathcal {O}}<{\mathcal {O}}'$ if ${\mathcal {O}}\le {\mathcal {O}}'$ and ${\mathcal {O}}\ne {\mathcal {O}}'$. By Corollary B.2 below this implies an inequality on the dimensions of complexifications: $\dim {\mathcal {O}}_{{\mathbb C}}<\dim {\mathcal {O}}'_{{\mathbb C}}$.

Lemma 2.4.2

If ${\mathcal {O}}'\ge {\mathcal {O}}$, then for any place $\nu$ of ${\mathbb K}$, the closure of ${\mathcal {O}}'$ in ${\mathfrak {g}}({\mathbb K}_{\nu }) := {\mathfrak {g}}\otimes _\mathbb {K} \mathbb {K}_\nu$ (in the local topology) contains ${\mathcal {O}}$.

Proof

Fix a place $\nu$ of ${\mathbb K}$. Since $\mathbf{G}({\mathbb K})$ is dense in $\mathbf{G}({\mathbb K}_{\nu })$, the closure of ${\mathcal {O}}'$ equals the closure of the saturation $\mathbf{G}({\mathbb K}_{\nu }){\mathcal {O}}'$. Thus it is enough to show that for any $Z\in {\mathfrak {g}}$, $\varphi \in {\mathfrak {g}}^Z_0$ and $\psi \in {\mathfrak {g}}^Z_{>0}$, $\varphi$ lies in the closure of $\mathbf{G}({\mathbb K}_{\nu })(\varphi +\psi )$. By [18, Lemma 2.18], there exist a central element $Z'\in {\mathfrak {g}}$, a positive integer k and an algebraic group homomorphism $\mu :{\mathbb K}_{\nu }^{\times }\rightarrow \mathbf{G}^{\mathrm {ad}}({\mathbb K}_{\nu })$ such that $k(Z-Z')=d\mu (1)$, where $\mathbf{G}^{\mathrm {ad}}$ denotes the adjoint group of $\mathbf{G}$. Let $\lambda \in {\mathbb K}_\nu ^{\times }$ be small, and let $g':=\mu (\lambda )$. Since the natural projection $\mathbf{G} \rightarrow G^{\mathrm {ad}}$ is onto, the image of the map between the ${\mathbb K}_{\nu }$-points $\mathbf{G}({\mathbb K}_{\nu })\rightarrow \mathbf{G}^{\mathrm {ad}}({\mathbb K}_{\nu })$ has finite index in $\mathbf{G}^{\mathrm {ad}}({\mathbb K}_{\nu })$ (by [4, Corollary 6.4]), and thus for some power m, $(g')^m$ lies in this image. Let $g\in \mathbf{G}({\mathbb K}_{\nu })$ be a preimage of $(g')^{m}$. Since $\varphi \in {\mathfrak {g}}^Z_0$ and $\psi \in {\mathfrak {g}}^Z_{>0}$, we have $g^n\varphi =\varphi$, while $g^n\psi \rightarrow 0$ when $n\rightarrow \infty$. Thus $g^n(\varphi +\psi )\rightarrow \varphi$. $\square$

Let us use this opportunity to mention that a similar argument in the 4th paragraph in the proof of [18, Proposition 4.12] is imprecise, and should be replaced by the argument above.

Definition 2.4.3

For an automorphic function $\eta$, we define $\mathrm {WO}(\eta )$ to be the set of nilpotent orbits ${\mathcal {O}}$ in ${\mathfrak {g}}^*$ under the coadjoint action of $\mathbf{G}({\mathbb K})$ such that ${\mathcal {F}}_{h,\varphi }[\eta ]\ne 0$ for some neutral Whittaker pair $(h,\varphi )$ with $\varphi \in {\mathcal {O}}$. Using the partial order of Definition 2.4.1, we define the Whittaker support ${\text {WS}}(\eta )$ to be the set of maximal elements in $\mathrm {WO}(\eta )$.

Remark 2.4.4

Applying Lemma 2.4.2 to Archimedean places, we obtain that our order is stronger than the closure (Bruhat) order on complex orbits, i.e. if ${\mathcal {O}}'\ge {\mathcal {O}}$ then $\overline{{\mathcal {O}}'_{{\mathbb C}}}\supseteq {\mathcal {O}}_{{\mathbb C}}$. It is in general strictly stronger; for example one can show that the maximal elements with respect to our order are the ${\mathbb K}$-distinguished orbits. For $\mathbf{G}= {\text {GL}}_n,$ our order coincides with the Bruhat order on complex orbits by [18, Proposition 7.5].

The definition of the Whittaker support ${\text {WS}}$ of automorphic representations given in [18, Sect. 8] uses the coarser Bruhat order on complex orbits. Thus, the ${\text {WS}}$ defined in Definition 2.4.3 includes the ${\text {WS}}$ used in [18, Sect. 8] but is frequently not equal to it.

3 Relating different Fourier coefficients

In this section we will introduce some standard tools used to relate different types of Fourier coefficients. In the subsequent subsection we first relate Fourier coefficients for different isotropic subspaces for a single Whittaker pair. Then we relate Fourier coefficients along deformations of Whittaker pairs, and lastly we explain the relationship between the conjugation of a Whittaker pair and the translation of the argument of a Fourier coefficient.

The statements in this section have partial local analogues in [18, 19]. However, the statements we give here are global and more explicit.

3.1 Relating different isotropic subspaces

We will now see how ${\mathcal {F}}_{S,\varphi ,\varphi '}$ and ${\mathcal {F}}_{S,\varphi ,\varphi '}^R$ can be expressed through each other.

Lemma 3.1.1

(cf. [19, Lemma 6.0.2] for a slightly weaker statement) Let $\eta \in C^{\infty }(\Gamma \backslash G)$, let $(S,\varphi ,\varphi ')$ be a Whittaker triple, ${\mathfrak {n}}_{S,\varphi }$ be as in (2.4), and ${\mathfrak {u}}:= \mathfrak {g}^S_{\ge 1}$.

Let ${\mathfrak {n}}_{S,\varphi } \subseteq \mathfrak {i}\subseteq \mathfrak {r}$ be isotropic subspaces of ${\mathfrak {u}}$, and let $\mathfrak {i}^\perp \supseteq \mathfrak {r}^{\perp }$ be their orthogonal complements with respect to $\omega _\varphi |_{\mathfrak {u}}$. Let also $I: = {\text {Exp}}(\mathfrak {i}(\mathbb {A}))$, $R = {\text {Exp}}(\mathfrak {r}(\mathbb {A}))$, $I^\bot :={\text {Exp}}( \mathfrak {i}^\bot (\mathbb {A}))$ and $R^\bot :={\text {Exp}}(\mathfrak {i}^\bot (\mathbb {A}))$. Then,

$$\begin{aligned} {\mathcal {F}}_{S,\varphi ,\varphi '}^R[\eta ](g)&= \mathop \int \limits _{[R/I]} {\mathcal {F}}^I_{S,\varphi ,\varphi '}[\eta ](ug) \, du \, \end{aligned}$$

(3.1)

and

$$\begin{aligned} {\mathcal {F}}^I_{S,\varphi ,\varphi '}[\eta ](g)&=\sum _{\gamma \in {\text {Exp}}(\mathfrak {i}^{\perp }/\mathfrak {r}^\bot )}{\mathcal {F}}_{S,\varphi ,\varphi '}^{R}[\eta ](\gamma g). \end{aligned}$$

(3.2)

We will mostly use this lemma in the case $\mathfrak {i}={\mathfrak {n}}_{S,\varphi }$ for which $\mathfrak {i}^{\bot }={\mathfrak {u}}$. The convergence in (3.2) is absolute convergence, uniform in compact subsets of G.

Proof

We assume that $\varphi$ is non-zero since otherwise $R=I=N_{S,\varphi }$. We have that $I \subseteq R$ with R/I abelian which means that (3.1) follows immediately from the definitions of ${\mathcal {F}}^I_{S,\varphi ,\varphi '}$ and ${\mathcal {F}}_{S,\varphi ,\varphi '}^R$. For (3.2) observe that the function $(\chi _{\varphi }^{R})^{-1}\cdot {\mathcal {F}}^I_{S,\varphi ,\varphi '}[\eta ]$ on R is left-invariant under the action of $I\cdot (R\cap \Gamma )$. In other words, we can identify it with a function on

$$\begin{aligned} (I\cdot (R\cap \Gamma )) \backslash R \cong {\text {Exp}}(\mathfrak {r}/\mathfrak {i}\bigr ) \backslash {\text {Exp}}((\mathfrak {r}/\mathfrak {i})(\mathbb {A})) =:[R/I], \end{aligned}$$

(3.3)

where the equality follows from the fact that R/I is abelian. Therefore, we have a Fourier series expansion

$$\begin{aligned} {\mathcal {F}}^I_{S,\varphi ,\varphi '}[\eta ](u)=\sum _{\psi \in [R/I]^{\wedge }}c_{\psi ,\chi _{\varphi +\varphi '}^{R}}(\eta )\psi (u)\chi _{\varphi }^{R}(u), \end{aligned}$$

(3.4)

where $[R/I]^{\wedge }$ denotes the Pontryagin dual group of [R/I] and

$$\begin{aligned} c_{\psi ,\chi _{\varphi }^{R}}(\eta )=\mathop \int \limits _{[R]}\psi (u)^{-1}\chi _{\varphi +\varphi '}^{R}(u)^{-1}\eta (u)du. \end{aligned}$$

(3.5)

The Fourier series (3.4) converges absolutely since the function $\eta$ is smooth.

In particular, denoting by ${\text {Id}}\in G$ the identity element we obtain

$$\begin{aligned} {\mathcal {F}}^I_{S,\varphi ,\varphi '}[\eta ]({\text {Id}})=\sum _{\psi \in [R/I]^{\wedge }}c_{\psi ,\chi _{\varphi +\varphi '}^{R}}(\eta ). \end{aligned}$$

(3.6)

Now observe that the map $X \mapsto \omega _{\varphi }(X,\cdot )=\varphi \circ {\text {ad}}(X)$ induces an isomorphism between $\mathfrak {i}^{\bot }/\mathfrak {r}^\bot$ and the dual space $(\mathfrak {r}/\mathfrak {i})^*$. Hence, according to equations (2.1) and (3.3), we can use the character $\chi$ to define a group isomorphism

$$\begin{aligned} \begin{array}{rcl} (I^{\bot }\cap \Gamma )/(R^\bot \cap \Gamma ) &{} \longrightarrow &{} [R/I]^{\wedge } \\ u &{} \mapsto &{} \psi _{u}, \end{array} \end{aligned}$$

(3.7)

where

$$\begin{aligned} \psi _{u}(r)=\chi (\varphi ([X,Y])), \qquad {u=\exp X}\quad \text{ and }\quad {r=\exp Y.} \end{aligned}$$

(3.8)

Hence, for all $u\in I^{\bot }\cap \Gamma$ and $r\in R$ we have

$$\begin{aligned} \psi _{u}(r)\chi _{\varphi +\varphi '}^{R}(r) = \chi (\varphi ([X,Y])+\varphi '([X,Y]))\chi (\varphi (Y)+\varphi '(Y)) = \chi ((\varphi +\varphi ')(Y+[X,Y]))\\ = \chi ((\varphi +\varphi ')(e^{{\text {ad}}(X)}(Y))) = \chi _{\varphi +\varphi '}(({\text {Ad}}(u)Y)) = \chi _{\varphi +\varphi '}^{R}(uru^{-1}). \end{aligned}$$

Here we are taking again $u=\exp X$, $r=\exp Y$ and the middle equality follows from the vanishing of $\varphi$ on ${\mathfrak {g}}^{S}_{>2}$. But now, from formula (3.5) and the fact that $\eta$ is automorphic, we have

$$\begin{aligned} c_{\psi _{u},\chi _{\varphi +\varphi '}^{R}}(\eta )= & {} \mathop \int \limits _{[R]}\psi _{u}(r)^{-1}\chi _{\varphi +\varphi '}^{R}(r)^{-1}\eta (r)dr = \mathop \int \limits _{[R]}\chi _{\varphi +\varphi '}^{R}(uru^{-1})^{-1}\eta (r)dr.\nonumber \\= & {} \mathop \int \limits _{[R]}\chi _{\varphi +\varphi '}^{R}(r)^{-1}\eta (u^{-1}ru)dr = {\mathcal {F}}_{S,\varphi ,\varphi '}^{R}[\eta ](u), \end{aligned}$$

(3.9)

for all $u\in U\cap \Gamma$. Combining this with (3.6) and (3.7) we obtain

$$\begin{aligned} {\mathcal {F}}^I_{S,\varphi ,\varphi '}[\eta ]({\text {Id}})=\sum _{u\in (I^{\bot }\cap \Gamma )/(R^\bot \cap \Gamma )}{\mathcal {F}}_{S,\varphi ,\varphi '}^{R}[\eta ](u). \end{aligned}$$

(3.10)

Applying this to $\eta$ and its right shifts we obtain (3.2). The convergence is uniform when g varies in compact subsets of G since $\eta$ is infinitely smooth. $\square$

Corollary 3.1.2

Let $\eta \in C^{\infty }(\Gamma \backslash G)$, let $(S,\varphi ,\varphi ')$ be a Whittaker triple, and ${\mathfrak {n}}_{S,\varphi }$, and ${\mathfrak {u}}$ be as above. Let $\mathfrak {r},\mathfrak {r}'\subseteq {\mathfrak {u}}$ be two isotropic subspaces that include $\mathfrak {n}_{S,\varphi }$. Assume $\dim \mathfrak {r} =\dim \mathfrak {r}'$ and $\mathfrak {r}\cap (\mathfrak {r}')^{\bot }\subseteq \mathfrak {r}'$. Then

$$\begin{aligned} {\mathcal {F}}_{S,\varphi ,\varphi '}^R[\eta ](g) = \mathop \int \limits _{R/(R\cap R')} {\mathcal {F}}^{R'}_{S,\varphi ,\varphi '}[\eta ](ug) \, du\,. \end{aligned}$$

(3.11)

Note that this is a non-compact, adelic, integral. It converges absolutely.

Proof

Let $\mathfrak {i}:=\mathfrak {r}\cap \mathfrak {r}'$. We claim that the natural map $p:\mathfrak {r}/\mathfrak {i} \rightarrow \mathfrak {i}^{\bot }/(\mathfrak {r}')^{\bot }$ is an isomorphism. Indeed, from the assumption $\mathfrak {r}\cap (\mathfrak {r}')^{\bot }\subseteq \mathfrak {r}'$ we have $\mathfrak {i} = \mathfrak {r}\cap (\mathfrak {r}')^{\bot }$ and thus p is an embedding. Further, from the assumption $\dim \mathfrak {r}=\dim \mathfrak {r}'$ we obtain that the source and the target spaces of p have the same dimension. Indeed,

$$\begin{aligned} \dim (\mathfrak {i}^{\bot }/(\mathfrak {r}')^{\bot })= \dim {\mathfrak {u}}-\dim \mathfrak {i} - (\dim {\mathfrak {u}}-\dim \mathfrak {r}')=\dim \mathfrak {r}-\dim \mathfrak {i}= \dim (\mathfrak {r}/\mathfrak {i}) \end{aligned}$$

(3.12)

Now, p defines a natural isomorphism . Let $I:={\text {Exp}}(\mathfrak {i}(\mathbb {A}))=R\cap R'$. From Lemma 3.1.1 we obtain

$$\begin{aligned} {\mathcal {F}}_{S,\varphi ,\varphi '}^R[\eta ](g) = \mathop \int \limits _{[R/I]} \sum _{\gamma \in {\text {Exp}}(\mathfrak {r}/\mathfrak {i})} {\mathcal {F}}^{R'}_{S,\varphi ,\varphi '}[\eta ](\gamma ug) \, du = \mathop \int \limits _{R/I} {\mathcal {F}}^{R'}_{S,\varphi ,\varphi '}[\eta ](ug) \, du\,. \end{aligned}$$

(3.13)

The convergence of the integral follows from the convergence of the Fourier series in (3.2). $\square$

This corollary can be seen as a version of the root exchange lemma in [15].

3.2 Relating different Whittaker pairs

Let $(H,\varphi )$ be a Whittaker pair.

Lemma 3.2.1

Let Z be as in Lemma 2.3.7. Then $(H-Z,\varphi )$ dominates $(H,\varphi )$.

Proof

Denote $h:=H-Z$. We have to show that (2.11) holds, i.e.

$$\begin{aligned} {\mathfrak {g}}_{\varphi }\cap {\mathfrak {g}}^h_{\ge 1}\subseteq {\mathfrak {g}}^Z_{\ge 0}\,. \end{aligned}$$

(3.14)

Since ${\mathfrak {g}}_{\varphi }$ is spanned by lowest weight vectors, we have ${\mathfrak {g}}_{\varphi }\subseteq {\mathfrak {g}}^h_{\le 0}$ and thus ${\mathfrak {g}}_{\varphi }\cap {\mathfrak {g}}^h_{\ge 1}=\{0\}$. $\square$

Corollary 3.2.2

Any Whittaker pair is dominated by a neutral Whittaker pair with the same character $\varphi$.

Another example of domination is provided by the following proposition, that immediately follows from [19, Proposition 3.3.3].

Proposition 3.2.3

If $\varphi$ is a PL nilpotent then there exists $Z\in {\mathfrak {g}}$ such that $(H+Z,\varphi )$ is a standard Whittaker pair and $(H,\varphi )$ dominates $(H+Z,\varphi )$.

From now till the end of the section let $(H,\varphi )$ be a Whittaker pair and $Z\in {\mathfrak {g}}_0^H$ be a rational semi-simple element such that $(H,\varphi )$ dominates $(H+Z,\varphi )$. We will now consider the deformation of the former Whittaker pair to the latter. For any rational number $t \ge 0$ define

$$\begin{aligned} H_t:=H+tZ,\quad {\mathfrak {u}}_t:={\mathfrak {g}}^{H_t}_{\ge 1},\quad {\mathfrak {v}}_t:={\mathfrak {g}}^{H_t}_{> 1},\text { and }{\mathfrak {w}}_t:={\mathfrak {g}}^{H_t}_{1}. \end{aligned}$$

(3.15)

Definition 3.2.4

We call $t \ge 0$ regular if ${\mathfrak {u}}_t = {\mathfrak {u}}_{t+\varepsilon }$ for any small enough $\varepsilon \in {\mathbb Q}$, or in other words ${\mathfrak {w}}_t\subset {\mathfrak {g}}^Z_0$. If t is not regular we call it critical. Equivalently, t is critical if ${\mathfrak {g}}^{H_t}_{1}\nsubseteq {\mathfrak {g}}^{Z}_{0}$ which we may interpret as something new has entered the 1-eigenspace of H. For convenience, we will say that $t=0$ is critical.

We also say that $t \ge 0$ is quasi-critical if either ${\mathfrak {g}}^{H_t}_{1}\nsubseteq {\mathfrak {g}}^{Z}_{0}$ or ${\mathfrak {g}}^{H_t}_{2}\nsubseteq {\mathfrak {g}}^{Z}_{0}$. We may interpret this as something new has entered either the 1-eigenspace or the 2-eigenspace. The latter is related to new characters being available in the Whittaker pairs. Note that there are only finitely many (quasi-)critical numbers.

Recall the anti-symmetric form $\omega _\varphi$ on ${\mathfrak {g}}$ given by $\omega _\varphi (X,Y)=\varphi ([X,Y])$ and the definition $\mathfrak {n}_{H_t,\varphi } := {\text {Ker}}(\omega _\varphi |_{{\mathfrak {u}}_t})$.

Lemma 3.2.5

([19, Lemma 3.2.6])

• (i) The form $\omega _\varphi$ is ${\text {ad}}(Z)$-invariant.

• (ii) ${\text {Ker}}\omega _\varphi = {\mathfrak {g}}_\varphi$.

• (iii) ${\text {Ker}}(\omega _\varphi |_{{\mathfrak {w}}_t})={\text {Ker}}(\omega _\varphi )\cap {\mathfrak {w}}_t$.

• (iv) ${\text {Ker}}(\omega _\varphi |_{{\mathfrak {u}}_t})={\mathfrak {v}}_t\oplus {\text {Ker}}(\omega _\varphi |_{{\mathfrak {w}}_t})$.

• (v) ${\mathfrak {w}}_s\cap {\mathfrak {g}}_\varphi \subseteq {\mathfrak {u}}_{t}$ for any $s<t$.

We will often suppress the deformation $H_t$ and character $\varphi$ and simply write ${\mathfrak {n}}_t = {\mathfrak {n}}_{H_t,\varphi }$. Similarly, define

$$\begin{aligned} \mathfrak {l}_t = \mathfrak {l}_{H_t,\varphi } := ({\mathfrak {u}}_t\cap {\mathfrak {g}}^{Z}_{< 0})+{\mathfrak {n}}_{H_t,\varphi } \quad \text { and } \quad \mathfrak {r}_t = \mathfrak {r}_{H_t,\varphi }:=({\mathfrak {u}}_t\cap {\mathfrak {g}}^{Z}_{> 0})+{\mathfrak {n}}_{H_t,\varphi }. \end{aligned}$$

(3.16)

We note that $\mathfrak {l}_t$ and $\mathfrak {r}_t$ are nilpotent subalgebras. The choice of notation for them comes from ‘left’ and ‘right’.

Lemma 3.2.6

For any $t\ge 0$ we have

• (i) $\mathfrak {l}_t$ and $\mathfrak {r}_t$ are ideals in ${\mathfrak {u}}_t$ and $[\mathfrak {l}_t,\mathfrak {r}_t]\subseteq \mathfrak {l}_t\cap \mathfrak {r}_t={\mathfrak {n}}_t$.

• (ii) $\mathfrak {l}_t$ and $\mathfrak {r}_t$ are isotropic subspaces of ${\mathfrak {u}}_t$ of the same dimension, and the natural projections $\mathfrak {l}_t/{\mathfrak {n}}_t\rightarrow {\mathfrak {u}}_t/\mathfrak {r}_t^\bot$ and $\mathfrak {r}_t/{\mathfrak {n}}_{t} \rightarrow {\mathfrak {u}}_t/\mathfrak {l}_t^\bot$ are isomorphisms. Furthermore, $\mathfrak {l}_t = \mathfrak {g}^{H_t}_1 \cap \mathfrak {g}^Z_{<0} \oplus \mathfrak {n}_t$.

• (iii) Suppose that $0\le s< t$, and all the elements of the interval (s, t) are regular. Then

  $$\begin{aligned}&{\mathfrak {v}}_{t}\oplus ({\mathfrak {w}}_{t}\cap {\mathfrak {g}}^Z_{<0})={\mathfrak {v}}_{s}\oplus ({\mathfrak {w}}_{s}\cap {\mathfrak {g}}^Z_{>0}) \end{aligned}$$

  (3.17)
  $$\begin{aligned}&\mathfrak {l}_{t}=\mathfrak {r}_{s}+ ({\mathfrak {w}}_{t}\cap {\mathfrak {g}}_{\varphi }) \text { and }\mathfrak {r}_{s}\cap ({\mathfrak {w}}_{t}\cap {\mathfrak {g}}_{\varphi })={\mathfrak {w}}_{0}\cap {\mathfrak {g}}^Z_0\cap {\mathfrak {g}}_{\varphi }. \end{aligned}$$

  (3.18)

  Moreover, $\mathfrak {r}_{s}$ is an ideal in $\mathfrak {l}_{t}$ and the quotient is commutative.

Proof

It is easy to see that ${\mathfrak {v}}_t$ is an ideal in ${\mathfrak {u}}_t$ with commutative quotient, and that ${\mathfrak {v}}_t\subseteq \mathfrak {l}_t\cap \mathfrak {r}_t={\mathfrak {n}}_t$. This proves (i). For the first part of (ii), note that $(\mathfrak {l}_t+\mathfrak {r}_t)/{\mathfrak {n}}_t$ is a symplectic space in which the projections of $\mathfrak {l}_t$ and $\mathfrak {r}_t$ are complementary Lagrangians.

For the second part, we have by Lemma 3.2.5 that $\mathfrak {g}^{H_t}_1 \cap \mathfrak {g}_\varphi \subseteq \mathfrak {g}^Z_{\ge 0}$ and thus,

$$\begin{aligned} \mathfrak {l}_t={\mathfrak {v}}_{t}\oplus ({\mathfrak {w}}_{t}\cap {\mathfrak {g}}^Z_{<0})\oplus ({\mathfrak {w}}_{t}\cap {\mathfrak {g}}_{\varphi }) \, . \end{aligned}$$

(3.19)

For part (iii) note that

$$\begin{aligned} {\mathfrak {v}}_{s}&=({\mathfrak {v}}_{s}\cap {\mathfrak {g}}^Z_{\ge 0}) \oplus ({\mathfrak {v}}_{s}\cap {\mathfrak {g}}^Z_{<0}) \,, \end{aligned}$$

(3.20)

$$\begin{aligned} {\mathfrak {v}}_{t}&= ({\mathfrak {v}}_{t}\cap {\mathfrak {g}}^Z_{<0}) \oplus ({\mathfrak {v}}_{t}\cap {\mathfrak {g}}^Z_{\ge 0}) \,, \end{aligned}$$

(3.21)

$$\begin{aligned} {\mathfrak {v}}_{t}\cap {\mathfrak {g}}^Z_{\ge 0}&=({\mathfrak {w}}_{s}\cap {\mathfrak {g}}^Z_{>0})\oplus ({\mathfrak {v}}_{s}\cap {\mathfrak {g}}^Z_{\ge 0})\,, \end{aligned}$$

(3.22)

$$\begin{aligned} {\mathfrak {v}}_{s}\cap {\mathfrak {g}}^Z_{<0}&=({\mathfrak {w}}_{t}\cap {\mathfrak {g}}^Z_{<0})\oplus ({\mathfrak {v}}_{t}\cap {\mathfrak {g}}^Z_{<0})\,. \end{aligned}$$

(3.23)

This implies (3.17). By Lemma 3.2.5 we have

$$\begin{aligned} {\mathfrak {n}}_s= {\mathfrak {v}}_s\oplus ({\mathfrak {g}}_{\varphi }\cap {\mathfrak {w}}_s)\subseteq {\mathfrak {v}}_s\oplus ({\mathfrak {w}}_s\cap {\mathfrak {g}}^{Z}_{\ge 0}), \end{aligned}$$

(3.24)

and thus

$$\begin{aligned} \mathfrak {r}_s={\mathfrak {v}}_{s}\oplus ({\mathfrak {w}}_{s}\cap {\mathfrak {g}}^Z_{>0})\oplus ({\mathfrak {w}}_0\cap {\mathfrak {g}}^Z_0\cap {\mathfrak {g}}_{\varphi }) \text { and }\mathfrak {r}_{s}\cap ({\mathfrak {w}}_{t}\cap {\mathfrak {g}}_{\varphi })={\mathfrak {w}}_{0}\cap {\mathfrak {g}}^Z_0\cap {\mathfrak {g}}_{\varphi }. \end{aligned}$$

(3.25)

Hence, (3.17) and (3.19) imply (3.18), and the rest is straightforward. $\square$

Using Lemma 2.3.7, choose an -triple $(e_\varphi ,h,f_\varphi )$ in ${\mathfrak {g}}^Z_0$ such that h commutes with H and with Z, and $\varphi$ is given by the Killing form pairing with $f = f_\varphi$. Define $L_{t} = L_{H_t,\varphi } :={\text {Exp}}(\mathfrak {l}_{H_t,\varphi }(\mathbb {A})),\, R_{t}=R_{H_t,\varphi }:={\text {Exp}}(\mathfrak {r}_{H_t,\varphi }(\mathbb {A}))$ and $V_t := {\text {Exp}}({\mathfrak {v}}_t(\mathbb {A}))$. From Lemmas 3.2.6 and 3.1.1 we get the following.

Lemma 3.2.7

Let $(H,\varphi )$ and $(S,\varphi )$ be Whittaker pairs such that $(H,\varphi )$ dominates $(S,\varphi )$. Let $Z = S - H$, $H_t = H + tZ$ and define $R_t$, $L_t$ and $V_t$ as above.

• (i) Let $t \ge 0$ and $\varphi '\in ({\mathfrak {g}}^*)^{H_t}_{>-2}$. Let also $I = L_t$ and $\mathfrak {j} = \mathfrak {r}_t$, or $I = R_t$ and $\mathfrak {j} = \mathfrak {l}_t$ respectively. Let $A:={\text {Exp}}(({\mathfrak {u}}_t/{\mathfrak {v}}_t)^Z_{<0})(\mathbb {A})$. Then,

  

  (3.26)
  

  (3.27)
  

  (3.28)

• (ii) Now, let $0 \le s < t$ such that there are no critical values in the interval (s, t) and let $\varphi '\in ({\mathfrak {g}}^*)^{H_t}_{>-2}\cap ({\mathfrak {g}}^*)^{H_s}_{>-2}$. Then,

  $$\begin{aligned} {\mathcal {F}}_{H_t,\varphi ,\varphi '}^{L_t}[\eta ](g)&= \mathop \int \limits _{[L_t/R_s]} {\mathcal {F}}_{H_s,\varphi ,\varphi '}^{R_s}[\eta ](ug) \, du \end{aligned}$$

  (3.29)
  $$\begin{aligned} {\mathcal {F}}_{H_s,\varphi ,\varphi '}^{R_s}[\eta ](g)&= \sum _{\psi ' \in \Psi '} {\mathcal {F}}_{H_t,\varphi ,\varphi '+\psi '}^{L_t}[\eta ](g), \end{aligned}$$

  (3.30)

  where $\Psi ' := ({\mathfrak {g}}^*)^{H_t}_{-1}\cap ({\mathfrak {g}}^*)^{e_\varphi }\cap ({\mathfrak {g}}^*)^Z_{<0}$.

• (iii) Let $0 \le s < t$ and $\varphi '$ as in (ii), and let $\psi \in ({\mathfrak {g}}^*)^{H_s}_{>-2}\cap ({\mathfrak {g}}^*)^{H_t}_{ -2}$. Then

  $$\begin{aligned} {\mathcal {F}}_{H_s,\varphi ,\psi +\varphi '}^{R_s}[\eta ](g) = \mathop \int \limits _{[R_s/V_t]} \sum _{\psi ' \in \Psi ''} {\mathcal {F}}_{H_t,\varphi +\psi ,\varphi '+\psi '}[\eta ](ug) \, du, \end{aligned}$$

  (3.31)

  where $\Psi '' := (\mathfrak {g}^*)^{H_t}_{-1} \cap (\mathfrak {g}^*)^{e_{\varphi +\psi }}$.

Proof

For part (i) we note that $\mathfrak {l}_t$ and $\mathfrak {r}_t$ are isotropic subspaces of ${\mathfrak {u}}_t$ containing ${\mathfrak {n}}_t$ by Lemma 3.2.6(ii). Thus, the statement follows from Lemma 3.1.1 and Corollary 3.1.2. The domain of summation for $\gamma$ in (3.26) follows from Lemma 3.2.6(ii). Note that for Corollary 3.1.2 we have that $\mathfrak {l}_t \cap \mathfrak {r}_t^\perp = \mathfrak {l}_t \cap \mathfrak {r}_t={\mathfrak {n}}_t$ and $\mathfrak {l}_t/{\mathfrak {n}}_t\cong ({\mathfrak {u}}^{t}/{\mathfrak {v}}^t)^Z_{<0}$ by Lemma 3.2.6.

For part (ii) we note from Lemma 3.2.6(iii) that ${\mathfrak {r}_s} \subseteq \mathfrak {l}_t$ and that $\mathfrak {l}_t/\mathfrak {r}_s$ is commutative with a set of representatives ${\mathfrak {g}}^{H_t}_1 \cap {\mathfrak {g}}_\varphi \cap {\mathfrak {g}}^Z_{>0}$. Indeed, $\mathfrak {l}_t/\mathfrak {r}_s$ projects naturally and isomorphically to $({\mathfrak {g}}^{H_t}_1 \cap {\mathfrak {g}}_{\varphi })/({\mathfrak {g}}^{H_t}_1 \cap {\mathfrak {g}}_\varphi \cap {\mathfrak {g}}^Z_0)$, and ${\mathfrak {g}}^{H_t}_1 \cap {\mathfrak {g}}_\varphi \subseteq {\mathfrak {g}}^Z_{\ge 0}$ by Lemma 3.2.5. Thus, (3.29) follows directly by integration, while (3.30) follows from a Fourier expansion in $L_t/R_s$ with characters given by $(\mathfrak {l}_t/{\mathfrak {r}_s})^*$ for which we have the set of representatives $\Psi ' := ({\mathfrak {g}}^*)^{H_t}_{-1}\cap ({\mathfrak {g}}^*)^{e_\varphi }\cap ({\mathfrak {g}}^*)^Z_{<0}$.

For part (iii), note first that $N'_t:=N_{H_t, \varphi +\psi }$ and $R_s = R_{H_s, \varphi }$ depend on different characters which means that we cannot directly relate them using Lemma 3.2.6(iii) as we did in part (ii). Instead, we will relate them both to $V_t = {\text {Exp}}({\mathfrak {v}}_t(\mathbb {A}))$ where ${\mathfrak {v}}_t := {\mathfrak {g}}^{H_t}_{>1}$ which does not depend on any character.

We have that ${\mathfrak {v}}_t$ is an ideal in $\mathfrak {l}_t := \mathfrak {l}_{H_t, \varphi }$ with commutative quotient. Since $\mathfrak {l}_t$ and ${\mathfrak {r}_s} := {\mathfrak {r}}_{H_s, \varphi }$ are part of the same deformation with the same character we can then use (3.18) in Lemma 3.2.6(iii) to see that ${\mathfrak {v}}_t$ is an ideal in ${\mathfrak {r}_s}$ with commutative quotient. Thus,

$$\begin{aligned} {\mathcal {F}}^{R_s}_{H_s,\varphi ,\psi +\varphi '}[\eta ](g)= & {} \mathop \int \limits _{[R_s/V_t]} {\mathcal {F}}_{V_t}[\eta ](ug)\, du \quad \text { where } \nonumber \\ {\mathcal {F}}_{V_t}[\eta ](g):= & {} \mathop \int \limits _{[V_t]} \eta (vg)\chi _{\varphi +\psi +\varphi '}(v)^{-1} dv. \end{aligned}$$

(3.32)

Furthermore, ${\mathfrak {v}}_t$ is an ideal in ${\mathfrak {n}}_t' := \mathfrak {l}_{H_t,\varphi +\psi }$ and ${\mathfrak {n}}_t'/{\mathfrak {v}}_t$ is commutative with a set of representatives $\mathfrak {g}^{H_t}_1 \cap \mathfrak {g}_{\varphi +\psi }$ by (2.5). The statement now follows from a Fourier expansion of ${\mathcal {F}}_{V_t}[\eta ](g)$ in $N_t'/V_t$ with characters given by $({\mathfrak {n}}_t'/{\mathfrak {v}}_t)^*$ for which we have the representatives $\Psi '' := (\mathfrak {g}^*)^{H_t}_{-1} \cap (\mathfrak {g}^*)^{e_{\varphi +\psi }}$. $\square$

Lemma 3.2.8

Let $(S, \varphi , \psi )$ be a Whittaker triple, $\eta$ an automorphic function and $\gamma \in \Gamma$. Then,

$$\begin{aligned} {\mathcal {F}}_{S, \varphi , \psi }[\eta ](g) = {\mathcal {F}}_{{\text {Ad}}(\gamma ) S, {\text {Ad}}^*(\gamma )\varphi , {\text {Ad}}^*(\gamma )\psi }[\eta ](\gamma g) \, . \end{aligned}$$

(3.33)

Proof

The proof is straightforward. We have that $\chi _{\varphi +\psi }(u) = \chi _{{\text {Ad}}^*(\gamma )(\varphi + \psi )}(\gamma u \gamma ^{-1})$. Indeed, the right-hand side equals

$$\begin{aligned} \chi \Bigl ( \bigl ({\text {Ad}}^*(\gamma )(\varphi + \psi ) \bigr ) ({\text {Ad}}(\gamma ) \log u ) \Bigr ) = \chi \bigl ((\varphi + \psi )({\text {Ad}}(\gamma ^{-1}) {\text {Ad}}(\gamma ) \log u )\bigr ) = \chi _{\varphi + \psi }(u)\,. \end{aligned}$$

(3.34)

We also have that $\mathfrak {g}^{{\text {Ad}}(\gamma )S}_\lambda = {\text {Ad}}(\gamma ) \mathfrak {g}^S_\lambda$ since, for $x \in \mathfrak {g}$, $[{\text {Ad}}(\gamma ) S, {\text {Ad}}(\gamma ) x] = {\text {Ad}}(\gamma )[S,x]$. Similarly, $\mathfrak {g}_{{\text {Ad}}^*(\gamma )\varphi } = {\text {Ad}}(\gamma ) \mathfrak {g}_\varphi$ and thus, $\mathfrak {n}_{{\text {Ad}}(\gamma ) S,{\text {Ad}}^*(\gamma ) \varphi } = {\text {Ad}}(\gamma ) \mathfrak {n}_{S,\varphi }$.

Hence, using the automorphic invariance of $\eta$, the right-hand side of (3.33) equals

$$\begin{aligned}&\mathop \int \limits _{[N_{{\text {Ad}}(\gamma )S,{\text {Ad}}^*(\gamma )}]} \eta (\gamma ^{-1} u \gamma g) \chi _{{\text {Ad}}^*(\gamma )(\varphi + \psi )}(u)^{-1} \, du\nonumber \\&\quad = \mathop \int \limits _{[\gamma ^{-1} (N_{{\text {Ad}}(\gamma )S,{\text {Ad}}^*(\gamma )} )\gamma ]} \eta (u' g) \chi _{{\text {Ad}}^*(\gamma )(\varphi + \psi )}(\gamma u' \gamma ^{-1})^{-1} \, du' \,. \end{aligned}$$

(3.35)

By the arguments above, this equals ${\mathcal {F}}_{S, \varphi , \psi }[\eta ](g)$. $\square$

4 The reduction algorithm

In this section we will present Algorithm A which details the steps for expressing a Fourier coefficient $\mathcal {F}_{H,\varphi }$ for a Whittaker pair $(H,\varphi )$ in terms of Fourier coefficients $\mathcal {F}_{S,\psi }$ with pairs $(S,\psi ) \succcurlyeq (H,\varphi )$ as described in Sect. 1.1 and detailed below in Proposition 4.1.3.

The proof of this proposition, which also shows that Algorithm A terminates in a finite number of steps and produces the stated result, is given in Sect. 4.2.

After stating and proving some additional results in Sect. 4.3 that follow from the algorithm (and are used in [20]) we show in Corollary 4.4.7 in Sect. 4.4 that $\mathcal {F}_{H, \varphi }$ can, by iteration, be expressed in terms of Levi-distinguished Fourier coefficients associated to orbits which are equal to or bigger than $\Gamma \varphi$. In the same subsection we also discuss some further results in Remark 4.4.8.

4.1 Statement of algorithm

Before stating Algorithm A, we first start with an algorithm that expresses any quasi-Fourier coefficient ${\mathcal {F}}_{H,\varphi ,\varphi '}$ in terms of Fourier coefficients ${\mathcal {F}}_{S,\psi }$ such that either $(H,\varphi )$ dominates $(S,\psi )$ or $\Gamma \psi >\Gamma \varphi$.

Algorithm 4.1.1

Given a Whittaker triple $(H,\varphi ,\varphi ')$, choose h that commutes with H such that $(h, \varphi )$ is a neutral pair. This is possible by Lemma 2.3.7. Let $Z = H - h$, and let $H_t:=H+tZ$ for any $t\in {\mathbb Q}$. Let $m_Z$ denote the minimal positive eigenvalue of Z and $M_h$ denote the maximal positive eigenvalue of h. Let $T:=(M_h+2)/m_Z$. If Z is central we set $m_Z$ to 1 for definiteness. Then for every $i\ge 0$ we have

$$\begin{aligned} {\mathfrak {g}}^{H_T}_{\ge i}=({\mathfrak {g}}^h_{\ge i})^Z_0\oplus ({\mathfrak {g}}^{H_T}_{\ge i})^Z_{>0} , \quad {\mathfrak {g}}^Z_{>0}\subset {\mathfrak {g}}^{H_T}_{> 2} \text { and } {\mathfrak {g}}^Z_{<0}\subset {\mathfrak {g}}^{H_T}_{< -2}. \end{aligned}$$

(4.1)

Since ${\mathfrak {g}}_{\varphi }\subset {\mathfrak {g}}^h_{\le 0}$, and since ${\text {ad}}(h)$ has integer eigenvalues, we obtain

$$\begin{aligned} {\mathfrak {n}}_{H_T,\varphi }={\mathfrak {v}}_T={\mathfrak {g}}^{H_T}_{>1}={\mathfrak {g}}^{H_T}_{\ge 2}. \end{aligned}$$

(4.2)

We get two cases depending on whether there are any quasi-critical t in the interval (0, T). Recall that the number of quasi-critical t is in any case finite.

• (i) If there are no quasi-critical $t\in (0,T)$ then, using Lemma 3.2.7(i) and (ii), we may express ${\mathcal {F}}_{H,\varphi ,\varphi '}$ linearly in terms of the set of all ${\mathcal {F}}_{H_T,\varphi ,\varphi '+\psi '}$ with $\psi ' \in \Psi ' = (\mathfrak {g}^*)^{H+TZ}_{-1} \cap (\mathfrak {g}^*)^{e_\varphi } \cap (\mathfrak {g}^*)^{Z}_{<0}$. By (4.1) we have $\Psi '=\{0\}$ and $L_T=N_{H_T,\varphi }$. By (4.2), $\varphi '$ vanishes on ${\mathfrak {n}}_{H_T,\varphi }$, thus ${\mathcal {F}}^{L_T}_{H_T,\varphi ,\varphi '}={\mathcal {F}}_{H_T,\varphi ,\varphi '}={\mathcal {F}}_{H_T,\varphi }$. Altogether we have

  

  (4.3)

• (ii) Now assume that there are quasi-critical numbers in (0, T) and let s be the smallest one. Since s is the first quasi-critical value we have that $(\mathfrak {g}^*)^{H}_{>-2} \subseteq (\mathfrak {g}^*)^{H_s}_{\ge -2}$ because this is the first point where something new may enter the $-2$-eigenspace. Decompose $\varphi ' = \psi + \varphi ''$ where $\psi \in (\mathfrak {g}^*)^{H_s}_{-2}$ and $\varphi '' \in (\mathfrak {g}^*)^{H_s}_{>-2}$. Using Lemma 3.2.7(i) and (iii), we may express ${\mathcal {F}}_{H,\varphi ,\varphi '}$ in terms of the Fourier coefficients ${\mathcal {F}}_{H_{s},\varphi +\psi ,\varphi '' + \psi ''}$ with $\psi ''\in \Psi '' = (\mathfrak {g}^*)^{H_s}_{-1} \cap (\mathfrak {g}^*)^{e_{\varphi +\psi }}$ in the following way:

  

  (4.4)

  Now, we repeat the procedure for each triple ${\mathcal {F}}_{H_{s},\varphi +\psi ,\varphi ''+\psi ''}$ and so on until we reach an expression that includes only Fourier coefficients.

$\square$

Lemma 4.1.2

Algorithm 4.1.1 terminates in a finite number of steps, and expresses ${\mathcal {F}}_{H,\varphi ,\varphi '}$ in terms of Fourier coefficients ${\mathcal {F}}_{S_i,\Phi _i}$ such that for each i, either $\Phi _i=\varphi$ and $(H,\varphi )$ dominates $(S_i,\Phi _i)$, or $\Gamma \Phi _i >\Gamma \varphi$.

We postpone the proof of the above lemma to the next subsection. Now let $(H,\varphi )$ and $(S,\varphi )$ be Whittaker pairs such that $(H,\varphi )$ dominates $(S,\varphi )$. The following algorithm expresses ${\mathcal {F}}_{H,\varphi }$ in terms of ${\mathcal {F}}_{S,\varphi }$ and Fourier coefficients ${\mathcal {F}}_{S',\psi }$ such that $\Gamma \psi >\Gamma \varphi$.

Algorithm A

Let $Z:=S-H$ and $H_t:=H+tZ$. Let first $s = 0$ and t be the first quasi-critical point in the interval (s, 1). If there are no critical points in this interval we let $t = 1$ which will be the end point of this algorithm. Using Lemma 3.2.7(i) and (ii), we have that ${\mathcal {F}}_{H, \varphi }[\eta ](g)$ equals

(4.5)

where ${\Psi }' = (\mathfrak {g}^*)^{H_t}_{-1} \cap (\mathfrak {g}^*)^{e_\varphi } \cap (\mathfrak {g}^*)^{Z}_{<0}$. The last step follows from Lemma 3.2.6(ii). For all non-zero $\psi '\in \Psi '$ we run Algorithm 4.1.1 to express the quasi-Fourier coefficient ${\mathcal {F}}_{H_t,\varphi ,\psi '}$ in terms of Fourier coefficients, that, as we will show, correspond to higher orbits. For the $\psi '=0$ term we iterate the same step as above using (4.5) and Algorithm 4.1.1 but now with $s = t$ and t being the next quasi-critical point, until we reach $t = 1$. $\square$

Proposition 4.1.3

Algorithm A terminates in a finite number of steps and produces an expression of the form

$$\begin{aligned} {\mathcal {F}}_{H,\varphi }[\eta ](g) = \mathcal {M}_H^S(\mathcal {F}_{S,\varphi }[\eta ]) + \mathcal {A}, \end{aligned}$$

(4.6)

where

$$\begin{aligned} \mathcal {M}_H^S(\mathcal {F}_{S,\varphi }[\eta ]) = \sum _{w\in \Omega }\,\,\, \mathop \int \limits _{V} \mathop \int \limits _{ {[ U]}} {\mathcal {F}}_{S,\varphi }[\eta ](wvug) \, dudv \, \end{aligned}$$

(4.7)

which is called the “main term” in the introduction,

$$\begin{aligned} \begin{aligned} U={\text {Exp}}({\mathfrak {u}}(\mathbb {A})),\quad V={\text {Exp}}({\mathfrak {v}}(\mathbb {A})),\quad \Omega ={\text {Exp}}({\mathfrak {w}}({\mathbb K})), \\ {\mathfrak {u}}:= ({\mathfrak {g}}^{H}_{>1})^S_1\cong ({\mathfrak {g}}^{H}_{>1})^S_{\ge 1}/({\mathfrak {g}}^{H}_{>1})^S_{>1}, \quad {\mathfrak {v}}:= ({\mathfrak {g}}^{H}_{>1})^S_{<1}\cong {\mathfrak {g}}^H_{> 1}/({\mathfrak {g}}^{H}_{>1})^S_{\ge 1}, \\ {\mathfrak {w}}:=({\mathfrak {g}}^{H}_{1})^S_{<1}\cong {\mathfrak {g}}^H_{\ge 1}/\left( {\mathfrak {g}}^H_{> 1}+ ({\mathfrak {g}}^{H}_1)^S_{\ge 1}\right) , \end{aligned} \end{aligned}$$

(4.8)

and $\mathcal {A}$ is a countable absolutely convergent sum of integral transforms of Fourier coefficients corresponding to orbits bigger than that of $\varphi$.

We prove this proposition in the next subsection.

4.2 Proofs of Theorem B and the validity of Algorithm A

We prove the following lemma in Appendix B.

Lemma 4.2.1

Let $(H,\varphi )$ be a Whittaker pair, and let Z be a rational semi-simple element that commutes with H and with $\varphi$. Let $\varphi '\in ({\mathfrak {g}}^*)^Z_{>0}\cap ({\mathfrak {g}}^*)^{S}_{2}$. Then either $\dim \mathbf{G}({\mathbb C})(\varphi +\varphi ')>\dim \mathbf{G}({\mathbb C})(\varphi )$, or $\varphi$ is conjugate to $\varphi +\varphi '$ by the stabilizer of H in $\Gamma$.

Let us now present the proofs that were postponed from the last subsection.

Proof of Lemma 4.1.2

In case (i) of the algorithm it terminates in one step, and (4.3) expresses ${\mathcal {F}}_{H,\varphi ,\varphi '}$ in terms of ${\mathcal {F}}_{H_T,\varphi }$, and $(H,\varphi )$ dominates $(H_T,\varphi )$. Thus assume that we are in case (ii), and let $s\in (0,T]$ be the smallest critical number. Decompose $\varphi ' = \psi + \varphi ''$ where $\psi \in (\mathfrak {g}^*)^{H_s}_{-2}$ and $\varphi '' \in (\mathfrak {g}^*)^{H_s}_{>-2}$. Note that $\psi \in ({\mathfrak {g}}^*)^{Z}_{<0}$ and thus the orbit $\Gamma (\varphi +\psi )$ of $\varphi +\psi$ is bigger than or equal to the orbit $\Gamma \varphi$ of $\varphi$.

If $\Gamma (\varphi +\psi )=\Gamma \varphi$ then, by Lemma 4.2.1, $\varphi +\psi$ is conjugate to $\varphi$ under the stabilizer of $H_s$ in $\Gamma$. Then, by shifting the argument, we express each ${\mathcal {F}}_{H_s,\varphi +\psi ,\varphi ''+\psi ''}$ in terms of ${\mathcal {F}}_{H_s,\varphi ,\varphi '''}$ for some $\varphi '''$. Then we run the algorithm on ${\mathcal {F}}_{H_s,\varphi ,\varphi '''}$ and it terminates by induction on the finite number of critical values in the interval (0, T).

If $\Gamma (\varphi +\psi )>\Gamma \varphi$ then, by Lemma 4.2.1, $\dim \mathbf{G}({\mathbb C})(\varphi +\psi )>\dim \mathbf{G}({\mathbb C})(\varphi )$, and thus the algorithm terminates by induction on $\dim \mathbf{G}({\mathbb C})(\varphi )$. $\square$

Lemma 4.2.2

Let $(H,\varphi )$ be a Whittaker pair, and let $(e_{\varphi },h,f_{\varphi })$ be an ${\mathfrak {sl}}_2$-triple such that $\varphi$ is given by Killing form pairing with $f_{\varphi },$ and h commutes with H. Let $0 \ne \varphi ' \in ({\mathfrak {g}}^*)^{H}_{>-2}\cap ({\mathfrak {g}}^*)^{H}_{\le -1} \cap ({\mathfrak {g}}^*)^{e_\varphi }$. Then Algorithm 4.1.1 expresses the quasi-Fourier coefficient ${\mathcal {F}}_{H,\varphi ,\varphi '}$ in terms of Fourier coefficients $(S_i,\Phi _i)$ with $\Gamma \Phi _i>\Gamma \varphi$.

Proof

As in the algorithm, we let $Z:=H-h$, and for any $t\ge 0$ denote $H_t:=H+tZ$. Then $\varphi '$ decomposes to a sum of eigenvectors $\psi _i$ of Z, and each of these lies in $(({\mathfrak {g}}^*)^{e_{\varphi }})^H_{\le -1}$. Since $({\mathfrak {g}}^*)^{e_{\varphi }}\subset ({\mathfrak {g}}^*)^{h}_{\ge 0}$, we obtain $\psi _i\in (({\mathfrak {g}}^*)^{h}_{\ge 0})^{Z}_{<0}$. Hence, while running Algorithm 4.1.1, some of the eigenvectors $\psi _i$ will join the space $({\mathfrak {g}}^*)^{H_t}_{-2}$ for some $t>0$, leading to an orbit that is greater than, or equal to, that of $\varphi$. To show that it is not equal to $\Gamma \varphi$ note that $\varphi +\psi _i$ lies in the Slodowy slice $\varphi +({\mathfrak {g}}^*)^{e_{\varphi }}$, and thus its complex orbit has bigger dimension than that of $\varphi$. More Whittaker triples will be produced while running Algorithm 4.1.1, but their third elements will lie in $({\mathfrak {g}}^*)^{H_t}_{-1} \cap ({\mathfrak {g}}^*)^{e_\varphi }$ for some $t>0$ and thus lead to bigger orbits using the argument as above. $\square$

Proof of Proposition 4.1.3

As in the statement of Algorithm A let $Z:=S-H$ and $H_t:=H+tZ$. If there are no quasi-critical points in the interval (0, 1) then $t=1$ and the algorithm terminates in one step. In this case we can decompose the expression in the right-hand side of (4.5) as

$$\begin{aligned} \sum _{\gamma \in {\text {Exp}}(({\mathfrak {g}}^H_1)^S_{<1}) }\int _{[{\text {Exp}}(({\mathfrak {g}}^S_1)^H_{>1}(\mathbb {A}))]} {\mathcal {F}}_{H_T, \varphi , \psi '}[\eta ](\gamma ug) du + \mathcal {A}, \end{aligned}$$

(4.9)

where

$$\begin{aligned} \mathcal {A}=\sum _{\begin{array}{c} \gamma \in {\text {Exp}}(({\mathfrak {g}}^H_1)^S_{<1}) \\ 0\ne \psi '\in \Psi ' \end{array}}\int _{[{\text {Exp}}(({\mathfrak {g}}^S_1)^H_{>1}(\mathbb {A}))]} {\mathcal {F}}_{H_T, \varphi , \psi '}[\eta ](\gamma ug) du\,. \end{aligned}$$

(4.10)

By Lemma 4.2.2, for every non-zero $\psi '\in \Psi '=(\mathfrak {g}^*)^{S}_{-1} \cap (\mathfrak {g}^*)^{e_\varphi } \cap (\mathfrak {g}^*)^{Z}_{<0}$ Algorithm 4.1.1 expresses ${\mathcal {F}}_{S,\varphi ,\varphi +\psi '}$ in terms of Fourier coefficients $(S_i,\Phi _i)$ with $\Gamma \Phi _i>\Gamma \varphi$. Thus the term $\mathcal {A}$ satisfies the conditions of the proposition.

Let us now assume that there exist quasi-critical points $0<t_1<\cdots<t_{n}<1$. Lemma 4.2.2 again implies that all non-zero $\psi '\in \Psi ''_i=(\mathfrak {g}^*)^{H_{t_i}}_{-1} \cap (\mathfrak {g}^*)^{e_\varphi } \cap (\mathfrak {g}^*)^{Z}_{<0}$ will lead to Fourier coefficients corresponding to bigger orbits, that will be accumulated in $\mathcal {A}$. It is left to track down the formula for the term that we get when we take all $\psi '$ in all the steps to be zero. Let

$$\begin{aligned} {\mathfrak {v}}_{i}:=({\mathfrak {g}}^{H_{t_i}}_{\ge 1}\cap {\mathfrak {g}}^{Z}_{<0})/({\mathfrak {g}}^{H_{t_i}}_{>1}\cap {\mathfrak {g}}^{Z}_{<0}) \quad \text { and } \quad V_i={\text {Exp}}({\mathfrak {v}}_i) \, . \end{aligned}$$

(4.11)

By Lemma 3.2.7(i) we have

$$\begin{aligned} {\mathcal {F}}^L_{H_{t_i},\varphi }[\eta ](g)=\mathop \int \limits _{V_i}{\mathcal {F}}^R_{H_{t_{i}},\varphi }[\eta ](v_ig) \, dv_i\,. \end{aligned}$$

(4.12)

Using Lemma 3.2.7(ii) (retaining only $\psi ' = 0$) we obtain

$$\begin{aligned} {\mathcal {F}}^R_{H,\varphi }[\eta ](g)=\mathop \int \limits _{V_1}\dots \mathop \int \limits _{V_{n-1}} \mathop \int \limits _{V_n} {\mathcal {F}}^L_{S,\varphi }[\eta ](v_n\dots v_1g) dv +\mathcal {A}. \end{aligned}$$

(4.13)

Since ${\mathfrak {v}}=\bigoplus _{i=1}^n({\mathfrak {g}}^{H_{t_i}}_1\cap {\mathfrak {g}}^{Z}_{<0})$, and as a commutative Lie algebra ${\mathfrak {g}}^{H_{t_i}}_1\cap {\mathfrak {g}}^{Z}_{<0}$ is naturally isomorphic to ${\mathfrak {v}}_i$, the group V is glued from $V_i$. Thus

$$\begin{aligned} \mathop \int \limits _{V_1}\dots \mathop \int \limits _{V_{n-1}} \mathop \int \limits _{V_n} {\mathcal {F}}^L_{S,\varphi }[\eta ](v_n\dots v_1g) dv = \mathop \int \limits _{V} {\mathcal {F}}^L_{S,\varphi }[\eta ](vg) \, dv \, . \end{aligned}$$

(4.14)

The proposition now follows from Lemma 3.2.7(i). $\square$

Proof of Theorem B

Under the assumption of Theorem B, for any $\Phi$ with $\Gamma \Phi >\Gamma \varphi ,$ the neutral coefficient ${\mathcal {F}}_{h,\Phi }[\eta ]$ vanishes. By [19, Theorem C], this implies the vanishing of ${\mathcal {F}}_{S,\Phi }[\eta ]$ for any pair $(S,\Phi )$ with $\Gamma \Phi >\Gamma \varphi$. Thus, the term $\mathcal {A}$ in Proposition 4.1.3 also vanishes. The theorem follows. $\square$

4.3 Additional results

Let us prove for future use some additional results that utilise the technique of this section. For the entire subsection we fix Whittaker pairs $(H,\varphi )$ and $(S,\varphi )$ such that $(H,\varphi )$ dominates $(S,\varphi )$. Let $Z:=S-H$ and let $H_t:=H+tZ$. Let $0<t_1<\dots<t_{n}<1$ be all the critical values between 0 and 1. Let $t_0:=0$ and $t_{n+1}:=1$. Lastly, for each $t_i$, let $R:=R_{t_i}$ and $L:=L_{t_i}$ be defined as in (3.16).

Lemma 4.3.1

Let $\eta$ be an automorphic function such that $\Gamma \varphi \in {\text {WS}}(\eta )$. Then we have ${\mathcal {F}}^R_{H_{t_i},\varphi }[\eta ]={\mathcal {F}}^L_{H_{t_{i+1}},\varphi }[\eta ]$.

Proof

Denote $H_{j}:=H_{t_{j}}$ for any j, and ${\mathfrak {c}}:=({\mathfrak {g}}^*)^{H_{i+1}}_{-1}\cap ({\mathfrak {g}}^*)^{e_{\varphi }}\cap (\mathfrak {g}^*)^Z_{<0}$. Arguing as in the proof of Lemma 3.2.7(ii), we obtain

$$\begin{aligned} {\mathcal {F}}^R_{H_{i},\varphi }[\eta ]=\sum _{\varphi '\in {\mathfrak {c}}}{\mathcal {F}}^L_{H_{i+1},\varphi ,\varphi '}[\eta ]. \end{aligned}$$

(4.15)

We have to show that for any non-zero $\varphi '\in {\mathfrak {c}}$, we have ${\mathcal {F}}^L_{H_{i+1},\varphi ,\varphi '}[\eta ]=0$. But ${\mathcal {F}}^L_{H_{i+1},\varphi ,\varphi '}[\eta ]$ is an integral of ${\mathcal {F}}_{H_{i+1},\varphi ,\varphi '}[\eta ]$, which by Lemma 4.2.2 is expressed through coefficients ${\mathcal {F}}_{S_j,\Phi _j}$ with $\Gamma \Phi _j>\Gamma \varphi$. Since $\varphi \in {\text {WS}}(\eta ),$ ${\mathcal {F}}_{S_j,\Phi _j}[\eta ]\equiv 0$ for all j, and thus ${\mathcal {F}}^L_{H_{i+1},\varphi ,\varphi '}[\eta ]\equiv 0$. $\square$

Corollary 4.3.2

Let $\eta$ be an automorphic function such that $\Gamma \varphi \in {\text {WS}}(\eta )$. Let ${\mathfrak {v}}:= {\mathfrak {g}}^H_{> 1}/({\mathfrak {g}}^H_{> 1}\cap {\mathfrak {g}}^{S}_{\ge 1})$, ${\mathfrak {v}}':= {\mathfrak {g}}^S_{> 1}/({\mathfrak {g}}^S_{> 1}\cap ({\mathfrak {g}}^{H}_{\ge 1}+{\mathfrak {g}}_{\varphi }))$, $V:={\text {Exp}}({\mathfrak {v}}(\mathbb {A}))$, and $V':={\text {Exp}}({\mathfrak {v}}'(\mathbb {A}))$. Then,

$$\begin{aligned} {\mathcal {F}}^R_{H,\varphi }[\eta ](g)=\int _V {\mathcal {F}}^L_{S,\varphi }[\eta ](vg)dv\quad \text { and }\quad {\mathcal {F}}^L_{H,\varphi }[\eta ](g)=\int _{V'} {\mathcal {F}}^R_{S,\varphi }[\eta ](vg)dv\,. \end{aligned}$$

(4.16)

Proof

With the notation introduced above, let

$$\begin{aligned} {\mathfrak {v}}_{i}:=({\mathfrak {g}}^{H_{t_i}}_{\ge 1}\cap {\mathfrak {g}}^{Z}_{<0})/({\mathfrak {g}}^{H_{t_i}}_{>1}\cap {\mathfrak {g}}^{Z}_{<0}) \quad \text { and } \quad V_i={\text {Exp}}({\mathfrak {v}}_i) \, . \end{aligned}$$

(4.17)

By Lemma 3.2.7(i) we have

$$\begin{aligned} {\mathcal {F}}^L_{H_{t_i},\varphi }[\eta ](g)=\mathop \int \limits _{V_i}{\mathcal {F}}^R_{H_{t_{i}},\varphi }[\eta ](v_ig) \, dv_i\,. \end{aligned}$$

(4.18)

By Lemma 4.3.1 we have ${\mathcal {F}}^R_{H_{t_i},\varphi }[\eta ]={\mathcal {F}}^L_{H_{t_{i+1}},\varphi }[\eta ]$. Thus

$$\begin{aligned} {\mathcal {F}}^R_{H,\varphi }[\eta ](g)=\mathop \int \limits _{V_1}\dots \mathop \int \limits _{V_{n-1}} \mathop \int \limits _{V_n} {\mathcal {F}}^L_{S,\varphi }[\eta ](v_n\dots v_1g) dv = \mathop \int \limits _{V} {\mathcal {F}}^L_{S,\varphi }[\eta ](vg) \, dv \, . \end{aligned}$$

(4.19)

Similarly, let

$$\begin{aligned} {\mathfrak {v}}'_{i}:=({\mathfrak {g}}^{H_{t_i}}_{\ge 1})^{Z}_{>0}/({\mathfrak {g}}_{\varphi }\cap {\mathfrak {g}}^{H_{t_i}}_{1}+ ({\mathfrak {g}}^{H_{t_i}}_{>1})^{Z}_{>0})\cong ({\mathfrak {g}}^{H_{t_i}}_{>1})^Z_{>0}/(({\mathfrak {g}}^{H_{t_i}}_{>1})^Z_{>0}\cap {\mathfrak {g}}_{\varphi }) \text { and } V'_i={\text {Exp}}({\mathfrak {v}}'_i) \, . \end{aligned}$$

(4.20)

Then the nilpotent group $V'$ is glued from the commutative groups $V'_i$ and

$$\begin{aligned} {\mathcal {F}}^L_{S,\varphi }[\eta ](g)=\mathop \int \limits _{V'_{n}}\dots \mathop \int \limits _{V'_{2}} \mathop \int \limits _{V'_1} {\mathcal {F}}^R_{H,\varphi }[\eta ](v_1\dots v_{n}g) dv = \mathop \int \limits _{V'} {\mathcal {F}}^L_{H,\varphi }[\eta ](vg) \, dv \, . \end{aligned}$$

(4.21)

$\square$

Proposition 4.3.3

(cf. [19, Theorem A] for a local analogue) For any $\eta \in C^{\infty }(\Gamma \backslash G)$ with ${\mathcal {F}}_{H,\varphi }[\eta ]\equiv 0$ we have ${\mathcal {F}}_{S,\varphi }[\eta ]\equiv 0$.

Proof

Let $Z:=S-H$, and for any $t\ge 0$ let $H_t:=H+tZ$. Let $0< t_1< \dots< t_k < 1$ be all the critical values of t between 0 and 1. Let $t_0:=0$ and $t_{k+1}:=1$. By Lemma 3.2.7(i) and (ii), for any $0\le i\le k$, ${\mathcal {F}}_{H_{t_{i+1}},\varphi }$ is expressed in terms of ${\mathcal {F}}_{H_{t_i},\varphi }$. Since $H_{t_0}=H$, we obtain by induction that ${\mathcal {F}}_{H_{t_i},\varphi }[\eta ]\equiv 0$ for all i. Since $H_{t_{k+1}}=S$, the proposition follows. $\square$

4.4 Levi-distinguished coefficients

Let us show that any Whittaker pair $(H,\varphi )$ dominates a Levi-distinguished Whittaker pair. Using Lemma 2.3.7, decompose $H=h+Z$, where $(h,\varphi )$ is a neutral pair, and Z commutes with h and $\varphi$.

Notation 4.4.1

Let $C\subseteq \Gamma$ denote the centralizer of $(h,\varphi )$. Let A denote a maximal split torus of C such that its Lie algebra ${\mathfrak {a}}$ includes Z, and let M denote the centralizer of ${\mathfrak {a}}$ in G. Then M is a Levi subgroup of G, $\mathfrak {m}$ includes h, Z and $\varphi$. Let z be a rational semi-simple element of ${\mathfrak {a}}$ that is generic in the sense that its centralizer is M.

Lemma 4.4.2

As an element of $\mathfrak {m}$, $\varphi$ is ${\mathbb K}$-distinguished.

Proof

Let $\mathfrak {l}$ be the Lie algebra of a Levi subgroup L of M defined over ${\mathbb K}$ such that $\varphi \in \mathfrak {l}^*$. We have to show that $L=M$. By replacing L by its conjugate by an element of $\Gamma$ we can assume $h\in \mathfrak {l}$, and that there exists a rational semi-simple element $z'\in \mathfrak {m}$ such that $\mathfrak {l}$ is the centralizer of $z'$. Then $z'$ commutes with h and $\varphi$ and we have to show that $z'$ is central in $\mathfrak {m}$.

Indeed, $z'\in \mathfrak {m}\cap {\mathfrak {c}}={\mathfrak {a}}$. Now, any $X\in \mathfrak {m}$ commutes with z, and thus with any element of ${\mathfrak {a}}$, since z is generic in ${\mathfrak {a}}$. Thus ${\mathfrak {a}}$ lies in the center of $\mathfrak {m}$ and thus $z'$ is central. $\square$

Note that the eigenvalues of the adjoint action of any Lie algebra element are symmetric around zero.

Notation 4.4.3

Let N be a positive integer that is bigger than the ratio of the maximal eigenvalue of ${\text {ad}}(z)$ by the minimal positive eigenvalue of ${\text {ad}}(Z)$. Let

$$\begin{aligned} Z':=NZ+z. \end{aligned}$$

(4.22)

From our choice of N we have

$$\begin{aligned} {\mathfrak {g}}^{Z'}_{>0}={\mathfrak {g}}^{Z}_{>0}\oplus ({\mathfrak {g}}^{Z}_{0}\cap {\mathfrak {g}}^{z}_{>0}) \text { and } {\mathfrak {g}}^{Z'}_{0}={\mathfrak {g}}^{z}_{0}=\mathfrak {m}\subseteq {\mathfrak {g}}^{Z}_{0}. \end{aligned}$$

(4.23)

That $\mathfrak {m} \subseteq \mathfrak {g}^Z_0$ follows from the fact that M is the centralizer of z which equals the centralizer of $\mathfrak {a}$ and $\mathfrak {a}$ includes Z.

Lemma 4.4.4

For rational $T > 0$, $(H, \varphi )$ dominates $(H+TZ', \varphi )$, that is, $H,\varphi$ and $T Z'$ commute, and satisfy (2.11).

Proof

By construction $H = h + Z$, $\varphi$ and Z commute, and since $h, Z, \varphi \in \mathfrak {m}$ they commute with z. Thus, $Z'$ commutes with H and $\varphi$. Furthermore, ${\mathfrak {g}}_{\varphi }\cap {\mathfrak {g}}^H_{\ge 1}\subseteq {\mathfrak {g}}^h_{\le 0}\cap {\mathfrak {g}}^H_{\ge 1}\subseteq {\mathfrak {g}}^Z_{>0} \subseteq {\mathfrak {g}}^{Z'}_{>0} = {\mathfrak {g}}^{TZ'}_{>0}.$ $\square$

Lemma 4.4.5

For a fixed $\lambda \in {\mathbb Q}$, and a rational $T > 0$ large enough,

$$\begin{aligned} {\mathfrak {g}}^{H+TZ'}_{> 1}={\mathfrak {g}}^{H+TZ'}_{\ge 2}={\mathfrak {g}}^{Z'}_{>0}\oplus ({\mathfrak {g}}^{Z'}_{0}\cap {\mathfrak {g}}^{H+TZ'}_{>1})={\mathfrak {g}}^{Z'}_{>0}\oplus \mathfrak {m}^{h}_{\ge 2}\text { and }{\mathfrak {g}}^{H+TZ'}_{ \lambda }=\mathfrak {m}^{H}_{\lambda }=\mathfrak {m}^{h}_{\lambda }. \end{aligned}$$

(4.24)

The Fourier coefficient $\mathcal {F}_{H+TZ', \varphi }$ is then Levi-distinguished.

Proof

For large enough T, we have that $\mathfrak {g}^{H+TZ'}_{>1} \cap \mathfrak {g}^{Z'}_{<0} = \{0\}$ and $\mathfrak {g}^{H+TZ'}_{>1} \cap \mathfrak {g}^{Z'}_{>0} = \mathfrak {g}^{Z'}_{>0}$. Thus $\mathfrak {g}^{H+TZ'}_{>1} = \mathfrak {g}^{H+TZ'}_{>1} \cap \bigl ( \mathfrak {g}^{Z'}_{<0} \oplus \mathfrak {g}^{Z'}_0 \oplus \mathfrak {g}^{Z'}_{>0} \bigr ) = \mathfrak {g}^{Z'}_{>0} \oplus \bigl ( \mathfrak {g}^{Z'}_0 \cap \mathfrak {g}^{H+TZ'}_{>1} \bigr )$. Since $H = h + Z$ and $\mathfrak {g}^{Z'}_0 = \mathfrak {m} \subseteq \mathfrak {g}^Z_0$ we have that $\mathfrak {g}^{Z'}_0 \cap \mathfrak {g}^{H+TZ'}_{>1} = \mathfrak {g}^{Z'}_0 \cap \mathfrak {g}^{h}_{>1}$ and since h is neutral $\mathfrak {g}^{h}_{>1} = \mathfrak {g}^{h}_{\ge 2}$. Now $\mathfrak {g}^{Z'}_0 = \mathfrak {m}$ and thus, $\mathfrak {g}^{H+TZ'}_{>1} = \mathfrak {g}^{Z'}_{>0} \oplus \bigl ( \mathfrak {g}^{Z'}_0 \cap \mathfrak {g}^{h}_{\ge 2} \bigr ) = \mathfrak {g}^{Z'}_{>0} \oplus \mathfrak {m}^h_{\ge 2}$. Doing the same manipulations for $\mathfrak {g}^{H+TZ'}_{\ge 2}$ one ends up with the same result, proving the equality $\mathfrak {g}^{H+TZ'}_{>1} = \mathfrak {g}^{H+TZ'}_{\ge 2}$.

Now, for any fixed $\lambda \in {\mathbb Q}$ and a large enough T, we have that $\mathfrak {g}^{H+TZ'}_\lambda = \mathfrak {g}^{H}_\lambda \cap \mathfrak {g}^{Z'}_0 = \mathfrak {g}^{H}_\lambda \cap \mathfrak {m} = \mathfrak {m}^H_\lambda$. Again, since $H = h + Z$ and $\mathfrak {m} \subseteq \mathfrak {g}^Z_0$, we get that $\mathfrak {m}^H_\lambda = \mathfrak {m}^h_\lambda$.

Since $H + TZ' = h + Z + TZ'$, the semi-simple element denoted by Z in Definition 2.3.5 is here $Z + TZ'$, which, for large enough T has the centralizer $\mathfrak {g}^Z_0 \cap \mathfrak {g}^{Z'}_0 = \mathfrak {g}^{Z'}_0 = \mathfrak {m}$. By Lemma 4.4.2, $\varphi$ is ${\mathbb K}$-distinguished in $\mathfrak {m}$. Since $\mathfrak {g}^Z_{>0} \subseteq \mathfrak {g}^{Z'}_{>0}$ we have that $\mathfrak {g}^{Z+TZ'}_{>0} = \mathfrak {g}^{Z'}_{>0}$ and thus (4.24) implies (2.12) which means that ${\mathcal {F}}_{H+TZ', \varphi }$ is Levi-distinguished. $\square$

Corollary 4.4.6

Any Whittaker pair $(H,\varphi )$ dominates a Levi-distinguished Whittaker pair.

Corollary 4.4.7

Algorithm A allows us to express any Fourier coefficient ${\mathcal {F}}_{H,\varphi }$ in terms of Levi-distinguished Fourier coefficients with characters in orbits which are equal to or bigger than $\Gamma \varphi$.

Proof

Choose a Levi-distinguished Whittaker pair $(S,\varphi )$ dominated by $(H,\varphi )$. Then Algorithm A expresses ${\mathcal {F}}_{H,\varphi }$ in terms of ${\mathcal {F}}_{S,\varphi }$, and Fourier coefficients ${\mathcal {F}}_{H'_i,\Phi _i}$ corresponding to higher orbits. Each of the pairs $(H'_i,\Phi _i)$ dominates a Levi-distinguished Whittaker pair $(S_i,\Phi _i)$. We repeat the procedure for each pair. The process terminates in a finite number of steps since the dimension of each complex orbit $\mathbf{G}({\mathbb C})\Phi _i$ is bigger than that of $\mathbf{G}({\mathbb C})\varphi$. $\square$

Remark 4.4.8

• (i) Note that $\eta ={\mathcal {F}}_{0,0}[\eta ]$. Thus, the algorithm allows us to express any automorphic function in terms of its Levi-distinguished Fourier coefficients.

• (ii) Algorithm A produces a general formula, that holds for all automorphic functions $\eta \in C^{\infty }(\Gamma \backslash G)$. However, if we put additional assumptions on $\eta$ the algorithm might terminate earlier and produce a shorter expression.

• (iii) By Lemma 2.3.9, the Levi-distinguished Fourier coefficients of PL elements are Whittaker coefficients. This implies that if all Levi-distinguished Fourier coefficients of some automorphic function $\eta$ corresponding to non-PL orbits vanish, then the algorithm allows us to express $\eta$ in terms of its Whittaker coefficients.

• (iv) Vanishing as in (iii) happens in two important cases. One is the case of ${\text {GL}}_n$ in which all orbits are PL orbits. We explain the results in this case in Sect. 5.3 below. Another case is the case when G is simply-laced, and $\eta$ is minimal or next-to-minimal. In this case the output of the algorithm is analyzed in great detail in [21].

Let us now go to the other extreme and consider cuspidal $\eta$.

Lemma 4.4.9

Let $\eta \in C^{\infty }(\Gamma \backslash G)$, and assume that the constant term $c_U[\eta ]:=\int _{[U]}\eta (u)du$ vanishes for any $U\subset G$ which is a unipotent radical of a proper parabolic subgroup. Let ${\mathcal {F}}_{S,\varphi }[\eta ]$ be a non-vanishing Levi-distinguished Fourier coefficient. Then the orbit $\Gamma \varphi \in {\mathfrak {g}}^*$ is ${\mathbb K}$-distinguished.

Proof

Recall that by Definition 2.3.5, there is a decomposition $S=h+Z$ such that $(h,\varphi )$ be a neutral Whittaker pair for $\mathfrak {l}:={\mathfrak {g}}^Z$, the orbit of $\varphi$ in $\mathfrak {l}^*$ is ${\mathbb K}$-distinguished, and

$$\begin{aligned} {\mathfrak {g}}^{h+Z}_{> 1}={\mathfrak {g}}^{h+Z}_{\ge 2}= {\mathfrak {g}}^{Z}_{>0}\oplus \mathfrak {l}^{h}_{\ge 2}\text { and }{\mathfrak {g}}^{h+Z}_{ 1}=\mathfrak {l}^{h}_{1}. \end{aligned}$$

(4.25)

Let $\mathfrak {p}:={\mathfrak {g}}^Z_{\ge 0}$, P be the corresponding parabolic subgroup, and U be the unipotent radical of P. By (4.25), ${\mathcal {F}}_{S,\varphi }[\eta ]={\mathcal {F}}_{h,\varphi }(c_U[\eta ])$, where we view $c_U[\eta ]$ as an element of $C^{\infty }(\Gamma \backslash G)$. Since ${\mathcal {F}}_{S,\varphi }[\eta ]$ does not vanish, neither does $c_U[\eta ]$ and thus $P=G$. Thus $L=G$ and thus the orbit $\Gamma \varphi \in {\mathfrak {g}}^*$ is ${\mathbb K}$-distinguished. $\square$

5 Applications and examples

In this section we will illustrate how to apply the framework introduced in this paper to compute certain Fourier coefficients in detail. We begin in Sect. 5.1 to consider the case when G is split and simply-laced and $P\subset G$ is a parabolic subgroup with unipotent radical U isomorphic to a Heisenberg group. We use Algorithm A to express any automorphic function on G in terms of its Fourier coefficients with respect to U. In Sect. 5.2 we then give an example of a Whittaker triple and a quasi-Fourier coefficient for the group $G={\text {SL}}_4$. In Sect. 5.3 we demonstrate Algorithm A, Corollary 4.4.7 and Remark 4.4.8 for $G= {\text {GL}}_n$. In Sect.5.4 we demonstrate them for $G= {\text {Sp}}_4$.

In [21] we apply Theorem B, Algorithm A, and Proposition 5.1.5 below to automorphic forms in small automorphic representations of all simple split simply-laced groups.

As many examples below are built on classical groups, we shall use matrix notation and denote by $e_{ij}$ the elementary matrix with a 1 at position (i, j) and zeroes elsewhere.

5.1 Fourier expansions along Heisenberg parabolics

Let G be split and simply-laced, and let ${\mathfrak {h}}\subset {\mathfrak {g}}$ be the Lie algebra of a maximal split torus. Fix a choice of positive roots. For any simple root ${\alpha }$ define $S_{{\alpha }}\in {\mathfrak {h}}$ by ${\alpha }(S_{{\alpha }})=2$ and $\beta (S_{{\alpha }})=0$ for any other simple root $\beta$.

Definition 5.1.1

We say that a simple root $\alpha$ is a Heisenberg root if $\mathfrak {g}^{S_\alpha }_{>0}$ is a Heisenberg Lie algebra, or, equivalently, if ${\mathfrak {g}}^{S_\alpha }_{4}$ has dimension one.

Lemma 5.1.2

If ${\mathfrak {g}}$ is simple of type $A_n$, there are no Heisenberg roots. If ${\mathfrak {g}}$ is simple of type $D_n$ or $E_n$ then there exists a unique Heisenberg root, and this is the unique simple root $\alpha$ satisfying $\langle {\alpha }, {\alpha }_{\max }\rangle =1,$ where ${\alpha }_{\max }$ denotes the highest root.

Proof

Let ${\mathfrak {g}}$ be simple and let ${\alpha }$ be a Heisenberg root. Then ${\mathfrak {g}}^{S_\alpha }_{4}$ has to be the highest weight space of the adjoint representation, i.e. the root space of ${\alpha }_{\max }$. Since ${\mathfrak {g}}^{S_\alpha }_{4}$ is one-dimensional, ${\alpha }_{\max }-\beta$ is not a root for any simple root $\beta \ne {\alpha }$. Thus ${\alpha }_{\max }-{\alpha }$ is a root, and thus $\langle {\alpha }, {\alpha }_{\max }\rangle =1$. The roots ${\alpha }$ with this property are precisely the nodes in the affine Dynkin diagram that are connected to the affine node (corresponding to $-{\alpha }_{\max }$).

Checking the affine Dynkin diagrams (see [6, Tables IV-VII]), we see that there is a unique simple root ${\alpha }$ with this property in types $D_n$ and $E_n$, and these roots are indeed Heisenberg. In the Bourbaki notation, these roots are ${\alpha }_2$ for $D_n$ and $E_6$, ${\alpha }_1$ for $E_7$ and ${\alpha }_8$ for $E_8$. In type $A_n$ there are two roots with this property but none of them is Heisenberg. In fact, $\mathfrak {g}^{S_\beta }_{>0}$ is abelian for any simple root $\beta$ in type $A_n$. This is so, since in type $A_n$, ${\alpha }_{\max }$ is the sum of all simple roots (with all coefficients being 1). $\square$

Notation 5.1.3

Let ${\alpha }$ be a root. Define $h_{{\alpha }}:={\alpha }^{\vee }\in {\mathfrak {h}}$ by requiring for all roots $\beta$

$$\begin{aligned} \beta (h_{{\alpha }})=2\frac{\langle {\alpha },\beta \rangle }{\langle {\alpha },{\alpha }\rangle }=\langle {\alpha },\beta \rangle \,. \end{aligned}$$

(5.1)

Denote also by ${\mathfrak {g}}^{\times }_{-{\alpha }}$ the set of non-zero covectors in the dual root space ${\mathfrak {g}}^{*}_{-{\alpha }}$.

Note that for $\beta \ne {\pm } {\alpha },$ $\beta (h_{{\alpha }})\in \{-1,0,1\}$. By [25, Proposition II.8.3], $(h_{{\alpha }},\varphi )$ is a neutral pair for any $\varphi \in {\mathfrak {g}}^{\times }_{-{\alpha }}$.

Notation 5.1.4

For any Heisenberg root ${\alpha }$, let $\Omega _{{\alpha }}\subset \Gamma$ be the abelian subgroup obtained by exponentiation of the abelian Lie algebra given by the direct sum of the root spaces of negative roots $\beta$ satisfying $\langle {\alpha }, \beta \rangle =1$. Let

$$\begin{aligned} \Psi _\alpha :=\{ \text { root } \varepsilon \mid \langle \varepsilon , {\alpha }\rangle \le 0, \varepsilon (S_{{\alpha }})=2\}. \end{aligned}$$

(5.2)

Note that all the roots in $\Psi _{{\alpha }}$ have to be positive.

In this subsection we use Algorithm A to deduce the following proposition, that will be used in the sequel paper [21].

Proposition 5.1.5

Let ${\alpha }$ be a Heisenberg root. Let $\gamma _{{\alpha }}\in \Gamma$ be a representative of a Weyl group element that conjugates ${\alpha }$ to ${\alpha }_{\max }$, where ${\alpha }_{\max }$ denotes the maximal root of the component of ${\mathfrak {g}}$ corresponding to ${\alpha }$. Then we have

$$\begin{aligned} \eta (g)=\sum _{\varphi \in ({\mathfrak {g}}^*)^{S_{{\alpha }}}_{-2}}{\mathcal {F}}_{S_{{\alpha }},\varphi }[\eta ](g)+\sum _{\varphi \in {\mathfrak {g}}^{\times }_{-{\alpha }}}\sum _{\omega \in \Omega _{{\alpha }}}\sum _{ \psi \in \bigoplus _{\varepsilon \in \Psi _\alpha }{\mathfrak {g}}^*_{-\varepsilon }}{\mathcal {F}}_{S_{{\alpha }},\varphi +\psi }[\eta ](\omega \gamma _{{\alpha }}g)\,. \end{aligned}$$

(5.3)

For the proof we will need the following lemma.

Lemma 5.1.6

Let ${\alpha }$ be a Heisenberg root. Then for any $\varphi \in {\mathfrak {g}}^{\times }_{-{\alpha }}$ we have

$$\begin{aligned} {\mathcal {F}}_{h_{{\alpha }},\varphi }[\eta ](g)=\sum _{\omega \in \Omega _{{\alpha }}}\sum _{ \psi \in \bigoplus _{\varepsilon \in \Psi _\alpha }{\mathfrak {g}}^*_{-\varepsilon }}{\mathcal {F}}_{S_{{\alpha }},\varphi +\psi }[\eta ](\omega g)\,. \end{aligned}$$

(5.4)

Proof

Following Algorithm A we consider the deformation $(1-t)h_{{\alpha }}+tS_{{\alpha }}$. By Lemma 3.1.1, we have

$$\begin{aligned} {\mathcal {F}}_{h_{{\alpha }},\varphi }[\eta ](g)=\sum _{\omega \in \Omega _{{\alpha }}}{\mathcal {F}}^{R_0}_{h_{{\alpha }},\varphi }[\eta ](\omega g)\,. \end{aligned}$$

(5.5)

Then, the critical values are 1/2 and 2/3, and the quasi-critical values are 1/3 and 1. At 1/3, we have no Whittaker triple entries yet and thus nothing moves into the $-2$-eigenspace. At 1/2, we get contributions in the third component of the Whittaker triple from the root spaces of all the roots $\varepsilon$ with $\langle \varepsilon , {\alpha }\rangle = 0$ and $\varepsilon (S_{\alpha }) =- 2$. At $t=2/3$ we also get all the negative roots with $\langle \varepsilon , {\alpha }\rangle = 1$ and $\varepsilon (S_{\alpha }) =- 2$. This means that we would get contributions from all these root spaces in the third component of the Whittaker triple. At $t=1$ the Whittaker triple becomes a Whittaker pair and thus we obtain

$$\begin{aligned} {\mathcal {F}}_{h_{{\alpha }},\varphi }[\eta ](g)=\sum _{\omega \in \Omega _{{\alpha }}}{\mathcal {F}}^{R_0}_{h_{{\alpha }},\varphi }[\eta ](\omega g)= \sum _{\omega \in \Omega _{{\alpha }}}\sum _{ \psi \in \bigoplus _{\varepsilon \in \Psi _\alpha }{\mathfrak {g}}^*_{-\varepsilon }}{\mathcal {F}}_{S_{{\alpha }},\varphi +\psi }[\eta ](\omega g)\,. \end{aligned}$$

(5.6)

$\square$

Since $\eta ={\mathcal {F}}_{0,0}[\eta ],$ in order to express $\eta$ in terms of Fourier coefficients of the form ${\mathcal {F}}_{S_{{\alpha }},\varphi }$ we need to consider the deformation $S_t:=tS_{{\alpha }}$. To simplify the exposition we do that in more elementary terms.

Proof of Proposition 5.1.5

By the conditions, the Lie algebra ${\mathfrak {g}}^{S_{{\alpha }}}_{>0}$ is a Heisenberg Lie algebra, with center ${\mathfrak {g}}^{S_{{\alpha }}}_{4}$, and abelian quotient ${\mathfrak {g}}^{S_{{\alpha }}}_{2}$. We restrict $\eta$ to the exponential of the center and decompose to Fourier series. The constant term with respect to the center ${\mathfrak {g}}^{S_{{\alpha }}}_{4}$ is ${\mathcal {F}}_{S_{{\alpha }}/3,0}[\eta ]$, and the other terms are ${\mathcal {F}}_{S_{{\alpha }}/2,\varphi }[\eta ]$ for $\varphi \ne 0\in ({\mathfrak {g}}^*)^{S_{{\alpha }}}_{-4} =({\mathfrak {g}}^*)^{S_{{\alpha }}/2}_{-2}$. We remark that this constant term can be denoted ${\mathcal {F}}_{cS_{{\alpha }},0}[\eta ]$ for any $1/4 \le c<1/2$ but not for $c=1/2$ since 0 defines a zero form on the 1-eigenspace, and thus ${\mathfrak {n}}_{cS_{{\alpha }},0}={\mathfrak {g}}^{cS_{{\alpha }}}_{\ge 1}={\mathfrak {g}}^{S_{{\alpha }}}_{\ge c^{-1}}$ and ${\mathfrak {n}}_{S_{{\alpha }}/2,0}={\mathfrak {g}}^{S_{{\alpha }}}_{\ge 2}$. Note also that $({\mathfrak {g}}^*)^{S_{{\alpha }}/2}_{-2}={\mathfrak {g}}^{\times }_{-{\alpha }_{\max }}$. Altogether we have

$$\begin{aligned} \eta (g)={\mathcal {F}}_{S_{{\alpha }}/3,0}[\eta ](g)+\sum _{\varphi \in {\mathfrak {g}}^{\times }_{-{\alpha }_{\max }}}{\mathcal {F}}_{S_{{\alpha }}/2,\varphi }[\eta ](g)\,. \end{aligned}$$

(5.7)

Note that $\gamma _{{\alpha }}$ conjugates $S_{{\alpha }}/2$ to $h_{{\alpha }}$. Thus, by Lemma 3.2.8, we have

$$\begin{aligned} {\mathcal {F}}_{S_{{\alpha }}/2,\varphi }[\eta ](g)={\mathcal {F}}_{h_{{\alpha }},{\text {Ad}}^*(\gamma _{{\alpha }})\varphi }[\eta ](\gamma _{{\alpha }}g) \end{aligned}$$

and

$$\begin{aligned} \sum _{\varphi \in {\mathfrak {g}}^{\times }_{-{\alpha }_{\max }}}{\mathcal {F}}_{S_{{\alpha }}/2,\varphi }[\eta ](g)=\sum _{\varphi \in {\mathfrak {g}}^{\times }_{-{\alpha }}}{\mathcal {F}}_{h_{{\alpha }},\varphi }[\eta ](\gamma _{{\alpha }}g)\,. \end{aligned}$$

(5.8)

We restrict the constant term of (5.7) to the maximal abelian quotient of ${\text {Exp}}({\mathfrak {g}}^{S_{{\alpha }}}_{>0})$, decompose to Fourier series and obtain

$$\begin{aligned} {\mathcal {F}}_{S_{{\alpha }}/3,0}[\eta ](g)=\sum _{\varphi \in ({\mathfrak {g}}^*)^{S_{{\alpha }}}_{-2}}{\mathcal {F}}_{S_{{\alpha }},\varphi }[\eta ](g)\,. \end{aligned}$$

(5.9)

Formula (5.3) follows now from (5.7), (5.8), (5.4) and (5.9). $\square$

Remark 5.1.7

Let us explain why we chose to use $S_{{\alpha }}$ in Proposition 5.1.5. In types $E_6,E_7,$ and $E_8$ this choice follows Sect. 4.4. Indeed, the starting point is $\eta ={\mathcal {F}}_{0,0}[\eta ]$. In the first step we choose a generic element z in the Cartan. Choose z to be 2 on the Heisenberg root ${\alpha }$, and to be very small positive rational numbers on other roots. This obtained deformation gives the same results as the deformation with $Z=S_{{\alpha }}$. The first critical value is 1/4, at which we obtain the decomposition described in (5.7). With the constant term we can proceed to the next critical value 1/2, at which we obtain the decomposition in (5.9).

Then we conjugate the non-constant terms obtained in (5.7) by $\gamma _{{\alpha }}$. This is not part of the algorithm, but we do that for convenience. Now we need to choose a generic z that commutes with the root space of ${\alpha }$. We choose it to be 1 on the only simple root non-orthogonal to ${\alpha }$, and very small positive rational numbers on other simple roots. The resulting decomposition appears in Lemma 5.1.6. Altogether, this gives Proposition 5.1.5.

To express $\eta$ in terms of its Levi-distinguished coefficients one should continue with each of the terms in the right-hand side of (5.3). However, the obtained expression would be very long and complicated. In [21] we provide the expression under the assumption that the Whittaker support of $\eta$ consists of orbits with Bala–Carter labels $A_1$ and $2A_1$. Since these orbits are PL orbits, in this case we obtain an expression for $\eta$ in terms of its Whittaker coefficients.

For groups of type $D_n$, Sect. 4.4 would provide us with a different formula, but we still prove formula (5.3) for all cases for its uniformity, beauty, and future applications.

Remark 5.1.8

Here we elaborate a little on the structure of the Fourier expansion (5.3) and comment on the relation to previous works on Heisenberg expansions. The semisimple element $S_{\alpha }$ defines a Heisenberg parabolic subgroup $P_{\alpha }\subset G$ with Levi decomposition $P_{\alpha }=LU$. The Lie algebra $\mathfrak {p}_{\alpha }\subset \mathfrak {g}$ of $P_{\alpha }$ exhibits the following grading

$$\begin{aligned} \mathfrak {p}_{\alpha }=\mathfrak {g}_0\oplus \mathfrak {g}_1\oplus \mathfrak {g}_2, \end{aligned}$$

(5.10)

where the subscripts indicate the values of the inner products $\langle \,\cdot \, , {\alpha }_{\max }\rangle$. Thus $\mathfrak {g}_1$ is spanned by all roots $\varepsilon$ such that $\langle \varepsilon , {\alpha }_{\max }\rangle =1$. Equivalently, these are all roots $\varepsilon$ such that ${\alpha }_{\max }-\varepsilon$ is also a root. Notice that all roots in $\mathfrak {g}_1$ are positive and the only simple root satisfying the condition $\langle \varepsilon , {\alpha }_{\max }\rangle =1$ is $\alpha$ itself. Since only $\varepsilon =\alpha _{\max }$ satisfies $\langle \varepsilon , {\alpha }_{\max }\rangle =2$ the space $\mathfrak {g}_2$ is one-dimensional and equals $\mathfrak {g}_{\alpha _{\max }}$. The zeroth subspace $\mathfrak {g}_0$ is the Lie algebra of the Levi $L\subset P$. The subspace $\mathfrak {g}_1\oplus \mathfrak {g}_2$ is thus the Heisenberg nilpotent subalgebra with center $\mathfrak {g}_2$. Notice that $\sum _{\gamma \in \Psi _\alpha }\mathfrak {g}_\gamma$ is a Lagrangian subspace of $\mathfrak {g}_1$. Indeed, $\mathfrak {g}_1$ has a canonical Lagrangian decomposition

$$\begin{aligned} \mathfrak {g}_1=\sum _{\gamma \in \Psi _\alpha }\mathfrak {g}_\gamma \oplus \sum _{\gamma \in \Psi ^{\perp }_\alpha }\mathfrak {g}_\gamma , \end{aligned}$$

(5.11)

where $\Psi _{\alpha }^{\perp }$ is the orthogonal complement

$$\begin{aligned} \Psi _{\alpha }^{\perp }=\{\text { root } \varepsilon \mid \langle \varepsilon , {\alpha }\rangle \ge 1, \langle \varepsilon , {\alpha }_{\max }\rangle =1 \}. \end{aligned}$$

(5.12)

Note that the root $\alpha$ belongs to $\Psi _{\alpha }^{\perp }$. The Fourier expansion (5.3) thus corresponds to the standard non-abelian Fourier expansion along the Heisenberg unipotent U, which exhibits a sum over the center $\mathfrak {g}_2$ along with a sum over a Lagrangian subspace $\Psi _{\alpha }$ of $\mathfrak {g}_1$. The choice of Lagrangian decomposition is usually referred to as a choice of “polarization”. Similar kinds of expansions have been treated in several places in the literature; see [2, 13, 30, 31, 36] for a sample. In the notation of the original paper by Kazhdan and Savin [31], the space $\Psi _{\alpha }$ corresponds to $\Pi _o^{*}$ while $\Psi _{\alpha }^{\perp }$ corresponds to $\Pi _o$.

5.2 Whittaker triples

We will now illustrate what type of quasi-Fourier coefficients we are able to describe using Whittaker triples that are not captured by Whittaker pairs in an example for $G = {\text {SL}}_4$.

Let $(S, \varphi , \psi )$ be the Whittaker triple with $S = \tfrac{1}{3}{\text {diag}}(3,1,-1,-3)$, $\varphi = e_{41}$ and $\psi = m e_{31} + n e_{42}$, where $m,n \in {\mathbb K}$ and $e_{ij}$ denote elementary matrices. The S-eigenvalues for the different elementary matrices can be illustrated by the following matrix

$$\begin{aligned} \left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 0 &{} 2/3 &{} 4/3 &{} 2 \\ -2/3 &{} 0 &{} 2/3 &{} 4/3 \\ -4/3 &{} -2/3 &{} 0 &{} 2/3 \\ -2 &{} -4/3 &{} -2/3 &{} 0 \\ \end{array} \, \right) , \end{aligned}$$

(5.13)

from which we may read out that $\varphi$ has eigenvalue $-2$ while $\psi$ has eigenvalue $-4/3$.

As seen from this matrix we get the following unipotent subgroup (independent of $\psi$)

$$\begin{aligned} N_{S,\varphi } = \left\{ \left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 1 &{} 0 &{} x_2 &{} x_1 \\ &{} 1 &{} 0 &{} x_3 \\ &{} &{} 1 &{} 0 \\ &{} &{} &{} 1 \end{array} \right) : x_1, x_2, x_3 \in \mathbb {A} \right\} \, , \end{aligned}$$

(5.14)

and the corresponding Fourier coefficient of an automorphic function $\eta$ can be expressed as

$$\begin{aligned} \mathcal {F}_{S,\varphi ,\psi }[\eta ](g) = \mathop \int \limits _{({\mathbb K}\backslash \mathbb {A})^3} \eta \left( \left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 1 &{} 0 &{} x_2 &{} x_1 \\ &{} 1 &{} 0 &{} x_3 \\ &{} &{} 1 &{} 0 \\ &{} &{} &{} 1 \end{array}\right) g\right) {\chi (x_1 + m x_2 + n x_3)}^{-1} \, d^3x \, , \end{aligned}$$

(5.15)

where we recall that $\chi$ is a fixed non-trivial character on $\mathbb {A}$ trivial on $\mathbb {K}$.

From this example we see that we require Whittaker triples in addition to Whittaker pairs if we want to construct Fourier coefficients with characters that are not only supported on $x_1$ but also on $x_2$ and $x_3$.

5.3 The case of ${\text {GL}}_n$

Let $\mathbf {G}:={\text {GL}}_n$, $G:={\text {GL}}_n(\mathbb {A})$, and $\Gamma :={\text {GL}}_n({\mathbb K})$. In this section we will follow Algorithm A and Sect. 4.4 to present any automorphic function $\eta \in C^{\infty }(\Gamma \backslash G)$ as a countable linear combination of its Whittaker coefficients. We will show that our proof amounts in this case morally to the same decomposition as in [28, 34, 37].

In [34, 37], $\eta$ is first restricted to the mirabolic nilradical, i.e.

$$\begin{aligned} U=\left\{ \left( \begin{array}{c@{\quad }c} {\text {Id}}_{n-1} &{} * \\ 0 &{}1\\ \end{array}\right) \right\} , \end{aligned}$$

(5.16)

and decomposed into Fourier series with respect to U. Our algorithm does the same thing, but in several steps. First let $(h,\varphi )=(0,0)$. Let $N\gg 0$,

$$\begin{aligned} z_1:={\text {diag}}(0,-1,-N,\dots ,-N^{n-3},-N^{n-2}), \end{aligned}$$

(5.17)

and consider the deformation $S_t:=tz_{1}$. Under this deformation, the first thing that happens is that the highest root space (spanned by $e_{1n}$) enters ${\mathfrak {g}}^{S_t}_1$. At this point $\eta$ decomposes into a sum of quasi-Fourier coefficients. At the next step $e_{1n}$ enters ${\mathfrak {g}}^{S_t}_2$, and the quasi-Fourier coefficients become Fourier coefficients. For the constant term, we continue with the same deformation, until $e_{2n}$ enters. For the non-constant term we have to change the deformation into something that will commute with the corresponding new $\varphi$. This $\varphi$ can be identified with $ae_{n1}$ under the trace form, for some $a\in {\mathbb K}^{\times }$. We take the deformation by

$$\begin{aligned} z_2:={\text {diag}}(-N^{n-2},-1,-N,\dots ,-N^{n-4},-N^{n-3},-N^{n-2}), \end{aligned}$$

(5.18)

and continue in the same way. Eventually, all of U enters and all possible characters (including the trivial one) appear.

Let us now analyze the summands. The constant term is ${\mathcal {F}}_{tz_1,0}$ for $t=2/(N^{n-2}-N^{n-3})$, and we can continue the deformation along $z_1$. Any non-trivial character of U can be conjugated using ${\text {GL}}_{n-1}$ (embedded into the upper left corner) to the one given by $e_{n,n-1}$. We can now choose the deformation

$$\begin{aligned} z_3:={\text {diag}}(-1,-N,\dots ,-N^{n-4},-N^{n-3},-N^{n-3}). \end{aligned}$$

(5.19)

In the same way as above, it will give a decomposition of ${\mathcal {F}}_{tz_1,0}$ into Fourier series with respect to the column $n-1$, i.e.

$$\begin{aligned} U'=\left\{ \left( \begin{array}{c@{\quad }c@{\quad }c} {\text {Id}}_{n-2} &{} * &{} 0 \\ 0 &{}1&{}0\\ 0&{}0&{}1 \end{array}\right) \right\} . \end{aligned}$$

(5.20)

Continuing in this way we obtain

$$\begin{aligned} \eta (g)=\sum _{x\in 2^{[n-1]}}\sum _{\gamma \in \Gamma _x}{\mathcal {F}}_{S,\varphi _x}[\eta ](\gamma g), \end{aligned}$$

(5.21)

where $[n-1]$ denotes the set $\{1,\dots ,n-1\}$, $2^{[n-1]}$ denotes the set of all its subsets, $S={\text {diag}}(n-1,n-3,\dots , 3-n,1-n)$, and for any $x\in 2^{[n-1]}$, $\psi _x:=\sum _{i\in x}e_{i+1,i}$ and $\Gamma _x$ is a certain subset of $\Gamma$.

For cuspidal $\eta$ and $x\ne [n-1]$, we have ${\mathcal {F}}_{S,\varphi _x}[\eta ]=0$ and (5.21) becomes the formula in [34, 37]. For $\eta$ in the discrete spectrum, ${\mathcal {F}}_{S,\varphi _x}[\eta ]$ vanishes for many x by [28, Lemma 3.2], and (5.21) reflects the formula in [28, Theorem 3.3]. If $\eta$ is minimal then ${\mathcal {F}}_{S,\varphi _x}[\eta ]=0$ for $|x|>1$ and if $\eta$ is next-to-minimal then ${\mathcal {F}}_{S,\varphi _x}[\eta ]=0$ for $|x|>2$. These cases were computed in [1], motivated by applications in string theory.

5.4 Examples for ${\text {Sp}}_{4}$

Let $G:={\text {Sp}}_4(\mathbb {A})$, $\Gamma :={\text {Sp}}_4({\mathbb K})$ and let $\eta \in C^{\infty }(\Gamma \backslash G)$. In this section we express $\eta$ in terms of its Levi-distinguished Fourier coefficients, providing an example for Algorithm A and Remark 4.4.8. Let ${\mathfrak {g}}:={\text {Lie}}(\Gamma )$, realized in ${\mathfrak {gl}}_4$ by the $2\times 2$ block matrices

$$\begin{aligned} \left( \begin{array}{c@{\quad }c} A &{} B=B^t \\ C=C^t &{} -A^t\\ \end{array} \right) . \end{aligned}$$

(5.22)

Let ${\mathfrak {n}}\subset {\mathfrak {g}}$ be the maximal unipotent subalgebra spanned by the matrices $e_{12}-e_{43}$, $e_{13}$, $e_{24}$, $e_{14}+e_{23}$ and let $N:={\text {Exp}}({\mathfrak {n}}(\mathbb {A}))$. For any $a,b\in {\mathbb K}$ denote by $\chi _{a,b}$ the character of ${\mathfrak {n}}$ given by $\chi _{a,b}(e_{12}-e_{43})=a$ and $\chi _{a,b}(e_{24})=b$, and let $\mathcal {W}_{a,b}$ denote the corresponding Whittaker coefficient. Let ${\mathfrak {u}}\subset {\mathfrak {n}}$ be the Siegel nilradical, i.e. the normal commutative subalgebra spanned by the matrices $e_{13}$, $e_{24}$, $e_{14}+e_{23}$ and let $U:={\text {Exp}}({\mathfrak {u}}(\mathbb {A}))$. Let L denote the Siegel Levi subgroup of $\Gamma$ given by ${\text {diag}}(g,(g^t)^{-1})$, where $g\in {\text {GL}}_2(\mathbb {A})$. Using the trace form on ${\mathfrak {g}}$, we can identify ${\mathfrak {u}}^*$ with the nilradical $\bar{\mathfrak {u}}$ of the opposite parabolic, i.e. with the space of matrices of the form (5.22) with $A=B=0$. Note that $\bar{\mathfrak {u}}\cong {\text {Sym}}^2({\mathbb K}^2)$, and L acts on it by the standard action on symmetric forms. For any $\varphi \in {\mathfrak {u}}^*\cong \bar{\mathfrak {u}}\cong {\text {Sym}}^2({\mathbb K}^2)$, denote by ${\mathcal {F}}_{{\mathfrak {u}},\varphi }$ the corresponding parabolic Fourier coefficient.

Let us now outline the strategy for this subsection. According to Sect. 4.4 we should choose a generic element z of the Cartan. Choose it to be 1 on the long simple root ${\alpha }_2=2\varepsilon _2$ that defines U and a small positive rational number on the short simple root. Then, the first steps of the algorithm will provide us with decomposition of $\eta$ into Fourier series along the abelian unipotent radical U. The coefficients will be parameterized by characters of ${\mathfrak {u}}$, that can be identified with quadratic forms on ${\mathbb K}^2$. For the zero form we can just continue the deformation all the way to a Whittaker coefficient. All forms of rank one are conjugate, so we can conjugate them to a convenient form and again continue the deformation, obtaining a Whittaker coefficient. The same goes for split forms of rank two. The non-split forms of rank two belong to ${\mathbb K}$-distinguished orbits, and thus the corresponding Fourier coefficients are already Levi-distinguished.

We will now give all the details of the decompositions we just mentioned. We will do this in elementary terms in a self-contained way.

Since U is abelian, the Fourier decomposition on it gives

$$\begin{aligned} \eta =\sum _{\varphi \in {\mathfrak {u}}^*}{\mathcal {F}}_{{\mathfrak {u}},\varphi }[\eta ]\,. \end{aligned}$$

(5.23)

We now decompose this sum into three different terms, by the rank of $\varphi$, viewed as a quadratic form. Let us first analyze the constant term ${\mathcal {F}}_{{\mathfrak {u}},0}[\eta ]$. We restrict it to L, and decompose to Fourier series on the abelian group $N\cap L$. We obtain

$$\begin{aligned} {\mathcal {F}}_{{\mathfrak {u}},0}[\eta ]=\sum _{a\in {\mathbb K}}\mathcal {W}_{a,0}[\eta ]\,. \end{aligned}$$

(5.24)

Next, any $\varphi$ of rank one is conjugate under L to $\varphi _1:= \left( \begin{array}{c@{\quad }c} 1 &{} 0 \\ 0 &{} 0\\ \end{array}\right)$. This $\varphi _1$ is normalized by N, and thus we can again decompose ${\mathcal {F}}_{{\mathfrak {u}},\varphi }[\eta ]$ on $N\cap L$. We obtain

$$\begin{aligned} {\mathcal {F}}_{{\mathfrak {u}},\varphi _1}[\eta ]=\sum _{a\in {\mathbb K}}\mathcal {W}_{a,1}[\eta ]\,. \end{aligned}$$

(5.25)

The non-degenerate forms (i.e. those of rank two) can be divided into two subsets: split and non-split. All the split ones are conjugate under L to . Let $w\in \Gamma$ denote a representative for the Weyl group element given by the simple reflection with respect to the long simple root ${\alpha }_2=2\varepsilon _2$, e.g. $w={\text {diag}}(1,1,1,-1)\sigma _{24}$, where $\sigma _{24}$ is the permutation matrix on indices 2 and 4. Then ${\mathfrak {u}}^w={{\text {Span}}}(e_{12}-e_{43}, e_{13}, e_{42})$, and $\varphi _2^w$ equals the restriction to ${\mathfrak {u}}^w$ of $\chi _{1,0}$. Using Corollary 3.1.2, we can express ${\mathcal {F}}_{{\mathfrak {u}}^w,\chi _{1,0}}$ through ${\mathcal {F}}_{{\mathfrak {u}}',\chi _{1,0}}$, where ${\mathfrak {u}}'={{\text {Span}}}(e_{12}-e_{43}, e_{13}, e_{24})\subset {\mathfrak {n}}$. The integration will be over matrices of the form $v_x:={\text {Id}}+xe_{24}\in G$. Using Fourier expansion by the remaining coordinate of $e_{14}+e_{23}\in {\mathfrak {n}}$, we obtain

$$\begin{aligned} {\mathcal {F}}_{{\mathfrak {u}},\varphi }[\eta ](g)=\int \limits _{\mathbb {A}} \mathcal {W}_{1,a}[\eta ](v_{x}wg) dx. \end{aligned}$$

(5.26)

Finally, let $X\subset \bar{u} \cong {\mathfrak {u}}^*$ denote the set of anisotropic non-degenerate forms. For $\varphi \in X$, we have no expression of ${\mathcal {F}}_{{\mathfrak {u}},\varphi }[\eta ]$ in terms of Whittaker coefficients. However, any $\varphi \in X$ is ${\mathbb K}$-distinguished. Indeed, let $h:={\text {Id}}\in \mathfrak {l}$. Then $(h,\varphi )$ is a neutral pair, and its centralizer is anisotropic. By Lemma 4.4.2 applied to $(h,\varphi )$ and $Z:=0$, $\varphi$ is ${\mathbb K}$-distinguished.

Combining (5.23)–(5.26) we obtain the following theorem, that exemplifies Algorithm A and Remark 4.4.8.

Theorem 5.4.1

For any $\eta \in C^{\infty }(\Gamma \backslash G)$ and $g\in G$, $\eta (g)$ equals

(5.27)

where $O(1,1)\subset L$ denotes the stabilizer of the split form $\varphi _2$.

If $\eta$ is cuspidal then $\mathcal {W}_{0,a}[\eta ]=\mathcal {W}_{a,0}[\eta ]=0$. If $\eta$ is non-generic, then $\mathcal {W}_{1,a}[\eta ]=\mathcal {W}_{a,1}[\eta ]=0$, unless $a=0$. Thus Theorem 5.4.1 implies the following corollary.

Corollary 5.4.2

Let $\eta \in C^{\infty }(\Gamma \backslash G)$ and $g\in G$.

• (i) If $\eta$ is cuspidal then

  

  (5.28)

• (ii) If $\eta$ is non-generic then

  

  (5.29)

• (iii) If $\eta$ is cuspidal and non-generic then $\eta =\sum _{\varphi \in X}{\mathcal {F}}_{{\mathfrak {u}},\varphi }[\eta ]$.

Appendices

Appendix A. On PL-orbits

A complex orbit is a PL-orbit if and only if its Bala–Carter label has no parenthesis. In particular, all complex minimal and next-to-minimal orbits are PL. The classification of PL orbits of complex classical groups in terms of the corresponding partitions is given in [22, Sect. 6].

The classification of rational PL-orbits is a more complicated task. In this subsection we discuss the PL property for small ${\mathbb K}$-rational orbits of simple split groups. A complex orbit ${\mathcal {O}}_{{\mathbb C}}$ may include several or even infinitely many rational orbits. If ${\mathcal {O}}_{{\mathbb C}}$ is non-PL then all its rational orbits are non-PL. If ${\mathcal {O}}_{{\mathbb C}}$ is PL then it includes at least one rational PL-orbit, but can also include non-PL rational orbits. In type $A_n$, all rational orbits are PL. Let us now describe the PL properties of minimal and next-to-minimal orbits. Here, minimal and next-to-minimal refers to the closure order on the complex orbits, which might be coarser than the order defined in Definition 2.4.1.

All minimal rational orbits are PL. Indeed, for classical groups it is easy to establish the Levi in which they are principal: for ${\text {SO}}_{n+1,n}$ it is ${\text {SO}}_{2,1}\times ({\text {GL}}_1)^{n-1}$, for ${\text {Sp}}_{2n}$ it is ${\text {Sp}}_2\times ({\text {GL}}_1)^{n-1}$ and for ${\text {SO}}_{n,n}$ it is ${\text {SO}}_{2,2}\times ({\text {GL}}_1)^{n-2}$. For exceptional groups, the rational minimal orbit is unique and thus PL. This uniqueness was explained to us by Joseph Hundley. Let us now deal with the next-to-minimal orbits.

Lemma A.1

All next-to-minimal rational orbits for ${\text {SO}}_{n,n}$ and ${\text {SO}}_{n+1,n}$ are PL.

Proof

One can give a classification of the rational orbits in the spirit of the classification of real orbits given in [11, Sect. 9.3]. Namely, a ${\mathbb K}$-rational orbit with a given partition is defined by a collection of quadratic forms $Q_{2i+1}$ on multiplicity spaces of the odd parts. If we add a hyperbolic form to the direct sum of these forms we get the initial form, which is also hyperbolic. Here, a hyperbolic form is a direct multiple of the 2-dimensional quadratic form given by $H(x,y)=xy$. By Witt’s cancellation theorem this implies that the direct sum of the forms on multiplicity spaces of the odd parts is hyperbolic.

An orbit for ${\text {SO}}_{n,n}$ is PL if and only if all $Q_{2i+1}$ are hyperbolic, except $Q_{2j+1}$ for a single index $j\ge 1$, which is a direct sum of a hyperbolic form and a one-dimensional quadratic form. For ${\text {SO}}_{n,n}$ there are two next-to-minimal partitions. One of them is $2^41^{2n-8}$. For it, $Q_1$ has to be hyperbolic. The other next-to-minimal partition is $3 1^{2n-3}$. Thus $Q_3$ is one-dimensional. Now, note that $H^n=Q_3\oplus -Q_3 \oplus H^{n-1}$. Thus, $Q_3\oplus Q_1=Q_3\oplus -Q_3 \oplus H^{n-1}$ and thus $Q_1=(-Q_3) \oplus H^{n-1}$, i.e. $Q_1$ is a direct sum of a hyperbolic form and a one-dimensional quadratic form.

Similarly, it is easy to see that the next-to-minimal orbits for ${\text {SO}}_{n+1,n}$ are principal in Levis isomorphic to $({\text {GL}}_2)^2\times ({\text {GL}}_1)^{n-4}$ or ${\text {SO}}_{2,1}\times ({\text {GL}}_1)^{n-1}$. $\square$

However, ${\text {Sp}}_{2n}({\mathbb K})$ has infinitely many rational next-to-minimal orbits, already for $n=2$. Moreover, there exist cuspidal next-to-minimal representations of ${\text {Sp}}_4(\mathbb {A})$. Note that cuspidal non-generic automorphic forms cannot expressed through their Whittaker coefficients, since the latter coefficients have to vanish on such forms. See [16, Sect. 4] for a discussion of cuspidal representations, in particular those of ${\text {Sp}}_4(\mathbb {A})$.

As for the exceptional groups, Joseph Hundley showed that the next-to-minimal orbit is unique, and thus PL, for $E_6,E_7,E_8$ and $G_2$ [27].

We prove the following lemma in AppendixThe group $F_4$ has infinitely many rational next-to-minimal orbits. We expect that infinitely many of them are not PL.

Appendix B. Some geometric lemmas

In this appendix we show some properties of the partial order on rational nilpotent orbits defined in Definition 2.4.1.

Lemma B.1

Let $Z\in {\mathfrak {g}}$ be rational semi-simple, let $\varphi \in {\mathfrak {g}}^Z_0$ and $\varphi '\in {\mathfrak {g}}^Z_{>0}$. Assume that $\varphi$ is conjugate to $\varphi +\varphi '$ by $\mathbf{G}({\mathbb C})$. Then there exist $X\in {\mathfrak {g}}^{Z}_{>0}$ such that ${\text {ad}}^*(X)(\varphi )=\varphi '$ and $v\in {\text {Exp}}({\mathfrak {g}}^{Z}_{>0})$ such that ${\text {Ad}}^*(v)(\varphi )=\varphi +\varphi '$.

Proof

Decompose $\varphi '=\sum _{i=1}^k\varphi '_i$ where $\varphi '_i\in ({\mathfrak {g}}^*)^{Z}_{{\lambda }_i}$ and ${\lambda }_1<{\lambda }_2<\dots <{\lambda }_k\in {\mathbb Q}_{>0}$ are all the positive eigenvalues of Z.

Let us first construct X using the fixed complex embedding $\sigma : \mathbb {K} {\hookrightarrow }\mathbb {C}$ from Sect. 2. For any $t\in {\mathbb R}$, we have the following identity in ${\mathfrak {g}}^{*}({\mathbb C}) := \mathfrak {g}^* \otimes _{\sigma (\mathbb {K})} \mathbb {C}$:

$$\begin{aligned} {\text {Ad}}^*(\exp (tZ))(\varphi +\varphi ')=\varphi +\sum _{i=1}^k {\text {Ad}}^*(\exp (t{\lambda }_i))\varphi '_i\,. \end{aligned}$$

(B.1)

Thus, $\varphi + \sum _i {\text {Ad}}^*(\exp (t{\lambda }_i))\varphi '_i\in G({\mathbb C})\varphi$. Differentiating by t at 0 we obtain that $\sum _i {\lambda }_i\varphi '_i$ lies in the tangent space to the orbit $\mathbf{G}({\mathbb C})\varphi$ at $\varphi$. This tangent space is the image of $\varphi$ under the coadjoint action. Thus there exists $Y_{{\mathbb C}}\in {\mathfrak {g}}({\mathbb C})$ with ${\text {ad}}^*(Y_\mathbb {C})(\varphi )=\sum _i {\lambda }_i\varphi '_i$. Since both $\varphi$ and $\sum _i {\lambda }_i\varphi '_i$ lie in the ${\mathbb K}$-points ${\mathfrak {g}}^*$, there exists $Y\in {\mathfrak {g}}$ satisfying the same linear equation. Decompose $Y=Y'+\sum _i Y_i$ with $Y_i\in {\mathfrak {g}}^Z_{{\lambda }_i}$. Since $\varphi$ commutes with Z, we obtain ${\text {ad}}^*(Y_i)(\varphi )={\lambda }_i\varphi '_i$. Now we take $X:=\sum _i {\lambda }_i^{-1}Y_i \in \mathfrak {g}^Z_{>0}$.

We now prove the existence of v by descending induction on the maximal index i such that $\varphi '\in {\mathfrak {g}}^Z_{>{\lambda }_i}$. The base case $i=k$ has $\varphi '=0$. For the induction step, let $i<k$ such that $\varphi '\in {\mathfrak {g}}^Z_{>{\lambda }_i}$. Then ${\text {Ad}}^*(\exp (-X))(\varphi +\varphi ')=\varphi +\psi$, where $\psi \in {\mathfrak {g}}^Z_{>{\lambda }_{i+1}}$. By the induction hypothesis, $\varphi +\psi \in {\text {Ad}}^*({\text {Exp}}({\mathfrak {g}}^{Z}_{>0}))\varphi$. $\square$

Corollary B.2

Let ${\mathcal {O}}\ne {\mathcal {O}}'$ be two nilpotent $\Gamma$-orbits with $({\mathcal {O}}, {\mathcal {O}}')\in R$ (see Definition 2.4.1), and let ${\mathcal {O}}_{{\mathbb C}},{\mathcal {O}}'_{{\mathbb C}}$ denote their complexifications. Then $\dim {\mathcal {O}}_{{\mathbb C}}<\dim {\mathcal {O}}'_{{\mathbb C}}$.

Proof

By Lemma 2.4.2 we have ${\mathcal {O}}_{{\mathbb C}}\subset \overline{{\mathcal {O}}'_{{\mathbb C}}}$. Thus either $\dim {\mathcal {O}}_{{\mathbb C}}<\dim {\mathcal {O}}'_{{\mathbb C}}$ or ${\mathcal {O}}_{{\mathbb C}}= {\mathcal {O}}'_{{\mathbb C}}$. If ${\mathcal {O}}_{{\mathbb C}}= {\mathcal {O}}'_{{\mathbb C}}$ then, by the definition of R, there exist a rational semi-simple $Z\in {\mathfrak {g}}$, $\varphi \in {\mathcal {O}}\cap {\mathfrak {g}}^Z_0$, and $\psi \in {\mathfrak {g}}^Z_{>0}$ such that $\varphi +\psi \in {\mathcal {O}}_{{\mathbb C}}$, but $\varphi +\psi \notin {\mathcal {O}}$. This contradicts Lemma B.1. $\square$

Corollary B.3

Definition 2.4.1 indeed defines an order relation.

Proof

We have to show that if ${\mathcal {O}}\ge {\mathcal {O}}'$ and ${\mathcal {O}}\le {\mathcal {O}}'$ then ${\mathcal {O}}={\mathcal {O}}'$. By Lemma 2.4.2 the complexifications ${\mathcal {O}}'_{{\mathbb C}}$ and ${\mathcal {O}}_{{\mathbb C}}$ coincide. Moreover, there exist orbits ${\mathcal {O}}^i$ such that ${\mathcal {O}}^1={\mathcal {O}}$, ${\mathcal {O}}^n={\mathcal {O}}'$ and $({\mathcal {O}}^{i+1}, {\mathcal {O}}^i) \in R$ with the same complexification. By Lemma B.1, all these orbits are the same. $\square$

Lemma B.4

Let $Z,S\in {\mathfrak {g}}$ be commuting rational semi-simple elements, let $q\in {\mathbb Q}$ and let $\varphi \in {\mathfrak {g}}^Z_0\cap {\mathfrak {g}}^{S}_{q}$ and $\varphi '\in {\mathfrak {g}}^Z_{>0}\cap {\mathfrak {g}}^{S}_{q}$. Assume that $\varphi$ is conjugate to $\varphi +\varphi '$ by $\mathbf{G}({\mathbb C})$. Then there exist $X\in {\mathfrak {g}}^{Z}_{>0}\cap {\mathfrak {g}}^{S}_{0}$ such that ${\text {ad}}^*(X)(\varphi )=\varphi '$ and $v\in {\text {Exp}}({\mathfrak {g}}^{Z}_{>0}\cap {\mathfrak {g}}^{S}_{0})$ such that ${\text {Ad}}^*(v)(\varphi )=\varphi +\varphi '$.

Proof

To construct X we proceed in the same way as in the proof of Lemma B.1, and then decompose it with respect to eigenspaces of S and take projection on the 0 eigenspace. Then we construct v in the same way as in the proof of Lemma B.1. $\square$

Lemma 4.2.1 follows now from Lemma B.4 (with $q=2$) and Corollary B.2.

We would now like to relate the notion of dominance to dimensions. For any Whittaker pair $(S,\varphi )$ define

$$\begin{aligned} d(S,\varphi ):= (\dim {\mathfrak {n}}_{S,\varphi }+ \dim ({\mathfrak {g}}^S_{\ge 1}))/2 \end{aligned}$$

(B.2)

Example B.5

If $(S,\varphi )$ is a neutral pair then $d(S,\varphi )=(\dim \Gamma \varphi )/2$. If $(S,\varphi )$ is a Levi-distinguished pair corresponding to a nilpotent orbit ${\mathcal {O}}$ in a Levi subalgebra $\mathfrak {l}\subset {\mathfrak {g}}$ then $d(S,\varphi )=(\dim {\mathfrak {g}}- \dim \mathfrak {l}+\dim {\mathcal {O}})/2$. Furthermore, if ${\mathfrak {g}}^{S}_1 = \{0\}$, as in for example a parabolic Fourier coefficient, then $d(S,\varphi ) = \dim {\mathfrak {n}}_{S,\varphi }$.

Lemma B.6

The number $d(S,\varphi )$ equals the dimension of any maximal isotropic subspace of ${\mathfrak {u}}_S:={\mathfrak {g}}^S_{\ge 1}$.

Proof

Any such subspace includes ${\mathfrak {n}}_{S,\varphi }$, and the quotient is Lagrangian in the symplectic space ${\mathfrak {u}}_S/{\mathfrak {n}}_{S,\varphi }$. $\square$

We study Fourier coefficients with respect to maximal isotropic subspaces, so-called Fourier–Jacobi coefficients, in [20].

Lemma B.7

Let $(H,\varphi )$ and $(S,\varphi )$ be Whittaker pairs with the same $\varphi$.

• (i) If $(H,\varphi )$ dominates $(S,\varphi )$ then $d(H,\varphi )\le d(S,\varphi )$.

• (ii) If $(S,\varphi )$ is Levi-distinguished then $\dim {\mathfrak {n}}_{H,\varphi }\le \dim {\mathfrak {n}}_{S,\varphi }$.

Proof

For part (i) let $Z:=S-H$ and choose a Lagrangian subspace $a\subset ({\mathfrak {g}}^H_1\cap {\mathfrak {g}}^Z_{0})/( {\mathfrak {g}}^H_1\cap {\mathfrak {g}}^Z_{0}\cap {\mathfrak {g}}_{\varphi })$. Let $a'$ denote the preimage of a in ${\mathfrak {g}}^H_1\cap {\mathfrak {g}}^Z_{0}$. For any rational $t\in [0,1]$ denote $H_t:=H+tZ,$ and define $\mathfrak {l}_t$ and $\mathfrak {r}_t$ as in (3.16). Define also $\mathfrak {l}_t^\text {max}:=\mathfrak {l}_t+a'$ and $\mathfrak {r}_t^\text {max}:=\mathfrak {r}_t+a'$. Then both $\mathfrak {l}^\text {max}_t$ and $\mathfrak {r}^\text {max}_t$ are maximal isotropic subspaces in ${\mathfrak {u}}_{H_t} := {\mathfrak {g}}^{H_t}_{\ge 1}$ with respect to the anti-symmetric form $\omega _\varphi : {\mathfrak {g}}\times {\mathfrak {g}}\rightarrow \mathbb {K}$ defined by $\omega _\varphi (X, Y) = \varphi ([X,Y])$. Indeed, the symplectic space ${\mathfrak {u}}_{H_t}/{\mathfrak {n}}_{H_t,\varphi }$ is naturally isomorphic to ${\mathfrak {w}}_t/({\mathfrak {w}}_t\cap {\mathfrak {g}}_\varphi )$, where ${\mathfrak {w}}_t:={\mathfrak {g}}^{H_t}_1$. Note that $({\mathfrak {w}}_t)^Z_0={\mathfrak {g}}^H_1\cap {\mathfrak {g}}^Z_{0}$ for all t. Now, ${\mathfrak {w}}_t=({\mathfrak {w}}_t)^{Z}_0\oplus ({\mathfrak {w}}_t)^{Z}_{<0}\oplus ({\mathfrak {w}}_t)^Z_{>0}$, with $({\mathfrak {w}}_t)^Z_{<0}$ and $({\mathfrak {w}}_t)^Z_{>0}$ both isotropic and orthogonal to $({\mathfrak {w}}_t)^Z_{0}$ with respect to $\omega _\varphi$. Let ${\mathfrak {w}}_t^+$ and ${\mathfrak {w}}_t^-$ denote the images of $({\mathfrak {w}}_t)^Z_{<0}$ and $({\mathfrak {w}}_t)^Z_{>0}$ in ${\mathfrak {w}}_t/({\mathfrak {w}}_t\cap {\mathfrak {g}}_\varphi )$.

Since $\mathfrak {l}^\text {max}_t$ projects onto ${\mathfrak {w}}_t^{-} \oplus a$ and $\mathfrak {r}^\text {max}_t$ projects onto ${\mathfrak {w}}_t^{+} \oplus a$, both project to Lagrangian subspaces of ${\mathfrak {u}}_{H_t}/{\mathfrak {n}}_{H_t,\varphi }\cong {\mathfrak {w}}_t/({\mathfrak {w}}_t\cap {\mathfrak {g}}_\varphi ),$ and thus are maximal isotropic. Hence, by Lemma B.6, we have

$$\begin{aligned} d(H_t,\varphi )=\dim \mathfrak {l}_t^\text {max}=\dim \mathfrak {r}_t^\text {max}. \end{aligned}$$

(B.3)

Now let $0=t_0< t_1< \cdots < t_n=1$ be all the critical numbers in the interval [0, 1]. Then by (3.18) for every i we have $\mathfrak {r}_{t_i}\subseteq \mathfrak {l}_{t_{i+1}},$ thus $\mathfrak {r}^\text {max}_{t_i}\subseteq \mathfrak {l}^\text {max}_{t_{i+1}},$ and thus $d(H_{t_i},\varphi )\le d(H_{t_{i+1}},\varphi )$. Since $H_{t_0}=H$ and $H_{t_n}=S$ part (i) follows.

For part (ii), let $Z'$ be as in Notation 4.4.3, and let $S':=H+TZ'$ with T large enough as in Lemma 4.4.5. Then ${\mathfrak {g}}^{S'}_1 \subseteq {\mathfrak {g}}^H_1$ and thus

$$\begin{aligned} d(H,\varphi )-\dim {\mathfrak {n}}_{H,\varphi }\le d(S',\varphi )-\dim {\mathfrak {n}}_{S',\varphi }. \end{aligned}$$

(B.4)

Furthermore, by Lemma 4.4.4, $(H,\varphi )$ dominates $(S',\varphi )$. Part (i) implies now that $\dim {\mathfrak {n}}_{(H,\varphi )}\le \dim {\mathfrak {n}}_{(S',\varphi )}$. Finally, by Lemma 4.4.5 the pair $(S',\varphi )$ is Levi-distinguished, and thus, by Lemma 2.3.10, we have $\dim {\mathfrak {n}}_{S,\varphi }= \dim {\mathfrak {n}}_{S',\varphi }$. $\square$

Remark B.8

In a previous arXiv preprint version of this paper we claimed that if $(H,\varphi )$ dominates $(S,\varphi )$ then $\dim {\mathfrak {n}}_{H,\varphi }\le \dim {\mathfrak {n}}_{S,\varphi }$, and that therefore $\dim {\mathfrak {n}}_{H,\varphi }$ is minimal for a neutral pair $(H, \varphi )$. Unfortunately, these statements are wrong. Indeed, it is possible that two pairs $(H,\varphi )$ and $(S,\varphi )$ dominate each other, with $(H,\varphi )$ neutral, and $\dim {\mathfrak {n}}_{H,\varphi } > \dim {\mathfrak {n}}_{S,\varphi }$.