The generalized doubling method: local theory

Abstract

A fundamental difficulty in the study of automorphic representations, representations of p-adic groups and the Langlands program is to handle the non-generic case. In a recent collaboration with David Ginzburg, we presented a new integral representation for the tensor product L-functions of $G\times {{\,\textrm{GL}\,}}_k$ where G is a classical group, that applies to all cuspidal automorphic representations, generic or otherwise. In this work we develop the local theory of these integrals, define the local $\gamma$-factors and provide a complete description of their properties. We can then define L- and $\epsilon$-factors at all places, and as a consequence obtain the global completed L-function and its functional equation.

Appendices

Faculty of Mathematics and Computer Science, Weizmann Institute of Science, POB 26, Rehovot 76100, Israel; e-mail: dmitry.gourevitch@weizmann.ac.il. Dmitry Gourevitch was supported by the ERC, StG grant number 637912 and by the Israel Science Foundation, grant number 249/17.

Appendix A. Technical results on analytic families of representations (Dmitry Gourevitch)

Faculty of Mathematics and Computer Science, Weizmann Institute of Science, POB 26, Rehovot 76100, Israel; e-mail: dmitry.gourevitch@weizmann.ac.il. Dmitry Gourevitch was supported by the ERC, StG grant number 637912 and by the Israel Science Foundation, grant number 249/17.

Let H be a real reductive group. Fix a maximal compact subgroup $K_H$ of H. Let P be a parabolic subgroup of H, and $M_P$ be its Levi quotient. Let $\rho$ be a (complex) smooth Fréchet representation of $M_P$, of moderate growth. For an algebraic character $\chi$ of $M_P$ and $s\in \mathbb {C}^l$, let $V(s,\rho )$ be the space of the smooth induced representation ${{\,\textrm{Ind}\,}}_{P}^{H}(|\chi |^{s}\rho )$ (l is determined by $M_P$). For example H is a classical group, P is a Siegel parabolic subgroup of H, $M_P$ is isomorphic to ${{\,\textrm{GL}\,}}_r(\mathbb {R})$ or ${{\,\textrm{GL}\,}}_r(\mathbb {C})$, $\rho$ is in addition admissible of finite length, $\chi$ is the determinant character and $s\in \mathbb {C}$.

By virtue of the Iwasawa decomposition, the spaces $V(s,\rho )$ where s varies are all isomorphic as representations of $K_H$ to the smooth induction $V:={{\,\textrm{Ind}\,}}_{{M_P\cap K_H}}^{K_H}(\rho |_{M_P\cap K_H})$.

Let W denote the space of functions from $\mathbb {C}^l$ to V that are holomorphic in the sense that their composition with every continuous functional on V is a holomorphic function. This notion was discussed by Grothendieck [Gro53, § 2]. Since V is a Fréchet space, by [Gro53, § 2, Remarque 1 and footnote 4] a function $f:\mathbb {C}^l \rightarrow V$ is holomorphic if and only if it is continuous, and in addition $\psi \circ f$ is a holomorphic function $\mathbb {C}^l\rightarrow \mathbb {C}$ for every $\psi$ in a separating set $\mathfrak {X}$ of functionals on V. Separating here means that they have no common zeros on V. For example, we can take $\mathfrak {X}$ to be the set of all functionals of the form $v \mapsto \langle w, v(k) \rangle$, where $k\in K_H$ (thus v(k) belongs to the space of $\rho$), and w is a $K_M$-finite vector in the space of the continuous dual representation of $\rho$. Here, $K_M$ is a maximal compact subgroup of M.

Define a topology on W by the system of semi-norms $||f||_D^{\nu }:=\max _{s\in D}\nu (f(s))$, where D runs over all closed balls in $\mathbb {C}^l$, and $\nu$ over all the semi-norms on V. Note that this family of semi-norms defines a Fréchet topology on W. Indeed, the topology stays equivalent if we keep only balls with rational centers and radii, and thus can be given by a countable family of semi-norms. Furthermore, the topology is complete since for any Cauchy sequence $f_n$ and any $s\in \mathbb {C}^l$, the sequence of vectors $f_n(s)$ converges, and the limit f(s) is holomorphic in s by the Cauchy formula, since for every continuous functional $\psi$ on V, the holomorphic functions $\psi (f_n(s))$ converge to $\psi (f(s))$ uniformly on compact sets.

Note that W is naturally a continuous representation of H of moderate growth (see e.g., [Jac09, § 3.3]). Furthermore, W is a smooth representation of H. Indeed, $V(s,\rho )$ is smooth for every s, and for every X in the Lie algebra of H and $f\in W$, the functions $t^{-1}(\exp (tX)f(s)-f(s))$ converge when $t\rightarrow 0$ to the derivative X(f(s)) uniformly on compact sets. The latter follows from the definition of the topology on the smooth induction (see e.g., [Cas89] for this definition).

Let R be a Lie subgroup of H. Let $C^{\infty }(R)$ denote the space of smooth functions on R, and let $C_c^{\infty }(R)$ be the subspace of compactly supported functions. Fix a (non-zero) left-invariant measure dx on R. For any $\phi \in C^{\infty }_c(R)$ and any $f\in W$, define $\phi (f)\in W$ by

$$\begin{aligned} \phi (f)(s)=\int \limits _R x\cdot f(s)\phi (x)\,dx. \end{aligned}$$

Equivalently, we can define $\phi (f)$ using the action of $\phi$ on the representation W, rather than separately on $V(s,\rho )$ for each s. The Dixmier–Malliavin Theorem [DM78] (see also [Cas] for a modern exposition and [Dor] for an extension to bornological spaces) applied to W implies the following statement.

Theorem A.1

For any $f\in W$ there exist $m\in \mathbb {N}$, $\phi _1,\ldots \phi _m\in C_c^{\infty }(R)$ and $f_1,\ldots ,f_m\in W$ such that $f=\sum _{i=1}^m\phi _i(f_i)$, i.e., $f(s)=\sum _{i=1}^m\phi _i(f_i)(s)$ for all s.

Remark A.2

As a rule, even if f does not depend on s, the sections $f_i$ will still depend on s, unless $R<K_H$.

In the discussion above, and in the theorem, one can restrict the domain of the functions to any open subset U of $\mathbb {C}^l$. One can also define meromorphic sections of W as functions f from $U\setminus S$ to V for some discrete set S such that for some holomorphic function $\alpha :U\rightarrow \mathbb {C}$, the product $\alpha f$ extends to an element of W. Multiplying by $\alpha$, Theorem A.1 implies the following corollary.

Corollary A.3

For any meromorphic section $f\in W$ there exist $m\in \mathbb {N}$, $\phi _1,\ldots ,\phi _m\in C_c^{\infty }(R)$ and meromorphic sections $f_1,\ldots ,f_m\in W$ such that for all s for which f(s) is defined, each $f_i(s)$ is also defined and we have $f(s)=\sum _{i=1}^m\phi _i(f_i)(s)$.

Consider $f\in W$ (a holomorphic section), and let $\mathcal {D}\subset \mathbb {C}^l$ be a domain (in the paper $l=1$ and the domains are vertical strips of finite width). We say that f is of finite order in $\mathcal {D}$ if for every continuous functional $\psi$ on V, the holomorphic $\mathbb {C}^l\rightarrow \mathbb {C}$ function $\psi \circ f$ has a finite order in $\mathcal {D}$.

Theorem A.4

For any $f\in W$ there exists a sequence $f_n\in W$ that converges to f, and for every n, $f_n$ is a finite sum of the form $f_n=\sum _{i=1}^{m_n}\vartheta _{n,i}f_{n,i}$ with the following properties:

(1):

Each $f_{n,i}\in W$ is a standard section, in the sense that $f_{n,i}(s)$ is independent of s.

(2):

Each $f_{n,i}$ is $K_H$-finite.

(3):

Each $\vartheta _{n,i}:\mathbb {C}^l\rightarrow \mathbb {C}$ is holomorphic.

(4):

If f is of finite order in $\mathcal {D}$, so are all the functions $\vartheta _{n,i}$.

Proof

According to Bishop [Bis62, Theorem 1], there exists a sequence $p_k$ of continuous mutually annihilating projections on V, whose ranges are one dimensional subspaces of V, such that $f=\sum _kp_k\circ f$. Choosing for each k a nonzero vector $v_k\in V$ in the image of $p_k$, we can write $f=\sum _k\alpha _kv_k$ where each $\alpha _k:\mathbb {C}^l\rightarrow \mathbb {C}$ is holomorphic.

The vectors $v_k$ uniquely define standard sections $h_k$. We then approximate each $h_k$ by a sequence of standard $K_H$-finite vectors $h_k^i$. Since $f=\sum _{k=1}^{\infty }\alpha _k h_k$, and the sequences $h_k^i$ converge to $h_k$ for every k, there exist sequences of indices $k_n$ and $i_n$ such that the sequence $f_n:=\sum _{k=1}^{k_n} \alpha _kh_k^{i_n}$ converges to f.

Finally if f is of finite order (in $\mathcal {D}$), each $p_k \circ f$ is of finite order, then so are the functions $\alpha _k$. $\square$

Appendix B. Proof of Theorem 3.2 (Eyal Kaplan)

We prove the result by adapting the arguments from [GK] to the present setup. We use the notation of § 1.1 and § 3. For brevity and to simplify the comparison to [GK], we put $D=Y_{k,c}$ and $\psi _D=\psi _{k,c}$ (D of loc. cit. is a different subgroup but plays the same role). Let $\rho$ be a (k, c) representation of finite length, not necessarily of the form $\rho _c(\tau )$. We prove $\dim {{\,\textrm{Hom}\,}}_{D}(V(s,\rho ),\psi _D)\le 1$ by analyzing distributions on the orbits of the right action of D on the homogeneous space $P\backslash H$. For $h,h'\in H$, write $h\sim h'$ if $PhD=Ph'D$, otherwise $h\not \sim h'$. Denote $P_h={}^{h^{-1}}P\cap D$. By the Frobenius reciprocity law, the space of distributions on the orbit PhD is given by

$$\begin{aligned} \mathcal {H}(h)={{\,\textrm{Hom}\,}}_{P_{h}}({}^{h^{-1}}(|\det |^{s-1/2}\rho )\otimes \psi _D^{-1}\otimes \Lambda _{h,\nu },\theta _h). \end{aligned}$$

(B.1)

Here $\Lambda _{h,\nu }$ is the trivial one dimensional representation if F is p-adic or $h\sim \delta _0$ ($\delta _0$ was defined in § 2.4), otherwise for each integer $\nu \ge 0$, $\Lambda _{h,\nu }$ is the algebraic dual of the symmetric $\nu$-th power of the normal bundle to PhD, and $\theta _h(x)=\delta _{P_h}(x)\delta _{D}^{-1}(x)\delta _P^{-1/2}({}^hx)$ ($x\in P_h$). We prove $\mathcal {H}(h)=0$ when $h\not \sim \delta _0$, and $\dim \mathcal {H}(\delta _0)=1$. The local analysis on the orbits implies the result: in the non-archimedean case this follows from the theory of Bernstein and Zelevinsky [BZ76] of distributions on l-sheafs, note that the action of D is constructive; in the archimedean case the analysis is far more involved, but now follows transparently from Kolk and Varadarajan [KV96] and Aizenbud and Gourevitch [GK, Appendix], exactly as explained in [GK, § 2.1.3]. Note that for the vanishing arguments we only use the equivariance properties with respect to unipotent subgroups of $P_h$, and for these the representations $\Lambda _{h,\nu }$ can be ignored (see [GK, § 2.1.1]).

Fix $H={{\,\textrm{Sp}\,}}_{2kc}$. At the end of the proof we explain how to adapt it to ${{\,\textrm{SO}\,}}_{2kc}$ and ${{\,\textrm{GSpin}\,}}_{2kc}$ (for ${{\,\textrm{GL}\,}}_{kc}$ the result already follows from [CFGoK, Proposition 2]).

Since $V_{(c^{k})} < imes {U_{P}}=D<P$, we have $P\backslash H/ D=\coprod _hPhD$ with $h=wu$, where w is a representative from $W(M_P)\backslash W(H)$ and $u\in N_H\cap M_{(c^{k})}<M_P$. Identify w with a kc-tuple of 0’s and 1’s, where the i-th coordinate corresponds to

E.g., . Writing $v\in D$ in the form $(v_{i,j})_{1\le i,j\le 2k}$ with $v_{i,j}\in \textrm{Mat}_c$, let $B_i$ be the i-th block $v_{i,i+1}$, $1\le i\le k$, then $B_{k}\in D\cap U_P$. Note that $B_i$ takes arbitrary coordinates in $\textrm{Mat}_c$ for $i<k$, while $B_{k}\in \{X\in \textrm{Mat}_c:J_c({}^tX)J_c=X\}$. Also $\psi _D|_{B_i}=\psi \circ {{\,\textrm{tr}\,}}$ for each i.

   As shown in [GK, § 2.1.2], the condition

$$\begin{aligned}&\psi _D|_{D\cap {}^{h^{-1}}U_P}\ne 1 \end{aligned}$$

(B.2)

implies $\mathcal {H}(h)=0$ (in loc. cit. $\psi _U$ was restricted to $U\cap {}^{h^{-1}}U_P$).

Let $h=wu$. We have the following analog of [GK, Lemma 2.6].

Lemma B.1

Condition (B.2) is implied by

$$\begin{aligned} \psi _D|_{D\cap {}^{w^{-1}}U_P}\ne 1. \end{aligned}$$

(B.3)

Proof

By (B.3), there exists a root in D such that for the subgroup $Y<D$ generated by this root, ${}^wY<U_P$ and $\psi _D|_Y\ne 1$. Since u normalizes D, it remains to show $\psi _D|_{{}^{u^{-1}}Y}\ne 1$. If this root belongs to $B_i$ for $i<k$, it is identified by a diagonal coordinate d of $B_i$, and if $i=k$, by two diagonal coordinates (d, d) and $(c-d+1,c-d+1)$ of $B_i$. In both cases, since $u\in N_H\cap M_{(c^{k})}$, the conjugation by u only changes coordinates above or to the right of these diagonal coordinates, whence $\psi _D|_{{}^{u^{-1}}Y}\ne 1$ (cf. the proof of [GK, Lemma 2.6]). $\square$

Recall the embedding ${{\,\textrm{GL}\,}}_c^{\triangle }$ of ${{\,\textrm{GL}\,}}_c$ in ${{\,\textrm{GL}\,}}_{kc}$, and further embed ${{\,\textrm{GL}\,}}_c^{\triangle }$ in $M_P$ by $g^{\triangle }\mapsto {{\,\textrm{diag}\,}}(g^{\triangle },(g^{\triangle })^*)$. We see that ${{\,\textrm{GL}\,}}_c^{\triangle }$ stabilizes the restriction of $\psi _D$ to $B_1,\ldots ,B_{k-1}$. Since $({}^{g^{\triangle }}\psi _D)|_{B_{k}}(X)=\psi ({{\,\textrm{tr}\,}}(J_c{}^tg^{-1}J_cg^{-1}X))$, the stabilizer of $\psi _D$ in $M_P$ is $\{g^{\triangle }:g\in {{\,\textrm{GL}\,}}_c,{}^tgJ_cg=J_c\}$. In particular, the stabilizer contains $W({{\,\textrm{O}\,}}_c)$ (the Weyl group of ${{\,\textrm{O}\,}}_c$) regarded as a subgroup of permutation matrices. The following result simplifies the structure of w, at the cost of slightly modifying u. See [GK, Propositions 2.7–2.8].

Proposition B.2

We have $\mathcal {H}(h)=0$, unless $h\sim \hat{w}\hat{u}\sigma$ such that for an integer $0\le l\le n$,

$$\begin{aligned} \hat{w}&=(1^n,0^{n-l},1^l,w_2,\ldots ,w_{k}), \quad \forall 1<i\le k, \\ w_i&=(1^{n},0^{n-l-d_{i-1}},1^{l+d_{i-1}}),\,0\le d_{1}\le \ldots \le d_{k-1}\le n-l, \end{aligned}$$

$\sigma =\sigma _0^{\triangle }$ for $\sigma _0\in W({{\,\textrm{O}\,}}_c)$ and ${}^{\sigma ^{-1}}\hat{u}\in N_H\cap M_{(c^{k})}$.

Proof

Put $w=(w_1,\ldots ,w_{k})$ with $w_i\in \{0,1\}^c$ and denote the j-th coordinate of $w_i$ by $w_i[j]$. For $1\le j\le n$, if $w_1[j]=w_1[c-j+1]=0$, (B.3) holds, then by Lemma B.1 (B.2) holds whence $\mathcal {H}(h)=0$. This already describes the first c coordinates of $\hat{w}$ up to a permutation. E.g., l is the number of coordinates with $w_1[j]=w_1[c-j+1]=1$. Assume $w_i[j]=1$ for some $1\le i<k$ and $1\le j\le c$. Hence the j-th column of $B_{k-i}$ is permuted into $U_P$, and if $w_{i+1}[j]=0$, the j-th row of $B_{k-i}$ is not permuted. Thus the (j, j)-th coordinate of $B_{k-i}$ is permuted into $U_P$, and as above (B.3) implies $\mathcal {H}(h)=0$.

Now as in the proof of [GK, Proposition 2.8], we can choose a suitable permutation $\sigma =\sigma _0^{\triangle }$ with $\sigma _0\in W({{\,\textrm{O}\,}}_c)$ such that $\hat{w}={}^{\sigma }w$ satisfies the required properties, then clearly so does $\hat{u}={}^{\sigma }u$, and $h\sim \sigma h=\hat{w}\hat{u}\sigma$. $\square$

Re-denote $w=\hat{w}$ and $u=\hat{u}$ with the properties of the proposition, then $h=wu\sigma$. To compute ${}^hD\cap M_P$ note that ${}^hD={}^wD$. We can further multiply h on the left by elements of $M_P$, to change the blocks $J_a$ appearing in the matrix corresponding to w to blocks $I_a$, then conjugate ${}^hD\cap M_P$ by permutation matrices in $M_P$ to obtain a subgroup of $N_{M_P}$ (see [GK, (2.26)] and the discussion after [GK, Proposition 2.8]).

Example B.3

For $k=2$, we first multiply h on the left by elements in $M_P$ to obtain

then conjugate ${}^hD\cap M_P$ by

We see that ${}^hD\cap M_P=V_{\beta }$ for the composition $\beta$ of kc given by

$$\begin{aligned} \beta =(n-l-d_{k-1},\ldots ,n-l-d_{1},n-l,n+l,n+l+d_{1},\ldots ,n+l+d_{k-1}). \end{aligned}$$

(B.4)

(Cf. [GK, (2.27)].) Denote $\psi _{V_{\beta }}={}^{h}\psi _D|_{V_{\beta }}$. First we describe ${}^{w}\psi _D|_{V_{\beta }}$, then handle $u\sigma$. For

(B.5)

(The sum $\sum _{j=k-1}^{2}$ is omitted if $k\le 2$.)

Proposition B.4

Assume $k>1$ and $l<n$. If $\mathcal {H}(h)\ne 0$, $\psi _{V_{\beta }}$ belongs to the orbit of

$$\begin{aligned}&v\mapsto \psi \left( \sum _{j=k-1}^{2}{{\,\textrm{tr}\,}}\left( b_{k-j}\left( {\begin{matrix}*_{d_j-d_{j-1} \times n-l-d_j} \\ *_{n-l-d_{j}}\end{matrix}}\right) \right) +{{\,\textrm{tr}\,}}\left( b_{k-1}\left( {\begin{matrix}*_{d_1\times n-l-d_1}\\ *_{n-l-d_1} \end{matrix}}\right) \right) +{{\,\textrm{tr}\,}}\left( b_{k}\left( {\begin{matrix}*_{l\times n-l}\\ I_{n-l} \\ *_{l\times n-l}\end{matrix}}\right) \right) \right. \nonumber \\ {}&\left. \quad \quad -{{\,\textrm{tr}\,}}\left( b_{k+1}\left( {\begin{matrix} I_n&{}0_{n\times l} \\ *_{d_1\times n}&{}*_{d_1\times l}\\ *_{l\times n}&{}*_l \end{matrix}}\right) \right) -\sum _{j=2}^{k-1}{{\,\textrm{tr}\,}}\left( b_{k+j}\left( {\begin{matrix} I_{n}&{} 0_{n\times d_{j-1}+l} \\ *_{d_{j}-d_{j-1}\times n} &{} *_{d_{j}-d_{j-1}\times d_{j-1}+l} \\ *_{d_{j-1}+l\times n} &{} *_{d_{j-1}+l}\end{matrix}}\right) \right) \right) . \end{aligned}$$

(B.6)

Here $*$ means undetermined block entries. When $u\sigma$ is the identity element, all coordinates were computed above and (B.6) coincides with (B.5).

Proof

The proof is a simplified version of [GK, Proposition 2.11]. We need some notation. Set $d_0=0$ and $d_{k}=d_{k-1}$. For each $1\le i\le k-1$, write $B_i$ as the upper right block of

$$\begin{aligned} \left( {\begin{matrix} I_{l+d_{k-i-1}}&{} \quad &{} \quad &{} \quad &{} \quad B_i^{1,1}&{} \quad B_i^{1,2}&{} \quad B_i^{1,3}&{} \quad B_i^{1,4}\\ &{} \quad I_{d_{k-i}-d_{k-i-1}}&{} \quad &{} \quad &{} \quad B_i^{2,1}&{} \quad B_i^{2,2}&{} \quad B_i^{2,3}&{} \quad B_i^{2,4}\\ &{} \quad &{} \quad I_{n-l-d_{k-i}}&{} \quad &{} \quad B_i^{3,1}&{} \quad B_i^{3,2}&{} \quad B_i^{3,3}&{} \quad B_i^{3,4}\\ &{} \quad &{} \quad &{} \quad I_{n}&{} \quad B_i^{4,1}&{} \quad B_i^{4,2}&{} \quad B_i^{4,3}&{} \quad B_i^{4,4}\\ &{} \quad &{} \quad &{} \quad &{} \quad I_{l+d_{k-i-1}}\\ &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad I_{d_{k-i}-d_{k-i-1}}\\ &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad I_{n-l-d_{k-i}}\\ &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad I_{n} \end{matrix}}\right) \end{aligned}$$

and $B_{k}$ as the upper right block of

$$\begin{aligned} \left( {\begin{matrix} I_l&{} \quad &{} \quad &{} \quad &{} \quad {B}_{k}^{1,1}&{} \quad {B}_{k}^{1,2}&{} \quad {B}_{k}^{1,3}&{} \quad {B}_{k}^{1,4}\\ &{} \quad I_{n-l}&{} \quad &{} \quad &{} \quad {B}_{k}^{2,1}&{} \quad {B}_{k}^{2,2}&{} \quad {B}_{k}^{2,3}&{} \quad {B}_{k}^{2,4}\\ &{} \quad &{} \quad I_{n-l}&{} \quad &{} \quad {B}_{k}^{3,1}&{} \quad {B}_{k}^{3,2}&{} \quad {B}_{k}^{3,3}&{} \quad {B}_{k}^{3,4}\\ &{} \quad &{} \quad &{} \quad I_l&{} \quad {B}_{k}^{4,1}&{} \quad {B}_{k}^{4,2}&{} \quad {B}_{k}^{4,3}&{} \quad {B}_{k}^{4,4}\\ &{} \quad &{} \quad &{} \quad &{} \quad I_l\\ &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad I_{n-l}\\ &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad I_{n-l}\\ &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad &{} \quad I_{l} \end{matrix}}\right) . \end{aligned}$$

With this notation $\psi _D$ is given by $\psi (\sum _{i=1}^{k}\sum _{j=1}^4{{\,\textrm{tr}\,}}(B_i^{j,j}))$. Denote the lists of blocks $B_i^{t,t'}$ conjugated by w into $M_P$, $U_P$ and $U_P^-$ by $\mathscr {M}_P$, $\mathscr {U}_P$ and $\mathscr {U}_P^-$ (resp.). We have

$$\begin{aligned} \mathscr {M}_P=&\{B_i^{1,1},B_i^{1,4},B_i^{2,1},B_i^{2,4},B_i^{3,2},B_i^{3,3},B_i^{4,1},B_i^{4,4}:1\le i\le k-1\}\\&\coprod \{{B}_{k}^{1,3},{B}_{k}^{2,1},{B}_{k}^{2,2},{B}_{k}^{2,4},{B}_{k}^{3,3},{B}_{k}^{4,3}\},\\ \mathscr {U}_P=&\{B_i^{3,1},B_i^{3,4}:1\le i\le k-1\}\coprod \{{B}_{k}^{2,3}\} \end{aligned}$$

and the remaining blocks belong to $\mathscr {U}_P^-$.

Recall $h=wu\sigma$. Since $\sigma$ fixes $\psi _D$, ${}^h\psi _D={}^{wu}\psi _D$, thus we can already assume $h=wu$ (but u is still given by Proposition B.2). Write $u={{\,\textrm{diag}\,}}(z_{1},\ldots ,z_{k})\in M_{(c^{k})}$ with $z_i={}^{\sigma _0}v_i$ and $v_i\in N_{{{\,\textrm{GL}\,}}_c}$ (recall ${}^{\sigma ^{-1}}u\in N_H\cap M_{(c^{k})}$). We can simplify the form of $z_i$ as follows. If $z_i=z_i'm_i$ such that ${}^w{{\,\textrm{diag}\,}}(z_1',\ldots ,z_{k}',z_{k}'^*,\ldots ,z_{1}'^*)\in M_P$, then because $h\sim ph$ for any $p\in P$, we can already assume $z_i=m_i$. We take for $1\le i\le k$,

$$\begin{aligned}&m_i= \left( {\begin{matrix} I_{l+d_{k-i}}+M_i^1M_i^2&{} \quad M_i^1&{} \quad 0\\ M_i^2&{} \quad I_{n-l-d_{k-i}}+M_i^3M_i^4&{} \quad M_i^3\\ 0&{} \quad M_i^4&{} \quad I_{n} \end{matrix}}\right) \in {{\,\textrm{GL}\,}}_c,\\&I_{l+d_{k-i}}+M_i^1M_i^2\in {{\,\textrm{GL}\,}}_{l+d_{k-i}},\qquad I_{n-l-d_{k-i}}+M_i^3M_i^4\in {{\,\textrm{GL}\,}}_{n-l-d_{k-i}}. \end{aligned}$$

These matrices are invertible because $m_i\in {}^{\sigma _0}N_{{{\,\textrm{GL}\,}}_c}$, and so are the matrices $I_{n-l-d_{k-i}}+M_i^2M_i^1$ (see the proof of [GK, Proposition 2.11]). Then

$$\begin{aligned}&m_i^{-1}= \left( {\begin{matrix} I_{l+d_{k-i}}&{}-M_i^1&{}M_i^1M_i^3\\ -M_i^2&{}I_{n-l-d_{k-i}}+M_i^2M_i^1&{}-(I_{n-l-d_{k-i}}+M_i^2M_i^1)M_i^3\\ M_i^4M_i^2&{}-M_i^4(I_{n-l-d_{k-i}}+M_i^2M_i^1)&{}I_{n} +M_i^4(I_{n-l-d_{k-i}}+M_i^2M_i^1)M_i^3 \end{matrix}}\right) . \end{aligned}$$

Also set for $X\in \textrm{Mat}_{a\times b}$, $X'=-J_b{}^tXJ_a$.

To determine $\psi _{V_{\beta }}$ we compute ${}^u\psi _D$ on the blocks of D conjugated by w into $b_{k},b_{k+1},\ldots ,b_{2k-1}$. First, $b_{k}=\left( {\begin{matrix}B_{k}^{2,1}&B_{k}^{2,2}&B_{k}^{2,4}\end{matrix}}\right)$. To compute ${}^u\psi _D$ on $b_{k}$ we consider $m_{k}^{-1}B_{k}(J_c{}^tm_{k}^{-1}J_c)$. Note that

$$\begin{aligned} J_c{}^tm_{k}^{-1}J_c=\left( {\begin{matrix} I_{n} +(M_{k}^3)'(I_{n-l}+(M_{k}^1)'(M_{k}^2)')(M_{k}^4)' &{} (M_{k}^3)'(I_{n-l}+(M_{k}^1)'(M_{k}^2)') &{} (M_{k}^3)'(M_{k}^1)'\\ (I_{n-l}+(M_{k}^1)'(M_{k}^2)')(M_{k}^4)' &{} I_{n-l}+(M_{k}^1)'(M_{k}^2)' &{}(M_{k}^1)'\\ (M_{k}^2)'(M_{k}^4)' &{} (M_{k}^2)' &{} I_{l} \end{matrix}}\right) . \end{aligned}$$

Since $\psi _D|_{B_{k}}=\psi \circ {{\,\textrm{tr}\,}}$, ${}^u\psi _D|_{B_{k}}=\psi ({{\,\textrm{tr}\,}}(J_c{}^tm_{k}^{-1}J_cm_{k}^{-1}B_{k}))$. The restriction of ${}^u\psi _D$ to $B_{k}^{2,3}$ is given by the product of rows $n+1,\ldots ,c-l$ of $J_c{}^tm_{k}^{-1}J_c$ and columns $l+1,\ldots ,n$ of $m_{k}^{-1}$, and because $B_{k}^{2,3}\in \mathscr {U}_P$, we have

(B.7)

otherwise $\mathcal {H}(h)=0$ by (B.2). Since the restriction of ${}^u\psi _D$ to is given by the product of rows $1,\ldots ,n$ of $J_c{}^tm_{k}^{-1}J_c$ and columns $l+1,\ldots ,n$ of $m_{k}^{-1}$,

(B.8)

where $a=I_{n-l}+ M^2_{k}M^1_{k}\in {{\,\textrm{GL}\,}}_{n-l}$. Set $d_a={{\,\textrm{diag}\,}}(I_{(k-1)c+l},a,I_c,a^*, I_{(k-1)c+l})\in M_P$. Since ${}^wd_a\in M_P$, $h\sim wd_au$ and when we repeat the computation above we obtain , hence ${}^u\psi _D$ belongs to an orbit of a character which agrees with (B.6) on $b_{k}$.

For $1\le i\le k-1$,

(B.9)

To compute ${}^u\psi _D$ on $b_{k+i}$ consider $m_{k-i}^{-1}B_{k-i}m_{k-i+1}$. Since $\psi _D|_{B_{k-i}}=\psi \circ {{\,\textrm{tr}\,}}$,

$$\begin{aligned} {}^u\psi _D|_{B_{k-i}}=\psi ({{\,\textrm{tr}\,}}(m_{k-i+1}m_{k-i}^{-1}B_{k-i})). \end{aligned}$$

This restriction must be trivial on $B_{k-i}^{3,4}\in \mathscr {U}_P$, otherwise $\mathcal {H}(h)=0$ by (B.2). Thus we obtain, if $\mathcal {H}(h)\ne 0$,

Hence

Then the restriction of ${}^u\psi _D$ to $B_{k-i}^{4,4}$, which corresponds to the bottom right $n\times n$ block of $m_{k-i+1}m_{k-i}^{-1}$, is $\psi \circ {{\,\textrm{tr}\,}}=\psi _D|_{B_{k-i}^{4,4}}$. Similarly, because $B_{k-i}^{3,1}\in \mathscr {U}_P$, $\mathcal {H}(h)=0$ unless

Hence

Therefore ${}^u\psi _D$ and $\psi _D$ are both trivial on $B_{k-i}^{4,1}$. It then follows from (B.9) that ${}^u\psi _D$ is given on the blocks which w conjugates into $b_{k+i}$ by

We conclude $\psi _{V_{\beta }}$ belongs to the orbit of (B.6). $\square$

Proposition B.5

If $l<n$, $\mathcal {H}(h)=0$.

Proof

The proof is a simplified version of [GK, Proposition 2.12]. The definitions imply any morphism in $\mathcal {H}(h)$ factors through $J_{V_{\beta },\psi _{V_{\beta }}}(\rho )$ (see [GK, § 2.1.1]). The pair $(V_{\beta },\psi _{V_{\beta }})$ defines a degenerate Whittaker model in the sense of [MW87]. Let $\varphi$ be the transpose of the nilpotent element defined by $\psi _{V_{\beta }}$, which is an upper triangular nilpotent matrix in $\textrm{Mat}_{kc}$. We show $\varphi$ is nilpotent of order at least $k+1$. Since $\rho$ is (k, c), we deduce $J_{V_{\beta },\psi _{V_{\beta }}}(\rho )=0$ by [GGS17, Theorem E] (which over non-archimedean fields is based on [BZ76, 5.9–5.12]).

By Proposition B.4 we can assume $\psi _{V_{\beta }}$ is given by (B.6), then the block $b_i$ of $\varphi$ is the transpose of the block appearing to the right of $b_i$ in (B.6), up to the signs $\pm 1$. Consider the blocks $b_{k},\ldots ,b_{2k-1}$ of $\varphi$: for $i>k$, the (n, n)-th coordinate of $b_{i}$ is nonzero and is the only nonzero coordinate in its column, and the same applies to the $(n-l,n)$-th coordinate of $b_{k}$. These are k coordinates, and it follows that $\varphi$ is nilpotent of order at least $k+1$. $\square$

Remark B.6

The above reasoning in [GK] only implied $d_1=n-l$; we had to use a third method to deduce vanishing (see [GK, Proposition 2.14]), and lose a discrete subset of s.

The remaining case to consider is $l=n$, which means $h\sim \delta _0$. Now since $P_{\delta _0}=V_{(c^{k})}$ and $\psi _D|_{P_{\delta _0}}$ is the (k, c) character (1.1) (see (B.5)),

$$\begin{aligned} \mathcal {H}(\delta _0)={{\,\textrm{Hom}\,}}_{V_{(c^{k})}}({}^{\delta _0^{-1}}\rho \otimes \psi _D^{-1},1)= {{\,\textrm{Hom}\,}}_{V_{(c^{k})}}(\rho \otimes \psi _D,1) ={{\,\textrm{Hom}\,}}_{V_{(c^{k})}}(\rho ,\psi _D^{-1}), \end{aligned}$$

which is one dimensional (but the space in the theorem can still vanish) because $\rho$ is (k, c) and $\psi _D^{-1}$ belongs to the orbit of $\psi _{k}$, $\psi _D^{-1}={}^{\textrm{d}_{k,c}}\psi _{k}$. The proof is complete.

We now explain the case of $H={{\,\textrm{SO}\,}}_{2kc}$. The main difference is that here the restriction of $\psi _D$ to the block $B_k$ is given by $X\mapsto \psi ({{\,\textrm{tr}\,}}({}^tAX))$ ($A$ was defined in § 2.1, now $A\ne I_c$).

Assume momentarily that kc is even. First, for the kc-tuple representing the element w, the sum of coordinates must be even. Lemma B.1 remains valid, but now for the proof if the root belongs to $B_k$ and c is odd, it is determined by a pair of coordinates $(d,d+1)$ and $(c-d,c-d+1)$ where $1\le d\le n$.

The stabilizer of $\psi _D$ in $M_P$ does not contain $W(O_c)$, but ${{\,\textrm{GL}\,}}_c^{\triangle }$ still fixes the restriction of $\psi _D$ to the blocks $B_1,\ldots ,B_{k-1}$. We argue as in Proposition B.2: Using conjugations by elements $\sigma =\sigma _0^{\triangle }$ for $\sigma _0\in {{\,\textrm{diag}\,}}(W(O_{2n}),I_{c-2n})$, we first deduce $\hat{w}=(a_1I_{c-2n},1^n,0^{n-l},1^l,w_2,\ldots ,w_k)$ for some $0\le l\le n$ and $w_i=(a_iI_{c-2n},1^{n},0^{n-l-d_{i-1}},1^{l+d_{i-1}})$ for $i>1$. Here $a_1,\ldots ,a_k\in \{0,1\}$ only appear when c is odd, and $a_1\le \ldots \le a_k$. If c is odd we now conjugate $\hat{w}$ by . Let $o\ge 1$ be minimal such that $a_o=1$, where if $a_k=0$ we set $o=k+1$. Then, for $j<o$ we have $w_{j}=(1^{n},0^{n+1-l-d_{j-1}},1^{l+d_{j-1}})$ ($d_0=0$) and for $j\ge o$, $w_{j}=(1^{n+1},0^{n-l-d_{j-1}},1^{l+d_{j-1}})$.

It follows that in the even case $\beta$ is still given by (B.4). In the odd case the leftmost $k-o+1$ parts of $\beta$ are $(n-l-d_{k-1},\ldots ,n-l-d_{o-1})$, the next $o-1$ parts are $(n+1-l-d_{o-2},\ldots ,n+1-l-d_{0})$, the following $o-1$ parts are $(n+l+d_{0},\ldots ,n+l+d_{o-2})$, and the rightmost $k-o+1$ parts are $(n+1+l+d_{o-1},\ldots ,n+1+l+d_{k-1})$.

Now consider Proposition B.4. Besides minor modifications to the sizes of the parts of $\beta$ in the odd case, the main difference concerns the restriction of (B.6) to $b_k$. This is because for $i\ne k$, $\psi _{V_{\beta }}|_{b_i}$ depends only on $\psi _D|_{B_j}$ for $j<k$ and then ${}^{\sigma }\psi _D|_{B_j}=\psi _D|_{B_j}$. However, $\sigma$ does not fix $\psi _D|_{B_k}$ (which determines $\psi _{V_{\beta }}|_{b_k}$). We can write ${}^{\sigma }\psi _D|_{B_k}(X)=\psi ({{\,\textrm{tr}\,}}(\varrho X))$ for $\varrho \in \textrm{Mat}_c$, $\varrho ={{\,\textrm{diag}\,}}(\varrho _1,\ldots ,\varrho _c)$ where $\varrho _i=\pm 1$ for all i if c is even, and when c is odd $\varrho _i=\pm 1$ for all $i\ne n+1$ and $\varrho _{n+1}=0$. The important observation is that ${}^{\sigma }\psi _D|_{B_k}$ will still be nonzero on n root subgroups. To determine ${}^{u\sigma }\psi _D$ on $b_k$ we multiply the rows of $J_c{}^tm_k^{-1}J_c\varrho$ by columns of $m_k^{-1}$. On the l.h.s. of both (B.7) and (B.8) we “inject" $\varrho$ into the product. The r.h.s. of (B.7) still vanishes because $B_k^{2,3}\in \mathscr {U}_P$ (if $o>1$, $B_k^{2,3}$ is taken to be an $n-l+1\times n-l+1$ block), and the r.h.s. of (B.8) becomes (if $o=1$, $I_n$ here is replaced by $I_{n+1}$). The only change to (B.6) (and in particular, to (B.5)) concerns the block $I_{n-l}$ appearing in the restriction to $b_k$ which is replaced by $\varrho ^{\circ }={{\,\textrm{diag}\,}}(\varrho _{l+1},\ldots ,\varrho _n)$ when c is even, by if $o>1$ and by for $o=1$.

This change does not cause any new complications in the proof of Proposition B.5 and we conclude $l=n$. When c is even this implies $h\sim \delta _0$ and we complete the proof as above. When c is odd the remaining compositions $\beta$ are uniquely determined by o, which varies over the numbers $1,\ldots ,k+1$ such that $k-o+1$ is even. For each such $\beta$, the associated partition is $p_{\beta }=(k+o-1,k^{2n-1},k-o+1)$ and the character $\psi _{V_{\beta }}$ is generic. For $o>1$ the partition $p_{\beta }$ is greater than $(k^c)$, thus $\mathcal {H}(h)=0$ (because $\rho$ is (k, c)). Since we are still considering the case where kc is even, $k-o+1$ is even for $o=1$. Then $h\sim \delta _0$ again, and the result holds. Lastly, when kc is odd we write $w=w'\jmath _1$ with $\det w'=-1$. Since now $D={}^{\jmath _1}V_{(c^{k})} < imes {}^{\jmath _1}{U_{P}}$ (see § 3), the same proof is applicable. In addition, since the proof only involves unipotent subgroups and the properties of (k, c) representations, the case of $H={{\,\textrm{GSpin}\,}}_{2kc}$ is now clear as well.