Criteria for the existence of cuspidal theta representations

Abstract

Theta representations appear globally as the residues of Eisenstein series on covers of groups; their unramified local constituents may be characterized as subquotients of certain principal series. A cuspidal theta representation is one which is equal to the local twisted theta representation at almost all places. Cuspidal theta representations are known to exist but only for covers of $GL_j$, $j\le 3$. In this paper we establish necessary conditions for the existence of cuspidal theta representations on the r-fold metaplectic cover of the general linear group of arbitrary rank.

1 Introduction and main results

Let $r\ge 2$, let F be a number field containing a full set of r-th roots of unity $\mu _r$, and let $\mathbb {A}$ denote the adeles of F. For $n\ge 2$, let $GL_n^{(r)}({\mathbb {A}})$ denote an r-fold cover of the general linear group, as in Kazhdan-Patterson [12]. This group is a cover of $GL_n(\mathbb {A})$ with fibers given by $\mu _r$ and multiplication defined by a certain two-cocycle $\sigma$. The group $GL_n^{(r)}(\mathbb {A})$ is obtained by piecing together local metaplectic groups $GL_n^{(r)}(F_\nu )$ over the places $\nu$ of F [the group $GL_n^{(r)}(F_\nu )$ is, however, not the $F_\nu$-points of an algebraic group]. Following Takeda [17], we shall use the local cocycle given by Banks-Levy-Sepanski [1], which is block-compatible, and adjust it by a coboundary to construct a global cocycle. Kazhdan and Patterson work with a different cocycle than Takeda but the groups are isomorphic. The choice of twisting parameter c in the sense of [12] is arbitrary.

Let $\Theta _n^{(r)}$ denote the theta representation on the group $GL_n^{(r)}({\mathbb {A}})$. This representation was defined in Kazhdan-Patterson [12] using the residues of Eisenstein series, as follows. Let $B_n$ be the standard Borel subgroup of $GL_n$, and $T_n\subseteq B_n$ denote the maximal torus of $GL_n$. Let $\mathbf {s}$ be a multi-complex variable, and define the character $\mu _{{\mathbf s}}$ of $T_n(\mathbb {A})$ by $\mu _{{\mathbf s}}(\text {diag}(a_1,\ldots ,a_n))=\prod _i |a_i|^{s_i}$. If H is an algebraic subgroup of $GL_n$, let $H^{(r)}(F_\nu )$ (resp. $H^{(r)}(\mathbb {A})$) denote the full inverse image of $H(F_\nu )$ (resp. $H(\mathbb {A})$) in $GL_n^{(r)}(F_\nu )$ (resp. $GL_n^{(r)}(\mathbb {A})$). Let $Z(T_n^{(r)}(\mathbb {A}))$ denote the center of $T_n^{(r)}(\mathbb {A})$. Let $\omega _{{\mathbf s}}$ be a genuine character of $Z(T_n^{(r)}(\mathbb {A}))$ such that $\omega _{{\mathbf s}}=\mu _{{\mathbf s}}\circ p$ on $\{(t^r,1)\mid t\in T_n(\mathbb {A})\}$, where p is the canonical projection from $T_n^{(r)}(\mathbb {A})$ to $T_n(\mathbb {A})$. Choose a maximal abelian subgroup A of $T_n^{(r)}(\mathbb {A})$, extend this character to a character of A, and induce it to $T_n^{(r)}(\mathbb {A}).$ Then extend trivially to $B_n^{(r)}(\mathbb {A})$ using the canonical projection from $B_n^{(r)}(\mathbb {A})$ to $T_n^{(r)}(\mathbb {A})$, and further induce it to the group $GL_n^{(r)}({\mathbb {A}})$. We abuse the notation slightly and write this induced representation $\text {Ind}_{B_n^{(r)}(\mathbb {A})}^{GL_n^{(r)}({\mathbb {A}})}\mu _{{\mathbf s}}$. It follows from [12] that this construction is independent of the choice of A and of the extension of $\omega _{{\mathbf s}}$. Forming the Eisenstein series $E(g,{\mathbf s})$ attached to this induced representation, it follows from [12] that when $\mu _{{\mathbf s}}=\delta _{B_n}^{\frac{r+1}{2r}}$ (with $\delta _{B_n}$ the modular function of $B_n$), this Eisenstein series has a nonzero residue representation. This is the representation $\Theta _n^{(r)}$.

Let $\nu$ be a finite place for F such that $|r|_\nu =1$. Defining similar groups over the local field $F_\nu$, it follows from [12] that the local induced representation $\text {Ind}_{B_n^{(r)}(F_\nu )}^{GL_n^{(r)}(F_\nu )}\delta _{B_n}^{\frac{r+1}{2r}}$ has a unique unramified subquotient which we again denote by $\Theta _n^{(r)}$. This representation is also the unique unramified subrepresentation of $\text {Ind}_{B_n^{(r)}(F_\nu )}^{GL_n^{(r)}(F_\nu )}\delta _{B_n}^{\frac{r-1}{2r}}$. If $\chi _\nu$ denotes an unramified character of $F_\nu ^\times$, then the twisted induced representation $\text {Ind}_{B_n^{(r)}(F_\nu )}^{GL_n^{(r)}(F_\nu )}\chi _\nu ^{\frac{1}{r}} \delta _{B_n}^{\frac{r-1}{2r}}$ is also reducible, and one can define the local twisted theta representation $\Theta _{n,\chi _\nu }^{(r)}$ as the unique unramified subrepresentation. Here the twisting means that the induction is from a genuine character of the group $Z\left( T_n^{(r)}(F_\nu )\right)$ such that

$$\begin{aligned} \chi _\nu ^{\frac{1}{r}} \delta _{B_n}^{\frac{r-1}{2r}}((t^r,1))=\chi _\nu (\det t)\, \delta _{B_n}^{\frac{r-1}{2}}(t)\quad \text {for all} \quad t\in T_n(F_\nu ). \end{aligned}$$

The group $\{(t^r,\zeta )\mid t\in T_n(F_\nu ),\zeta \in \mu _r\}$ is in general a proper subgroup of $Z\left( T_n^{(r)}(F_\nu )\right)$, so this condition does not uniquely specify the twisting character $\chi _\nu ^{\frac{1}{r}}$. Since the local calculations below are independent of this choice, we do not indicate it in the notation.

Returning to the global case, we have

Definition 1

An automorphic representation $\pi$ of $GL_n^{(r)}({\mathbb {A}})$ is called a theta representation if for almost all places $\nu$ there are unramified characters $\chi _\nu$ such that the unramified constituent of $\pi$ is equal to $\Theta _{n,\chi _\nu }^{(r)}$. If $\pi$ is cuspidal, we say that $\pi$ is a cuspidal theta representation.

The interesting cases of such theta representations are when the local characters $\chi _\nu$ are the unramified constituents of a global automorphic character $\chi$. We shall write $\Theta _{n,\chi }^{(r)}$ for such a representation.

Examples of such representations may be constructed as follows. Suppose that $\chi =\chi _1^r$ for some global character $\chi _1$. Then one can construct theta representations $\Theta _{n,\chi }^{(r)}$ by means of residues of Eisenstein series, by [12] (the case $\chi =1$ was described above). However, these representations are never cuspidal.

In Flicker [6], a classification of all theta representations for the covering groups $GL_2^{(r)}({\mathbb {A}})$, $r\ge 2$, with $c=0$ in the sense of [12] was given using the trace formula. The case $n=r=2$ was also studied by Gelbart and Piatetski-Shapiro [7]. When $n=r=3$, Patterson and Piatetski-Shapiro [15] constructed a cuspidal theta representation $\Theta _{n,\chi }^{(r)}$ for any $\chi$ which is not of the form $\chi _1^3$, again for the cover with $c=0$. This construction applied the converse theorem. This approach was used when $n=r=4$ by Wang [18], and results were obtained subject to certain technical hypotheses. No other examples of such representations are known.

The basic problem is then to understand for what values of r and n, and for what characters $\chi$, there exists a cuspidal theta representation $\Theta _{n,\chi }^{(r)}$. We shall give a necessary condition for the existence of such a representation. However, we do not determine whether or not these conditions are sufficient.

First, if $r<n$ such cuspidal representations do not exist. This follows trivially since every cuspidal automorphic representation of $GL_n^{(r)}({\mathbb {A}})$ must be generic, but the local unramified representation $\Theta _{n,\chi _\nu }^{(r)}$ is not generic if $r<n$. Hence we may assume that $r\ge n$. Our main result is

Theorem 1

Fix a natural number r, and an automorphic character $\chi$ of $GL_1({\mathbb {A}})$. Then there is at most one natural number n such that there is a nonzero cuspidal theta representation $\Theta _{n,\chi }^{(r)}$. Moreover, if such n exists, then n divides r. If a cuspidal theta representation $\Theta _{n,\chi }^{(r)}$ exists for some n which divides r, $n\ge 3$, then $\chi \ne \chi _1^r$ for any character $\chi _1$.

For $n=2$ and twisting parameter $c=0$, this result follows from [6]. This includes the last assertion: if a cuspidal theta representation $\Theta _{2,\chi }^{(r)}$ exists for some even r, then $\chi \ne \chi _1^r$ for any character $\chi _1$. In [6], a character $\chi$ such that $\chi \ne \chi _1^r$ is called an odd character for the number r.

To establish the Theorem, we need to prove three things. First that n divides r. We prove this in Sect. 3. Second, the uniqueness property of the number n. We prove this in Sect. 4. Then in Sect. 5 we prove the condition on the character $\chi$ when $n\ge 3$. The basic tool in these sections is the study of Eisenstein series obtained by inducing copies of cuspidal theta representations, and their residues. There is a unipotent orbit attached to the automorphic representation generated by these residues, and we determine this. However, if any of the conditions of the Theorem are violated then this leads to a contradiction. For example, if there are two cuspidal theta representations ${\Theta }_{m,\chi }^{(r)}$ and ${\Theta }_{n,\chi }^{(r)}$ attached to the same character $\chi$ with $m<n$, a suitable Eisenstein series obtained by mixing them in the inducing data has a residue. The Fourier coefficient of this residue attached to its unipotent orbit can be analyzed in two different ways, with contradictory vanishing properties. The contradictory properties are due to a lack of symmetry for this representation under the outer automorphism of the Dynkin diagram, which takes the relevant parabolic used in making the induction to its associated parabolic, in terms of the constant terms that it supports.

We will establish Theorem 1 using residues of Eisenstein series, but parts of it follows easily if one accepts Conjecture 1.2, which is a local statement, in Bump and Friedberg [3]. To see this, suppose that for some character $\chi$, there is a cuspidal theta representation $\Theta _{r,\chi }^{(r)}$ defined on the group $GL_r^{(r)}({\mathbb {A}})$. Assume that for some $m<r$, one can define a theta representation $\Theta _{m,\chi }^{(r)}$ which need not be cuspidal, but corresponding to the same character $\chi$. Then, assuming Conjecture 1.2 in [3], the following identity follows from [3], Proposition 2.1

$$\begin{aligned}&\int \limits _{GL_m(F)\backslash GL_m({\mathbb {A}})} \overline{\theta }_{r,\chi }^{(r)}\begin{pmatrix} g&{}\quad \\ &{}\quad I_{r-m}\end{pmatrix}\theta _{m,\chi }^{(r)}(g)\, |\text {det} g|^{s-\frac{r-m}{2}}\,dg\nonumber \\&\quad = Z_S(\chi ,s) L^S\left( rs- \frac{r-1}{2},\chi ^{-1}\otimes \Theta _{m,\chi }^{(r)}\right) . \end{aligned}$$

(1)

Here ${\theta }_{r,\chi }^{(r)}$ is a vector in the space of ${\Theta }_{r,\chi }^{(r)}$, and similarly for ${\theta }_{m,\chi }^{(r)}$. Also, S is a set of places, including the archimedean places, such that outside of S all data is unramified. Finally, $L^S$ is the partial L-function interpreted as in [3], and $Z_S(\chi ,s)$ is a product of local integrals defined on the places in S.

Now from the definition of the partial L-function, it follows that this term contributes a finite product of partial zeta functions to the right-hand-side of (1). Hence for suitable s the term $L^S\left( rs- \frac{r-1}{2},\chi ^{-1}\otimes \Theta _{m,\chi }^{(r)}\right)$ has a simple pole. Since the integrals involved in $Z_S(\chi ,s)$ are all Whittaker type integrals, it is not hard to prove that given any complex number s, there is a choice of data such that $Z_S(\chi ,s)$ is not zero at s. Hence, for suitable s and suitable data, the right hand side of (1) has a simple pole. But the left hand side of (1) is holomorphic for all s since ${\Theta }_{r,\chi }^{(r)}$ is cuspidal. This is a contradiction, and hence for all $m<r$ the group $GL_m^{(r)}({\mathbb {A}})$ has no cuspidal theta representation associated with $\chi$. Moreover, if $\chi =\chi _1^r$ for some character $\chi _1$, then for any $m<r$, we can consider the theta representation ${\Theta }_{m,\chi }^{(r)}$ as constructed in [12]. Since the left-hand side of (1) still represents a holomorphic function even if ${\Theta }_{m,\chi }^{(r)}$ is not cuspidal while the right-hand side has a pole, we once again derive a contradiction. Hence $\chi \ne \chi _1^r$.

We thank Erez Lapid for helpful conversations.

2 Residues of Eisenstein series

Given l natural numbers $n_1\ge n_2\ge \cdots \ge n_l>0$, let ${\Theta }_{n_i,\chi }^{(r)}$ denote theta representations attached to a fixed character $\chi$. Let $k=n_1+\cdots +n_l$, so $\lambda :=(n_1,n_2,\ldots ,n_l)$ is a partition of k. Let $P_{n_1,\ldots ,n_l}$ be the standard parabolic subgroup of $GL_k$ whose Levi part $M_{n_1,\ldots ,n_l}$ is $GL_{n_1}\times \cdots \times GL_{n_l}$ embedded diagonally

$$\begin{aligned} (g_1,g_2,\ldots ,g_l)\mapsto \text {diag}(g_1,g_2,\ldots ,g_l)\quad :\quad g_j\in GL_{n_j}, \end{aligned}$$

and let $U_{n_1,\ldots ,n_l}$ denote the unipotent radical of $P_{n_1,\ldots ,n_l}$.

Let ${\mathbf s}=(s_1,\dots ,s_{l})$ be a multiple complex variable. Then one may form an Eisenstein series $E_{\lambda ,\chi }^{(r)}(g,{\mathbf s})$ on the group $GL_k^{(r)}({\mathbb {A}})$ attached to the representations $\left( {\Theta }_{n_1,\chi }^{(r)},{\Theta }_{n_2,\chi }^{(r)},\ldots ,{\Theta }_{n_l,\chi }^{(r)}\right)$ by a variant of standard parabolic induction. Once one has a representation of $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$ the construction is the standard ‘averaging’ one (see, for example, Mœglin-Waldspurger [13], II.1.5); we frequently suppress the dependence of this series on the test vector used in the averaging from the notation for the Eisenstein series. However, since the inverse images of the groups $GL_{n_i}(\mathbb {A})$ in $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$ do not commute, one must restrict to a smaller subgroup and then induce or extend from that. Let

$$\begin{aligned} GL_{n_j,0}(\mathbb {A})=\left\{ g\in GL_{n_j}(\mathbb {A}) \mid \det g \in (\mathbb {A}^\times )^r F^\times \right\} . \end{aligned}$$

Then the inverse images of these groups in $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$ commute, and the group S that they generate is thus isomorphic to the fibered direct product of the $GL^{(r)}_{n_j,0}(\mathbb {A})$ over $\mu _r$. Accordingly, one first restricts each representation $\Theta _{n_i,\chi }^{(r)}|\text {det}(\cdot )|^{s_i}$ to $GL_{n_i,0}^{(r)}(\mathbb {A})$, and takes the usual tensor product to obtain a genuine representation of S. One may then proceed to extend this representation to $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$, either by extending functions by zero (as in Suzuki [16], Section 8), by inducing to $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$ (as in Brubaker and Friedberg [2], though that paper is written in the language of S-integers as a substitute for the adeles), or by first extending to a larger subgroup of $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$, under certain hypotheses, and then inducing from that subgroup to $M^{(r)}_{n_1,\ldots ,n_l}(\mathbb {A})$ (as in Takeda [17]). Our main computations will take place in the subgroup generated by S and by unipotent subgroups, which split via the trivial section $u\mapsto (u,1)$, hence any of these (slightly different) foundations are sufficient for the arguments given here. We will sometimes abuse the notation (as we already did in the case of induction from the Borel subgroup) and describe $E_{\lambda ,\chi }^{(r)}(g,{\mathbf s})$ as the Eisenstein series attached to the induced representation

$$\begin{aligned} \text {Ind}_{P_{n_1,\ldots ,n_l}^{(r)}(\mathbb {A})}^{GL_k^{(r)}({\mathbb {A}})}\left( {\Theta }_{n_1,\chi }^{(r)}|\text {det}(\cdot )|^{s_1}\otimes {\Theta }_{n_2,\chi }^{(r)}|\text {det}(\cdot )|^{s_2}\otimes \cdots \otimes {\Theta }_{n_l,\chi }^{(r)}|\text {det}(\cdot )|^{s_l}\right) . \end{aligned}$$

(2)

As a representation, this induced space is the vector space spanned by the functions $E_{\lambda ,\chi }^{(r)}(g,{\mathbf s})$ as one varies over all test vectors.

The Eisenstein series $E_{\lambda ,\chi }^{(r)}(g,{\mathbf s})$ has a simple pole, similarly to the case $n_i=1$ for all i which is described in Sect. 1. Indeed, by Definition 1, the unramified constituent at a place $\nu$ of the representation ${\Theta }_{n_i,\chi }^{(r)}$ is a quotient of $\text {Ind}_{B_{n_i}^{(r)}(F_\nu )}^{GL_{n_i}^{(r)}(F_\nu )}\chi _\nu ^{\frac{1}{r}} \delta _{B_{n_i}}^{\frac{r+1}{2r}}$, where $\chi =\prod _\nu \chi _\nu$. This means that the unramified constituent of the induced representation (2) is an induced representation of the form $\text {Ind}_{B_k^{(r)}(F_\nu )}^{GL_k^{(r)}(F_\nu )}\chi _\nu ^{\frac{1}{r}} \mu _{{\mathbf s}}$ where $\mu _{{\mathbf s}}$ is a genuine character of the group $Z\left( T^{(r)}_k(F_\nu )\right)$ defined as follows. Let $t=\text {diag}(A_1,\ldots , A_l)$ where each $A_i$ is a diagonal matrix in $GL_{n_i}(F_\nu )$ which consists of r-th powers. Let $\tilde{t}=(t,\zeta )\in Z\left( T^{(r)}_k(F_\nu )\right)$. Then we define

$$\begin{aligned} \mu _{{\mathbf s}}(\tilde{t})=\zeta \,\delta _{B_{n_1}}^{\frac{r+1}{2r}}(A_1)\ldots \delta _{B_{n_l}}^{\frac{r+1}{2r}}(A_l)\prod _i |A_i|^{s_i}. \end{aligned}$$

Arguing as in [12], one sees that the Eisenstein series $E_{\lambda ,\chi }^{(r)}(g,{\mathbf s})$ has a simple pole at the point $\mu _{{\mathbf s}}= \delta _{B_k}^{\frac{r+1}{2r}}$.

We remark that the existence of this pole does not depend on whether some of the representations ${\Theta }_{n_i,\chi }^{(r)}$ are cuspidal or not. A similar construction with all of the representations being cuspidal was studied by Suzuki [16], Sections 8 and 9. In that reference, the author also assumes that the Shimura lifts of the cuspidal representations in question are also cuspidal. In our case this does not happen, but the argument about the existence of the pole is the same.

Let ${\mathcal L}_{k,\lambda ,\chi }^{(r)}$ denote the residue representation of the above Eisenstein series at the above point. Then the construction of the representation ${\mathcal L}_{k,\lambda ,\chi }^{(r)}$ is inductive in the following sense. For $1\le j\le l$ let $\lambda _j$ be a partition of $n_j$. Form the representations ${\mathcal L}_{n_j,\lambda _j,\chi }^{(r)}$. Then we can form the Eisenstein series attached to the representations $(\eta _{{\mathbf s}}{\mathcal L}_{n_1,\lambda _1,\chi }^{(r)}, \eta _{{\mathbf s}}{\mathcal L}_{n_2,\lambda _2,\chi }^{(r)},\ldots , \eta _{{\mathbf s}}{\mathcal L}_{n_l,\lambda _l,\chi }^{(r)})$ where $\eta _{{\mathbf s}}$ is an unramified character of $P_{n_1,\ldots ,n_l}$. We shall denote this Eisenstein series by $E_{\lambda _1,\ldots ,\lambda _l,\chi }^{(r)}(g,{\mathbf s})$. As in the above, and also as in Sect. 1, we deduce that this Eisenstein series has a simple pole at $\eta _{{\mathbf s}}= \delta _{P_{n_1,\ldots ,n_l}}^{\frac{r+1}{2r}}$, and the representation generated by the residues is ${\mathcal L}_{k,\lambda ,\chi }^{(r)}$.

3 The divisibility condition

Suppose that ${\Theta }_{n,\chi }^{(r)}$ is a cuspidal theta representation defined on $GL_n^{(r)}({\mathbb {A}})$ and that n does not divide r. We shall derive a contradiction. First, we construct the residue representation ${\mathcal L}_{nl,\lambda ,\chi }^{(r)}$ on $GL_{nl}^{(r)}({\mathbb {A}})$ where l is a natural number and $\lambda =(n^l)$. For convenience we sometimes omit $\lambda$ from the notation, writing ${\mathcal L}_{nl,\chi }^{(r)}$ instead of ${\mathcal L}_{nl,\lambda ,\chi }^{(r)}$. Thus ${\mathcal L}_{n,\chi }^{(r)}={\Theta }_{n,\chi }^{(r)}$.

In general, if $\varphi$ is an automorphic function on a group $H({\mathbb {A}})$ and U is any unipotent subgroup of H, we write $\varphi ^U$ for the constant term of $\varphi$ along U

$$\begin{aligned} \varphi ^U(h)=\int \limits _{U(F)\backslash U({\mathbb {A}})}\varphi (uh)\,du. \end{aligned}$$

Also, if $\psi _U$ is a character of $U(F)\backslash U({\mathbb {A}})$, we write

$$\begin{aligned} \varphi ^{U,\psi _U}(h)=\int \limits _{U(F)\backslash U({\mathbb {A}})}\varphi (uh)\,\psi _U(u)\,du. \end{aligned}$$

We shall be concerned with the case that $U=U_{(l-m)n, \,mn}$, the unipotent radical of the maximal parabolic subgroup of $GL_{nl}$ whose Levi part is $GL_{(l-m)n}\times GL_{mn}$, with $1\le m<l$.

We start with the following

Proposition 1

Fix m, $1\le m<l$, let $P=P_{(l-m)n, \,mn}$ and $U=U_{(l-m)n, \,mn}$.

• (i) Let $\varphi _{nl,\chi }^{(r)}$ be a function in the space of ${\mathcal L}_{nl,\chi }^{(r)}$. Then there are functions $\varphi _{(l-m)n,\chi }^{(r)}$ in the space of ${\mathcal L}_{(l-m)n,\chi }^{(r)}$ and $\varphi _{mn,\chi }^{(r)}$ in the space of ${\mathcal L}_{mn,\chi }^{(r)}$ such that

  $$\begin{aligned} {(\varphi _{nl,\chi }^{(r)}})^U(t(v_1,v_2))=\delta _{P}^{\frac{r-1}{2r}}(t) \varphi _{(l-m)n,\chi }^{(r)}(v_1)\,\varphi _{mn,\chi }^{(r)}(v_2) \end{aligned}$$

  (3)

  for all unipotent elements $v_1\in GL_{(l-m)n}(\mathbb {A})$ and $v_2\in GL_{mn}(\mathbb {A})$ and all t which are r-th powers and in the center of the Levi subgroup of P.

• (ii)  Let $V_1$ (resp. $V_2)$ be the group of upper triangular unipotent elements in $GL_{(l-m)n}(\mathbb {A})$ (resp. $GL_{mn}(\mathbb {A})$), and for $i=1,2$, let $\psi _{V_i}$ be characters of $V_i(F)\backslash V_i({\mathbb {A}})$. Then the integral

  $$\begin{aligned} \int \limits _{V_1(F)\backslash V_1({\mathbb {A}})}\int \limits _{V_2(F)\backslash V_2({\mathbb {A}})}{(\varphi _{nl,\chi }^{(r)}})^U((v_1,v_2))\,\psi _{V_1}(v_1)\,\psi _{V_2}(v_2)\,dv_1\,dv_2 \end{aligned}$$

  is zero for all $\varphi _{nl,\chi }^{(r)}$ in the space of ${\mathcal L}_{nl,\chi }^{(r)}$ if $(\varphi _{(l-m)n,\chi }^{(r)})^{V_1,\psi _{V_1}}$ or $(\varphi _{mn,\chi }^{(r)})^{V_2,\psi _{V_2}}$ is zero for all functions $\varphi _{(l-m)n,\chi }^{(r)}$ in the space of ${\mathcal L}_{(l-m)n,\chi }^{(r)}$ or all $\varphi _{mn,\chi }^{(r)}$ in the space of ${\mathcal L}_{mn,\chi }^{(r)}$.

When $r=1$ and ${\Theta }_{n,\chi }^{(r)}$ is a cuspidal representation of $GL_n({\mathbb {A}})$, a similar statement is given in Offen-Sayag [14], Lemma 2.4 and Jiang-Liu [11], Lemma 4.2.

Proof

The proof is based on a standard argument using unfolding of the Eisenstein series, and closely follows [11, 14], and [13] II.1.7. We sketch it briefly. Let $E_{nl,\chi }^{(r)}(g,s)$ denote the Eisenstein series attached to the induced representation

$$\begin{aligned} \text {Ind}_{P_{mn,(l-m)n}^{(r)}({\mathbb {A}})}^{GL_{nl}^{(r)}({\mathbb {A}})}\left( {\mathcal L}_{mn,\chi }^{(r)} \otimes {\mathcal L}_{(l-m)n,\chi }^{(r)}\right) \delta _{P_{mn,(l-m)n}}^s. \end{aligned}$$

Then, as explained in Sect. 2 above, ${\mathcal L}_{nl,\chi }^{(r)}$ is the residue of this Eisenstein series at $s=\frac{r+1}{2r}$. Consider the constant term $E_{nl,\chi }^{(r),U}(g,s)$ for $\text {Re}(s)$ large. Unfolding this constant term as in [4, 11, 13, 14], we obtain a sum of Eisenstein series (and degenerate Eisenstein series), where the sum is over Weyl elements that give a complete set of representatives for the double cosets $P_{mn,(l-m)n}(F)\backslash GL_{nl}(F)/P_{(l-m)n,mn}(F)$ [(see for example Bump–Friedberg–Ginzburg [4], Eq. (1.2)]. Let

$$\begin{aligned} w_0= \begin{pmatrix} &{}\quad I_{mn}\\ I_{(l-m)n}&{}\quad \end{pmatrix}. \end{aligned}$$

Then as in the references above, for every Weyl element not equal to $w_0$ which contributes a nonzero term, the corresponding Eisenstein series is holomorphic at $s=\frac{r+1}{2r}$. The contribution from $w_0$ is just the intertwining operator $M_{w_0,s}$ which clearly has a simple pole at $s=\frac{r+1}{2r}$, and as a function of $(v_1,v_2)$ is as in (3).

The claim about the dependence of ${(\varphi _{nl,\chi }^{(r)}})^U(t(v_1,v_2))$ on t follows since ${\mathcal L}_{nl,\chi }^{(r)}$ is a sub-representation of the induced representation

$$\begin{aligned} \text {Ind}_{P_{mn,(l-m)n}^{(r)}({\mathbb {A}})}^{GL_{nl}^{(r)}({\mathbb {A}})}\left( {\mathcal L}_{mn,\chi }^{(r)} \otimes {\mathcal L}_{(l-m)n,\chi }^{(r)}\right) \delta _{P_{mn,(l-m)n}}^s \end{aligned}$$

at the point $s=\frac{r-1}{2r}$. $\square$

We next give an application of Proposition 1.

Lemma 1

The representation ${\mathcal L}_{nl,\chi }^{(r)}$ is square integrable.

Proof

We use Jacquet’s criterion as stated in [13], the Lemma in I.4.11. Note that ${\mathcal L}_{nl,\chi }^{(r)}$ consists of automorphic forms so the Lemma there is applicable. Let U denote a unipotent radical of a maximal parabolic subgroup P of $GL_{nl}$. Let $U_{n,\ldots , n}^-=\widetilde{w}U_{n,\ldots , n}\widetilde{w}^{-1}$. Here $\widetilde{w}$ is the longest Weyl element in $GL_{nl}$.

Suppose first that U is such that there is no Weyl element w of $GL_k$ such that $wUw^{-1}\subset U_{n,\ldots , n}^-$. Then a standard unfolding argument implies that the constant term

$$\begin{aligned} \int \limits _{U(F)\backslash U({\mathbb {A}})}E_{nl,\chi }^{(r)}(ug,{\mathbf s})\,du \end{aligned}$$

is zero for all choices of data.

On the other hand, if $U=U_{(l-m)n,mn}$ for some m, then it follows from Proposition 1, part (i), that for all t in the center of $P=P_{(l-m)n, \,mn}$ we obtain the exponent $\delta _P^{\frac{r-1}{2r}}=\delta _P^{\frac{-1}{2r}}\delta _B^{\frac{1}{2}}$. Here B is the Borel subgroup of $GL_{ln}$. Lemma 1 follows. $\square$

The above Proposition and Lemma can be extended to the general case. That is, both statements hold for the representation ${\mathcal L}_{k,\lambda ,\chi }^{(r)}$ as well.

Since we are in the case $r\ge n$, the representation ${\mathcal L}_{n,\chi }^{(r)}={\Theta }_{n,\chi }^{(r)}$ is clearly generic. On the other hand if l is chosen so that $nl>r$, then ${\mathcal L}_{nl,\chi }^{(r)}$ is not generic. Hence, there is a minimal natural number, which we denote by a, such that ${\mathcal L}_{an,\chi }^{(r)}$ is generic, but ${\mathcal L}_{(a+1)n,\chi }^{(r)}$ is not. Notice that since n does not divide r then $an<r$. Let b be the smallest natural number so that $abn>r$.

For the proof of the next Proposition we need to modify our construction. Let ${\mathcal E}_{an,\chi }^{(r)}$ denote an irreducible generic summand of the representation ${\mathcal L}_{an,\chi }^{(r)}$. The existence of such a summand follows from Lemma 1 and from the assumption that ${\mathcal L}_{an,\chi }^{(r)}$ is generic. Forming the Eisenstein series on $GL_{nl}^{(r)}({\mathbb {A}})$ attached to $({\mathcal E}_{an,\chi }^{(r)},{\mathcal E}_{an,\chi }^{(r)},\ldots ,{\mathcal E}_{an,\chi }^{(r)})\eta _{{\mathbf s}}$ then it follows as in the previous section that this series has a simple pole at the point $\eta _{{\mathbf s}}=\delta _{P_{n,\ldots ,n}}^{\frac{r+1}{2r}}$. Denote the residue representation by ${\mathcal E}_{abn,\chi }^{(r)}$. It is clear from this construction that Proposition 1 holds if we replace the representation ${\mathcal L}_{ln,\chi }^{(r)}$ with the representation ${\mathcal E}_{ln,\chi }^{(r)}$.

Given an automorphic representation $\pi$ defined on a reductive group $H({\mathbb {A}})$, let ${\mathcal O}(\pi )$ be its set of unipotent orbits as defined in Ginzburg [9] (For information about unipotent orbits see Collingwood and McGovern [5]). A unipotent orbit ${\mathcal O}$ is in the set ${\mathcal O}(\pi )$ if first, $\pi$ has no nonzero Fourier coefficients attached to any unipotent orbit which is greater than ${\mathcal O}$ and second, $\pi$ has a nonzero Fourier coefficient corresponding to the unipotent orbit ${\mathcal O}$. Since unipotent groups split in any covering group, this definition extends without change to representations of metaplectic groups. Moreover, for the general linear group the unipotent orbits are parametrized by partitions, a manifestation of the Jordan decomposition. If $\mathcal {O}(\pi )$ consists of a single unipotent orbit parametrized by a partition $\lambda$ we write $\mathcal {O}(\pi )=\lambda$. In our case, we have:

Proposition 2

If $n\not \mid r$, then ${\mathcal O}\left( {\mathcal E}_{abn,\chi }^{(r)}\right) =\left( (an)^b\right)$.

Proof

This proof is similar to Jiang and Liu [11]; see also the proof of Proposition 5.3 in Ginzburg [9].

We need to prove two things. First, let ${\mathcal O}=(n_1n_2\ldots n_r)$ be a partition of abn. Assume that this partition is greater than or is not related to the partition $\left( (an)^b\right)$. Then we need to prove that any Fourier coefficient of ${\mathcal E}_{abn,\chi }^{(r)}$ associated with this partition is zero. As explained in [9] at the beginning of the proof of Proposition 5.3, it is enough to prove that the functions in ${\mathcal E}_{abn,\chi }^{(r)}$ have no nonzero Fourier coefficients associated with the partitions $(m1^{abn-m})$ for all $m>an$.

The proof of this statement about the Fourier coefficients is similar to [9, 11]. In [9] this was proved by local means, and this was replaced in [11] by a version of Proposition 1. To indicate the approach, suppose that m is even. The case that m is odd is similar and will be omitted. If m is even, then the Fourier coefficient associated with the unipotent orbit $(m1^{abn-m})$ is given as follows. Let $P_m$ denote the parabolic subgroup of $GL_{abn}$ whose Levi part is $GL_1^m\times GL_{abn-m}$. We embed the Levi part in $GL_{abn}$ as all matrices of the form $\text {diag}(a_1,\ldots ,a_{m/2},h,b_1,\ldots ,b_{m/2})$, with $a_i, b_i\in GL_1$, $h\in GL_{abn-m}$. Let $V_m^0$ denote the unipotent radical of $P_m$, and $V_m$ denote the subgroup of $V_m^0$ which consists of all matrices $v=(v_{i,j})$ such that $v_{i,abn-\frac{m}{2}+1}=0$ for all $\frac{m}{2}+1\le i\le abn-\frac{m}{2}$. Let $\psi _{V_m}$ denote the character of $V_m$ defined as follows. For $v=(v_{i,j})\in V_m$ set

$$\begin{aligned} \psi _{V_m}(v)=\psi \left( v_{\frac{m}{2},abn-\frac{m}{2}+1}+ \sum _{i=1}^{m/2 -1}\left( v_{i,i+1}+v_{abn-\frac{m}{2}+i,abn-\frac{m}{2}+i+1}\right) \right) . \end{aligned}$$

Then, the Fourier coefficient corresponding to the partition $(m1^{abn-m})$ is given by

$$\begin{aligned} \int \limits _{V_m(F)\backslash V_m({\mathbb {A}})}E_{abn,\chi }^{(r)}(v)\,\psi _{V_m}(v)\,dv. \end{aligned}$$

Let $w_1$ denote the Weyl element of $GL_{abn}$ defined by

$$\begin{aligned} w_1= \begin{pmatrix} I_{\frac{m}{2}}&{}\quad &{}\quad \\ &{}\quad &{}\quad I_{\frac{m}{2}}\\ &{}\quad I_{abn-m}&{}\quad \end{pmatrix}. \end{aligned}$$

Conjugating by $w_1$ and performing some Fourier expansions, one deduces that the vanishing of the above integral for all choices of data is equivalent to the vanishing of

$$\begin{aligned} \int \limits _{U_m(F)\backslash U_m({\mathbb {A}})}E_{abn,\chi }^{(r)}(u)\,\psi _{U_m}(u)\,du \end{aligned}$$

(4)

for all choices of data. Here $U_m$ is the unipotent radical of the standard parabolic subgroup of $GL_{abn}$ whose Levi part is $GL_1^{m-1}\times GL_{abn-m+1}$, with the Levi part embedded in $GL_{abn}$ as $\text {diag}(a_1,\ldots , a_{m-1},h)$ ($a_i\in GL_1$, $1\le i\le m-1$, and $h\in GL_{abn-m+1}$), and $\psi _{U_m}$ is the character

$$\begin{aligned} \psi _{U_m}(u)=\psi (u_{1,2}+u_{2,3}+\cdots +u_{m-1,m}). \end{aligned}$$

Note that when $m=abn$, the group $U_{abn}$ is the maximal upper triangular unipotent subgroup of $GL_{abn}$.

Let $\underline{\alpha }=(\alpha _i)_{m\le i\le abn-1}$ with $\alpha _i\in \{0,1\}$ for all i and define

$$\begin{aligned} \psi _{U_m,\underline{\alpha }}(u)=\psi \left( \sum _{i=1}^{m-1} u_{i,i+1}+ \sum _{i=m}^{abn-1} \alpha _{i}u_{i,i+1}\right) . \end{aligned}$$

Then performing Fourier expansions, one sees that the vanishing of the integral (4) is equivalent to the vanishing of all the integrals

$$\begin{aligned} \int \limits _{U_{abn}(F)\backslash U_{abn}({\mathbb {A}})}E_{abn,\chi }^{(r)}(u)\, \psi _{U_m,\underline{\alpha }}(u)\,du. \end{aligned}$$

(5)

If $\alpha _i=1$ for all i, then the integral (5) is the Whittaker coefficient of $E_{abn,\chi }^{(r)}$ which is zero. If instead $\alpha _i=0$ for some i, let $k\ge m$ be the first integer such that $\alpha _i=1$ for all $m\le i\le k$ and $\alpha _{k+1}=0$. If $k\ne np$ for some natural number p, then the corresponding integral (5) is zero. Indeed, since ${\Theta }_{n,\chi }^{(r)}$ is a cuspidal representation, the constant term $E_{abn,\chi }^{(r),U}(g,s)$ is zero if U is not equal to $U_{(ab-l)n,ln}$ for some l. (Note that it is precisely at this point in the argument that we use the cuspidality hypothesis.) On the other hand, if $k=np$, then it follows from Proposition 1 that integral (5) is zero if the residue representation ${\mathcal E}_{np,\chi }^{(r)}$ is not generic. But since $np=k\ge m>an$, it follows from the definition of a that ${\mathcal E}_{np,\chi }^{(r)}$ is indeed not generic. This completes the proof that ${\mathcal E}_{abn,\chi }^{(r)}$ has no nonzero Fourier coefficient corresponding to any unipotent orbit which greater than or not related to $\left( (an)^b\right)$.

The last step is to prove that ${\mathcal E}_{abn,\chi }^{(r)}$ has a nonzero Fourier coefficient corresponding to the partition $\left( (an)^b\right)$. This is proved similarly to [9] pp. 338–339; see also [11] and [14]. Let $E_{abn,\chi }^{(r)}$ be a vector in the space of ${\mathcal E}_{abn,\chi }^{(r)}$. Then it follows from [9], p. 338, that the Fourier coefficient of $E_{abn,\chi }^{(r)}$ with respect to the orbit $\left( (an)^b\right)$ is given by the integral

$$\begin{aligned} f(h)=\int \limits _{V(F)\backslash V({\mathbb {A}})}E_{abn,\chi }^{(r)}(vh)\,\psi _V(v)\,dv. \end{aligned}$$

(6)

Here we let the $V_{k,p}$ be the unipotent subgroup of $GL_{kp}$ consisting of all matrices of the from

$$\begin{aligned} \begin{pmatrix} I_{k}&{}\quad X_{1,2}&{}\quad *&{}\quad *&{}\quad \cdots &{}\quad *\\ &{}\quad I_{k}&{}\quad X_{2,3}&{}\quad *&{}\quad \cdots &{}\quad *\\ &{}\quad &{}\quad I_{k}&{}\quad X_{3,4}&{}\quad \cdots &{}\quad *\\ &{}\quad &{}\quad &{}\quad I_{k}&{}\quad \cdots &{}\quad *\\ &{}\quad &{}\quad &{}\quad &{}\quad \ddots &{}\quad *\\ &{}\quad &{}\quad &{}\quad &{}\quad &{}\quad I_{k} \end{pmatrix} \end{aligned}$$

(7)

with $I_k$ appearing p times and each $X_{i,j}$ a matrix of size k. The group V in the integral (6) is the group $V_{k,p}$ with $k=b$ and $p=an$. Also, define a character $\psi _{V_{k,p}}$ on $V_{k,p}$ by $\psi _{V_{k,p}}(v)=\psi (\text {tr} (X_{1,2}+X_{2,3}+\cdots +X_{p-1,p}))$. Then the character $\psi _V$ in (6) is $\psi _{V_{b,an}}$.

Let $U_{an}$ denote the maximal upper unipotent subgroup of $GL_{an}$, and let $U'=U_{an}\times \cdots \times U_{an}$ where the group $U_{an}$ appears b times. This group is embedded in $GL_{anb}$ as $(u_1,\ldots ,u_b)\mapsto \text {diag}(u_1,\ldots ,u_b)$. Let $\psi _{U'}$ be the character given by

$$\begin{aligned} \psi _{U'}(u')=\psi _{U_{an}}(u_1)\ldots \psi _{U_{an}}(u_b), \end{aligned}$$

where $\psi _{U_{an}}$ is the standard Whittaker character of $U_{an}$. Then as in [9] p. 338, the integral (6) is nonzero for some choice of data if and only if the integral

$$\begin{aligned} \int \limits _{U'(F)\backslash U'({\mathbb {A}})}\int \limits _{V_{b,an}(F)\backslash V_{b,an}({\mathbb {A}})} E_{abn,\chi }^{(r)}(vu')\,\psi _{U'}(u')\,dv\,du' \end{aligned}$$

(8)

is not zero for some choice of data. Using (3) inductively and the irreducibility of the representation ${\mathcal E}_{an,\chi }^{(r)}$, we deduce that the integral (8) is not zero for some choice of data if the representation ${\mathcal E}_{an,\chi }^{(r)}$ is generic. This last assertion follows from our assumption on the number a. $\square$

We can now prove the first part of Theorem 1.

Proposition 3

Let $n\le r$ be a natural number, and suppose there exists a cuspidal theta representation ${\Theta }_{n,\chi }^{(r)}$ on $GL_n^{(r)}({\mathbb {A}})$. Then n divides r.

Proof

Suppose instead that n does not divide r. Construct the representation ${\mathcal E}_{abn,\chi }^{(r)}$ on the group $GL_{abn}^{(r)}({\mathbb {A}})$ as above. It follows from Proposition 2 that ${\mathcal O}\left( {\mathcal E}_{abn,\chi }^{(r)}\right) \,=\,\left( (an)^b\right)$. Let $E_{abn,\chi }^{(r)}$ be a vector in the space of ${\mathcal E}_{abn,\chi }^{(r)}$. Then the Fourier coefficient of $E_{abn,\chi }^{(r)}$ with respect to the orbit $\left( (an)^b\right)$ is given by the integral (6) above.

Since $an<r$ and $b\ge 2$, the Fourier coefficient (6) defines a genuine automorphic function on some covering group of $GL_b({\mathbb {A}})$ of degree greater than one. Hence f(h) cannot be the constant function. Note that at this step we are using the hypothesis that n does not divide r. Indeed, this assumption implies that $an\ne r$. By contrast, if $an=r$, then it might happen that the above embedding of the group $GL_b$ splits under the r-fold cover, and we would not be able to assert that f(h) is not constant.

Let $\sigma$ denote the representation generated by all functions f(h) as above. Since a non-constant automorphic function cannot equal a constant term along any unipotent subgroup, it follows that the integral

$$\begin{aligned} \int \limits _{F\backslash {\mathbb {A}}}f(x(l))\,\psi (l)\,dl\qquad \text {where~}\, x(l)=I_b+le_{1,b} \end{aligned}$$

(9)

is not zero for some function f in $\sigma$ (here and below $e_{i,j}$ denotes the (i, j)th elementary matrix). Using this nonvanishing, we will show that the representation ${\mathcal E}_{abn,\chi }^{(r)}$ has a nonzero Fourier coefficient corresponding to the unipotent orbit $((an+1)(an)^{b-2}(an-1))$.

To do so, we introduce two families of unipotent subgroups of $GL_{abn}$. First, let $Z_i$, $1\le i\le an-1$, denote the unipotent subgroup with

$$\begin{aligned} Z_i(\mathbb {A})=\left\{ r_1e_{b,1}+r_2e_{b,2}+\cdots +r_{b-1}e_{b,b-1}:\ \ r_j\in {\mathbb {A}}\right\} \subset X_{i,i+1} \end{aligned}$$

and let $Z_0$ denote the group with

$$\begin{aligned} Z_0(\mathbb {A})=\left\{ r_2e_{2,1}+r_3e_{3,1}+\cdots +r_{b-1}e_{b-1,1}:\ \ r_j\in {\mathbb {A}}\right\} \subset X_{1,2}. \end{aligned}$$

Here each $X_{i,i+1}$ is embedded in $GL_{abn}$ as in (7). Notice that $Z_0$ and $Z_1$ are two distinct subgroups of $X_{1,2}$. Second, for $1\le i\le an-1$ let

$$\begin{aligned} Y_i(\mathbb {A})=\left\{ I_{b}+l_1e_{1,b}+l_2e_{2,b}+\cdots + l_{b-1}e_{b-1,b} :\ \ l_j\in {\mathbb {A}}\right\} \end{aligned}$$

and let

$$\begin{aligned} Y_0(\mathbb {A})=\left\{ I_{b}+l_2e_{1,2}+l_3e_{1,3}+\cdots + l_{b-1}e_{1,b-1} :\ \ l_j\in {\mathbb {A}}\right\} . \end{aligned}$$

These groups are embedded in $GL_{abn}$ as $\text {diag}(Y_0,Y_1,\ldots ,Y_{an-1})$. Also, let Z be the unipotent subgroup of $GL_{abn}$ generated by all $Z_i$ with $0\le i\le an-1$, and let Y be the unipotent subgroup of $GL_{abn}$ generated by the $Y_i$, $0\le i\le an-1$, together with the one dimensional unipotent subgroup x(l) defined in (9).

Substituting (9) into (6) we then expand the integral along the unipotent subgroups $Y_i$ where $0\le i\le an-1$. Then using the unipotent subgroups $X_i$, we obtain that the integral (9) is equal to

$$\begin{aligned} \int \limits _{Z({\mathbb {A}})}\int \limits _{V_1(F)\backslash V_1({\mathbb {A}})}E_{abn,\chi }^{(r)}(v_1zh)\,\psi _{V_1}(v_1)\, dv_1\,dz, \end{aligned}$$

(10)

where $V_1$ is the subgroup of all upper triangular unipotent matrices in $GL_{abn}$ generated by Y and all the one-parameter unipotent subgroups $\{x_\alpha (t)\}$, $\alpha$ a positive root, that are in V but not in Z. The character $\psi _{V_1}$ matches $\psi _{V}$ on the one-parameter subgroups $\{x_\alpha (t)\}$ in V that are not in Z and is $\psi (y_{1,b})$ on $Y(\mathbb {A})$.

To conclude the proof, we note that the inner integration over $V_1$ in (10) is a Fourier coefficient corresponding to the unipotent orbit $((an+1)(an)^{b-2}(an-1))$. Since it is not zero this contradicts Proposition  2. $\square$

4 The uniqueness property

In this section we prove the uniqueness property given in Theorem 1. To do so, suppose that there are two natural numbers, n and m, $m<n$, with cuspidal theta representations ${\Theta }_{n,\chi }^{(r)}$ and ${\Theta }_{m,\chi }^{(r)}$ attached to the same character $\chi$. We shall derive a contradiction.

As above, let $a\ge 1$ be the smallest natural number such that ${\mathcal E}_{an,\chi }^{(r)}$ is an irreducible generic representation. From Sect. 3 we know that n divides r, and hence, using [12], we have $an\le r$. Choose the smallest integer $b\ge 1$ such that $abn+m>r$. Construct the residue representation ${\mathcal E}_{abn+m,\chi }^{(r)}$ as in Sect. 2. This representation is the residue of the Eisenstein series on $GL_{nl+m}^{(r)}({\mathbb {A}})$ attached to the induced representation

$$\begin{aligned} \text {Ind}_{P_{n,\ldots ,n,m}^{(r)}({\mathbb {A}})}^{GL_{nl+m}^{(r)}({\mathbb {A}})}\left( {\mathcal E}_{an,\chi }^{(r)}\otimes {\mathcal E}_{an,\chi }^{(r)}\otimes \cdots \otimes {\mathcal E}_{an,\chi }^{(r)} \otimes \Theta _{m,\chi }^{(r)}\right) \eta _{{\mathbf s}} \end{aligned}$$

Denote by $U_{n,\ldots ,n,m}^-$ the transpose of the unipotent group $U_{n,\ldots ,n,m}$ defined in Sect. 2.

The following Lemma is standard.

Lemma 2

Let U denote the unipotent radical of a maximal parabolic subgroup of $GL_{abn+m}$. If there is no Weyl element w in $GL_{abn+m}$ such that $wUw^{-1}$ is a subgroup of $U_{n,\ldots ,n,m}^-$, then the constant term $E_{abn+m,\chi }^{(r),U}(g)$ is zero for all choices of data.

With this we have the following analogue of Proposition 1.

Proposition 4

Let U denote the unipotent radical of the maximal parabolic subgroup of $GL_{abn+m}$ whose Levi part is $GL_{r_1}\times GL_{r_2}$ with $r_1=m+kn$ and $r_2=(ab-k)n$ for some $k\ge 0$. Suppose that $wUw^{-1}$ is a subgroup of $U_{n,\ldots ,n,m}^-$ for some Weyl element w. Let $E_{abn+m,\chi }^{(r)}$ be a vector in the space of ${\mathcal E}_{abn+m,\chi }^{(r)}$. Then for $i=1,2$ there exist $E_{r_i,\chi }^{(r)}$ in the space of ${\mathcal E}_{r_i,\chi }^{(r)}$ such that

$$\begin{aligned} E_{abn+m,\chi }^{(r),U}\left( \begin{pmatrix} v_1&{}\quad \\ &{}\quad v_2\end{pmatrix}\right) =E_{m+kn,\chi }^{(r)}(v_1) E_{(ab-k)n,\chi }^{(r)}(v_2) \end{aligned}$$

for all unipotents $v_i\in GL_{r_i}(\mathbb {A})$. Moreover a statement similar to Proposition 1, part (ii), holds in this case as well.

With these properties we can prove

Proposition 5

Under the hypotheses of this section, ${\mathcal O}\left( {\mathcal E}_{abn+m,\chi }^{(r)}\right) =\left( (an)^bm\right)$.

Proof

There are two things to establish. The first is the vanishing property of the Fourier coefficients with respect to orbits that are greater than or incomparable with $\left( (an)^bm\right)$. This vanishing is established similarly to the proof of Proposition 2 above. We omit the details.

The second part of the assertion is the nonvanishing of a Fourier coefficient attached to the partition $\left( (an)^bm\right)$. We now describe such a coefficient. The description depends on the parity relation between an and m. We shall give the details in the case where both numbers are odd. The other cases are similar.

Let V denote the unipotent subgroup of $GL_{abn+m}$ consisting of all matrices of the form

$$\begin{aligned} \begin{pmatrix} v_1&{}\quad v_4&{}\quad v_6\\ &{}\quad v_2&{}\quad v_5\\ &{}\quad &{}\quad v_3\end{pmatrix}\qquad v_1,v_3\in V_{b,(an-m)/2},\quad v_2\in V_{b+1,m}. \end{aligned}$$

Here the groups $V_{k,p}$ were defined in (7) above, and $v_4,v_5,v_6$ are general suitably-sized matrices. Write $v_4=\begin{pmatrix} *&{}\quad *\\ v_4'&{}\quad *\end{pmatrix}$ where $v_4'\in \text {Mat}_{b\times b}$, and let $\psi _1(v_4)=\psi (\text {tr}\ v_4')$. Similarly, write $v_5=\begin{pmatrix} *&{}\quad *\\ v_5'&{}\quad *\\ v_5''&{}\quad *\end{pmatrix}$ with $v_5'\in \text {Mat}_{b\times b}$ and $v_5''\in \text {Mat}_{1\times b}$, and let $\psi _2(v_5)=\psi (\text {tr}\ v_5')$. Let $\psi _V$ be the character

$$\begin{aligned} \psi _V(v)=\psi _{V_{b,(an-m)/2}}(v_1)\,\psi _{V_{b+1,m}}(v_2)\,\psi _{V_{b,(an-m)/2}}(v_3)\, \psi _1(v_4)\,\psi _2(v_5). \end{aligned}$$

Then a Fourier coefficient associated with the partition $\left( (an)^bm\right)$ is given by

$$\begin{aligned} \int \limits _{V(F)\backslash V({\mathbb {A}})}E_{abn+m,\chi }^{(r)}(vh)\,\psi _V(v)\,dv. \end{aligned}$$

(11)

Let $\nu _1$ be the Weyl element of $GL_{abn+m}$ defined as follows. Write

$$\begin{aligned} \nu _1=\begin{pmatrix} w_0\\ w_1\\ \vdots \\ w_b\end{pmatrix}\qquad w_0\in \text {Mat}_{m\times (anb+m)},\qquad w_j\in \text {Mat}_{an\times (anb+m)},~ 1\le j\le b. \end{aligned}$$

Here the matrix $w_0$ has $(i, b(i+t))$ entries equal to 1 , $1\le i\le m$, and all other entries 0, where $t=(an-m)/2$. The matrices $w_j$, $1\le j\le b$, have entries of 1 at the $(i_1,j+(i_1-1)b),\ (t+i_2, j+tb+i_2(b+1))$ and $(t+m+i_3+1, j+tb+m(b+1)+i_3b)$ positions for all $1\le i_1\le t+1$, $1\le i_2\le m$ and $1\le i_3\le t-1$, and all other entries 0. This Weyl element may be characterized as follows. As explained in [9], to any unipotent orbit ${\mathcal O}$ one can attach a one dimensional torus $\{h_{\mathcal O}(t)\}$. In our case, for the unipotent orbit ${\mathcal O}=\left( (an)^bm\right)$,

$$\begin{aligned} h_{\mathcal O}(t)= & {} \text {diag}\ \left( t^{an-1}I_b, t^{an-3}I_b,\ldots ,t^{m+1}I_b,t^{m-1}I_{b+1},\ldots , t^{-(m-1)}I_{b+1},\right. \\&\qquad \left. t^{-(m+1)}I_b,\ldots , t^{-(an-1)}I_b\right) . \end{aligned}$$

The Weyl element $\nu _1$ is the shortest Weyl element in $GL_{abn+m}$ which conjugates the torus $\{h_{\mathcal O}(t)\}$ to the torus $\{h(t)\}$ with $h(t)=\text {diag}(d_m(t),d_{an}(t),\ldots , d_{an}(t))$, where for all $i>0$ we have $d_i(t)=\text {diag}\left( t^{i-1}, t^{i-3},\ldots , t^{-(i-3)}, t^{-(i-1)}\right)$.

Using the invariance of $E_{abn+m,\chi }^{(r)}$ by $\nu _1$ and moving it rightward via conjugation, the integral (11) is equal to

$$\begin{aligned} \int \limits _{Z(F)\backslash Z({\mathbb {A}})}\int \limits _{U'(F)\backslash U'({\mathbb {A}})} \int \limits _{Y(F)\backslash Y({\mathbb {A}})}E_{abn+m,\chi }^{(r)}\left( yu'z\nu _1 h\right) \,\psi _{U'} \left( u'\right) \,dy\,du'\,dz. \end{aligned}$$

(12)

The notation here is as follows. Let $U_k$ denote the maximal unipotent subgroup of $GL_k$ consisting of upper triangular matrices. Then $U'=U_m\times U_{an}\times \cdots \times U_{an}$ where the group $U_{an}$ appears b times. This group is embedded inside $GL_{anb+m}$ as $(u_0,u_1,\ldots ,u_b)\mapsto \text {diag}(u_0,u_1,\ldots ,u_b)$. The character $\psi _{U'}$ is given by

$$\begin{aligned} \psi _{U'}(u')=\psi _{U_m}(u_0)\psi _{U_{an}}(u_1)\ldots \psi _{U_{an}}(u_b), \end{aligned}$$

where $\psi _{U_k}$ is the standard Whittaker character of $U_k$. The group Y is the upper triangular unipotent group defined by $Y=\nu _1V\nu _1^{-1}\cap U_{m,an,\ldots ,an}$. The group Z is the lower triangular unipotent group consisting of all elements $v\in V$ such that $\nu _1v\nu _1^{-1}\in U_{m,an,\ldots ,an}^-$ where the group $U_{m,an,\ldots ,an}^-$ is the transpose of the unipotent group $U_{m,an,\ldots ,an}$. Another way of characterizing these groups is by means of the torus $\{h(t)\}$. The group Y is generated by the matrices $y_{i,j}(k)=I_{abn+m}+ke_{i,j}\in U_{m,an,\ldots ,an}$ such that $h(t)y_{i,j}(k)h(t)^{-1}=y_{i,j}(t^\ell k)$ for some $\ell >0$. Similarly, the group Z is generated by all matrices $z_{i,j}(k)=I_{abn+m}+ke_{i,j}\in U_{m,an,\ldots ,an}^-$ such that $h(t)z_{i,j}(k)h(t)^{-1}=z_{i,j}(t^\ell k)$ for some $\ell >0$.

The next step is to perform certain Fourier expansions on the integral (12), using root exchange (as in Ginzburg–Rallis–Soudry [10], Section 7.1) and the vanishing of the Fourier coefficients of the representation ${\mathcal E}_{abn+m,\chi }^{(r)}$ corresponding to unipotent orbits which are greater than or not comparable to $\left( (an)^bm\right)$. This process is fairly standard—see for example the proof of Ginzburg–Rallis–Soudry [8], Lemma 2.4—and so we only sketch the ideas. View $U_{m,an,\ldots ,an}$ as the group of matrices generated by $u_{i,j}(k)=I_{abn+m}+ke_{i,j}$. Similarly for the groups Y and Z. Consider the subgroup $u_{an+m-1,abn+m}(k)$. Since $h(t)u_{an+m-1,abn+m}(k)h(t)^{-1}= u_{an+m-1,abn+m}(k)$, this one dimensional unipotent group is not in Y. Similarly, conjugating by h(t) we deduce that $u_{i,abn+m}(k)$ is in Y for all $1\le i\le an+m-2$, and that $z_{abn+m-1,an+m-1}(k)$ is in Z. We may continue this process, going from the last to the first column in $U_{m,an,\ldots ,an}$. When we encounter a unipotent group of the form $u_{i,j}(k)$ in $U_{m,an,\ldots ,an}$ which is not in Y we look for a suitable unipotent subgroup of Z. If such a subgroup exists, we perform a root exchange. If not, we check that the Fourier coefficient obtained corresponds to a unipotent orbit which is greater than or not related to $\left( (an)^bm\right)$. This implies that all non-trivial characters of the expansion contribute zero, and we are left with only the trivial character.

By this argument, we see that integral (11) is not zero for some choice of data if and only if the integral

$$\begin{aligned} \int \limits _{U'(F)\backslash U'({\mathbb {A}})} \int \limits _{U_{m,an,\ldots ,an}(F)\backslash U_{m,an,\ldots ,an}({\mathbb {A}})}E_{abn+m,\chi }^{(r)}\left( uu'h\right) \,\psi _{U'}\left( u'\right) \,du\,du' \end{aligned}$$

(13)

is not zero for some choice of data.

Notice that $U_{m,abn}$ is a subgroup of $U_{m,an,\ldots ,an}$, and it is the unipotent radical of the maximal parabolic subgroup $P_{m,abn}$. Hence we can apply Proposition 4 inductively with $k = 0$ to deduce that the integral (13) is not zero for some choice of data if the two integrals

$$\begin{aligned} \int \limits _{U_m(F)\backslash U_m({\mathbb {A}})} \theta _{m,\chi }^{(r)}(u)\,\psi _{U_m}(u)\,du \end{aligned}$$

(14)

and

$$\begin{aligned} \int \limits _{U_{abn}(F)\backslash U_{abn}({\mathbb {A}})} E_{an,\chi }^{(r)}(u)\,\psi _{U_{an}}(u)\,du \end{aligned}$$

(15)

are each nonzero for suitable data. The integral (14) is not zero since $\Theta _{m,\chi }^{(r)}$ is an irreducible cuspidal representation, and hence generic. It follows from the irreducibility of ${\mathcal E}_{an,\chi }^{(r)}$ and from the definition of a that the second integral (15) is also nonzero for some choice of data. $\square$

With the above we can now prove

Proposition 6

Fix r and $\chi$. Then there is at most one natural number n such that a cuspidal theta representation ${\Theta }_{n,\chi }^{(r)}$ exists.

Proof

Recall that we are supposing that $m<n$ and there exist cuspidal theta representations ${\Theta }_{n,\chi }^{(r)}$ and ${\Theta }_{m,\chi }^{(r)}$. We will derive a contradiction. Form the residue representation ${\mathcal E}_{abn+m,\chi }^{(r)}$ as above. Then the integral (11) is not zero for some choice of data. We claim that this implies that the integral

$$\begin{aligned} \int \limits _{U'(F)\backslash U'({\mathbb {A}})} \int \limits _{U_{an,\ldots ,an,m}(F)\backslash U_{an,\ldots ,an,m}({\mathbb {A}})}E_{abn+m,\chi }^{(r)}\left( uu'h\right) \,\psi _{U'}\left( u'\right) \,du\,du' \end{aligned}$$

(16)

is not zero for some choice of data. Notice the difference between the integrals (16) and (13). In (13) the integration is over $U_{m,an,\ldots ,an}(F)\backslash U_{m, an,\ldots ,an}({\mathbb {A}})$, while in (16) it is over $U_{an,\ldots ,an,m}(F)\backslash U_{an,\ldots ,an,m}({\mathbb {A}})$. These are two different groups.

To prove that the integral (16) is not zero for some choice of data, we start with the integral (11), which we have already shown is nonzero for some choice of data. Let $\nu _2$ be the Weyl element

$$\begin{aligned} \nu _2=\begin{pmatrix} w_1\\ \vdots \\ w_b\\ w_0\end{pmatrix}\qquad w_0\in \text {Mat}_{m\times (anb+m)}, \quad w_j\in \text {Mat}_{an\times (anb+m)},~ 1\le j\le b, \end{aligned}$$

where the matrices $w_i$ are as above. Inserting $\nu _2$ into (11) and performing similar Fourier expansions, we deduce that the integral (16) is not zero for some choice of data. Here $U'=U_{an}\times \cdots \times U_{an}\times U_m$, and the character $\psi _{U'}$ is defined accordingly. But notice that $U_{abn,m}$ is a subgroup of $U_{an,\ldots ,an,m}$ which is also the unipotent radical of a maximal parabolic subgroup. However, there is no Weyl element w such that $wU_{abn,m}w^{-1}$ is contained in $U_{n,\ldots ,n,m}^-$. Hence, by Lemma 2 we obtain that the integral (16) is zero for all choices of data. This is a contradiction. $\square$

5 The condition on the character $\chi$

Let $n\ge 3$. Suppose that $\Theta _{n,\chi }^{(r)}$ is a cuspidal theta representation and $\chi =\chi _1^r$ for some character $\chi _1$. We shall derive a contradiction. The idea is similar to the one we used in Sect. 4.

Similarly to [12], we may construct the theta representation $\Theta _{2,\chi }^{(r)}$ by means of a residue of an Eisenstein series. This is possible since $\chi =\chi _1^r$. This representation has a nonzero constant term and so is not cuspidal. Define a and b as in Sect. 4 above and let m defined in that section equal 2 in the present case. Then construct the residue representation ${\mathcal E}_{abn+m,\chi }^{(r)}={\mathcal E}_{abn+2,\chi }^{(r)}$ as above. Although the representation $\Theta _{2,\chi }^{(r)}$ is not cuspidal, most of the results stated in Sect.  4 go through with small adaptations. In particular, we have

Proposition 7

Under the hypotheses of this section, ${\mathcal O}({\mathcal E}_{abn+m,\chi }^{(r)})=\left( (an)^b2\right)$.

Then we obtain a contradiction as in Sect. 4. Indeed, from Proposition 7 we obtain that the Fourier coefficient

$$\begin{aligned} \int \limits _{U'(F)\backslash U'({\mathbb {A}})} \int \limits _{U_{an,\ldots ,an,2}(F)\backslash U_{an,\ldots ,an,2}({\mathbb {A}})}E_{abn+2,\chi }^{(r)}\left( uu'h\right) \,\psi _{U'}\left( u'\right) \,du\,du' \end{aligned}$$

is not zero for some choice of data. Notice that $U_{abn,2}$ is a subgroup of $U_{an,\ldots ,an,2}$. However, even though $\Theta _{2,\chi }^{(r)}$ is not cuspidal, the constant term of $E_{abn+2,\chi }^{(r)}$ along $U_{abn,2}$ is still zero for all choices of data. Indeed, it is not hard to check that there is no Weyl element w in $GL_{abn+2}$ such that $wU_{abn,2}w^{-1}$ is contained in $U^-_{an,\ldots ,an,2}$. The vanishing of this constant term then follows as in Lemma 2. However, this is a contradiction, and the result follows.