On the length of global integrals for

Abstract

In this paper we study global integrals defined on the group $GL_n({\mathbf A})$. We prove a vanishing result for certain integrals which involve Speh representations and certain Eisenstein series, and which satisfies a certain dimension equation. This is part of a general Conjecture which states that a global nonzero integral which satisfies a dimension equation involves at most three nontrivial representations.

1 Background

Let F denote a global field and let ${\mathbf A}$ denote its adele ring. As is well known, in the Rankin–Selberg method one writes down a global integral which depends on a complex parameter s, and the basic problem is to determine when this integral is Eulerian. One of the useful tools to study this problem is the so-called dimension equation. For a genral definition of the dimension equation and related results and conjectures, see [2] Definition 3, [3,4,5]. Conjecture 1 as stated in [3] is one of the basic conjectures in this topic. We will now state it in the context of this paper.

Let $\pi _{l+1}$ denote an irreducible cuspidal representation realized on the space of automorphic forms on $GL_n(F)\backslash GL_n({\mathbf A})$, and denote by $\pi _{l+2}$ an Eisenstein series defined on the group $GL_n({\mathbf A})$. For $1\le i\le l$, let $\pi _i$ denote l representations of the group $GL_n({\mathbf A})$ which are invariant under $GL_n(F)$, and such that the integral

$$\begin{aligned} \int \limits _{Z({\mathbf A})GL_n(F)\backslash GL_n({\mathbf A})} \varphi _1(g)\varphi _2(g)\ldots \varphi _{l+1}(g)E(g,s)dg \end{aligned}$$

(1)

converges for almost all values of s. Thus, for $1\le i\le l$, one can take $\pi _i$ to be cuspidal representations, Eisenstein series and also residues or special values of Eisenstein series. Here, Z is the center of $GL_n$, and we assume that the product of all central characters of the above representations is one. Also, $\varphi _i$ is a vector in the space of $\pi _i$, and E(g, s) is a function in the space of the representation $\pi _{l+2}$. We assume that none of the representations involved is a one dimensional representation, and we refer to the number $l+2$ as the length of the integral.

To define the dimension equation attached to the integral (1), we first define the notion of the Gelfand-Kirillov dimension of a representation. As explained in [4], to every irreducible automorphic representation $\pi$ of $GL_n({\mathbf A})$ one can attach a set of unipotent orbits which we denote by ${\mathcal O}(\pi )$. This set is defined as follows. To each unipotent orbit ${\mathcal O}$ one can attach a set of Fourier coefficients. The way to do it is explained in [4] Sect. 2. As explained in [1], one can define a partial order on the set of all unipotent orbits. We say that a unipotent orbit ${\mathcal O}$ is in the set ${\mathcal O}(\pi )$ if the representation $\pi$ has a nontrivial Fourier coefficient attached to the unipotent orbit ${\mathcal O}$, but for all unipotent orbits which are greater than ${\mathcal O}$, the representation $\pi$ has no nontrivial Fourier coefficient. From this definition it is clear that the set ${\mathcal O}(\pi )$ can contain more than one unipotent orbit. It is an open conjecture that for any irreducible automorphic representation this set contains only one element. In many cases this conjecture is known. For example, it trivially holds for any irreducible cuspidal representation of $GL_n({\mathbf A})$, since any such representation is generic. It is also known for Speh representations which were defined in [6] as residues of certain Eisenstein series. This is established in [4] Proposition 5.3. Henceforth, we shall assume that it holds for all representations involved. In other words, we will always assume that the set ${\mathcal O}(\pi )$ contains a unique unipotent orbit.

As follows from [1], to each unipotent orbit one can define its dimension. For the group $GL_n$, one can parameterize unipotent orbits of $GL_n$ by all partitions of n. With this identification the dimension of a unipotent orbit is given by the second equality of equation (7). Thus, since we assume that the set ${\mathcal O}(\pi )$ contains only one element, one can then define the dimension of $\pi$, denoted by $\text {dim}\ \pi$, to be half the number $\text {dim}\ {\mathcal O}(\pi )$. For example, the representation $\pi _{l+1}$ is a cuspidal representation, and hence it is a generic representation. Hence ${\mathcal O}(\pi _{l+1})=(n)$ and $\text {dim}\ {\mathcal O}(\pi _{l+1})=\frac{1}{2}n(n-1)$.

With these notations, the dimension equation attached to integral (1) is defined by,

$$\begin{aligned} \sum _{i=1}^{l+2}\text {dim}\ \pi _i=\text {dim}\ GL_n -1. \end{aligned}$$

(2)

As explained in [2, 3, 5], all known global unipotent integrals which are non-zero and Eulerian, do satisfy the dimension equation (2). In [3] Sect. 2 there is a detailed description of the integrals we refer to. In particular the notion of unipotent global integral is defined in [3] Definition 1. One of the requirements is that the global integral (1) unfolds to an integral which involves unique functionals defined on the various representations in question. This is not always the case with Eulerian global integrals. For example, this does not happen with global integrals which are known in the literature as New Way type integrals. See [8].

The main Conjecture in this topic is

Conjecture 1

Assume that the integrals (1) satisfies the dimension equation (2). Suppose that $l>1$. Then these integrals are zero for all choice of data.

Even though the motivation for stating and studying this conjecture is the desire to classify global Rankin–Selberg integral, we emphasize that this conjecture is stated regardless to what type of integral we consider, in other words, regardless as whether we expect this integral to be a global unipotent integral or if we expect this integral to unfold to any other type of integral.

In particular, if true, this Conjecture asserts that if a global unipotent integral satisfies the dimension equation, and is not zero then $l=1$. It is well known that such integrals exists. For example the Rankin product integral is such an integral. See [5] Theorem 1 for a partial classification of such integrals when $l=1$.

There are two main difficulties in studying Conjecture 1. The first difficulty is that it is not practical to classify all solutions to Eq. (2). For low values of n it is not hard but the number of solutions grows quite fast. The second difficulty is the fact that for $1\le i\le l$, the representations $\pi _i$ are arbitrary and hence when unfolding the integral and performing Fourier expansions, there are many cases to consider.

To illustrate this, let us consider the case which motivates the integrals we study in this paper. Consider the special case of integral (1) where the Eisenstein series is a minimal representation of $GL_n({\mathbf A})$. In other words, let E(g, s) denote the Eisenstein series attached to the induced representation $Ind_{P({\mathbf A})}^{GL_n({\mathbf A})}\delta _P^s$. Here P is the maximal parabolic of $GL_n$ whose Levi part is $GL_{n-1}\times GL_1$. A simple unfolding process which we will perform in the next section implies that the integral

$$\begin{aligned} \int \limits _{U_n(F)\backslash U_n({\mathbf A})} \varphi _1(u)\varphi _2(u)\ldots \varphi _{l}(u)\psi _U(u)du \end{aligned}$$

(3)

is an inner integration to integral (1). Here $U_n$ is the maximal unipotent subgroup of $GL_n$ and $\psi _U$ is the Whittaker character of $U_n(F)\backslash U_n({\mathbf A})$. For more details see Sect. 2.

Moreover, as we will explain below, if integral (1) satisfies the dimension Eq. (2), then the integral (3) also satisfies a similar equation. Namely, we have

$$\begin{aligned} \sum _{i=1}^{l}\text {dim}\ \pi _i=\text {dim}\ U_n=\frac{1}{2}n(n-1). \end{aligned}$$

(4)

Hence, in this case, Conjecture 1 reduces to

Conjecture 2

Suppose that integral (3) satisfies the dimension Eq. (4). Suppose that $l>1$. Then the integral (3) is zero for all choice of data.

Notice that in integral (3) the integration domain is a compact group. Hence the integral clearly converges. Conjecture 2 is interesting by itself regardless of its application to Conjecture 1.

When $l=1$, integral (3) reduces to the Fourier coefficient of the representation $\pi _1$ which corresponds to the unipotent orbit (n). In other words, this is just the Whittaker coefficient of $\pi _1$. Since there are generic representations defined on the group $GL_n({\mathbf A})$, clearly there are such nonzero global integrals. An important example of such integrals, with $l=1$, is the well known Shahidi type integrals. These are the cases when $\pi _1$ is an Eisenstein series induced from generic automorphic representations [9].

In this paper we study Conjecture 2 for two important types of representations. The first type of representation is what we refer to as representations of Speh type. In our context, a representation $\pi$ is a representation of Speh type if ${\mathcal O}(\pi )=(q^m)$. Here m and q are two natural numbers such that $n=mq$. The motivation is that every Speh representation is such a representation. See [4] Proposition 5.3. Notice that by our definition, every generic representation is a representation of Speh type, and it is not hard to find examples of Eisenstein series which are also such representations.

The second type of representations are Eisenstein series. In detail, for $1\le i\le r$, let $\tau _i$ denote an automorphic representation of the group $GL_{m_i}({\mathbf A})$ where we assume that $m_i\ge m_{i+1}$. Let Q denote a parabolic subgroup of $GL_n$, whose Levi part is $M=GL_{m_1}\times \ldots \times GL_{m_r}$. Then $\tau =\tau _1\times \tau _2\times \ldots \times \tau _r$ is a representation of $M({\mathbf A})$. Denote by $E_\tau (g,\widetilde{s})$ the Eisenstein series of $GL_n({\mathbf A})$ attached to the induced representation $Ind_{Q({\mathbf A})}^{GL_n({\mathbf A})}\tau \delta _Q^{\widetilde{s}}$. Since Q need not be a maximal parabolic subgroup, it is possible that this Eisenstein series depends on several complex variables. To emphasize this we shall use the notation $\widetilde{s}$. We will assume that the Eisenstein series in question is in general position. By that we mean that we are in the domain where it is given by a convergent series, and hence we can carry out an unfolding process. When we need to consider more than one Eisenstein series we will use the notation $E_\sigma (g,\widetilde{\nu })$, or if we need to consider a collection of Eisenstein series, we will also use the notation $E_{\tau ^{(i)}}(g,\widetilde{s_i})$. The definition of these Eisenstein series is similar to the definition of $E_\tau (g,\widetilde{s})$ given above.

Our first main result concerns the case when $l=2$. Because of Lemma 2, to prove Conjecture 2 for $l=2$, for the set of representations we consider, it is enough to prove,

Theorem 1

Let $E_\tau (g,\widetilde{s})$ and $E_\sigma (g,\widetilde{\nu })$ denote two Eisenstein series defined as above. Let $\pi$ be a representation of Speh type or the Eisenstein series $E_\sigma (g,\widetilde{\nu })$. Assume that $\tau _1$ is either a representation of Speh type, or any one dimensional representation of $GL_{m_1}({\mathbf A})$. Assume that $\text {dim}\ \pi +\text {dim}\ E_\tau (g,\widetilde{s})= \text {dim}\ U_n$. Then the integral

$$\begin{aligned} \int \limits _{U_n(F)\backslash U_n({\mathbf A})}\varphi (u)E_\tau (u,\widetilde{s}) \psi _U(u)du \end{aligned}$$

(5)

is zero for all choice of data. Here $\varphi$ is a vector in the space of $\pi$.

The proof of this Theorem is given in Sect. 3. From Theorem 1, using the above and the argument stated in the beginning of Sect. 2, we deduce

Theorem 2

Let $\pi _1$ denote either a representation of Speh type, or an Eisenstein series $E_\tau (g,\widetilde{s})$ as in Theorem 1. Let $\pi _2$ denote a representation of Speh type, or an Eisenstein series $E_\sigma (g,\widetilde{\nu })$ as defined above. Let $\pi _3$ denote an irreducible cuspidal automorphic representation of $GL_n({\mathbf A})$, and let $\pi _4=E(g,s)$ denote the Eisenstein series defined right before integral (3). Then Conjecture 1 holds with $l=2$.

In the last section we consider Conjecture 2 in the case when $l\ge 3$. We prove the following

Theorem 3

Assume that $l\ge 3$, and that for all $1\le i\le l-1$ we have $\pi _i=E_{\tau ^{(i)}}(g,\widetilde{s_i})$. Assume that $\pi _l$ is a representation of Speh type, or that $\pi _l=E_{\tau ^{(l)}}(g,\widetilde{s_l})$ with the property that for some j we have that $\tau _j^{(l)}$ is a one dimensional representation. Then Conjecture 2 holds.

We are aware that there are other representations which are not of the types mentioned above. For example Eisenstein series at some special values, or residues of Eisenstein series. However, the above two types are the most important. Every representation in the discrete spectrum is included in them. We hope to consider other cases in the future.

2 Notations and preliminary results

We keep the notations of the Introduction. We start by unfolding the global integral (1) in the case where E(g, s) is the Eisenstein series defined right before integral (3). Notice first that since $\pi _{l+1}$ is a cuspidal representation, the integral (1) converges for almost all s. Furthermore, for $\text {Re}(s)$ large, the following integrals converge and therefore we can carry out the unfolding process. Assuming $\text {Re}(s)$ large this integral is equal to

$$\begin{aligned} \int \limits _{Z({\mathbf A})P(F)\backslash GL_n({\mathbf A})} \varphi _1(g)\varphi _2(g)\ldots \varphi _{l+1}(g)f(g,s)dg. \end{aligned}$$

(6)

Here f(g, s) is a section in the induced space used to construct the Eisenstein series E(g, s). Since we assume that $\pi _{l+1}$ is a cuspidal representation, we can use the well known expansion for such representations, see [7],

$$\begin{aligned} \varphi _{l+1}(g)=\sum _{\gamma \in U_n(F)\backslash P(F)} W_{l+1}(\gamma g). \end{aligned}$$

Here $W_{l+1}$ is the Whittaker coefficient of $\varphi _{l+1}$, defined by

$$\begin{aligned} W_{l+1}(g)=\int \limits _{U_n(F)\backslash U_n({\mathbf A})} \varphi _{l+1}(ug)\overline{\psi _U(u)}du. \end{aligned}$$

The character $\psi _U$ is defined as follows. Let $u=(u_{i,j})\in U_n$. Then $\psi _U(u)=\psi (u_{1,2}+u_{2,3}+\cdots +u_{n-1,n})$. Plugging the above expansion in integral (6), we obtain

$$\begin{aligned} \int \limits _{Z({\mathbf A})U_n(F)\backslash GL_n({\mathbf A})} \varphi _1(g)\varphi _2(g)\ldots \varphi _{l}(g)W_{l+1}(g)f(g,s)dg. \end{aligned}$$

Factoring the measure, we obtain the integral

$$\begin{aligned} \int \limits _{Z({\mathbf A})U_n({\mathbf A})\backslash GL_n({\mathbf A})} \int \limits _{U_n(F)\backslash U_n({\mathbf A})} \varphi _1(ug)\varphi _2(ug)\ldots \varphi _{l}(ug)\psi _U(u)du W_{l+1}(g)f(g,s)dg. \end{aligned}$$

Thus, we obtain integral (3) as inner integration to the above integral.

Suppose that integral (1) satisfies the dimension Eq. (2). Since $\pi _{l+1}$ is a generic representation, then $\text {dim}\ \pi _{l+1}=\frac{1}{2}n(n-1)$. The Eisenstein series E(g, s) used in the above integral is attached to the unipotent orbit $(21^{n-2})$ and has dimension $n-1$. This follows from [4] Proposition 5.2. Plugging these numbers into Eq. (2) we obtain the Eq. (4).

Let $\pi$ denote an automorphic representation, and suppose that ${\mathcal O}(\pi )=\lambda =(k_1k_2\ldots k_p)$ which is a partition of n. In other words, we have $k_i\ge k_{i+1}$ and $\sum k_i=n$. Then, as follows from [1], see also [5], we have

$$\begin{aligned} \text {dim}\ \pi =\frac{1}{2}\text {dim}\ \lambda = \frac{1}{2}\left( n^2-\sum _{i=1}^p(2i-1)k_i\right) =\frac{1}{2}(n^2+n)-\sum _{i=1}^p ik_i. \end{aligned}$$

(7)

In the following Lemma we compute a certain relation between the dimensions of certain type of partitions. We recall that if $\lambda =(k_1k_2\ldots k_p)$ where $k_p\ge 1$, then p is called the length of the partition. Also, we denote by $\lambda ^t$ the transpose of the partition $\lambda$ [1].

Lemma 1

Let $\mu$ be a nontrivial partition of n, and assume that $\mu ^t=(m_1m_2\ldots m_r)$. Then, for any partition $\lambda$ of n, whose length is at most $n-m_1+1$, we have

$$\begin{aligned} \text {dim}\ \lambda +\text {dim}\ \mu > n^2-n. \end{aligned}$$

(8)

Proof

Using [1], see also [4] Proposition 5.16, we have $\text {dim}\ \mu =2\sum _{1\le i<j\le r}m_im_j$. Using Eq. (7) we need to prove that

$$\begin{aligned} I=n+\sum _{1\le i<j\le r}m_im_j-\sum _{i=1}^p ik_i>0 \end{aligned}$$

(9)

for all partitions $\lambda =(k_1\ldots k_p)$ where $p\le n-m_1+1$. The partition $(m_11^{n-m_1})$ is a partition of length $n-m_1+1$. It is not hard to check that it satisfies inequality (9), and every unipotent orbit ${\mathcal O}=(k_1'k_2'\ldots k_q')$ such that $k_1'\ge m_1$ and $q\le n-m_1+1$, is greater than or equal to $(m_11^{n-m_1})$. If $k_1'\le m_1-1$, then there is a number $2\le a \le m_1$ such that ${\mathcal O}$ is greater than or equal to $(a^{p_1}(a-1)^{p_2})$ with the following conditions. First, we have

$$\begin{aligned} p_1+p_2=n-m_1+1\ \ \ \ \ \ ap_1+(a-1)p_2=n \end{aligned}$$

(10)

If $a>2$, then we also have

$$\begin{aligned} \frac{am_1-(a+1)}{a-1} < n\le \frac{(a-1)m_1-a}{a-2}. \end{aligned}$$

(11)

When $a=2$, we have the condition $n\ge 2m_1-2$.

Thus, it is enough to prove that $I>0$ for the partitions $(a^{p_1}(a-1)^{p_2})$ with the above conditions. To do that we compute I for these partitions. It is equal to

$$\begin{aligned} n+\sum _{1\le i<j\le r}m_im_j-a\sum _{i=1}^{p_1} i -(a-1)\sum _{i=p_1+1}^{p_1+p_2}i=n+\sum _{1\le i<j\le r}m_im_j-a\sum _{i=1}^{p_1+p_2} i+\sum _{i=p_1+1}^{p_1+p_2}i. \end{aligned}$$

From this we obtain

$$\begin{aligned} I=n+\sum _{1\le i<j\le r}m_im_j+\sum _{i=p_1+1}^{p_1+p_2}i- \frac{a}{2}(n-m_1+1)(n-m_1+2). \end{aligned}$$

(12)

Assume first that $a\ge 4$ and that $p_2\ge 1$. From the right hand side of the inequality (11), we deduce that $n-m_1+1\le \frac{m_1-2}{a-2}$. Hence, it is enough to prove that

$$\begin{aligned} I_0=n+\sum _{1\le i<j\le r}m_im_j+\sum _{i=p_1+1}^{p_1+p_2}i- \frac{a(m_1-2)}{2(a-2)}(n-m_1+2)>0. \end{aligned}$$

Write

$$\begin{aligned} \sum _{1\le i<j\le r}m_im_j=m_1(m_2+\cdots +m_r)+\sum _{2\le i<j\le r}m_im_j=m_1(n-m_1)+\sum _{2\le i<j\le r}m_im_j. \end{aligned}$$

(13)

Plugging this into $I_0$, we deduce that $I_0$ is equal to

$$\begin{aligned} \sum _{2\le i<j\le r}m_im_j+\sum _{i=p_1+1}^{p_1+p_2}i+ \left( 1-\frac{a}{2(a-2)}\right) m_1(n-m_1)+n+\frac{a(n-m_1+2)}{a-2} -\frac{am_1}{a-2}. \end{aligned}$$

Since $a\ge 4$, the third term from the left is not negative. From the assumption that $p_2\ge 1$, and from the fact that $p_1+p_2=n-m_1+1$ we deduce that the second term from the left is equal to $n-m_1+1+\epsilon$ where $\epsilon \ge 0$. Hence, to conclude that $I_0>0$, it is enough to check that

$$\begin{aligned} n+\frac{a(n-m_1+2)}{a-2}+n-m_1+1\ge \frac{am_1}{a-2}. \end{aligned}$$

This is equivalent to $n\ge (3a-2)(m_1-1)/(3a-4)$. Using the left inequality of (11), it is enough to prove that

$$\begin{aligned} \frac{am_1-(a+1)}{a-1}\ge \frac{(3a-2)(m_1-1)}{(3a-4)}. \end{aligned}$$

This inequality is easy to verify.

To conclude the case when $a\ge 4$, we still have to consider the case when $p_2=0$. When this happens, then it follows from (10) that $a(n-m_1+1)=n$. Plugging this into (12) we obtain $I=\sum m_im_j+(nm_1-n^2)/2$. Since $n=\sum m_i$, then $2\sum m_im_j-n^2=-\sum m_i^2$, and it is easy to check that $I>0$.

Finally we need to prove that when $a=2,3$, then $I>0$. We start with $a=2$. In this case we have $n\ge 2m_1-2$, and $p_1=m_1-1$. Hence,

$$\begin{aligned} I=\sum _{2\le i<j\le r} m_im_j +2m_1n+2m_1-\frac{1}{2}(n^2+n) -2m_1^2-1. \end{aligned}$$

Notice that in the right hand side the first sum is over $2\le i<j\le r$. This follows from the identity $m_1(m_2+\cdots m_r)=m_1(n-m_1)$.

Let $m_i=m_1-\mu _i$ where $\mu _2\le \mu _3\le \ldots \le \mu _r$. Then $n=rm_1-\mu$ where $\mu =\mu _2+\cdots +\mu _r$. Plugging all this into the right hand side of the above equation, we obtain

$$\begin{aligned}&\sum \limits _{2\le i<j\le r}(m_1-\mu _i)(m_1-\mu _j)+2m_1(rm_1+\mu )+2m_1- \frac{1}{2}((rm_1+\mu )^2+rm_1+\mu ) -2m_1^2-1 \end{aligned}$$

Simplifying, this is equal to

$$\begin{aligned} \frac{1}{2}((r-2)m_1^2-(r-4)m_1-(\mu _2^2+\cdots +\mu _r^2)+\mu )-1 \end{aligned}$$

We have $(r-1)m_1^2-(\mu _2^2+\cdots +\mu _r^2)=\sum _{i=2}^r(m_1^2-\mu _i^2) =\sum _{i=2}^rm_i(m_1+\mu _i)$, and $\mu =(r-1)m_1-(m_2+\cdots +m_r)$. Plugging this in, we find that the above is equal to $\frac{1}{2}(\sum _{i=2}^r(m_i(\mu _i-1)+m_1(n-2m_1+3)))-1$. Since $n\ge 2m_1-2$, then $n-2m_1+3>0$. The first term could have some negative terms. This will happen if $m_i=m_1$ for some $i>1$. However, a direct computation shows that even in this case $I>0$.

The last case to consider is the case when $a=3$. From (10) we obtain that $p_1=2m_1-n-2$ and $p_2=2n-3m_1+3$. Using the factorization in Eq. (13), it is enough to prove that

$$\begin{aligned} n+m_1(n-m_1)+\sum _{i=p_1+1}^{p_1+p_2}i-a\sum _{i=1}^{p_1+p_2}i >0. \end{aligned}$$

Denote the left hand side by $I_0$. Plugging in the values for $p_1$ and $p_2$, we obtain $I_0=5m_1n+6m_1-3n^2/2-7n/2-4m_1^2-3$. From (11) we obtain that $3m_1-4<2n\le 4m_1-6$. Hence , we can write $2n=3m_1+\alpha$ where $-3\le \alpha \le m_1-6$. Plugging in $I_0$, we obtain

$$\begin{aligned} 8I_0=m_1^2+2m_1\alpha +6m_1-3\alpha ^2-14\alpha -24. \end{aligned}$$

It is easy to check that for the values of $\alpha$ we are interested in, we obtain that $8I_0>0$ for $m_1\ge 4$. For the values $1\le m_1 \le 3$, we have from $n\le 2m_1-3$ that the result holds. $\square$

We have the following,

Lemma 2

Suppose that $l\ge 2$. Assume that at least two of the representations $\pi _i$ are representations of Speh type. Then the dimension Eq. (4) is not satisfied.

Proof

Let $\mu =(2^{\frac{n}{2}})$ if n is even, and $\mu =(2^{\frac{n-1}{2}}1)$ if n is odd. Then $\mu ^t=\left( \left( \frac{n}{2}\right) ^2\right)$ if n is even, and $\mu ^t=\left( \left( \frac{n+1}{2}\right) \left( \frac{n-1}{2}\right) \right)$ if n is odd. It follows from Lemma 1, or by direct calculation, that Eq. (8) holds with $\lambda =\mu$.

Consider the l representations $\pi _i$, and assume that $\pi _1$ and $\pi _2$ are two representations of Speh type. Then ${\mathcal O}(\pi _i)=(p_i^{q_i})$ for $i=1,2$. Hence ${\mathcal O}(\pi _i)\ge \mu$ where $\mu$ was defined above. Thus, recalling that the dimension of a representation is half of the dimension of the corresponding partition, we have

$$\begin{aligned} \sum _{i=1}^2\text {dim}\ \pi _i=\frac{1}{2} \sum _{i=1}^2\text {dim}\ {\mathcal O}(\pi _i)\ge \text {dim}\ \mu >\frac{1}{2}(n^2-n) \end{aligned}$$

where the last inequality follows from Eq. (8), or by direct calculation. $\square$

For a root $\gamma$ for $GL_n$ we shall denote by $\{x_\gamma (m)\}$ the one dimensional unipotent subgroup of $GL_n$. We need the following trivial Lemma, whose proof is obtained by simple Fourier expansion,

Lemma 3

Let $\alpha , \beta$ be two roots for the group $GL_n$ such that $\alpha +\beta$ is also a root. Let f denote an automorphic function of $GL_n({\mathbf A})$. Consider the integral

$$\begin{aligned} \int \limits _{(F\backslash {\mathbf A})^2}f(x_{\alpha +\beta }(m_1)x_\beta (m_2))\psi (m_1)dm_1dm_2. \end{aligned}$$

(14)

Then it is equal to

$$\begin{aligned} \int \limits _{\mathbf A} \int \limits _{(F\backslash {\mathbf A})^2}f(x_\alpha (m_3)x_{\alpha +\beta }(m_1)x_\beta (m_2))\psi (m_1)dm_1dm_3dm_2. \end{aligned}$$

Proof

Denote integral (14) by I. Applying Fourier expansion, we obtain

$$\begin{aligned} I=\int \limits _{F\backslash {\mathbf A}}\ \sum _{\delta \in F} \int \limits _{(F\backslash {\mathbf A})^2}f(x_\alpha (m_3)x_{\alpha +\beta }(m_1)x_\beta (m_2))\psi (m_1+\delta m_3)dm_1dm_2dm_3 \end{aligned}$$

Since f is invariant on the left by elements in $GL_n(F)$, we have $f(x_\beta (\delta )g)=f(g)$ for all $g\in GL_n({\mathbf A})$ and $\delta \in F$. Hence, conjugating the element $x_\beta (\delta )$ to the right, we obtain after a change of variables in $m_1$,

$$\begin{aligned} I=\int \limits _{F\backslash {\mathbf A}}\ \sum _{\delta \in F} \int \limits _{(F\backslash {\mathbf A})^2}f(x_\alpha (m_3)x_{\alpha +\beta }(m_1)x_\beta (m_2+\delta ))\psi (m_1)dm_1dm_2dm_3 \end{aligned}$$

Collapsing summation and integration, the Lemma follows. $\square$

In particular, integral (14) is zero for all choice of data if and only if the integral

$$\begin{aligned} \int \limits _{(F\backslash {\mathbf A})^2}f(x_{\alpha +\beta }(m_1)x_\alpha (m_3))\psi (m_1)dm_1dm_3 \end{aligned}$$

is zero for all choice of data. This follows by a simple convolution with a Schwartz function on ${\mathbf A}$.

3 The proof of Theorem 1

In this section we prove Theorem 1. As mentioned in the Introduction, in this paper we study Conjecture 2 for Speh type representations, and for Eisenstein series. These representations were defined in the introduction right before Theorem 1. Thus, because of Lemma 2, we may assume that one of the two representations is an Eisenstein series.

It will be convenient to separate the proof into two cases.

3.1 Eisenstein series: the trivial case

In this subsection we assume that the representation $\tau _1$ of the group $GL_{m_1}({\mathbf A})$ is the trivial representation. We recall that by construction, $m_1\ge m_i$ for all i.

Let $\pi$ denote an irreducible automorphic representation of $GL_n({\mathbf A})$. The integral we consider is

$$\begin{aligned} \int \limits _{U_n(F)\backslash U_n({\mathbf A})}\varphi (u)E_\tau (u,\widetilde{s})\psi _U(u)du. \end{aligned}$$

(15)

Here $\varphi$ is a vector in the space of $\pi$. We have, see [4] Proposition 5.16, $\text {dim}\ E(g,\widetilde{s})=\ \text {dim} U(Q)+\text {dim}\ \tau$. Hence, $\text {dim}\ E(g,\widetilde{s})=\text {dim}\ \tau +\sum _{1\le i<j\le r}m_im_j$. Then the dimension equation attached to integral (15) is given by

$$\begin{aligned} \text {dim}\ \pi +\text {dim}\ \tau +\sum _{1\le i<j\le r}m_im_j=\frac{1}{2}(n^2-n). \end{aligned}$$

(16)

To prove Theorem 1 in this case we prove,

Proposition 1

Assume that $\pi$ satisfies Eq. (16). Then the integral (15) is zero for all choice of data.

Proof

Unfolding the Eisenstein series, integral (15) is equal to

$$\begin{aligned} \sum _{w\in Q(F)\backslash GL_n(F)/U_n(F)}\ \ \int \limits _{w^{-1}U_n(F)w\cap U_n(F)\backslash U_n({\mathbf A})}\varphi (u)f_\tau (wu,\widetilde{s})\psi _U(u)du. \end{aligned}$$

(17)

The sum is finite and representatives can be taken to be Weyl elements. Factoring the measure, we obtain the integral

$$\begin{aligned} \int \limits _{U_n^w(F)\backslash U_n^w({\mathbf A})}\varphi (u)\psi _U(u)du \end{aligned}$$

(18)

as inner integration to integral (17). Here, given a Weyl element w, we denote $U_n^w=w^{-1}U_nw\cap U_n$. By means of Fourier expansions, we can express integral (18) as a sum of Fourier coefficients corresponding to a set of unipotent orbits ${\mathcal O}_1,\ {\mathcal O}_2,\ldots ,{\mathcal O}_q$. Suppose that we show that for all $1\le i\le q$ we have

$$\begin{aligned} \frac{1}{2}\text {dim}\ {\mathcal O}_i+\sum _{1\le i<j\le r}m_im_j> \frac{1}{2}(n^2-n). \end{aligned}$$

(19)

Then it follows from Eq. (16) and (19) that integral (18) is zero for all choice of data. Indeed, from (16) and (19) we obtain that $\frac{1}{2}\text {dim}\ {\mathcal O}_i> \text {dim}\ \pi$ for all $1\le i\le q$. Hence $\text {dim}\ {\mathcal O}_i > \text {dim}\ {\mathcal O}(\pi )$. By the definition of ${\mathcal O}(\pi )$, we deduce that integral (18) is zero for all choice of data. But the vanishing of all these integrals implies that integral (15) is zero for all choice of data which is what we want to prove.

We fix some notations. For $1\le i\le n$, let $V_i$ denote the unipotent subgroup of $U_n$ generated by all matrices of the form $I_n+x_je_{i,j}$ where $i<j\le n$. Here $e_{i,j}$ is the matrix of size n whose (i, j) entry is one, and all other entries are zero. We define $V_n$ to be the identity group. Let $1\le i_1<i_2<\ldots <i_k\le n$ denote a set of natural numbers. Denote $V_{i_1,\ldots ,i_k}=V_{i_1}V_{i_2}\ldots V_{i_k}$. Then we claim that given a Weyl element $w\in Q\backslash GL_n/U_n$, there is a set $1\le i_1<i_2<\ldots <i_k\le n$ where $k\le n-m_1$, such that the integral

$$\begin{aligned} I(i_1,\ldots ,i_{k-1},i_k)=\int \limits _{U_n(F)V_{i_1,\ldots ,i_k}({\mathbf A})\backslash U_n({\mathbf A})}\varphi (u)\psi _U(u)du \end{aligned}$$

(20)

is an inner integration to integral (18). This claim follows from the fact that every such w can be chosen to be a permutation matrix. Hence, if w has an entry one at the $(a,j_a)$ position, where $1\le a\le m_1$, then the group $V_{j_1,\ldots , j_{m_1}}$ is contained in $U_n^w$. This means that there are at most $n-m_1$ indices and subgroups $V_i$ which are not contained inside $U_n^w$. Hence, we can factor the group $U_n$ as a disjoint product $U_n=V_{i_1,\ldots ,i_k}V_{l_1,\ldots ,l_{n-k}}$ such that $k\le n-m_1$, and such that $V_{l_1,\ldots ,l_{n-k}}$ is contained in $U_n^w$. Factoring the integration domain in integral (18), we obtain that the integral

$$\begin{aligned} \int \limits _{V_{l_1,\ldots ,l_{n-k}}(F)\backslash V_{l_1,\ldots ,l_{n-k}}({\mathbf A})}\varphi (u)\psi _U(u)du \end{aligned}$$

is an inner integration to integral (18). Since this integral is equal to $I(i_1,\ldots ,i_{k-1},i_k)$, the above claim follows. Hence, it is enough to prove that given a representation $\pi$ which satisfies Eq. (16), then for all sets $\{ i_1,\ldots , i_k\}$ as above, the integrals $I(i_1,\ldots ,i_{k-1},i_k)$ are zero for all choice of data.

To prove that we argue by induction. First, let $\epsilon =(\epsilon _1,\ldots ,\epsilon _{n-1})$ where $\epsilon _i=0,1$. Define the character $\psi _{U,\epsilon }$ of the group $U_n$ as follows. Given $u=(u_{i,j})\in U_n$, define $\psi _{U,\epsilon }(u)= \psi (\epsilon _1 u_{1,2}+\cdots +\epsilon _{n-1} u_{n-1,n})$. We now define the set of Fourier coefficients

$$\begin{aligned} I(i_1,\ldots ,i_{k-1},i_k; \epsilon _{j_1},\ldots , \epsilon _{j_p})=\int \limits _{U_n(F)V_{i_1,\ldots ,i_k}({\mathbf A})\backslash U_n({\mathbf A})}\varphi (u)\psi _{U,\epsilon }(u)du \end{aligned}$$

(21)

where all the $j_1,\ldots , j_p$ components of $\epsilon$ are zeros, and all other components are one. Notice that when there are no $i_m$ indices then the integration is over $U_n(F)\backslash U_n({\mathbf A})$. We shall denote these integrals by $I(\epsilon _{j_1},\ldots , \epsilon _{j_p})$. If further there are also no $\epsilon _{j_m}$ then integral (21) is the Whittaker coefficient of $\pi$. We shall denote this integral by $I_0$.

Start with integral $I(i_1,\ldots ,i_{k-1},i_k)$. By means of Fourier expansions we will prove that this integral is equal to a sum of integrals such that the integrals $I(i_1,\ldots ,i_{k-1},i_k+a)$ and the integral $I(i_1,\ldots ,i_{k-1}; \epsilon _{i_k})$ appear as inner integrations to each summand. Here $1\le a\le n-i_k$. Notice that when $a=n-i_k$, we have $I(i_1,\ldots ,i_{k-1},i_k+a)=I(i_1,\ldots ,i_{k-1})$. Repeating this process with each of the integrals $I(i_1,\ldots ,i_{k-1},i_k+a)$, we deduce that the integral $I(i_1,\ldots ,i_{k-1},i_k)$ is a sum of integrals such that $I(i_1,\ldots ,i_{k-1})$ and $I(i_1,\ldots ,i_{k-1}; \epsilon _{i_k+a})$ appear as inner integration in each summand. Here $0\le a\le n-i_k-1$. Continuing this process with this set of integrals we finally deduce that $I(i_1,\ldots ,i_{k-1},i_k)$ is a sum of integrals such that $I(\epsilon _{j_1},\ldots , \epsilon _{j_p})$ appear as an inner integration for some set of indices $1\le j_1<j_2<\ldots <j_p\le n-1$ and $0\le p\le n-m_1$. Notice that the bound on p follows from the fact that the number k as defined in (20) is bounded by $n-m_1$. Thus, to complete the proof we will first relate $I(i_1,\ldots ,i_{k-1},i_k)$ to the integrals $I(i_1,\ldots ,i_{k-1},i_k+a)$ and $I(i_1,\ldots ,i_{k-1}; \epsilon _{i_k})$ as mentioned above. Then we prove that the integrals $I(\epsilon _{j_1},\ldots , \epsilon _{j_p})$ are zero for all choice of data.

Consider the integral

$$\begin{aligned} I(i_1,\ldots ,i_{k-1},i_k)=\int \limits _{U_n(F)V_{i_1,\ldots ,i_{k-1}}({\mathbf A})V_{i_k}({\mathbf A})\backslash U_n({\mathbf A})}\varphi (u)\psi _U(u)du \end{aligned}$$

(22)

where we used the fact that $V_{i_1,\ldots ,i_k}=V_{i_1,\ldots ,i_{k-1}} V_{i_k}$. Expand this integral along the one dimension unipotent subgroup $\{x_1(y_1)=I_n+y_1e_{i_k,n}\}$. Thus, $I(i_1,\ldots ,i_{k-1},i_k)$ is equal to

$$\begin{aligned} \int \int \limits _{F\backslash {\mathbf A}} \varphi (ux_1(y_1))\psi _U(u)dy_1du+\sum _{\eta \in F^*}\int \int \limits _{F\backslash {\mathbf A}}\varphi (ux_1(y_1)t(\eta ))\psi _U(u)\psi (y_1)dy_1du \end{aligned}$$

(23)

where the integration over u is as in integral (22), and $t(\eta )$ is the torus element $t(\eta )=\text {diag}(\eta I_{i_k},I_{n-i_k})$. Consider each term of the right most integral in Eq. (23). In the notations of Lemma 3, we denote $x_{\alpha +\beta }(y_1)=x_1(y_1)$. Also we denote $x_\alpha (z_1)=I_n+z_1e_{i_k,n-1}$ and $x_\beta (z_2)=I_n+z_2e_{i_k+1,n}$. Then the conditions of the Lemma hold and we can apply it. We repeat this process with $x_\alpha (z_1)=I_n+z_1e_{i_k,j}$ and $x_\beta (z_2)=I_n+z_2e_{i_k+n-j,n}$ in decreasing order in j for all $i_k+1\le j\le n-2$. Then, after applying Lemma 3 for $n-i_k-1$ times, we conjugate by the Weyl element

$$\begin{aligned} \begin{pmatrix} I_{i_k}&{}&{} \quad \\ &{}&{} \quad 1\\ &{} \quad I_{n-i_k-1} &{} \quad \end{pmatrix} \end{aligned}$$

Then it is not hard to check that we obtain the integral $I(i_1,\ldots ,i_{k-1},i_k+1)$ as inner integration. Thus, we conclude that each summand in the right term integral of Eq. (23) contains the integral $I(i_1,\ldots ,i_{k-1},i_k+1)$ as inner integration. Next consider the left term integral in Eq. (23). We expand it along the unipotent subgroup $\{x_2(y_2)=I_n+y_2e_{i_k,n-1}\}$. There are two terms. In the first, which corresponds to the non trivial terms in the expansion, we deduce as in the case of $\{x_1(y_1)\}$ that the integral $I(i_1,\ldots ,i_{k-1},i_k+2)$ appear as inner integration. In the second term, which corresponds to the trivial term in the expansion, we can further expand along $\{x_3(y_3)=I_n+y_3e_{i_k,n-2}\}$. Continuing this process we get the first above claim, stated before integral (22), regarding the induction process. Notice that the integral $I(i_1,\ldots ,i_{k-1}; \epsilon _{i_k})$ is obtained by taking in each expansion the constant term.

Finally, we need to prove that the integrals $I(\epsilon _{j_1},\ldots , \epsilon _{j_p})$ are zero for all choice of data. But this follows easily from Lemma 1. Indeed, it follows from [4], that this Fourier coefficient corresponds to the following unipotent orbit. Consider the numbers $\{j_1,j_2-j_1,\ldots ,j_p-j_{p-1}, n-j_p\}$. Rearranging them in decreasing order, we obtain a partition $\lambda$ of n whose length is $p+1\le n-m_1+1$. From Lemma 1, and from Eq. (19), we deduce that

$$\begin{aligned} \frac{1}{2}\ \text {dim}\ \lambda + \text {dim} E_\tau (g,s)>\frac{1}{2}(n^2-n) \end{aligned}$$

But, as explained above, this contradicts the dimension Eq. (16). $\square$

3.2 Eisenstein series: the nontrivial case

We keep the notations of the previous Subsection. The second case to consider is integral (15) where the Eisenstein series is the representation $E_\tau (g,\widetilde{s})$ as was defined in the Introduction, and such that the representation $\tau _1$ is a Speh type representation. Then we may assume that the other representation in integral (15), is either a Speh type representation, or it is a similar Eisenstein series denoted by $E_\sigma (g,\widetilde{\nu })$. By that we mean the following. Let R denote a parabolic subgroup of $GL_n$ whose Levi part is $L=GL_{n_1}\times GL_{n_2}\times \cdots \times GL_{n_k}$ with $n_i\ge n_{i+1}$. Let $\sigma _i$ denote an irreducible automorphic representation of $GL_{n_i}({\mathbf A})$, and denote $\sigma = \sigma _1\times \ldots \times \sigma _k$. Form the Eisenstein series $E_\sigma (g,\widetilde{\nu })$ attached to the induced representation $Ind_{R({\mathbf A})}^{GL_n({\mathbf A})}\delta _R^{\widetilde{\nu }}$ where $\widetilde{\nu }$ is a multi complex variable. We also may assume that $\sigma _1$ is a Speh type representation. For if it is the trivial representation, then we may apply the argument of the Sect. 3.1.

With these notations, to prove Theorem 1 in this case we prove,

Proposition 2

With the above notations, let $\pi$ denote a Speh type representation, or assume that $\pi =E_\sigma (g,\widetilde{\nu })$. Then

$$\begin{aligned} \text {dim}\ E_\tau (g,\widetilde{s})+\text {dim}\ \pi >\frac{1}{2}(n^2-n) \end{aligned}$$

In particular, the dimension Eq. (4) does not hold in this case.

Proof

We use Lemma 1. From [4] we deduce that the orbit ${\mathcal O}(E_\tau (g,\widetilde{s}))$ is the suitable induced orbit as defined in [1]. This implies that

$$\begin{aligned} {\mathcal O}(E_\tau (g,\widetilde{s}))={\mathcal O}(\tau _1)+\cdots + {\mathcal O}(\tau _r) \end{aligned}$$

The definition of addition of two partitions is given in [1] as follows. If $\lambda _1=(k_1k_2\ldots k_p)$ and $\lambda _2=(k_1'k_2'\ldots k_q')$, then $\lambda _1+\lambda _2=((k_1+k_1')(k_2+k_2')\ldots )$. Since $\tau _1$ is a representation of Speh type of the group $GL_{m_1}({\mathbf A})$, the length of ${\mathcal O}(\tau _1)$ is not greater than $\frac{m_1}{2}+1$. Since $\frac{m_1}{2}+1< \frac{n}{2}$ and $m_i\le m_1$ for all i, the length of ${\mathcal O}(\tau _i)$ is at most n / 2. From this we deduce that the length of ${\mathcal O}(E_\tau (g,\widetilde{s}))$ is at most n / 2. Similarly, if $\pi =E_\sigma (g,\widetilde{\nu })$ or if $\pi$ is a representation of Speh type then its length is at most n / 2. But every partition of n whose length is at most n / 2 is greater than or equal to $\mu =(2^{\frac{n}{2}})$ if n is even, and $\mu =(2^{\frac{n-1}{2}}1)$ if n is odd. Hence

$$\begin{aligned} \text {dim}\ E_\tau (g,\widetilde{s})+\text {dim}\ \pi = \frac{1}{2}(\text {dim}\ {\mathcal O}(E_\tau (g,\widetilde{s}))+ \text {dim}\ {\mathcal O}(\pi ))\ge \text {dim}\ \mu \ge \frac{n^2-1}{2} \end{aligned}$$

From this the proof follows. $\square$

4 The proof of Theorem 3

In this section we consider the case when $l\ge 3$. Let $\pi _i$ denote l automorphic representations of $GL_n({\mathbf A})$. Assume that $\pi _i=E_{\tau ^{(i)}}(g,\widetilde{s}_i)$ for all $1\le i\le l-1$. Here, the representations $E_{\tau ^{(i)}}(g,\widetilde{s}_i)$ were defined at the Introduction right before Theorem 1. Assume also that $\tau _1^{(i)}$ is the trivial representation for all $1\le i\le l-1$.

The integral we consider is integral (3). We can write it as

$$\begin{aligned} \int \limits _{U_n(F)\backslash U_n({\mathbf A})} E_{\tau ^{(1)}}(u,\widetilde{s}_1)\Phi (u)\psi _U(u)du. \end{aligned}$$

(24)

Here $\Phi (g)=E_{\tau ^{(2)}}(g,\widetilde{s}_2)\ldots E_{\tau ^{(l-1)}}(g,\widetilde{s}_{l-1})\varphi _{\pi _l}(g)$.

Unfold the Eisenstein series. Then carry out the same Fourier expansion process as described in the proof of Proposition 1. Doing so, we deduce that integral (24) is zero for all choice of data if the integrals

$$\begin{aligned} I_\Phi (\epsilon _{j_1},\ldots ,\epsilon _{j_p})= \int \limits _{U_n(F)\backslash U_n({\mathbf A})} \Phi (u)\psi _{U,\epsilon }(u)du \end{aligned}$$

(25)

are all zero for all choice of data. All the notations were defined in the proof of Proposition 1, and we have that $p\le n-m_1^{(1)}$. It follows from the definition of $\psi _{U,\epsilon }$ that this character is not trivial at least on $(n-1)-(n-m_1^{(1)})=m_1^{(1)}-1$ simple roots.

Next, in the integral $I_\Phi (\epsilon _{j_1},\ldots ,\epsilon _{j_p})$ we unfold the Eisenstein series $E_{\tau ^{(2)}}(g,\widetilde{s}_2)$ and repeat this process again. Then we deduce that integrals (25) are zero for all choice of data, if the integrals

$$\begin{aligned} \int \limits _{U_n(F)\backslash U_n({\mathbf A})} \Phi _1(u)\psi _{U,\epsilon '}(u)du \end{aligned}$$

(26)

are zero for all choice of data. Here $\Phi _1(g)=E_{\tau ^{(3)}}(g,\widetilde{s}_3)\ldots E_{\tau ^{(l-1)}}(g,\widetilde{s}_{l-1})\varphi _{\pi _l}(g)$, and $\epsilon '=(\epsilon _1',\ldots ,\epsilon _{n-1}')$ is such that at least $(m_1^{(1)}-1)-(n-m_1^{(2)})=m_1^{(1)}+m_1^{(2)}-n-1$ of the entries are one.

Continuing by induction, we eventually get as inner integrations the integral (26) with $\Phi _1=\varphi _{\pi _l}$ and $\epsilon '$ is a vector with at least $m_1^{(1)}+\ldots +m_1^{(l-1)}-(l-2)n-1$ entries which are equal to one. We conclude that if all such integrals are zero for all choice of data, then the integral (24) is zero for all choice of data.

We have

Corollary 1

For $1\le i\le l$, let $\pi _i$ denote l Eisenstein series $E_{\tau ^{(i)}}(g,\widetilde{s}_i)$ as defined in the Introduction. Assume that for all i the representation $\tau _1^{(i)}$ is the trivial representation. If

$$\begin{aligned} \sum _{i=1}^lm_1^{(i)}\ge n(l-1)+2 \end{aligned}$$

(27)

then the integral (3) is zero for all choice of data.

Proof

Applying the above process, the condition (27) implies that we obtain the integral $\int \psi (r)dr$ as inner integration to each of the integrals of the type of integral (26). Here r is integrated over $F\backslash {\mathbf A}$. From this the Corollary follows. $\square$

Notice that in the above Corollary we did not assume that the dimension Eq. (4) holds.

Let $\pi _i$ denote l automorphic representations of $GL_n({\mathbf A})$. Because of Proposition 2 we may assume that there is at most one representation, denoted by $\pi _l$, such that ${\mathcal O}(\pi _l)\ge \mu =(2^{\frac{n}{2}})$ if n is even, and ${\mathcal O}(\pi _1)\ge \mu =(2^{\frac{n-1}{2}}1)$ if n is odd. This means that all other $l-1$ representations are of the form $E_{\tau ^{(i)}}(g,\widetilde{s}_i)$, with the conditions that $\tau _1^{(i)}$ is the trivial representation and that $m_1^{(i)}> n/2$. Indeed, as argued in Proposition 2 if for some i we have $m_1^{(i)}\le n/2$, then ${\mathcal O}(E_{\tau ^{(i)}}(g,\widetilde{s}_i))\ge \mu$ where $\mu$ was defined above. We have,

Proposition 3

For $1\le i\le l-1$, let $\pi _i$ denote the $l-1$ Eisenstein series defined above. Assume that $\pi _l=E_{\tau ^{(l)}}(g,\widetilde{s}_l)$ and that there is a j such that $\tau _j^{(l)}$ is the trivial representation. Assume also that the dimension Eq. (4) holds for these l representations. Then

$$\begin{aligned} \sum _{i=1}^{l-1}m_1^{(i)}+m_j^{(l)}\ge n(l-1)+2. \end{aligned}$$

In particular, integral (3) is zero for all choice of data.

Proof

For all $1\le i\le l-1$ we have $\text {dim}\ {\mathcal O}(E_{\tau ^{(i)}}(g,\widetilde{s}_i))=m_1^{(i)}(n-m_1^{(i)})+b_i$. We also have $\text {dim}\ {\mathcal O}(E_{\tau ^{(l)}}(g,\widetilde{s}_l))=m_j^{(l)}(n-m_j^{(l)})+b_l$. Here, the $b_i$’s are some non-negative integers. Thus, we can write the dimension Eq. (4) as

$$\begin{aligned} m_j^{(l)}(n-m_j^{(l)})+\sum _{i=1}^{l-1}m_1^{(i)}(n-m_1^{(i)})+A= \frac{1}{2}n(n-1) \end{aligned}$$

where A is a non-negative integer. With these notations, to prove the Proposition, it is enough to prove that the value at the minimum point of the function $\sum _{i=1}^{l-1}m_1^{(i)}+m_j^{(l)}$, subject to the condition $m_j^{(l)}(n-m_j^{(l)})+\sum _{i=1}^{l-1}m_1^{(i)}(n-m_1^{(i)})\le \frac{1}{2}n(n-1)$, is greater or equal to $(l-1)n+2$. Using Lagrange multipliers, it is easy to check that the minimum is obtained when all $m_1^{(i)}$ and $m_j^{(l)}$ are equal. Denote this value by m. Notice, that since $m_1^{(i)}>n/2$, we have $m>n/2$. Thus we need to prove that $lm\ge (l-1)n+2$ if $lm(n-m)\le \frac{1}{2}n(n-1)$ and $m>n/2$. Solving the quadratic inequality, we obtain using that $m>n/2$ that

$$\begin{aligned} lm\ge \frac{ln}{2} +\frac{1}{2}\left[ (l^2-2l)n^2+2ln\right] ^{1/2} \end{aligned}$$

It is easy to check that the right hand side is greater or equal to $(l-1)n+2$. $\square$

Next we consider the case when $\pi _l$ is a representation of Speh type. Thus we assume that $n=pq$ and that ${\mathcal O}(\pi _l)=(p^q)$. Since $(p^q)\ge \mu$, arguing as after Corollary 1, we may assume that for all $1\le i\le l-1$ the representations $\pi _i$ are the Eisenstein series $E_{\tau ^{(i)}}(g,\widetilde{s}_i)$ such that $\tau _1^{(i)}$ is the trivial representation of $GL_{m_1^{(i)}}({\mathbf A})$, and $m_1^{(i)}>n/2$. From Eq. (7) we have $\text {dim}\ \pi _l=\frac{1}{2}n(n-q)$. Hence the dimension equation in this case is

$$\begin{aligned} \frac{1}{2}n(q-1)+\sum _{i=1}^{l-1}m_1^{(i)}(n-m_1^{(i)})+A= \frac{1}{2}n(n-1) \end{aligned}$$

(28)

where A is a non-negative integer. In a similar way as in the previous case, this time we want to minimize $\sum _{i=1}^{l-1}m_1^{(i)}-(l-2)n-1$ subject to the conditions $\sum _{i=1}^{l-1}m_1^{(i)}(n-m_1^{(i)})\le \frac{1}{2}n(q-1)$ and $m_1^{(i)}>n/2$. The first term is the expression which appears in Eq. (27) with $l-1$ instead of l. The above inequality is derived from the Eq. (28). As in Proposition 3, the minimum of the function $\sum _{i=1}^{l-1}m_1^{(i)}-(l-2)n-1$ is derived when all $m_1^{(i)}$ are equal, and so we need to minimize the function $(l-1)m-(l-2)n-1$ with the condition $(l-1)m(n-m)\le \frac{1}{2}n(q-1)$. The solution in m of the quadratic inequality which satisfies $m>n/2$ is

$$\begin{aligned} m=\frac{n}{2}+\frac{1}{2(l-1)}\left[ (l-1)^2n^2-2n(l-1)(q-1) \right] ^{1/2} \end{aligned}$$

Plugging this value in $(l-1)m-(l-2)n-1$ it is easy to prove that it is greater than $n-q+1$.

From all this we deduce that after unfolding the Eisenstein series, as we did in the case of integral (24), we obtain as inner integration to integral (24) the integrals

$$\begin{aligned} \int \limits _{U_n(F)\backslash U_n({\mathbf A})} \varphi _{\pi _l}(u)\psi _{U,\epsilon '}(u)du \end{aligned}$$

(29)

where the vector $\epsilon '$ has at least $n-q+1$ nonzero entries. Hence, it follows from Proposition 5.3 in [4], and from the assumption that ${\mathcal O}(\pi _l)=(p^q)$, that integral (29) is zero for $\psi _{U,\epsilon '}$ as above. Indeed, the Fourier coefficient corresponding to the unipotent orbit $(p^q)$ is given by integral (29) where $\epsilon '$ has exactly $n-q$ nonzero entries. It is not hard to check that if $\epsilon '$ has at least $n-q+1$ nonzero entries, we obtain a Fourier coefficient corresponding to a unipotent orbit which is greater than or not related to $(p^q)$. We summarize,

Proposition 4

For $1\le i\le l-1$, let $\pi _i$ denote the $l-1$ Eisenstein series defined above. Assume that $\pi _l$ is a representation of Speh type. Assume also that the dimension Eq. (4) holds for these l representations. Then the integral (3) is zero for all choice of data.

Acknowledgements

The author is partly supported by the Israel Science Foundation Grant Number 259/14.