1 Introduction

The Bruggeman–Kuznetsov formula [4, 12, 27] is one of the most powerful tools in the analytic theory of automorphic forms on GL(2) and the cornerstone for the investigation of moments of families of L-functions, including striking applications to subconvexity and non-vanishing. It can be viewed as a relative trace formula for the group G=GL(2) and the abelian subgroup U 2×U 2G×G where U 2 is the group of unipotent upper triangular matrices. The Kuznetsov formula in the simplest case is an equality of the shape

(1.1)

where

  • n,m∈ℤ∖{0},

  • δ n,m is the Kronecker symbol,

  • the sum on the left-hand side runs over an orthogonal basis of Hecke–Maaß cusp forms u j for the group SL 2(ℤ) having spectral parameter t j and Hecke eigenvalues λ j (n) for n∈ℕ (and λ j (−n):=±λ j (n) depending on whether u j is even or odd),

  • σ t (n) is the Fourier coefficient of an Eisenstein series defined by

    $$ \sigma_t(n) = \sum_{d_1d_2 = |n|} d_1^{it} d_2^{-it}, $$
  • d spec t=π −2 ttanh(πt) dt is the spectral measure,

  • h is some sufficiently nice, even test function, and

  • h ± is a certain integral transform of h, the sign being sgn(nm), described in (1.2).

There have been many generalizations of the Kuznetsov to other groups of real rank one or products thereof, see e.g. [5, 28, 31], the first of which covers also the groups SL 2(ℂ), SO(n,1) and SU(2,1); see also [11, 15] for interesting applications. For the groups GL(n), n>2, Kuznetsov-type formulae are available [17, Theorem 11.6.19], [36], but they are in considerably less explicit form.

The power of the GL(2) Kuznetsov formula lies in the fact that one can choose arbitrary (reasonable) test functions on either side of the formula, and the relevant integral transforms are completely explicit in terms of Bessel functions. In fact, we have

$$ h^{\pm}(x) = \int_0^{\infty} \mathcal{J}^{\pm}(t, x) h(t)\, d_{\text{spec}} t $$
(1.2)

where

$$ \mathcal{J}^{\pm}(t, x) =2\pi i \sinh(\pi t)^{-1} \left \{ \begin{array}{c} J_{2 i t}(4\pi\sqrt{x}) - J_{-2it}(4\pi\sqrt{x})\\[3pt] I_{2 i t}(4\pi\sqrt{x}) - I_{-2it}(4\pi\sqrt{x}) \end{array} \right \}; $$

this is best understood in terms of its Mellin transform:

$$ \widehat{\mathcal{J}}^{\pm}(t, s) = \int_0^{\infty} \mathcal{J}^{\pm}(t, x) x^{s-1}\, dx = \frac{G_t(s)}{G_t(1-s)} \mp \frac{G_t(1 + s)}{G_t(2 - s)} $$

where

$$ G_t(s) = \pi^{-s} \varGamma \biggl(\frac{s+it}{2} \biggr) \varGamma \biggl(\frac{s-it}{2} \biggr). $$

In addition, this transform can be inverted and is essentially unitary:

$$ h(t) \approx\int_0^{\infty} \mathcal{J}^{\pm}(t, x) h^{\pm}(x) \,\frac{dx}{x}. $$

There is a subtlety, as in the + case the image of the map hh + is not dense, but its complement is well-understood. These formulas together with standard facts about Bessel functions make it possible to apply the Kuznetsov formula in both directions. Unfortunately, such explicit knowledge is not available for GL(n), n≥3.

The aim of this article to provide a “semi-explicit” version of the Kuznetsov formula for GL(3) together with some careful analysis of the various terms occurring on both sides of the formula, and to give some applications in Theorems 1–4 below. On the way we will prove a number of useful auxiliary results for GL(3) Whittaker functions, Eisenstein series and Kloosterman sums that may be helpful for further investigation of GL(3) automorphic forms. The proof of the Kuznetsov formula proceeds along classical lines: we compute the inner product of two Poincaré series in two ways: by spectral decomposition and by unfolding and computing the Fourier expansion of the Poincaré series. The latter has been worked out in great detail in [8].

The spectral side (8.1) of the GL(3) formula consists of three terms:

  • the contribution of the cuspidal spectrum,

  • the contribution of the minimal parabolic Eisenstein series,

  • the contribution of the maximal parabolic Eisenstein series.

The arithmetic side (8.2) contains four terms:

  • the diagonal contribution corresponding to the identity element in the Weyl group,

  • two somewhat degenerate termsFootnote 1 corresponding to and ,

  • the contribution of the long Weyl element.

Interestingly, the two remaining elements in the Weyl group do not contribute as long as n 1,n 2,m 1,m 2 are non-zero; in fact, these two furnish the GL(2) formula which is hidden in the degenerate terms of the Fourier expansion of the Poincaré series. On the arithmetic side, the various variables n 1,n 2,m 1,m 2,D 1,D 2 appearing in the test function are explicitly coupled similarly as on the right hand side of (1.1).

The spectral side (8.1) contains the weight function

$$ h(\nu_1, \nu_2) = \biggl \vert \int _0^{\infty} \int_0^{\infty} \tilde{W}_{\nu _1, \nu_2}(y_1, y_2) F(y_1, y_2)\, \frac{dy_1\, dy_2}{(y_1y_2)^3} \biggr \vert ^2 $$

where \(\tilde{W}_{\nu_{1}, \nu_{2}}\) is a normalized Whittaker function on GL(3) and F is any compactly supported test function (or with sufficiently rapid decay at 0 and ∞ in both variables). In principle this integral transform is invertible: it has been shown in [18] that a natural generalization of the Kontorovich–Lebedev transform inversion formula holds for Whittaker functions on GL(n), hence we have a recipe to find a suitable F to construct our favorite non-negative function h. Proceeding in this way would however considerably complicate the analysis of the arithmetic side, and hence we take a different route which is somewhat less precise, but more convenient for applications. In Proposition 3 below we show roughly the following: taking

$$ F(y_1, y_2) = \bigl(\tau_1 \tau_2( \tau_1+\tau_2) \bigr)^{1/2} f_1(y_1) f_2(y_2) y_1^{ i(\tau_1+2\tau_2)} y_2^{ i(2\tau_1 + \tau_2)} $$

for some fixed functions f 1, f 2 with compact support in (0,∞) yields a non-negative smooth bump function h with h(ν 1,ν 2)≍1 for ν j = j +O(1) and rapid decay outside this range. In other words, h is a good approximation to the characteristic function of a unit square in the (ν 1,ν 2)-plane. Integration over τ 1,τ 2 can now give a good approximation to the characteristic function of any reasonable shape. Passing to a larger region in this way will in fact improve the performance of sum formula and ease the estimations on the arithmetic side.

The test functions on the arithmetic side are completely explicit in (8.3), (8.4) and given as a multiple integral. At least in principle a careful asymptotic analysis should yield a complete description of the behavior of this function, but this seems very complicated. Nevertheless, we are able to give some non-trivial (and in some cases best possible) bounds in Proposition 5 that suffice for a number of applications that we proceed to describe.

The commutative algebra \(\mathcal{D}\) of invariant differential operators of SL 3(ℝ) acting on L 2(SL 3(ℝ)/SO 3) is generated by two elements (see [17, p. 153]), the Laplacian and another operator of degree 3. One class of eigenfunctions of \(\mathcal{D}\) is given by the power functions \(I_{\nu_{1}, \nu_{2}}\) defined in (2.11) below. A Maaß form ϕ for the group SL 3(ℤ) with spectral parameters ν 1,ν 2 is an element in L 2(SL 3(ℤ)∖SL 3(ℝ)/SO 3) that is an eigenfunction of \(\mathcal{D}\) with the same eigenvalues as \(I_{\nu_{1}, \nu_{2}}\) and vanishes along all parabolics, that is,

$$ \int_{(\mathit{SL}_3(\mathbb{Z}) \cup U)\backslash U} \phi(uz)\, du = 0 $$

for and (and then automatically for the minimal parabolic). We choose an orthonormal basis \(\{\phi_{j}\} = \mathcal{B} \subseteq L^{2}(\mathit {SL}_{3}(\mathbb{Z})\backslash\mathit{SL}_{3}(\mathbb{R})/\allowbreak \mathit{SO}_{3})\) of Hecke–Maaß cusp forms (i.e. Maaß forms that are eigenfunctions of the Hecke algebra as in [17, Sect. 6.4]) with spectral parameters \(\nu^{(j)}_{1}, \nu_{2}^{(j)}\). If no confusion can arise, we drop the superscripts (j).

This can be re-phrased in more representation theoretic terms. Let SL 3(ℝ)=NAK be the Iwasawa decomposition where K=SO 3, N is the standard unipotent subgroup and A is the group of diagonal matrices with determinant 1 and positive entries, and let \(\mathfrak{a}\) be the Lie algebra of A. An infinite-dimensional, irreducible, everywhere unramified cuspidal automorphic representation π of \(\mathit{GL}_{3}(\mathbb{A}_{\mathbb {Q}})\) with trivial central character is generated by a Hecke–Maaß form ϕ j for SL 3(ℤ) as above. The local (spherical) representation π is an induced representation from the parabolic subgroup NA of the extension of a character χ:A→ℂ×, \(\text{diag}(x_{1}, x_{2},x_{3}) \mapsto x_{1}^{\alpha_{1}} x_{2}^{\alpha_{2}} x_{3}^{\alpha_{3}}\) with α 1+α 2+α 3=0. In this way we can identify the spherical cuspidal automorphic spectrum with a discrete subset of the Lie algebra \(\mathfrak{a}^{\ast}_{\mathbb{C}}/W\) (W the Weyl group), where we associate to each Maaß form \(\phi_{j} \in\mathcal{B}\) the linear form \(l = (\alpha_{1}, \alpha_{2}, \alpha_{3}) \in \mathfrak{a}^{\ast }_{\mathbb{C}}/W\) that contains the (archimedean) Langlands parameters. A convenient basis in \(\mathfrak{a}^{\ast}_{\mathbb{C}}\) is given by the fundamental weights diag(2/3,−1/3,−1/3), diag(1/3,1/3,−2/3) of SL 3. The coefficients of l=(α 1,α 2,α 3) with respect to this basis can be obtained by evaluating l at the two co-roots \(\text{diag}(1, -1, 0), \text{diag}(0, 1, -1) \in\mathfrak{a}\) and are given by 3ν 1,3ν 2. We then have α 1=2ν 1+ν 2, α 2=−ν 1+ν 2, α 3=−ν 1−2ν 2. With this normalization, ϕ is an eigenform of the Laplacian with eigenvalue

$$ \lambda= 1 - 3\nu_1^2 - 3 \nu_1\nu_2 - 3\nu_2^2 = 1 - \frac{1}{2} \bigl(\alpha_1^2 + \alpha_2^2 + \alpha_3^2 \bigr), $$
(1.3)

and the trivial representation is sitting at (ν 1,ν 2)=(1/3,1/3). The Ramanujan conjecture states that the Langlands parameters α 1,α 2,α 3 of Maaß forms are purely imaginary (equivalently, the spectral parameters ν 1,ν 2 are purely imaginary). A Maaß form is called exceptional if it violates the Ramanujan conjecture. Modulo the action of the Weyl group, we can always assume that ℑν 1,ℑν 2≥0 (positive Weyl chamber). Switching to the dual Maaß form if necessary, we can even assume without loss of generality 0≤ℑν 1≤ℑν 2.

A count of the Maaß forms \(\phi\in\mathcal{B}\) inside the ellipse λT 2 described by (1.3) is referred to as Weyl’s law. The number of such forms is known to be cT 5+O(T 3) for some constant c, see [24, 29]. As a first test case of the Kuznetsov formula we show a result of comparable strength as [24, Proposition 4.5] that turns out to be a simple corollary of the Kuznetsov formula. A similar upper bound has recently been proved by X. Li [26]. Let \(L(\phi\times \tilde{\phi}, s)\) be the Rankin–Selberg L-function (see (4.2) below). Then the following weighted count of the cuspidal spectrum in a small ball of radius O(1) in \(\mathfrak{a}_{\mathbb{C}}^{\ast}\) holds.

Theorem 1

There are absolute constants c 1,c 2>0, T 0,K≥1 with the following property: for all T 1,T 2T 0 we have

$$ c_1 T_1T_2(T_1+T_2) \leq \sum_{\substack{\scriptsize | \nu^{(j)}_1 -iT_1| \leq K\\ | \nu^{(j)}_2 - iT_2| \leq K}} \bigl(\underset{s=1}{\mathrm{res}} L( \phi_j \times\tilde{\phi}_j, s) \bigr)^{-1} \leq c_2 T_1T_2(T_1+T_2). $$

It is standard to estimate the residue from above, but due to possible Siegel zeros a good lower bound is not known. If ϕ=sym2 u for some Hecke–Maaß form \(u \in L^{2}(\mathit{SL}_{2}(\mathbb {Z})\backslash\mathfrak{h}^{2})\) with spectral parameter νiℝ, then Ramakrishnan and Wong [30] have shown that no Siegel zeros exist:

$$ \underset{s=1}{\mathrm{res}} L \bigl(\text{sym}^2u \times \text{sym}^2u, s \bigr) = \bigl(1+|\nu|\bigr)^{o(1)}. $$

In general we will only be able to prove the following bounds: if ϕ has spectral parameters ν 1,ν 2, then setting C:=(1+|ν 1+ν 2|)(1+|ν 1|)(1+|ν 2|) we have

$$ C^{-1}\ll \underset{s=1}{\mathrm{res}} L(\phi\times \tilde{ \phi}, s) \ll C^{\varepsilon}. $$
(1.4)

In particular it follows (after possibly enlarging the constant K in Theorem 1) that in each ball inside \(i \mathfrak{a}^{\ast}\) of sufficiently large constant radius, there exist cusp forms. We will prove (1.4) in Lemma 2 below.

Miller [29] proved that almost all forms are non-exceptional, that is, the number of exceptional forms \(\phi_{j} \in\mathcal{B}\) with λ j T 2 is o(T 5). This was, among other things, strengthened in [24] to O(T 3). By unitaricity and the standard Jacquet–Shalika bounds towards the Ramanujan conjectureFootnote 2 (cf. (2.4) below) the spectral parameters ν 1,ν 2 of an exceptional Maaß form are of the form (assuming 0≤ℑν 1≤ℑν 2)

$$ (\nu_1, \nu_2) = (2\rho/3, -\rho/3 + i\gamma), \quad \gamma\geq0, \ |\rho| \leq1/2, $$

see (2.8) below. It is an easy corollary of Theorem 1 that there are O(T 2+ε) exceptional eigenvalues with γ=T+O(1), but more can be shown which can be viewed as a density theorem for exceptional eigenvalues and interpolates nicely between the Jacquet–Shalika bounds and the tempered spectrum.

Theorem 2

For any ε>0 we have

$$ \sum_{\substack{ \phi_j\ \mathit{exceptional}\\ \gamma_j = T+O(1)}} T^{4|\rho_j|} \ll_{\varepsilon} T^{2+\varepsilon}. $$

Next we prove a large sieve type estimate for Hecke eigenvalues. Let A j (n,1) denote the Hecke eigenvalues of the Hecke–Maaß cusp form ϕ j .

Theorem 3

Let N≥1, T 1,T 2T 0 sufficiently large, and let α(n) be a sequence of complex numbers. Then

$$ \sum_{\substack{T_1 \leq|\nu^{(j)}_1| \leq2T_1\\ T_2 \leq |\nu ^{(j)}_2| \leq2 T_2}} \bigg|\sum _{n \leq N} \alpha(n) A_j(n, 1) \bigg|^2 \ll_{\varepsilon} \bigl(T_1^2T_2^2(T_1+T_2) + T_1T_2N^2 \bigr)^{1+\varepsilon} \|\alpha \|_2^2 $$
(1.5)

for any ε>0 whereα2=(∑ n |α(n)|2)1/2.

The first term is optimal on the right hand side is optimal. Most optimistically one could hope for an additional term of size N (instead of T 1 T 2 N 2), but in any case our result suffices for an essentially optimal bound of the second moment of a family of genuine GL(3) L-functions. This seems to be the first bound of this kind in the literature. For large sieve inequalities in the level aspect (with very different proofs) see [14, Theorem 4] and [35].

Theorem 4

For T≥1 and any ε>0 we have

$$ \sum_{ T \leq|\nu^{(j)}_1|, |\nu^{(j)}_2| \leq 2T} \big|L(\phi_j, 1/2) \big|^2 \ll_{\varepsilon} T^{5+\varepsilon}. $$

More applications of the GL(3) Kuznetsov formula to the Sato–Tate distribution of GL(3) Hecke eigenvalues and a version of Theorem 2 for the Langlands parameters at finite places will be given in a forthcoming paper [2, Theorems 1–3].

After the paper was submitted, two other interesting approaches to the GL(3) Kuznetsov formula have been developed independently by Buttcane [10] and Goldfeld–Kontorovich [19]. The present technique, however, gives the strongest bounds for the Kloosterman terms in the Kuznetsov formula which are indispensable for applications to L-functions as in Theorems 3 and 4. One may compare, for instance, with [19] for which the reader is referred to the appendix which features in Theorem 5 another result of independent interest.

It would be very interesting to generalize the present results and techniques to congruence subgroups of SL 3(ℤ) of the type

The analytic parts of the present argument (in particular the bounds for Whittaker functions and the corresponding integral transforms) work without any change. One needs a more general Bruhat decomposition to calculate the Fourier expansion of the relevant Poincaré series, and it would be useful to have an explicit spectral decomposition for the space \(L^{2}(\varGamma_{0}(q)\backslash \mathfrak{h}^{3})\). This along with further applications will be addressed in [1].

2 Whittaker functions

Let ν 1,ν 2∈ℂ. We introduce the notation

$$ \nu_0 := \nu_1 + \nu_2 $$
(2.1)

and (as in the introduction)

$$ \alpha_1 = 2\nu_1 +\nu_2, \qquad\alpha_2 = -\nu_1 + \nu_2, \qquad \alpha_3 = -\nu_1 - 2\nu_2. $$
(2.2)

The transformations

(2.3)

leave {α 1,α 2,α 3} invariant, and they also leave {|ℑν 0|,|ℑν 1|,|ℑν 2|} invariant. For convenience we assume the Jacquet–Shalika bounds towards the Ramanujan conjecture

$$ \max\bigl(|\Re\alpha_1|, |\Re\alpha_2|, | \Re \alpha_3|\bigr) \leq \frac{1}{2}, $$
(2.4)

and we always assume unitaricity

$$ \{\alpha_1, \alpha_2, \alpha_3 \} = \{-\overline{\alpha_1}, - \overline { \alpha_2}, - \overline{\alpha_3}\}. $$
(2.5)

It is elementary to deduce from (2.4) that

$$ \max\bigl(|\Re\nu_0|, |\Re\nu_1|, | \Re \nu_2|\bigr) \leq\frac{1}{3} $$
(2.6)

and to deduce from (2.5) that

$$ \nu_0, \nu_1, \nu_2, \alpha_1, \alpha_2, \alpha_3 \in i\mathbb{R} $$
(2.7)

or

(2.8)

with ρ,γ∈ℝ and |ρ|≤1/2 by (2.4). The choice

$$ \begin{array}{l} (\alpha_1, \alpha_2, \alpha_3) = (\rho+i\gamma, -\rho+i\gamma, -2 i\gamma), \\[6pt] (\nu_1, \nu_2, \nu_0) = (2\rho/3, - \rho/3 + i\gamma, \rho/3 + i \gamma) \end{array} $$
(2.9)

is unique if we require ℑν 2≥ℑν 1≥0, γ≥0.

Let

The group SL 3(ℤ) acts faithfully on \(\mathfrak {h}^{3}\) by left multiplication.

The Whittaker function \(\mathcal{W}^{\pm}_{\nu_{1}, \nu_{2}} : \mathfrak {h}^{3} \rightarrow\mathbb{C}\) is given byFootnote 3 (analytic continuation in ν 1,ν 2 of)

(2.10)

with

$$ I_{\nu_1, \nu_2}(z) = y_1^{1 + 2\nu_1 + \nu_2} y_2^{1+\nu_1+2\nu_2}. $$
(2.11)

Compared to [7]Footnote 4 we have re-normalized the indices ν j →1/3+ν j . By the formula of Takhtadzhyan–Vinogradov we have \(\mathcal{W}^{\pm}_{\nu_{1}, \nu_{2}}(z) = e(x_{1} \pm x_{2}) W_{\nu_{1}, \nu_{2}}(y_{1}, y_{2})\) whereFootnote 5

(2.12)

It is convenient to slightly re-normalize this function: let

$$ c_{\nu_1, \nu_2} := \pi^{-3\nu_0} \prod _{j=0}^2 \varGamma \biggl(\frac {1}{2} + \frac{3}{2} \nu_j \biggr) \biggl \vert \varGamma \biggl( \frac{1}{2} + \frac {3}{2} i \Im\nu_j \biggr) \biggr \vert ^{-1} $$
(2.13)

and

If ν 1,ν 2iℝ, this changes the original Whittaker function only by a constant on the unit circle, in the situation (2.9) it changes the order of magnitude by a bounded factor. Often the Whittaker function is defined entirely without the normalizing Gamma-factors in the denominator of (2.12) in which case it is often referred to as the completed Whittaker function. It is convenient not to work with the completed Whittaker function here, see Remark 3 below. (Of course, \(\tilde{W}_{\nu_{1}, \nu_{2}}\) is not analytic in the indices any more.)

The GL(2)-analogue of this function is

$$ W_{\nu}(y) := 2 \pi^{\frac{1}{2} } \biggl \vert \varGamma \biggl(\frac{1}{2} + \nu \biggr) \biggr \vert ^{-1} \sqrt{y} K_{\nu}(2 \pi y), $$
(2.14)

see [17, p. 65].

We proceed to collect analytic information on the GL(3) Whittaker function. We have the double Mellin inversion formula ([17, p. 155], [7, (10.1)])

(2.15)

This implies in particular that \(\tilde{W}_{\nu_{1}, \nu_{2}}\) is invariant under the transformations (2.3) which is the reason for including the normalization constant \(c_{\nu_{1}, \nu_{2}}\). Note that

$$ \tilde{W}_{\nu_1, \nu_2}(y_1, y_2) = \tilde{W}_{\nu_2, \nu_1}(y_2, y_1) = \overline{ \tilde{W}_{\overline{\nu_1}, \overline{\nu_2}}(y_1, y_2)}. $$
(2.16)

Uniform bounds for Bessel functions are rare in the literature, but frequently needed in the GL(2) theory. We are not aware of any uniform bound for a GL(n) Whittaker function with n>2. Although the proofs of Theorems 1–5 do not require bounds for individual Whittaker functions, we record here for future reference the following uniform result.

Proposition 1

Let ν 1,ν 2∈ℂ satisfy (2.6)(2.8) and write θ=max(|ℜα 1|,|ℜα 2|,|ℜα 3|)≤1/2. Let θ<σ 1<σ 2 and ε>0. Then for any σ 1c 1,c 2σ 2 we have

Remark 1

This result can be refined somewhat, in particular for small and large y 1,y 2. In the “transitional range” it is not too far from the truth. For instance, if ν 1=ν 2=iT are large (and purely imaginary) and \(y_{1} = y_{2} = \frac{3}{2\pi}T - \frac{1}{100} T^{1/3}\), then the integral in (2.12) is non-oscillating, and it follows from (2.12) and known properties of the K-Bessel function that \(\tilde{W}_{\nu_{1}, \nu_{2}}(y_{1}, y_{2}) \asymp y_{1}y_{2}/T^{4/3}\), whereas our bound gives \(\tilde{W}_{\nu_{1}, \nu_{2}}(y_{1}, y_{2}) \ll y_{1}y_{2}T^{\varepsilon- \frac{1}{2}}\).

Proof

Let us first assume that ν 1,ν 2 are purely imaginary; we write \(\tilde{\nu}_{j} := \Im\nu_{j}\). By (2.16) and the invariance under (2.3) we can assume without loss of generality that

$$ 0 \leq \tilde{\nu}_1 \leq\tilde{\nu}_2. $$
(2.17)

By (2.15) and Stirling’s formula, we have

(2.18)

for σ 1<c 1,c 2<σ 2. It is elementary to check that

$$ \frac{3\pi}{4}\sum_{j=0}^2| \nu_j| - \frac{\pi}{4}\sum_{j=1}^3|it_1 + \alpha_j| - \frac{\pi}{4}\sum_{j=1}^3|it_2 - \alpha_j| + \frac{\pi}{4} |t_1+t_2| \leq0 $$
(2.19)

with equality if and only if

$$ \tilde{\nu}_1-\tilde{\nu}_2 \leq t_1 \leq\tilde{\nu}_1 + 2\tilde{\nu }_2, \qquad\tilde{\nu}_2 -\tilde{\nu}_1 \leq t_2 \leq2\tilde{\nu}_1 + \tilde{\nu}_2, $$
(2.20)

or

$$ -2\tilde{\nu}_1-\tilde{\nu}_2 \leq t_1 \leq\tilde{\nu}_1 - \tilde{\nu }_2, \qquad-\tilde{\nu}_1 -2\tilde{\nu}_2 \leq t_2 \leq\tilde{\nu}_2 - \tilde{\nu}_1. $$
(2.21)

For a<b let w a,b (t) be defined by

$$ w_{a, b} := \min \bigl(1, e^{-\frac{\pi}{4}(t-b)}, e^{-\frac{\pi}{4}(a-t)} \bigr); $$

then the exp-factor in (2.18) is bounded by

$$ w_{ \tilde{\nu}_1-\tilde{\nu}_2, \tilde{\nu}_1 + 2\tilde{\nu}_2}(t_1) w_{ \tilde{\nu}_2 -\tilde{\nu}_1, 2\tilde{\nu}_1 + \tilde{\nu}_2}(t_2) + w_{ -2\tilde{\nu}_1-\tilde{\nu}_2, \tilde{\nu}_1 - \tilde{\nu}_2}(t_1) w_{ -\tilde{\nu}_1 -2\tilde{\nu}_2 , \tilde{\nu}_2 - \tilde{\nu}_1}(t_2), $$

and hence we have

We consider only the first summand in the first line of the preceding display. The second summand is similar. By a shift of variables, the first term equals

(2.22)

It is straightforward to estimate this expression. For convenience we provide the details. We recall our assumption (2.17) and split the t 1,t 2 integration into several ranges. Let \(\tilde{\nu}_{1} \leq R \leq\tilde {\nu_{2}}\), and define

We estimate the double integral in all 6 ranges for c 1,c 2>0:

Combining all 6 previous bounds, and summing over dyadic numbers R, we obtain the bound of the proposition if ν 1,ν 2iℝ.

It remains to consider the situation (2.8). The exponential factor does not change, but the fraction in the first line of (2.18) becomes now

(2.23)

This is independent of ρ, and the same calculation goes through. □

We recall an important formula of Stade [33] (cf. also (2.16) and observe that Stade’s definition [33, (1.1)] of the Whittaker function has ν 1 and ν 2 interchanged, and his Whittaker function is, up to Gamma-factors, twice our Whittaker function).

Proposition 2

Let ν 1,ν 2,μ 1,μ 2,s∈ℂ. We use the notation (2.1) and (2.2), and define similarly μ 0 and β 1,β 2,β 3 in terms of μ 1,μ 2. Then we have an equality of meromorphic functions in s:

3 Integrals over Whittaker functions

For our purposes it is convenient to consider the double Mellin transform of the product of \(\tilde{W}_{\nu_{1}, \nu_{2}}\) with some rapidly decaying function. We are not aware of any explicit formula in the literature, but the next proposition gives an asymptotic result which is sufficient for our purposes. This is one of the key ingredients in this paper, and therefore we present all details of the lengthy proof.

Proposition 3

Let A,X 1,X 2≥1, τ 1,τ 2≥0, and assume that τ 1+τ 2 is sufficiently large in terms of A. Let ν 1,ν 2∈ℂ satisfy (2.6)(2.8) and in additionν 1,ℑν 2≥0. Let

$$ t_1 = \tau_1 + 2\tau_2, \qquad t_2 = 2\tau_1 + \tau_2. $$

Fix two non-zero smooth functions f 1,f 2:(0,∞)→[0,1] with compact support. Let ε>0 and let

If X 1=X 2=1, then

(3.1)

Moreover, there is a constant c depending on f 1,f 2 such that the following holds: if

$$ X_1 = X_2 = 1, \qquad\tau_1, \tau_2 \gg 1, \qquad |\Im\nu_1 - \tau_1| \leq c, \qquad|\Im\nu_2 - \tau_2| \leq c, $$
(3.2)

then

$$ I \asymp \prod_{j=0}^2 \bigl(1+| \nu_j| \bigr)^{-1/2}; $$
(3.3)

and if ν 1,ν 2 are given by (2.9) and in addition

$$ X_2 \gg1, \qquad|\rho| \geq\varepsilon, \qquad \gamma\gg (X_1X_2)^{\varepsilon}, \qquad|\gamma- \tau_2| \leq c, \qquad|\tau_1| \leq c, $$
(3.4)

then

$$ I \asymp X_1 X_2^{1+ |\rho|} \prod _{j=0}^2 \bigl(1+|\nu_j| \bigr)^{-1/2}. $$
(3.5)

All implied constant depend at most on A,ε,f 1,f 2 and the signshould be interpreted as “up to a sufficiently large constant”.

Remark 2

This should be roughly interpreted as follows: given t 1,t 2,X 1,X 2 as above, I as a function of ν 1,ν 2∈{z∈ℂ:|ℑz|≤1/2} is under some technical assumptions a function with a bump at ℑν 1=τ 1 and ℑν 2=τ 2 of size \(X_{1} X_{2}^{1+\max_{j}|\Re\alpha_{j}|}(\nu_{0}\nu_{1}\nu_{2})^{-1/2}\) with rapid decay away from this point. Most of the time we will put X 1=X 2=1. Only if we need a test function that blows up at exceptional eigenvalues we will choose X 2 to be large. The asymmetry in X 1,X 2 in (3.4) and (3.5) is due to the special choice (2.9).

Proof

By Parseval’s formula and (2.15) the double integral in question equals

(3.6)

Let us first assume that

$$ |\nu_1| + |\nu_2| \leq\frac{1}{100}( \tau_1 + \tau_2). $$
(3.7)

In this case the conditions (3.2) and (3.4) are void, so we only need to show (3.1) and take X 1=X 2=1. We apply Stirling’s formula to the Γ-quotient. We argue as in the proof of Proposition 1, see (2.19) and the surrounding discussion. The exponential part is given by

As before let us write \(\tilde{\nu}_{j} = \Im\nu_{j}\) and assume without loss of generality (2.17). Using (2.20) and (2.21) together with the rapid decay of \(\widehat{f}_{1}\) and \(\widehat{f}_{2}\) it is easy to see that by our present assumption (3.7) we can bound I by

$$ \ll_{A, f_1, f_2} \,(t_1+t_2)^{-A}. $$

In the range (3.7) this is acceptable for (3.1).

Let us now assume

$$ |\nu_1| + |\nu_2| \geq \frac{1}{100}(\tau_1 + \tau_2). $$
(3.8)

We want to shift the two contours in (3.6) to −∞. To check convergence, we first shift the u 1-integral to ℜu 1=−2A−1 for some large integer A. We observe that

$$ \widehat{f}(s) \ll_{B, f} |s|^{-B} C^{\Re s+B} $$

for ℜs>0, any B≥0 and some constant C>0 depending only on f (one can take \(C := \sup\{x > 0 \mid f(x) \not= 0\}\)). We also recall that the reflection formula for the Gamma-function implies the uniform bound

$$ \varGamma(-s) \ll e^{-\frac{\pi}{2}|\Im s|} \biggl(\frac{|s|}{e} \biggr)^{-\Re s - \frac{1}{2}} $$

for ℜs>0, min n∈ℤ(ℜsn)>1/50. It is now easy to see that the remaining u 1-integral for A→∞ vanishes, and we are left with the sum over the residues. Next we shift in the same way the u 2-integral to −∞, and express (3.6) as an absolutely convergent double sum over residues. Let us first assume that \(\nu_{1} \not=0\) so that α 1,α 2,α 3 are pairwise distinct. For j∈{1,2,3} we denote by k,l two integers such that {j,k,l}={1,2,3}. Similarly for r∈{1,2,3} let s,t be such that {r,s,t}={1,2,3}. Then (3.6) equals

(3.9)

We can bound the second and third line of (3.9) by

$$ \sum_{n, m} \ll_{A, f_1, f_2} X_1^{1-\Re\alpha_j} X_2^{1+\Re\alpha_r} \bigl(1+| \alpha_j + it_1| \bigr)^{-A} \bigl(1+| \alpha_r-it_2| \bigr)^{-A}. $$
(3.10)

Here we have used that by (2.2) we have the following equality of multisets:

(3.11)

for a certain choice of signs (depending on j,r) whenever \(j \not= r\). In addition we see that we have in the special case j=3, r∈{1,2} (that is, α j =−ν 1−2ν 2, α r =2ν 1+ν 2 or −ν 1+ν 2)

$$ \sum_{(n, m) \not= (0, 0)} \ll_{A, f_1, f_2} X_1^{1-\Re\alpha_j} X_2^{1+\Re\alpha_r} \frac{(1+|\alpha_j + it_1|)^{-A} (1+|\alpha_r-it_2|)^{-A} }{\min ( (1+ |\nu_1|)X_2^2, (1+ |\nu_2|)X_1^2) }. $$
(3.12)

We will now carefully analyze all 6 terms 1≤α j ,α r ≤3, \(j \not= r\) in the main term under the assumption τ 1,τ 2≥0,ℑν 2≥ℑν 1≥0 and show that they all satisfy the bound (3.1). Moreover, under the assumption (3.2), the term j=3, r=1 is of order of magnitude (3.3) and dominates all other terms. Similarly we will show (3.5). We will first make the extra assumption

$$ |\nu_1| \geq\varepsilon. $$

This ensures that α 1,α 2,α 3 are not too close together (note that |ν 2| must be large by (3.8)). By Stirling’s formula, (3.11) and (2.9) (in the non-tempered case),

(3.13)

The non-negativity of f 1,f 2 implies that the absolute values of the Mellin transforms \(|\widehat{f}_{1}(-1+\alpha_{3} + it_{1})|, |\widehat{f}_{2}(-1-\alpha_{1}+ i t_{2})|\) are bounded from below in the range (3.2) and (3.4) if c is sufficiently small. Combining (3.13) and (3.12), we see that under the assumption (3.2) the term j=3, r=1 satisfies (3.3). Note that (3.2) forces |ν 1|,|ν 2| to be sufficiently large and excludes (2.8). Similarly, under the assumption (3.4) the term j=3, r=1 (if ρ>0) or the term j=3, r=2 (if ρ<0) satisfies (3.5), whereas the other term is of smaller order of magnitude. This is also consistent with (3.1).

It remains to show that all other terms satisfy (3.1), and are of lesser order of magnitude than (3.3) and (3.5) under the respective conditions. Under the assumption (3.2) all 5 terms \((j, r) \not= (3, 1)\) satisfy |α j +it 1|+|α r it 2|≥3min(|ℑν 1|,|ℑν 2|)+O(1) and can therefore be bounded by (recall (2.17))

$$ \ll_{\varepsilon, A, f_1, f_2} |\nu_1|^{-A} \prod _{n=0}^2 \bigl(1+|\nu_n| \bigr)^{-\frac{1}{2}} $$

which is dominated by (3.3). Similarly, under the assumption (3.4) the 4 terms \((\alpha_{j}, \alpha_{r}) \not\in\{ (3, 1), (3, 2)\}\) satisfy |α j +it 1|+|α r it 2|≥3|ℑν 2|+O(1) and can therefore be bounded by

$$ \ll_{\varepsilon, A, f_1, f_2} |\nu_2|^{-A} \prod _{n=0}^2 \bigl(1+|\nu_n| \bigr)^{-\frac{1}{2}} $$

which is again dominated by (3.5). We proceed now to show (3.1) for X 1=X 2=1. It follows from (3.10) and (3.13) that all 6 terms \(\alpha_{j} \not= \alpha_{r}\) contribute

to the main term. This is in agreement with (3.1) if we can show

$$ |\Im\alpha_j + t_1| + |\Im\alpha_r - t_2 | \geq \frac{1}{2} \bigl(|\tilde{ \nu}_1 - \tau_1| + |\tilde{\nu}_2 - \tau_2| \bigr). $$

This is clear for j=1 or r=3 by the positivity assumption \(\tilde{ \nu}_{1}, \tilde{\nu}_{2}, \tau_{1},\allowbreak \tau_{2} \geq0\) (recall the notation \(\tilde{\nu}_{j} = \Im\nu_{j}\)), even without the factor 1/2. We check the other 3 cases. In the case j=3,r=1 we have again the stronger inequality

$$ |{-}\tilde{\nu}_1 - 2\tilde{\nu}_2 + \tau_1 + 2 \tau_2| + |2\tilde{ \nu }_1 + \tilde{ \nu}_2 - 2\tau_1- \tau_2| \geq| \tau_1 -\tilde{ \nu}_1| + |\tau_2 - \tilde{ \nu}_2| $$

which follows from the easy to check inequality |a|+|b|≤|a+2b|+|b+2a|. In the case j=3, r=2 we need to show

$$ |{-}\tilde{ \nu}_1 -2\tilde{ \nu}_2 + \tau_1 + 2 \tau_2| + |{-}\tilde{ \nu }_1 +\tilde{ \nu}_2 - 2\tau_1- \tau_2| \geq \frac{1}{2} \bigl(| \tau_1 -\tilde{\nu}_1| + | \tau_2 - \tilde{ \nu}_2| \bigr). $$

If \(\tau_{1} -\tilde{ \nu}_{1}\) and \(\tau_{2}-\tilde{ \nu}_{2}\) are of the same sign, the first term dominates the right hand side; if \(\tau_{1} - \tilde {\nu}_{1} \leq0, \tau_{2} - \tilde{ \nu}_{2} \geq0\), the second term dominates the right hand side; in either case we do not need the factor 1/2. Finally if \(\tau_{1} - \tilde{\nu}_{1} \geq0\), \(\tau_{2} - \tilde{\nu }_{2} \leq0\), we distinguish the two cases \(\tau_{1} - \tilde{\nu}_{1}\) greater or smaller than \(\tilde{\nu}_{2} - \tau_{2}\): in the former case the second term dominates the right hand side, because \(|{-}\tilde{\nu}_{1} +\tilde{\nu}_{2} - 2\tau_{1}- \tau_{2}| \geq2(\tau_{1} - \tilde{\nu}_{1}) - (\tilde{\nu}_{2} - \tau_{2})\), and in the latter the first term dominates the right hand side, because \(|{-}\tilde{ \nu}_{1} -2\tilde{ \nu}_{2} + \tau_{1} + 2 \tau_{2}| \geq2(\tilde{\nu}_{2} -\tau_{2}) - (\tau_{1} - \tilde{\nu }_{1})\). Finally the case j=2,r=1 amounts to showing

$$ |{-}\tilde{ \nu}_1 +\tilde{\nu}_2 + \tau_1 + 2 \tau_2| + |2\tilde{ \nu }_1 +\tilde{\nu}_2 - 2\tau_1- \tau_2| \geq\frac{1}{2} \bigl(| \tau_1 -\tilde{\nu}_1| + |\tau_2 - \tilde{ \nu}_2| \bigr) $$

which can be seen as above after interchanging indices.

Finally we need to treat the case \(0 \not= |\nu_{1}| < \varepsilon\) and |ν 2|≫|τ 1|+|τ 2|. Here the condition (3.4) is empty, and if τ 1,τ 2 are sufficiently large, the condition (3.2) is also empty, so we only need to show the upper bound (3.1) for X 1=X 2=1. We return to (3.9) and partition the 6 terms (j,r) into three pairs

$$ \bigl\{(3, 2), (3, 1) \bigr\}, \quad \bigl\{(2, 3), (1, 3) \bigr\}, \quad \bigl \{(2, 1), (1, 2) \bigr\}. $$

The contribution of the first pair is

For |ν 1|<ε the second line can be bounded by the mean value theorem. Then we use the functional equation (s)=Γ(s+1) of the Gamma-function in connection with Stirling’s formula as before and bound the preceding display by

and argue as before. The same argument with different indices works for the pair {(2,3),(1,3)}. The last pair is only a small variation; its contribution is given by

For small ν 1, this can again be estimated by the mean value theorem giving the crude bound

$$ \ll_{A, f_1, f_2} \bigl(| \nu_2 + it_1| + | \nu_2 -i t_2|\bigr)^{-A} \bigl(1+|\nu_2|\bigr)^{O(1)} $$

which is admissible for (3.1). This completes the proof of the proposition under the additional assumption that α 1, α 2, α 3 are pairwise distinct, that is \(\nu_{1} \not= 0\). The case ν 1=0 follows by continuity. □

An inspection of the proof, in particular (3.9)–(3.13), shows that for τ 1, τ 2 sufficiently large one has

(3.14)

for ν 1,ν 2iℝ in a neighborhood of 1, 2, respectively, and it is very small outside this region.

4 Maass forms

Let Γ=SL 3(ℤ). We denote by

the maximal parabolic subgroup, and by

the standard unipotent group. Analogously, let .

A Maaß cusp form \(\phi: \varGamma\backslash\mathfrak{h}^{3} \rightarrow\mathbb{C}\) with spectral parameters ν 1,ν 2 (that is, of type (1/3+ν 1,1/3+ν 2) in the notation of [17]) for the group Γ has a Fourier expansion of the type

(4.1)

with \(\mathcal{W}^{\pm}_{\nu_{1}, \nu_{2}}\) as in (2.10) and \(c_{\nu_{1}, \nu_{2}}\) as in (2.13). The Fourier coefficients are given by

We have

$$ A_{\phi}(m_1, m_2) = A_{\phi}(m_1, -m_2), $$

see [17, Proposition 6.3.5]. Hence one can alternatively write the Fourier expansion as a sum over m 1,m 2≥1, γU 2GL 2(ℤ). We will use this observation in the proof of Lemma 1.

It is expected that ν 1,ν 2 are imaginary, but we certainly know that (2.6)–(2.8) hold. If ϕ is an eigenfunction of the Hecke algebra (see [17, Sect. 6.4]), we define its L-function by L(ϕ,s):=∑ m A ϕ (1,m)m s, and the Rankin–Selberg L-function by

$$ L(\phi\times\tilde{\phi}, s) := \zeta(3s) \sum _{m_1, m_2} \frac{|A_{\phi}(m_1, m_2)|^2}{m_1^{2s} m_2^{s}}. $$
(4.2)

It follows from [25, Theorem 2] or [6, Corollary 2] that the coefficients are essentially bounded on average, uniformly in ν:

$$ \sum_{m \leq x} \big|A_{\phi}(m, 1)\big|^2 \ll x \bigl(x\bigl(1+|\nu_1|+|\nu_2|\bigr) \bigr)^{\varepsilon}. $$
(4.3)

The space of cusp forms is equipped with an inner product

$$ \langle\phi_1, \phi_2 \rangle:= \int _{\varGamma\backslash\mathfrak {h}^3}\phi_1(z) \overline{\phi_2(z)} \,dx_1\, dx_2\, dx_3\, \frac{dy_1\, dy_2}{(y_1y_2)^3}. $$

It is known that \(L(\phi\times\tilde{\phi}, s)\) can be continued holomorphically to ℂ with the exception of a simple pole at s=1 whose residue is proportional to ∥ϕ2 [17, Theorem 7.4.9]. The proportionality constant is given in the next lemma.

Lemma 1

For a Hecke eigenform ϕ as in (4.1) with A ϕ (1,1)=1 we have \(\|\phi\|^{2} \asymp \operatorname{\mathrm{res}}_{s=1} L(\phi\times\tilde{\phi}, s)\).

Remark 3

This lemma shows that the normalization of the Whittaker functions \(\tilde{W}_{\nu_{1}, \nu_{2}}\) is well chosen in the sense that an arithmetically normalized cusp form A ϕ (1,1)=1 should roughly have norm 1. The main point is that \(\tilde{W}_{\nu_{1}, \nu_{2}}\) has roughly norm 1 with respect to the inner product

$$ (f, g) := \int_0^{\infty} \int _0^{\infty} f(y_1, y_2) \overline{g(y_1, y_2)} \det \left(\begin{array}{c@{\quad}c@{\quad}c} y_1y_2 & & \\ & y_1 & \\ & & 1\end{array} \right) \,\frac{dy_1 \, dy_2}{(y_1y_2)^3}. $$

Proof

This is standard Rankin–Selberg theory. We use the maximal parabolic Eisenstein series

$$ E(z, s; \textbf{1}) := \sum_{\gamma\in P \backslash\varGamma} \det (\gamma z)^s = \frac{1}{2} \sum_{\gamma\in P_1 \backslash\varGamma} \det (\gamma z)^s , \quad\Re s > 1. $$

It follows from (5.7) below that

$$ \| \phi\|^2 = \frac{3\zeta(3)}{2\pi} \underset{s=1}{\text{res}} \bigl \langle\phi, \phi E(., \bar{s}, \textbf{1}) \bigr\rangle. $$

We follow the unfolding argument of [17, pp. 227–229] and [16, Sect. 3]. Unfolding the Eisenstein series, we see

$$ \bigl\langle\phi, \phi E(., \bar{s}, \textbf{1}) \bigr\rangle= \frac{1}{2} \int_{P_1\backslash\mathfrak{h}^3} \big|\phi(z)\big|^2 \bigl(y_1^2y_2 \bigr)^s \,dx_1\, dx_2\, dx_3\, \frac{dy_1\, dy_2}{(y_1y_2)^3}. $$

Let \(\mathcal{F}\) denote a fundamental domain for , and let . Then \(P_{1}\backslash\mathfrak {h}^{3}\) is in 2-to-1 correspondence with \(\widetilde{\mathit {GL}}_{2}(\mathbb{Z}) \backslash\mathcal{F}\). Inserting the Fourier expansion of one factor and unfolding once again, we obtain

The lemma follows now easily from Stade’s formula. □

We are now ready to prove (1.4).

Lemma 2

For an arithmetically normalized Hecke–Maaß cusp form ϕ with spectral parameters ν 1, ν 2 as above we have

$$ \bigl(\bigl(1+|\nu_0|\bigr) \bigl(1+|\nu_1|\bigr) \bigl(1+|\nu_2|\bigr) \bigr)^{-1} \ll\| \phi\|^2 \ll_{\varepsilon} \bigl(\bigl(1+| \nu_0|\bigr) \bigl(1+|\nu_1|\bigr) \bigl(1+|\nu_2|\bigr) \bigr)^{\varepsilon} $$

for any ε>0.

Proof

We conclude from Lemma 1 that as in (4.3) the upper bound follows directly from [25, Theorem 2] or [6, Corollary 2]. We proceed to prove the lower bound. The idea is taken from [13, Lemma 4]. We can assume that one of ν 1,ν 2 is sufficiently large. Since A ϕ (1,1)=1, we have

$$ \tilde{W}_{\nu_1, \nu_2}(y_1, y_2) = \int _0^1\int_0^1 \int_0^1 \phi(z) e(-x_1-x_2) \,dx_1 \, dx_2 \, dx_3. $$

for any y 1,y 2>0. By Cauchy–Schwarz we get

$$ \big| \tilde{W}_{\nu_1, \nu_2}(y_1, y_2)\big|^2 \leq \biggl(\int_0^1\int_0^1 \int_0^1 \big|\phi(z)\big|^2 \,dx_1 \, dx_2 \, dx_3 \biggr)^{1/2} \big| \tilde {W}_{\nu_1, \nu_2}(y_1, y_2)\big|. $$

Integrating this inequality and using Cauchy–Schwarz again, we find

Since [1,∞)2×[0,1]3 is contained in a fundamental domain for \(\mathit{SL}_{3}(\mathbb{Z}) \backslash\mathfrak{h}^{3}\) (see e.g. [21]), we obtain together with Proposition 2 that the right hand side is

$$ \ll\| \phi\| \biggl( \frac{ \prod_{j, k=1}^3 \varGamma(\frac{1+ \alpha_j + \overline{\alpha}_k}{2}) }{ \prod_{j=0}^2 |\varGamma(\frac{1}{2} + \frac {3}{2}i\Im\nu_j)|^2 } \biggr)^{1/2} \asymp\|\phi\|. $$

The left hand side is

Again by Proposition 2, this is

if one of ν 1,ν 2 is sufficiently large. □

We briefly discuss cusp forms \(u : \mathit{SL}_{2}(\mathbb{Z}) \backslash\mathfrak{h}^{2}\rightarrow\mathbb{C}\) for the group SL 2(ℤ) and spectral parameter νiℝ (Selberg’s eigenvalue conjecture is known for SL 2(ℤ)). A cusp form u has a Fourier expansion

$$ u(z) = \sum_{m \not= 0} \frac{\rho_u(m)}{\sqrt{m}} W_{\nu}\bigl(|m|y\bigr) e(mx) $$

where W ν was defined in (2.14). Similarly as in Lemma 1 we see that an arithmetically normalized newform u has norm

$$ \| u \|^2 = \int_{\mathit{SL}_2(\mathbb{Z}) \backslash\mathfrak{h}^2} \big|u(z)\big|^2 \,\frac{dx\, dy}{y^2} = 2 L \bigl(\mathrm{Ad}^2 u, 1 \bigr). $$
(4.4)

Indeed, the Eisenstein series \(E(z, s) = \frac{1}{2} \sum_{\gamma\in U_{2}\backslash \mathit{SL}_{2}(\mathbb{Z})} \Im(\gamma z)^{s}\) has residue 3/π at s=1, hence by (2.14)

the evaluation of the integral follows from [20, 6.576.4] or Stade’s formula for GL(2). Again we see that an arithmetically normalized cusp form u is essentially L 2-normalized, and W ν has roughly norm one with respect to the inner product

$$ (f, g) := \int_0^{\infty} f(y) \overline{g(y)} \det \left(\begin{array}{c@{\quad}c} y & \\ & 1\end{array} \right) \frac{dy}{y^2}. $$

5 Eisenstein series

There are three types of Eisenstein series on the space \(L^{2}(\varGamma \backslash\mathfrak{h}^{3})\) according to the decomposition

$$ L^2 \bigl(\mathit{SL}_2(\mathbb{Z})\backslash \mathfrak{h}^2 \bigr) = L^2_{\text{Eis}} \oplus L^2_{\text{cusp}} \oplus\mathbb{C} \cdot\textbf{1}. $$

The first term gives rise to minimal parabolic Eisenstein series: for \(z \in\mathfrak{h}^{3}\) and ℜν 1,ℜν 2 sufficiently large we define the minimal parabolic Eisenstein series

$$ E(z, \nu_1, \nu_2) := \sum_{\gamma\in U_3 \backslash \varGamma} I_{\nu _1, \nu_2}(\gamma z) $$

where \(I_{\nu_{1}, \nu_{2}}\) was defined in (2.11). It has meromorphic continuation in ν 1 and ν 2, and its non-zero Fourier coefficients are given by

(5.1)

(cf. (2.12) and (2.13) for the notation) where

$$ A_{(\nu_1, \nu_2)}(m_1, m_2) = |m_1|^{\nu_1+2\nu_2} |m_2|^{2\nu_1+\nu _2} \sigma_{-3\nu_2, -3\nu_1}\bigl(|m_1|, |m_2|\bigr) $$

and \(\sigma_{\nu_{1}, \nu_{2}}(m_{1}, m_{2})\) is the multiplicative function defined by

This is a combination of [7, (6.5), (6.7), (6.8), (7.3), Theorem 7.2]. An alternative description is given as follows: \(A_{(\nu_{1}, \nu_{2})}(m_{1}, m_{2})\) is defined by

$$ A_{(\nu_1, \nu_2)}(1, m) = \sum_{d_1d_2d_3 = m} d_1^{\alpha_1} d_2^{\alpha_2} d_3^{\alpha_3} $$

and the symmetry and Hecke relation

(5.2)

cf. [17, Theorems 6.4.11 and 10.8.6] and note his different choice of the I-function.

Next we define maximal parabolic Eisenstein series. Let s∈ℂ have sufficiently large real part and let \(u : \mathit {SL}_{2}(\mathbb{Z}) \backslash\mathfrak{h}^{2} \rightarrow\mathbb{C}\) be a Hecke–Maaß cusp form with ∥u∥=1, spectral parameter νiℝ and Hecke eigenvalues λ u (m). Then we define

$$ E(z,s; u) := \sum_{\gamma\in P\backslash\varGamma} \det( \gamma z)^{s} u \bigl(\pi(\gamma z) \bigr) $$
(5.3)

where

is the restriction to the upper left corner. It has a meromorphic continuation in s, and as the minimal parabolic Eisenstein series it is an eigenform of all Hecke operators; in particular for s=1/2+μ it is an eigenform of T(1,m) with eigenvalue

$$ B_{(\mu, u)}(1, m) = \sum_{d_1d_2 = |m|} \lambda_u(d_1) d_1^{-\mu }d_2^{2\mu}, $$

see [17, Proposition 10.9.3]. We extend this definition to all pairs of integers by the Hecke relations (5.2). Coupling this with [17, Proposition 10.9.1], we conclude that the non-zero Fourier coefficients

$$ \int_0^1\!\!\int_0^1\!\! \int_0^1 E(z, 1/2 + \mu; u) e(-m_1x_1-m_2x_2) \,dx_1 \, dx_2 \, dx_3 $$

are proportional to

$$ \frac{B_{(\mu, u)}(m_1, m_2)}{|m_1m_2|} \tilde{W}_{\mu- \frac{1}{3}\nu , \frac{2}{3} \nu}\bigl(m_1y_1, |m_2|y_2\bigr), $$
(5.4)

and the proportionality constant is

$$ \frac{c }{L(u, 1 + 3 \mu) L( \text{Ad}^2 u, 1)^{1/2} } $$
(5.5)

for some absolute non-zero constant c. This can be seen by setting m 1=m 2=1 and comparing with [32, Theorem 7.1.2] in the special case G=GL(3), M=GL(2)×GL(1), m=1, s=3μ and observing (4.4).

A degenerate case of (5.3) occurs if we choose ϕ to be the constant function (of course, this is not a cusp form). For ℜs sufficiently large and \(z \in\mathfrak{h^{3}}\) let

$$ E(z, s, \textbf{1}) := \sum_{\gamma\in P\backslash \varGamma} \det (\gamma z)^s. $$
(5.6)

This function has a meromorphic continuation to all s∈ℂ, and it has a simple pole at s=1 with constant residue

$$ \underset{s=1}{\text{res}} E(z, s, \textbf{1}) = \frac{1}{3} \bigl( \varGamma(3/2)\zeta(3)\pi^{-3/2} \bigr)^{-1} = \frac{2 \pi}{3\zeta(3)}, $$
(5.7)

see [16, Corollary 2.5].Footnote 6 As the constant function on \(\mathit{SL}_{2}(\mathbb{Z}) \backslash\mathfrak{h}^{2}\) is the residue of an Eisenstein series on \(\mathit{SL}_{2}(\mathbb{Z}) \backslash\mathfrak{h}^{2}\), the Eisenstein series (5.6) is a residue of a minimal parabolic Eisenstein series and has only degenerate terms in its Fourier expansion.

6 Kloosterman sums

As usual we write

$$ S(m, n, c) := \underset{d \, (\text{mod }c)}{\left .\sum \right .^{\ast }}e \biggl(\frac{md + n\bar{d}}{c} \biggr) $$

for the standard Kloosterman sum. We introduce now GL(3) Kloosterman sums; the following account is taken from [8].

For n 1,n 2,m 1,m 2∈ℤ, D 1,D 2∈ℕ we define

(6.1)

where Y 1,Y 2,Z 1,Z 2 are chosen such that

$$ Y_1B_1 + Z_1C_1 \equiv1 \ (\text{mod }D_1), \qquad Y_2B_2 + Z_2C_2 \equiv1 \ (\text{mod } D_2). $$

It can be shown that this expression is well-defined [8, Lemmas 4.1, 4.2]. Clearly it depends only on m 1,n 1 (mod D 1) and m 2,n 2 (mod D 2), and satisfies [8, Properties 4.4, 4.5]

(6.2)

Moreover, if p 1 p 2q 1 q 2≡1 (mod D 1 D 2), then [8, Property 4.3]

$$ S(p_1m_1, p_2m_2, q_1m_1, q_2m_2, D_1, D_2) = S(m_1, m_2, m_1, m_2, D_1, D_2). $$

Finally we have the factorization rule [8, Property 4.7]

(6.3)

whenever \((D_{1}D_{2}, D_{1}'D_{2}') = 1\) and inverses are taken with respect to the product of the respective moduli, that is,

$$ \overline{D_1} D_1 \equiv\overline{D_2}D_2 \equiv1 \ \bigl(\text{mod }D_1'D_2' \bigr), \qquad\overline{D_1'} D_1' \equiv\overline{D_2'}D_2' \equiv1 \ (\text{mod }D_1D_2). $$

This implies in particular

$$ \begin{array}{l} S(m_1, m_2, n_1, n_2, D_1, D_2) = S(D_2m_1, n_1, D_1)S(D_1 m_2, n_2, D_2), \\[6pt] \quad (D_1, D_2) = 1. \end{array} $$
(6.4)

For a prime p and l≥1 we have [8, Property 4.10]

(6.5)

Essentially best possible (“Weil-type”) upper bounds for S(m 1,m 2,n 1,n 2,D 1,D 2) have been given by Stevens [34, Theorem 5.1]. The dependence on m 1,m 2,n 1,n 2 has been worked out in [9, p. 39].

Lemma 3

For any integers n 1,n 2,m 1,m 2∈ℤ∖{0}, D 1,D 2∈ℕ and any ε>0 we have

where [.,.] denotes the least common multiple. In particular,

$$ \sum_{D_1 \leq X_1} \sum _{D_2 \leq X_2} \big|S(m, \pm1, n, 1, D_1, D_2)\big| \ll(X_1X_2)^{3/2+\varepsilon} (n, m)^{\varepsilon} $$
(6.6)

if \(mn \not= 0\). All implied constants depend only on ε.

Proof

It remains to show (6.6) which is straightforward:

 □

Next we define a different class of Kloosterman sums: If D 1D 2, we put

Again this sum depends only on m 1,n 1 (mod D 1) and n 2 (mod D 2/D 1), and for p 1 q 1≡1 (mod D 1), p 2 q 2≡1 (mod D 2) we have [8, Property 4.13]

$$ \tilde{S}(m_1p_1, n_1q_1p_2, n_2q_2, D_1, D_2) = \tilde{S}(m_1, n_1, n_2, D_1, D_2). $$

We have the factorization rule [8, Property 4.15]

whenever \((D_{2}, D_{2}') = 1\) and all terms are defined. Finally we have for a prime number p and 1≤l<k [8, Properties 4.16, 4.17]

$$ \tilde{S} \bigl(m_1, n_1, n_2, p^l, p^l \bigr) = \begin{cases} p^{2l} - p^{2l-1}, & p^l \mid m_1, \, p^l \mid n_1\\[3pt] -p^{2l-1}, & p^{l-1} \parallel m_1, \, p^l \mid n_1\\[3pt] 0, &\text{otherwise} \end{cases} $$

and

$$ \tilde{S} \bigl(m_1, n_1, n_2, p^l, p^k \bigr) = 0 $$

unless

  • k<2l and p 2lkn 1, or

  • k=2l, or

  • k>2l and p k−2ln 2.

In particular

$$ \tilde{S}(m_1, n_1, n_2, D_1, D_2) = 0 \quad\text{unless} \quad D_1^2 \mid n_1D_2. $$

A sharp bound was proved by Larsen [8, Appendix]:

$$ \tilde{S}(m_1, n_1, n_2, D_1, D_2) \ll\min \bigl((n_2, D_2/D_1)D_1^2, (m_1, n_1, D_1)D_2 \bigr) (D_1D_2)^{\varepsilon}. $$
(6.7)

7 Poincaré series

Let F:(0,∞)2→ℂ be a smooth compactly supported function (or sufficiently rapidly decaying at 0 and ∞ in both variables). Let

$$ F^{\ast}(y_1, y_2) := F(y_2, y_1). $$

For two positive integers m 1,m 2 and \(z \in\mathfrak{h}^{3}\) let \(\mathcal{F}_{m_{1}, m_{2}}(z) := e(m_{1}x_{2} + m_{2}x_{2}) F(m_{1}y_{1}, m_{2}y_{2})\). Then we consider the following Poincaré series:

$$ P_{m_1, m_2}(z) := \sum_{\gamma\in U_3 \backslash \varGamma} \mathcal {F}_{m_1, m_2}(\gamma z). $$

Unfolding shows

(7.1)

for an arbitrary automorphic form ϕ. In particular, if ϕ is given as in (4.1), we find

(7.2)

We want to apply (7.1) also with \(\phi= P_{n_{1}, n_{2}}\) where n 1,n 2 is another pair of positive integers. The Fourier expansion of \(P_{n_{1}, n_{2}}\) has been computed explicitly in [8, Theorem 5.1]: For m 1,m 2>0 we have

$$ \int_0^1\int _0^1 \int_0^1 P_{n_1, n_2}(z) e(-m_1x_1 - m_2x_2) \,dx_1 \, dx_2 \, dx_3 = S_1 + S_{2a} + S_{2b} + S_3, $$
(7.3)

where

The Kloosterman sums have been defined in Sect. 6 and the weight functions are given as follows:

(7.4)
(7.5)

8 Spectral decomposition

We have the following spectral decomposition theorem [17, Proposition 10.13.1]: If \(\phi\in L^{2}(\varGamma\backslash\mathfrak{h}^{3})\) is orthogonal to all residues of Eisenstein series, then

where the first j-sum runs over an orthonormal basis of cusp forms ϕ j for SL 3(ℤ) and the second j-sum runs over an orthonormal basis of cusp forms u j for SL 2(ℤ).

Therefore we have for 4 positive integers n 1,n 2,m 1,m 2 an equality of the type

We refer to the right hand side as the spectral side and to the left hand side as the arithmetic side.

We proceed to describe the spectral side and the arithmetic side more precisely. We define an inner product on L 2((0,∞)2,dy 1dy 2/(y 1 y 2)3) by

$$ \langle f, g \rangle:= \int_0^{\infty} \int _0^{\infty} f(y_1, y_2) \overline{g(y_1, y_2)} \,\frac{dy_1\,dy_2}{(y_1y_2)^3}. $$

Let {ϕ j } denote an arithmetically normalized orthogonal basis of the space of cusp forms on \(L^{2}(\mathit{SL}_{3}(\mathbb{Z}) \backslash\mathfrak{h}^{3})\) that we assume to be eigenfunctions of the Hecke algebra with eigenvalues A j (m 1,m 2). Let {u j } be an arithmetically normalized orthogonal basis of the space of cusp forms on \(L^{2}(\mathit{SL}_{2}(\mathbb{Z})\backslash\mathfrak{h}^{2})\) that we assume to be eigenfunctions of the Hecke algebra with eigenvalues λ j (m) and spectral parameter ν j iℝ.

Proposition 4

Keep the notation developed so far. Let F:(0,∞)2→ℂ be a smooth compactly supported function, and let m 1,m 2,n 1,n 2∈ℕ. Then for some absolute constant c>0 the following equality holds:

(8.1)

where

(8.2)

The weight functions \(\tilde{\mathcal{J}}\) and \(\mathcal{J}\) are given by

(8.3)
(8.4)

Proof

The spectral side (8.1) follows fromFootnote 7 (7.2) in combination with (5.1) and (5.4), (5.5). Note that E(z,1/2+μ,1) does not contribute because it has only degenerate terms in its Fourier expansion.

Upon combining (7.1) and (7.3), we obtain the arithmetic side (8.2) after applying a linear change of variables

$$ y_1 \mapsto\sqrt{\frac{n_1n_2}{D_1D_2m_1}} y_1, \qquad y_2 \mapsto \frac{y_2}{m_2} $$

in (7.4) and observing \(m_{2}D_{1}^{2} = n_{1}D_{2}\) (and with interchanged indices for Σ 2b ), and

$$ y_1 \mapsto\sqrt{\frac{n_2D_1}{m_1D_2^2}} y_1, \qquad y_2 \mapsto \sqrt{\frac{n_1D_2}{m_2D_1^2}} y_2 $$

in (7.5). □

Formally (8.1) and (8.2) resemble the GL(2) Kuznetsov formula, but in its present form it is relatively useless as long as we do not understand the transforms \(|\langle\tilde {W}_{\nu_{1}, \nu_{2}}, F\rangle|^{2}\) and \(\tilde{\mathcal{J}}, \mathcal{J}\) for a given test function F. The present formulation has the important advantage that the weight functions on the arithmetic side (8.2) do not depend on n 1,n 2,m 1,m 2,D 1,D 2 individually, but only in a coupled fashion. This is, of course, a well-known phenomenon in the GL(2) world.

We choose now

(8.5)

for X 1,X 2,R 1,R 2≥1, τ 1,τ 2≥0, τ 1+τ 2≥1 and f a fixed smooth, nonzero, non-negative function with support in [1,2]. Analytic properties of \(\langle\tilde{W}_{\nu _{1}, \nu_{2}}, F \rangle\) have been obtained in Proposition 3. We summarize some bounds for the weight functions occurring on the arithmetic side in the following proposition.

Proposition 5

With the notation developed so far, we have

$$ \|F \|^2 \asymp (X_1X_2)^2 R_1 R_2 (R_1 + R_2). $$
(8.6)

Let C 1,C 2≥0, ε>0. Then

$$ \tilde{\mathcal{J}}_{\epsilon; F}(A) \begin{cases} \ll X_1^2X_2 R_1R_2(R_1+R_2) \bigl(\frac{1+A^{2/3} }{\tau_1+\tau_2} \bigr)^{C_1}, \\[6pt] = 0, \quad\mbox{\textit{if} } A\leq(100X_1)^{-3/2} + (100 X_1X_2)^{-3/4}, \end{cases} $$
(8.7)

and

$$ \begin{array}{l} \mathcal{J}_{\epsilon_1, \epsilon_2}(A_1, A_2) \ll (X_1X_2)^{2}R_1R_2(R_1+R_2) \\[6pt] \quad {} \times \biggl(\displaystyle\frac{1+A_1^{4/3}A_2^{2/3}(X_1+X_2)}{\tau_1+\tau_2} \biggr)^{C_1} \biggl( \frac{1+A_2^{4/3}A_1^{2/3}(X_1+X_2)}{\tau_1+\tau_2} \biggr)^{C_2}\\[9pt] \quad{}\times \bigl((X_1+A_1) (X_2+A_2) \bigr)^{\varepsilon}, \\[6pt] \mathcal{J}_{\epsilon_1, \epsilon_2}(A_1, A_2) = 0, \quad \mbox{\textit{if} } \min \bigl(A_1A_2^2, A_2A_1^2 \bigr) \leq(100 X_1X_2)^{-3/2}. \end{array} $$
(8.8)

In the special case when A 1,A 2≤1, X 1=1, X 2=X≥1, R 1+R 2τ 1+τ 2 this can be improved to

(8.9)

Let g be a fixed smooth function with compact support in (0,∞). Then for R 1,R 2≫1 sufficiently large we have

(8.10)

On the left hand side we have suppressed the dependence of \(\mathcal {J}_{\epsilon_{1}, \epsilon_{2}}\) on τ 1, τ 2.

Remark 4

The bounds (8.7), (8.8), (8.10) are not best possible, but (8.9) is likely to be best possible. The important feature is that (8.7) and (8.8) effectively bound A 1,A 2 from below, and therefore D 1,D 2 in (8.2) from above. For example, for the contribution of the long Weyl element, we can essentially assume

$$ D_1 \leq\frac{(m_1n_2)^{1/3}(m_2n_1)^{2/3}}{\tau_1 + \tau_2}, \qquad D_2 \leq \frac{(m_1n_2)^{2/3}(m_2n_1)^{1/3}}{\tau_1 + \tau_2} $$

if X 1=X 2=1. It is instructive to compare this with the GL(2) situation: one can construct a sufficiently nice test function h on the spectral side with essential support on [T,T+1] such that the integral transforms h ± in (1.1) are negligible unless \(c \leq\frac{(nm)^{1/2}}{T}\).

The bound (8.10) shows that integration over τ 1,τ 2 can be performed at almost no cost, in other words, we save a factor (R 1 R 2)1−ε compared to trivial integration.

Remark 5

Choosing \(f(y_{1}, y_{2}) = e^{-2\pi(y_{1}+y_{2})} (y_{1}y_{2})^{100} \) (say), the two y-integrals in (8.4) can be computed explicitly using [20, 3.471.9], giving two Bessel-K-functions with general complex arguments. It is not clear how to take advantage of this fact.

Proof

Equation (8.6) is clear. We proceed to prove (8.7). Let us write

$$ \xi_1 := x_1^2 + 1, \qquad\xi_2 = x_1^2 + x_2^2 + 1 $$

in (8.3). The support of f restricts the variables to

(8.11)

The second set of conditions in (8.11) implies \(\xi_{1} \asymp A^{4/3} X_{1}^{2}\) and ξ 2A 8/3(X 1 X 2)2. Hence the second part of (8.7) is clear and a trivial estimation shows

$$ \tilde{\mathcal{J}}_{\epsilon; F}(A) \ll R_1R_2(R_1+R_2) X_1^2X_2. $$

In certain ranges this can be improved by partial integration. We have

We can assume that C 1 is an integer. Then C 1 successive integrations by parts with respect to y 1 yield an additional factor

$$ \ll_{C_1} \biggl( \biggl(\frac{y_1}{\tau_1+\tau_2} \biggr) \biggl( A|x_1| + \frac{A|x_2|}{\xi_2y_1^2y_2} + \frac{1}{y_1} \biggr) \biggr)^{C_1} \ll_{C_1} \biggl(\frac{1+A^{2/3}}{\tau_1+\tau_2} \biggr)^{C_1} $$
(8.12)

in the support of f.

The bound (8.8) can be shown similarly, but the estimations are a little more involved. Here we write

$$ \xi_1 := (x_1x_2 - x_3)^2 + x_1^2 + 1, \qquad \xi_2 = x_3^2 + x_2^2 + 1 $$
(8.13)

and truncate

(8.14)

This implies

$$ \xi_2 \asymp\varXi_2 := A_1^{4/3}A_2^{8/3} (X_1X_2)^2, \qquad \xi_1 \asymp \varXi_1 := A_2^{4/3}A_1^{8/3}(X_1X_2)^2 $$
(8.15)

which yields in particular the second part of (8.8) as well as

(8.16)

For future purposes we study the volume of the set of (x 1,x 2,x 3) defined by (8.15) or by

$$ \xi_2 = \varXi_2 \bigl(1+O(1/R_2) \bigr), \qquad\xi_1 = \varXi_1 \bigl(1+O(1/R_1) \bigr). $$
(8.17)

Lemma 4

For Ξ 1,Ξ 2≥1 and any ε>0 we have

$$ \underset{\substack{x_1, x_2, x_3\\ \mbox{\scriptsize\textup{satisfying} } (8.15)}}{\int \int\int} dx_1\, dx_2\, dx_3 \ll(\varXi_1\varXi_2)^{1/2+\varepsilon}. $$

Moreover,

$$ \underset{\substack{x_1, x_2, x_3 \\ \mbox{\scriptsize\textup{satisfying} } (8.17)}}{\int\int\int} dx_1\, dx_2\, dx_3 \ll (\varXi_1 \varXi_2R_1R_2)^{\varepsilon} \frac{(\varXi_1\varXi_2)^{1/2}}{R_1R_2}. $$

We postpone the proof to the end of this section. A trivial estimation now implies

Alternatively we write

(8.18)

using the notation (8.13). We can assume that C 1,C 2 are integers. Integrating by parts C 1 times with respect to y 1 and C 2 times with respect to y 2 introduces an additional factor

and (8.8) follows by (8.16) and the same argument that led to (8.7).

The proof of (8.9) is a small variant of the preceding argument. We need to save an additional power of R 1+R 2 which comes from a more careful treatment of the y 1,y 2-integral. Let η>0 be small. If \(1+A_{1}^{4/3} A_{2}^{2/3}X \leq(R_{1}+R_{2})^{1-\eta}\), we replace C 1 in (8.8) by C 1+η −1 saving a factor

$$ \biggl(\frac{1+A_1^{4/3}A_2^{2/3}X}{R_1+R_2} \biggr)^{-1/\eta} \geq R_1+R_2. $$

The same argument works if \(1+A_{1}^{2/3} A_{2}^{4/3}X \leq (R_{1}+R_{2})^{1-\eta}\). In the remaining case

$$ 1+ A_1^{2/3}A_2^{4/3}X \geq(R_1+R_2)^{1-\eta}\quad\text{and} \quad 1+A_2^{2/3}A_1^{4/3}X \geq(R_1+R_2)^{1-\eta} $$

it is enough to show

(8.19)

then the bound (8.9) follows with ε+η(C 1+C 2) instead of ε. To this end, we combine as before (8.18) and the first part of Lemma 4, and need to show that the y 1 and y 2 integral in (8.18) are both ≪(τ 1+τ 2)−1/2≍(R 1+R 2)−1/2. Our present assumption X 1=1, X 2=X≥1, A 1,A 2≤1 together with the size constraints (8.14)–(8.16) imply that the y 1 integral is of the form

$$ \int_0^{\infty} y_1^{-3i(\tau_1+\tau_2)} e(-\epsilon_1A_1x_1y_1) w(A_1y_1) \,\frac{dy_1}{y_1} $$
(8.20)

where w is a smooth function with support in [1,2] and w (j)(y)≪ j 1 uniformly in all other variables. We can assume that ϵ 1=sgn(x 1) and |x 1|≍τ 1+τ 2, otherwise we can save as many powers of τ 1+τ 2 as we wish by repeated partial integration. In that case we make another change of variables and re-write (8.20) as

$$ \int_0^{\infty} e \biggl(\frac{3}{2\pi}( \tau_1+\tau_2) (y_1 - \log y_1) \biggr) w \biggl(\frac{3(\tau_1+\tau_2)}{2\pi|x_1|} y_1 \biggr)\, \frac {dy_1}{y_1}. $$

A standard stationary phase argument bounds this integral by (τ 1+τ 2)−1/2: we cut out smoothly the region y 1=1+O((τ 1+τ 2)−1/2) which we estimate trivially. For the rest we apply integration by parts. The treatment of the y 2 integral is very similar. Here our assumptions imply that the integral is of the form

$$ \int_0^{\infty} y_2^{-3i(\tau_1+\tau_2)} e \biggl(-\frac{A_2}{y_2} \cdot\frac{x_1x_3+x_2}{x_3^2 + x_2^2 + 1} \biggr)\tilde{w}(XA_2w)\, \frac{ dy_2}{y_2}, $$

and the same stationary phase-type argument gives a saving of (τ 1+τ 2)−1/2.

Finally we prove (8.10). Let

$$ Z := R_1R_2(X_1+A_1) (X_2+A_2). $$

As before we see that we can assume

$$ \begin{array}{l} 1+ A_1^{2/3}A_2^{4/3}(X_1+X_2) \geq(R_1+R_2)^{1-\eta}\quad\text{and}\\[6pt] 1+A_2^{2/3}A_1^{4/3}(X_1+X_2) \geq(R_1+R_2)^{1-\eta}; \end{array} $$
(8.21)

otherwise we integrate trivially over τ 1,τ 2. In the situation (8.21) it is enough to showFootnote 8

(8.22)

then the bound (8.10) follows with ε+η(C 1+C 2) instead of ε. In order to show (8.22), we integrate (8.18) explicitly over τ 1,τ 2 and observe that

Since g is smooth, \(\tilde{g}\) is rapidly decaying, and up to a negligible error of Z A we can restrict ξ 1, ξ 2 to

$$ \begin{array}{l} \xi_1 = \displaystyle\frac{A_1^{2/3}}{A_2^{2/3} (y_1y_2)^{2}} \biggl(1 + O \biggl( \frac{1}{R_1Z^{\varepsilon}} \biggr) \biggr),\\[9pt] \xi_2 = \displaystyle\frac{A_2^{2/3}}{A_1^{2/3} (y_1y_2)^{2}} \biggl(1 + O \biggl(\frac {1}{R_2Z^{\varepsilon}} \biggr) \biggr). \end{array} $$
(8.23)

We note that \(A_{1}^{2/3}(A_{2}^{2/3} (y_{1}y_{2})^{2})^{-1} \asymp\varXi_{1}\), \(A_{2}^{2/3}(A_{1}^{2/3} (y_{1}y_{2})^{2})^{-1} \asymp\varXi_{2}\) in the notation of (8.15). Hence a trivial estimate bounds the left hand side of (8.22) by

$$ \ll\frac{(R_1R_2)^{2}(R_1+R_2)}{(A_1A_2)^2} \underset{\substack{x_1, x_2,x_3 \\ \text{satisfying } (8.23)}}{\int\int\int}\,dx_1\, dx_2\, dx_3 $$

and the desired bound follows from the second part of Lemma 4. □

It remains to prove Lemma 4: the conditions ξ 1Ξ 1, ξ 2Ξ 2 are equivalent to

Hence

This proves the first part of the lemma. The second part is more technical. The conditions (8.17) imply

(8.24)

for some constant c>0. We separate four cases for the range of x 3.

Case 1. If \((\varXi_{1}(1+c/R_{1})-1)(x_{2}^{2} + 1) < x_{3}^{2}\), then the condition on x 1 is empty.

Case 2. Let us assume

$$ \bigl( \varXi_1(1-c/R_1)-1 \bigr) \bigl(x_2^2 + 1 \bigr) \leq x_3^2 \leq \bigl(\varXi_1(1+c/R_1)-1 \bigr) \bigl(x_2^2 + 1 \bigr). $$
(8.25)

Then the volume of the x 1-region is

$$ \leq2 \biggl(\frac{\varXi_1(1+c/R_1)-1}{x_2^2+1} - \frac {x_3^2}{(x_2^2+1)^2} \biggr)^{1/2} \leq2 \biggl(\frac{2c\, \varXi_1}{R_1(x^2_2+1)} \biggr)^{1/2}. $$

The region (8.25) and the second inequality in (8.24) have a non-empty intersection only if

$$ \frac{\varXi_2(1-c/R_2) }{\varXi_1(1+c/R_1)} -1\leq x_2^2 \leq \frac{\varXi_2(1+c/R_2) }{\varXi_1(1-c/R_1)}-1. $$
(8.26)

If R 1,R 2>2c (which we may assume), this condition is empty unless \(\varXi_{2} \geq\frac{1}{3}\varXi_{1}\). This implies \(x_{2}^{2} \leq3\varXi_{2}/\varXi_{1}\) and \(x_{2}^{2}+1 \asymp\varXi_{2}/\varXi_{1}\). In the following we will frequently use the inequality \(\sqrt{A}-\sqrt{B} \leq(A-B)A^{-1/2}\) for AB≥0. Since we are assuming that Ξ 1,Ξ 2 are sufficiently large, we can deduce from (8.25) and the second inequality in (8.24) that the volume of the x 3-region is

$$ \ll\min \biggl(\frac{\varXi_2^{1/2}}{R_2}, \frac{\varXi_2^{1/2}}{R_1} \biggr) $$
(8.27)

(uniformly in x 2) and hence the total contribution under the assumption (8.25) is

$$ \int_{x_2 \in(8.26)} \frac{(\varXi_1\varXi_2)^{1/2}}{R_1^{1/2}(R_1+R_2) (x_2^2+1)^{1/2}} \,dx_2 \asymp \int _{x_2 \in(8.26)} \frac{\varXi_1}{R_1^{1/2}(R_1+R_2) } \,dx_2. $$

The region (8.26) describes an interval of length \(O( (R_{1}^{-1} + R_{2}^{-1}) \varXi_{2}/\varXi_{1})\) for \(x_{2}^{2}\), hence the total contribution is

$$ \ll \frac{\varXi_1}{R_1^{1/2} (R_1+R_2)} \frac{\varXi_2^{1/2}}{\varXi_1^{1/2}} \biggl(\frac{1}{R_1^{1/2}} + \frac{1}{R_2^{1/2}} \biggr) \ll \frac{(\varXi_1\varXi_2)^{1/2}}{R_1R_2} $$

as claimed.

Case 3. For a parameter 1/3≤αc/R 1 consider the region

$$ \bigl( \varXi_1(1-2\alpha)-1 \bigr) \bigl(x_2^2 + 1 \bigr) \leq x_3^2 \leq \bigl(\varXi_1(1- \alpha )-1 \bigr) \bigl(x_2^2 + 1 \bigr). $$
(8.28)

The procedure here is very similar to case 2. The x 1-volume is at most

$$ \leq \frac{4\varXi_1c/R_1}{(x_2^2+1)^{1/2}} \biggl(\varXi_1 \biggl(1+ \frac{c}{R_1} \biggr)-1 - \frac{x_3^2}{x^2_2+1} \biggr)^{-1/2} \ll \frac{\varXi_1^{1/2}}{R_1(x_2^2+1)^{1/2}\alpha^{1/2}}. $$

The region (8.28) and the second inequality in (8.24) have a non-empty intersection only if

$$ \frac{\varXi_2(1-c/R_2) }{\varXi_1(1-\alpha)} -1 \leq x_2^2 \leq \frac{\varXi_2(1+c/R_2) }{\varXi_1(1-2\alpha)} -1. $$
(8.29)

In particular this implies \(x_{2}^{2}+1 \asymp\varXi_{2}/\varXi_{1}\). As in (8.27) we see that the x 3-volume is \(\ll\varXi_{2}^{1/2} \min(R_{2}^{-1}, \alpha)\), hence the total contribution in the present subcase is

Case 4. Finally we consider the region \(x_{3}^{2} \leq (\varXi_{1}/3-1) (x_{2}^{2} + 1)\). In this case the x 1-volume is

$$ \leq\frac{4\varXi_1c/R_1}{(x_2^2+1)^{1/2}} \biggl(\varXi_1 \biggl(1+\frac {c}{R_1} \biggr)-1 - \frac{x_3^2}{x^2_2+1} \biggr)^{-1/2} \ll\frac{\varXi_1^{1/2}}{R_1(x_2^2+1)^{1/2}}. $$

The length of the x 3 interval is at most

$$ \leq \frac{4c\varXi_2}{R_2(\varXi_2(1+c/R_2) -1 - x_2^2)^{1/2}}. $$

Hence the total contribution is at most

This last integral can be computed explicitly:

$$ \int\frac{dx}{x(Z-x^2)^{1/2}} = \frac{\log(x) - \log(Z + \sqrt {Z(Z-x^2)})}{\sqrt{Z}}, $$

and the desired bound follows. This completes the proof of the lemma. □

9 Proofs of the theorems

For the proof of Theorem 1 we choose n 1=n 2=m 1=m 2=1 and combine (8.1), (8.2), Lemma 1 and Propositions 3 and 5. We choose τ 1=R 1=T 1, τ 2=R 2=T 2 and X 1=X 2=1 in (8.5), fix a function f and drop all these parameters from the notation of F. By the second part of (8.7) and (8.8), the Kloosterman terms Σ 2a ,Σ 2b ,Σ 3 are finite sums over D 1,D 2 and hence are O((T 1 T 2)−100) by the first part of (8.7) and (8.8). The diagonal term (8.6) is ≍T 1 T 2(T 1+T 2). On the spectral side, we drop the Eisenstein spectrum and large parts of the cuspidal spectrum to conclude by (3.3) and Lemma 1 the upper bound

$$ \sum_{\substack{| \nu^{(j)}_1 -iT_1| \leq c\\ | \nu^{(j)}_2 - iT_2| \leq c}} \bigl(\underset{s=1}{\mathrm{res}} L(s, \phi_j \times\tilde{\phi }_j) \bigr)^{-1} \ll T_1T_2(T_1+T_2) $$

for some sufficiently small c and T 1,T 2T 0, and hence

$$ \sum_{\substack{| \nu^{(j)}_1 -iT_1| \leq K\\ | \nu^{(j)}_2 - iT_2| \leq K}} \bigl(\underset{s=1}{\mathrm{res}} L(s, \phi_j \times\tilde{\phi }_j) \bigr)^{-1} \ll_K T_1T_2(T_1+T_2) $$
(9.1)

for any K≥1 by adding the contribution of O K (1) balls. To prove the lower bound, we choose (once and for all) K so large that

$$ \sum_{\substack{\max(| \nu^{(j)}_1 -iT_1|, | \nu^{(j)}_2 - iT_2|) \geq K}} \|\phi_j\|^{-2}\big|\langle\tilde{W}_{\nu_1, \nu_2}, F\rangle\big|^2 \leq\frac{1}{2} \| F \|^2 $$

which is possible by (9.1) and (3.1). We bound the Eisenstein spectrum trivially: the second line of (8.1) contributes O((T 1+T 2)ε) by known bounds for the zeta function on the line ℜs=1, the third line contributes similarly O((T 1+T 2)1+ε) by Weyl’s law for SL 2(ℤ) and lower bounds for the L-functions in the denominator [22, 23]. Hence we obtain

$$ \sum_{\substack{ | \nu^{(j)}_1 -iT_1| \leq K\\ | \nu^{(j)}_2 - iT_2| \leq K}} \|\phi_j\|^{-2} \big|\langle\tilde{W}_{\nu_1, \nu_2}, F\rangle \big|^2 \geq \frac{1}{2} \| F \|^2 + O \bigl((T_1+T_2)^{1+\varepsilon} \bigr), $$

and the lower bound in Theorem 1 follows from (3.1) and (8.6) for T 1,T 2 sufficiently large. □

The proof of Theorem 2 proceeds similarly. As mentioned in the introduction, as a direct corollary of Theorem 1 we find that the number of exceptional Maaß forms ϕ j with γ j =T+O(1) is O(T 2). In order to prove Theorem 2, it is therefore enough to consider those Maaß forms with |ρ j |≥ε. Moreover, by symmetry it is enough to bound only Maaß forms satisfying (2.9). In (8.5) we take τ 2=R 2=T, R 1=1, τ 1=0, X 1=1, X 2=X=T δ for some δ>0 to be chosen later. With this data, the spectral side, after dropping

  • the tempered spectrum,

  • the Eisenstein spectrum, and

  • those parts of the non-tempered spectrum not of the form (2.9) with |ρ j |≥ε,

is by (3.5) (note that (3.4) is satisfied) and the upper bound of (1.4) at least

$$ \gg T^{-\varepsilon} X^2 \sum_{\substack{ \phi_j \text{ as in } (2.9)\\ \gamma_j = T+O(1)\\ |\rho_j| \geq\varepsilon}} X^{2|\rho_j|}. $$

On the arithmetic side, the diagonal term is ≍T 2 X 2 by (8.6). Next by (8.7) we have

$$ \varSigma_{2a} \ll T^2X \sum _{D_1 \ll1} \frac{|\tilde{S}(\pm1, 1, 1, D_1, D_1^2)|}{D_1^3} T^{-102} \ll X T^{-100} $$

and

$$ \varSigma_{2b} \ll T^2X^2 \sum _{D_2 \ll X^{1/2}} \frac{|\tilde{S}(\pm1, 1, 1, D_2, D_2^2)|}{D_2^3} T^{-102} \ll X^{2+\varepsilon} T^{-100} $$

by (6.7). (Note that we are exchanging X 1 and X 2 for Σ 2b .) The long Weyl element contributes at most

(9.2)

which follows by combining (8.9) and (6.6). Choosing X=T 2 completes the proof of Theorem 2. □

We proceed to prove Theorem 3. Again we choose X 1=X 2=1, R 1=T 1, R 2=T 2 in (8.5), fix a function f and then drop R 1,R 2,X 1,X 2,f from the notation of F and keep only τ 1,τ 2. We also fix a suitable non-negative smooth function g with support in [1/2,3] as in Proposition 5. Let T:=max(T 1,T 2). The left hand side of (1.5) is, by (3.3) and the upper bound of (1.4),

$$ \ll T^{\varepsilon} \sum_{j} \frac{1}{\|\phi_j\|^2} \int_{T_1}^{2T_1} \int_{T_2}^{2T_2} \big|\langle\tilde{W}_{\nu_1, \nu_2}, F_{\tau_1, \tau _2}\rangle\big|^2 \,d \tau_1 \,d\tau_2 \bigg| \sum_{n \leq N} \alpha(n) A_j(n, 1) \bigg|^2. $$

We cut the n-sum into dyadic intervals, insert artificially the function g and bound the preceding display by

Next we add artificially the continuous spectrum getting the upper bound

We open the squares and apply the Kuznetsov formula, that is, we replace the three terms of the shape (8.1) with the four terms (8.2). We estimate each of them individually. The diagonal term contributes by (8.6)

This is the first term on the right hand side of (1.5). In the term Σ 2a in (8.2) the condition D 1D 2, \(D_{1}^{2} = nD_{2}\) is equivalent to D 1=nd, D 2=nd 2 for some d∈ℕ; hence its contribution is at most

By (8.7), the d-sum is finite, hence in combination with (6.7) this is bounded by

$$ \ll(NT)^{\varepsilon} \max_{\substack{ M \leq N}} \sum_{M \leq n, m \leq2M}\big|\alpha(n) \alpha(m)\big| T^{-101} \ll N^{\varepsilon} T^{-100} \| \alpha\|^2. $$

In the term Σ 2b in (8.2) the condition D 2D 1 is redundant, and the argument of \(\tilde{\mathcal {J}}_{\epsilon, F^{\ast}}\) equals \((n/(mD_{2}^{3}))^{1/2}\). As before we see that this contributes at most N ε T −100α2.

Finally the long Weyl element finally contributes by (8.10)

(9.3)

for any C 1,C 2≥0. Recalling the notation T=max(T 1,T 2) and using (6.6), it is straightforward to see that the previous display is

This is the second term on the right hand side of (1.5). □

Finally we prove Theorem 4. To this end, we express L(ϕ j ,1/2) by an approximate functional equation. As we are summing over the archimedean parameters of the L-functions, we need an approximate functional equation whose weight function is essentially independent of the underlying family. This has been obtained in [3, Proposition 1], and we quote the following special case. For a Maaß form ϕ j as in Theorem 4 put

Moreover, for a multi-index \(\textbf{n} \in\mathbb{N}_{0}^{6}\) we write |n|=n(1)+⋯+n(6) and

$$ \boldsymbol{\eta}_j^{-\textbf{n}} := \prod_{l=1}^3 (\eta_j)_l^{-\textbf {n}(2l-1)} (\overline{ \eta}_j)_l^{-\textbf{n}(2l)}. $$

Proposition 6

Let G 0:(0,∞)→ℝ be a smooth function with functional equation G 0(x)+G 0(1/x)=1 and derivatives decaying faster than any negative power of x as x→∞. Let M∈ℕ and fix a Maaß form ϕ as above. There are explicitly computable rational constants c n, ∈ℚ depending only on n, , M such that the following holds for

$$ G(x) := G_0(x)+\sum_{\substack{0< |\boldsymbol{n}| < M \\ 0< \ell< |\boldsymbol{n}|+M}} c_{\boldsymbol{n},\ell} \boldsymbol{\eta}_j^{-\boldsymbol{n}} \biggl(x \frac{\partial }{\partial x} \biggr)^{\ell} G_0(x). $$

For any ε>0 one has

(9.4)

where |κ j |=1 and the implied constant depends at most on ε, M, and the function G 0.

It is now a simple matter to prove Theorem 4. We can assume that T is sufficiently large. Let

$$ G_{\ell}(x) := \biggl(x\frac{\partial}{\partial x} \biggr)^{\ell} G_0(x). $$

Then the Mellin transform \(\widehat{G}_{\ell}(s)\) is rapidly decaying on vertical lines ℜs=σ>0. By (9.4) and Mellin inversion we have

noting that C j η j T 2T 3. This is at most

By Theorem 3 and (4.3) the first term is O(T 5+ε). By Theorem 1 or (9.1) it is easy to see that the second term is also O(T 5+ε). This completes the proof of Theorem 4. □