A nonabelian trace formula

Let $E/F$ be an extension of number fields with $\mathrm{Gal}(E/F)$ simple and nonabelian. In [G] the first named author suggested an approach to nonsolvable base change and descent of automorphic representations of $\mathrm{GL}_2$ along such an extension. Motivated by this we prove a trace formula whose spectral side is a weighted sum over cuspidal automorphic representations of $\mathrm{GL}_2(\mathbb{A}_E)$ that are isomorphic to their $\mathrm{Gal}(E/F)$-conjugates.

one should have a corresponding transfer of packets of automorphic representations of H(A F ) to packets of automorphic representations of G(A F ) compatible with local transfers at every place v of F . If the rank of G is large, in most cases where this conjecture is known the image of the transfer map induced by (1.0.1) has a very simple description, namely it consists of L-packets fixed by a finite-order automorphism of G. One has a standard technique available to isolate representations that are isomorphic to their twists under an automorphism of G, namely the twisted trace formula, and one can compare this to a trace formula on H to establish the desired transfer; see [AC] for the first set of examples in arbitrary rank and [A] [M] for more general examples that represent the state of the art. The alternate approach via converse theory is different, but the set of L-maps treated in arbitrary rank is roughly the same [CKPSS].
If one could isolate automorphic representations of a group G invariant under a noncyclic group of automorphisms then one could hope to establish new cases of Langlands functoriality. In this paper we take the first step in this direction in a very special case.
1.1. The trace formula. It is convenient to first explicitly describe the trace formula we will employ. Let F be a number field, let K ∞ ≤ GL 2 (F ∞ ) be the standard maximal compact subgroup and let GL 2 (A F ) 1 := ker (| · | • det : where | · | is the usual idelic norm. Moreover let B = T U ≤ GL 2 be the standard Borel subgroup of upper triangular matrices with T ≤ GL 2 the subgroup of diagonal matrices, U the unipotent radical of B and T (A F ) 1 = t 1 t 2 ∈ T (A F ) : |t 1 | = |t 2 | = 1 . (1.1.1) Using these subgroups one defines spaces of test functions C(GL 2 (A F ) 1 ) as in §4 below. The key property of these spaces of test functions is that, although their elements are not compactly supported functions at any place, the trace formula is still valid for them, as proven in [FiL1] (and in greater generality in [FiLM] and [FiL2]). Explicitly, in [FiL1] one finds a proof of the following theorem: is absolutely convergent and equal to tr R cusp (f ).
Here the sum on γ is over elliptic conjugacy classes in GL 2 (F ) and R res (f ) (resp. R cusp (f )) denotes the restriction of the usual convolution operator R(f ) to the residual (resp. discrete) spectrum. Moreover GL 2γ denotes the centralizer of γ in GL 2 .
If f ∈ C ∞ c (GL 2 (A F ) 1 ) then this formula is a special case of the Arthur-Selberg trace formula. In general this is [FiL1,Theorem 1], with a caveat: the theorem in loc. cit. is stated only for F = Q, but this is only for convenience (see the introduction of [FiL1]). We will not explicate the definition of the undefined terms and measures in the expression above; instead we refer the reader to loc. cit. From a qualitative point of view, the key point is that the first line of the equality in Theorem 1.1 is equal to tr R cusp (f ) up to "error terms," i.e. the rest of the expression.
Remark. The result is actually better than stated above, for the terms in the trace formula define continuous seminorms on C(GL 2 (A F ) 1 ) (see [FiL1,Theorem 1]). In particular, it is shown in loc. cit. that the trace norm of tr R cusp (f ) is bounded.

The main theorem. Let E/F be a Galois extension with
Gal(E/F ) = ι, τ where |ι| = 2 and τ has odd order.
Remark. Any nonabelian finite simple group can be generated by an involution and an element of odd order by a result of Guralnick and Malle (see [GuM,Theorem 1.6] and the paragraph after it). Let F ′ := ι=1 E and F 0 := τ =1 E (1.2.1) denote the fixed fields of ι and τ , respectively. We also let Let K ∞ ≤ G(F ∞ ) be the standard maximal compact subgroup and let . As usual we set K = K ∞ K ∞ . Finally let S be a Gal(E/F )-invariant set of places of E containing all infinite places, all places where E/Q is ramified, and all dyadic places. We also denote by S the set of places of any subfield E ≥ k ≥ F dividing S.
Let Ψ ∈ C ∞ c (G(A F ) 1 //K). In §5 we define certain functions such that the following theorem holds: where the sum is over cuspidal automorphic representations of G(A F ) 1 spherical at every place and Π ′ is a cuspidal automorphic representation of GL 2 (A F ′ ) 1 whose base change to E is Π. The where T F is defined as in Theorem 1.1.
The proof of Theorem 1.2 boils down to carefully constructing the function V and then proving that it lies in the space of test functions to which the trace formula applies. We will explain this in broad terms in §1.4 below.
1.3. Possible applications. The authors view Theorem 1.2 as a nontrivial step towards proving the existence of base change and descent of automorphic representations of GL 2 along the extension E/F . We illustrate this with a typical example. Assume that Gal(E/F ) and Gal(E/F ) = ι, τ with |ι| = 2 and |τ | odd. One does not expect all cuspidal representations π of GL 2 (A E ) that are Gal(E/F )-invariant to descend to GL 2 (A F ), but Langlands functoriality predicts that one can choose a finite set of characters χ : E × \A × E → C × depending on Gal(E/F ) such that if π is Gal(E/F )-invariant then π ⊗ χ descends to GL 2 (A F ) for some χ [Ge, Proof of Lemma 5.1]. This additional complication disappears if we assume in addition that Gal(E/F ) is the universal perfect central extension of a finite simple group (compare loc. cit.). Thus let us assume that Gal(E/F ) is a universal perfect central extension of a finite simple group. For simplicity we exclude the case where Gal(E/F ) is the universal perfect central extension of the icosahedral group (this case is treated from a slightly different perspective in [Ge]). Let

Consider the two quantities
are transfers of f . Notice that by Theorem 1.1 the quantities (1.3.1) and (1.3.2) are given by (admittedly complicated) absolutely convergent sums and integrals that are geometric in nature. It is not hard to argue as in [Ge] that the equality of (1.3.1) and (1.3.2) for all f implies that any Gal(E/F )-invariant cuspidal automorphic representation of G(A E ) spherical at all places descends to GL 2 (A F ) (compare [Ge,Theorem 1.2]).
Remark. The existence of base changes of cuspidal automorphic representations π of GL 2 (A F ) 1 spherical at all places would also follow if we knew somehow that tr π(b E/F (V )) = 0, but it is not clear how to prove this without knowing the existence of the base change.
We emphasize, once again, that (1.3.1) and (1.3.2) are built out of absolutely convergent geometric sums. This is not easy to come by in this subject. The primary difficulty in all cases where Langlands' beyond endoscopy proposal [La2] has been carried out [V], [Her], [W] has been obtaining an absolutely convergent sum out of limiting forms of trace formulae, and, to date, it has never been carried out to prove functoriality in a case that was not known by some other method. As explained in [Sa], this task is expected to become even more difficult in other situations. One motivation for proving Theorem 1.2 is that we can contemplate proving that (1.3.1) is equal to (1.3.2), and this might provide a way to move around the analytic difficulties of the beyond endoscopy proposal in certain cases.
1.4. Outline. In the following section we give our conventions on Haar measures for the paper. In §3 we record some known results on Whittaker functions, and prove the crucial result Corollary 3.5, which, combined with the Casselman-Shalika-Shintani formula, allows us to construct Hecke operators W g whose Hecke eigenvalues are the values of Whittaker functions 1 at g ∈ G(A F ). We then record what we mean by transfers of test functions in our setting in §4.
Finally in §5 we construct V . The idea is to use the special Hecke operators W g to build a weight consisting of the period of a cusp form over a unitary group into the trace formula. This weight isolates representations of G(A F ) invariant under ι by work of Jacquet (we actually require the refinements of [FLO]). These V are then used in §5.2 to prove our main result, Theorem 1.2.

Haar measures
Throughout this paper we normalize Haar measures as in the following table, which is consistent with the normalizations in [FiL1]: Here (A × E ) 1 = ker(| · |). We will also require a Haar measure on H(A F ) and H(F )\H(A F ), where H is the group of §5.1, but for the purposes of this paper there is no need to specify it; we simply fix one.

Whittaker functions
In this section we collect some generalities on Whittaker functions. The main results are corollaries 3.2 and 3.5, which construct Hecke operators whose Hecke eigenvalues are the values of Whittaker functions.
None of the results of this section use anything about the extension E/F , so for this section E is an arbitrary number field. We let K ∞ := GL 2 ( O E ) and let K ∞ ≤ GL 2 (E ∞ ) be the standard maximal compact subgroup.
3.1. Generalities. Let ϕ be a smooth cuspidal function in L 2 (GL 2 (E)\GL 2 (A E ) 1 ) and let ψ : E\A E → C × be the additive character Here the principal part pr(a) ∈ Z[p −1 ] is any element such that a ≡ pr(a) (mod Z p ). We set for m ∈ E. As usual for each m ∈ E we have Whittaker functions To ease notation we set is identically zero as a function of g if m = 0. By taking Fourier expansions we obtain the Whittaker expansion to the Whittaker model W(π, ψ m ) of π with respect to ψ m . Here W(π, ψ m ) and its local analogues are defined as in [C, §1.2].
We recall that Whittaker functions factor into local Whittaker functions: where the product is over places w of E and W ϕ m,w ∈ W(π w , ψ mw ), the Whittaker model of π w with respect to ψ mw .
We note that for a smooth ϕ, a β ∈ E × , and g ∈ GL 2 (A E ) one has Finally we record a definition used in the statement of Theorem 1.2 above. Let be an idele whose associated ideal is the absolute different D E of E over Q. Let π w be a generic irreducible admissible representation of GL 2 (E w ). We say that W ∈ W(π w , ψ w ) is a Whittaker newform if the following conditions hold: (1) It is K w -finite.
(2) The identity holds and the integral on the left is absolutely convergent for Re(s) large enough.
The factor |δ| s−1/2 w occurs here because we have chosen our standard character ψ w to have conductor D Ew . If ϕ is a smooth function in the space of a cuspidal automorphic representation π realized in the cuspidal subspace of Whittaker newforms always exist. For the archimedian case see [P]. In the nonarchimedian case the Whittaker newform is a choice of normalization of a newform (or new vector) in the usual sense (see [G,Proposition 6.17] or [Sc,§2,Summary]).
3.2. The Casselman-Shalika-Shintani formula. For the remainder of this section we work in a local setting. Thus we fix a place w and omit it from notation, writing E := E w .
Assume for this subsection that w is nonarchimedian.
Here and below, unless otherwise specified, ½ X is the characteristic function of a set X.
The following is the Casselman-Shalika formula, due in the case of GL n to Shintani: Here δ is as in (3.1.7).
Proof. The second assertion is an immediate consequence of the fact that in each admissible irreducible representation π the space of newforms is one-dimensional. The identity (π(½ m )W ) 1/δ 1 = |m| −1 W m/δ 1 is well-known. One reference is [KLi,Proposition 4.6].
Let B ≤ GL 2 denote the Borel subgroup of upper-triangular matrices.
where ψ is as in (3.1.1). We have the following corollary of Theorem 3.1: Corollary 3.2. Let π be a generic representation of GL 2 (E). If W ∈ W(π, ψ) is a newform and g ∈ B(E)K then π(W g )W = W (g)W.
Proof. Recall that if W ∈ W(π, ψ) is a Whittaker newform then W m/δ 1 = 0 only if m ∈ O E and W m/δ 1 = 1. One reference for this fact is [Sc,§2], bearing in mind that in loc. cit. the character ψ is normalized to have conductor O E , whereas we normalized our ψ to have conductor D E . In view of this the corollary is an immediate consequence of Theorem 3.1.

3.3.
A replacement for Theorem 3.1 in the archimedian case. Assume that E = E w is an archimedian local field. In this section we construct test functions in which is defined to be the Frechet space of smooth functions f on GL 2 (E) such that for all The implicit constant in the choice of Haar measure is irrelevant for our purposes.
In order to construct these functions we recall the explicit description of the standard model of an induced irreducible admissible representation of GL 2 (E). The basic reference for the constructions is [JL,[5][6]. There is a brief summary in [Z, §2.2].
Let (π, V π ) be an irreducible representation of GL 2 (E) with central character χ. Consider the space B(µχ, µ −1 ) of K-finite functions ϕ ∈ C ∞ (GL 2 (E)) satisfying where µ : E × → C × is a quasicharacter. The group GL 2 (E) acts on B(µχ, µ −1 ) via right translations, and upon replacing µ with χ −1 µ −1 if necessary we can and do assume that π is equivalent to a subrepresentation (not merely a subquotient) of B(µχ, µ −1 ). The space B(µχ, µ −1 ) consists of vectors of the form where P (g) is a polynomial in the entries of g and their complex conjugates. It is clear that F P restricts to a smooth function on GL 2 (E).
For real numbers r ∈ R >0 let δ r denote the distribution on GL 2 (E) given on test functions and dg 1 is chosen to be compatible with the decomposition and the choices of measures in §2. Less formally, δ r is the Dirac distribution associated to the GL 2 (E) 1 -homogeneous submanifold Assume that the representation K × K −→ Aut x → F P (k −1 1 xk 2 ) : k 1 , k 2 ∈ K is finite-dimensional and equivalent to an irreducible representation of the form ρ ⊗ ρ ∨ of K × K, where ρ is an irreducible representation of K.
Lemma 3.3. Assume that π is an admissible representation of GL 2 (E). One has a welldefined bounded operator π(F P δ r ) on π. Let ϕ be a vector in π, viewed as a subrepresentation of B(µχ, µ −1 ). If ϕ does not lie in the ρ-isotypic subspace of π, then π(F P δ r )ϕ = 0. If ϕ lies in the ρ-isotypic subspace of π, then π(F P δ r ) acts via the scalar Moreover, the integral in this expression is absolutely convergent.
Proof. To prove that the operator is bounded we proceed as in [GoJ,§1.1]. Let G 1 ∈ S(M 2 (E)) be a nonnegative Schwartz function. Then there are nonnegative Schwartz functions G 2 ∈ S(M 2 (E)) and G 3 ∈ S(E) such that This implies that π(F P δ r ) is a well-defined bounded operator.
If ϕ does not lie in the ρ-isotypic subspace of π, then the assertion of the lemma is clear. Suppose therefore that ϕ = 0 does lie in the ρ-isotypic subspace (which implies in particular that this subspace is nonempty). Then our assumptions on the K-type of F P and Schur's lemma imply that the operator π(F P δ r ) acts via a scalar on the ρ-isotypic subspace. We must compute this scalar.
Choose Φ ∈ S(E × E) such that ϕ Φ is in the ρ-isotypic subspace of π and That this is always possible can be seen by realizing that the integral defining ϕ Φ is a Tate integral, compare [JL,Lemma 5.13.1 and Lemma 6.3.1].
Using the Iwasawa decomposition we compute By (3.3.8) we conclude that the scalar by which π(F P δ r ) acts on the ρ-isotypic subspace is For the remainder of this subsection assume that π is spherical. For each h = ( 1 t 1 ) ( zm z ) k ∈ GL 2 (E) with k ∈ K define a distribution W h on GL 2 (E) by setting Lemma 3.4. For any Ψ ∈ C ∞ c (GL 2 (E)) one has Ψ * W g ∈ C(GL 2 (E)). In fact, for any Proof. It clearly suffices to prove the second statement.
As explained in [GoJ,§1.8,, we have X * F 1 * Y = F P (3.3.10) for some P . Moreover any operator in the subalgebra is insensitive to the difference between F P and F P δ |m| , and the algebra on the right is generated (as an algebra) by these elements together with central operators corresponding to the Lie algebra of Z Res E/R GL 2 . These operators can be absorbed into Ψ. It therefore suffices to show that for Ψ ∈ C ∞ c (GL 2 (E)) and polynomials P as above we have ||Ψ * F P (g)δ r || 1 ≪ Ψ,T,A,P r −A for r > 1 (3.3.11) ||Ψ * F P (g)δ r || 1 ≪ Ψ,T,B,P r −B for r ≤ 1.
for any A > 0 and some B > 0. But Applying (3.3.7) the assertion (3.3.11) is clear.
The reason for the notation W g is apparent given the following Corollary: Corollary 3.5. Let π be unitary, and let W ∈ W(π, ψ) be the spherical Whittaker newform. Then π(W g ) annihilates the orthogonal complement of the line spanned by W , and acts via the scalar W (g) on the line spanned by W .
Proof. It is enough to verify the corollary in the special case where g = ( m 1 ). By Lemma 3.3 it suffices to show that tr π W ( r 1 ) = W ( r 1 ) (3.3.12) for r ∈ R >0 . Both sides of this expression are smooth functions of r. It is well-known that the right hand side is rapidly decreasing as r → ∞ and of order at most r −B for some B > 0 as r → 0 and the same is true of the left hand side by Lemma 3.4. Thus to prove that the two are equal it is enough to verify that their Mellin transforms are equal. Recall (see §2) that our choice of Haar measure on E is the Lebesgue measure if E is real and twice the Lebesgue measure (on R 2 ) if E is complex. Thus, essentially by definition, for Re(s) sufficiently large one has On the other hand, by Lemma 3.3 for Re(s) sufficiently large :R](yy+zz) |yz| s µχ(y)µ −1 (z)dy × dz × = L(s, µχ)L(s, µ −1 ) = L(s, π).

Test functions and transfer
For this section we let E/F be a finite extension of number fields (not necessarily Galois) that is everywhere unramified and split at all infinite places.
Consider the Frechet space We also have the analogous Frechet space C(G(F ∞ )) at infinity; it consists of elements of C ∞ (G(F ∞ )) such that (4.0.2) is valid for all X, Y with L 1 (G(A F ) 1 ) replaced by L 1 (G(F ∞ )). Note that this is not the same as the Harish-Chandra Schwartz space, whose elements may not be integrable.
The spaces )) and C(GL 2 (F ∞ )) are defined similarly. We note that these spaces are algebras under convolution. Moreover, if Π is a cuspidal automorphic representation of G(A F ) 1 , then is a continuous linear functional (this is part of the main result of [FiL1], for example, see Theorem 1.1 above). The assumption that Π is cuspidal can be weakened somewhat, but it is not true for an arbitrary admissible representation.
Let v be a place of F . We say that Ψ is valid for all irreducible admissible representations π of GL 2 (F v ) and base change Π to G(F v ) (the notion of base change in this context is recalled in [Ge,§1]).

Remarks.
(1) This notion does not coincide with the usual notion of transfer in the theory of cyclic base change.
(2) It would be interesting to define transfers of arbitrary functions in C(G(A F ) 1 ), but we will not attempt this. Instead we will make do with something weaker (see Lemma 4.2 below).
In this paper we need to make explicit our choices of transfers. For the remainder of this section let and let K F := K ∩ GL 2 (A F ). We will define a continuous algebra homomorphism First recall that for finite places v of F the base change maps Second, recall that E/F is assumed to be split at all infinite places. We can therefore by choosing an identification (4.0.5) On the dense subspace this map just coincides with the [E : F ]-fold convolution, and therefore (4.0.5) is continuous. There is ambiguity in the choice of isomorphism (4.0.4), but since spherical Hecke algebras are commutative this does not affect the definition of the morphism b E/F . Lemma 4.1. Let π be a cuspidal automorphic representation of GL 2 (A F ) and base change Proof. Under the isomorphism (4.0.4) the representation π E∞ corresponds to the [E : F ]-fold tensor product of π ∞ . Thus if f lies in the dense subset C ∞ c (GL 2 (F ∞ )//K F ∞ ) ⊗[E:F ] then the identity is trivial. The general case follows from the continuity of the distributions tr π E on C(G(A F ) 1 ) and tr π on C(GL 2 (A F ) 1 ).

The considerations above yield continuous algebra morphisms
(4.0.6) Lemma 4.2. The map (4.0.6) extends to a continuous algeba morphism Moreover, if π is a cuspidal automorphic representation of GL 2 (A F ) with cuspidal base change π E to GL 2 (A E ) and f ∈ C(G( tr π E (f ) = tr π(b E/F (f )).
Proof. By Lemma 4.1 b E/F is continuous and linear on the dense subspace therefore extends uniquely to a continuous algebra morphism. To deduce the final assertion of the lemma we invoke the continuity of the distributions tr π E on C(G(A F ) 1 ) and tr π on C(GL 2 (A F ) 1 ) together with Lemma 4.1.

The construction of V
In this section we construct V ∈ C(G(A F ) 1 ) and use it to prove our main theorem, Theorem 1.2. We assume that E/F is a Galois extension of number fields that is everywhere unramified and split at all infinite places. We also assume that Gal(E/F ) = ι, τ where τ has odd order. We let F ′ be the fixed field of ι and F 0 the fixed field of τ . 5.1. The subgroup H. Let U 2 be the unitary group over O F ′ whose points in an O F ′algebra R are given by Here we regard the matrices ( t t −1 ), t ∈ R × as elements of B H (F ∞ ) via the diagonal embedding.
Proof. This follows from [Bo,Theorem 5.3] together with the Iwasawa decomposition.
Let F be a fundamental domain as in Lemma 5.1 and let Ψ ∈ C ∞ c (G(A F ) 1 //K). We define Proposition 5.2. One has that V ∈ C(G(A F ) 1 ).
Proof. Let X, Y ∈ U(g). Then by the Iwasawa decomposition where t ∈ R >0 is embedded diagonally in E × ∞ , dt × (resp. da × ) is our standard Haar measure on R × (resp. (A × E ) 1 ), db is a translation invariant measure on the 1-dimensional free A Fmodule {x ∈ A E : tr E/F ′ (x) = 0}, and dk H is a Haar measure on K ∩ H(A F ). As above, let U ≤ GL 2 be the unipotent radical of the Borel subgroup of upper-triangular matrices.
by a scalar of norm 1 independent of g and hence the above is equal to By definition, W yt 2 a ι a 1 vanishes identically if (ya ι a) ∞ ∈ D −1 E . Thus, since the integral over a is compactly supported, the sum over y above can be truncated to the subset y ∈ a −1 − 0 for some ideal a ⊆ O E . Moreover, for any ε > 0, one has , and hence the above is bounded by a constant depending on ε times Notice also that the integral over t can be truncated to the region t > T for some T > 0 by our choice of F (see Lemma 5.1). Thus the integral above is bounded by Lemma 3.4.
Finally, we prove a proposition which motivated our definition of V in the first place. Recall that if Π is a cuspidal automorphic representation of G(A F ) 1 distinguished by H(A F ) 1 = H(A F ) then it is the base change of a cuspidal automorphic representation Π ′ of GL 2 (A F ′ ) 1 . Moreover, in this case there is a certain H(A F )-invariant linear functional on the space of Whittaker functions of Π S with respect to ψ S denoted in [FLO,§3.3 and Definition 10.1] by where the product is over places v ′ of F ′ dividing S. Since H is quasi-split, these linear functionals α Π ′ S are nonzero by [FLO,Corollary 10.3 and Remark 10.4]. We note that these distributions depend on the choice of measure on H(F )\H(A F ) (see loc. cit.).
Proposition 5.3. If Π is a cuspidal automorphic representation of G(A F ) 1 , spherical at all places and f ∈ C ∞ c (G(A F ) 1 //K S ) then Here Π ′ is a cuspidal automorphic representation of G ′ (A F ) 1 = GL 2 (A F ′ ) 1 whose base change to G(A F ) 1 is Π and W 0S ∈ W(Π S , ψ S ) is the Whittaker newform.
Proof. By construction the operator Π(V ) projects the space of Π onto the line spanned by the Whittaker newform ϕ of Π as does Π(Ψ). Thus Recognizing the Whittaker expansion in the expression above (compare (3.1.4)) we obtain The proposition now follows from [FLO,Theorem 10.2].

5.2.
Proof of Theorem 1.2. By Theorem 1.1 we have that where the (absolutely convergent) sum is over all cuspidal automorphic representations of G 0 (A F ) 1 . Recall that each π 0 admits a unique cuspidal base change π 0E to G(A F ), and π 0E ∼ = σ 0E if and only if π 0 ∼ = σ 0 ⊗ χ for some character χ of F 0 \A × F 0 trivial on N E/F 0 (A × F 0 ). This was proven by Langlands [La1] in the prime degree case, and for the deduction of the general case, which is easy since the extension E/F ′ has odd degree, see [LR].
Thus, by Lemma 4.2, the sum (5.2.1) is equal to where the latter sum is over all cuspidal automorphic representations of G(A F ) 1 invariant under τ . The theorem now follows from Proposition 5.3.