Geometrization of Trace Formulas

Following our joint work arXiv:1003.4578 with Robert Langlands, we make the first steps toward developing geometric methods for analyzing trace formulas in the case of the function field of a curve defined over a finite field. We also suggest a conjectural framework of geometric trace formulas for curves defined over the complex field, which exploits the categorical version of the geometric Langlands correspondence.


Introduction
Recently, a new approach to functoriality of automorphic representations [L1] has been proposed in [FLN] following earlier work [L2, L3, L4] by Robert Langlands. The idea may be roughly summarized as follows. Let G and H be two reductive algebraic groups over a global field F (which is either a number field or a function field; that is, the field of functions on a smooth projective curve X over a finite field), and assume that G is quasi-split. Let L G and L H be the Langlands dual groups as defined in [L1]. The functoriality principle states that for each homomorphism L H → L G there exists a transfer of automorphic representations from H(A) to G(A), where A is the ring of adéles of F . In other words, to each L-packet of automorphic representations of H(A) should correspond an L-packet of automorphic representations of G(A), and this correspondence should satisfy some natural properties. Functoriality has been established in some cases, but is unknown in general.
In [L2,L3,L4,FLN] the following strategy for proving functoriality was proposed. In the space of automorphic functions on G(F )\G(A) one should construct a family of integral operators which project onto those representations which come by functoriality from other groups H. One should then use the trace formula to decompose the traces of these operators as sums over these H (there should be sufficiently many operators to enable us to separate the different H). The hope is that analyzing the orbital side of the trace formula and comparing the corresponding orbital integrals for G and H, one should ultimately be able prove functoriality.
In the present paper we make the first steps in developing geometric methods for analyzing these orbital integrals in the case of the function field of a curve X over a finite field F q . We also suggest a conjectural framework of "geometric trace formulas" in the case of curves defined over the complex field.
In [FLN] a family of averaging operators K d,ρ has been constructed. They depend on a positive integer d and an irreducible representation ρ of L G. The operator K d,ρ is expected to have the following property: for sufficiently large d, the representations of G(A) that contribute to the trace of K d,ρ are those coming by functoriality from the groups H such that the dual group L H has non-zero invariant vectors in ρ. To prove this, we wish to use the trace formula (1.1) Tr K d,ρ where K d,ρ (x, y) is the integral kernel of K d,ρ on G(F )\G(A) × G(F )\G(A). 2 Thus, we need to express the right hand side of this formula as the sum of orbital integrals and to compare these orbital integrals for G and the groups H described above. 3 Here an important remark is in order. There are two types of automorphic representations of G(A): tempered (or Ramanujan) and non-tempered (or non-Ramanujan), see Section 3.3 for more details. As explained in [FLN], for the above strategy to work, one first needs to remove the contribution of the non-tempered representations from the trace of K d,ρ . This was the focus of [FLN], where it was shown how to separate the contribution of the trivial representation of G(A) (which is in some sense the most non-tempered).
In the first part of this paper we show that the kernel K d,ρ may be obtained using the Grothendieck's faisceaux-fonctions dictionary from a perverse sheaf K d,ρ on a certain algebraic stack over the square of Bun G , the moduli stack of G-bundles on X. Hence the right hand side of the trace formula (1.1) may be written as the trace of the Frobenius (of the Galois group of the finite field k, over which our curve X is defined) on theétale cohomology of the restriction of K d,ρ to the diagonal in Bun G × Bun G . We show that the latter is closely related to the Hitchin moduli stacks of Higgs bundles on X. The idea is then to use the geometry of these moduli stacks to prove the desired identities of orbital integrals for G and H by establishing isomorphisms between the corresponding cohomologies. In Section 4 we formulate a precise conjecture relating the cohomologies in the case that G = SL 2 and H is a twisted torus.
A prototype for this is the proof of the fundamental lemma (in the setting of Lie algebras) given by one of the authors [N3]. However, our situation is different. The argument of [N3] used decomposition of the cohomology of the fibers of the Hitchin map under the action of finite groups, whereas in our case the decomposition of the cohomology we are looking for does not seem to be due to an action of a group.
One advantage of the geometric approach is that the moduli stacks and the sheaves on them that appear in this picture have natural analogues when the curve X is defined over k = C, rather than a finite field. Some of our questions may be reformulated over C as well, and we can use methods of complex algebraic geometry (some of which have no obvious analogues over a finite field) to tackle them.
2 More precisely, we should consider the stabilized version of the trace formula, which is more efficient for comparing different groups. 3 We note that in general we should insert additional operators at the finite set S of closed points in X. However, in this paper, in order to simplify the initial discussion, we will restrict ourselves to the case when S is either empty or the operators inserted belong to the spherical Hecke algebra (with respect to a particular choice of a maximal compact subgroup of G(A)). Thus, only the unramified representations of G(A) (with respect to this subgroup) will contribute to (1.1).
This leads to a natural question: what is the analogue of the trace formula (1.1) over C? We discuss this question in the second, more speculative, part of the paper and formulate two conjectures in this direction.
The geometric analogue of the right hand (orbital) side of (1.1) is the cohomology of a sheaf on a moduli stack, as we explain in the first part. We would like to find a similar interpretation of the left hand (spectral) side of (1.1). Again, to simplify our discussion, consider the unramified case, but with a more general kernel K instead of K d,ρ which is bi-invariant with respect to a fixed maximal compact subgroup G(O) of G(A). Then the left hand side of (1.1) may be rewritten as follows (we ignore the continuous spectrum for the moment): where the sum is over irreducible representations π of G(A), unramified with respect to G(O), m π is the multiplicity of π in the space of automorphic functions, and N π is the eigenvalue of K d,ρ on a spherical vector in π fixed by this subgroup. We wish to interpret this sum as the Lefschetz trace formula for the trace of the Frobenius on thé etale cohomology of an -adic sheaf. It is not obvious how to do this, because the set of the π's appearing in (1.2) is not the set of points of a moduli space (or stack) in any obvious way.
However, recall that according to the Langlands correspondence [L1], the L-packets of irreducible (unramified) tempered automorphic representations of G(A) are supposed to be parametrized by (unramified) homomorphisms where W F is the Weil group of the function field F and L G is the Langlands dual group to G. This has been proved by V. Drinfeld for G = GL 2 [D1, D2] and L. Lafforgue for G = GL n , n > 2 [LLaf], but is unknown in general. Nevertheless, assume that this is true. Then the eigenvalues N π are determined by the Hecke eigenvalues of π, and in particular they are the same for all π in the L-packet L σ corresponding to any given σ : W F → L G. We will denote them by N σ . Further, we expect that if σ is unramified, then there is a unique representation π in the L-packet L σ which is unramified with respect to the subgroup G(O). Hence we will write m σ for m π . Then, assuming the Langlands correspondence, we obtain that (1.2) is equal to Ideally, we would like to describe (1.3) as the trace of the Frobenius on theétale cohomology of a sheaf on a moduli stack, whose set of k-points is the set of equivalence classes of homomorphisms σ. Such a stack does not exist if the curve X is defined over a finite field k. But when X is defined over C, we have an algebraic stack LocL G of de Rham local systems on X. Hence we can pose the following question: define a sheaf on this stack such that its cohomology (representing the left hand side of (1.1) in the complex case) is isomorphic to the cohomology representing the right hand side of (1.1). It would then be natural to call this isomorphism a geometrization of the trace formula (1.1).
Our main observation is that the answer may be obtained in the framework of a categorical form of the geometric Langlands correspondence, which is a conjectural equivalence between derived categories of coherent sheaves on the moduli stack LocL G of L G-local systems on a complex curve X and D-modules on the moduli stack Bun G of G-bundles on X. Such an equivalence has been proved in the abelian case by L. Laumon [Lau1] and M. Rothstein [R], and in the non-abelian case it has been suggested as a conjectural guiding principle by A. Beilinson and V. Drinfeld (see, e.g., [F1,VLaf,LL] for an exposition). 4 So far, this correspondence has been mostly studied at the level of objects. For example, the skyscraper sheaf supported at a given L G-local system E on X should go to the Hecke eigensheaf on Bun G with "eigenvalue" E. But if we have an equivalence of categories, then we also obtain non-trivial information about morphisms; namely, the Hom's between the objects corresponding to each other on the two sides should be isomorphic. The main point of the second part of this paper is that for suitable objects the isomorphism of Hom's yields the sought-after geometric trace formula. (More precisely, these objects are sheaves on the squares of Bun G and LocL G , which may be interpreted as "kernels" of functors acting on the above two categories. ) We propose a conjectural geometrization of the trace formula (1.1) in this framework. This is still a tentative answer, because several important issues remain unresolved, as we explain in Section 6.5. Nevertheless, we believe that it contains interesting features and hence even in this rough form it might provide a useful framework for a better geometric understanding of the trace formula as well as the geometric Langlands correspondence.
In deriving this formula, we apply the categorical form of the geometric Langlands correspondence to the sheaf K, which is the geometric incarnation of the kernel of the integral operator K. This sheaf is defined on an algebraic stack over the square of Bun G . It is natural to ask whether we can obtain meaningful analogues of the trace formula by applying the categorical Langlands correspondence to sheaves on Bun G itself.
In Section 7 we show that there is in fact an analogous statement which may be interpreted as a geometric analogue of the "relative trace formula" (also known as Kuznetsov trace formula, see, e.g., [J]). This formula has some interesting features. First, since it involves Whittaker functionals, only generic automorphic representations appear on the left hand side and hence, conjecturally, the non-tempered representations should not appear at all. Second, conjecturally, the factor m σ in the sum (1.3) should disappear because each generic L-packet is expected to contain a single irreducible representation with a non-zero Fourier coefficient.
The price we pay for this is that in the sum (1.3) appears a weighting factor, L(σ, ad, 1) −1 , the reciprocal of the value of the L-function of σ in the adjoint representation at s = 1 (here we assume that G is simple, but our analysis may be extended to more general reductive groups). Thus (not counting automorphisms of σ), we obtain the sum (1.4) σ N σ · L(σ, ad, 1) −1 .
Conjecturally, for non-generic L-packets the L-function L(σ, ad, s) has a pole at s = 1 (see [GP, Ic, IcIk]) and hence these representations should disappear from the sum (1.4).
The insertion of this factor in the context of the trace formulas discussed in [L2] (which are very close to the trace formulas considered here) was originally suggested by P. Sarnak [S] and further studied by A. Venkatesh [Ve], for the group GL 2 in the number field context. From our point of view, this factor has a natural geometric interpretation, as one coming from the Atiyah-Bott-Lefschetz fixed point formula (see Sections 5.2, 6.4, and 7.4 for more details).
On the right hand side of the relative trace formula we obtain certain analogues of the Kloosterman sums, which are represented geometrically by cohomologies of moduli stacks that are similar to the ones appearing in the geometrization of the ordinary trace formula and have been previously studied in [N1] and [FGV1]. Understanding relations between these Kloosterman sums, and the corresponding cohomologies, for G and the groups H discussed above may give us another way to approach functoriality.
The paper is organized as follows. In Section 2 we introduce the main ingredient of the geometric trace formula, the sheaf K d,ρ on the square of Bun G , and its analogues. This sheaf, viewed as a kernel, gives rise to a functor acting on the derived category of sheaves on Bun G . In Section 3 we consider the action of this functor on Hecke eigensheaves. We prove that for large d this functor annihilates those eigensheaves which do not come from smaller groups H by functoriality. Next, we look at the right hand (orbital) side of the trace formula in Section 4. We show that it corresponds to the cohomology of the restriction of our sheaf to the diagonal. We study this restriction in more detail and connect it to Hitchin type moduli stacks of Higgs bundles. We then formulate a precise conjecture linking the cohomology in the case that G = SL 2 and H is a twisted torus.
In the second part of the paper, we begin by describing a conjectural geometrization of the left hand (spectral) side of (1.1) Section 5. Then we explain in Section 6 how a geometrization of the trace formula appears in the context of the categorical Langlands correspondence. We state the geometric trace formula in Conjecture 6.1. Finally, we develop the needed formalism and state the relative geometric trace formula (Conjecture 7.2) in Section 7. As a byproduct of this discussion, we formulate a conjectural generalization of the Atiyah-Bott-Lefschetz fixed point formula for algebraic stacks in Section 6.4. of the Whittaker functional, and David Nadler for helpful discussions of the conjectural generalization of the Atiyah-Bott-Lefschetz fixed point formula for stacks presented in Section 6.4. E.F. is grateful to David Ben-Zvi, David Eisenbud, Alexander Givental, Dick Gross, Michael Harris, Atsushi Ichino, David Kazhdan, Vincent Lafforgue, Sergey Lysenko, and Martin Olsson for valuable discussions.

2.
The right hand side of the trace formula Let X be a smooth projective geometrically connected curve X over a field k which will be a finite field F q or the field of complex numbers. Denote by F the field of rational functions on X and A the ring of adéles of F . For each closed point x ∈ |X| we denote by F x the completion of F at x and by O x its ring of integers.
Let G be a reductive group scheme over X which is a quasi-split form of a split connected reductive algebraic group G over the field k. This means that G becomes split after pull-back to anétale cover X of X.
We wish to find a geometric incarnation of the trace formula where K is an operator on the space of automorphic functions corresponding to a kernel K(x, y): K(x, y)f (y)dy.
Here we choose Haar measure on G(A) normalized so that the volume of the fixed maximal compact subgroup In what follows we will restrict ourselves to unramified automorphic representations and we will assume that K is in the (restricted) tensor product of the spherical Hecke algebras (with respect to G(O x )) over all closed points x ∈ |X|.
Let Bun G be the algebraic moduli stack of G-bundles on X. Denote by ker 1 (F, G) the kernel of the map For simplicity, we will assume that ker 1 (F, G) = 0. Then we have where Bun G (k) is the set (more precisely, groupoid) of k-points of the moduli stack Bun G of G-bundles on X. Hence the kernel K may be viewed as a function on Bun G (k) × Bun G (k).
Then the right hand side of (1.1) may be rewritten as (see, e.g., [BeDh]). For a class of kernels K described below, we will give a geometric interpretation of this sum as the trace of the Frobenius on the cohomology of an -adic sheaf on an algebraic stack over the diagonal in Bun G × Bun G .
2.1. The dual group. Let L G be the Langlands dual group to G, HereǦ is the dual group of G taken here to be defined over the field of complex numbers if k = C but to be over Q (where is relatively prime to q) if k = F q , and Γ is the (étale) fundamental group π 1 (X, x) of X for some geometric point x of X that acts onǦ through a finite quotient. This action determines the quasi-split form G. In particular, if this action is trivial, then L G = G × Γ and G = X × G. We will fix a splitting of G andǦ so that in particular we can talk about dominant (co)weights. The fundamental group Γ acts on the abelian group X * of all weights (or characters) of the maximal torus ofǦ, preserving the cone X + consisting of the dominant ones. Recall that X + is equipped with a partial order: µ ≥ µ if and only if µ − µ is a sum of simple roots ofǦ. Since the action of Γ preserves the set of simple roots, it also preserves the partial order on X + . It also induces a partial order on the set [X + /Γ] of orbits of Γ on X + .
which is an ind-scheme over k [BD, MV]. These orbits are called Schubert cells, and they are labeled by elements µ the set X + of dominant wights of the maximal torus in the dual groupǦ. We denote the orbit corresponding to µ by Gr µ . We will write inv(E, E ) = µ if the pair (E, E ) belongs to Gr µ . Recall that Gr µ is contained in the closure Gr µ of Gr µ if and only if µ ≥ µ .
Recall the Hecke stack H = H(X, G) (see [BD]) that classifies quadruples where x ∈ X, E, E ∈ Bun G and φ is an isomorphism We have two natural morphisms p, p : H → Bun G sending such a quadruple to E or E and the morphism s : H → X. Since Bun G is an algebraic stack, so is H(X, G). However, if we fix E , then we obtain an ind-scheme over X, which is called the Beilinson-Drinfeld Grassmannian (see [BD, MV]). Let H (X, G) be the stack classifying the quadruples x is the punctured disc around x. We have a natural morphism H(X, G) → H (X, G) (restricting E to D x and φ to D • x ), which is in fact an isomorphism, according to a strong version of a theorem of Beauville-Laszlo [BL] given in [BD], Sect. 2.3.7. Therefore we obtain that the morphism s × p : H(X, G) → X × Bun G sending the above quadruple to (x, E) is a locally trivial fibration with fibers isomorphic to the affine Grassmannian Gr = G((t))/G [[t]].
For every orbit µ ∈ X + we define the closed substack H µ of H(X, G) by imposing the inequality It is a scheme over X × Bun G with fibers isomorphic to Gr µ .
Suppose now that G is a quasi-split form of G whose dual group L G is a semi-direct productǦ Γ given by a finite action of Γ onǦ, fixing a given splitting. Let Γ be the kernel of this action which is a normal subgroup of finite index in Γ. This is equivalent to a finiteétale covering π : X → X with a geometric point x ∈ X over x ∈ X such that the pull-back of G to X is split. The group scheme G over X is isomorphic to and a G-torsor on X is nothing but a G-torsor E on X which is Γ/Γ -equivariant in the sense that for each γ ∈ Γ/Γ we are given an isomorphism i γ : γ * ( E) ∼ −→ E (where γ * is induced by both the action of γ on X and on G), and these isomorphisms satisfy the compatibility conditions i γ 1 i γ 2 = i γ 1 γ 2 for all γ 1 , γ 2 ∈ Γ/Γ .
We define the Hecke stack H(G, X) as the algebraic stack which classifies quadruples where x ∈ X, E, E ∈ Bun G and φ is an isomorphism Equivalently, it classifies (x, E, E , φ) where x ∈ X, E, E are Γ/Γ -equivariant Gbundles on X (in the above sense), and φ is an isomorphism which is invariant under the Γ/Γ -action on E and E . For every geometric point x of X, the restriction of G to the formal disc D x around x is isomorphic to G, and there is an isomorphism between G| Dx and the constant group scheme over D x with the fiber G, well-defined up to the finite action of Γ on G. It follows that the morphism H(X, G) → X × Bun G sending the quadruple (x, E, E , φ) to (x, E) is a fibration over X × Bun G with fibers over geometric points isomorphic to the affine Grassmannian Gr = G((t))/G [[t]].
Furthermore, if E and E are G-torsors over D x equipped with an isomorphism over the punctured disc D • x , then inv x (E, E ) is well-defined as an orbit [µ] of Γ on X + . Since we have a partial order on the set [X + /Γ] of orbits of Γ on X + , it makes sense to write inv x (E, E ) ≤ [µ]. Hence, for every orbit [µ] of Γ acting on X + , we define the closed substack H [µ] of by imposing the inequality It is again a fibration over X × Bun G with fibers isomorphic to the union Gr [µ] of Gr µ ⊂ Gr with µ in the Γ-orbit [µ].
The following description of H(X, G) will be useful in Section 2.5. Let H be the stack which classifies quadruples (

The morphism
H → X × Bun G sending (x , E, E , φ) to (x , E) makes it into an ind-scheme over X ×Bun G whose fibers are isomorphic to the affine Grassmannian Gr of G. We also have a natural action of Γ/Γ on H lifting the action on X , and (2.5) H(X, G) H/(Γ/Γ ).
2.3. Symmetric powers. For every positive integer d, we introduce a sort of dth symmetric power of H [µ] . Let X d be the dth symmetric power of X which is defined as the scheme-theoretic (also known as GIT) quotient of X d by the action of the symmetric group S d . Thus, X d classifies effective divisors of degree d on X. This is a smooth algebraic variety (see, e.g., [S], p.95). The algebraic stack H d, [µ] over k classifies the data is an effective divisor on our curve X of degree d (equivalently, a point of X d ), E and E are two principal G-bundles on X, and φ is an isomorphism between them over X − supp(D), such that for each i the inequality is satisfied. Consider the morphism sending the quadruple (2.6) to (D, E). Its fiber over a fixed D = i n i [x i ] and E ∈ Bun G isomorphic to the product The algebraic stack H d, [µ] is a closed substack of the dth symmetric power H d of the Hecke stack H which is defined as the classifying stack of the quadruples (2.6) as above, but with the condition inv x i (E, E ) ≤ [n i µ] removed. To verify that these conditions define a closed substack, it is convenient to introduce a chain version of H [µ] . Let H d [µ] be the algebraic stack classifying We have a morphism and φ is obtained by composing φ 1 , . . . , φ d on their common domain of definition X − supp(D). The morphism (2.11) is proper and H d, [µ] is its image; its geometric points are characterized by the inequality (2.8).
We also have a chain version H d of the entire Hecke stack H, which classifies the same objects as above, but without the condition that inv

2.4.
Examples. Let us discuss examples of the above construction. Let G be the multiplicative group G m over X. Recall that G m -principal bundles on X are equivalent to line bundles, or invertible O X -modules. Their classifying stack Pic has a well-known structure. The degree of a line bundle induces a bijective map from the set of connected components of Pic to Z. The kernel of this map, denoted by Pic 0 , classifying line bundles of degree 0, is isomorphic to the quotient of an abelian variety -namely, the Jacobian Jac of X -by the trivial action of G m (which is the group of automorphisms of line bundles) In this case, the reduced part of the affine Grassmannian is the discrete set Z (however, the non-reduced structure at each point is highly non-trivial 5 ). The Hecke stack H classifies quadruples (x, L, L , φ), where (x, L, L ) ∈ X × Pic × Pic and φ is an isomorphism At the level of k-points we then have L = L(µx) for some integer µ ∈ Z. 6 Hence the reduced part of H is Pic ×X × Z in this case. For every µ ∈ Z, which is the set X + in this case, we have the inclusion (of the reduced parts, which we denote by the same symbols by abuse of notation) The other projection onto Pic is (L, x, µ) → L = L(µx).
Since G m is commutative, H d can be identified with The dth symmetric power of Z consists of elements of the form ((d 1 , µ 1 ), . . . , (d r , µ r )) where d 1 , . . . , d r are positive integers satisfying d 1 + · · · + d r = d and µ 1 < · · · < µ r is a strictly increasing sequence of integers. The fiber of (X × Z) d over ((d 1 , µ 1 ), . . . , (d r , µ r )) is X (d 1 ) × · · · × X (dr) projecting onto X d by the obvious map of adding the divisors. In particular, for the element (d, µ) ∈ Z d , we have the component Our second example is a non-split one-dimensional torus H over X. Such a torus is given by a µ 2 -torsor (equivalently,étale double cover) π : X → X. Let us denote by τ the non-trivial involution on X . The group Γ ⊂ Γ has index 2 in this case and Γ/Γ = {1, τ }. We assume that X is geometrically connected so that H remains non-split over X ⊗k. An H-torsor on X is a line bundle L over X whose norm down to X is equipped with a trivialization β : The group of connected component of Bun H is Z/2Z. The neutral component of Bun H is a quotient of an abelian variety -namely, the Prym variety of X -by the trivial action of Z/2Z which is the group of automorphism of any pair (L, β) as above.
The set X + = Z is acted on by the involution τ (µ) = −µ. The set of orbits [X + /Γ] may therefore be indexed by non-negative integers m ∈ Z ≥0 . The moduli stack H 5 Indeed, for a general k-algebra R the quotient R((t))/R [[t]] is isomorphic to the product of Z and the set of polynomials of the form 1 + r−1t −1 + r−2t −2 + . . ., where the ri are nilpotent elements of R.
6 However, this is not true at the level of R-points, where R is a ring with nilpotents, as explained in the previous footnote.
(again, the reduced part) decomposes into connected components H m with H 0 = Bun H ×X and H m = Bun H ×X for m ≥ 1. The second projection onto Bun H may be described as follows: for m = 0 we map (L, x) → L, and for m ≥ 1 we map (L, x ) → L(m(x − τ (x ))). For every d ≥ 1 we have the following description of H d : The connected components of (X (X × Z >0 )) d are the fibers over elements of (Z >0 ) d , which are r-tuples ((d 0 , m 0 ), (d 1 , m 1 ), . . . , (d r , m r )) with 0 = m 0 < m 1 < · · · < m r , where d 0 ≥ 0 and d 1 , . . . , d r > 0 are integers such that 2.5. The perverse sheaf. Let ρ be a finite-dimensional representation of L G. For simplicity, we will restrict ourselves to irreducible representations. If G is split, for example, then such ρ must be of the form ρ = ρ µ ⊗ ρ L , where ρ µ is the irreducible representation ofǦ with highest weight µ and ρ L is the representation of Γ attached to an irreducible local system L on X.
For any semi-simple finite-dimensional representation ρ of L G, we will define perverse sheaves K ρ on H and K d,ρ on H d . They may be constructed more conceptually using an equivalence of categories suggested in the Appendix to [FGV2], but for our purposes the explicit construction given below will suffice.
First, we assume that G is split and ρ is an irreducible representation of L G =Ǧ × Γ which is trivial on Γ. Recall that we have defined a closed substack H µ of H such that the projection H µ → X × Bun G is a locally trivial fibration with fibers isomorphic to the Schubert variety Gr µ .
Using the geometric Satake correspondence [MV], we associate to each irreducible representation ρ = ρ µ of L G of highest weight µ ∈ X + the sheaf With the above cohomological shift and Tate twist, locally over X × Bun G , K ρ is isomorphic to where IC(Gr µ ) denotes the intersection cohomology sheaf of Gr µ and Q is the constant sheaf on X × Bun G . Hence K ρ is perverse and pure of weight 0 along the fibers of the projection to X × Bun G . By abuse of notation, we will use the same symbol for i * K ρ , where i : H µ → H, so that K ρ will also be viewed as a sheaf on H.
We define the Hecke functor H ρ = H µ as the integral transform corresponding to the kernel K ρ (see [BD]): For x ∈ |X|, let H x be the fiber of H over x, and p x , p x : H x → Bun G the corresponding morphisms. Denote by K ρ,x the restriction of K ρ to H x . Define the functor H ρ,x by the formula The function corresponding to the sheaf K ρ,x is the kernel K ρ,x of the Hecke operator H ρ,x and so the functor H ρ,x on the derived category D(Bun G ) of -adic sheaves on Bun G is a geometric analogue of the Hecke operator H ρ,x .
Next, suppose that G is split and ρ is a more general irreducible representation of L G =Ǧ × Γ of the form ρ = ρ µ ⊗ ρ L , where ρ µ is the irreducible representation ofǦ of highest weight µ ∈ X + and ρ L is the representation of Γ attached to an irreducible local system L on X. Now we put (2.14) where s is the obvious projection onto X. By abuse of notation, we will use the same symbol for i * K ρ where i : H µ → H, so that K ρ could also be viewed as a sheaf on H (again, it is perverse on the entire H only up to a cohomological shift, but perverse along the fibers of the projection to X × Bun G ).
Let ρ be an irreducible representation ofǦ Γ. Suppose that its restriction toǦ × Γ is semi-simple, so that over a finite collection of pairs (µ, L), where µ ∈ X + and L is an irreducible local system on X . Here ρ µ is an irreducible representation ofǦ of highest weight, ρ L is the representation of Γ attached to L, and the integer n µ,L is the multiplicity. We attach to this ρ a sheaf K ρ on H(X, G) as follows. First, we construct, in the same way as above, the sheaf K ρ on the stack H introduced in Section 2.2, using the fact that this stack is a locally trivial fibration over X × Bun G with fibers isomorphic to the affine Grassmannian Gr of G. Since ρ is a representation ofǦ Γ (and not jusť G × Γ ), we obtain that K ρ is a Γ/Γ -equivariant sheaf on H. Hence it descends to a sheaf on the quotient of H by Γ/Γ , which is nothing but H(X, G), by formula (2.5). This gives us the desired sheaf K ρ on H(X, G).
We note that this construction yields a generalization of the geometric Satake correspondence to the quasi-split case.
2.6. Symmetric power. Next, we will construct the symmetric power K d,µ of K µ for any positive integer d.
First, we recall the construction of the symmetric power of a local system L on X. From our point of view, this is a particular case of the general construction corresponding to the trivial G (or general G but µ = 0). Let L be a rank n local system on a curve X over a field k (which, as before, is either C or F q ); that is, a rank n locally constant sheaf ( -adic, if k = F q ) on X. Recall from Section 2.3 that X d = X d /S d is the dth symmetric power of X. This is a smooth algebraic variety defined over k, whose k-points are effective divisors on X of degree d.
We define a sheaf on X d , denoted by L d and called the dth symmetric power of L, as follows: where π d : X d → X d is the natural projection. The stalks of L d are easy to describe: they are tensor products of symmetric powers of the stalks of L.
where S n i (L x i ) is the n i -th symmetric power of the vector space L x i . In particular, the dimensions of the stalks are not the same, unless n = 1. In the case when n = 1 the sheaf L d is in fact a rank Proof. The natural projection π d : X d → X d is a finite flat map which isétale over the open subscheme X d,• of X d . As a finite map, π d is in particular small in the sense of Goresky and MacPherson. If p : Y → Z is a small map and F a local system on Y , then p * (F)[dim Y ] is a perverse sheaf on Z isomorphic to the intermediate extension of its restriction to any open dense subset of Z. Hence we obtain that this is true for the sheaf π d * (L d ) on X d . Taking the S d -invariants, we obtain the statement of the lemma.
This lemma suggests the following general construction of F d,ρ . We note that a similar construction was proposed by Laumon in [Lau2] By restriction, (K ρ which is equipped with an action of S d . By finiteétale Galois descent, we get a perverse sheaf over Now, we can apply the functor of intermediate extension to obtain a perverse sheaf In the above discussion, we have neglected a shift, since K ρ defined as in (2.14) is only a perverse sheaf after a shift. Recall that the effect of this shift makes the restrictions of K ρ to the fibers of the map H [µ] → X × Bun G perverse sheaves. With the above definition, the restrictions of K d,ρ to the fibers of the maps over the open subset X • d × Bun G are also perverse sheaves. We will see that this is also true for all other fibers.
The above definition of K d,ρ is probably the quickest one, but it is impractical for many purposes as the intermediate extension is not very explicit. The following lemma, essentially proved by Mirković and Vilonen in [MV], uses the whole diagram (2.17) to make the definition of K d,ρ more explicit.
Lemma 2.2. The morphism π d H of the diagram (2.17) is small in the stratified sense of [MV]. Moreover, for all Assume now till the end of this subsection that G is split, L G =Ǧ×Γ and ρ = ρ µ ⊗ρ L . In this case the construction of K d,ρ can be made more explicit along the lines of Springer's construction of representations of the symmetric group. Consider the closed substack H d µ of H d defined in Section 2.3 and the perverse sheaf The first assertion of Lemma 2.2 guarantees the equivalence of the two definitions.
According to the second assertion of Lemma 2.2, the restriction of K d,ρ to the fibers of the map H d,µ → X d × Bun G is also perverse. Now it is possible to describe the fibers of K d,ρ using the geometric Satake equivalence [MV]. Fix an effective divisor of degree d Fix a principal G-bundle E ∈ Bun G and choose trivializations of E on the formal discs D x i . Then the fiber of H d over (D, E) is the product of copies of the affine Grassmannian attached to the points x i , The restriction K d,ρ to this fiber is the external tensor product of spherical perverse sheaves where S n i is the n i -th symmetric power based on the symmetrical monoidal structure of the category of spherical perverse sheaves on the affine Grassmannian. In other words, S n i (IC(Gr x i ,µ ) is the perverse sheaf on the affine Grassmannian which corresponds to the n i th symmetric power of ρ under the geometric Satake equivalence [MV] (pr i denotes the projection of (2.23) onto the ith factor). This formula follows from the second assertion of Lemma 2.2 and from the fact that the definition of Drinfeld's commutativity constraint used in [MV] was also based on this lemma. Let K d,ρ be the function on Bun G (k) × Bun G (k) corresponding to the sheaf K d,ρ . Then it is given by the formula We fix an irreducible three-dimensional representation of L G as follows. Let X → X be anétale double cover of X which is geometrically connected. Choose a geometric point x of X over the given point x ∈ X. This gives rise to a non-trivial character ρ E of order two of Γ. The first component ρ µ is the irreducible three-dimensional representation of PGL 2 . Let V be the tautological twodimensional representation of GL 2 . Then we have the following formula for the lifting of ρ µ to a representation of PGL 2 : If we chose the splitting of SL 2 with the diagonal torus and the subgroup of upper triangular matrices as Borel subgroup, then µ is simply the where D ∈ X d is an effective divisor of degree d, E, E are rank two vector bundles on X with trivialized determinant and φ is an isomorphism of the restrictions of the vector bundles E and E to X − supp(D) such that det(φ) = 1, and φ can be extended as a morphism of vector bundles E → E (D). The representation ρ µ corresponds to the intersection cohomology complex with the appropriate shift where pr X d is the projection on X d and E d is the dth symmetric power of E, which is a rank one local system on X d of order two corresponding to ρ E .
Next, we consider a closely related example for a one-dimensional twisted torus H over X as in Section 2.4. The dual group L H is G m Γ, where Γ acts through a quotient n H : Γ → Z/2Z, and Z/2Z acts by the formula τ (t) = t −1 for the non-trivial element τ ∈ Z/2Z and t ∈ G m . Now, we have a homomorphism L H → L G mapping G m to the diagonal torus of PGL 2 and an element σ ∈ Γ to the element where w 0 is the permutation matrix and n H (σ) denotes the image of σ in Z/2Z = {0, 1}.
Consider the representation ρ = ρ µ ⊗ ρ E of L G, where ρ µ is the irreducible threedimensional representation of PGL 2 and ρ E is the one-dimensional representation of Γ of order two as above. If we restrict ρ to L H, it decomposes into a direct sum where ρ 1 is a two-dimensional representation of L H and ρ 0 is the one-dimensional representation which is trivial on G m , and on the factor Γ is given by the formula Assume from now on that E = E H . Then the representation ρ H of L H corresponds to the direct sum where the sum is over all non-negative integers d 0 and d 1 such 2.8. Integral transforms. We have defined a morphism and a perverse sheaf K d,ρ on H d attached to any semi-simple finite-dimensional representation ρ of L G. Let us denote by p d and p d the two projections H d → Bun G mapping the quadruple (2.6) to E and E . We use the sheaf K d,ρ to define an integral transform functor K d,ρ on the derived category D(Bun G ) of -adic sheaves on Bun G by the formula Functors of this kind were first introduced in [FGV2] in the case of GL n under the name "averaging functors." An example is the functor Av d E of [FGV2] which plays the role of a "projector" on the Hecke eigensheaf corresponding to an irreducible rank n local system E on X, in the category of D-modules, or -adic sheaves, on Bun G . The functors K corresponding to these kernels are combinations of the Hecke functors, in which the positions of the points at which these functors are applied are allowed to vary over the dth symmetric power X d of X.
We generalize this construction to a larger family of integral transforms on D(Bun G ) as follows. For any scheme over the symmetric power S → X d we form the Cartesian product Denote by K d,ρ,S the restriction of K d,ρ to H d,S . Let p S and p S the two projections from H d,S to Bun G . Then we have an integral transform U is an open subset of X, we have an averaging operator over U that will be useful in the study of the ramified case.
2.9. Vector space. Let us form the Cartesian square S be a sheaf of the type introduced in the previous section and K the corresponding integral transform functor on the derived category D(Bun G ) of sheaves on Bun G . Let K = p * (K) (note that p is proper over the support of K). This is a sheaf on Bun G × Bun G which is the kernel of the functor K.
Note that the sheaf ∆ ! (Q ) on Bun G × Bun G is the kernel of the identity functor Id on the category D(Bun G ). Hence the vector space may be viewed as a concrete realization of the "categorical trace" RHom(Id, K) of the functor K. By adjunction and base change, the space (2.31) is isomorphic to ). Let D be the Verdier duality on H d,µ . It follows from the construction and the fact that . Therefore, up to a shift and Tate twist (and replacing L by L * ), the last space in (2.32) is isomorphic to Here we use the results of Y. Laszlo and M. Olsson [LO] on the six operations on -adic sheaves on algebraic stacks and formula (2.33).
Using the Lefschetz formula for algebraic stacks developed by K. Behrend [Be], we find that, formally, the trace of the (arithmetic) Frobenius on ) is equal (up to a power of q) to the sum (2.2), which is the right hand side of the trace formula (1.1).
For this reason, we will view the vector space (2.35) as a geometric incarnation of the right hand side of the trace formula (1.1).
There are some technical issues that have to be dealt with in order to make this precise, because M d is not an algebraic stack of finite type. In general, in order to make the trace of the Frobenius on (2.35) finite, it is necessary to introduce some stability condition on the objects that M d classifies (see Section 4.1). Imposing this stability condition should be viewed as the geometric counterpart of Arthur's truncation process of the trace formula (see [CL], where this is explained in the case of the trace formula for Lie algebras), but we will not discuss this issue in the present paper. Here we will consider the moduli stack M d without any stability conditions.
In Section 4 we will give a description of the stack M d that is reminiscent and closely related to the Hitchin moduli stack of Higgs bundles on the curve X [H1]. We will also conjecture that ∆ ! H (K d,ρ ) is a pure perverse sheaf on M d . If X is a curve over C, then the vector space (2.35) still makes sense if we consider K as an object of either the derived category of constructible sheaves or of D-modules on Bun G .

Eigenvalues of K d,ρ on Hecke eigensheaves
In this section we apply the functor K d,ρ to Hecke eigensheaves on Bun G .
3.1. Cohomology of symmetric powers. Recall the definition of the symmetric power of a local system L on X given in Section 2.6. We generalize this definition slightly, by allowing L to be a complex of local systems on X. 7 The resulting object L d will then be a complex of sheaves on X d .
Let us now compute the cohomology of X d with coefficients in L d . By Künneth formula, we have where the action of S d on the cohomology is as follows: it acts by the ordinary transpositions on the even cohomology and by signed transpositions on the odd cohomology. Thus, we find that This is true for any complex of local systems. The cohomological grading is computed according to the rule that d 0 does not contribute to cohomological degree, d 1 contributes d 1 , and d 2 contributes 2d 2 . In addition, we have to take into account the cohomological grading on L.
3.2. Connection to L-functions. Suppose that k = F q . Let us form a generating function where , thus the alternating sum of traces on the groups H j (X d , L d ).
There are two ways to compute it.
I. By the Lefschetz formula, we find that the generating function (3.2) is equal to where |X| is the set of closed points of X.
If we set t = q −s in formula (3.3), we obtain the L-function L(L, s) of the Galois representation associated to L (and defining representation def of GL n ). Thus we find that Tr(Fr, II. Formula (3.1) gives us the following expression for the generating function (3.2): .
If we substitute t = q −s , we obtain the Grothendieck-Lefschetz formula for the Lfunction L(L, s).

Hecke eigensheaves.
Recall that X is a geometrically connected smooth proper curve over a finite field k = F q , and G a reductive group scheme over X which is a quasi-split form of a constant group G. The Langlands dual group of G, defined over Q , is L G =Ǧ Γ where Γ is the (étale) fundamental group Γ = π 1 (X, x) that acts oň G by the data defining the quasi-split form G. Let W F be the Weil group of the function field F of the curve X. An Arthur parameter is an equivalence class of homomorphisms According to the conjectures of Langlands and Arthur, equivalence classes of irreducible automorphic representations of G(A) may be parametrized by Arthur parameters. Such a representation π is called tempered (or Ramanujan) if σ| SL 2 is trivial. Otherwise, it is called non-tempered (or non-Ramanujan).
If an automorphic representation π = x∈|X| π x of the adélic group G(A) has the Arthur parameter σ, then for all closed points x ∈ |X| where π x is unramified the restriction σ| W F is also unramified and the Satake parameter of π x is equal to the conjugacy class Now we discuss a geometric analogue of this -in the unramified case, in order to simplify the discussion. The geometric analogue of the notion of automorphic representations (or, more precisely, the corresponding spherical automorphic function) is the notion of Hecke eigensheaf which we now recall.
Let ρ be a continuous representation of L G on a finite-dimensional vector space V and let σ be an Arthur parameter. Then we obtain a representation ρ where each ρ • σ i is a continuous representation of W F . Assume that each of them is unramified. Then it gives rise to an -adic local system L (ρ•σ) i on X. Now we define L ρ•σ to be a complex of local systems on X with the trivial differential Recall the Hecke functors H ρ and H ρ,x introduced in Section 2.5 (geometric analogues of the Hecke operators corresponding to ρ). The following definition is a slight generalization (to the case when σ| SL 2 is non-trivial) of the definition given in [BD].
and these isomorphisms are compatible for different ρ with respect to the structures of tensor categories.
It follows from the above identity that for every x ∈ X, we have an isomorphism where L ρ•σ,x is the stalk of ρ • σ at x. Formula (3.6) follows from (3.5) by base change for the cohomology with compact support.
In X is a smooth projective connected curve over C, then the above definition also makes sense if we take as σ a homomorphism SL 2 × π 1 (X, x) → L G and as L (ρ•σ) i the corresponding local system on X.
Now we compute the "eigenvalues" of the integral transform K d,ρ on F σ . The following result is Lemma 2.6 of [FLN].
Lemma 3.2. If F = F σ is a Hecke eigensheaf with eigenvalue σ, then for every representation ρ of L G and every positive integer d we have Proof. We will restrict ourselves to the split case. The general case is obtained from the split case by the descent method. From now on, the group scheme G is of the form G × X, L G isǦ × Γ, the representation ρ is assumed to be of the form ρ = ρ µ ⊗ ρ L where ρ µ is a irreducible representation ofǦ of highest weight µ and ρ L is the continuous representation of Γ attached to an irreducible local system L over X.
Consider the chain version of the Hecke stack from Section 2.3 with truncation parameter µ, . As in Section 2.8, let us denote by p d , p d the two projections to Bun G and q d = pr X d ×p d , where pr X d is the projection to X d . Consider the integral transform . By using repeatedly (3.5), we obtain By pushing forward along X d , we get The symmetric group S d acts on the right hand side in the obvious way and the invariant part is On the other hand, recall the morphism Hence the the S d -invariant part of the left hand side of (3.8) is K d,ρ (F σ ), and we obtain the desired formula (3.7).
is then equal to the Euler characteristic of the constant local system of rank dim ρ on X). Hence we obtain that and the statement of the corollary follows.
We can translate the above lemma and corollary to the level of functions: the eigenvalue of the operator K d,ρ (see Section 2.8) on an unramified Hecke eigenfunction f σ is equal to the q −ds -coefficient of the automorphic L-function L(ρ, σ, s). This can indeed be shown directly at the level of functions as follows (see also the proof of Lemma 2.6 in [FLN]).
When K d,ρ acts on F, for a fixed effective divisor D = i n i [x i ] of degree d we apply modifications at the closed points x i . These modifications are described by the Schubert varieties Gr n i µ with the "kernel" S n i (IC(Gr µ )), which is the perverse sheaf on Gr n i µ corresponding to the n i th symmetric power of the representation ρ.
When we apply the Hecke functor corresponding to S n i (IC(Gr µ )) at the point x i to our sheaf F, we simply tensor it by (note that in general this is a complex of vector spaces). But this is precisely the stalk of (ρ • σ) d at D (see formula (2.16)). In other words, because we are applying K d,ρ to a Hecke eigensheaf F, we are effectively replacing each Hecke operator by the corresponding "eigenvalue", that is, Then we need to integrate over all possible D. This simply means taking the cohomology of X d with coefficients in (ρ • σ) d . The result is in Lemma 3.2. Now, each step of this calculation makes sense at the level of functions. Hence we obtain that the eigenvalue of the operator K d,ρ on f σ is equal to the trace of the Frobenius on H Let F 0 = Q | Bun G , the constant sheaf on Bun G . This is the geometric analogue of the trivial representation of G(A). Let us apply the Hecke operator H µ,x to F 0 . For any G-principal bundle E, the fiber of p −1 (E) ∩ H x is isomorphic to Gr x , once we have chosen a trivialization of E on the formal disc D x . Thus the fiber of H ρ,x (F 0 ) at E is isomorphic to This isomorphism is actually canonical in the sense that it does not depend on the choice of the trivialization of E on D x , so that the above isomorphism can be put in a family with respect to the parameter E. Hence we obtain that By the geometric Satake correspondence [MV], a complex of vector spaces, which is isomorphic to the representation ρ µ with the cohomological grading corresponding to the principal grading on ρ µ . In other words, and is the principal embedding on SL 2 . We conclude that the constant sheaf on Bun G is a Hecke eigensheaf with the eigenvalue σ 0 . This is in agreement with the fact that σ 0 is the Arthur parameter of the trivial automorphic representation of G(A). For example, if ρ µ is the defining representation of GL n , then the corresponding Schubert variety is P n−1 , and we obtain its cohomology shifted by (n − 1)/2, because the intersection cohomology sheaf IC(Gr µ ) is the constant sheaf placed in cohomological degree −(n − 1), that is where θ 1/2 = Q [−1](−1/2). This agrees with the fact that the principal grading takes values (n − 1)/2, . . . , −(n − 1)/2 on the defining representation of GL n , and each of the corresponding homogeneous components is one-dimensional.
Next, we find that where ρ µ • σ 0 is the complex described in the previous section. At the level of functions, we are multiplying the constant function by where P (ρ µ ) is the set of possible values of the principal grading on ρ µ and ρ µ,i is the corresponding subspace of ρ µ . For example, if ρ µ is the defining representation of GL n , then formula (3.9) reads (3.10) n−1 k=0 ζ(s + k − (n − 1)/2).
3.5. Decomposition of the trace. Now we wish to use Corollary 3.3 to decompose the trace of K d,ρ for d > (2g − 2) dim ρ as a sum over subgroups of L G. Let σ be an Arthur parameter. We attach to it two subgroups of L G: λ G = λ G σ is the centralizer of the image of SL 2 in L G under σ, and λ H = λ H σ is the Zariski closure of the image of W F in λ G σ under σ. The representation ρ • σ has non-zero invariants if and only if the restriction of ρ to λ H σ has non-zero invariants. Thus, assuming that Arthur's conjectures are true, we obtain that the trace of K d,ρ decomposes as a double sum: first, over different homomorphisms φ : SL 2 → L G, and second, for a given φ, over the subgroups λ H of the centralizer λ G φ of φ having non-zero invariants in ρ: The summands Φ triv, λ H correspond to the Ramanujan representations. Denote their sum by (Tr K d,ρ ) R . Note that if φ is non-trivial, then the rank of λ G φ is less than that of L G. As explained in [FLN], we would like to use induction on the rank of L G to isolate the Ramanujan part (Tr K d,ρ ) R in Tr K d,ρ . In other words, we need to isolate and remove the contribution of the non-Ramanujan representations. In [FLN] it was shown how to isolate the contribution of the trivial representation.
We then wish to decompose (Tr K d,ρ ) R over λ H, Here the sum should be over all possible λ H ⊂ L G such that λ H has non-zero invariant vectors in ρ • σ, and K H d,ρ H is the operator corresponding to ρ H = ρ|λ H for the group H(A). Proving formula (3.12) is the main step in the strategy to prove functoriality outlined in [FLN]. More specifically, we would like to use the orbital side of the trace formula (1.1) to establish (3.12).
Actually, comparisons of trace formulas should always be understood as comparisons of their stabilized versions. Therefore the traces in (3.12) should be replaced by the corresponding stable traces (see [FLN] for more details). Since the fundamental lemma has been proved [N3], the connection between the actual trace formula and the stabilized trace formula is now well-understood.
The spectral side of (3.12) (again, assuming Arthur's conjectures) is the sum of the eigenvalues of K d,ρ which are expressed in terms of the coefficients of the L-function of the corresponding Arthur parameter σ : W F → λ H, for different λ H. A precise formula for these eigenvalues is complicated in general, but we can compute its asymptotics as d → ∞. If we divide K d,ρ by q d (geometrically, this makes sense because of the dimension of X d ), then the asymptotics will be very simple: where N σ is the multiplicity of automorphic representations in the corresponding Lpacket. Indeed, the highest power of q comes from the highest cohomology, which in this case is (d 0 = 0, d 1 = 0 and d 2 = d in the notation of formula (3.1)). We have dim H 2 (X, ρ•σ) = m σ (ρ), the multiplicity of the trivial representation in ρ • σ (we are assuming here that this trivial representation splits off as a direct summand in ρ • σ), and Thus, as a function of q d , the eigenvalues of q −d K d,ρ on the Ramanujan representations grow as O(1) when d → ∞. For the non-Ramanujan representations corresponding to non-trivial φ : SL 2 → L G, they grow as a higher power of q d . For instance, the eigenvalue corresponding to the trivial representation of G(A) (for which φ is a principal embedding) grows as O(q d(ρ,µ) ), where µ is the highest weight of ρ. In general, it grows as O(q da ), where 2a is the maximal possible highest weight of the image of SL 2 ⊂ L G under φ acting on ρ. Thus, the asymptotics of the non-Ramanujan representations dominates that of Ramanujan representations. This is why we wish to isolate the Ramanujan part first and then decompose it over λ H.
The main question we want to answer is the following: how to observe the decompositions (3.11) and (3.12) on the orbital side of the trace formula, using geometry? 3.6. Example of SL 2 and the twisted torus continued. Let us go back to the case that G = X × SL 2 and the Langlands dual group is L G = PGL 2 × Γ. We consider the representation ρ = ρ µ ⊗ ρ E of L G, where ρ µ is the three-dimensional adjoint representation of PGL 2 and E is a non-trivial rank one local system over X of order two which is non trivial. We are now looking at the effect of the operator K d,ρ on a Hecke eigenfunction f σ attached to an Arthur parameter σ : SL 2 × Γ → L G.
Consider first the case of the Arthur parameter σ = σ 0 whose restriction to the factor SL 2 is non-trivial, and hence an isomorphism. It then follows that the restriction of σ to W F must be trivial. The automorphic representation of SL 2 (A) attached to this parameter is the trivial one-dimensional representation. In this case F ρ•σ is the graded local system that is, the direct sum of three copies of E put in cohomological degrees −2, 0 and 2.
Since H i (X, E) = 0 for i = 1, we have the direct sum of three copies of H 1 (X, E) put in degrees −1, 1 and 3. In particular H • (X, F ρ•σ 0 ) vanishes in even degrees. It follows from an obvious generalization of formula (3.1) that for a large integer d, we have Hence we obtain that the integral transform K d,ρ annihilates the contribution of the trivial representation of SL 2 (A), which is the only non-Ramanujan representation in this case. Thus, only Ramanujan representations of SL 2 (A) contribute to the trace formula for large d in this case.
Now we look at the Arthur's parameters σ whose restriction to SL 2 factor is trivial. The restriction of σ to W F is of the form where ϕ : W F → Γ is the canonical homomorphism, for some homomorphism σ + : W F →Ǧ. The local system L ρ•σ is of the form where L ρµ•σ + is the rank three local system given by the representation of W F given by the composition of σ + : W F → PGL 2 and the three-dimensional representation ρ µ of PGL 2 .
If either H 0 (X, L ρ•σ ) or H 2 (X, L ρ•σ ) is non-zero, then the three-dimensional representation ρ • σ must have either non-zero space of invariants or coinvariants. Since L ρµ•σ is semi-simple because of the purity, ρ • σ has non-zero invariants. Let λ H be the closure of the image of σ + . We want to understand when this subgroup has an invariant vector v in the adjoint representation V of PGL 2 .
We identify PGL 2 with the group SO 3 preserving the Killing form on V . There are two possibilities for the invariant vector: be isotropic or anisotropic.
If the invariant vector is v is isotropic, v belongs to the two-dimensional vector space v ⊥ . Since we have assumed σ + to be semi-simple, there exists another vector v 1 / ∈ v ⊥ such that σ + (W F ) preserves the line v 1 . Let v 2 be a vector generating the one-dimensional vector space v ⊥ ∩ v 1 ⊥ . We have a decomposition of V into lines preserved by σ + (W F ) V = v ⊕ v 1 ⊕ v 2 . One can check that σ + (W F ) is contained in the split torus SO 2 ⊂ SO 3 . This means that σ is a parameter of an Eisenstein series, so that the rank three local system L ρµ•σ + is of the form L ρµ•σ + = E 1 ⊕ E −1 1 ⊕ Q for some rank one local system E 1 on X.
If L ρµ•σ = L ρµ•σ + ⊗E contains the trivial local system, the only possibility is If the invariant vector v is anisotropic, then v / ∈ v ⊥ and we have an orthogonal If σ + (W F ) is contained in SO 2 , we end up in the previous case. If it is not contained in SO 2 , then σ + induces a surjective homomorphism ρ E 1 : W F → Z/2Z corresponding to a certain non-trivial rank one local system E 1 of order two. In this case L ρµ•σ + has the form L ρµ•σ + = E 1 ⊕ V 1 , where V 1 is a rank two local system of determinant det(V 1 ) = E 1 . If the image of σ + contains only two elements, then we again end up in the same situation as above, i.e., V 1 = Q ⊕ E 1 and σ is the Arthur parameter of an Eisenstein series that we have encountered above.
Finally, if the image of σ + contains more than two elements, then V 1 is an irreducible rank two local system. Now if contains the trivial local system, the only possibility is E 1 = E, because V 1 ⊗ E is an irreducible rank two local system. This means that σ factors through the dual group L H of the twisted one-dimensional torus H = H E attached to E.
To summarize, apart from the Eisenstein series, only (cuspidal) automorphic representations with the Arthur parameter factorizing through the dual group L H of the twisted one-dimensional torus H L contribute to the trace of the operator K d,ρ . Actually, the parameter of the Eisenstein series which contributes to the trace also factors through L H as follows: The contribution of these representations must be exactly the same as the trace of the operator K d,ρ H for the twisted torus H corresponding to the restriction ρ H of ρ from L G to L H.

The moduli stack of G-pairs
As we have seen in Section 2.9, the right hand side of the trace formula (1.1) is equal (up to a power of q) to the trace of the (arithmetic) Frobenius on the vector space ). Recall from Section 2.6 that the sheaf K d,ρ is supported on the substack H d, [µ] of H. In this section we will mostly consider the split case, so in order to simplify our notation we will write H d,µ for H d, [µ] . Let M d,µ be the fiber product of X d × Bun G and H d,µ with respect to the two morphisms to X d × Bun G × Bun G . In other words, we replace H d by H d,µ in the upper right corner of the diagram (2.30). The sheaf ∆ ! H (K d,ρ ) is supported on M d,µ ⊂ M d , and hence the vector space (4.1) is equal to ρ )). In this section we show that the stack M d,µ has a different interpretation as a moduli space of objects that are closely related to Higgs bundles. More precisely, we need "group-like" versions of Higgs bundles (we will call them "G-pairs"). The moduli spaces of (stable) Higgs bundles has been introduced by Hitchin [H1] (in characteristic 0) and the corresponding stack (in characteristic p) has been used in [N3] in the proof of the fundamental lemma. In addition, there is an analogue A d,µ of the Hitchin base and a morphism h d,µ : M d,µ → A d,µ analogous to the Hitchin map. We would like to use this Higgs bundle-like realization of M d,µ and the morphism h d,µ in order to derive the decompositions (3.11) and (3.12).
4.1. Definition of the moduli stack. Let us assume that G is split over X and µ is a fixed dominant coweight. The groupoid M d,µ (k) classifies the triples  (k)). There is in fact a functorial description of H d,µ and of M d,µ that we will now explain.
We will assume that G is semi-simple and simply-connected. The general case is not much more difficult. Let ω 1 , . . . , ω r denote the fundamental weights of G and the Weyl modules of highest weight ω i . Using the natural action of G on End(V ω i ), we can attach to any G-principal bundle E on X the vector bundle (4.4) End The section ϕ of Ad(E) on X− supp(D) induces a section End ω i (ϕ) of the vector bundle End ω i (E) on X− supp(D). The local conditions (4.3) are equivalent to the property that for all i, End ω i (ϕ) may be extended to a section Though the ϕ i determine ϕ, we will keep ϕ in the notation for convenience. Thus, we obtain a provisional functorial description of M d,µ as the stack classifying the data Sometimes it will be more convenient to package the data (ϕ, ϕ i ) as a single object ϕ which has values in the closure of where g ∈ G and t 1 , . . . , t r ∈ G m are invertible scalars. This way Vinberg's semi-group [Vi] makes its appearance naturally in the description of M d,µ . We will discuss this in more detail elsewhere, and for the time being will stick to the more concrete description (4.5) of the moduli space.

Comparison with the Hitchin fibration.
It is instructive to note that the stack M d,µ is very similar to the moduli stacks of Higgs bundles (defined originally by Hitchin [H1] and considered, in particular, in [N2, N3]). The latter stack N D also depends on the choice of an effective divisor on X and classifies pairs (E, φ), where E is again a G-principal bundle on X and φ is a section of the adjoint vector bundle , which is allowed to have a pole of order at most n i at x i . In other words, This φ is usually referred to as a Higgs field.
In both cases, we have a section which is regular almost everywhere, but at some (fixed, for now) points of the curve these sections are allowed to have singularities which are controlled by a divisor. In the first case we have a section ϕ of adjoint group bundle Ad(E), and the divisor is D · µ, considered as an effective divisor with values in the lattice of integral weights of L G. In the second case we have a section φ of the adjoint Lie algebra bundle ad(E), and the divisor is just the ordinary effective divisor.
An important tool in the study of the moduli stack N D is the Hitchin map [H2] from N D to an affine space where the m i 's are the exponents of G. It is obtained by, roughly speaking, picking the coefficients of the characteristic polynomial of the Higgs field φ (this is exactly so in the case of GL n ; but one constructs an obvious analogue of this morphism for a general reductive group G, using invariant polynomials on its Lie algebra). A point a ∈ A E then records a stable conjugacy class in g(F ), where F is the function field, and the number of points in the fiber over a is related to the corresponding orbital integrals in the Lie algebra setting (see [N2, N3] It then follows from [Bou], Th. VI.3.1 and Ex. 1, that k[G] G is a polynomial algebra generated by the functions g → tr(ρ ω i (g)), where ω 1 , . . . , ω r are the fundamental weights of G. For a fixed divisor D, the analogue of the Hitchin map for M d,µ (D) (the fiber of M d,µ over D) is the following map: defined by attaching to (D, E, ϕ, ϕ i ) the collection of traces tr(ϕ i ) ∈ H 0 (X, O X ( µ, ω i D)).
By letting D vary in X d , we obtain a fibration The morphism h d,µ : M d,µ → A d,µ is very similar to the Hitchin fibration. We are now going to outline the geometric properties of the Hitchin fibration that can be carried over to our new situation. We will discuss in more detail the case of SL 2 and postpone the general case for another occasion.
There exists a Picard stack P d,µ → A d,µ that plays the same role as P → A constructed in [N2]. In particular, for every a ∈ A d,µ , the Picard stack P a acts on the fiber M a of h d,µ over a containing a principal homogeneous space as an open substack. For generic a, P a acts simply transitively on M a . In general, there exists a product formula that expresses the quotient [M a /P a ] in terms of local data as in [N2, N3].
Assume from now on that G is semisimple. As in [CL] whosek-points are the pairs (D, b) such that as an element of (T /W )(F ⊗ kk ), b corresponds to a regular semisimple and anisotropic conjugacy class in G(F ⊗ kk ). The preimage M ani d,µ of A ani d,µ is contained in M st d,µ for all stability conditions. In particular, the morphism M ani d,µ → A ani d,µ is proper. Let P tor d,µ be the open substack of P d,µ such that for every a ∈ A d,µ , the fiber P tor a is an open subgroup of P a and the component group π 0 (P tor a ) is the torsion part of π 0 (P a ). Whatever stability condition we choose, P tor d,µ acts on M st d,µ . Our goal is to understand the cohomology (4.2). According to formula (2.33), up to a shift and Tate twist (and replacing the local system L by L * ), it is isomorphic to the dual of the cohomology with compact support of ∆ * H (K d,ρ (K d,ρ ) is a pure complex. Hence, geometrically, it is isomorphic to a direct sum of shifted simple perverse sheaves.
As we explained in Section 3.5, we wish to compare the stable trace formulas for the given group G and the groups H (depending on ρ and d). Hence we need to isolate geometrically a part in the cohomology of (h st d,µ ) * ∆ * H (K d,ρ ) which corresponds to the stable trace. For this, we follow the strategy of [N2, N3].
Let π 0 (P tor d,µ ) denote the sheaf of connected components of P tor d,µ . This is a sheaf of finite abelian groups for theétale topology of A d,µ . As in [N3], we define the stable part on which π tor 0 (P d,µ ) acts unipotently. Note that in the above formula the subscript "st" and the superscript "st" have completely different meaning: the subscript refers to the stable trace formula (see Section 4.3 below), and the superscript refers to imposing a stability condition in the sense of geometric invariant theory.
By analogy with the main theorem of [N3,CL], we state the following conjecture. We will now explain the arithmetic meaning of this conjecture.

Adélic description. Let M d,µ (D) be the fiber of
If the curve X is defined over a finite field k, we can give an adélic expression for the groupoid M d,µ (D) of k-points of M d,µ (D). Namely, let The map h d,µ sends a point (D, γ, (g x )) ∈ M d,µ (k) to (D, b) ∈ A d,µ (k) where b = (b 1 , . . . , b r ) with b i = tr(γ) ∈ H 0 (X, O X ( µ, ω i D)).

Let (D, b) ∈ A ani
d,µ (k). According to the Lefschetz trace formula (see Section 2.9), the trace of the Frobenius acting on the stalk of (h st d,µ ) * ∆ * H (K d,ρ ) over (D, b) may be expressed as where γ runs over the set of conjugacy classes of G(F ) that map to b. The global orbital integral O γ (K d,ρ,D ) is defined by the formula where the G γ (F ) stabilizer of γ in G(F ) is a discrete subgroup of G(A) and dg is the Haar measure on G(A) such that G(O A ) has volume one. The function under the integration sign is and H Sym n (ρ) is the element of the spherical Hecke algebra of G(O x ) bi-invariant functions on G(F x ) corresponding to the representation Sym n (ρ) under the Satake isomorphism. In particular, for all but finitely many x ∈ |X|, H Sym nx(D) (ρ) is the characteristic function of G(O x ), the unit element of the Hecke algebra.
As in [N2], for every (D, b) ∈ A ani d,µ (k) the trace of the Frobenius operator on the fiber of (h st d,µ ) * ∆ * H (K d,ρ ) st over (D, b) can be expressed in terms of stable orbital integrals (4.8) x∈|X| SO b (H Sym nx(D) (ρ) ).
If (D, b) / ∈ A ani d,µ (k) the expression is more complicated. Nevertheless, following [CL], it is reasonable to expect that it is the correct contribution of b to the Arthur stable trace formula. 4.4. The case of SL 2 . Let us consider again our example with G = SL 2 and µ the highest weight of the adjoint representation of PGL 2 . As cocharacter of the maximal torus of SL 2 , µ is simply t → diag(t, t −1 ). The moduli stack M d,µ classifies the data (D, E, ϕ) where D ∈ X d is an effective divisor on X of degree d, E is a rank two vector bundle over X equipped with a trivialization of its determinant and ϕ is an automorphism of E| X− supp(D) of determinant one that can be extended to a homomorphism of vector bundles ϕ 1 : E → E(D).
Let A d,µ be a fibration over X d , whose fiber over D ∈ X d is the vector space H 0 (X, O X (D)). We have a Hitchin-like map Using Riemann-Roch theorem, it is easy to see that dim(A d,µ ) = 2d − g + 1 for every d ≥ 2g − 1. This formula is in fact true for all d ≥ g and we have a formula for d ≤ g as well.
Recall that A d,µ classifies all pairs (D, b) with D ∈ X d and b ∈ H 0 (X, O X (D)). If b is a non-zero section of H 0 (X, O X (D)), the divisor D = div(b) + D is an effective divisor of degree d that is linearly equivalent to D. Here div(b) is the principal divisor attached to b viewed an a non-zero rational function on X. Such a D determines b up to a scalar in k × . If A × d,µ denote the complement of the zero section b = 0, then we have a map Here g denotes the genus of X.
Proof. Suppose first 1 ≤ d ≤ g − 1. According to Martens' theorem [ACGH,p.191 After Martens' theorem the subvariety of line bundles L of degree d such that dim(H 0 (X, L )) = r is less or equal to d − 2r + 2. It follows that the part of X d × which is less than 2d − g.
We can use Hitchin's device called spectral curve to describe the fibers of h d,µ . Recall that for any sections b 1 ∈ H 0 (X, O X (D)) and b 2 ∈ H 0 (O X (2D)), we have a curve on the total space of the line bundle O X (D) defined by the equation For every (D, b) ∈ A d,µ , the fiber -modules F such that by pushing along the finite flat map p D,b : Y D,b → X, we get a rank two locally free O X -module p D,b * F, equipped with a trivialization of the determinant. If Y D,b is reduced, the fact that p D,b * F is a rank two vector bundle implies that F is a torsion-free module of generic rank one and vice versa. If Y D,b is smooth, F is in fact an invertible sheaf.
In the present SL 2 case, the group P D,b consists of invertible sheaves L on Y D,b with the trivial norm down to X. It acts on M D,b by tensor product For b ∈ H 0 (X, O X (D)) such that the section b 2 − 4 2D of the line bundle O X (2D) has a multiplicity free divisor, the spectral curve Y D,b is smooth. In this case, P D,b is isomorphic to the quotient of an abelian variety by the trivial action of Z/2Z. In particular, P D,b is proper and connected. It is also known that P D,b acts simply transitively on M D,b in this case.
For a general parameter (D, b), as in [N3], we need to keep track of two invariants attached to P D,b : the component group π 0 (P D,b ) and its invariant δ.
The component group π 0 (P D,b ) is related to the theory of endoscopy. In the present situation, it is fairly easy to compute. If the spectral curve Y D,b has at least one unibranch ramification point, in particular irreducible, then π 0 (P D,b ) = 0. If this curve has only ramification points with two branches but still irreducible, then π 0 (P D,b ) = Z/2Z. This is the case if and only if the normalization of Y D,b is an unramified covering of X. Finally, if the spectral curve Y D,b has two irreducible components, then π 0 (P D,b ) = Z. See the last section of [N2] for this calculation.
The δ invariant δ(D, b) which is defined as the dimension of the affine part of the group P D,b is a rough but efficient measure of singularity of M D,b . This invariant can be calculated from the discriminant as follows. Let discr(D, b) be the effective divisor attached to the section b 2 − 4 2D of O X (2D). In other words, we have  We will stratify A d,µ by this invariant δ where A d,µ (δ) is the locally closed subscheme of A d,µ consisting of points (D, b) such that δ(D, b) = δ. In view of some general results proved in [N3], the following estimate of dimensions provides some evidence in favor of Conjecture 4.2.
Proof. For every integer e ≤ d, let denote A d,µ (δ, e) the locally closed subscheme of A d,µ (δ) defined by the condition δ(D, b) = δ and the degree of the zero divisor of the rational function b equals e. Let (D, b) be a geometric point of A d,µ (δ, e). Let us denote E + = div(b) + the divisor of zeros of b and E − = div(E) − the divisor of poles; we have then deg(E + ) = deg(E − ) = e. The requirement that the div(b)+D be effective, implies that D = E − + D 0 for a certain effective divisor D 0 of degree deg We observe that b defines a finite flat morphism b : X → P 1 of degree e and in terms of this map div(b − 2 D ) − = b −1 (2) and div(b + 2) = b −1 (−2). In particular, these effective divisors are disjoint. Now we write div(b − 2 D ) + = D 1 + 2D 2 and div(b + 2 D ) + = D 1 + 2D 2 where D 1 and D 1 are multiplicity free effective divisors. We will use the obvious notation d 1 = deg(D 1 ), d 2 = deg(D 2 ) etc. Then we have We consider the map from A d,µ (δ, e) to This is a G m -torsor over the closed subset of the above product defined by only one condition: D 1 + 2D 2 and D 1 + 2D 2 are linearly equivalent. Now this subvariety of the quintuple product is smooth of dimension if at least one of the integers d 1 , d 2 , d 1 , d 2 is larger or equal to 2g − 1. We prove that either d 1 or d 1 is larger than equal to 2g − 1.
Recall the equalities Under the assumption d − δ ≥ 2g − 1, we can proceeds as follows: assume d 2 ≤ d 2 , then 2d 2 + d 0 ≤ δ so that the inequality 4.5. The case of the twisted torus. Now we consider a non-split example. Namely, let H be the one-dimensional torus over X attached to a µ 2 -torsor E H : X → X as in Section 2.4. The dual group of H is L H = G m Γ where Γ acts non-trivially on G m through the quotient corresponding to E H . We have defined a homomorphism L H → PGL 2 × Γ in (2.27). We will consider the restriction ρ H of the representation ρ = ρ µ ×ρ E H of PGL 2 ×Γ to L H, where ρ µ is the adjoint representation of PGL 2 and ρ E H is the representation of Γ corresponding to E H . The three-dimensional representation ρ H breaks into a direct sum of a one-dimensional and a two-dimensional representations: We have seen earlier that the perverse sheaf K d,ρ also breaks into a direct sum (4.11) The relevant moduli stack M d, [µ] , which we will denote here by M d,ρ H , is a disjoint union and θ is an isomorphism of line bundles such that τ (θ)θ = 1. Recall that τ is the involution on X such that X /τ = X. We derive from θ a symmetric isomorphism O X (D 1 ) → τ * (O X (D 1 )) or, equivalently a descent datum of the line bundle O X (D 1 ) to X. This defines a point in the cartesian product X d 1 × Pic d 1 (X ) Pic d 1 /2 (X). Let us denote This space is empty for odd integers d 1 . The Hitchin base is then the disjoint union The Hitchin map, restricted to each component M d,ρ H (d 0 , d 1 ), is just the projection that has relative dimension dim(Bun H ) = g − 1.
4.6. Conjecture. We recall that our goal is to analyze the (stable parts) of the cohomologies of the restrictions of the sheaves K d,ρ and K d,ρ H to M d,ρ and M d,ρ H , respectively, and to use them to establish the desired identities (3.12). We return to our main example (see Section 3.6): G = SL 2 , ρ = ρ µ ⊗ L, where ρ µ is the adjoint representation of L G = PGL 2 and L is an order two local system on X, and H is the twisted torus over X corresponding to L. We state the following conjecture in this case.
Recall that π 0 (Bun H ) = Z/2Z and let us denote by Bun 0 H the neutral component of Bun H . Let us also denote by τ the involution of A d,ρ H given by This involution acts on the cohomology of A d,ρ H (with compact support) and we will denote by the upper script τ the space of invariants of this action.
the contribution of the Eisenstein series mentioned at the end of the Section 3.6.
We remark that the vector space on the right hand side may be viewed as a subspace of RΓ c (M d,ρ H , Q ), since M d,ρ H Bun H ×A d,ρ H . Taking τ -invariants is due to the fact that the map A d,ρ H → A d,ρ is generically two-to-one, and taking the neutral component Bun 0 H is the geometric counterpart of the factor 1/2 in the corresponding stable trace formula.
We hope that this conjecture may provide a starting point for further investigation of trace formulas by geometric methods.

Part II
In this, more speculative, part of the paper we propose two conjectural "geometric trace formulas" in the case of a complex algebraic curve X.

The geometric trace formula
In Section 2.9 we have interpreted geometrically the right hand side of the trace formula (1.1). Now we turn to the left hand side. From now on we will assume that G is a constant group scheme G × X, where G is a split reductive group defined over the ground field k. Henceforth, in order to simplify our notation, we will not distinguish between G and G.
5.1. The left hand side of the trace formula. As we discussed in Section 3.3, the L-packets of (unramified) irreducible automorphic representations should correspond to (unramified) homomorphisms W F ×SL 2 → L G. Assuming this conjecture and ignoring for the moment the contribution of the continuous spectrum, we may write the left hand side of (2.1) as where σ runs over the unramified homomorphisms W F × SL 2 → L G, m σ is the multiplicity of the irreducible automorphic representation unramified with respect to G(O) in the L-packet corresponding to σ, and N σ is the eigenvalue of the operator K on an unramified automorphic function f σ on Bun G (F q ) corresponding to a spherical vector in this representation: Thus, recalling (2.2), the trace formula (2.1) becomes Consider, for example, the case of K = H ρ,x , the Hecke operator corresponding to a representation ρ of L G and x ∈ |X|. Then, according to the conjectures of Langlands and Arthur, The operators K introduced in Section 2.8 are generated by the Hecke operators H ρ,x . Therefore the eigenvalue N σ for such an operator K is expressed in terms of the traces of σ(Fr x ) on representations of L G.

Lefschetz fixed point formula interpretation.
It is tempting to try to interpret the left hand side of (5.2) as coming from the Lefschetz trace formula for the trace of the Frobenius on the cohomology of an -adic sheaf on a moduli stack, whose set of k-points is the set of σ's. Unfortunately, such a stack does not exist if k is a finite field F q (or its algebraic closure). On the other hand, if X is over C, then there is an algebraic stack LocL G of (de Rham) L G-local systems on X; that is, L G-bundles on X with flat connection. But in this case there is no Frobenius acting on the cohomology whose trace would yield the desired number (the left hand side of (5.2)). Nevertheless, we will define a certain vector space (when X is over C), which we will declare to be a geometric avatar of the left hand side of (5.2) (we will give an heuristic explanation for this in Section 6.4). We will then conjecture that this space is isomorphic to (2.35) -this will be the statement of the "geometric trace formula" that we propose in this paper.
Let us first consider the simplest case of the Hecke operator K = H ρ,x . In this case the eigenvalue N σ is given by formula (5.3), which is essentially the trace of the Frobenius of x on the vector space which is the stalk of the local system on X corresponding to σ and ρ. These vector spaces are fibers of a natural vector bundle on X × LocL G (when X is defined over C).
Indeed, we have a tautological L G-bundle T on X ×LocL G , whose restriction to X ×σ is the L G-bundle on X underlying σ ∈ LocL G . For a representation ρ of L G, let T ρ be the associated vector bundle on X × LocL G . It then has a partial flat connection along X. Further, for each point x ∈ |X|, we denote by T x and T ρ,x the restrictions of T and T ρ , respectively, to x × LocL G .
It is tempting to say that the geometric incarnation of the left hand side of (5.2) in the case K = H ρ,x is the cohomology However, this would only make sense if the vector bundle T ρ,x carried a flat connection (i.e., a D-module structure) and this cohomology was understood as the de Rham cohomology (which is the analogue of theétale cohomology of an -adic sheaf that we now wish to imitate when X is defined over C).
Unfortunately, T ρ,x does not carry any natural connection. If we have a vector bundle V with a flat connection on a smooth algebraic variety Y , then its de Rham cohomology may be defined as the coherent cohomology of the tensor product where T Y is the tangent sheaf to Y . This cohomology is well-defined even if V does not have a flat connection, and so, by making a leap of faith, we can declare the cohomology as the geometric incarnation of the left hand side of (5.2). However, this formula is only the first approximation to the right formula, because LocL G is not a variety, but an algebraic stack. We suggest that the correct formula is where ∆ is the diagonal. This will be discussed further in Sections 6.3 and 6.4.
On the other hand, as explained in Section 2.9, the geometric avatar of the right hand side of (5.2) is the (de Rham) cohomology The conjectural geometric trace formula for K = H ρ,x would then be an isomorphism 5.3. The case of K = K d,ρ . Now we generalize this to other integral operators K, in particular, the one that is of most interest to us, K = K d,ρ (see formula (2.25)). In this case N σ on the left hand side of the trace formula (5.2) is given by the formula The corresponding sheaf on LocL G is defined by symmetrization of the vector bundles T ρ , as follows.
First, let us recall from Section 2.6 that if E is a flat vector bundle on X, then we define its dth symmetric power E d by the formula where π d : X d → X d is the symmetrization map. This is a D-module on X d (but not a vector bundle if the rank of E is greater than 1). We may perform the same construction over any base B parametrizing a family of flat vector bundles. In other words, suppose that we have a vector bundle E over X × B equipped with a flat connection along X.
Then we obtain a coherent sheaf E d on X d × B which is moreover a D-module along X d . The restriction of E d to X d × b is E d given by formula (5.6), where E = E| X×b . Let us apply this in the case of B = LocL G . If ρ is again a representation of L G, we have a tautological vector bundle T ρ on X × LocL G equipped with a flat connection along X. Let T d ρ be the sheaf on X d × LocL G obtained by taking the Cartesian power of T ρ along X. It carries a flat connection along X d . Denote by π d the symmetrization morphism X d × LocL G → X d × LocL G along the first factor and set This is a coherent sheaf on X d × LocL G which carries the structure of a D-module along ρ,x i is the tautological vector bundle on LocL G corresponding to the representation Sym n (ρ) and x i ∈ |X|.
Let π : X d × LocL G → LocL G be the projection onto the second factor. Then (where the direct image corresponds to taking the fiberwise de Rham cohomology) is the coherent sheaf on LocL G that captures the functor K = K d,ρ the way the vector bundle T ρ,x captures the Hecke functor K = H ρ,x . Hence, following the same argument as in the case of K = H ρ,x , we may view the vector space as the first approximation to the geometric avatar of the left hand side of the trace formula (5.2) for K = K d,ρ (it is approximate because LocL G is an algebraic stack).
On the other hand, as explained in Section 2.9, the geometric avatar of the right hand side of (5.2) is the (de Rham) cohomology Therefore, in the first approximation, we obtain the following conjectural geometrization of the trace formula (5.2): However, since LocL G is an algebraic stack in general, the left hand side of (5.9) has to be modified. In the next section we will argue that (5.9) should be replaced by the isomorphism (6.9) which we will propose as the sought-after "geomeric trace formula". We will also show that this isomorphism naturally appears as a corollary of the categorical Langlands correspondence (also known as "non-abelian Fourier-Mukai transform").

Categorical Langlands correspondence
According to the geometric Langlands conjecture [BD] (see, e.g., [F1] for an exposition), for each (sufficiently generic) L G-local system σ on X there exists a D-module (or perverse sheaf) F σ on Bun G which is a Hecke eigensheaf with "eigenvalue" σ. This property means that we have a system of isomorphisms compatible with the tensor product structures, where H ρ is the Hecke functor associated to the representation ρ of L G, acting from the category of D-modules on Bun G to the category of D-modules on X × Bun G (see Section 2.2).
6.1. Equivalence of categories. Categorical Langlands correspondence extends this to a kind of spectral decomposition of the derived category of D-modules on Bun G . More precisely, one hopes that there exists an equivalence of derived categories 8 (6.2) derived category of O-modules on LocL G ←→ derived category of D-modules on Bun G 8 It is expected (see [FW], Sect. 10) that in general there is not a canonical equivalence, but rather a Z2-gerbe of such equivalences. This gerbe is trivial, but not canonically trivialized. One gets a particular trivialization of this gerbe, and hence a particular equivalence C, for each choice of the square root of the canonical line bundle KX on X.
under which the skyscraper coherent sheaf supported at σ ∈ LocL G goes to F σ . This has been proved in the abelian case by Laumon [Lau1] and Rothstein [R], as a version of the Fourier-Mukai transform. In the non-abelian case this categorical correspondence ("non-abelian Fourier-Mukai transform") has been suggested by Beilinson and Drinfeld (see, e.g., [F1,VLaf]). It also follows from the S-duality/Mirror Symmetry picture of Kapustin and Witten [KW] (see [FW,F2] for an exposition). It has not yet been made into a precise conjecture in the literature, so we only use it as a guiding principle.
In what follows, we will denote by D(LocL G ) and D(Bun G ) the categories that should appear on the two sides of (6.2).
This conjectural equivalence has been mostly studied at the level of objects (for instance, in the case when C is applied to the skyscraper sheaf O σ on LocL G , see the next subsection). But it should also yield important information at the level of morphisms. In particular, if we denote the equivalence going from left to right in the above diagram by C, then we should have isomorphisms where F 1 and F 2 are arbitrary two functors from the category D(LocL G ) to itself. We will use the latter to produce the geometric trace formula (5.9).

Compatibility with Wilson/Hecke functors.
There is also another important property that the equivalence C is expected to satisfy. On both sides we have natural functors labeled by representations ρ of L G. On the right hand side these are the Hecke functors H ρ acting from the category of D-modules on Bun G to the category of D-modules on X × Bun G (see Section 2.2). On the left hand side these are the functors acting from the category of O-modules on LocL G to the category of sheaves on X × LocL G , which are D-modules along X and O-modules along LocL G . Following [KW], we will call them Wilson functors and denote them by W ρ . By definition, (6.5) W ρ (F) = T ρ ⊗ p * 2 (F), where T ρ is the tautological vector bundle on X × LocL G defined in Section 5.2 and p 2 : X × LocL G → LocL G is the natural projection. Note that, by construction, T ρ carries a connection along X and so the right hand side of (6.5) is a D-module along X. The Wilson functor W ρ may also be written as the "integral transform" functor corresponding to the O-module ∆ * (T ρ ) on X × LocL G × LocL G , where ∆ is the diagonal embedding of X × LocL G : where q and q are the projections onto X × LocL G (the first factor) and LocL G (the second factor), respectively.
We may also consider more general functors built from the Wilson functors -for example, the functor W d,ρ defined by the formula where F d,ρ is an O-module on LocL G given by formula (5.7). The compatibility of the equivalence C with the Wilson/Hecke functors is the statement that we should have a family of isomorphisms, which are compatible with the tensor product structures on the Wilson and Hecke functors, This implies that we have functorial isomorphisms In particular, observe that the skyscraper sheaf O σ at σ ∈ LocL G is obviously an eigensheaf of the Wilson functors: Indeed, if we tensor a skyscraper sheaf with a vector bundle, this is the same as tensoring it with the fiber of this vector bundle at the point of support of this skyscraper sheaf. Therefore (6.7) implies that C(O σ ) must be a Hecke eigensheaf on Bun G with "eigenvalue" σ (see formula (6.1)). Hence we recover the traditional formulation of the geometric Langlands correspondence.
6.3. Application to the geometric trace formula. Consider now the isomorphism (6.4) in the case F 1 = Id and F 2 = W d,ρ . It follows from the construction of the functors W d,ρ and H d,ρ and formulas (6.6) and (6.7) that Hence (6.4) gives us an isomorphism RHom(Id, W d,ρ ) RHom(Id, K d,ρ ).
We should also have an isomorphism of the RHoms's of the kernels defining these functors. On the right hand side this is the RHom (where C is the constant sheaf on Bun G and K d,ρ = p * (K d,ρ )) in the derived category of D-modules on Bun G × Bun G , which we have proposed in formula (2.31) as a geometric analogue of the right hand side of the trace formula.
On the left hand side of the categorical Langlands correspondence, the kernel corresponding to W d,ρ is just ∆ * (F d,ρ ), where F d,ρ is the O-module on LocL G defined by (5.7). In other words, it is supported on the diagonal in LocL G × LocL G and is just the O-module push-forward of F d,ρ from the diagonal.
Thus, we find that the categorical version of the geometric Langlands correspondence should yield the following isomorphism We recall from Section 2.9 that the right hand side is isomorphic to ρ )). We may also rewrite the left hand side of (6.8) using adjunction as follows: Hence we obtain the following isomorphism, which we conjecture as a geometric trace formula.
Conjecture 6.1. There is an isomorphism of vector spaces: Thus, starting with the categorical Langlands correspondence, we have arrived at what we propose as a geometrization of the trace formula.
We have a similar conjecture for more general functors K built from the Hecke functors of the type introduced in Section 2.8. On the other side of the categorical Langlands correspondence it corresponds to a functor W built in the same way from the Wilson functors, so that C(K) W. We then expect to have an analogous isomorphism (6.10) where K is the kernel of K and F is the O-module on LocL G such that ∆ * (F) is the kernel of W.
Remark 1. It is tempting to rewrite the left hand side of (6.10) as where ∆ ! is the right adjoint functor to ∆ * . Then (6.10) becomes more symmetrical: However, the definition of the functor ∆ ! for the derived category of coherent sheaves on the stack LocL G is tricky. Defining the left hand side of (6.11) essentially amounts to rewriting it as the left hand side of (6.10). That is why we stated the conjecture in the form (6.10).
As a special case, let both K and W be the identity functors. Then (6.10) gives rise to an isomorphism (6.12) . The right hand side of this formula is known [T, NS, HS], and hence this special case already gives us a good test for the conjecture. The isomorphism (6.12) holds in the abelian case G = G m , as we will see at the end of in Section 7.8. 6.4. Connection to the Atiyah-Bott-Lefschetz fixed point formula. Here we give an heuristic explanation why we should think of the space (6.10) as a geometric incarnation of the sum appearing on the left hand side of (5.2).
Let F be a coherent sheaf on LocL G built from the vector bundles T ρ,x , x ∈ |X| (like F d,ρ constructed below). We would like to interpret taking the trace of the Frobenius on the (coherent!) cohomology H • (LocL G , F) as the sum over points σ of LocL G , which we think of as the fixed points of the Frobenius automorphism acting on a moduli of homomorphisms Recall the Atiyah-Bott-Lefshetz fixed point formula [AB] (see also [Il], §6). Let M be a smooth proper scheme, V a vector bundle on M , and V the coherent sheaf of sections of V . Let u be an automorphism acting on M with isolated fixed points, and suppose that we have an isomorphism γ : u * (V) V. Then (6.13) is the set of fixed points of u, the fiber product of the graph Γ u of u and the diagonal ∆ in M × M , which we assume to be transversal to each other (as always, the left hand side of (6.13) stands for the alternating sum of traces on the cohomologies). Now, if we take the cohomology not of V, but of the tensor product V⊗Ω • (M ), where Ω • (M ) = Λ • (T * (M )) is the graded space of differential forms, then the determinants in the denominators on the right hand side of formula (6.13) will get cancelled and we will obtain the following formula: (6.14) Tr We would like to apply formula (6.14) in our situation. However, LocL G is not a scheme, but an algebraic stack (unless L G is a torus), so we need an analogue of (6.14) for algebraic stacks (and more generally, for derived algebraic stacks, since LocL G is not smooth as an ordinary stack).
Let M be as above and ∆ : M → M 2 the diagonal embedding. Observe that and hence we can rewrite (6.14) as follows: (6.16) Tr(γ, H • (M, ∆ * ∆ * (V))) = p∈M u Tr(γ, V p ). Now we propose formula (6.16) as a conjectural generalization of the Atiyah-Bott-Lefschetz fixed point formula in the case that M is a smooth algebraic stack, and more generally, smooth derived algebraic stack (provided that both sides are well-defined). We may also allow here V to be a perfect complex (as in [Il]).
Even more generally, we drop the assumption that u has fixed points (or that the graph Γ u of u and the diagonal ∆ are transversal) and conjecture that (6.17) This seems to be the most general fixed point formula of Atiyah-Bott type.
Let us apply (6.16) to the left hand side of (6.10): This is not exactly in the form of the left hand side of (6.16), but it is very close. Indeed, if M is a smooth scheme, then so we obtain the exterior algebra of the tangent bundle instead of the exterior algebra of the cotangent bundle. In our setting, we want γ to be the Frobenius, and so a fixed point is a homomorphism π 1 (X) → L G. The tangent space to σ (in the derived sense) should then be identified with the cohomology H • (X, ad • σ) [1], and the cotangent space with its dual. Hence the trace of the Frobenius on the exterior algebra of the tangent space at σ is the L-function L(σ, ad, s) evaluated at s = 0. Poincaré duality implies that the trace of the Frobenius on the exterior algebra of the cotangent space at σ is L(σ, ad, 1) = q −d G L(σ, ad, 0), so the ratio between the traces on the exterioir algebras of the tangent and cotangent bundles at the fixed points results in an overall factor which is a power of q. 9 Hence, by following this argument and switching from C to F q , we obtain (up to a power of q) the trace formula (5.2) from the isomorphism (6.10).
Remark 2. The sheaf ∆ * ∆ * (O M ) is the structure sheaf of the self-intersection of the diagonal of a derived stack M . In [BN] it is interpreted as the loop space of M . Hence the left hand side of the conjectural generalization (6.16) of the Atiyah-Bott-Lefschetz formula may be viewed as the cohomology of the pullback of V from M to its loop space. David Nadler has outlined to us a possible proof of (6.16) using the methods of [BFN, BN]. 6.5. Discussion. The isomorphism (6.9) is still tentative, because there are some unresolved issues in the definition of the two sides of the isomorphism (6.9).
The first issue is the definition of the stack LocL G . Presumably, it should be defined not as a stack, but as a derived (or differential graded, or DG) stack. This should correspond to the inclusion of the non-tempered (or non-Ramanujan) automorphic representations, which should be parametrized, according to Arthur, by homomorphisms σ : W F × SL 2 → L G (for tempered representations, σ| SL 2 is expected to be trivial). For instance, the trivial representation of G(A) corresponds to σ that is trivial on W F and is a principal SL 2embedding on the second factor. Recent results of V. Lafforgue [VLaf] show that one might be able to take such σ into account by considering a derived version of the stack LocL G . There are also additional insights pointing in the same directions coming from the study of S-duality in physics (see a brief discussion at the end of [F2] and references therein). 9 The exact power of q depends on whether we choose the geometric or the arithmetic Frobenius here. We recall from Section 2.9 that the right hand side of the trace formula (5.2) is equal to the trace of the Frobenius also up to an overall factor which is a power of q. Since this is an heuristic argument, we choose to ignore these factors.
The second issue is the definition of RHom's on the two sides of the isomorphism (6.9). This is in fact the main question in finding the correct formulation of the categorical Langlands correspondence (6.2).
Finally, as we have discussed in Part I, in order to become an effective tool, trace formulas have to be stabilized. Hence an analogue of stabilization needs to be worked out in the geometric setting as well.
But suppose we could solve these problems and really make sense of (5.9) as a geometric trace formula. What would we be able to learn from it?
First of all, we would have a good framework for the geometric trace formula that works for curves over F q and over C. Second, we could try to establish the isomorphism (5.9) by using fixed-point formulas, for example. Here the description of G m -fixed points in the moduli space of stable flat bundles due to Hitchin [H1, H3] and Simpson [Si] could be useful.
The main application we have in mind is to use the isomorphism (5.9) to establish functoriality. Namely, we wish to express each side as a direct sum of two vectors spaces corresponding to the tempered (Ramanujan) and non-tempered (non-Ramanujan) representations, and then to decompose further the former as a direct sum over the groups H (see Section 3.5). In Section 4 we made the first steps toward this goal for the right hand (orbital) side of (5.4), using an analogue of the Hitchin fibration. The fact that we now have the left hand (spectral) side of the isomorphism (5.9) may help us understand better how to make this decomposition. For instance, the locus of G m -fixed points in the moduli space of stable L G-local systems decomposes into the union of the φ = 0 locus (where φ is the Higgs field) and other loci with φ = 0. It is possible that the former corresponds to the tempered part and the latter to the non-tempered part.

Relative geometric trace formula
In the previous section we discussed a geometrization of the trace formula (1.1). We have seen in the previous section that a geometric analogue of this formula may be constructed within the framework of the categorical form of the Langlands correspondence, as the statement that the RHom's of kernels of certain natural functors are isomorphic. These kernels are sheaves on algebraic stacks over the squares Bun G × Bun G and LocL G × LocL G .
It is natural to ask what kind of statement we may obtain if we consider instead the RHom's of sheaves on the stacks Bun G and LocL G themselves.
In this section we will show that this way we obtain what may be viewed as a geometric analogue of the so-called relative trace formula. On the spectral side of this formula we also have a sum like (5.1), but with one important modification; namely, the eigenvalues N σ are weighted with the factor L(σ, ad, 1) −1 . The insertion of this factor was originally suggested by Sarnak in [S] and further studied by Venkatesh [Ve], for the group GL 2 in the number field context. Another difference is that the summation is restricted to the generic representations, which means (at least, conjecturally) that we remove the contribution of the non-Ramanujan representations as well as the multiplicity factors m σ .
We will also analyze the corresponding orbital side and express is as the cohomology of a sheaf on an algebraic stack. This will give us what we may call a relative geometric trace formula. It will be the statement about an isomorphism of two cohomology spaces, one on Bun G and the other on LocL G . We will see that this isomorphism arises naturally from the categorical form of the geometric Langlands correspondence (6.2). 7.1. Relative trace formula. First, let us recall the setup of the relative trace formula.
Let G be a split simple algebraic group over k = F q . In order to state the relative trace formula, we need to choose a non-degenerate character of N (F )\N (A), where N is a maximal unipotent subgroup of G. A convenient way to define it is to consider a twist of the group G. Let us pick a maximal torus T such that B = T N is a Borel subgroup. If the maximal torus T admits the cocharacterρ : G m → T equal to half-sum of all positive roots (corresponding to B), then let Kρ X be the T -bundle on our curve X which is the pushout of the G m -bundle K × X (the canonical line bundle on X without the zero section) underρ. Ifρ is not a cocharacter of T , then its square is, and hence this T -bundle is well-defined for each choice of the square root K 1/2 X of K X . We will make that choice once and for all. 10 Now set where T acts via the adjoint action. For instance, if G = GL n , then GL K n is the group scheme of automorphism of the rank n bundle O ⊕ K X ⊕ . . . ⊕ K ⊗(n−1) X on X (rather than the trivial bundle O ⊕n ).
We have Now let ψ : F q → C × be an additive character, and define a character Ψ of N K (A) as follows via the isomorphism (7.1). We denote by k x the residue field of x, which is a finite extension of the ground field k = F q .
By the residue formula, Ψ is trivial on the subgroup N K (F ) (this was the reason why we introduced the twist). It is also trivial on In what follows, in order to simplify notation, we will denote G K and N K simply by G and N .
Given an automorphic representation π of G(A), we have the Whittaker functional W : π → C, where du is the Haar measure on N (A) normalized so that the volume of N (O) is equal to 1.
We choose, for each unramified automorphic representation π, a non-zero G(O)invariant function f π ∈ π on G(F )\G(A).
Let K again be a kernel on the square of Bun G (k) = G(F )\G(A)/G(O) and K the corresponding integral operator acting on unramified automorphic functions. The simplest unramified version of the relative trace formula reads (here we restrict the summation to cuspidal automorphic representations π) (see, e.g., [J] and dg denotes the invariant Haar measure normalized so that the volume of G(O) is equal to 1. Note that T is the norm corresponding to the Tamagawa measure, d G = (g − 1) dim G, where the m i are the exponents of G.
The following conjecture was communicated to us by B. Gross and A. Ichino. In the case of G = SL n or P GL n , formula (7.3) follows from the Rankin-Selberg convolution formulas, as we explain in Section 7.2 below. Other cases have been considered in [GP, Ic, IcIk]. For a general semi-simple group G of adjoint type formula (7.3) has been conjectured by A. Ichino and T. Ikeda assuming that π is square-integrable. Note that if a square-integrable representation is tempered, then it is expected to be cuspidal, and that is why formula (7.3) is stated only for cuspidal representations.
Recall that an L-packet of automorphic representations is called generic if each irreducible representation π = π x from this L-packet has the property that the local L-packet of π x contains a generic representation (with respect to a particular choice of non-degenerate character of N (F x )).
Conjecture 7.1. Suppose that the L-packet corresponding to an unramified σ : W F → L G is generic. Then it contains a unique, up to an isomorphism, irreducible representation π such that W Ψ (f π ) = 0, with multiplicity m π = 1. Moreover, if this π is in addition cuspidal, then the following formula holds: where S σ is the (finite) centralizer of the image of σ in L G, d N = −(g − 1)(4 ρ,ρ − dim N ) (see formula (7.6)), and d G = (g − 1) dim G.
Ichino has explained to us that according to Arthur's conjectures, square-integrable non-tempered representations are non-generic. Recall that we have K · f π = N σ f π . Therefore Conjecture 7.1 and formula (7.2) give us the following: On the left hand side we sum only over unramified σ, and only those of them contribute for which the corresponding L-packet of automorphic representations π is generic. We expect that the left hand side of formula (7.5) has the following properties: (1) It does not include homomorphisms σ : SL 2 × W F → L G which are non-trivial on the Artur's SL 2 .
(2) The multiplicity factor m σ of formula (5.2) disappears, because only one irreducible representation from the L-packet corresponding to σ shows up (with multiplicity one). (3) Since S σ is the group of automorphisms of σ, the factor |S σ | −1 makes the sum on the left hand side (7.5) look like the Lefschetz fixed point formula for stacks.
7.2. The case of G = P GL n . Here we derive formula (7.3) for G = P GL n using the Rankin-Selberg convolution method, following [Lys]. Denote by Bun n the moduli stack of rank n bundles on X and Bun m n the connected component corresponding to vector bundles of degree m. Let σ be an irreducible n-dimensional -adic representation of W F and F σ the pure perverse sheaf of weight 0 on Bun n (irreducible on each Bun m n ) which is a Hecke eigensheaf with respect to σ (see [FGV2]). Let f σ be the corresponding function on Bun GLn (F q ). The following formula is derived in [Lys]: (here ad denotes the adjoint representation of GL n ). In fact, S. Lysenko [Lys] has given a geometric interpretation of this formula, which is compatible with the categorical Langlands correspondence. In addition, for irreducible σ we have see [LL]. Using this formula, it is straightforward to obtain for G = PGL n that where ad denotes the adjoint representation of L G = SL n . This agrees with formula (7.3), because S σ is the center of L G in this case (here we are again under the assumption that σ is irreducible).
7.3. Geometric meaning: right hand side. Now we discuss the geometric meaning of formula (7.5), starting with the right hand side. Let Bun F T N be the moduli stack of B = B K bundles on X such that the corresponding T -bundle is F T = Kρ X . Note that . Let ev : Bun F T N → G a be the map constructed in [FGV1]. For instance, if G = GL 2 , then Bun F T N classifies rank two vector bundles V on X which fit in the exact sequence X is a square root of K X which we have fixed. The map ev assigns to such V its extension class in Ext(O X , K X ) = H 1 (X, K X ) G a . For other groups the construction is similar (see [FGV1]).
On G a we have the Artin-Schreier sheaf L ψ associated to the additive character ψ. We define the sheaf Ψ = ev * (L| ψ ) on Bun F T N . Next, let p : Bun F T N → Bun G be the natural morphism. Let Then the right hand side of (7.5) is equal to the trace of the Frobenius on the vector space (7.8) RHom(Ψ, K(Ψ)).
Here we use the fact that D 7.4. Geometric meaning: left hand side. As discussed above, we don't have an algebraic stack parametrizing homomorphisms σ : W F → L G if our curve X is defined over a finite field F q . But such a stack exists when X is over C, though in this case there is no Frobenius operator on the cohomology whose trace would yield the desired number (the left hand side of (7.5)). In this subsection we will define a certain vector space (when X is over C) and conjecture that it is isomorphic to the vector space (7.14) which is the geometric incarnation of the right hand side of (7.5) (and which is well-defined for X over both F q and C). This will be our "relative geometric trace formula". In the next subsection we will show that this isomorphism is a corollary of the categorical version of the geometric Langlands correspondence.
In order to define this vector space, we will use the coherent sheaf F d,ρ on LocL G introduced in formula (5.7). We propose that the geometric incarnation of the left hand side of (7.5) in the case when K = K d,ρ is the cohomology The heuristic explanation for this proceeds along the lines of the explanation given in the case of the ordinary trace formula in Section 6.4, using the Atiyah-Bott-Lefschetz fixed point formula. 11 We wish to apply it to the cohomology (7.9). If LocL G were a smooth scheme, then we would have to multiply the number N σ which corresponds to the stalk of F d,ρ at σ, by the factor (7.10) det(1 − Fr, T * σ LocL G ) −1 . Recall that the tangent space to σ (in the derived sense) may be identified with the cohomology H • (X, ad • σ) [1]. Using the Poincaré duality, we find that the factor (7.10) is equal to L(σ, ad, 1) −1 . Therefore, if we could apply the Lefschetz fixed point formula to the cohomology (7.9) and write it as a sum over all σ : W F → L G, then the result would be the left hand side of (7.5) (up to a factor that is a power of q). (Note however that since LocL G is not a scheme, but an algebraic stack, the weighting factor should be more complicated for those σ which admit non-trivial automorphisms, see the conjectural fixed point formula (6.16) in Section 6.4.) This leads us to the following relative geometric trace formula (in the case of the functor K d,ρ ).
It is clear how to generalize this to other functors K of the form described in Section 2.8. Now we explain how the isomorphism (7.11) fits in the framework of a categorical version of the geometric Langlands correspondence. 7.5. Interpretation from the point of view of the categorical Langlands correspondence. We start by asking what is the D-module on Bun G corresponding to the structure sheaf O on LocL G under the functor C discussed in Section 6. The following answer was suggested by Drinfeld (see [VLaf]): it is the sheaf Ψ that we have discussed above.
The rationale for this proposal is the following: we have where O σ is again the skyscraper sheaf supported at σ. Therefore, since C(O σ ) = F σ , we should have, according to (6.3), RHom Bun G (C(O), F σ ) = C, ∀σ. 11 We note that applications of the Atiyah-Bott-Lefschetz fixed point formula in the context of Galois representations have been previously considered by M. Kontsevich in [K].
According to the conjecture of [LL], the sheaf Ψ has just this property: is the one-dimensional vector space in cohomological degree 0 (if we use appropriate normalization for F σ ). This vector space should be viewed as a geometric incarnation of the Fourier coefficient of the automorphic function corresponding to F σ .
This provides some justification for the assertion that 12 (7.12) C(O) = Ψ.
Let us take this for granted. Now let us look at the left hand side of the isomorphism (7.11). We can rewrite it as the de Rham cohomology of a sheaf on X d , whose fiber at D = n i [x i ] ∈ X d is ρ,x i is the Wilson operator corresponding to the representation Sym n i ρ, specialized to the point x i ∈ X (see formula (6.5)).
Using the compatibility (6.7) with the Wilson/Hecke operators and formulas (6.3) and (7.12), we obtain that (7.13) should be isomorphic to (7.14) RHom where H (n i ) ρ,x i is the Hecke operator corresponding to the representation Sym n i ρ, specialized to the point x i ∈ X. Hence the right hand side of (7.11) is isomorphic to the de Rham cohomology of a sheaf on X d whose fiber at D = n i [x i ] ∈ X d is (7.14). But varying D over X d gives us precisely our averaging functor K d,ρ . Therefore the result is (7.15) RHom Bun G (Ψ, K d,ρ (Ψ)) , which is the right hand side of (7.11). Thus, we obtain that the relative geometric trace formula (7.11) follows from the categorical version of the geometric Langlands correspondence.
Let's recap. We started out with the classical relative trace formula (7.5). The right hand side of (7.5) had a simple geometric interpretation as the trace of the Frobenius on the vector space (7.14). To interpret the left hand side of (7.5) we made a "leap of faith" and replaced it by the cohomology (7.9). However, we have just shown that the resulting geometric relative trace formula (7.11) is a meaningful statement from the point of view of the categorical geometric Langlands correspondence. This is an indication that (7.11) is actually a reasonable conjecture (and hence so is our "leap of faith"). 12 As explained in the footnote on page 44, C corresponds to a particular choice of K 1/2 X . Given such a choice, C(O) should be the character sheaf Ψ associated to that K 1/2 X . 7.6. Applications to functoriality. As explained in [FLN] (following [L2, L3, L4]), our goal is to express the trace of the averaging operator K d,ρ for the group G as the sum of traces for the groups H dual to subgroups λ H ⊂ L G for which representation ρ contains non-zero invariant vectors. This has important applications to functoriality.
Likewise, in the setting of the relative trace formula, we are interested in decomposing the left hand side as a sum over the groups H. It is instructive to see why and how this decomposition should come about, first for an arbitrary G and then in the example of G = GL 2 and ρ the second symmetric power of its defining representation.
Let us consider the general case first. Look at the left hand side of formula (7.5). According to Corollary 3.3 (see also [FLN], Lemma 2.6), N σ = 0 for d > (2g − 2) dim ρ, unless (ρ • σ) contains non-zero invariant vectors. From now on we will assume that d > (2g − 2) dim ρ. Then we obtain that the left hand side of formula (7.5) decomposes into a sum (7.16) λ H⊂ L G σ :W F → λ H N σ · L(σ , adL G , q −1 ) −1 |S σ | −1 over all possible λ H ⊂ L G such that λ H has a trivial subrepresentation in ρ • σ. Then each σ : W F → λ H gives rise to σ : W F → L G and to ρ • σ : W F → Aut ρ (we view ρ as a representation of λ H obtained by restriction from L G). We have = q −ds -coeff. of L(σ , ρ, s).
Observe that L g = λ h ⊕ R H , where λ h is the Lie algebra of λ H and R H is a certain representation of λ H. We have L(σ, adL G , 1) = L(σ , adλ H , 1) · L(σ , R H , 1).
The summation is over the λ H described above.
In the same way as for the ordinary trace formula (as explained in [FLN]), our goal is to obtain the sum decomposition (7.20) using the right hand side of formula (7.5).
Geometrically, we would like to have a decomposition of the right hand side of (7.11) as a direct sum of spaces labeled by the λ H. We will also need to construct the geometric counterpart of the extra factor (7.19).
We now look more closely at what happens in one example. 7.7. The case of GL 2 . Let ρ be the second symmetric power of the defining representation of GL 2 . We want to decompose the vector space (7.21) RHom Bun GL 2 (Ψ, K d,ρ (Ψ)) , into a direct sum of subspaces labeled by the groups H, which in this case are the unramified tori (split and non-split) in GL 2 . They are labeled by the unramified double covers of X.
Let us look at the vector space (7.21) more closely. The sheaf Ψ is supported on the locus of rank two bundles which have the form (7.7): The support of Ψ consists of bundles V of this form. We will fix a point ∞ ∈ X and identify rank two bundles V and V(n[∞]) for all n ∈ Z. In other words, we replace Bun GL 2 by its quotient by the group Z generated by tensoring rank two vector bundles with the line bundle O([∞]). When we apply our averaging functor K d,ρ , we average over the Hecke modifications of the bundle V: for each divisor D = n i [x i ] of degree d we apply all possible Hecke modifications corresponding to the representation Sym n i ρ at the point x i , for all i, and then take the resulting bundle V and replace it by V(−d [∞]). This bundle should then again be of the form (7.22). Thus, the space (7.21) can be interpreted as the cohomology of a certain sheaf over the moduli space W d,ρ of data (V, V , D, φ), where D = n i [x i ], V and V are rank two vector bundles on X the form (7.22), together with an embedding where T x i are torsion sheaves supported at x i : for some k i = 0, . . . , n i (these correspond to the strata in the affine Grassmannian which lie in the closure of Gr n i ρ ). We need to compute the cohomology of this sheaf and relate it to the cohomologies corresponding to the contributions of λ H in the sum on the right hand side of (7.20). We hope that we could use the methods of [N1] and [FGV1] to do this. 7.8. The abelian case. Finally, we consider the case of the group GL 1 and trivial ρ. We wish to check the statement of Section 7.4 that the cohomology (7.9) is really the geometric incarnation of the sum appearing on the left hand side of (7.5).
Since ρ is trivial, the symmetric power of the curve X d decouples from the formulas, and we might as well set K = Id, and so N σ = 1 in formula (7.5). Note that the adjoint representation of GL 1 is trivial, and so q −d G L(σ, ad, s) = q −(g−1) ζ(s), where ζ(s) is the zeta-function of X. However, since GL 1 is not simple, we have to make an adjustment and replace ζ(1) by its residue at s = 1, which we denote by ζ(1). In other words, we remove the factor corresponding to H 2 (X, Q ), which is independent of σ. (One can argue that this factor decouples because the structure sheaf of the full derived stack of rank one local systems on X contains the exterior algebra of H 2 (X, Q ) placed in the cohomological degree 1.) Thus, we have to compute the sum (7.23) σ q −(g−1) ζ(1) −1 = q −(g−1) ζ(1) −1 · #{σ : W F → GL 1 }, where σ runs over all unramified homomorphisms W F → GL 1 . Let α i , i = 1, . . . , 2g, be the eigenvalues of the Frobenius on H 1 (X, Q ). We have α i = β i q 1/2 , where |β i | = 1 and for each β i there is β j = β −1 i . Therefore .
On the other hand, by the abelian class field theory, the number of unramified σ : W F → GL 1 is equal to the dimension of the space of unramified automorphic forms on GL 1 (A), which is the number of F q -points of the Jacobian of X. This number is in turn equal to the alternating sum of the traces of the Frobenius of the cohomology of the Jacobian, which is Λ • H 1 (X, Q ). Hence the answer is 2g i=1 (1 − α i ).
Thus, the sum (7.23) is equal to q − 1. Its geometric incarnation is the vector space (7.24) H • (Loc GL 1 , O) (since we have replaced K d,ρ with the identity functor, we replace the bundle T d,ρ with the trivial bundle O).
It is easy to see that H 0 (Loc GL 1 , O) is one-dimensional, and H i (Loc GL 1 , O) = 0 for all i > 0, so that the total space (7.24) is one-dimensional, situated in cohomological degree 0. This means that the (non-existent) Frobenius should act on it by q − 1. Let us compare this number with the number appearing on the right hand side.
First, let us look at the right hand side (7.15) of the relative geometric trace formula (7.11). As above, we replace K d,ρ by the identify functor. The resulting vector space is (7.25) RHom D(Pic 0 (X)) (Ψ, Ψ).