RETRACTED ARTICLE: Curves with prescribed symmetry and associated representations of mapping class groups

Abstract

Let C be a complex smooth projective algebraic curve endowed with an action of a finite group G such that the quotient curve has genus at least 3. We prove that if the G-curve C is very general for these properties, then the natural map from the group algebra ${{\mathbb {Q}}}G$ to the algebra of ${{\mathbb {Q}}}$-endomorphisms of its Jacobian is an isomorphism. We use this to obtain (topological) properties regarding certain virtual linear representations of a mapping class group. For example, we show that the connected component of the Zariski closure of such a representation often acts ${{\mathbb {Q}}}$-irreducibly in a G-isogeny space of $H^1(C; {{\mathbb {Q}}})$ and with image a ${{\mathbb {Q}}}$-almost simple group.

1 Introduction

A classical theorem of Hurwitz (1886) asserts that for a very general complex projective smooth curve C of genus $\ge 2$, the endomorphism ring of its Jacobian J(C) is as small as possible, namely ${{\mathbb {Z}}}$. Lefschetz [14] proved in 1928 that this is still true if we restrict ourselves to hyperelliptic curves and we refer to Ciliberto’s survey [4] for what other results of that type were known before 1989. As of then the theorem of Lefschetz has been refined and generalized in several directions. For example, Ciliberto, van der Geer and Teixidor i Bigas [5] studied (and to some extent characterized) the locus of curves for which the endomorphism ring of the Jacobian is strictly larger than ${{\mathbb {Z}}}$, while Zarhin addressed this problem over other base fields and also considered in [24, 25] curves with a specified automorphism of a particular type other than a hyperelliptic involution: he proved that for a general such curve the endomorphism ring of its Jacobian is indeed as small as it could possibly be.

In the present paper we obtain a natural generalization of these results, at least over the base field ${{\mathbb {C}}}$, by proving such a minimality property for curves endowed with an action of a given (but arbitrary) finite group. More precisely, we prove (cf. Corollary 1.9):

Theorem A

For a very general G-curve C for which the genus of the orbit curve C/G is at least 3, the natural homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is an isomorphism.

As we show in Remark 1.13, without further hypotheses on the G-curve C, this result is optimal. On the other hand, with further hypotheses, we were able to prove a more general result (cf. Theorems 1.1 and 1.11).

Although Theorem A has an interest of its own, our original motivation was of a more topological nature, since this result has implications for certain virtual linear representations of the (pointed) mapping class groups, as we shall now explain.

Let S be a closed oriented connected surface of genus $\ge 2$ endowed with an action of a finite group G by orientation preserving diffeomorphisms. Then G embeds in the mapping class group ${\text {Mod}}(S)$ of S. The centralizer ${\text {Mod}}(S)^G$ of G in ${\text {Mod}}(S)$ maps to a finite index subgroup of the mapping class group ${\text {Mod}}(S_G)$ of the orbifold surface $S_G$ (these are mapping classes of the underlying surface which take orbifold points to orbifold points of the same type), with kernel the center Z(G) of G. Thus ${\text {Mod}}(S)^G$ and ${\text {Mod}}(S_G)$ have in common a subgroup of finite index so that the natural representation $\rho _G: {\text {Mod}}(S)^G\rightarrow {\text {Sp}}(H^1(S,{{\mathbb {Q}}}))^G$, where we denote by ${\text {Sp}}(H^1(S,{{\mathbb {Q}}}))^G$ the centralizer of G in the symplectic group ${\text {Sp}}(H^1(S,{{\mathbb {Q}}}))$, can be regarded as a virtual linear representation of the mapping class group ${\text {Mod}}(S_G)$. Let us describe this centralizer in more detail.

For an irreducible ${{\mathbb {Q}}}G$-module $V_\chi$, of character $\chi$, let $D_\chi :={End}_{{{\mathbb {Q}}}G}(V_\chi )^\mathrm {op}$ be the associated division ring. We then regard $V_\chi$ as a left ${{\mathbb {Q}}}G$-module and a right $D_\chi$-module. Let then $H^1(S;{{\mathbb {Q}}})[\chi ]:={\text {Hom}}_{{{\mathbb {Q}}}G}(V_\chi ,H^1(S;{{\mathbb {Q}}}))$. This is a left $D_\chi$-module and the $\chi$-primary component $H^1(S;{{\mathbb {Q}}})_\chi$ of $H^1(S;{{\mathbb {Q}}})$ is naturally isomorphic to $V_\chi \otimes _ {D_\chi } H^1(S;{{\mathbb {Q}}})[\chi ]$. Let be the set of rational irreducible characters of G. Then, there are isomorphisms:

where ${\text {U}}_{D_\chi }(H^1(S;{{\mathbb {Q}}})[\chi ])$ denotes the group of $D_\chi$-linear automorphisms of ${H^1(S;{{\mathbb {Q}}})[\chi ]}$ which preserve an antisymmetric sesquilinear form $\langle \_,\_\rangle _\chi$, induced by the intersection product on the homology group $H^1(S;{{\mathbb {Q}}})$ (cf. Sect. 2 for the details). Note that the above decomposition of ${\text {Sp}}(H^1(S; {{\mathbb {Q}}}))^G$ mirrors the isogeny decomposition of the Jacobian of a G-curve C of the same topological type of (S, G) (cf. Theorem 2.2 in [13]).

Let us denote by ${{\mathscr {M}}}{on}^\circ (S)$ the identity component of the Zariski closure of the image of $\rho _G$ in ${\text {Sp}}(H^1(S,{{\mathbb {Q}}}))^G$ and by ${{{\mathscr {M}}}{on}^\circ (S)[\chi ]}$ the projection of ${{\mathscr {M}}}{on}^\circ (S)$ to the factor ${{\text {Sp}}_G(H_1(S;{{\mathbb {Q}}})_\chi )\cong {\text {U}}(H_1 (S;{{\mathbb {Q}}})[\chi ])}$ of ${\text {Sp}}_G(H_1(S; {{\mathbb {Q}}}))$.

A natural question is whether ${{{\mathscr {M}}}{on}^\circ (S)[\chi ]}$ is the derived subgroup of the group ${\text {Sp}}_G(H_1(S;{{\mathbb {Q}}})_\chi )$. For an unramified covering $S\rightarrow S/G$ which is either abelian or “$\phi$-redundant” (see [10] for the definition), this has been proved to be the case in [10, 15], respectively. But the problem is open in general and it can be shown that a positive answer implies the Putman-Wieland conjecture (cf. [19]).

From standard results on variations of polarizable Hodge structures of weight 1, we then deduce from Theorem A the following properties of the connected algebraic monodromy group (cf. Theorem 3.2 for a more precise result):

Theorem B

Let us assume that the genus of $S_G$ is at least 3. Then we have:

• (i) The algebraic group ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ acts ${{\mathbb {Q}}}$-isotypically on the $D_\chi$-module $H_1(S;{{\mathbb {Q}}})[\chi ]$.

• (ii) If the ${{\mathbb {Q}}}$-rank of ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is positive, then ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is a quasi-simple ${{\mathbb {Q}}}$-algebraic group which acts irreducibly on the $D_\chi$-module $H_1(S;{{\mathbb {Q}}})[\chi ]$.

We also observe (cf. Remark 3.6) that, in particular, the hypothesis of item (ii) is always satisfied in case the covering $S\rightarrow S/G$ is of the type considered in either [10, 15]. Hence item (ii) of Theorem B may also be regarded as a generalization of the results of those papers to an arbitrary covering.

The above result gives rise to a number of questions, which we collected in 3.4. The paper also ends in this spirit: with a conjecture (3.7) and a question (3.8). Still, we will use the results obtained here to answer some of these questions in certain cases (this is made more explicit in Remark 3.6).

Conventions. If a group G acts properly discretely on a smooth object S in a category where this makes sense, then $S_G$ denotes the associated orbifold object.

When G is a finite group and k is a field of characteristic zero, then we denote by X(kG) the set of irreducible characters of kG. For every $\chi \in X(k G)$, we let $V_\chi$ be a finitely generated (irreducible) kG-module with that character. Is H is a kG-module of finite rank, then we write $H[\chi ]$ for the $V_\chi$ isogeny space ${\text {Hom}}_{kG}(V_\chi ,H)$. The $\chi$-isotypical summand of H is the image of the evaluation map $V_\chi \otimes _k H[\chi ]\rightarrow H$ and denoted by $H_\chi$. We indeed have $H=\oplus _{\chi \in X(k G)} H_\chi$.

We write ${{\mathbb {K}}}$ for the Hamilton quaternions, considered as a division ring containing ${{\mathbb {R}}}$.

We follow the tradition in algebraic topology to use the semicolon in denoting the (co)homology of a space with values in a coefficient group (the comma being reserved for relative cohomology e.g., $H_r(X, A; {{\mathbb {Q}}})$). But we adhere to the tradition in algebraic geometry (where sheaf cohomology with supports takes the place of relative cohomology) to use the comma for denoting sheaf cohomology, e.g., $H^r(X, {{\mathscr {O}}}_X)$.

2 Endomorphisms of Jacobians of G-curves

A central role in this paper is played by a moduli stack whose definition we will give in a moment. We do this in some detail, as it involves some subtleties. More on this can be found in a paper by Collas and Maugeais [6] which also addresses the question over which number field such a stack is defined.

Let S be a fixed closed oriented surface of genus $g\ge 2$, G a finite group and $\phi$ a monomorphism of G in the mapping class group ${\text {Mod}}(S)$ of S. By a theorem of Nielsen-Kerckhoff, we can lift $\phi$ to a monomorphism $\Phi$ of G to a group of orientation preserving diffeomorphisms of S. This implies that S admits a G-invariant complex structure so that we obtain a smooth projective complex G-curve C. We consider the G-curves which so arise. To be precise, we consider pairs $(C,G\hookrightarrow {\text {Aut}}(C))$, where C is a smooth projective complex algebraic curve C of genus $\ge 2$ and $G\hookrightarrow {\text {Aut}}(C)$ a monomorphism of groups such that there exists an orientation preserving G-equivariant homeomorphism of C onto S. Observe that the last property only depends on the ${\text {Mod}}(S)$-conjugacy class of $\phi$. So if we precompose $\phi$ with an automorphism $\sigma$ of G, then we get an isomorphic object if and only if there exists an orientation preserving diffeomorphism h of S such that $h \phi (g)h^{-1}=\phi \sigma (g)$ for all $g\in G$. We denote a pair as above simply by (C, G) (which is admittedly somewhat of an abuse of notation, as it hides the datum of the ${\text {Inn}}({\text {Mod}}(S))$-orbit of $\phi$) and we will often refer to it as a G-curve of topological type $\phi$.

The G-curves of topological type $\phi$ are parameterized by a D-M stack ${{\mathscr {M}}}^\phi$, which is defined as follows. Recall that the space of isotopy classes of complex structures on S compatible with the given orientation is parameterized by a connected complex manifold, the Teichmüller space ${\text {Teich}}(S)$, which supports a universal holomorphic family of curves ${\text {Univ}}(S)\rightarrow {\text {Teich}}(S)$. The mapping class group ${\text {Mod}}(S)$ acts on this family. We let G act on ${\text {Teich}}(S)$ via the natural map $\phi : G\rightarrow {\text {Mod}}(S)$ and then its fixed point locus ${\text {Teich}}(S)^G$ (¹) parameterizes the curves of topological type $\phi$. Since the G-invariant complex structures on S are in bijective correspondence with the complex structures on $S_G$, we may identify ${\text {Teich}}(S)^G$ with the Teichmüller space ${\text {Teich}}(S_G)$ of the orbifold $S_G$.

The centralizer ${\text {Mod}}(S)^G$ of G in ${\text {Mod}}(S)$ meets ${\text {Mod}}(S)$ in the center Z(G) of G and the product $G\cdot {\text {Mod}}(S)^G$ is a subgroup of the normalizer of G in ${\text {Mod}}(S)$. The center Z(G) acts trivially on ${\text {Teich}}(S)^G$ and the quotient ${\text {Mod}}(S)^G/Z(G)$ can be identified with a finite index subgroup of the mapping class group ${\text {Mod}}(S_G)$ (this is the group of mapping classes of the underlying surface which respect the type of the orbifold points). Note that an element of ${\text {Mod}}(S_G)$ lies in this image if and only if it lifts to a mapping class of S which commutes with the G-action.

The group $G\cdot {\text {Mod}}(S)^G\subset {\text {Mod}}(S)$ acts on the restriction ${\text {Univ}}(S,G)\rightarrow {\text {Teich}}(S)^G$ of ${\text {Univ}}(S)\rightarrow {\text {Teich}}(S)$ to ${\text {Teich}}(S)^G$ and since G acts trivially on ${\text {Teich}}(S)^G$, this turns the restriction ${\text {Univ}}(S,G)\rightarrow {\text {Teich}}(S)^G$ into a G-curve over ${\text {Teich}}(S)^G$. It is universal as a marked G-curve, the marking being given by an equivariant isotopy class of diffeomorphisms with (S, G) (so $\phi$ is here still being suppressed in the notation). If we divide out this universal marked G-curve by the action of ${\text {Mod}}(S)^G$, that is, if we pass to the stack or orbifold quotient, then we obtain a smooth complex D-M stack, denoted ${{\mathscr {M}}}^\phi$, which parameterizes the G-curves of topological type $\phi$. It is characterized by the property that if Y is a complex variety (or scheme for that matter), then a smooth G-curve over Y of topological type $\phi$ gives rise to a (stack) morphism $Y\rightarrow {{\mathscr {M}}}^\phi$, and that thus is obtained a bijection between the isomorphism classes of smooth G-curves over Y of topological type $\phi$ and the morphisms from Y to ${{\mathscr {M}}}^\phi$. Beware that if we ignore the stack structure by passing to the underlying variety, the fiber over every closed point is a quotient of a G-curve by a subgroup of G which contains Z(G).

It is clear that ${{\mathscr {M}}}^\phi$ only depends on the ${\text {Inn}}({\text {Mod}}(S))$-orbit of $\phi$. Since ${\text {Teich}}(S)^G$ is a simply-connected manifold, ${{\mathscr {M}}}^\phi$ is irreducible and the fundamental group of the underlying analytic orbifold is ${\text {Mod}}(S)^G$. Precomposition with an automorphism $\rho$ of G may lead to a different D-M complex stack ${{\mathscr {M}}}^{\phi \rho }$ to which we shall refer as the $\rho$-twist of ${{\mathscr {M}}}^\phi$. If however, $\rho$ is inner and defined by conjugation with some $g\in G$, say, then $g: C\rightarrow C$ makes the G-curve isomorphic to its $\rho$-twist. So the union ${{\mathscr {M}}}^{[G]}$ of ${\text {Aut}}(G)$-twists (denoted by Collas-Maugeais in [6] by ${{\mathscr {M}}}_{g}[G]$, where g is the genus of S) comes with an action of ${\text {Aut}}(G)$, which permutes its irreducible components through ${\text {Out}}(G)$. If we pass to the stack quotient with respect to this ${\text {Aut}}(G)$-action, we obtain a D-M stack ${{\mathscr {M}}}^{(G)}$ which parameterizes curves C endowed with a subgroup of ${\text {Aut}}(C)$ isomorphic to G (but with no such isomorphism specified).

The forgetful morphism ${{\mathscr {M}}}^{\phi }\rightarrow {{\mathscr {M}}}(S)$ is finite and the universal G-curve ${{\mathscr {C}}}_{{{\mathscr {M}}}^\phi }/{{\mathscr {M}}}^\phi$ fits in a forgetful cartesian diagram which maps to the universal curve ${{\mathscr {C}}}_{{{\mathscr {M}}}(S)}/{{\mathscr {M}}}(S)$. Note that we have a factorization ${{\mathscr {M}}}^{\phi }\rightarrow {{\mathscr {M}}}^{(G)}\rightarrow {{\mathscr {M}}}(S)$. In fact, ${{\mathscr {M}}}^{(G)}$ can be regarded as the normalization of the image of ${{\mathscr {M}}}^{\phi }\rightarrow {{\mathscr {M}}}(S)$ (this expresses the fact that a curve C whose automorphism group contains a copy of G, will in general have only one such copy) and ${{\mathscr {M}}}^{\phi }\rightarrow {{\mathscr {M}}}^{(G)}$ will be étale.

The following observation is important for what follows. For every G-curve (C, G) of type $\phi$ (so defining a point of ${{\mathscr {M}}}^\phi$), G acts on $H^0(C,\Omega _C)$ but the character of this action will be independent of the point we choose, simply because ${{\mathscr {M}}}^\phi$ is irreducible. In other words, this only depends on $\phi$, so that this character may be regarded as a topological invariant. We are therefore justified in denoting it by $\chi ^\phi$. It is clear that for every $\rho \in {\text {Aut}}(G)$, $\chi ^{\phi \sigma }=\rho ^*\chi ^{\phi }$.

Let us say that a geometric point $\{x\}\rightarrow {{\mathscr {M}}}^\phi$ is a very general point of ${{\mathscr {M}}}^\phi$ if it is not contained in a given countable union of closed proper substacks of ${{\mathscr {M}}}^\phi$.

Theorem 1.1

Let G be a finite group and $\phi : G\rightarrow {\text {Mod}}(S)$ a monomorphism such that the centralizer of the image of $\phi$ does not contain a hyperelliptic involution. Assume that the associated moduli stack ${{\mathscr {M}}}^\phi$ has positive dimension and that for a G-curve C representing a point of ${{\mathscr {M}}}^\phi$ the action of G on $H^0(C, \Omega _C)$ satisfies the property that, for every complex G-character $\chi$ afforded by $H^0(C, \Omega _C)$, where we denote by $m_\chi$ its multiplicity:

• (i) if $\chi$ is symplectic, then $m_\chi \ge 3$;

• (ii) if $\chi \ne \overline{\chi }$, then $m_\chi ,m_{\overline{\chi }}\ge 2$.

Then the natural homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is surjective, provided the G-curve C represents a very general member of ${{\mathscr {M}}}^\phi$.

Remark 1.2

We can avoid the notion of a very general point by using the following more concrete (and equivalent) formulation of the conclusion of this theorem: if ${{\mathscr {C}}}/Y$ is a G-curve over a complex variety Y of type $\phi$ such that the associated morphism $Y\rightarrow {{\mathscr {M}}}^\phi$ is dominant, then ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J({{\mathscr {C}}}/Y))$ is onto. This is essentially what we shall prove and it is also the formulation through which we will use it.

Remark 1.3

If the genus of the quotient curve C/G is zero, then the natural homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is not injective, as the trivial G-module is not represented in $H^0(C,\Omega _C)$. But as we will see, the Chevalley-Weil formula implies that this map is injective when the genus of C/G is at least 2. This formula will also show that the conditions we imposed on the G-action on $H^0(C, \Omega _C)$ are topological, in the sense that they only depend on how G acts on $\phi$ (as we should expect).

The assumption that ${{\mathscr {M}}}^\phi$ has positive dimension translates into asking that the moduli space of the quotient orbifold curve (so endowed with the finite subset representing irregular G-orbits) is not rigid. This simply amounts to demanding that the regular G-orbit space of C has negative Euler characteristic, but is not a 3-punctured sphere. It is also equivalent to $H^0(C, \Omega _C^{\otimes 2})^G$ being nonzero.

The hypothesis that G does not contain a hyperelliptic involution in its center is needed in order to ensure that some point of ${{\mathscr {M}}}^\phi$ parameterizes a nonhyperelliptic curve. Then, since the hyperelliptic involution of a hyperelliptic curve is contained in the center of its automorphism group, every point of ${{\mathscr {M}}}^\phi$ has that property.

Observe that if a nontrivial automorphism of C acts trivially on its canonical image in $\check{{\mathbb {P}}}(H^0(C,\Omega _C))$, then C is hyperelliptic and this automorphism is the hyperelliptic involution. The latter acts as $-1$ on $H^0(C,\Omega _C)$, and so G always acts faithfully on $H^1(C,{{\mathbb {C}}})$.

The hypothesis on the G-module structure of $H^0(C,\Omega _C)$ comes from the use of the following lemma in the proof.

Lemma 1.4

Let F be a finite dimensional ${{\mathbb {C}}}G$-module with the property that, for every complex G-character $\chi$ afforded by F:

• (i) if $\chi$ is symplectic, then $m_\chi \ge 3$;

• (ii) if $\chi \ne \overline{\chi }$, then $m_\chi ,m_{\overline{\chi }}\ge 2$.

Let L be a compact Lie subgroup of ${\text {GL}}(F)$, containing the image of G and such that $({\text {Sym}}^2F)^L= ({\text {Sym}}^2F)^G$. Then L is contained in the image of ${{\mathbb {R}}}G$ in ${End}(F)$.

The proof uses the decomposition of ${{\mathbb {R}}}G$ as an ${{\mathbb {R}}}$-algebra in terms of the decomposition of ${{\mathbb {C}}}G$ as a ${{\mathbb {C}}}$-algebra and so we need to recall how this comes about. Remember that $X({{\mathbb {C}}}G)$ stands for the set of characters of irreducible ${{\mathbb {C}}}G$-modules, that for every $\chi \in X({{\mathbb {C}}}G)$ we have chosen a representative ${{\mathbb {C}}}G$-module $V_\chi$, and that the obvious map

$$\begin{aligned} {{\mathbb {C}}}G\rightarrow \prod _{\chi \in X({{\mathbb {C}}}G)} {End}(V_\chi ) \end{aligned}$$

is an isomorphism of ${{\mathbb {C}}}$-algebras. ‘Taking the dual’ defines an involution $\chi \mapsto \chi ^*$ in $X({{\mathbb {C}}}G)$. Then for all $\chi , \mu \in X({{\mathbb {C}}}G)$, the space of G-invariant bilinear forms $V_\chi \times V_{\mu }\rightarrow {{\mathbb {C}}}$ is nonzero unless $\mu =\chi ^*$, in which case it is generated by a nondegenerate pairing ${\alpha _\chi : V_\chi \times V_{\chi ^*}\rightarrow {{\mathbb {C}}}}$. The fixed point set of this involution decomposes into the orthogonal characters $X({{\mathbb {C}}}G)_+$ (the $\chi$ for which $\alpha _\chi$ is symmetric) and the symplectic characters $X({{\mathbb {C}}}G)_-$ (the $\chi$ for which $\alpha _\chi$ is anti-symmetric). Let $X({{\mathbb {C}}}G)_o\subset X({{\mathbb {C}}}G)$ be a system of representatives for the free orbits, so that $X({{\mathbb {C}}}G)$ is the disjoint union of $X({{\mathbb {C}}}G)_+$, $X({{\mathbb {C}}}G)_-$, $X({{\mathbb {C}}}G)_o$ and $X({{\mathbb {C}}}G)_o^*$. Choose a G-invariant inner product $h_\chi$ in $V_\chi$. When $\chi \in X({{\mathbb {C}}}G)_\varepsilon$ with $\varepsilon =\pm$, let $c_\chi$ be the anti-linear automorphism of $V_\chi$ characterized by the property that $\alpha _\chi (x,y)=h_\chi (c_\chi (x), y)$. Note that $c_\chi$ is G-equivariant. Hence so is $c_\chi ^2$. As $c_\chi ^2$ is also ${{\mathbb {C}}}$-linear, Schur’s lemma (applied to the irreducible ${{\mathbb {C}}}G$-module $V_\chi$) implies that $c_\chi ^2$ must be a scalar, $\lambda$, say. The identity

$$\begin{aligned} \lambda h_\chi (x, x)=h_\chi (c_\chi ^2(x), x)=\alpha _\chi (c_\chi (x),x)=\varepsilon \alpha _\chi (x, c_\chi (x))=\varepsilon h_\chi (c_\chi (x), c_\chi (x)) \end{aligned}$$

then shows that $\lambda$ is real and has the sign $\varepsilon$. Upon replacing $\alpha _\chi$ by $|\lambda |^{-1/2}\alpha _\chi$, we arrange that $c_\chi ^2=\varepsilon 1$. In the orthogonal case ($c_\chi ^2=1$), the fixed point space of $c_\chi$ is a real form $V_\chi ({{\mathbb {R}}})$ of $V_\chi$ and so ${{\mathbb {R}}}G$ maps to ${End}_{{\mathbb {R}}}(V_\chi ({{\mathbb {R}}}))$. In the symplectic case ($c_\chi ^2=-1$), $c_\chi$ endows $V_\chi$ with the structure of a right ${{\mathbb {K}}}$-module (recall that ${{\mathbb {K}}}$ denotes the Hamilton quaternions: here we let $c_\chi$ act as j), which extends the ${{\mathbb {C}}}$-vector space structure, so that ${{\mathbb {R}}}G$ maps to ${End}_{{\mathbb {K}}}(V_\chi )$. It is known that the disjoint union of $\{V_\chi ({{\mathbb {R}}})\}_{\chi \in X({{\mathbb {C}}}G)_+}$, $\{V_\chi \}_{\chi \in X({{\mathbb {C}}}G)_-}$ and $\{V_\chi \}_{\chi \in X({{\mathbb {C}}}G)_o}$ represent the distinct elements of $X({{\mathbb {R}}}G)$ (complexification as a ${{\mathbb {R}}}G$-module reproduces resp. $V_\chi$, $V_\chi \oplus V_\chi$ and $V_\chi \oplus V_{\chi *}$) and that the obvious ${{\mathbb {R}}}$-algebra homomorphism

$$\begin{aligned}&{{\mathbb {R}}}G\rightarrow \big (\prod _{\chi \in X({{\mathbb {C}}}G)_+} {End}_{{\mathbb {R}}}(V_\chi ({{\mathbb {R}}})) \big )\times \nonumber \\&\quad \big (\prod _{\chi \in X({{\mathbb {C}}}G)_-} {End}_{{\mathbb {K}}}(V_\chi )\big ) \times \big (\prod _{\chi \in X({{\mathbb {C}}}G)_o} {End}(V_\chi )\big ). \end{aligned}$$

(1)

is an isomorphism (see for instance [21], §13.2).

Proof of Lemma 1.4

For $\chi \in X({{\mathbb {C}}}G)$ we put $F[\chi ]:={\text {Hom}}_{{{\mathbb {C}}}G}(V_\chi , F)$, so that the obvious evaluation map $\oplus _{\chi \in X({{\mathbb {C}}}G)} V_\chi \otimes F[\chi ]\rightarrow F$ is an isomorphism of G-modules. Via this isomorphism, the image of ${{\mathbb {C}}}G$ in ${End}(F)$ is identified with the product of the ${End}(V_\chi )$ taken over all $\chi \in X({{\mathbb {C}}}G)$ with $F[\chi ]\not =0$. The generators $\alpha _\chi$ determine an isomorphism

$$\begin{aligned}&({\text {Sym}}^2 F)^G \cong \big (\underset{\chi \in X({{\mathbb {C}}}G)_+}{\oplus }{\text {Sym}}^2 F[\chi ]\big ) \oplus \big (\underset{\chi \in X({{\mathbb {C}}}G)_-}{\oplus }\wedge ^2 F[\chi ]\big ) \oplus \big (\underset{\chi \in X({{\mathbb {C}}}G)_o}{\oplus }F[\chi ]\otimes F[\chi ^*]\big ). \end{aligned}$$

We are given that this space is also equal to $({\text {Sym}}^2 F)^L$. We claim that for every $\chi \in X({{\mathbb {C}}}G)$ and every line $\ell \subset F[\chi ]$, L preserves $V_\chi \otimes \ell$. To see this, note that with any $q\in {\text {Sym}}^2 F$ is associated a subspace $F_q\subset F$, namely the smallest subspace of F such that $q\in {\text {Sym}}^2F_q$ (so that q is nondegenerate, when regarded as a quadratic form on the dual of $F_q$). It is clear that any linear transformation of F which fixes q, leaves $F_q$ invariant. In our case this means that L leaves invariant every subspace $F_q$ with $q\in ({\text {Sym}}^2 F)^G$. Among the nonzero subspaces $F_q$ we thus obtain are those of the form $V_\chi \otimes \ell$ with $\chi \in X({{\mathbb {C}}}G)_+$ and $\ell$ any line in $F[\chi ]$, $V_\chi \otimes P$ with $\chi \in X({{\mathbb {C}}}G)_-$ and P any plane in $F[\chi ]$, and $(V_\chi \otimes \ell )\oplus (V_{\chi ^*}\otimes \ell ')$ with $\chi \in X({{\mathbb {C}}}G)_o$ and $\ell \subset F[\chi ]$ and $\ell '\subset F[\chi ^*]$ arbitrary lines. Then our assumptions (i) and (ii) imply that for every $\chi \in X({{\mathbb {C}}}G)$ and every line $\ell$ in $F[\chi ]$, $V_\chi \otimes \ell$ is an intersection of such subspaces (and hence preserved by L). It follows that L acts trivially on each ${{\mathbb {P}}}(F[\chi ])$, or rather, that L is contained in $\prod _{\chi \in X({{\mathbb {C}}}G)} {\text {GL}}(V_\chi )\otimes \mathbf {1}_{F[\chi ]}$, so that L is at least contained in the image of ${{\mathbb {C}}}G$ in ${End}(F)$. If $F[\chi ]=0$, then the factor ${\text {GL}}(V_\chi )\otimes \mathbf {1}_{F[\chi ]}$ is of course trivial, but otherwise L will act on $V_\chi$ in such a way that it preserves $\alpha _\chi$.

Assume that $F[\chi ]\not =0$. Since L is compact, it leaves invariant an inner product in $V_\chi$. Since this inner product is also left invariant by G, we may assume that this is the one we used to define the $c_\chi$ above in case $\chi \in X({{\mathbb {R}}}G)_\pm$. So then L preserves $c_\chi$ when defined. Hence in all cases L lands in the corresponding factor of the decomposition given by Eq. (1) so that it is contained in the image of ${{\mathbb {R}}}G$ in ${End}(F)$. $\square$

Remark 1.5

This lemma can fail when a symplectic representation occurs with multiplicity 2. To see this, let V be an irreducible ${{\mathbb {R}}}G$-module which is of quaternionic type. This means that the ${{\mathbb {R}}}$-algebra $k:={End}_{{{\mathbb {R}}}G}(V)$ is isomorphic to ${{\mathbb {K}}}$ and the image of ${{\mathbb {R}}}G$ in ${End}_{{\mathbb {R}}}(V)$ is equal to ${End}_k(V)$. Let $s: V\times V\rightarrow {{\mathbb {R}}}$ be a G-invariant positive definite quadratic form. Then $L:={\text {O}}(V, s)$ is clearly not contained in the image of ${{\mathbb {R}}}G$. The ${{\mathbb {C}}}G$-module $F:={{\mathbb {C}}}\otimes _{{\mathbb {R}}}V$ is a direct sum of two copies of a symplectic irreducible ${{\mathbb {C}}}G$-module, but both G and L have the same subspace of invariants in ${\text {Sym}}^2(V_{{\mathbb {C}}})$, namely the line spanned by the tensor determined by s.

Before we begin the proof of Theorem 1.1, we recall that for a polarized abelian variety A, the polarization determines in the algebra ${End}_{{\mathbb {Q}}}(A)$ an anti-involution (the Rosati involution) $b\mapsto b^\dagger$ which is positive in the sense that for every nonzero $b\in {End}_{{\mathbb {Q}}}(A)$, ${\text {tr}}(bb^\dagger )> 0$. This remains so after tensoring with ${{\mathbb {R}}}$. We recall a few facts about such algebras.

Given a finite dimensional ${{\mathbb {R}}}$-algebra ${{\mathscr {B}}}$ with a positive anti-involution $\dagger$, then by the classification of such algebras, ${{\mathscr {B}}}$ naturally decomposes into a product of ordinary matrix algebras with coefficients ${{\mathbb {R}}}$, ${{\mathbb {C}}}$, or ${{\mathbb {K}}}$, and $\dagger$ is then in each factor equivalent to taking the conjugate transpose. So the set of $f\in {{\mathscr {B}}}$ with $ff^\dagger =1$ is a subgroup $L ({{\mathscr {B}}})$ of the group of units in ${{\mathscr {B}}}$. It is a finite extension of a product of orthogonal groups, unitary groups and unitary quaternion groups, so in particular, compact. Moreover, it generates ${{\mathscr {B}}}$ as an ${{\mathbb {R}}}$-algebra. Given a finite dimensional ${{\mathscr {B}}}$-module F, then we say that an element of ${\text {Sym}}^2F$ is ${{\mathscr {B}}}$-invariant if for all $b\in {{\mathscr {B}}}$, it is killed by $b\otimes 1-1\otimes b^\dagger$. Since $L({{\mathscr {B}}})$ generates ${{\mathscr {B}}}$ as an algebra, this is equivalent to this element being $L({{\mathscr {B}}})$-invariant in the usual sense. There is similar notion of covariance, but in view the reductive nature of ${{\mathscr {B}}}$ it is not worthwhile to make the distinction.

This situation shows up when we consider a polarized abelian variety A, endowed with a homomorphism ${{\mathscr {B}}}\rightarrow {End}_{{\mathbb {R}}}(A)$ of algebras with involution, where we let ${End}_{{\mathbb {R}}}(A):={{\mathbb {R}}}\otimes {End}(A)$. If we put $F:=H^0(A,\Omega _A)$, then ${End}_{{\mathbb {R}}}(A)$ (and hence ${{\mathscr {B}}}$) acts on F. As it is well-known, the space of first order deformations of A as a polarized variety can be identified with the complex dual of ${\text {Sym}}^2F$. The first order deformations for which ${{\mathscr {B}}}$ still maps to the deformed object can be identified with the dual of the ${{\mathscr {B}}}$-coinvariants of ${\text {Sym}}^2F$, i.e., the dual of $({\text {Sym}}^2F)_{L({{\mathscr {B}}})}\cong ({\text {Sym}}^2F)^{L({{\mathscr {B}}})}$. The more precise statement is that we have locally a universal deformation in the sense of Schlessinger with smooth base and whose cotangent space at the closed point can be identified with $({\text {Sym}}^2F)^{L({{\mathscr {B}}})}$.

We now turn to the proof of Theorem 1.1. We first reduce it to Lemma 1.6 below. We write ${{\mathscr {B}}}$ for ${End}_{{\mathbb {R}}}(J(C))$ and F for $H^0(C, \Omega _C)$. The property that ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ be surjective is equivalent to ${{\mathbb {R}}}G\rightarrow {{\mathscr {B}}}$ being surjective and since $L({{\mathscr {B}}})$ generates ${{\mathscr {B}}}$, the latter is equivalent to $L({{\mathscr {B}}})$ being contained in the image of ${{\mathbb {R}}}G$. This is what we will show.

Since $L({{\mathscr {B}}})$ is compact, $({\text {Sym}}^2 F)^{L({{\mathscr {B}}})}$ has a ${L({{\mathscr {B}}})}$-stable supplement $({\text {Sym}}^2F)^{{L({{\mathscr {B}}})}\not =1}$ in ${\text {Sym}}^2 F$. This is also G-stable and so

$$\begin{aligned} ({\text {Sym}}^2F)^{G,{L({{\mathscr {B}}})}\not =1}:=({\text {Sym}}^2F)^G\cap ({\text {Sym}}^2F)^{{L({{\mathscr {B}}})}\not =1} \end{aligned}$$

is a G-invariant supplement of $({\text {Sym}}^2 F)^{L({{\mathscr {B}}})}$ in $({\text {Sym}}^2 F)^G$. In view of our hypotheses and Lemma 1.4, Theorem 1.1 will follow from:

Lemma 1.6

We have $({\text {Sym}}^2 F)^{G,{L({{\mathscr {B}}})}\not = 1}=0$.

The proof of Lemma 1.6 involves a Hilbert scheme. We write ${{\mathbb {P}}}$ for the projective space $\check{{\mathbb {P}}}(F)$ of hyperplanes of F. The very general G-curve yields a canonical embedding $C\subset {{\mathbb {P}}}$. Let us denote by ${{\mathscr {H}}}$ the open subscheme of the irreducible component of ${\text {Hilb}}({{\mathbb {P}}})$ which parameterizes canonically embedded smooth projective curves. The G-equivariant canonical embeddings of G-curves in ${{\mathbb {P}}}$ of our given topological type define an irreducible closed subscheme ${{\mathscr {H}}}^G$ of ${{\mathscr {H}}}$. The action of G on F induces one on ${{\mathscr {H}}}$ and then ${{\mathscr {H}}}^G$ is an irreducible component of the fixed point locus for this action.

Lemma 1.7

Let X be a G-invariant quadric hypersurface of ${{\mathbb {P}}}$ containing the G-invariant canonical curve C. Then the G-invariant Hilbert scheme ${{\mathscr {H}}}_X^G$ (of G-curves contained in X) is a proper subscheme of the G-invariant Hilbert scheme ${{\mathscr {H}}}^G$.

Proof

In what follows we use the following notational convention. Given an immersion $Y\hookrightarrow Z$ of schemes, we write ${{\mathcal {C}}}_{Y/Z}$ for its conormal sheaf on Y. This is an ${{\mathscr {O}}}_Y$-module. The normal sheaf ${{\mathcal {N}}}_{Y/Z}$ of this immersion is by definition the ${{\mathscr {O}}}_Y$-dual of ${{\mathcal {C}}}_{Y/Z}$.

In the proof of Lemma 2.2 in [2], we observed that to the nested embeddings $C\subset X\subset {{\mathbb {P}}}$ is associated a short exact sequence

$$\begin{aligned}0\rightarrow (\Omega _C^{\otimes 2})^\vee \rightarrow {{\mathcal {C}}}_{C/{{\mathbb {P}}}}\rightarrow {{\mathcal {C}}}_{C/X}\rightarrow 0,\end{aligned}$$

which, by Lemma 2.2 in [2], determines an exact sequence

$$\begin{aligned} 0\rightarrow H^0(C,{{\mathcal {N}}}_{C/X})\rightarrow H^0(C,{{\mathcal {N}}}_{C/{{\mathbb {P}}}})\rightarrow H^0(C, \Omega ^{\otimes 2}_C)\rightarrow {\text {Ext}}_C^1({{\mathcal {C}}}_{C/X}, {{\mathscr {O}}}_C)\rightarrow 0. \end{aligned}$$

Passing to G-invariants is an exact functor, and so we get the exact sequence

$$\begin{aligned} 0\rightarrow H^0(C,{{\mathcal {N}}}_{C/X})^G&\rightarrow H^0(C,{{\mathcal {N}}}_{C/{{\mathbb {P}}}})^G\rightarrow H^0(C, \Omega ^{\otimes 2}_C)^G \nonumber \\&\rightarrow {\text {Ext}}_C^1({{\mathcal {C}}}_{C/X}, {{\mathscr {O}}}_C)^G\rightarrow 0, \end{aligned}$$

(2)

where $H^0(C,{{\mathcal {N}}}_{C/X})^G$ and $H^0(C,{{\mathcal {N}}}_{C/{{\mathbb {P}}}})^G$ identify with the tangent space at $[C\subset X]$ to the scheme ${{\mathscr {H}}}_X^G$ resp. the tangent space at $[C\subset {{\mathbb {P}}}]$ to the scheme ${{\mathscr {H}}}^G$, while $H^0(C, \Omega ^{\otimes 2}_C)^G$ identifies with the cotangent space to ${{\mathscr {M}}}^\phi$ at s. By assumption $H^0(C, \Omega ^{\otimes 2}_C)^G\ne \{ 0\}$. It is well-known (see [2], Lemma 2.1) that ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/{{\mathbb {P}}}},{{\mathscr {O}}}_C)^G= H^1(C,{{\mathcal {N}}}_{C/{{\mathbb {P}}}})^G=0$, so that the G-invariant Hilbert scheme ${{\mathscr {H}}}^G$ is smooth at $[C\subset {{\mathbb {P}}}]$. In case ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)^G=0$, then the exact sequence (2) shows that the inclusion ${{\mathscr {H}}}_X^G\subseteq {{\mathscr {H}}}^G$ is strict.

When ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)^G\not =0$, we need the interpretation of this space as an obstruction space, as provided by Lemma 1.8 below. It then tells us that the G-equivariant deformation theory of C in the quadric X is obstructed. So the G-invariant Hilbert scheme ${{\mathscr {H}}}_X^G$ is singular at $[C\subset X]$ and then the inclusion ${{\mathscr {H}}}_X^G\subseteq {{\mathscr {H}}}^G$ is also strict. $\square$

Lemma 1.8

A minimal obstruction space for the G-equivariant deformation theory of C in the quadric X is ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)^G$.

Proof

We proved in Theorem 1.1 in [2] that ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)$ is a minimal obstruction space for the deformation theory of C in the quadric X and so we only need to check that this retains this interpretation in a G-equivariant setting. This is a matter of carefully going through the definitions.

Let $\omega$ be the map which assigns to every small extension $0\rightarrow I\rightarrow A'\rightarrow A\rightarrow 0$ of Artin ${{\mathbb {C}}}$-algebras and every deformation $\xi : {{\mathscr {C}}}\subset X_A$ over A of the embedding $C\subset X$ (where $X_A=X\times {\text {spec}}(A)$) an element $\omega (\xi ,A')\in {\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)\otimes I$ which is the obstruction to lifting $\xi$ to $A'$ (cf. Definition 5.1 in [22]).

The group G acts on deformations of the embedding $C\subset X$ by changing the identification of the central fiber. Let us assume that $0\rightarrow I\rightarrow A'\rightarrow A\rightarrow 0$ is a tiny extension, i.e. that $I\cong {{\mathbb {C}}}$. Then, from the canonicity of minimal obstruction spaces (cf. Exercise 5.8 in [22]), it follows that $\omega (\alpha \cdot \xi ,A')=\alpha \cdot \omega (\xi ,A')$, for all $\alpha \in G$. Therefore, obstructions arising from G-equivariant lifting problems are contained in ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)^G$.

We have to show that all elements of ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)^G$ are obtained as obstructions to G-equivariant lifting problems. For this, it is enough to observe the following. Every element of ${\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)$ is of the form $\omega (\xi ,A')$, for some tiny extension $0\rightarrow I\rightarrow A'\rightarrow A\rightarrow 0$ and some deformation $\xi : {{\mathscr {C}}}\subset X_A$ over A of the embedding $C\subset X$. By the construction appearing in the proof of Proposition 5.6 in [22], the sum of the elements appearing in the G-orbit of $\omega (\xi ,A')$:

$$\begin{aligned} \sum _{\alpha \in G}\alpha \cdot \omega (\xi ,A')=\sum _{\alpha \in G}\omega (\alpha \cdot \xi ,A')\in {\text {Ext}}^1_C({{\mathcal {C}}}_{C/X},{{\mathscr {O}}}_C)^G \end{aligned}$$

is realized as the obstruction to lifting the G-equivariant deformation $\times _{\alpha \in G}\alpha \cdot \xi$ over $\otimes ^{|G|}A$ of the embedding $C\subset X$ in a G-equivariant fashion. $\square$

Proof of Lemma 1.6

For this we use an interpretation of $({\text {Sym}}^2 F)^{G,{L({{\mathscr {B}}})}\not = 1}$ as a conormal space of moduli spaces. We first note that the restriction map

$$\begin{aligned} \Phi : ({\text {Sym}}^2F)^G=({\text {Sym}}^2 H^0(C, \Omega _C))^G \rightarrow H^0(C, \Omega ^{\otimes 2}_C)^G \end{aligned}$$

can be understood as follows: the dual of $H^0(C, \Omega ^{\otimes 2}_C)^G$ is the space of first order deformations of C as a G-curve and the dual of $({\text {Sym}}^2F)^G$ is the space of first order deformations of deformations of the Jacobian J(C) as a ppav with G-action, and $\Phi$ is then the obvious map. (By Max Noether’s theorem, the natural map ${\text {Sym}}^2F={\text {Sym}}^2H^0(C, \Omega _C)\rightarrow H^0(C, \Omega ^{\otimes 2}_C)$ is onto and hence the same is true for $\Phi$.) As mentioned above, the dual of $({\text {Sym}}^2F)^{L({{\mathscr {B}}})}$ has the interpretation of the space of first order deformations of J(C) as a ppav endowed with an involution preserving homomorphism of ${{\mathscr {B}}}$ to the endomorphism algebra tensored with ${{\mathbb {R}}}$. In other words, $({\text {Sym}}^2 F)^{G,{L({{\mathscr {B}}})}\not = 1}$ vanishes on the first order deformations of J(C) as G-ppav which retain ${{\mathscr {B}}}$ in their endomorphism algebra tensored with ${{\mathbb {R}}}$. By construction, the first order deformations of J(C) as a G-Jacobian are of this type. It follows that the kernel of $\Phi$ contains $({\text {Sym}}^2 F)^{G,{L({{\mathscr {B}}})}\not = 1}$. As C is very general, it is also very general as a member of ${{\mathscr {H}}}^G$ and so this will then be true for all members parameterized by ${{\mathscr {H}}}^G$ (we here use that the associated morphism ${{\mathscr {H}}}^G\rightarrow {{\mathscr {M}}}(G)$ is dominant, see Remark 1.2). It follows that any nonzero element of $({\text {Sym}}^2 F)^{G,{L({{\mathscr {B}}})}\not = 1}$ gives us a G-invariant quadric X for which ${{\mathscr {H}}}_X^G={{\mathscr {H}}}^G$. As this would contradict Lemma 1.7, we must have $({\text {Sym}}^2 F)^{G,{L({{\mathscr {B}}})}\not = 1}=0$. $\square$

The hypotheses of Theorem 1.1 are satisfied in most of the situations of interest. For instance, if the genus of the quotient curve C/G is positive, then so is the dimension of the corresponding moduli stack ${{\mathscr {M}}}^\phi$ and G does not contain a hyperelliptic involution. In order to establish when the condition on the structure of the G-module of abelian differentials $H^0(C,\Omega _C)$ is also satisfied, we use the classical Chevalley-Weil formula, which we now recall.

Let $C\rightarrow C/G$ be a Galois covering defined by a smooth projective G-curve and let $\chi \in X({{\mathbb {C}}}G)$ be nontrivial (that is, not the character of the trivial irreducible ${{\mathbb {C}}}G$-module ${{\mathbb {C}}}$, which is constant equal to 1). Denote by $B\subset C/G$ the branch locus (a finite set). For every $Q\in B$, we choose a representative ${\tilde{Q}}\in C$ and let $\gamma _{{\tilde{Q}}}$ be the positive generator of the (cyclic) stabilizer $G_{{\tilde{Q}}}\subset G$. Since the conjugacy class of $\gamma _{{\tilde{Q}}}$ only depends on Q, the same is true for the characteristic polynomial of $\gamma _{{\tilde{Q}}}$ in $V_\chi$. For $a\in {{\mathbb {Q}}}\cap (0,1]$, let $N_{\chi , Q}(a)$ be the multiplicity of $\exp (2\pi \sqrt{-1}a)$ as a root of this polynomial and put $N_\chi (a):=\sum _{Q\in B} N_{\chi , Q}(a)$. Then the Chevalley-Weil formula [3] asserts that

$$\begin{aligned} {\text {dim}}_{{\mathbb {C}}}H^0(X,\Omega _X)[\chi ] =(g(C/G)-1) \chi (1)+ \sum _{a\in {{\mathbb {Q}}}\cap (0,1]} N_{\chi }(a)(1-a). \end{aligned}$$

(3)

If $\chi$ is the trivial character, the left hand side is of course equal to g(C/G). This shows that when $g(C/G)\ge 2$, all irreducible characters of G are afforded by the ${{\mathbb {C}}}G$-module $H^0(C,\Omega _C)$, so that the natural homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is injective.

Corollary 1.9

For a very general G-curve C for which the genus of the orbit curve C/G is at least 3, the natural homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is an isomorphism.

Proof

The Chevalley-Weil formula shows that every $\chi \in X({{\mathbb {C}}}G)$ appears in $H^0(C,\Omega _C)$ with multiplicity $\ge 2$ and even with multiplicity $\ge 4$, unless $\chi$ is the character of the trivial representation. Now apply Theorem 1.1. $\square$

We shall see (Remark 1.13) that this corollary can fail if $g(C/G)=2$.

Since the dual of $H^0(C,\Omega _C)$ can be identified with its Hodge supplement in $H^1(C; {{\mathbb {C}}})$, the character of $H^1(C; {{\mathbb {C}}})$ must be equal to $\chi ^\phi +\overline{\chi ^\phi }$. The character of $H^1(C,{{\mathbb {C}}})$ can also be computed by means of a G-invariant cellular decomposition of the underlying oriented surface S: choose one such that the induced decomposition S/G has as zero skeleton the union of B and a base point of $S/G\smallsetminus B$, has $2g(S/G)+|B|$ 1-cells and has one 2-cell. The associated cellular complex is then

$$\begin{aligned} 0\rightarrow {{\mathbb {Z}}}G \rightarrow ({{\mathbb {Z}}}G)^{2g(C/G)}\oplus ({{\mathbb {Z}}}G)^B\rightarrow {{\mathbb {Z}}}(G)\oplus \underset{Q\in B}{\oplus }{{\mathbb {Z}}}(G/G_{{\tilde{Q}}}) \rightarrow 0. \end{aligned}$$

(4)

The complex cohomology of S as a ${{\mathbb {C}}}G$-module is that of the complex obtained by applying ${\text {Hom}}(\; , {{\mathbb {C}}})$ and from this we see that for a nontrivial irreducible ${{\mathbb {C}}}G$-module $V_\chi$,

$$\begin{aligned} {\text {dim}}_{{\mathbb {C}}}H^1(X,{{\mathbb {C}}})[\chi ] =2( g(S/G)-1)\chi (1)+\sum _{Q\in B} {\text {dim}}(V_\chi /V_\chi ^{G_{{\tilde{Q}}}}). \end{aligned}$$

(5)

So if $\chi ^\phi$ is self-dual, then the character $H^0(C,\Omega _C)$ will then be half that of $H^1(C,{{\mathbb {C}}})$. Conversely, when G acts freely on S, then a comparison with the Chevalley–Weil formula (3) shows that $\chi ^\phi$ is half the character of $H^1(C,{{\mathbb {C}}})$, so that $\chi ^\phi$ must be self-dual.

Remark 1.10

It is not hard to see that complex conjugation sends ${{\mathscr {M}}}^\phi$ to a ${{\mathscr {M}}}^{{\bar{\phi }}}$, where ${\bar{\phi }}$ is represented by retaining our $\phi : G\hookrightarrow {\text {Mod}}(S)$ and changing the orientation of S. We then have $\chi ^{{\bar{\phi }}}=\overline{\chi ^\phi }$. In particular, if ${{\mathscr {M}}}^\phi$ is real, then so is $\chi ^\phi$. Whether this is a useful result is another matter. Let us just note that it fits in a more general Galois setting: if ${{\mathscr {M}}}^{(G)}$ is defined over the number field K (Collas and Maugeais have shown in [6, 7] that we can take $K={{\mathbb {Q}}}$ for G cyclic), then the individual ${{\mathscr {M}}}^\phi$ are defined over a (common) finite extension L/K so that the Galois group of ${\overline{K}}/K$ permutes them through ${\text {Gal}}(L/K)$. The induced action of ${\text {Gal}}(L/K)$ on the characters $\chi ^\phi$ will then be the obvious one (via a cyclotomic character, since all the characters of G take their values in the extension of ${{\mathbb {Q}}}$ generated by the |G|-th roots of unity).

Even though we cannot expect that the hypotheses of Theorem 1.1 are always satisfied by G-curves C whose orbit curve has genus zero, there is a particular case for which this happens:

Theorem 1.11

Let C be a smooth projective G-curve and $N\subset G$ a subgroup of index 2 which acts freely on C and is such that C/N is of genus $\ge 2$ with G/N acting on C/N as a hyperelliptic involution (so that C/G is of genus zero). Then

• (i) G is a semi-direct product of N and a group of order 2 and the G-invariant subspace $H^0(C/N,\Omega _{C/N})$ of $H^0(C,\Omega _C)$ is acted on via G/N with the nontrivial element acting as scalar multiplication by $-1$.

• (ii) Every irreducible ${{\mathbb {C}}}G$-submodule V of $H^0(C,\Omega _C)$ Which is not contained in $H^0 (C/N,\Omega _{C/N})$ decomposes into two non-isomorphic irreducible ${{\mathbb {C}}}N$-modules, which are characterized by the property that if we induce them up to G, we get a ${{\mathbb {C}}}G$-module equivalent to V. The two are exchanged by the outer action of G/N and V appears in $H^0(C,\Omega _C)$ with multiplicity $\tfrac{1}{2}(g(C/N)-1) {\text {dim}}V$.

• (iii) If N is nontrivial, then C is nonhyperelliptic.

Moreover, if C/N is of genus $\ge 4$ and (C, G) is very general as a G-curve, then the natural homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is surjective.

Remark 1.12

In case N is trivial (so that C is hyperelliptic), the three assertions are trivial or empty, and the last clause follows from Theorem 1.2 in [5]. We will therefore assume in the proof below that N is nontrivial.

Proof

We first show that the involution $\iota$ of C/N induced by the nontrivial element of G/N lifts to an involution $\tilde{\iota }$ in G. To see this, let $x\in C$ be such that its image $x_{N}$ in C/N is fixed under $\iota$. Since N acts freely on C, the G-stabilizer $G_x$ of x meets N trivially. So $G_x$ is of order two and its nontrivial element $\tilde{\iota }$ is a lift of $\iota$. In particular, G is a semi-direct product $\langle \tilde{\iota }\rangle \ltimes N$. Part (i) now follows, because the hyperelliptic involution $\iota$ acts on $H^0(C/N,\Omega _{C/N})$ as multiplication by $-1$.

Since N is of index 2 in G, we have for every irreducible G-submodule V of $H^0(C,\Omega _C)$ two possibilities, according to whether the outer action of $\iota \in G/N$ fixes the character of V or not (cf. Proposition 5.1 in [9]):

• (irr) V is also irreducible as a ${{\mathbb {C}}}N$-module (with $\tilde{\iota }$ acting on it trivially or by multiplication by $-1$) or

• (red) V splits into two non-isomorphic irreducible ${{\mathbb {C}}}N$-modules $V'\oplus V''$, where $V''$ is obtained from $V'$ by precomposition of N with conjugation by $\tilde{\iota }$, and ${\text {Ind}}^G_N V'$ and ${\text {Ind}}^G_N V''$ are equivalent to V.

We now invoke the Chevalley–Weil formula (3). The group N acts freely on C, while the involution $\iota$ in C/N is hyperelliptic, and so G has $2g(C/{N})+2$ irregular orbits in C, each of which contains a fixed point of $\tilde{\iota }$. So the sum in the right hand side of (3) is in fact a single term $N_V(a)(1-a)$ with $N_V(a)=(2g(C/{N})+2){\text {dim}}V$ and either $a=1$ or $a=\tfrac{1}{2}$, according to whether $\tilde{\iota }$ acts in V as 1 or as $-1$. We find that in the first case, the multiplicity of V in $H^0(C,\Omega _C)$ is $-{\text {dim}}V$ (which is of course absurd) and so $\tilde{\iota }$ must act in V as $-1$ and V appears with multiplicity $-{\text {dim}}V+ (2g(C/{N})+2){\text {dim}}V.\frac{1}{2}=g(C/{N}){\text {dim}}V$. But N acts freely on C and hence the Chevalley–Weil formula (3) applied to the N-action implies that the only irreducible representation of N which appears in $H^0(C, \Omega _C)$ with multiplicity g(C/N) is the trivial one, in other words, V is then contained in $H^0(C/N, \Omega _{C/N})$.

The above dichotomy also shows that if the irreducible ${{\mathbb {C}}}G$-module V is not contained in $H^0(C/N, \Omega _{C/N})$, then V decomposes as a ${{\mathbb {C}}}N$-module as in (ii): $V=V'\oplus V''$. Since $V'$ appears in $H^0(C, \Omega _C)$ with multiplicity $(g(C/N)-1){\text {dim}}V'$, the ${{\mathbb {C}}}G$-module $V\cong {\text {Ind}}^G_N V'$ appears in $H^0(C, \Omega _C)$ with the same multiplicity, i.e., $\frac{1}{2}(g(C/N)-1){\text {dim}}V$. This proves (ii). Since $\tilde{\iota }$ has the effect of exchanging $V'$ and $V''$, this also shows that $\tilde{\iota }$ is not hyperelliptic.

Suppose now $g(C/N)\ge 4$ and (C, G) is very general. The one-dimensional nontrivial representation of G/N appears in $H^0(C, \Omega _C)$ and its isotypical subspace is $H^0(C/N, \Omega _{C/N})$. So it has multiplicity $\ge 4$ in $H^0(C, \Omega _C)$. According to (ii), any other nontrivial irreducible representation V which appears in $H^0(C, \Omega _C)$ is of the form $V'\oplus V''$ and hence appears with multiplicity $(g(C/N)-1){\text {dim}}V'\ge 3$. It then remains to apply Theorem 1.1. $\square$

Remark 1.13

Theorem 1.11 implies that the image ${{\mathbb {Q}}}N\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ cannot contain the lift $\tilde{\iota }$ of the hyperelliptic involution (for $\tilde{\iota }$ permutes non-isomorphic irreducible ${{\mathbb {Q}}}N$-modules), so that in particular, ${{\mathbb {Q}}}N\rightarrow {End}_{{\mathbb {Q}}}(J(C))$ is not surjective. Since it is easy to find nontrivial unramified covers of a closed genus 2 surface such that a hyperelliptic involution lifts, we see that Corollary 1.9 can fail when the genus of the orbit curve is 2.

3 Symplectic G-modules and associated unitary groups

We here collect (or derive) some facts regarding symplectic G-modules and the algebraic groups that they give rise to, insofar they are needed in this paper.

3.1 Irreducible G-modules

We review the basics of the theory, as can be found for instance in [11] and use this opportunity to establish notation. Let k be a totally real number field or be equal to ${{\mathbb {R}}}$ (but for us only the cases $k={{\mathbb {Q}}}$ and $k={{\mathbb {R}}}$ will be relevant). The set X(kG) of irreducible characters of kG is a set of k-valued class functions on G, which is invariant under the Galois group of $k/{{\mathbb {Q}}}$. For every $\chi \in X({{\mathbb {Q}}}G)$, there is always a positive integer m and an orbit in the Galois group of ${{\mathbb {R}}}/{{\mathbb {Q}}}$ in $X({{\mathbb {R}}}G)$ such that $\chi$ is m times the sum over this Galois orbit (see e.g., Theorem (9.21) in [11]). We shall express this shortly in terms of the representations themselves.

For a given $\chi \in X(k G)$, Schur’s lemma implies that ${End}_{k G}(V_\chi )$ is a division algebra. We denote the opposite algebra by $D_\chi$ and regard $V_\chi$ as a left kG-module and as a right $D_\chi$-module. We also fix a positive definite G-invariant inner product $s_\chi :V_\chi \times V_\chi \rightarrow k$. For any bilinear form $\phi : V_\chi \times V_\chi \rightarrow k$, there exists a unique $\sigma \in {End}_k(V_\chi )$ such that $\phi (v, v')= s_\chi (\sigma (v), v')$. The form $\phi$ is G-invariant if and only if $\sigma$ is G-equivariant, so that the latter will be given by right multiplication with some $\lambda \in D_\chi$. For the same reason there exists a $\lambda ^\dagger \in D_\chi$ such that $\phi (v, v')= s_\chi (v, v'\lambda ^\dagger )$. Note that thus is defined an anti-involution $\lambda \mapsto \lambda ^\dagger$ on $D_\chi$ characterized by the property that $s_\chi (v\lambda , v')=s_\chi (v, v'\lambda ^\dagger )$. So if $\lambda \in D_\chi$ is symmetric in the sense that $\lambda =\lambda ^\dagger$, then $(v, v')\mapsto s_\chi (v\lambda , v')=s_\chi (v, v'\lambda )$ is also a G-invariant symmetric bilinear form with values in k. In particular, $s_\chi$ need not be unique up to scalar. The identity $s(v\lambda , v'\lambda )=s(v\lambda \lambda ^\dagger , v')$, shows that $\dagger$ is positive in the following sense. Denote by $L_\chi$ the center of $D_\chi$ and by $K_\chi \subset L_\chi$ the subfield fixed under the anti-involution $\dagger$. Then $K_\chi$ is finite extension of k that is totally real and for every nonzero $\lambda \in D_\chi$, the trace ${\text {tr}}_{D_\chi /K_\chi }(\lambda \lambda ^\dagger )\in K_\chi$ has positive image under every field embedding $K_\chi \hookrightarrow {{\mathbb {R}}}$.

Recall that for a finitely generated kG-module H and $\chi \in X(kG)$, we write $H[\chi ]$ for ${\text {Hom}}_{k G}(V_\chi , H)$. The right $D_\chi$-module structure on $V_\chi$ determines a left $D_\chi$-module structure on $H[\chi ]$ by the rule $(du)(v):=u(vd)$ and then the natural map

$$\begin{aligned} \oplus _{\chi \in X(k G)} V_\chi \otimes _{D_\chi } H[\chi ]\rightarrow H, \quad v\otimes _{D_\chi } u\in V_\chi \otimes _{D_\chi } H[\chi ]\mapsto u(v) \end{aligned}$$

is an isomorphism of kG-modules. This is the kG-isotypical decomposition of H with the image $H_\chi$ of $V_\chi \otimes _{D_\chi } H[\chi ]$ being its $\chi$-isotypical summand. Any kG-linear automorphism of H preserves its G-isotypical decomposition and acts in the summand $H_\chi \cong V_\chi \otimes _{D_\chi } H[\chi ]$ through a $D_\chi$-linear transformation of $H[\chi ]$. Thus, the G-centralizer ${End}_k(H)^G\subset {End}_k(H)$ is identified with $\prod _{\chi \in X(k G)} {End}_{D_\chi }(H[\chi ])$. The natural map

$$\begin{aligned} kG\rightarrow \prod _{\chi \in X(k G)} {End}^{D_\chi }(V_\chi ), \end{aligned}$$

where ${End}^{D_\chi }(V_\chi )$ stands for the algebra of endomorphisms of $V_\chi$ as a right $D^\chi$-module, is an isomorphism of k-algebras (this can be regarded as an instance of one of Wedderburn’s structure theorems).

We discuss the passage from $k={{\mathbb {Q}}}$ to $k={{\mathbb {R}}}$. For this, let us for now fix $\chi \in X( kG)$ and temporarily omit $\chi$ in the notation. So we write $D=D_\chi$, $L=L_\chi$, $V=V_\chi$ etc.

When $k={{\mathbb {R}}}$, we have the three classical cases: $D={{\mathbb {R}}}$ (so $\dagger$ is then the identity) or D is isomorphic to ${{\mathbb {K}}}$ or ${{\mathbb {C}}}$, with $\dagger$ in both cases corresponding to the usual conjugation. (These three cases correspond to $\chi \in X({{\mathbb {R}}}G)$ being resp. an element of $X({{\mathbb {C}}}G)$, twice an element of $X({{\mathbb {C}}}G)$, the sum of a complex conjugate distinct pair in $X({{\mathbb {C}}}G)$.)

When $k={{\mathbb {Q}}}$ there are in some sense also three cases. Given a field embedding $\sigma :K\hookrightarrow {{\mathbb {R}}}$, let us use write $\sigma ^*$ for the associated base change: if F is a K-vector space, then $\sigma ^*F={{\mathbb {R}}}\otimes _\sigma F$. For the irreducible ${{\mathbb {Q}}}G$-module $V=V_\chi$, the ${{\mathbb {R}}}G$-module $\sigma ^*V$ is in general no longer irreducible, but it will be isotypical for an irreducible ${{\mathbb {R}}}G$-module $V^\sigma$ with character $\chi ^\sigma \in X({{\mathbb {R}}}G)$, say with multiplicity $m^\sigma$. The characters $\{\chi ^\sigma \}_\sigma$ are pairwise distinct and make up an orbit of the Galois group of ${{\mathbb {R}}}/{{\mathbb {Q}}}$ and $m^\sigma$ is independent of $\sigma$: $m^\sigma =m$. Denote by $D^\sigma$ the opposite of ${End}_{{{\mathbb {R}}}G} (V^\sigma )$. It is a real division algebra, which will be isomorphic to ${{\mathbb {R}}}$, ${{\mathbb {C}}}$ or ${{\mathbb {K}}}$ and whose isomorphism type is independent of $\sigma$. We put $I^\sigma ={\text {Hom}}_{{{\mathbb {R}}}G} (V^\sigma , \sigma ^*V)$ (so we could also write this as $\sigma ^*V[\chi ^\sigma ]$). Then $I^\sigma$ is a left $D^\sigma$-module (of rank m) and a right $\sigma ^*D$-module and we have the isotypical decomposition as ${{\mathbb {R}}}G$-modules

$$\begin{aligned} V_{{\mathbb {R}}}(:={{\mathbb {R}}}\otimes _{{\mathbb {Q}}}V)=\oplus _\sigma \, \sigma ^*V \cong \oplus _\sigma \, V^\sigma \otimes _{D^\sigma } I^\sigma . \end{aligned}$$

This also identifies $\sigma ^*D$ with the opposite of ${End}_{{{\mathbb {R}}}G}(\sigma ^*V)={End}_{D^\sigma }(I^\sigma )$ and gives rise to the isomorphism

$$\begin{aligned} \sigma ^*(H[\chi ])=\sigma ^*({\text {Hom}}_{{{\mathbb {Q}}}G}(V, H))\cong {\text {Hom}}_{{{\mathbb {R}}}G}(\sigma ^*V, H_{{\mathbb {R}}})\cong {\text {Hom}}_{{{\mathbb {R}}}G}(V^\sigma \otimes _{D^\sigma } I^\sigma , H_{{\mathbb {R}}})\cong \\ \cong {\text {Hom}}_{D^\sigma } (I^\sigma , {\text {Hom}}_{{{\mathbb {R}}}G}(V^\sigma , H_{{\mathbb {R}}}))= {\text {Hom}}_{D^\sigma } (I^\sigma , H_{{\mathbb {R}}}[\chi ^\sigma ]) \end{aligned}$$

as left $\sigma ^*D$-modules. The anti-involution $\dagger$ on D yields one on $\sigma ^*D$ and via the identification of the latter with ${End}_{D^\sigma }(I^\sigma )^\mathrm {opp}$ becomes ‘taking the adjoint’ with respect to a hermitian form $h^\sigma$ on $I^\sigma$ obtained as follows. There is up to scalar exactly one G-invariant inner product on the ${{\mathbb {R}}}G$-module $V^\sigma$. Let us fix one and denote it $s^\sigma$. Then the G-invariant symmetric bilinear form s on V determines a positive definite hermitian form $h^\sigma : I^\sigma \times I^\sigma \rightarrow D^\sigma$ (relative to the standard conjugation in $D^\sigma$) characterized by the property that for all $v, v'\in V^\sigma$ and $w, w'\in I^\sigma$, $s^\sigma ( w(v), w'(v'))=s^\sigma (vh^\sigma (w, w'),v')$.

3.2 Symplectic G-modules

Suppose that our kG-module H comes with a nondegenerate G-invariant symplectic form $(a,b)\in H\times H\mapsto \langle a, b\rangle \in k$ (for example $H=H^1(S; k)$, and the symplectic form being the intersection product). We then first observe that we have a decomposition

$$\begin{aligned} {\text {Sp}}(H)^G=\prod _{\chi \in X(k G)} {\text {Sp}}(H_\chi )^G. \end{aligned}$$

Let us now fix some $\chi \in X(k G)$. Given $u,u'\in H[\chi ]$, then $(v, v')\mapsto \langle u(v), u'(v')\rangle$ is a G-invariant bilinear form on $V_\chi$. Since $s_\chi$ is nondegenerate, there is an $h\in {End}(V_\chi )$ such that $\langle u(v), u'(v')\rangle =s_\chi (h(v), v')$ for all $v, v'\in V_\chi$. The G-invariance of the form implies that h is right multiplication with some $h_\chi (u, u')\in D_\chi$:

$$\begin{aligned} \langle u(v), u'(v')\rangle =s_\chi (vh_\chi (u, u'),v'). \end{aligned}$$

It is then straightforward to verify that $h_\chi$ is $D_\chi$-linear in the first variable and satisfies $h_\chi (u, u')=-h_\chi (u',u)^\dagger$ (so that $h_\chi (u, du')=h_\chi (u',u)d^\dagger$). In other words, it is a skew-hermitian form on the $D_\chi$-module $H[\chi ]$. We shall therefore denote this form by $\langle \; ,\; \rangle _\chi$. It is clearly nondegenerate. We thus find an identification

$$\begin{aligned} {\text {Sp}}(H_\chi )^G\cong {\text {U}}_{D_\chi }(H[\chi ]), \end{aligned}$$

where ${\text {Sp}}(H_\chi )^G$ stands for the G-centralizer in ${\text {Sp}}(H_\chi )$ and ${\text {U}}_{D_\chi } (H[\chi ])$ denotes the group of automorphisms of $H[\chi ]$ as a skew-hermitian $D_\chi$-module (but we write ${\text {U}}(H[\chi ])$ when $D_\chi$ is clear from the context). The group on the left is in a natural manner the group of k-points of a reductive k-algebraic group ${{\mathscr {Sp}}}(H_\chi )^G$ and the group on the right can be regarded as the group of $K_\chi$-points of an algebraic group ${{\mathscr {U}}}(H[\chi ])$ defined over $K_\chi$. In other words, ${{\mathscr {Sp}}}(H_\chi )^G$ is obtained from ${{\mathscr {U}}}(H[\chi ])$ by restriction of scalars (from $K_\chi$ to k).

This enables us to understand what the group of real points of ${{\mathscr {Sp}}}(H_\chi )^G$ is like when $k={{\mathbb {Q}}}$. Given a field embedding $\sigma :K_\chi \rightarrow {{\mathbb {R}}}$, then, as we noted, the ${{\mathbb {R}}}$-algebra $\sigma ^*D_\chi$ is isomorphic to a matrix algebra over ${{\mathbb {R}}}$, ${{\mathbb {C}}}$ or ${{\mathbb {K}}}$. Clearly, $\sigma ^*H[\chi ]$ is a left $\sigma ^*D_\chi$-module and $\langle \; , \, \rangle$ determines a skew-hermitian form $\sigma ^*\langle \; , \, \rangle$ on it. The above identification then yields an isomorphism

$$\begin{aligned} {{\mathscr {Sp}}}(H_\chi )^G({{\mathbb {R}}})\rightarrow \prod _\sigma {\text {U}}_{\sigma ^*D_\chi } (\sigma ^*H[\chi ]), \end{aligned}$$

where the product is over all field embeddings $\sigma : K_\chi \hookrightarrow {{\mathbb {R}}}$.

The factor ${\text {U}}_{\sigma ^*D_\chi } (\sigma ^*H[\chi ])$ can be identified with a unitary group over the classical division algebra $D^\sigma =D_{\chi ^\sigma }$ as follows. Let us recall that we write $V^\sigma$ for $V_{\chi ^\sigma }$ and $I^\sigma$ for ${\text {Hom}}_{{{\mathbb {R}}}G} (V^\sigma , \sigma ^*V)$ and that we have isomorphisms

$$\begin{aligned}&\sigma ^*D_\chi \cong {End}_{D^\sigma }(I^\sigma )^\mathrm {opp}\text { (of algebras)},\\&\quad \quad \sigma ^*H[\chi ]\cong {\text {Hom}}_{D^\sigma } (I^\sigma , H_{{\mathbb {R}}}[\chi ^\sigma ]) \text { (of }\sigma ^*D_\chi -\text {modules)}. \end{aligned}$$

If we apply the preceding to the case $k={{\mathbb {R}}}$, then we see that $H_{{{\mathbb {R}}}, \chi ^\sigma }$ is a nondegenerate subspace of $H_{{\mathbb {R}}}$ for the symplectic form and that the $D^\sigma$-module $H_{{\mathbb {R}}}[\chi ^\sigma ]$ comes with a nondegenerate $D^\sigma$-valued skew-hermitian form. Let us denote the latter by $\langle \; ,\; \rangle ^\sigma$. So if $\phi , \phi '\in \sigma ^*H[\chi ]\cong {\text {Hom}}_{D^\sigma } (I^\sigma , H_{{\mathbb {R}}}[\chi ^\sigma ])$, then a skew-hermitian form on $I^\sigma$ is given by

$$\begin{aligned} (u,u')\mapsto \langle \phi (u) ,\phi ' (u') \rangle ^\sigma \end{aligned}$$

We had already defined a positive definite hermitian form $h^\sigma : I^\sigma \times I^\sigma \rightarrow D^\sigma$. Hence there is a unique $g(\phi , \phi ')\in {End}_{D^\sigma }(I^\sigma )$ such that

$$\begin{aligned} \langle \phi (u) ,\phi ' (u') \rangle ^\sigma = h^\sigma (g(\phi , \phi ')(u), u') \end{aligned}$$

for all $u, u'\in I^\sigma$. It is straightforward to verify that g defines a ${End}_{D^\sigma }(I^\sigma )^\mathrm {opp}$-valued skew-hermitian form on ${\text {Hom}}_{D^\sigma } (I^\sigma , H_{{\mathbb {R}}}[\chi ^\sigma ])$. Via the isomorphisms above this is up to a scalar the $\sigma ^*D_\chi$-valued skew-hermitian form on $\sigma ^*H[\chi ]$. We thus find:

Corollary 2.1

We have a natural identification ${\text {U}}_{\sigma ^*D_\chi } (\sigma ^*H[\chi ])\cong {\text {U}}_{D_{\chi ^\sigma }}(H_{{\mathbb {R}}}[\chi ^\sigma ])$ and hence an identification ${{\mathscr {Sp}}}(H_\chi )^G({{\mathbb {R}}})\cong \prod _\sigma {\text {U}}_{D_{\chi ^\sigma }}(H_{{\mathbb {R}}}[\chi ^\sigma ])$.

So if r denotes the rank of $H_{{\mathbb {R}}}[\chi ^\sigma ]$ as a $D_{\chi ^\sigma }$-module, then

• real $H_{{\mathbb {R}}}[\chi ^\sigma ]$ is a real symplectic vector space so that ${\text {U}}(\sigma ^*H[\chi ])$ is isomorphic to a symplectic group in r real variables.

• quaternion $H_{{\mathbb {R}}}[\chi ^\sigma ]$ is a skew-hermitian module over $D_{\chi ^\sigma }\cong {{\mathbb {K}}}$ and ${\text {U}}(\sigma ^*H[\chi ])$ is isomorphic to a unitary group in r quaternionic variables.

• complex $H_{{\mathbb {R}}}[\chi ^\sigma ]$ is a skew-hermitian module $D_{\chi ^\sigma }=L_{\chi ^\sigma }\cong {{\mathbb {C}}}$ and ${\text {U}}(\sigma ^*H[\chi ])$ is isomorphic to a unitary group in r complex variables.

4 Virtual linear representations of the mapping class group

4.1 Hodge group and connected monodromy group

We recall that a weight m Hodge structure on a ${{\mathbb {Q}}}$-vector space W amounts to giving an action of the circle group ${\text {U}}(1)$ on $W_{{\mathbb {R}}}$ so that $W_{{\mathbb {C}}}=\oplus _{p+q=m}W^{p,q}$ can be regarded as the eigenspace decomposition of an ${\text {U}}(1)$-action, where $\lambda \in {\text {U}}(1)$ acts as multiplication by $\lambda ^{q-p}$. The Hodge group of this Hodge structure is defined to be the smallest ${{\mathbb {Q}}}$-subgroup $\mathscr {H}\! g(W)$ of ${\text {GL}}(W)$ whose group of real points contains the image of this action. It is a connected reductive group. Note that W is indecomposable as a Hodge structure, if and only if $\mathscr {H}\! g(W)$ acts ${{\mathbb {Q}}}$-irreducibly on W. The case of interest here is when W is polarized and has (1, 0) and (0, 1) as its only Hodge bidegrees of W. It is known that the simple factors of $\mathscr {H}\! g(W)$ are then all of classical type, i.e., forms of a symplectic, orthogonal or special linear group.

Now let ${{\mathbb {W}}}$ be a variation of polarizable Hodge structure ${{\mathbb {W}}}$ over a connected complex manifold Y with Hodge bidegrees (1, 0) and (0, 1) (our definition assumes that there exists a sublattice ${{\mathbb {W}}}_{{\mathbb {Z}}}\subset {{\mathbb {W}}}$, but this is a property and not part of the data). Since the Hodge group gets smaller as the Hodge structure becomes more special and is locally constant on a dense subset, we have a locally constant family of subgroups $\{\mathscr {H}\! g({{\mathbb {W}}})_y\subseteq {\text {GL}}({{\mathbb {W}}}_y)\}_{y\in Y}$ characterized by the property that $\mathscr {H}\! g({{\mathbb {W}}}_y)\subseteq \mathscr {H}\! g({{\mathbb {W}}})_y$ for all $y\in Y$ and with equality for some y. This is what is called the generic Hodge group (of ${{\mathbb {W}}}$ at y). Deligne’s semi-simplicity theorem ([8], Thm. 4.2.6, Cor. 4.2.8 and Cor. 4.2.9) tells us that ${{\mathbb {W}}}$ is semi-simple as a variation of Hodge structure and that its complexification ${{\mathbb {W}}}_{{\mathbb {C}}}$ is semisimple as a local system. The last property implies that the identity component of the Zariski closure of the monodromy group of ${{\mathbb {W}}}$ in ${\text {GL}}({{\mathbb {W}}}_y)$—which we shall denote by ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ and refer to as the connected monodromy group—is semisimple as well. Note that ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ does not change under base change by an unramified finite cover of Y.

The endomorphism algebra of the local system underlying ${{\mathbb {W}}}$, ${End}({{\mathbb {W}}})$, has a natural Hodge structure with Hodge bidegrees $(1,-1)$, (0, 0) and $(-1, 0)$. Note that the subalgebra ${End}_{HS}({{\mathbb {W}}})$ of endomorphisms preserving the Hodge structure is ${End}({{\mathbb {W}}})^{(0,0)}({{\mathbb {Q}}})= {End}({{\mathbb {W}}}_{{\mathbb {C}}})^{(0,0)}\cap {End}({{\mathbb {W}}})$. If ${{\mathbb {W}}}$ is indecomposable as a VHS, this will be a division algebra.

Taking into account the results quoted above, then the essential part of the following lemma is second part, which is due to Masa–Hiko Saito [20] (see also the discussion in section (1.5) of [18]).

Lemma 3.1

Let ${{\mathbb {W}}}$ be a polarized variation of Hodge structure over a connected complex manifold Y with Hodge bidegrees (1, 0) and (0, 1). Suppose ${{\mathbb {W}}}$ is indecomposable and has infinite monodromy (or equivalently, is not isotrivial). Then:

• (i) ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ is a product of ${{\mathbb {Q}}}$-simple factors of the derived Hodge group ${\mathscr {DH}}\!g({{\mathbb {W}}})_y$, and its representation on ${{\mathbb {W}}}_y$ is isotypical. The group $\mathscr {H}\! g({{\mathbb {W}}})_y$ acts on ${End}({{\mathbb {W}}})$ via $\mathscr {H}\! g({{\mathbb {W}}})_y/{{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ and the latter’s action on ${End}({{\mathbb {W}}})$ has as its kernel a central subgroup.

• (ii) The variation of Hodge structure ${{\mathbb {W}}}$ is rigid in the sense that ${End}({{\mathbb {W}}})$ is of type (0, 0) if and only if ${\mathscr {DH}}\!g({{\mathbb {W}}})_y={{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$.

• (iii) When ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ has positive ${{\mathbb {Q}}}$-rank, then ${\mathscr {DH}}\!g({{\mathbb {W}}})_y={{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ and this group is ${{\mathbb {Q}}}$-almost simple. In particular, ${{\mathbb {W}}}$ is rigid.

Proof

We merely show how this follows from the literature. We may assume that ${{\mathbb {W}}}_y$ is indecomposable as Hodge structure and that $\mathscr {H}\! g({{\mathbb {W}}}_y)=\mathscr {H}\! g({{\mathbb {W}}})_y$. It is a general result of André [1] that ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ is normal in ${\mathscr {DH}}\!g({{\mathbb {W}}})_y$. Since ${\mathscr {DH}}\!g({{\mathbb {W}}})_y$ is a semisimple ${{\mathbb {Q}}}$-algebraic group, ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ will be a product of ${{\mathbb {Q}}}$-simple factors of it. The isotypical decomposition of ${{\mathbb {W}}}_y$ as a representation of ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ will respect the Hodge decomposition on ${{\mathbb {W}}}_y$ and hence must be trivial in the sense that ${{\mathbb {W}}}_y$ will be an isotypical ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$-module. We can state this as follows. The isogeny type of ${{\mathbb {W}}}$ is represented by an irreducible local system ${{\mathbb {V}}}$ of ${{\mathbb {Q}}}$-vector spaces over Y such that ${{\mathbb {W}}}$ is as local system isomorphic a sum of copies of ${{\mathbb {V}}}$. More precisely, $D:={End}({{\mathbb {V}}})$ is a division algebra and if we put $U:={\text {Hom}}({{\mathbb {V}}}, {{\mathbb {W}}})$, then U is in a natural manner a right D-module such that the evident map $U\otimes _D {{\mathbb {V}}}\cong {{\mathbb {W}}}$ is an isomorphism of local systems. This isomorphism also identifies ${End}({{\mathbb {W}}})$ with the matrix algebra ${End}^D(U)$ (the endomorphisms of U as a right D-module). But note that the tensor decomposition ${{\mathbb {W}}}\cong U\otimes _D {{\mathbb {V}}}$ cannot respect the Hodge structure, unless this decomposition is trivial. In particular, U need not have a Hodge structure. On the other hand, ${End}^D(U)\cong {End}({{\mathbb {W}}})$ has one. Since this Hodge structure is invariant under the monodromy, $\mathscr {H}\! g({{\mathbb {W}}})_y$ acts on it through $\mathscr {H}\! g({{\mathbb {W}}})_y/{{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$. It is clear that the kernel of this action is a central subgroup of $\mathscr {H}\! g({{\mathbb {W}}})_y/{{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$.

We now turn to the second assertion. Since Saito does not express his results in terms of the Hodge group, we must make the translation. We address the nontrivial direction: if ${{\mathbb {W}}}$ is rigid, then we must show that every ${{\mathbb {Q}}}$-almost simple factor ${{\mathscr {G}}}$ of $\mathscr {H}\! g({{\mathbb {W}}})_y$ is contained in ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$. Let ${{\mathscr {G}}}'$ be a complementary normal ${{\mathbb {Q}}}$-subgroup of $\mathscr {H}\! g({{\mathbb {W}}})_y$ so that ${{\mathscr {G}}}$ and ${{\mathscr {G}}}'$ commute with each other and have finite intersection. The Hodge structure on ${{\mathbb {W}}}_y$ is given by a morphism ${\text {U}}(1)\rightarrow \mathscr {H}\! g({{\mathbb {W}}})_y$. Denote its composite with the projection $\mathscr {H}\! g({{\mathbb {W}}})_y\rightarrow \mathscr {H}\! g({{\mathbb {W}}})_y/{{\mathscr {G}}}'$ by $\rho _y$. This composite is independent of y if and only if ${{\mathscr {G}}}$ is not an irreducible constituent of ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ (we could paraphrase this by saying that variations of Hodge structures manufactured out of ${{\mathbb {W}}}$ which do not involve ${{\mathscr {G}}}'$ will be constant). But as Saito makes clear, this just means that $\rho _y$ describes a (necessarily) nontrivial Hodge structure on ${End}({{\mathbb {W}}})$. As this precludes rigidity, this shows that ${{\mathscr {G}}}\subseteq {{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$.

His argument also shows that when every factor of ${{\mathscr {G}}}({{\mathbb {R}}})$ is noncompact, we have in fact equality: ${{\mathscr {G}}}=\mathscr {H}\! g({{\mathbb {W}}})_y$. (Apply Cor. 6.27 to the family of abelian varieties which deforms the Hodge structures in the ${{\mathscr {G}}}$-direction only: it implies that the group denoted there by $G'_{{\mathbb {R}}}$—in our situation $G'$ is essentially the centralizer of ${{\mathscr {G}}}$ in the the symplectic group of ${{\mathbb {W}}}_y$—has to be compact, so that the defining homomorphism ${\text {U}}(1)\rightarrow {\text {Sp}}({{\mathbb {W}}}_y)$ actually lands in ${{\mathscr {G}}}({{\mathbb {R}}})$.) If ${{\mathscr {M}}}{on}^\circ ({{\mathbb {W}}})_y$ has positive ${{\mathbb {Q}}}$-rank, then it has a ${{\mathbb {Q}}}$-simple factor ${{\mathscr {G}}}$ of positive ${{\mathbb {Q}}}$-rank. Since every factor of such a ${{\mathscr {G}}}({{\mathbb {R}}})$ is noncompact, (iii) also follows. $\square$

4.2 Application to G-curves

We are going to apply the preceding to $H^1(S; {{\mathbb {Q}}})$, where S is as before: a closed connected oriented surface with a faithful G-action. So then for every $\chi \in X({{\mathbb {Q}}}G)$, we have a skew-hermitian $D_\chi$-module $H^1(S; {{\mathbb {Q}}})[\chi ]$.

We equip $V_\chi$ with the trivial Hodge structure of bidegree (0, 0); it is then polarized by $s_\chi$ (but this can be one of many polarizations). When we endow S with a G-invariant complex structure, resulting in a G-curve C, then $H^1(C; {{\mathbb {Q}}})[\chi ]$ comes with a polarizable $D_\chi$-invariant Hodge structure with Hodge bidegrees (1, 0) and (0, 1) and $D_\chi$ acts by endomorphisms of type (0, 0). If the natural algebra homomorphism ${{\mathbb {Q}}}G\rightarrow {End}_{{\mathbb {Q}}}(J(C))\cong {End}_{HS} (H^1(C; {{\mathbb {Q}}}))$ is onto, then the G-isotypical decomposition cannot be refined any further in a G-equivariant manner such that it also respects the Hodge structure: otherwise we would get a projector in ${End}_{{\mathbb {Q}}}(J(C))$ that is not in the image of ${{\mathbb {Q}}}G$. In other words, the endomorphism algebra of the Hodge structure on $H^1(C; {{\mathbb {Q}}})[\chi ]$ is then the division algebra $D_\chi$.

This becomes more interesting in a relative setting. We aim to formulate the consequences in topological language.

Let (S, G) be a closed connected oriented G-surface. Let us denote by ${{\mathscr {M}}}{on}^\circ (S)^G$ the identity component of the Zariski closure of the image of the action of ${\text {Mod}}(S)^G$ on $H^1(S;{{\mathbb {Q}}})$. It respects the isotypical decomposition $H^1(S; {{\mathbb {Q}}})\cong \oplus _\chi V_\chi \otimes _{D_\chi } H^1(S; {{\mathbb {Q}}})[\chi ]$ and ${{\mathscr {M}}}{on}^\circ (S)^G$ acts on the $\chi$-summand via $H^1(S; {{\mathbb {Q}}})[\chi ]$. We denote the image of this action by ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$. A G-invariant conformal structure on S yields a G-curve C. The Hodge group of $H^1(C; {{\mathbb {Q}}})$ is a reductive ${{\mathbb {Q}}}$-subgroup of ${\text {Sp}}(H^1(S; {{\mathbb {Q}}}))^G$. Since these conformal structures are (up to G-equivariant isotopy) parameterized by the (connected) complex manifold ${\text {Teich}}(S)^G={\text {Teich}}(S_G)$, the preceding discussion shows that for a very general choice of a G-invariant conformal structure, this Hodge group is in fact independent of that choice. We therefore denote it by $\mathscr {H}\! g(G,S)$. This group respects the isotypical decomposition and like ${{\mathscr {M}}}{on}^\circ (S)^G$, it acts on the $\chi$-summand via $H^1(S; {{\mathbb {Q}}})[\chi ]$. We denote the image of this action by $\mathscr {H}\! g(S)[\chi ]$.

Our main application is Theorem 3.2 below. We have already shown that its hypotheses are fulfilled when S/G has genus at least 3 or when we are in the situation of Theorem 1.11 (where G contains a subgroup N of index 2 which acts freely on S such that $S_N$ is of genus $\ge 4$ and G/N acts on $S_N$ as a hyperelliptic involution).

Theorem 3.2

Let (S, G) be a closed connected oriented G-surface. Assume that for a very general G-invariant conformal structure on S, the resulting G-curve C has the property that ${{\mathbb {Q}}}G$ maps onto ${End}_{HS}(H^1(C; {{\mathbb {Q}}}))$. Then:

• (i) The centralizer of $\mathscr {H}\! g(S)[\chi ]$ in ${End}(H^1(S; {{\mathbb {Q}}})[\chi ])$ is $D_\chi$.

• (ii) ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is a normal subgroup of the derived group ${\mathscr {DH}}\!g(S)[\chi ]$ of $\mathscr {H}\! g(S)[\chi ]$ and the quotient $\mathscr {H}\! g(S)[\chi ]/{{\mathscr {M}}}{on}^\circ (S)[\chi ]$ acts on ${End}_{{{\mathscr {M}}}{on}^\circ (S)[\chi ]}(H^1(S; {{\mathbb {Q}}})[\chi ])$ with finite central kernel.

• (iii) The group $\mathscr {H}\! g(S)[\chi ]$ (resp. ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$) acts ${{\mathbb {Q}}}$-irreducibly (resp. ${{\mathbb {Q}}}$-isotypically) on the $D_\chi$-module $H^1(S;{{\mathbb {Q}}})[\chi ]$.

• (iv) If ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ has positive ${{\mathbb {Q}}}$-rank, then ${{\mathscr {M}}}{on}^\circ (S)[\chi ]={\mathscr {DH}}\!g(S)[\chi ]$ and this group is ${{\mathbb {Q}}}$-almost simple.

• (v) If ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is trivial, then ${\text {Mod}}(S)^G$ acts through a finite group on $H^1(S; {{\mathbb {Q}}})_\chi$ and all G-invariant conformal structures on S define the same indecomposable Hodge structure on $H^1(S;{{\mathbb {Q}}})[\chi ]$.

Remark 3.3

It is not known whether the case (v) of finite monodromy occurs at all. Putman and Wieland conjecture (1.2 of [19]) that it does not (when the genus of S/G is $\ge 2$), and prove that this non-occurrence is essentially equivalent to a conjecture of Ivanov, which states that the first Betti number of a finite index subgroup of a mapping class group of genus $\ge 3$ is trivial.

(Proof of Theorem 3.2) In order to prove the theorem for a cofinite subgroup $\Gamma \subset {\text {Mod}}(S)^G$, there is no loss of generality in passing to a smaller one and so we may assume that $\Gamma \subset {\text {Mod}}(S)^G$ is a normal subgroup which acts trivially on $H^1(S, {{\mathbb {Z}}}/3)$. By a classical observation of Serre, $\Gamma$ is then torsion free, so if we divide out ${\text {Univ}}(S,G)\rightarrow {\text {Teich}}(S)^G$ by the freely acting $\Gamma$ we obtain a G-curve $f: {{\mathscr {C}}}\rightarrow Y$, where Y has the structure of a smooth quasi-projective variety étale over ${{\mathscr {M}}}^\phi$. For $y\in Y$, we have an isomorphism $\pi _1(Y,y)\cong \Gamma$, unique up to inner automorphism and via such an isomorphism, the $\pi _1(Y,y)\times G$-action on $H^1(C_y)$ is equivalent to the $\Gamma \times G$-action on $H^1(S)$. We are now in a situation in which we can apply Lemma 3.1 to ${{\mathbb {W}}}=R^1f_*{{\mathbb {Q}}}[\chi ]$ and all the assertions then follow. $\square$

Questions 3.4

It is natural to ask whether in the case of infinite monodromy (under the hypotheses of Theorem 3.2) which of the inclusions

$$\begin{aligned} {{\mathscr {M}}}{on}^\circ (S)[\chi ]\subseteq {\mathscr {DH}}\!g(C)[\chi ]\subseteq \mathscr {D\!U}(H^1(S, {{\mathbb {Q}}})[\chi ]) \end{aligned}$$

is strict. Theorem 3.2-iv tells us that the first inclusion is always an isomorphism when ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ has positive ${{\mathbb {Q}}}$-rank, but we do not know whether that is always the case; see also Corollary 3.5 below. Nor do we know whether the image of the ${\text {Mod}}(S)^G$-action on $H^1(S;{{\mathbb {Q}}})[\chi ]$ is an arithmetic subgroup. When G is abelian and acts freely on S and is such that $g(S/G)\ge 2$, then it was verified in [15] that the answer is yes; the essential case is then G cyclic, which means that $S\rightarrow S/G$ is trivial over a connected subsurface of genus $g(S_G)-1$. Grünewald, Larsen, Lubotzky and Malestein [10] generalized this to the case when $S\rightarrow S/G$ is trivial over a connected subsurface of genus one that is the complement of $g(S/G)-1$ pairwise disjoint loops in S/G (with G still acting freely and $g(S/G)\ge 2$). This implies that G has $\le g(S/G)-1$ generators, but can be very much non-abelian. Venkataramana [23] proved that the answer is also yes when $g(S/G)=0$, G abelian and such that the number of irregular orbits does not exceed half the order of G. In none of these cases a nontrivial $H^1(S, {{\mathbb {Q}}})[\chi ]$ occurs with finite monodromy group.

Another question is whether these representations are independent, in the sense that the natural map ${{\mathscr {M}}}{on}^\circ (S)^G\rightarrow \prod _\chi {{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is an isomorphism (again assuming that $g(S/G)\ge 3$). The results in [15] imply that this is so when $B=\emptyset$ and G is abelian.

4.3 Virtual representations

Let us now explain the title of this section. For this we need to elaborate a bit more on the structure of the D-M stack ${{\mathscr {M}}}^\phi$. Choose a base point $y\in S_G$ away from the set of inertia points. The data of $S_G$ as the G-orbifold quotient of S, determines and is determined by an epimorphism $p:\pi _1(S_G, y)\rightarrow G$ up to an inner automorphism of G. Here $\pi _1(S_G, y)$ is to be understood as the fundamental group of the orbifold $S_G$. The kernel K of p is of course intrinsic to this situation, but it does not quite determine the topological type $\phi$ up to a G-isomorphism, because it is also necessary to specify an ${\text {Inn}}(G)$-orbit of isomorphisms $\pi _1(S_G, y)/K\cong G$. However, the relevant stabilizer subgroup in ${\text {Mod}}(S_G)$, that is, the group of mapping classes of ${\text {Mod}}(S_G)$ which, when viewed as a group of outer automorphisms of $\pi _1(S_G, y)$, preserve K and change p by an inner automorphism of G, only depends on K, and can therefore be denoted by ${\text {Mod}}(S_G)[K]$. It consists of the mapping classes which lift to S and commute with the G-action. It is of finite index in ${\text {Mod}}(S_G)$. We thus obtain an étale covering ${{\mathscr {M}}}(S_G)[K]\rightarrow {{\mathscr {M}}}(S_G)$ and a short exact sequence:

$$\begin{aligned} 1\rightarrow Z(G)\rightarrow {\text {Mod}}(S)^G\rightarrow {\text {Mod}}(S_G)[K]\rightarrow 1. \end{aligned}$$

(This exact sequence expresses the fact that the smooth D–M stack ${{\mathscr {M}}}^\phi$ is a Z(G)-gerbe over ${{\mathscr {M}}}(S)[K]$.) If ${\overline{g}}$ is the genus of $S_G$ and ${\overline{n}}$ the cardinality of the inertia set, then ${\text {Mod}}(S_G)$ contains ${\text {Mod}}_{{\overline{g}}, {\overline{n}}}$ as a subgroup of finite index. Since Z(G) is finite and the group ${\text {Mod}}(S_G)[K]$ is of finite index in ${\text {Mod}}(S_G)$, the group ${\text {Mod}}(S)^G$ has a subgroup of finite index which is also of finite index in ${\text {Mod}}_{{\overline{g}}, {\overline{n}}}$. So in that sense, the monodromy representation of ${\text {Mod}}(S)^G$ appearing in Theorem 3.2 can be regarded as a virtual representation of ${\text {Mod}}_{{\overline{g}}, {\overline{n}}}$.

4.4 The Dehn groups

Given an oriented surface $\Sigma$, we shall denote by ${\mathscr {X}}(\Sigma )$ the vertices of the curve complex of $\Sigma$, that is, the set of isotopy classes of embedded circles in $\Sigma$ which do not bound a disk or a once punctured disk. We recall that every $\alpha \in {\mathscr {X}}(\Sigma )$ defines a Dehn twist $\tau _\alpha \in {\text {Mod}}(\Sigma )$ and carries, after an orientation, a homology class $[\alpha ]\in H_1(\Sigma )$ (which is therefore only given up to sign). The action of $\tau _\alpha$ on $H_1(\Sigma )$ is given by the transvection $T_{[\alpha ]}: c\in H_1(\Sigma )\mapsto c+ \langle c ,[\alpha ]\rangle [\alpha ]$, where $\langle \; , \; \rangle$ is the intersection form on $H_1(\Sigma )$. Here the choice of orientation of $\alpha$ clearly doesn’t matter.

Let us write $S_G^\circ$ for $S_G{\smallsetminus } B$, where $B\subset S_G$ denotes the set of inertia points. We observe that every ${\bar{\alpha }}\in {\mathscr {X}}(S_G^\circ )$ determines a G-orbit $A=A_{{\bar{\alpha }}}\subset {\mathscr {X}}(S)$, namely the set of connected components of its preimage of a representative of ${\bar{\alpha }}$. We denote by ${\mathscr {X}}(S, G)\subset \mathscr {X}(S)$ the union of the G-orbits A so obtained. It is clear that a G-orbit A as above spans an isotropic subspace for the intersection pairing. The Dehn twists $\{\tau _{\alpha }\}_{\alpha \in A}$ commute pairwise and their product $\tau _{A}$ is an element of ${\text {Mod}}(S)^G$ which canonically lifts $\tau _{{\bar{\alpha }}}^{m_{{\bar{\alpha }}}}\in {\text {Mod}}(S_G)$, where $m_{{\bar{\alpha }}}$ denotes the common degree of the members of A over ${\bar{\alpha }}$. Then $T_{A}:=\tau _{A*}\in {\text {Sp}}(H_1(S))^G$ is given by

$$\begin{aligned} T_{A}(x)=x+\sum _{\alpha \in A} \langle x,[\alpha ]\rangle [\alpha ] \end{aligned}$$

and hence generates the one-parameter subgroup

$$\begin{aligned} \mathscr {T}_{A}:{\mathbb {G}}_a\rightarrow {{\mathscr {Sp}}}(H)^G, \quad \mathscr {T}_{A}(t) (x)= x+ \sum _{\alpha \in A} \langle x,[\alpha ]\rangle t[\alpha ]. \end{aligned}$$

It is clearly contained in the connected monodromy group ${{\mathscr {M}}}{on}^\circ (S)^G$. Let us observe that $\langle x,\mathscr {T}_{A}(t)(x)\rangle = \langle x, x\rangle +t\sum _{\alpha \in A} \langle x, [\tilde{\alpha }]\rangle ^2$. So if x fixed under $\mathscr {T}_{A}$, then that $\langle x, [\tilde{\alpha }]\rangle =0$ for every $\alpha \in A$. If on the other hand $\langle x, [\alpha ]\rangle \not =0$ for some $\alpha \in A$, then $\sum _{\alpha \in A}\langle x, [\alpha ]\rangle ^2>0$ and we see that $n\in {{\mathbb {Z}}}\mapsto \langle x, T_{A}^n(x)\rangle =\langle x, \mathscr {T}_{A}(n)(x)\rangle$ is strictly increasing so that $T_A^{{\mathbb {Z}}}x$ is infinite.

Corollary 3.5

Assume that we are in the situation of Theorem 3.2 and that $\chi \in X({{\mathbb {Q}}}G)$ is such that ${\mathscr {X}}(S,G)$ has a nonzero image in $H_1(S;{{\mathbb {Q}}})_\chi$. Then this image spans $H_1(S;{{\mathbb {Q}}})_\chi$ over ${{\mathbb {Q}}}$. Moreover, ${\text {Mod}}(S)^G$ acts irreducibly on $H_1(S;{{\mathbb {Q}}})[\chi ]$ and the identity component of the Zariski closure of this representation, ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$, is ${{\mathbb {Q}}}$-almost simple.

Proof

Let $A\subset \mathscr {X}(S,G)$ be a G-orbit whose image in $H_1(S;{{\mathbb {Q}}})_\chi$ is nontrivial. This implies that there exists an $x\in H_1(S;{{\mathbb {Q}}})_\chi$ such that $\langle x, \alpha \rangle \not =0$ for some $\alpha \in A$. So by the observation above, the one-parameter subgroup $\mathscr {T}_{A}$ of ${{\mathscr {M}}}{on}^\circ (S)^G$ has a nontrivial image in ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$. Hence ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ will have positive ${{\mathbb {Q}}}$-rank. It then follows from Theorem 3.2-iv that ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is ${{\mathbb {Q}}}$-almost simple and acts irreducibly on $H_1(S;{{\mathbb {Q}}})[\chi ]$. It then also follows that $G\times {\text {Mod}}(S)^G$ acts irreducibly on $H_1(S;{{\mathbb {Q}}})_\chi$. Since the ${{\mathbb {Q}}}G$-submodule of $H_1(S;{{\mathbb {Q}}})_\chi$ spanned by the image of ${\mathscr {X}}(S,G)$ is $G\times {\text {Mod}}(S)^G$-invariant, it must be all of $H_1(S;{{\mathbb {Q}}})_\chi$. $\square$

Remark 3.6

Suppose that $S_G\smallsetminus B$ contains an embedded circle $\gamma$ over which the covering is trivial and is such that its preimage $\tilde{\gamma }$ in S has the property that $S\smallsetminus \tilde{\gamma }$ is still connected. This implies that that ${{\mathbb {Z}}}G\cong H_1(\tilde{\gamma })\rightarrow H_1(S)$ is an injection of ${{\mathbb {Z}}}G$-modules. Hence, if in addition $g(S_G)\ge 3$, then the hypotheses of Corollary 3.5 are satisfied for all $\chi$, so that for every $\chi \in X({{\mathbb {Q}}}G)$, ${\text {Mod}}(S)^G$ acts ${{\mathbb {Q}}}$-irreducibly on $H_1(S;{{\mathbb {Q}}})[\chi ]$ and ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$ is ${{\mathbb {Q}}}$-almost simple.

We close the paper with asking a question and proposing a conjecture.

Conjecture 3.7

In the situation of Corollary 3.5, the group generated by the isotropic transvections $\{T_{A}[\chi ]\}_{A\in G\backslash {\mathscr {X}}(S,G)}$ maps onto an arithmetic subgroup of ${{\mathscr {M}}}{on}^\circ (S)[\chi ]$.

Questions 3.8

Malestein and Putman [17] have recently shown that it can happen that $\mathscr {X}(S, G)$ does not span $H_1(S;{{\mathbb {Q}}})$. It then follows from Corollary 3.5 that for some $\chi \in X({{\mathbb {Q}}}G)$, the image of $\mathscr {X}(S, G)$ in $H_1(S;{{\mathbb {Q}}})[\chi ]$ is trivial. So then any Dehn twist in $S_G\smallsetminus B$ admits a power which lifts to a multi-Dehn twist in ${\text {Mod}}(S)^G$ that acts trivially on $H_1(S;{{\mathbb {Q}}})_\chi$. Since a Dehn twist in $S_G\smallsetminus B$ represents a loop in the moduli space of curves of orbifold type $S_G$, there exists then an étale cover $Y\rightarrow {{\mathscr {M}}}^\phi$ of our moduli space ${{\mathscr {M}}}^\phi$ with the property that for the resulting pull-back $f:{{\mathscr {C}}}_Y\rightarrow Y$ (a family of G-curves), the local variation of Hodge structure $R^1f_*{{\mathbb {Q}}}[\chi ]$ extends across the Deligne-Mumford compactification of Y. Since the kind of representations of ${\text {Mod}}(S_G)$ which come from the theory of conformal blocks (the so-called quantum representations) also have this property, one of us [16] has raised the question whether in such a case the (virtual) representation of ${\text {Mod}}(S_G)$ on $H_1(S; {{\mathbb {Q}}})[\chi ]$ is of this type. (See also [12].) It is likely that the Zariski closure of such a representation is anisotropic (i.e., has ${{\mathbb {Q}}}$-rank zero).