The nilpotent cone in the Mukai system of rank two and genus two

Abstract

We study the nilpotent cone in the Mukai system of rank two and genus two. We compute the degrees and multiplicities of its irreducible components and describe their cohomology classes.

1 Introduction

Let (S, H) be a polarized K3 surface of genus g and fix two coprime integers $n\ge 1$ and s. The moduli space $M=M_H(v)$ of H-Gieseker stable coherent sheaves with Mukai vector $v=(0,nH,s)$ is a smooth Hyperkähler variety of dimension $2(n^2(g-1)+1)$. A point in M corresponds to a stable sheaf $\mathcal {E}$ on S such that $\mathcal {E}$ is pure of dimension one with support in the linear system |nH|. Taking the (Fitting) support defines a Lagrangian fibration

$$\begin{aligned} f :M \longrightarrow |nH| \cong \mathbb {P}^{n^2(g-1)+1},\ [\mathcal {E}] \mapsto {{\,\mathrm{Supp}\,}}(\mathcal {E}) \end{aligned}$$

known as the Mukai system [5, 22]. Over a general point in |nH| which corresponds to a smooth curve $D \subset S$ the fibers of f are abelian varieties isomorphic to ${{\,\mathrm{Pic}\,}}^{\delta }(D)$, where $\delta = s-n^2(1-g)$. So, M can also be viewed as a relative compactified Jacobian associated to the universal curve $\mathcal {C} \rightarrow |nH|$.

The Mukai system is of special interest because of its relation to the classical and widely studied Hitchin system, see [14] for a survey. Let C be a smooth curve of genus g. A Higgs bundle on C is a pair $(\mathcal {E},\phi )$ consisting of a vector bundle $\mathcal {E}$ on C and a morphism $\phi :\mathcal {E}\rightarrow \mathcal {E}\otimes \omega _C$, called Higgs field. The moduli space $M_\mathrm{Higgs}(n,d)$ of stable Higgs bundles of rank n and degree d is a smooth and quasi-projective symplectic variety. Sending $(\mathcal {E},\phi )$ to the coefficients of its characteristic polynomial $\chi (\phi )$ defines a proper Lagrangian fibration

$$\begin{aligned} \chi :M_\mathrm{Higgs}(n,d) \longrightarrow \bigoplus _{i = 1}^{n}H^0(C,\omega _C^i). \end{aligned}$$

It is equivariant with respect to the $\mathbb {C}^*$-action that is given by scaling the Higgs field on $M_\mathrm{Higgs}(n,d)$ and by multiplication with $t^i$ in the corresponding summand on the base. As a corollary the topology of $M_\mathrm{Higgs}(n,d)$ is controlled by the fiber over the origin. This fiber

$$\begin{aligned} N :=\chi ^{-1}(0) = \{(\mathcal {E},\phi )\in M_\mathrm{Higgs}(n,d)\mid \phi \ \text {is nilpotent} \} \end{aligned}$$

is called the nilpotent cone. In the late ’80s Beauville, Narasimhan, and Ramanan discovered a beautiful interpretation of the space of Higgs bundles [4]. They showed that a Higgs bundle $(\mathcal {E},\phi )$ with characteristic polynomial s corresponds to a pure sheaf of rank one on a so called spectral curve $C_s \subset T^*C$ inside the cotangent bundle of C. The curve $C_s$ is defined in terms of $s =\chi (\phi )$ and is linearly equivalent to nC, the n-th order thickening of the zero section $C \subset T^*C$. This idea was taken up by Donagi, Ein, and Lazarsfeld in [9]: The space $M_\mathrm{Higgs}(n,d)$ appears as a moduli space of stable sheaves on $T^*C$ that are supported on curves in the linear system |nC|. Consequently, $M_\mathrm{Higgs}(n,d)$ has a natural compactification $\overline{M}_\mathrm{Higgs}(n,d)$ given by a moduli space of sheaves on the projective surface $S_0 = \mathbb {P}(\omega _C\oplus \mathcal {O}_C)$ with respect to the polarization $H_0=\mathcal {O}_{S_0}(C)$. The Hitchin map extends to $\overline{M}_\mathrm{Higgs}(n,d) \rightarrow |nH_0| \cong \mathbb {P}(\oplus _{i=0}^nH^0(\omega _C^i))$ and is nothing but the support map; the nilpotent cone is the fiber over the point $nC \in |nH_0|$. However, $\overline{M}_\mathrm{Higgs}(n,d)$ cannot admit a symplectic structure as it is covered by rational curves. At this point the Mukai system enters the picture. If S is a K3 surface that contains the curve C as a hyperplane section, one can degenerate (S, H) to $(S_0,H_0)$ and consequently the Mukai system $M_H(v) \rightarrow |nH|$ with $v=(0,nH,d+n(1-g))$ degenerates to the compactified Hitchin system [9, §1]. From our perspective, this is a powerful approach to studying the Hitchin system. For instance, in a recent paper [7], de Cataldo, Maulik and Shen prove the P=W conjecture for $g=2$ by means of the corresponding specialization map on cohomology.

In this note, we study the geometry of the nilpotent cone in the Mukai system, which is defined in parallel to the Hitchin system

$$\begin{aligned} N_C :=f^{-1}(nC), \end{aligned}$$

for some curve $C \in |H|$. Alternatively, one could say that we study the most singular fiber type, see (3.1). We will fix the invariants $n=2$ and $g=2$ and the Mukai vector $v=(0,2H,-1)$. In this case and if C is irreducible, the nilpotent cone has two irreducible components

$$\begin{aligned} (N_C)_\mathrm{red} = N_0 \cup N_1, \end{aligned}$$

where the first component is isomorphic to the moduli space $M_C(2,1)$ of stable vector bundles of rank two and degree one on C and the second component is the closure of $N_C\setminus N_0$. Both components are Lagrangian subvarieties of $M=M_H(v)$. If C is smooth, then $N_0$ is smooth and the singularities of $N_1$ are contained in $N_0 \cap N_1$ (each understood with their reduced structure). However, both components occur with multiplicities.

Our first result is the computation of the multiplicities of the components as well as their degrees. Here, the degree is meant with respect to a naturally defined distinguished ample class $u_1 \in H^2(M,\mathbb {Z})$, see Definition 4.7.

Theorem 1.1

Let $C \in |H|$ be an irreducible curve. The degrees of the two components of the nilpotent cone $N_C$ are given by

$$\begin{aligned} \begin{array}{lcr} \deg _{u_1}N_0 = 5 \cdot 2^9&\text {and}&\deg _{u_1}N_1 = 5^2\cdot 2^{11} \end{array} \end{aligned}$$

and their multiplicities are

$$\begin{aligned} \begin{array}{lcr} {{\,\mathrm{mult}\,}}_{N_C}N_0 = 2^3&\text {and}&{{\,\mathrm{mult}\,}}_{N_C}N_1 = 2. \end{array} \end{aligned}$$

Moreover, any fiber F of the Mukai system has degree $5 \cdot 3 \cdot 2^{13}$.

As the smooth locus of every component with its reduced structure deforms from the Mukai to the Hitchin system, the multiplicities and degrees must coincide. Here, indeed, the same multiplicities can be found in [23, Propositions 34 and 35] and [15, Proposition 6], whereas, up to our knowledge, the degrees have not been determined in the literature. In our case, the degrees determine the multiplicities.

Our second result is a description of the classes $[N_0]$ and $[N_1] \in H^{10}(M,\mathbb {Z})$. The projective moduli spaces of stable sheaves on K3 surfaces are known to be deformation equivalent to Hilbert schemes of points. In our case, M is actually birational to $S^{[5]}$ [6, Lemma 3.2.7]. In particular, there is an isomorphism $H^*(M,\mathbb {Z}) \cong H^*(S^{[5]},\mathbb {Z})$. The cohomology ring of $S^{[5]}$ is well understood, e.g. [19, §4] and the references therein. Recall that for any Hyperkähler variety X of dimension 2n there is an embedding $S^iH^2(X,\mathbb {Q}) \hookrightarrow H^{2i}(X,\mathbb {Q})$ for all $i\le n$ [24, Theorem 1.7].

Theorem 1.2

The classes $[N_0]$ and $[N_1] \in H^{10}(M,\mathbb {Q})$ are linearly independent and span a totally isotropic subspace of $H^{10}(M,\mathbb {Q})$ with respect to the intersection pairing. They are given by

$$\begin{aligned}{}[N_0] = \frac{1}{48} [F] + \beta \ \text {and}\ [N_1] = \frac{5}{12} [F] - 4 \beta , \end{aligned}$$

where [F] is the class of a general fiber of the Mukai system and $0 \ne \beta \in (S^5H^2(M,\mathbb {Q}))^\bot$ satisfies $\beta ^2 = 0$. As $\deg _{u_1}\beta = 0$, the class $\beta$ is not effective.

1.1 Outline

In Sect. 2 we introduce the Mukai system. In Sect. 3 we reduce the study of $N_C$ to the case of a smooth curve C. We describe the irreducible components of the nilpotent cone following [9, §3], where it is shown that any point $[\mathcal {E}]\in N_C\setminus N_0$ fits into an extension of the form

$$\begin{aligned} 0 \rightarrow \mathcal {L}(x)\otimes \omega _C^{-1} \longrightarrow \mathcal {E}\longrightarrow \mathcal {L}\rightarrow 0 , \end{aligned}$$

where $\mathcal {L}\in {{\,\mathrm{Pic}\,}}^1(C)$ is a line bundle and $x \in C$ a point. We specify a space $W \rightarrow {{{\,\mathrm{Pic}\,}}^1(C)} \times C$ parameterizing such extensions, and a compactification $\overline{W}$ of W that comes with a birational map $\nu :\overline{W}\rightarrow N_1$. In the Hitchin case, this idea originates from [23].

In Sect. 4 we prove Theorem 1.1. The proof relies on the functorial properties of the defintion of $u_1$ via the determinant line bundle construction, see Sect. 4.1. It allows us to relate $u_1\!\left| _{F}\right.$ and $u_1\!\left| _{N_0}\right.$ with the (generalized) theta divisor on $F = f^{-1}(D) \cong {{\,\mathrm{Pic}\,}}^3(D)$ for $D \in |nC|$ smooth and $M_C(2,1)$, respectively, see Propositions 4.8 and 4.10. For $[N_1]$ the degree computation is achieved by determining $\nu ^*u_1 \in H^2(\overline{W},\mathbb {Z})$. Finally, the multiplicities are infered from knowing the degrees. The last Sect. 5 is devoted to the proof of Theorem 1.2. It uses our previous results.

1.2 Notation

All schemes are of finite type over $k = \mathbb {C}$. In the entire paper, S is a K3 surface polarized by a primitive, ample class $H \in {{\,\mathrm{NS}\,}}(S)$ with $H^2 = 2g-2$.

2 The Mukai system

In this section, we give a brief recollection on moduli spaces of sheaves on K3 surfaces and define the Mukai system. First recall that the Mukai vector induces an isomorphism

$$\begin{aligned} v :{{\,\mathrm{K}\,}}(S)_\mathrm{num} \xrightarrow \sim H^*_\mathrm{alg}(S,\mathbb {Z}) = H^0(S,\mathbb {Z}) \oplus {{\,\mathrm{NS}\,}}(S) \oplus H^4(S,\mathbb {Z}). \end{aligned}$$

It is given by

$$\begin{aligned} v(\mathcal {E}) :={{\,\mathrm{ch}\,}}(\mathcal {E})\sqrt{{{\,\mathrm{td}\,}}(S)} = ({{\,\mathrm{rk}\,}}(\mathcal {E}),c_1(\mathcal {E}),\chi (\mathcal {E})-{{\,\mathrm{rk}\,}}(\mathcal {E})). \end{aligned}$$

We write $M_{H}(v)$ for the moduli space of pure, H-Gieseker stable sheaves on S with Mukai vector v. If v is primitive and positive and H is v-generic then $M_H(v)$ is an irreducible holomorphic symplectic manifold of dimension $\langle v,v \rangle +2$, which is deformation equivalent to the Hilbert scheme of $\tfrac{1}{2}\langle v,v \rangle +1$ points on S [18, Theorem 10.3.1]. Here, $\langle \ ,\ \rangle$ is the Mukai pairing given by

$$\begin{aligned} \langle (r,c,s),(r',c',s') \rangle = cc' -rs'-r's. \end{aligned}$$

Consider the Mukai vector

$$\begin{aligned} v :=(0,nH,s) \in H^*_\mathrm{alg}(S,\mathbb {Z}), \end{aligned}$$

and assume that v is primitive. A pure sheaf $\mathcal {F}$ of Mukai vector v has one-dimensional support, first Chern class nH and Euler characteristic s. In particular, $\mathcal {F}$ admits a length one resolution by two vector bundles of the same rank r [16, §1.1]. We define the (Fitting) support of $\mathcal {F}$ to be

$$\begin{aligned} {{\,\mathrm{Supp}\,}}(\mathcal {F}) :=V(\det f) \subset S \end{aligned}$$

the vanishing scheme of the induced morphism $\det f= \wedge ^r f :\wedge ^r\mathcal {V}\rightarrow \wedge ^r\tilde{\mathcal {V}}$, for any resolution $0 \rightarrow \mathcal {V}\xrightarrow f \tilde{\mathcal {V}}$ of $\mathcal {F}$ as above. This definition is well-defined, i.e. independent of the chosen resolution [13, Definition 20.4].

Example 2.1

Let $i :C \hookrightarrow S$ be an integral curve and $\mathcal {E}$ a vector bundle of rank n on C. Then

$$\begin{aligned} {{\,\mathrm{Supp}\,}}(i_*\mathcal {E}) = nC \end{aligned}$$

is the n-th order thickening of C in S.

By definition, ${{\,\mathrm{Supp}\,}}(\mathcal {F})$ is linearly equivalent to $c_1(\mathcal {F})$ and ${{\,\mathrm{Supp}\,}}(\mathcal {F})$ contains the usual support defined by the annihilator of $\mathcal {F}$. Moreover, the reduced locus ${{\,\mathrm{Supp}\,}}(\mathcal {F})_\mathrm{red}$ is the set-theoretic support of $\mathcal {F}$. The advantage of the above definition is, that it behaves well in families and thus induces a morphism [20, §2.2]

$$\begin{aligned} f :M_H(v) \longrightarrow |nH| \cong \mathbb {P}^{\tilde{g}},\quad [\mathcal {E}] \mapsto {{\,\mathrm{Supp}\,}}(\mathcal {E}). \end{aligned}$$

Here, $\tilde{g}= n^2(g-1)+1$. Moreover, $M_H(v)$ is irreducible holomorphic symplectic of dimension $n^2H^2+2= 2\tilde{g}$ and hence, by Matsushita’s result [21, Corollary 1] this morphism is a Lagrangian fibration (for an explicit proof see [9, Lemma 1.3]), called the Mukai system (of rank n and genus g).

3 The nilpotent cone for $n=2$ and $g=2$

We now specialize to the case that $n = 2$ and $s = 3-2g$ with $g=2$, i.e. we fix the Mukai vector

$$\begin{aligned} v = (0,2H,-1). \end{aligned}$$

In particular, a stable vector bundle of rank two and degree one on a smooth curve $C \in |H|$ defines a point in $M :=M_H(v)$. We have $\dim M = 8g-6 = 10$ and M is birational to the Hilbert scheme $S^{[5]}$ of five points on S.

Taking (Fitting) supports defines a Lagrangian fibration

$$\begin{aligned} f :M \longrightarrow |2H| \cong \mathbb {P}^{5}. \end{aligned}$$

We have a natural morphism $|H|\times |H| \rightarrow |2H|$. We define $\Sigma \subset |2H|$ as its image and $\Delta \subset \Sigma$ as the image of the diagonal. Then $\Sigma \cong {{\,\mathrm{Sym}\,}}^2|H|$ and $\Delta \cong |H|$. If every curve in |H| is irreducible (e.g. if ${{\,\mathrm{Pic}\,}}(S) = \mathbb {Z}\cdot H$) then $\Delta$ and $\Sigma$ are exactly the loci of non-reduced and non-integral curves, respectively. And in this case we can distinguish three cases following [6, Proposition 3.7.1]:

$$\begin{aligned} f^{-1}(x)\ {\left\{ \begin{array}{ll} \text {is reduced and irreducible} &{} \text {if}\ x\in |2H|\setminus \Sigma \\ \text {is reduced and has two irreducible components} &{} \text {if}\ x\in \Sigma \setminus \Delta \\ \text {has two irreducible components with multiplicities} &{} \text {if}\ x\in \Delta . \end{array}\right. } \end{aligned}$$

(3.1)

In the general case, the list is still valid for for the geometric generic point in the respective subvariety. However, over points that correspond to curves with more irreducible components, one also finds more irreducible components in the fiber [6, Proof of Lemma 3.3.2].

We will study fibers of the third type, namely

$$\begin{aligned} N_C :=f^{-1}(2C), \end{aligned}$$

where $C \in |H|$ is irreducible. In analogy with the Hitchin system, we call $N_C$ nilpotent cone.

For the rest of the paper, we fix a smooth curve $C \in |H|$ and write N instead of $N_C$. We will now identify the irreducible components of N following the ideas of [9].

3.1 Pointwise description of the nilpotent cone $N = N_C$

Let $[\mathcal {E}] \in N$ and consider its restriction $\mathcal {E}\!\left| _C\right.$ to C. There are two cases, either $\mathcal {E}\!\left| _C\right.$ is a stable rank two vector bundle on C or $\mathcal {E}\!\left| _C\right.$ has rank one. By dimension reasons, the sheaves of the first kind contribute an irreducible component $N_0$ of N isomorphic to the moduli space ${{\,\mathrm{M}\,}}_C(2,1)$ of stable rank two and degree one vector bundles on C. In the second case, $\mathcal {E}\!\left| _C\right. \cong \mathcal {L}\oplus \mathcal {O}_D$, where the first factor $\mathcal {L}:=\mathcal {E}\!\left| _C\right. /\text {torsion}$ is a line bundle on C and $D\subset C$ is an effective divisor. We set

$$\begin{aligned} E_1 :=N \setminus N_0 \end{aligned}$$

with reduced structure.

Lemma 3.1

Let $[\mathcal {E}] \in E_1$ and write $\mathcal {E}\!\left| _C\right. = \mathcal {L}\oplus \mathcal {O}_D$. There is a short exact sequence of $\mathcal {O}_S$-modules

$$\begin{aligned} 0 \rightarrow i_*(\mathcal {L}(D) \otimes \omega _C^{-1}) \longrightarrow \mathcal {E}\longrightarrow i_*\mathcal {L}\rightarrow 0. \end{aligned}$$

(3.2)

Moreover, $k :=\deg \mathcal {L}=1$ and $d :=\deg D = 2g-2k-1 =1$.

Proof

Noting that $\omega _C^{-1}$ is the conormal bundle of C in S, it is straightforward to obtain the sequence (3.2). Let us prove the numerical restrictions. From (3.2) we have

$$\begin{aligned} 1 + 2(1-g) = \chi (\mathcal {E}) = \chi (\mathcal {L}(D)\otimes \omega _C^{-1}) + \chi (\mathcal {L}) = 2k+d-(2g-2) + 2(1-g). \end{aligned}$$

Thus $d = 2g -2k -1$ and we find $k \le g-1$. On the other hand, $\mathcal {E}$ is stable and therefore the reduced Hilbert polynomials [16, Definition 1.2.3] of $\mathcal {E}$ and $\mathcal {L}$ satisfy $p(\mathcal {E},t) < p(\mathcal {L},t)$, which amounts to

$$\begin{aligned} \tfrac{1}{2}{(1+2(1-g))} < {k+1-g} \end{aligned}$$

or equivalently $k\ge 1$. $\square$

Remark 3.2

For $n=2$ and arbitrary genus g, one has $\deg \mathcal {L}\in \{1,\ldots ,g-1\}$ and a decomposition into locally closed subsets $N_\mathrm{red}=N_0 \sqcup E_1 \sqcup \ldots \sqcup E_{g-1}$ corresponding to the degree of $\mathcal {L}$. In fact, $N_0$ and the closures of $E_k$ are the irreducible components of N.

We conclude that every point in $E_1$ defines a class in ${{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1}))$ for some point $x \in C$ and some line bundle $\mathcal {L}\in {{\,\mathrm{Pic}\,}}^1(C)$. Conversely, an extension class in ${{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1}))$ defines a point in $E_1$ if and only if its middle term is stable and has the point x as support of its torsion part when restricted to C, i.e. if it is not pushed forward from C. It turns out that all such extensions are stable.

Lemma 3.3

Consider a coherent sheaf $\mathcal {E}$ on S that is given as an extension

$$\begin{aligned} 0 \rightarrow i_*\mathcal {L}\rightarrow \mathcal {E}\rightarrow i_*\mathcal {L}' \rightarrow 0, \end{aligned}$$

where $\mathcal {L}'$ and $\mathcal {L}$ are line bundles on C of degree k and $1-k$, respectively, with $k \ge 1$. Moreover, assume that $\mathcal {E}$ itself does not admit the structure of an $\mathcal {O}_C$-module. Then $\mathcal {E}$ is H-Gieseker stable.

Proof

We have to prove $p(\mathcal {E},t) < p(\mathcal {M},t)$ or, equivalently, $\tfrac{\chi (\mathcal {E})}{c_1(\mathcal {E}).H} < \tfrac{\chi (\mathcal {M})}{c_1(\mathcal {M}).H}$ for every surjection $\mathcal {E}\twoheadrightarrow \mathcal {M}$. We can assume that ${{\,\mathrm{Supp}\,}}(\mathcal {M})=C$ and $\mathcal {M}= i_*\mathcal {M}'$, where $\mathcal {M}'$ is a line bundle on C. Then because $\mathcal {E}\!\left| _C\right. \cong \mathcal {L}' \oplus \mathcal {T}$ for some torsion sheaf $\mathcal {T}$, we find

$$\begin{aligned} {{\,\mathrm{\mathcal {H}{{ om}}}\,}}_{\mathcal {O}_S}(\mathcal {E},i_*\mathcal {M}') \cong {{\,\mathrm{\mathcal {H}{{ om}}}\,}}_{\mathcal {O}_C}(\mathcal {E}\!\left| _C\right. ,\mathcal {M}') \cong {{\,\mathrm{\mathcal {H}{{ om}}}\,}}_{\mathcal {O}_C}(\mathcal {L}',\mathcal {M}') \end{aligned}$$

and thus $i_*\mathcal {L}' \xrightarrow \sim \mathcal {M}$.

Corollary 3.4

The closed points of $E_1$ are in bijection with the following set

$$\begin{aligned} \bigsqcup \limits _{{\mathop {x \in C}\limits ^{\mathcal {L}\in {{\,\mathrm{Pic}\,}}^1(C)}}} \mathbb {P}({{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1})))\setminus \mathbb {P}({{\,\mathrm{Ext}\,}}^1_C(\mathcal {L},\mathcal {L}(x)\otimes \omega _C^{-1})), \end{aligned}$$

i.e. with extension classes $[v] \in \mathbb {P}({{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1})))$ such that v is not pushed forward from C. Here, $\mathcal {L}$ varies over all line bundles on C with $\deg \mathcal {L}= 1$, and x varies over all points in C. The bijection is established by Lemma 3.1.

In Proposition 3.5 below, we will see that there is a short exact sequence

$$\begin{aligned} 0 \rightarrow {{\,\mathrm{Ext}\,}}^1_C(\mathcal {L},\mathcal {L}(x)\otimes \omega _C^{-1}) \rightarrow {{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1})) \xrightarrow {\rho _{\mathcal {L},x}} H^0(C,\mathcal {O}_C(x)) \rightarrow 0, \end{aligned}$$

where $\rho _{\mathcal {L},x}$ has the following interpretation modulo a scalar factor. If $\mathcal {E}$ is the middle term of a representing sequence of $v \in {{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1}))$, then

$$\begin{aligned} \mathcal {E}\!\left| _C\right. \cong \mathcal {L}\oplus \mathcal {O}_{V(\rho _{\mathcal {L},x}(v))}. \end{aligned}$$

Hence, another way to phrase Corollary 3.4 is by fixing for every $x \in C$ a defining section $s_x \in H^0(C,\mathcal {O}_C(x))$ as follows. Let $\Delta \hookrightarrow C \times C$ be the diagonal, yielding a section $s_{\Delta } \in H^0(C \times C,\mathcal {O}(\Delta ))$. For every $x \in C$, we set $s_x = s_{\Delta }\!\left| _{\{x\}\times C}\right.$. Then we can write

$$\begin{aligned} \begin{array}{ccc} \text {points of}\ E_1&{\mathop {\longleftrightarrow }\limits ^{1:1}}&\bigsqcup \limits _{{\mathop {x \in C}\limits ^{\mathcal {L}\in {{\,\mathrm{Pic}\,}}^1(C)}}}\{ v \in {{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1}))\mid \rho _{\mathcal {L},x}(v) = s_x \}. \end{array} \end{aligned}$$

(3.3)

3.2 Extension spaces

So far, we have given a pointwise description of the nilpotent cone. Next, we will identify its irreducible components and their scheme structures. This subsection is a technical parenthesis in this direction. The reader may like to skip it.

Let S be a smooth projective surface and $i :C \hookrightarrow S$ a smooth curve with normal bundle $\mathcal {N}_{C/S} \cong \mathcal {O}_C(C)$. Let T be any scheme and let $\mathcal {F}$ and $\mathcal {F}'$ be two vector bundles on $T\times C$ considered as families of vector bundles on C. Denote by $\pi :T \times S \rightarrow T$ and $\pi _C:T \times C\rightarrow T$ the projections. For a morphism $f:X \rightarrow Y$, we write ${{\,\mathrm{\mathcal {E}{{ xt}}}\,}}_f$ instead of $Rf_*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}$.

Proposition 3.5

There is a short exact sequence of $\mathcal {O}_T$-modules

$$\begin{aligned} 0 \rightarrow {{\,\mathrm{\mathcal {E}{{ xt}}}\,}}^1_{\pi _C}(\mathcal {F}',\mathcal {F})\rightarrow {{\,\mathrm{\mathcal {E}{{ xt}}}\,}}^1_{\pi }(({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}) \xrightarrow {\rho } {{\,\mathrm{\mathcal {E}{{ xt}}}\,}}^0_{\pi _C}(\mathcal {F}'\boxtimes \mathcal {O}_C(-C),\mathcal {F}) \rightarrow 0, \end{aligned}$$

(3.4)

as well as for every $t \in T$ a short exact sequence of vector spaces

(3.5)

Note that the fibers of (3.4) must, in general, not coincide with (3.5), see Lemma 3.6.

Proof

Apply ${R\pi _C}_* R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(\;\;,\mathcal {F})$ or $R{{\,\mathrm{Hom}\,}}(\;\;,\mathcal {F}_t)$, to the exact triangle

$$\begin{aligned} \mathcal {F}' \boxtimes \mathcal {O}_C(-C) [1] \rightarrow L({{\,\mathrm{id}\,}}\times i)^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}' \rightarrow \mathcal {F}' \xrightarrow {[1]} \end{aligned}$$

in $D^b(T\times C)$ (see [17, Corollary 11.4]) or its counterpart in $D^b(C)$, respectively, and consider the induced cohomology sequence. $\square$

We can explicitly describe the morphism $\rho _t$ in the sequence (3.5). Represent $v \in {{\,\mathrm{Ext}\,}}^1_{S}(i_*\mathcal {F}_t',i_*\mathcal {F}_t)$ by $0 \rightarrow i_*\mathcal {F}_t \rightarrow \mathcal {E}\rightarrow i_*\mathcal {F}_t' \rightarrow 0.$ Restriction to C yields

$$\begin{aligned} \cdots \rightarrow \mathcal {F}_t' \otimes \mathcal {O}_C(-C) \xrightarrow {\delta (v)} \mathcal {F}_t \rightarrow \mathcal {E}\!\!\left| _C\right. \rightarrow \mathcal {F}_t' \rightarrow 0, \end{aligned}$$

where we inserted ${{\,\mathrm{\mathcal {T}{{ or}}}\,}}_1^{\mathcal {O}_S}(i_*\mathcal {F}_t',i_*\mathcal {O}_C)\cong \mathcal {F}_t' \otimes _{\mathcal {O}_C} {{\,\mathrm{\mathcal {T}{{ or}}}\,}}_1^{\mathcal {O}_S}(i_*\mathcal {O}_C,i_*\mathcal {O}_C) = \mathcal {F}_t' \otimes \mathcal {O}_C(-C).$ This gives a well-defined, linear map

$$\begin{aligned} \delta :{{\,\mathrm{Ext}\,}}^1_{S}(i_*\mathcal {F}_t',i_*\mathcal {F}_t) \rightarrow {{\,\mathrm{Ext}\,}}^0_C(\mathcal {F}_t' \otimes \mathcal {O}_C(-C),\mathcal {F}_t). \end{aligned}$$

As ${{\,\mathrm{im}\,}}\xi = \ker \delta$, it follows by dimension reasons, that $\delta$ has to be surjective. So, $\rho _t = \delta$ up to post-composition with an isomorphism of ${{\,\mathrm{Ext}\,}}^0_C(\mathcal {F}_t' \otimes \mathcal {O}_C(-C),\mathcal {F}_t)$.

Lemma 3.6

For every $t \in T$ there is a commutative diagram of short exact sequences

where the first vertical arrow is an isomorphism. If ${{\,\mathrm{Ext}\,}}^0_C(\mathcal {F}_t' \otimes \mathcal {O}_C(-C),\mathcal {F}_t)$ has constant dimension for all $t \in T$ all vertical arrows are isomorphisms.

Proof

The vertical morphisms are the usual functorial base change morphisms. The lower line is (3.5) and hence also exact on the left. The first vertical arrow is an isomorphism because ${{\,\mathrm{Ext}\,}}^2_C(\mathcal {F}'_t,\mathcal {F}_t)=0$. Consequently, also the upper line is exact on the left.

3.3 Irreducible components of N

In this section, we show that $E_1$ is irreducible and has the same dimension as N. Therefore its closure

$$\begin{aligned} N_1 :=\overline{E}_1 \subset N_\mathrm{red}, \end{aligned}$$

with reduced structure is an irreducible component of N. For the proof, we need some more notation. Let $\mathcal {P}_1$ be a Poincaré line bundle on ${{{\,\mathrm{Pic}\,}}^1(C)} \times C$ and $\Delta \subset C \times C$ the diagonal. Set $T :={{{\,\mathrm{Pic}\,}}^1}(C) \times C$ and on T define the following sheaves

$$\begin{aligned} \mathcal {V}&:=R^1{p_{\scriptscriptstyle 12}}_*(p_{\scriptscriptstyle 23}^*\mathcal {O}(\Delta )\otimes p_3^*\omega _C^{-1}),\\ \mathcal {W}&:=R^1{p_{\scriptscriptstyle 12}}_*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(({{\,\mathrm{id}\,}}\times i)_*p_{\scriptscriptstyle 13}^*\mathcal {P}_1,({{\,\mathrm{id}\,}}\times i)_*(p_{\scriptscriptstyle 13}^*\mathcal {P}_1\otimes p_{\scriptscriptstyle 23}^*\mathcal {O}(\Delta ) \otimes p_{\scriptscriptstyle 3}^*\omega _C^{-1} )\ \text {and}\\ \mathcal {U}&:={p_{\scriptscriptstyle 12}}_*p_{\scriptscriptstyle 23}^*\mathcal {O}(\Delta ), \end{aligned}$$

where $p_{ij}$ are the appropriate projections from ${{{\,\mathrm{Pic}\,}}^1(C)} \times C \times C$. Considering the fiber dimensions, we see that $\mathcal {V}$ and $\mathcal {U}$ are vector bundles of rank 2 and 1, respectively. In fact, $s_\Delta$ induces an isomorphism . Moreover, by Proposition 3.5 they fit into a short exact sequence

(3.6)

Consequently, also $\mathcal {W}$ is a vector bundle and $\rho$ induces a map of geometric vector bundles

$$\begin{aligned} \underline{\smash {\rho }} :\underline{\smash {{{\,\mathrm{Spec}\,}}}}_T({{\,\mathrm{Sym}\,}}^\bullet \mathcal {W}^\vee ) \longrightarrow T \times \mathbb {A}^1. \end{aligned}$$

We set

$$\begin{aligned} W :=\underline{\smash {\rho }}^{-1}(T\times \{1\}) \end{aligned}$$

with the projection $\tau :W \rightarrow T$. We retain some immediate consequences of the construction.

• (i) W is a principal homogeneous space under $\underline{\smash {{{\,\mathrm{Spec}\,}}}}_T({{\,\mathrm{Sym}\,}}^\bullet \mathcal {V}^\vee )$. In particular, it is an affine bundle over T.

• (ii) Let $t=(\mathcal {L},x) \in T$. Then by Lemma 3.6 we have

  $$\begin{aligned} W_t = \tau ^{-1}(t) \cong \mathbb {P}({{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {L},i_*(\mathcal {L}(x)\otimes \omega _C^{-1})))\setminus \mathbb {P}({{\,\mathrm{Ext}\,}}^1_C(\mathcal {L},\mathcal {L}(x)\otimes \omega _C^{-1})). \end{aligned}$$

• (iii) $\dim W = 5$.

• (vi) W is compactified by the projective bundle $\overline{W}:=\mathbb {P}(\mathcal {W})$ with boundary isomorphic to $\mathbb {P}(\mathcal {V})$, i.e.

  $$\begin{aligned} \overline{W}= W \cup \mathbb {P}(\mathcal {V}). \end{aligned}$$

Remark 3.7

Actually, $\mathcal {V}\cong p_2^*(\omega _C\oplus \omega _C)$ and hence $\mathbb {P}(\mathcal {V}) \cong \mathbb {P}^1 \times {{{\,\mathrm{Pic}\,}}^1(C)} \times C$.

Next, we relate $E_1$ and $\overline{W}$. Recall that $N_1 :=\overline{E}_1 \subset N_\mathrm{red}$. We keep all the notations from the previous section, and

Proposition 3.8

There exists a ‘universal’ extension represented by

$$\begin{aligned} 0 \rightarrow \tau _S^*({{\,\mathrm{id}\,}}\times i)_*(\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta ) \boxtimes \omega _C^{-1})\boxtimes \mathcal {O}_\tau (1) \rightarrow \mathcal {G}_\mathrm{univ} \rightarrow \tau _S^*({{\,\mathrm{id}\,}}\times i)_*p_{\scriptscriptstyle 13}^*\mathcal {P}_1 \rightarrow 0, \end{aligned}$$

such that $\mathcal {G}_\mathrm{univ} \in {{\,\mathrm{Coh}\,}}(\overline{W} \times S)$ defines a birational morphism

$$\begin{aligned} \nu :\overline{W} \longrightarrow N_1. \end{aligned}$$

In particular, $N_\mathrm{red} = N_0 \cup N_1$ is a decomposition into irreducible components.

Proof

We set $\mathcal {F}:=\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta ) \boxtimes \omega _C^{-1}$ and $\mathcal {F}' :=p_{\scriptscriptstyle 13}^*\mathcal {P}_1$. We are looking for a ‘universal’ extension, i.e. for

$$\begin{aligned} v_{\scriptscriptstyle \mathrm univ} \in {{\,\mathrm{Ext}\,}}^1_{\overline{W}\times S}(\tau _S^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',\tau _S^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}\otimes \pi '^*\mathcal {O}_\tau (1)), \end{aligned}$$

such that for $w \in W \subset \overline{W}$ the restriction of $v_{\scriptscriptstyle \mathrm univ}$ to $\{w\} \times S$ is the extension corresponding to $w \in W_{\tau (w)} \subset {{\,\mathrm{Ext}\,}}^1_S(i_*\mathcal {F}'_{\tau (w)},i_*\mathcal {F}_{\tau (w)})$.

By definition, $\mathcal {W}= R^1{\pi }_*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',({{\,\mathrm{id}\,}}\times i)_*\mathcal {F})$. Hence, there is a base change map

$$\begin{aligned} \tau ^*\mathcal {W}\rightarrow R^1{\pi '}_*L\tau _S^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}). \end{aligned}$$

We get

$$\begin{aligned}&H^0(\overline{W}, \tau ^*\mathcal {W}\otimes \mathcal {O}_\tau (1))\rightarrow H^0(\overline{W},R^1{\pi '}_*L\tau _S^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}) \otimes \mathcal {O}_{\tau }(1))\nonumber \\&\quad \xleftarrow {\sim } H^1(\overline{W}\times S,L\tau _S^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}) \otimes \pi '^* \mathcal {O}_{\tau }(1)))\nonumber \\&\quad = {{\,\mathrm{Ext}\,}}^1_{\overline{W}\times S}(\tau _S^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',\tau _S^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}\otimes \pi '^*\mathcal {O}_\tau (1)), \end{aligned}$$

(3.7)

where the indicated isomorphism comes from the Leray spectral sequence. It is an isomorphism, because

$$\begin{aligned}&R^0{\pi '}_*L\tau _S^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',({{\,\mathrm{id}\,}}\times i)_*\mathcal {F})\\&\cong R^0{\pi _C'}_*L\tau _C^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(L({{\,\mathrm{id}\,}}\times i)^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',\mathcal {F})= 0, \end{aligned}$$

where $\pi _C :T \times C \rightarrow T$. The last equality follows from the long exact sequence

$$\begin{aligned} \cdots \rightarrow 0= & {} R^{0}{\pi _C'}_*L\tau _C^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(\mathcal {F}',\mathcal {F}) \rightarrow R^0{\pi _C'}_*L\tau _C^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(L({{\,\mathrm{id}\,}}\times i)^*({{\,\mathrm{id}\,}}\times i)_*\mathcal {F}',\mathcal {F})\\ \rightarrow 0= & {} R^0{\pi _C'}_*L\tau _C^*R{{\,\mathrm{\mathcal {H}{{ om}}}\,}}(\mathcal {F}'\boxtimes \mathcal {O}_C(-C)[1],\mathcal {F})\rightarrow \ldots . \end{aligned}$$

Finally, we consider the universal surjection as an element in $H^0(\overline{W}, \tau ^*\mathcal {W}\otimes \mathcal {O}_\tau (1)))$ and take its image under (3.7). This produces the desired extension.

By construction, $\mathcal {G}_\mathrm{univ} \in {{\,\mathrm{Coh}\,}}(\overline{W} \times S)$ defines a morphism $\nu :\overline{W} \rightarrow N_1 \subset M$ which restricts to a bijection $W \rightarrow E_1$ (see Corollary 3.4 and (3.3)). By degree reasons an extension on C of the form $0 \rightarrow \mathcal {L}\rightarrow \mathcal {E}\rightarrow \mathcal {L}' \rightarrow 0,$ where $\deg \mathcal {L}' = 1$ and $\deg \mathcal {L}= 0$ is stable or split. However, the split extensions do not occur in $\mathbb {P}(\mathcal {V})$. Hence, $\nu$ is everywhere defined. Moreover, the boundary $\overline{W}\setminus W = \mathbb {P}(\mathcal {V})$ maps to $N_1\setminus E_1 = N_0 \cap N_1$. $\square$

Remark 3.9

One can show that $\nu :W \rightarrow E_1$ is actually an isomorphism of schemes. Moreover, $\nu :\overline{W} \rightarrow N_1$ is finite and hence a normalization map. Its tangent map is analyzed in [11, Proposition 7.5] and provides a characterization of the singularities of $N_1$.

4 Proof of Theorem 1.1

We will now prove Theorem 1.1.

Theorem

Let $C \in |H|$ be an irreducible curve. The degrees of the two components of the nilpotent cone $N_C = N_0 \cup N_1$ are given by

$$\begin{aligned} \begin{array}{lcr} \deg _{u_1}N_0 = 5 \cdot 2^9&\text {and}&\deg _{u_1}N_1 = 5^2\cdot 2^{11} \end{array} \end{aligned}$$

and their multiplicities are

$$\begin{aligned} \begin{array}{lcr} {{\,\mathrm{mult}\,}}_{N_C}N_0 = 2^3&\text {and}&{{\,\mathrm{mult}\,}}_{N_C}N_1 = 2. \end{array} \end{aligned}$$

Moreover, any fiber F of the Mukai system has degree $5 \cdot 3 \cdot 2^{13}$.

All degrees will be computed with respect to a naturally defined distinguished ample class $u_1 \in H^2(M,\mathbb {Z})$, which we construct in Sect. 4.1. We set

$$\begin{aligned} d_i = \deg _{u_1}(N_i) :=\int _M [N_i]u_1^5 \end{aligned}$$

for $i= 0,1$, where by abuse of notation $[N_i] \in H^{10}(M,\mathbb {Z})$ is the Poincaré dual of the fundamental homology class $[N_i] \in H_{10}(M,\mathbb {Z})$.

The multiplicity is defined as follows. Let $\eta _i$ be the generic point of $N_i$. Then

$$\begin{aligned} m_i = {{\,\mathrm{mult}\,}}_N N_i :=\lg _{\mathcal {O}_{N,\eta _i}}\mathcal {O}_{N_i,\eta _i} = \lg _{\mathcal {O}_{N_i,\eta _i}}\mathcal {O}_{N_i,\eta _i}. \end{aligned}$$

In particular, we have an equality $[F] = m_0[N_0]+m_1[N_1] \in H^{10}(M,\mathbb {Z})$ for any fiber F. Consequently, inserting $m_0 = 2^3$ and $m_1= 2$, we find

$$\begin{aligned} \deg _{u_1}(F) = 5 \cdot 2^{12} + 5^2 \cdot 2^{12} =5 \cdot 3 \cdot 2^{13}. \end{aligned}$$

as stated in the theorem. Luckily, it turns out that the multiplicities are small in comparison with the degrees so that it is possible to determine the multiplicities from the knowledge of the degrees but not vice versa.

Proof of the multiplicities knowing all the degrees

Let $F \subset M$ be a smooth fiber. Then, we have $\deg F = m_0d_0 + m_1d_1$ and hence

$$\begin{aligned} 5\cdot 3 \cdot 2^{13} = m_0 \cdot 5 \cdot 2^9 + m_1 \cdot 5^2 \cdot 2^{11}. \end{aligned}$$

The only possible solutions are $(m_0,m_1) = (28,1)$ or $(m_0,m_1) = (8,2)$. However, by [8, Proposition 4.11]

$$\begin{aligned} \dim T_{[\mathcal {E}]}N= \dim {{\,\mathrm{Ext}\,}}^1_{2C}(\mathcal {E},\mathcal {E}) = \dim N + 1\ \text {for all}\ [\mathcal {E}] \in E_1. \end{aligned}$$

Hence, $N_1$ is not reduced and the first solution is ruled out.

Remark 4.1

We will prove Theorem 1.1 for a fixed smooth curve $C \in |H|$, which implies the case of an irreducible and possibly singular curve by a deformation argument as follows. According to the careful analysis in [6, Section 3.7, in particular Propositions 3.7.23 & 3.7.19] the above description of the irreducible components of $f^{-1}(2C)$ is valid for every irreducible curve $C \in |H|$. Hence, if one deforms from a smooth to a singular, irreducible curve in |H|, the irreducible components of the fiber with their reduced structure deform as well. Consequently, degrees and multiplicities remain constant.

4.1 Construction of the ample class $u_1$

We use the determinant line bundle construction [16, Lemma 8.1.2] in order to produce an ample class on the moduli space M.

Let X and T be two projective varieties and assume that X is smooth. Let $p:T\times X \rightarrow T$ and $q:T \times X \rightarrow X$ denote the two projections. For any $\mathcal {W}\in {{\,\mathrm{Coh}\,}}(X\times T)$ flat over T, we define $\lambda _{\mathcal {W}} :{{\,\mathrm{K}\,}}(X)_\mathrm{num} \rightarrow H^2(T,\mathbb {Z})$ to be the following composition

$$\begin{aligned} {{\,\mathrm{K}\,}}(X)_\mathrm{num} \xrightarrow {q^*} {{\,\mathrm{K}\,}}^0(T \times X)_\mathrm{num} \xrightarrow {\cdot [\mathcal {W}]} {{\,\mathrm{K}\,}}^0(T \times X)_\mathrm{num} \xrightarrow {Rp_*} {{\,\mathrm{K}\,}}^0(T)_\mathrm{num} \xrightarrow {\det } {{\,\mathrm{NS}\,}}(T) \subset H^2(T,\mathbb {Z}). \end{aligned}$$

We will take advantage of the functorial properties of this definition. These are

• (i) $f^*\lambda _{\mathcal {W}} = \lambda _{(f \times {{\,\mathrm{id}\,}})^*\mathcal {W}}$ for any morphism $f:T' \rightarrow T$ and

• (ii) $\lambda _{({{\,\mathrm{id}\,}}\times i)_*\mathcal {W}}(x) = \lambda _\mathcal {W}(Li^*x)$ for all $x \in {{\,\mathrm{K}\,}}(X)_\mathrm{num}$ if $i:Y \hookrightarrow X$ is the inclusion of a closed, smooth subscheme and $\mathcal {W}\in {{\,\mathrm{Coh}\,}}(T \times Y)$.

The construction is especially interesting if $X={{\,\mathrm{M}\,}}_T(c)$ is a fine moduli space, that parametrizes coherent sheaves of class c on T. Let $\mathcal {E}_\mathrm{univ}$ be a universal sheaf on ${{\,\mathrm{M}\,}}_T(c)\times T$, then

$$\begin{aligned} \lambda _{\mathcal {E}_\mathrm{univ} \otimes p^*\mathcal {M}}(x) = \lambda _{\mathcal {E}_\mathrm{univ}}(x) + \chi (c\cdot x) c_1(\mathcal {M}) \end{aligned}$$

for all $\mathcal {M}\in {{\,\mathrm{Pic}\,}}({{\,\mathrm{M}\,}}_T(c))$. Hence,

$$\begin{aligned} \lambda _{{{\,\mathrm{M}\,}}_T(c)} :=\lambda _{\mathcal {E}_\mathrm{univ}} :c^{\bot ,\chi } \longrightarrow {{\,\mathrm{NS}\,}}({{\,\mathrm{M}\,}}_T(c)) \end{aligned}$$

is well-defined and does not depend on the choice of universal sheaf. Here,

$$\begin{aligned} c^{\bot ,\chi } = \{x \in {{\,\mathrm{K}\,}}(T)_\mathrm{num}\mid \chi (x\cdot c) = 0 \}. \end{aligned}$$

Example 4.2

Let C be a smooth curve of any genus $g \ge 0$. Then

Fix $n \ge 1$ and $d \in \mathbb {Z}$ coprime and let $c = (n,d)\in {{\,\mathrm{K}\,}}(C)_\mathrm{num}$. Then ${{\,\mathrm{M}\,}}_C(c) = {{\,\mathrm{M}\,}}_C(n,d)$ is the moduli space of stable vector bundles of rank n and degree d on C and we find $c ^{\bot ,\chi } = \langle (-n, d+n(1-g)\rangle$. The generalized Theta divisor can be defined by

$$\begin{aligned} \Theta _{{{\,\mathrm{M}\,}}_C(n,d)} :=\lambda _{{{\,\mathrm{M}\,}}_C(n,d)}(-n,d+n(1-g)), \end{aligned}$$

see [12, Théorème D]. A special case is ${{\,\mathrm{M}\,}}_C(1,k) = {{\,\mathrm{Pic}\,}}^k(C)$. In this case, we find $c^{\bot ,\chi } = \langle (-1,k + 1-g) \rangle$ and

$$\begin{aligned} \Theta _k :=\lambda _{{{\,\mathrm{Pic}\,}}^k(C)}(-1,k+1-g) \end{aligned}$$

is the class of the canonical Theta divisor in ${{\,\mathrm{Pic}\,}}^k(C)$.

Remark 4.3

Denote by ${{\,\mathrm{SM}\,}}_C(n,d)$ the moduli space of vector bundles with fixed determinant, i.e. a fiber of $\det :{{\,\mathrm{M}\,}}_C(n,d) \rightarrow {{\,\mathrm{Pic}\,}}(C)$ and by $\Theta _{{{\,\mathrm{SM}\,}}_C(n,d)}$ the restriction of $\Theta _{{{\,\mathrm{M}\,}}_C(n,d)}$ to ${{\,\mathrm{SM}\,}}_C(n,d)$. Taking the tensor product defines an étale map

$$\begin{aligned} h :{{\,\mathrm{SM}\,}}_C(n,d) \times {{\,\mathrm{Pic}\,}}^0(C) \longrightarrow {{\,\mathrm{M}\,}}_C(n,d) \end{aligned}$$

of degree $n^{2g}$. Using [10, Corollary 6], we find the following relation if (n, d) are coprime

$$\begin{aligned} h^* \Theta _{{{\,\mathrm{M}\,}}_C(n,d)} = p_1^*\Theta _{{{\,\mathrm{SM}\,}}_C(n,d)} + n^2p_2^*\Theta _0. \end{aligned}$$

(4.1)

Lemma 4.4

Let C be a smooth curve of genus g and $\mathcal {P}$ a Poincaré line bundle on ${{\,\mathrm{Pic}\,}}^k(C) \times C$. Then

$$\begin{aligned} \lambda _{\mathcal {P}} :{{\,\mathrm{K}\,}}(C)_\mathrm{num} \rightarrow H^2({{\,\mathrm{Pic}\,}}^k(C),\mathbb {Z}) \end{aligned}$$

is given by

$$\begin{aligned} (r,d) \mapsto (d+(k+1-g)r)\mu -r\Theta _k, \end{aligned}$$

where $p_1^*\mu =c_1^{2,0}(\mathcal {P}) \in H^2({{\,\mathrm{Pic}\,}}^k(C)\times C,\mathbb {Z})$ is the (2, 0) Künneth component of $c_1(\mathcal {P})$.

By tensoring with a suitable line bundle on ${{\,\mathrm{Pic}\,}}^k(C)$, we can assume $c_1^{2,0}(\mathcal {P}) = 0$.

Proof

Let us abbreviate ${{\,\mathrm{Pic}\,}}^k(C)$ to ${{\,\mathrm{Pic}\,}}^k$. We decompose

$$\begin{aligned} c_1(\mathcal {P}) =c^{2,0} + c^{1,1} + c^{0,2} \end{aligned}$$

into its Künneth components and write $c^{2,0}= p^*\mu$ for some $\mu \in H^2({{\,\mathrm{Pic}\,}}^k,\mathbb {Z})$. Then by [1, VIII §2] the class $\gamma = c^{1,1}$ satisfies $\gamma ^2 = -2\rho p^*\Theta _k$. Moreover, by definition, $c^{0,2} = k\rho$, where $\rho$ is the pullback of the class of a point on C. Together, $c_1(\mathcal {P}) = p^*\mu + \gamma + k\rho$ and

$$\begin{aligned} {{\,\mathrm{ch}\,}}(\mathcal {P}) = 1 + p^*\mu + \gamma + k\rho + \rho p^*(k\mu - \Theta _k). \end{aligned}$$

Now, let $x = (r,d) \in {{\,\mathrm{K}\,}}(C)_\mathrm{num}$. The Grothendieck–Riemann–Roch theorem gives

$$\begin{aligned} {{\,\mathrm{ch}\,}}(Rp_*(\mathcal {P}\otimes q^*x))&= p_*({{\,\mathrm{ch}\,}}(\mathcal {P}\otimes q^*x){{\,\mathrm{td}\,}}({{\,\mathrm{Pic}\,}}^k \times C)) = p_*({{\,\mathrm{ch}\,}}(\mathcal {P}){{\,\mathrm{ch}\,}}(q^*x)q^*{{\,\mathrm{td}\,}}(C)) \\&=p_*({{\,\mathrm{ch}\,}}(\mathcal {P})(r+((1-g)r+d)\rho ))\\&= kr + (1-g)r + d +(kr + (1-g)r) + d)\mu - r\Theta _k. \end{aligned}$$

In particular, $\lambda _{\mathcal {P}}(x) = c_1(Rp_*(\mathcal {P}\otimes q^* x)) = ((k+1-g)r + d)\mu -r\Theta _k$. $\square$

We come back to our original situation, i.e. (S, H) is a polarized K3 surface of genus 2 and $M= M_H(v)$ parametrizes H-stable sheaves with fixed Mukai vector $v = (0,2H,-1)$ or equivalently, with Chern character $v_\mathrm{ch} =(0,2H,-1)$. In this setting $\lambda _M$ induces an isomorphism, [16, Theorem 6.2.15]

(4.2)

As v and $v_\mathrm{ch}$ coincide, we will notationally not distinguish between them anymore. We find

$$\begin{aligned} {v}^{\bot ,\chi } = \{ (2c.H,c,s)\mid c\in {{\,\mathrm{NS}\,}}(S), s\in \mathbb {Z}\}. \end{aligned}$$

Warning 4.5

In this setting, one usually wants to consider the morphism $\lambda _M$ in terms of the Mukai vector and the Mukai pairing instead of the chern character and the intersection product, i.e. one considers the composition

which identifies the Mukai pairing on the left hand side with the Beauville–Bogomolov form on the right hand side. Here $v_\mathrm{ch}\cdot \sqrt{{{\,\mathrm{td}\,}}(S)}=v$. Explicitly, if $v=(r,c,s)$, then $v_\mathrm{ch} =(r,c,s-r)$ and

$$\begin{aligned} \langle (r',c',s'),(r,c,s)\rangle = \chi ((-r',c',-s'-r')\cdot (r,c,s-r)).\end{aligned}$$

Thus the first arrow is given by $(r',c',s') \mapsto (-r',c',-s'-r')$.

Definition 4.6

For all $s \in \mathbb {Z}$ we define

$$\begin{aligned} l_s :=\lambda _ {M}((-4,-H,s)) \in H^2(M,\mathbb {Z}). \end{aligned}$$

The value of s does not have any relevance for our computations. However, with the results of [3], it can be proven that $l_s$ is ample for $s \gg 0$ and one can even compute the precise boundary of the ample cone.

Definition 4.7

For everything what follows, we fix $s_0 \gg 0$ such that $l_{s_0}$ is ample and set

$$\begin{aligned} u_1 :=l_{s_0}. \end{aligned}$$

4.2 Degree of a general fiber

We compute the degree of a general fiber.

Proposition 4.8

Let $D \in |2H|$ be a smooth curve and let $F :=f^{-1}(D)$ be the corresponding fiber. Let $u= \lambda _{M}(x)$ with $x=(2c.H,c,s) \in v^{\bot ,\chi }$. Then

$$\begin{aligned} u\!\left| _{F}\right. = -2c.H \cdot \Theta _3, \end{aligned}$$

where $\Theta _3 \in H^2({{\,\mathrm{Pic}\,}}^3(D),\mathbb {Z})$ is the class of the Theta divisor. In particular, we have

$$\begin{aligned} \deg _{u_1}F = 5!\cdot 2^{10}. \end{aligned}$$

Proof

Let $i :D \hookrightarrow S$ be the inclusion. The inclusion ${{\,\mathrm{Pic}\,}}^3(D) \cong F \hookrightarrow M$ is defined by $({{\,\mathrm{id}\,}}\times i)_*\mathcal {P}_3$, where $\mathcal {P}_3$ is a Poincaré line bundle on ${{\,\mathrm{Pic}\,}}^3(D) \times D$. Hence,

$$\begin{aligned} u|_{{{\,\mathrm{Pic}\,}}^3(D)} = \lambda _{({{\,\mathrm{id}\,}}\times i)_*\mathcal {P}_3}(x) = \lambda _{\mathcal {P}_3}(Li^*x). \end{aligned}$$

Now, $Li^* :K(S)_\mathrm{num} \rightarrow K(D)_\mathrm{num} \cong \mathbb {Z}^{\oplus 2}$ maps (r, c, s) to (r, c.D) and thus $Li^*x$ to $2c.H\cdot (1,1)$, whereas by definition $\theta _3 = \lambda _{\mathcal {P}_3}(-1,-1).$ Finally,

$$\begin{aligned} \deg _{u_1}F = \int _{{{\,\mathrm{Pic}\,}}^3(D)}(4\Theta _3)^5 = 2^{10} \cdot 5!. \end{aligned}$$

Remark 4.9

One can also prove the above result using the Beauville–Bogomolov form $(\ ,\ )_{BB}$ on $H^2(M,\mathbb {Z})$. Let $u_0 = f^*c_1(\mathcal {O}(1)) \in H^2(M,\mathbb {Z})$. Then $[F] = u_0^5 \in H^{10}(M,\mathbb {Z})$ and

$$\begin{aligned} \deg _{u_1}(F) = \int _M u_0^5u_1^5 = 5!\cdot (u_0,u_1)_{BB}^5, \end{aligned}$$

where we use that $(u_0,u_0)_{BB}=0$ and that M is birational to $S^{[5]}$ in order to determine the correct Fujiki constant. One verifies that $u_0= \lambda _M((0,0,1))$ [25, Lem 4.4] whereas, by definition, $u_1 = \lambda _M(-4,-H,s_0)$ with $s_0 \gg 0$. After correct identification (cf. Warning 4.5), one has

$$\begin{aligned} (\lambda _{M}(r,c,s), \lambda _{M}(r',c',s'))_{BB} = \langle (r,c,s),(r',c',s')\rangle +2rr'. \end{aligned}$$

This gives $(u_0,u_1)_{BB} =4$.

4.3 Degree of the vector bundle component $N_0$

Next, we deal with the component $N_0$, which is isomorphic to $M_C(2,1)$.

Proposition 4.10

Let $x= \lambda _{M}(u)$ with $u=(2c.H,c,s) \in v^{\bot ,\chi }$. Then

$$\begin{aligned} x\!\left| _{N_0}\right. = -c.H\Theta , \end{aligned}$$

where $\Theta \in H^2(N_0,\mathbb {Z})$ is the the generalized Theta divisor. In particular,

$$ u_1\!\left| _{N_0}\right. = 2 \Theta , $$

and given $x_i = \lambda _{M}(2c_i.H,c_i,s_i)$ for $i=1,\ldots ,5$, we find

$$\begin{aligned} \int _M x_1\ldots x_5 [N_0] = -\prod _{i=1}^5c_i.H \int _{N_0}\Theta ^5 = -5\cdot 2^4\prod _{i=1}^5c_i.H. \end{aligned}$$

Hence, $\deg _{u_1}N_0 = 5 \cdot 2^9$.

Proof

Let $i :C \hookrightarrow S$ be the inclusion. The inclusion $N_0 \hookrightarrow M$ is defined by $({{\,\mathrm{id}\,}}\times i)_*\mathcal {E}_\mathrm{univ}$, where $\mathcal {E}_\mathrm{univ}$ is the universal vector bundle on $N_0 \times C$. Hence,

$$\begin{aligned} x\!\left| _{N_0} \right. = \lambda _{({{\,\mathrm{id}\,}}\times i)_*\mathcal {E}_\mathrm{univ}}(u) = \lambda _{N_0}(Li^*u). \end{aligned}$$

Now, $Li^* :K(S)_\mathrm{num} \rightarrow K(C)_\mathrm{num} \cong \mathbb {Z}^{\oplus 2}$ maps (r, c, s) to (r, c.H). In particular, $Li^*u = c.H(2,1)$, whereas by definition $\theta = \lambda _{N_0}(-2,-1).$

Next, we compute $\int _{N_0}\Theta ^5$ by pulling back along $h :{{\,\mathrm{SM}\,}}_C(2,1) \times {{\,\mathrm{Pic}\,}}^0(C) \rightarrow N_0$ from Remark 4.3.

$$\begin{aligned} \int _{{{\,\mathrm{M}\,}}_C(2,1)} \Theta ^5&{\mathop {=}\limits ^{(4.1)}} \frac{1}{2^4}\int _{{{\,\mathrm{SM}\,}}_C(2,1)\times {{\,\mathrm{Pic}\,}}^0(C)}(p_1^*\Theta _{{{\,\mathrm{SM}\,}}} + 4p_2^*\Theta _0)^5 \\&=\frac{1}{2^4}\left( {\begin{array}{c}5\\ 3\end{array}}\right) \int _{{{\,\mathrm{SM}\,}}_C(2,1)}\Theta _{{{\,\mathrm{SM}\,}}}^3\int _{{{\,\mathrm{Pic}\,}}^0(C)}(4\Theta _0)^2= 5\cdot 2^4. \end{aligned}$$

The value $\int _{{{\,\mathrm{SM}\,}}_C(2,1)}\Theta _{{{\,\mathrm{SM}\,}}}^3 = 4$ is given by the leading term of the Verlinde formula [27].

Remark 4.11

The general formula is

$$\begin{aligned} \int _{{{\,\mathrm{M}\,}}_C(n,d)}\Theta ^{\dim {{\,\mathrm{M}\,}}_C(n,d)} = \dim {{\,\mathrm{M}\,}}_C (n,d)!(2^{2g-2}-2)\frac{(-1)^g2^{2g-2}B_{2g-2}}{(2g-2)!}, \end{aligned}$$

where $B_i$ is the i-th Bernoulli number. The second Bernoulli number is $B_2= \frac{1}{6}$

Remark 4.12

In the general case, where $[ v = (0,nH,s)\ \text {and}\ u_1= \lambda _M(-n(2g-2),sH,*)\ \text {with}\ s=n+d(1-g),]$ we find

$u_1\!\left| _{F}\right. = n(2g-2)\Theta _\delta$ and $u_1\!\left| _{N_0}\right. = (2g-2)\Theta$. Thus

$$\begin{aligned}&\deg _{u_1}F= (n(2g-2))^{\dim N}\cdot \dim N!\\&\text {and} \deg _{u_1}N_0 = (2g-2)^{\dim N}\int _{{{\,\mathrm{M}\,}}_C(n,d)}\Theta ^{\dim {{\,\mathrm{M}\,}}_C(n,d)}. \end{aligned}$$

Here, $\dim N = n^2(2g-2)+2$.

4.4 Degree of the other component $N_1$

We complete the proof of Theorem 1.1 by dealing with the remaining component $N_1$. Recall from Proposition 3.8 that there is a birational map $\nu :\overline{W} \rightarrow N_1,$ where $\tau :\overline{W} = \mathbb {P}(\mathcal {W}) \rightarrow T = {{{\,\mathrm{Pic}\,}}^1(C)} \times C$.

Proposition 4.13

Let $x_i= \lambda _{M}(u_i)$ with $u_i=(2c_i.H,c_i,s_i) \in v^{\bot ,\chi }$ for $i=1,\ldots ,5$. Then

$$\begin{aligned} \int _M x_1\ldots x_5 [N_1] = \int _{\overline{W}}\prod _{i=1}^5\nu ^*(x_i\!\left| _{N_1}\right. ) = -5^2\cdot 2^6\prod _{i=1}^5c_i.H. \end{aligned}$$

(4.3)

In particular, $\deg _{u_1}N_1 = 5^2 \cdot 2^{11}$.

Note that the first equality in (4.3) is immediate, because $\nu :\overline{W} \rightarrow N_1$ is birational. For the proof of the proposition, we need to introduce some more notation. We abbreviate ${{\,\mathrm{Pic}\,}}^1(C)$ to ${{\,\mathrm{Pic}\,}}^1$ and in the following all cohomology groups have $\mathbb {Z}$ coefficients. We set

$$\begin{aligned} \begin{array}{lcr} \zeta = c_1(\mathcal {O}_\tau (1)) \in H^2(\overline{W})&\text {and write}&\rho = p_2^*[\mathrm {pt}] \in H^2({{\,\mathrm{Pic}\,}}^1 \times C) \end{array} \end{aligned}$$

for the pullback of the class of a point on C. If no confusion is likely, we suppress pullbacks from our notation, e.g. we will write $\Theta _1 \in H^2({{\,\mathrm{Pic}\,}}^1 \times C)$ and also $\Theta _1 \in H^2(\mathbb {P}(\mathcal {W}))$ instead of $p_1^*\Theta _1$ and $\tau ^*p_1^*\Theta _1$, respectively. Moreover, we define

$$\begin{aligned} \pi :=c_1(\mathcal {P}) -c_1^{2,0}(\mathcal {P}) \in H^2({{\,\mathrm{Pic}\,}}^1 \times C), \end{aligned}$$

where $\mathcal {P}$ is a Poincaré line bundle. Note that $\pi$ is independent of the choice of $\mathcal {P}$.

Proof of Proposition 4.13

We will split the proof into the following three steps.

• (i) Let $x = \lambda _{M}(2c.H,c,s)$. Then

  $$\begin{aligned} \nu ^*(x\!\left| _{N_1}\right. ) = \lambda _{\mathcal {G}_\mathrm{univ}}(x) = c.H(-4\Theta _1+ 2\pi -7 \rho -\zeta ) \in H^2(\overline{W}). \end{aligned}$$

• (ii) We have

  $$\begin{aligned} (-4\Theta _1+ 2\pi -7 \rho -\zeta )^5 = -5^2 2^{5}\zeta ^2\rho \Theta _1^2 \in H^{10}(\overline{W}). \end{aligned}$$

• (iii) The top cohomology group $H^{10}(\overline{W})$ generated by $\tfrac{1}{2}\zeta ^2\rho \Theta _1^2$ and we have

  $$\begin{aligned} \int _{\mathbb {P}(\mathcal {W})}\zeta ^2\rho \Theta _1^2 = 2. \end{aligned}$$

$\square$

Proof of (i)

In Proposition 3.8, we defined the morphism $\nu :\overline{W} \rightarrow N_1$ by means of $\mathcal {G}_\mathrm{univ} \in {{\,\mathrm{Coh}\,}}(\overline{W} \times S)$, which sits in the (universal) extension

$$\begin{aligned} 0 \rightarrow \tau _S^*({{\,\mathrm{id}\,}}\times i)_*(\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta ) \boxtimes \omega _C^{-1} )\boxtimes \mathcal {O}_\tau (1) \rightarrow \mathcal {G}_\mathrm{univ} \rightarrow \tau _S^*({{\,\mathrm{id}\,}}\times i)_*p_{\scriptscriptstyle 13}^*\mathcal {P}_1 \rightarrow 0, \end{aligned}$$

where $\tau _S = \tau \times {{\,\mathrm{id}\,}}_S :\overline{W} \times S \rightarrow {{\,\mathrm{Pic}\,}}^1 \times C \times S$. So, by construction, we have

$$\begin{aligned} \lambda _{\mathcal {G}_\mathrm{univ}}(x)&= \lambda _{\tau _S^*({{\,\mathrm{id}\,}}\times i)_*(\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta ) \boxtimes \omega _C^{-1}) \boxtimes \mathcal {O}_\tau (1)}(x) + \lambda _{\tau _S^*({{\,\mathrm{id}\,}}\times i)_*p_{\scriptscriptstyle 13}^*\mathcal {P}_1}(x)\\&= \lambda _{\tau _S^*({{\,\mathrm{id}\,}}\times i)_*(\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta ) \boxtimes \omega _C^{-1})}(x) + k(Li^*x)\cdot \zeta + \tau ^*p_1^*\lambda _{\mathcal {P}_1}(Li^*x)\\&= \tau ^*(\lambda _{\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta )}(Li^*x\cdot \omega ^{-1})+ p_1^*\lambda _{\mathcal {P}_1}(Li^*x))+ k(Li^*x)\cdot \zeta ,\\ \end{aligned}$$

where $\omega = c_1(\omega _C)$ and $k(Li^*x) = {{\,\mathrm{rk}\,}}Rp_* (\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta )\boxtimes \omega _C^ {-1} \boxtimes Li^*x) = \chi (Li^*x ) = - c.H$.

The term $\lambda _{\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta )}(Li^*x\cdot \omega ^{-1})+ p_1^*\lambda _{\mathcal {P}_1}(Li^*x)$, is determined in Lemmas 4.14 and 4.4. Note that each summand depends on the choice of a Poincaré line bundle, whereas the sum does not. Together,

$$\begin{aligned} \nu ^*(x\!\left| _{N_1}\right. )&= \tau ^*(\lambda _{\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta )}(Li^*x\cdot \omega ^{-1})+ p_1^*\lambda _{\mathcal {P}_1}(Li^*x))-c.H\zeta \\&= c.H (p_1^*(\lambda _{\mathcal {P}_1}(2,-3)+ \lambda _{\mathcal {P}_1}(2,1)) + 2c_1(\mathcal {P}_1) -7\rho -\zeta )\\&= c.H ( -4\Theta _1 + 2\pi -7\rho -\zeta ). \end{aligned}$$

Lemma 4.14

Let $\mathcal {F}\in {{\,\mathrm{Coh}\,}}(X \times C)$. Then

$$\begin{aligned} \lambda _{\mathcal {F}\boxtimes \mathcal {O}(\Delta )} (x) = {p_1}^*\lambda _{\mathcal {F}}(x) + rc_1(\mathcal {F}) + c_0(\mathcal {F})(d-2r)\rho \end{aligned}$$

for all $x = (r,d) \in {{\,\mathrm{K}\,}}(C)_\mathrm{num}$. In particular,

$$\begin{aligned} \lambda _{\mathcal {P}_1 \boxtimes \mathcal {O}(\Delta )}(Li^*x\cdot \omega ^{-1}) = c.H (p_1^*\lambda _{\mathcal {P}}(2,-3) + 2c_1(\mathcal {P}_1) -7\rho ). \end{aligned}$$

Proof

We have

$$\begin{aligned}{}[\mathcal {F}\boxtimes \mathcal {O}(\Delta )] = [p_{13}^*\mathcal {F}] + [({{\,\mathrm{id}\,}}\times i_\Delta )_*(p_2^*\omega _C^{-1} \otimes \mathcal {F})] \in {{\,\mathrm{K}\,}}(X\times C\times C), \end{aligned}$$

where $i_\Delta :C \rightarrow C \times C$ is the diagonal and thus

$$\begin{aligned} \lambda _{\mathcal {F}\boxtimes \mathcal {O}(\Delta )}(x) = \lambda _{p_{13}^*\mathcal {W}}(x) + \lambda _{({{\,\mathrm{id}\,}}\times i_\Delta )_*(p_2^*\omega _C^{-1} \otimes \mathcal {F})}(x) \in H^2(X \times C) \end{aligned}$$

for all $x \in {{\,\mathrm{K}\,}}(C)_\mathrm{num}$. Now,

$$\begin{aligned} \lambda _{({{\,\mathrm{id}\,}}\times i_\Delta )_*(p_2^*\omega ^{-1} \cdot [\mathcal {F}])}(x)&= \det R{p_{12}}_*(({{\,\mathrm{id}\,}}\times i_\Delta )_*(p_2^*\omega ^{-1} \cdot [\mathcal {F}]) \cdot p_3^*x)\\&= \det R(p_{12}\circ ({{\,\mathrm{id}\,}}\times i_\Delta ))_*([\mathcal {F}]\cdot p_2^*(\omega ^{-1} \cdot x))\\&= \det ([\mathcal {F}] \cdot p_2^*(\omega ^{-1} \cdot x)) = rc_1(\mathcal {F}) + c_0(\mathcal {F})(d-r(2g-2))\rho . \end{aligned}$$

$\square$

To prove the remaining steps, we need to understand the cohomology ring $H^*(\overline{W})$.

Lemma 4.15

We have

$$\begin{aligned} H^*(\overline{W})\cong H^*({{\,\mathrm{Pic}\,}}^1 \times C)[\zeta ]/\zeta ^3+4\rho \zeta ^2. \end{aligned}$$

In particular,

$$\begin{aligned} H^{10}(\overline{W}) = \zeta ^2 \cdot H^6({{\,\mathrm{Pic}\,}}^1 \times C). \end{aligned}$$

Proof

By definition, $\overline{W} = \mathbb {P}(\mathcal {W})$. Hence,

$$\begin{aligned} H^*(\overline{W})\cong H^*({{{\,\mathrm{Pic}\,}}^1} \times C)[\zeta ]/\zeta ^3+c_1(\mathcal {W})\zeta ^2+ c_2(\mathcal {W})\zeta + c_3(\mathcal {W}). \end{aligned}$$

We use the short exact sequence $0 \rightarrow \mathcal {V}\rightarrow \mathcal {W}\rightarrow \mathcal {O}_T \rightarrow 0$ from (3.6) to compute the Chern classes of $\mathcal {W}$. Note that

$$\begin{aligned} \mathcal {V}= R^1{p_{\scriptscriptstyle 12}}_*(p_{\scriptscriptstyle 23}^*\mathcal {O}(\Delta )\otimes p_3^*\omega _C^{-1}) \cong p_2^*R^1 {p_1}_*(\mathcal {O}(\Delta )\otimes p_2^*\omega _C^{-1}). \end{aligned}$$

So the Chern classes of $\mathcal {V}$ can be computed by the push forward along the first projection of the following the short exact sequence

$$\begin{aligned} 0 \rightarrow p_2^*\omega _C^{-1} \rightarrow \mathcal {O}(\Delta ) \boxtimes \omega _C^{-1} \rightarrow p_2^*\omega _C^{- 2}\!\left| _\Delta \right. \rightarrow 0. \end{aligned}$$

We find

$$\begin{aligned} 0 \longrightarrow \omega _C^{-2} \rightarrow \mathcal {O}_C \otimes H^1(C,\omega _C^{-1}) \rightarrow R^1{p_1}_*(\mathcal {O}(\Delta )\otimes p_2^*\omega _C^{-1}) \longrightarrow 0. \end{aligned}$$

Hence, $c_1(\mathcal {W}) = 4 \rho$ and $c_i(\mathcal {W}) = 0$ if $i \ge 2$. $\square$

Proof of (ii) and (iii)

We want to show that

$$\begin{aligned} (-4\Theta _1+ 2\pi -7 \rho -\zeta )^5 = -5^2 2^{5}\zeta ^2\rho \Theta _1^2 \in H^{10}(\overline{W}). \end{aligned}$$

We compute

$$\begin{aligned} (-4\Theta _1+2\pi -7 \rho -\zeta )^5&= \left( {\begin{array}{c}5\\ 3\end{array}}\right) (-\zeta ^3)(-4\Theta _1+2\pi -7 \rho )^2 + \left( {\begin{array}{c}5\\ 2\end{array}}\right) \zeta ^2(-4\Theta _1+2\pi -7 \rho )^3\\&=10\cdot \zeta ^2 ((4\rho (-4\Theta _1+2\pi -7 \rho )^2 + (-4\Theta _1+2\pi -7 \rho )^3). \end{aligned}$$

The result is a combination of $\pi ,\theta$ and $\rho$, which are classes of type $(1,1)+(0,2), (2,0)$ and (0, 2), respectively. Moreover, in the proof of Lemma 4.4 we computed $\pi = \rho + \gamma and \pi ^2 = \gamma ^2 = -2\rho \Theta _1.$ Hence, the only non-zero combinations are $\pi ^2\Theta _1 = -2\rho \Theta _1^2 = -2 \pi \Theta _1^2.$ We find

$$\begin{aligned}&10\cdot \zeta ^2 ((4\rho (-4\Theta _1+2\pi -7 \rho )^2 + (-4\Theta _1+2\pi -7 \rho )^3)\\&\quad = 10\cdot \zeta ^2 (2^6\rho \Theta _1^2 + 3(-2^4\pi ^2\Theta _1 + 2^5\pi \Theta _1^2 - 7\cdot 2^4\rho \Theta _1^2)\\&\quad = 10 (2^6+ 3(2^5 +2^5 - 7\cdot 2^4))\zeta ^2\rho \Theta _1^2 = -5^2 2^{5} \zeta ^2\rho \Theta _1^2. \end{aligned}$$

Finally, we want to show that $\int _{\overline{W}}\zeta ^2\rho \Theta ^2 = 2$. Indeed,

$$\begin{aligned} \int _{\overline{W}}\zeta ^2\rho \Theta ^2 =\tau _*\zeta ^2 \int _{{{\,\mathrm{Pic}\,}}^1}\Theta ^2\int _{C}\rho =2. \end{aligned}$$

$\square$

This concludes the proof of the proposition.

5 Proof of Theorem 1.2

In this section, we prove Theorem 1.2.

Theorem

The classes $[N_0]$ and $[N_1] \in H^{10}(M,\mathbb {Q})$ are linearly independent and span a totally isotropic subspace of $H^{10}(M,\mathbb {Q})$ with respect to the intersection pairing. They are given by

$$\begin{aligned}{}[N_0] = \frac{1}{48} [F] + \beta \ \text {and}\ [N_1] = \frac{5}{12} [F] - 4 \beta , \end{aligned}$$

where [F] is the class of a general fiber of the Mukai system and $0 \ne \beta \in (S^5H^2(M,\mathbb {Q}))^\bot$ satisfies $\beta ^2 = 0$. As $\deg _{u_1}\beta = 0$, the class $\beta$ is not effective.

From now on, all cohomology groups have $\mathbb {Q}$-coefficients.

Before coming to the proof, we want to point out, that the irreducible components over points in $\Sigma \setminus \Delta$ (see (3.1)) are of different cohomological nature. Let $D \in \Sigma \setminus \Delta$ be a reducible curve with two smooth components $C_1$ and $C_2$ meeting transversally. Then the two components $N'_1$ and $N'_2$ of $f^{-1}(D)$ contain an open sublocus parametrizing line bundles on D of bi-degree (2, 1) and (1, 2), respectively [6, Proposition 3.7.1 and Lemma 3.3.2]. The monodromy around $\Sigma \setminus \Delta$ exchanges $C_1$ and $C_2$ and consequently the classes of the irreducible components. We find

$$\begin{aligned}{}[N'_1] = [N'_2] = \tfrac{1}{2}[F]. \end{aligned}$$

In particular, the two components are linearly dependent. This is not true over $\Delta$.

Proposition 5.1

The classes $[N_0]$ and $[N_1] \in H^{10}(M)$ are linearly independent.

The proof uses the following simple observation.

Lemma 5.2

Let $M \rightarrow B$ be a Lagrangian fibration and F a smooth fiber. Then

$$\begin{aligned} c_i(\mathcal {T}_M)\!\left| _F\right. = 0\ \text {for all}\ i>0. \end{aligned}$$

Proof

We have a short exact sequence $0 \rightarrow \mathcal {T}_F \longrightarrow \mathcal {T}_M\!\left| _F\right. \longrightarrow \mathcal {N}_{F/M} \rightarrow 0.$ Now, $F \subset M$ is Lagrangian and hence $\mathcal {N}_{F/M} \cong \Omega _F$. Moreover, F is an abelian variety and hence all its Chern classes of degree greater than zero are trivial.

Proof of Proposition 5.1

Assume that $[N_0]$ and $[N_1]$ are linearly dependent. Then there is some $\lambda \in \mathbb {Q}$ such that $[F] = \lambda [N_0]$, where $F \subset M$ is a smooth fiber. In particular, by the above lemma, any product of $[N_0]$ and the Chern classes of M vanishes. However, we will show that

$$\begin{aligned} \int _M c_2(\mathcal {T}_M)\cdot u_1^3\cdot [N_0] \ne 0, \end{aligned}$$

leading to the desired contradiction. We have $c(\mathcal {T}_M\!\left| _{N_0}\right. ) = c(\mathcal {T}_{N_0})c(\Omega _{N_0})$ and thus

$$\begin{aligned} c_2(\mathcal {T}_M)\!\left| _{N_0}\right. = (2c_2 -c_1^2)(\mathcal {T}_{N_0}). \end{aligned}$$

Moreover, our computation will use the following two inputs. Let $\alpha \in H^2({{\,\mathrm{SM}\,}}_C(2,1))$ be the degree two Künneth component of $(c_1^2-c_2)(\mathcal {V}_\mathrm{univ})$ with $\mathcal {V}_\mathrm{univ}$ being a universal bundle on ${{\,\mathrm{SM}\,}}_C(2,1)\times C$. It is known, e.g. [26, §5A], that

$$\begin{aligned} \begin{array}{cccc} c_1(\mathcal {T}_{{{\,\mathrm{SM}\,}}_C(2,1)}) = 2\alpha ,&c_2(\mathcal {T}_{{{\,\mathrm{SM}\,}}_C(2,1)}) = 3\alpha ^2&\text {and}&\int _{{{\,\mathrm{SM}\,}}_C(2,1)}\alpha ^3 = 4. \end{array} \end{aligned}$$

Further, by [12, Théorème F] it is known that $\mathcal {O}_{{{\,\mathrm{SM}\,}}_C(2,1)}(-2\Theta ) \cong \omega _{{{\,\mathrm{SM}\,}}_C(2,1)}$. Hence,

$$\begin{aligned} \Theta = - \frac{1}{2} c_1(\omega _{{{\,\mathrm{SM}\,}}_C(2,1)}) = \frac{1}{2}c_1(\mathcal {T}_{{{\,\mathrm{SM}\,}}_C(2,1)}). \end{aligned}$$

This gives,

$$\begin{aligned} \int c_2(\mathcal {T}_M)u_1^3[N_0]&{{\mathop {=}\limits ^{4.10}}}&\int _{N_0}(2c_2-c_1^2)(\mathcal {T}_{N_0})\cdot (2\Theta )^3\\= & {} \frac{1}{2^4}\int _{{{\,\mathrm{SM}\,}}_C(2,1)\times {{\,\mathrm{Pic}\,}}^0}h^*((2c_2-c_1^2)(\mathcal {T}_{N_0})\cdot (2\Theta )^3)\\&{{\mathop {=}\limits ^{(4.1)}}}&\frac{2^3}{2^4}\int _{{{\,\mathrm{SM}\,}}_C(2,1)\times {{\,\mathrm{Pic}\,}}^0}p_1^*(2c_2-c_1^2)(\mathcal {T}_{{{\,\mathrm{SM}\,}}})\cdot (p_1^* \Theta _{{{\,\mathrm{SM}\,}}}+ 4 p_2^*\Theta _{0})^3\\= & {} \frac{1}{2} \int _{{{\,\mathrm{SM}\,}}_C(2,1)\times {{\,\mathrm{Pic}\,}}^0} p_1^*(2c_2-c_1^2)(\mathcal {T}_{{{\,\mathrm{SM}\,}}})\cdot (3p_1^* \Theta _{{{\,\mathrm{SM}\,}}}\cdot 4^2 p_2^*\Theta _{0}^2)\\= & {} 3\cdot 2^3 \int _{{{\,\mathrm{SM}\,}}_C(2,1)}(2c_2-c_1^2)(\mathcal {T}_{{{\,\mathrm{SM}\,}}})\cdot \frac{1}{2}c_1(\mathcal {T}_{{{\,\mathrm{SM}\,}}}) \int _{{{\,\mathrm{Pic}\,}}^0}\Theta _{0}^2\\= & {} 3 \cdot 2^4 \int _{{{\,\mathrm{SM}\,}}_C(2,1)}(6\alpha ^2-4\alpha ^2)\alpha = 3\cdot 2^7 \ne 0. \end{aligned}$$

Proof of Theorem 1.2

We set $V :=S^5H^2(M)\subset H^{10}(M)$ so that we have an orthogonal decomposition with respect to the cup product $H^{10}(M) = V \oplus V^\bot .$ Accordingly, we write $[N_i] = \alpha _i + \beta _i$ with $\alpha _i \in V$ and $0 \ne \beta _i \in V^\bot$ for $i=1,2$. We claim that

$$\begin{aligned} 20 [N_0] - [N_1]\in V^\bot . \end{aligned}$$

(5.1)

To see this, we decompose the second cohomology group into its transcendental and algebraic part, i.e. $H^2(M) = T(M) \oplus {{\,\mathrm{NS}\,}}(M)$. Now, for $i=1,2$ consider

$$\begin{aligned} T(M) \rightarrow H^{12}(M),\quad \alpha \mapsto \alpha \cdot [N_i]. \end{aligned}$$

(5.2)

As the symplectic form $\sigma \in T(M)$ vanishes on $N_i$, it follows by irreducibility of the Hodge structure T(M) that the assignment (5.2) is trivial. Hence, it suffices to show that $20 [N_0] - [N_1] \in (S^5{{\,\mathrm{NS}\,}}(M))^\bot$. By (4.2) any element in $S^5{{\,\mathrm{NS}\,}}(M)$ is of the form $x_1x_2\ldots x_5$, where $x_i = \lambda _{M}(2c_i.H,c_i,s_i)$. According to Propositions 4.10 and 4.13

$$\begin{aligned} \int [N_1]x_1x_2\ldots x_5 = -5^2 2^6\prod _{i=1}^5 c_i.H = 20 \int [N_0]x_1x_2\ldots x_5. \end{aligned}$$

This proves (5.1).

Next, we write $[N_1] - 20 [N_0] = \alpha _1 -20 \alpha _0 + \beta _1 - 20 \beta _0 \in V^\bot$ and conclude $\alpha _1 = 20 \alpha _0$. We set $\alpha = \alpha _0$. On the one hand, we have by Theorem 1.1

$$\begin{aligned} 2^3 [N_0] + 2[N_1] = [F] = u_0^5 \in V, \end{aligned}$$

but also

$$\begin{aligned} u_0^5 = 48\alpha + 8 \beta _0 + 2\beta _1. \end{aligned}$$

This gives $48\alpha = u_0^5$ and $\beta _1 = -4 \beta _0$. Setting $\beta = \beta _0$ gives the desired expression.

The last assertion follows from $[N_0]^2 = (\tfrac{1}{48}u_0^5 + \beta )^2 = \beta ^2$ and

$$\begin{aligned}{}[N_0]^2= \int _{N_0}c_5(\mathcal {N}_{N_0/M}) = \int _{N_0}c_5(\Omega _{N_0}) = - e(N_0), \end{aligned}$$

which is known to vanish, see [2, §9]. Hence $\beta ^2 =0$, which implies $[N_1]^2=0$ and finally, as $[F]^2=0$, also $[N_0]\cdot [N_1]=0$. $\square$