1 Introduction

Since the discovery of Yang–Mills instantons as topologically nontrivial field configurations that minimize the Yang–Mills action in four-dimensional Euclidean spacetime [14], many important developments on the applications of instantons arose in both physics [27, 120, 126] and mathematics [33, 43]. In the Atiyah–Drinfield–Hitchin–Manin (ADHM) construction [8], the moduli space of Yang–Mills instantons on \(\mathbb {R}^{4}\) is given as a hyperkahler quotient. In addition, the ADHM construction can be derived in a physically intuitive way using string theory [36, 37, 129]. For example, the moduli space \(\mathcal {M}_{n,k}\) of \(\mathrm {SU}(n)\) instantons of charge k is given by the Higgs branch of the supersymmetric gauge theory living on k D1-branes probing a stack of n coincident D5-branes in type IIB superstring theory. To avoid the noncompactness of \(\mathcal {M}_{n,k}\) due to small instantons, Nakajima introduced a smooth manifold \(\widetilde{\mathcal {M}}_{n,k}\), which can be obtained from the Uhlenbeck compactification of \(\mathcal {M}_{n,k}\) by resolving the singularities [86]. Thereafter Nekrasov and Schwarz interpreted \(\widetilde{\mathcal {M}}_{n,k}\) as the moduli space of \(\mathrm {U}(n)\) instantons on noncommutative \(\mathbb {R}^{4}\) [106], where the noncommutativity of the spacetime coordinates can be produced in string theory by turning on a proper constant B-field [123].

The moduli space \(\widetilde{\mathcal {M}}_{n,k}\) admits a \(\mathrm {U}(1)^{2}\) action which stems from the rotation symmetry of the spacetime \(\mathbb {R}^{4}\), and a \(\mathrm {U}(n)\) action which rotates the gauge orientation at infinity. Although \(\widetilde{\mathcal {M}}_{n,k}\) is still noncompact because the instantons can run away to infinity of the spacetime \(\mathbb {R}^{4}\), the \(\mathbf {T}\)-equivariant symplectic volume \(\mathcal {Z}_{k}\) of \(\widetilde{\mathcal {M}}_{n,k}\) is well-defined [82], with \(\mathbf {T}\) being the maximal torus of \(\mathrm {U}(1)^{2}\times \mathrm {U}(n)\). Using the equivariant localization theorem [7], \(\mathcal {Z}_{k}\) can be evaluated exactly and is given by a sum over a collection of random partitions. Assembling \(\mathcal {Z}_{k}\) with all \(k\ge 0\) into a generating function, Nekrasov obtained the instanton partition function \(\mathcal {Z}=\sum _{k\ge 0}{} \texttt {q}^{k}\mathcal {Z}_{k}\) of four-dimensional \(\mathcal {N}=2\) \(\mathrm {SU}(n)\) supersymmetric Yang–Mills theory in the \(\varOmega \)-background [110]. It turns out that both the Seiberg–Witten effective prepotential [121, 122] and the couplings to the background gravitational fields [5, 18] can be derived rigorously from \(\mathcal {Z}\) [76, 89, 99, 101, 107, 134]. The instanton partition function is also related to the A-model topological strings on two-dimensional Riemann surfaces [75, 77, 112, 133], the Virasoro/W-algebra conformal blocks [3, 132], and quantum integrable systems [102, 105, 113]. Recently, based on the computation of the elliptic genus using supersymmetric localization techniques [16, 17], an alternative general approach to computing \(\mathcal {Z}\) was provided in [56]. The major advantage of this approach is that we no longer need to know \(\widetilde{\mathcal {M}}_{n,k}\) explicitly, and \(\mathcal {Z}\) is given in terms of contour integrals with Jeffrey–Kirwan (JK) residue prescription [58].

Over the past few years, there have been several fascinating generalizations of the Yang–Mills instantons on \(\mathbb {R}^{4}\). The ADHM-type constructions of Yang–Mills instantons on some other four-manifolds have been found [24, 38, 68, 71, 131]. Instantons also appear in higher-dimensional gauge theories [1, 11, 35], and we can get an ADHM-type construction of instanon moduli spaces from the low-energy worldvolume theory on \(\hbox {D}1\)-branes probing \(\hbox {D}(2p+1)\)-branes with \(p=3,4\) [81]. The instanton partition function \(\mathcal {Z}\) is given by a statistical sum over random plane partitions (\(p=3\)) or solid partitions (\(p=4\)), see [64] for a recent review. The \(p=3\) case gives the equivariant Donaldson–Thomas invariants of toric Calabi–Yau threefolds [10, 15, 25, 26, 40, 78,79,80], while the \(p=4\) case defines the magnificent four model [95, 103], and can be interpreted in terms of equivariant Donaldson–Thomas invariants of toric Calabi–Yau fourfolds [20,21,22]. The partition function of the magnificent four model is envisioned to be the mother of all instanton partition functions [95].

In yet a different line of research, the concept of generalized field theory, which is constructed by merging several ordinary field theories across defects, has been emerging in recent years. The spacetime X of a generalized gauge theory contains several intersecting components, \(X=\bigcup _{A}X_{A}\). The fields and the gauge groups \(G_{A}=\left. G\right| _{A}\) on different components can be different, and the matter fields living on the intersection \(X_{A}\bigcap X_{B}\) transform in the bifundamental representation of the product group \(G_{A}\times G_{B}\). For instance, \(\hbox {D}1\)-branes probing a configuration of intersecting (anti-)\(\hbox {D}5\)-branes, in the presence of a proper B-field, give rise to the spiked instantons in a generalized gauge theory [92, 93, 104, 119]. When each component \(X_{A}\) of the spacetime is a noncompact toric surface, the generating function of equivariant symplectic volumes of the instanton moduli spaces can be similarly defined and is called the gauge origami partition function [96]. Applying the equivariant localization theorem, the gauge origami partition function can be expressed as a statistical sum over collections of random partitions, and provides a unified treatment of instanton partition functions of four-dimensional \(\mathcal {N}=2\) supersymmetric gauge theories [101], possibly with local or surface defects [2, 4, 9, 46, 65, 69, 97, 118]. Nekrasov also derived an infinite set of nonperturbative Dyson–Schwinger equations from the gauge origami partition function [92], leading to a number of interesting applications [59, 60, 62, 63, 72, 94, 108].

It is the goal of the present work to piece together the jigsaw puzzle of instantons by studying \(\hbox {D}1\)-branes probing a configuration of intersecting \(\hbox {D}7\)-branes. With a proper B-field, the ground state of the D-brane system is supersymmetric, and the low-energy theory on \(\hbox {D}1\)-branes preserves \(\mathcal {N}=\left( 0,2\right) \) supersymmetry in two dimensions. Since the arrangement of various D-branes and open strings attached to them can be naturally associated with the vertices, edges and faces of a tetrahedron, we will call these instantons the tetrahedron instantons. We carefully work out the moduli space of tetrahedron instantons, which can be viewed as an interpolation between other instanton moduli spaces that have been explored extensively. It is also a generalization of the moduli space of solutions to the Donaldson–Uhlenbeck–Yau equations [34, 124], which describe the BPS configurations in higher dimensional super-Yang–Mills theory [1, 11, 35]. We compute the instanton partition function \(\mathcal {Z}\) using two approaches. Furthermore, we show that \(\mathcal {Z}\) admits a free field representation, suggesting the existence of a novel kind of symmetry which acts on the cohomology of the moduli spaces of tetrahedron instantons.

The paper is organized as follows. In Sect. 2 we provide a string theory construction of tetrahedron instantons and work out the instanton moduli spaces. In Sect. 3, we describe the tetrahedron instantons in the framework of noncommutative field theory. In Sect. 4 we analyze the moduli space of tetrahedron instantons. In Sect. 5, we compute the instanton partition function of the tetrahedron instantons using the equivariant localization theorem. In Sect. 6, we calculate the instanton partition function of the tetrahedron instantons from the elliptic genus of the worldvolume theory on the \(\hbox {D}1\)-branes probing a configuration of intersecting \(\hbox {D}7\)-branes, and match it with the equivariant localization computation. In Sect. 7, we give the free field representation of the instanton partition function. We conclude in Sect. 8 with some comments and a list of interesting open questions. We have included several appendices with relevant background material.

2 Tetrahedron Instantons from String Theory

We begin our discussion by describing the realization of tetrahedron instantons from string theory, as this is the most natural setting they can be constructed.

Fig. 1
figure 1

A tetrahedron with the sets \(\underline{4}\) and \(\underline{4}^{\vee }\) associated with vertices and faces, respectively. Each vertex is labeled by \(a\in \underline{4}\) and represents a complex plane \(\mathbb {C}_{a}\). The edge connecting two vertices labeled by a and b represents a complex two-plane \(\mathbb {C}_{ab}^{2}=\mathbb {C}_{a}\times \mathbb {C}_{b}\). The face \(A=\left( abc\right) \in \underline{4}^{\vee }\) has three vertices a, b, and c, and represents a complex three-plane \(\mathbb {C}_{A}^{3}=\prod _{a\in A}\mathbb {C}_{a}\). The remaining vertex that is not in the face \(A\in \underline{4}^{\vee }\) is denoted by \(\check{A}\in \underline{4}\)

Let us identify the ten-dimensional spacetime \(\mathbb {R}^{1,9}\) with \(\mathbb {R}^{1,1}\times \mathbb {C}^{4}\) by choosing a complex structure on \(\mathbb {R}^{8}\). We take the coordinates on \(\mathbb {R}^{1,1}\) to be \(x^{0},x^{9}\). The set of coordinate labels of four complex planes is denoted by

$$\begin{aligned} \underline{4}=\left\{ 1,2,3,4\right\} , \end{aligned}$$
(1)

with the complex coordinate on \(\mathbb {C}_{a}\subset \mathbb {C}^{4}\) being \(z_{a}=x^{2a-1}+\mathrm {i}x^{2a}\). There are four complex three-planes, \(\mathbb {C}_{A}^{3}=\prod _{a\in A}\mathbb {C}_{a}\subset \mathbb {C}^{4}\) for \(A\in \underline{4}^{\vee }\), where

$$\begin{aligned} \underline{4}^{\vee }=\begin{pmatrix}\underline{4}\\ 3 \end{pmatrix}=\left\{ \left( 123\right) ,\left( 124\right) ,\left( 134\right) ,\left( 234\right) \right\} . \end{aligned}$$
(2)

For each \(A\in \underline{4}^{\vee }\), we define

$$\begin{aligned} \check{A}=\left\{ \left. a\in \underline{4}\right| a\notin A\right\} . \end{aligned}$$
(3)

It is beneficial to introduce a tetrahedron (see Fig. 1) to visualize the above data.

The tetrahedron instantons can be realized by k \(\hbox {D}1\)-branes along \(\mathbb {R}^{1,1}\) probing a system of \(n_{A}\) \(\hbox {D}7_{A}\)-branes along \(\mathbb {R}^{1,1}\times \mathbb {C}_{A}^{3}\) for \(A\in \underline{4}^{\vee }\) in type IIB superstring theory. We summarize the configuration of D-branes in Table 1. This can also be visualized by a tetrahedron \(\mathcal {T}_{{\mathbf {n}},k}\), where \(n_{A}\) \(\hbox {D}7_{A}\)-branes sit on the face A, and k \(\hbox {D}1\)-branes can move on the surface of the tetrahedron. Here we denote

$$\begin{aligned} \mathbf {n}=\left( n_{A}\right) _{A\in \underline{4}^{\vee }}. \end{aligned}$$
(4)

We denote the Chan–Paton spaces of the \(\hbox {D}1\)-branes and \(\hbox {D}7_{A}\)-branes by vector spaces \(\mathbf {K}\) and \(\mathbf {N}_{A}\), respectively. The presence of the D-branes breaks the ten-dimensional Lorentz group \(\mathrm {SO}\left( 1,9\right) \) down to \(\mathrm {SO}\left( 1,1\right) _{09}\times \prod _{a\in \underline{4}}\mathrm {SO}\left( 2\right) _{a}\), where \(\mathrm {SO}\left( 1,1\right) _{09}\) is the two-dimensional Lorentz group of \(\mathbb {R}^{1,1}\), and \(\mathrm {SO}\left( 2\right) _{a}\) rotates the plane \(\left( x^{2a-1},x^{2a}\right) \).

Table 1 Tetrahedron instantons constructed from \(\hbox {D}1\)-branes probing intersecting \(\hbox {D}7\)-branes in type IIB superstring theory

2.1 Condition for unbroken supersymmetry

Let \(Q_{L}\) and \(Q_{R}\) be the supercharges which originate from the left- and right-moving worldsheet degrees of freedom. They are Majorana–Weyl spinors of the same chirality,

$$\begin{aligned} \Gamma _{c}Q_{L}=Q_{L},\quad \Gamma _{c}Q_{R}=Q_{R}, \end{aligned}$$
(5)

where

$$\begin{aligned} \Gamma _{c}=\Gamma ^{0}\Gamma ^{1}\cdots \Gamma ^{9},\quad \Gamma _{c}^{2}=1. \end{aligned}$$
(6)

A set of parallel \(\hbox {D}p\)-branes along \(x^{0},x^{i_{1}},\cdots ,x^{i_{p}},i_{1}<\cdots <i_{p}\) preserves a linear combination \(\epsilon _{L}Q_{L}+\epsilon _{R}Q_{R}\) of the supercharges with

$$\begin{aligned} \epsilon _{R}=\Gamma ^{0}\Gamma ^{i_{1}}\cdots \Gamma ^{i_{p}}\epsilon _{L}. \end{aligned}$$
(7)

Hence, the presence of both the \(\hbox {D}1\)-branes and the \(\hbox {D}7_{A}\)-branes imposes a constraint on the preserved supercharges,

$$\begin{aligned} \Gamma _{A}\epsilon _{L}=\epsilon _{L}, \end{aligned}$$
(8)

where \(\Gamma _{A}=\Gamma ^{2a-1}\Gamma ^{2a}\Gamma ^{2b-1}\Gamma ^{2b}\Gamma ^{2c-1}\Gamma ^{2c}\) for \(A=\left( abc\right) \). The Eq. (8) has no nonzero solutions, since \(\Gamma _{A}^{2}=-1\). Hence, this configuration is not supersymmetric.

We can also reach the conclusion that supersymmetry is completely broken in the absence of a B-field by inspecting the ground state energy. As reviewed in Appendix A, the zero-point energy in the Ramond sector is always zero due to worldsheet supersymmetry, while that in the Neveu–Schwarz sector is given by

$$\begin{aligned} E^{(0)}=-\frac{1}{2}+\frac{\kappa }{8}, \end{aligned}$$
(9)

where \(\kappa \) is the number of DN and ND directions. For the D1–D1 and D7\(_{A}\)–D\(7_{A}\) strings, \(\kappa =0\) and \(E^{(0)}=-\frac{1}{2}\). This state is tachyonic and is killed by the GSO projection. The physical ground state that survives the GSO projection has zero energy and therefore is supersymmetric. For the D1–D\(7_{A}\) strings, \(\kappa =6\) and \(E^{(0)}=\frac{1}{4}\), indicating that the ground state is stable but not supersymmetric.

The condition for unbroken supersymmetry is modified in the presence of a nonzero Neveu–Schwarz B-field [123]. We turn on a constant B-field along \(\mathbb {C}^{4}\) in the canonical form,

$$\begin{aligned} B=\sum _{a\in \underline{4}}b_{a}dx^{2a-1}\wedge dx^{2a},\quad b_{a}\in \mathbb {R}, \end{aligned}$$
(10)

and define

$$\begin{aligned} e^{2\pi \mathrm {i}v_{a}}=\frac{1+\mathrm {i}b_{a}}{1-\mathrm {i}b_{a}},\quad -\frac{1}{2}<v_{a}<\frac{1}{2}. \end{aligned}$$
(11)

Then the constraint (8) is modified to be

$$\begin{aligned} \exp \left( \sum _{a\in A}\vartheta _{a}\Gamma ^{2a-1}\Gamma ^{2a}\right) \epsilon _{L}=\epsilon _{L}, \quad \vartheta _{a}=\pi \left( v_{a}+\frac{1}{2}\right) \in \left( 0,\pi \right) . \end{aligned}$$
(12)

There are also conditions for unbroken supersymmetry due to the \(\hbox {D}7_{acd}\)- and the \(\hbox {D}7_{bcd}\)-branes,

$$\begin{aligned} \exp \left( \vartheta _{a}\Gamma ^{2a-1}\Gamma ^{2a}\right) \epsilon _{L} =\exp \left( \vartheta _{b}\Gamma ^{2b-1}\Gamma ^{2b}\right) \epsilon _{L},\quad a\ne b\in \underline{4}. \end{aligned}$$
(13)

If we label the 32 components of the ten-dimensional supercharges by the eigenvalues \(\left( s_{0},s_{1},s_{2},s_{3},s_{4}\right) \) of

$$\begin{aligned} \left( \Gamma ^{0}\Gamma ^{9},-\mathrm {i}\Gamma ^{1}\Gamma ^{2},-\mathrm {i} \Gamma ^{3}\Gamma ^{4},-\mathrm {i}\Gamma ^{5}\Gamma ^{6},-\mathrm {i}\Gamma ^{7}\Gamma ^{8}\right) , \end{aligned}$$
(14)

with \(s_{i}\in \left\{ \pm 1\right\} \), the preserved supercharges obey

$$\begin{aligned} \exp \left( \mathrm {i}s_{a}\vartheta _{a}+\mathrm {i}s_{b}\vartheta _{b}+\mathrm {i}s_{c}\vartheta _{c}\right)= & {} 1,\quad \forall \left( abc\right) \in \underline{4}^{\vee }, \end{aligned}$$
(15)
$$\begin{aligned} \exp \left( \mathrm {i}s_{a}\vartheta _{a}-\mathrm {i}s_{b}\vartheta _{b}\right)= & {} 1,\quad \forall a\ne b\in \underline{4}, \end{aligned}$$
(16)

whose solutions are

$$\begin{aligned} \vartheta _{1}= & {} \vartheta _{2}=\vartheta _{3}=\vartheta _{4} = \frac{2\pi }{3},\nonumber \\ s_{1}= & {} s_{2}=s_{3}=s_{4} = \pm \,1,\quad s_{0}=+\,1. \end{aligned}$$
(17)

Hence, when the B-field is chosen to be

$$\begin{aligned} b_{1}=b_{2}=b_{3}=b_{4}=\tan \frac{\pi }{6}=\frac{1}{\sqrt{3}}, \end{aligned}$$
(18)

or equivalently,

$$\begin{aligned} v_{1}=v_{2}=v_{3}=v_{4}=\frac{1}{6}, \end{aligned}$$
(19)

there are two preserved supercharges, which are right-moving supercharges \(\mathcal {Q}_{+}\) and \(\bar{\mathcal {Q}}_{+}\) from the viewpoint of the common two-dimensional intersection \(\mathbb {R}^{1,1}\).

One should keep in mind that the condition (18) for unbroken supersymmetry is valid for the original string theory vacuum. However, a nonzero constant B-field away from the supersymmetric locus in (18) can introduce instability in the form of open-string tachyons. After tachyon condensation, the system may roll from the original unstable and nonsupersymmetric vacuum down to a nearby supersymmetric vacuum [104, 123, 130]. Indeed, this phenomenon happens and plays an essential role in our system.

2.2 Low-energy spectrum

We are interested in the low-energy spectrum of open strings ending on D-branes. Since the system preserves \(\mathcal {N}=\left( 0,2\right) \) supersymmetry in two dimensions when (18) is satisfied, it is convenient to organize fields obtained from the quantization of open strings in terms of two-dimensional \(\mathcal {N}=\left( 0,2\right) \) supermultiplets. For simplicity, we assume that the B-field can at most be a small deviation from the locus (19),

$$\begin{aligned} v_{a}=\frac{1}{6}+\tilde{v}_{a},\quad \left| \tilde{v}_{a}\right| \ll 1. \end{aligned}$$
(20)

As reviewed in Appendix A, we can have Neumann (N), Dirichlet (D), and twisted (T) boundary conditions at the two ends of the open strings.

2.2.1 D1–D1 strings

The D1–D1 strings satisfy NN boundary conditions along \(\mathbb {R}^{1,1}\) and DD boundary conditions along \(\mathbb {C}^{4}\).

In the Ramond sector, the zero-point energy vanishes. There are ten zero modes, giving rise to 32 degenerate ground states \(\left| s_{0},s_{1},s_{2},s_{3},s_{4}\right\rangle _{\mathrm {R}}\) which form a representation of the gamma matrix algebra in \(\mathbb {R}^{1,9}\). After the GSO projection, we keep half of the ground states, which become eight left-moving fermions and eight right-moving fermions in two dimensions. They transform under \({\mathrm {Spin}}(8)\cong {\mathrm {SU}}(2)_{-}\times {\mathrm {SU}}(2)_{+} \times {\mathrm {SU}}(2)_{-}^{\prime }\times {\mathrm {SU}}(2)_{+}^{\prime }\) as \(\left( \mathbf {1},\mathbf {2},\mathbf {1},\mathbf {2}\right) \oplus \left( \mathbf {1},\mathbf {2}, \mathbf {2},\mathbf {1}\right) \oplus \left( \mathbf {2},\mathbf {1},\mathbf {1},\mathbf {2}\right) \oplus \left( \mathbf {2},\mathbf {1},\mathbf {2},\mathbf {1}\right) \).

In the Neveu–Schwarz sector, the ground state \(\left| 0\right\rangle _{\mathrm {NS}}\) is unique and has zero-point energy \(E^{(0)}=-\frac{1}{2}\). This tachyonic mode is eliminated by the GSO projection. The excited states \(b_{-\frac{1}{2}}^{\mu }\left| 0\right\rangle _{\mathrm {NS}}\) have zero energy and survive the GSO projection. In the light-cone gauge, they give rise to eight real scalar fields for \(\mu =1,\cdots ,8\). These scalar fields describe the positions of the \(\hbox {D}1\)-branes in \(x^{1},\cdots ,x^{8}\), and transform in the vector representation of \({\mathrm {Spin}}(8)\). We can combine them into four complex scalars \(B_{a},a\in \underline{4}\) and their complex conjugates.

The worldvolume theory on k coincident \(\hbox {D}1\)-branes is the two-dimensional \(\mathcal {N}=\left( 8,8\right) \) super-Yang–Mills theory with gauge group \(\mathrm {U}\left( k\right) \), which is the dimensional reduction of the ten-dimensional \(\mathcal {N}=1\) \(\mathrm {U}\left( k\right) \) super-Yang–Mills theory.

2.2.2 D1–D7 strings

The boundary conditions of D1–D\(7_{A}\) strings are NN along \(\mathbb {R}^{1,1}\), DT along \(\mathbb {C}_{A}^{3}\), and DD along \(\mathbb {C}_{\check{A}}\).

In the Ramond sector, the zero-point energy vanishes. In the light-cone gauge, we have two zero modes from worldsheet fermions along \(\mathbb {C}_{\check{A}}\). Quantization of these zero modes leads to a pair of massless states with \(s_{\check{A}}=\pm \,1\). The GSO projection kills one of them.

In the Neveu–Schwarz sector, the ground state has zero-point energy

$$\begin{aligned} E^{(0)}=\frac{1}{4}-\frac{1}{2}\sum _{a\in A}\left| v_{a}\right| =-\frac{1}{2}\sum _{a\in A}\tilde{v}_{a}. \end{aligned}$$
(21)

We should consider three different cases:

  1. 1.

    When \(E^{(0)}>0\), the ground state is stable, but supersymmetry is broken.

  2. 2.

    When \(E^{(0)}=0\), the ground state is supersymmetric. It is unique since there are no worldsheet zero modes to be quantized. It survives the GSO projection, and gives rise to a real scalar field which transforms as a singlet under the \({\mathrm {Spin}}(8)\) group. Combining with the similar state of D\(7_{A}\)–D1 strings and the fermions from the Ramond sectors, we get a massless \(\mathcal {N}=\left( 2,2\right) \) chiral multiplet, transforming as \(\left( \mathbf {k},\overline{\mathbf {n}_{A}}\right) \) under the \(\mathrm {U}\left( k\right) \times \mathrm {U}\left( n_{A}\right) \) symmetry.

  3. 3.

    When \(E^{(0)}<0\), the ground state is tachyonic and unstable. The lowest excited states are obtained by acting the fermionic creation operators on it. For small \(\tilde{v}_{a}\), all of these excited states have positive energy, and it is reasonable to neglect them when we study the low-energy theory.

2.2.3 D7–D7 strings

The analysis of D\(7_{A}\)–D\(7_{A}\) strings is similar to that of D1–D1 strings, and we will get the dimensional reduction of the ten-dimensional \(\mathcal {N}=1\) \(\mathrm {U}\left( n_{A}\right) \) supersymmetric Yang–Mills theory. The worldsheet bosons have position and momentum zero modes along \(\mathbb {R}^{1,1}\times \mathbb {C}_{A}^{3}\). Hence the result is the eight-dimensional \(\mathrm {U}\left( n_{A}\right) \) supersymmetric Yang–Mills theory with sixteen supercharges on \(\mathbb {R}^{1,1}\times \mathbb {C}_{A}^{3}\).

On the other hand, the boundary conditions of \(\hbox {D}7_{acd}\)\(\hbox {D}7_{bcd}\) strings are NN along \(\mathbb {R}^{1,1}\), TD along \(\mathbb {C}_{a}\), DT along \(\mathbb {C}_{b}\), and TT along \(\mathbb {C}_{c}\) and \(\mathbb {C}_{d}\). The worldsheet bosons have position and momentum zero modes along \(\mathbb {R}^{1,1}\times \mathbb {C}_{c}\times \mathbb {C}_{d}\).

In the Ramond sector, the zero-point energy vanishes. In the light-cone gauge, we have four zero modes from worldsheet fermions along \(\mathbb {C}_{c}\) and \(\mathbb {C}_{d}\). Quantization of these zero modes leads to four massless states with \(\left( s_{c},s_{d}\right) =\left( \pm \,1,\pm \,1\right) \). The GSO projection kills two states with \(\left( s_{c},s_{d}\right) =\left( +1,-1\right) ,\left( -1,+1\right) \), and two states with \(\left( s_{c},s_{d}\right) =\left( +1,+1\right) ,\left( -1,-1\right) \) survive.

In the Neveu–Schwarz sector, the ground state has zero-point energy \(E^{(0)}=-\frac{1}{2}\left( \left| v_{a}\right| +\left| v_{b}\right| \right) \), and the lowest excited states increase the energy by \(\left| v_{a}\right| \) and \(\left| v_{b}\right| \). Thus, the energy of the first four states are \(\frac{1}{2}\left( \pm \, v_{a}\pm \, v_{b}\right) \). The states that survive the GSO projection have energy \(\pm \,\frac{1}{2}\left( v_{a}-v_{b}\right) =\pm \,\frac{1}{2}\left( \tilde{v}_{a}-\tilde{v}_{b}\right) \). Combining with the similar states of \(\hbox {D}7_{bcd}\)\(\hbox {D}7_{acd}\) strings, we get two complex scalar fields, which are massless when \(\tilde{v}_{a}=\tilde{v}_{b}.\) All the other excited states can be ignored in the low-energy theory when \(\left| \tilde{v}_{a}\right| ,\left| \tilde{v}_{b}\right| \ll 1\).

Combining states in the Ramond sectors and those in the Neveu–Schwarz sector for both \(\hbox {D}7_{acd}\)\(\hbox {D}7_{bcd}\) strings and \(\hbox {D}7_{bcd}\)\(\hbox {D}7_{acd}\) strings, we get a four-component Weyl spinor and two complex scalar fields, which are component fields of a six-dimensional \(\mathcal {N}=(1,0)\) hypermultiplet on \(\mathbb {R}^{1,1}\times \mathbb {C}_{c}\times \mathbb {C}_{d}\). These fields transform in the bifundamental representation \(\left( \mathbf {n}_{acd},\overline{\mathbf {n}_{bcd}}\right) \) under \(\mathrm {U}\left( n_{acd}\right) \times \mathrm {U}\left( n_{bcd}\right) \).

2.3 Tetrahedron instantons from the supersymmetric vacua

We are now ready to write down the low-energy worldvolume theory on \(\hbox {D}1\)-branes probing a configuration of intersecting \(\hbox {D}7\)-branes in the presence of a nonzero constant B-field. Our goal is to find the stable ground state of the low-energy theory. Since the \(\hbox {D}7\)-branes are heavy from the point of view of \(\hbox {D}1\)-branes, the degrees of freedom supported on them are frozen to their classical expectation values. Therefore, the \(\mathrm {U}\left( n_{A}\right) \) symmetry from \(\hbox {D}7_{A}\)-branes will be treated as a global symmetry.

Table 2 Field content from D1–D1 and D1–D7\(_{A}\) open strings in terms of \(\mathcal {N}=\left( 2,2\right) \) and \(\mathcal {N}=\left( 0,2\right) \) supermultiplets

The low-energy worldvolume theory on D1-branes probing a single stack of \(\hbox {D}7_{A}\)-branes with a constant B-field was obtained in [130]. The field content is summarized in Table 2. In addition to the standard kinetic terms, the theory has a superpotential

$$\begin{aligned} \mathcal {W}=\frac{1}{6}\epsilon ^{abc}{\mathrm {Tr}}\varPhi _{a}\left[ \varPhi _{b},\varPhi _{c}\right] ,\quad A=\left( abc\right) , \end{aligned}$$
(22)

and a Fayet-Iliopoulos term with coupling

$$\begin{aligned} r=\left( \sum _{a\in A}v_{a}\right) -\frac{1}{2}. \end{aligned}$$
(23)

We can deduce from the analysis in Sect. 2.1 that there are four preserved supercharges specified by

$$\begin{aligned} s_{a}=s_{b}=s_{c}=\pm \,1,\quad s_{\breve{A}}=\pm \,1,\quad s_{0}=\pm \,1,\quad s_{a}s_{\breve{A}}s_{0}=1. \end{aligned}$$
(24)

The theory at the classical level has a \(\mathrm {U}(1)_{\mathcal {R}}\times \mathrm {U}(1)_{\breve{A}}\) R-symmetry, where the \(\mathrm {U}(1)_{\mathcal {R}}\) (\(\mathrm {U}(1)_{\breve{A}}\)) symmetry comes from rotations of \(\mathbb {C}_{A}^{3}\) and \(\mathbb {C}_{\breve{A}}\) in the same (opposite) directions.

The bound states of \(\hbox {D}1\)\(\hbox {D}7_{A}\)-branes and those of \(\hbox {D}1\)\(\hbox {D}7_{B}\)-branes for \(A\ne B\) share the common \(\mathrm {U}(1)_{\mathcal {R}}\) R-symmetry, but the \(\mathrm {U}(1)_{\breve{A}}\) symmetry and the \(\mathrm {U}(1)_{\breve{B}}\) symmetry are different. Accordingly, only an \(\mathcal {N}=\left( 0,2\right) \) supersymmetry will be preserved if we have four stacks of \(\hbox {D}7\)-branes. In terms of the two-dimensional \(\mathcal {N}=\left( 0,2\right) \) superspace, the Lagrangian of the low-energy worldvolume theory is

$$\begin{aligned} \mathcal {L}= & {} \int d\theta ^{+}d\bar{\theta }^{+}{\mathrm {Tr}}\left( \frac{1}{2e^{2}}\bar{\varUpsilon }\varUpsilon -\frac{\mathrm {i}}{2}\sum _{a\in \underline{4}}\bar{\varPhi }_{a}\mathcal {D}_{-}\varPhi _{a}-\frac{1}{2}\sum _{i=1}^{3}\bar{\varPsi }_{i,-}\varPsi _{i,-}\right) \nonumber \\&-\frac{1}{\sqrt{2}}{\mathrm {Tr}}\left( \int d\theta ^{+}\left. \sum _{i=1}^{3}\varPsi _{-,i}J^{i}\right| _{\bar{\theta }^{+}=0}+c.c.\right) +\left( \frac{\mathrm {i}r}{2}\int d\theta ^{+}\left. \varUpsilon \right| _{\bar{\theta }^{+}=0}+\mathrm {c.c.}\right) \nonumber \\&-\frac{1}{2}{\mathrm {Tr}}\sum _{A\in \underline{4}^{\vee }}\left( \mathrm {i}\bar{\varPhi }_{A}\mathcal {D}_{-}\varPhi _{A}+\bar{\varPsi }_{A,-}\varPsi _{A,-}\right) , \end{aligned}$$
(25)

where

$$\begin{aligned} J^{i}=\frac{1}{2}\epsilon ^{iab4}\left[ \varPhi _{a},\varPhi _{b}\right] ,\quad E_{i}=\left[ \varPhi _{4},\varPhi _{i}\right] ,\quad E_{A}=\varPhi _{\breve{A}}\varPhi _{A}. \end{aligned}$$
(26)

We also need to impose a consistency condition on the B-field,

$$\begin{aligned} v_{1}=v_{2}=v_{3}=v_{4}=\frac{1}{6}+\frac{r}{3}, \end{aligned}$$
(27)

so that all the fields \(I_{A}\) have the same mass, which may be real or imaginary depending on the sign of the parameter r. This also avoid tachyons from the quantization of the \(\hbox {D}7\)\(\hbox {D}7\) open strings. Integrating out the auxiliary fields, we obtain the scalar potential of (25),

$$\begin{aligned} V= & {} {\mathrm {Tr}}\left( \sum _{a\in \underline{4}}\left[ B_{a},B_{a}^{\dagger }\right] +\sum _{A\in \underline{4}^{\vee }}I_{A}I_{A}^{\dagger }-r\cdot \mathbb {1}_{\mathrm {U}\left( k\right) }\right) ^{2}\nonumber \\&+\sum _{A\in \underline{4}^{\vee }}{\mathrm {Tr}}\left| B_{\breve{A}}I_{A}\right| ^{2}+\sum _{a<b\in \underline{4}}{\mathrm {Tr}}\left| \left[ B_{a},B_{b}\right] \right| ^{2}. \end{aligned}$$
(28)

Since the scalar potential \(V\ge 0\), the ground state is always stable. The original string theory vacuum is given by \(B_{a}=0\), \(I_{A}=0\). For \(r<0\), this vacuum has positive energy, and the supersymmetry is spontaneously broken. For \(r=0\), the original string theory vacuum preserves supersymmetry. For \(r>0\), the original string theory vacuum is not supersymmetric and does not give the global minimum of V. However, the system restore supersymmetry after transitioning to a nearby vacuum via tachyon condensation. Moreover, the theory has a family of classical vacua, and the classical moduli space \(\mathfrak {M}_{{\mathbf {n}},k}\) is given by the space of solutions to \(V=0\) modulo the gauge symmetry \(\mathrm {U}\left( k\right) \),

(29)

where

$$\begin{aligned} \mathbf {B}=\left( B_{a}\right) _{a\in \underline{4}},\quad \mathbf {I}=\left( I_{A}\right) _{A\in \underline{4}^{\vee }}. \end{aligned}$$
(30)

We will call \(\mathfrak {M}_{{\mathbf {n}},k}\) the moduli space of tetrahedron instantons in the generalized gauge theory on \(\bigcup _{A\in \underline{4}^{\vee }}\mathbb {C}_{A}^{3}\) with gauge groups \(\left. G\right| _{A}=\mathrm {U}\left( n_{A}\right) \) and instanton number k.

3 Tetrahedron Instantons in Noncommutative Field Theory

As shown in [123], the dynamics of open strings connecting D-branes in the presence of a strong constant B-field can be described by a noncommutative gauge theory. The noncommutative deformation is advantageous since the position-space uncertainty smooths out the singularities in the conventional field theory, and it allows us to treat uniformly the worldvolume theories of D-branes of various dimensions [39]. It also provides a natural framework for the description of generalized gauge theories. In the section, we will construct tetrahedron instantons as particular solutions of general instanton equations in noncommutative field theory, to put it in perspective.

3.1 General instanton equations

We deform the ten-dimensional space \(\mathbb {R}^{1,1}\times \mathbb {C}^{4}\) to the noncommutative space \(\mathbb {R}^{1,1}\times \mathbb {C}_{\varTheta }^{4}\), where the coordinates of \(\mathbb {C}_{\varTheta }^{4}\) satisfy the commutation relations

$$\begin{aligned} \left[ z_{a},z_{b}\right] =\left[ \bar{z}_{a},\bar{z}_{b}\right] =0,\quad \left[ z_{a},\bar{z}_{b}\right] =-2\varTheta _{a}\delta _{ab},\quad a,b\in \underline{4}, \end{aligned}$$
(31)

and the coordinates of \(\mathbb {R}^{1,1}\) remain commutative. The coordinates of \(\mathbb {C}_{\varTheta }^{4}\) are not simultaneously diagonalizable. We introduce the creation and annihilation operators,

$$\begin{aligned} c_{a}^{\dagger }=\frac{z_{a}}{\sqrt{2\varTheta _{a}}},\quad c_{a}=\frac{\bar{z}_{a}}{\sqrt{2\varTheta _{a}}},\quad \left[ c_{a},c_{b}^{\dagger }\right] =\delta _{ab}, \end{aligned}$$
(32)

and replace the underlying spacetime manifold by the Fock module,

$$\begin{aligned} \mathcal {H}_{1234}=\mathbb {C}\left[ c_{1}^{\dagger },c_{2}^{\dagger },c_{3}^{\dagger },c_{4}^{\dagger }\right] \left| \mathbf {0}\right\rangle =\bigoplus _{\mathbf {\mathfrak {N}}\in \mathbb {Z}_{\ge 0}^{\otimes 4}}\mathbb {C}\left| \mathbf {\mathfrak {N}}\right\rangle , \end{aligned}$$
(33)

where \(\left| \mathbf {0}\right\rangle \) is the Fock vacuum defined by

$$\begin{aligned} c_{a}\left| \mathbf {0}\right\rangle =0,a\in \underline{4}, \end{aligned}$$
(34)

and \(\mathbf {\mathfrak {N}}=\left( \mathfrak {N}_{1},\mathfrak {N}_{2},\mathfrak {N}_{3},\mathfrak {N}_{4}\right) \). We define \(\mathcal {H}_{S}\) for a set \(S\subset \left\{ 1,2,3,4\right\} \) to be the Fock module that can be obtained from \(\mathcal {H}_{1234}\) by setting \(\mathfrak {N}_{a}=0\) for all \(a\notin S\). We denote

$$\begin{aligned} \mathfrak {N}=\sum _{a\in \underline{4}}\mathfrak {N}_{a}. \end{aligned}$$
(35)

The creation and annihilation operators satisfy

$$\begin{aligned} c_{a}\left| \cdots ,\mathfrak {N}_{a},\cdots \right\rangle= & {} \sqrt{\mathfrak {N}_{a}}\left| \cdots ,\mathfrak {N}_{a}-1,\cdots \right\rangle ,\nonumber \\ c_{a}^{\dagger }\left| \cdots ,\mathfrak {N}_{a},\cdots \right\rangle= & {} \sqrt{\mathfrak {N}_{a}+1}\left| \cdots ,\mathfrak {N}_{a}+1,\cdots \right\rangle . \end{aligned}$$
(36)

The derivatives and the integrals are replaced in the noncommutative space by

$$\begin{aligned} \frac{\partial }{\partial z_{a}}f\rightarrow & {} \frac{1}{2\varTheta _{a}}\delta _{ab}\left[ \bar{z}_{b},f\right] , \end{aligned}$$
(37)
$$\begin{aligned} \int \prod _{a\in \underline{4}}dz_{a}d\bar{z}_{a}f\rightarrow & {} \prod _{a\in \underline{4}}\left( 2\pi \varTheta _{a}\right) {\mathrm {Tr}}_{\mathcal {H}_{1234}}f. \end{aligned}$$
(38)

We now fix \(\varTheta _{a}=\varTheta \) for all \(a\in \underline{4}\). Following [93, 109], the general instanton equations can be written as

$$\begin{aligned} \left[ \mathbf {Z}_{a},\mathbf {Z}_{b}\right] +\frac{1}{2}\epsilon _{abcd}\left[ \bar{\mathbf {Z}}_{c},\bar{\mathbf {Z}}_{d}\right]= & {} 0, \end{aligned}$$
(39)
$$\begin{aligned} \sum _{a\in \underline{4}}\left[ \mathbf {Z}_{a},\bar{\mathbf {Z}}_{a}\right]= & {} -\zeta \cdot \mathbb {1}_{\mathscr {H}}, \end{aligned}$$
(40)
$$\begin{aligned} \left[ {\varvec{\Phi }},\mathbf {Z}_{a}\right] =\left[ {\varvec{\Phi }},\bar{\mathbf {Z}}_{a}\right]= & {} 0, \end{aligned}$$
(41)

where \(\mathbf {Z}_{a}\) and \(\bar{\mathbf {Z}}_{a}\) are the covariant coordinates of \(\mathbb {C}_{\varTheta }^{4}\),

$$\begin{aligned} \mathbf {Z}_{a}=c_{a}^{\dagger }+\mathrm {i}\sqrt{\frac{\varTheta }{2}}\left( A_{2a-1}+\mathrm {i}A_{2a}\right) ,\quad \bar{\mathbf {Z}}_{a}=c_{a}-\mathrm {i}\sqrt{\frac{\varTheta }{2}}\left( A_{2a-1}-\mathrm {i}A_{2a}\right) , \end{aligned}$$
(42)

and \({\varvec{\Phi }}\) is a holomorphic coordinate of \(\mathbb {R}^{1,1}\). The constant \(\zeta >0\) depends on the choice of \(\mathscr {H}\).

The Eq. (41) are modified in the \(\varOmega \)-background to

$$\begin{aligned} \left[ {\varvec{\Phi }},\mathbf {Z}_{a}\right] =\varepsilon _{a}\mathbf {Z}_{a},\quad \left[ {\varvec{\Phi }},\bar{\mathbf {Z}}_{a}\right] =-\varepsilon _{a}\bar{\mathbf {Z}}_{a}. \end{aligned}$$
(43)

In order to preserve the holomorphic top form \(\frac{1}{4}\epsilon _{abcd}dz_{a}\wedge dz_{b}\wedge dz_{c}\wedge dz_{d}\) that is involved in (39), we should impose the constraint

$$\begin{aligned} \sum _{a\in \underline{4}}\varepsilon _{a}=0. \end{aligned}$$
(44)

In the following, we will give various interesting solutions to the Eqs. (39, 40, 43) by making different choices of \(\mathscr {H}\).

3.2 Noncommutative instantons

The \(\mathrm {U}(n)\) noncommutative instantons on \(\prod _{a=1}^{p}\mathbb {C}_{a}\) correspond to the choice

$$\begin{aligned} \mathscr {H}=\mathbf {N}\otimes \mathcal {H}, \end{aligned}$$
(45)

where \(\mathbf {N}\cong \mathbb {C}^{n}\), and

$$\begin{aligned} \mathcal {H}=\mathcal {H}_{1\cdots p}=\mathbb {C}\left[ c_{1}^{\dagger },\cdots ,c_{p}^{\dagger }\right] \left| 0,\cdots ,0\right\rangle =\bigoplus _{\mathbf {\mathfrak {N}}\in \mathbb {Z}_{\ge 0}^{\otimes p}}\mathbb {C}\left| \mathfrak {N}_{1},\cdots ,\mathfrak {N}_{p}\right\rangle . \end{aligned}$$
(46)

Here \(p=2,3,4\) correspond to the noncommutative Yang–Mills instantons on \(\mathbb {C}_{12}^{2}\) [106], the noncommutative instantons on \(\mathbb {C}_{123}^{3}\) [57], and the instantons of the magnificent four model on \(\mathbb {C}_{1234}^{4}\) [19, 95], respectively. They can be obtained from the supersymmetric bound states of \(\hbox {D}1\)-branes and n \(\hbox {D}\left( 2p+1\right) \)-branes with the B-field taken to infinity [28, 123, 130]. The vacuum solution is given by

(47)

where \(\texttt {a}_{\alpha }\) are Coulomb parameters, and we have fixed \(\zeta =p\). In the vacuum, there is no instanton, and the gauge field \(A=0\). If we set \(\varepsilon _{a}=0\) for one direction \(a\in \left\{ p+1,\cdots ,4\right\} \), then \(\mathbf {Z}_{a}\) is allowed to be nonzero,

(48)

For the vacuum, we have the normalized character

$$\begin{aligned} \mathcal {E}_{\emptyset }=\left( \prod _{a=1}^{p}\left( 1-e^{-\beta \varepsilon _{a}}\right) \right) {\mathrm {Tr}}_{\mathscr {H}}e^{-\beta \varPhi }=\sum _{\alpha =1}^{n}e^{\beta \texttt {a}_{\alpha }}. \end{aligned}$$
(49)

A large class of nontrivial solutions can be produced using the solution generating technique [51, 53, 70]. For simplicity, we present here only the \(\mathrm {U}(1)\) case. We make an almost gauge transformation of the vacuum solution,

$$\begin{aligned} \mathbf {Z}_{a}= & {} {\left\{ \begin{array}{ll} \mathcal {U}_{\ell }c_{a}^{\dagger }f_{\ell }\left( \sum _{a=1}^{p}c_{a}^{\dagger }c_{a}\right) \mathcal {U}_{\ell }^{\dagger }, &{}\quad a=1,\cdots ,p\\ 0, &{}\quad a=p+1,\cdots ,4 \end{array}\right. },\nonumber \\ {\varvec{\Phi }}= & {} \mathcal {U}_{\ell }\left( \sum _{a=1}^{p}\varepsilon _{a}c_{a}^{\dagger }c_{a}\right) \mathcal {U}_{\ell }^{\dagger }-\texttt {a}\cdot \mathbb {1}_{\mathcal {H}}. \end{aligned}$$
(50)

Here \(\mathcal {U}_{\ell }\) is a partial isometry on \(\mathcal {H}\) obeying

$$\begin{aligned} \mathcal {U}_{\ell }\mathcal {U}_{\ell }^{\dagger }=\mathbb {1}_{\mathcal {H}}, \quad \mathcal {U}_{\ell }^{\dagger }\mathcal {U}_{\ell }=\mathbb {1}_{\mathcal {H}}-\varPi _{\ell }, \end{aligned}$$
(51)

where \(\varPi _{\ell }\) is a Hermitian projector onto a finite-dimensional subspace of \(\mathcal {H}\),

$$\begin{aligned} \varPi _{\ell }=\sum _{\mathfrak {N}<\ell }\left| \mathfrak {N}_{1},\cdots ,\mathfrak {N}_{p}\right\rangle \left\langle \mathfrak {N}_{1},\cdots ,\mathfrak {N}_{p}\right| . \end{aligned}$$
(52)

The real function \(f_{\ell }\left( \mathfrak {N}\right) \) satisfies the initial condition \(f_{\ell }\left( \mathfrak {N}\right) =0\) for \(\mathfrak {N}=0,\cdots ,\ell -1\) and the finite action condition \(\lim _{\mathfrak {N}\rightarrow \infty }f_{\ell }\left( \mathfrak {N}\right) =1\), and can be found by substituting (50) into (39, 40, 43),

$$\begin{aligned} f_{\ell }\left( \mathfrak {N}\right) =\sqrt{1-\frac{\ell \left( \ell +1\right) \cdots \left( \ell +p-1\right) }{\left( \mathfrak {N}+1\right) \left( \mathfrak {N}+2\right) \cdots \left( \mathfrak {N}+p\right) }} \left( \mathbb {1}_{\mathcal {H}}-\varPi _{\ell }\right) . \end{aligned}$$
(53)

Since \(\mathcal {U}_{\ell }\) fails to be unitary only in the subspace of \(\mathcal {H}\) with \(\mathfrak {N}<\ell \), (50) is a true gauge transformation away from a region of characteristic size \(\sqrt{\ell \varTheta }\) around the origin. The solution (50) with (53) describes localized instantons near the origin. These instantons would sit on top of each other if they were commutative instantons, and the space of such configurations would have been rather singular. The noncommutative deformation precisely resolves these singularities, and the position-space uncertainty principle (31) prevents the instantons from getting closer than the characteristic size \(\sqrt{\varTheta }\). The topological charge is given by

$$\begin{aligned} k={\mathrm {ch}}_{p}=\frac{\left( 2\pi \varTheta \right) ^{p}}{p!}{\mathrm {Tr}}_{\mathcal {H}} \left( \frac{F}{2\pi }\right) ^{p}=\frac{\ell \left( \ell +1\right) \cdots \left( \ell +p-1\right) }{p!}, \end{aligned}$$
(54)

which is the number of states removed by the operator \(\mathcal {U}_{\ell }\). Of course, these solutions are only a subset of all the solutions, and we can get more general solutions by relaxing the condition (51) [70]. In all these solutions, \(\mathcal {U}_{\ell }\) identifies \(\mathcal {H}\) with its subspace

$$\begin{aligned} \mathcal {H}_{\mathcal {I}}=\mathscr {I}\left( c_{1}^{\dagger },\cdots ,c_{p}^{\dagger }\right) \left| 0,\cdots ,0\right\rangle , \end{aligned}$$
(55)

where \(\mathscr {I}\left( w_{1},\cdots ,w_{p}\right) \subset \mathbb {C}\left[ w_{1},\cdots ,w_{p}\right] \) is an ideal in the ring of polynomials, generated by monomials, and

$$\begin{aligned} \dim _{\mathbb {C}}\mathbb {C}\left[ w_{1},\cdots ,w_{p}\right] \big /\mathscr {I}=k. \end{aligned}$$
(56)

Any such ideal defines a partition (\(p=2\)), a plane partition (\(p=3\)), or a solid partition (\(p=4\)),

$$\begin{aligned} \mathscr {I}\longleftrightarrow \mathcal {Y}=\left\{ \left. \left( x_{1},\cdots ,x_{p}\right) \in \mathbb {Z}_{+}^{p}\right| \prod _{a=1}^{p}w_{a}^{x_{a}-1}\notin \mathscr {I}\right\} . \end{aligned}$$
(57)

Let us now describe in detail the case \(p=3\), which plays an important role in this paper. The plane partition is customarily denoted by \(\pi \), and can be formed by putting \(\pi _{x,y}\in \mathbb {Z}_{\ge 0}\) boxes vertically at the position (xy) in a plane,

$$\begin{aligned} \pi =\begin{pmatrix}\pi _{1,1} &{} \pi _{1,2} &{} \pi _{1,3} &{} \cdots \\ \pi _{2,1} &{} \pi _{2,2} &{} \pi _{2,3} &{} \cdots \\ \pi _{3,1} &{} \pi _{3,2} &{} \pi _{3,3} &{} \cdots \\ \vdots &{} \vdots &{} \vdots &{} \ddots \end{pmatrix}, \end{aligned}$$
(58)

such that \(\pi _{x,y}\ge \pi _{x+1,y},\pi _{x,y+1}\) for all \(x,y\ge 1\). The volume of \(\pi \) is denoted by \(\left| \pi \right| \), and is given by

$$\begin{aligned} \left| \pi \right| =\sum _{\left( x,y\right) }\pi _{x,y}. \end{aligned}$$
(59)

We can also view the plane partition \(\pi \) as the set of boxes sitting in \(\mathbb {Z}_{+}^{3}\),

$$\begin{aligned} \pi =\left\{ \left. \left( x,y,z\right) \in \mathbb {Z}_{+}^{3}\right| 1\le z\le \pi _{x,y}\right\} , \end{aligned}$$
(60)

so that there can be at most one box at \(\left( x,y,z\right) \), and a box can occupy \(\left( x,y,z\right) \) only if there are boxes in \(\left( x^{\prime },y,z\right) ,\left( x,y^{\prime },z\right) ,\left( x,y,z^{\prime }\right) \) for all \(1\le x^{\prime }<x\), \(1\le y^{\prime }<y\), \(1\le z^{\prime }<z\). The volume of \(\pi \) is then simply the total number of boxes in \(\pi \).

In general, the normalized character evaluated at the solution labeled by \(\mathcal {Y}\) is given by

$$\begin{aligned} \mathscr {E}_{\mathcal {Y}}= & {} \left( \prod _{a=1}^{p}\left( 1-e^{-\beta \varepsilon _{a}}\right) \right) \left. {\mathrm {Tr}}_{\mathscr {H}}e^{-\beta \varPhi }\right| _{\mathcal {Y}}\nonumber \\= & {} e^{\beta \texttt {a}}-\left( \prod _{a=1}^{p}\left( 1-e^{-\beta \varepsilon _{a}}\right) \right) \sum _{\left( x_{1},\cdots ,x_{p}\right) \in \mathcal {Y}}e^{\beta \texttt {a}-\beta \sum _{a=1}^{p} \varepsilon _{a}\left( x_{a}-1\right) }. \end{aligned}$$
(61)

Once we generalize the gauge group to \(\mathrm {U}(n)\), we will have a collection of n (plane, solid) partitions labeled by \(\mathcal {Y}=\left\{ \mathcal {Y}^{(\alpha )},\alpha =1,\cdots ,n\right\} \), and the normalized character becomes

$$\begin{aligned}&\mathscr {E}_{\mathcal {Y}} = \left( \prod _{a=1}^{p}\left( 1-e^{-\beta \varepsilon _{a}}\right) \right) \left. {\mathrm {Tr}}_{\mathscr {H}} e^{-\beta \varPhi }\right| _{\mathcal {Y}}\nonumber \\&\quad = \sum _{\alpha =1}^{n}e^{\beta \texttt {a}_{\alpha }} -\left( \prod _{a=1}^{p}\left( 1-e^{-\beta \varepsilon _{a}}\right) \right) \sum _{\alpha =1}^{n}\sum _{\left( x_{1},\cdots ,x_{p}\right) \in \mathcal {Y}^{(\alpha )}} e^{\beta \texttt {a}_{\alpha }-\beta \sum _{a=1}^{p}\varepsilon _{a}\left( x_{a}-1\right) }. \end{aligned}$$
(62)

3.3 Spiked instantons

We can generalize the noncommutative Yang–Mills instantons on \(\mathbb {C}_{12}^{2}\) by taking

$$\begin{aligned} \mathscr {H}=\bigoplus _{\mathfrak {A}\in \underline{6}}\left( \mathbf {N}_{\mathfrak {A}} \otimes \mathcal {H}_{12}\right) ,\quad \mathbf {N}_{\mathfrak {A}} \cong \mathbb {C}^{n_{\mathfrak {A}}}, \end{aligned}$$
(63)

where

$$\begin{aligned} \underline{6}=\begin{pmatrix}\underline{4}\\ 2 \end{pmatrix}=\left\{ \left( 12\right) ,\left( 13\right) ,\left( 14\right) ,\left( 23\right) , \left( 24\right) ,\left( 34\right) \right\} . \end{aligned}$$
(64)

The solutions of generalized instanton equations with (63) are called the spiked instantons, which can be realized in string theory by \(\hbox {D}1\)-branes probing a stack of \(n_{\mathfrak {A}}\) (anti-)\(\hbox {D}5_{\mathfrak {A}}\)-branes in the presence of a constant B-field [93, 104].

The vacuum solution of spiked instantons is given by

(65)

Here \(\mathbf {Z}_{a}\) contains a piece in \(\mathbf {N}_{\mathfrak {A}}\) if and only if \(a\in \mathfrak {A}\), and \(c_{a}^{\dagger }\) are assigned to make \(\left[ \mathbf {Z}_{a},\mathbf {Z}_{b}\right] =0\), which are sufficient conditions for (39). We have no \(\hbox {D}1\)-brane in the vacuum. The parameters \(\varepsilon _{\mathfrak {A}}\) are given in terms of the \(\varOmega \)-deformation parameters \(\varepsilon _{a}\) appearing in (43) by

$$\begin{aligned} \varepsilon _{\mathfrak {A}}=\sum _{a\in \mathfrak {A}}\varepsilon _{a}. \end{aligned}$$
(66)

The Coulomb parameters associated with the stack of (anti-)\(\hbox {D}5_{\mathfrak {A}}\)-branes are \(\texttt {a}_{\mathfrak {A},\alpha }\).

We can produce nontrivial solutions of spiked instantons by substituting in (65)

(67)
(68)

where \(\left( \widetilde{\mathbf {Z}}_{\mathfrak {A},1},\widetilde{\mathbf {Z}}_{\mathfrak {A},2}\right) ,\mathfrak {A}\in \underline{6}\) are solutions of noncommutative \(\mathrm {U}\left( n_{\mathfrak {A}}\right) \) Yang–Mills instantons on \(\mathbb {C}^{2}\) with \(\mathscr {H}=\mathbf {N}_{\mathfrak {A}}\otimes \mathcal {H}_{12}\) and partial isometry \(\mathcal {U}_{\mathfrak {A},\ell }\). Clearly, \(\left. \mathbf {Z}_{a}\right| _{\mathbf {N}_{\mathfrak {A}}}=0\) whenever \(a\notin \mathfrak {A}\). All these solutions are in one-to-one correspondence with a collection of \(\sum _{\mathfrak {A}\in \underline{6}}n_{\mathfrak {A}}\) partitions

$$\begin{aligned} \mathbf {\mathcal {Y}}=\left\{ \mathcal {Y}^{\left( \mathfrak {A},\alpha \right) },\alpha =1,\cdots ,n_{\mathfrak {A}},\mathfrak {A}\in \underline{6}\right\} . \end{aligned}$$
(69)

3.4 Tetrahedron instantons

The tetrahedron instantons can be viewed as a generalization of spiked instantons and noncommutative instantons on \(\mathbb {C}^{3}\). We take

$$\begin{aligned} \mathscr {H}=\bigoplus _{A\in \underline{4}^{\vee }}\left( \mathbf {N}_{A} \otimes \mathcal {H}_{123}\right) ,\quad \mathbf {N}_{A}\cong \mathbb {C}^{n_{A}}. \end{aligned}$$
(70)

The construction of the vacuum solution is similar to that for spiked instantons,

(71)

We can check that (71) indeed solves the Eqs. (39, 40, 41). The vacuum solution describes that there is no \(\hbox {D}1\)-brane but \(n_{A}\) \(\hbox {D}7_{A}\)-branes, with the associated Coulomb parameters given by \(\texttt {a}_{A,\alpha }\). The parameters \(\varepsilon _{A}\) are given in terms of the \(\varOmega \)-deformation parameters \(\varepsilon _{a}\) appearing in (43) by

$$\begin{aligned} \varepsilon _{A}=\sum _{a\in A}\varepsilon _{a}. \end{aligned}$$
(72)

We can obtain nontrivial tetrahedron instantons by substituting in (71)

(73)
(74)

where \(\left( \widetilde{\mathbf {Z}}_{A,1},\widetilde{\mathbf {Z}}_{A,2},\widetilde{\mathbf {Z}}_{A,3}\right) ,A\in \underline{4}^{\vee }\) are solutions of noncommutative instantons on \(\mathbb {C}^{3}\) with \(\mathscr {H}=\mathbf {N}_{A}\otimes \mathcal {H}_{123}\) and partial isometry \(\mathcal {U}_{A,\ell }\). These solutions satisfy \(\left. \mathbf {Z}_{\check{A}}\right| _{\mathbf {N}_{A}}=0\). They describe bound states of \(\hbox {D}1\)-branes and \(n_{A}\) \(\hbox {D}7_{A}\)-branes in the presence of a strong B-field. All these solutions are in one-to-one correspondence with a collection of \(\sum _{A\in \underline{4}^{\vee }}n_{A}\) plane partitions

$$\begin{aligned} \varvec{\uppi }=\left\{ \pi ^{\left( \mathcal {A}\right) },\mathcal {A}\in \underline{n}\right\} . \end{aligned}$$
(75)

where we combined \(\left( A,\alpha \right) \) into \(\mathcal {A}\), and define

$$\begin{aligned} \underline{n}=\left\{ \left. \mathcal {A}=\left( A,\alpha \right) \right| \alpha =1,\cdots n_{A},A\in \underline{4}^{\vee }\right\} . \end{aligned}$$
(76)

We can obtain the normalized characters

(77)

4 Moduli Space of Tetrahedron Instantons

In this section, we will carefully analyze the moduli space \(\mathfrak {M}_{\mathbf {n},k}\) of tetrahedron instantons.

4.1 Basic properties of the moduli space

Let \({\mathbf {B}}=\left( B_{a}\right) _{a\in \underline{4}}\) and \({\mathbf {I}}=\left( I_{A}\right) _{A\in \underline{4}^{\vee }}\) be two quartets of matrices,

$$\begin{aligned} B_{a}\in {\mathrm {End}}\left( \mathbf {K}\right) ,\quad I_{A}\in {\mathrm {Hom}}\left( \mathbf {N}_{A},\mathbf {K}\right) , \end{aligned}$$
(78)

with the vector spaces \(\mathbf {K}\cong \mathbb {C}^{k}\) and \(\mathbf {N}_{A}\cong \mathbb {C}^{n_{A}},A\in \underline{4}^{\vee }\). The moduli space \(\mathfrak {M}_{\mathbf {n},k}\) has been derived from the string theory realization of tetrahedron instantons,

(79)

where

$$\begin{aligned} \mu ^{\mathbb {R}}= & {} \sum _{a\in \underline{4}}\left[ B_{a},B_{a}^{\dagger }\right] +\sum _{A\in \underline{4}^{\vee }}I_{A}I_{A}^{\dagger }, \end{aligned}$$
(80)
$$\begin{aligned} \mu ^{\mathbb {C}}= & {} \left( \mu _{ab}^{\mathbb {C}}=\left[ B_{a},B_{b}\right] \right) _{a,b\in \underline{4}}, \end{aligned}$$
(81)
$$\begin{aligned} \sigma= & {} \left( \sigma _{A}=B_{\check{A}}I_{A}\right) _{A\in \underline{4}^{\vee }}, \end{aligned}$$
(82)

and the \(\mathrm {U}(k)\) symmetry acts on \(B_{a}\) in the adjoint representation and \(I_{A}\) in the fundamental representation,

$$\begin{aligned} \left( B_{a},I_{A}\right) \rightarrow \left( gB_{a}g^{-1},gI_{A}\right) ,\quad g\in \mathrm {U}(k). \end{aligned}$$
(83)

The metric on \(\mathfrak {M}_{\mathbf {n},k}\) is inherited from the flat metric on \(\left( \mathbf {B},{\mathbf {I}}\right) \). Since the moduli space \(\mathfrak {M}_{\mathbf {n},k}\) is invariant under the scaling transformation

$$\begin{aligned} B_{a}\rightarrow \kappa B_{a},\quad I_{A}\rightarrow \kappa I_{A},\quad r\rightarrow \kappa ^{2}r,\quad \kappa >0, \end{aligned}$$
(84)

the value of r is inconsequential as long as \(r>0\).

If we drop the equations \(\sigma =0\), we can combine the quartet of matrices \({\mathbf {I}}\) into a single matrix \(I\in {\mathrm {Hom}}\left( \bigoplus _{A\in \underline{4}^{\vee }}\mathbf {N}_{A},\mathbf {K}\right) \), and \(\mathfrak {M}_{\mathbf {n},k}\) becomes the moduli space of instantons in the rank n magnificent four model [95, 103].

The moduli space \(\mathfrak {M}_{\mathbf {n},k}\) admits an equivalent description using the geometric invariant theory quotient [83],

$$\begin{aligned} \mathfrak {M}_{\mathbf {n},k}\cong \left. \left\{ \left( \mathbf {B},{\mathbf {I}}\right) \right| \mu ^{\mathbb {C}}=\sigma =0\right\} ^{\mathrm {stable}}\big /\mathrm {GL}(k,\mathbb {C}), \end{aligned}$$
(85)

where the stability condition states that

$$\begin{aligned} \sum _{A=\left( abc\right) \in \underline{4}^{\vee }}\mathbb {C}\left[ B_{a},B_{b},B_{c}\right] I_{A}\left( \mathbf {N}_{A}\right) =\mathbf {K}. \end{aligned}$$
(86)

The virtual dimension of \(\mathfrak {M}_{\mathbf {n},k}\) can be computed by subtracting the number of constraints and gauge degrees of freedom from the total number of components of the matrices,

$$\begin{aligned} {\mathrm {vdim}}_{\mathbb {C}}\mathfrak {M}_{\mathbf {n},k} =\left( 4k^{2}+\sum _{A\in \underline{4}^{\vee }}n_{A}k\right) -\left( 3k^{2} +\sum _{A\in \underline{4}^{\vee }}n_{A}k\right) -k^{2}=0. \end{aligned}$$
(87)

We emphasize that the vanishing virtual dimension does not mean that the space \(\mathfrak {M}_{\mathbf {n},k}\) is empty or a set of discrete points. In fact, we will see that \(\mathfrak {M}_{\mathbf {n},k}\) generally consists of several smooth manifolds of positive actual dimensions.

We can also substitute the equations \(\mu ^{\mathbb {C}}=0\) with the equations \(\rho =0\) using the identity

$$\begin{aligned} \sum _{1\le a<b\le 4}{\mathrm {Tr}}\left[ B_{a},B_{b}\right] \left[ B_{a},B_{b}\right] ^{\dagger } =\frac{1}{2}\sum _{1\le a<b\le 4}{\mathrm {Tr}}\rho _{ab}\rho _{ab}^{\dagger }, \end{aligned}$$
(88)

where

$$\begin{aligned} \rho _{ab}=\left[ B_{a},B_{b}\right] +\frac{1}{2}\epsilon _{abcd}\left[ B_{c}^{\dagger },B_{d}^{\dagger }\right] . \end{aligned}$$
(89)

4.2 Geometric interpretation

In this subsection we discuss geometric interpretations for the moduli space \(\mathfrak {M}_{\mathbf {n},k}\).

Let us start with the simplest case \(\mathbf {n}=\left( n_{123},0,0,0\right) \), which can be realized in string theory as the bound states of k \(\hbox {D}1\)-branes with \(n_{123}\) \(\hbox {D}7_{123}\)-branes [25, 57, 111]. In this case, the matrices \(I_{A}\) and equations \(\sigma _{A}=0\) are nontrivial only for \(A=(123)\). It is useful to review two equivalent geometric interpretations for the moduli space \(\mathfrak {M}_{\left( n_{123},0,0,0\right) ,k}\).

Let \(\mathbb {CP}^{3}=\mathbb {C}^{3}\cup \mathbb {CP}_{\infty }^{2}\) be a compactification of \(\mathbb {C}^{3}\), where the homogeneous coordinates on \(\mathbb {CP}^{3}\) are \(\left[ z_{0}:z_{1}:z_{2}:z_{3}\right] \), and \(\mathbb {CP}_{\infty }^{2}=\left[ 0:z_{1}:z_{2}:z_{3}\right] \) is the plane at infinity. We define the canonical open embedding \(\iota :\mathbb {C}^{3}\hookrightarrow \mathbb {CP}^{3}\). The moduli space \(\mathfrak {M}_{\left( n_{123},0,0,0\right) ,k}\) coincides with the moduli space of \(\left( \mathcal {E},\varPhi \right) \), where \(\mathcal {E}\) is a torsion free sheaf on \(\mathbb {CP}^{3}\) with the Chern character

$$\begin{aligned} {\mathrm {ch}}\left( \mathcal {E}\right) =\left( n_{123},0,0,-k\right) , \end{aligned}$$
(90)

and the framing \(\varPhi \) is a trivialization of \(\mathcal {E}\) on \(\mathbb {CP}_{\infty }^{2}\),

$$\begin{aligned} \varPhi :\left. \mathcal {E}\right| _{\mathbb {CP}_{\infty }^{2}}\cong \mathbf {N}_{123}\otimes \mathcal {O}_{\mathbb {CP}_{\infty }^{2}}. \end{aligned}$$
(91)

There is a short exact sequence

$$\begin{aligned} 0\rightarrow \mathcal {E}\rightarrow \mathcal {F}\rightarrow \mathcal {S}_{\mathfrak {Z}}\rightarrow 0, \end{aligned}$$
(92)

where \(\mathcal {F}\) is the coherent sheaf of sections of the trivial rank \(n_{123}\) holomorphic vector bundles on \(\mathbb {CP}^{3}\) with framing on \(\mathbb {CP}_{\infty }^{2}\),

$$\begin{aligned} \mathcal {F}\cong \mathbf {N}_{123}\otimes \mathcal {O}_{\mathbb {CP}^{3}}, \end{aligned}$$
(93)

and \(\mathcal {S}_{\mathfrak {Z}}\) is a coherent sheaf supported on the subspace \(\mathfrak {Z}\subset \mathbb {C}^{3}=\mathbb {CP}^{3}\setminus \mathbb {CP}_{\infty }^{2}\),

$$\begin{aligned} \mathcal {S}_{\mathfrak {Z}}\cong \iota _{*}\mathcal {O}_{\mathfrak {Z}}. \end{aligned}$$
(94)

In our case, \(\mathfrak {Z}\) is a union of k points \(\mathfrak {p}_{i}\). The sheaf \(\mathcal {F}\) is a locally free sheaf, and the torsion free sheaf \(\mathcal {E}\) fails to be locally free only along \(\mathfrak {Z}\). As a result of (92), the Chern characters are related by

$$\begin{aligned} {\mathrm {ch}}\left( \mathcal {E}\right) ={\mathrm {ch}}\left( \mathcal {F}\right) -{\mathrm {ch}} \left( \mathcal {S}_{\mathfrak {Z}}\right) , \end{aligned}$$
(95)

with

$$\begin{aligned} {\mathrm {ch}}\left( \mathcal {F}\right) =\left( n_{123},0,0,0\right) ,\quad {\mathrm {ch}} \left( S_{\mathfrak {Z}}\right) =\left( 0,0,0,\sum _{i=1}^{k}\mathrm {PD}\left[ \mathfrak {p}_{i}\right] \right) . \end{aligned}$$
(96)

Here we denote the Poincare dual of the fundamental class \(\left[ X\right] \) associated to X by \(\mathrm {PD}\left[ X\right] \). From the perspective of string theory, \(\mathcal {F}\) and \(\mathcal {S}_{\mathfrak {Z}}\) correspond to the \(\hbox {D}7_{123}\)-branes and the \(\hbox {D}1\)-branes, respectively. Moreover, (93) is realized in noncommutative field theory by the vacuum solution (47) with \(p=3\) and \(\mathbf {N}=\mathbf {N}_{123}\).

As proven in [23], \(\mathfrak {M}_{\left( n_{123},0,0,0\right) ,k}\) is isomorphic to a Quot scheme

$$\begin{aligned} \mathfrak {M}_{\left( n_{123},0,0,0\right) ,k}\cong {\mathrm {Quot}}_{\mathbb {C}^{3}}^{k} \left( \mathcal {O}^{\oplus n_{123}}\right) , \end{aligned}$$
(97)

which parametrizes isomorphism classes of the quotients \(\mathcal {O}^{\oplus n_{123}} \twoheadrightarrow \mathcal {S}_{\mathfrak {Z}}\) such that the Hilbert–Poincare polynomial of \(\mathcal {S}_{\mathfrak {Z}}\) is k [115]. When \(n_{123}=1\), this Quot scheme is the same as the Hilbert scheme \(\mathrm {Hilb}^{k}\left( \mathbb {C}^{3}\right) \) of k points on \(\mathbb {C}^{3}\). In noncommutative field theory, each quotient \(\mathcal {O}^{\oplus n_{123}}\twoheadrightarrow \mathcal {S}_{\mathfrak {Z}}\) corresponds to a choice of the partial isometry \(\mathcal {U}_{\ell }\) with the identification (55) satisfying (56).

Now we sketch a possible geometric interpretation for \(\mathfrak {M}_{\mathbf {n},k}\) by generalizing the Quot scheme description for \(\mathfrak {M}_{\left( n_{123},0,0,0\right) ,k}\). We regard the worldvolume of the \(\hbox {D}7_{123}\)-branes as the physical spacetime, and \(\mathcal {F}\) is still a locally free sheaf given by (93). The additional \(\hbox {D}7_{A}\)-branes for \(A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} \) are located on the real codimension-two hyperplane \(\mathfrak {h}_{A}\subset \mathbb {C}^{3}\) defined by \(z_{\breve{A}}=0\), and produce real codimension-two defects in the physical spacetime. Accordingly, \(\mathfrak {Z}\) becomes a union of hyperplanes and points,

$$\begin{aligned} \mathfrak {Z}=\left( \bigcup _{A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} }\mathfrak {h}_{A}\right) \cup \left( \bigcup _{i=1}^{k}\mathfrak {p}_{i}\right) , \end{aligned}$$
(98)

and \(\mathcal {S}_{\mathfrak {Z}}\) is a complex of sheaves whose entries are \(\mathbf {N}_{A}\otimes \iota _{*}\mathcal {O}_{\mathfrak {h}_{A}}\) for \(A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} \), \(\iota _{*}\mathcal {O}_{\mathfrak {p}_{i}}\) for \(i=1,\cdots ,k\), and differentials specified by strings stretching between the D-branes. To define the Quot scheme, we need to further specify the quotients \(\mathcal {O}^{\oplus n_{123}}\twoheadrightarrow \mathcal {S}_{\mathfrak {Z}}\) by giving the Hilbert–Poincare polynomial \(\mathcal {P}\), which describes the configuration of \(\hbox {D}1\)-branes and \(\hbox {D}7_{A}\)-branes for \(A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} \). From the classical configuration of the D-branes, we can write down their coordinate ring in a suitable basis. For example, if \(\mathfrak {Z}\) arises from \(n_{124}\) \(\hbox {D}7_{124}\)-branes and a single \(\hbox {D}1\)-brane, their coordinate ring is given by

$$\begin{aligned} \mathbb {C}\left[ z_{1},z_{2},z_{3}\right] \big /\mathscr {I}_{\mathfrak {h}}\cdot \mathscr {I}_{\mathfrak {p}}, \end{aligned}$$
(99)

where \(\mathscr {I}_{\mathfrak {h}}=\left\langle Q\left( z_{3}\right) \right\rangle \) is an ideal generated by a degree \(n_{124}\) polynomial \(Q\left( z_{3}\right) \) which encodes the positions of \(\hbox {D}7_{124}\)-branes in \(\mathbb {C}_{3}\), and \(\mathscr {I}_{\mathfrak {p}}=\left\langle z_{1}-\xi _{1},z_{2}-\xi _{2},z_{3}-\xi _{3}\right\rangle \) is an ideal which encodes the location \(\left( \xi _{1},\xi _{2},\xi _{3}\right) \) of \(\hbox {D}1\)-branes in \(\mathbb {C}_{123}^{3}\). From the coordinate ring, we can calculate the Hilbert–Poincare polynomial \(\mathcal {P}\left( t;n_{124}\right) \), which is a formal power series of t and depends on \(n_{124}\). In general, the Hilbert–Poincare polynomial \(\mathcal {P}\left( t;n_{124},n_{134},n_{234},k\right) \) will depend on \(n_{A}\) for \(A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} \) and k. We can also read off the Chern character from \(\mathcal {P}\).

Since \(\mathfrak {M}_{\mathbf {n},k}\) is symmetric under the permutation of \(\mathbf {n}\), it is natural to expect the isomorphisms such as

$$\begin{aligned} {\mathrm {Quot}}_{\mathbb {C}^{3}}^{\mathcal {P}\left( t;n_{124},n_{134},n_{234},k\right) }\left( \mathcal {O}^{\oplus n_{123}}\right) \cong {\mathrm {Quot}}_{\mathbb {C}^{3}}^{\mathcal {P}\left( t;n_{123},n_{134},n_{234},k\right) }\left( \mathcal {O}^{\oplus n_{124}}\right) . \end{aligned}$$
(100)

We can interpret such isomorphisms as four possible projections of tetrahedron instantons to the faces of the tetrahedron (see Fig. 1), and each shadow contains the same information.

Furthermore, the geometric interpretation for \(\mathfrak {M}_{\mathbf {n},k}\) as the Quot scheme leads to a natural forgetful projection,

$$\begin{aligned} \varrho :\mathfrak {M}_{\mathbf {n},k}\rightarrow \bigcup _{k^{\prime }\le k}\mathfrak {M}_{\left( n_{123},0,0,0\right) ,k'}, \end{aligned}$$
(101)

where we drop all the information of \(\hbox {D}7_{A}\)-branes for \(A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} \) in the Hilbert–Poincare polynomial.

It is rather difficult to give a geometric interpretation for \(\mathfrak {M}_{\mathbf {n},k}\) if we want to keep the permutation symmetry of \(\mathbf {n}\) manifest. Here we propose a possible approach, leaving the mathematical rigor for future work. Instead of considering four stacks of \(\hbox {D}7_{A}\)-branes on different \(\mathbb {C}_{A}^{3}\), we imagine that they would be unified into a single \(\hbox {D}7\)-brane which wraps a complicated hyperplane in \(\mathbb {C}^{4}\). We compactify \(\mathbb {C}^{4}\) into the projective space \(\mathbb {CP}^{4}=\mathbb {C}^{4}\cup \mathbb {CP}_{\infty }^{3}\) with homogeneous coordinates \(\left[ z_{0}:z_{1}:z_{2}:z_{3}:z_{4}\right] \), and the hyperplane at infinity is \(\mathbb {CP}_{\infty }^{3}=\left[ 0:z_{1}:z_{2}:z_{3}:z_{4}\right] \). We also define \(\mathbb {CP}_{A}^{3}\subset \mathbb {CP}^{4}\) and \(\mathbb {CP}_{\infty ,A}^{2}\subset \mathbb {CP}_{\infty }^{3}\) by \(z_{\check{A}}=0\) for each \(A\in \underline{4}^{\vee }\), respectively. The hyperplane becomes an algebraic variety,

$$\begin{aligned} X_{\xi }=\left\{ \left[ z_{0}:z_{1}:z_{2}:z_{3}:z_{4}\right] \in \mathbb {CP}^{4}\left| \left( \prod _{A\in \underline{4}^{\vee }}z_{\check{A}}^{n_{A}}\right) =\xi z_{0}^{\sum _{A\in \underline{4}^{\vee }}n_{A}}\right. \right\} , \end{aligned}$$
(102)

where we introduced a small deformation parameter \(\xi \) in order to make \(X_{\xi }\) a smooth manifold, and we will finally take \(\xi \) to zero. We also introduce a noncompact space \(\mathring{X}_{\xi }\), which is obtained from \(X_{\xi }\) by removing all points on \(\mathbb {CP}_{\infty }^{3}\). Then we can take \(\mathcal {F}\) to be a rank one locally free sheaf on \(X_{\xi }\),

$$\begin{aligned} \mathcal {F}\cong \mathcal {O}_{X_{\xi }}, \end{aligned}$$
(103)

and the sheaf \(\mathcal {S}_{\mathfrak {Z}}\) is

$$\begin{aligned} \mathcal {S}_{\mathfrak {Z}}\cong \iota _{*}\mathcal {O}_{\mathfrak {Z}}, \end{aligned}$$
(104)

where the support \(\mathfrak {Z}\subset X_{\xi }\setminus \mathbb {CP}_{\infty }^{3}\) is a union of k points \(\mathfrak {p}_{i}\), and \(\iota :\mathring{X}_{\xi }\hookrightarrow X_{\xi }\) is the natural embedding. We expect that the moduli space \(\mathfrak {M}_{\mathbf {n},k}\) coincides with the \(\xi \rightarrow 0\) limit of the Hilbert scheme \(\mathrm {Hilb}^{k}\left( \mathring{X}_{\xi }\right) \) of k points on \(\mathring{X}_{\xi }\). Equivalently, \(\mathfrak {M}_{\mathbf {n},k}\) should also be identical to the \(\xi \rightarrow 0\) limit of the moduli space of framed rank one torsion free sheaves \(\mathcal {E}\) on \(X_{\xi }\) with the framing

$$\begin{aligned} \varPhi :\left. \mathcal {E}\right| _{X_{\xi }\cap \mathbb {CP}_{\infty }^{3}}\cong \mathcal {O}_{X_{\xi }\cap \mathbb {CP}_{\infty }^{3}}. \end{aligned}$$
(105)

There are particularly interesting points on the moduli space \(\mathfrak {M}_{\mathbf {n},k}\) such that the framed torsion free sheaf \(\left( \mathcal {E},\varPhi \right) \) admits an isomorphism,

$$\begin{aligned} \left( \mathcal {E},\varPhi \right) \cong \bigoplus _{\mathcal {A}=(A,\alpha )\in \underline{n}}\left( \mathcal {I}_{\mathcal {A}},\varPhi _{\mathcal {A}}\right) , \end{aligned}$$
(106)

where \(\mathcal {I}_{A,\alpha }\) is a rank one torsion free sheaf supported on \(\mathbb {CP}_{A}^{3}\) with the framing \(\varPhi _{A,\alpha }:\left. \mathcal {I}_{A,\alpha }\right| _{\mathbb {CP}_{A,\infty }^{2}}\cong \mathcal {O}_{\mathbb {CP}_{A,\infty }^{2}}\). The tetrahedron instantons corresponding to such decompositions are given in noncommutative field theory in Sect. 3.4.

4.3 One-instanton examples

In order to gain a better understanding of \(\mathfrak {M}_{\mathbf {n},k}\), we will work out explicitly the one-instanton moduli spaces step by step. When \(k=1\), the matrix \(B_{a}\) is simply a complex number, and \(I_{A}=\left( I_{A,1},\cdots ,I_{A,n_{A}}\right) \) is a \(1\times n_{A}\) matrix if \(n_{A}\ge 1\). The equations \(\mu ^{\mathbb {C}}=0\) are satisfied automatically, so we only need to consider

$$\begin{aligned} \sum _{A\in \underline{4}^{\vee }}I_{A}I_{A}^{\dagger }= & {} 1, \end{aligned}$$
(107)
$$\begin{aligned} B_{\check{A}}I_{A}= & {} 0, \end{aligned}$$
(108)

where we set \(r=1\) using the scaling invariance (84). Meanwhile, the group \(\mathrm {U}(k)=\mathrm {U}(1)\) acts trivially on \(B_{a}\) and gives an equivalence relation \(I_{A}\sim e^{\mathrm {i}\theta }I_{A}\).

4.3.1 Instanton on \(\mathbb {C}^{3}\)

We start with the rank n instanton on \(\mathbb {C}_{123}^{3}\) corresponding to \(\mathbf {n}=\left( n_{123}=n,0,0,0\right) \) [25, 111]. There is only one \(I_{A}\), namely \(I_{(123)}\), and the equation (107) becomes

$$\begin{aligned} \sum _{\alpha =1}^{n}\left| I_{123,\alpha }\right| ^{2}=1. \end{aligned}$$
(109)

After modding out the \(\mathrm {U}(1)\) phase, we obtain from \(I_{(123)}\) a complex projective space \(\mathbb {CP}^{n-1}\). Meanwhile, we get \(B_{4}=0\) from (108), and \(B_{1},B_{2},B_{3}\) are three unconstrained complex numbers. Therefore, the one-instanton moduli space of the rank n instanton on \(\mathbb {C}_{123}^{3}\) is given by

$$\begin{aligned} \mathfrak {M}_{\left( n,0,0,0\right) ,1}\cong \mathbb {C}^{3}\times \mathbb {CP}^{n-1}. \end{aligned}$$
(110)

Here the factor \(\mathbb {C}^{3}\) stands for the center of the instanton, and the factor \(\mathbb {CP}^{n-1}\) stands for the size and the gauge orientation of the instanton.

4.3.2 Generalized folded instanton

We go one step further by allowing \(\mathbf {n}\) to have two nonzero elements,

$$\begin{aligned} \mathbf {n}=\left( n_{123}=n,n_{124}=n^{\prime },0,0\right) , \end{aligned}$$
(111)

which can be viewed as a generalization of the folded instantons [93]. In this case, the nonzero \(I_{A}\) are \(I_{123}\) and \(I_{124}\). \(B_{1},B_{2}\) are unconstrained complex numbers. When \(B_{3}\) and \(B_{4}\) are both nonzero, we know from (108) that \(I_{123}=I_{124}=0\), which contradicts (107). When \(B_{4}=0\) and \(B_{3}\ne 0\), \(I_{124}=0\) and \(I_{123}\) satisfies (109). Modding out the \(\mathrm {U}(1)\) phase, we get a \(\mathbb {CP}^{n-1}\) from \(I_{123}\). Similarly, by exchanging \(3\leftrightarrow 4\), we get a \(\mathbb {CP}^{n^{\prime }-1}\) from \(I_{124}\). When \(B_{3}=B_{4}=0\), we have

$$\begin{aligned} \sum _{\alpha =1}^{n}\left| I_{123,\alpha }\right| ^{2}+\sum _{\alpha =1}^{n^{\prime }}\left| I_{124,\alpha }\right| ^{2}=1, \end{aligned}$$
(112)

which gives a \(\mathbb {CP}^{n+n^{\prime }-1}\) after modding out the \(\mathrm {U}(1)\) phase. Therefore, the moduli space \(\mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\) consists of three smooth manifolds with different actual dimensions for generic n and \(n^{\prime }\),

$$\begin{aligned} \mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\cong \mathbb {C}^{2} \times \mathbb {C}^{*}\times \mathbb {CP}^{n-1}\bigcup \mathbb {C}^{2}\times \mathbb {C}^{*} \times \mathbb {CP}^{n^{\prime }-1}\bigcup \mathbb {C}^{2}\times \mathbb {CP}^{n+n^{\prime }-1}. \end{aligned}$$
(113)

The first and the second components of \(\mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\) correspond to the instanton being only on \(\mathbb {C}_{123}^{3}\) and \(\mathbb {C}_{124}^{3}\), respectively. The factor \(\mathbb {C}^{2}\times \mathbb {C}^{*}\) parametrizes the center of the instanton, while \(\mathbb {CP}^{n-1}\) or \(\mathbb {CP}^{n^{\prime }-1}\) parametrizes the size and the gauge orientation of the instanton. The last component of \(\mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\) corresponds to the instanton being on the intersection \(\mathbb {C}_{12}^{2}=\mathbb {C}_{123}^{3}\cap \mathbb {C}_{124}^{3}\), and the center of the instanton gives the factor \(\mathbb {C}^{2}\).

Recall that the moduli space of vortices with charge k in the \(\mathrm {U}\left( n+n'\right) \) gauge theory is given by the symplectic quotient [52]

$$\begin{aligned} \mathcal {V}_{n+n^{\prime },k}\cong \left\{ \left( \texttt {B},\texttt {I}\right) \left| \left[ \texttt {B},\texttt {B}^{\dagger }\right] +\texttt {I}{} \texttt {I}^{\dagger }=r_{v} \cdot \mathbb {1}_{k}\right. \right\} \big /\mathrm {U}(k),\quad r_{v}>0, \end{aligned}$$
(114)

where \(\texttt {B}\in {\mathrm {End}}\left( \mathbb {C}^{k}\right) \), \(\texttt {I}\in {\mathrm {Hom}}\left( \mathbb {C}^{n+n^{\prime }},\mathbb {C}^{k}\right) \), and the \(\mathrm {U}(k)\) action is

$$\begin{aligned} \left( \texttt {B},\texttt {I}\right) \rightarrow \left( g\texttt {B}g^{-1},g\texttt {I}\right) ,\quad g\in \mathrm {U}(k). \end{aligned}$$
(115)

We introduce the following actions on \(\mathcal {V}_{n+n^{\prime },k}\),

$$\begin{aligned} \mathbb {T}_{1}&:&\left( \texttt {B},\texttt {I}\right) \rightarrow \left( q\texttt {B},\texttt {I}\right) ,\quad q\in \mathbb {C}^{*} \end{aligned}$$
(116)
$$\begin{aligned} \mathbb {T}_{2}&:&\left( \texttt {B},\texttt {I}\right) \rightarrow \left( \texttt {B},\texttt {I}h^{-1}\right) ,\quad h={\mathrm {diag}}\left( {\mathop {\overbrace{1,\cdots ,1}}\limits ^{n}},{\mathop {\overbrace{-1,\cdots ,-1}}\limits ^{n^{\prime }}}\right) . \end{aligned}$$
(117)

Now we focus on the simple case \(k=1\). The fixed points of \(\mathcal {V}_{n+n^{\prime },1}\) under the \(\mathbb {T}_{1}\) action satisfy

$$\begin{aligned} \texttt {B}=0,\quad \texttt {I}{} \texttt {I}^{\dagger }=r_{v}\cdot \mathbb {1}_{k}, \end{aligned}$$
(118)

and therefore the \(\mathbb {T}_{1}\)-fixed points of \(\mathcal {V}_{n+n^{\prime },1}\) form a manifold

$$\begin{aligned} \mathcal {V}_{n+n^{\prime },1}^{\mathbb {T}_{1}}\cong \mathbb {CP}^{n+n'-1}. \end{aligned}$$
(119)

On the other hand, if we write

$$\begin{aligned} \texttt {I}=\begin{pmatrix}{} \texttt {I}_{n} &{}\quad 0\\ 0 &{}\quad \texttt {I}_{n^{\prime }} \end{pmatrix}, \end{aligned}$$
(120)

then the fixed points of \(\mathcal {V}_{n+n^{\prime },k}\) under the \(\mathbb {T}_{2}\) action satisfy

$$\begin{aligned} \left\{ \texttt {I}_{n}{} \texttt {I}_{n}^{\dagger }=r_{v}\cdot \mathbb {1}_{k},\texttt {I}_{n^{\prime }}=0\right\} \ \mathrm {or}\ \left\{ \texttt {I}_{n^{\prime }}{} \texttt {I}_{n^{\prime }}^{\dagger }=r_{v}\cdot \mathbb {1}_{k},\texttt {I}_{n}=0\right\} , \end{aligned}$$
(121)

and consequently the \(\mathbb {T}_{2}\)-fixed points of \(\mathcal {V}_{n+n^{\prime },1}\) are given by

$$\begin{aligned} \mathcal {V}_{n+n^{\prime },1}^{\mathbb {T}_{2}}\cong \mathbb {C}\times \left( \mathbb {CP}^{n-1}\cup \mathbb {CP}^{n^{\prime }-1}\right) . \end{aligned}$$
(122)

It is interesting that the moduli space \(\mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\) can be rewritten as

$$\begin{aligned} \mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\cong \mathbb {C}^{2} \times \left( \mathcal {V}_{n+n^{\prime },1}^{\mathbb {T}_{1}} \cup \mathcal {V}_{n+n^{\prime },1}^{\mathbb {T}_{2}}\right) , \end{aligned}$$
(123)

which is manifestly symmetric between n and \(n^{\prime }\). Here we used the fact that

$$\begin{aligned} \mathcal {V}_{n+n^{\prime },1}^{\mathbb {T}_{1}}\cap \mathcal {V}_{n+n^{\prime },1}^{\mathbb {T}_{2}}\cong \left\{ 0\right\} \times \left( \mathbb {CP}^{n-1}\cup \mathbb {CP}^{n^{\prime }-1}\right) . \end{aligned}$$
(124)

It is straightforward to generalize this relation between \(\mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,k}\) and \(\mathcal {V}_{n+n^{\prime },k}\) to any positive integer k.

4.3.3 Generic tetrahedron instanton

Now it is clear how to obtain the one-instanton moduli space \(\mathfrak {M}_{\mathbf {n},1}\) for generic \(\mathbf {n}\). The Eqs. (107) and (108) have no solutions when all \(B_{a}\) are nonzero. When there are r nonzero \(B_{\check{A}}\) with \(r=3,2,1,0\), the Eq. (108) require the corresponding r of \(I_{A}\) to be zero, and the remaining \((4-r)\) of \(I_{A}\) are constrained by (107), producing a complex projective space after modding out the \(\mathrm {U}(1)\) phase. Combining all the possibilities, we get

$$\begin{aligned} \mathfrak {M}_{\mathbf {n},1}\cong & {} \left[ \bigcup _{A\in \underline{4}^{\vee }} \left( \mathbb {C}^{*}\right) ^{3}\times \mathbb {CP}^{n_{A}-1}\right] \cup \left[ \bigcup _{A\ne B\in \underline{4}^{\vee }}\left( \mathbb {C}^{*}\right) ^{2} \times \mathbb {CP}^{n_{A,B}-1}\right] \cup \nonumber \\&\cup \left[ \bigcup _{A\ne B\ne C\in \underline{4}^{\vee }}\mathbb {C}^{*} \times \mathbb {CP}^{n_{A,B,C}-1}\right] \cup \left[ \mathbb {CP}^{n_{\underline{4}^{\vee }}-1}\right] , \end{aligned}$$
(125)

where

$$\begin{aligned} n_{S}=\sum _{A\in S}n_{A},\quad S\subset \underline{4}^{\vee }. \end{aligned}$$
(126)

We see that \(\mathfrak {M}_{\mathbf {n},1}\) for generic \(\mathbf {n}\) consists of \(2^{4}-1=15\) smooth manifolds of different actual dimensions. The interpretation of each component of \(\mathfrak {M}_{\mathbf {n},1}\) is a straightforward generalization of that of \(\mathfrak {M}_{\left( n,n^{\prime },0,0\right) ,1}\).

4.4 Symmetries of the moduli space

In the definition of the moduli space \(\mathfrak {M}_{\mathbf {n},k}\), we have the freedom to pick the basis for the vector space \(\mathbf {N}_{A}\). This induces a \(\mathrm {U}\left( n_{A}\right) \) symmetry, which acts on \(I_{A}\) in the anti-fundamental representation and acts trivially on other operators,

$$\begin{aligned} B_{a}\rightarrow B_{a},\quad I_{B}\rightarrow \delta _{A,B}I_{B}h^{-1},\quad h\in \mathrm {U}\left( n_{A}\right) . \end{aligned}$$
(127)

We parametrize the Cartan subalgebra of the Lie algebra of \(\mathrm {U}\left( n_{A}\right) \) by

$$\begin{aligned} \texttt {a}_{A}={\mathrm {diag}}\left( \texttt {a}_{A,1},\cdots ,\texttt {a}_{A,n_{A}}\right) . \end{aligned}$$
(128)

Since the common center \(\mathrm {U}(1)_{c}\) of \(\prod _{A\in \underline{4}^{\vee }}\mathrm {U}\left( n_{A}\right) \) is contained in \(\mathrm {U}(k)\), it is the group

$$\begin{aligned} {\mathrm {PU}}\left( \mathbf {n}\right) =\frac{\prod _{A\in \underline{4}^{\vee }} \mathrm {U}\left( n_{A}\right) }{\mathrm {U}(1)_{c}} \end{aligned}$$
(129)

that acts nontrivially on \(\mathfrak {M}_{\mathbf {n},k}\). Accordingly, the parameters \(\texttt {a}_{A,\alpha }\) are defined up to the simultaneous shift \(\texttt {a}_{A,\alpha }\rightarrow \texttt {a}_{A,\alpha }+\xi \), where \(\xi \) is a constant number. Sometimes it is useful to separate the \(\mathrm {U}\left( n_{A}\right) \) into the \(\mathrm {U}(1)\) part and the \({\mathrm {SU}}\left( n_{A}\right) \) part, and their respective Cartan subalgebras are parametrized by

$$\begin{aligned} \bar{\texttt {a}}_{A}=\frac{1}{n_{A}}\sum _{\alpha =1}^{n_{A}} \texttt {a}_{A,\alpha },\quad \tilde{\texttt {a}}_{A,\alpha }=\texttt {a}_{A,\alpha }-\bar{\texttt {a}}_{A}. \end{aligned}$$
(130)

In addition, \(\mathfrak {M}_{\mathbf {n},k}\) has an \({\mathrm {SU}}(4)\) symmetry which acts on \(\left( \mathbf {B},{\mathbf {I}}\right) \) as

$$\begin{aligned} B_{a}\rightarrow U_{ab}B_{b},\quad I_{A}\rightarrow I_{A},\quad U\in {\mathrm {SU}}(4). \end{aligned}$$
(131)

This \({\mathrm {SU}}(4)\) symmetry is induced from the rotation symmetry of \(\mathbb {C}^{4}\) that leaves the holomorphic top form invariant. We parametrize the Cartan subalgebra of the Lie algebra of \({\mathrm {SU}}(4)\) by

$$\begin{aligned} \varepsilon ={\mathrm {diag}}\left( \varepsilon _{1},\varepsilon _{2}, \varepsilon _{3},\varepsilon _{4}\right) ,\quad \sum _{a\in \underline{4}}\varepsilon _{a}=0. \end{aligned}$$
(132)

For any \(S\subset \underline{4}\), we define

$$\begin{aligned} \varepsilon _{S}=\sum _{a\in S}\varepsilon _{a}. \end{aligned}$$
(133)

In total, the symmetry group of \(\mathfrak {M}_{\mathbf {n},k}\) is \({\mathrm {PU}}\left( \mathbf {n}\right) \times {\mathrm {SU}}(4)\). If we adopt the holomorphic description (85), the symmetry group gets complexified, and its maximal torus is

$$\begin{aligned} \mathbf {T}=\mathbf {T}_{\mathbf {a}}\times \mathbf {T}_{\mathbf {\varepsilon }}= {\mathrm {GL}}(1,\mathbb {C})^{n_{\underline{4}^{\vee }}-1}\times {\mathrm {GL}}(1,\mathbb {C})^{3}. \end{aligned}$$
(134)

We denote

$$\begin{aligned} \mathbf {\texttt {a}}=\left\{ \texttt {a}_{\mathcal {A}},\mathcal {A}\in \underline{n}\right\} ,\quad \mathbf {\varepsilon }=\left\{ \varepsilon _{a},a\in \underline{4}\right\} , \end{aligned}$$
(135)

and

$$\begin{aligned} \mathbf {t}=\left\{ t_{\mathcal {A}}=e^{\beta \texttt {a}_{\mathcal {A}}},\mathcal {A}\in \underline{n}\right\} ,\quad \mathbf {q}=\left\{ q_{a}=e^{\beta \varepsilon _{a}},a\in \underline{4}\right\} . \end{aligned}$$
(136)

5 Instanton Partition Function from Equivariant Localization

In this section, we will compute the instanton partition function using equivariant localization theorem.

5.1 Fixed points

Generalizing the arguments of [88, 110], we can find the set \(\mathfrak {M}_{\mathbf {n},k}^{\mathbf {T}}\) of all \(\mathbf {T}\)-fixed points of \(\mathfrak {M}_{\mathbf {n},k}\). It is convenient to work with the holomorphic description (85). We also assume that all the parameters \(\mathbf {a},\mathbf {\varepsilon }\) take generic values. The nongeneric case is more complicated but can still be handled following [93].

We choose suitable bases for \(\mathbf {N}_{A},A\in \underline{4}^{\vee }\) so that they decompose into one-dimensional vector spaces,

$$\begin{aligned} \mathbf {N}_{A}=\bigoplus _{\alpha =1}^{n_{A}}\mathbf {N}_{A,\alpha }, \end{aligned}$$
(137)

with \(\mathbf {N}_{A,\alpha }\) being the eigenspace of \(\mathbf {T}_{\mathbf {a}}\) action with eigenvalue \(t_{A,\alpha }\). If \(\left( \mathbf {B},{\mathbf {I}}\right) \) is a \(\mathbf {T}\)-fixed point, it must be invariant under the combination of an arbitrary \(\mathbf {T}\)-transformation and a related \(\mathrm {GL}(k,\mathbb {C})\) gauge transformation,

$$\begin{aligned} B_{a}= & {} q_{a}gB_{a}g^{-1},\quad a\in \underline{4},\nonumber \\ I_{A,\alpha }= & {} gI_{A,\alpha }t_{A,\alpha }^{-1},\quad A\in \underline{4}^{\vee }. \end{aligned}$$
(138)

Hence \(g\left( \mathbf {t},\mathbf {q}\right) =e^{\beta \phi }\in \mathrm {GL}(k,\mathbb {C})\) defines a representation \(\mathbf {T}\rightarrow \mathrm {GL}(k,\mathbb {C})\). Since every irreducible complex representation of an abelian group is one-dimensional, we can decompose \(\mathbf {K}\) into the orthogonal direct sum

$$\begin{aligned} \mathbf {K}=\bigoplus _{A\in \underline{4}^{\vee }}\mathbf {K}_{A}= \bigoplus _{\mathcal {A}\in \underline{n}}\mathbf {K}_{A,\alpha }, \end{aligned}$$
(139)

where \(\mathbf {K}_{A,\alpha }\) is the eigenspace of \(\mathbf {T}_{\mathbf {a}}\) action with eigenvalue \(t_{A,\alpha }\), and can be further decomposed into a direct sum of eigenspaces of \(\mathbf {T}_{\mathbf {\varepsilon }}\). From (138), we have

$$\begin{aligned}&gB_{a}^{x-1}B_{b}^{y-1}B_{c}^{z-1}I_{A}\left( \mathbf {N}_{A,\alpha }\right) \nonumber \\&\quad = q_{a}^{1-x}q_{b}^{1-y}q_{c}^{1-z}t_{A,\alpha }B_{a}^{x-1}B_{b}^{y-1}B_{c}^{z-1}I_{A} \left( \mathbf {N}_{A,\alpha }\right) ,\quad x,y,z\ge 1. \end{aligned}$$
(140)

Thus, \(B_{a}^{x-1}B_{b}^{y-1}B_{c}^{z-1}I_{A}\left( \mathbf {N}_{A,\alpha }\right) \) is an eigenspace of \(\mathbf {T}\) with eigenvalue \(q_{a}^{1-x}q_{b}^{1-y}q_{c}^{1-z}t_{A,\alpha }\). Due to the stability condition, we must have

$$\begin{aligned} \mathbf {K}_{A=(abc),\alpha }=\bigoplus _{\left( x,y,z\right) \in \pi ^{(A,\alpha )}} B_{a}^{x-1}B_{b}^{y-1}B_{c}^{z-1}I_{A}\left( \mathbf {N}_{A,\alpha }\right) , \end{aligned}$$
(141)

where the set \(\pi ^{(A,\alpha )}\subset \mathbb {Z}_{+}^{3}\) contains \(k_{A,\alpha }=\dim \mathbf {K}_{A,\alpha }\) elements. It has been shown explicitly in [25] that all possible \(\pi ^{(A,\alpha )}\) are in one-to-one correspondence with plane partitions. Hence, each \(\mathbf {T}\)-fixed points of \(\mathfrak {M}_{\mathbf {n},k}\) is labeled by a collection of plane partitions

$$\begin{aligned} \varvec{\uppi }=\left\{ \pi ^{(\mathcal {A})},\mathcal {A}\in \underline{n}\right\} , \end{aligned}$$
(142)

such that the total volume of \(\varvec{\uppi }\) is k,

$$\begin{aligned} k=\left| \varvec{\uppi }\right| =\sum _{\mathcal {A}\in \underline{n}}\left| \pi ^{(\mathcal {A})}\right| . \end{aligned}$$
(143)

From the point of view of noncommutative field theory, each \(\mathbf {T}\)-fixed point is given by a tetrahedron instanton sitting near the origin of the spacetime whose solution is labeled with \(\varvec{\uppi }\). On the other hand, in the geometric language, each \(\mathbf {T}\)-fixed point corresponds to a decomposition \(\bigoplus _{\mathcal {A}=(A,\alpha )\in \underline{n}}\left( \mathcal {I}_{\mathcal {A}},\varPhi _{\mathcal {A}}\right) \), where \(\mathcal {I}_{\mathcal {A}}\) is an \(\mathbf {T}_{\mathbf {\varepsilon }}\)-invariant ideal sheaf supported on the \(\mathbf {T}_{\mathbf {\varepsilon }}\)-fixed zero-dimensional subscheme contained in \(\mathbb {C}_{A}^{3}=\mathbb {CP}_{A}^{3}\setminus \mathbb {CP}_{A,\infty }^{2}\), and the framing \(\varPhi _{A,\alpha }:\left. \mathcal {I}_{A,\alpha }\right| _{\mathbb {CP}_{A,\infty }^{2}} \cong \mathcal {O}_{\mathbb {CP}_{A,\infty }^{2}}\).

5.2 Tangent space

Now let us look at the holomorphic tangent space \(T_{\varvec{\uppi }}\mathfrak {M}_{\mathbf {n},k}\), where \(\varvec{\uppi }\) labels a fixed point \(\left( {\mathbf {B}},{\mathbf {I}}\right) \in \mathfrak {M}_{\mathbf {n},k}^{\mathbf {T}}\). If \(\left( {\mathbf {B}}+\mathbf {b},{\mathbf {I}}+\mathbf {i}\right) \in \mathfrak {M}_{\mathbf {n},k}\) is a nearby point, then \(\left( \mathbf {b},\mathbf {i}\right) \) should obey the linearization of the equations \(\mu ^{\mathbb {C}}=\sigma =0\),

$$\begin{aligned} d_{2}\left( \mathbf {b},\mathbf {i}\right) \equiv \left( \left[ b_{a},B_{b}\right] +\left[ B_{a},b_{b}\right] ,b_{\check{A}}I_{A}+B_{\check{A}}i_{A}\right) =0, \end{aligned}$$
(144)

up to an infinitesimal \(\mathrm {GL}(k,\mathbb {C})\)-transformation,

$$\begin{aligned} \left( b_{a},i_{A}\right) \sim \left( b_{a},i_{A}\right) +d_{1}\left( \phi \right) ,\quad d_{1}\left( \phi \right) \equiv \left( \left[ \phi ,B_{a}\right] ,\phi I_{A}\right) ,\quad \phi \in \mathfrak {gl}(k,\mathbb {C}). \end{aligned}$$
(145)

We have the following deformation complex,

(146)

whose middle cohomology group is isomorphic to the tangent space \(T_{\varvec{\uppi }}\mathfrak {M}_{\mathbf {n},k}\). We can compute the \(\mathbf {T}\)-equivariant Chern character of \(T_{\varvec{\uppi }}\mathfrak {M}_{\mathbf {n},k}\),

$$\begin{aligned} \chi _{\varvec{\uppi }}= & {} {\mathrm {Ch}}_{\mathbf {T}}\left( T_{\varvec{\uppi }}\mathfrak {M}_{\mathbf {n},k}\right) \nonumber \\= & {} -K_{\varvec{\uppi }}^{*}K_{\varvec{\uppi }}+K_{\varvec{\uppi }}^{*}K_{\varvec{\uppi }}{\mathrm {Ch}}_{\mathbf {T}} \left( \mathbb {C}^{4}\right) +N_{A}^{*}K_{\varvec{\uppi }}\nonumber \\&-K_{\varvec{\uppi }}^{*}K_{\varvec{\uppi }}{\mathrm {Ch}}_{\mathbf {T}}\left( \wedge ^{2,+}\mathbb {C}^{4}\right) -\sum _{A\in \underline{4}^{\vee }}N_{A}^{*}K_{\varvec{\uppi }}{\mathrm {Ch}}_{\mathbf {T}}\left( \wedge ^{3} \mathbb {C}_{A}^{3}\right) \nonumber \\= & {} -K_{\varvec{\uppi }}^{*}K_{\varvec{\uppi }}L+\sum _{A\in \underline{4}^{\vee }}N_{A}^{*}K_{\varvec{\uppi }} \left( 1-q_{A}^{-1}\right) , \end{aligned}$$
(147)

where \(\left( e^{\beta w}\right) ^{*}=e^{-\beta w}\), and

$$\begin{aligned} N_{A}= & {} {\mathrm {Ch}}_{\mathbf {T}}\left( \mathbf {N}_{A}\right) =\sum _{\alpha =1}^{n_{A}}t_{A,\alpha }, \end{aligned}$$
(148)
$$\begin{aligned} K_{\varvec{\uppi }}= & {} {\mathrm {Ch}}_{\mathbf {T}}\left( \mathbf {K}_{\varvec{\uppi }}\right) =\left. \sum _{i=1}^{k}e^{\beta \phi }\right| _{\varvec{\uppi }}\nonumber \\= & {} \sum _{A=(a,b,c)\in \underline{4}^{\vee }}\sum _{\alpha =1}^{n_{A}}t_{A,\alpha } \sum _{(x,y,z)\in \pi ^{(A,\alpha )}}q_{a}^{1-x}q_{b}^{1-y}q_{c}^{1-z}, \end{aligned}$$
(149)
$$\begin{aligned} L= & {} 1-{\mathrm {Ch}}_{\mathbf {T}}\left( \mathbb {C}^{4}\right) +{\mathrm {Ch}}_{\mathbf {T}} \left( \wedge ^{2,+}\mathbb {C}^{4}\right) \nonumber \\= & {} 1-\sum _{a\in \underline{4}}q_{a}^{-1}+q_{1}^{-1}q_{2}^{-1}+q_{1}^{-1}q_{3}^{-1}+q_{2}^{-1}q_{3}^{-1}. \end{aligned}$$
(150)

Notice that the normalized character (77) computed in noncommutative field theory can be related to \(N_{A}\) and \(K_{\varvec{\uppi }}\) by

$$\begin{aligned} \mathscr {E}_{A,\varvec{\uppi }}=N_{A}-\left( \prod _{a\in A}\left( 1-q_{a}^{-1}\right) \right) \left. K_{\varvec{\uppi }}\right| _{A}. \end{aligned}$$
(151)

5.3 Equivariant integrals

The \(\mathbf {T}\)-equivariant symplectic volume of \(\mathfrak {M}_{\mathbf {n},k}\) is defined as the integral of the \(\mathbf {T}\)-equivariant cohomology class \(1\in H_{\mathbf {T}}^{*}\left( \mathfrak {M}_{\mathbf {n},k}\right) \) over the virtual fundamental cycle [13, 73] of \(\mathfrak {M}_{\mathbf {n},k}\),

$$\begin{aligned} \mathcal {Z}_{k}\left( \mathbf {\texttt {a}};\mathbf {\varepsilon }\right) =\int _{\left[ \mathfrak {M}_{\mathbf {n},k}\right] ^{\mathrm {vir}}}1, \end{aligned}$$
(152)

where \(\left( \mathbf {\texttt {a}},\mathbf {\varepsilon }\right) \) are generators of \(H_{\mathbf {T}}^{*}\left( \mathrm {pt}\right) \). Since \(\mathfrak {M}_{\mathbf {n},k}\) is noncompact and is a union of manifolds of different actual dimensions, we should apply the Atiyah–Bott equivariant localization theorem [7] in the virtual approach [50] to evaluate the \(\mathbf {T}\)-equivariant integral,

$$\begin{aligned} \mathcal {Z}_{k}\left( \mathbf {\texttt {a}};\mathbf {\varepsilon }\right) =\sum _{\varvec{\uppi }, \left| \varvec{\uppi }\right| =k}\frac{1}{\mathrm {e_{\mathbf {T}}}\left( T_{\varvec{\uppi }}\mathfrak {M}_{\mathbf {n},k} \right) }=\sum _{\varvec{\uppi },\left| \varvec{\uppi }\right| =k}\mathbb {E}\left\{ -\chi _{\varvec{\uppi }}\right\} , \end{aligned}$$
(153)

where \(\mathrm {e_{\mathbf {T}}}\left( T_{\varvec{\uppi }}\mathfrak {M}_{\mathbf {n},k}\right) \) is the \(\mathbf {T}\)-equivariant Euler class of the tangent space of \(\mathfrak {M}_{\mathbf {n},k}\) at \(\varvec{\uppi }\), and the operator \(\mathbb {E}\) converts additive Chern characters to multiplicative classes,

(154)

where the \(w_{i}=0\) term should be excluded in the product. The instanton partition function is the generating function of \(\mathcal {Z}_{k}\left( \mathbf {\texttt {a}},\mathbf {\varepsilon }\right) \),

$$\begin{aligned} \mathcal {Z}\left( \mathbf {\texttt {a}};\mathbf {\varepsilon };\texttt {q}\right) =\sum _{k=0}^{\infty }{} \texttt {q}^{k}\mathcal {Z}_{k}\left( \mathbf {\texttt {a}}; \mathbf {\varepsilon }\right) =\sum _{\varvec{\uppi }}{} \texttt {q}^{\left| \varvec{\uppi }\right| } \mathbb {E}\left\{ -\chi _{\varvec{\uppi }}\right\} , \end{aligned}$$
(155)

where \(\texttt {q}\) is the instanton counting parameter. Notice that \(\chi _{\varvec{\uppi }}\) is not invariant under the permutations of \(q_{a}\). However, we have

$$\begin{aligned} L+L^{*}=\prod _{a\in \underline{4}}\left( 1-q_{a}^{-1}\right) . \end{aligned}$$
(156)

Therefore, \(\mathbb {E}\left\{ -\chi _{\varvec{\uppi }}\right\} \) is invariant under the permutations of \(q_{a}\), up to an overall \(\pm \,\) sign that depends on the ordering in \(a\in \underline{4}\). The orientation problem also appeared in the study of magnificent four model [95, 103].

We can obtain the K-theoretic and elliptic versions of the instanton partition function by replacing the integrand in (152) from 1 to the arithmetic genus \(\hat{A}_{\beta }\left( \mathfrak {M}_{\mathbf {n},k}\right) \) and the elliptic genus \(\varphi _{\mathrm {ell}}\left( \mathfrak {M}_{\mathbf {n},k}\right) \), respectively [12, 74]. Correspondingly, the definition of the operator \(\mathbb {E}\) becomes

(157)

In fact, the result (155) suggests a more refined version of the instanton partition function with four independent instanton counting parameters \(\texttt {q}_{A}\) for \(A\in \underline{4}^{\vee }\),

$$\begin{aligned} \mathcal {Z}^{\mathrm {ref}}\left( \mathbf {\texttt {a}}; \mathbf {\varepsilon };\mathbf {\texttt {q}}\right) =\sum _{\varvec{\uppi }}\prod _{A\in \underline{4}^{\vee }}{} \texttt {q}_{A}^{\left| \pi ^{(A)}\right| }\mathbb {E}\left\{ -\chi _{\varvec{\uppi }}\right\} , \end{aligned}$$
(158)

where \(\mathbf {\texttt {q}}=\left\{ \texttt {q}_{A},A\in \underline{4}^{\vee }\right\} \) and \(\left| \pi ^{(A)}\right| =\sum _{\alpha =1}^{n_{A}}\left| \pi ^{(A,\alpha )}\right| \).

6 Instanton Partition Function from Elliptic Genus

In this section, we will compute the instanton partition function from the elliptic genus of the low-energy worldvolume theory on \(\hbox {D}1\)-branes, where all the heavy stringy modes are decoupled.

6.1 Definition via elliptic genus

We have shown that the low-energy worldvolume theory on \(\hbox {D}1\)-branes probing a system of intersecting \(\hbox {D}7\)-branes is a two-dimensional \(\mathcal {N}=\left( 0,2\right) \) supersymmetric gauge theory, with two supercharges \(Q_{+}\) and \(\bar{Q}_{+}\). This theory has a \(\mathrm {U}(1)^{4}\) global symmetry induced from \(\prod _{a\in \underline{4}}\mathrm {SO}\left( 2\right) _{a}\) rotating \(\mathbb {C}^{4}\). The corresponding bosonic generators \(\mathcal {J}_{a}\) commute with each other, but do not commute with \(Q_{+}\) and \(\bar{Q}_{+}\),

$$\begin{aligned} \left[ \mathcal {J}_{a},Q_{+}\right] =-Q_{+},\quad \left[ \mathcal {J}_{a},\bar{Q}_{+}\right] =\bar{Q}_{+}. \end{aligned}$$
(159)

We can choose three linearly independent combinations of \(\mathcal {J}_{a}\), for instance

$$\begin{aligned} \left( \mathcal {J}_{1}-\mathcal {J}_{4},\mathcal {J}_{2} -\mathcal {J}_{4},\mathcal {J}_{3}-\mathcal {J}_{4}\right) , \end{aligned}$$
(160)

which commute with \(Q_{+}\) and \(\bar{Q}_{+}\). They generate a group \(\mathrm {U}(1)^{3}\subset \mathrm {U}(1)^{4}\), and can be identified with \(\mathbf {T}_{\mathbf {\varepsilon }}\).

The elliptic genus of the two-dimensional worldvolume theory on k \(\hbox {D}1\)-branes probing intersecting \(\hbox {D}7\)-branes is defined to be

$$\begin{aligned}&Z_{k}\left( \tau ;\varvec{\texttt {a}};\varepsilon _{1},\varepsilon _{2},\varepsilon _{3}\right) \nonumber \\&\quad = {\mathrm {Tr}}_{\mathcal {H}_{k}}\left[ \left( -1\right) ^{F}q^{H_{L}}\bar{q}^{H_{R}} \prod _{a=1}^{3}e^{2\pi \mathrm {i}\varepsilon _{a}\left( \mathcal {J}_{a}-\mathcal {J}_{4}\right) } \prod _{\mathcal {A}\in \underline{n}}e^{2\pi \mathrm {i}{} \texttt {a}_{\mathcal {A}}T_{\mathcal {A}}} \right] , \end{aligned}$$
(161)

where the trace is taken in the RR sector of the Hilbert space \(\mathcal {H}_{k}\) of the worldvolume theory, \(H_{L}\) and \(H_{R}\) are the left- and right-moving Hamiltonians respectively, and \(T_{\mathcal {A}=\left( A,\alpha \right) },\alpha =1,\cdots ,n_{A}\) are the Cartan generators of the symmetry group \(\mathrm {U}\left( n_{A}\right) \). The parameter

$$\begin{aligned} q=e^{2\pi \mathrm {i}\tau } \end{aligned}$$
(162)

specifies the complex structure \(\tau \) of a torus. The fugacities \(\left( \varepsilon _{1},\varepsilon _{2},\varepsilon _{3}\right) \) and \(\texttt {a}_{\mathcal {A}=\left( A,\alpha \right) }\) are associated with \(\mathrm {U}(1)^{3}\) and \(\mathrm {U}\left( n_{A}\right) \), respectively. We can introduce \(\varepsilon _{4}=-\varepsilon _{1}-\varepsilon _{2}-\varepsilon _{3}\) to make the expression more symmetric,

$$\begin{aligned} Z_{k}\left( \tau ;\mathbf {\texttt {a}};\mathbf {\varepsilon }\right) ={\mathrm {Tr}}_{\mathcal {H}_{k}}\left[ \left( -1\right) ^{F}q^{H_{L}} \bar{q}^{H_{R}}\prod _{a\in \underline{4}}e^{2\pi \mathrm {i} \varepsilon _{a}\mathcal {J}_{a}}\prod _{\mathcal {A}\in \underline{n}}e^{2\pi \mathrm {i}{} \texttt {a}_{\mathcal {A}}T_{\mathcal {A}}}\right] _{\sum _{a\in \underline{4}}\varepsilon _{a}=0}. \end{aligned}$$
(163)

It is clear that \(\varepsilon _{a},a\in \underline{4}\) can be identified with the standard \(\varOmega \)-deformation parameters [110]. The instanton partition function is then the grand canonical partition function of the elliptic genus,

$$\begin{aligned} Z^{\mathrm {inst}}\left( \tau ;\mathbf {\texttt {a}};\mathbf {\varepsilon };\texttt {q}\right) =1+\sum _{k=1}^{\infty }{} \texttt {q}^{k}Z_{k}\left( \tau ;\mathbf {\texttt {a}};\mathbf {\varepsilon }\right) . \end{aligned}$$
(164)

The elliptic genus can be calculated using the supersymmetric localization techniques, and is given by contour integrals [17],

$$\begin{aligned} Z_{k}=\frac{1}{k!}\int \prod _{i=1}^{k}d\phi _{i}\left( Z_{k}^{1-1} \prod _{A\in \underline{4}^{\vee }}Z_{k}^{1-7_{A}}\right) , \end{aligned}$$
(165)

where k! is the order of the Weyl group of \(\mathrm {U}(k)\). The contributions from the \(\hbox {D}1\)\(\hbox {D}1\) strings and \(\hbox {D}1\)\(\hbox {D}7_{A}\) strings are [15]

$$\begin{aligned} Z_{k}^{1-1}= & {} \left[ \frac{2\pi \eta (\tau )^{3}\prod _{1\le a<b\le 3}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}} \theta \left( \varepsilon _{a}\right) }\right] ^{k}\nonumber \\&\times \prod _{\begin{array}{c} i,j=1\\ i\ne j \end{array} }^{k}\frac{\theta \left( \phi _{ij}\right) \prod _{1\le a<b\le 3}\theta \left( \phi _{ij}+\varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( \phi _{ij}+\varepsilon _{a}\right) }, \end{aligned}$$
(166)
$$\begin{aligned} Z_{k}^{1-7_{A}}= & {} \prod _{i=1}^{k}\prod _{\alpha =1}^{n_{A}} \frac{\theta \left( \phi _{i}-\texttt {a}_{A,\alpha }-\varepsilon _{A}\right) }{\theta \left( \phi _{i}-\texttt {a}_{A,\alpha }\right) }, \end{aligned}$$
(167)

where \(\phi _{ij}=\phi _{i}-\phi _{j}\), and we use the abbreviationFootnote 1

$$\begin{aligned} \theta \left( z\right) \equiv \theta _{1}\left( \left. z\right| \tau \right) . \end{aligned}$$
(168)

We emphasize that the detailed description of \(\mathfrak {M}_{\mathbf {n},k}\) is not used in the computation. We only need to know the supermultiplets that appear in the worldvolume theory, as well as their charges under the symmetry group \(\mathbf {T}\).

We see that \(Z^{\mathrm {inst}}\) is invariant under an overall shift,

$$\begin{aligned} \phi _{i}\rightarrow \phi _{i}-\xi ,\quad \texttt {a}_{A,\alpha }\rightarrow \texttt {a}_{A,\alpha }+\xi . \end{aligned}$$
(169)

This confirms the claim that the center \(\mathrm {U}(1)_{c}\) of \(\prod _{A\in \underline{4}^{\vee }}\mathrm {U}\left( n_{A}\right) \) acts trivially, and the final result of the partition function should not dependent on the overall shift of \(\texttt {a}_{A,\alpha }\).

The integral in (165) make sense only when the integrand is invariant under the large gauge transformations \(\phi _{i}\rightarrow \phi _{i}+r+s\tau \) for \(r,s\in \mathbb {Z}\) [15, 17]. From the transformation property of the Jacobi theta function \(\theta _{1}\left( \left. z\right| \tau \right) \) under shifts of z,

$$\begin{aligned} \theta _{1}\left( \left. z+r+s\tau \right| \tau \right) =\left( -1\right) ^{r+s} \exp \left( -\pi \mathrm {i}s^{2}\tau -2\pi \mathrm {i}sz\right) \theta _{1} \left( \left. z\right| \tau \right) ,\quad r,s\in \mathbb {Z}, \end{aligned}$$
(170)

we obtain that

$$\begin{aligned} Z_{k}\rightarrow \left( \prod _{A\in \underline{4}^{\vee }}\prod _{i=1}^{k}\prod _{\alpha =1}^{n_{A}}e^{2\pi \mathrm {i}s\varepsilon _{A}}\right) Z_{k}. \end{aligned}$$
(171)

To get rid of the extra phase factor for all \(k\in \mathbb {Z}^{+}\), we should impose the consistency condition

$$\begin{aligned} \sum _{A\in \underline{4}^{\vee }}n_{A}\varepsilon _{A}\in \mathbb {Z}, \end{aligned}$$
(172)

which generalizes the similar condition obtained in [15].

6.2 \(k=1\)

When \(k=1\), the elliptic genus is given by

$$\begin{aligned} Z_{1}=\left[ \frac{2\pi \eta (\tau )^{3}\prod _{1\le a<b\le 3}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( \varepsilon _{a}\right) }\right] \int d\phi \prod _{\mathcal {A}\in \underline{n}} \frac{\theta \left( \phi -\texttt {a}_{\mathcal {A}}-\varepsilon _{\mathcal {A}}\right) }{\theta \left( \phi -\texttt {a}_{\mathcal {A}}\right) }. \end{aligned}$$
(173)

It is straightforward to evaluate this integral explicitly. The set of poles in the integrand are

$$\begin{aligned} \mathscr {M}_{*}^{\mathrm {sing}}=\left\{ \left. \phi \right| \phi -\texttt {a}_{\mathcal {A}}=0\mod \mathbb {Z}+\tau \mathbb {Z}\right\} . \end{aligned}$$
(174)

We should take all of them and the result is given by

$$\begin{aligned} Z_{1}= & {} \left[ \frac{2\pi \eta (\tau )^{3}\prod _{1\le a<b\le 3}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( \varepsilon _{a}\right) }\right] \sum _{\phi _{*}\in \mathscr {M}^{\mathrm {sing}}}\oint _{\phi _{*}} d\phi \prod _{\mathcal {A}\in \underline{n}}\frac{\theta \left( \phi -\texttt {a}_{\mathcal {A}} -\varepsilon _{\mathcal {A}}\right) }{\theta \left( \phi -\texttt {a}_{\mathcal {A}}\right) }\nonumber \\= & {} \sum _{\mathcal {A}=\left( A,\alpha \right) \in \underline{n}}\left[ \frac{\prod _{a<b\in A}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in A}\theta \left( \varepsilon _{a}\right) }\prod _{\mathcal {B}\in \underline{n}\setminus \left\{ \mathcal {A}\right\} }\frac{\theta \left( \texttt {a}_{\mathcal {A}}-\texttt {a}_{\mathcal {B}}-\varepsilon _{\mathcal {B}}\right) }{\theta \left( \texttt {a}_{\mathcal {A}}-\texttt {a}_{\mathcal {B}}\right) }\right] , \end{aligned}$$
(175)

where we have used \(\sum _{a\in \underline{4}}\varepsilon _{a}=0\). Due to the product over \(a<b\in A\), the result depends on the ordering of \(a\in \underline{4}\).

6.3 General k

Now we proceed with general k. As shown in [17], we should apply the JK residue formula [58] to evaluate the contour integrals in (165).

6.3.1 Classification of potential poles in terms of trees

We first classify all the potential poles in the integrand that can have nonzero JK-residues, temporarily ignoring the numerator.

The denominator of the integral (165) becomes zero along the hyperplanes

$$\begin{aligned} H_{A,ij,a}= & {} \left\{ \phi _{i}-\phi _{j}=-\varepsilon _{a}\right\} , \end{aligned}$$
(176)
$$\begin{aligned} H_{F,i,\mathcal {A}}= & {} \left\{ \phi _{i}=\texttt {a}_{\mathcal {A}}\right\} , \end{aligned}$$
(177)

where the identifications up to \(\mathbb {Z}+\tau \mathbb {Z}\) are understood. We introduce the standard basis \(\left\{ \mathbf {e}_{i}\right\} _{i=1,\cdots ,k}\) of \(\mathbb {R}^{k}\),

$$\begin{aligned} \mathbf {e}_{i}=\left( 0,\cdots ,0,\overset{i}{1},0,\cdots ,\overset{k}{0}\right) . \end{aligned}$$
(178)

The charge vectors associated with (176) and (177) are \(\mathbf {h}_{A,ij}=\mathbf {e}_{i}-\mathbf {e}_{j}\) and \(\mathbf {h}_{F,i}=\mathbf {e}_{i}\), respectively.

A singularity is called nondegenerate if exactly k linearly independent hyperplanes intersect at the point, and is called degenerate if the total number of hyperplanes through the point is greater than k. A practical way to deal with the degenerate singularities is to blowup them into nondegenerate ones by introducing small generic nonphysical fugacities to deform the hyperplane arrangement. In the end of the computation, we remove the deformation by sending the nonphysical fugacities to zero in a continuous way. In the following, we will only consider the situation where all singularities are nondegenerate.

We denote the charge vectors of the k hyperplanes by

$$\begin{aligned} \mathbf {Q}=\begin{pmatrix}\mathbf {Q}_{1}\\ \vdots \\ \mathbf {Q}_{k} \end{pmatrix},\quad \mathbf {Q}_{I}\in \left\{ \mathbf {h}_{A},\mathbf {h}_{F}\right\} . \end{aligned}$$
(179)

The JK-residue can be nonzero only if \(\varvec{\eta }\in {\mathrm {Cone}}\left( \mathbf {Q}\right) \), i.e.,

$$\begin{aligned} \sum _{I=1}^{k}\lambda _{I}\mathbf {Q}_{I}=\varvec{\eta },\quad \lambda _{I}>0. \end{aligned}$$
(180)

In our problem, the result will depend on \(\varvec{\eta }\), and we should take the standard choiceFootnote 2

$$\begin{aligned} \varvec{\eta }=\sum _{i=1}^{k}\mathbf {e}_{i}=\left( 1,1,\cdots ,1\right) . \end{aligned}$$
(181)

Since charge vectors of type \(\mathbf {h}_{A}\) only generate at most a \((k-1)\)-dimensional subspace of \(\mathbb {R}^{k}\), \(\mathbf {Q}\) must contain \(M\ge 1\) charge vectors of type \(\mathbf {h}_{F}\), which are taken to be \(\mathbf {e}_{1},\cdots ,\mathbf {e}_{M}\) using Weyl permutations. We will show that it is possible to divide \(\mathbf {Q}\) into M subsets in such a way that each subset contains exactly one charge vector of type \(\mathbf {h}_{F}\). Let us start with \(\mathbf {e}_{1}\). If \(\mathbf {Q}_{j_{1}}=\mathbf {e}_{1}-\mathbf {e}_{j_{1}}\) is also in \(\mathbf {Q}\), the condition (180) gives

$$\begin{aligned} \left( \lambda _{1}+\lambda _{j_{1}}\right) \mathbf {e}_{1}+\sum _{I\ne 1,j_{1}}\lambda _{I}\mathbf {Q}_{I} =\lambda _{j_{1}}\mathbf {e}_{j_{1}}+\sum _{i=1}^{k}\mathbf {e}_{i}. \end{aligned}$$
(182)

Since the coefficient of \(\mathbf {e}_{j_{1}}\) on the right-hand side is positive, \(\mathbf {Q}\) must contain \(\mathbf {e}_{j_{1}}-\mathbf {e}_{j_{2}}\) for at least one \(j_{2}\). Notice that \(\mathbf {e}_{j_{1}}\) cannot be in \(\mathbf {Q}\), since it is not linearly independent with \(\mathbf {e}_{1}\) and \(\mathbf {e}_{1}-\mathbf {e}_{j_{1}}\) that are already in \(\mathbf {Q}\). Then the same argument for \(\mathbf {e}_{j_{2}}\) leads to the requirement that \(\mathbf {Q}\) must contain \(\mathbf {e}_{j_{2}}-\mathbf {e}_{j_{3}}\) for at least one \(j_{3}\ne 1,j_{1}\). Since there are only a finite number of elements in \(\mathbf {Q}\), this procedure cannot be carried on forever, and finally it is impossible to match the coefficient of one \(\mathbf {e}_{i}\). Therefore, \(\mathbf {e}_{1}-\mathbf {e}_{j}\) is not allowed to be in \(\mathbf {Q}\).

On the contrary, \(\mathbf {Q}\) can contain one or more charge vectors \(\mathbf {e}_{j_{1}^{(\mu )}}-\mathbf {e}_{1}\), which are labeled by \(\mu =1,\cdots \) , and we require \(j_{1}^{(\mu )}>M\) in order to avoid linearly dependent combinations of charge vectors. We can draw an oriented rooted tree. The root vertex is labeled by \(\mathbf {e}_{1}\). For each \(\mathbf {e}_{j_{1}^{(\mu )}}-\mathbf {e}_{1}\in \mathbf {Q}\), we put an arrow from \(\mathbf {e}_{1}\) to the vertex \(\mathbf {e}_{j_{1}^{(\mu )}}\). We can go on and add \(\mathbf {e}_{j_{2}^{(\nu )}}-\mathbf {e}_{j_{1}^{(\mu )}}\) in \(\mathbf {Q}\), with \(j_{2}^{(\nu )}\) being different from \(1,\cdots ,M\) and \(j_{1}^{(\mu )}\) so that there are no linear relations among selected charge vectors. The tree grows by adding the vertices \(\mathbf {e}_{j_{2}^{(\nu )}}\) and arrows from \(\mathbf {e}_{j_{1}^{(\mu )}}\) to \(\mathbf {e}_{j_{2}^{(\nu )}}\). We can repeat this construction until no charge vectors can be further added in this way, ending up with an oriented rooted tree with root \(\mathbf {e}_{1}\) and arrows corresponding to charge vectors of type \(\mathbf {h}_{A,ij},i>j\). The linearly independent condition ensures that there can be no cycles. Subsequently, we can proceed with \(\mathbf {e}_{2}\), and produce a similar oriented rooted tree. The trees with root \(\mathbf {e}_{1}\) and \(\mathbf {e}_{2}\) must be disconnected, otherwise there will be linear relations among charge vectors. After performing this construction for all \(\mathbf {e}_{1},\cdots ,\mathbf {e}_{M}\), we divide all the charge vectors in \(\mathbf {Q}\) into a disjoint union of M oriented rooted trees, with k vertices in total.

It is convenient to perform a Weyl permutation of \(\phi _{i}\) so that \(\mathbf {Q}\) form a block diagonal matrix,

$$\begin{aligned} \mathbf {Q}={\mathrm {diag}}\left( \mathbf {Q}^{(1)},\cdots ,\mathbf {Q}^{(M)}\right) , \end{aligned}$$
(183)

where the block \(\mathbf {Q}^{(m)}\) is a square matrix of order \(k_{m}\),

$$\begin{aligned} \mathbf {Q}^{(m)}=\begin{pmatrix}1 &{} 0 &{} 0 &{} 0 &{} \cdots &{} 0\\ -1 &{} 1 &{} 0 &{} 0 &{} \cdots &{} 0\\ *&{} *&{} 1 &{} 0 &{} \cdots &{} 0\\ *&{} *&{} *&{} 1 &{} \cdots &{} 0\\ \vdots &{} \vdots &{} \vdots &{} \vdots &{} \ddots &{} \vdots \\ *&{} *&{} *&{} *&{} \cdots &{} 1 \end{pmatrix},\quad m=1,\dots ,M,\quad \sum _{m=1}^{M}k_{m}=k. \end{aligned}$$
(184)

The first row in \(\mathbf {Q}^{(m)}\) corresponds to the root of the m-th tree, and the remaining rows correspond to the other vertices of the m-th tree. Each \(*\) can be either 0 or \(-1\), and there is exactly one \(-1\) in each row containing \(*\). We relabel the poles \(\phi _{i}\) by

$$\begin{aligned} \phi _{m,l}\equiv \phi _{l+\sum _{j=1}^{m-1}k_{j}},\quad l=1,\dots ,k_{m}. \end{aligned}$$
(185)

The positions of poles are solutions to the equations

$$\begin{aligned} \mathbf {Q}^{(m)}\begin{pmatrix}\phi _{m,1}\\ \vdots \\ \phi _{m,k_{m}} \end{pmatrix}=\begin{pmatrix}\gamma _{m,1}\\ \vdots \\ \gamma _{m,k_{m}} \end{pmatrix}, \end{aligned}$$
(186)

where

$$\begin{aligned} \gamma _{m,l}\in {\left\{ \begin{array}{ll} \left\{ \texttt {a}_{\mathcal {A}},\mathcal {A}\in \underline{n}\right\} , &{}\quad l=1\\ \left\{ -\varepsilon _{a},a\in \underline{4}\right\} , &{}\quad l>1 \end{array}\right. }. \end{aligned}$$
(187)

In particular, we can have an injective map

$$\begin{aligned} \varrho :\left\{ 1,\cdots ,M\right\} \rightarrow \underline{n}, \end{aligned}$$
(188)

and the pole corresponding to the root of the m-th tree is

$$\begin{aligned} \phi _{m,1}=\texttt {a}_{\varrho (m)}. \end{aligned}$$
(189)

We can decorate the trees associated with \(\mathbf {Q}\) into trees describing potential poles that can have nonvanishing JK-residues by assigning \(\varrho (m)\) to the root of the m-th tree, and painting each arrow by the a-th color if the pole associated with the target vertex differs from the pole associated with the source vertex by \(-\varepsilon _{a}\).

6.3.2 Classification of genuine poles in terms of colored plane partitions

There is an important flaw in the above classification of poles that can give nonvanishing JK-residues, because the denominator can have extra zeros from linearly dependent hyperplanes, and the zeros in the numerator will cancel some zeros in the denominator. We define the genuine poles to be the poles that indeed give nonvanishing JK-residues. These genuine poles must be contained in the set of potential poles found above.

We claim that the genuine poles \(\phi _{m,l}\) are completely classified by a collection of colored plane partitions,

$$\begin{aligned} \varvec{\uppi }=\left\{ \pi ^{(\mathcal {A})},\mathcal {A}\in \underline{n}\right\} , \end{aligned}$$
(190)

where each \(\pi ^{(\mathcal {A})}\) is restricted to be a plane partition, and we allow some of \(\pi ^{(\mathcal {A})}\) to be empty. If there are M non-empty plane partitions in \(\varvec{\uppi }\), then we can introduce a bijective map

$$\begin{aligned} \varrho :\left\{ 1,\cdots ,M\right\} \rightarrow \left\{ \left. \mathcal {A}\in \underline{n} \right| \pi ^{(\mathcal {A})}\ne \emptyset \right\} , \end{aligned}$$
(191)

and the poles labeled by \(\varvec{\uppi }\) are at

$$\begin{aligned} \phi _{m,s}=\texttt {a}_{\mathcal {A}}+(1-x)\varepsilon _{a}+(1-y)\varepsilon _{b}+(1-z)\varepsilon _{c}, \end{aligned}$$
(192)

where \(\varrho (m)=\mathcal {A}=\left( abc,\alpha \right) \) and \(s=\left( x,y,z\right) \in \pi ^{(\mathcal {A})}\). This claim can be proved by induction on k as follows.

For \(k=1\), all the allowed poles are at \(\left\{ \texttt {a}_{\mathcal {A}},\mathcal {A}\in \underline{n}\right\} \). For each given pole, there is only one nonempty plane partition in \(\varvec{\uppi }\), and is given by \(\left\{ (1,1,1)\right\} \in \mathbb {Z}_{+}^{3}\). We have shown that they give nonvanishing contributions to \(Z_{1}\). Hence, the claim holds for the base case.

We assume that the claim is true for \(k-1\) and examine it for k. If all the blocks of \(\mathbf {Q}\) are one dimensional, then the poles are at

$$\begin{aligned} \phi _{i}=\texttt {a}_{\varrho (i)}, \end{aligned}$$
(193)

with the map \(\varrho :\left\{ 1,\cdots ,k\right\} \rightarrow \left\{ \left. \mathcal {A}\in \underline{n}\right| \pi ^{(\mathcal {A})}\ne \emptyset \right\} \). There are k nonempty plane partitions in \(\varvec{\uppi }\), with each one being \(\left\{ (1,1,1)\right\} \in \mathbb {Z}_{+}^{3}\). All of them will give nonzero contributions to the JK-residue, and the claim holds.

We then consider the case when \(\mathbf {Q}\) is that it contains at least one charge vector of type \(\mathbf {h}_{A,ij},i>j\). Up to Weyl permutations, we can always arrange the k hyperplanes so that the charge vectors of the first \((k-1)\) hyperplanes only contain \(\mathbf {e}_{1},\cdots ,\mathbf {e}_{k-1}\), and the charge vector of the last hyperplane \(H_{k}\) is \(\mathbf {Q}_{k}=\mathbf {e}_{k}-\mathbf {e}_{J}\) with a fixed J. From the picture of trees, \(H_{k}\) is associated with the arrow from \(\mathbf {e}_{J}\) to \(\mathbf {e}_{k}\) and \(\mathbf {e}_{k}\) is not the source of any other arrow. In other words, \(\mathbf {e}_{k}\) corresponds to an end of a tree with multiple vertices. The integrand which contains \(\phi _{1},\cdots ,\phi _{k-1}\) but not \(\phi _{k}\) is precisely the integrand for the instanton number \(k-1\). The poles \(\phi _{1},\cdots ,\phi _{k}\) can contribute to the JK-residue if \(\sum _{I=1}^{k}\lambda _{I}\mathbf {Q}_{I}=\varvec{\eta }\) with \(\lambda _{I}>0\), which leads to

$$\begin{aligned} \sum _{I=1}^{k-1}\lambda _{I}\mathbf {Q}_{I}=\left( \sum _{i=1}^{k}\mathbf {e}_{i}\right) -\lambda _{k}\left( \mathbf {e}_{k}-\mathbf {e}_{J}\right) . \end{aligned}$$
(194)

Because the left-hand side does not contain \(\mathbf {e}_{k}\), we need \(\lambda _{k}=1\) and

$$\begin{aligned} \sum _{I=1}^{k-1}\lambda _{I}\mathbf {Q}_{I}=\left( \sum _{i=1}^{k-1}\mathbf {e}_{i}\right) +\mathbf {e}_{J}. \end{aligned}$$
(195)

Since the right-hand side is in the same chamber as \(\left( \sum _{i=1}^{k-1}\mathbf {e}_{i}\right) \), we know that the poles \(\phi _{1},\cdots ,\phi _{k-1}\) must also contribute to the JK-residue. Therefore, the genuine poles for \(\phi _{1},\cdots ,\phi _{k}\) can be obtained by first giving the genuine poles for \(\phi _{1},\cdots ,\phi _{k-1}\), and then determining the proper position of the pole \(\phi _{k}\) by choosing \(H_{k}\). By the induction hypothesis, the genuine poles for \(\phi _{1},\cdots ,\phi _{k-1}\) are labeled by a collection \(\varvec{\uppi }\) of colored plane partitions with \(\left| \varvec{\uppi }\right| =k-1\). We need to show that there is a bijection between the possible choices of \(H_{k}\) giving nonzero k-dimensional JK-residue and the ways of making a collection \(\varvec{\uppi }^{\prime }\) of colored plane partitions with \(\left| \varvec{\uppi }^{\prime }\right| =k\) from \(\varvec{\uppi }\) by adding a box. Without loss of generality, we assume that \(\phi _{k}\) is in a tree whose root vertex corresponds to the pole at \(\texttt {a}_{123,1}=\texttt {a}_{*}\). Based on our assumption of \(H_{k}\), the potential pole for \(\phi _{k}\) is \(\phi _{k}=\phi _{J}-\varepsilon _{a}\) for \(a\in \underline{4}\). Accordingly, adding \(H_{k}\) can only deform \(\pi ^{(123,1)}=\pi _{*}\), leaving the other colored plane partitions invariant. We can factorize the integrand of \(Z_{k}\) into two parts,

$$\begin{aligned} Z_{k}^{1-1}\prod _{A\in \underline{4}^{\vee }}Z_{k}^{1-7_{A}}=\left( Z_{k}^{1-1} \prod _{A\in \underline{4}^{\vee }}Z_{k}^{1-7_{A}}\right) ^{\mathrm {reg}}\times I_{k}. \end{aligned}$$
(196)

Here the regular part contains neither zeros nor poles in the neighborhood of \(\phi _{k}\rightarrow \phi _{J}-\varepsilon _{a}\), and \(I_{k}\) is given by

$$\begin{aligned} I_{k}=\frac{f\left( 0\right) f\left( \varepsilon _{12}\right) f\left( \varepsilon _{13} \right) f\left( \varepsilon _{23}\right) }{\prod _{a\in \underline{4}}f\left( \varepsilon _{a}\right) } \times \theta \left( \phi _{k}-\texttt {a}_{*}-\varepsilon _{123}\right) , \end{aligned}$$
(197)

where

$$\begin{aligned} f(x)=\prod _{s\in \pi _{*}}\left( \theta \left( \phi _{k}-c_{s}+x\right) \theta \left( c_{s}-\phi _{k}+x\right) \right) , \end{aligned}$$
(198)

and

$$\begin{aligned} c_{s=(x,y,z)}=\texttt {a}_{*}+(1-x)\varepsilon _{1} +(1-y)\varepsilon _{2}+(1-z)\varepsilon _{3}. \end{aligned}$$
(199)

If \(\phi _{J}=\texttt {a}_{*}\) corresponding to the box \((1,1,1)\in \pi _{*}\), then the factor \(\theta \left( \phi _{k}-\texttt {a}_{*}-\varepsilon _{123}\right) \) in the numerator cancels the factor \(\theta \left( \phi _{k}-\texttt {a}_{*}+\varepsilon _{4}\right) \) in the denominator using the constraint \(\sum _{a\in \underline{4}}\varepsilon _{a}=0\), and the genuine poles are \(\phi _{k}=\texttt {a}_{*}-\varepsilon _{a}\) for \(a\in \left\{ 1,2,3\right\} \). In the following, we assume that \(\phi _{J}\) corresponds to the box \(\left( x,y,z\right) \in \pi _{*}\setminus \left\{ (1,1,1)\right\} \), then the potential poles are

$$\begin{aligned} \phi _{k}=\texttt {a}_{*}+\left( 1-x^{\prime }\right) \varepsilon _{1} +\left( 1-y^{\prime }\right) \varepsilon _{2}+\left( 1-z^{\prime }\right) \varepsilon _{3}, \end{aligned}$$
(200)

with four possibilities

$$\begin{aligned} \left( x^{\prime },y^{\prime },z^{\prime }\right) \in \left\{ \left( x+1,y,z\right) , \left( x,y+1,z\right) ,\left( x,y,z+1\right) ,\left( x-1,y-1,z-1\right) \right\} . \end{aligned}$$
(201)

When the box \(\left( x^{\prime },y^{\prime },z^{\prime }\right) \) is already contained in \(\pi _{*}\), the numerator of \(I_{k}\) contains a double zero from

$$\begin{aligned} \theta \left( \phi _{k}-c_{\left( x^{\prime },y^{\prime },z^{\prime }\right) } \right) \theta \left( c_{\left( x^{\prime },y^{\prime },z^{\prime }\right) }-\phi _{k}\right) , \end{aligned}$$
(202)

and the residue vanishes. Therefore, there can be at most one box at each \(\left( x,y,z\right) \in \pi _{*}^{\prime }\). We denote the combination of the plane partition \(\pi _{*}\) and the box \(\left( x^{\prime },y^{\prime },z^{\prime }\right) \) by \(\pi _{*}^{\prime }\). We need to show that if \(\pi _{*}^{\prime }\) is not a plane partition, then the residue is zero.

If \(\left( x^{\prime },y^{\prime },z^{\prime }\right) \) is one of the boxes \(\left( x+1,y,z\right) \), \(\left( x,y+1,z\right) \), and \(\left( x,y,z+1\right) \), the box \(\left( x,y,z\right) \in \pi _{*}\setminus \left\{ (1,1,1)\right\} \) must sit on the boundary of \(\pi _{*}\). We can focus on the case \(\left( x^{\prime },y^{\prime },z^{\prime }\right) =\left( x+1,y,z\right) \), and the other cases can be obtained by simple permutations. We want to count the order \(\varDelta \) of singularity for a potential pole \(\phi _{k}\), which is the number of poles from the denominator minus the number of zeros from the numerator. The residue is nonzero when \(\varDelta =1\). We need to further make the following distinction:

  • When \(y=z=1\), \(\pi _{*}^{\prime }\) is a plane partition. \(I_{k}\) only contains a pole from \(\theta \left( \phi _{k}-c_{\left( x,1,1\right) }+\varepsilon _{1}\right) \), and therefore the residue is nonzero.

  • When \(y>1\) and \(z=1\) (by exchanging y and z we can get results for \(z>1\) and \(x=y=1\)), \(\pi _{*}^{\prime }\) is a plane partition if and only if

    $$\begin{aligned} \left( x+1,y-1,1\right) \in \pi _{*}. \end{aligned}$$
    (203)

    The poles and the zero of \(I_{k}\) are

    $$\begin{aligned} {\mathrm {poles}}&:&{\left\{ \begin{array}{ll} \theta \left( \phi _{k}-c_{\left( x,y,1\right) }+\varepsilon _{1}\right) ,\\ \theta \left( \phi _{k}-c_{\left( x+1,y-1,1\right) }+\varepsilon _{2}\right) , &{} \mathrm {if}\ \left( x+1,y-1,1\right) \in \pi _{*}^{\prime } \end{array}\right. }\nonumber \\ {\mathrm {zero}}&:&\theta \left( \phi _{k}-c_{\left( x,y-1,1\right) }+\varepsilon _{12}\right) . \end{aligned}$$
    (204)

    If \(\pi _{*}^{\prime }\) is a plane partition, \(\varDelta =1\), and the residue is nonzero. On the other hand, if \(\left( x+1,y-1,1\right) \notin \pi _{*}\) so that \(\pi _{*}^{\prime }\) is not a plane partition, \(\varDelta =0\), and the residue vanishes.

  • When \(y,z>1\), \(\pi _{*}^{\prime }\) is a plane partition if and only if

    $$\begin{aligned} \left( x+1,y-1,z\right) ,\left( x+1,y,z-1\right) \in \pi _{*}. \end{aligned}$$
    (205)

    The poles and zeros of \(I_{k}\) are

    $$\begin{aligned} {\mathrm {poles}}&:&{\left\{ \begin{array}{ll} \theta \left( \phi _{k}-c_{\left( x,y,z\right) }+\varepsilon _{1}\right) ,\\ \theta \left( c_{\left( x,y-1,z-1\right) }-\phi _{k}+\varepsilon _{4}\right) ,\\ \theta \left( \phi _{k}-c_{\left( x+1,y-1,z\right) }+\varepsilon _{2}\right) , &{} \mathrm {if}\ \left( x+1,y-1,z\right) \in \pi _{*}\\ \theta \left( \phi _{k}-c_{\left( x+1,y,z-1\right) }+\varepsilon _{3}\right) , &{} \mathrm {if}\ \left( x+1,y,z-1\right) \in \pi _{*} \end{array}\right. }\nonumber \\ \mathrm {zeros}&:&{\left\{ \begin{array}{ll} \theta \left( \phi _{k}-c_{\left( x,y-1,z\right) }+\varepsilon _{12}\right) ,\\ \theta \left( \phi _{k}-c_{\left( x,y,z-1\right) }+\varepsilon _{13}\right) ,\\ \theta \left( \phi _{k}-c_{\left( x+1,y-1,z-1\right) }+\varepsilon _{23}\right) , &{} \mathrm {if}\ \left( x+1,y-1,z-1\right) \in \pi _{*} \end{array}\right. } \end{aligned}$$
    (206)

    Since \(\left( x+1,y-1,z-1\right) \in \pi _{*}\) is automatically satisfied when \(\left( x+1,y-1,z\right) \in \pi _{*}\) or \(\left( x+1,y,z-1\right) \in \pi _{*}\), we can have \(\varDelta =1\) so that the residue is nonzero only if \(\pi _{*}^{\prime }\) is a plane partition.

If \(\left( x^{\prime },y^{\prime },z^{\prime }\right) =\left( x-1,y-1,z-1\right) \), the residue can be nonzero only if \(\left( x-1,y,z\right) \notin \pi _{*}\), since the numerator would contain a zero from \(\theta \left( c_{\left( x-1,y,z\right) }-\phi _{k}+\varepsilon _{23}\right) \) otherwise. Similarly, \(\pi _{*}\) cannot contain \(\left( x,y-1,z\right) \) and \(\left( x,y,z-1\right) \). However, this is in contradiction to the assumption that \(\left( x,y,z\right) \in \pi _{*}\setminus \left\{ (1,1,1)\right\} \) and \(\pi _{*}\) is a plane partition. Therefore, taking \(\left( x^{\prime },y^{\prime },z^{\prime }\right) =\left( x-1,y-1,z-1\right) \) will always lead to a vanishing residue.

In summary, we have shown that all the genuine poles of \(\phi _{k}\) are in one-to-one correspondence with the possibilities of adding a box to \(\varvec{\uppi }\) to make a collection of colored plane partitions.

6.3.3 Expression

Eventually, we obtain the elliptic genus \(Z_{k}\),

$$\begin{aligned} Z_{k}=\sum _{\varvec{\uppi },\left| \varvec{\uppi }\right| =k}Z_{\varvec{\uppi }}. \end{aligned}$$
(207)

We define

$$\begin{aligned} \mathcal {C}_{\mathcal {A},s}=\texttt {a}_{\mathcal {A}}+(1-x) \varepsilon _{a}+(1-y)\varepsilon _{b}+(1-z)\varepsilon _{c}, \end{aligned}$$
(208)

for \(\mathcal {A}=\left( abc,\alpha \right) \in \underline{n}\) and \(s=\left( x,y,z\right) \in \pi ^{(\mathcal {A})}\), and

$$\begin{aligned} \mathcal {D}_{\mathcal {B},t}^{\mathcal {A},s}=\mathcal {C}_{\mathcal {A},s}-\mathcal {C}_{\mathcal {B},t}. \end{aligned}$$
(209)

We also introduce the notation

$$\begin{aligned} \mathcal {R}\left\{ \theta \left( \left. x\right| \tau \right) \right\} =\frac{\theta \left( x\right) \prod _{1\le a<b\le 3}\theta \left( x+\varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( x+\varepsilon _{a}\right) }. \end{aligned}$$
(210)

Then \(Z_{\varvec{\uppi }}\) can be expressed as

$$\begin{aligned} Z_{\varvec{\uppi }}=\left( \prod _{\mathcal {A}\in \underline{n}}Z_{\varvec{\uppi }}^{(\mathcal {A})} \right) \left( \prod _{\mathcal {A}\ne \mathcal {B}\in \underline{n}}Z_{\varvec{\uppi }}^{(\mathcal {A},\mathcal {B})}\right) , \end{aligned}$$
(211)

where

(212)

and

$$\begin{aligned} Z_{\varvec{\uppi }}^{(\mathcal {A},\mathcal {B})}=\left( \prod _{s\in \pi ^{(\mathcal {A})}} \prod _{t\in \pi ^{(\mathcal {B})}}\mathcal {R}\left\{ \theta \left( \mathcal {D}_{\mathcal {B},t}^{\mathcal {A},s}\right) \right\} \right) \left( \prod _{s\in \pi ^{(\mathcal {A})}}\frac{\theta \left( \mathcal {C}_{\mathcal {A},s} -\texttt {a}_{\mathcal {B}}-\varepsilon _{\mathcal {B}}\right) }{\theta \left( \mathcal {C}_{\mathcal {A},s} -\texttt {a}_{\mathcal {B}}\right) }\right) . \end{aligned}$$
(213)

The instanton partition function is

$$\begin{aligned} Z=\sum _{\varvec{\uppi }}{} \texttt {q}^{\left| \varvec{\uppi }\right| }Z_{\varvec{\uppi }}, \end{aligned}$$
(214)

which is identical to \(\mathcal {Z}^{\mathrm {inst}}\) in (155) if we use the elliptic version (157) of the operator \(\mathbb {E}\).

6.3.4 Example: \(k=2\) and \(\mathbf {n}=\left( 1,1,0,0\right) \)

Let us present here explicitly the result for the simplest nontrivial example, \(k=2\) and \(\mathbf {n}=\left( n_{123}=1,n_{124}=1,0,0\right) \). We are dealing with the integral

$$\begin{aligned} Z_{2}= & {} \frac{1}{2}\left[ \frac{2\pi \eta (\tau )^{3}\prod _{1\le a<b\le 3}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}} \theta \left( \varepsilon _{a}\right) }\right] ^{2}\nonumber \\&\int d\phi _{1}d\phi _{2}\frac{\theta ^{2}\left( \phi _{1} -\phi _{2}\right) \prod _{1\le a<b\le 3}\theta \left( \phi _{1} -\phi _{2}\pm \,\varepsilon _{ab}\right) }{\prod _{a\in \underline{4}} \theta \left( \phi _{1}-\phi _{2}\pm \varepsilon _{a}\right) }\nonumber \\&\times \prod _{i=1}^{2}\frac{\theta \left( \phi _{i}-\texttt {a}_{123} -\varepsilon _{123}\right) }{\theta \left( \phi _{i}-\texttt {a}_{123}\right) } \frac{\theta \left( \phi _{i}-\texttt {a}_{124}-\varepsilon _{124}\right) }{\theta \left( \phi _{i} -\texttt {a}_{124}\right) }. \end{aligned}$$
(215)

Due to the invariance under the overall shift (169), the result can only depend on the difference

$$\begin{aligned} \delta =\texttt {a}_{123}-\texttt {a}_{124}. \end{aligned}$$
(216)

The genuine poles are completely classified by a collection of two (possibly empty) colored plane partitions,

$$\begin{aligned} \varvec{\uppi }=\left\{ \pi ^{(123)},\pi ^{(124)}\right\} , \end{aligned}$$
(217)

and the total number of boxes

$$\begin{aligned} \left| \pi ^{(123)}\right| +\left| \pi ^{(124)}\right| =k=2. \end{aligned}$$
(218)

There are three possibilities:

  1. 1.

    If \(\left| \pi ^{(123)}\right| =2,\left| \pi ^{(124)}\right| =0\), the genuine poles are at \(\left( \texttt {a}_{123},\texttt {a}_{123}-\varepsilon _{a}\right) ,a=1,2,3\), and the corresponding residues are

    $$\begin{aligned} z_{\left( \texttt {a}_{123},\texttt {a}_{123}-\varepsilon _{1}\right) }= & {} \frac{\theta \left( \varepsilon _{12}\right) \theta \left( \varepsilon _{13}\right) \theta \left( \varepsilon _{23}\right) }{\theta \left( \varepsilon _{1}\right) \theta \left( \varepsilon _{2}\right) \theta \left( \varepsilon _{3}\right) }\frac{\theta \left( \delta +\varepsilon _{3}\right) \theta \left( \delta -\varepsilon _{1}+\varepsilon _{3}\right) }{\theta \left( \delta \right) \theta \left( \delta -\varepsilon _{1}\right) }\nonumber \\&\times \frac{\theta \left( 2\varepsilon _{1}+\varepsilon _{2}\right) \theta \left( 2\varepsilon _{1} +\varepsilon _{3}\right) \theta \left( \varepsilon _{23}-\varepsilon _{1}\right) }{\theta \left( 2\varepsilon _{1}\right) \theta \left( \varepsilon _{1}-\varepsilon _{2}\right) \theta \left( \varepsilon _{1}-\varepsilon _{3}\right) }, \end{aligned}$$
    (219)
    $$\begin{aligned} z_{\left( \texttt {a}_{123},\texttt {a}_{123}-\varepsilon _{2}\right) }= & {} \frac{\theta \left( \varepsilon _{12}\right) \theta \left( \varepsilon _{13}\right) \theta \left( \varepsilon _{23}\right) }{\theta \left( \varepsilon _{1}\right) \theta \left( \varepsilon _{2}\right) \theta \left( \varepsilon _{3}\right) } \frac{\theta \left( \delta +\varepsilon _{3}\right) \theta \left( \delta -\varepsilon _{2} +\varepsilon _{3}\right) }{\theta \left( \delta \right) \theta \left( \delta -\varepsilon _{2}\right) }\nonumber \\&\times \frac{\theta \left( 2\varepsilon _{2}+\varepsilon _{1}\right) \theta \left( 2\varepsilon _{2}+\varepsilon _{3}\right) \theta \left( \varepsilon _{13} -\varepsilon _{2}\right) }{\theta \left( 2\varepsilon _{2}\right) \theta \left( \varepsilon _{2} -\varepsilon _{1}\right) \theta \left( \varepsilon _{2}-\varepsilon _{3}\right) }, \end{aligned}$$
    (220)
    $$\begin{aligned} z_{\left( \texttt {a}_{123},\texttt {a}_{123}-\varepsilon _{3}\right) }= & {} \frac{\theta \left( \varepsilon _{12}\right) \theta \left( \varepsilon _{13}\right) \theta \left( \varepsilon _{23} \right) }{\theta \left( \varepsilon _{1}\right) \theta \left( \varepsilon _{2}\right) \theta \left( \varepsilon _{3}\right) }\frac{\theta \left( \delta +\varepsilon _{3}\right) }{\theta \left( \delta -\varepsilon _{3}\right) }\nonumber \\&\times \frac{\theta \left( 2\varepsilon _{3}+\varepsilon _{1}\right) \theta \left( 2\varepsilon _{3}+\varepsilon _{2}\right) \theta \left( \varepsilon _{12} -\varepsilon _{3}\right) }{\theta \left( 2\varepsilon _{3}\right) \theta \left( \varepsilon _{3} -\varepsilon _{1}\right) \theta \left( \varepsilon _{3}-\varepsilon _{2}\right) }, \end{aligned}$$
    (221)
  2. 2.

    If \(\left| \pi ^{(123)}\right| =0,\left| \pi ^{(124)}\right| =2\), the genuine poles are at \(\left( \texttt {a}_{124},\texttt {a}_{124}-\varepsilon _{a}\right) ,a=1,2,4\), and the corresponding residues are

    $$\begin{aligned} z_{\left( \texttt {a}_{124},\texttt {a}_{124}-\varepsilon _{1}\right) }= & {} \frac{\theta \left( \varepsilon _{12}\right) \theta \left( \varepsilon _{14}\right) \theta \left( \varepsilon _{24}\right) }{\theta \left( \varepsilon _{1}\right) \theta \left( \varepsilon _{2}\right) \theta \left( \varepsilon _{4}\right) }\frac{\theta \left( \delta -\varepsilon _{4}\right) \theta \left( \delta +\varepsilon _{1}-\varepsilon _{4}\right) }{\theta \left( \delta \right) \theta \left( \delta +\varepsilon _{1}\right) }\nonumber \\&\times \frac{\theta \left( 2\varepsilon _{1}+\varepsilon _{2}\right) \theta \left( 2\varepsilon _{1}+\varepsilon _{4}\right) \theta \left( \varepsilon _{24}-\varepsilon _{1}\right) }{\theta \left( 2\varepsilon _{1}\right) \theta \left( \varepsilon _{1}-\varepsilon _{2}\right) \theta \left( \varepsilon _{1}-\varepsilon _{4}\right) }, \end{aligned}$$
    (222)
    $$\begin{aligned} z_{\left( \texttt {a}_{124},\texttt {a}_{124}-\varepsilon _{2}\right) }= & {} \frac{\theta \left( \varepsilon _{12}\right) \theta \left( \varepsilon _{14}\right) \theta \left( \varepsilon _{24}\right) }{\theta \left( \varepsilon _{1}\right) \theta \left( \varepsilon _{2}\right) \theta \left( \varepsilon _{4}\right) }\frac{\theta \left( \delta -\varepsilon _{4}\right) \theta \left( \delta +\varepsilon _{2}-\varepsilon _{4}\right) }{\theta \left( \delta \right) \theta \left( \delta +\varepsilon _{2}\right) }\nonumber \\&\times \frac{\theta \left( 2\varepsilon _{2}+\varepsilon _{1}\right) \theta \left( 2\varepsilon _{2}+\varepsilon _{4}\right) \theta \left( \varepsilon _{14}-\varepsilon _{2}\right) }{\theta \left( 2\varepsilon _{2}\right) \theta \left( \varepsilon _{2}-\varepsilon _{1}\right) \theta \left( \varepsilon _{2}-\varepsilon _{4}\right) }, \end{aligned}$$
    (223)
    $$\begin{aligned} z_{\left( \texttt {a}_{124},\texttt {a}_{124}-\varepsilon _{4}\right) }= & {} \frac{\theta \left( \varepsilon _{12}\right) \theta \left( \varepsilon _{14}\right) \theta \left( \varepsilon _{24}\right) }{\theta \left( \varepsilon _{1}\right) \theta \left( \varepsilon _{2}\right) \theta \left( \varepsilon _{4}\right) }\frac{\theta \left( \delta -\varepsilon _{4}\right) }{\theta \left( \delta +\varepsilon _{4}\right) }\nonumber \\&\times \frac{\theta \left( 2\varepsilon _{4}+\varepsilon _{1}\right) \theta \left( 2\varepsilon _{4}+\varepsilon _{2}\right) \theta \left( \varepsilon _{12}-\varepsilon _{4}\right) }{\theta \left( 2\varepsilon _{2}\right) \theta \left( \varepsilon _{2}-\varepsilon _{1}\right) \theta \left( \varepsilon _{2}-\varepsilon _{4}\right) }, \end{aligned}$$
    (224)
  3. 3.

    If \(\left| \pi ^{(123)}\right| =\left| \pi ^{(124)}\right| =1\), the genuine pole can only be at \(\left( \texttt {a}_{123},\texttt {a}_{124}\right) \), and the corresponding residue is

    $$\begin{aligned} z_{\left( \texttt {a}_{123},\texttt {a}_{124}\right) }= & {} \frac{\theta ^{2}\left( \varepsilon _{12}\right) \theta ^{2}\left( \varepsilon _{13}\right) \theta ^{2}\left( \varepsilon _{23}\right) }{\theta ^{2}\left( \varepsilon _{1}\right) \theta ^{2}\left( \varepsilon _{2}\right) \theta \left( \varepsilon _{3}\right) \theta \left( \varepsilon _{4}\right) }\nonumber \\&\times \frac{\theta \left( \pm \delta +\varepsilon _{12}\right) \theta \left( \pm \delta +\varepsilon _{13}\right) \theta \left( \pm \delta +\varepsilon _{23}\right) }{\theta \left( \pm \delta +\varepsilon _{1}\right) \theta \left( \pm \delta +\varepsilon _{2}\right) \theta \left( -\delta +\varepsilon _{3}\right) \theta \left( \delta +\varepsilon _{4}\right) }. \end{aligned}$$
    (225)

In the above calculations, we have taken a particular ordering of \(\phi _{1}\) and \(\phi _{2}\) to cancel the factor of 2 in the denominator of (215). Summing up these contributions, we get

$$\begin{aligned} Z_{2}=\sum _{a\in \left( 123\right) }z_{\left( \texttt {a}_{123},\texttt {a}_{123}-\varepsilon _{a}\right) }+\sum _{a\in \left( 124\right) }z_{\left( \texttt {a}_{124},\texttt {a}_{124}-\varepsilon _{a}\right) }+z_{\left( \texttt {a}_{123},\texttt {a}_{124}\right) }. \end{aligned}$$
(226)

One can check that (226) matches the general expression given in the previous subsection.

6.4 Expectation value of real codimension-two defects

Up to now, we treat all \(\hbox {D}7_{A}\)-branes on equal footing, but the string theory construction of tetrahedron instantons and the geometric interpretation of the moduli space suggest a different point of view of the instanton partition function. We choose the physical spacetime to be \(\mathbb {R}^{1,1}\times \mathbb {C}_{123}^{3}\), so that the bound states of \(\hbox {D}1\)- and \(\hbox {D}7_{123}\)-branes give rise to instantons on \(\mathbb {C}_{123}^{3}\). The remaining \(\hbox {D}7_{A}\)-branes for \(A\in \underline{4}^{\vee }\setminus \left\{ (123)\right\} \) will produce real codimension-two defects from the viewpoint of the physical spacetime. This provides the physical realization of the projection of the moduli space \(\mathfrak {M}_{\mathbf {n},k}\) of tetrahedron instantons to the moduli spaces \(\mathfrak {M}_{\left( n_{123},0,0,0\right) ,k^{\prime }}\) of instantons on \(\mathbb {C}_{123}^{3}\) discussed in Sect. 4. Thus we identify the instanton partition function as the expectation value of real codimension-two defects \(\mathcal {O}_{A}\) in the instanton partition function of the Donaldson–Thomas theory,

$$\begin{aligned} Z= & {} \sum _{k=0}^{\infty }\frac{\texttt {q}^{k}}{k!}\int \prod _{i=1}^{k}d\phi _{i} \left[ \left( Z_{k}^{1-1}Z_{k}^{1-7_{123}}\right) \left( \prod _{A\in \underline{4}^{\vee }\setminus \left\{ \left( 123\right) \right\} }Z_{k}^{1-7_{A}}\right) \right] \nonumber \\= & {} \left\langle \prod _{A\in \underline{4}^{\vee }\setminus \left\{ \left( 123\right) \right\} }\mathcal {O}_{A}\right\rangle _{\mathrm {DT}}, \end{aligned}$$
(227)

where the bracket denotes the unnormalized vacuum expectation value in the Donaldson–Thomas theory on \(\mathbb {C}_{123}^{3}\) whose instanton partition function is given by

$$\begin{aligned} Z_{\mathrm {DT}}=\sum _{k=0}^{\infty }\frac{\texttt {q}^{k}}{k!}\int \prod _{i=1}^{k}d\phi _{i}Z_{k}^{1-1}Z_{k}^{1-7_{123}}. \end{aligned}$$
(228)

6.5 Dimensional reductions

We now briefly discuss dimensional reductions of the system.

Performing a T-duality along \(x^{9}\) of the configuration in Table 1, we get \(\hbox {D}0\)-branes probing a configuration of intersecting \(\hbox {D}6\)-branes in type IIA superstring theory. The generating function of the generalized Witten indices of the supersymmetric gauged quantum mechanical models on \(\hbox {D}0\)-branes is the K-theoretical version of the instanton partition function of tetrahedron instantons. Since there are no anomalies of large gauge transformations, we no longer impose the constraint (172). Taking the limit \(q\rightarrow 0\) of Z, we get the dimensionally reduced instanton partition function \(Z^{\downarrow }\),

$$\begin{aligned} Z^{\downarrow }= & {} \sum _{\varvec{\uppi }}{} \texttt {q}^{\left| \varvec{\uppi }\right| }Z_{\varvec{\uppi }}^{\downarrow }\nonumber \\= & {} \sum _{\varvec{\uppi }}{} \texttt {q}^{\left| \varvec{\uppi }\right| } \left( \prod _{\mathcal {A}\in \underline{n}}Z_{\varvec{\uppi }}^{\downarrow (\mathcal {A})}\right) \left( \prod _{\mathcal {A}\ne \mathcal {B}\in \underline{n}}Z_{\varvec{\uppi }}^{\downarrow (\mathcal {A},\mathcal {B})} \right) , \end{aligned}$$
(229)

where \(Z_{\varvec{\uppi }}^{\downarrow (\mathcal {A})}\) and \(Z_{\varvec{\uppi }}^{\downarrow (\mathcal {A},\mathcal {B})}\) are obtained from \(Z_{\varvec{\uppi }}^{(\mathcal {A})}\) and \(Z_{\varvec{\uppi }}^{(\mathcal {A},\mathcal {B})}\) by substituting

$$\begin{aligned} \theta \left( z\right) \rightarrow 2\sinh \left( \frac{\beta z}{2}\right) , \end{aligned}$$
(230)

and \(\beta \) is the circumference of the circle of the supersymmetric quantum mechanics. The instanton partition function \(Z^{\downarrow }\) matches \(\mathcal {Z}\) in (155) with the K-theoretical version (157) of the operator \(\mathbb {E}\).

We can further perform a T-duality along \(x^{0}\) direction to get D-instantons probing a configuration of intersecting \(\hbox {D}5\)-branes in type IIB superstring theory. The instanton partition function \(Z^{\Downarrow }\) is obtained by

$$\begin{aligned} Z^{\Downarrow }=\sum _{\varvec{\uppi }}{} \texttt {q}^{\left| \varvec{\uppi }\right| }Z_{\varvec{\uppi }}^{\Downarrow } =\sum _{\varvec{\uppi }}{} \texttt {q}^{\left| \varvec{\uppi }\right| }\left( \prod _{\mathcal {A} \in \underline{n}}Z_{\varvec{\uppi }}^{\Downarrow (\mathcal {A})}\right) \left( \prod _{\mathcal {A} \ne \mathcal {B}\in \underline{n}}Z_{\varvec{\uppi }}^{\Downarrow (\mathcal {A},\mathcal {B})}\right) . \end{aligned}$$
(231)

Here \(Z_{\varvec{\uppi }}^{\Downarrow (\mathcal {A})}\) and \(Z_{\varvec{\uppi }}^{\Downarrow (\mathcal {A},\mathcal {B})}\) are obtained from \(Z_{\varvec{\uppi }}^{(\mathcal {A})}\) and \(Z_{\varvec{\uppi }}^{(\mathcal {A},\mathcal {B})}\) by substituting \(\theta \left( z\right) \rightarrow z\),

$$\begin{aligned} Z_{\varvec{\uppi }}^{\Downarrow \left( \mathcal {A}=\left( A,\alpha \right) \right) }= & {} \varepsilon _{\check{A}}\left( \frac{\prod _{a<b\in A}\varepsilon _{ab}}{\prod _{a\in \underline{4}} \varepsilon _{a}}\right) ^{\left| \pi ^{(\mathcal {A})}\right| }\left( \prod _{s\ne t\in \pi ^{(\mathcal {A})}} \mathcal {R}\left\{ \mathcal {D}_{\mathcal {A},t}^{\mathcal {A},s}\right\} \right) \nonumber \\&\times \left( \prod _{s\in \pi ^{(\mathcal {A})}\setminus (1,1,1)}\frac{\mathcal {C}_{\mathcal {A},s} -a_{\mathcal {A}}-\varepsilon _{\mathcal {A}}}{\mathcal {C}_{\mathcal {A},s}-a_{\mathcal {A}}}\right) , \end{aligned}$$
(232)
$$\begin{aligned} Z_{\varvec{\uppi }}^{\Downarrow (\mathcal {A},\mathcal {B})}= & {} \left( \prod _{s\in \pi ^{(\mathcal {A})}} \prod _{t\in \pi ^{(\mathcal {B})}}\mathcal {R}\left\{ \mathcal {D}_{\mathcal {B},t}^{\mathcal {A},s}\right\} \right) \left( \prod _{s\in \pi ^{(\mathcal {A})}}\frac{\mathcal {C}_{\mathcal {A},s}-a_{\mathcal {B}} -\varepsilon _{\mathcal {B}}}{\mathcal {C}_{\mathcal {A},s}-a_{\mathcal {B}}}\right) , \end{aligned}$$
(233)

where

$$\begin{aligned} \mathcal {R}\left\{ x\right\} =\frac{x\left( x+\varepsilon _{12}\right) \left( x+\varepsilon _{13}\right) \left( x+\varepsilon _{23}\right) }{\left( x+\varepsilon _{1}\right) \left( x+\varepsilon _{2}\right) \left( x+\varepsilon _{3}\right) \left( x+\varepsilon _{4}\right) }. \end{aligned}$$
(234)

The partition function \(Z^{\Downarrow }\) matches \(\mathcal {Z}\) in (155) exactly.

7 Free Field Representation

Following [15, 66, 75, 99, 110], we give a free field representation of the instanton partition function. This is in the general spirit of the BPS/CFT correspondence [92].

Recall that the torus propagator for a free massless r-component scalar field \(\varphi =\left( \varphi _{1},\cdots ,\varphi _{r}\right) \) is given by [30]

$$\begin{aligned} G_{i,j}\left( z,\bar{z}\right)= & {} \left\langle \varphi _{i}\left( z,\bar{z}\right) \varphi _{j} \left( 0,0\right) \right\rangle _{\mathbb {T}^{2}}\nonumber \\= & {} -\log \left| \frac{\theta _{1}\left( \left. z\right| \tau \right) }{2\pi \eta (\tau )^{3}} \exp \left( -\frac{\pi \left( \mathrm {Im}z\right) ^{2}}{\mathrm {Im}\tau }\right) \right| ^{2} \delta _{i,j},\quad i,j=1,\cdots ,r, \end{aligned}$$
(235)

where the torus \(\mathbb {T}^{2}\) is described by a complex z-plane with the identification \(z\cong z+1\cong z+\tau \), and \(G_{i,j}\left( z,\bar{z}\right) \) is the normalized doubly periodic solution of the Laplacian on \(\mathbb {T}^{2}\),

$$\begin{aligned} -\varDelta G_{i,j}\left( z,\bar{z}\right) =\left( 2\pi \delta ^{2}(z) -\frac{4\pi }{\mathrm {Im}\tau }\right) \delta _{i,j}. \end{aligned}$$
(236)

The basic vertex operators of the theory are the exponential fields parameterized by a r-component vector parameter \(\alpha =\left( \alpha _{1},\cdots ,\alpha _{r}\right) \),

$$\begin{aligned} \mathbb {V}_{\alpha }\left( z,\bar{z}\right) =:e^{\mathrm {i}\sum _{i=1}^{r} \alpha _{i}\varphi _{i}\left( z,\bar{z}\right) }:. \end{aligned}$$
(237)

We require that the complex structure \(\tau \) of \(\mathbb {T}^{2}\) is the same as that in the definition of the elliptic genus (161). Then we will use the abbreviation (168) in the following.

We shall take \(r=7\) and introduce a slightly deformed vertex operator

$$\begin{aligned} \mathscr {V}_{\alpha ,\rho }\left( z,\bar{z}\right) =:e^{\mathrm {i} \sum _{i=1}^{7}\alpha _{i}\varphi _{i}\left( z+\rho _{i},\bar{z}+\rho _{i}\right) }::e^{-\mathrm {i} \sum _{i=1}^{7}\alpha _{i}\varphi _{i}\left( z-\rho _{i},\bar{z}-\rho _{i}\right) }:, \end{aligned}$$
(238)

where

$$\begin{aligned} \alpha =\left( \mathrm {i},\mathrm {i},\mathrm {i},\mathrm {i},1,1,1\right) ,\quad \rho = \frac{1}{2}\left( \varepsilon _{1},\varepsilon _{2},\varepsilon _{3},\varepsilon _{4},\varepsilon _{12}, \varepsilon _{13},\varepsilon _{23}\right) . \end{aligned}$$
(239)

It is an important fact that when \(\sum _{a\in \underline{4}}\varepsilon _{a}=0\) we have

$$\begin{aligned} \sum _{i=1}^{7}\alpha _{i}^{2}\left( \mathrm {Im}\left( \rho _{i}\right) \right) ^{2}=0. \end{aligned}$$
(240)

Performing the Wick contraction, we can get

$$\begin{aligned} \mathscr {V}_{\alpha ,\rho }\left( z,\bar{z}\right) =\left| \frac{2\pi \eta (\tau )^{3}\prod _{1\le a<b\le 3}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( \varepsilon _{a}\right) }\right| ^{2}:\mathscr {V}_{\alpha , \rho }\left( z,\bar{z}\right) :, \end{aligned}$$
(241)

and

$$\begin{aligned}&\left\langle :\mathscr {V}_{\alpha ,\rho }\left( z,\bar{z}\right) ::\mathscr {V}_{\alpha ,\rho } \left( w,\bar{w}\right) :\right\rangle _{\mathbb {T}^{2}}\nonumber \\= & {} \left| \frac{\theta ^{2}\left( z-w\right) \prod _{1\le a<b\le 3}\theta \left( z-w\pm \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( z-w\pm \varepsilon _{a}\right) }\right| ^{2}, \end{aligned}$$
(242)

where \(\theta \left( z\pm \tilde{\varepsilon }\right) =\theta \left( z+\tilde{\varepsilon }\right) \theta \left( z-\tilde{\varepsilon }\right) \). Since (242) takes the form of an absolute square, we can define the holomorphic part as

$$\begin{aligned}&\left\langle :\mathscr {V}_{\alpha ,\rho }\left( z,\bar{z}\right) ::\mathscr {V}_{\alpha ,\rho } \left( w,\bar{w}\right) :\right\rangle _{\mathbb {T}^{2}}^{\mathrm {hol}}\nonumber \\= & {} \frac{\theta ^{2}\left( z-w\right) \prod _{1\le a<b\le 3}\theta \left( z-w\pm \varepsilon _{ab} \right) }{\prod _{a\in \underline{4}}\theta \left( z-w\pm \varepsilon _{a}\right) }. \end{aligned}$$
(243)

We further introduce a linear source operator,

$$\begin{aligned} \varUpsilon =\frac{1}{2\pi \mathrm {i}}\oint _{\Gamma }dz\sum _{A\in \underline{4}^{\vee }} \varpi _{A}(z)\partial _{z}\varphi _{\check{A}}(z), \end{aligned}$$
(244)

where the contour \(\Gamma \) is chosen to be a loop around \(z=0\) encircling all \(\pm \rho _{i}\) for \(i=1,\cdots ,7\), and \(\varpi _{A}(z)\) is a locally analytic function inside \(\Gamma \),

$$\begin{aligned} \varpi _{A}(z)=\sum _{\alpha =1}^{n_{A}}\log \theta \left( z-\texttt {a}_{A,\alpha } -\frac{1}{2}\varepsilon _{A}\right) . \end{aligned}$$
(245)

Then

$$\begin{aligned} \left\langle e^{\varUpsilon }:\mathscr {V}_{\alpha ,\rho }(z):\right\rangle _{\mathbb {T}^{2}} =\prod _{\mathcal {A}=\left( A,\alpha \right) \in \underline{n}}\frac{\theta \left( z-\texttt {a}_{\mathcal {A}}-\varepsilon _{A}\right) }{\theta \left( z-\texttt {a}_{\mathcal {A}}\right) }, \end{aligned}$$
(246)

which is already holomorphic.

Therefore, we have the expansion

$$\begin{aligned}&\left\langle e^{\varUpsilon }e^{\texttt {q}\oint _{\mathcal {C}}\mathscr {V}_{\alpha ,\rho }(z)dz} \right\rangle _{\mathbb {T}^{2}}^{\mathrm {hol}}\nonumber \\= & {} \sum _{k=0}^{\infty }\frac{\texttt {q}^{k}}{k!}\left[ \frac{2\pi \eta (\tau )^{3} \prod _{1\le a<b\le 3}\theta \left( \varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( \varepsilon _{a} \right) }\right] ^{k}\nonumber \\&\times \oint _{\mathcal {C}}dz_{1}\cdots \oint _{\mathcal {C}}dz_{k}\prod _{i=1}^{k}\prod _{\mathcal {A} =\left( A,\alpha \right) \in \underline{n}}\frac{\theta \left( z-\texttt {a}_{\mathcal {A}}-\varepsilon _{A}\right) }{\theta \left( z-\texttt {a}_{\mathcal {A}}\right) }\nonumber \\&\times \prod _{\begin{array}{c} i,j=1\\ i\ne j \end{array} }^{k}\frac{\theta \left( z_{ij}\right) \prod _{1\le a<b\le 3}\theta \left( z_{ij}+\varepsilon _{ab}\right) }{\prod _{a\in \underline{4}}\theta \left( z_{ij}+\varepsilon _{a}\right) }, \end{aligned}$$
(247)

which coincides with the instanton partition function (164) if the contour \(\mathcal {C}\) is chosen to give the Jeffrey–Kirwan residues. We see that the contributions from the \(\hbox {D}1\)\(\hbox {D}1\) and \(\hbox {D}1\)\(\hbox {D}7\) strings are reproduced by the Wick contractions within the exponentiated integrated vertex, and the Wick contractions between the exponentiated integrated vertex and the linear source, respectively.

8 Conclusions and Future Directions

In this paper, we introduced tetrahedron instantons and explained how to construct them from string theory and from noncommutative field theory. We analyzed the moduli space of tetrahedron instantons and discussed its geometric interpretations. We computed the instanton partition function in two different approaches: the infrared approach which computes the partition function via equivariant localization on the moduli space of tetrahedron instantons, and the ultraviolet approach which computes the partition function as the elliptic genus of the worldvolume theory on the \(\hbox {D}1\)-branes probing a configuration of intersecting \(\hbox {D}7\)-branes. Both approaches lead to the same result. Our instanton partition function can also be viewed as the expectation value of the most general real codimension-two defects in the instanton partition function of the Donaldson–Thomas theory. Finally, we find a free field representation of the instanton partition function, indicating the existence of a novel kind of symmetry acting on the cohomology of the moduli spaces of tetrahedron instantons.

There are still many interesting aspects of tetrahedron instantons that remain to be better understood. Some of the future directions in which this work could be continued are listed in the following.

  1. 1.

    According to [130], a supersymmetric bound state can be formed by the \(\hbox {D}1\)\(\hbox {D}9\) system if we turn on a constant B-field satisfying

    $$\begin{aligned} \sum _{a\in \underline{4}}v_{a}\ge 1. \end{aligned}$$
    (248)

    Therefore, we can generalize the tetrahedron instantons by adding a stack of \(\hbox {D}9\)-branes without further breaking the supersymmetry if the B-field satisfies the condition

    $$\begin{aligned} v_{1}=v_{2}=v_{3}=v_{4}\ge \frac{1}{4}. \end{aligned}$$
    (249)

    This generalization can also be viewed as instantons in the magnificent four model with all possible real codimension-two defects. Furthermore, it is fascinating to also incorporate in the system the spiked instantons, which can be realized by \(\hbox {D}1\)-branes probing six stacks of (anti-)\(\hbox {D}5\)-branes in type IIB superstring theory. As analyzed in [104], supersymmetry is completely broken when we have six stacks of \(\hbox {D}5\)-branes, and two supercharges can be preserved when we have four stacks of \(\hbox {D}5\)-branes and two stacks of anti-\(\hbox {D}5\)-branes in the presence of a B-field obeying

    $$\begin{aligned} v_{1}=-v_{2}=v_{3}=-v_{4}. \end{aligned}$$
    (250)

    In both cases, no supersymmetry will be preserved when we put together the magnificent four model, the tetrahedron instantons and the spiked instantons. On the other hand, the configuration of \(\hbox {D}1\)-branes with six stacks of anti-\(\hbox {D}5\)-branes preserve two supercharges when the B-field obeysFootnote 3

    $$\begin{aligned} v_{1}=v_{2}=v_{3}=v_{4}. \end{aligned}$$
    (251)

    In this case, we can study the supersymmetric combination of the magnificent four model, the tetrahedron instantons and the spiked instantons. This combined system can be understood as instantons in the magnificent four model with all possible real codimension-two and real codimension-four defects.

  2. 2.

    It was proposed that the partition function of the magnificent four model is the mother of all instanton partition functions [95, 103]. In particular, it was shown in [103] that the partition function of the magnificent four model at a degenerate limit reduces to the instanton partition function of the Donaldson–Thomas theory on \(\mathbb {C}^{3}\). The magnificent four model can be realized in string theory using \(\hbox {D}0\)-branes probing a collection of \(\hbox {D}8\)- and anti-\(\hbox {D}8\)-branes wrapping a Calabi–Yau fourfold, with an appropriate B-field. Here the \(\hbox {D}0\)\(\hbox {D}8\) system gives an ADHM-type construction for instantons in the eight-dimensional gauge theory, while the presence of the anti-\(\hbox {D}8\)-branes introduces certain fundamental matter fields. The degenerate limit corresponds to a fine-tuned position of the anti-\(\hbox {D}8\)-branes, and it was conjectured that anti-\(\hbox {D}8\)-branes will annihilate the \(\hbox {D}8\)-branes, leaving a configuration of \(\hbox {D}6\)-branes after the tachyon condensation. It is natural to imagine that by taking more general degenerate limits, the instanton partition function of our model can always be obtained from that of the magnificent four model. The matching of the instanton partition function will then be a highly nontrivial test of the tachyon condensation in nontrivial string backgrounds.

  3. 3.

    It is well known that the partition function of the Donaldson–Thomas theory on a toric Calabi–Yau threefold and the partition function of the magnificent four model play important roles in the study of the compactification of M-theory on Calabi–Yau fivefolds [29, 54, 100]. An equivalence between the Donaldson–Thomas invariants and Gromov-Witten invariants was conjectured [78,79,80]. Together with the Gopakumar-Vafa invariants [47,48,49], they arise from different expansions of the same topological string amplitude. A fascinating direction is to explore our model from this viewpoint. We consider the bound state of k \(\hbox {D}0\)-branes and \(n_{A}\) \(\hbox {D}6_{A}\)-branes on \(\mathbb {S}^{1}\times \mathbb {C}_{A}^{3}\) for all \(A\in \underline{4}^{\vee }\), which can be lifted to M-theory as a bound state of k graviton Kaluza-Klein modes on \(\mathbb {S}^{1}\times \mathscr {X}\), where \(\mathscr {X}\) is a noncompact Calabi–Yau fivefolds. When only one of the \(n_{A}\) is nonzero, \(\mathscr {X}\) becomes \(\mathbb {C}_{A}^{3}\times \mathrm {TN}_{n_{A}}\), where \(\mathrm {TN}_{n_{A}}\) is the \(n_{A}\)-centered Taub-NUT space. After introducing the \(\varOmega \)-deformation, the eleven-dimensional spacetime \(\mathbb {S}^{1}\times \mathscr {X}\) is replaced by a fiber bundle over \(\mathbb {S}^{1}\) with fiber \(\mathscr {X}\), such that the fiber is rotated by an element \(g\in {\mathrm {SU}}(5)\) as we go around \(\mathbb {S}^{1}\). The eleven-dimensional supergravity partition function on this background is defined as the twisted Witten index,

    $$\begin{aligned} Z^{\mathrm {sugra}}\left[ \mathbb {S}^{1}\rtimes _{g}\mathscr {X}\right] \left( \mathfrak {q}_{1},\cdots ,\mathfrak {q}_{5}\right)= & {} {\mathrm {Tr}}_{\mathcal {H}(\mathscr {X})}(-1)^{F}e^{-\beta \left\{ \mathfrak {Q},\mathfrak {Q}^{*}\right\} }g\nonumber \\= & {} \exp \left[ \sum _{\ell =1}^{\infty }\frac{1}{\ell }\mathcal {F}^{\mathrm {sugra}}\left( \mathfrak {q}_{1}^{\ell },\cdots ,\mathfrak {q}_{5}^{\ell }\right) \right] , \end{aligned}$$
    (252)

    where \(\mathcal {H}(\mathscr {X})\) is the Hilbert space of the supergravity theory on \(\mathscr {X}\), F is the fermion number operator, \(\beta \) is the circumference of \(\mathbb {S}^{1}\), \(\mathfrak {Q}\) is a preserved supercharge that commutes with g, and \(\mathfrak {q}_{1},\cdots ,\mathfrak {q}_{5}\) satisfying \(\prod _{i=1}^{5}\mathfrak {q}_{i}=1\) are the fugacities associated with the \({\mathrm {SU}}(5)\) action. We can decompose the single-particle index \(\mathcal {F}\) into two parts,

    $$\begin{aligned} \mathcal {F}^{\mathrm {sugra}}=\mathcal {F}^{\mathrm {sugra,pert}}+\mathcal {F}^{\mathrm {sugra,inst}}, \end{aligned}$$
    (253)

    where \(\mathcal {F}^{\mathrm {sugra,pert}}\) is the perturbative contribution from \(\hbox {D}6\)-branes in the absence of \(\hbox {D}0\)-branes, and \(\mathcal {F}^{\mathrm {sugra,inst}}\) should coincide with the single-particle index (147) of the instanton partition function. An extraordinary feature of this correspondence is that the instanton counting parameter \(\texttt {q}\) in the instanton partition function will be expressed in terms of the fugacities \(\mathfrak {q}_{1},\cdots ,\mathfrak {q}_{5}\) in \(\mathcal {F}^{\mathrm {sugra,inst}}\).

  4. 4.

    In this paper we only considered the simplest spacetime geometry \(\mathbb {R}^{1,1}\times \mathbb {C}^{4}\). It is definitely interesting to generalize our analysis to \(\mathbb {R}^{1,1}\times \mathscr {Y}\), where \(\mathscr {Y}\) is an arbitrary toric Calabi–Yau fourfold. For example, one can consider the orbifold \(\mathscr {Y}=\mathbb {C}^{4}/\Gamma \), where \(\Gamma \) is a finite subgroup of \({\mathrm {SU}}(4)\). The moduli space will be a generalization of Nakajima quiver varieties [19, 85, 87, 93] and the chain-saw quiver [41, 42, 65]. The instanton partition function on the orbifold can be obtained by projecting onto the \(\Gamma \)-invariant part. Another nature choice is to blowup the origin of \(\mathbb {C}^{4}\) in the spirit of [89,90,91], and it may be useful for the study of BPS/CFT correspondence [61, 98]. These instanton partition functions should lie between the Donaldson–Thomas invariants of toric Calabi–Yau threefolds [78, 79] and fourfolds [20,21,22]. We can even generalize our computations by including extra D-branes wrapping compact divisors.

  5. 5.

    The instanton partition function of the Donaldson–Thomas theory was identified with the classical statistical mechanics of melting crystal [116], and can be interpreted as a quantum gravitational path integral involving fluctuations of geometry and topology [57]. It would be wonderful if one can provide a similar interpretation for the instanton partition function of tetrahedron instantons, in particular from the expression (227).

  6. 6.

    It would be interesting if we can have a better understanding of the free field representation of the instanton partition function, generalizing the discussion in [32].

  7. 7.

    We may consider the tetrahedron instantons with supergroups by adding negative branes [31, 117, 125] in our construction. We can then calculate the instanton partition function as in [67].