General instanton counting and 5d SCFT

Instanton partition functions of 5d N=1 gauge theories are Witten indices for the ADHM gauged quantum mechanics with (0,4) SUSY. We derive the integral contour prescriptions for these indices using the Jeffrey-Kirwan method, for gauge theories with hypermultiplets in various representations. The results can be used to study various 4d/5d/6d QFTs. In this paper, we study 5d SCFTs which are at the UV fixed points of 5d SYM theories. In particular, we focus on the Sp(N) theories with N_f \leq 7 fundamental and 1 antisymmetric hypermultiplets, living on the D4-D8-O8 systems. Their superconformal indices calculated from instantons all show E_{N_f+1} symmetry enhancements. We also discuss some aspects of the 6d SCFTs living on the M5-M9 system. It is crucial to understand the UV incompleteness of the 5d SYM, coming from small instantons in our problem. We explain in our examples how to fix them. As an aside, we derive the index for general gauged quantum mechanics with (0,2) SUSY.


Introduction and summary
Instantons play important roles in understanding non-perturbative physics of gauge theories. In this paper, we study the partition function for multi-instantons in gauge theories preserving 8 SUSY, on Omega-deformed R 4 or R 4 × S 1 [1,2]. This partition function was first studied to understand the Seiberg-Witten solutions [3] of the 4d N = 2 gauge theories. It however turned out to have much wider applications. In particular, [1,2] considered circle uplifts of the 4d partition functions to 5d. They are Witten indices which capture the BPS spectrum of 5d SYM theories. The usefulness of this partition function for computing the 4d and 5d SUSY partition functions on curved spaces was also shown in [4,5] and [6,7,8,9,10,11,12,13].
Instanton partition functions for the pure N = 2 theories with classical gauge groups were studied in [14], with generalizations [15] to matter hypermultiplets in various representations. These partition functions are given by contour integrals. To the best of our knowledge, a systematic derivation of the contour has not been available, although working prescriptions are known with examples. In particular, the contour issue becomes subtler when there are hypermultiplets in matrix-valued or higher representations of the gauge group. Gauge theories with such matters are important for many reasons. For instance, they appear in various quiver gauge theories, engineered by D-branes or M-theory branes. Adjoint, bi-fundamental, and even tri-fundamental representations [16] appear. In this paper, they appear in the 5d SYM descriptions of 5d/6d superconformal field theories. In particular, we are interested in the 5d SCFTs which admit low energy descriptions by 5d SYMs after relevant deformations [17,18,19].
In this paper, we address a few technical/conceptual issues related to the 5d instanton partition functions. The subjects that we discuss are: (1) contour choice in Nekrasov's instanton partition function, and more generally indices of gauged quantum mechanics with (0, 2) SUSY; (2) the physics of 5d SCFTs from instanton partition functions, addressing various properties of the UV fixed points such as E n symmetry enhancements; (3) the role of small instantons in UV incomplete 5d gauge theories, and the meaning of Nekrasov's partition functions there. We explain these issues below, also summarizing our results.
Indices of SUSY gauged quantum mechanics: On the technical side, we would like to clarify a step which was left somewhat incomplete in the literature, concerning the choice of integral contour when the matter hypermultiplet is in general representation of the gauge group. In the ADHM quantum mechanics description of instantons, hypermultiplets with higher rank (≥ 2) representations in the gauge theory yield bosonic degrees in the mechanics. The contour integrand encounters poles coming from these bosons. The question is how the contour goes around these poles. For the poles coming from the vector multiplets, [1] stated the famous contour prescription. For some gauge theories with matrix-valued hypermultiplets, such as the 4d N = 2 * theory with U(N) gauge group and its 5d uplift, [1] also found the prescription. 1 Also, [21] explains the working contour prescription with tri-fundamental hypermultiplets of SU(2) 3 or bi-fundamental hypermultiplets of Sp(1) × SO (4). [22] recently studied the quiver gauge theories with many SU(n) gauge groups and matters in the adjoint/bi-fundamental representations. Our general derivation of the contour will explain or reconfirm these results. See the last four paragraphs of section 2.3 for the contour prescription.
In fact, a similar problem was recently solved for the indices for the circle compactified 2d gauge theories [23,24,25]. This is a SUSY partition function on a torus, called an elliptic genus. Following their derivation in the context of the gauged quantum mechanics, we derive a similar contour for the instanton partition function, with new aspects which do not have 2d analogues. The ADHM quantum mechanics for instantons is formally obtained by a 1d reduction of 2d N = (0, 4) gauge theories. This is also called N = 4B SUSY quantum mechanics. Exceptionally for 5d maximal SYM, the ADHM quantum mechanics preserves N = (4, 4) SUSY. Regarding our (0, 4) mechanics as a (0, 2) system, we derive the general form of the N = (0, 2) index and apply it to our ADHM quantum mechanics.
The zero modes appearing in the 2d path integral of [23,24,25] live on tori, while the path integral for our quantum mechanics has zero modes living on cylinders. The non-compactness of the zero mode space is a new aspect in the quantum mechanical index. This could be subtle because noncompact moduli develop a continuum in the spectrum above the BPS states. In the context of instanton quantum mechanics, the noncompact direction corresponds to the Coulomb branch moduli space through which instantons can 'escape to infinity.' For instance, in D-brane realization, this is the direction in which Dp-branes (string theory uplift of instantons) move away from D(p+4)-branes. These degrees do not represent any degrees of freedom in the 5d QFT, but enter while one attempts to engineer the UV incomplete instanton quantum mechanics by a UV complete ADHM quantum mechanics. When the 5d SYM has a 5d UV fixed point [17,18,19], the instantons 'generically' cannot move away in this noncompact direction. This is because the 1-loop effects provide linear growths in the instanton masses in the Coulomb branch, and confine them. So in our index, the zero mode integral is convergent in the asymptotic regions of the cylinder. There are interesting exceptions to this confinement, which we study in detail. The 5d gauge theories which uplift to the 6d SCFTs on S 1 also have more nontrivial asymptotic behaviors, which we explain in section 3.
5d superconformal field theories: While we make a general study on a large class of 5d SCFTs classified in [19], we shall also focus on a special class of them in section 4, which we explain in some detail here. We shall study the 5d Sp(N) N = 1 gauge theories coupled to N f fundamental hypermultiplets and one anti-symmetric hypermultiplet. From string theory, this system is engineered by N D4-branes near N f D8-branes and an O8-plane. At 0 ≤ N f ≤ 7, these systems were used to predict the existence of a class of 5d SCFTs at the UV fixed point [17]. Although the SYM theory at small coupling only exhibits SO(2N f ) × U(1) global symmetry, E N f +1 symmetry enhancements were predicted at the strong coupling fixed point. At N f = 8, the 5d SYM is a suitable circle reduction of the 6d (1, 0) SCFT living on the M5-M9 sytem. We mainly study 5d SCFTs with N f ≤ 7 in this paper, although we also explain how to compute the 6d SCFT partition function at N f = 8 (section 3.4.2). The 5d SCFTs for N f ≤ 7 have been studied recently, starting from [10]. They studied the Nekrasov's partition function, and also the superconformal index [26] which one can calculate using the former. [10] mainly discussed the Sp(1) theories. Since the anti-symmetric representation of Sp(1) is neutral, they naturally considered the systems with N f fundamental hypermultiplets only. For N f ≤ 5, the superconformal indices they calculate showed the E N f +1 symmetry enhancement, in a series expansion with the fugacities. For N f = 6, 7, where we expect E 7 , E 8 symmetries, the calculations of [10] did not exhibit the spectra with these symmetries. Our studies began by a rather small motivation to have things to work correctly at N f = 6, 7, as SCFTs with higher E n symmetries are important in gauge theories and string theory. Here we note that [27] computed the E 7 SCFT index from topological strings by Higgsing the 5d T 4 theory, and [28,27] computed the E 6 index from the 5d T 3 theory.
We start by observing that, even if the Sp(1) anti-symmetric hypermultiplet decouples in the perturbative gauge dynamics, the details of the ADHM instanton calculus in [15,10] depend on whether one includes this matter contribution or not. Such phenomenon is somewhat well known. For instance, consider a U(1) gauge theory with adjoint hypermultiplet. As U(1) adjoint is neutral, one may think that this partition funciton is totally trivial, free from any influence from the adjoint hypermultiplet. Instead, [2] captures the U(1) small instantons' contributions to the index, whose singular configurations see the perturbatively decoupled fields. In string theory, this index counts the marginal bound states of single D4-brane and D0-branes. Similar reasoning can be given to our problem with an Sp(1) anti-symmetric hypermultiplet. These nonperturbative couplings are possible due to the small instantons which violate the perturbative intuitions of 5d SYM. Namely, the instanton quantum mechanics given by a SUSY sigma model is incomplete, as the instanton moduli space develops small instanton singularities. To get a complete description, one uses the ADHM gauged quantum mehcanics. The last mechanics is often motivated from string theory, most naturally on Dp-D(p+4) systems. It contains extra degrees of freedom, which have to be carefully treated to compute the QFT observable correctly.
From the brane perspective, the perturbatively decoupled Sp(1) antisymmetric hypermultiplet contains the degrees of freedom for D4-branes moving along the worldvolume of D8-O8. So the related degrees in the ADHM quantum mechanics represent D0-branes moving away from the D4's, bound only to the D8-O8. In other words, the Sp(1) partition function with an antisymmetric hypermultiplet not only counts the 4 + 1 dimensional BPS states living on the D4-branes, but also captures the 8 + 1 dimensional bound states for D0-D8-O8. This phenomenon was observed in [10] at one instanton order. In this paper, we call these '4 dimensional' and '8 dimensional' particles, respectively. The index for the 8d and 4d particles will factorize. Here, [10] further assumed that the effect of including the Sp(1) antisymmetric hypermultiplet in the instanton calculus was exactly providing this extra factorized 8d superparticle index and nothing else, to all orders in the instanton number. As their main interest was the 5d QFT and not these 8d particles, [10] then omitted the antisymmetric hypermultiplet and proceeded with the instanton calculus having N f fundamental hypermultiplets only.
We find that naively discarding the antisymmetric hypermultiplet's effect in the instanton calculus is not always the same as discarding the factorized 8d index. The naive expectation of [10] is correct for 0 ≤ N f ≤ 5, precisely when their indices exhibit E N f +1 symmetry enhancement. However, for N f = 6, 7, we find that Z(with antisymmetric) Z(without antisymmetric) = Z(8d particle) . (1.1) So one has to include the antisymmetric hypermultiplet in the calculation, and then divide by the 8d index afterwards. (We can evaluate the last 8d index separately.) This procedure completely restores the E 7 and E 8 symmetries in the superconformal index.
We can understand in many ways why [10] got good results by simply discarding the antisymmetric hypermultiplet at low N f . Firstly, for 0 ≤ N f ≤ 4, the 5d system has a good 4d limit, which yields either asymptotically free or conformal QFT. Since 4d QFT is well defined, any instanton calculus there should be free of ambiguities, especially concerning small instantons. 2 So including the decoupled Sp(1) anti-symmetric hypermultiplet should not affect the QFT dynamics, and in particular the Seiberg-Witten solution. This implies Z 4d (with antisymmetric) Z 4d (without antisymmetric) = (independent of Coulomb VEV) . (

1.2)
Suppose that the 5d version (1.1) of the relation holds. Then the right hand side is independent of the Coulomb VEV, as the decoupled 8d particles do not see the 5d gauge group. So it will not affect the 4d Seiberg-Witten solution, as required by (1.2). We may understand the success of [10] at N f ≤ 4 from the constraints that the 4d QFT is complete.
There is another way to understand the result of [10] at N f ≤ 5, and actually to improve it to N f ≤ 6 in a way different from what we explained above. We are studying 5d SCFTs with 1 dimensional Coulomb branch and E N f +1 symmetry. There are two ways of engineering them from string theory [19]. We start by explaining two (generally inequivalent) classes of 5d rank N SCFTs engineered by sting theory. Both admit relevant deformations to Sp(N) SYM with N f fundamental hypermultiplets. The first class comes with an extra antisymmetric hypermultiplet, while the second class does not. The first class has a UV interacting fixed point for N f ≤ 7, and the second class has one for N f ≤ 2N + 4. The first can be engineered either by D4-D8-O8 [17], or M-theory on certain singular Calabi-Yau 3-folds [19], by taking low energy decoupling limit. The second class is engineered by M-theory on a different type of singular CY 3 's [19]. At N = 1, the two classes are expected to yield the same 5d SCFTs. The M-theory descriptions for both classes become identical, as the two CY 3 's become the same. However, the D4-D8-O8 description provides a different string theory background for the first class. In both descriptions, the Hilbert spaces at low energy are expected to factorize into the sectors of 5d QFT and the rest. The QFT sector is expected to be the same in any descriptions, but the extra sectors are not. These aspects descend to the ADHM quantum mechanics descriptions of instantons.
The D-brane description of Sp(1) theories exists for N f ≤ 7, while the M-theory description exists for N f ≤ 2N + 4 = 6. The ADHM quantum mechanics used in [10] (without degrees coming from 'Sp(1) antisymmetric hypermultiplet') can be understood as the latter. This ADHM system is rather simple for N f ≤ 5, as studied in [10]. At N f = 6, there exists a continuum in the ADHM quantum mechanics which comes from the extra degrees in the string theory. So we find Z  (1) theory with N f ≤ 6 can be studied as [10], taking into account the subtle Z string factor. For N f = 7 with E 8 , only the D-brane engineering should work. The D-brane approach also generalizes to N f = 8, which has an interesting 6d UV fixed point. 3 So to summarize, it is crucial to carefully follow the string theory guidance rather than the perturbative QFT intuition, when one makes nonperturbative studies on the UV incomplete theories. We shall also study other 5d SCFTs classified in [19] which exhibit similar phenomena. Separating out the factors from string theory which are irrelevant for QFT is the key step, which we shall explain with many examples.
Partition functions of UV incomplete theories: Recently, 5d gauge theories played important roles in understanding many enigmatic SCFTs in 6 [31,32,33,34] and 5 dimensions. The 5d Yang-Mills theories that one uses to compute the CFT observables are non-renormalizable, at least apparently. They are low energy effective descriptions of UV SCFTs with relevant deformations (5d SCFT) or circle compactifications (6d SCFT). The instanton partition functions on R 4 × S 1 are often related to other SCFT observables, given by SYM partition functions on curved 5-manifolds such as S 5 or S 4 ×S 1 [6,7,8,9,10,11,12,13]. So it is important to develop a more abstract and instrinsic notion of the instanton partition functions of 5d SYMs as those of 5d/6d SCFTs on Omega-deformed R 4 × S 1 or R 4 × T 2 , not referring to the descriptions one uses to compute them. The computation of our 5d SYM observable on R 4 × S 1 boils down to the 1d path integral for the instanton quantum mechanics [1]. The UV incompleteness of the original 5d SYM leaves a remnant on the instanton quantum mechanics, by exhibiting small instanton singularities. We UV-complete it to ADHM gauged quantum mechanics, with extra UV degrees, which reduces to the instanton quantum mechanics in the limit of strong gauge coupling (or equivalently low energy) in the 'Higgs branch' of the mechanics. Doing computations this way, one should either find a method to decouple the extra UV degrees, or should separately compute the extra stringy contribution and divide it.
A simple example is the U(1) theory with one adjoint hypermultiplet. This is naively a free QFT in 5d, but [20,8,9,11] could get the spectrum of circle compactified 6d (2, 0) theory physics of the free tensor multiplet from small instantons. So this is clearly an extra UV input beyond the naive 5d SYM, but here we do not acquire extra states' contribution from string theory. Similarly, the U(N) N = 1 * theory that was used to study the circle compactified non-Abelian (2, 0) theory could also contain such ambiguities resolved by string theory considerations. Our example of Sp(1) gauge theory with one antisymmetric hypermultiplet exhibits a more noticeable subtlety of the 5d gauge theory. This is partly because the subtlety of the 5d SYM theory becomes manifest in a regime (N f ≥ 6) that 4d QFT limit does not exist. So it should be clear that this is really the subtlety of 5d non-renormalizable gauge theory. We hope that the nontrivial examples studied in this paper could help clarify the usefulness and subtleties of the 5d gauge theories in studying higher dimensional quantum field theories.
The rest of this paper is organized as follows. In section 2, we derive the index of general gauged quantum mechanics with (0, 2) SUSY, and apply it to the ADHM gauged quantum mechanics with (0, 4) SUSY. In section 3, we explain examples which have 5d or 6d UV fixed points. In particular in section 3.4, we explain how to factor out the extra contribution to the index from the string theory states, decoupled from the QFT. In section 4, we explain the 5d SCFT indices on the D4-D8-O8 system, from our SYM theories with N f ≤ 7. We show the E N f +1 symmetry enhancements of these superconformal indices.
Note added: Our sections 2.2 and 2.3 on the general index have an overlap with [35]. Also, [36] discusses the same index as ours presented in section 2.3.

Instantons and their indices
We consider 5d N = 1 SYM, with U(N), SO(N), Sp(N) gauge groups. We are interested in selfdual instantons which satisfy F mn = ⋆ 4 F mn = 1 2 ǫ mnpq F pq on spatial R 4 (m, n, p, q = 1, 2, 3, 4). Let us decompose the 8 SUSY into Q A α ,Q Ȧ α , where α = 1, 2,α =1,2 are for the doublets of SU(2) l × SU(2) r = SO(4) rotating R 4 , and A = 1, 2 is for the doublet of SU(2) R R-symmetry. Q A α ,Q Ȧ α are subject to symplectic-Majorana conditions. The SUSY algebra is given by M, N = 1, 2, 3, 4 are Dirac spinor indices, C M N is the charge conjugation matrix, k is the instanton number, v is the scalar VEV in the Coulomb branch, Π is the electric charge. The topological U(1) I charge of self-dual instantons is given by These instantons preserve 4 real SUSYQ Ȧ α . They can also form marginal bound states with perturbative particles with electric charges, namely the W-bosons and their superpartners. The bound states preserve the same 4 SUSY, with the BPS mass provided that the signs of the electric charges are properly chosen. The signs should be chosen with tr(vΠ) ≥ 0.

ADHM quantum mechanics with N = (0, 4) SUSY
The zero modes of self-dual instantons can be described by the ADHM data [37] for classical gauge groups, subject to the ADHM constraint equations. In 5d SYM, the moduli space approximation of these instantons is given by a supersymmetric sigma model with the target space given by the instanton moduli space. The partition function of [1] can be understood as that of this mechanical system. The moduli space has small instanton singularities, implying that the quantum mechanics description is incomplete. It admits a UV completion by a gauged SUSY quantum mechanics, often called gauged linear sigma model (GLSM). The ADHM data provide the dynamical degrees, and the ADHM constraints are realized as the vanishing condition of the D-term potential. The incomplete mechanical system is obtained at low energy, or equivalently when quantum mechanical gauge coupling is infinite. When the gauge theory and instantons are engineered by D-branes in string theory, the ADHM mechanics is the D-brane quantum mechanics. As we shall explain below, the UV completion includes extra degrees of freedom, on top of the degrees relevant for QFT.
We first explain the ADHM data for various classical gauge groups. The SU(N) gauge group is replaced by U(N), which has no effect in the classical gauge theory viewpoint (as the overall U(1) plays no role in constructing self-dual solutions). For each 5d gauge group, G = U(N), Sp(N), SO(N), the quantum mechanical gauge groupĜ for k instantons is given byĜ = U(k), O(k), Sp(k), respectively. The ADHM data consists of the following matrices: qα which is in bi-fundamental representation of G×Ĝ, and a αβ = 1 √ 2 a m σ m αβ which assumes a matrixvalued representation ofĜ. Here, σ m = (i τ , 1). qα with givenα is an N × k matrix for U(N), an N ×2k matrix for SO(N), and an 2N ×k matrix for Sp(N). For G = U(N), Sp(N), SO(N), a m is in the adjoint, symmetric, and antisymmetric representation ofĜ = U(k), O(k), Sp(k), respectively. In the last two cases with Sp(N), SO(N), qα and a m are subject to suitable reality conditions. Together with superpartner degrees, their quantum mechanics action preserves 4 SUSYQ Ȧ α of the self-dual instantons. They form N = (0, 4) SUSY. It comes with an SO(4) = SU(2) r × SU(2) R 'R-symmetry.' Let us explain their supermultiplet structures. Together with the fermionic superpartners ψ A , λ A α , the above ADHM degrees form the following multiplets under the (0, 4) SUSY The scalars are in the (2, 1) representations of the SO(4) = SU(2) r × SU(2) R . Let us call them (0, 4) hypermultiplets, in the sense that it is obtained by truncating some fermions from the hypermultiplets of N = (4, 4) theory. The UV quantum mechanics also comes with a vector multiplet forĜ, which consists of a worldline gauge field A t , a scalar ϕ, fermionsλ Ȧ α and 3 auxiliary fields Dαβ = Dβα. Although A t , D do not possess physical degrees, ϕ,λ Ȧ α do. They are extra degrees present only in the UV theory. Their action is given by v is the Coulomb VEV of the 5d SYM, and ζ I for I = 1, 2, 3 are FI parameters. For instance, this action can be obtained by starting from the N = (4, 4) ADHM instanton mechanics for 5d maximal SYM, and truncating to the (0, 4) system. We used the notation of [20]. 4 Also, one can possibly add 1 dimensional Chern-Simons term This is induced by the Chern-Simons term of 5d SYM at level κ [38,39]. Our (0, 4) supersymmetry transformation can also be obtained from [20] by discarding half of the (4, 4) SUSY. In our discussion below, we shall only need for all hypermultiplet fields (Φα, Ψ A ). g QM is the quantum mechanical gauge coupling. The strong coupling limit g QM → ∞ of this theory is the quantum mechanics of instantons in the 'Higgs branch,' where the mechanical hypermultiplet degrees in (2.4) are nonzero [40]. This can be seen from the interaction |qαϕ| 2 on the first line (say, at v = 0). The mass for ϕ in the Higgs branch is g QM |qα|, which becomes large in the strong coupling limit.
If the 5d SYM has hypermultiplets, there are more zero modes in the instanton background, which can be determined by an index theorem. They are fermionic degrees in the UV incomplete supersymmetric sigma model, and no more normalizable bosonic zero modes appear. In the UV complete ADHM gauged quantum mechanics, the degrees of freedom depend on the representation of the 5d hypermultiplet in G, and also possibly on the string theory engineering. There may be extra UV bosonic degrees in the ADHM quantum mechanics. In appendix A, we explain a few examples of ADHM mechanical degrees coming from the 5d hypermultiplets. The 5d hypermultiplets induce mechanical hypermultiplets and/or Fermi multiplets. The hypermultiplet in quantum mechanics takes a structure different from (2.4), Its representation under G×Ĝ depends on the 5d hypermultiplet: see appendix A. An important difference from the hypermultiplets (2.4) in the ADHM data is that the scalars in (2.9) are in the (1, 2) representation of SU(2) r × SU(2) R . The existence of two (0, 4) multiplets in the ADHM mechanics was known well, e.g. see [41]. There could also be various 'Fermi multiplets,' whose on-shell degree is only a complex fermion Ψ. The representation of Ψ in G ×Ĝ depends on 5d hypermultiplets. The (0, 2) decompositions of these multiplets are and Ψ reduces to a (0, 2) Fermi multiplet. Again φ 2 ,φ 1 ∼φ 2 are chosen because Q acts on them nontrivially. The scalars have J = − 1 2 .
Including all these degrees, it suffices for us to know the off-shell (0, 2) action and SUSY to compute the index. Apart from the 1d Chern-Simons term (2.6), these can be obtained by a 1d reduction of a suitable 2d (0, 2) action and SUSY, e.g. given in the appendices of [23,24] 2.2 (0, 2) indices for instantons: rank 1 gauge groups The full partition function of 5d SYM takes the form of Z = Z pert Z inst , where Z pert is the perturbative partition function which acquires contribution only from the W-bosons and superpartners. Z inst acquires contribution from instantons, and takes the form of Z inst = ∞ k=0 Z k q k with Z 0 = 1. In the ADHM quantum mechanics, we will study a Witten index Z k QM for k instantons, which counts states preserving a pair of mutually Hermitian conjugate supercharges Q, Q † , chosen amongQ Ȧ α . The index Z QM ≡ ∞ k=0 Z k QM q k is essentially Z inst , up to a possible multiplicative factor which we call Z string . We explain Z string in section 3.4 with examples. Following [1], we choose Q ≡Q 1 1 = −Q 12 and Q † ≡Q 2 2 =Q 21 . The index for the ADHM mechanics is defined by (2.11) J 1 , J 2 are the Cartans of SO(4), rotating two orthogonal 2-planes of R 4 . In the SU(2) l ×SU(2) r notation, the two Cartans J l , J r are given by J l = J 1 −J 2 2 , J r = J 1 +J 2 2 . J R is the Cartan of the SU(2) R R-symmetry. α i is the chemical potential for the electric charges in the Coulomb branch, where i runs from 1 to the rank of G. All other flavor symmetries are collectively called F , conjugate to the chemical potential which we call z. β is the standard regulator of the Witten index, which does not appear in Z k QM . Z k QM admits a supersymmetric path integral representation, in which the time direction is compactified with circumference β.
The index (2.11) can be regarded formally as a circle reduction of the elliptic genus partition function of a 2d N = (0, 4) gauge theory on T 2 . The multiplets that we explained in section 2.1 uplift to those in 2d, where we uplift A t , ϕ as the 2d vector potential. In particular, our index (2.11) has the same structure as the 2d (0, 2) ellipitic genus, by regarding Q ≡ −Q 12 , Q † as the N = (0, 2) SUSY. The index Z k QM viewed this way can be computed closely following [23,24]. Following [23], we shall first present the 'naive' computation, highlight the subtlety and then explain the proper derivation. The index does not depend on continuous parameters of the theory preserving Q, Q † (up to a caveat we discuss in a moment). So we can tune these parameters to any values which ease the computation. The action is multiplied by the gauge coupling 1 g 2

QM
, and the index for the QFT instantons is defined at g QM → ∞ in the Higgs branch of mechanics. We replace 1 g 2 QM by 1 e 2 for the gauge kinetic term, and by 1 g 2 for the matter kinetic term, and send e, g → 0. It naively appears that this will yield a steep Gaussian integral around zero modes. So one first identifies all the zero modes, and keep them fixed and integrate over the non-zero modes first. After that one should integrate over the zero modes exactly.
The zero modes are given by the holonomy of the gauge field A τ on the temporal circle S 1 , and the scalar ϕ in the vector multiplet. Here and below, we multiply a suitable power of β to these variables, as well as all other variables appearing in the index calculus, to make them dimensionless. In particular, the rescaled eigenvalues of A I τ have period 2π. So there are r complex eigenvalues φ I = ϕ I + iA I τ living on cylinders, where r is the rank ofĜ. Note that in [23,24], the analogous zero modes for 2d gauge theories are the holonomies A 1 + iA 2 , whose eigenvalues live on tori. Also, there are fermionic zero modes fromλ˙1 1 ,λ˙2 2 .
We first explain the 1-loop determinant obtained by integrating over massive modes [1,15,23,24]. Each (0, 2) chiral multiplet Φ contributes to a factor of (2.12) ρ is the weight of Φ inĜ, J = J r + J R is the sum of Cartans of SU(2) r × SU(2) R , F collectively denotes the rest of global charges, in our case for G, SU(2) l , and possibly extra flavor symmetries. z also collectively denotes their chemical potentials, α, ǫ − , m. All charges are for the complex scalar of the chiral multiplet. A (0, 2) Fermi multiplet Ψ contributes to a factor of In particular, the (0, 2) Fermi multipletλ 1 2 ,λ 2 1 from the (0, 4) vector multiplet yields where α is the weight of the adjoint representations ofĜ: it runs over all roots as well as Cartans. Physically, these represent the complex ADHM constraints. The contribution of the vector multiplet A t , ϕ,λ 1 1 ,λ 2 2 takes the same form as that of a Fermi multiplet. Sinceλ 1 1 ,λ 2 2 carry charges J = 0, F = 0, the determinant is given by We multiplied the integral measure for φ I in foresight. The index formula that we will get is with the contour to be derived below. W is the Weyl group ofĜ. (2.12), (2.13), (2.14) can be obtained by taking the 'q → 0 limit' of (2.13), (2.14), (2.15) in [24], ignoring the overall 'vacuum energy' factor of the form q # . 5 When the gauge groupĜ is disconnected, one should turn on discrete holonomies which change the above formula. We shall encounter them in our example with Sp(N) instantons, asĜ = O(k) is disconnected. We shall explain the effect of discrete holonomies in sections 3.2 with this example, following [10].
There are various dangerous regions in the zero mode space, some analogous to those of the 2d index [23,24] and some being intrinsic to the quantum mechanics. Let us explain the latter first, and then the former.
After integrating over non-zero modes, we should perform an integral over the cylinders. The integral is over a noncompact region, implying that there could possibly exist a continuum in the spectrum if the 1-loop measures Z Φ , Z Ψ , Z V do not provide quantum suppression. (Other noncompact zero modes from instanton positions and sizes are lifted by the fugacities ǫ 1 , ǫ 2 , α i .) If a true continuum is developed on top of the BPS states, the Witten index loses its topological nature. Continuous deformations of the theory like e, g → 0 are no longer guaranteed to leave the index invariant, due to the contributions from the continuum. However, this continuum is formed by making a UV completion of the instanton quantum mechanics. This is because the continuum comes from the extra vector multiplet degrees in the ADHM quantum mechanics, while the instanton quantum mechanics of our interest is obtained by going to the IR limit (or the g QM → ∞ limit) in the Higgs branch [40]. So all subtle continuum contributions are expected to come from the extra stringy degree, which we will eventually factor out to get the QFT index. This procedure of factoring out the extra UV degrees is explained in section 3. The noncompact regions of the zero modes are handled in the following way. For the r eigenvalues φ I (I = 1, 2, · · · , r), we take −Λ 1 ≤ ϕ I = Re(φ I ) ≤ Λ 2 , with large IR cut-offs Λ 1 , Λ 2 > 0. After we compute the integral over all the modes, we can take these cutoffs back to infinity. Now we discuss the second kind of dangerous regions in the zero mode space, whose structure is similar to those of [23,24] in 2d. It is legitimate to perform the Gaussian integration over the nonzero modes with fixed φ = ϕ + iA τ only when the non-zero modes carry large masses. This assumption fails near the values of φ for which the integrand diverges. These dangerous points are provided by the poles from the chiral multiplet determinant, at ρ(φ * ) + Jǫ + + F z = 0. The path integral thus has to be done carefully near φ * . Following [23,24], we keep the zero modes of the auxiliary field D and carefully take the e, g → 0 limit. With given nonzero D, the only determinant that changes is Z Φ , which is given by . (2.17) The calculus which takes into account these dangerous regions was done (for 2d theories) in [23] when the gauge group has rank 1, and then in [24] for the gauge group with general rank.
In this subsection, we repeat the analysis of [23] for the rank 1 case in our quantum mechanics version, taking care of small differences in 1d. In the next subsection we obtain the result with general rank. At rank 1, all weights are replaced by the charge Q i of the i'th mode.
We take e, g small but finite, and keep also the D zero modes. The path integral then reduces to an integral of the form ζ is the Fayet-Iliopoulos parameter that one can turn on ifĜ contains U(1) factors. M is the space of the zero mode φ, f e,g is obtained by the path integral except φ, D (and also integrated over the zero modes of vector multiplet fermion). As in [23], the g → 0 limit exists for any φ, as long as e is not zero, since the coupling of chiral multiplet scalars with D yields a potential of the form e 2 (|φ| 2 − ζ) 2 . Now before sending the e → 0 limit, we identify the dangerous regions of the φ integral following [23]. We call M sing the set of all pole locations φ * , and let ∆ ε with small ε to be the ε-neighborhood of M sing . We divide the integral over φ as Following [23], we take the e → 0 limit in a way that the second integration in ∆ ε does not contribute. This can be done if ε is sent to zero much faster than e # with a positive number #, so that the small volume factor of ∆ ε dominates over the divergent behavior as e → 0. See section 3.2.1 of [23] for the precise scaling form of the limit e, ε → 0.
Thus we are left with The factor f e = lim g→0 f e,g (φ,φ, D) is given in the e → 0 limit by [23] f e (φ,φ, D) where λ 0 is the zero mode of the gaugino (λ 1 1 ,λ 2 2 in our ADHM mechanics), (2.23) The constant c is taken to be c = i/π following [23], with comments in the footnote 5 on the sign in mind. Here, one can show that h(φ,φ, D, ǫ + , z)g(φ,φ, D, ǫ + , z) = 1 πD ∂ ∂φ g(φ,φ, D, ǫ + , z) , (2.24) again following [23]. Now consider the integral over D along R. We deform the contour slightly away from the real line [23], to one of the following two contours Γ ± , D ∈ R + iδ with 0 < ±δ < ε 2 . Our final result will be independent of the sign of δ. So using (2.24), one obtains where Γ ± is the deformed D-contour with positive/negative δ, respectively. We also used Let us consider the above integral expression for each patch of ∂∆ ε +∂M 0 +∂M ∞ . The patch can either surround a pole φ * , or can be at ϕ = −Λ 1 , Λ 2 . We first consider the former. With small ε, the pole for D (apart from D = 0) which is closest to the real axis is located at D = iQ i ε 2 . This pole approaches the real axis of the D plane in the e, ε → 0 limit. If the sign of Q i is opposite to the sign of the contour shift parameter δ, then the closest pole approaching the real axis does not hit the D integration contour. If the two signs are the same, the pole D = iQ i ε 2 will cross the contour. Let us denote by ∂∆ (+) ε the union of small contours surrounding poles with Q i > 0, and by ∂∆ (−) ε the union of small contours surrounding poles with Q i < 0. Firstly, consider the case with φ ∈ ∂∆ (∓) ε and D ∈ Γ ± (upper/lower signs correlated). Then there are no poles, including D = iQ i ε 2 , which cross Γ ± as ε → 0. So we can relax the condition |δ| < ε 2 and keep δ = Im(D) fixed as ε → 0. So one obtains since g(φ, D, ǫ + , z) with given nonzero D is a bounded integrand, while the dφ integral region shrinks as ε → 0. On the other hand, consider the case with φ ∈ ∂∆ (±) ε and D ∈ Γ ± . The poles D = iQ i ε 2 approach to zero in the ε → 0 limit and cross the contour Γ ± . To compute the integral avoiding these crossings, we deform the contour Γ ± to Γ ± = Γ ∓ ∓ C 0 , where C 0 is a small contour surrounding D = 0 counterclockwise [23]. Following the similar argument which led to (2.27) where g(φ, ǫ + , z) ≡ g(φ, D = 0, ǫ + , z) is the holomorphic integrand in φ. Thus, the ∂∆ ε part of the integral (2.25) is given by depending on the deformed contour Γ ± . R i is the residue of g(φ, ǫ + , z) at the i'th pole.
We finally consider the contribution to (2.25) from the patch φ ∈ ∂M 0 + ∂M ∞ . If φ is on one of these contours with large IR cutoffs Λ 1 , Λ 2 , ϕ is very large so that D appearing in the denominator of (2.17) is negligible. So we replace g(φ, D, ǫ + , z) in (2.25) by holomorphic g(φ, ǫ + , z) = g(φ, D = 0, ǫ + , z) and obtain where R 0 and R ∞ are the residues of dz z g(φ, ǫ + , z) in z ≡ e φ = 0 and w ≡ z −1 = 0, respectively. Its contribution to the partition function (2.25) is given by which is the same for both contour choices Γ ± . So both f ± (eζ) are given in terms of the error function erf(x) defined by Namely, one obtains where Pr denotes the principal value. These computations are done after deforming the contours Γ ± to those which are almost at the real axis and go around the pole at D = 0 in two different ways, as shown in Fig. 1. So we find that Figure 1: Deformations of the contours Γ ± to compute f ± (0) Note that these functions satisfy f − (eζ)−f + (eζ) = 1. The contribution to the partition function from the two residues at the infinities of the cylinder is thus Note that the residues R 0 , R ∞ at infinity may be zero for some 5d SYM theories. In fact, in many (but not all) interesting examples, we shall find in section 3 that R 0 + R ∞ = 0. It will also happen very often that R 0 = R ∞ = 0 separately. Note that the computation of this part depends on the FI term, which is allowed only when the gauge group contains a U(1) factor. In the instanton calculus, this is possible only for the U(N) instantons, for whichĜ = U(k). For other cases, with G = Sp(N), SO(N), we shall always find R 0 + R ∞ = 0 from the Weyl symmetry ofĜ, so that Z ± ∂M 0 +∂M∞ = 0.
Adding all the contributions Z ± ∂∆ε and Z ± ∂M 0 +∂M∞ , the index Z is given by where the last equation holds because the sum of all residues on the cylinder is zero if one includes those at infinities. So from now on we call Z = Z + = Z − . The FI term ζ does affect the result. For instance, at eζ = −∞, 0 and +∞, one obtains Since ζ is a parameter of the theory, the index can depend on it only when there is a continuum contribution from the Coulomb branch (with nonzero ϕ). In particular, Z(eζ) depends continuously on eζ, so that Z(eζ) expanded in the fugacities cannot generally have integral coefficients. This is also expected in general, with continuum contributions. The point ζ = 0 is where the Coulomb branch with nonzero φ meets the Higgs branch. Nonzero ζ generates a mass gap for the Coulomb branch degrees with the mass proportional to e 2 ζ, so that the continuum cannot affect the Witten index. Since we computed the index in the e → 0 limit, the above result as a function of finite eζ generates vanishing mass gap e 2 ζ → 0. The indices with finite gaps are thus obtained in the eζ → ±∞ limit. Indeed in section 3.4, we shall illustrate that Z(eζ = ±∞) have integral coefficients, as expected from general considerations.
To ease the analysis with higher rank in the next subsection, we rephrase the rank 1 result as follows. The functions f ± (eζ) satisfy f ± (eζ = ∓∞) = 0. 6 So when δ and ζ have opposite signs, the contribution (2.37) from the IR cutoff contours vanishes. One thus obtains which does not refer to the contributions from residues at infinities.
The possible difference between the Witten indices in two limits eζ = ±∞, implies a wall crossing of the index across ζ = 0, at which the system of our interest meets a continuum from the Coulomb branch.
All discussions till here go through for general quantum mechanical index with (0, 2) SUSY. Now let us consider the instanton partition function with our (0, 4) mechanics. We find that ζ dependence of the 5d SYM index is unacceptable. This is because the ζ dependence incurs a strange dependence of Z on ǫ + , conjugate to the Cartan of the diagonal combination of SU(2) r × SU(2) R . Nekrasov's partition function counts half-BPS states of the 5d N = 1 gauge theory, preserving 4 Hermitian supercharges. Their short multiplets break neither SU(2) r nor SU(2) R symmetry. So although our index only refers to 2 SUSY, it should be an even function of ǫ + if it only captures the spectrum of these half-BPS states with a further refinement with ǫ + . Since both ζ, ǫ + break SU(2) R , the sign flip of ζ effectively induces the sign flip of ǫ + in the partition function. So if Z(ζ ≶ 0) are same, it is an even function of ǫ + . However, if R 0 + R ∞ = 0, this measures the failure of Z(ζ > 0) and Z(ζ < 0) being even in ǫ + . Thus, ζ dependence and ǫ + → −ǫ + asymmetry is unreasonable if this index is counting half-BPS bound states of instantons and W-bosons in the 5d super-Yang-Mills theory. In 5d SCFTs, this is also a consequence of the superconformal symmetry which contains SU(2) R .
To find the resolution to this puzzle, we should understand the true nature of Z. The possible wall crossing happens because the Coulomb branch continuum appears at ζ = 0. From our ADHM quantum mechanics, the Coulomb branch degrees appear only when we go to the UV complete gauge theory description of the instanton quantum mechanics. So there may appear contribution to the index from the extra UV degrees of the ADHM quantum mechanics which do not belong to the QFT Hilbert space. This can either be a fractional contribution from the continuum, or integral contribution coming from marginal bound states involveing the extra stringy states. In any case, these have to factorize with the true QFT index, since the field theories that we shall study from string theory are obtained by taking suitable decoupling limits. We should identify Z string and factor it out to study the true QFT index. Whether we can do so or not depends on our knowledge on the string theory background which engineers the QFT. We explain this point further in section 3.4.
Coming back to the ζ dependence and ǫ + → −ǫ + asymmetry, the wall crossing of the index should happen only in Z string , but not in Z QFT . As we are not aware of the half-BPS state interpretation of the BPS states captured in Z string , we find this to be a consistent resolution to the puzzle stated above.
When R 0 = R ∞ = 0 separately, the continuum from ϕ is lifted by quantum effects. See section 3 for classifications and examples. In this case, the index is independent of the continuous parameters of the theory, and its coefficients are integers. When R 0 + R ∞ = 0 but R 0 = −R ∞ = 0, there is no wall crossing but still is a continuum from ϕ. In this case, since a continuum is attached to the Higgs branch (the latter being our main interest), the index may have fractional coefficients. However, still these non-integral contributions will go to Z string , and not to Z QFT .

(0, 2) indices with higher rank gauge groups
Now we study the index with gauge symmetryĜ of general rank. In the case of instanton quantum mechanics, the rank ≤ 1 case covers single instantons for the U(N) theory when G = U(1), 1 or 2 instantons for the Sp(N) theory whenĜ = O(1), O(2), and the single instantons for the SO(N) theory whenĜ = Sp(1). All higher instanton sectors require the study of higher rank quantum mechanics.
The path integral again can be computed in the e → 0 and g → 0 limit as the above rank 1 case. Again the limit should be taken with care. We again start by identifying the zero modes in the limit, do Gaussian integration over non-zero modes which yield product of various Z v , Z Φ , Z Ψ factors, and then integrate over the zero modes. The last integral becomes a multi-variable contour integral, determined by a careful computation similar to the previous subsection. In the k instanton quantum mechanics, the rank r of the gauge groupĜ is proportional to k.
for O(k) − . Following [24], we explain the general structure of the index for the connected groupĜ in this section, and explain examples with disconnected O(k) in section 3 following [10]. The zero modes consists of r holonomy eigenvalues of A τ along the temporal circle or circumference β, and also the r eigenvalues of the scalar ϕ in the vector multiplet. The r complex eigenvalues φ I = ϕ I + iA τ with I = 1, 2, · · · , r thus locally form r copies of cylinders. More precisely, let h C be the complexified Cartan subalgebra and Q ∨ the coroot lattice. Then defining M = h C /Q ∨ , the space of zero modes is given by M/W , where W is the Weyl group. The 'naive' integrand Using the variables z I ≡ e φ I , r copies of cylinders map locally to C r , where the two infinite regions of a cylinder map to the origin and infinity of each C. Since the space of zero mode is again noncompact, we have to introduce an IR cutoff for large ϕ I and then remove the regulator after the path integral. As in the rank 1 case, there may appear contributions from the boundaries of the IR cutoff points. Namely, the holomporphic measure g(φ I , ǫ + , z) may not converge to zero as |ϕ I | → ∞. Each chiral multiplet provides a factor of 1 2 sinh(ρ i (φ)/2+··· ) , so that a convergent factor e −|ρ i (ϕ)|/2 is provided. On the other hand, each vector or Fermi multiplet provides a divergent factor of e |ρ(ϕ)|/2 , where ρ is the weight of the mode. So if there are too many divergent factors, g may not converge to 0 depending on the direction in which we take ϕ I to be large on R r . As in the rank 1 case, the important quantity which can contribute to the index is the sum of two residues as one goes to ρ i (ϕ) → ±∞ (together with specifying other singular hyperplanes so that a pole φ * is picked), analogous to R 0 + R ∞ in the rank 1 case.
We shall explain the expression for the higher rank index in terms of the residue sum shortly. As in the rank 1 index, one should perform a careful path integral near the above hyperplanes (including infinities of the cylinders). Just as in the rank 1 case, one slightly shifts the integration contours for r eigenvalues of D field, from R r to R r + iδ, where δ is an r dimensional real vector in h. In the case of 2d index, the expressions appearing after the different choices of δ are apparently different, but they should turn out to be the same after taking into account that suitable 'sums of residues' are zero for a meromorphic function. δ is constrained by fixing a vector η ∈ h * and demanding η(δ) > 0 [24], which was enough to specify the index as a suitable residue sum. In quantum mechanics, ζ dependence may appear due to the poles at infinities, but we already know that there are simple choices of δ (and thus η) which provides an expression for the index without referring to the residues at infinities. In (2.41) and footnote 6, we have seen that the contributions from infinities vanish when the direction of the contour shift and the sign of ζ are correlated, ζδ < 0. In the higher rank case, this amounts to taking ζ(δ) < 0. So we choose the vector η to be η = −ζ in all expressions below, whenever necessary. When the gauge groupĜ does not contain any overall U(1) factor, such as Sp(k) or O(k), the sum of two residues from the two infinites ρ(ϕ) → ±∞ of a cylinder always vanishes, from the Weyl symmetry ϕ I → −ϕ I ofĜ. In this case the result for the quantum mechanical index will not depend on the choice of η, just like the 2d index of [24].
The contour choice for the r variables φ I for our quantum mechanical index follows from the same consideration as [24], assuming η = −ζ below whenever necessary. [24] phrased the 'contour' by specifying the set of residues that have to be summed over, with sign factors. The r complex variables encounter a 'pole' when n ≥ r hyperplanes among H i meet at a point φ = φ * . When n = r, the intersection (or 'hyperplane arrangement') is called non-degenerate. If n > r, it is called degenerate. Hyperplane arrangements are called projective when the n charge vectors Q(φ * ) ≡ {Q i |φ ∈ H i } responsible for the pole φ * are contained in a halfspace of h * . The results of [24] apply straightforwardly when all hyperplane arrangements are projective. In all the examples of instanton calculus that we studied in this paper, we find that the projective condition is met. For U(N) instantons for which we know the general pole selection rules, one can show that there may be degenerate intersections but they are all projective. For Sp(1), Sp (2), we made numerical studies up to k = 3, 4, 5 orders depending on theories. We find that there may be degenerate poles, but they are all projective.
Near the pole φ * , one can Laurent expand the integrand Z 1-loop , in negative powers of Q i (φ − φ * ). The nonzero residues are obtained only from the 'simple pole' parts, which are linear combinations of the functions of the form The so-called Jeffrey-Kirwan residue JK-Res, that is relevant for specifying the index, also refers to the auxiliary vector η. JK-Res is defined by [24] JK-Res(Q * , η) (2.46) 'Cone' denotes the cone spanned by the r independent vectors. Namely, η ∈ Cone(Q 1 , · · · , Q r ) if η = r i=1 a i Q i with positive coefficients a i . Although this definition apparently looks overdetermining, it is known to be consistent: see [24] and references therein. As explained in the previous paragraph, η specifies how the D contour is deformed to the imaginary direction +iδ by demanding η(δ) > 0. The index is given by [24] Note that the result depends on the choice of η when the residue sum at the two infinities of a cylinder (spanned by any ρ(φ) appearing in Z 1-loop ) does not vanish. In this case, the above expression is understood with η = −ζ. So Z in this case is a piecewise constant function of ζ.
We note that, for non-Abelian gauge groupĜ, the co-vector ζ is restricted to be along the overall U(1) factors only. η is chosen in [24] not to coincide with the weights Q i associated with the poles. (More generally η is taken not to lie at the boundary of the 'chambers.' See [24] for explanations.) So one might think that it would be troublesome to impose η = −ζ if ζ is aligned along the forbidden direction for −η, e.g. being proportional to a weight in the problem. For Abelian theories, such as U(1) r theories, ζ can be a generic vector in h * so that ζ on the boundary of a chamber is potentially a wall-crossing point. ζ can be displaced from such a value in the Abelian theories, and one obtains different results across the wall by setting η = −ζ for these displaced ζ. However, for U(r), its FI term is along a fixed direction on h * = R r , proportional to (1, 1, · · · , 1). This might be at the boundary of chambers. For instance, the weight (1, 1, · · · , 1) appears with the rank r totally symmetric representation of U(r), or a totally antisymmetric representation of it. If ζ is at the boundary, then one can remove the ambiguity by slightly shifting η away from −ζ(1, 1, · · · , 1). Different shifts may leave η in different chambers. However, since −ζ(1, · · · , 1) is a Weyl invariant point of U(r), these different chambers map to one another by Weyl reflection. Due to this symmetry, different shifts of η should yield the same result.
In all examples that we study, we find that the Jeffrey-Kirwan residue rule is equivalent to the following prescription, which is well known in the instanton calculus for some theories. We shall perform the contour integral over r variables φ I one by one. The contour integral takes the form of The poles from a chiral multiplet factor takes the form where Q i is the charge vector of the chiral multiplet Φ i . So the pole one picks up for the z I = e φ I variables are determined by r different equations of the kind In the instanton calculus, the value of charge J conjugate to ǫ + is always positive when the quantum mechanical chiral multiplet comes from the 5d SYM theory's vector multiplet (namely, the ADHM degrees). On the other hand, one always finds that J < 0 for the mechanical chiral multiplet coming from 5d hypermultiplets. At this point, we temporarily treat the e −ǫ + factors appearing in the Z Φ 's from the 5d vector multiplet and those from the 5d hypermultiplet independently. Namely, we substitute e −ǫ + → t < 1 for the measure coming from 5d vector multiplet, and e −ǫ + → T > 1 for the measure from 5d hypermultiplet. This makes the pole for e Q i (φ) to be all inside the unit circle in (2.49).
With these understood, the alternative residue prescription which turns out to give the same result is obtained by regarding each integral variable z I as living on the unit circle on the complex plane. Then we integrate over these variables one by one, for which we have to pick up poles inside the contour and sum over their residues (assuming t < 1, T > 1). After the residue sums of all r integrals, we set t, T back to the same value t = T = e −ǫ + . This yields the same result as the index obtained by the sum of JK-Res. Of course, to see the agreement most clearly, one should choose η carefully, related to the order of integral for φ 1 , φ 2 , · · · , φ r in the prescription. Whenever one encounters a pole at z I = 0, one does not include their residues, as part of the prescription. Also, one occasionally encounters poles which are not clearly inside or outside the unit circle with t ≪ 1, T ≫ 1 only. Here, one may choose other fugacities arbitrarily to shift such poles away from the unit circle. The arbitrary shift will not affect the result, as we will illustrate. So far, this is the prescription which works when theindex has zero residue sums at infinities over a cylinder. For some U(k) instanton calculus for which the sign of FI term matters, we chose η = −ζ and summed over JK-Res. In the alternative prescription, ζ dependence appears as a failure of the index to be an even function of ǫ + . So the two different indices are obtained by either running through the above prescription as explained above, or alternatively taking t = e ǫ + < 1 and T = e ǫ + > 1 temporarily and going through the unit circle contour prescription. This yields results which are related to each other by flipping ǫ + → −ǫ + , or equivalently ζ → −ζ.
In the rank 1 case, it is immediate that the alternative prescription yields the same result as the result of section 2.2. This is because (2.49) for rank 1 is given by for the chiral multiplet originating from 5d vector and hypermultiplet, respectively. The rule in section 2.2 was to sum over the residues with Q i > 0. The poles to be kept are These are all inside the unit circle |z| < 1 since J i ≷ 0 respective for chiral multiplets from 5d vector/hypermultiplet, and t < 1, T > 1. So this is agrees with our unit circle contour prescription. In fact the temporary relaxation e −ǫ + → t < 1 and e −ǫ + → T > 1 can be understood as pushing all poles with nonzero JK-Res inside the unit circles. So even for the higher rank case, this prescription is quite heuristic but we are not aware of a general proof that the two are equivalent. We shall just provide comparisons of the two rules with higher rank examples in sections 3. Although the final result is the same, the latter prescription picks more poles and residues in the intermediate stage compared to the JK-Res rule: the extra residues however all cancel out in pairs, as explained in sections 3.1 and 3.3.
We seek for such an alternative rule as this is known and widely used in the instanton calculus, starting from [1]. Comparisons of the two rules above in the next sections will thus rigorously justify the existing prescriptions from the JK-Res rules. When there are subtleties due to the poles at the infinities of the cylinders, the JK residue rule also justifies various vague parts in the existing prescriptions. We also note that temporarily substituting t < 1 and T > 1 for the vector/hypermultiplet poles is essentially the 'iǫ' and '−iǫ' prescriptions given to the vector and hypermultiplet poles, observed in [21]. We can rephrase our alternative prescription as picking all the poles/residues inside the unit circles from the 5d vector multiplets, and picking all of them outside the unit circles from the 5d hypermultiplets, assuming e −ǫ + ≪ 1.

Examples
Since 5d SYM is non-renormalizable, it is important to pick theories which are related to consistent quantum systems. In this section, we explain various classes of 5d SYM theories related to interesting QFTs. Some examples are explained in more detail in the subsections.
Among others, 5d SYMs and their partition functions could be useful as follows.
1. Compactifying the 5d theory on a small S 1 , we can study the effective action of 4d N = 2 SYM in their Coulomb phase from the instanton partition functions. The theories in this class should be asymptotically free or conformal in 4d.
2. Some 5d SYM theories are relevant deformations of 5d SCFTs. A necessary condition for such 5d Yang-Mills theories was studied in [19]. The condition is that the 1-loop correction to the coupling matrix is nonzero, with non-negative eigenvalues everywhere, in the Coulomb branch. Then one can take the bare coupling to infinite, which yields 5d SCFTs. Some of them admit string theory engineerings. One setting is the M-theory on singular Calabi-Yau 3-folds. When the Calabi-Yau is toric, this is dual to type IIB (p, q) 5-brane webs [42]. D4-D8-O8 systems also generate a class of 5d SCFT [17].
3. Some 5d SYM theories are obtained at low energy by circle compactification of 6d (2, 0) or (1, 0) superconformal field theories. A necessary condition for the 5d SYM in this class is to have vanishing 1-loop correction to the coupling matrix in the Coulomb branch.
The first class has various examples. The second class is classified in [19]. There are known examples in the third class, some of which we explain in detail below. We now explain these three classes above in some detail, as our general instanton counting results are applicable to all of them.
In 4d, the non-positivity of the 1-loop beta function for the gauge group G demands where i runs over hypermultiplets, C(R) is given by tr R (T a T b ) = C(R)δ ab . For instance, C(adj(G)) = c 2 (G) with c 2 (G) being the dual Coxeter number, which are given by c 2 (SU(N)) = N, c 2 (Sp(N)) = N +1, c 2 (SO(N)) = N −2. Also, C(fund) = 1 2 for fundamental representations of classical groups. When the gauge theory has the simple gauge group G = ABCD, the allowed matter contents are (we list the number of hypermultiplets in various representations): [19] engineers many 5d SCFTs from M-theory on singular Calabi-Yau 3-folds. One can also engineer some QFTs using type IIB 5-brane webs [42]. Sp(N) theories at n A = 1 and N f ≤ 7 fundamental hypermultiplets can be engineered by D4-D8-O8 systems [17].
Weakly coupled 5d SYMs for circle compactified 6d SCFTs should have zero 1-loop correction to the coupling constant in the Coulomb branch. The coupling constant thus remains constant throughout the moduli space. This 'microscopic' coupling is identified with the radius of the compactified circle of the 6d SCFT. 5d maximal SYM is the simplest example, related to the circle compactified 6d (2, 0) SCFT. Another example is the Sp(N) theory with 1 antisymmetric and 8 fundamental hypermultiplets, related to the circle compactified 6d SCFT for the M5-M9 system. These two examples will be briefly discussed in this paper, to explain how to compute their partition functions.
A tricky point of the instanton calculus from ADHM quantum mechanics is the continuum coming from extra string theory states. This can be measured by the asymptotic behaviors of the integrand g(φ, α, ǫ + , z), since V (ϕ) = − log g provides the effective potential to ϕ. Let us first explain the indices for 5d/6d SCFTs. We study when g does not vanish with large ϕ I . V (ϕ) should approach to a constant when there is a continuum, without attractive force. Since different eigenvalues ϕ I are for identical multi-instantons, we expect on physical grounds that it suffices to study the behaviors of one ϕ I going to ±∞, while other eigenvalues are fixed. Equivalently, it should be sufficient to investigate the asymptotic behaviors for the rank 1 case. We use the formulae for g for the U(N) or Sp(N) theories explained in sections 3.1, 3.2, 3.3, or use the formulae of [15] that we did not record in this paper for other theories. Among others, we obtain the following whose instanton partition functions see the continua: In all but the first case, the residue sum R 0 + R ∞ is zero. In the first case, R 0 = 0 and R ∞ = 0 when κ > 0, R 0 = 0 and R ∞ = 0 when κ < 0, and both R 0 = R ∞ are nonzero when κ = 0.
All but the third and fifth cases are in the 5d SCFTs classified by [19]. The interpretations of these continua are given in the following subsections. When the 5d SYM describes a circle compactified 6d SCFT, the string theory uplift of instantons can move away from the QFT, developing continua. An example is the D0-D4 system, for the circle compactified 6d (2, 0) SCFT. Another example is the D0-branes in the background of N D4-branes, 8 D8-branes, and an O8-plane, for the circle compactified 6d (1, 0) SCFT of the M5-M9 system. Since the D8-brane charges cancel, the type I' dilaton remains constant transverse to the 8-branes. This implies that D0-brane mass remains constant as it moves farther away from the D8-O8 (unlike the cases with N f ≤ 7 in which D0-branes are confined). They can escape the D4-D8-O8 system, developing a continuum.
For the 4d asymptotically free or conformal theories with simple gauge groups G, their 5d uplifts can sometimes develop poles at infinities. For instance, SO(N) theory is allowed with N f ≤ N − 2 fundamental hypermultiplets in 4d. Their 5d uplifts do not have poles at infinities when N f ≤ N − 5, exhibit simple/double/triple poles at N f = N − 4, N − 3, N − 2, respectively. In the last three potentially subtle cases, the sums of two residues at the both infinities of cylinders are always zero, due to the Weyl invariance of the Sp(k) eigenvalues of ϕ I inĜ. We have not made further studies on these 4d examples.

5d N = 1 * theories
We first discuss the SYM theory with one adjoint hypermultiplet, with arbitrary classical gauge group G = U(N), Sp(N), SO(N). The contour integral takes the form of Z vec is the 1-loop determiant for the quantum mechanical modes which come from the 5d gauge theory's vector multiplet. For brevity, we shall introduce the notation 2 sinh (a ± b) ≡ 2 sinh (a + b) · 2 sinh (a − b) and so on. For G = U(N) andĜ = U(k), it is given by 4) where R is symmetric/antisymmetric representation of O(k) + /Sp(k), respectively. The result for O(k) − is more complicated [10], which we review in section 3.3. Z adj is the 1-loop determiant for the quantum mechanical modes coming from the 5d theory's adjoint hypermultiplet. For G = U(N),Ĝ = U(k), it is given by and for other groups, where R is chosen in the same way for each group as in Z vec . Now we consider the contour integral. One can show that the sums of two residues at the infinities of cylinders are always zero, so η can be arbitrarily chosen without referring to ζ. Here we start from the well-known case with G = U(N),Ĝ = U(k) [1]. For k instantons, the covector space h * for charges is R k . We choose η = (1, 1, · · · , 1) = e 1 + e 2 + · · · + e k . Let us first explain all possible choices of k charges {Q i 1 , Q i 2 , · · · , Q i k } satisfying η ∈ Cone(Q i 1 , · · · , Q i k ). They determine Q(φ * ) for poles with nonzero JK-Res, both in non-degenerate cases (n = k) and in degenerate cases (n > k) where the k charges form a subset of Q(φ * ).
Possible Q i 's are {±e I } from the fundamental/anti-fundamental weights, and {e I −e J } from adjoint. With η having all positive components, we first note that −e I can never be chosen in the k charges which contain η in their cone. Using the Weyl invariance of U(k) which permutes k e I 's, it suffices to show that Q 1 = −e 1 = (−1, 0, · · · , 0) cannot be chosen. Suppose that we can. Then we should choose the remaining k − 1 charge vectors which satisfy with a 1 , a 2 , · · · , a k > 0. For this to be true, at least one of the k − 1 Q I 's should have positive first component, which we take to be Q 2 . Q 2 = e 1 is impossible, because then Q 1 , Q 2 are linearly dependent. Other choices are Q 2 = e 1 − e I for I = 1, which we take with I = 2 using Weyl symmetry. Then nonzero second component of a 1 Q 1 + a 2 Q 2 = (a 2 − a 1 , −a 2 ) requires that Q 3 = e 2 − e 3 up to Weyl reflection, and so on. This step repeats, until one finds (up to Weyl reflections) all the k vectors given by (Q 1 , Q 2 , · · · , Q k ) = (−e 1 , e 1 − e 2 , e 2 − e 3 , · · · , e k−1 − e k ), (3.8) for the first k − 1 components of (3.7) to be positive. Then one finds that the last component of (3.7) is −a k < 0, arriving at a contradiction.
So we choose k charges among {e I } and {e I − e J } only. Using the arguments similar to the previous paragraph based on positivity and linear independence, the allowed charges are given as follows. Firstly, there should be one or more charges chosen among {e I }, since the latter set {e I − e J } only generates k − 1 dimensional subspace of R k . Let us choose p(≤ k) of them, which we can take to be e 1 , e 2 , · · · , e p again up to Weyl reflections. For each chosen e I with 1 ≤ I ≤ p, the other charges can be divided into p groups, each group containing exactly one e I . As an example, let us pick the group containing e 1 and explain its structure, as other groups will be similar. First, charges of the type e 1 − e Ja should not be selected. Once we choose both e 1 and e 1 − e Ja , say J a = k, we require (a 1 + a 2 , 0, · · · , 0, −a 2 ) + k I=3 a I Q I = (1, 1, · · · , 1). Since the first component can always be a unity by adjusting a 1 , we can simply drop it so that the problem gets reduced to picking k − 2 additional charges in which −e k has been chosen. The previous paragraph showed that this is impossible. Second, there are charges of the form e Ja − e 1 with J a = 2, · · · , p. J a 's have to be different from 2, · · · , p, since otherwise there will be a linearly dependent combination of charge vectors. We can say that these make a tree graph, with a branch e Ja − e 1 attached to e 1 . Then, with e 1 and (possibly more than one) e Ja − e 1 chosen, one can further find Q i vectors which branch out from one of e Ja 's, taking the form of e K b − e Ja . K b are again different from all the subscripts which appeared so far (I = 1, 2, · · · , p, J a 's), to avoid linear relations among selected Q vectors. This procedure can be repeated, attaching adjoint charge vectors to e K b , and so on. This forms a tree graph originating from e 1 . The same tree graph can be formed starting from e 2 . It starts from e Lc − e 2 , with L c being different from all indices that appeared so far. In this way, we can make p possible trees with k charges. This tree structure will be further constrained below, by considering whether there actually exist poles which refer to these charges in Q(φ * ).
For instance, for k = 2 with η = (1, 1), the selected charge vectors are Now we consider the pole φ * whose Q * forms a tree that we just explained (non-degenerate), or contains it (degenerate). The poles φ * that actually arise from the integrand are labeled as follows, which we shall prove below by induction. It is the famous colored Young diagram rule [1]. For each element Q i in the chosen {Q 1 , Q 2 , · · · , Q n }, we assign a hyperplane equation which constrains φ * . When Q i is one of the fundamental weights, {e I }, then one should impose an equation of the form with i = 1, · · · , N. When Q i belongs to the type of e I − e J , one should impose one of the following equations, where the first two come from Z vec and the latter two come from Z adj . When n > k, n − k of them should be redundant for deciding φ * . So we first pick k independent hyperplane equations which we shall use to define φ * . Since we are interested in the poles with nonzero JK-Res, there should be at least one choice {Q 1 , Q 2 , · · · , Q k } in Q * which contains η in their cone. We work with k hyperplane equations picked in this way, whenever necessary.
The 'Young diagram rule' first states that there are no poles with nonzero JK-Res which refer to any of the hyperplane equation of the latter two types in (3.14) (containing m). Namely, [1] asserts that the poles coming from the 5d hypermultiplet measure Z adj can be completely ignored when classifying relevant JK-Res. Then [1] focuses on the hyperplanes (3.13) and the first two types of hyperplanes in (3.14), all coming from Z vec . The set of hyperplanes from the poles of Z vec with nonzero residues are classified by the N-colored Young diagrams with k boxes. A colored Young diagram consists of N Young diagrams Y = (Y 1 , · · · , Y N ) which satisfy |Y 1 | + · · · + |Y N | = k, where k i = |Y i | is the number of boxes of the Young diagram. Each box in the diagram (Y 1 , · · · , Y N ) corresponds to a hyperplane among (3.13) and the first two of (3.14). We explain how n i ≥ k i hyperplanes are chosen for a given Young diagram Y i . Firstly, assign to each of the k i boxes one of the k i variables φ I 1 , · · · , φ I k i . Let us say that φ I 1 maps to the box at the upper-left corner. The corresponding hyperplane is given by Then, consider all possible pairs of boxes one can form in Y i , by grouping horizontally attached boxes or vertically attached boxes. For a horizontal pair, with φ I 1 and φ I 2 mapping to the left and right box respectively, we assign the hyperplane For a vertical pair, with φ I 1 and φ I 2 mapping to the upper and lower box respectively, we assign the hyperplane One obtains at least k i independent hyperplanes this way. For instance, the diagram Y i = 1 2 3 defines k i = 3 hyperplanes while the diagram Y i = 1 2 3 4 with k i = 4 defines n i = 5 > k i hyperplanes In all hyperplane assignments, one can easily see that n i ≥ k i equations determine unique φ * and never over-determine it. Repeating the process for all N Young diagrams, one picks n = N i=1 n i ≥ k independent hyperplanes. By taking a look, one can convince oneself that the chosen Q(φ * ) is always projective. For instance, the 5 charges responsible for (3.19) are e 1 , e 21 , e 31 , e 43 , e 42 on R 4 . They are projective, since they are contained in the half-space x 4 + ǫ(x 2 + x 3 ) + ǫ 2 x 1 > 0 with small enough ǫ. The mapping of φ I variables to the k boxes of Y can be done in a unique way, by eating up the Weyl symmetry factor 1 k! .
To derive the above 'colored Young diagram rules,' we shall make the inductive argument. Firstly, we show that this is true at k = 1. At k = 1, there is no pole from Z adj so that ignoring all possible poles from Z adj is trivially true. Then, we only have to choose the pole value of single φ variable. By the JK residue rule, or equivalently the rank 1 residue choice rule of section 2.2, this is given by choosing one of the N equations φ I − α i + ǫ + = 0 for the pole. These choices correspond to N different colored Young diagrams with 1 box, confirming the rule at k = 1. Now assume that the 'Young diagram rule' is true at rank k − 1. To use induction, we pick the k independent hyperplane equations with η ∈ Cone(Q 1 , · · · , Q k ) in Q(φ * ). Here, recall the 'tree structure' of these k charge vectors. Apart from the case with k independent trees without any branches from e I 's, corresponding to the colored Young diagram in which each Y i contains only a single box, there are always one or more charge vectors of the form Q = e I − e J which are at the end of a tree (not having further branches attached to them). The hyperplane equation Q(φ) + · · · = 0 with such a Q is the only one which refers to φ I coordinate among the k hyperplane equations. Using the Weyl symmetry, we take φ k to be such a coordinate which appears only once in the k hyperplane equations. So we take the set of k hyperplanes to be (k − 1 hyperplanes referring to φ 1 , · · · , φ k−1 only) ∩ H k . (3.20) H k is the only hyperplane whose equation contains the φ k coordinate, so that the other k − 1 hyperplanes refer to φ 1 , · · · , φ k−1 only. The charges appearing in these k − 1 hyperplanes are those for the U(k − 1). Also, the integrand which contains φ 1 , · · · , φ k−1 but not φ k is the integrand for the k − 1 instantons in the U(N) theory. Finally, if {Q 1 , Q 2 , · · · , Q k } is the set of charges which contains k dimensional η (k) = (1, · · · , 1) in their cone, then {Q 1 , · · · , Q k−1 } contains η k−1 = (1, · · · , 1, 0) in R k−1 ⊂ R k . This is obvious from the fact that the charge Firstly, we explain that the hyperplanes φ k − φ I − ǫ + ± m = 0 yield zero residues. If φ I maps to the box at the upper-left corner of a Young diagram, then φ I = α j − ǫ + for some j. Then, the factor 2 sinh in the numerator of Z adj vanishes so that the pole does not exist. If φ I does not map to the box at the upper-left corner, then there should be a box with φ J which is left-adjacent or upper-adjacent to the box with φ I , namely φ J = φ I − ǫ 1,2 for either ǫ 1 or ǫ 2 . Here we note that there are factors in the numerator of (3.5). Inserting either of φ J = φ I − ǫ 1,2 , one finds that the factor ± 2 sinh φ kI − ǫ + ± m 2 (3.24) is always contained in the numerator, which vanishes due to the hyperplane condition φ kI −ǫ + ± m = 0 for one of the two signs. This shows that the corresponding poles do not exist. Next, we consider the first two types of hyperplanes in (3.22). The hyperplane of the first two sorts will correspond to adding a box to Y (k−1) when the box corresponding to φ I is at the 'border' of Y (k−1) . See Fig. 3 for what we mean by the boxes at the border of a Young diagram. One can show that the first two equations with I not at the border has zero residue, as follows. The box φ I not at the border always has a right-adjacent and lower-adjacent boxes, which we call φ J 1 , φ J 2 , respectively. These variables are determined by the hyperplane equations φ J 1 I + ǫ 1 = 0 and φ J 2 I + ǫ 2 = 0. So if φ I is not at the border of Y (k−1) , the factor sinh φ kJ 1 2 or sinh φ kJ 2 2 in the numerator of (3.3) vanishes, yielding zero residue. The remaining hyperplane conditions in (3.22) that are not ruled out are φ kI + ǫ 1,2 = 0 with φ I at the border. Now using the 'box' language, the box φ k may either attach to two boxes φ I , φ J of Y (k−1) like P I J k , attach to one box at the 'corners' of the Young diagram like I k or I k (see Fig. 3), or attach to one box φ I in the middle of the border of Y (k−1) like P I k and P I k . The first three are stacking the k'th box to form a colored Young diagram Y (k) , while the last two are not. In the last two cases, the factor 2 sinh φ kP +2ǫ + 2 in the numerator of Z vec vanishes so that the corresponding poles do not exist. In the first case, the factor partly cancels to keep a simple pole. The second and third cases also develop poles. So only the first three types of hyperplanes survive, exhausting all possible ways of putting the k'th box to Y (k−1) to make Y (k) . This finishes the inductive proof of the map between poles with nonzero JK-Res and colored Young diagrams.
Having identified the poles, one can compute the JK-Res at these poles. For this, one expands the integrand in the Laurent series of Q i (φ − φ * ), and the computation boils down to knowing various JK-Res(Q(φ * ), η) dφ 1 ∧···∧dφ k Q j 1 (φ−φ * )···Q j k (φ−φ * ) given by (2.46). In particular, all JK-Res for the poles labeled by the colored Young diagrams can be regarded as iterated contour integrals. Firstly, JK-Res factorizes into N groups, each group mapping to a Young diagram Y i . Within a given Young diagram Y i , the iterated integral goes in the reverse order of stacking the boxes. For instance, for the Young diagram 1 2 3 4 5 6 , the integral over the relevant pole terms is given by where for each φ I is done around a small counterclockwise circle surrounding the pole. Such iterated integral formula holds for all poles labeled by Young diagrams. This yields the following Integral over z 1 Integral over z 2 (|z 1 |, |z 2 |) Jeffrey-Kirwan expression for the U(N) instanton partition function [43,44]: Here, s runs over the boxes in the i'th Young diagram Y i . h i (s) is the distance from the box s to the edge of the right side of Y i that one reaches by moving to the right. v j (s) is the distance from s to the edge on the bottom side of Y j that one reaches by moving down [43,44,20]. Now, we discuss the alternative prescriptions for the U(N) contour integral as stated at the end of section 2.3. Namely, with the relaxation understood, e −ǫ + → t ≪ 1 and e −ǫ + → T ≫ 1, we take all the e φ I variables to live on the unit circles on the complex plane. Multiple unit circle integrals can be done in any order. In fact, this should be the original method used by [1,44] to get the result (3.27). For the purpose of illustrating how the alternative contour prescription works, we repeat it for U(N) index at k = 2.
We first integrate over z 1 = e φ 1 . We sum over all residues for poles in |z 1 | < 1 inside the unit circle, keeping z 2 fixed with |z 2 | = 1. Then we integrate over z 2 = e φ 2 , again picking all residues for poles in |z 2 | < 1. The rule excludes all the poles at the origin, z 1 = 0 or z 2 = 0. The possible poles in this procedure are shown on the first two columns of Table 1. At a given row, one first chooses an equation from the left column, which gives the poles for z 1 inside the unit contour. Then one moves on to the second column on the same row, which gives possible poles for z 2 inside its unit contour. The third column shows the values of |z 1 |, |z 2 | at the moment we are going to decide whether the pole is within the unit circle or not. For z 1 , it does not necessarily agree with its actual value after the pole for z 2 is selected, since we keep |z 2 | = 1 while integrating over z 1 . Table 1 contains only those selected by the z 1 unit contour rule. Some of them evidently stay inside the z 2 unit contour, while the four cases which are labeled by the superscripts a, · · · , d are rather ambiguous with the unit contour rule for z 2 . One finds that all the poles which are unambiguously inside the unit contour T 2 = S 1 × S 1 map to the poles which are chosen by the JK-Res rule. (Of course, we saw that some of the residues within this class can be zero, by using extra structures of the U(N) index.) As for the four ambiguous cases, whether they are inside or outside the unit contour for z 2 may depend on the scales of other fugacities which we did not specify. But one can notice that there always exists a pair of poles at an ambiguous location of z 2 . a, c and b, d are such pairs. So the paired poles are either simultaneously inside or outside the unit contour of z 2 . When they are outside the z 2 unit contour, they provide no contribution so that the result is consistent with the JK-Res rule. When they are both inside the z 2 unit contour, the two residues cancel. The sum of two paired residues is actually a result of doing the contour integral of the form: The integral is given by The two terms −f (a, a − b) and f (a, a − b) are precisely the pair of residues in all the above cases, guaranteeing cancelation when they are in the unit contour. This illustrates that the unit contour rule with e −ǫ + → t ≪ 1 and e −ǫ + → T ≫ 1 yields the same result as the JK-Res rule: although the unit contour rule may appear to keep more residues, after pairwise cancelations the two rules become equivalent. We confirmed that similar pairwise cancelations happen for the Sp(N) indices at k = 4, as summarized in section 3.3. In some other cases, such as the Sp(1) index at k = 5 in section 4.1, we just used the iterated integral rule along unit contour without checking its equivalence with the Jeffrey-Kirwan rule. Possibly, one could be able to prove the equivalence in full generality.
We emphasize here that the above type of pole classification goes through for U(N) instanton partition functions with other matters. For fundamental hypermultiplets, there are no extra poles incurred by the hypermultiplets. Then the above arguments can be reused, simply ignoring all the discussions involving the hyperplanes from Z adj . We also checked that the bi-fundamental hypermultiplets in the U(n) quiver theories do not provide any poles with nonzero JK-Res at two instanton order, as derived in [22]. The absence of poles coming from the hypermultiplet factor Z adj is an accidental property of the U(N) theory. This simplification does not happened for the N = 1 * theory with other gauge groups. One should just use the Jeffrey-Kirwan residue rule, or alternatively use the unit contour integration rule after suitably replacing e −ǫ + by t ≪ 1 and T ≫ 1. We leave the studies on these indices to the future.

U (N ) theories with matters and Chern-Simons term
In this section, we consider the instanton partition function of 5d U(N) SYM, with N f fundamental matters and nonzero Chern-Simons term at level κ. We shall only consider the theories which are related to the 5d SCFTs at the UV fixed points. The contour integral has the same structure as what we explained in the previous section, picking up poles and residues labeled by the colored Young diagrams. However, there occasionally arise subtleties in this class of theories. The residue sums at the two ends of cylinders will not be zero when N f + 2|κ| = 2N. The nonzero residues at the infinity regions of ϕ I imply a continuum in the ADHM quantum mechanics. The nonzero sum of two residues at the infinites of a cylinder implies a wall crossing as the FI parameter changes. These can be naturally understood with the string theory realizations of these 5d SYMs and the UV SCFTs. Before proceeding, we emphasize that most of the studies in this section and section 3.4.4 are already done in [28,27,45,46,47,48]. Mostly, we just reproduce their results, sometimes filling the missing values of N, N f , κ not checked by them, to illustrate the (absence of unphysical) wall crossing issue.
In Fig. 4, various (p, q) 5-brane webs are shown which engineers the U(2) gauge theory with fundamental hypermultiplets and/or bare Chern-Simons term κ is integral when the number N f of fundamental hypermultiplets is even, and is half an odd integer when N f is odd. The overall U(1) of the U(2) is non-dynamical in QFT. Note that there cannot be a bare Chern-Simons term for the SU(2) theory. Thus the bare U(2) Chern-Simons term means the mixed Chern-Simons term for the U(1)-SU(2)-SU(2), inducing the background U(1) electric charge to the SU(2) instantons. The two horizontal lines are D5-branes on which the U(2) theory lives. The overall U(1) has infinite inertia, as the overall displacement of the two D5-branes induce translations of the asymptotic branes. When N f + 2|κ| = 2N, one finds horizontal D1-branes stretched between the two parallel vertical lines (NS5-branes). These D1-branes, shown by the red lines in Fig. 4(c),(d), can escape up/down from the D5-branes on which the 5d QFT is defined. This implies that the ADHM quantum mechanics for the D1-D5 system (UV completing the instanton mechanics) has a continuum in the Coulomb branch. In the contour integrand for the instanton index, this continuum causes a nonzero pole at one or two ends of the cylinder. The case with N = 3, κ = 3, N f = 0 is shown in Fig. 5. So in these examples, the interpretation of the poles at infinities is the continuum developed by the stringy D1-brane states which appear in the ADHM mechanics, which are not in the QFT spectrum. These were studied well in [28,27,45,46,47,48].
The k instanton partition function, which could possibly include the extra decoupled Z string factor when N f + 2|κ| = 2N, is given by the following contour integral (see, e.g. [10]) where Z vec takes the same form as (3.3), and The overall sign (−1) κ+N f /2 was found in [45,46] to be the physically sensible one, from various indirect evidences. [45,46] conjectured that it will have to do with the effect of 5d Chern-Simons term, but its microscopic derivation seems to be unavailable yet. The pole selection derived from the JK-Res rule is exactly the same as what we derived for the N = 1 * theory in the previous subsection, labeled by the colored Young diagram. In the previous subsection, we chose η = (1, · · · , 1). Here, we note in foresight that the index may depend on ζ. The choice of η in section 3.1 is for ζ < 0. For the theories in this subsection, the result is given by (3.34) E ij is defined by (3.28), and φ(s) is given by where s = (m, n) ∈ Y i with m, n being the vertical and horizontal positions of the box s from the upper-left corner of Y i [10]. When ζ > 0, one would have to choose η = −(1, · · · , 1) and use the JK-Res rule. It is easy to get the result for ζ > 0. Since ζ → −ζ can be undone by the SU(2) r Weyl reflection, or the Weyl reflection of the diagonal of SU(2) r × SU(2) R , the sign flip of ζ is equivalent to that of ǫ + . So by flipping all signs of ǫ + in the above result, we obtain the index for ζ > 0. The two results will be the same unless N f + 2|κ| = 2N.
At N f + 2|κ| = 2N, Z k QM factorizes into Z QFT Z string with nontrivial Z string , and furthermore Z string exhibits a wall crossing as ζ flips sign. Nontrivial Z string was analyzed and factored out from Z QM in [28,27,45,46,47,48]. We will explain/review these indices and their ζ dependence in section 3.4.

Sp(N ) theories
In this subsection, we study the instanton partition function for the Sp(N) gauge theories with N f fundamental and n A = 0, 1 antisymmetric hypermultiplets.
Let us first write down the contour integral expression. The integral variables are the zero modes of the ADHM quantum mechanics for the Sp(N) instantons. Part of the zero modes is the holonomy ofĜ on the temporal circle. For k instantons, they come withĜ = O(k) gauge group. Since O(k) has two components O(k) + and O(k) − , one should also turn on discrete holonomies for e iAτ . These can all be labeled by the complexified group element U = e φ = e ϕ+iAτ , which can be taken as [10] U + = e φ + = diag(e iσ 2 φ 1 , · · · , e iσ 2 φn ) for even k = 2n diag(e iσ 2 φ 1 , · · · , e iσ 2 φn , 1) for odd k = 2n+1 (3.36) for O(k) + , and for O(k) − . The above expressions with imaginary φ I are the O(k) ± group elements, while their complexifications come with ϕ I = Re(φ I ). Below we shall write k = 2n + χ, with χ = 0, 1. We will get two intermediate indices Z k ± from the path integral. Each of them is obtained by taking the complexified holonomy in either U ± , performing Gaussian integration over non-zero modes, and then exactly summing or integrating over U ± (with contours explained in section 2.3). The final index is given by [10] There is a variation of this result due to nontrivial π 4 (Sp(N)) = Z 2 , which sometimes defines new 5d SCFTs. With nontrivial Z 2 element, one would have to take [45] This will be discussed more in section 3.4.4. Z k ± are given by The Weyl factors |W | for O(k) ± are |W | χ=0 The integrands are given as follows: from the ADHM data of 5d vector multiplet with O(k) + , with O(k) − when k = 2n + 1;  for O(k) − when k = 2n. When one considers the index with n A = 0, of course Z ± anti factors are dropped from the integrand.
In all of the above integrands, the arguments are written in the form of sinh Q(φ)+··· 2 , where Q is the weight of the chiral or Fermi multiplet responsible for this factor. The contour integral is understood as the sum of Jeffrey-Kirwan residues with a chosen η. Here, any choice of η will provide the same result. We checked the behavior of poles carefully for the Sp(1) theory with one antisymmetric hypermultiplet, up to k = 4 instanton order. The case with k = 1 has no integral. The case with k = 2 either has rank 1 for O(2) + , where the formulae of section 2.2 applies, or has no integral for O(2) − . The case with k = 3 again has at most 1 integral. The case with k = 4 has rank 2 for O(4) + . In 2 dimensional h * , we take η in the shaded chamber in Fig. 6. One can show that all hyperplane arrangements are projective, fulfilling the condition for the results of [24] to be applicable. To see this, it suffices to check all the degenerate hyperplane arrangements. We find that they are either e 1 e 2 Figure 6: The charges for the Sp(1) index at k = 4. The charges ±2e 1 , ±2e 2 are not shown. We chose η in the shaded chamber.
with ± signs correlated, or others obtained by making Weyl reflections on all the charges Q i appearing in the hyperplane equations. These are obviously projective.
In the analysis of sections 3 and 4, we used the iterated integrals over z I = e φ I with e −ǫ + → t ≪ 1, T ≫ 1 replacements. We have checked the equivalence of the two rules for Sp(1) theory till k = 4, similar to what we explained for U(N) k = 2 in section 3.2. With η chosen in the shaded chamber shown in Fig. 6, we integrate over z 1 = e φ 1 first and then over z 2 = e φ 2 . From the integrals over unit circles, we encounter 372 = 292 + 80 possible poles. 292 poles are unambiguously inside the unit circle, and are those kept from the Jeffrey-Kirwan rule. The 80 extra poles are ambiguous but show pairwise cancelations, as explained around (3.30), proving the equivalence. In Z 1-loop , some poles are actually absent because sinh factors in the numerators vanish at the poles. (Similar phenomena were repeatedly observed for the U(N) case, while deriving the Young diagram rules.) Taking these into account, we have 324 nonzero residues from our unit circle integrations, and 260 nonzero Jeffrey-Kirwan residues: 64 extra residues from the former cancel pairwise. Finally, identifying t and T at the final stage, 188 nonzero poles remain. Similar structures are found for Sp(N) at O(4) + , although there are more poles.

Extra string theory states and continua
In this subsection, we explain in various examples how one can factor out Z string from the index of ADHM quantum mechanics, and obtain Z QFT = Z QM Z string of our interest. The examples that we shall mainly discuss are Sp(N) gauge theories with 0 ≤ N f ≤ 8 fundamental and n A = 1 antisymmetric hypermultiplets, and U(N) gauge theories with N f fundamental hypermultiplets and 5d Chern-Simons level κ satisfying N f + 2|κ| ≤ 2N.

Sp(N) theories for 5d SCFTs
We first discuss the Sp(N) theories with N f ≤ 7 fundamental and 1 antisymmetric hypermultiplets. The case with N f = 8 fundamental hypermultiplets is discussed in the next subsection separately. The ADHM quantum mechanics describes the k D0-branes along 0 direction, N D4-branes along 01234 directions, N f D8-branes and one O8-plane along 0 · · · 8 directions. The scalars ϕ I from the ADHM vector multiplet represent D0-branes' positions along the 9 direction, transverse to all D-branes. The general analysis at the beginning of this section says that there is no pole at infinities of ϕ I . One can expect this, since N f ≤ 7 D8-branes do not completely cancel the 8-brane charge of the O8-plane, so that the dilaton runs along the 9 direction. D0-brane's mass increases linearly in ϕ I , explaining the absence of the continuum for ϕ I . However, there is an extra contribution Z string from D0-branes which are unbound to D4-branes, but are bound to D8-O8 only. Since the motion of D0's along the worldvolume of D8-O8 is fully gapped by the chemical potentials ǫ 1 , ǫ 2 , m, one could compute the multi-particle index for the D0-particles in 8 + 1 dimensions. These D0-D8-O8 bound states' index will never refer to the electric charge fugacities α i on D4. So to detect the possible Z string factor, it suffices to examine the expansion of Z QM in the Coulomb VEV e −α i with α 1 > α 2 > · · · > 0, and study the sector which carries zero electric charges. The index can be written as Z QM (α, ǫ 1,2 , v, q) = Z (0) (ǫ 1,2 , v, q)Z (1) (α, ǫ 1,2 , v, q) . (3.53) v = e −m is the flavor fugacity rotating the antisymmetric hypermultiplet. Z (1) is given by One can write where t = e −ǫ + , u = e −ǫ − , v = e −m , y i = e m i /2 with i = 1, · · · , N f . f N f is the single particle index. One finds .
2 N f −1 is the chiral spinor representation of SO(2N f ) (in our convention), whose highest weight state contributes y 1 y 2 · · · y N f to the character. We have checked these forms of f N f up to q 4 order from the Sp(1) index with all fugacities kept, and the same result up to q 3 from the Sp (2) index. In section 3.4.3, we shall derive these indices from the D0-D8-O8 system, which proves that Z (0) In the remaining part of this subsection, we show that this Z (0) is precisely what one expects from the type I' string theory with N f D8-branes, which should exhibit E N f +1 gauge symmetry on the 8-branes' worldvolume (from its duality to heterotic strings [17]). To see this, one has to combine Z (0) with the contribution to the index from perturbative type I' string theory. N f D8-branes and an O8-plane host massless degrees given by the 9d SYM theory with SO(2N f ) gauge group. Nonperturbative enhancement SO(2N f ) → E N f +1 is expected from string duality, where E N f +1 includes the D0-brane charge in its Cartan [17]. So the nonperturbative index of the type I' theory should be that of the 9d E N f +1 SYM theory.
Let us explain the perturbative index first. The index of the 9d SO(2N f ) SYM is defined referring to the same 2 supercharges that we used to define our instanton index. The 16 supercharges preserved by the D8-O8 system can be decomposed according to their representations of SO(4) × SO(4) = SU(2) l × SU(2) r × SU(2) R × SU(2) F symmetry. The first SO(4) = SU(2) l × SU(2) r is the spatial rotation on the common worldvolume of D4-D8-O8. Second SO(4) = SU(2) R × SU(2) F is the rotation on D8-O8 worldvolume transverse to D4. SU(2) R was the R-symmetry of 5d N = 1 theory. SU(2) F with the chemical potential m rotates the Sp(N) antisymmetric hypermultiplet. Denoting by a = 1, 2 the doublet index for SU(2) F , the 16 supercharges can be written as where α,α, A indices are for SU(2) l , SU(2) r , SU(2) R doublets as before, and each SU(2)×SU(2) doublet satisfies symplectic-Majorana condition. 9d SYM has half-BPS W-bosons and their superpartners in their BPS spectrum, in the Coulomb branch where one real scalar is given nonzero VEV. The SO(2N f ) electric charges have fugacities y i ≡ e m i /2 , which were introduced in 5d SYM as flavor fugacities. Let us write the 32 × 32 gamma matrix in 10d as (Γ 0 , Γ 9 ) = 1 8 ⊗ (σ 2 , σ 1 ), Γ i = γ i ⊗ σ 3 , with γ i given by the SO(8) gamma matrices (i = 1, · · · , 8). The BPS condition for the half-BPS W-boson is one of Γ 09 ǫ = ±iǫ in the 10d chiral Majorana spinor notation, where 9 stands for the scalar direction. The SUSY parameter ǫ satisfies the 10d chirality condition γ 1···8 ⊗ σ 3 ǫ = ǫ. W-bosons' BPS condition says that ǫ is either chiral or anti-chiral SO(8) spinors. In our notation, the preserved supercharges are either Q a α ,Q Ȧ α or Q A α ,Q ȧ α . Since our indices are always defined usingQ A=1 α=1 andQ A=2 α=2 , the sector which is captured by our index contains W-bosons preserving the former. The broken supercharges Q A α ,Q ȧ α provide Goldstone fermion zero modes, which contribute to the single particle index of 9d W-bosons. The 4 pairs of fermionic oscillator from these Goldstinos provide a factor 2 sinh where to the index. One should also consider χ SO(2N f ) adj (y i ) + factor for the the W-bosons, where the + superscript denotes that only the positive roots contribute to this character. This is because we are counting only W-bosons and their superpartners in the Coulomb branch of the 9d theory, without anti-W-bosons or the massless Cartans. So one obtains (3.61) Note that the four factors in the denominator can be understood as the four complex zero modes on C 4 = R 8 , indicating that this is coming from 8 dimensional particles.
Combining (3.61) and (3.56) together, we now show that one obtains the single particle index for the W-bosons of 9d E N f +1 SYM. One first finds that at N f = 0, E 1 = SU(2) adjoint decomposes into 3 states which have U(1) I instanton charges 0, +1, −1, respectively. The latter two are the non-perturbative enhanced symmetry generators. Adjoint representation of E 2 = SU(2) × U(1), which is 3 + 1 in SU(2), decomposes in SO(2) × U(1) I to two neutral generators, and two non-perturbative generators carrying q ±1 y ±1 1 . E 3 = SU(3)×SU(2) contains the perturbative SO(4) × U(1) I = SU(2) × SU(2) × U(1) I in the following way. The second SU(2) of SO(4) is the same as the SU(2) factor of E 3 , while SU(3) adjoint branches to the remaining SU(2) × U(1) I irreps as (3.62) The branching rules of the E N f +1 adjoints, with N f ≥ 4, to SO(2N f ) × U(1) I irreps are The subscripts all denote the U(1) I instanton number. The first 1 0 's all denote the generator of the U(1) I , while the next U(1) I singlets are all adjoints of SO(2N f ). As explained around (3.61), only the positive roots from the SO(2N f ) adjoints contribute to the index. Among the remaining non-singlets on the right hand side, only the states which have positive U(1) I charge will contribute to the index, as our index counts instantons but not anti-instantons.
The instanton contribution to the 9d E N f +1 SYM index required from (3.63) and the preceding branching rules indeed appear in (3.56) for all N f . So the addition of (3.61) and (3.56) precisely captures the contribution from the W-bosons of 9d E N f +1 SYM.
So we conclude that Z (0) with f N f given by (3.56) is precisely Z string , the extra string theory or UV contribution to Z QM . This will be reconfirmed in section 3.4.3 by a direct computation of the D0-D8-O8 index, without assuming type I'-heterotic duality. The index for the 5d SCFT is given by Z QFT = Z QM Z string , which shall be used in section 4.
Before closing this subsection, we discuss the Sp(N) partition function with N f fundamental hypermultiplets at n A = 0. This engineers another class of 5d SCFTs, which can be realized by M-theory on suitable CY 3 [19]. For Sp(1), this should yield the same 5d SCFT indices as those obtained from the quantum mechanics with n A = 1. The only issue is that the two descriptions may have different Z string factors. At all N, including N = 1, the condition for the contour integrand Z 1-loop to vanish at |ϕ| → ∞ is N f < 2N + 4. Z 1-loop approaches a constant asymptotically for N f = 2N + 4. So we study the Sp(1) ADHM instanton calculus at N f ≤ 6. In the Calabi-Yau engineering, instantons are realized as M2-branes wrapping certain 2-cycles. When Z 1-loop does not vanish at the infinity of ϕ, the BPS 2-cycle for these M2brane worldvolume has a noncompact modulus so that this M2-brane can continuously move to infinity. When N f ≤ 5, there is no noncompact moduli so that Z string = 1. This is supported by the analysis of [10]. When N f = 6, we find Z QM = Z QFT Z string , where Z QFT is the SCFT partition function that we derived with n A = 1, and This fact was confirmed up to q 4 order. Since (3.64) comes with a fractional coefficient, it clearly has to do with the continuum. It may be possible to explain the t, u, q dependence by understanding the CY 3 geometry of [19], which we do not attempt in this paper.
In all cases with N f ≤ 6, we confirmed that Z QFT computed from the ADHM mechanics with n A = 0 and n A = 1 are the same, up to q 4 order.

Sp(1) theory for 6d SCFT on M5-M9
Now we turn to the case with N f = 8, for D0-branes probing N D4, 8 D8's and an O8. Again the t 9 direction is a half-line R + . The difference from the cases with N f ≤ 7 is that the D8-brane charges completely cancel between 8 D8's and one O8. The dilaton asymptotically becomes a constant as one moves away from the brane system along t 9 . So this system uplifts to M-theory on R 8,1 × R + × S 1 at strong coupling. Our 5d SYM is thus a low energy description of circle compactified 6d (1, 0) theory for the M5-M9 system. In this case, there are poles at infinities of cylinders in Z 1-loop , since D0's can move away from the 8-branes with a continuum. Following the same strategy as the cases with N f ≤ 7, we first extract out Z (0) as this should contain all possible Z string factors. Again writing Z (0) = P E[f ], f is given by where we checked the q dependence up to 4-instanton order from the Sp(1) theory. Namely, the above expression is obtained with q 2 1−q 2 → q 2 + q 4 and q 1−q 2 → q + q 3 . So all properties that we show below are proven up to this order. 120 and 128 are the adjoint and chiral spinor representations of SO (16). We now explain the terms in (3.65) which should go to Z string .
We first study the second line of (3.65). This provides a single particle index for certain 8+1 dimensional particles, thus should go to the factorized Z string from bulk degrees. Let us first explain what we expect from the string dualities and heterotic M-theory. Heterotic M-theory was proposed in [49] as a strong coupling limit of E 8 × E 8 heterotic string theory. It has a low energy limit described by 11d supergravity on R 9,1 × I, where I = S 1 /Z 2 is an interval. There are two fixed planes of the Z 2 action at both ends of I, which we call the M9-planes. Each M9-plane hosts E 8 gauge symmetry, having a massless sector of 10d E 8 super-Yang-Mills theory. One can compactify the heterotic M-theory on a small circle with radius R to R 8,1 × I. The circle compactification can be made with nonzero E 8 × E 8 Wilson lines on two 10d SYM theories on M9-planes. In particular, consider the following Wilson line The holonomy (3.66) is such that e 2πiRA leaves 120 invariant, while giving −1 sign to the spinors. So the compactification with this holonomy yields a 10d theory with SO(16) × SO (16) symmetry. This is the type I' string theory on R 8,1 × I, which has two orientifold 8-planes (O8-planes) at the two ends of I. Each O8-plane has 8 D8-branes on top of it. A crucial part of this identification is that the nonperturbative D0-brane physics of type I' theory should enable us to see the 11th circle's KK modes. We will show that the second line of (3.65) achieves it.
Note that the fugacities q, y i in (3.65), especially on the second line, probe the momentum and SO(16) charges in the background of Wilson line (3.66). Here, note that the charges of the type I' theory and the heterotic M-theory are related by [50] where k is the type I' instanton charge, P is the circle momentum of heterotic M-theory, F E 8 are the E 8 charges, A E 8 is the holonomy (3.66). The expression in [50] has more shifts to k on the right hand side, depending on the string winding number, which is zero for all states captured by Z (0) . The fugacities conjugate to k, F 8 are more naturally viewed in the heterotic M-theory as If one replaces all y 8 's in Z (0) by y 8 q, this effectively turns off the background holonomy (3.66). Namely, q 2 and y i should be regarded as the fugacities conjugate to P and E 8 charges after the replacement. Note that this replacement y 8 → y 8 q makes a rearrangement of the instanton series expansion, as the original expansion is made with q ≪ y i , y −1 i . After the replacement, one temporarily decomposes the SO(16) characters into SO (14) characters. The characters appearing on the second line of (3.65) are decomposed as χ SO(16) 120 after the replacement y 8 → y 8 q. The second line of (3.65) thus rearranges as . The first term in the square bracket is exactly what we expect from the heterotic M-theory with nonzero momentum P , since there should be contributions from the 10d E 8 SYM with P = 1, 2, 3, · · · . Note also that the prefactor is also the index for the vector multiplet in the R 8 part, as computed around (3.61). Therefore, the replacement y 8 → y 8 q makes SO(16) → E 8 enhancement visible by turning off the background Wilson line.
It only remains to understand the last three terms in the square bracket. These can be understood again by combining it with the perturbative 9d SO(16) SYM of type I' theory. Depending on the Coulomb VEV conjugate to F 8 , the W-boson and anti-W-boson indices would be one of . (3.72) After the replacement y 8 → y 8 q in f ± pert , the second term of f + pert becomes . (3.73) This precisely cancels with the last term of (3.71). The second and third terms of (3.71) combine with χ ± 91 in (3.72), to provide the positive/negative roots of E 8 at O(q 0 ). (Other roots at q 0 are provided by shifts from anti-instanton sector.) Thus, the KK tower of 10d E 8 SYM with k > 0 is completely reproduced by the second line of (3.65), with the last term of (3.71) provided by the W-bosons of 9d SO(16) SYM. This states that the second line of (3.65) should go to Z string . Then in (3.65), we consider the term on the first line. The overall coefficient 1 2 shows that this is clearly the continuum contribution. In fact, there is no way to turn on the FI term with O(k) gauge group, so that we cannot decouple the continuum from the Witten index calculus. Although we do not have an account for the factor 1 2 , in a way similar to [51,52], we can surely show that all the dependence on the fugacities is that for the continuum states in our problem.
To show this, we investigate the 11d supergravity spectrum on R 8,1 × S 1 × R + . The continuum is formed by the states which propagate along R + . The R 8 × S 1 part of the space has a fully gapped spectrum, either by having compact space or by having nonzero chemical potentials for the rotations. So the gapped part of the spectrum can be computed by investigating the supergravity multiplet, setting aside an overall fractional coefficient which can only be determined by knowing the dynamics along R + (and our deformations in the index computation). The factor q 2 1−q 2 simply shows that the KK fields of the 11d gravity on circle have all same spin contents in 10d. So the t, u, v dependence of this term can be computed from the 10d type I' supergravity. Also, since we are only paying attention to the R 8 part of the spectrum, we can replace R + by I = S 2 /Z 2 and apply T-duality along this direction, after which the well known type I supergravity spectrum will be relevant. The type I supergravity contains a dilaton φ, RR 2-form C 2 , graviton g µν , dilatino λ, and the gravitino ψ µ . All of them are in the following representation of SO(8), rotating R 8 : (3.75) The SU(2) 4 characters of 8 v , 8 s , 8 c on the right hand sides (with (−1) F signs for 8 s , 8 c ) are From this, one can compute the index for the right hand side of (3.75). Note that the symmetrized and anti-symmetrized characters are given by , where f is the character of 8 v,c appearing in the (anti)symmetrization. Multiplying this with the factor t 4 (1−tu) 2 (1−t/u) 2 (1−tv) 2 (1−t/v) 2 which comes from the translation zero modes on R 8 , one obtains So this proves that (3.74) is the continuum contribution from 11d supergravity KK modes.
Having explained the second line of (3.65) and the t, u, v, q dependence of (3.74), there is no ambiguity in the separation of the 4d index and the higher dimensional index in Z (0) . So we have shown Z string = P E[f string ] with is the contribution from the string theory or UV sector. The first term of (3.65), is not included in Z string . It is part of the 6d QFT spectrum.
So far, we relied on the heterotic M-theory physics to factor out Z string from the ADHM quantum mechanics index Z QM . Again, we can directly compute Z string from the index of D0-D8-O8 quantum mechanics, without relying on unproved properties. See section 3.4.3.
So we provided a clear recipe to compute the index of the circle compactified 6d (1, 0) SCFT on the M5-M9 system, Z QFT = Z QM Z string , with replacement y 8 → y 8 q, etc. It will be very interesting to study its physics from this observable.

Direct computations of the D0-D8-O8 indices
The computations reported in this short section supplement the discussions of sections 3.4.1 and 3.4.2. There we extracted out the neutral part Z (0) of the D0-D4-D8-O8 index and argued that this contains Z string which is deducible from string dualities, etc. Instead, we can simply derive the Z string factors of the previous subsections directly from the D0-D8-O8 quantum mechanics. One can start from the gauged quantum mechanics for the open strings connecting D0-D8-O8 with O(k) gauge group. The field contents can be easily obtained from the previous D0-D4-D8-O8 fields by dropping all N × k bi-fundamental fields. The index is also obvious: one just uses the index in section 3.3 after dropping all the determinant factors for the fields charged in Sp(N). (In fact, the expansion in e −α i that we discussed in the last subsections is almost the same as doing this.) So we compute these indices, in all examples up to q 4 order, and obtain and These all directly justify the Z string factors that we argued using string dualities. In particular, (3.80) supports the non-perturbative duality between the type I' and heterotic strings by finding a spectrum which allows E N f +1 enhancement. (3.81) supports that non-perturbative physics of type I' strings reconstructs the physics of M9-plane compactified on a circle.

U(N) theories for 5d SCFTs
Our last example is the U(N) SYM with N f fundamental hypermultiplets and bare Chern-Simons term at level κ, satisfying N f + 2|κ| ≤ 2N. The indices for the theories saturating the last inequality have Z string contributions. These partition functions are studied in great detail in [28,27,45,46,47,48].
We first discuss the theories with U(2) gauge group, with N f ≤ 4 fundamental matters and CS level κ satisfying N f + 2|κ| ≤ 2N. The 5-brane webs engineering some of these theories are shown in Fig. 4. The SU(2) part of the U(2) gauge group is identified with the Sp(1) gauge group, while the overall U(1) is non-dynamical. The information on the overall U(1), especially the CS level κ, should be irrelevant for Z QFT , since the QFT is just the Sp(1) theory coupled to N f fundamental hypermultiplets. So we expect QFT is the QFT index that one obtains by dividing Z string . (We suppressed the α i , ǫ 1,2 , m, y i , ζ dependence.) At N f + 2|κ| < 2N, there is no continuum from the string theory which are attached to the instanton quantum mechanics, and the right hand side is 1. At N f + 2|κ| = 2N, the right hand side is not 1 and further experiences a wall crossing as the FI parameter ζ changes.
Before explaining the results, one should realize that the 5d Sp(N) theories can be classified into two [53], labeled by two discrete theta angles. Namely, there are two topologically distinct configurations due to π 4 (Sp(N)) = Z 2 . This also descends to the two topologically distinct configurations in the O(k) ADHM quantum mechanics, due to π 0 (O(k)) = Z 2 [45]. In both 5d/1d cases, the sector with nontrivial element of Z 2 has a relative −1 sign in the path integral. So the instanton calculus rule Z k θ=0 = The overall factor of (−1) k was argued in [45] at k = 1, 2 in a somewhat indirect way. At N f = 0, the two cases with θ = 0, π were shown (based on the instanton partition function calculus) in [45] to uplift to the so-called E 1 andẼ 1 theories, respectively [18]. With N f ≥ 1, the relative minus signs from Z 2 nontrivial sector can be canceled by flipping the sign of a mass parameter. In the following, we stick to our previous definition of m i parameters, which implies that we should insert the relative minus sign for the Z k − when we explicitly write θ = π. But this is related to new SCFT only when N f = 0. In other cases, inserting extra minus sign is simply changing our convention for m i . [45] finds that Z To make the comparison between the U(2) and Sp(1) observables, we identify α 1 + α 2 = 0 in the U(2) result below.
One first finds [45,46] Z 2 ) is even/odd, respectively. We checked this fact for N = 2, and all possible N f , κ satisfying N f + 2|κ| < 2N up to q 3 order. Although this was already analyzed in [45,46], we checked it for our own sake. In proving this, it is crucial to insert the factor (−1) k(κ+N f /2) in the k instanton index of the U(2) theory, as explained in [45,46]. Secondly, one finds (1) QFT (N f = 2N − 2|κ|, e iθ = ±1) = Z string (ζ) (3.85) when the 5d SCFT bound N f + 2|κ| ≤ 2N is saturated. The theta angle is chosen between e iθ = ±1 depending on whether N −(κ+ N f 2 ) is even or odd. Namely, when κ ≥ 0 and saturates 5d SCFT bound κ = N − N f 2 , one takes e iθ = +1. On the other hand, when κ < 0, one takes e iθ = (−1) N f . Note here that whenever N f = 0, the choice of θ is purely a convention on the masses m i . At N f = 0 when θ acquires meaning, we find e iθ = 1 for both κ = ±2. 7 The extra Z string factor is naturally expected, since there always exist D1-branes which can be separated from the QFT system in this case, as explained in section 3.2. This is given by for κ > 0, for κ < 0, and for κ = 0 and N f = 2N. Here we defined y i ≡ e m i /2 . We have checked these results for N = 2 and (N f , κ) = (0, ±2), (1, ± 3 2 ), (2, ±1), (3, ± 1 2 ), (4, 0) up to q 3 order. These results are known from [27,28,45,46,47,48]. In particular, [27] explains that it is consistent with the structure of the index for M2-branes wrapping 2-cycles in CY 3 which can escape from the QFT. Note that Z U (N ) QM at N f + 2|κ| = 2N lacks the ǫ + → −ǫ + (or t → t −1 ) invariance, which is inconsistent either as a half-BPS index of 5d SYM (as we explained in section 2.2) or the index of 5d SCFT with SU(2) R symmetry [45]. This asymmetry all goes to Z string , leaving Z U (2) (1) QFT invariant under the sign flip of ǫ + . The bulk contribution is not invariant under ǫ + → −ǫ + . We are not aware of the half-BPS state interpretation of this part of the index: i.e. the index is well-defined only at ζ = 0, and we only have a continuum at ζ = 0 with unbroken SU(2) R symmetry. So there appears to be no contradiction. Also, the bulk spectrum is not constrained by the 5d superconformal symmetry, so ǫ + → −ǫ + asymmetry is fine.
Note that the ratios are always given by 89) where w i ≡ e α i , and we have listed the results without taking w 1 w 2 = 1. R 0 , R ∞ are the residues of the holomorphic measure for the rank 1 integrand. This is consistent with what we found for the rank 1 case in section 2.2. At w 1 w 2 = 1, it just reduces to the ratio of two Z string factors at ζ ≶ 0 that we found by comparing Z QFT Z string . However, from the 5-brane web diagram, we could naturally expect that the structure of D1-branes escaping from the QFT is exactly the same as those for the U(2) theory. For instance, see Fig. 5 where horizontal D1-branes can escape from the QFT by moving downwards. [46] used this strategy to extract Z SU (3) QFT , by dividing out the Z string that one could get from the U(2) theory at κ = 2. This is also consistent with the ratio of Z U (3) QM at ζ < 0 and ζ > 0, which is

5d SCFT from D4-D8-O8 and enhanced symmetry
In this section, we use the QFT instanton partition function Z QFT = Z QM Z string for the Sp(N) theory with 1 antisymmetric and N f ≤ 7 fundamental hypermultiplets to study the 5d SCFT of [17]. The relevant Z string factors are all identified in section 3.4.1. In particular, we would like to study the superconformal index [26] for the 5d SCFTs. This index is a supersymmetric partition function on S 4 × S 1 . When the 5d SCFT admits a relevant deformation to a 5d SYM, [10] studied this quantity in detail. One can define it by I(t, u, m i , q) = Tr (−1) F e −β{Q,S} t 2(Jr +J R ) u 2J l e −F ·m q k . (4.1) J r , J l are rotations of SO(4) ⊂ SO(5) on S 4 , being the Cartans of SU(2) r × SU(2) l ⊂ SO(4). The J r , J l have two fixed points at the north and south poles of S 4 . J R is the Cartan of the SU(2) R symmetry of the F (4) superconformal symmetry. F are the global symmetries of the SCFT which are visible in the 5d SYM as Noether charges. k is the instanton number in 5d SYM. This index counts BPS local operators on R 5 , or BPS states on S 4 × R, which saturate the following bound for the scale dimension (or energy) E.
In 5d SYM, [10] showed that this index can be expressed as a unitary matrix integral with group G, the gauge group of 5d SYM. The measure of the integrand is given by a product of two instanton partition functions of the 5d gauge theory, or more abstractly the partition function of 5d SCFT on Omega-deformed R 4 × S 1 . Especially in the latter abstract viewpoint, one should be using Z QFT rather than Z QM . The precise form is given by [da] is the integral over holonomies of G, including its Haar measure. Z pert is given by [10] Z pert = PE f vec (t, u, e ia ) + f fund (t, u, e ia , e m l ) + f anti (t, u, e ia , e m ) , (4.5) Here we use the notation e ±x = e +x + e −x , and so on. Of course for Sp(1), we do not include f anti in Z pert . Each Z inst is the instanton contribution, which is given by our Z QFT in section 3.

Sp(1) indices
Since we checked that Z QFT from the ADHM quantum mechanics with n A = 1 (our work) and with n A = 0 computed in [10] are same for N f ≤ 5, we do not have to compute the superconformal indices again. For N f = 0, one obtains where χ n (u) is the character of n-dimensional representation of SU(2). The enhanced symmetry E 1 = SU(2) appears rather trivially, as the superconformal index is manifestly invariant under the q → q −1 Weyl symmetry. For N f = 1, one obtains For 2 ≤ N f ≤ 5, one obtains I = 1 + χ adj t 2 + χ 2 (u) 1 + χ adj t 3 + χ 3 (u) 1 + χ adj + 1 + χ adj 2 t 4 + χ 4 (u) 1 + χ adj + χ 2 (u) 1 + χ adj 2 + χ (adj⊗adj) A t 5 where adj denotes the adjoint representation of E N f +1 , and (adj ⊗ adj) A denotes antisymmetrized tensor product of two adjoint representations. A brief explanation of E n characters is provided at Appendix B.
Before proceeding, we comment on the calculations of the superconformal index in series expansion. Unlike Nekrasov's partition function in which the instanton fugacity q is the main expansion parameter, the superconformal index is expanded in t = e −ǫ + , and comes in both positive and negative powers in q. One should first fix the order t n to which one wishes to expand I. Then one investigates the q expansion or q −1 expansion of the two Z inst 's, and see how many instantons one has to keep. Now we get to the Sp(1) theory with N f = 6 matters. One obtains To completely obtain all contributions up to t 6 order, we count the orders as follows. Firstly, one can check that Z QM at 4-instanton order starts from t 6 , while Z QM at 5-instanton order starts from t 9 . So it may appear that the result up to t 6 will be consistently obtained by making a 4-instanton expansion in both Z inst 's in (4.3). However, note that Z inst in (4.3) should be Z QFT = Z QM Z string , and Z string obeys a different upper bound on instanton number with given order in t. Namely, in (3.56), the single particle index f 6 contains t 2 q 2 . So in Z string = P E[f 6 ], t 6 can come with t 6 q 6 = (t 2 q 2 ) 3 , which contain more than 4-instanton order at t 6 . Actually this is the reason why the branching rule of 238602 contains 5, 6 instanton contributions. However, we know Z string exactly so that all contributions at k > 4 can be easily traced. Thus, we expand Z QM that appear in (4.3) up to 4-instantons, and Z string up to 6-instantons, which consistently yields all contributions till t 6 order.
Finally, we consider the Sp(1) index at N f = 7. The superconformal index is given by (4.12) The instanton order counting for the t expansion up to t 6 goes as follows. We computed Z QM up to 5-instantons to get these results. 5-instanton results start at t 6 , so assuming that higher instantons come with higher powers in t, our result should be reliable up to t 6 order. 9 Again Z string up to t 6 order can come with higher instantons. Since f 7 in (3.56) comes with t 2 q 2 , we can maximally have q 6 from Z string = P E[f 7 ] at t 6 . This is the reason why we find contribution at k = ±6 in the branching rule of 1763125. Again, since we know Z string exactly, we expand it up to t 6 and also expand Z QM up to 5-instantons to consistently get all terms up to t 6 .
This finishes our illustration that the Sp(1) index at N f = 6, 7 exhibits E 7 and E 8 enhancement, respectively, complementing the results of [10] at N f ≤ 5. We close this subsection by a few comments on related works. The first line of the index (4.9) was obtained in [27], by computing Z inst from a suitablly Higgsed 5d T 4 theory [55]. The microscopic computation of the index (4.11) with E 8 symmetry appears to be new.
is natural for them to have internal SU(2) R doublet indices rather than the spacetime SU(2) r doublet indices as (2.4). This indeed is the case with examples that we explain below.
Consider a 5d hypermultiplet in the fundamental representation of G. The Dirac fermion in this multiplet can be written as a pair of chiral and anti-chiral fermions in SO(4) = SU(2) l × SU(2) r ⊂ SO(4, 1). The chiral fermion has k complex zero modes in the background of k selfdual instantons. In the ADHM gauged quantum mechanics, these zero modes become Fermi multiplets in the fundamental representation ofĜ. These multiplets are responsible for the Z 1-loop factors (3.33), (3.48), (3.49), (3.50).
We make a brief comment on the nature of the degrees ϕ aA ,λ ȧ α , λ a α , ψ a . There is a good sense in which they all 'come from 5d hypermultiplet,' since they all carry the index a for SU(2) F which rotates it. However, some degrees are clearly extra UV degrees of the ADHM quantum mechanics. In particular, zero modes like ϕ aA never appear in the QFT instanton background. Their eigenvalues actually represent the location of D0-branes transverse to D4, and together with ϕ form the transverse space R 5 . One finds the following bosonic potentials ∼ |qαϕ aA | 2 , ∼ [a αβ , ϕ aA ] 2 , which make ϕ aA to decouple with the ADHM fields (a m , qα) in the g QM → ∞ limit (just like ϕ).
As emphasized in many places in this paper, the ADHM quantum mechanics may contain extra degrees even when the 5d hypermultiplet (perturbatively) decouples from the gauge dynamics. An example is the U(1) adjoint hypermultiplet, or Sp(1) antisymmetric hypermultiplet. Even in these cases, the above ADHM degrees are nontrivial. So determining their couplings in the ADHM mechanics in principle should depend on how we embed the instanton mechanics into string theory (to make it well defined).
However, here we note that, [15] more abstractly considered the ADHM degrees coming from 5d hypermultiplets of the classical groups in tensor product representations of the fundamental representations. In fact, [15] just discussed the equivariant index for the ADHM degrees coming from 5d hypermultiplets, whose Plethystic exponential provides their Z 1-loop . The Z 1-loop computed from the recipe of [15] agrees with all examples discussed in this paper (with small corrections on factors, etc. which we believe are simply typos or minor errors). Since [15] writes down Z 1-loop which only requires our knowledge on the free theory, it may not be too sensitive to the physical string theory embedding of our QFT problem. We sometimes blindly used the rules of [15] to compute the indices, even when we cannot naturally motivate the system from D-branes as in this appendix. Such cases are: classification of the possible poles R 0 , R ∞ at the infinities, provided at the beginning of section 3; the Sp(1) theory with N f ≤ 6 flavors at n A = 0.

B Characters of SO(2N f )
SO(2N f ) characters in this paper can be obtained by the Weyl character formula [56]: where h denotes the highest weight of the representation with h 1 ≥ h 2 ≥ · · · ≥ h N f −1 ≥ |h N f | and m denotes chemical potential. For example, two spinor representations of the highest weights ( 1 2 , · · · , ± 1 2 ) have the following characters: where we use chemical potential y i = e m i /2 . In our paper we use two chirality conventions for such spinor representations. In section 3 we call ( 1 2 , · · · , 1 2 ) the chiral spinor and call ( 1 2 , · · · , − 1 2 ) the anti-chiral spinor, which has a bar on its name. On the other hand, in section 4 we follow the convention of [54] for computational convenience. In that case we call ( 1 2 , · · · , − 1 2 ) the chiral spinor for N f = 2, 3, 6, 7 while we still call ( 1 2 , · · · , 1 2 ) the chiral spinor for N f = 4, 5.
All of the E n characters can be read off from the branching rules of E n into its subgroup specified in the main text. For example, E 5 = S0(10) adjoint has the following decomposition under SO (8)