Instanton counting and O-vertex

We present closed-form expressions of unrefined instanton partition functions for gauge groups of type $BCD$ as sums over Young diagrams. For $\mathrm{SO}(n)$ gauge groups, we provide a fivebrane web picture of our formula based on the vertex-operator formalism of the topological vertex with a new type called O-vertex for an O5-plane.


Introduction
At the dawn of the twentieth century, a Young diagram was introduced to represent a partition of a positive integer. It is a basic tool in physics to "count states". The harmonic oscillator in quantum mechanics is a salient example in which states can be counted by Young diagrams. In mathematics, it was first used in representation theory, and it gradually became versatile in other fields due to its fundamental nature. From the modern viewpoint, it plays the role of bridges that connect different areas.
The seminal paper [1] considers 4d and 5d gauge theories with eight supercharges on the Ω-background and employs equivariant localizations on instanton moduli spaces. An exact instanton partition function reproduces non-perturbative dynamics of a 4d N = 2 theory [2,3]. Moreover, for a gauge group of type A, fixed points of the equivariant action are classified by ordered sets of Young diagrams and the computation of instanton partition functions significantly simplifies to combinatorics of Young diagrams. Consequently, exact results [1,4] in 4d and 5d theories with eight supercharges connect instantons to BPS/CFT correspondence in physics (for instance [5][6][7][8][9]), and geometric representation theory in mathematics (for instance [10][11][12][13]).
A 5d theory on the Ω-background effectively reduces to supersymmetric quantum mechanics on instanton moduli spaces. For gauge groups of type BCD, an instanton moduli space admits an ADHM description [14], which is a Nakajima quiver variety [15] of Jordan type [16]. Using the ADHM descriptions, instanton partition functions are successfully expressed as Jeffrey-Kirwan contour integrals in outstanding works [17,18]. Moreover, SO (7) and G 2 instanton partition functions are written by sums over Young diagrams in [19], employing ingenious representation theoretic methods. Nonetheless, unlike gauge groups of type A, it has been an open problem for a long time to classify poles and obtain closed-form expressions as residue sums from the ADHM descriptions for gauge groups of type BCD. This paper provides a solution for SO(n) gauge groups and a conjectural expression for Sp(N ) gauge groups to the long-standing problem at the unrefined level in §2.
String theory provides other perspectives for instanton partition functions. For a gauge group of type A, the topological vertex [20] computes a partition function from a toric Calabi-Yau three-fold thanks to geometric engineering [21,22]. Also, the duality [23] between a toric Calabi-Yau three-fold and a (p, q)-fivebrane web allows us to identify it with the partition function of a 5d gauge theory on an intersection of fivebranes where instantons are realized by D1-branes on D5-branes. A 5d gauge theory with a gauge group of type BCD is realized on a stack of N D5-branes on an O5-plane in string theory. Recently, an analog of the topological vertex for an O5-plane is proposed in [24], called an O-vertex. Using the O-vertex, we obtain a closed-form expression of a partition function of an SO(n) gauge theory in §3. Remarkably, it coincides with the expression obtained from the ADHM description. At the end of §3, we also show that the O-vertx can provide a closed-form expression of the G 2 instanton partition function by using fivebrane web diagrams [25,26].
This paper aims to present unrefined instanton partition functions for gauge groups of type BCD as concisely as possible. Since Jeffrey-Kirwan contour integrals from the ADHM descriptions are rather complicated, we delegate the pole analysis to Appendix B. Also, detailed computations for correlation functions involving the Overtex are given in Appendix C. In Appendix D, we give expressions of 4d unrefined instanton partition functions for gauge groups of type BCD. We believe that we use the standard notations in literature. Nevertheless, the reader can refer to Appendix A for notations and definitions necessary for this paper.

SO(n) instanton partition functions
A 5d theory on the Ω-background effectively reduces to supersymmetric quantum mechanics on the instanton moduli spaces. Fortunately, the k-instanton moduli space for SO(n) gauge group admits a standard ADHM formulation. Hence, the supersymmetric quantum mechanics on the k-instanton moduli space is described by the Sp(k) gauge theory with one hypermultiplet in the second rank antisymmetric representation and n hypermultiplets in the fundamental representation [16,17]. The partition function [17] of the supersymmetric quantum mechanics is given by the contour integral (2.1) where n = 2N + χ with N = n 2 , n ≡ χ (mod 2). Remarkably, in the unrefined limit 1 = − 2 = , the JK-poles are classified by N -tuples λ of Young diagrams with the total number of boxes N s=1 |λ (s) | = k as analyzed in Appendix B. We specify a pole location associated with a content (i, j) ∈ λ (s) as By defining a familiar factor [1] the residue sum is then expressed as Here we define a total ordering on boxes of the Young diagrams: (2.5) The partition function (2.4) is invariant under the inverse A i ↔ A −1 i of the Coulomb branch parameters. In the unrefined limit, the set of poles (2.2) are valid even when 5d hypermultiplets in the fundamental representation of SO(N ) [27, (4.14)] are included.
As we will see in §3, the formula admits an interpretation of the extended topological vertex formalism with an O5-plane. In fact, the terms colored by red in (2.4) correspond to the Nekrasov factors (A.6) that appear in the U(N ) instanton partition functions. On the other hand, the other black part corresponds to the contribution from O5-plane, and we call it the M -factor (3.15). Comparing with (3.10) and (3.18) from the topological vertex, we observe that we can rewrite (2.4) as The formulas (2.4) and (2.6) allow us to check the Lie algebra theoretic relations among instanton partition functions. For instance, the isomorphism so(4) su(2) ⊕ su (2) indicates the identity of the partition functions Here q is the instanton counting parameter, and we confirm this identity up to 10instanton. The isomorphism so (6) su(4) of Lie algebras leads to the equality of the partition functions which we verify up to 6-instanton. Furthermore, using an embedding SU(4) ⊂ SO (7), the SO(7) instanton partition function has been written as a sum over 4-tuples of Young diagrams in [19, (2.18)]. Although the expression (2.4) for SO (7) is written in terms of a sum over 3-tuples of Young diagrams, we check that it agrees with the unrefined limit of [19, (2.18)] up to 6-instanton.

Sp(N ) instanton partition functions
Similarly, the Sp(N ) instanton moduli space can be also described by the ADHM quiver. Supersymmetric quantum mechanics on the k-instanton moduli space is described by the O(k) gauge theory with one hypermultiplet in the second rank symmetric representation and N hypermultiplets in the fundamental representation [16,17]. Since O(k) has two connected components O(k) ± (det = ±), the instanton partition function receives contribution Z k ± from each connected component O(k) ± . A choice of taking the sum or difference of Z k ± corresponds to the one of the discrete θ-angle that originates from π 4 (Sp(N )) = Z 2 [28]. Thus, the k-instanton partition function with trivial θ-angle is given by whereas that with non-trivial Z 2 element is given by (2.8) Using the description of supersymmetric quantum mechanics on the ADHM description, the contour integral expressions of Sp(N ) instanton partition functions are given in [18,29,30]. The formulas are rather involved so that we summarize the integral expressions in Appendix B. In the unrefined limit, the JK-residues of Z k ± are classified by (N + 4)-tuples of Young diagrams with the total number of boxes N +4 s=1 |λ (s) | = . The relation between the instanton number k and the total number of boxes will be given below. More remarkably, they can be expressed by a single universal formula where φ i,j (s) and N i,j (s, t) are given in (2.2) and (2.3), and C λ, A is a constant Each constant is defined as follows: C λ (s) =∅,As = 1, and C λ (s) ,As=±1,±q where m is the number of rows with λ (s) where m is the number of rows with λ (s) j ≥ j + 1 .
(2.11) (See also (B.10) and (B.11) for more illustrative description.) These constants are in principle determined by the number of poles and the determinant of JK vectors that define a cone, and we conjecture (2.11) by comparing (2.9) and JK-residues of (B.5). The formula (2.9) is a close cousin of that of SO(2N ) instanton partition function (2.4). The difference is that the non-trivial constant (2.10) is involved, and poles of four other types (s = N + 1, . . . , N + 4) do not involve the Coulomb branch parameters. Note that (2.9) can be written as in the form of (2.6).
Performing the JK residues as in Appendix B, Z k ± can be written as (2.12) The pole structure will remain as it is even if we include 5d hypermultiplets in the fundamental representations [30, (3.48)-(3.50)]. Again, we can check the Lie algebra theoretic relations of instanton partition functions. The isomorphisms sp(1) su(2) and sp(2) so(5) of Lie algebras lead to the equality of the partition functions . We check the first and second equality up to 27-instanton and 16-instanton, respectively. The first equality is further checked up to 28-instanton in the 4d limit.
We can also use string duality to check the formulas. The 5d Sp(N ) pure gauge theory can be realized by a stack of N D5-branes with an O7 − -plane. By splitting an O7 − -plane into two (p, q)-sevenbranes [31] and performing Hanany-Witten transitions [32], one can obtain a fivebrane diagram [33, Figure 14] from which the partition function is computed by the topological vertex. The resulting formula is expressed as a sum over 2N -tuples of Young diagrams with k the total number of boxes. We verify the agreement up to 5-instanton for Sp(N ) (N < 7) with both trivial and non-trivial θ-angle. Thus, the string duality yields the highly non-trivial identities for the Sp(N ) instanton partition function.

Instantons from O-vertex
Topological string theory provides an alternative and powerful way to compute instanton partition functions of 5d N = 1 gauge theories. In the case of quiver gauge theories with gauge groups of type A, a chain of string dualities takes the gauge theory to the A-model of topological string theory defined on toric Calabi-Yau manifold [21][22][23]. The topological vertex formalism [20,34,35] expresses partition functions in terms of sums over Young diagrams, which strikingly simplifies the evaluations of the partition functions. Recently, inspired by the work of [36] and its S-dual configuration [37], an extended version of topological vertex formalism was developed for 5d gauge theories with SO(n) gauge groups [24] in which a new topological vertex, called the O-vertex, was introduced for O5 − -plane. In this section, we obtain a closed-form analytic expression of the instanton partition function of the 5d SO(n) pure gauge theory in terms of Young diagrams from the extended topological vertex formalism. The results are consistent with the expressions obtained from the ADHM descriptions in the previous section. As a by-product, we also present a closed-form expression of the G 2 instanton partition function by using the O-vertex.

SO(2N ) instantons
The brane construction of the 5d N = 1 SO(2N ) pure gauge theory is well-known in string theory: one can put N D5-branes on the top of O5 − orientifold (see (3.1)).
It is proposed in [24] that an extended version of the topological vertex can be applied to the above brane web with orientifold. In this proposal, topological vertices C µνλ are assigned to trivalent vertices as usual, while a new type of topological vertex called O-vertex is attached to the intersection point of the orientifold and (2,1)-brane: where V ν is defined as the following entry of the vertex operator, Here, we use the vertex-operator formalism of the topological vertex (see Appendix A), and J n are the modes of the free boson. The intersection point in the following diagram, By applying the extended topological vertex formalism to the web diagram (3.1), the partition function can be written as where the summation is performed over (3N )-tuples Young diagrams ( λ, ν, µ) = (λ (s) , ν (s) , µ (s) ) (s = 1, . . . N ) and we set ν (N +1) = ∅ = µ (N +1) . We expect the above partition function to reproduce the 5d partition function with SO(2N ) gauge group; more precisely, we have where is the one-loop factor of the 5d gauge theory, and Z inst G is the instanton partition function investigated in §2. Note that Z root SO(2N ) can be obtained in the extended topological formalism by setting all λ (s) = ∅. Summations over ν (s) , µ (s) can be expressed as a correlation function of vertex operators including the O-vertex (3.4) where V − is the vertex operator (A.3) for the skew Schur function, and the Coulomb branch parameters are written as This simplifies an expression of the 5d instanton partition function where N µν is the Nekrasov factor (A.6) and the M -factor is defined by Thus, the k-instanton partition function with SO(2N ) gauge group receives contributions from λ = (λ (1) , . . . , λ (N ) ) with k = N s=1 |λ (s) |. Applying the Baker-Campbell-Hausdorff formula the M -factor can be written as One can show that this can be written as .
The equality between (3.13) and (3.15) is given in Appendix C. Note that the Mfactor is invariant under the permutation group for ∀σ ∈ S N . Plugging the expression (3.15) into (3.10), it is equal to (2.6). Let us remark that another approach to the SO(2N ) instanton partition functions was proposed in [38] by the topological vertex with "trivalent gluing". Although the method in [38] has its own advantage (such as applications to gauge groups of other types), it is computationally expensive for SO(2N ) gauge theories. The vertexoperator formalism of the O-vertex in this section provides a closed-form expression (3.10) as a sum over N -tuples of Young diagrams. Our formula is computationally efficient so that it would be desirable to find a synergy between our approach and the trivalent gluing in [38].

SO(2N + 1) instantons
In string theory, the 5d N = 1 SO(2N + 1) pure gauge theory is realized by N D5-branes with an O5 − -plane, which is equivalent to an O5 − -plane with 1 2 D5-brane. The "effective" brane web of 5d SO(2N +1) theories that can be used in the extended topological vertex is given in [24,39,40] where we need to take Q 0 → 1 limit at the end of the calculation. Applying the extended topological vertex to the brane web (3.16), the partition function can be written as where the summation is performed over (3N . The summations over ν (s) , µ (t) can be packaged into the M -factors (3.11), and the instanton partition function of 5d SO(2N + 1) theory can be extracted out in the same way as It follows from (3.15) that the second M -factor can be written as (3.17) Thus, we have (3.18) Using the expression (3.15), it is equal to (2.6).

G 2 instantons
We can also use the extended topological vertex formalism to the 5d G 2 pure gauge theory. Indeed, there are two different ways in the fivebrane web realizations of the 5d G 2 pure gauge theory [25,26]. (We take the limit like (3.16) in which the uncolored leg touches the O5-plane in Figure 1.) Applying the extended topological vertex formalism and using the identity (3.17) of the M -factor, we can show that the two diagrams provide the same formula for the G 2 instanton partition function This expression is equal to [19, (2.53)] with the change of variables if we ignore the matter and take the unrefined limit there. Figure 1. Two web diagrams for the 5d G 2 pure gauge theory.

Future directions
The results in this paper will lead to a variety of natural generalizations. More importantly, we expect that many new research directions will be opened up.
• More general 5d theories: As briefly mentioned in §2, the partition function with matters in the fundamental representation can be naturally obtained. However, more analysis has to be carried out to obtain closed-form expressions of instanton partition functions with matters in higher rank representations or spinor representations of SO(n) [19,30,39] as well as quiver gauge theories with gauge groups of type BCD.
• O-vertex for Sp(N ) instantons: In §3, we have realized SO(n) instanton partition functions by applying the extended topological vertex to fivebrane webs with an O5 − -plane in string theory. Although an approach to the Sp(N ) instanton partition functions by the topological vertex has been proposed in [36], the forms of the partition functions in (2.12) strongly suggest that there is a formalism of the extended topological vertex analogous to §3 for fivebrane webs with an O5 + -plane in string theory.
• Codimension-4 defects and qq-characters: We can also consider instanton partition functions with various defects. In particular, partition functions with codimension-4 defects appear as the Y -operator in the qq-characters [8,41]. At the refined level, the qq-characters for gauge groups of type BCD have infinitely many terms [42]. However, they truncate to finite terms at the unrefined level. This allows us to study Lie algebra theoretic relations of qq-characters, interpretations as quantum Seiberg-Witten curves, and actions of quantum toroidal algebras, which will be addressed in the forthcoming paper [43].
• Relation with blowup equations: The blowup equation, originally introduced in [44,45], provides a powerful tool to compute the instanton partition function recursively. Given the input data such as the one-loop and oneinstanton partition function, one can fix the full instanton partition function completely from the blowup equation in principle. It is necessary to show that our expressions for BCD gauge theories satisfy the corresponding blowup equations developed in [46,47] as a future work. On the other hand, the relation between the blowup equation and the topological string was revealed in [48], and it will be very interesting to explore the role of an orientifold in this context.
• AGT correspondence and W -algebras: The AGT correspondence has been studied exclusively for gauge groups of type A [5] in the last decade because closed-form expressions of instanton partition functions in terms of Young diagrams were available only for type A. The results in this paper need to be examined from the viewpoint of the AGT correspondence and W-algebras [49][50][51].
• Surface defects and isomonodromic deformations: The discrete Fourier transformations of the unrefined instanton partition functions can be interpreted as the τ -functions of Painleve-type equations [52]. This can be formulated in the framework of the blowup equations with surface defects [53,54]. This program was generalized to arbitrary simple gauge groups in [55] and the connection with the analytic expression obtained in this paper is to be explored in the future.
• Geometric engineering of O5-plane: For gauge groups of type A, the geometric engineering [21,22] connects a 5d gauge theory to a topological string theory of a toric Calabi-Yau three-fold, and a skeleton of the toric Calabi-Yau three-fold is identified with the corresponding fivebrane web [23]. In this paper, we show that fivebrane webs with an O5 − -plane can be formulated by the extended topological vertex. However, we miss an interpretation of fivebrane webs with an O5 − -plane in terms of a topological string theory of a Calabi-Yau three-fold. In particular, we expect that the instanton partition function obtained in this paper is a generating function of Gopakumar-Vafa invariants [56] of a certain Calabi-Yau three-fold. It would be intriguing to find a duality between fivebrane webs with an O5-plane and a Calabi-Yau three-fold.
• Open topological string with an O5-plane: The topological vertex was originally developed from the large N duality between the topological string theory and Chern-Simons theory. It is natural to ask whether the extended topological vertex can be understood as the large N limit of SO(N )/ Sp(N ) Chern-Simons theory. It is expected that open topological string theory with the O-vertex gives ingenious perspectives of Kauffman polynomials [57].
• Refinement: As elucidated in Appendix B, JK-residues from the ADHM descriptions of type BCD are not classified in the conventional way by Young diagrams at the refined level. Nevertheless, it is desirable to obtain closed-form expressions of fully refined instanton partition functions for arbitrary gauge groups like SO (7) and G 2 gauge groups in [19].

Acknowledgments
We are grateful to the authors (Sung-Soo and Futoshi) of [36] and the authors (Hee-Cheol, Joonho, Seok, Ki-Hong, Jaemo) of [19] for explanations and clarifications of their papers. We would also like to thank Hirotaka Hayashi, Yuji Tachikawa

A Notations and definitions
In this appendix, we summarize notations and definitions necessary for the paper. For more details, we refer to [24]. Let λ = (λ 1 , λ 2 , · · · ) be a Young diagram (i.e. non-negative integers such that λ i ≥ λ i+1 and |λ| = i λ i < ∞). Let (λ) be the length of the Young diagram, which is the number of non-zero λ i . We write the transposition of λ by λ ∨ . The arm length a i,j (λ) of a box at (i, j) is the number of boxes to the right of the box in the diagram λ, and the leg length l i,j (λ) is the number of boxes below the box. If it is clear from the context, we simply write a i,j and l i,j . Note that A s are called Coulomb branch parameters. Often we use the notation 2 + = 1 + 2 , and it becomes zero in the unrefined limit 1 = − 2 = . Therefore, unrefined instanton partition function depends only on q = e − . In this paper as well as the original U(N ) case [1], a k-instanton partition function Z k G is expressed as a sum over N -tuples of Young diagrams, which we denote The total instanton partition function of gauge group G is expressed as a generating function with instanton counting parameter q Next, let us fix the notation in the formalism of the topological vertex. We follow the standard notation of the unrefined topological string, in which a Schur function s λ (x) and a skew Schur function s λ/µ (x) play pivotal roles. The topological string partition function can be constructed from fermionic operators, ψ and ψ * , and bosonic current J [35,58,59] {ψ n , ψ m } = {ψ * n , ψ * m } = 0, {ψ n , ψ * m } = δ n+m,0 , J n := j∈Z+1/2 : ψ −j ψ * j+n : , Given the Frobenius coordinate λ = (α 1 , α 2 , . . . |β 1 , β 2 . . . ) [60], we define a state Then, the skew Schur function can be expressed as The topological vertex labeled by Young diagrams ν, µ, λ in the clockwise direction is given by The framing factor is given by The unrefined Nekrasov factor is defined by It can be related to the Schur function as The vertex functions are defined by A plethystic exponential function is defined by

B Contour integrals from ADHM
This appendix analyzes the residue structure of the Jeffrey-Kirwan (JK) contour integrals [61,62] of instanton partition functions for gauge groups of type BCD obtained from the ADHM descriptions. There are many works [29,30,49,50,[63][64][65][66] to carry out the contour integrals for SO(n) gauge group [17] and for Sp(N ) gauge group [18]. As analyzed in these works, the JK residues for gauge groups of type BCD are not classified by the conventional way with Young diagrams as in [1] at the refined level. Here, we will show that the JK poles are indeed classified by ordered sets of Young diagrams at the unrefined level. The analysis here leads to closed-form expressions (2.4) and (2.12).

B.1 Residues for SO(n)
For the sake of concreteness, let us look at the example of the SO(4) instanton partition function from (2.1). Choosing the reference vector as η = 1, the JK residues of the 1-instanton Z k=1 SO(4) are When we take the unrefined limit + = 0, the two residues at φ 1 = ±a s coincide. Therefore, the poles essentially correspond to 2-tuples of Young diagrams: ( , ∅) for φ 1 = ±a 1 and (∅, ) for φ 1 = ±a 2 . Let us move to the case of 2-instanton with η = (1, 1 + ε) for a sufficiently small ε. Then, the charge vectors are The corresponding JK poles for {φ 1 , φ 2 } are as follows: where s = 1, 2 and j = 1, 2. Hence, even if we take into account the difference by the sign in front of the Coulomb branch parameter, the poles in the forth do not appear in the computation of U(N ) instanton partition functions, and they do not fit into the conventional pole classification by Young diagrams as in type A at the refined level. However, once we take the unrefined limit + = 0, the poles in (i)-(iv) are related by relative signs and the exchange φ 1 ↔ φ 2 . More remarkably, they provide the same residues corresponding to ( , ∅), ( , ∅), (∅, ), (∅, ) in (2.4). Also, all the poles in (v) give the same residue corresponding to ( , ). All the residues from (vi) are zero. Consequently, we obtain the unrefined instanton partition function in terms of a sum over 2-tuples of Young diagrams where the terms correspond to Young diagrams ( , ∅), ( , ∅), (∅, ), (∅, ), ( , ), respectively. It is also worth mentioning about the shift by πi in (vi) of (B.3), called the holonomy saddle [67,68]. This shift appears because the determinant of the corresponding charge vectors in (B.2) is greater than one, but it is indeed equal to two in this case. Even at the refined level, the SO(4) instanton partition function receives no contribution from the poles in (vi) of (B.3). On the other hand, the SO(5) instanton partition function has the same pole structure as in (B.3), and it receives a non-trivial contribution from the holonomy saddles: This is the universal feature of SO(2N +1) instanton partition functions at the refined level, and these residues apparently do not fit into the conventional pole classification by Young diagrams as in type A. Nevertheless, as can be seen in (B.4), it turns out that the contributions from the holonomy saddles vanish at the unrefined level t = q.
This structure holds even at higher instantons. Despite the complicated pole structure of the contour integral (2.1) at the refined level illustrated above, the nontrivial JK poles of the k-instanton partition function are classified by n 2 -tuples of Young diagrams with the total number of boxes k as (2.2) [63,64], and the JK residues yield (2.4).

B.2 Residues for Sp(N )
As briefly described in §2.2, the ADHM description of the k-instanton moduli space leads to supersymmetric quantum mechanics with O(k) gauge group so that the instanton partition functions receive contributions Z k ± from each connected component of O(k). The complete contour integral formulas are given in [18,29,30] based on the result in [17], and we summarize the formulas below. Writing k = 2 + χ, with χ = 0, 1, Z k ± are given by The order |W | of the Weyl group for O(k) ± is The integrands for the 5d vector multiplet are given as follows: (B.5) Now let us analyze the JK residues of these integrals. Since the JK poles at the refined level are rather complicated, we focus only on the unrefined limit with Sp(1) gauge group. We will see below that the JK residues are expressed as a sum over 5-tuples of Young diagrams as in (2.12). The integral is trivial at 1-instanton so that we will investigate JK residues from 2-instanton.

3-instanton
The JK poles for the integrand Z k=3 vec,+ with η = (+1) are located at The difference from (B.6) lies at the last one coming from sinh φ 1 ±
The charge vectors with determinant two are The JK poles corresponding to (a)-(d) are located at as well as their simultaneous πi shifts. The poles from the first line give the term for (∅, , ∅, ∅, ∅) in (2.12) whereas at the second line provide the term for (∅, , ∅, ∅, ∅).
The terms for (∅, ∅, , ∅, ∅) and (∅, ∅, , ∅, ∅) come from their holonomy saddles. The JK poles corresponding to (e),(f) are located at as well as the πi shifts ± 2 → ± 2 + πi. These poles yield the residues corresponding to ( (2.12). In total, the 4-instanton partition function can be written as a sum over 5-tuples of Young diagrams with the total number of boxes two as in (2.12).
On the other hand, JK poles for the other part Z k=4 vec,− are simpler. The JK poles with determinant one are located at φ 1 = ±a 1 , which corresponds to ( , ∅, ∅, ∅, ∅). The JK poles with determinant two are located at

higher instantons
At this moment, we familiarize ourselves with the JK residues of the contour integrals (B.5). Similarly, we can analyze the JK residues at higher instantons. The JK poles of (B.5) are classified by 5-tuples of Young diagrams λ (s) (s = 1, · · · , 5): where the total number of boxes is equal to the number of integration variables in (B.5). As seen above, a 1 is the Coulomb branch parameter of Sp (1) Up to constants, it is straightforward though tedious to derive that the residues at these poles are given by (2.12), which can be fittingly packaged into the single formula (2.9).
In principle, the constants in (2.9) are determined by the number of poles with the same residue and the determinant of the corresponding charge vectors. By performing explicit calculations at high instanton numbers, we propose the following conjectural expressions. (It would be desirable to derive the expressions rigorously.) For effective Coulomb branch parameters A s = ±1, ±q (B.10) For effective Coulomb branch parameters A s = ±q, we assign the following coefficient to a Young diagram λ (s) that is created by adding boxes to the right and bottom of the m × (m + 1) rectangular Young diagram Note that a Young diagram with a single column corresponds to m = 0. The analysis for higher rank gauge groups goes in parallel. In conclusion, we obtain a sum (2.12) over (N + 4)-tuples of Young diagrams after the JK-residue integral of (B.5) for Sp(N ) gauge group.

C Derivation of the M -factor
In this appendix, we prove the equivalence between (3.13) and (3.15) by recursion relation. Let us suppose we add one box at (m, n) to λ (s) as in Figure 3 and we denote the resulting Young diagram by λ (s) := λ (s) + . Then, it follows from (3.13) that the ratio of the M -factors is expressed in the form of a plethystic exponential function M λ (1) ,...,λ (s) + ,...,λ (N ) M λ (1) ...,λ (s) ,..., . Now, we will first bring the part of Y 1 into a product form. We can manipulate Y 1 into The terms with red, gray and blue in the first line correspond to those from boxes in Young diagrams (Figure 3) with the corresponding colors in the second line. Note that the terms with blue provide non-trivial contributions when λ As a result, we have (C.1) Next, we will manipulate the part of Y (t) 2 into a product form by cases. If (λ (t) ) < n, we can write it as Combining this with (C.1), we obtain a recursion relation of (3.15) for λ (s) → λ (s) . If (λ (t) ) ≥ n, we can write (q l n,j (λ (t) )−j+1+m − q l n,j (λ (t) )−j+m ) .
As a result, we have Combining this with (C.1), we obtain a recursion relation of (3.15) for λ (s) → λ (s) .

D 4d instanton partition functions
As a 5d theory is put on S 1 , the zero radius limit of S 1 leads to an effective 4d N = 2 gauge theory. In this sense, it is straightforward to write down 4d instanton partition functions for gauge groups of type BCD from the analytic expressions in this paper.
To take the 4d limit, we need to restore the dependence on R, the radius of S 1 in the parameters. We replace a s → a s R , → R , in (2.4), (2.6) and (2.12). As a consequence, all trigonometric functions in the 4d limit R → 0 reduce to the rational form We remark that specific power of the radius, R −n G | λ| , is absorbed into the instanton counting parameter q, i.e.
Taking this limit, we obtain the 4d k-instanton partition function for SO(n) gauge group where the summation is taken over N -tuples of Young diagrams with k = N s=1 |λ (s) |. The other functions such as φ i,j (s) and N i,j (s, t) are as in (2.2) and (2.3).
Although there are N + 4 effective Coulomb branch parameters in (2.12), two of them disappear in the 4d limit since two parameters have an imaginary part iπ in (B.9), which changes sinh to cosh. Therefore, the basic building block for 4d Sp(N ) instanton partition functions becomes where the summation is taken over (N + 2)-tuples of Young diagrams with = N s=1 |λ (s) |. Here, C λ, a is defined as an appropriate rational version of (2.10): where C λ (s) ,as=0, 2 = 2 2m−1 where m is the number of rows with λ This is consistent with the fact that the non-trivial discrete θ-angle exists only in 5d theories.