More on topological vertex formalism for 5-brane webs with O5-plane

We propose a concrete form of a vertex function, which we call O-vertex, for the intersection between an O5-plane and a 5-brane in the topological vertex formalism, as an extension of the work of [1]. Using the O-vertex it is possible to compute the Nekrasov partition functions of 5d theories realized on any 5-brane web diagrams with O5-planes. We apply our proposal to 5-brane webs with an O5-plane and compute the partition functions of pure SO(N) gauge theories and the pure G2 gauge theory. The obtained results agree with the results known in the literature. We also compute the partition function of the pure SU(3) gauge theory with the Chern-Simons level 9. At the end we rewrite the O-vertex in a form of a vertex operator. ar X iv :2 01 2. 13 30 3v 1 [ he pth ] 2 4 D ec 2 02 0


Introduction
The topological vertex formalism [2][3][4][5] has long been known as an alternative method to compute the partition function of five-dimensional (5d) N = 1 supersymmetric field theories on the Ω-background C 2 q,t × S 1 . Unlike the original way using the localization technique [6][7][8], the topological vertex formalism computes the partition function in a Feynman-diagram-like manner. That is to say, given a diagram corresponding to a local Calabi-Yau threefold realizing a 5d theory by an M-theory compactification, we assign a function called the topological vertex to each vertex in the diagram, and sum over all possible internal states labeled by Young diagrams and weighted by Kähler parameters (which plays the role of a propagator). Through a chain of string dualities [9], one can see that these toric diagrams are mapped to (p, q) 5-brane web diagrams in type IIB string theory. From this viewpoint 5d N = 1 supersymmetric field theories are realized on the 5-brane webs [10,11].
Although the topological vertex formalism is an elegant and systematic way to compute related physical quantities, the original formalism only applies to (dual) toric diagrams. In terms of the corresponding 5d gauge theories, the application had been limited mostly to linear quiver theories with SU-type gauge groups. Since then, the topological vertex formalism has been extended in various ways to compute the partititon functions of gauge theories with different gauge groups. One is the application to web diagrams which contain configurations where 5-branes jump over other 5-branes [12][13][14]. Then it is possible to compute the partition functions of Sp(N ) gauge theories [15,16] from the 5-brane webs constructed in [17]. Another extension is the application to web diagrams with O5-planes [1]. By combining the two methods the partition functions of G 2 gauge theories have been computed [18]. Furthermore, web diagrams each of which are made by gluing three or four (dual) toric diagrams have been constructed in [19] and the method developed there computes the partition functions of SO(2N ) gauge theories and also the pure E 6 , E 7 , E 8 gauge theories. Recently the topolgical vertex formalims has been also extended in other directions [20][21][22][23][24][25][26][27][28][29].
In this paper we consider a further extension of the formalism in [1] for 5-brane webs with O5-planes. To apply the topological vertex to 5-brane webs with an O5-plane ref. [1] utilizes "generalized flop transitions" in [30]. After the transition and taking a different fundamental region, the brane configuration around an O5-plane becomes locally a strip diagram and hence we can apply the topological vertex. However the generalized flop transition is applicable to configurations that yield an Sp-type gauge group and the formalism has not been applied to for example the brane webs for pure SO(N ) gauge theories. In this paper we propose a vertex function labeled by a Young diagram which applies to an intersection point between an O5 − -plane, a (2, 1) 5-brane and an O5 + -plane, where the Young diagram is assigned to the (2, 1) 5-brane, by making use of the formalism in [1]. We call the vertex function O-vertex. With the O-vertex, it is not necessary to contain configurations of an Sp-type gauge group for applying the topological vertex and it becomes possible to apply the topological vertex to any 5-brane web diagrams with O5-planes. We will exemplify the O-vertex by computing the partition functions of various pure SO(N ) gauge theories and also the pure G 2 gauge theory. We also compute the partition function of the pure SU(3) gauge theory with the Chern-Simons (CS) level 9, which was proposed in [31] and was also constructed from a geometry [32] and from a 5-brane web [33] approach. Furthermore we consider integrating the O-vertex into computations using vertex operators [5,34]. We propose an expression of a vertex operator associated to the O-vertex. This paper is organized as follows: in section 2, we propose the O-vertex by utilizing the formalism in [1]. In section 3, we validate our proposal by computing the partition functions of the pure SO(N ) (N = 4, 5, 6,7,8) gauge theories and also the pure G 2 gauge theory from 5-brane webs with an O5-plane. We compare the results with the known expressions and then find perfect agreement. We also compute the partition function of the pure SU(3) gauge theory with the CS level 9. Finally in section 4, we reconsider the O-vertex from vertex operator computations and we propose a vertex operator associated to the O-vertex. Section 5 summarizes our results and discuss possible future directions. In appendix A we list explicit expressions of the O-vertex and describe its properties. Appendix B summarizes some formulae of the Nekrasov partition functions and the Schur functions, which we use in various computations.

O-vertex
In this section, we propose a vertex function, O-vertex, which can be applied to the intersection point of a 5-brane with an O5-plane for computing the partition functions of 5d theories realized on 5-brane webs with an O5-plane. We first give a brief review on the topological vertex formalism for 5-brane webs with an O5-plane proposed in [1], and then extend the formalism by introducing the O-vertex in section 2.2. In section 2.3, we further explain how to use the O-vertex for 5-brane webs with an O5-plane by exploiting the Higgsing procedure discussed in [35].

Topological vertex formalism with an O5-plane
The topological vertex formalism is a convenient method to compute the topological string partition functions associated to toric Calabi-Yau threefolds. Given a toric diagram, or equivalently a (p, q) 5-brane web diagram in type IIB string theory, we assign the topological vertex to each vertex appearing in the diagram. The topological vertex is labeled by Young diagrams which are assigned to all the internal lines. Then we take the product of all the topological vertices weighted by Kähler parameters and framing factors for the internal lines and sum over all the Young diagram. The explicit form of the (unrefined) topological vertex labeled by Young diagrams λ 1 , λ 2 , λ 3 in clockwise direction is given by [3], where κ(λ) := 2 (i,j)∈λ (j − i), and (2.5) The information above is enough to compute the topological string partition functions for toric Calabi-Yau threefolds. In fact one can apply the topological vertex to certain nontoric diagrams. Higgsing a 5d theory realized on a 5-brane web dual to a toric diagram may yield a non-toric diagram where a line in the diagram jumps over other lines [36]. A typical configuration in such a case is given by (2.6) where the lines with the slope 1 are external lines. For applying the topological vertex to the diagram (2.6) we deform the diagram in the following way [12,14] ∅ ∅ (2.7) Then we apply the topological vertex to the diagram (2.7) with the trivial Young diagrams assigned to both the lines with the slope 1. Kähler parameters we assign in a deformed diagram need to respect the configuration before the deformation. For example, the legnth between the vertical lines in (2.7) should be equal to the length between the horizontal lines in the diagram. We will make use of this technique in section 3.
In [1] the topological vertex formalism has been further extended to 5-brane webs with O5-planes. We review the formalism by using an example which we will utilize later. We start from the following diagram, with the Kähler paramter Q . This diagram may be thought of as the one for the pure "Sp(0)" gauge theory, which is trivial. Instead of directly applying the topological vertex to the diagram of (2.8), we may change the Kähler parameter Q and deform the diagram into [30] O5 − Q 2 (2.9) where Q = Q −1 . Then we use the mirror image for the half of the diagram of (2.9), which yields Namely we take a different fundamental region compared to (2.9). Since this is a strip diagram it is possible to apply the topological vertex to (2.10). The only unusual thing compared to an ordinary strip diagram is that we need to sum over all possible Young diagram configurations λ assigned to the horizontal legs. A subtlety is that the Young diagram λ assigned to the horizontal leg of the mirror image is transposed compared to the original diagram and we need to be careful of the framing factor for the glued horizontal legs. For the original diagram of (2.9) the contribution around the horizontal leg is locally given by On the other hand, the contribution from the horizontal legs from (2.10) is locally given by with some factor g λ . Since they represent the same diagram (2.11) should be equal to (2.12). In fact by usuing the identity (2.13) it is possible to rewrite (2.11) as (2.14) Therefore the factor g λ in (2.12) turns out to be g λ = (−1) |λ| f 2 λ and this is the framing factor assigned to the horizontal legs when we use the diagram of (2.10).
It is now possible to compute the partition function for the diagram (2.10) which is equivalent to (2.9). The partition function is given by Inserting the explicit expressions (2.1) and (2.3) gives , (2.16) where the Nekrasov factor in the unrefined limit is given by (B.35) and P.E. represents the Plethystic exponential defined by . (2.18) In this article, we compute such partition functions as series expansions by Kähler parameters using the mathematica package developed in [19]. Since the diagram (2.10) or (2.9) gives the pure "Sp(0)" gauge theory, the partition function (2.16) should be trivial, namely Indeed, one can check this statement with mathematica to find (2.20)

Proposal for O-vertex
We are interested in extending this formalism to compute the partition function for a diagram which involves the following configuration, where we also introduced the parameter P . As explained in section 2.1, the partition function of the diagram (2.22) can be computed by deforming the diagram into the following one, where Q = Q −1 and P = P Q and taking the fundamental region as in (2.10). The partition function for the diagram (2.23) is then Therefore the function associated to the diagram (2.21) is obtained by applying the limit Q → ∞ with P Q fixed in (2.24). Then it seems that (2.24) diverges from the contribution coming from Young diagrams satisfying 2|µ| + 2|λ| > |ν|. However we observe that there is non-trivial cancellation for higher orders of Q. When ν = ∅, then (2.24) reduces to (2.16) and it is just 1 as we checked in (2.20). Even when ν is non-trivial, interestingly, the result as an expansion over Q 2 stops at a finite order. For example, for ν = (5, 1), we have (2.25) For various choices of ν we have checked the function (2.24) terminates at the order Q |ν| . Hence we conjecture that the expansion by Q of (2.24) terminates at the order Q |ν| for any ν. Then it is possible to take the limit Q → ∞ with P Q fixed and (2.24) becomes Therefore we define a vertex function V ν by We will call V ν O-vertex as it is associated to a line intersecting with the O5-plane.
In the same way, we can compute another O-vertex which is associated to the diagram given by For computing this type of O-vertex we start from the diagram in (2.23) and take the fundamental region as The associated partition function is given by Again we observe that (2.35) terminates at the finite order Q |ν| for various choices of ν.
Hence we conjecture that the expansion of (2.35) by Q terminates at the order Q |ν| . Then we can take the limit Q → ∞ with P Q fixed to obtain another O-vertex W ν assoicated the diagram of (2.33). Namely we define We also observe that whileP ν is not necessarily a polynomial here, but is still zero for |ν| odd. Some examples ofP ν are listed below, and more details can be found in appendix A. In fact (2.36) is related to (2.27) in a simple way. To see that note the topological vertex satisfies, Then using (2.40) to the two topological vertices in the sum of (2.36) yields Identifying the explicit forms of the O-vertices with (2.29) and (2.36), we expect that it is possible to compute the partition functions of any 5d gauge theories that are realized on 5-brane webs with an O5-plane. We will confirm this proposal by computing the partition functions of pure SO(2N ) gauge theories and compare them with the results obtained by the localization method in section 3.1.

Higgsing and O-plane
So far we have obtained the O-vertices (2.29) and (2.36) which are applicable to 5-brane webs with an O5-plane. In fact it turns out that it is possible to apply the O-vertices to 5brane webs with an O5-plane. To see that we first review the relation between an O5-plane and an O5-plane [35]. We start from the 5-brane web diagram for the SO(8) gauge theory with a hypermultiplet in the vector representation, which is depicted in Figure 1(a). The presence of the flavor is shown by the flavor D5-brane which ends on a D7-brane. This theory has a Higgs branch on which the low energy theory becomes the pure SO(7) gauge theory. The Higgs branch opens up when we tune the mass parameter for the flavor as well as a Coulomb branch modulus. By applying the same tuning to the 5-brane web in Figure 2. The brane web for the pure SO(7) gauge theory. The green dotted line represents the branch cut for a half D7-brane.    Figure 1(b). Then we take T → 1 and the D7-brane is on top of the O5-plane. When a D7-brane is on top of an O5-plane it can be split into two half D7-branes [37]. A piece of D5-brane between the two half D7-branes can move in the direction which the D7-branes extend, which represents the Higgs branch. After removing the piece of the D5-brane, we pull the two half D7-branes in the opposite directions into the infinity. This operation leaves a configuratioin with a branch cut of one of the half D7-branes on top of the whole O5-plane and also a half D5-brane on top of the O5 − -plane, which is depicted in Figure  2. Then the resulting theory should be interpreted as the pure SO(7) gauge theory. Since the SO(7) gauge group can be realized on an O5-plane, we may interpret the orientifold in Figure 2 as an O5-plane. Namely an O5-plane is effectively realized as an O5-plane with a branch cut of a half D7-brane on it (and a half D5-brane for an O5 − -plane). Since it is not clear how to apply the topological vertex for a configuration with the branch cut, we find it convenient to use the diagram before moving the half D7-branes into infinity. Moving the D7-branes into the left or the right direction in the diagram does not change the theory and we can equally use the diagram before the moving depicted in Figure 3(a). For practically applying the topological vertex to the diagram in Figure  3(a), we can use the diagram in Figure 3(b) instead of the one in Figure 3(a), Then the partition function for the diagram in Figure 3(b) can be computed using the topological vertex and the O-vertex. The Young diagram assigned to the bottom D5-brane is empty as reviewed in section 2.1 for the application of the topological vertex to certain non-toric diagrams. The Kähler parameter T is reintroduced and it is set to 1 at the end of the computation. Although it seems that we need to sum over infinitely many Young diagrams assigned to the line with the Kähler parameter T , it turns out that a finite sum gives the correct answer, which we will see in section 3.2.
We can use this technique to compute the partition functions of SO(2N + 1) gauge theories and also G 2 gauge theories utilizing the construction in [18]. Furthermore it is also possible to compute the pure SU(3) gauge theory with the CS level κ cs = 9 by applying the brane web proposed in [33], which may be obtained by a twisted compactification of 6d pure SU(3) gauge theory with a tensor multiplet [38].

Examples
In this section, we give several examples of calculations that involve the usage of the Overtex defined in section 2. More concretely, we first check the validity of our formalism in section 3.1, 3.2 and 3.3 by comparing the partition functions of the SO(2N ), SO(2N + 1) and G 2 gauge theories obtained in the topological vertex formalism with the O-vertex with the known one-instanton and two-instanton partition functions in the literature. In section 3.4, we will further apply the O-vertex to the calculation of the partition function of the 5d pure SU(3) gauge theory with the CS level κ cs = 9, by using the brane diagram proposed for it in [33].

SO(2N ) gauge theories
We start from the computation of the partition function of the pure SO(2N ) theory. The brane web diagram is obtained by putting a stack of N D5-branes on the top of an O5 −plane. Hence we can directly apply the O-vertex proposed in section 2.2.

Pure SO(4) gauge theory
The first example is the simplest case N = 2, i.e. SO(4) gauge theory. The 5-brane web diagram for the pure SO(4) theory is given by The partition function of the pure SO(4) gauge theory realized on the web in (3.1) can be computed by applying the topological vertex formalism with the O-vertices given in (2.27) and (2.36) to the diagram in (3.1). Since the brane web has parallel external legs, the partition function computed by the topological vertex formalism contains an extra factor related to contributions from strings between the parallel external legs [12,[39][40][41]. The extra factor in this case is given by Hence the partition function of the pure SO(4) gauge theory is given bŷ where Z

SO(4) top
is the partition function which is directly obtained by applying the topological vertex and the O-vertex to the web diagram in (3.1) and the explicit form is given by The Kähler parameters Q, Q , P are related to gauge theory parameters of the pure SO(4) gauge theory. The height of the color D5-branes in (3.1) is related to the Coulomb branch moduli. The simple roots of the so(4) Lie algebra are represented by e 1 − e 2 and e 1 + e 2 where e 1 , e 2 are the orthonormal basis of R 2 . A string between the top color D5brane and the bottom color D5-brane yields a W-boson in the simple root e 1 − e 2 . On the other hand, a string which extends from the top color brane and reflects on the O5-plane, ending on the bottom color brane corresponds to a W-boson in the simple root e 1 + e 2 . Hence we associate the Coulomb branch moduli A 1 = e −a 1 , A 2 = e −a 2 with the Kähler parameters by The Kähler parameter Q is a mass parameter of the SO(4) gauge theory and there is only one mass parameter in the theory, which is the instanton fugacity q. Namely we have Let us see if the partition function (3.3) reproduces the known result. The perturbative part of the partition function is obtained by taking the limit Q → 0. Then the sum of the Young diagrams λ and σ becomes trivial and the partition function (3.4) becomes is the partition function associted with the diagram (3.10) and Z

SO(4)
right is associated with (3.11) Using the O-vertex (2.27) for (3.8) we checked that (3.12) where o(X k 1 1 , X k 2 2 , · · · ) means that the computation was done until the order X l 1 1 X l 2 2 · · · with l 1 ≤ k 1 , l 2 ≤ k 2 , · · · . Similarly applying the O-vertex (2.36) to (3.9) yields (3.13) (3.14) Since the topological vertex reproduces only the root part of the perturbative partition function we compare (3.14) with (B.2). The root contribution to the perturbative partition funciton of the pure SO(4) gauge theory from (B.2) is given by Hence we see the agreement between (3.14) and (3.15) until the orders we computed. Namely the O-vertices (2.27) and (2.36) reproduced the correct result until the orders we computed. Let us move on to the comparison of the instanton partẐ From the identification in (3.6), the one-instanton partition function can be extracted from (3.3) as the coefficient of Q 1 in (3.16) and it is given bŷ One can guess that the above result may be extrapolated aŝ . (3.19) in terms of the Coulomb branch moduli of the SO(4) gauge theory. We can compare (3.19) with the result computed by the localization method. The result of the instanton partition function for the pure SO(2N + δ) gauge theory for δ = 0, 1 is given in (B.12). Applying N = 2 to (B.12) with the unrefined Ω-deformation parameter : where we took δ → 0 in the second to the last line of the evaluation. For the comparison using (3.20) we also need to be careful of the extra factor in the instanton partition function. In general, the localization result Z loc, inst of the supersymmetric index of an ADHM quantum mechanics is related to the instanton partition function Z loc, inst for a ultraviolet (UV) complete 5d theory bŷ where the extra factor Z loc, extra is the Coulomb branch moduli independent part of Z loc, inst . Namely one needs to subtract a Coulomb branch moduli independent term from (3.20) if there is such a term in the 1-instanton partition function. Indeed the expansion of (3.20) by and hence we can identify the extra term by (3.23) Note that (3.23) is the 1-instanton part of Z loc, extra . Therefore we expect that the full extra factor for the partition function for the pure SO(4) gauge theory computed by (B.12) is given by (3.2). Subtracting (3.23) from (3.20) one finds which agrees with the 1-instanton partition function (3.19) computed by the topological vertex formalism. One can also see that (3.24) or (3.19) is given by the sum of the oneinstanton partition function of the pure SU(2) gauge theory with discrete theta angle zero, which follows from (B.8) with N = 2. This is consistent with the isomorphism of the Lie algebra so(4) su(2) ⊕ su (2).
where Z SU (2) inst (q, A) is the instanton partition function of the pure SU(2) gauge theory with the zero discrete theta angle. The closed-form expressions of the SU(N ) instanton partition functions can be found in appendix B.1. We also checked that the two-instanton partition function computed from the integral (B.12) with the extra factor removed matches with the 2-instanton part of (3.27) until the order P 4 Q 4 .

Pure SO(6) and SO(8) Theories
Let us then consider the partition functions of the pure SO(6) and SO(8) gauge theories. The method of the computation is essentally parallel to that in the case of the pure SO(4) gauge theory discussed in section 3.1.1.
Pure SO(6) gauge theory. We start from the pure SO(6) gauge theory. The pure SO(6) gauge theory can be realized on the following brane diagram, The relation between the Kähler parameters Q, Q , P, R and the gauge theory parameters may be also read off from the diagram in (3.28). The Coulomb branch moduli On the other hand, the instanton fugacity q is proportional to Q. More precisely we consider extrapolating the external (1, 1) 5-brane and the (1, −1) 5-brane until the O5-plane and measure the length between the 5-branes on the O5-plane. Namely, we have The application of the topological vertex and the O-vertex to the diagram in (3.28) gives The 5-brane web diagram (3.28) does not have parallel external legs and hence there is no extra factor in this case. The perturbative part is obtained by taking the limit Q → 0. Then only λ = σ = τ = ∅ contributes in the sum in (3.31) and (3.31) becomes The explicit evaluation of the summation (3.33) yields is satisfied by the O-vertex. The instanton part is obtained from (3.31) by removing the perturabtive part (3.32). Namely the instanton partition function of the pure SO(6) gauge theory is given by We can compare (3.39) with the integral formula (B.12) for the pure SO(6) gauge theory. In fact since so(6) su(4), we can also use the instanton partition function of the pure SU(4) gauge theory with the zero CS level. The one instanton partition function for the pure SO(6) gauge theory from (B.12) becomes (3.40) From (B.8) with N = 4, the one instanton partition function of the pure SU(4) gauge theory with the zero CS level can be found as where we denote the Coulomb branch moduli for SU (4) Since the fundamental representation of SU(4) is mapped to the spinor representation of SO (6), the relation between the Coulomb branch moduli is With this map it is possible to check (3.40) agrees with (3.41). We can then check if (3.40) or (3.41) agrees with the 1-instanton part of (3.39). Indeed we found the agreement between them until the order P 4 Q 4 R 7 .
We can extend the comparison to the two-instanton partition function. We checked that the two-instanton part from (3.39) agrees with the two-instanton part of (B.12) for the pure SO(6) gauge theory until the order of P 4 Q 2 R 2 .
Pure SO(8) gauge theory. Now we turn to the computation of the simplest non-trivial example of the SO(2N )-type, the pure SO(8) gauge theory. The brane diagram for the pure SO(8) gauge theory is given by The Coulomb branch moduli a 1 , a 2 , a 3 , a 4 are the height of the color D5-brane. Hence A i = e −a i , (i = 1, 2, 3, 4) are parametrized by Extrapolating the external (2, 1) and (2, −1) 5-brane webs on the O5-plane, the instaton fugacity is related to the length between the 5-branes on the orientifold. Namely it is given by From the web diagram in (3.43) it is possible to compute the partition function using the topological vertex and the O-vertex and the partition function becomes Since there is no parallel external lines, (3.46) does not contain an extra factor. The perturbative part is obtained by the limit Q → 0 and the partition functin splits into two parts, The computing the summation of the left part yields (3.53) The localization result for the instanton partition function for the pure SO(8) gauge theory is obtained from (B.12). For the 1-instanton part, we find the perfect match between the expression from the integral (B.12) and the result obtained from the topological vertex formalism (3.53) until the order P 2 Q 2 R 2 T 2 . We also computed the two-instanton partition functions in the two methods and the results agreee with each other until the order

SO(2N + 1) gauge theories
As we dicussed in section 2.3 it is also possible to use the O-vertex to compute the partition functions of SO(2N + 1) gauge theories using their 5-brane web realization with an O5plane. We will apply the formalism to the examples of the pure SO(5) and SO (7) gauge theories.
Pure SO(5) gauge theory. We start from the pure SO(5) gauge theory. The 5-brane web diagram which we use for the computation of the partition function of the pure SO(5) gauge theory is given by in the limit T → 1. Note that we first introduce T in (3.54) so that we can apply the Overtex (2.36) in the partition function computation, and then take the limit T → 1 at the end of the calculation. The relation between the gauge theory parameters and the Kähler parameters can be again read from the diagram. Let a 1 , a 2 be the Coulomb branch moduli of the pure SO(5) gauge theory. They are related to the height of the color D5-branes and hence A i = e −a i (i = 1, 2) are given by For the identification of the instanton fugacity we extrapolate the (0, 1) and (1, 1) 5-brane and measure the length between the 5-branes intersecting with the orientifold. Namely it is given by Let us compute the partition function of the pure SO(5) gauge theory from the web diagram in (3.54). Since we take T → 1 at the end of the computation it seems that we need to sum over the Young diagram assigned to the line with the Kähler parameter T . It turns out that the series in terms of T terminates at a fixed order of P . In order to see it, we consider the following diagram, The partition function associated to the diagram (3.57) is given bỹ We observed that the expansion of (3.58) by T terminates for a fixed ν. More precisely for a fixed ν we observed that (3.58) is a polynomial of T with the lowest order T −|ν| and the highest order T |ν| . We checked this feature for |ν| ≤ 4 with the sum taken until |µ| = 9. Therefore we conjecture that the expansion of (3.58) by T terminates at the order |ν| and it is enough to compute the summation until |µ| = 2|ν|. Namely instead of (3.58) we can useZ Then the number of the Young diagrams µ over which we need to sum for a fixed |ν| becomes finite. Therefore we can obtain the exact result after taking T → 1 for coefficient of P |ν| when we perform the summation of µ which satisfies |µ| ≤ 2|ν|. With this prescription it is possible to compute the partition function for the 5-brane web in (3.54) and it is given by In this case there is no extra factor as the diagram in (3.54) does not have parallel external legs. We first work out the perturbative part of (3.61) or (3.60) which is obtained by taking the limit Q → 0. Then (3.61) splits into two factors Since (3.12), (3.35) have already reproduced the square root of the root contribution to the perturbative part for the pure SO(4) gauge theory and the pure SO(6) gauge theory respectively, we assumed that (3.12) and (3.35) are exact results. From (3.66) one can see that the order of T is always less than or equal to the order of P . This gives another support for the the conjecture on the restriction of the Young diagram summation (3.59) since the restriction also implies that the order of T is less than or equal to that of P . Taking the limit T → 1 and substituting the Coulomb branch moduli of the SO(5) into (3.65) and (3.66), we obtain (3.69) We checked that the one-instanton part of (3.69) agrees with the known SO(5) oneinstanton and two-instanton partition functions computed from (B.12) until the order P 2 Q 2 .
Pure SO(7) gauge theory. SO(7) gauge theories are more interesting for us, as we will see later that it may be related to the construction of the pure G 2 gauge theory. The 5-brane web of the pure SO(7) gauge theory is given by The relation between the Coulomb branch moduli A 1 = e −a 1 , A 2 = e −a 2 , A 3 = e −a 3 and the Kähler parameters is On the other hand, the instanton fugacity is obtained in the same way as before and it is given by Applying the topological vertex and the O-vertex with the prescription (3.59) to the web diagram in (3.70), the partition function becomes (3.74) The perturbative part of (3.74) or (3.73) is obtained by taking the limit Q → 0, Similar to the case of the pure SO(5) gauge theory, the expression (3.75) before taking the limit T → 1 and restricting the Young diagram in summation is the product of (3.33) and (3.49). Hence instead of computing (3.75) directly we here simply reuse the result (3.35) and (3.50), and each factor now becomes  (7) gauge theory is obtained by taking the T → 1 limit and we have We checked that the one-instanton part of (3.79) agrees with the known SO(7) oneinstanton partition function calculated from the integral (B.14) until the order P 2 Q 2 R 2 . We also checked that the two-instaton partition function computed from (B.12) for the pure SO(7) gauge theory matches with the result computed from the topological vertex formalism with O-vertex (3.79) as an expansion until the order P 2 Q 2 R 2 .
3.3 Pure G 2 gauge theory 5-brane webs with an O5-plane yield not only SO(2N + 1) gauge theories but also gauge theories with an exceptional gauge group G 2 [18]. The construction utilizes the fact that the pure G 2 gauge theory may be realized as a low energy theory on a Higgs branch of the SO(7) gauge theory with a hypermultiplet in the spinor representation. In fact the spinor matter can be introduced into 5-brane webs with an O5-plane or an O5-plane [35]. Therefore Higgsing the 5-brane web for the SO (7) gauge theory with spinor matter should yield a diagram for the pure G 2 gauge theory. In [18], two types of the 5-brane web diagrams were proposed for the pure G 2 gauge theory. It turns out that it is possible to apply the formalism of [1] to one of the diagrams and the partition function for the pure G 2 gauge theory has been computed using the topological vertex in [18]. When we use the O-vertex it is possible to compute the partition function of the pure G 2 gauge theory from the other type of the diagrams. We will explicitly compute the partition function here and compare the result with the known expression of the partition function of the pure G 2 gauge theory. Let us first recall the argument presented in [18]. The brane diagram of the SO(7) theory with a hypermultiplet in the spinor representation is given by [35] (3.80) The presence of two parallel external (2, 1) 5-branes implies an SU(2) flavor symmetry associated to the spinor matter. To perform the Higgsing which breaks the SU(2) one needs to put the (2, 1) 5-branes on top of each other. For that we perform a generalized flop transition [30] which yields Then we can tune the length of 5-branes to put the two parallel external (2, 1) 5-branes on top of each other. Supposing that a 7-brane ends on each external 5-brane, we can remove a piece of a (2, 1) 5-brane between two (2, 1) 7-branes. This corresponds to the Higgsing and sending the piece of the 5-brane far away giving rises to a diagram of the pure G 2 gauge theory given by (3.83) where the 7-brane on which the (2, 1) 5-branes end is depicted as a black dot.
We then consider applying the topological vertex as well as the O-vertex to the diagram in (3.83). At the practical level, we use the following diagram for computation.
The constraint that the two (2, 1) 5-branes with empty Young diagrams assigned coincide at the infinity forces the Kähler parameters to satisfy Q 2 = Q 2 . The Coulomb branch moduli of the pure G 2 gauge theory can be parametrized by On the other hand the instanton fugacity can be obtained by extrapolating the external 5-branes to the orientifold and then measuring the length between the intersection points. The instanton fugacity q is given by With the parameterization (3.85) and (3.86) we can compute the partition function of the pure G 2 gauge theory from the diagram in (3.84). For the part where the (2, 1) 5-brane intersects with the orientifold we can use the standard O-vertex. For the right part where 5-branes intersect with the orientifold it is possible to apply the original formalism in [1]. Then the full partition function obtained by applying the topological vertex as well as the O-vertex to the diagram in (3.84) is given by The perturbative part can be obtained by taking the limit Q → 0 since Q is proportional to the instanton fugacity. In this limit, the web diagram (3.84) splits into the left part and the right part. The left part is identical to the left part of the SO(6) diagram or that of the SO(7) diagram and the right part is the same as the half of the web diagram for a pure SU(4) gauge theory. Hence we can make use of (3.33) with the change R → Q 2 , Q → Q 1 , P → Q 2 and the partition function from the left part is given by For the right part, it is possible to sum over the Young diagrams. In terms of the SU(4), the simple roots of SU(4) correspond to the Kähler parameter Q 1 , Q 2 , Q 1 Q 2 2 . Then the partition function from the right part yields The total perturbative part becomes the product of (3.88) and (3.89) and in addition to α 1 , α 2 . Associating the Kähler parameters Q 1 and Q 2 to the simple roots α 1 and α 2 respectively, we can see that (3.90) or (3.91) reproduces the root contribution to the perturbative part of the partition function. Let us move on to the instanton part. The instanton part is given by removing the perturbative contribution from the full partition function It is possible to compare (3.93) with the result in the literature. For the one-instanton part, the universal formula (B.14) gives where we can see that the denominator is completely determined by the set of positive long roots, {α 1 , α 1 + 3α 2 , 2α 1 + 3α 2 }. The two-instanton part may be computed by using the blow up formula in [42]. Then we found a perfect match between (3.93) and the twoinstanton part obtained from the blow up formula as well as the one-instanton part (3.94) until the order of Q 5 1 Q 5 2 . In the diagram in (3.80), a (2, 1) 5-brane was introduced on the right side. It is also possible to realize spinor matter by having a (1, 1) 5-brane on the left side of the diagram. When the diagram is reflected along a vertical axis, the following diagram (3.95) also realizes the SO(7) gauge theory with a hypermultiplet in the spinor representation. One can perform the Higgsing to the diagram, which gives a diagram of the pure G 2 gauge theory [33] (3.96) It is also possible to apply the topological vertex and the O-vertex to the diagram in (3.96). For that we use the following diagram, We need to take the limit P → 1 at the end of the computation. The parameterization of the Coulomb branch moduli is the same as (3.85). The instanton fugacity is also given by (3.86). Before taking the limit P → 1, the computational result by the topological vertex applied to the diagram (3.97) gives rise tõ Since we take P → 1 at the end, it seems that we need to sum over ν to obtain a result at each order of Q a Q b 1 Q c 2 . However, as observed in section 3.2, it is enough to sum over ν until the order |ν| = 2|α| for a fixed |α|. Namely the partition function can be computed by The perturbative part is obtained by applying the limit Q → 0 to (3.99). The diagram splits into the left part and the right part. The left half of the diagram before the limit P → 1 is the left part of the pure SO(8) diagram whereas the right part diagram is the half of the diagram of an SU(3) gauge theory with a flavor. The partition function from the left part of the pure SO(8) diagram is given in (3.50) and it becomes in this casẽ where we identified Q = Q 2 /P, R = Q 1 , T = Q 2 and we assumed that (3.50) gives the exact result. Then P → 1 limit gives On other hand the square root of the perturbative part of the partition function of an SU(3) gauge theory with a flavor is given by

) are Kähler parameters for the Coulomb branch moduli of the SU(3) and M is the mass parameter for the flavor. Identifying
(3.103) Combining (3.101) with (3.103), we obtain which reproduces the perturbative part of the pure G 2 partition function (3.90). The instanton part of the pure G 2 partition function is given by removing the perturbative part (3.104) from (3.99), We checked that both the one-instanton partition function and the two-instanton partition function extracted out from the above expression agree with the result (3.94) and the blow up result until the order Q 2 1 Q 2 2 .

Pure SU(3) gauge theory with the Chern-Simons level 9
Lastly we compute the partition function of the 5d pure SU(3) gauge theory with the Chern-Simons level 9 found in [31,32]. This theory may be obtained by a circle compactification with a twist of the 6d pure SU(3) gauge theory with a tensor multiplet [38]. A 5-brane web diagram for the theory has been also found in [33] and it is given by In (3.107) the dotted lines represent O5-planes which is the S-dual of the ON-planes in (3.106). Here the vertical direction becomes periodic and it implies that this theory is a marginal theory whose UV completion is related to a 6d theory. Using the O-vertex, it is also possible to compute the partition function of the pure SU(3) gauge theory with the CS level 9 from the diagram (3.107). For that, first note that the upper half or the lower half of the diagram (3.107) is the same as the lower half of the diagram (3.96) for the pure G 2 gauge theory. Hence we can compute the partition function in the same way as we have done in the latter part of section 3.3. Namely for applying the topological vertex and O-vertex we use the following diagram, where we need again to take the P → 1 limit at the end. Q 1 and Q 2 are related to the Coulomb branch moduli A 1 , A 2 , A 3 (A 1 A 2 A 3 = 1) of SU(3) by One can see that (3.113) is exactly the same as (2.35) and the limit Q → ∞ with P Q fixed yields the O-vertex W ν . Therefore we can use W ν for (3.111). Then by applying the topological vertex and the O-vertex, we find that the partition function of the diagram (3.108) is given by where the summation of ν and µ is restricted since it gives the exact result for a fixed order of |α| and |γ| as argued in section 3.2. One can extract out the perturbative part in the sense of SU(3) by setting Q → 0 for (3.114), which forces β, δ and ζ in the above summation to be ∅. Then the diagram (3.107) splits into the upper part and the lower part, and the two diagrams are related to each other by the reflection with respect to a horizontal axis. Hence the partition function from the upper part will be the same as that from the lower part, and the perturbative part of the partition function is given by (3.115) We checked until the order of Q 2 1 Q 2 2 that the above expression agrees with This is the standard perturbative part of the partition function of a pure SU(3) gauge theory. Let us then evaluate the full partition function, There are several sub-diagrams in this theory that can be relatively easily computed. The first one is the following half diagram, From the periodicity in the vertical direction, one would expect the exact partition function of the above diagram to be given by an infinite product of plethystic exponentials. For example we checked with our explicit computation that the partition function from the diagram (3.117) gives We note that the above expression can alternatively be reproduced from the P → 1 limit of the positive root system of affine D 4 Lie algebra. In general the positive roots of an affine Lie algebraĝ associated to a Lie algebra g is given bŷ where ∆ is the set of the roots of g, ∆ + is the set of the positive roots of g and δ is the null (or imaginary) root. In this case, the Kähler parameters for the simple roots of SO (8) are Q 2 P −1 , Q, Q 2 P −1 , Q 2 P and the Kähler parameter for the affine node is Q 2 P . From this we can see that the Kähler parameter for the null root becomes Q 2 Q 4 2 . Then, until the order Q 2 Q 2 2 , the positive roots with n = 0 gives while the positives roots with n = 1 gives after taking the P → 1 limit, and their product indeed reproduces (3.118).
To have more insight let us consider the following diagram, where there is no restriction on the Kähler parameters R i (i = 1, · · · , 5). The application of the topological vertex and the O-vertex to the diagram (3.122) gives ν,µ,α,β,γ (3.123) When we associate R 4 , R 3 , R 2 , R 2 R 2 1 for the simple roots of SO(8) and R 4 R 2 5 for the affine node of the Dynkin diagram of the affine D 4 Lie algebra, we can see that (3.123) reproduces the contribution of the positive roots of the affine D 4 Lie algebra for n = 0, 1 except for the terms proportional to 5 i=1 R 2 i which is the Kähler parameter corresponding to the imaginary root. We expect that the property that the contribution is associated to the positive roots of the affine D 4 Lie algebra holds for higher orders except for terms associated 2 k for k = 1, 2, · · · when we take the limit to the diagram (3.117).
The presence of the affine D 4 structure is in fact expected from the two orientifolds. When the two orientifolds in (3.117) are decoupled, then the diagram gives a half of the diagram for a pure SU(4) gauge theory. When we introduce one orientifold, then the diagram is a half of the diagram of the pure SO(8) gauge theory and the change of the Dynkin diagram from SU(4) to SO(8) is given by (3.124) Hence having another orientifold further changes the Dynkin diagram [43], (3.125) which is nothing but the Dynkin diagram of the affine D 4 Lie algebra.
As for the second sub-web let us consider the right half of the diagram (3.108), which is given by Taking into account the mirror images, this diagram is like an infinitely long strip diagram. The partition function for the diagram (3.126) becomes Here the Young diagram ζ is assigned to the line which connects the diagram back to the original one through the orientifold. The partition function of the form (3.127) has appeared in [44][45][46] and it is given by (3.128) accounts for strings wound around the periodic direction. Lastly we compute the full partition function. Until some orders of the parameters the explicit form of the partition function is given by where · · · stands for higher order terms. Unfortunately, not much is known about the partition function of the pure SU(3) gauge theory with the CS level 9. It has been argued in [38] that the theory is obtained by a circle compactification with a twist of the 6d pure SU(3) gaue theory with a tensor multiplet. The elliptic genus of the 6d SU(3) strings has been also computed in [47,48], but its relation with the partition function of the 5d pure SU(3) gauge theory with the CS level 9 remains unclear at the moment. In particular, one can see from the brane diagram (3.108) that there are only three Kähler parameters in the 5d theory, while there are 4 parameters (including the string parameter that weights different number of strings) in the 6d case. We leave the comparison between these partition functions after a proper twisting to a future work. We can also obtain the plethystic exponential form of (3.114) and it is given by (3.130) From the expression (3.130), it is possible to extract the Gopakumar-Vafa (GV) invariants. The GV invariant n g β , which was originally introduced in the series of papers [49][50][51][52], is defined so that the partition can be expressed as where q =: e igs , X is the corresponding Calabi-Yau manifold and Q β is the Kähler parameter for a curve β ∈ H 2 (X, Z). These topological invariants are related to the number of BPS states in the underlining theory. Then comparing (3.131) with (3.130), one can obtain the GV invariants for various lines in the brane web. The partition function (3.114) is the one we obtain when we apply the topological vertex and the O-vertex to the diagram (3.108) or (3.107). To obtain the Nekrasov partition function of the pure SU(3) gauge theory with the CS level 9, we need to remove a factor which does not depend on the Coulomb branch moduli defined in (3.109) [12,[39][40][41]. Therefore the Nekrasov partition function of the pure SU(3) gauge theory with the CS level 9 iŝ , and the extra factor is (3.133) We remark that the instaton part of the partition function is expected to be symmetric under the Weyl group of SU (3). Until the order we computed we can for example see a part of the symmetry from the part (3.134) in (3.132), which is symmetric under the exchange A 1 ↔ A 2 4 .

O-vertex as a vertex operator
The topological vertex computation may be rephrased by expectation values of some vertex operators [5,34]. In this section we propose a vertex operator corresponding to the O-vertex introduced in section 2 along the line of the vertex operator formalism. Some technical details of the vertex operator formalism are summarized in appendix B.2.

O-vertex operator
We consider reformulating the computation using the O-vertex obtained in section 2 by using operators and their expectation values. This may be regarded as a first step to develop a purely analytic method to compute SO(N ) and G 2 partition functions in a closed form at each order of the instanton number. More details will be presented in a future work [53]. Let us try to express V µ as an expectation value of a vertex operator O(q), The ket state |ν here is the fermion basis labeled by the Frobenius coordinates of ν (see appendix B.2 for more details). It is straightforward to obtain the operator O(P, q) as a form in the expansion of the fermion basis. Since the fermion basis is orthonormal ν | ν = δ ν ν , one way to express the operator O(P, q) is given by where (α 1 , α 2 , · · · , α s |β 1 , β 2 , · · · , β s ) is the Frobenius coordinate of a Young diagram µ (see Figure 4). For example, the expression of the operator O(P, q) until the order P 4 is given by from the explicit form of V ν , which can be computed by (2.27). In fact one can infer a candidate for another form of the vertex operator O(P, q) which satisfies (4.1). In order to see that, let us compute the perturbative part of the pure SO(2N + 2) gauge theory from the O-vertex. We focus on the contribution from the left half of the 5-brane web diagram of the pure SO(2N + 2) gauge theory as it contains the O-vertex V ν . The partition function from the left half of the diagram contains the square root of the perturbative part of the partition function of a pure SU(N + 1) gauge theory. From the topological vertex computation, it is possible to factor out such a part by using the identity (B.33) and the remaining contribution is given by (4.6) It is possible to write (4.5) in terms of the expectation value of vertex operators. First note that the skew Schur function may be written by where V − ( x) is defined in (B.19). Then using (4.1) and (4.7), (4.5) can be written as where we used the completeness of the Frobenius basis to sum over the Young diagrams from the second line to the third line. Since (4.8) gives a part of the perturbative part of the partition function of the pure SO(2N + 2) gauge theory except for the pure SU(N + 1) part, (4.8) should be equal to Then comparing (4.8) with (4.9), we find that a candidate for another form of the vertex operator O(P, q) in the bosonic basis may be given by (4.10) Indeed we can see that inserting (4.10) into (4.8) yields and that J n |0 = 0| J −n = 0 for n > 0. Similarly, we can associate a vertex operator to the O-vertex W ν in (2.36). Note that the partition function from the right half of the 5-brane web diagram for the pure SO(2N + 2) gauge theory is given by (4.13) where we again factored out the square root of the perturbative part of the partition function of a pure SU(N + 1) gauge theory. Since W µ comes with a factor q κ(µ t ) 2 , it will be useful to define a vertex operatorÕ(P, q) by (4.14) From the relation (2.42), the righthand side of (4.14) is ThereforeÕ(P, q) may be identified with O(P, q).

Towards refinement of O-vertex
So far we have focused on the unrefined cases where the O-vertices V ν , W ν and the vertex operator O(P, q) depends only on q. Let us see how the vertex operator may be modified in the refined case. We here assume that there exist vertex functions V ν (t, q), W ν (t, q) associated with the intersection between an O5-plane and a 5-brane and also the partition function from a diagram away from the orientifold can be computed from the standard refined topological vertex. This refined version of the topological vertex is known to take the form [5] C µνλ (t, q) = q t ||ν|| 2 +||λ|| 2 2 t κ(ν) where ||λ|| 2 = i λ 2 i for λ = {λ i } and P λ is the Macdonald function with the normalization such that To use the refined topological vertex, we need to choose a preferred direction and we assign the refined topological vertex (4.16) so that λ is the preferred direction. The framing factor is also refined [5,54] and we assigñ for non-preferred directions and for preferred directions. Then, suppose that the refined version of the O-vertex V ν (q) is given by V ν (t, q), the partition function corresponding to (4.5) will become 5 From the localization result with the two Ω-deformation parameters turned on, (4.20) should be equal to Then from the comparison of (4.20) with (4.22), we obtain 1−q n J −n yields polynomials generated by (1 + N i=1Q n i ) for each order of P 2k . Also each coefficient of P 2k in (4.22) can be also written by a polynomial of (1+ N i=1Q n i ). We can then compare the two expressions to determine O(P, t, q). Until the orders we computed we have enough equations to determine the parameters for O(P, t, q).
We remark that in the unrefined case, the vertex operators for V ν can be the same as that for W ν q κ(ν t ) 2 , i.e.Õ(P, q) = O(P, q). In the refined case, there is one more step we need 5 The direct application of the refined topological vertex gives a factor t q |ν| 2 and we defined Vν (t, q) with the factor included.
to take. Note that the perturbative part (B.2) we ought to reproduce in the topological vertex formalism has two parts. The perturbative part we use for obtaining the vertex operator O(P, t, q) from the left part is given by (4.22) and this comes from the the first term in the exp in (B.2). The the partition function from the right half of the diagram needs to reproduce the second term in (B.2), namely P.E.
On the other hand the partition function computed by applying the refined topological vertex to the right half of the diragram with the factors for the perturabtive part of the pure SU(N + 1) gauge theory removed gives We also included the factor t − ||ν|| 2 2 q ||ν t || 2 2 q t − |ν| 2 which comes from the refined topological vertex computation in the definition of W ν (t, q). Then (4.25) has the same form as (4.20) but the final result obtained from (4.25) needs to be equal to (4.24), rather than (4.22). Note that (4.24) is reproduced by changing P into P t q 1 2 in (4.22). Hence a vertex operator associated to W ν (P, t, q) is expected to be given byÕ(P, t, q) = O P t q 1 2 , t, q . More details and consistency checks of this formulation will be presented in our sequel of this paper [53].

Conclusion
In this paper, we introduced a new type of the topological vertex called O-vertex, which is associated to the intersection point between an O5 − -plane, (2, 1) (or (2, −1)) 5-brane and an O5 + -plane, by making use of the formalism in [1]. The O-vertex is labeled by a Young diagram assigned to the (2, 1) (or (2, −1)) 5-brane. With the O-vertex we computed the partition functions of the pure SO(N ) (N = 4, 5, 6, 7, 8) gauge theories and also of the pure G 2 gauge theory. The comparison between the results obtained by the O-vertex and the known results showed the perfect agreement until some orders for the perturbative part, the one-instanton part and the two-instanton part, which legitimates our proposal. We then applied the O-vertex to the 5-brane web diagram of the pure SU(3) gauge theory with the CS level 9 and computed its partition function. It would be interesting to perform more consistency checks for the result of the pure SU(3) gauge theory with the CS level 9. We also presented the GV-invariants of low orders. Furthermore we also provided a vertex operator formalism for the O-vertex. We proposed an explicit expression for the vertex operator associated to the O-vertex.
There are two crucial steps in the study of the O-vertex and the instanton partition function in the future. One is to find a closed form for the O-vertex. The O-vertex V ν we proposed is given by the sum of finite terms for each order of the associated Kähler parameter. Since it is give by the sum of finite terms, we can compute the O-vertex exactly at each order. However it would be still interesting to find a closed formula for the O-vertex. We also proposed a candidate in the form of the vertex operator in section 4, but we do not have a proof of this proposal at the moment. The polynomial P ν appearing in the O-vertex has various properties as observed in appendix A. It will also be interesting to prove these observed properties. The link of the O-vertex with the dimer model and the melting crystal [34,[55][56][57][58][59] will also be a topic to work on in the future.
The other is to establish the refinement of this generalized topological vertex formalism. By achieving that, we can start to discuss the relation with the formalism constructed in the S-dual setup [20,26], and to derive the qq-characters [60-69] 6 associated to BCD-type gauge theories. The fundamental qq-characters of BCD-type gauge theories was discussed in a recent work [70], and unlike the beautiful results obtained for A-type gauge theories, they contain infinite number of terms and there is no known closed form for these qq-characters. Since the topological vertex formalism computes the partition function in a different basis (labeled by Young diagrams), it might simplify the expression of the qq-characters beyond the A-type gauge group. The relation with the blow-up equation [48,[71][72][73][74], which requires the refinement, would be also of great interest in the current context.
Then we observe that the polynomial P ν (q) satisfies the following properties, • P ν (q) is a polynomial of q of degree at most m(|ν|) = n(n+1) 2 for n = |ν| 2 and is zero for |ν| odd.
• When we denote then we have All coefficients a i seem to have the same sign, and the overall sign is determined from the property described below.
• As for the sign of P ν , when we move a box in the i-th line of ν to the (i + j)-th line to form a new Young diagram ν , then the sign of P ν differs from that of P ν by (−1) j .
(A. 8) In general W ν is related to V ν by (2.42) as shown in (2.41).

B Some useful formulae
In this appendix, we summarize some useful formulae that we make use of in the computations in this paper.

B.1 Nekrasov partition functions
The partition functions of certain 5d N = 1 supersymmetric gauge theories may be computed by the localization method. It consists of two factors, the perturbative part and the instanton part. The perturbative part of a pure gauge theory with a gauge group G is in general given by and ∆ + is the set of positive roots of the Lie algebra g of G and a = (a 1 , · · · , a rank(G) ) is the Coulomb branch moduli in the Cartan subalgebra. q, t are related to the Ω-deformation parameters 1 , 2 by q = e − 1 , t = e 2 . The unrefined case corresponds to q = t.
On the other hand, the partition function of the instanton part is more involved. The instanton part of the pure SU(N ) gauge theory with the zero CS level is given by where the Nekrasov factor N λν (a; 1 , 2 ) is given by [8,75,76] N λν (a; 1 , 2 ) := (B.9) The instanton part of the partition function of the pure SO(2N + δ) gauge theory for δ = 0, 1 is also given by a contour integral. The Losev-Moore-Nekrasov-Shatashvili (LMNS) integrand of the integral for the k-instanton partition function of the pure SO(2N +δ) gauge theory is [77][78][79][80][81] Z SO(2N +δ) k Then the instanton partition function is found via the contour integral The contour integral (B.12) may be evaluated by the Jeffrey-Kirwan residues [82][83][84]. Or alternatively one can choose the contour in the following way in the unrefined cases with 1 + 2 = 0. We first assume that Im( 1 ) = Im( 2 ) = δ, + = iδ ∈ iR and evaluate the contour integral by picking up the poles in the upper half plane. Then we take the limit δ → 0 at the end. As argued in [77,79], the poles may be labeled by colored Young diagrams in general.
There is in fact a subtlety for the instanton partition function computed by the localization method. The instanton partition function may contain a factor which does not depend on Coulomb branch moduli In that case, one needs to remove the extra factor Z loc, extra from the instanton partition function Z loc, inst to obtain the correct instanton partition function of a UV complete 5d theory [12,[39][40][41]83]. Namely we consider Z loc, inst = Z loc, inst /Z loc, extra . (B.13) Similarly we also need to remove an extra factor from the instanton partition function computed by the topological vertex. We see that this indeed happens for the pure SO(4) gauge theory in section 3.1.1.
As for the one-instanton partition function there is a universal formula for an arbitrary gauge group G and it is given by [85][86][87][88][89], γ∈∆ l e (h ∨ −1)γ/2 (1 − e ( 1 + 2 )+γ )(e γ/2 − e −γ/2 ) α,γ ∨ ·α=1 (e α/2 − e −α/2 ) , (B.14) where ∆ l is the set of long roots in the Lie algebra, g of G, and for each of simple root α i , e α i here is identified with the Coulomb branch moduli A i . This formula will be repeatedly used in the consistency check in this article. Furthermore, the higher-instanton partition function can be obtained using the blow up formula in [42,74]. Again, if the partition function contains an extra factor which is independent of the Coulomb branch moduli, one needs to remove the extra factor to obtain the correct instanton partition function.