Topological vertex for 6d SCFTs with $\mathbb{Z}_2$-twist

We compute the partition function for 6d $\mathcal{N}=1$ $SO(2N)$ gauge theories compactified on a circle with $\mathbb{Z}_2$ outer automorphism twist. We perform the computation based on 5-brane webs with two O5-planes using topological vertex with two O5-planes. As representative examples, we consider 6d $SO(8)$ and $SU(3)$ gauge theories with $\mathbb{Z}_2$ twist. We confirm that these partition functions obtained from the topological vertex with O5-planes indeed agree with the elliptic genus computations.


Introduction
In [1], it was proposed that 5-brane webs [2,3] for 6d N = (1, 0) superconformal field theories (SCFTs) with SO(N ) gauge symmetry coupled to a tensor multiplet on a circle [4]. Such 5-brane webs are constructed with two O5-planes whose separation can be naturally identified with the Kaluza-Klein (KK) momentum [5,6]. Given a 5-brane web, one can compute the prepotential [7,8] for the corresponding theories on Coulomb branch. It was checked [1] that the prepotentials obtained from the proposed 5-branes webs indeed agree with the prepotentials that one can compute the triple intersection numbers in their geometric descriptions [9,10].
Of particular significance to this 5-brane construction of 6d SCFTs with SO(N ) gauge symmetry is a realization of such 6d SCFTs with Z 2 twist, providing a new perspective on RG flows on Higgs branches of D-type conformal matters [11,12]. With non-zero holonomies turned on, one introduces a light charged scalar mode carrying non-zero KK momentum along the 6d circle, giving rise to new Higgs branch associated with a vev of the light mode. In particular, RG flows from Higgsings on two O5-planes leads to 5-brane webs for twisted compactifications of 6d theories.
In this paper, we compute R 4 × T 2 partition functions for 6d theories with Z 2 twist, based on 5-brane configurations proposed in [1]. As representative examples, we consider two 6d SCFTs with Z 2 twist: one is the 6d SO (8) gauge theory with Z 2 twist and the other is the 6d SU (3) theory with Z 2 twist. The SO (8) theory is obtained through the Higgsing sequence from the 6d SO (10) gauge theory with two hypermultiplets in the fundamental representation (flavors) to the SO(9) gauge theory with a flavor and then to the SO(8) theory with Z 2 twist, while the SU (3) theory is obtained through the Higgsing sequence from 6d G 2 gauge theory with a flavor to the SU (3) theory with Z 2 twist, whose 5-brane configuration yields 5d SU (3) 9 gauge theory with the Chern-Simons level 9, as predicted in [9,13,14].
The organization of the paper is as follows. In section 2, we discuss the construction of 5-brane webs for the 6d SO (8) gauge theory with Z 2 twist and compute the partition function using topological vertex based on 5-brane webs as well as using the ADHM method. In a similar way, the 6d SU (3) gauge theory with Z 2 twist is discussed in section 3. We then summarize the results and discuss some subtle issues in section 4. In appendices A and B, we discuss properties of Z 2 twisting of 6d theories and the perturbative partition function from the perspective of twisted affine Lie algebras, and also provide various identities that are useful in actual computations.
While completing this paper, we became aware of [22] which has some overlap with this paper.

SO(8) theory with Z twist
We first consider twisted compactification of 6d pure SO (8) gauge theory on −4 curve. As proposed in [1], a 5-brane configuration for the twisted compactification of the 6d pure SO (8) gauge theory can be obtained from a Higgsing of 6d SO(10) gauge theory with two hypermultiplets in the fundamental representation (SO(10)+2F) on −4 curves. Before we discuss twisting, let us recall the standard Higgsing, which is to give the vev's to two fundamental scalars charged under a color D5-brane. This leads to the 6d SO (8) gauge theory on −4 curve. To realize twisted compactification, one instead gives independent vev's to two individual scalars. Namely, first give a vev to one fundamental scalar to yield the 6d SO (9) gauge theory with a fundamental hypermultiplet (SO(9) + 1F), and then give a different vev to another fundamental Figure 1: A 5-brane web for 6d SO(10) theory with two fundamentals. Two fundamental hypermultiplets are represented by two external D5-branes ending on a D7-brane (black dot) and the monodromy cuts of D7-branes point the outward directions.
scalar carrying a unit KK momentum. This leads to the 6d SO (8) gauge theory on −4 curve with Z 2 twist. In this section, we first review construction of 5-brane configuration leading to the twisted compactification of the 6d pure SO (8) gauge theory on −4 curve, and then compute the partition function of the twisted SO (8) gauge theory on R 4 × T 2 , using topological vertex [15] on a 5-brane web with two O5-planes [6]. We compare our result with Higgsing or twisting of the elliptic genus from the ADHM construction using localization technique, introduced in [23, 24].

5-brane web for 6d SO(8) gauge theory with Z 2 twist
Let us begin with a 5-brane configuration for the 6d SO(10) gauge theory with two fundamental hypermultiplets on −4 curve. It is depicted in Figure 1, where there are two O5-planes whose separation naturally gives a compactification direction associated with 6d circle. Two fundamental hypermultiplets are denoted by two D5branes ending on D7-branes. Their masses are m 1 , m 2 . The W-bosons W i of the theory are denoted by wiggly lines connecting color D5-branes in Figure 1. The masses of the W-bosons are given by where φ i (i = 1, · · · , 5) are the Coulomb branch vev and φ 0 is the tensor branch vev. As it is a configuration for the 6d theory on a circle, one can see that the W-bosons masses are consistent with the affine Cartan matrix 1 of untwisted affine Lie algebra plane. The half D7-branes and corresponding half monodromy cuts are denoted by blue dots and blue zigzag lines. The half D5brane is denoted by violet line. (d) A 5-brane configuration after Higgsing away two half D5-branes between D7-branes and then taking two half D7-branes to the left and right infinities, respectively.
As discussed, to twist, we perform the Higgsing such that we first bring a D7brane down to one of the O5-planes to obtain SO(9) gauge theory with a fundamental as depicted in Figure 2, and then push up the remaining D7-brane to the other O5plane to give a different vev to the remaining scalar, as in Figure 3. More precisely, the Higgsing from a SO(10) gauge theory with a flavor to a SO(9) gauge theory is achieved as follows: Through successive flop transitions, one can bring a D7brane near the bottom O5-plane as depicted in Figure 2(a). Then one brings the D7-brane inside the Coulomb branch of SO (10) gauge theory, where the D7-brane becomes floating as in Figure 2(b), since the flavor D5-brane is annihilated due to the Hanany-Witten transition. We give a vev to the flavor which locates the flavor 7-brane and a color D5-brane on an O5-plane. That is to set the masses of the flavor hypermultiplet and the associated W-boson φ 1 − φ 0 to zero, We note that when it is placed on an O5-plane as depicted in Figure 2(b) and Figure 2(c), a D7-brane (black dot) is split into two half D7-branes (blue dots), creating a half D5-brane between two half D7-branes stuck on an O5-plane, which hence turns the O5-plane between these split half D7-branes to an O5 − -plane [25][26][27][28]. In doing so, a new Higgs branch opens up such that two half color D5-branes (or a full color D5-brane) are suspended between two half D7-branes on the O5 − plane which can be Higgsed away, resulting in a 5-brane configuration for a SO (9) gauge theory. After moving half D7-branes away from each other to the opposite along the orientifold plane, one gets a 5-brane configuration with an O5-plane as in Figure 2(d). Figure 3(a) is the resulting 5-brane web diagram describing the 6d SO(9) gauge theory on a circle with a fundamental hypermultiplet of mass m 2 , implementing (2.3). Here we have redefined the scalars φ i to make the masses of the W-bosons form the affine Cartan matrix of untwisted affine Lie algebra B 4 , or more explicitly, We further Higgs the remaining fundamental hypermultiplet. As the 5-brane configuration given in Figure 3 has two different types of orientifold planes, O5and O5-planes, we have two ways of Higgsing the remaining hypermultiplet. In the previous Higgsing SO(10) + 2F → SO(9) + 1F, the Higgsing procedure converts an O5-plane to an O5-plane or vice versa. In a similar fashion, if one brings a flavor D5-brane (or equivalently a flavor D7-brane) down to the O5-plane, then it is to perform the same Higgsing, and hence it would yield a 5-brane web for a pure SO(8) theory on a circle. This is the standard Higgsing SO(9) + 1F → SO (8), hence corresponding to untwisted compactification SO(10) + 2F → SO(9) + 1F → SO (8). Indeed, shifting φ 3 → φ 3 + φ 4 straightforwardly yields the W-boson masses forming the affine Cartan matrix of untwisted affine Lie algebra D (1) 4 , as expected, Now, let us take the flavor D5-brane and one of color D5-branes to the O5plane instead. One can do the same kind of Higgsing as done for the previous the SO(10) theory to the SO(9) theory, which then makes a 5-brane configuration with two O5-planes. As discussed in [1], this is the Higgsing that one Higgses the SO(9) theory with one fundamental hypermultiplet by giving a vev to a scalar field carrying Kaluza-Klein momentum which leads to the SO(8) gauge theory with a Z 2 twist. Namely, we first revive the 6d circle radius R in Figure 3(a) by introducing Kaluza-Klein momentum, which can be done by redefining Kähler parameters to be where other parameters are unaltered in Figure 3(a). By successive flop transitions, the fundamental hypermultiplet can be placed near the O5-plane located on the top in Figure 3(a). This yields a 5-brane configuration in Figure 3(b). We then Higgs by setting The resulting 5-brane configuration for the 6d SO(8) gauge theory with a Z 2 twist is depicted in Figure 4. Here the W-boson masses are given by which form the affine Cartan matrix of twisted affine Lie algebra D 4 , (2.11)

Partition function from 5-brane webs
We now compute the partition function based on the 5-brane configuration for the 6d SO(8) gauge theory with a Z 2 twist, using topological vertex with an O5-plane [6]. With an O5-plane, we can easily realize a 5d SO(N ) or Sp(N ) gauge theory with hypermultiplets in the fundamental representation. For SO(N ) gauge theories, we can also depict 5-brane configurations for hypermultiplets in the spinor representation [29]. Our strategy to carry out topological vertex computation with O5-planes is first to introduce auxiliary spinor hypermultiplets and then to decouple them after the computation.
To introduce four auxiliary spinors, we start with a 5-brane configuration before twisting. Namely, we consider 5-brane web for 6d SO(10) gauge theory with two hypermultiplets in the fundamental representation as well as four auxiliary hypermultiplets in the spinor representation. One can imagine that given a 5-brane web for 6d SO(10) gauge theory with two fundamentals as in Figure 2(a), one introduces each spinor as a charge conserving distant 5-brane on the left and on the right of the main web for the SO(10) gauge theory. On the bottom O5-plane, they are a 5-brane of charge (2, −1) on the left and a 5-brane of charge (2, 1) on the right. On the top O5-plane, they are a 5-brane of charge (2, −1) on the left and a 5-brane of charge  Figure 6: A 5-brane web diagram for Z 2 twisted compactification of 6d SO (8) theory with four auxiliary spinors.
(2, −1) on the right. Higgsing toward SO(8) gauge theory with a twist is straightforward except there are additional half D5-branes introduced along O5-planes due to Hanany-Witten transitions when taking half D7-brane to infinity. For instance, in Figure 5, Higgsing from SO(10) gauge theory with a fundamental is depicted in the presence of two spinors, a distant 5-brane of charge (1, −1) due to the monodromy of D7-brane (fundamental hyper) on the left and another distant 5-brane of charge (2, 1) on the right of the bottom O5-plane. Figure 5(a) is a configuration when the Higgsing is performed, where there are two half D7-branes are on the orientifold plane. Unlike the previous case where we took each half D7-brane to the opposite directions to realize an O5-plane, this time, for computation ease, we take two half D7-branes to the same direction, to the left, to keep the orientifold plane an O5-plane. This is depicted in Figure 5(b), where half D5-branes are created due to the Hanany-Witten transitions. Taking the spinor 5-branes closer to the center of the O5-plane, they undergo "generalized flop" transitions [30] but still preserve the charge conservation as giving in Figure 5(c). Repeating the same procedure on the top O5-plane, one finds a 5-brane web for the 6d SO(8) gauge theory with four auxiliary spinors with a Z 2 twist, as in Figure 6. Again, to obtain the partition function for the 6d SO(8) gauge theory with a Z 2 twist, we first compute the partition function based on the 5-brane web for the 6d SO(8) gauge theory with four spinors with the twist in Figure 6, and then decouple the spinors by taking all the masses of the spinor matter to infinity.
To perform the topological vertex method with an O5-plane, we assign arrows, Young diagrams λ, µ i , ν i and Kähler parameters Q i on the edges as in Figure 7. The topological string partition function Z can be computed by evaluating the edge factor and the vertex factor: The Edge Factor is given by the product of Kähler parameters and the framing factor of the associated Young diagram λ = (λ 1 , λ 2 , · · · , λ (λ) ), where the framing factor is defined as Here, g = e − is the Ω-deformation parameter that is unrefined 1 = − 2 = , and λ t denotes the transposed Young diagram which corresponds to the conjugate partition of λ. The power of the framing factor n is defined with the incoming and outgoing edges connected to Q and their antisymmetric product, for instance, in Figure 7, The Vertex Factor is given by where λ, µ, ν are the Young diagrams assigned to three out-going edges connected to a given vertex, arranged in the clockwise order such that the last Young diagram ν is assigned to the preferred direction which is associated with instanton sum. We note that C λµν = C µνλ = C νλµ for unrefined vertex factors. When the orientation of an edge is reversed, the corresponding Young diagram is transposed. Here,Z ν (g) is defined byZ where s λ/η (x) is skew Shur functions and ρ = (− 1 2 , − 3 2 , − 5 2 , · · · ). When summing skew Shur functions, one needs to repeatedly use the Cauchy identities See also Appendix B for various other Cauchy identities and other related special functions. Now, we assign the Young diagrams and Kähler parameters to a 5-brane web diagram for the 6d SO(8) gauge theory with four auxiliary spinors as in Figure 8. We compute the partition function based on this 5-brane web with two O5-planes as follows: First we reflect the 5-brane web with respect to each O5-plane, which creates a mirror image of the original 5-brane web. We also cut the 5-brane configuration by half so that we can glue over the color D5-branes associated with Young diagrams µ 1 , µ 2 , µ 3 . As we have mirror images, we can choose a convenient fundamental 5brane configuration, which contains both the original and the mirror images such that when crossing the O5-plane, 5-brane webs are smoothly connected. Together with the mirror images, we then see that we can extract two pieces of strip diagrams given in Figure 9. We note here in Figure 9 that unlike the edge factors µ i following the conventional rule, the edge factors of ν i should be treated carefully. When ν i are reflected over the orientifold planes, they get the additional factor (−1) |ν i | f ±1 ν i , where ±1 is determined from the orientation of the edges. For the vertex factor, the assigned arrows associated with Young diagrams are reversed when they are reflected over, which makes the ordering of the corresponding Young diagrams counterclockwise. It also follows from the identity [6] Figure 8: 5-brane web diagram for SO (8) theory with a Z 2 twist where we first couple four auxiliary spinors and then later decouple all the spinors after obtaining the partition function. In this way, one can compute the partition function for 6d SO(8) theory with a Z 2 twist. Figure 9: Strip diagrams for 6d SO(8) gauge theory with Z 2 twist, which is coupled to 4 auxiliary spinors. They are the left part (a) and the right part (b) of Figure 8. We use mirror images to make each part as a form of a strip diagram.
that the reversal of the Young diagrams is translated into the transposition of the Young diagrams, and in turn, it is equivalent to changing the direction of the arrows. The topological string partition function is then obtained by gluing two strip diagrams in Figure 9, by summing up the Young diagrams µ i and ν i which are associated with the preferred direction where two strip parts are as follows: With the shorthand notation the left strip part, in Figure 9(a), is given by , (2.23) and the right strip part, in Figure 9(b), is given by . (2.24) As we are computing the partition function of a 6d theory on the Ω-background which is compactified on a circle with a twist, it is a partition function on R 4 × T 2 . Hence, it can be compared with the elliptic genus of 6d theories. To this end, we expand the partition function (2.22) as where Z pert is the perturbative part and Z n are the n-string elliptic genus, and u is the string fugacity which is given by a product of Kähler parameters. We proceed with the computation by expressing the Kähler parameters in terms of physical parameters. First we eliminate the dependence of φ 0 along the W-bosons. For that, we perform the following shifts where α i and Q B are the Kähler parameters assigned in Figure 8. Here, we have identified that 1/R as τ , since it parametrizes the KK-momentum. One can see the Cartan matrix of so(7) from the W-bosons in (2.27), which is the invariant subalgebra under order two outer automorphism of so(8) algebra. The string fugacity u is As the perturbative part of the partition function Z pert is the zeroth order in u, we obtain Z pert by setting µ i = ∅ in (2.22). It is convenient to separate out the spinor matter parts, as we will decouple them. To this end, we first sum over ν i and express the Kähler parameters in terms of the α i associated with the W-bosons, e.g., Then we see that Z pert is written as a function of g, α i and Q 1 , Q 3 , Q 5 , Q 7 that are the Kähler parameters associated with the spinor matter. Next, we take the plethystic logarithm, defined in Appendix B, and expand Z pert in terms of α i and Q i . The shift (2.27) then yields where PE stands for the plethystic exponential, and q = e −τ , the terms · · · are the terms depend on the spinor matter. Here the characters for the positive roots and the fundamental representation of so(7) algebra take the form χ so(7) We note that the · · · terms in (2.29) denote the terms that involve Q 1 , Q 3 , Q 5 and Q 7 which we will decouple. In the 5d limit where the KK-momentum becomes very large, the states with KK-momenta are truncated. Thus, only q 0 terms in Z pert survive. It is then easy to see that that Z pert involves the contribution from the positive roots of so (7) which is expected on the Coulomb branch where masses of W-bosons are all positive. This thus shows that the corresponding 5d theory is an SO(7) gauge theory, as expected. The partition function for 6d self-dual strings can be expanded in terms of the string fugacity where Z n is the n-string elliptic genus. Again, having in mind the decoupling of the spinor contributions, when summing over Young diagrams (2.22), we express it as a function of α i and Q 1 , Q 3 , Q 5 , Q 7 . To decouple the spinors, one needs to go through a flop transition, where masses of the spinors are proportional to We take each mass of the spinors to infinite or the corresponding Kähler parameters to zero while keeping α 1 , α 4 finite. So, after decoupling the spinors, we get the desired n-string elliptic genus.
To organize n-string elliptic genus, we expand Z n in terms of the Kähler parameters associated with the W-bosons, given in (2.27). In particular, using α 4 = q 1/2 (α 1 α 2 α 3 ) −1 , we expand Z n in terms of q and α 1 . The one-string elliptic genus Z 1 is expressed as The two-string elliptic genus Z 2 is rather complicated, so we write only the leading terms in α 1 : (2. 35) In the following subsections, we will compare this obtained result with the elliptic genus computations in two different ways: First, as explained through 5-brane webs, we perform the Higgsing SO(10) + 2F → SO(9) + 1F → SO (8) with Z 2 twist. Secondly, we apply the Z twist directly to the elliptic genus of the 6d SO(8) theory.

Partition function from the Higgsing sequence
We now compute the partition function of the SO(8) theory with Z 2 twist from direct Higgsing of the partition function for the 6d SO(10) gauge theory with two fundamental hypermultiplets. The perturbative part is given by are the positive and negative root parts of the character for the adjoint representation of so(10), respectively. More explicitly, for instance, χ so(10) ∆ + = α e −α for the positive roots α, which can be also expressed in terms of φ i in Figure 2(a).
To realize the Higgsing SO(10) + 2F → SO(9) + 1F , let us consider the contribution of the first hypermultiplet of mass m 1 to the perturbative part, which is given by where χ so (10)  10 is the character for the fundamental representation 10 of so (10), and M 1 = e −m 1 is the fugacity for the mass of the fundamental hypermultiplet m 1 . To Higgs the first hypermultiplet, we recover the affine node φ 0 using Figure 2(a) and impose (2.3). If one changes the variables as in Figure 3(a) and eliminate the affine node φ 0 , one can find that the perturbative part (2.38) up to the Cartan part, where χ so (9) ∆ ± are the positive and negative parts of the characters for the adjoint representation of so(9), respectively. This reflects that the theory becomes the SO(9) gauge theory as in Figure 3(a).
Higgsing the second hypermultiplet of mass m 2 needs more caution because of 1/R factor in Figure 3(b). From string tensions associated with the hypermultiplet in Figure 3(b), one can read that the contribution of the second hypermultiplet after Higgsing the first hypermultiplet becomes Figure 3(b). To Higgs the second hypermultiplet, we reintroduce the affine node φ 0 again and impose (2.9). Lastly, shifting variables   Table 1. The U (1) 1 and U (1) 2 are Cartans of SO(4) rotating transverse R 4 to worldsheet.
Using the field contents in Table 1, one can calculate the elliptic genus of strings in the 6d SO(10) theory using localization technique [24]. The n-string elliptic genus is expressed as where G is the gauge group on the 2d worldsheet. The contour integral is performed using the Jeffrey-Kirwan (JK) residue prescription in [31]. The one-loop determinant Z 1−loop is the product of the contributions from each multiplets. The N = (0, 2) vector multiplet contribution in Z 1−loop is given as a product of all the roots α of G of rank r, where η is the Dedekind eta function, θ 1 (x) is the Jacobi theta function. See Appendix B for the definition and some useful properties of the theta function. The N = (0, 2) chiral and Fermi multiplet contributions are given as where ρ is the weight vector of the representation of each multiplet in gauge and flavor symmetries.
In the case of the SO(10) theory, the 2d gauge group is Sp(n). Note that in orthogonal basis {e i } n i=1 of sp(n) algebra, the root vectors are given by ±e i ± e j , the weight vectors of the fundamental and the antisymmetric representations are ±e i and ±e i ± e j (i = j), respectively. Denote chemical potential parameters of sp(n) gauge algebra be u i and so (10), sp(1) 1 , sp(1) 2 flavor symmetry algebras be v i , m 1 and m 2 , respectively. Then the n-string elliptic genus from (2.41) is where ± = 1 ± 2 2 and we use shorthand notation To evaluate the integral (2.44), one needs to find residues of the integrand. However, not all poles contributes to elliptic genus. The contributing residues are called Jeffrey-Kirwan (JK) residue, determined as follows. Suppose the singular hyperplanes H n are r I=1 a (n) I u I +· · · = 0. Fix an auxiliary vector ζ in the space which contains vectors a (n) . If ζ can be written as linear combination of a (n 1 ) , · · · , a (n l ) with positive coefficients, then the singularities H 1 , · · · , H l contribute to the integral.
We explicitly calculate the one string and two string elliptic genus to compare with topological vertex calculation. For n = 1, the singular hyperplanes are If one takes an auxiliary vector ζ = (1), only singular hyperplanes +u 1 ± v J + + = 0 contribute to the elliptic genus. Using the one-string elliptic genus is For n = 2, the singular hyperplanes are for I = 1, 2 and J = 1, · · · , 5. We choose an auxiliary vector ζ as in Figure 10, so that the contributing poles are The residue around poles (2.49)(i) is The poles (2.49)(ii) give  We now Higgs this SO(10) elliptic genera. We only consider the unrefined case, The brane web construction gives hints for Higgsing procedure. In the elliptic genus calculated from the field theory construction, there are so(10) parameters v i . They are written in the orthogonal basis, so first convert it to fundamental weight basis (Dynkin basis) and recover an affine node: Note that v 2 , v 3 , v 4 form the orthogonal basis of so(7) algebra. Thus, by redefinition of the variables, the Higgsing sets the parameters as Substituting (2.56) into (2.47) and (2.53) and taking the unrefined limit 1 = − 2 , we find that the one-string elliptic genus is given by and the two-string elliptic genus is given by where (2.59) (2.60) Since v i are the so(7) parameters, we change the variables (2.62) and expand Z 1 and Z 2 in terms of q and α 1 to compare with topological vertex calculation in the previous subsection. By an explicit calculation, we see that two results are in complete agreement.

Partition function from a direct twisting of the SO(8) theory
There is yet another way to obtaining the partition function of the SO(8) theory with Z 2 twist from the field theory construction. The perturbative part of the partition function of the 6d SO(8) theory is given by where χ so (8) ∆ ± are the positive and negative parts of the character for the adjoint representation of so (8), for the fugacities x i of the orthonormal basis of so (8).
The elliptic genus of the 6d pure SO(8) theory can be obtained from the 2d worldsheet theory with matters in Table 1, with p = 0. As there is no Fermi multiplet Λ Q and ΛQ for p = 0, the n-string elliptic genus is given by v J are the chemical potentials for the SO(8) symmetry and they can be regarded as being in the orthogonal basis of SO (8).
Let the eight states in the fundamental representation be |±e i where i = 1, 2, 3, 4. They satisfy H j |±e j = ±δ ij |±e j for the orthonormal basis H j of the Cartan subalgebra of so (8). The order two outer automorphism σ of so(8) algebra exchanges the simple root α 3 and α 4 . In terms of the orthonormal basis of the Cartan subalgebra, σ(H 4 ) = −H 4 . Thus, the invariant elements are ±H 1 , ±H 2 , ±H 3 and H 4 − H 4 , corresponding to the states |±e 1 , |±e 2 , |±e 3 and |e 4 + |−e 4 . On the other hand, H 4 − (−H 4 ) has the eigenvalue −1, which corresponds to the state |e 4 − |−e 4 . Upon compactification, |e 4 and |−e 4 are identified so that the invariant states |±e 1 , |±e 2 , |±e 3 and |0 form the fundamental representation of the invariant algebra so(7) under the order two outer automorphism. The state of eigenvalue −1 becomes a singlet and gets additional half KK-momentum shift [32].
Consequently, we change (x 4 + 1 Figure 11: A 5-brane web for 6d G 2 gauge theory with one fundamental.
part becomes which agrees with the perturbative part obtained from 5-brane webs in (2.29). For the instanton partition function, we replace ±v 4 in (2.65) to τ 2 and 0. The elliptic genus is then expressed as .

(2.67)
For n = 1, the singular hyperplanes are Take an auxiliary vector ζ = (1). The residues of the last two singular hyperplanes are zero since θ 1 (kτ ) = 0 for k ∈ Z. Thus, only poles +u 1 + + ± v j = 0 contribute to the one string elliptic genus. One can find that the JK-residue sum for Z 1 with the unrefined limit 1 = − 2 gives the same expression as (2.57). A similar calculation for the n = 2 case leads to (2.58) as well.

SU (3) theory with Z 2 twist
The 6d SU (3) gauge theory with Z 2 twist can be obtained by Higgsing from the 6d G 2 gauge theory with a hypermultiplet in the fundamental representation. The Higgsing procedure can be realized through 5-brane webs as given in [1]. It is also worthy of noting that from the perspective of 5d SCFTs, it is 5d pure SU (3) gauge theory at the Chern-Simons level κ = 9 [9]. The corresponding 5-brane web is given in [14]. In order not to be confused with this 5d SU (3) theory, we denote it by SU (3) 9 . In this section, we compute the partition function for the 6d SU (3) gauge theory with Z 2 twist from 5-brane webs using topological vertex and also from performing a Higgsing on the elliptic genera of self-dual strings in the 6d G 2 theory.

5-brane web for 6d SU (3) gauge theory with Z 2 twist
We first review Type IIB 5-brane construction of the 6d pure SU (3) gauge theory with Z 2 twist. We start from the 6d G 2 gauge theory with one fundamental hypermultiplet, whose 5-brane configuration is given in Figure 11 3 , where the mass of the fundamental hyper is denoted by m and the masses of W-bosons W i are expressed in terms of the scalar fields as which form the affine Cartan matrix of untwisted affine Lie algebra G 2 . To obtain the SU (3) gauge theory with Z 2 twist from the G 2 gauge theory with a fundamental, we Higgs the hypermultiplet of the G 2 gauge theory by giving vevs to the scalar fields which carry KK-momentum. To this end, we redefine the scalars as while other parameters in Figure 11 remain unaltered. This recovers the 6d circle radius R in the brane configuration. The Higgsing procedure to the 6d SU (3) gauge theory with Z 2 twist is then similar to the Higgsing explained through Figure 2(b)-(d) in the previous section. We first bring the flavor D7-brane down to the lower O5-plane as depicted in Figure 12. The Higgsing condition is then given by 3) The first condition in (3.3) comes from the flavor D7-brane put on the lower O5-plane. As discussed in the previous section, this D7-brane is brought to the lower chamber of the Coulomb branch and then is put to the O5 − -plane on the bottom. It follows that this full D7-brane is split into two half D7-branes generating an O5 − plane in between these two half D7-branes. The second condition in (3.3) comes from putting the lower color D5-brane on the O5 − -plane in the middle, which also becomes Figure 12: Lowering the flavor brane in Figure 11 through a series of flop transitions, so as to apply the Higgsing procedure discussed in Figure 2 Figure 13: A 5-brane web for 6d SU (3) theory with Z 2 twist. The half monodromy cut extends from the right to the left.
two half D5-branes. Through Higgsing, one of half D5-branes remains intact, while the other half D5-brane is recombined onto the two half D7-branes on the O5 − -plane. So there are effectively three half D5-branes in between two half D7-branes so that the two half D5-branes can be Higgsed away. Now, by taking each half D7-brane in the opposite directions, leaving the half D7 monodromy cut being presented from the right to the left, one obtains a 5-brane configuration given in Figure 13. The masses of W-bosons after the Higgsing are then given by which form the affine Cartan matrix of twisted affine Lie algebra A 2 .

Partition function from 5-brane webs
With this 5-brane web, we are ready to compute the topological string partition function for the 6d SU (3) gauge theory with Z 2 twist. Like 5-brane configurations for the SO(8) theory with Z 2 twist in the previous section, this 5-brane configuration is not smoothly connected to its reflected image over the orientifold plane. In order to make a desirable 5-brane configuration where the original 5-brane and its reflected mirror image are smoothly connected, we introduce an auxiliary spinor hypermultiplet on the 5-brane configuration, as done in the previous section. It allows us then to apply topological vertex formulation with an O5-plane to compute the partition function for the 6d SU (3) gauge theory with Z 2 twist, followed by taking the decoupling limit of the newly introduced hypermultiplet. For instance, see Figure 14. The lower part of the 5-brane configuration for 6d SU (3) gauge theory with Z 2 twist is given in Figure 14(a) where we introduced a hypermultiplet 4 on the left hand side of the web configuration. Here two half D7-branes are still in the middle of the brane web which are denoted by blue dots and their monodromy cuts are the wiggly lines pointing to the left. For computational ease, we take these half D7-branes to the far left instead of taking them in the opposite directions. This makes a 5-brane configuration with only O5-plane, as depicted in Figure 14(b). When taking all the half D7-branes to the left, half D5-branes are created as a result of the Hanany-Witten transitions, which is depicted in Figure 14(b), where half D5-branes are solid lines in violet. In actual computation, they can be viewed as follows: one full external D5-brane (or two half D5-branes) is connected to the right (0, 1) and (1, 1) 5-branes, and an additional full D5-brane is connected to the left (0, 1) and (1, 1) 5-branes.
For the upper part of the 5-brane web in Figure 13, the 5-brane configuration is not smoothly connected to its mirror image as well. One may again couple another hypermultiplet as in Figure 15. It is also possible to use the symmetry of the 5-brane web. Instead of computing the partition function based on Figure 15, we use the following trick. We recycle the lower part of 5-brane configuration with a hyper for the upper part computation. In other words, when implementing the topological vertex for 5-brane configuration on the upper orientifold plane, we re-use the lower part configuration with a hyper as if it is a suitable 5-brane configuration for the upper part and perform the partition function computation by properly gluing 5branes. For instance, as in Figure 16, we first consider strip diagrams. For the upper part, in particular, we use the strip diagram of the lower part and glue the upper and lower part together. In the figure, the two edges marked with || symbols are identified. Figure 15: A brane web of SU (3) theory with Z 2 twist coupled with two spinors. Figure 16: Strip diagrams for 5-brane webs for SU (3) theory with Z 2 twist given in Figure 15. The left part (a) and the right part (b) of the web diagram in Figure 15. Z left is based on (a) and Z right is based on (b). The edges denoted with || are identified.
This trick is not harmful, as the point of introducing extra hypermultiplets is to arrange the 5-brane configuration in a way that we can apply the topological vertex. Moreover, we decouple the contribution of these newly introduced hypermultiplets. It follows then that the the strip diagram on Figure 16(a) yields 1,2,3 )R µ 1 ν 2 (Q 3,4,5,5 )R µ 2 ν 2 (Q 4,5,5 )R ν 1 ν 2 (Q 1,1,2,3,4,5,5 )R ν t where we used the Kähler parameters Q i assigned in Figure 15 with the convention The strip on Figure 16(b) can be treated in a similar way except that the upper and lower edges marked with || symbol are identified.
The topological string partition function for the strip in Figure 16(b) is a bit involving but can be obtained in a straightforward manner following [33,34]. One needs to repeatedly use various Cauchy identities in Appendix B. The contribution of the strip diagram in Figure 16(b) then reads The full topological string partition function Z is obtained by combining Z left and Z right , It is convenient to further rescale the scalars in Figure 13 as which gives that the Kähler parameters are expressed in terms of physical parameters where u is the string fugacity. We note that the relation q = (Q 1 Q 2 Q 3 Q 4 Q 5 ) 2 = e −τ is not changed under the rescaling. With A = e −φ 2 , we can write the Kähler parameters Q 2 and Q 4 as Q 2 = q 1/4 A −1 Q −1 1 and Q 4 = q 1/4 A −1 Q −1 5 where Q 1 , Q 5 are the Kähler parameters associated with the spinors which we will decouple.
The perturbative partition function Z pert is obtained by setting µ 1 = µ 2 = ∅ in (3.7) and by summing up over the Young diagrams ν i , as a function of g, A, q, Q 1 and Q 5 . As an expansion of q, we find the perturbative part is given by where the · · · denotes the terms involving Q 1 and Q 5 which we will decouple later. This is expected result as in Appendix A. If we take the large radius limit corresponding to q → 0, the states depending on KK-momentum will be truncated, and as a result, only q 0 term will remain. The partition function for 6d self-dual strings Z string is obtained by summing up over all Young diagrams in (3.7), expand about q and A, and also by decoupling the auxiliary spinor hypermultiplets that we introduced for computational ease, Q 1 , Q 5 → ∞ as discussed in the previous section, where Z n correspond to the n-string elliptic genus, Though we presented the result of one-and two-string elliptic genus only, higher n-string elliptic genus can be computed in a straightforward manner.

Elliptic genus by Higgsing 6d G 2 + 1F
In this subsection, we compute the elliptic genus to cross-check the partition for the SU (3) theory with Z 2 twist obtained from 5-brane webs in the previous subsection.
As discussed, the SU (3) theory with Z 2 twist can be obtained by Higgsing the 6d G 2 gauge theory with one fundamental hypermultiplet (G 2 + 1F). We hence start by computing the elliptic genus for the 6d G 2 gauge theory with one fundamental hypermultiplet and apply the Higgsing that leads to the SU (3) theory with Z 2 twist. The perturbative part of the partition function comes from the contributions from the vector multiplet and the hypermultiplet, Z G 2 +1F pert = Z gauge pert Z hyper pert . As we have the correspond 5-brane web in Figure 12, we can readily obtain the perturbative part. The vector multiplet contribution to the perturbative part for G 2 + 1F takes the following form (3.13) where χ G 2 ∆ ± are the positive and negative parts from the characters for the adjoint representation of G 2 . The hypermultiplet contribution to the perturbative part is given by (3.14) where M = e −m and x i = e −φ i . These perturbative part can be easily computed from the web diagram in Figure 12.
To perform the Higgsing leading to the SU (3) gauge theory with Z 2 twist, we give a vev to the hypermultiplet carrying KK momentum. For that, we recover the affine node φ 0 as in Figure 12, and impose the Higgsing conditions (3.3). By rescaling the scalars (3.8), we obtain the perturbative part of the partition function for the SU (3) gauge theory with Z 2 twist: (3.15) where A = x 2 = e −φ 2 . This result is the same as that obtained from the topological vertex calculation (3.10) up to the Cartan parts.
We now compute the elliptic genus of the 6d G 2 theory with one fundamental hypermultiplet, following [35]. The symmetries of the 2d worldsheet theory on selfdual strings are U (n) gauge symmetry for string number n, SU (3) global symmetry which is enhanced to G 2 in IR, U (1) J and SU (2) l global symmetries. Here, for SU (2) R R-symmetry and SO(4) = SU (2) l × SU (2) r rotation symmetry of the transverse R 4 , the charge J is identified as the sum of the SU (2) r charge and the SU (2) R charge, J = J r + J R . The worldsheet fields can be written in N = (0, 2) multiplets. There are the vector multiplet V , the Fermi multiplets λ, (λ,λ), (Ψ,Ψ), and the chiral multiplets (q,q), (a,ã), (φ i , φ 4 ), (b,b). Their charges under the symmetries in the 2d worldsheet theory are summarized in Table 2. Using the matter content in Table 2, one can write down the n-string elliptic genus: Here, u I are the U (n) parameters, η is the Dedekind eta function, v I are the SU (3) parameters subject to v 1 + v 2 + v 3 = 0. First consider the n = 1 case. We choose an auxiliary vector ζ = (1). Then contributing poles are from + + u − v J = 0. The contour integral converts to the JK-residue sum. We thus obtain the one-string elliptic genus for the G 2 gauge theory with one hypermultiplet , (3.17) where the SU (3) condition v 1 + v 2 + v 3 = 0 is used. For the n = 2 case, we choose an auxiliary vector ζ as in Figure 17. The poles which survive in the JK-residue sum are The first poles (i) in (3.18) give The second poles (ii) give Notice that the third poles (iii) can be obtained by u 1 ↔ u 2 in (ii). It follows that the residue for (iii) is the same as that for (ii), Res 3 = Res 2 . The two-string elliptic genus of the G 2 theory with one hypermultiplet is then given by We now Higgs the 6d G 2 theory. The procedure is similar to that for the SO(10) case in the previous section. As the elliptic genus results are written in terms of the SU (3) parameters, rather than the G 2 parameters, some additional implementation is required for the proper Higgs from the 6d G 2 theory to the SU (3) theory with Z 2 twist.
SO (8) and SU (3) gauge theories with Z 2 twist, from the 6d SO(10) gauge theory with two flavors and from the 6d G 2 gauge theory with a flavor, respectively. We computed the partition functions of these theories following the topological vertex formalism in the presence of O5-planes developed in [6], by introducing and decoupling spinor matter fields to implement the topological vertex as done [36]. The other is to directly apply the outer automorphism twists on the elliptic genera for the SO (10) and G 2 gauge theories, by Higgsing the hypermultiplets with proper KK-momentum dependence. We checked that the partition functions based on topological vertex with O5-planes agree with the elliptic genera after Higgsings. As the elliptic genera are fully refined, we compared them in the unrefined limit as well as by double expanding each in terms of the KK momentum and the Coulomb branch parameters.
When computing the elliptic genus for the SU (3) theory with Z 2 twist in section 3.3, one may wonder whether one can apply the twisting directly on the elliptic genus for the SU (3) theory, implementing the KK-momentum shifts as done for the SO (8) case in section 2.4. In the case of SO(8) theory with Z 2 twist, the order two outer automorphism maps fundamental representation to itself, and hence KK-momentum shifts can be easily understood. For su(3) algebra, however, the fundamental representation 3 maps to the anti-fundamental representation3 under the order two outer automorphism. From the ADHM perspective, there are one chiral multiplet which transforms as 3 and two chiral multiplets which transform as3, and hence it is not clear how to perform the automorphism twist on these SU (3) states in the ADHM construction. It would be good if a more systematic study along this direction is carried out so that it would be even applicable to order three outer automorphism twist.
Yet as another independent check, we computed, in [37], the BPS spectrum of the 6d theories with Z 2 twist using the bootstrap method via the Nakajima-Yoshioka's blowup equations, which provides fully refined partition functions. We checked our results against the BPS spectrum from the blowup formula and found that two results completely match.

A Twisted boundary condition and affine Lie algebras
Consider a 6d gauge theory with a simple Lie algebra g and compactification of the theory on a circle S 1 of radius R. For the gauge fields A µ = A a µ T a , where T a lie in the adjoint representation of g, we impose the periodic boundary condition Here, i = 1, 2, · · · , 5 and x 6 is the coordinate along S 1 . Fourier expansion of the gauge fields preserving the periodic boundary condition takes the form where T a n = T a e inx 6 /R . The new basis T a n satisfies where f abc is the structure constant of g, [T a , T b ] = f abc T c . The basis {T a n } with possible central extensions generates an untwisted affine Lie algebra. For a simple Lie algebra of type X , the untwisted affine Lie algebra constructed using the periodic boundary condition (A.1) is denoted by X (1) , and their Dynkin diagram is shown in Figure 18. The black circles are affine nodes. Excluding it gives the Dynkin diagram of corresponding simple Lie algebra X . Instead of the periodic boundary condition, one can impose a twisted boundary condition where σ is a finite order automorphism of g. If σ is an order m automorphism, i.e., σ m = 1, then the Fourier expansion of the gauge fields preserving the twisted boundary condition (A.4) is . The commutation relation of new basis is given by If σ is a conjugation, it is called an inner automorphism. In this case, the resultant algebra is the same as untwisted affine Lie algebra [38]. If σ is not an inner automorphism, it is called an outer automorphism and the resultant algebra becomes different from untwisted affine Lie algebras. The new algebra generated by {T a n+m/l } with possible central extensions is called a twisted affine Lie algebra. An outer automorphism can be viewed as a graph automorphism of Dynkin diagrams. Only the simple Lie algebras of types A n , D n and E 6 have nontrivial outer automorphisms. A n has an Z 2 Dynkin diagram automorphism which exchanges the simple roots α i and α n+1−i as in Figure 19(a) and Figure 19(b). D n has an order two outer automorphism which exchanges the simple roots α n−1 and α n as in Figure 19(c). The exceptional algebra E 6 also has an order two outer automorphism shown in Figure 19(e). The D 4 algebra additionally has an order three automorphism as Figure 19(d). For these simple Lie algebras X , twisted affine Lie algebras associated with order r = 2, 3 diagram automorphism are denoted by X (r) , and their Dynkin diagrams are given in Figure 20. Note that unlike the untwisted case, excluding the affine node does not yield X algebra. For example, deleting the affine node from A 2 and D +1 algebras give the Dynkin diagram of C and B algebras, respectively. Since a 6d gauge theory compactified on a circle naturally has an affine Lie algebra structure, its perturbative spectrum can be read off from the representation theory of Lie algebras. One can write down the perturbative part of the partition function by using either affine root system or by decomposing the representation of 6d gauge algebra into the representation of the invariant subalgebra under the automorphism. For more details, see [39,40] and also Appendix A of [37].
We list some relevant results of the perturbative part of the partition function  Z pert , used in this paper.
(i) The untwisted compactification: for a 6d gauge theory with gauge algebra g, the W-boson contribution to the perturbative part is given by, in the unrefined limit, where χ g ∆ ± are the character associated with the positive/negative roots, respectively. The twisted compactifications that we discussed in the main text are SO(8) gauge theory with Z 2 twist and SU (3) gauge theory with Z 2 twist. Their invariant subalgebras carry fractional KK-momentum. These fractional momenta contribute to the perturbative part.
(ii) SO(8) gauge theory with Z 2 twist: the adjoint representation of so(8) decomposes into the adjoint and the fundamental representations of so (7). The adjoint representation of so(7) carries integer KK charge, while the fundamental representation carries half-integer KK charge, where the subscript denotes the KK charge. From this, one finds that the perturbative contribution to the partition function for SO (8) gauge theory with Z 2 twist is given by (7) ∆ + + q 1/2 χ so(7) 7 + q χ so(7) ∆ − . (A.9) In the dimensional reduction limit where R → 0, all the states with non-zero KKmomentum are truncated so that only the adjoint representation of so(7) algebra remains. We note that the invariant subalgebra so (7) can be readily read off, as it it nothing but the algebra obtained by removing the affine node of the Dynkin diagram of D 4 .
We also note that skew Schur function s λ/µ is zero unless λ ⊃ µ.