M5-branes, orientifolds, and S-duality

We study the instanton partition functions of 5d maximal super Yang-Mills theories with all classical gauge groups. They are computed from the ADHM quantum mechanics of the D0-D4-O4 systems. Our partition functions respect S-dualities of the circle compactified Yang-Mills theories and various orientifold backgrounds. We also compute and study the S5 partition functions that correspond to the 6d (2, 0) superconformal indices. Our SO(2N ) index takes the form of the vacuum character of WD\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\mathcal{W}}_D $$\end{document} algebra in a special limit, supporting the W\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \mathcal{W} $$\end{document} algebra conjecture. We propose new indices for (2, 0) theories with outer automorphism twists along the temporal circle, obtained from non-simply-laced SYMs on S5.


Introduction
String theory provides a powerful framework for exploring quantum field theories. It leads to the discovery of quantum field theories in spacetime dimensions higher than 4. Important examples are six-dimensional maximally superconformal field theories, called (2, 0) theories, which come with the ADE classification [1]. The (2, 0) theories of AD-types describe the low energy dynamics of M5-branes [2]. It is very difficult to study these systems, due to the lack of their microscopic definitions.
Compactifying these 6d theories on a circle, one finds 5d maximal super Yang-Mills theories at low energy. Although these Yang-Mills theories are non-renormalizable, it has been suggested that they contain useful information about the 6d UV theories [3,4]. Instanton solitons of 5d gauge theories play an essential role in understanding the 6d physics. They are non-perturbative solitons carrying the topological U(1) charges which are interpreted as the Kaluza-Klein momenta along the circle.
In this work, we study these 5d N = 2 gauge theories preserving 16 supercharges, obtained from circle compactifications of 6d (2, 0) SCFTs. A remarkable point is that even if one begins with 5d SYMs which do not recognize the 6d circle, the six-dimensional physics is recovered by incorporating non-perturbative instantons [3][4][5][6][7][8][9]. At least for ADtypes, this claim is inspired by the duality relation between type IIA and M-theory. The Yang-Mills coupling constant g 2 5 is proportional to the radius of M-theory circle R M via the IIA string coupling constant. D0-branes bound to D4-branes are realized as instantons in 5d gauge theories, whose mass is inversely proportional to R M , i.e.,
We study the instanton partition functions for 5d N = 1 * gauge theories on Omegadeformed R 4 × S 1 . The N = 1 * theory is deformed from the maximal SYM by adding an N = 1 hypermultiplet mass m. We consider the Coulomb branch where the gauge symmetry is completely broken to its Abelian subgroup. The instanton partition function was first studied in [10,11] to understand Seiberg-Witten solutions of 4d N = 2 gauge theories. Generalization to all classical gauge groups and inclusion of various hypermultiplets were considered in [12,13]. Throughout this work, we regard this observable as the Witten index of 5d SYM wrapped on the temporal circle S 1 of radius β 2π . The 5d SU(N ) partition function was computed in [10,11] via supersymmetric localization, being further interpreted as the Kaluza-Klein index of SU(N )-type (2, 0) theory [5]. Besides instantons, the Nekrasov partition function also gets contribution from charged W-bosons which constitute 1 4 -BPS bound states with instantons. W-bosons are uplifted to self-dual strings of (2, 0) theories. They are electric and magnetic sources of tensor multiplets existing in 6d SCFTs. They are tensionless at the conformal fixed point. They obtain non-zero tension T ∝ Φ in the tensor branch, where tensor multiplet scalars Φ obtain non-zero VEVs Φ = 0 [1,2,14,15]. The SU(N ) instanton partition function played important roles in the recent studies on (2, 0) theory of A N −1 -type [5][6][7][8][9].
In this work, we extend the analyses made for the SU(N ) instanton partition functions to those of other classical gauge groups: SO(2N + 1), Sp(N ), SO(2N ). The SO(2N ) gauge theories are circle reductions of (2, 0) theories of D N -type. The SO(2N + 1) and Sp(N ) gauge theories are obtained from circle compactified (2, 0) theories of AD-type with outer automorphism twists [16,17]. We use these instanton partition functions to study the following subjects of the (2, 0) theories.
We first use the instanton partition functions to explore S-dualities of maximal super Yang-Mills theories. S-duality asserts that a pair of 4d N = 4 gauge theories are equal, where their gauge groups G and G ∨ are Langlands dual [18][19][20][21]. The W-bosons and monopoles in one theory correspond to the monopoles and W-bosons in the other theory, if their gauge couplings τ 4 = θ 4 2π + 4πi are related as τ ∨ 4 ∼ − 1 τ 4 [18][19][20][21]. It identifies a weakly-coupled theory and a strongly-coupled theory. Regarding 4d N = 4 SYMs as (2, 0) theories wrapped on tori, whose complex structures are translated to gauge couplings, S-duality is realized as exchanging two sides of the torus. Our instanton partition functions are 6d observables. Since they depend only on complex structures τ of the tori, they are also expected to respect the geometric S-dualities. In particular, for non-simplylaced gauge theories, S-duals of the instanton partition functions are expected to be those for 5d SYMs on S 1 with twisted boundary conditions [17]. The instanton partition functions depend on various chemical potentials. Keeping all of these chemical potentials, their Sdualities are hard to explore. However, the instanton partition functions simplify after taking special limits of the chemical potentials. In section 4.2, we discuss the S-dualities of the instanton partition functions in these limits.

JHEP12(2016)148
As a byproduct, we study the S-dualities of type IIA orientifold backgrounds compactified on S 1 which uplift to M-theory on T 2 . This is because one can study the D0-brane partition functions in various O4-plane backgrounds using the same techniques. Remarkably, one can perform the exact S-duality transformations on these partition functions with all chemical potentials turned on. See section 3 for the details.
The instanton partition functions are also useful as building blocks for curved space partition functions. We compute the 5-sphere partition functions that were studied in [6,7,22,23]. They are related to the partition functions of (2, 0) theories on S 5 × S 1 , which are called the (2, 0) superconformal indices [24]. For example, [6,7] obtained the (2, 0) superconformal index for SU(N )-type theory which agrees with the vacuum character of W A N −1 algebra. It leads to the conjecture that there is an underlying W algebra structure in 6d (2, 0) SCFTs of A-type. This has been explicitly conjectured and further tested using 3-point functions in [25]. We compute the S 5 partition functions for SO(2N ) gauge theories, and show that they take the form of the vacuum character of W D N algebra. 1 The outline of this paper is as follows: in section 2, we review the ADHM quantum mechanics of D0-D4-O4 systems and compute their Witten indices. In section 3, we study the S-dualities of pure orientifold systems compactified on a circle. In section 4, we study S-dualities of the instanton partition functions in special limits. We also compute the (2, 0) superconformal indices from the S 5 partition functions, displaying the 6d operator spectra. Our results extend the W algebra conjecture to SO(2N ) theories and also propose the new indices for (2, 0) theories with outer automorphism twists. Concluding remarks are given in section 5.

Instantons in 5d maximal SYM
We consider 5d maximal SYM on R 1,4 having a classical gauge group G with rank N . It has SO(1, 4) Lorentz symmetry and SO(5) R R-symmetry. We study the Coulomb branch where the vector multiplet scalars φ I acquire non-zero VEVs α I , breaking the gauge symmetry G into the Abelian subgroup U(1) N ⊂ G. The maximal SUSY algebra is given by where M, N = 1, 2, 3, 4 are SO(1, 4) spinor indices, i, j = 1, 2, 3, 4 are SO(5) R spinor indices, I is the SO(5) R vector index, C M N is the charge conjugation matrix, ω ij is the SO(5) R Sp(2) R symplectic form, Π denote U(1) N gauge symmetry generators. Supercharges are subject to the symplectic-Majorana condition. The U(1) instanton charge k is defined as which is integer-valued. We write Yang-Mills kinetic term as 1 Tr (F µν F µν ), setting the unit instanton mass to be 4π 2 /g 2 5 .

JHEP12(2016)148
The maximal SYM contains an N = 2 vector multiplet whose N = 1 decomposition gives a vector multiplet plus an adjoint hypermultiplet. We look at the N = 1 Coulomb branch where only the N = 1 vector multiplet scalar has a non-zero VEV α. This set-up has SO(4) 1 little group of 5d massive particles and SO(4) 2 ⊂ SO(5) R R-symmetry. We further decompose it into SU(2) 1L × SU(2) 1R ⊂ SO(4) 1 and SU(2) 2L × SU(2) 2R ⊂ SO(4) 2 . We denote 16 generators of maximal SUSY by There are two types of massive 1 2 -BPS particles in the Coulomb phase: W-bosons and instantons. W-bosons are electrically charged objects under U(1) N ⊂ G, which we choose to satisfy Tr(α · Π) > 0 and preserve Q Ȧ α and Q a α . Instantons are solitonic particles that carry the topological U(1) charge defined in (2.2), which we choose to satisfy the self-duality condition F mn = 1 2 mnpq F pq and preserve Q Ȧ α and Q ȧ α . Notice that our choice restricts the instanton charge k to be a positive integer. These BPS particles may form 1 4 -BPS bound states, whose masses are given by In this section, we study the instanton partition functions which count these BPS bound states of instantons and W-bosons in 5d maximal super Yang-Mills theories.

N = (4, 4) ADHM quantum mechanics
The moduli space approximation is a technique to describe the low energy dynamics of solitons [26]. Being applied to 5d SYM instantons, it gives a SUSY quantum mechanics for instanton zero modes. When the 5d gauge group is classical, the zero modes are described as the ADHM data satisfying the ADHM constraint equation [27]. The instanton quantum mechanics is a non-linear sigma model whose target space is the instanton moduli space.
However, one cannot expect the instanton quantum mechanics to be a UV complete description for instanton solitons since the instanton moduli space suffers from the small instanton singularity where the instanton size shrinks to zero. It is generally very demanding task to find a UV completion of the instanton quantum mechanics. In certain type of theories, string theory supplies the UV completions from D-brane realizations of 5d SYM instantons. We call them the ADHM quantum mechanics. 5d maximal SYM with a classical gauge group G is engineered from D4-branes possibly on top of an O4-plane. Its instantons are realized as D0-branes stuck on D4-branes. The worldvolume theory of D0-branes is the ADHM quantum mechanics that we study. It is a gauge theory whose gauge groupĜ is determined from G. It has SO(4) 1 × SO(5) R symmetries inherited from underlying 5d theories. It preserves N = (4, 4) supersymmetry 2 generated by Q Ȧ α and Q ȧ α . Its field contents are induced from massless modes of open
Here bif(Ĝ × G) denotes the bifundamental representation ofĜ × G. Type of the O4-plane determines the 5d group G, the 1d groupĜ, and the representation R as follows [28].
Type of O4 5d gauge group G 1d gauge groupĜ Representation R ofĜ k denotes the instanton number. The action of the U(k) ADHM quantum mechanics is given in [5]. Generalization to other gauge groups is straightforward. The ADHM quantum mechanics include more fields than the instanton quantum mechanics, which are massless only at the small instanton singularity. The vector multiplet scalars ϕ, ϕ aA are such extra fields, which open up a new branch of the moduli space touching the instanton moduli space exactly at the small instanton singularity. We call it the Coulomb branch of the ADHM quantum mechanics. This UV-completion of the instanton moduli space allows D0-branes to move away from D4-branes. One can regard D0-branes as 5d SYM instantons only in the Higgs branch of the quantum mechanics, which is spanned by the ADHM data a αα , qα subject to the ADHM constraint equation.
We want to use the ADHM quantum mechanics for counting 5d SYM states involving instantons. However, the Coulomb branch may give extra contributions to the Witten indices which are irrelevant to the 5d SYM partition functions. See section 3 for related discussions.

Witten index
In this section, we compute the index of the ADHM quantum mechanics defined as (2.6) It counts BPS states annihilated by Q ≡ Q A=1 α=1 and Q † ≡ Q A=2 α=2 . J 1L , J 1R , J 2L , J 2R are the Cartan generators for the SU(2) 1L , SU(2) 1R , SU(2) 2L , SU(2) 2R global symmetries. Π a denote the Cartan generators of the 5d gauge group G. Various fugacities are conjugate to the Cartans of global symmetries commuting with Q, Q † . Besides Π a 's, J 1R +J 2R , J 1L , J 2L are all commuting combinations. We often express the fugacities using chemical potentials as follows:

JHEP12(2016)148
The chemical potentials deform the underlying 5d gauge theory. 1 = + + − 2 and 2 = + − − 2 put the 5d gauge theory on Omega-deformed background. m becomes the mass of the N = 1 adjoint hypermultiplet, yielding the mass-deformed N = 1 * SYM. α is a complexified background Wilson line of the 5d gauge group G, breaking it into the Abelian subgroup. We set α to be 3 Such deformations regulate the 5d gauge theory at long distances, defining the instanton partition function with IR regulators [10]. They give mass to the scalars ϕ aA , a αα , qα in the ADHM quantum mechanics, making the BPS spectrum to be gapped. However, there still remains a flat direction in the moduli space. As the vector multiplet scalar ϕ is not charged under any global symmetries, chemical potentials cannot give it a mass.
If an Fayet-Iliopoulos deformation is available, i.e., ifĜ contains an overall U(1) factor, ϕ acquires mass. In general, we encounter a continuous BPS spectrum that makes the index (2.6) hard to compute. Nevertheless we can obtain the instanton partition function from the indices (2.6) of the quantum mechanics. The flat direction spanned by ϕ belongs to the Coulomb branch that decouples from the Higgs branch in IR [29,30]. Index contributions of the decoupled Coulomb branch can be identified and removed from the multi-particle indices, which are factorized as [31] where q ≡ e −4π 2 β/g 2 5 denotes the instanton number fugacity. The instanton partition function Z inst comes from the Higgs branch contributions which do not involve any subtleties from the Coulomb branch continuum.
We now turn to the path integral localization of the index (2.6), following [31,32]. The path integral measure is given by the Euclidean action of the ADHM quantum mechanics. The temporal direction is compactified as the circle with circumference β, used in (2.6). Once we take the weak coupling limit g 1 → 0, where g 1 denotes the coupling constant of the quantum mechanics, the path integrals are reduced to Gaussian integrals around zero modes. The most important zero modes are holonomies of the gauge field A 0 along the temporal circle and the vector multiplet scalar ϕ. They are combined into dimensionless, complexified holonomies φ = iβA 0 + βϕ. A 0 is subject to a large gauge transformation on the temporal circle, making the imaginary parts of φ eigenvalues to be periodic. The
We perform the Gaussian integrals over massive fluctuations, fixing the zero modes for a while. Integration gives the 1-loop determinants I 1-loop which are products of the following factors [13].
for all other cases (2.13) We use the ± notation: 2 sinh (a ± b) ≡ 2 sinh (a + b) · 2 sinh (a − b). r is the number of continuous parameters in the φ holonomies. One-loop determinants for fields in certain representations ofĜ and G involve the parametersρ(φ) and ρ(α). We refer to [13] for the parameters in all rank-1 and rank-2 representations of classical groups. For example, the parameters for the U(k) representations are The root is defined such that the coupled parametersρ(φ) arê

JHEP12(2016)148
Note that 1-loop determinants of real scalars and fermions involve square roots of sinh factors. A pair of such factors are always arranged as sinh z+a 2 sinh −z−a 2 ∼ sinh ( z+a 2 ), following [31]. This is why the first line of (2.13) has twice of sinh factors than the second line of (2.13).
The next step is to integrate over the zero modes. Carefully treating the zero modes of the gaugino and the auxiliary scalar, the integral becomes a contour integral over the space of φ holonomies, i.e., r copies of a cylinder [31,32]. The integrand I 1-loop develops various poles, some being inside a finite region and the others being at infinities. We need a proper choice of contour to complete the integral. 4 For r = 1, the index contribution of the residues R ±∞ at infinities appears as the sum R +∞ + R −∞ with a coefficient depending on a Fayet-Iliopoulos parameter ζ [31,32]. We expect the same for r > 1. One can show that the residue sums R +∞ + R −∞ , for the N = (4, 4) ADHM quantum mechanics, are zero for all cylinders. So the index is independent of ζ. It receives the contributions only from the residues inside the finite region, chosen by the Jeffrey-Kirwan residue operation [31,32,35]. For non-degenerate poles, where r distinct sinh factors in the denominator of I 1-loop become zero, the Jeffrey-Kirwan residue can be expressed as It is a linear functional which refers to a pole location φ * and an auxiliary vector η in an r-dimensional charge space. Q * = (Q 1 , · · · , Q r ) is a set of r charge vectors, associated to the sinh factors in the denominator being zero at φ * . The vector η has to be generic for JK-Res φ * (Q * , η) to be well-defined. We refer to [32] for treatment of degenerate poles. We recall thatĜ = O(k) allows the two disconnected Wilson lines in O(k) + and O(k) − . In general, ifĜ is disconnected, there can exist multiple, disconnected Wilson line backgrounds ofĜ. The Witten index includes a sum over distinct holonomy sectors such that where a labels the disconnected holonomy sectors, |W a | and I 1-loop are the Weyl group order and the 1-loop determinant in a given holonomy sector a. φ * runs over all existing poles in the integrands. The Weyl group orders |W a | that preserve the holonomies given in (2.10) are

JHEP12(2016)148
Note that 5d Sp(N ) SYMs have two discrete choices of θ angles 0, π related to π 4 (Sp(N )) = Z 2 [36]. The 5d θ parameters induce the 1d discrete θ parameters related to π 0 (O(k)) = Z 2 , which are the Wilson lines along the temporal circle. The sums over holonomy sectors in (2.17) are given as [37] 3 Orientifolds from M-theory The multi-particle index Z ADHM of the ADHM quantum mechanics is computable using the formula (2.17). Z ADHM captures BPS states of D0-branes, some of which are decoupled states from 5d SYM. The multi-particle index Z ADHM is factorized into two parts [31]: Z inst is the instanton partition function. Z extra is the index for the extra states. To obtain the instanton partition function, we need to identify and remove Z extra from Z ADHM . In this section, we determine Z extra for various D0-D4-O4 systems on a case-by-case basis.
We already discussed that the extra contribution comes from the Coulomb branch, which is parametrized by ϕ and ϕ aA . At a generic point of the Coulomb branch, these scalar fields acquire non-zero values which represent D0-branes' transverse positions to D4branes. We expect to determine Z extra by analyzing BPS states of D0-branes far away from D4-branes. For this purpose, we separately compute the multi-particle indices of D0-branes in the pure orientifold backgrounds without any dynamical D4-branes.
The pure orientifold backgrounds are obtained by formally taking N = 0 in the ADHM quantum mechanics. 5 The extra indices Z extra are included in the N = 0 indices which can be written in a simple way using the plethystic exponential
The subtlety arising here is that there exists a flat direction, spanned by ϕ, even after turning on all chemical potentials. This damages the robustness of the Witten index, which is the critical assumption for localizing the path integrals. The Witten index is defined in the regime g 2 1 β 3 1, while the index formula (2.17) is derived in the regime g 2 1 β 3 1. If there were no flat direction, such deformation would not change the index. However, the ADHM quantum mechanics has the flat direction developed by ϕ, so that the N = 0 indices exhibit fractional coefficients 1 2 's in (3.3)-(3.6), as discussed in [39][40][41]. Due to the decoupling of the Higgs and Coulomb branch at low energy [29,30], we still expect that the continuum from ϕ spoils Z extra , not Z inst .
In order to identify the extra indices Z extra from the N = 0 indices, we study the origins of D0-brane bound states in the N = 0 indices from M-theory perspectives. It also tells us interesting information on the orientifold backgrounds in string theory. While SO(9) is the little group for 10d massive particles, we only try to understand the SO(8) ⊂ SO(9) spin contents of D0-branes, setting aside 1 2 's in (3.3)-(3.6), where the fugacities t, u, v are conjugate to the SO(8) spins.
First of all, we remark on the geometric interpretations of the instanton number fugacities q. They are defined as q ≡ e −S 0 using the unit instanton action S 0 = 4π 2 β g 2 5 . Since D0-branes carry the Kaluza-Klein momenta along the M-circle, their masses are related to the inverse of the M-circle radius. The O(k) theories describe the dynamics of k 'half' D0-branes, while the U(k) and Sp(k) theories describe the dynamics of k 'full' D0-branes. This observation implies that the relations (1.1) between the 5d gauge couplings and the M-circle radii should be precisely stated as

JHEP12(2016)148
Accordingly, the instanton number fugacities (3.8) are interpreted as the M-circle momentum fugacities with a half-integer unit (forĜ = O(k)) and with integer units (for all other cases). In all cases, M-theory wraps the torus T 2 . It is made of the M-circle and the temporal circle, whose radii are R M and β 2π respectively. Its complex structure τ is given by τ ≡ i β/2π R M . Note that S-dualities of IIA backgrounds, often called the "9-11 flip" [42], correspond to exchanging the two sides of the torus T 2 , i.e., τ −→ − 1 τ . Now we analyze (3.3)-(3.6) from M-theory perspectives, apart from the 1 2 factors.
Pure D0. We consider M-theory on R 1,9 × S 1 and its massless particles with non-zero momentum P 11 along the M-circle, satisfying |P 0 | = |P 11 | > 0 and P i=1,··· ,9 = 0. The 11d SUSY algebra is written as {Q, Q † } ∼ (−1 ± Γ 0 Γ 11 )|P 0 |. The 11d gravity multiplet preserves half of the 32 supercharges, whose component fields are the metric g (11) µν , the 3-form tensor C µνρ , and the gravitino ψ µ . They sit in irreducible representations of the SO(9) little group, i.e., 44, 84, and 128. We define the index f bulk to be the trace  10) and the sum of SO(8) characters X 0 for the following irreducible representations: Taking (−1) F into account, X 0 is given by where χ(R) denotes a character for an irreducible representation R. Using the fugacities t, u, v introduced in (2.6), the SO (8) D0-O4 − . The O4 − background uplifts to M-theory on R 1,4 × R 5 /Z 2 × S 1 [43]. The Z 2 action inverts the coordinates (x 5 , · · · , x 9 ) → (−x 5 , · · · , −x 9 ) and flips the sign of the 3form tensor C 3 → −C 3 . One can divide the Z 2 action into two operations: (1) the rotation of x 5,6 and x 7,8 planes by +π and −π, (2) the x 9 → −x 9 parity along with the sign flip of the 3-form tensor C 3 → −C 3 . Regarding the index computation, the first operation P 4 causes the shift of chemical potential m → m + iπ conjugate to 2J 2L ≡ J 3 − J 4 , where J 3 and J 4 are the rotation generators for x 5,6 and x 7,8 planes. The second operation P 1 provides the grading of all SO(8) representations in (3.11). The grading rules are stated as follows. When P 1 acts, • Each vector index referring the x 9 direction yields the negative sign (−1).
• Each spinor index undergoes the multiplication by Γ 9 where Γ µ denotes the 11d gamma matrix.
• The 3-form tensor C 3 goes through the extra sign flip C 3 → −C 3 .
We require 8 s to be even and 8 c to be odd under the multiplication by Γ 9 , such that the supercharges Q and Q † in 8 s which used to define the index (2.6) are invariant. 56 c and 56 s are parts of 8 v ⊗ 8 s and 8 v ⊗ 8 c , inheriting the parities of 8 s and 8 c .

JHEP12(2016)148
where P 4 replaces v ≡ e −m by −v = e −m−iπ in (3.10) and X 1 . After all, the index f bulk becomes 3) counts the 11d supergravity fields on R 1,4 ×R 5 /Z 2 × S 1 , up to the 1 2 factor, which do not belong to the 5d SYM Hilbert space. We conclude that Z D0-O4 0 . The O4 0 background uplifts to M-theory on R 1,4 × (R 5 × S 1 )/Z 2 , whose Z 2 action is the combination of P 4 , P 1 , and the M-circle shift x 11 → x 11 + πR M by the halfperiod [43]. In addition to the P 4 and P 1 actions described above, the M-circle shift induces the phase factor e iπ = −1 to a single D0-brane wavefunction exp (ix 11 /R M ). It causes the sign change of the momentum fugacity q → −q. The index f bulk over the Kaluza-Klein fields of 11d supergravity on R 1,4 × (R 5 × S 1 )/Z 2 becomes , (3.16) whose multi-particle spectrum agrees with Z D0-O4 + . The O4 + background uplifts to M-theory on R 1,4 × R 5 /Z 2 × S 1 with the full M5-brane frozen at the Z 2 fixed plane [43]. q is the fugacity for half-integral momenta along the M-circle. The first term of Z Sp(0) θ=0 ADHM in (3.5) comes from the 11d bulk gravity, using the same argument used for the O4 − background. Here we take into account that q 2 is the fugacity for integral momenta in the bulk. We identify this as the extra states' index Z (3.17) The second term in (3.5) comes from the full M5-brane frozen at the Z 2 fixed plane, which hosts the SO(2)-type (2, 0) theory on T 2 with the Z 2 outer automorphism twist [17]. Beginning from the Abelian (2, 0) theory on S 1 , whose index is given by the result in (3.5) shows that all states with integral momenta are eliminated by the Z 2 projection, while the twisted states having half-integral momenta are newly introduced. The perturbative degrees for 5d Abelian SYM are projected out, yielding the pure orientifold background. Our result is consistent with [44] which shows a half D0-brane can bind to an O4 + plane while a full D0-brane do not. Applying the S-duality of M-theory on T 2 , which swaps the temporal circle and the Mcircle, the O4 + background is mapped to the full D4-brane frozen at the O4 − plane [43]. In

JHEP12(2016)148
the S-dualized background, the O(2) − Wilson line σ 3 = diag (+1, −1) along the temporal circle prevents the D4-brane from moving away from the orientifold [45]. We shall verify the S-duality relation between two orientifold backgrounds using the multi-particle indices of the relevant ADHM quantum mechanices. First, we compute the 5d O(2) instanton partition function in the Wilson line background σ 3 = diag (+1, −1) ∈ O(2) − . It can be done by replacing the continuous holonomies ±α 1 of G = SO(2) by the discrete holonomies 0 and iπ of O(2) − in the 1-loop determinants I 1-loop . Such replacement means the full D4brane cannot move from the O4 − plane. Following the computation in section 2.2, the index Z O(2) − is expressed using the plethystic exponential as follows: whose q-dependence was checked up to q 3 -order. The last term comes from the 11d bulk gravity. The remaining two terms are expected to be S-dual to the second term of Z Sp(0) θ=0 ADHM . We observe that the second term of Z Sp(0) θ=0 ADHM can be written as The S-duality transformation of this function has been studied in [9], since Z is the partition function of a single M5-brane on Omega-deformed R 4 × T 2 . Let us apply the S-duality transformation that exchanges the two sides of the T 2 , i.e., τ → − 1 τ , and transforms the chemical potentials such that + → + τ , − → − τ , and m → m τ . The S-dual partition function Z(− 1 τ , + τ , − τ , m τ ) is related to the original one in the following simple manner [9], (3.22) in a particular parameter regime. See [9] for the details. Z(τ, + , − , m) is essentially

JHEP12(2016)148
invariant under S-duality, up to a simple overall factor [9]. Accordingly, (3.20) becomes under the S-duality transformation. The second line equals the first line of Z O(2) − in (3.19). Apart from the τ independent factor on the third line of (3.23), one finds that Z ADHM is dual to Z O(2) − . This analysis confirms the S-duality relation studied in [45] between the O4 + plane and the O4 − plane with the frozen D4-brane.

D0-O4
+ . The O4 + background uplifts to M-theory on R 1,4 × (R 5 × S 1 )/Z 2 with a stuck M5-brane at the origin of R 5 [43]. The Z 2 action consists of the R 5 parity and the Mcircle shift x 11 → x 11 + πR M by the half-period. The M-circle shift induces the phase factor e iπ/2 = i to a half D0-brane wavefunction exp (ix 11 /2R M ), causing the half-integral momentum fugacity q to undergo the phase rotation q → iq. Repeating the same argument used for the O4 0 background, we understand the first term of Z . (3.24) The second term of Z (3.5). Both are obtained from single M5-branes stuck at the Z 2 fixed point of R 5 . The difference is that the M5-brane wrapping the freely-acting Z 2 orbifold sees the halved M-circle radius. It effectively doubles the complex structure τ of the T 2 , explaining the difference between (3.20) and Similar to the O4 + case, the Z 2 projection kills all integral-momentum states in the Abelian (2, 0) theory on S 1 . What contribute to (3.25) are the twisted states, having half-integral momenta along the M-theory circle. The Z 2 projection eliminates the perturbative degrees of 5d Abelian SYM, leaving the pure orientifold background. Its S-dual configuration in M-theory has been suggested in [43,45]. It would be interesting to see the S-duality of the two backgrounds explicitly. We remark that the multi-particle indices (3.3)-(3.6) of the ADHM quantum mechanics for various D0-O4 systems capture the non-perturbative (curvature) 2 terms of orientifold JHEP12(2016)148 backgrounds. Recall that the genus expansion of the topological string amplitude F takes the form of [46][47][48] F ≡ log Z ADHM = ∞ n,g≥0 (3.26) Turning off + , the remaining Omega-deformation parameter − denotes the self-dual part of the graviphoton field strength on R 4 . The amplitude gives the (curvature) 2 correction to M-theory on CY3 × S 1 [47]. Considering the case with m = + = 0, the genus-1 terms at n = 0 are given from (3.3)-(3.6) as follows.
After performing the S-duality transformation τ → − 1 τ , they agree with the (curvature) 2 terms in the O4-plane actions, computed from graviton scattering amplitudes with nonperturbative effects [49], up to an overall factor − 1 4π that we have not precisely traced.

6d SCFTs from 5d SYM instantons
In this section, we study the BPS spectra of 6d (2, 0) theories using the instanton partition functions. String theory predicts that 5d maximal SYM, supplemented by the non-perturbative instanton effect, has a UV fixed point which corresponds to 6d (2, 0) SCFT [3,4]. The instanton partition function Z inst would capture the BPS spectrum of 6d (2, 0) SCFT wrapped on R 4 × T 2 , where instantons play the role of Kaluza-Klein momenta along the M-circle. For example, the interpretation of the SU(N ) instanton partition function as the tensor branch index of 6d SU(N )-type (2, 0) SCFT was justified in [5,8].
We want to extend some aspects of the above analysis to other gauge groups: SO(2N ), SO(2N + 1), and Sp(N ). In section 4.1, we make some consistency checks of the instanton partition functions for lower rank gauge groups. This is because the SU(N ), SO(2N + 1), Sp(N ), SO(2N ) Lie algebras sometimes coincide at low ranks. We confirm that apparently different ADHM approaches yield the same prepotentials. In section 4.2, we obtain the closed form expressions for the instanton partition functions in special limits, and study the S-duality relations of them. In section 4.3, we study the 6d superconformal indices using the results of section 4.2.

Consistency checks
Let us compare the instanton partition functions with coincident gauge groups, which belong to different ABCD-types of Lie algebras. For instance, the classical actions of the maximal super Yang-Mills theories with SU(2) and SO(3) gauge groups are identical, both in 4d and 5d. In 4d, this means that the two quantum theories are completely the same. So

JHEP12(2016)148
the physical QFT observables that one can extract out of the instanton partition functions should be the same, although the two ADHM descriptions look different. However, in 5d, having identical classical actions do not necessarily imply that the two instanton calculus are the same, because different ADHM quantum mechanics for 5d solitons may provide different UV completions of non-renormalizable 5d Yang-Mills theories.
The requirement that SU(2) and SO(3) partition functions should yield identical 4d observables imposes non-trivial constraints on the two partition functions, which we shall check in this section. One important 4d observable is the prepotential, whose second derivative with the Coulomb VEVs α gives the Coulomb branch effective action. Thus, the Coulomb VEV dependent parts of the SU(2) and SO(3) partition functions should be the same. We check in the 5d partition functions that the Coulomb VEV dependent parts are indeed identical to each other, even before taking the 4d limits, which confirms that the two partition functions satisfy the 4d constraint. However, Coulomb VEV independent parts do not have to agree with each other, and we find that they indeed disagree for SU(2) and SO(3). We make similar consistency checks for gauge theories with low ranks, whenever we have more than one ADHM descriptions: SO(2) vs U(1), SO(4) vs SU(2) 2 , SO(6) vs SU(4), SU(2) vs SO(3) vs Sp(1), SO(5) vs Sp (2).
However, in 5d, even Coulomb VEV independent terms are physically meaningful, as the partition function itself is a Witten index. They are simply counting electrically neutral BPS states. Different ADHM descriptions may provide different UV completions of the two 5d gauge theories which are identical at the classical level. Such phenomena happen when the 5d gauge groups belong to the B or C classes, which uplift to different 6d theories on a circle with outer automorphism twists. It also affects the 5-sphere partition functions that we study in section 4.3, which we interpret as 6d superconformal indices with outer automorphism twists. SO(2) vs U(1), SO(4) vs SU(2) 2 , SO(6) vs SU(4) theories, which formally belong to the AD classes, are expected to have identical UV uplifts. In these cases, we expect that both the charged and neutral sectors are the same.
We start by commenting on our parameterizations of the SU(N ) chemical potentials. The SU(N ) instanton partition functions can be obtained from the U(N ) instanton partition functions by imposing the traceless conditions on the chemical potentials. Among the U(N ) holonomies α 1 , · · · , α N , we turn off the overall U(1) holonomy N i=1 α i = 0 and replace all α i by the SU(N ) holonomies α i defined by α i ≡ α i+1 − α i . The final expressions depend on α 1 , · · · , α N −1 .
SO(2) vs U(1). The instanton partition functions for Abelian theories are, checked up to q 3 order, They are independent of the gauge holonomies, since the Abelian theories do not involve electrically charged particles. They agree with the 6d index on R 4 × T 2 for a free (2, 0) tensor multiplet [5].

JHEP12(2016)148
SO(4) vs SU(2) 2 . The SO(4) and SU(2) 2 instanton partition functions satisfy the following relation which was checked by taking the series expansion in w 1,2 , up to w 3 1 w 3 2 q 2 order. The SO(4) instanton partition function depends on the SO(4) holonomies α 1 and α 2 , which are distances of two D4-branes from the orientifold. There are two kinds of charged W-bosons in the SO(4) gauge theory, induced from open strings whose lengths are |α 1 ±α 2 |. The relation (4.2) shows that each type of W-bosons is that of the SU(2) gauge theory. SO(6) vs SU(4). The SO(6) and SU(4) instanton partition functions satisfy the following relation which was checked by taking the series expansion in w 1,2,3 , up to w 1 1 w 1 2 w 1 3 q 2 order. The SO(6) gauge holonomies α 1,2,3 , measuring the distances of three D4-branes from the orientifold, appear in the SO(6) instanton partition function. There are three types of charged W-bosons in the SO(6) gauge theory, induced from open strings whose lengths are |α 1 ± α 2 | and |α 2 − α 3 |. They correspond to three kinds of SU(4) W-bosons, according to the relation (4.3). (1)| θ=0 . The instanton partition functions for these theories are written as

SU(2) vs SO(3) vs Sp
where α 1 = α 1 − α 2 . We divide the instanton partition functions into two parts: the first term f is dependent on the Coulomb VEVs. The second term is independent of the Coulomb VEVs. We find that the 4d QFT constraint is satisfied, , checked up to q 2 w 3 order. However, the Coulomb VEV independent term of the SO(3) theory disagrees with those of the SU(2) and Sp(1) theories, using the same identification of q's. It implies that these gauge theories do not have the same UV completion. The Sp(1) gauge theory with θ = π should satisfy the same constraint, which can be straightforwardly checked but we have not checked. The four 5d theories with SU(2), SO(3), Sp(1) θ=0 , Sp(1) θ=π uplift to 6d (2, 0) theories of A 1 -type, A 1 -type with Z 2 outer automorphism twist, D 2 -type with Z 2 outer automorphism twist, and A 2 -type with Z 2 outer automorphism twist, respectively [17]. We expect the differences in the Coulomb VEV independent parts to reflect such distinct UV completions.

Index on R 4 × T 2 and S-duality
Now we study if the instanton partition functions respect S-dualities of 4d N = 4 gauge theories, and their 6d uplifts on tori. S-duality identifies certain pairs of 4d N = 4 gauge theories, such as [18][19][20][21]  are related as τ ∨ 4 = − 1 n G τ 4 . Recall that 4d N = 4 SYMs are obtained from (2, 0) theories on tori, whose complex structures τ ≡ i β/2π R M are translated to gauge couplings τ 4 [1]. The precise relations between complex structures τ and gauge couplings τ 4 are given in [50]. From this viewpoint, S-dualities merely exchange two sides of the tori.
Our instanton partition functions are six-dimensional observables. However, they depend only on the complex structures τ of the tori, and not on their volumes. So these partition functions are expected to respect the geometric S-dualities in certain forms. For instance, the S-duality transformation of the partition function for the Abelian (2, 0) theory has been studied in [9], as we reviewed in section 3. Even in such simple theory, the transformation of the 6d partition function is fairly non-trivial. For general non-Abelian theories, we do not know the S-duality transformations of the partition functions, keeping all the chemical potentials. However, the partition functions may simplify after taking JHEP12(2016)148 special limits of the chemical potentials, so that we can explicitly see the S-duality of the partition functions.
We are interested in two different limits of the partition functions, which are closely related to each other. One is to take all the fugacities w i for the U(1) N electric charges to be zero. In this limit, the partition function acquires contributions only from the neutral states. Another limit we are interested in is m = + . This limit also makes the contributions from charged sectors automatically vanish. It is because W-bosons always carry a factor of 2 sinh m± + 2 in their indices, coming from the supersymmetries broken by these BPS states [5]. Since this factor vanishes at m = + , the index acquires contributions only from neutral instantons unbound to W-bosons. The 6d partition functions in both limits can be easily dualized to test the S-dualities suggested in the literatures.
We first consider the limit w i → 0. For various gauge groups, we observe that the instanton partition functions in electrically neutral sectors are given as checked for N ≤ 3, up to q 5 order.
All of these partition functions include the 6d index on R 4 × T 2 for N free (2, 0) tensor multiplets, whose S-duality transformations were studied in [9]. Extra terms in the Sp(N ) partition functions are already discussed in section 3.
We are further interested in the special limit m → + where the instanton partition functions simplify into the following forms,

JHEP12(2016)148
up to the overall factors e −N πiτ /12 and e −(N ±1)πiτ /12 . These expressions will be useful to study the 6d superconformal indices in section 4.3. The SO(2N ) partition functions are invariant under the S-duality τ → − 1 τ as expected. We expect the other two Sp(N ) partition functions to be S-dualized to the partition functions of the 5d gauge theories on a circle with twisted boundary conditions [17]. It will be interesting to explicitly study the dual partition functions with twisted boundary conditions.

6d superconformal index and W algebra
Here we study the S 5 partition functions that correspond to the (2, 0) superconformal indices [6,7,22,23]. The superconformal indices [24] are the SUSY partition functions on S 5 × S 1 which capture the 6d BPS operator spectra, due to the operator-state correspondence of radially quantized CFTs.
It was proposed in [7,23,51] that the S 5 partition functions are computed by merging three Nekrasov partition functions on R 4 × S 1 . They depend on the dimensionless coupling σ = g 2 5 2πr 5 , made of the S 5 radius r 5 and the Yang-Mills coupling g 5 . σ can be interpreted as the chemical potential analogous to inverse temperature on S 5 × S 1 . The expressions in [7,23,51] take the form of weak coupling expansions (σ 1), while the 6d spectral data are easier to read off in the strong coupling regime (σ 1) by making a series expansion in the fugacity e −σ . For example, strong coupling expansion was made for the S 5 partition functions of U(N ) gauge theories in a special unrefined limit [6,7]. The results give the (2, 0) superconformal indices of SU(N )-type, reproducing the BPS Kaluza-Klein spectrum of AdS 7 × S 4 supergravity in the large N limit. Also, they take the form of the vacuum character of W A N −1 algebra for finite N , leading to the W algebra conjecture [25]. We want to compute the S 5 partition functions for other gauge groups: SO(2N ), SO(2N + 1), Sp(N ). 6 We follow the notations of [6,7]. The path integrals for the S 5 partition functions are reduced to the following matrix integrals inst . (4.14) N is the rank of 5d gauge group. a, b, c are the S 5 squashing parameters a, b, c satisfying a + b + c = 0. σ = g 2 5 2πr 5 is the dimensionless coupling made of the S 5 radius r 5 and the Yang-Mills coupling g 5 . We use the normalized trace Tr(T a T b ) = δ ab and set a long root ϑ of G to satisfy |ϑ| 2 = 2. The S 5 Yang-Mills action is written as 1 Tr (F µν F µν ). The 6 In the proposal to use 5 dimensional Yang-Mills theory on S 5 to study the 6d index, it is unclear to us whether one should use O(n) or SO(n) as the 5d gauge groups. If the orientifold picture works even in curved 5d background with Euclidean signature, it might be more natural to use O(n) gauge theories. Even if the gauge groups are O(n), the following analysis in our paper will still be unchanged if there are no extra saddle points in the path integral on S 5 which use the O(n)− group elements. In fact, S 5 does not have 1-cycle on which discrete Z2 holonomies can be turned. So we think it is natural to assume the absence of extra saddle points in O(n) theories. This is in contrast with the path integral over R 4 × S 1 , as we explored in section 3 for O(2)− theories.

JHEP12(2016)148
unit instanton action is 4π 2 σ for all gauge groups. Z pert are the indices for BPS bound states of perturbative W-bosons only. Z inst are the instanton partition functions that we have studied so far. Omega-deformation parameters 1 , 2 of Z pert Z inst are identified with the squashing parameters a, b, c, such that ( 1 , 2 ) = (b − a, c − a), (c − b, a − b), (a − c, b − c) at each fixed point. See [7] for the details.
Re-arranging the S 5 partition functions into strong coupling expressions is generally very difficult. Only the Abelian (2, 0) index [24] was reproduced from the 5d U(1) partition function on S 5 [7]. Instead, we consider the special limit m = 1 2 and a, b, c → 0 in which the S 5 partition functions preserve 8 SUSYs. We can make the precise analysis in this limit as follows. The instanton partition functions are simplified such that the net instanton contributions come from only one fixed point.
inst → Z inst | m= + (4.15) if the limit a, b, c → 0 is taken in a suitable order [7]. For all classical gauge groups, the instanton corrections are given as follows: We can also handle the matrix integrals (4.14) over the perturbative determinants e −2π 2 Tr λ 2 /σ · Z pert . For classical gauge groups G, the integrals become [7]  where ∆ + G is the set of positive roots of G. c 2 is the dual Coxeter number. |G| is the dimension of the semi-simple part of G. ρ is the Weyl vector. The final results are products of (4. 19) and (4.20).

JHEP12(2016)148
We summarize the exact S 5 partition functions for all classical gauge groups as follows: twists. However, we have no good explanations at the moment. These indices are compatible with the twisted compactification realizations of non-simply-laced gauge theories, in that they have smooth large N limits. In the large N limit, these indices are given as . (4.28) These results may shed more lights on what one precisely means by outer automorphism twists of 6d (2, 0) theories.

Conclusion
In this paper, we studied the instanton partition functions for 5d N = 2 SYM with classical gauge groups. Our approach was to utilize the ADHM quantum mechanics of the D0-D4-O4 systems. The instanton partition functions were obtained by computing their Witten indices and extracting out the extra states' index contribution. The extra factor captured the BPS spectrum of D0-branes in pure orientifold backgrounds. We also studied S-dualities of various O4-planes on S 1 . The instanton partition functions respect 6d S-dualities which are uplifts of 4d Sdualities. Using them, we also computed the S 5 partition functions which correspond to the 6d superconformal indices. In the special limit, our final results showed that the superconformal indices of SO(2N )-type (2, 0) theories agree with the vacuum character of W D N algebra. The instanton partition functions for non-simply-laced SO(2N + 1), Sp(N )| θ=0 , Sp(N )| θ=π groups also produced interesting results. They are proposed to be the indices of (2, 0) theories with outer automorphism twists.
It would be important to understand S-dualities of the instanton partition functions on R 4 × S 1 in full generality. This will be useful to study the spectrum of (2, 0) theories via the superconformal indices. More generally, it would allow us to have better understanding on the high temperature behavior of (2, 0) theories.