Instanton Effects in Orientifold ABJM Theory

We investigate another supersymmetric Chern-Simons theory called the orientifold ABJM theory, which replaces the unitary supergroup structure of the ABJM theory with an orthosymplectic one. Its non-perturbative structure is completely clarified by considering the duplication of the quiver.


Introduction
Recently, there has been much progress in understanding the partition function of supersymmetric Chern-Simons theories. The first of such theories was, of course, the N = 6 supersymmetric one called ABJM theory [1], which has the gauge group U(N) k ×U(N) −k (with the subscripts denoting the Chern-Simons levels) and two pairs of bifundamental matters between the two U(N) factors. This theory is the worldvolume theory of multiple M2-branes placed on a background C 4 /Z k . Using the localization theorem, the partition function Z k (N) on S 3 is reduced to a matrix model [2], where the resulting expression can be interpreted in terms of the hidden supergroup U(N|N) [3]. After the perturbative terms of the partition function Z k (N) were summed to obtain an Airy function [4] following the study of the large N behavior N 3 2 in [5,6], the Fermi gas formalism was proposed [7], where the Airy function is easily rederived in the disguise of a cubic polynomial in the grand canonical ensemble. Switching to the grand potential J k (µ), defined with the oscillatory terms, which originates from the trivial 2πi shift of µ, properly subtracted [8], we found that the grand potential has both perturbative and non-perturbative contributions, where the result is simplified using the redefined effective chemical potential µ eff = µ+O(e −2µ ). In addition to the perturbative part, giving rise to the Airy function, we were able to study the non-perturbative effects explicitly [8][9][10][11], which turned out to be expressed in terms of the free energy of topological strings on local P 1 × P 1 (s L = 2j L + 1, s R = 2j R + 1), s R sin 2πg s ns L n(2 sin πg s n) 2 sin 2πg s n e −nd·T eff + ∂ ∂g s g s − sin πn gs s L sin πn gs s R 4πn 2 (sin πn gs ) 3 e − nd·T eff gs . (1.4) Here the Kähler parameters T 1 eff , T 2 eff and the coupling constant g s are identified with the chemical potential and the inverse level by with the B-field effect ±πi, and the overall integral coefficient N d j L ,j R is the BPS index. It can be proved that N d j L ,j R is non-vanishing only when 2j L + 2j R − 1 is even (see e.g. [11]). It is then interesting to ask how the grand potential is constrained so that it reduces to the free energy of topological strings. We are also interested in whether there are any other theories that have the same topological string expression and whether there are any expressions other than the topological one that can appear as the grand potential in the supersymmetric Chern-Simons theories.
To answer these questions, some generalizations may be helpful. The ability to make generalizations depends on the number of supersymmetries. Basically, the more supersymmetries the theory possesses, the more solvable the theory is. The N = 3 supersymmetric theories were constructed for all gauge groups and all matters. Among them, the A M quiver gauge theories with the gauge group M +1 a=1 U(N) ka (having the vanishing total level a k a = 0) and pairs of the bifundamental matters cyclically connecting two adjacent factors U(N) a and U(N) a+1 were proposed. It was found that when the levels are expressed as k a = (k/2)(s a − s a−1 ) with s a = ±1, the theories have enhanced N = 4 supersymmetries [12].
Two major examples of the generalizations are the orbifold ABJM theory and the (q, p) model. The (q, p) model [13] is described by a quiver with q sequential edges of s a = +1 and p edges of s a = −1. In particular, it was exciting to find that the grand potential of the (2, 2) model takes the same topological string expression (1.4) with the diagonal Gopakumar-Vafa invariant being that of the D 5 del Pezzo geometry [14]. See [15] for further intensive studies on general values of (q, p).
In terms of the quiver diagram, the orbifold ABJM theory is an r-times repetition of the original ABJM theory [16,17]. In [18] (see also [19]) it was found that as long as the new quiver is an r-times repetition of the original quiver, the grand potential of the new theory J k (µ) is expressed in terms of that of the original theory J [1] k (µ) by More explicitly, for r = 2, we have Here, we usually project out some terms in the first line but we need to add the second line to preserve the 2πi shift symmetry. In this sense, we will often refer to the first line with n = 0 as the untwisted sector and the second line with n = 0 as the twisted sectors using the terminology of the string orbifold theory. This analogy works even for r > 2. Although the expression (1.6) appears to contain infinite terms, the summation terminates at a finite n for a fixed instanton number because the exponent of each twisted (n = 0) sector always contains a decaying factor originating from the perturbative terms.
In this paper, we will further explore generalizations of the ABJM theory. Although the progress is guided by the number of supersymmetries, as we have already mentioned, there remains a theory with a large number of supersymmetries for which the discussion of its nonperturbative effects is still missing: the N = 5 orientifold ABJM theory [20,21]. This theory can be realized in a type IIB setup as the worldvolume theory of a brane system by adding an O3 − -plane in parallel with the original D3-branes used to realize the ABJM theory. In the dual M-theory description, this theory describes multiple M2-branes in M-theory on a D-type orbifold [20,21]. Again, after the supersymmetric localization, the partition function is reduced to a matrix model. Instead of the previous interpretation in terms of the supergroup U(N|N) for the ABJM theory, this theory can be interpreted in terms of the supergroup OSp(2N|2N). Hereafter, we will call this theory the orthosymplectic (OSp) theory. We follow the recent progress in establishing the Fermi gas formalism and compute exact values of the partition function Z k (N) for various values of k and N. Surprisingly, when we compare the resultant grand potential with that of the ABJM theory, we encounter the same structure as (1.6). Namely, we find that the grand potential of the OSp theory is related to that of the ABJM theory by is the grand potential of a duplicate OSp theory in the sense of (1.7), while J is the grand potential of the original ABJM theory. Since we already know J ABJM k (µ), this relation almost fixes the grand potential of the OSp theory. In this sense, we claim that the partition function of the OSp theory on S 3 is solved. See figure 1 for a schematic expression and section 2.5 for further explanation.
The organization of this paper is as follows. In the next section, we will study the OSp matrix model in full detail, first presenting the Fermi gas formalism and then analyzing it. Then, we conclude with some discussion on future directions. The appendix is devoted to the integration formulas necessary for the analysis of the OSp matrix model.

OSp matrix model
Let us start our analysis of the partition function of the OSp theory. This theory was proposed in [20,21] and possesses the enhanced N = 5 supersymmetries. The gauge group of the theory is O(2N)×USp(2N), whose Chern-Simons levels are k and k/2, respectively, in the convention of [22] ‡ . According to [22], k must be an even integer. ‡ Note that the parameter k appearing in [20] is different from k in this paper, k here = 2k HLLLP , owing to the difference in the normalization of the trace.
After applying the localization technique [2], the partition function on S 3 is given by § with the integration originating from the tree-level contribution and the measure originating from the one-loop contributions of the vector multiplets and the hypermultiplets Note that the integration measure is closely related to the invariant measure of OSp(2N|2N) in the same way that the integration measure of the ABJM theory is related to U(N|N).

Fermi gas formalism
Now let us propose a Fermi gas formalism for the OSp matrix model. The Fermi gas formalism for this matrix model was first studied in [23] and many important tools have already been given in [24].
Using the Cauchy determinant formula we can rewrite the partition function (2.1) as . (2.5) § We assume that the level k is positive. For negative k, we can simply consider its complex conjugate.
After expanding the determinant and rescaling the integration variables, we obtain Recall that to rewrite the partition function of the ABJM matrix model into that of a Fermi gas system [7], the phase space operators q and p satisfying [ q, p] = i and the coordinate eigenstate |µ normalized by µ|ν = 2πδ(µ − ν) were introduced. Using them, we were able to express the entries of the determinant as with the identification = 2πk. To deal with the extra factor of the hyperbolic sine functions, the use of the following reflection operator and projectors was proposed in [23]: Then, using we can rewrite the partition function (2.6) as where the Hamiltonian is given by (2.11) Note that R and (2 cosh p 2 ) −1 are commutative. This is why the expressions in (2.9) make sense. The two projectors appearing in (2.11) are redundant. We can always combine them, but let us rewrite (2.11) as (2.12) The Hamiltonian looks considerably more difficult than that in the original ABJM theory. However, a key observation for studying this Hamiltonian has already been proposed in [24]. From the Baker-Campbell-Hausdorff formula, we can show Naively, it appears that using (2.13) we can cancel the unwanted hyperbolic sine functions to obtain −1 in the Hamiltonian (2.12). It is not, however, as simple. When we consider the cancellation, the following remaining term appears: (2.14) After combining (2.14) with the projector Π + , the µ and ν dependences decouple, Since the right-hand side can be expressed as with 0| being the p = 0 eigenstate which is normalized such that after the cancellation we end up with two projectors, Using (2.19), we can rewrite the Hamiltonian as (2.20) Hence the grand canonical partition function can be expressed as if we use the formula for a general operator X and states |Y , Y | normalized as Y |Y = 1.
Before ending this subsection, we note that there is another way of rewriting the partition function (2.6) as the form (2.10). Namely, instead of (2.11) we can define a different Hamiltonian by exchanging the projectors in (2.11), (2.24) to obtain the same expression (2.6) via (2.10). However, one may notice that the expression (2.24) is not as helpful as (2.11) since the singularity of the matrix elements µ|e − H ′ |ν (due to the presence of a hyperbolic sine function in the denominator) is not cancelled by the projectors. In the following, we employ the Hamiltonian given by (2.11).

Even-parity determinant
In this subsection we explain the computation of the determinant in (2.22). After a similarity transformation, (2. 25) we find that the determinant is rewritten as (2.27) The matrix element of the density matrix ρ + (µ, ν) = µ| ρ + |ν is simply the even-parity one (2.28) which has already been introduced in [24]. The only difference is that, owing to the extra minus sign appearing in (2.19), the Fermi statistics is changed into the Bose statistics (2.26). See [9] for the appearance of the Bose statistics in the TBA equations.
As was already noted in [8], similarly to the original density matrix of the ABJM theory, we can rewrite this density matrix in the form This expression implies that we can compute the powers of the density matrix ρ n + simply by selecting one vector E and repeatedly multiplying it by the density matrix ρ + , .

Matrix elements
Next, let us turn to the new term in (2.22). As in the previous subsection, we perform the similarity transformation and rewrite each order of z as (2.33) To study it, let us first compute the matrix elements where we have used Using these formulas, we find that the matrix elements are given by Note that in the computation of either the determinant using (2.31) in the previous subsection or the matrix elements using (2.36) in this subsection, our starting point is the same. We can select one vector E and repeatedly multiply it by the density matrix ρ + .
In the last step of the computation, we must perform integrations of the form dν e n 1 +n 2 2 ν (cosh m 1 2k ν)(sinh m 2 2k ν)ν p (e ν + 1) n 1 (e ν − 1) n 2 e i 4πk ν 2 , (2.38) for non-negative integers n 1 , n 2 , m 1 , m 2 and p. For the integration, we first apply the partial fraction decomposition to make either n 1 or n 2 equal to zero. For the case of non-vanishing n 2 , we can shift the integration variable ν by πi so that the integration reduces to the case of non-vanishing n 1 . Finally, all we have to consider is the integration of (2.39) The integration formulas for I m,p n are given in appendix A.

Numerical results
Using the Fermi gas formalism we have established, we find the first few exact values of the partition function Z k (N) via (2.21). We first observe that the complex phase is given by After finding these exact values of the partition function, we can read off the perturbative and non-perturbative coefficients of the grand potential J k (µ) (1.1) by numerical fitting. As in the analysis [25] of the ABJ theory [20,21], the partition function has a simple phase dependence (2.40) and we define the grand potential J k (µ) by taking absolute values. Details of the numerical fitting can be found in [26], which we only explain briefly in the following. We make an ansatz for the grand potential J k (µ) = J with polynomials f a (µ), and find the best fit for the coefficients of the polynomials f a (µ) by comparing the absolute values of the partition function |Z k (N)| in table 1 with the inverse transformation of (1.1), Then, the perturbative coefficients are found to be given by while the non-perturbative coefficients are summarized in table 2. It is interesting to observe that when k = 1, J OSp 1 (µ) coincides with the grand potential constructed from det(1 + zρ + ) in [24,27].
If we compare the results in table 2 with those of the ABJM theory, we can make the following observations.
• For the perturbative part, the coefficients in the two theories are related by (2.44) • The non-perturbative part consists of exponents e − 4m k µ and e −ℓµ (m, ℓ ∈ N). These two exponents are identified as the worldsheet instanton and the membrane instanton, respectively, from their k-dependences. ¶ Formulas for N = 1, 2 and a general value k have been given by Tomoki Nosaka. We have checked their consistency with table 1. We are grateful to him for sharing his results with us. , , , |Z 3 (4)| = 1188 + 36(9 + 10 √ 6)π + (−4721 + 240 , , |Z 6 (2)| = −216 + (209 − 108 √ 3)π 2 41472π 2 ,   • Compared with the membrane instanton of the ABJM theory, where the exponents are e −2ℓµ , we also encounter odd powers of e −µ for the OSp theory. However, the coefficients of these terms are quite simple, (2.45) • The coefficient polynomials of e −2ℓµ proportional to π −2 are the same for the OSp theory and the ABJM theory except for an extra relative factor 2(−1) ℓ .

From OSp to ABJM
In the previous subsection, we found an explicit expression for the grand potential for k = 1, 2, 3, 4, 5, 6, 8, 12 and made several observations on the exponents and the coefficients.
These observations suggest that the argument µ ± πi/2 should be considered. * * In fact, it is not difficult to find that and notice that, when generalizing the above equation non-perturbatively, the extra nonperturbative e −ℓµ terms with odd ℓ on the left-hand side cancel among themselves.
As reviewed in the introduction, the analysis in [18] implies that the grand potential has an intricate constraint and, in particular, it is not appropriate to consider only the sum of two terms such as in the above equation. Instead, we can study the duplicate quiver using (1.7). Surprisingly, we find that the result correctly reproduces the grand potential of the ABJM theory as in (1.8), including the non-perturbative terms. For a comparison, see appendix A in [10].
Since the partition function of the ABJM theory is known, from this observation, we can solve the whole non-perturbative expansion except for the e −ℓµ terms with odd ℓ. Combining Owing to the coincidence noted above, J OSp (odd MB) k=1 (µ) in (2.45) has already appeared in [27]. Although the results for k ≡ 7 mod 8 are missing and we only have one result for k ≡ 1, 2, 3, 5, 6 mod 8, we believe that our formula (2.45) is a natural extrapolation. * * We are grateful to Tomoki Nosaka for valuable discussions on this point.
this with the observation (2.45), we claim that we have solved the partition function of the OSp theory for all integers k.
Note that, since in [18] the formula (1.7) is derived under the assumption Z k (N) > 0, it must be applied with care to the OSp theory because of the phase factor (2.40). Because of this phase factor and the rescaling by a factor of 2, the relation (2.47) does not literally mean that the grand potential of the duplicate OSp theory is the same as that of the ABJM theory. Instead, the grand partition function of the duplicate OSp theory is

Summary and discussion
We have studied the partition function of the OSp theory, which is the N = 5 supersymmetric orthosymplectic Chern-Simons theory with the gauge group O(2N)×USp(2N). Its nonperturbative structure was clarified by considering the duplication of the quiver.
We list a few technical issues that remain unresolved in solving the OSp theory, and then discuss some future directions.
First, the WKB expansion of the OSp theory is missing. We directly proceeded to the evaluation of the partition function but it is desirable to also study the WKB expansion to obtain a full understanding of the membrane instantons. For example, we do not know how the odd membrane instantons behave for non-integral k.
Secondly, we have followed the standard method to obtain the non-perturbative expansion of the grand potential and substituted the results into the repetition formula (1.7) to find the relation (1.8) between the OSp theory and the ABJM theory. However, it would be interesting to attempt to prove the relation directly from the density matrix in (2.11). A direct proof would lead to more general relations between the families of orientifold theories and those of original theories.
Here we have only studied the partition function of the OSp theory with equal ranks. As in the ABJM theory, it would be interesting to study the BPS Wilson loop in the theory and/or the ABJ deformation O(2N 1 )×USp(2N 2 ). A particularly interesting question is when the BPS Wilson loop reduces to the character of the OSp (2N 1 |2N 2 ) supergroup in the localization. For the OSp character, we hope that the formalism in [25,28] will directly apply to the OSp theory. Using this formalism, among others, we hope to establish whether our relation between the OSp theory and the ABJM theory (1.8) holds for the ABJ deformation. Also, it will be interesting to study the OSp theory in terms of topological string theory as in the ABJM theory. For example, study from the viewpoint of the dual spectral problem [29][30][31][32] may also be worthwhile. We would like to pursue these directions following our previous attempts [33,34].
Technically, the Fermi gas Hamiltonians in the OSp matrix model and the D-type quiver matrix model [35,36] are similar. It would be interesting to determine the relation between the two Hamiltonians.
Let us now return to our original question of the constraint on the partition function. Partition functions in physical theories often have a certain kind of modular invariance. The most standard example is, of course, the one-loop string worldsheet partition function, which is expressed by modular invariant combinations of theta functions. Owing to the constraints imposed by the modular invariance, we cannot construct a new theory simply by projecting out some excitation modes. In addition to the untwisted sector, which is obtained simply from the orbifold or orientifold projection, we need to add twisted sectors to recover the broken modular invariance.
Similarly, the expression for the supersymmetric Chern-Simons theory is considered to be strongly constrained. In parallel to the worldsheet partition function we have called the n = 0 part in (1.7) the untwisted sector and the n = 0 part the twisted sector. This similarity led us to (2.47) from (2.46). However, we do not have a clear picture of the modular invariance behind the constraint. A study of the modular invariance in five-dimensional partition functions [37] or in multiple sine functions [38] may increase our understanding of the matrix model studied in this work.
discussions. The work of S.M. was supported by JSPS Grant-in-Aid for Scientific Research (C) # 26400245. S.M. would like to thank Yukawa Institute for Theoretical Physics at Kyoto University for hospitality.

A Integrals
To compute the grand potential at each order of z exactly, we need to evaluate the integrals I m,p n given by (2.39). Using integration by parts, we easily obtain the recursive formula

A.1 Odd k
For the evaluation of I m,0 n with even m, we first rewrite the integrand using 1 (e ν + 1) n = e ν + 1 (e ν + 1) n+1 . (A.2) Then, we find that the e ν term is simply the same integral with the integration contour inversed and shifted by 2πik,