Orientifolding of the ABJ Fermi gas

The grand partition functions of ABJ theory can be factorized into even and odd parts under the reflection of fermion coordinate in the Fermi gas approach. In some cases, the even/odd part of ABJ grand partition function is equal to that of $\mathcal{N}=5$ $O(n)\times USp(n')$ theory, hence it is natural to think of the even/odd projection of grand partition function as an orientifolding of ABJ Fermi gas system. By a systematic WKB analysis, we determine the coefficients in the perturbative part of grand potential of such orientifold ABJ theory. We also find the exact form of the first few"half-instanton"corrections coming from the twisted sector of the reflection of fermion coordinate. For the Chern-Simons level $k=2,4,8$ we find closed form expressions of the grand partition functions of orientifold ABJ theory, and for $k=2,4$ we prove the functional relations among the grand partition functions conjectured in arXiv:1410.7658.


Introduction
The Fermi gas approach, first introduced in [1], to study the partition function of certain 3d Chern-Simons-matter theories on S 3 enables us to extract the valuable information of the non-perturbative effects in the holographically dual M-theory side. In particular, in the case of N = 6 U (N + M ) k × U (N ) −k ABJ(M) theory, which is dual to M-theory on AdS 4 × S 7 /Z k , we have a very detailed understanding of the instanton effects coming from the M2-branes wrapping some 3-cycles, thanks to the relation to the (refined) topological string on local P 1 × P 1 [2] (see [3] for a review). Remarkably, for the special values of the Chern-Simons level k = 1, 2, 4, it is found that the grand partition functions of ABJ(M) theories can be written explicitly in closed forms in terms of the Jacobi theta functions [4,5]. The grand partition functions of ABJ(M) theories can be naturally factorized into the even part and the odd part under the reflection x → −x of the fermion coordinate, and it is conjectured in [5] that the factorized grand partition functions of ABJ theories enjoy certain functional relations, which are reminiscent of the quantum Wronskian relation obeyed by the spectral determinant of some quantum mechanical system [6,7]. Recently, it is realized that such projection to the even/odd sectors under the reflection of Fermi gas system has a close relation to the orientifolding on the bulk M-theory side [8][9][10][11][12][13]. In this paper, we will consider the grand partition functions of the even/odd sectors of the ABJ Fermi gas under the reflection, which we call the "orientifold ABJ theory". For some special values of parameter M , it is found that such orientifold ABJ theory is related to the Fermi gas representation of the N = 5 O(n) × U Sp(n ) Chern-Simons-matter theory [11,13], which is obtained by introducing an orientifold plane into the brane construction of ABJ theory [14]. Thus, the "orientifold ABJ theory" in such cases are indeed related to the orientifolding in the bulk M-theory side.
In this paper, we will study the (grand) partition function of orientifold ABJ theory in detail. We develop a systematic method to compute the small k expansion (WKB expansion) of the grand potential of orientifold ABJ theory, and determine the coefficients C, B and A in the perturbative part of grand potential. We also study the coefficients of "half-instanton" corrections to the grand potential, which have half-weight O(e −µ ) of the membrane instanton O(e −2µ ) in ABJ theory. This type of half-instanton corrections has also appeared in other theories [11,12] which have dual description as M-theory on certain orientifolds, and it is natural to identify such half-instantons as the effect coming from orientifold planes.
We also compute the exact values of the canonical partition functions of orientifold ABJ theory up to some N = N max for various values of k, M . Using this data of exact values of partition functions, one can extract the instanton corrections to the grand potential which are summarized in Appendix B. For the case of k = 2, 4, 8 in Appendix B, by looking at the first few coefficients of this instanton expansion, one can easily guess the all-order resummation of instanton corrections and write down the closed form expression of the grand potential. Once we know the exact grand potential, we can reconstruct the grand partition function by performing the "periodic sum" in (2.5) [15]. In this way, we find the closed form expressions of the grand partition functions of orientifold ABJ theory with k = 2, 4, 8. It turns out that the grand potential at k = 2, 4 are essentially determined by the genus-zero and genus-one free energies of the (refined) topological string on local P 1 × P 1 , and the exact grand partition functions of orientifold ABJ theory for k = 2, 4 are proportional to the Jacobi theta function ϑ 3 (v, τ ), as in the case of ABJ(M) theory before orientifolding [4,5]. We find that the exact forms of grand partition functions with k = 2, 4 we obtained indeed satisfy the functional relations conjectured in [5]. On the other hand, the exact grand partition function for k = 8 is qualitatively different from the k = 2, 4 cases, due to the all-genus corrections in the worldsheet instanton part of grand potential. This paper is organized as follows. In section 2, we compute the exact values of the canonical partition functions of orientifold ABJ theories for various k and M , and determine the instanton corrections to the grand potential, which are summarized in Appendix B.
In section 3, we study the WKB expansion of the grand potential of orientifold ABJ theory, and find the coefficients C, B and A in the perturbative part of grand potential in closed forms. We also find the closed form expression of the first few coefficients of "half-instantons" coming from the twisted sector of the reflection of fermion coordinate. In section 4, 5, and 6, we write down the exact grand partition functions of orientifold ABJ theory at k = 2, k = 4, and k = 8, respectively. In section 7, for the k = 2, 4 cases we prove the functional relations among grand partition functions conjectured in [5]. We conclude in section 8. Some useful properties of theta functions are summarized in Appendix A.

Exact partition functions of orientifold ABJ theory
In this section, we will compute the exact values of the canonical partition functions Z ± (N, k, M ) of orientifold ABJ theory for various integral k, M up to some N = N max . Using the exact data of Z ± (N, k, M ), we determine the instanton corrections to the grand potential of orientifold ABJ theory.

Review of ABJ Fermi gas
Before going to the orientifold ABJ theory, let us first review the Fermi gas approach of U (N 1 ) k × U (N 2 ) −k ABJ theory. It is convenient to parametrize the theory by (N, k, M ), where k is the Chern-Simons level, N is the rank of the second gauge group, and M is the difference of the ranks of the first and the second gauge groups: (2.1) Without loss of generality, we can assume k > 0 and M ≥ 0. Note that the ABJM theory is a special case of ABJ theory with M = 0. We are interested in the grand partition function of ABJ theory on S 3 obtained from the canonical one by summing over N with fixed k, M where µ is the chemical potential conjugate to N . In the Fermi gas approach [1], the grand partition function of ABJ theory (2.2) is written as a system of fermions with no multi-body interactions (ideal Fermi gas) Ξ(µ, k, M ) = det(1 + e µ ρ). (2. 3) The density matrix ρ of ABJ Fermi gas is given by [16][17][18]  As discussed in [15], the large µ expansion of the total grand potential log Ξ(µ, k, M ) consists of two parts: the oscillatory part and the non-oscillatory part. We will focus on the latter part, known as the modified grand potential J(µ, k, M ). The grand partition function can be reconstructed from the modified grand potential J(µ, k, M ) by the following periodic sum [15] Ξ(µ, k, M ) = n∈Z e J(µ+2πin,k,M ) . (2.5) This summation over the 2πi-shift of µ recovers the periodicity of grand partition function Ξ(µ + 2πi, k, M ) = Ξ(µ, k, M ), (2.6) which is required from the very definition of Ξ(µ, k, M ) in (2.2). Moreover, as discussed in [15] one can obtain the canonical partition function at fixed N from the integral transform of the modified grand potential where C is a contour on the complex µ-plane running from e − πi 3 ∞ to e + πi 3 ∞. According to the conjecture of [2], the modified grand potential of ABJ theory is completely determined by the refined topological string on local P 1 × P 1 J(µ, k, M ) = F top (T 1 , T 2 , g s ) + 1 2πi where g s = 2/k denotes the string coupling and F top and F NS represent the free energy of the standard topological string and the refined topological string in the Nekrasov-Shatashvili limit, respectively. The Kähler parameters T 1 , T 2 of local P 1 ×P 1 are related to the chemical potential µ of ABJ Fermi gas by Here we introduced the notation b = M k , (2.10) and the "effective" chemical potential µ eff in (2.9) is related to µ by the so-called quantum mirror map [20]. See [2] for more details. In [18], it is shown that this conjecture (2.8) is in complete agreement with the exact values of the canonical partition functions of ABJ theory.
It is useful to decompose the modified grand potential (2.8) into two parts J(µ, k, M ) = J pert (µ, k, M ) + J np (µ, k, M ), (2.11) where J pert (µ, k, M ) is the perturbative part J pert (µ, k, M ) = C(k) 3 µ 3 + B(k, M )µ + A(k, M ) (2.12) and J np (µ, k, M ) is the non-perturbative part which is exponentially suppressed in the large µ limit. The coefficients C(k) and B(k, M ) in the perturbative part (2.12) are given by [18,19] 13) and the constant A(k, M ) in (2.12) is given by [18] A(k, M ) = A c (k) + F coni (k, M ), (2.14) where A c (k) is the constant term in the ABJM theory, which is closely related to a resummation of the constant map contributions in the topological string [21][22][23] The second term in (2.14) denotes the free energy of the N = 2 U (M ) k pure Chern-Simons theory on S 3 2 (2. 16) Via large N duality, this is equal to the free energy of the topological string on the resolved conifold [25]. As discussed in [23], by a certain resummation of the genus expansion, we can write down a useful integral representation of (2.16) where F 0 (b) is the genus-zero free energy of resolved conifold Here we assumed k, b ∈ R, and Li s (z) = ∞ n=1 z n n s denotes the polylogarithm. This integral representation (2.17) defines a natural analytic continuation of k, M to non-integer values.
Plugging the decomposition (2.11) of the modified grand potential into (2.7) and expanding the non-perturbative part, we find that the canonical partition function Z(N ) at fixed N can be also decomposed into perturbative and non-perturbative parts: (2.19) where the perturbative part Z pert (N ) is given by the Airy function [1,26] 20) and the non-perturbative correction Z np (N ) is given by a sum of the derivatives of the Airy functions [15]. We should mention one important property of the grand partition function of ABJ theory. Due to the Seiberg-like duality [14], In terms of the parameter b in (2.10), the grand partition function is invariant under b → 1 − b, which corresponds to the exchange of two Kähler parameters T 1 ↔ T 2 in (2.9). For the physical ABJ theory with integer k and M , it is argued that the supersymmetry is spontaneously broken when M > k [14]. Therefore, for a fixed k, the independent values of M are M = 0, 1, · · · , k. Taking account of the Seiberg-like duality (2.21), the physically independent values of M are further reduced to

Orientifold ABJ theory
In this subsection, we will consider the even/odd projection of the ABJ Fermi gas under the reflection R of the fermion coordinate This kind of projection of Fermi gas system into R = ±1 sectors appeared previously in some examples [11][12][13], and those examples have holographically dual description as the M-theory on certain orientifolds. We expect that the projection of ABJ Fermi gas into even/odd sectors is also holographically dual to the M-theory on some orientifolds. Since the density matrix ρ(x, y) of ABJ theory is invariant under the reflection of coordinates we can consider the projection of ρ into the even/odd parts ρ ± which, in the operator language, is simply written as Then the grand partition function of ABJ theory is naturally factorized into the even/odd part where Ξ ± (µ, k, M ) are the Fredholm determinant of ρ ± Ξ ± (µ, k, M ) = det(1 + e µ ρ ± ). (2.28) From this grand partition function Ξ ± (µ, k, M ), we can read off the canonical partition function of orientifold ABJ theory Z ± (N, k, M ) with fixed N by expanding in e µ Note that the Seiberg-like duality (2.21) holds for each sector From (2.4), one can see that ρ + (x, y) can be written as This is exactly the form to which we can apply the Tracy-Widom lemma [27] and we can easily compute the exact values of spectral traces. Once we know the trace Tr ρ + from = 1 to = N , we can compute the canonical partition function Z + (N, k, M ) at fixed N . Using the lemma in [27], the th power of ρ + can be systematically computed by constructing a sequence of functions φ (x) ( = 0, 1, 2, · · · ) (2.32) Then Tr ρ + is given by The integrals in (2.32) and (2.33) can be evaluated by rewriting them as contour integrals as in [15,18,28]. On the other hand, the traces of ρ − can be computed by using the following relation found in [15] Using the above algorithm, we have computed the exact values of the partition functions Z ± (N, k, M ) for various (k, M ) up to N = N max , where N max is about 10-30 3 . For the case of k = 8, 12, we have computed Z ± (N, k, M ) only for odd M . As in the case of ABJM theory [15], by matching the exact values of Z ± (N, k, M ) with the expansion in terms of Airy function and its derivatives (2.19) and (2.20), we can fix the coefficients in the modified grand potential order by order for the first few instanton numbers. In this way, we find that the modified grand potential for ρ ± can be written in a similar form as that in the ABJ theory (2.11) Again, the perturbative part is a cubic polynomial in µ where the coefficients C ± (k) and B ± (k, M ) in (2.36) are related to those in the ABJ theory by By matching the exact values of Z ± (N, k, M ), we conjecture that the constant term A ± (k, M ) in (2.36) is given by Here A c (k) and F coni (k, M ) are given by (2.15) and (2.16), respectively. Very interestingly, A ± (k, M ) contains the contributions from non-orientable worldsheet instantons F non-ori (k, M ) on the orientifold of resolved conifold, which in turn is related to the SO(M + 1) k pure Chern-Simons theory on S 3 via large N duality [29] − log Z CS (SO(M + 1) (2.40) The appearance of non-orientable contributions in A ± (k, M ) is the first indication that the projection of ABJ Fermi gas into R = ±1 is closely related to the orientifolding.
In the next section, we will directly derive (2.37) and (2.38) from the small k expansion (WKB expansion) of the grand potential. k increases from the right curve (k = 3) to the left curve (k = 12). Note that the horizontal axis is N 3/2 . The dots are the exact values of free energy at integer N , while the solid curves represent the perturbative free energy given by the Airy function (2.20) with C, B in (2.37) and A in (2.38).
In Figure 1, we show the plot of free energy F = − log Z − (N, k, M ) as a function of N for M = 3 with k = 3, 4, 6, 8, 12, as an example. As we can see, the exact values of the free energy exhibit a nice agreement with the perturbative one given by the Airy function (2.20) if we use the correct coefficients C, B in (2.37) and A in (2.38). We find the similar agreement for all other cases. As discussed in [21], we can estimate the numerical values of A ± (k, M ) by In practice, we compute the numerical value of A ± (k, M ) from the exact value of Z ± (N max , k, M ) by setting N = N max in (2.41). In Figure 2 we show the plot of A − (k, M ) for M = 3, as an example. One can clearly see a nice agreement between the numerical value of A − (k, 3) estimated by using ( which is in good agreement with the numerical estimation (2.42). The difference between (2.42) and (2.43) can be attributed to the instanton corrections. The first instanton correction is of order e − 4µ k = e − µ 2 for k = 8 in the grand canonical picture, which in the canonical picture corresponds to the correction of the order This is indeed the same order as the difference between (2.42) and (2.43).
Once we know the perturbative grand potential, we can continue the above procedure to fix the coefficients in the non-perturbative part J np ± (µ, k, M ) using our exact data of partition functions Z ± (N, k, M ). The results are summarized in Appendix B. We find that there are three types of instanton corrections with the weight: The first two types have direct analogue in the ABJ theory, namely, worldsheet instantons and membrane instantons. On the other hand, the last type in (2.45) is a new contribution coming from the effect of orientifold projection, which we call "half-instantons", following [12]. It is curious to observe that there is no contribution with the half-weight of worldsheet instanton of the order O(e −2µ/k ). Only the half-weight of membrane instantons of order O(e −µ ) appear. For the cases of k = 2, 4, 8, by looking at the coefficients of expansion in Appendix B, one can easily guess the all order resummation of instanton expansion, and find the closed form expression of grand partition function Ξ ± (µ, k, M ), which we will consider in section 4, 5, and 6. By expanding the closed form expression of Ξ ± (µ, k, M ) around e µ = 0, one can read off the exact values of canonical partition functions Z ± (N, k, M ) up to arbitrarily high N , in principle. Therefore, one can "bootstrap" the computation of partition functions for k = 2, 4, 8: As discussed in [14], by introducing orientifold planes into the brane construction of ABJ theory, we can obtain N = 5 Chern-Simons-matter theories with gauge group O(n) × U Sp(n ) [14,30]. Now it is natural to ask whether such N = 5 theories are related to the projection of ABJ Fermi gas into even/odd sectors. In a recent paper [13], it is observed that the grand partition function constructed from ρ − of ABJM theory (M = 0) is equivalent to the grand partition function of Also, a Fermi gas formalism of N = 5 theory with different gauge group O(2N ) 2k × U Sp(2N ) −k is studied in [11]. We find that the exact values of the canonical partition functions in Table.1 of [11] all coincide with the canonical partition functions Z + (N, k, 1) computed from the density matrix ρ + of orientifold ABJ theory with M = 1. From this non-trivial agreement of the canonical partition functions at small N , we conjecture that the grand partition functions of these two theories are actually equivalent It would be interesting to find a direct proof of this equality.

WKB expansion
In this section, we will consider the small k expansion (WKB expansion) of the grand potential J ± (µ, k, M ) of orientifold ABJ theory. As discussed in [12], the total grand potential can be written as with ρ being the density matrix of ABJ theory. As we will see in the next section, at the level of modified grand potential On the other hand, we find that the difference J + − J − is always equal to J R in the examples listed in Appendix B. We should stress that the total grand grand partition function is completely factorized and the total grand potentials satisfy log Ξ + (µ, k, M ) + log Ξ − (µ, k, M ) = log Ξ(µ, k, M ). (3.5) The difference in (3.4) comes from rewriting a double sum into a single sum in (2.5) After the submission of this paper to arXiv, the paper [31] appeared. We realized that Table.2 in [31] for the ABJM case (M = 0) can be generalized to the orientifold ABJ theory as where Q1,2 = e −T 1,2 .
For the case of k = 2, 4, we can write down the difference in (3.4) explicitly using the exact form of grand partition functions, as we will see in section 4 and section 5.
To find the large µ expansion of J ± (µ, k, M ), it is useful to rewrite them in the Mellin-Barnes representation [32] (3.7) The contour γ is taken to be parallel to the imaginary s-axis with 0 < (s) < 1. Picking up the poles at s = ( = 1, 2, · · · ), we recover (3.2). On the other hand, closing the contour in the direction (s) ≤ 0 and picking up the poles on the negative real s-axis, we can find the large µ expansion. Thus, the behavior of J ± (µ, k, M ) is encoded in the analytic properties of the spectral trace Tr(ρ s ) and the twisted spectral trace Tr(ρ s R). In this section, we will consider the WKB expansion of Tr(ρ s ) and Tr(ρ s R), following a similar computation in [12].

The density matrix of local F 0
Since the density matrix of ABJ Fermi gas (2.4) depends on k explicitly, it is not so obvious how can we compute the WKB expansion of (twisted) spectral traces. Also, the expression of density matrix in (2.4) makes sense only for integer M . In [33] an analytic continuation of ρ to arbitrary M is found using the relation to the mirror curve of local F 0 (note that F 0 = P 1 × P 1 ). It turns out that we can systematically compute the small k expansion with fixed b = M/k.
Let us briefly recall the construction in [33]. The density matrix of ABJ theory is identified as where O F 0 is the quantized mirror curve of local F 0 with The Planck constant is related to the Chern-Simons level k by = 2πk, (3.11) and the mass parameter m in (3.9) is related to the parameters in the ABJ theory as As shown in [33], the inverse operator of O F 0 can be written in terms of the quantum dilogarithm Φ(x) defined by (3.14) From (3.13), the quantum dilogarithm exhibits the following quasi-periodicity (3.20) Finally, after redefining the variables we arrive at the following expression of the density matrix ρ = O −1 F 0 (up to an overall constant and similarity transformation) We are interested in the small expansion of ρ. In this case the denominator of (3.13) can be ignored, since it is non-perturbative in . Then taking the log of (3.13) we find 6 Plugging this into (3.22) and expanding around = 0, the density matrix is rewritten as where E n (b) is the Euler polynomial defined by since E 2n (0) = 0 (n ≥ 1). Now we are ready to perform the WKB expansion (with b fixed) of the spectral trace Tr(ρ s ) and the twisted spectral trace Tr(ρ s R).

WKB expansion of spectral trace Tr ρ s
Let us first consider the WKB expansion of the spectral trace Tr(ρ s ). As discussed in [12], this is easily done by using the Wigner transform ρ W of the density matrix ρ. In general, the Wigner transform A W of the operator A is defined by Using the property of Wigner transformation we find that the Wigner transform of the density matrix ρ (3.24) is given by (3.29) From (3.28), the Wigner transform of the th power of ρ is easily obtained by the star- The WKB expansion of (ρ ) W can be computed recursively from the obvious relation (ρ ) W = ρ W (ρ −1 ) W . Finally, the WKB expansion of trace Tr(ρ ) can be found by integrating (ρ ) W on the classical phase space As discussed in [32], the WKB expansion of the spectral trace Tr(ρ s ) takes the following form (3.32) The leading term is simply given by The correction terms D n (s) can be systematically computed by making an ansatz that D n (s) is a rational function of s and fixing the coefficients in the ansatz by matching the values at integer s [12,32]. In this way, we have computed D n (s) up to n = 10. The first three terms are (3.34) Then, the WKB expansion of grand potential can be found by acting the differential operator on the leading term where the leading term J 0 (µ, k, M ) is given by The perturbative part comes from the pole at s = 0 To find the corrections to the perturbative grand potential, it is sufficient to expand D n (s) up to s 3 , since J pert 0 (µ, k, M ) is a cubic polynomial in µ. In the small s expansion, we find that D n (s) behaves as (3.38) Here B 2n (b) and B 2n = B 2n (0) denote the Bernoulli polynomial and the Bernoulli number, respectively. By acting the differential operator (3.35) on (3.37), we find that the constant C(k) and B(k, M ) in (2.13) are correctly reproduced. For instance, the µ-linear term in J pert becomes which correctly reproduces B(k, M ) in (2.13). Also, we can read off the WKB expansion of constant term A(k, M ) (3.40) The first line of (3.40) agrees with the constant term in ABJM theory [22] A As we will see below, the second line in (3.40) corresponds to the WKB expansion of the free energy F coni (k, M ) of the U (M ) k pure Chern-Simons theory. To see this, let us consider the WKB expansion of F coni (k, M ). In terms of = 2πk (3.11), F coni (k, M ) in (2.17) is rewritten as By expanding the integrand in and making use of the identities one can see that the small expansion of F coni (k, M ) indeed agrees with the second line of (3.40). To summarize, the perturbative grand potential of ABJ theory with the coefficients C(k), B(k, M ) in (2.13) and A(k, M ) in (2.14) are correctly reproduced from the WKB expansion of the spectral trace Tr(ρ s ).

WKB expansion of twisted spectral trace Tr(ρ s R)
In a similar manner as in the previous subsection 3.2, we can compute the WKB expansion of the twisted spectral trace Tr(ρ s R). This can be done using the fact that Tr(AR) for some operator A is easily obtained from its Wigner transform A W by simply setting X = P = 0 [12] Tr We find that the WKB expansion of Tr(ρ s R) is written as The leading term is given by We have computed the correction terms D R n (s) up to n = 12. The first three terms are given by (3.48) Again, the large µ expansion of J R (µ, k, M ) is found by acting differential operators on the leading term where the leading term J R 0 (µ, k, M ) is given by Perturbative part. Let us first consider the perturbative part of J R (µ, k, M ), which comes from the pole at s = 0. The leading term is give by This is consistent with the ±1/4 shift of B ± (k, M ) in (2.37). As in the previous subsection 3.2, the contribution F non-ori in (2.38) is obtained by summing all order corrections in .
Since (3.51) is a linear function in µ, in order to compute the term F non-ori it is sufficient to expand D R n (s) up to the linear order in s. We find that D R n (s) behaves as As we will see below, this indeed reproduces the WKB expansion of F non-ori in (2.40). By integration by parts and rescaling of integral variable, (2.40) is rewritten as .
and using the formula the small expansion of F non-ori with fixed b is found to be We can compare this with the WKB expansion of twisted spectral trace in (3.52). By acting the differential operator D n (∂ µ ) (3.52) on the leading term J R,pert 0 in (3.51), one can easily show that the constant term of the perturbative part of 1 2 J R agrees with the WKB expansion of F non-ori in (3.56) This correctly reproduces the shift in A ± (k, M ) in (2.38).
Instanton corrections. J R (µ, k, M ) can be written as a sum of the perturbative part and instanton corrections By picking up the poles at negative integers s = − ( = 1, 2, · · · ) in (3.50), we can compute the non-perturbative correction J R,np (µ, k, M ), which we identify as the "half-instanton" corrections in (2.45). From (3.49) and (3.50), one can show that the instanton corrections are given by (3.59) By matching the WKB expansion, we find the first few instanton coefficients r ( , b) in closed forms (3.60) Here we have introduced the notation θ b and θ 0 by When k and M are both integers, r ( , b) becomes simpler and we can easily guess the instanton coefficients r ( , b) for all order in . From (3.61) and the periodicity of trigonometric functions, the result depends on the value of k modulo 8. For odd M we find while for even M we find [11]. Also, for all cases in Appendix B with various integral k and M , the difference between the modified grand potentials J + and J − agrees with the results in (3.62) and (3.63) 4 Exact grand partition functions for k = 2 In this section, we will write down the exact grand partition functions of orientifold ABJ theory Ξ ± (µ, k, M ) for k = 2 with M = 0, 1. In general, in order to find the exact grand partition function, first we need to find the closed form expression of the modified grand potential. Then, by performing the periodic sum (2.5), we can construct the grand partition function. It turns out that for k = 2 the modified grand potential is determined by the genus-zero and genus-one free energies of topological string, and the exact grand partition function is written in terms of Jacobi theta functions, as in [4,5]. The genus-zero free energy of ABJ(M) theory is encoded in the classical periods of local P 1 × P 1 , which we review first in the next subsection. Then we proceed to write down the exact grand partition functions of orientifold ABJ theory for k = 2.

Periods of diagonal local
In this subsection, we summarize the known results of classical periods of the spectral curve of ABJM matrix model, or the mirror curve of diagonal local P 1 × P 1 [5,34]. The spectral curve of ABJM matrix model has genus one, hence there are two independent periods: A-period and B-period. These two periods are characterized by the Picard-Fuchs equation To study the small z expansion and the large z expansion of periods systematically, it is useful to write the solution of (4.1) in a Mellin-Barnes type integral representation. By plugging the ansatz into (4.1), we find that f (s) should satisfy One can easily find two independent solutions: .
In the large radius frame (small z region), these two solutions correspond to the flat coordinate t (A-period) and the derivative of the genus-zero free energy (B-period) where the integration contour γ is taken to be parallel to the imaginary s-axis with −1/2 < (s) < 0. The small z expansion of those periods are easily obtained by picking up the poles at s = 0, 1, 2, · · · , Note that the constant π 2 /6 in the B-period comes from the pole at s = 0 w a and w b in (4.7) are given by where ψ(n) denotes the digamma function. The A-period and the B-period in (4.7) can be written in closed forms in terms of a hypergeometric function and a Meijer G-function, respectively (4.10) From these relations, one can find the genus-zero free energy F 0 (t) as a function of t. The integration constant in F 0 (t) is found to be −2ζ (3), and the large t expansion of F 0 (t) is It turns out that the logarithmic derivatives of t and ∂ t F 0 with respect to z are simpler than t and ∂ t F 0 themselves, and they are given by the complete elliptic integral of the first Taking the ratio of them, we find the modulus of the spectral curve Here the subscript L stands for the "large radius frame". On the other hand, by deforming the contour γ of (4.6) in the direction (s) < 0 and picking up the poles at s = −(n + 1/2), (n = 0, 1, 2, · · · ), we find the small κ expansion of the periods, where κ is defined by (4.14) Then the two periods in (4.6) become By performing the sum of residues in (4.15), we find that the flat coordinate λ and the genus-zero free energy F 0 (λ) in the "orbifold frame" are given by (4.16) 7 One could absorb the constant π 2 /6 into the definition of F0(t) However, we will not do so and we will stick to the convention in [5]. Note that the t-linear term in F0(t) drops out in the combination of f0 in (4.26). On the other hand, this constant π 2 /6 plays an important role in ξL (4.41) and the modular property of the grand partition function. 8 We follow the convention of elliptic integral in Mathematica The small λ expansion of the free energy F 0 (λ) reads In this case, the ordinary derivatives of λ and ∂ λ F 0 with respect to κ become the complete elliptic integral 18) and the torus modulus in the orbifold frame is given by As we will see below, the difference of the derivatives between (4.12) and (4.18) plays an important role in the modular covariance of the grand partition functions. Since (4.6) and (4.15) are just the different expansions of the same integrals, they are related by 9 In other words, the role of A-cycle and B-cycle are essentially exchanged when going from the large radius frame to the orbifold frame 10 . In fact, τ L (4.13) and τ orb (4.19) are related by the S-transformation Finally, the genus-zero free energy of the large radius frame F 0 and the orbifold frame F 0 are related by 4.2 The case of (k, M ) = (2, 0) In this subsection, we write down the closed form expression of the grand partition functions Ξ ± (µ, 2, 0) of orientifold ABJ theory. As in [4,5], we find that Ξ ± (µ, 2, 0) are written in terms of the Jacobi theta functions. Also, they have a nice property under modular transformation, which enables us to find the expansion around the orbifold point κ → 0 and extract the canonical partition function at fixed N .
As observed in [34], this shifted 't Hooft parameter λ naturally appears in the large λ expansion of the free energy. However, we will not use λ in this paper. 10 We should emphasize that the s-plane in the Mellin-Barnes representation is an auxiliary object, and the contour γ on the s-plane has nothing to do with the A-cycle and B-cycle on the mirror curve of local Let us recall the grand partition function Ξ(µ, 2, 0) of ABJM theory [4] before projection to R = ±1. In [4], it was found that the modified grand potential is essentially determined by the genus-zero and genus-one data of the (refined) topological string on local P 1 × P 1 .
The large µ expansion of the modified grand potential is given by [15] J(µ, where the first line corresponds to the perturbative part, while the remaining terms are instanton corrections. As observed in [4], the terms with and without 1/π 2 coefficient, which we call f 0 and f 1 , respectively, have different origins. Accordingly, it is natural to decompose the modified grand potential as (4.25) It turns out that f 0 can be written in a closed form in terms of the genus-zero free energy F 0 (t) On the other hand, f 1 is given by the sum of the genus-one free energies of the standard topological string and the refined topological string in the Nekrasov-Shatashvili limit where the first term comes from the perturbative grand potential 11 . Explicit forms of F 1 and F NS 1 are given by [35][36][37] (4.28) 11 Our definition of f1 includes the perturbative term µ/4, which is different from the definition of f1 in [5]. and the sum of them becomes Note that −z∂ z t is given by the complete elliptic integral (4.12). One can easily check that (4.26) and (4.29) correctly reproduce the large µ expansion (4.25).
Modular expression of genus-one part. The genus-one free energies F 1 and F NS 1 have a nice expression as modular forms. To see this, it is convenient to introduce τ by Using the formula in Appendix A, various quantities appearing in the genus-one free energies (4.28) can be written in terms of τ Plugging (4.31) into (4.28), and using the expression of Dedekind eta function η(τ ) in (A.10), we recover the well known expression of the genus-one free energy of the standard topological string In a similar manner, one can also show that 12 For later purposes, it is also convenient to rewrite the genus-one free energy in terms of the original variable τ L . We find The first term µ/4 in f 1 (4.27) can also be written in terms of τ L using (4.31) From (4.35) and (4.34), we find that f 1 is written as 12 (4.29) and (4.32) imply This agrees with the result in [37] (note that G in [37] is equal to our −F NS 1 ).
Exact grand partition function. Once we find the modified grand potential, we can construct the grand partition function by summing over the periodic shift µ → µ + 2πin (2.5). As noticed in [4], using the relation the n 3 term coming from the cubic term µ 3 3π 2 in J pert (µ, 2, 0) can be rewritten as a liner term in n Then, the summand in (2.5) is written as 39) and the periodic sum (2.5) becomes the Jacobi theta function where τ L is defined in (4.13) and ξ L is given by 13 Note that the constant term −1/24 in (4.41) comes from the 2πin-shift of µ/4 in J pert (µ, 2, 0) and (4.38) 1 2 Using the expression of f 1 in (4.36), Ξ(µ, 2, 0) can also be written as (4.43) As we will see below, the constant −1/24 in (4.41) is necessary for the grand partition function to behave properly under the modular transformation. In fact, ξ L can be written as and the last two terms are exactly the B-period in (4.6). One can rewrite ξ L in a form that the modular property is more transparent. As in [38], by introducing the appropriately normalized A-period Π A and B-period Π B ξ L in (4.44) is written as Also, one can show that f 0 can be written in terms of Π A and Π B in (4.45) as After integration by parts, we find a surprisingly simple expression for f 0 In a similar manner, we can also rewrite ξ L as The integration constants in (4.49) and (4.50) should be fixed so that the large t expansions of f 0 and ξ L are correctly reproduced (4.51) To understand the modular property of grand partition function better, it is desirable to write f 0 and ξ L explicitly as modular forms. We find that the derivative of A-period with respect to τ L can be written in a closed form but we could not find a closed form expression of Π A itself. We leave this as an interesting future problem.
S-transformation and orbifold expansion. As discussed in [4,5], in order to find the small κ expansion (orbifold expansion) of the grand partition function, we have to perform the modular S-transformation. Indeed, the combination e f 0 +f 1 ϑ 3 (2ξ L , 2τ L ) is exactly the form of "non-perturbative partition function" proposed in [38] which has a nice modular property. Using the S-transformation of theta function in (A.3), we find where ξ orb and τ orb are given by and f 0 and f 1 in (4.53) are given by One can easily show that ξ orb is written as where the two periods in the orbifold frame are given by Also, one can show that f 0 and ξ orb can be recast in the same form as f 0 in (4.49) and ξ L in (4.50), as expected and the derivative of Π B with respect to τ orb is given by the S-transformation of (4.52) In terms of the orbifold free energy F 0 (λ), f 0 and ξ orb are written as For the genus-one part, we find Note that ∂ κ λ is given by the complete elliptic integral (4.18). Interestingly, the difference of the logarithmic derivative −z∂ z t in (4.29) and the ordinary derivative ∂ κ λ in (4.61) is compensated by the perturbative term µ/4 and the term − 1 2 log(−2iτ L ) coming from the S-transformation of theta function in (4.55). Also, it is interesting to observe that f 1 in (4.61) has a similar form as F 1 + F NS 1 in (4.29) and (4.34). Using the expression of f 1 as a theta function (4.61), the grand partition function can also be written as The small κ expansion of the grand partition function can be found from the expansion of ξ orb , τ orb and f 0 Putting all together, finally we find Ξ(µ, 2, 0) = 1 + κ 8 + κ 2 32π 2 + 10 − π 2 512π 2 κ 3 + · · · . (4.64) The coefficient of κ N (N = 1, 2, · · · ) correctly reproduces the canonical partition function Z(N, 2, 0) of ABJM theory computed in [15].
In this subsection, we consider the closed form expression of Ξ ± (µ, 2, 0). For even k, it is convenient to redefine the constants in the perturbative part of grand potential as we find the modified grand potentials in closed forms The first term ξ L + 1/24 in (4.69) comes from the periodic shift of f 0 /2, while the second term 1/3 and the third term B ± (2, 0) come from (4.68) and the B ± µ term in J pert ± , respectively. Finally, we find the grand partition function in a closed form which can also be written as We note in passing that all terms in J ± (µ, 2, 0) (4.67), except for the genus-zero term f 0 /2, can be written in terms of Jacobi theta functions, but the resulting expression is not so illuminating. This is true for all other cases considered in this paper. Using the identity (A.11) of theta functions in Appendix A, the product of Ξ ± (µ, 2, 0) becomes Ξ + (µ, 2, 0)Ξ − (µ, 2, 0) = e J + (µ,2,0)+J − (µ,2,0) ϑ 4 (0, 2τ L )ϑ 3 (2ξ L , 2τ L ). Orbifold expansion. The small κ expansion of Ξ ± (µ, 2, 0) can be found by performing the S-transformation of theta function, as in the previous subsection 4.2.1. This is easily done by noticing that Ξ ± (µ, 2, 0) in (4.70) can be written as a theta function with characteristic (a, b) = (0, ∓ 1 4 ) 14 where J ± (µ, 2, 0) is given by (4.76) Note that the 1 2 log 2 term in (4.76) appears from our definition of f 1 in (4.55) One can easily show that the theta function with characteristic (a, b) = (± 1 4 , 0) in (4.75) can also be written in terms of the Jacobi theta functions as (4.78) The − 1 8 log κ term in (4.76) is precisely canceled by the factor e πiτ orb /16 coming from the theta function, and hence the fractional power of κ does not appear in the expansion of Ξ ± (µ, 2, 0), as expected. Finally, the small κ expansion of Ξ ± (µ, 2, 0) is found to be For even k with integral M , the effective chemical potential is given by [18,19] In the case of (k, M ) = (2, 1), this sign is ε = +1 and the modulus in the large radius frame is identified with z = −e −2µ . In this case, 2µ eff is related to the A-period t in (4.10) by the shift but we have to subtract the imaginary part of − log z = 2µ + πi in (4.10). Therefore, 2µ eff and the A-period t are related by 2µ eff = t − πi. Here and in what follows, we put prime to indicate the shift (4.82), except for the subtlety in the genus-zero part as we will mention below. Again, the modified grand potential can be written in a closed form. For the perturbative part, we find where A(k, M ) is defined in (4.65). From the results in [18,19], the modified grand potential is given by where µ = µ + πi/2. The genus-zero part f 0 is obtained from (4.49) by replacing where Π A = −t /2. Thus we find 15 where ξ L and τ L are given by On the other hand, the genus-one part f 1 is literally related to f 1 (4.27) by the shift µ → µ + πi/2 By performing the periodic sum (2.5), the grand partition function becomes a theta function Note that the first three terms correspond to the three terms in f 0 (4.87), and the last term −1/3 comes from (4.38). The τ -parameter in the theta function ϑ 3 (v, τ ) is given by where the additional constant +2 comes from the second term −iπξ L in f 0 (4.87). Then the grand partition function becomes Ξ(µ, 2, 1) = e J(µ,2,1) ϑ 3 2ξ L − τ L + 1 2 , 2τ L = e J(µ,2,1) ϑ 4 2ξ L − τ L 2 , 2τ L . (4.92) As we will see below, the combination ξ L − τ L /4 appearing in (4.92) behave nicely under the S-transformation when combined with f 0 in (4.87). The small κ expansion can be found by performing the S-transformation with Again, the small κ expansion of Ξ(µ, 2, 1) does not contain the fractional power of κ, since the − 1 4 log(iκ) term in (4.96) is precisely canceled by the factor e πiτ orb /8 coming from the theta function ϑ 2 . Finally, the small κ expansion of Ξ(µ, 2, 1) is found to be Ξ(µ, 2, 1) = 1 + κ 4π + π 2 − 8 128π 2 κ 2 + 5π 2 − 48 4608π 3 κ 3 + · · · · · · , (4.97) which correctly reproduces the results of canonical partition function Z(N, 2, 1) of ABJ theory computed in [18,19].

Ξ ± (µ, 4, 0)
In this subsection, we will write down the closed form expression of Ξ ± (µ, 4, 0). In this case the sign in (4.81) is ε = +1, and the effective chemical potential is 2µ eff = t − πi as in the case of (k, M ) = (2, 1). The constants in the perturbative part are given by From the results (B.3) in Appendix B, the modified grand potential can be found in a closed form By performing the periodic sum, the grand partition function becomes the Jacobi theta function ϑ 3 (v, τ ) with the τ -parameter The constant 1/2 comes from the second term of f 0 in (4.87). Using (4.37), the n 3 term coming from the perturbative part µ 3 12π 2 is rewritten as a linear term in n Then, the parameter v in the theta function is found to be and we arrive at the following form of the exact grand partition functions From the property of theta functions in Appendix A, they can be also written as (5.9) Then using the identity in (A.11), the product of Ξ ± (µ, 4, 0) becomes which is indeed proportional to Ξ(µ, 4, 0) in (5.1). The difference of modified grand potential J + + J − and J(µ, 4, 0) comes from the factor ϑ 3 (0, τ L ) (5.11) From the results in (B.3) and [15,39], one can check that this indeed reproduces the difference between J + + J − and J.
Orbifold expansion. The small κ expansion of Ξ ± (µ, 4, 0) can be found, again, by performing the S-transformation. However, before doing the S-transformation, we have to rewrite the theta function ϑ a (v, τ ) in such a way that the τ parameter is proportional to τ L . Using the identity (A.12) in Appendix A, we find Now we are ready to perform the S-transformation. We get where J ± (µ, 4, 0) is given by (5.14) One can easily find the small κ expansion of (5.13) which correctly reproduces the exact computation of Z ± (N, 4, 0) in section 2.2.

Ξ ± (µ, 4, 2)
Next we consider Ξ ± (µ, 4, 2). In this case, the sign in (4.81) is ε = +1, hence the expression of modified grand potential is similar to the (k, M ) = (4, 0) case in the previous subsection 5.1. The constants in the perturbative part are given by From the results (B.5) in Appendix B, the modified grand potential can be found in a closed form By performing the periodic sum (2.5), the grand partition function becomes the Jacobi theta function ϑ 3 (v, ( Then, we arrive at the following closed form expression of the grand partition functions which can be also written as (5.20) Using the identity (A.11), the product of Ξ ± (µ, 4, 2) becomes which should be equal to Ξ(µ, 4, 2) in (5.1). Again, the difference between J + + J − and J(µ, 4, 2) comes from the factor ϑ 3 (0, τ L ) Orbifold expansion. The small κ expansion of Ξ ± (µ, 4, 2) can be found by rewriting them in a similar manner as in the previous subsection 5.1: (5.23) Then, performing the S-transformation, we find where J ± (µ, 4, 2) is given by The small κ expansion of Ξ ± (µ, 4, 2) in (5.24) reads which correctly reproduces the exact computation of Z ± (N, 4, 2) in section 2.2.

Ξ ± (µ, 4, 1)
Here we consider Ξ ± (µ, 4, 1). This case is similar to the (k, M ) = (2, 0) case in the previous section 4 with sign ε = −1. The constants in the perturbative part are given by From the results (B.4) in Appendix B, the modified grand potential can be found in a closed form J ± (µ, 4, 1) = A ± (4, 1) One can easily show that the grand partition function is proportional to Finally, we find the following form of the exact grand partition functions Again, the product of Ξ ± (µ, 4, 1) can be simplified by using (A.11) in Appendix A as Orbifold expansion. The small κ expansion of Ξ ± (µ, 4, 1) can be easily found by rewriting (5.30) in terms of the theta function with characteristic Ξ ± (µ, 4, 1) = e J ± (µ,4,1) ϑ 0 where J ± (µ, 4, 1) are given by J ± (µ, 4, 1) = A ± (4, 1) + 1 4 Note that log 2 term comes from The small κ expansion of Ξ ± (µ, 4, 1) is easily found from (5.33)  For k = 8, up to Seiberg-like duality (2.21) the independent grand partition functions are Ξ ± (µ, 8, M ) with M = 0, 1, · · · , 4. We find that the grand partition functions at k = 8 are qualitatively different from the k = 2, 4 cases studied in the previous sections 4 and 5. In the case of k = 8, the worldsheet instanton part of the modified grand potential receives all order corrections in the genus expansion. In general, it is hopeless to sum over all order corrections, but somewhat miraculously for the k = 8 case the all-order corrections in the genus expansion can be resummed in a closed form.
Let us first consider Ξ ± (µ, 8,1). The constants in the perturbative part are given by From the results in (B.6), we find the modified grand potential in a closed form Here j ± (µ, 8, 1) represents the perturbative part and the membrane instanton corrections On the other hand, log G o (µ) term in (6.2) represents the worldsheet instanton corrections. Note that the worldsheet 1-instanton factor for k = 8 is By looking at the worldsheet instanton coefficients in (B.6) for the first few instanton numbers, one can easily guess the all order resummation of such worldsheet instanton corrections In the periodic sum (2.5), we find that the summand can be written as where v is given by Above, the second term −1/6 comes from the 2πin-shift of perturbative part µ 3 (6.9) we have to consider the periodic sum (2.5) with even n and odd n separately. Finally, we find the closed from expression of Ξ ± (µ, 8, 1) (6.10) Let us consider the small κ expansion of Ξ ± (µ, 8, 1). By performing the S-transformation, we find Now it is straightforward to find the small κ expansion of (6.11) A few comments are in order here: 1. Ξ ± (µ, 8, 1) is a sum of two theta functions multiplied by some factor G ±1 o , and Ξ ± (µ, 8, 1) cannot be simply written as a single theta function. This is qualitatively different from the expression of the grand partition functions at k = 2, 4. This also implies that the zeros of the grand partition function Ξ ± (µ, 8, 1) = 0, which determines the exact quantization condition for the spectrum of the kernel ρ ± , cannot be simply expressed as the zeros of the Jacobi theta function ϑ 3 (v, τ ) as in the case of k = 2, 4 studied in [5]. As we will see below, this is also the case for (k, M ) = (8, 3), and we believe this is a general phenomenon for k = 1, 2, 4. but it is difficult to rewrite this product in a simple form and extract the modified grand potential J(µ, 8, 1) in a closed form. This suggests that for general k, Ξ ± (µ, k, M ) and J ± (µ, k, M ) are more simpler than Ξ(µ, k, M ) and J(µ, k, M ), and we should regard Ξ ± (µ, k, M ) as the fundamental building block.
4. This factor G o (µ) can be written as a modular form where G o (µ) is the same function (6.5) appeared in J ± (µ, 8,1), and j ± (µ, 8, 3) is given by Under the shift µ → µ + 2πin, the summand in the periodic sum in (2.5) becomes where v is given by Finally, by performing the periodic sum (2.5) for even n and odd n separately, we arrive at the closed form expression of grand partition functions The small κ expansion can be found by doing the S-transformation (6.24) and the small κ expansion of Ξ ± (µ, 8, 3) reads (6.25) One can show that this agrees with the computation of Z ± (N, 8, 3) in section 2.2.

Ξ ± (µ, 8, 0)
Then let us move on to the even M case. In this case, the sign ε in (4.81) is (−1) k/2−M = 1 and the natural variable is t in (4.83).
In this subsection we consider the (k, M ) = (8, 0) case. From the results in (B.9), we find the closed form of modified grand potential J ± (8, 0) = j ± (8, 0) + log G e (µ), (6.26) where log G e (µ) denotes the worldsheet instanton corrections log G e (µ) = 1 2 arcsinh(2e − µ 2 ), (6.27) and j ± (8, 0) is given by (6.28) From this exact form of modified grand potential, we can construct the exact grand partition function by performing the periodic sum (2.5). As in the case of (k, M ) = (8, 1), (8,3) in the previous subsections, the worldsheet instanton factor G e (µ) in (6.27) is not invariant under the 2πi-shift of µ but invariant under the 4πi-shift. Therefore, we have to devide the summation into two parts. In a similar manner as in the previous subsections, we arrive at the exact form of Ξ ± (8, 0) (6.29) The small κ expansion can be found by performing the S-transformation where and (6.32) Finally we find where G e (µ) is the same function in (6.27) appeared in the (k, M ) = (8, 0) case, and j ± (8, 4) is given by By performing the periodic sum (2.5), we find the exact form of grand partition function as Again, the small κ expansion can be found by the S-transformation where G e is defined in (6.31) and j ± (8, 4) is given by Ξ − (8, 4) = 1 + 1 sin 2 π 8 sin π 16 sin 3π (6.39) Let us consider Ξ ± (8, 2). From the result in (B.10), we find the closed form of modified grand potential Interestingly, in this case the modified grand potential J ± (8, 2) is invariant under the 2πishift of µ, and the periodic sum is written as a single theta function Using the formula of theta function in Appendix A, the product of Ξ + (8, 2) and Ξ − (8, 2) can be rewritten as One can show that this is equal to Ξ − (4, 2) in (5.20), as conjectured in [5] Ξ(8, 2) = Ξ − (4, 2) = e J − (4,2) ϑ 4 The small κ expansion of Ξ ± (8, 2) can be found by performing the S-transformation where J ± (8, 2) is given by .
Finally, we find (6.46) 6.6 Comment on the k = 6 case For k = 3, 6, 12, we find the instanton expansion of grand potential from the exact values of Z ± (N, k, M ), which is summarized in Appendix B. For these cases, however, it is difficult to find the closed form expression of the modified grand potential. In particular, it is difficult to resum the series of worldsheet instanton corrections, since these corrections come from all genera in the string coupling expansion with g s = 2/k. However, for k = 6, we find that the membrane instanton corrections of the form e −2nµ (n = 1, 2, · · · ) can be resummed in a closed form. The terms with 1/π 2 coefficient is given by the genus-zero free energy, as in the case of k = 2, 4 We find that the terms without 1/π 2 coefficient can be also written in closed forms. For even M we find while for odd M we find Suppose we have managed to sum over the worldsheet instanton corrections to the grand potential, which we denote as log W (µ). For definiteness, let us consider the (k, M ) = (6, 0) case. In this case, the perturbative coefficients are given by A ± (6, 0) = − 1 3 log 3 ± 1 2 log 2, B ± (6, 0) = 11 72 ± 1 4 (6.50) and the modified grand potential can be written as J ± (µ, 6, 0) = j ± (6, 0) + log W (µ), (6.51) where j ± (6, 0) includes the perturbative part and the membrane instanton corrections From the results in (B.14), log W (µ) has the following expansion On the other hand, the worldsheet instanton corrections (and bound states with membrane instantons) of ABJM theory at k = 6 is [15] J WS (µ, 6, 0) = 4 3 e − 2µ 3 − 2e − 4µ 3 + 88 9 e − 8µ 3 + 108 5 e − 10µ 3 + 25208 189 e − 14µ 3 + · · · , (6.54) which is different from 2 log W (µ) J WS (µ, 6, 0) = 2 log W (µ). (6.55) This suggests that the worldsheet instanton corrections in the orientifold ABJ theory receive additional contributions from the orientifolding. From (6.51), we can consider the possible form of the exact grand partition function Ξ ± (µ, 6, 0) by performing the periodic sum (2.5). Since W (µ) is not invariant under the 2πi-shift of µ but invariant under the 6πi-shift, we have to decompose the periodic sum (2.5) into three parts according to n modulo 3. Then, we find Ξ ± (µ, 6, 0) = e j ± (µ,6,0) This is similar to the structure of Ξ ± (µ, 8, M = 0, 1, 3, 4) in the previous subsections. Also, we speculate that the factor W (µ) can be written as a modular form, as in the case of G(µ) for k = 8. For general even k, the worldsheet 1-instanton factor is given by and we expect that W (µ) is invariant under the modular transformation S and T k : τ L → τ L + k. It would be interesting to find the closed form expression of the function W (µ). Also, we have detailed data of instanton coefficients for k = 3 (see Appendix B.4). It would be nice to find the closed form expression of the modified grand potential for k = 3.

Functional relations
In [5], it was conjectured that the grand partition functions of orientifold ABJ theory satisfy certain functional relations. These relations involve grand partition functions at fixed k but M differing by ±1. The first relation is Here we suppressed the µ-dependence of grand partition functions and introduced the notation f ± for a function of µ The second set of relations is Following [5], we will call these relations "quantum Wronskian relations". Using our exact data of Z ± (N, k, M ), for k = 2, 3, 4, 6, we have checked that these relations are indeed satisfied in the small κ expansion. We first notice that the perturbative part of grand potential J ± (k, M ) is consistent with those relations. For the first relation (7.1), the perturbative part J ±,pert (k, 1) appearing in the left hand side is One can show that Next consider the quantum Wronskian relations (7.3) and (7.4), at the level of perturbative part. We find that the constant term satisfies 9) and the perturbative grand potential satisfies − log 2 cos πM 2k . (7.11) We would like to show that not only the perturbative part, but also the exact grand partition functions including the instanton corrections, indeed satisfy the functional relations. In [5], it is found that the the exact grand partition function at k = 1 satisfies the relation (7.1). In this section, we will prove the functional relations (7.1), (7.3), and (7.4) for k = 2, 4 using the exact from of grand partition functions obtained in the previous sections.

Proof of functional relations for k = 2
In this subsection we will show the functional relations (7.1), (7.3), and (7.4) for k = 2. Interestingly, they all follow from an identity N 2 (τ ) = 1 proved in Appendix A.1.

Functional relation (7.1) for k = 2
Let us first consider the first relation (7.1) for k = 2. For readers convenience, let us recall the exact expression of grand partition functions appearing in the relation (7.1) 12) and the modified grand potentials are given by (7.13) Note that F 1 = F 1 (t ) and F NS 1 = F NS 1 (t ). First, let us consider the transformation of the modified grand potential J ± (2, 1) under the shift µ → µ ± πi/2. As discussed in [5], the genus-zero and genus-one free energies transforms as (7.14) Using these relations (7.14) we find In the first line of (7.15), the first term on the right hand side comes from the perturbative part (7.7), and the remaining terms comes from (7.14). Note that the linear term in µ in the first line of (7.15) is canceled on the left hand side, hence on the right hand side we subtracted the linear term from ∂ 2 t F 0 so that The origin of the last term in the second line of (7.15) is similar. Also, note that the "half-instanton" contribution J R,np (2, 1) in (3.62) vanishes by adding the ±πi/2 shift in (7.15) J R,np (2, 1) + + J R,np (2, 1) − = 0. (7.18) Next, let us consider the transformation of theta functions under the shift µ → µ±πi/2. Using the relation we find Then, as discussed in [5], using the formula (A.14) one can rewrite the the product of Ξ ± ± (2, 1) appearing on the left hand side of the relation (7.1) as (7.21) One can easily see that the first term in (7.21) is proportional to Ξ(2, 0), and the ϑ 2 terms in (7.21) are canceled by adding the two lines in (7.21) with the proper weight in the functional relation (7.1). Finally, we arrive at the desired functional relation (7.1) (7.22) up to a factor N 2 (τ L ) 16 where q L = e 2πiτ L . Using the expression of f 1 in (4.36), we find that N 2 (τ L ) can be written as One can show that this factor is actually unity N 2 (τ L ) = 1 (see Appendix A.1 for a proof), which implies that the functional relation (7.1) is indeed satisfied for k = 2. In the case of k = 2, M = 1, the second quantum Wronskian relation (7.4) is equivalent to the first quantum Wronskian relation (7.3), therefore it is sufficient to consider (7.3) only. Again, for readers convenience we recall the exact expression of the grand partition functions appearing in (7.25) 26) and the modified grand potentials are given by (7.27) 16 The same factor N2(τL) appeared in the proof of the functional relation (7.1) for k = 1 [5].
As in the previous subsection 7.1.1, one can show that the modified grand potential satisfy (7.28) The first two terms come from the perturbative part and remaining terms come from the transformation of f 0 and F 1 + F NS 1 as in (7.14). Next we consider the transformation of the theta function part. From the relation we find Again, using the formula (A.14) one can show that the functional relation (7.25) holds up to the same factor N 2 (τ L ) in (7.23), with a simple replacement of the argument τ L → τ L in (7.23). From the identity N 2 (τ L ) = 1 proved in Appendix A.1, we find that the functional relation (7.25) is indeed satisfied for k = 2.

Proof of functional relations for k = 4
In this subsection we will show the functional relations (7.1), (7.3), and (7.4) for k = 4. As in the case of k = 2, these relations all follow from a single identity N 4 (τ ) = 1 proved in Appendix A.2.

Comments on other k's
For other values of k's we do not have a general proof of the relations (7.1), (7.3), and (7.4). However, we can check these relations by comparing the small κ expansion on both sides using our data of the exact values of Z ± (N, k, M ) and the known values of the canonical partition functions Z(N, k, M ) of ABJ(M) theories computed in [15,18,19]. We find a complete agreement for k = 3, 6. On the other hand, for k = 8, 12 the exact values of ABJ(M) partition functions Z(N, k, M ) are not known in the literature. Rather, assuming that the functional relations are correct, we can predict the values of Z(N, k, M ) with k = 8, 12 for all 0 ≤ M ≤ k using the functional relations. We have checked that those values of Z(N, k, M ) for k = 8, 12 obtained in this manner are all consistent with the conjectured modified grand potential (2.8) given by the free energy of (refined) topological string on local P 1 × P 1 [2].

Conclusion
In this paper, we have studied the orientifolding of the ABJ Fermi gas by projecting to the even/odd sectors under the reflection of fermion coordinate. Ξ − (µ, k, 0) and Ξ + (µ, k, 1) are indeed related to the N = 5 O(n) × U Sp(n ) theories, and we expect that more general cases Ξ ± (µ, k, M ) are also related to M-theory on some orientifold background. It would be interesting to understand the spacetime picture (if any) for the general cases Ξ ± (µ, k, M ) and study the precise relation between the reflection of fermion coordinate and the orientifolding of spacetime. The relation to the orientifold is also suggested by the appearance of non-orientable contribution F non-ori in the constant term A ± (k, M ) (2.38) in the perturbative part of grand potential. The grand potential of orientifold ABJ theory receives three types of instanton corrections (2.45): worldsheet instantons, membrane instantons, and "half-instantons". We find that the half-instanton corrections come from the twisted spectral trace Tr(ρ s R), and we determined the first few half-instanton coefficients in closed forms (3.60) as functions of k and b = M/k. We also observed that the coefficients of worldsheet instanton corrections are different from the worldsheet instanton coefficients of ABJ theory before orientifolding. It would be interesting to understand the structure of worldsheet instantons in orientifold ABJ theory better, and see if they have some relation to the "real topological string" on the local P 1 × P 1 [40,41].
For k = 2, 4, 8 we have found the closed form expression of the grand partition function of orientifold ABJ theory in terms of the theta functions, and we have also checked for k = 2, 4 that the functional relations conjectured in [5] are indeed satisfied. We found that the structure of grand partition function at k = 8 is qualitatively different from the cases of k = 2, 4. For k = 2, 4, the modified grand potentials are basically determined by the genus-zero and genus-one free energies, while for the k = 8 case the modified grand potentials receives all genus corrections in the string coupling expansion. Nevertheless, we have managed to find the closed form expression of the all order resummation of worldsheet instanton corrections G o (µ) in (6.5) and G e (µ) in (6.27). Since G o,e (µ) are not invariant under the 2πi-shift, we divided the periodic sum into two parts. In general, the worldsheet 1-instanton factor e −4µ/k is not invariant under the 2πi-shift of µ, except for the k = 1, 2, 4 cases. Therefore, for generic k ∈ Z >0 , the periodic sum (2.5) should be taken separately according to n modulo k (or n modulo k/2 for even k), and the resulting exact grand partition function will be written as a sum of certain building blocks carrying definite Z k -charge (or Z k/2 -charge for even k). It would be very interesting to find a closed form expression of the grand partition functions for general integer k, say k = 3, 6.
As observed in [4,5], the exact forms of the grand partition functions of (orientifold) ABJ theories have the same form as the "non-perturbative partition functions" proposed in [38]. Our exact grand partition functions have a nice modular property, and we expect that they are invariant under some subgroup of SL(2, Z). It would be nice to understand the modular property of Ξ ± (µ, k, M ) better. Interestingly, the exact form of Ξ ± (µ, k, M ) we found is a holomorphic function of µ at fixed k, M . As emphasized in [38], the holomorphic anomaly might be an artifact of the perturbative genus expansion and the holomorphicity and modularity are restored at the non-perturbative level. It would be very interesting to study this mechanism in a concrete example, and we believe that the (orientifold) ABJ theory serves as a good testing ground for this purpose.
The exact grand partition functions of (orientifold) ABJ theories we obtained are reminiscent of the torus partition functions of chiral fermions in 2d CFT. It has long been suspected that the partition functions of matrix models have a natural interpretation in 2d CFT living on spectral curves (see e.g. [42] and references therein). It would be interesting to clarify the precise relation between our exact grand partition functions and 2d CFT.